<!-- Title: Owl shift. Mon May 14, 2001 -->
<!-- SciCo: Greg Veramendi -->
<!-- DAQAce: Greg Veramendi -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: &nbsp; -->
<!-- Notes: -->
<pre></pre>


<!-- Date: Mon May 14 00:58:17 2001 -->
<pre>Inherited run from last shift... 
busy TO from ccal5,6,7,13,14,15 
forced to end run with 2.2M events</pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114638' target=_top>114638</a>) -->
<!-- Comment: Mon May 14 01;17;12 comment by...greg -->
<pre>It appears this problem puts the evb in some bad state..
Cannot start new run,
L3 took 7 min. to coldstart
And Evb would not activate getting errors:
4'6" readout sent events mismatch[EOR]
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb20 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb24 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb14 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb13 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb16 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb15 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb23 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb19 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb18 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb22 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb17 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb12 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb11 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb25 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NoSuchEvent: SCPU b0eb21 claims to have loaded non-existent event 2232651.
(MLE) b0l3gate1.fnal.gov:main:1:06:01 AM-&gt;Host b0eb10.fnal.gov, task tsmDisp
SM-P0-E-NotRunning: Partition has no run in progress.
(RC)   1:10:36 Config -&gt; READY

Then I Reset, and I clean up evb and I get the following
message from csl while at the "Start" position:
13'41" readout sent events mismatch[EOR]b0dau32.fnal.gov:csl_mon_send:1:13:39 AM-&gt;Logger partition 0 cannot log this event. Run 114639 not accepted. Last run = 114639.

And a red screen saying: "Events in this run will NOT be written
to disk!!! END-OF-RUN is STRONGLY RECOMMENDED."


</pre>


<!-- Date: Mon May 14 02:08:50 2001 -->
<pre>Recieved Busy TO from the same ccal crates again... 
And the same errors from the evb when trying to start another 
run... 
</pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114641' target=_top>114641</a>) -->
<!-- Comment: Mon May 14 02;33;01 comment by...greg -->
<pre>cleaning up EVB seems to fix the problem
until of course we get the same error...
</pre>


<!-- Date: Mon May 14 03:25:47 2001 -->
<pre>PHA birds crate 8 wedge 4</pre>
<!-- Author: greg -->
<!-- Comment: Mon May 14 05;11;41 comment by...greg -->
PHA birds crate 11 wedge 15,16
<!-- Comment: Mon May 14 06;56;00 comment by...greg -->
<pre>PHA birds crate 11 wedge 15,16</pre>
<!-- Date: Mon May 14 04:51:48 2001 -->
<pre>Got a Done TO from ccal 01: SMX in slot 3 not responding... 
 
</pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114642' target=_top>114642</a>) -->



<!-- Date: Mon May 14 05:33:09 2001 -->
<pre>I notice that the Number of events is not increasing 
on the RC display, but the L3 farm is still processing 
events.. 
There is another javea.lang.OutOfMemoryError 
But everything seems to continue going regardless... 
 
Also one of the L3 output nodes (b0l3u04) "Ended" 
so three subfarms are no longer processing... 
 
stopping the run to do calibrations.. 
 </pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114642' target=_top>114642</a>) -->


<!-- Date: Mon May 14 06:20:33 2001 -->
<pre>Finished Calibrations: 
 
Calibration	Run Number	DB? 
-----------    	----------	--- 
QIE_NoBeam	114643		 y 
TOF		114645		 n 
LED		114646		 y 
Xenon (Flash)   114647 		 y 
Xenon (XEF)	114648		 y 
COT		114649 		 y 
Muon		114650		N/A 
SMQIE		114651		 y 
TOF (second try)114652	      not yet 
 
</pre>
<!-- Author: greg -->


<!-- Date: Mon May 14 06:26:30 2001 -->
<pre>TOF consumers are taking too long to finish... 
I don't think either of the calibrations will make in  
into the DB... 
 
taking rick run.. 
I got a Done TO from ccal01 slot 3 again.. 
will take about 30k events.. 
</pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114654' target=_top>114654</a>) -->



<!-- Date: Mon May 14 06:58:01 2001 -->
<pre>Going back to taking cosmics.. 
I attempted to get the L2 back into the run.. 
The setup I inherited had the L2 crate in the run.. 
but the slot 3 had been removed.. settings were for Auto-accept 
 
I failed in this attempt.. 
L2de00 fails coldstart 
even after a bus reset and a crate reboot... 
 
going back to old configuration.. 
 
Error Display messages from L2: 
(MLE) b0l2de00:Messenger:6:39:33 AM-&gt;VISIONread/write failed in slot 3 
(MLE) b0l2de00:Messenger:6:39:34 AM-&gt;VISIONread/write failed in slot 3 
(RC)   6:39:25 Config -&gt; ERROR 
(MLE) b0l2de00:Messenger:6:39:34 AM-&gt;Setup failed with error code 5 
(MLE) b0l2de00:Messenger:6:39:34 AM-&gt;Setup failed with error code 1 
(MLE) b0l2de00:Messenger:6:39:34 AM-&gt;Statechange failed 
</pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114656' target=_top>114656</a>) -->


<!-- Date: Mon May 14 07:11:31 2001 -->
<pre>Again b0l3u04 is VERY slow in sending events through... 
Looking at it's error messages I see: 
 
in L3_csl_socket_connect().. 
info: connect()-&gt;error 
perror: No route to host 
 
and  
 
in l3_drain_output_put()... 
info: Error opening drain output 
 
going to clean up l3 to see if that fixes this output node.. 
 
</pre>
<!-- Author: greg :: (run <a href='/cgi-bin/runSummaryRun.cgi?114657' target=_top>114657</a>) -->


<!-- Date: Mon May 14 07:47:51 2001 -->
<pre>Rack alarms 
all 8 silicon crates tripped 
quad log trip.. 
</pre>
<!-- Author: greg -->


<!-- Date: Mon May 14 08:04:43 2001 -->
<font color=red>Shift Summary: </font><pre>It's another day in the neighborhood 
Won't you be mine??</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   a bunch
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             260
   
</td></tr></table></pre>
<!-- Author: greg -->