<!-- Title: Owl shift. Tue Mar 20, 2001 -->
<!-- SciCo: Alan Sill -->
<!-- DAQAce: Dave Ambrose  -->
<!-- MonAce: Francesco Palmonari -->
<!-- CO: Un-Ki Yang  -->
<!-- OpManager: Kevin Burkett -->
<!-- Notes: -->
<pre>Continue running revised torture test until 7 am, if possible -- 
supervised access scheduled to begin then.</pre>


<!-- Date: Tue Mar 20 00:25:46 2001 -->
<pre>Right at shift change, got a water system alarm (harbor-style 
double boop) on collision hall relative humidity.  Techs were 
aware of it, and had changed a filter 15 minutes ago to try to 
take care of it. 
 
Looking at it more closely, the dew point was below the lower 
limit of 34 degrees F, which would imply that the humidity was 
in fact too low in the CH (so changing a water system filter 
doesn't sound like the right fix to me).  The alarm has been 
silenced and the cryo techs claim this can wait until tomorrow 
morning for further diagnosis.  If in fact the RH is too low  
rather than too high, this sounds like a correct assessment...</pre>
<!-- Author: AFS -->


<!-- Date: Tue Mar 20 00:56:43 2001 -->
<pre>Continuing with run 109657, run config ACE_CALORIMETER_MINIMAL: 
 
--> TriggerType: INTEGRATION_SUMET_2[2,78] 
--> HardEvb, subfarms 0-14 
--> calorimeters:  CCAL (all except 03,05; removed on eve shift) 
                   PCAL (all, 00-11) 
                   WCAL (all, 00-07) 
--> COT: all (00-19) 
--> XFT: all (00-06) 
--> triggers, scalers:  LEVEL1_CAL 00-05 
                        LEVEL1_GLOBAL 
                        LEVEL2_CAL 00-05 
                        TRIGGER_SCALERS 
--> current rates: L1 bit 0 (SUM_ET) 174.8 Hz 
                   TS rate 62 Hz 
                   </pre>
<!-- Author: dave :: (run 109657) -->
<!-- Comment: Tue Mar 20 02;20;38 comment by...dave -->
<pre>noticed that L3 output node u04 has frozen in input state, with
267717 events, along with processors 099-122.  Also, converter
c02 is in error state.  Still on run 109657, event 1096992.</pre>
<!-- Comment: Tue Mar 20 06;57;20 comment by...Frank Chlebana -->
<pre>For these runs we increased the data size to about 330 KB.
The TMPD bank for b0l1cl00-05 and b0l2cl00-05 was increase
by 4000 words while b0wcal crates were increased by 2000 words.

We can now control this through the hardware database.</pre>
<!-- Date: Tue Mar 20 02:19:40 2001 -->
(just to make an entry) 
iFix: you can't open the "COT systems" page if previously you 
have open the "CDF COT" page (an error pop up) 
<!-- Author: FP  :: (run (run 109657)) -->



<!-- Date: Tue Mar 20 04:25:22 2001 -->
<pre>Run 109657 still proceeding.  At about 3:00, started losing more 
L3 processors, individually at first, and then whole columns (or 
converter nodes) at a time.  At about 3:30, we lost all of output 
nodes u03 and u02 within about 5 minutes.  For most of these, it 
appeared that the upper analysis chain stopped counting first. 
After losing u02 and u03, the TS rate dropped to 29 Hz, with the 
usual fluctuation gone (see plots below).  The dead time stayed 
at about 99.99%, but switched from 85/15% (BUSY/READOUT) to 95/5. 
For any of this, we get no Merlin or Zephyr messages, and the L3 
Ace panel passes all Evb,L3,SM checks. 
 
Stopping run at 1451152 events to do COT tests prior to access.</pre>
<!-- Author: dave :: (run 109657) -->
<!-- Comment: Tue Mar 20 10;54;55 comment by...Ilya -->
<pre>   L3 experts should have been paged!
</pre>
<!-- Date: Tue Mar 20 04:27:56 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-244&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-244  width='300' align='top'></td><td>EVB rate</a></td></tr></table>
<!-- Author: dave -->


<!-- Date: Tue Mar 20 04:29:09 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-245&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-245  width='300' align='top'></td><td>...and L3 display (above).</a></td></tr></table>
<!-- Author: dave -->


<!-- Date: Tue Mar 20 04:38:59 2001 -->
<pre>Began run 109658 (less COT crates), but otherwise the same.</pre>
<!-- Author: dave :: (run 109658) -->
<!-- Comment: Tue Mar 20 04;44;35 comment by...dave -->
<pre>All L3 nodes appear to be back online, without doing any stop/
starting of proxies or L3 cleanups after last run.

The converter nodes c13-15 (c16 not in run) associated with
output node u04 have a noticeably higher number of events
(120 instead of 90, for example).  These are the nodes which
died first on the last run.</pre>

<!-- Comment: Tue Mar 20 04;44;53 comment by...dave -->
<pre><pre>All L3 nodes appear to be back online, without doing any stop/
starting of proxies or L3 cleanups after last run.

The converter nodes c13-15 (c16 not in run) associated with
output node u04 have a noticeably higher number of events
(120 instead of 90, for example).  These are the nodes which
died first on the last run.</pre></pre>

<!-- Comment: Tue Mar 20 04;51;37 comment by...dave -->
<pre>I guess that makes sense if events are getting divvied up by
output node, and u04 has one fewer converter node, so 25% more
events.</pre>


<!-- Comment: Tue Mar 20 06;48;26 comment by...dave -->
<pre>Ended run 109658 with no errors, to yank XFT crates for Jay.
Restarting new run.</pre>
<!-- Date: Tue Mar 20 06:36:19 2001 -->
MCR calls to say that they have turned off beam a bit early, 
and we can begin radiation survey/access as soon as we are ready. 
Paged ops mgr to relay this.
<!-- Author: Alan Sill -->




<!-- Date: Tue Mar 20 07:04:40 2001 -->
Dee, Rob here -- starting rad survey.  Access to begin when 
they are done.
<!-- Author: AFS -->


<!-- Date: Tue Mar 20 07:16:14 2001 -->
Train whistles from MCS, apparently signalling either the arrival 
of the Naperville Special, or alarm status of the HVAC system 
(one of these). 
 
Took 10 minutes to find the cryo tech and get him to silence the 
alarm, which is on differential pressure in the CH (due to fans 
for access??)...  
<!-- Author: Alan Sill -->


<!-- Date: Tue Mar 20 07:39:27 2001 -->
Access begins in earnest, many peoiple go in.
<!-- Author: AFS -->


<!-- Date: Tue Mar 20 07:57:57 2001 -->
<font color=red>Shift Summary: </font><pre>Shift started with long L3 torture test.  Rate dropped over 
time as nodes were lost, see above.  Noted holes in accumulated 
ET distribution... 
 
Took special COT ASDQ calibrations (map of dead channels), 
which worked fine. 
 
Access at end of shift, lots of people go in.  Note also Jay has 
COT02 and all of the XFT for special studies.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   109657 - 109659
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             100
   
</td></tr></table></pre>
<!-- Author: Alan Sill -->