<!-- Title: Eve shift. Sat Jan  1, 2005 -->
<!-- SciCo: A. Foland -->
<!-- DAQAce: E. Lipeles -->
<!-- MonAce: H. Sun -->
<!-- CO: F. Azfar -->
<!-- OpManager: B. Wagner -->
<!-- Notes: -->
<pre>Start shift with Si run 191778 underway in store 3893.  Plan is to take data, until 2 hours before the end of store when SVT wants to do a test.</pre>


<!-- Date: Sat Jan 1 16:17:00.789 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-53&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-54' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-55&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-56' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-57&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-58' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 16;17;22 comment by...Hao -->
<pre>Hourly plots (15:18-16:00)</pre>


<!-- Date: Sat Jan 1 16:39:50.912 2005 -->
<br> 
<b>SVT Tests at End of Store 3893</b>, from the silicon side: 
see  
<a href="http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=229&time=16:11:25"> 
this</a> entry.<br>
<!-- Author: Sebastian -->


<!-- Date: Sat Jan 1 17:02:56.973 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-59&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-60' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-61&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-62' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-63&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-64' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 17;03;48 comment by...Hao -->
<pre>Hourly plots (16:00-17:00)</pre>


<!-- Date: Sat Jan 1 17:37:23.585 2005 -->
<pre>3 HRRs didn't fix L2 decision timeout, reboot_alpha did</pre>
<!-- Author: lipeles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->


<!-- Date: Sat Jan 1 18:02:21.243 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-65&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-66' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-67&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-68' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-69&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-70' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 18;03;00 comment by...Hao -->
<pre>Hourly plots (17:00-18:00)</pre>


<!-- Date: Sat Jan 1 18:07:12.640 2005 -->
train whistle alarm again; again nothing appears out of order
<!-- Author: adf -->
<!-- Comment: Sat Jan  1 18;17;11 comment by...Steve in cryo -->
I got the alarm ~ 7 minutes later and noticed "out of service" on the COT alarm page. Busy network! 


<!-- Date: Sat Jan 1 19:06:22.603 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-71&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-72' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-73&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-74' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-75&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-76' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 19;07;05 comment by...Hao -->
<pre>Hourly plots (18:00-19:00)</pre>


<!-- Date: Sat Jan 1 20:07:23.704 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-77&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-78' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-79&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-80' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-81&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-82' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 20;08;00 comment by...Hao -->
<pre>Hourly plots (19:00-20:00)</pre>


<!-- Date: Sat Jan 1 20:16:13.631 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-83&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-83  width='300' align='top'></a></td><td>We have paged the L2 expert about the high incidence of pulsar errors. We've asked him to please add a comment. 
</td></tr></table>
<!-- Author: Farrukh Azfar -->
<!-- Comment: Sat Jan  1 20;28;32 comment by...Tom Wright -->
These mismatches come from the internal sync problems on the Alpha that require it to be rebooted occassionally.  The data coming in from the interface boards get corrupted, and sometimes it takes a few events before the Alpha notices it and generates a decision timeout because we don't always use the XTRP data for the trigger decision.  If some of those corrupted events get into the monitoring stream then you'll see entries in that plot.  As long as the number of errors is not steadily increasing but rather just jumps up whenever you have to reboot the Alpha then there is nothing to worry about.



<!-- Date: Sat Jan 1 20:30:08.997 2005 -->
<pre>We got a TOF inhibit alarm, indicating PM channel 159 is off. 
 
Also from HV summary screen the min bar of TOF dissapeared.</pre>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 20;39;36 comment by...Hao -->
<pre>The whole upper part of TOF (both east and west) was blue when we checked the TOF details.

We hit the HV "on" button on HV summary menu to recover the TOF,
the inhibit was cleared after we did that.   </pre>


<!-- Comment: Sat Jan  1 21;43;40 comment by...adf -->
<pre>for the record, we paged tof without response</pre>
<!-- Comment: Sat Jan  1 21;53;15 comment by...adf -->
<pre>Tof just called, apparently only got the page a moment ago.  Our actions were approved :)</pre>
<!-- Date: Sat Jan 1 20:42:19.242 2005 -->
<pre>&lt;strong&gt;L2 triggger people please read:&lt;/strong&gt;&lt;BR&gt; 
Some times after an HRR the L1 and L2 trigger rates will be identical and the dead time will be 100%. Ussual this can be fixed with a second HRR. This has happened three times tonight and doesn't seem to correlate with the reason for the original HRR (tonight is was   a L2 done time out, a hardware EVB error, and TOF HV error).</pre>
<!-- Author: lipeles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->
<!-- Comment: Sat Jan  1 20;44;35 comment by...lipeles -->
That should be:
<strong>L2 triggger people please read:</strong>


<!-- Comment: Sat Jan  1 21;17;02 comment by...Tom Wright -->
What is the L1/L2 accept rate when this happens?

<!-- Comment: Sat Jan  1 21;44;09 comment by...lipeles -->
The rate is ~300-400 Hz.
<!-- Comment: Sat Jan  1 22;13;59 comment by...Tom Wright -->
My guess is that a L1 trigger with a L2 auto-accept following it is running at the crossing rate for some reason.  Does ScalerMon show instaneous rates for individual triggers or just cumulative?  If it happens again maybe you can look on the ScalerMon display and see which L2 triggers are firing at whatever the total rate is at that time.  You might have to wait for awhile for the numbers to update.

<!-- Date: Sat Jan 1 21:07:47.242 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-84&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-85' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-86&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-87' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-88&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-89' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 21;08;45 comment by...Hao -->
<pre>Hourly plots (20:00-21:00)</pre>



<!-- Date: Sat Jan 1 21:19:15.829 2005 -->
<pre>CER_SVXMON_HALT_RECOVER_RUN_ERROR: 
Stuck Cellid S/B1/W5/L2/C0-4 . 
HRR recovered it.</pre>
<!-- Author: lipeles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->


<!-- Date: Sat Jan 1 21:35:04.621 2005 -->
<pre>Unmarked 3 pink cells on IMON after SVXMon error. 
 
They're: 
 
SVX AVDD B2W4L4 
SVX DVDD B2 W5 L4 
SVX VBAS B2 W9 L4 
 
Turned to green after unmark.</pre>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 21;44;05 comment by...Hao -->
<pre>The DVDD B2 W5 L4 cell got pink again, current read as 195.0, voltage 5.75

Will perfrom one more HRR and unmark.</pre>



<!-- Date: Sat Jan 1 21:50:18.965 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-90&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-91  width='300' align='top'></a></td><td>The history plot for SVX cell DVDD b2W5L4</td></tr></table>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 22;13;47 comment by...Hao -->
<pre>Turned pink again, the current value below 190 will be marked as "good" for this cell, it remained around 195.0 during this pink period. </pre>
<!-- Date: Sat Jan 1 22:04:50.116 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-94&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-95' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-96&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-97' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-98&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-99' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 22;08;49 comment by...Hao -->
<pre>Hourly plots (21:00-22:00)

LOSTP rate was not flat as before, it's still quite safe overall, though.</pre>




<!-- Date: Sat Jan 1 22:15:07.200 2005 -->
FERML_HEAP_CORRUPT ERROR !!! 
Heap Corrupt for crate b0pcal08. 
Sheperd successful
<!-- Author: lipeles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->


<!-- Date: Sat Jan 1 22:17:57.640 2005 -->
<pre>CER_SVXMON_HALT_RECOVER_RUN_ERROR 
Stuck Cellid S/B1/W5/L4/C0-6 . 
HRR fixed it</pre>
<!-- Author: lipeles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->


<!-- Date: Sat Jan 1 22:52:30.746 2005 -->
<pre>another t2 decision time out that require reboot_alphas to fix.</pre>
<!-- Author: lipeles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->


<!-- Date: Sat Jan 1 23:00:25.604 2005 -->
<pre>CMP HV tripped, node Sense Bottom A2 and Field Bottom D2 lost the power. Put them to standby and then on to recover the HV.</pre>
<!-- Author: Hao -->


<!-- Date: Sat Jan 1 23:07:41.782 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-100&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-101' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-102&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-103' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-104&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-105' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sat Jan  1 23;08;32 comment by...Hao -->
<pre>Hourly plots (22:00-23:00)</pre>

<!-- Date: Sat Jan 1 23:55:24.642 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778>
191778</A> 
x2ED22</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_00 [2,560,510]</TD>                                       
<TD ALIGN=CENTER>15:32:55</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>07:29:01</TD>                                          
<TD ALIGN=RIGHT>    544,994,375</TD>                                    
<TD ALIGN=RIGHT>     7,113,253</TD>                                     
<TD ALIGN=RIGHT>   1,815,981</TD>                                       
<TD ALIGN=RIGHT> 1186.623</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:03</TD>                                                       
<TD ALIGN=CENTER>07:29:01</TD>                                          
<TD ALIGN=RIGHT>     544,994,375</TD>                                   
<TD ALIGN=RIGHT>      7,113,253</TD>                                    
<TD ALIGN=RIGHT>     1,815,981</TD>                                     
<TD ALIGN=RIGHT> 1186.623</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Jan 2 00:01:52.346 2005 -->
<font color=red>Shift Summary: </font><pre>-Shift started with store 3893, silicon run 191778 
-A ToF power supply crate went off;  bringing up ToF HV again recovered 
-SVX cell DVDD B2W5L4 stubbornly remained pink even after several HRR's; 
Sebastien adjusted the current limits 
-CMP HV tripped (no apparent spikes in beam), recovered 
-Power has been lost in the village, apparently due to a car hitting a pole 
-When there are 2 hours left in the store, SVT wants to do a test; page 
SVT and main silicon pager 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   191778
Delivered Luminosity   1241  
Acquired Luminosity    1112  
Efficiency             89.6
   
</td></tr></table></pre>
<!-- Author: adf -->


<!-- Date: Sun Jan 2 00:02:14.143 2005 -->
<pre>Done Timeout detected in crates: 
COT_12 
HRR fixed it</pre>
<!-- Author: Ilsung :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191778' target=_top>191778</a>) -->


<!-- Date: Sun Jan 2 00:04:16.685 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-106&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-107' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-108&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-109' ></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-110&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-111' ></a></center>
<!-- Author: Hao -->
<!-- Comment: Sun Jan  2 00;05;34 comment by...Hao -->
<pre>Hourly plots (23:00-24:00)

Noticed that abort gap loss has a slightly increasing trend.</pre>

<!-- Comment: Sun Jan  2 00;08;25 comment by...Hao -->
<pre>Oops, the plots showed above are from 22:00 to 24:00, not 23:00.</pre>