<!-- Title: Owl shift. Tue Apr 19, 2005 -->
<!-- SciCo: R.J. Tesarek -->
<!-- DAQAce: Kwangzoo Chung, Veronica Sorin -->
<!-- MonAce: Christobal, Almenar -->
<!-- CO: Fabrizio Scuri -->
<!-- OpManager: Aron Soha -->
<!-- Notes: -->
<pre>Store: 4104 
Inherit run 196737 
Luminosity: 75.2 e30 
 
Shift Goals:  Keep the detector running, try to avoid problems from 
yesterday's shift.</pre>


<!-- Date: Tue Apr 19 00:34:25.703 2005 -->
<br><b>Accelerator Status:</b> 
<br>No stack, stash = 5.6e10, accelerator studies underway. 
We may be spinning this store for some time. 
<br>  
<!-- Author: R.J. Tesarek -->


<!-- Date: Tue Apr 19 00:44:25.142 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6542&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6542' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6543&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6543' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6544&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6544' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 00;44;45 comment by...Cristobal -->
Hourly plots: 11:30 to 12:30


<!-- Date: Tue Apr 19 01:00:07.305 2005 -->
<pre>We found an error message in MUON3 machine: 
 
Workspace application 
Error Number: -2147467259 
Unspecified error 
 
It keeps poping up when we ok the message.</pre>
<!-- Author: Kwangzoo & Cristobal :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->


<!-- Date: Tue Apr 19 01:22:32.445 2005 -->
<br><b>Stacking resumes:</b> 
<br>Stacking resumed about 10 minutes ago.
<!-- Author: R.J. Tesarek -->


<!-- Date: Tue Apr 19 01:31:49.969 2005 -->
CLC - In the check list found 
 
CLAD/CLTD occupancy plot and CLTD Delta-plot with no entries for  
 
West Layer 1, ch.13 
East layer 1, ch.62 
West Layer 2, ch. 17
<!-- Author: F.Scuri :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->
<!-- Comment: Tue Apr 19 02;11;47 comment by...F.Scuri -->
Sorry, those chs were already empty in the previous runs, I thought 
it was something new
<!-- Date: Tue Apr 19 01:34:35.693 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6545&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6545' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6546&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6546' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6547&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6547' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 01;35;01 comment by...Cristobal -->
Hourly plots: 12:30 to 1:30


<!-- Date: Tue Apr 19 02:01:14.453 2005 -->
<br><b>L2 problem:</b> 
<br>Wojtek called to inform us of a problem with L2 muons 
and stated that the problem was not likely due to the Pulsar. 
Per instructions we stop and restart TrigMon and the problems no longer appear.  The problem appears with emulation of the  
muon trigger.   Unfortunately, we (the shift crew) only have a 
good/bad assessment of this portion of the system. 
<br>  
  
<!-- Author: R.J. Tesarek :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->
<!-- Comment: Tue Apr 19 02;22;43 comment by...Wojtek -->
talked to Cheng-Ju- working hypothesis is that it is most likely upstream throwing L2muon off. this problem seems to have occured nearby the entry about MUON3, exact time correlation will be difficult to get. Then it persisted for some time (I also can't say how long exactly since we only get 2Mb of disk space and that flled up) 
The error condition is no longer occuring.



<!-- Comment: Tue Apr 19 03;12;06 comment by...Cristobal & Kwangzoo -->
MUON3 is the computer that monitors the muon systems HV (HVMon++). When we passed by (we wanted to check CLC1) we saw the error messeage we reported earlier tonight. This error message didn't give much information and only had an "OK" button, that we pressed. This action made the error message vanish and pop up again. We clicked OK several times, but the error messages would come back every single time.
However, the HV monitoring seemed to be working properly, so we thought that this error wasn't critical for the system. We didn't see anything strange in the behaviour of VNODE1 HV summary or in the comsumers (muon related) plots.
<!-- Comment: Tue Apr 19 03;25;59 comment by...R.J. Tesarek -->
<br><b>Summary:</b>
<br>Based on the information here, it seems unlikely that MUON3
initiated the problems with the L2 muon trigger observed earlier.
It would be nice to have a clear description of what may cause
the "bad" status on the Pulser summary slide and how we might
investigate such a problem (eg: where to look for log files). <br>
<!-- Date: Tue Apr 19 02:30:58.048 2005 -->
<pre>Follow up on XFT pager.  None of the XFT pager carriers that I 
could contact has the pager.  The remaining carriers are either 
at APS or out of state.  Before I track down the pager carrier, 
just page me directly for any XFT related problems (218-9236).</pre>
<!-- Author: Cheng-Ju -->


<!-- Date: Tue Apr 19 02:36:47.936 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6548&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6548' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6549&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6549' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6550&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6550' width='300'></a></center>
<!-- Author: Cridtobal -->
<!-- Comment: Tue Apr 19 02;38;02 comment by...Cristobal -->
Hourly plots: 1:30 to 2:30


<!-- Date: Tue Apr 19 02:54:26.684 2005 -->
L2 Decision TO. Auto HRR
<!-- Author: Kwangzoo & Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->




<!-- Date: Tue Apr 19 03:29:01.958 2005 -->
<br><b>Beam Status:</b> 
<pre> 
  Luminosity:     56.4 E30 /cm^2/s 
  Stack:          35.3 E10 pbars 
  Stacking rate:  16.3 E10 pbars/hr 
  Stash:           5.6 E10 pbars 
</pre>
<!-- Author: R.J. Tesarek -->
<!-- Comment: Tue Apr 19 03;32;00 comment by...R.J. Tesarek -->
Based on the above data, I might project the next store to 
occur on the next owl shift.
<!-- Date: Tue Apr 19 03:29:14.560 2005 -->
<pre>Done TO for b0pcal05. Auto HRR didn't fix it. Shepherded it successfully!</pre>
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->
<!-- Comment: Tue Apr 19 03;37;13 comment by...Kwangzoo -->
<pre>Bunchcounter mismatch between slot 15 and slot 14, 16, ..., 21 as below
...
(MLE) b0pcal05:Messenger:Apr 19, 2005 3:24:34 AM->Event 5133071: Bunchcounters in slot 15 (BC=109) and slot 17 (BC=121) disagree (L2B=1) 
L2B 15:  86 109 139 139
L2B 17:  86 121 139 139
EHx: 000E0026 001F0026 00240026 003E0026
...
</pre>
<!-- Date: Tue Apr 19 03:35:53.846 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6555&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6555' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6552&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6552' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6551&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6551' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 03;36;12 comment by...Cristobal -->
Hourly plots: 2:30 to 3:30



<!-- Date: Tue Apr 19 03:41:46.798 2005 -->
Done TO for b0clc00. Auto HRR.
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->
<!-- Comment: Tue Apr 19 03;55;51 comment by...Veronica & Kwangzoo -->
Just one Done TO this time. It might not be related but found a  <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=307&frame=2&anchor=01:43:40.061'>  recent replacement</a> of the TDC board in the slot 21 of b0clc00.
<pre>
(MLE) b0clc00:Messenger:Apr 19, 2005 3:40:11 AM->Event 5324105: Bunchcounters in slot 16 (BC=81) and slot 21 (BC=84) disagree (L2B=0) 
L2B 16:  81  81  57  81
L2B 21:  84   0   0   0
EHx: 00090077 00170077 00EB0077 00360077
(MLE) b0clc00:Messenger:Apr 19, 2005 3:40:12 AM->Runtime Error 2, Event 5324105: Bunch counter mismatch, mismatch count = 1
CT: 2005.04.19 03:40:16
40'16" 1 crate/s: b0clc00(118),  Done TO. [RXPT]
</pre>


<!-- Date: Tue Apr 19 04:24:33.101 2005 -->
Done TO for b0pcal09. Auto HRR.
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->
<!-- Comment: Tue Apr 19 04;26;48 comment by...Kwangzoo -->
<pre>
 Additional Information: 

 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb16 (forwarded by FER crate PCAL_09). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
</pre>


<!-- Date: Tue Apr 19 04:36:41.576 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6556&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6556' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6557&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6557' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6558&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6558' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 04;37;09 comment by...Cristobal -->
Hourly plots: 3:30 to 4:30


<!-- Date: Tue Apr 19 05:44:14.703 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6559&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6559' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6560&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6560' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6561&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6561' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 05;44;41 comment by...Cristobal -->
Hourly plots: 4:30 to 5:30


<!-- Date: Tue Apr 19 05:46:22.453 2005 -->
We got a heartbeat alarm for the plug. We went to upstairs to check the monitoring pc. All the windows opened were frozen except the client and server windows. Tried to kill the client by hitting ctrl-c, which didn't kill the client but brought the frozen windows back somehow. From HV Monitoring, we stopped and started monitoring. About 5 minutes later, the alarm went way. Everything is ok now. 
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->


<!-- Date: Tue Apr 19 05:50:12.246 2005 -->
L2 Decision TO. Auto HRR.
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->


<!-- Date: Tue Apr 19 06:12:06.147 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-6562&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-6562  width='300' align='top'></a></td><td>Looks like we are getting continuous reformatter errors from 
b0l3269. 
 
</td></tr></table>
<!-- Author: Frank Chlebana -->


<!-- Date: Tue Apr 19 06:31:33.887 2005 -->
Consumer Error! HRRed. 
<pre> 
 Additional Information:  
 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!!  
  
 DAQErrMon:DataMissingError:  
 Error 4 Event 7631906 Run 196737 Bank EMTD_StorableBank missing from event record, nErrors 1  
 
 - We suggest issuing an HRR.  
  
 - if the problem persists for more than a few minutes,  
   starting a new Run may be necessary.  
[MLE] 
29'18" b0dap84.fnal.gov:ConsumerErrorRe:Apr 19, 2005 6:29:18 AM->DAQErrMon:DataMissingError: Error 5 Event 7631906 Run 196737 Bank HATD_StorableBank missing from event record, nErrors 1  --> [MLE] 
29'18" b0dap84.fnal.gov:ConsumerErrorRe:Apr 19, 2005 6:29:18 AM->DAQErrMon:DataMissingError: Error 6 Event 7631906 Run 196737 Bank MPAD_StorableBank missing from event record, nErrors 1  --> [MLE] 
29'18" b0dap84.fnal.gov:ConsumerErrorRe:Apr 19, 2005 6:29:18 AM->DAQErrMon:DataMissingError: Error 7 Event 7631906 Run 196737 Bank SL1D_StorableBank missing from event record, nErrors 1  --> [MLE] 
(MLE) b0svx02:Messenger:Apr 19, 2005 6:29:44 AM->Silicon Resonator: Sl 16 Ch 8 -> e273: Bytes: 00126 
 
(RC)   6:29:47 Halt -> HALTED 
(RC)   6:29:54 Recover -> RECOVERED 
(RC)   6:29:56 Run -> ACTIVE 
</pre>
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=`196737' target=_top>196737</a>) -->


<!-- Date: Tue Apr 19 06:34:42.782 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6563&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6563' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6564&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6564' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6565&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6565' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 06;35;03 comment by...Cristobal -->
Hourly plots: 5:30 to 6:30


<!-- Date: Tue Apr 19 07:21:04.297 2005 -->
I'm trying to fix the popup error message on MUON3 as noted <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=325&frame=2&anchor=02:01:14.453'> 
above</a>.  It seems a bit sluggish but I just closed and reopened iFIX WorkSpace without restarting the monitoring.  It appears to have worked. 
<br><br> 
On a side note with respect to the L2 stuff <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=325&frame=2&anchor=02:01:14.453'> 
above</a> Muon3 (a HV control PC) knows <i>nothing</i> about L2 muons or the like.  
<!-- Author: Dale Stentz -->
<!-- Comment: Tue Apr 19 07;44;36 comment by...Phil -->
My recollection is that restarting is the only way to fix this particular error; otherwise it persists. I believe this one (-2147467259 ) is also the one that clogs the queue slowing the response to other business (e.g. trigger inhibits).
<!-- Date: Tue Apr 19 07:27:42.117 2005 -->
SVX - In the check-list of 07:00 a.m, I marked "bad" the svx reformatter errors becausde I read 2%, but it actually is 0.02%, so don't worry....
<!-- Author: F.Scuri :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->


<!-- Date: Tue Apr 19 07:33:27.696 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6566&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6566' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6567&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6567' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6568&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6568' width='300'></a></center>
<!-- Author: Cristobal -->
<!-- Comment: Tue Apr 19 07;33;58 comment by...Cristobal -->
Hourly plots: 6:30 to 7:30


<!-- Date: Tue Apr 19 07:41:56.995 2005 -->
RXPT Error for b0svx02. Manually halted run. 
<pre> 
Attention !!!. FERML_SRC_FATALITY ERROR !!!  
SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
</pre> 
Then Recovered and Running.
<!-- Author: Veronica & Kwangzoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737' target=_top>196737</a>) -->


<!-- Date: Tue Apr 19 07:55:40.078 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196737>
196737</A> 
x30081</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_02 [1,639,563]</TD>                                       
<TD ALIGN=CENTER>22:19:20</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>08:47:42</TD>                                          
<TD ALIGN=RIGHT>    661,831,521</TD>                                    
<TD ALIGN=RIGHT>     9,053,447</TD>                                     
<TD ALIGN=RIGHT>   2,273,089</TD>                                       
<TD ALIGN=RIGHT> 1901.973</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:05</TD>                                                       
<TD ALIGN=CENTER>08:47:42</TD>                                          
<TD ALIGN=RIGHT>     661,831,521</TD>                                   
<TD ALIGN=RIGHT>      9,053,447</TD>                                    
<TD ALIGN=RIGHT>     2,273,089</TD>                                     
<TD ALIGN=RIGHT> 1901.973</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Apr 19 08:00:22.531 2005 -->
<font color=red>Shift Summary: </font><pre>Owl shift summary 
 
Start of shift:  
 Store:      4104 
 Luminosity: 75.2 E30/cm^2/s 
 Run:        196737 
  
 o Problem with MUON3:  Undecipherable error code on routine check of the 1st  
   floor HV PCs. 
 o 01:50 Wojtek phones to tell us about high error rates with the L2 muons. 
   Restarting TrigMon clears the alarm state. Problem appears to be a discrepency 
   between the Trigger result and the emulation of the trigger. 
 o L2 decision time outs continue (at low rate). 
 o Plug HV heartbeat problem. 
 
Totals 
Date:	2005.04.19 
Shift:	owl 
Delivered luminosity:	1592.5 nb-1 
Acquired luminosity:	1510.0 nb-1 
Efficiency:	94.9 
 
Record efficency  
 
R.J.Tesarek 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   196737
Delivered Luminosity   1592  
Acquired Luminosity    1510  
Efficiency             94.9
   
</td></tr></table></pre>
<!-- Author: R.J.Tesarek -->