<!-- Title: Day shift. Sun Jun 18, 2006 -->
<!-- SciCo: Andreas Warburton -->
<!-- DAQAce: Yanjun Tu -->
<!-- MonAce: Gianluca de Lorenzo -->
<!-- CO: Seog Oh -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Store 4778 in progress, and anticipated to persist until this afternoon; B0Lum 18.7e30; Stack 57.9e10; RR 62.4e10.</pre>


<!-- Date: Sun Jun 18 08:09:28.199 2006 -->
When I came, COT in DQMON is red. in DQMOn Message Log, I found 
Run Number #218620@17:54:40:Warning!!!, problems in subsystem COT---> disable if problems persis. 
It seems this warning has been showing from yesterday
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218624' target=_top>218624</a>) -->
<!-- Comment: Sun Jun 18 08;11;35 comment by...Yanjun -->
If I left-click on this botton, I found most things are good except the Slide/SlideCOT1(SL3)

<!-- Comment: Sun Jun 18 08;15;32 comment by...Yanjun -->
oops, I made mistake. NOT red, It is yellow. Sorry

<!-- Date: Sun Jun 18 08:21:07.195 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218624>218624</A>
 Terminated at 2006.06.18 08:19:52
<!-- Author: RunControl -->


<!-- Date: Sun Jun 18 08:21:08.037 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218624>218624</A>
TERMINATE: end run to make some effort on COT
<!-- Author: Yanjun  x2080 -->


<!-- Date: Sun Jun 18 08:29:25.201 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
 Activated at 2006.06.18 08:27:56
<!-- Author: RunControl -->
<!-- Comment: Sun Jun 18 08;31;07 comment by...Yanjun -->
Put TDC_COT_210 back to slot 10,
It works now.
<!-- Date: Sun Jun 18 08:29:26.131 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
ACTIVATE: RunSet: AAA_CURRENT_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Yanjun  x2080 -->




<!-- Date: Sun Jun 18 08:37:43.415 2006 -->
COT in DQMon is still yellow after we put TDC back
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625' target=_top>218625</a>) -->
<!-- Comment: Sun Jun 18 09;20;48 comment by...A. Warburton -->
It turns out that the TDC wasn't actually put back.
<!-- Date: Sun Jun 18 08:56:30.337 2006 -->
A couple of HB (heart beat) alarms.  Gianluca (monitoring Ace) is investigating.  Global Alarms page is now showing CMU/CMP/CMX/BMU and CES all red.  Also just heard a "node unreachable" alarm.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625' target=_top>218625</a>) -->
<!-- Comment: Sun Jun 18 09;22;29 comment by...A. Warburton -->
See <a href=
'/cgi/elog/elog.pl?nb=2006&action=view&page=412&frame=2&anchor=09:14:51.474'>
log entry below </a> for explanation.
<!-- Date: Sun Jun 18 09:01:17.715 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218624>218624</A>
central: <tt>COT SL3 8 dead cells.</tt>
<!-- Author: Consumer Operator( seog oh) -->


<!-- Date: Sun Jun 18 09:01:18.779 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218624>218624</A>
muons: <tt>The plots are empty but it is green.</tt>
<!-- Author: Consumer Operator( seog oh) -->



<!-- Date: Sun Jun 18 09:09:15.461 2006 -->
<pre>After MUON3 PC rebooting we had a trip in the CMX NW 21.  trip recovered, we're taking data again.</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625' target=_top>218625</a>) -->


<!-- Date: Sun Jun 18 09:14:51.474 2006 -->
<p>Problem started with all detectors (CMU CMP CMX BMU CES) in 
iFIX PC MUON3 showing heartbeat alarms.  I went downstairs to 
help Gianluca.  HVMON++ program had died with usual obscure 
error dialog.  Eventually, had to kill iFIX (CAEN account on 
MUON3 has sufficient privs) and restart computer.</p> 
 
<p>When the computer and iFIX came back up, and we started  
HVMON++, we found that during the interval, we had a trip on 
CMX, also.</p>
<!-- Author: Steve Hahn, Gianluca de Lorenzo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625' target=_top>218625</a>) -->
<!-- Comment: Sun Jun 18 09;15;59 comment by...Steve Hahn -->
One more item: during the time the program was not running or
the computer was off, the slow/software inhibits did the correct
action and went to the inhibit state, so we were not taking data
during this time.


<!-- Comment: Sun Jun 18 10;39;57 comment by...Dale -->
<br><u>Techical Note</u>:<br><br>
The run summary page shows the trip was the first thing to pull the inhbit and as normal we also went into the halt state.  However, ~16 seconds later we were (incorrectly) back in the run state.  Later the PC inhibits add to the uncleared trip and once the PC was brought back the iFIX (also PC) inhibits come back.  There is a window of about 6 seconds when it looks like we were active and took data while the CMX was still being ramped up.<br><br>
The moral of the story is to keep RC in the halt state until the condition/error/issue causing the inhibit is understood.  As is the case here, there can be a gap between the trip inhibit and the slow control iFIX inhbiit from ramping up the HV.  Another older example: the monitoring on a certain non-muon PC goes down and turns off half of a detector and later the inhbits clear while we are in a non-optimal run state.  Also silicon folks need to be in the halt state anyway to recover their trips.  All the more reason to keep things there until the problem as been sorted out and made clear.
<br>
<!-- Date: Sun Jun 18 09:21:56.428 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
 Terminated at 2006.06.18 09:21:43
<!-- Author: RunControl -->


<!-- Date: Sun Jun 18 09:22:29.305 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
TERMINATE: end to run because slot 10 is still out
<!-- Author: Yanjun  x2080 -->



<!-- Date: Sun Jun 18 09:26:39.303 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626>218626</A>
 Activated at 2006.06.18 09:26:07
<!-- Author: RunControl -->


<!-- Date: Sun Jun 18 09:27:35.852 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626>218626</A>
ACTIVATE: RunSet: AAA_CURRENT_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Yanjun  x2080 -->
<!-- Comment: Sun Jun 18 09;43;26 comment by...Yanjun -->
slot 10 is still out, because somebody mark hardare in database yesterday
<!-- Date: Sun Jun 18 09:33:52.589 2006 -->
Paged DAQ LRP at Steve's request.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626' target=_top>218626</a>) -->
<!-- Comment: Sun Jun 18 10;01;49 comment by...A. Warburton -->
Bill Badgett responded while Steve was on the phone with Eric James; Bill offers his help if needed.
<!-- Date: Sun Jun 18 09:42:09.466 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4273&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4273' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4274&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4274' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4275&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4275' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 09;42;42 comment by...gianluca -->
hourly ace plots: 08:00->09:37.


<!-- Date: Sun Jun 18 09:45:34.384 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
fwd: <tt>bottom right plot is empty</tt>
<!-- Author: Consumer Operator( seog oh) -->


<!-- Date: Sun Jun 18 09:45:35.634 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
central: <tt>cot sl3 eight dead channel</tt>
<!-- Author: Consumer Operator( seog oh) -->


<!-- Date: Sun Jun 18 09:45:36.713 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218625>218625</A>
muons: <tt>BSU : two bottom plots are mising<br>
TSU : bottom plot is missing<br>
But still all are greeen</tt>
<!-- Author: Consumer Operator( seog oh) -->



<!-- Date: Sun Jun 18 09:49:38.199 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4276&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4276  width='300' align='top'></a></td><td>CDF Beam and Abort Gap Losses since the begin of the store.</td></tr></table>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 10;15;12 comment by...Ankush -->
<pre>Gianluca, thanks for putting this plot in...that's really useful for us :-)</pre>
<!-- Date: Sun Jun 18 10:05:45.283 2006 -->
<pre>errata corrige: 
I commented an entry (0922-0926) in the downtime acnet log saying that we ended and restarted the run to include the tdc slot 10 in the bocot06 crate.  Actually, the tdc is still out because we were not able to call it back from the rc configuration edit.  We told the COT experts to investigate the problem.</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626' target=_top>218626</a>) -->


<!-- Date: Sun Jun 18 10:12:46.011 2006 -->
<pre>svx error and resonances HRR recover the run 
 
 
(MLE) b0control21.fnal.gov:Thread-32:Jun 18, 2006 10:07:35 AM->FrontEnd Error: VRB_SVX_02  
(TS)   10:08:11 ********* L2 trigger rate has dropped to 0 Hz! Events are not being logged ! 
(CSL)  10:08:16 ********* L3 trigger rate has dropped to 0 Hz! Events are not being logged ! 
(MLE) b0svx08:Messenger:Jun 18, 2006 10:08:35 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 18, 2006 10:08:35 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
 
b0svx02:Messenger:Jun 18, 2006 10:08:36 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE]</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626' target=_top>218626</a>) -->
<!-- Comment: Sun Jun 18 10;17;18 comment by...Ankush -->
<pre>That's our resonnnce protection kicking in.  Thanks for posting this in Yanjun :-)
</pre>
<!-- Date: Sun Jun 18 10:14:28.050 2006 -->
<pre>Error! HRR recover the run 
 
 
b0server03.fnal.gov:ConsumerErrorRe:Jun 18, 2006 10:11:02 AM->Runtime Error 1, Event 11746, RunNum 218626: SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B4/W6/L3/C6-9 . -->   
 Additional Information:  
 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid S/B4/W6/L3/C6-9 .  
 AUTO HRR will be issued  
[MLE] 
(MLW) b0control21.fnal.gov:Thread-32:Jun 18, 2006 10:11:02 AM->Received error message: errmon Active TurboHRR HRR suggested for:b0server03.fnal.gov:ConsumerErrorRe:Jun 18, 2006 10:11:02 AM->Runtime Error 1, Event 11746, RunNum 218626: SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B4/W6/L3/C6-9 .(MLE) b0control21.fnal.gov:Thread-32:Jun 18, 2006 10:11:02 AM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0svx08:Messenger:Jun 18, 2006 10:11:08 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 18, 2006 10:11:08 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 18, 2006 10:11:09 AM->{ 10:11:9, 6/18/2006 }: L2-Pulsar-Linux: Halted while  processing happily 
(RC)   10:11:14 Halt -> HALTED 
(RC)   10:11:21 Recover -> RECOVERED 
(RC)   10:11:29 Run -> ACTIVE 
</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626' target=_top>218626</a>) -->



<!-- Date: Sun Jun 18 10:27:09.736 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626>218626</A>
 Terminated at 2006.06.18 10:26:20
<!-- Author: RunControl -->


<!-- Date: Sun Jun 18 10:27:42.492 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218626>218626</A>
TERMINATE: end run to make another try getting slot 10 back
<!-- Author: Yanjun  x2080 -->



<!-- Date: Sun Jun 18 10:30:20.376 2006 -->
<P> 
I have put TDC in slot 10 of COT_06 (b0cot06) crate back  
<I>online</I> so that the shiftCrew can test it once  
again, before requesting an access.   The TdcReadoutMode  
has also been changed to value <I>0</I> − which is  
global done mode. 
<P>  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 10:30:54.639 2006 -->
Bill Badgett has put b0cot06 slot 10 COT TDC card back online in the hardware database.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->

<!-- Date: Sun Jun 18 10:35:06.600 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
ACTIVATE: RunSet: AAA_CURRENT_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Yanjun  x2080 -->



<!-- Date: Sun Jun 18 10:37:08.855 2006 -->
shepherded b0cot03 because FERML_HEAP_CORRUPT ERROR 
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 10:38:53.969 2006 -->
<pre>COT_06 Done timeout  
HRR, recoverd  
 
 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 - Timeout on global TDC Done  for crate b0cot06   
 - HRR is strongly RECOMMENDED.    
 </pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 10:45:55.332 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
 Activated at 2006.06.18 10:30:26
<!-- Author: RunControl -->



<!-- Date: Sun Jun 18 10:56:22.199 2006 -->
<pre>RXPT error: 
recovered 
 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-32:Jun 18, 2006 10:44:12 AM->FrontEnd Error: VRB_SVX_02  
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0svx03:Messenger:Jun 18, 2006 10:44:52 AM->Silicon Resonator: Sl 16 Ch 4 -> e364: Bytes: 00100 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 18, 2006 10:44:52 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 18, 2006 10:44:51 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
 
b0svx02:Messenger:Jun 18, 2006 10:44:52 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
 
</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 10:57:33.900 2006 -->
COT done timeout, automatic HRR issued, recovered
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 11:00:26.245 2006 -->
The plots in IMU/BSUCOMain and TSUCOMain 
 
Time to time the plots in the two catagories show up emnpty. 
Right now the bottom two plots  in BSUCOMain are empty. 
Beginning of morning shift the plots in TSUCOMain are empty, but it is ok now.
<!-- Author: seog :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 11:02:44.015 2006 -->
COT_06 done timeout. auto HRR failed. do manually HRR  
success
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 11:08:24.609 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4277&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4277' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4278&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4278' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4279&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4279' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 11;08;56 comment by...gianluca -->
hourly ace plots: 09:30 -> 11:00


<!-- Date: Sun Jun 18 11:10:58.047 2006 -->
COT_06 done timeout again, HRR
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 11:35:59.984 2006 -->
Paged main Silicon LRP re many RXPT error conditions from svx02.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 11:36:28.265 2006 -->
<pre>1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-32:Jun 18, 2006 11:29:20 AM->FrontEnd Crate Error Condition from: 
VRB_SVX_02 
 
(MLE) b0control21.fnal.gov:Thread-32:Jun 18, 2006 11:29:20 AM->FrontEnd Error: VRB_SVX_02  
(RC)   11:29:23 Run -> ACTIVE 
L1 Done Timeout.[RXPT] 
Resuming AUTOMATIC HRRs.[EH Info] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
(TS)   11:30:05 ********* L2 trigger rate has dropped to 0 Hz! Events are not being logged ! 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(CSL)  11:30:14 ********* L3 trigger rate has dropped to 0 Hz! Events are not being logged ! 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0svx04:Messenger:Jun 18, 2006 11:31:44 AM->Silicon Resonator: Sl 12 Ch 7 -> e452: Bytes: 00146 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx00:Messenger:Jun 18, 2006 11:31:45 AM->Silicon Resonator: Sl 08 Ch 7 -> e012: Bytes: 00314 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 18, 2006 11:31:44 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 18, 2006 11:31:44 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
 
b0svx02:Messenger:Jun 18, 2006 11:31:44 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 18, 2006 11:31:45 AM->{ 11:31:45, 6/18/2006 }: L2-Pulsar-Linux: Halted while  waiting for SVT data 
(RC)   11:31:45 Halt -> HALTED 
RXPT Error LOOP !!! Disabling AUTOMATIC HRRs.[EH Info] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLW) b0control21.fnal.gov:Thread-32:Jun 18, 2006 11:33:37 AM->Inhibit flag SET (MLW) b0control21.fnal.gov:Thread-32:Jun 18, 2006 11:33:41 AM->Inhibit flag CLEAR </pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 12:04:10.168 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4280&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4280' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4281&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4281' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4282&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4282' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 12;04;40 comment by...gianluca -->
horly ace plots: 11:00 -> 12:00.

<!-- Date: Sun Jun 18 12:21:38.231 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
fwd: <tt>empty plots - may still good</tt>
<!-- Author: Consumer Operator( seog oh) -->


<!-- Date: Sun Jun 18 12:21:39.780 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
muons: <tt>Stil empty plots and green - should be ok</tt>
<!-- Author: Consumer Operator( seog oh) -->


<!-- Date: Sun Jun 18 12:21:40.517 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
trigger: <tt>I should note that the plot with silicon 3 layer track occupancy is empty (in TrigMon slide show) - this has been the case since last week (as far as i can check). Assume this is ok.</tt>
<!-- Author: Consumer Operator( seog oh) -->



<!-- Date: Sun Jun 18 12:54:04.486 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4283&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4283  width='300' align='top'></a></td><td>Beam and Abort Gap Losses since the begin of the store.</td></tr></table>
<!-- Author: gianluca -->


<!-- Date: Sun Jun 18 13:03:00.702 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4284&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4284' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4285&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4285' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4286&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4286' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 13;03;40 comment by...gianluca -->
hourly ace plots: 12:00 -> 13:00


<!-- Date: Sun Jun 18 13:25:42.800 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4287&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4287  width='300' align='top'></a></td><td>Ymon CSX Occupancy Plots</td></tr></table>
<!-- Author: Yanjun -->
<!-- Comment: Sun Jun 18 13;30;14 comment by...Yanjun -->
DQMon Alarm:
PartSE erro there are 4 or more contiguous dead channels


<!-- Date: Sun Jun 18 13:55:46.129 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4290&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4290  width='300' align='top'></a></td><td>empty TSUCOMain plot (YMon #106)</td></tr></table>
<!-- Author: seog -->


<!-- Date: Sun Jun 18 13:57:09.639 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4291&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4291  width='300' align='top'></a></td><td>Empty BSUCOMain plot</td></tr></table>
<!-- Author: seog -->


<!-- Date: Sun Jun 18 14:07:05.087 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4294&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4294  width='300' align='top'></a></td><td>wtyh</td></tr></table>
<!-- Author: greg feild -->
<!-- Comment: Sun Jun 18 14;07;51 comment by...gianluca -->
Sorry I posted the wrong plot, with the wrong name, with the wrong comment...


<!-- Date: Sun Jun 18 14:08:25.100 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4292&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4292' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4293&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4293' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4294&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4294' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 14;09;00 comment by...gianluca -->
hourly ace plots: 13:00 -> 14:00


<!-- Date: Sun Jun 18 14:27:28.158 2006 -->
WHA Alarm in DQMon
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 14:27:40.618 2006 -->
Darren called from the MCR.  The plan now is to end the store at 16:00.  The change from an hour earlier is because we are no longer requesting an access.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 14:49:56.991 2006 -->
<pre>SVX error again: 
 
 
(MLE) b0svx05:Messenger:Jun 18, 2006 02:46:52 PM-&gt;Silicon Resonator: Sl 10 Ch 4 -&gt; e524: Bytes: 00140 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 18, 2006 02:46:53 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 18, 2006 02:46:52 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
b0svx02:Messenger:Jun 18, 2006 02:46:52 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected  
 --&gt;   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 18, 2006 02:46:53 PM-&gt;{ 14:46:53, 6/18/2006 }: L2-Pulsar-Linux: Halted while  waiting for SVT data 
(RC)   14:46:54 Halt -&gt; HALTED 
RXPT Error LOOP !!! Disabling AUTOMATIC HRRs.[EH Info] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(RC)   14:47:01 Recover -&gt; RECOVERED 
(RC)   14:47:05 Run -&gt; ACTIVE 
</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 15:03:32.156 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4295&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4295' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4296&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4296' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4297&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4297' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Sun Jun 18 15;04;06 comment by...gianluca -->
hourly ace plots: 14:00 -> 15:00

<!-- Date: Sun Jun 18 15:05:31.134 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
fwd: <tt>MiniPlug- one empty plot - probabily ok</tt>
<!-- Author: Consumer Operator( seog oh) -->


<!-- Date: Sun Jun 18 15:05:32.026 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
muons: <tt>BSU/TSU empty plots - probabily ok</tt>
<!-- Author: Consumer Operator( seog oh) -->



<!-- Date: Sun Jun 18 15:21:37.952 2006 -->
<pre>b0cot19 done time out. HRR works 
svx resonance, but no warning. 
 
MLE) b0cot19:Messenger:Jun 18, 2006 03:17:08 PM-&gt;TDC Header Mismatch evt 2547010 slot 12: static/fifo 0x93804682/0x33211112 [0D0,0D4] 
(MLE) b0cot19:Messenger:Jun 18, 2006 03:17:09 PM-&gt;TDC evt 2547010 slot 12 static/fifo: ModID 295/102 L2B 0/0 Hits 70/273 BC 130/18 
(MLE) b0cot19:Messenger:Jun 18, 2006 03:17:09 PM-&gt;Event 2547010: Bunchcounters in slot 5 (BC=130) and slot 12 (BC=18) disagree (L2B=0)  
(MLE) b0cot19:Messenger:Jun 18, 2006 03:17:09 PM-&gt;Runtime Error 2, Event 2547010: Bunch counter mismatch, mismatch count = 1 
1 crate/s: b0cot19(67),  Done TO. [RXPT] 
(MLW) b0control21.fnal.gov:Thread-32:Jun 18, 2006 03:17:09 PM-&gt;Received error message: errmon Not a state HaltRecoverRun 1 crate/s: b0cot19(67),  Done TO. [RXPT](MLE) b0control21.fnal.gov:Thread-32:Jun 18, 2006 03:17:09 PM-&gt;Requested Halt-Recover-Run issued (errmon) 
(MLE) b0svx04:Messenger:Jun 18, 2006 03:17:15 PM-&gt;Silicon Resonator: Sl 12 Ch 3 -&gt; e443: Bytes: 00086 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 18, 2006 03:17:16 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 18, 2006 03:17:15 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 18, 2006 03:17:16 PM-&gt;{ 15:17:16, 6/18/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data 
(RC)   15:17:17 Halt -&gt; HALTED 
(RC)   15:17:24 Recover -&gt; RECOVERED 
(RC)   15:17:26 Run -&gt; ACTIVE 
</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 15:41:41.793 2006 -->
Roberto (CLC pager carrier) called.  He'd like to take the HV crate bcsc1 over for 10 minutes of tests.  This shouldn't be disruptive to calibrations, etc., and apparently can be done between the stores.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->


<!-- Date: Sun Jun 18 15:48:08.690 2006 -->
Darren (MCR) called.  They'd like to end the store in 15 minutes.  We will end the run and bring down the HV.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627' target=_top>218627</a>) -->

<!-- Date: Sun Jun 18 15:50:16.337 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
 Terminated at 2006.06.18 15:49:25
<!-- Author: RunControl -->


<!-- Date: Sun Jun 18 15:51:16.013 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218627>218627</A>
TERMINATE: end the run before the store end
<!-- Author: Yanjun  x2080 -->


<!-- Date: Sun Jun 18 15:55:27.875 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Jun 18 15:57:27.736 2006 -->
<font color=red>Shift Summary: </font><pre>08:00 Store 4778; B0Lum 18.7e30; run 218624 in progress. 
 
08:50ish HB alarms for muons and CES; HVMON++ program died; 
killed iFix and restarted PC; there was also a concurrent 
CMX trip. 
 
10:30 b0cot06 slot 10 COT TDC card was put back online to 
test if it's still causing problems that would require 
access. 
 
10:12 Error conditions from b0svx crates.  Occurred four 
additional times. 
 
15:50 Run 218627 ended for planned end of store 4778. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   218624-7
Delivered Luminosity     
Acquired Luminosity      
Efficiency             
   
</td></tr></table></pre>
<!-- Author: Andreas Warburton -->