<!-- Title: Day shift. Thu Nov 15, 2007 -->
<!-- SciCo: Stefano Zucchelli -->
<!-- DAQAce: Adrian Buzatu -->
<!-- MonAce: &nbsp; -->
<!-- CO: Miguel Vidal -->
<!-- OpManager: Maxim Goncharov -->
<!-- Notes: -->
<pre>Shot setup in progress: now ending the pbar loading. </pre>


<!-- Date: Thu Nov 15 08:07:59.026 2007 -->
Scraping started, we are waiting for Silicon expert to tell us if silicon will potentially be included.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253829' target=_top>253829</a>) -->

<!-- Date: Thu Nov 15 08:15:42.674 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253829>253829</A>
TERMINATE: done, to start AAA_NONSILICON.
<!-- Author: Adrian Buzatu -->



<!-- Date: Thu Nov 15 08:15:42.931 2007 -->
Store 5724- scraping completed
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253829' target=_top>253829</a>) -->

<!-- Date: Thu Nov 15 08:23:19.276 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
 Activated at 2007.11.15 08:22:58
<!-- Author: RunControl -->



<!-- Date: Thu Nov 15 08:23:19.672 2007 -->
TeV PR = 7216 10**9     PB = 1326 10**9  Initial B0 Luminosity 115 10**30
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->


<!-- Date: Thu Nov 15 08:25:33.275 2007 -->
Initial luminosity: 114 E30
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->


<!-- Date: Thu Nov 15 08:26:52.983 2007 -->
Running  No SILICON for the time beeing
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->

<!-- Date: Thu Nov 15 08:27:56.344 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_5_01 [2,1002,710] 
<!-- Author: Adrian Buzatu -->



<!-- Date: Thu Nov 15 08:28:27.592 2007 -->
BMU alarm
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->
<!-- Comment: Thu Nov 15 08;35;47 comment by...Adrian -->
Because by mistake I activated the physics run before turning the HVs on at "at step 2". Max (OpsMg) was here and he turned this BMU wedge on standby and then on and when it was all green, we activated the run again, we are taking data now. 


<!-- Date: Thu Nov 15 08:32:21.221 2007 -->
Proton (7730) and antiproton losses( 5408 ) too high , on request of silicon expert called MCR to ask them to try to reduce it.
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->



<!-- Date: Thu Nov 15 08:41:40.550 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-6895&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-6895  width='300' align='top'></a></td><td>Loss/AbGaps D44 Lumberjack plot for store 5724</td></tr></table>
<!-- Author: Adrian Buzatu -->


<!-- Date: Thu Nov 15 08:43:16.597 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-6896&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-6896  width='300' align='top'></a></td><td>D44 SVRAD Integ</td></tr></table>
<!-- Author: Adrian Buzatu -->


<!-- Date: Thu Nov 15 08:44:37.197 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-6897&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-6897  width='300' align='top'></a></td><td>Current losses</td></tr></table>
<!-- Author: Adrian Buzatu -->


<!-- Date: Thu Nov 15 08:50:52.241 2007 -->
In DQMIN, in Trig and Lumi column, CLC is yellow. The text output is hard to read, but it is just a warning, I guess.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->


<!-- Date: Thu Nov 15 09:13:49.404 2007 -->
L2 TO, no automatic HRR, manual HRR did the trick.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->


<!-- Date: Thu Nov 15 09:14:02.737 2007 -->
Flying wire, all OK
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->

<!-- Date: Thu Nov 15 09:32:55.368 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
si: <tt>No Silicon yet</tt>
<!-- Author: Consumer Operator( Miguel Vidal) -->


<!-- Date: Thu Nov 15 09:32:56.816 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
trigger: <tt>No silicon yet.<br>
East IMB wedge 3 is bad</tt>
<!-- Author: Consumer Operator( Miguel Vidal) -->



<!-- Date: Thu Nov 15 09:33:47.792 2007 -->
Beam losses hasn't change. MCR said they are still investigating. 
We asked them to call us back in no more than 20 minutes from now  
So we can take the  decision if to put silicon in or not based on the best condition MCR claim they can reach.
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->

<!-- Date: Thu Nov 15 09:43:11.989 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
 Terminated at 2007.11.15 09:42:44
<!-- Author: RunControl -->


<!-- Date: Thu Nov 15 09:43:12.931 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
TERMINATE: Silicon experts say we can start running with silicon
<!-- Author: Adrian Buzatu -->


<!-- Date: Thu Nov 15 09:43:16.268 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
TERMINATE: Silicon experts say we can start running with silicon
<!-- Author: Adrian Buzatu -->



<!-- Date: Thu Nov 15 09:44:12.516 2007 -->
MCR called to say they are stable now :  C:LOSTP = 5930  C:LOSTPB=5430 
We are turning on Silicon 
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830' target=_top>253830</a>) -->

<!-- Date: Thu Nov 15 09:56:33.958 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
 Activated at 2007.11.15 09:56:26
<!-- Author: RunControl -->


<!-- Date: Thu Nov 15 09:56:34.962 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [2,1002,710] 
<!-- Author: Adrian Buzatu -->



<!-- Date: Thu Nov 15 09:57:42.967 2007 -->
<pre> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error   
  
 L3 Instantaneous error rate is 1.8068535 per cent.  
  
 Level3 Reformatter detects abnormal amount of corrupted data! 
 Reformatter errors are a corruption in the basic integrity of the data 
 received by Level3 and are not due to problems with the Level3 hardware 
 We recommend: 
    - try HRR; if no effect, try starting a new run 
    - page Level3 for assistance in finding the source of the problem</pre>
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->
<!-- Comment: Thu Nov 15 09;58;00 comment by...Adrian -->
<b>We did an HRR </b>


<!-- Date: Thu Nov 15 10:01:10.791 2007 -->
After I lift the threshold from 20 to 30 in ASD13, ( see <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=884&frame=2&anchor=13:07:56.300'> 
change threshold to suppress noise </a> ( I checked the first physics run after the change. It seems have killed the noise that was found in wedge 14(w13&14 are swapped back in May to investigate this noise). The physics runs after the change don't have noise anymore. 
<!-- Author: Xiaojian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 10:08:02.960 2007 -->
SVX inhibit a few minutes ago, Si experts worked on it, now we are taking data again.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 10:27:21.612 2007 -->
  RXPT Error for crate/s:  
    b0svx02    
  
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 10:31:07.404 2007 -->
Run is halted now, experts are looking into this.  
 
 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
 
 
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run      
 - record any relevant information in e-log      
 - RECOVER - RUN
<!-- Author: ADrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 10:53:06.423 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Automatic HRR works.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 11:01:10.557 2007 -->
Again as above...
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 11:04:52.112 2007 -->
Done Timeout detected in crates: 
CLC_00 
 
Automatic HRR worked.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 11:09:14.889 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Automatic HRR works
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 11:18:59.411 2007 -->
From CO: alarm on IMU : IMBHAT EAST Wedge 3 plot is read . 
Follwing instruction we paged TDC expert
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->
<!-- Comment: Thu Nov 15 11;52;40 comment by...Stefano Zucchelli -->
Eric called back, no worry it is a known problem. 
<!-- Date: Thu Nov 15 11:23:17.154 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Automatic HRR works
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 11:27:16.515 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6898&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6898' width='300'></a></center>
<!-- Author: miguel -->
<!-- Comment: Thu Nov 15 11;30;11 comment by...Miguel -->
East IMB HAT wedge 3 BAD
<!-- Comment: Thu Nov 15 14;22;10 comment by...Phil Schlabach -->
After the <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2007&action=view&page=888&frame=2&anchor=22:17:24.615'>
discussion on evening shift</a>, this belongs on the CO fix list.
<!-- Date: Thu Nov 15 11:27:43.332 2007 -->
FYI: The source of the L2 timeouts is RECES 
<!-- Author: gene -->


<!-- Date: Thu Nov 15 11:32:26.169 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
trigger: <tt>East IMB HAT wedge 3</tt>
<!-- Author: Consumer Operator( Miguel Vidal) -->



<!-- Date: Thu Nov 15 11:50:17.614 2007 -->
<pre>  RXPT Error for crate/s:  
    b0svt07    
 
FrontEnd Crate Error Condition from: 
SVT_PULSAR_08 
SVT_FANOUT_07 
 
Manual HRR did not work, we got immediately: 
 
  L2 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!! 
 
Manual HRR did not work, we got immediately" 
 
  L2 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!! 
 
Now it worked!!! We are geting data once more. 
 
</pre>
<!-- Author: Adroam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 11:51:27.697 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Automatic HRR worked.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->



<!-- Date: Thu Nov 15 12:03:02.537 2007 -->
   
  RXPT Error for crate/s:  
    b0svt0 
 
Manual HRR worked.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:03:43.686 2007 -->
 
    L3 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!! 
 
Manual HRR worked
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:15:59.929 2007 -->
TrigMon stopped and i restarted again
<!-- Author: MIguel :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:22:45.126 2007 -->
Done Timeout detected in crates: 
CLC_00 
 
Manual HRR worked.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:37:03.334 2007 -->
<pre>FrontEnd Crate Error Condition from: 
SVT_PULSAR_09 
SVT_PULSAR_08 
SVT_FANOUT_07 
 
   
  RXPT Error for crate/s:  
    b0svt08    
 
   
  RXPT Error for crate/s:  
   L1 Done Timeout.[RXPT]  
 
  L2 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!! 
 
A few manual HRR worked. 
 
 
 
 
</pre>
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:41:46.935 2007 -->
FrontEnd Crate Error Condition from: 
SVT_PULSAR_09 
SVT_PULSAR_08 
 
   
  RXPT Error for crate/s:  
    b0svt
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:42:37.586 2007 -->
FrontEnd Crate Error Condition from: 
SVT_PULSAR_09 
 
    L2 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!! 
 
Manual HRR worked.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->

<!-- Date: Thu Nov 15 12:42:54.437 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=253830 target=_runSummary>253830</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->



<!-- Date: Thu Nov 15 12:45:54.659 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Manual HRR worked/
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 12:49:59.921 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2 
 
After two manual HRRs, it worked.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 13:00:44.208 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Manual HRR works.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 13:08:10.146 2007 -->
SVX alarm istantly cleared, BMU alarm, SE156
<!-- Author: Stefano Zucchelli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 13:10:32.570 2007 -->
<pre>  Attention!!!.   RXPT Errors LOOP!!!.  
   
  &gt;&gt; Please Check the crate/s in the following list: &lt;&lt;  
   
   b0svx02     
     
  &gt;&gt;        and take the appropriate actions.        &lt; 
 
 
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatality Error for b0svx02: Sl 6 Too Many L1A 2 L1A to Buff  
  
 - HALT the run      
 - record any relevant information in e-log      
 - RECOVER - RUN  
 
did Manual Halt, REcover, Run (not HRR) 
 
at the smae time we hot BMU trip, I restarted ladder 156 of SE. 
</pre>
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 13:14:17.257 2007 -->
Attention!!!. FERML_HIGH_DEADTIME !!!  
 - Trigger/DAQ Deadtime is too high (at 100.24236 %).  
 - Look for source of hot triggers.  
 - Contact a trigger expert for further assistance.  
 
As soon as BMU HV was up and the trigger inhibit was cleared, we got this error, but total deadtime was small. so I did an HRR to restart. We are getting data. But the Monitors appear as red in DQMon.  
 
Did HRR. Got this error again: 
 
 
 Attention!!!. FERML_HIGH_DEADTIME !!!  
 - Trigger/DAQ Deadtime is too high (at 100.24236 %).  
 - Look for source of hot triggers.  
 - Contact a trigger expert for further assistance 
 
Did HRR and got this 
 
 
Done Timeout detected in crates: 
CLC_00 
 
Did HRR again and it is working
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->

<!-- Date: Thu Nov 15 13:20:42.128 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
trigger: <tt>East IMB HAT wedge 3 bad</tt>
<!-- Author: Consumer Operator( Miguel Vidal) -->


<!-- Date: Thu Nov 15 13:20:43.668 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
monitors: <tt>SVT track chi-sqr RED</tt>
<!-- Author: Consumer Operator( Miguel Vidal) -->



<!-- Date: Thu Nov 15 13:20:45.700 2007 -->
In DQMon the monitors appeared red after the previous errors, as if the communication had died, so I restarted DQMon, at first all monitors in DQMon appeared as white (no information), no all appear Green, except TrigMon, Beammon, Lummon, Svtspymon.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->
<!-- Comment: Thu Nov 15 15;22;55 comment by...Adrian -->
It came back to yellow, so OK. Theh warning is understood by experts. Also happened yesterday, please see <a href=
'/cgi/elog/elog.pl?nb=2007&action=view&page=889&frame=2&anchor=02:14:36.137'>
here </a> .
<!-- Date: Thu Nov 15 13:23:20.547 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6899&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6899' width='300'></a></center>
<!-- Author: Miguel -->
<!-- Comment: Thu Nov 15 13;29;52 comment by...Miguel -->
Present in other runs.
<!-- Date: Thu Nov 15 13:23:36.343 2007 -->
*** Node b0fib04 in Error *** 
Last data more than 120s ago!  Node is not responsive 
 
b0fib04 not responsible, red in VxWorks, red in Replies and Acknowledgments.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->

<!-- Date: Thu Nov 15 13:26:42.029 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253830>253830</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVX 
ISL 
L00 
SVT 
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 15 13:29:33.474 2007 -->
b0fib04 crate red, could not respond to ping, so I followed intructions and I sphephered this crate, talking data now. There was also a RTServer ProcMon error for a few seconds, recovered at the same time as the sphephered. Were they related maybe?
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->



<!-- Date: Thu Nov 15 14:12:34.718 2007 -->
 Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
Manual HRR solves it. 
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 14:18:19.852 2007 -->
  Done TO. for crate/s:  
    b0clc00  
 
Manual HRR worked.
<!-- Author: ADrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->


<!-- Date: Thu Nov 15 14:30:22.707 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
 Terminated at 2007.11.15 14:30:01
<!-- Author: RunControl -->


<!-- Date: Thu Nov 15 14:30:33.984 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
TERMINATE: Alison and OpsManager: end run, start a new one and include L2stereo.
<!-- Author: Adrian Buzatu -->


<!-- Date: Thu Nov 15 14:30:44.808 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831>253831</A>
TERMINATE: Alison and OpsManager: end run, start a new one and include L2stereo.
<!-- Author: Adrian Buzatu -->



<!-- Date: Thu Nov 15 14:32:43.635 2007 -->
 Attention!!!. L3_RELAY_FAILURE Error !!!.  
 
 One or more L3 nodes is likely to have died. 
 Do not panic - this message is mainly for your information. 
 Level3 should be able to recover by itself. 
 Please read the error log for more specific instructions. 
 
while ending the run, it came to idle succesfully.
<!-- Author: ADrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253831' target=_top>253831</a>) -->

<!-- Date: Thu Nov 15 14:35:46.840 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833>253833</A>
 Activated at 2007.11.15 14:35:35
<!-- Author: RunControl -->


<!-- Date: Thu Nov 15 14:36:04.644 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833>253833</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [2,1002,710] 
<!-- Author: Adrian Buzatu -->



<!-- Date: Thu Nov 15 14:36:27.056 2007 -->
Done Timeout detected in crates: 
SVT_PULSAR_09 
 
Manual HRR works.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833' target=_top>253833</a>) -->


<!-- Date: Thu Nov 15 15:06:36.423 2007 -->
CLC in DQMon turned red, this pop-up window appeared: "CLC. There is an error in the subsystem CLC. This is a serious problem. Page the relevant detector expert now"/ 
 
However, after 2 second it turned yellow again. Should we still page CLC?
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833' target=_top>253833</a>) -->


<!-- Date: Thu Nov 15 15:08:49.395 2007 -->
TOMCAT red in procmon. I checked again and now it is running.
<!-- Author: Adrian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833' target=_top>253833</a>) -->

<!-- Date: Thu Nov 15 15:11:27.765 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833>253833</A>
trigger: <tt>East IMB HAT wedge 3 bad</tt>
<!-- Author: Consumer Operator( Miguel Vidal) -->




<!-- Date: Thu Nov 15 15:32:32.779 2007 -->
currently halted because of PLUG HV inhibit. Trying to contact the experts.
<!-- Author: Donatella  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833' target=_top>253833</a>) -->
<!-- Comment: Thu Nov 15 15;53;47 comment by...Donatella -->
PLUG Caen crate 10 tripped. We (kind of...) followed the recovery procedures but forgot to Kill the monitoring program first! After doing that, pressed the crate 10 reset button, everything came back to live in no time. Many thanks to Steva Hahn for his help.

<!-- Comment: Thu Nov 15 15;55;06 comment by...Donatella  -->
i mean.. Steve, of course
<!-- Date: Thu Nov 15 15:55:06.041 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=253830 target=_runSummary>253830</A> x3DF86</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1002,710]</FONT></TD>
<TD ALIGN=CENTER>08:22:58</TD>
<TD ALIGN=CENTER>09:42:44</TD>
<TD ALIGN=CENTER>01:10:22</TD>
<TD ALIGN=RIGHT>     357,881</TD>   <TD ALIGN=RIGHT>    471.15</TD>
<TD ALIGN=RIGHT>    415.03</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=253831 target=_runSummary>253831</A> x3DF87</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1002,710]</FONT></TD>
<TD ALIGN=CENTER>09:56:26</TD>
<TD ALIGN=CENTER>14:30:01</TD>
<TD ALIGN=CENTER>03:46:55</TD>
<TD ALIGN=RIGHT>   1,205,269</TD>   <TD ALIGN=RIGHT>  1,115.26</TD>
<TD ALIGN=RIGHT>    908.71</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=253833 target=_runSummary>253833</A> x3DF89</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1002,710]</FONT></TD>
<TD ALIGN=CENTER>14:35:35</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:52:12</TD>
<TD ALIGN=RIGHT>     206,937</TD>   <TD ALIGN=RIGHT>    246.88</TD>
<TD ALIGN=RIGHT>    163.93</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:00</TD>
<TD ALIGN=CENTER>05:49:30</TD>
<TD ALIGN=RIGHT>     1,770,087</TD> <TD ALIGN=RIGHT>  1,833.29</TD>
<TD ALIGN=RIGHT>  1,487.67</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Thu Nov 15 15:57:36.532 2007 -->
<font color=red>Shift Summary: </font><pre>Thursday November 15 2007  - Day Shift 
 
8:00 TeV status: Pbar injection completed going into scraping 
8:30 TeV PR = 7216 10**9 PB = 1326 10**9 Initial B0 Luminosity 115 10**30 
     running NO-SILICON 
    No BMU HV in run 253829 and 253830 see Adrian comment on e-log 
   Proton (7730) and antiproton losses( 5408 ) too high ,  
   on request of silicon expert called MCR to ask them to try to 
   reduce it before turning silicon on 
9:40 Running with  SILICON 
10:30 RXPT Error for crate/s: b0svx02. Run is halted now, experts are 
      looking into this. AM-&gt;SRC Fatal Error:Sl 6 Resonance Detected  
13:10 SVX alarm istantly cleared, BMU alarm, SE156 tripped - </pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   
Delivered Luminosity   1637  
Acquired Luminosity    1295  
Efficiency             79%
   
</td></tr></table></pre>
<!-- Author: MIA scico -->


<!-- Date: Thu Nov 15 16:01:06.869 2007 -->
cot06 Done TO . manual HRR since auto-HRR was disabled (?) re-enable it. 
Back to running.
<!-- Author: Donatella  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253833' target=_top>253833</a>) -->


<!-- Date: Thu Nov 15 16:02:01.036 2007 -->
<font color=red>Shift Summary: </font><pre>Please use caution if you choose to put HTML into an entry.  Forgetting to use a close tag for each open tag will certainly cause problems. 
 
If you choose to use plain text, PLEASE USE CR's since this text window automatically wraps the text and gives the illusion of CR's.  (click to clear this message) 
</pre> 
<!-- Author: mia -->