<!-- Title: Eve shift. Thu Jun 15, 2006 -->
<!-- SciCo: Matthew Jones -->
<!-- DAQAce: Gianluca De Lorenzo -->
<!-- MonAce: Michael Milnik -->
<!-- CO: &nbsp; -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Very smooth running with silicon at the moment.  Basic plan 
is to run like this for a few hours, then try out Scientific 
Linux, then hopefully do TOF HV scans and whatever else Steve 
has in store for us...</pre>


<!-- Date: Thu Jun 15 16:08:58.072 2006 -->
Mark Mattson called back about DQMon's complaint about a dead 
WHA channel...  he expects it is just an issue of low statistics 
in the plot and said he would talk to DQMon experts about it. 
<!-- Author: M Jones :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400' target=_top>218400</a>) -->
<!-- Comment: Thu Jun 15 16;14;38 comment by...Brian -->
Note.  Mark made a comment on this a few days ago in the e-log.  As a temporary ace, I did not read back that far.  No need to pay attention to this warning with low stats.  Mark mentioned he would talk to DQMon experts to see if the warning could be disabled for low stats or something similar.
<!-- Comment: Thu Jun 15 16;18;11 comment by...m mattson (ADMEM) -->
<br>I've sent an e-mail to Mario or Oriol, as listed on the DQMon instructions page.  Hopefully they can tell me the current settings, and we can figure out the right level so the shift crew get the accurate warnings.  (The DQMon warning happened earlier in the week as well.)<p>
The low eta towers (6 and 7) in WHA often require 20 or 30K events in YMon until all towers get at least one event.  DQMon reported "dead" towers with about 4K events to YMon.
<!-- Comment: Thu Jun 15 16;55;54 comment by...Oriol (DQMon) -->
DQMon was tunned for every individual histogram to start checking after there were enough entries, based on some weeks of data taking.<br>
We use to have this kind of false alarms due to low statistics during stores with low luminosity.<br>
It's a good a idea to re-check all the values after the shutdown, but I would prefer to wait for higher luminosity stores to do that.<br>
In the meanwhile, ACES, before paging athe experts, remember to visually check the histograms in the CO monitors, look for the checklist requirements, look at the statistics required, and use the common sense (DQMon is an automated tool, it does not have common sense :-) )
<!-- Comment: Thu Jun 15 17;04;57 comment by...Larry N -->
I see CHA 23W 1R hot, 13E 4R cold or off<br>
WHA 18W 11 R hot, 2W 11R and 9E 21L cold or dead<br>
If that is what it sees in WHA, it is a real problem, should mark as known.
I guess DQM is not good enough to catch the bad (near dead) CHA guy.

<!-- Date: Thu Jun 15 16:12:39.145 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4032&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4032' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4033&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4033' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4034&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4034' width='300'></a></center>
<!-- Author: Brian -->
<!-- Comment: Thu Jun 15 16;12;58 comment by...Brian -->
Hourly plots 15:00 - 16:00.




<!-- Date: Thu Jun 15 16:50:49.922 2006 -->
Done TO for b0cot19.  Auto HRR recovered.  This crate had this same problem towards the end of the previous shift.
<!-- Author: Brian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400' target=_top>218400</a>) -->
<!-- Comment: Thu Jun 15 16;53;52 comment by...Brian -->
Again.  Will consider shepherding this crate if problems continue.




<!-- Date: Thu Jun 15 17:08:41.277 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4035&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4035' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4036&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4036' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4037&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4037' width='300'></a></center>
<!-- Author: Brian -->
<!-- Comment: Thu Jun 15 17;09;00 comment by...Brian -->
Hourly plots 16:00 - 17:00.


<!-- Date: Thu Jun 15 17:23:41.379 2006 -->
Heartbeat alarm for ICICLE.  ICICLE seemed to have not updated for at least 5 minutes.  Restarted ICICLE, though I doubt this is the source of the problem.  ICICLE is restarted, but Heartbeat problem continues.  Will give it a bit more time.  No other associated alarms.
<!-- Author: Brian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400' target=_top>218400</a>) -->
<!-- Comment: Thu Jun 15 17;29;26 comment by...Brian -->
Alarm has cleared.


<!-- Comment: Thu Jun 15 17;50;24 comment by...Brian -->
Heartbeat alarm for ICICLE returns.  Will keep an eye on it.
<!-- Date: Thu Jun 15 17:35:06.003 2006 -->
Done TO for b0pcal07.  Auto HRR did not recover.  Shepherding.  Successful and back to running.
<!-- Author: Brian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400' target=_top>218400</a>) -->



<!-- Date: Thu Jun 15 18:00:10.740 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4038&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4039  width='300' align='top'></a></td><td>DIRAC occupancy looks strange.</td></tr></table>
<!-- Author: R.J. Tesarek -->
<!-- Comment: Thu Jun 15 18;03;23 comment by...R.J. Tesarek -->
The above TrigMon plot shows high occupancy in wedge 10 of the
total trigger occupancy.  The DCAS 125 MeV plots look ok.  Also,
the plots for the XFT (Linker) shows a reasonably flat distribution in phi.  I'm not sure where the problem lies.

<!-- Comment: Thu Jun 15 18;08;31 comment by...M Jones -->
XTRP pager carrier called and will investigate.


<!-- Comment: Thu Jun 15 18;19;56 comment by...M Jones -->
They think it is related to a problem they were tracking down earlier today.  They will look into it in more detail to see if
what we are looking at is the same thing.
<!-- Comment: Thu Jun 15 18;59;32 comment by...KPitts -->
The high dirac occupancy came from a single pin input error on xtrp databoard #5.   We attempted to repair this problem during the run this afternoon (see note in day shift e-log).   We thought we failed, and were planning to address it between stores.  However, in the new run, 218406, this error is gone...so maybe we did better than we thought.   With the error gone in the new run, the corresponding dirac wedge now looks normal.
<!-- Date: Thu Jun 15 18:12:09.428 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4040&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4040' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4041&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4041' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4042&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4042' width='300'></a></center>
<!-- Author: Brian -->
<!-- Comment: Thu Jun 15 18;12;32 comment by...Brian -->
Hourly plots 17:00 - 18:00.


<!-- Date: Thu Jun 15 18:20:46.772 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400>218400</A>
 Terminated at 2006.06.15 18:20:33
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 18:20:47.586 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400>218400</A>
TERMINATE: end for SL L3 run
<!-- Author: Niharika -->



<!-- Date: Thu Jun 15 18:20:47.247 2006 -->
L3 people/Ankush are here to test scientific linux on L3. 
<!-- Author: M Jones :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218400' target=_top>218400</a>) -->

<!-- Date: Thu Jun 15 18:26:34.658 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406>218406</A>
 Activated at 2006.06.15 18:26:26
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 18:26:43.836 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406>218406</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] using SL L3 subfarms
<!-- Author: Niharika -->
<!-- Comment: Thu Jun 15 18;36;30 comment by...Conor, Khaldoun, Georgios -->
We are now taking data using Scientific Linux on Level3. We would like to take at least 3 hours of data like this, so trigger people can study the performance of the SL L3 filter.
<!-- Date: Thu Jun 15 18:29:02.925 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4043&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4043  width='300' align='top'></a></td><td>Empty bin in PPR Tower Occupancy at WEST eta -15 phi 15</td></tr></table>
<!-- Author: Rick T. & Michael -->



<!-- Date: Thu Jun 15 18:37:31.043 2006 -->
DOne TO for b0cot19 - HRR
<!-- Author: Conor (sitting at the DAQ ACE controls) :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->


<!-- Date: Thu Jun 15 18:41:23.402 2006 -->
From the error log at the start of this run: 
"Error fetching L3CalibTag info for run 218406" 
The Level3 manager is currently running on b0server04, as we are running SL Level3. I will forward this to Veronique and Dave Waters.
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->


<!-- Date: Thu Jun 15 18:43:46.593 2006 -->
The ProcMon check for 'L3 Manager Parent' has been disabled. For this SL Level3 run, we run the L3Manager on b0server04 instead of b0dap83, so the ProcMon check is temporarily failing.
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->


<!-- Date: Thu Jun 15 18:47:52.074 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4044&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4044' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4045&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4045' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4046&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4046' width='300'></a></center>
<!-- Author: Michael -->
<!-- Comment: Thu Jun 15 18;48;16 comment by...Michael -->
Got some errors. Issued HRR and paged L2 as is said on the slide.



<!-- Comment: Thu Jun 15 19;31;09 comment by...Michael -->
The entries are increasing. Right now we have 61868 and 15209
<!-- Comment: Thu Jun 15 20;25;05 comment by...Michael -->
Still increasing. Right now we have 72672 and 17012
<!-- Comment: Thu Jun 15 20;26;02 comment by...Michael -->
Expert called and wanted to call back, when he thinks there is a problem.
<!-- Comment: Thu Jun 15 22;17;40 comment by...jdl -->
The TFRD Monitor errors occur because Trigmon does not know about UPS, FPS or NCLC64 cuts in the trigger.  There is a new Trigmon guy.  Maybe he can fix this so the program stops crying "Wolf!" and confusing every new set of aces for the past 3 years.  The PulsarMonitor errors should be regarded as real.
<!-- Date: Thu Jun 15 19:04:16.787 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4047&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4047' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4048&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4048' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4049&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4049' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 15 19;04;51 comment by...gianluca -->
Hourly ace plots.  18:00->19:00


<!-- Date: Thu Jun 15 19:27:00.794 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><P> 
There are some zombie cpu-eating root processes running  
on the b0www web server machine.    Via some incomprehensible  
logic, I can create these zombies, but I cannot kill them. 
Anyhow, I have paged the so-called "Rex" system administration  
group to kill these processes. 
<P> 
Until they are killed, the web access may be painfully  
slow! 
<P></td></tr></td></table>
<!-- Author: badgett -->
<!-- Comment: Thu Jun 15 20;07;25 comment by...Willis -->
The Consumer operator "C.O. Results (and more) for last 100 runs"
web page was also wiped out. It may not be related, but also
strange. I had to remake it.
<!-- Date: Thu Jun 15 19:42:07.406 2006 -->
This SL Level3 run seems to be going quite smoothly. Dave Waters came in to eyeball the L3 trigger rates. I will go home now. I have asked to be paged at 9.30pm. At that time, we will end this run and go back to RedHat. I will be on the phone in case of any problems.
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->



<!-- Date: Thu Jun 15 20:09:34.470 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4053&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4053' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4054&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4054' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4055&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4055' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 15 20;10;03 comment by...gianluca -->
hourly ace plots: 19:00->20:00


<!-- Date: Thu Jun 15 20:21:51.162 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4057&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4057  width='300' align='top'></a></td><td>We got an DQMon warning about WHA. This problems seemed to be spotted before, so we did not page the expert. 
There are multiple empty entries and one hot entry.</td></tr></table>
<!-- Author: Michael -->
<!-- Comment: Thu Jun 15 20;23;36 comment by...Michael -->
The dead one outside of eta region 6 or 7 caused the proglem.



<!-- Comment: Thu Jun 15 20;39;21 comment by...Michael -->
Was an old plot, sorry.
Anyhow, the error is gone now.


<!-- Comment: Thu Jun 15 23;00;29 comment by...Larry N -->
If you look at the high threshold plot, it really is bad
<!-- Date: Thu Jun 15 20:59:31.835 2006 -->
<pre>BMU trip (NE #5).  recovery successful. 
CMP trip (Bottom, the same as all this week long). recovery successful.</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->


<!-- Date: Thu Jun 15 21:01:16.354 2006 -->
<pre>We got three errors simultanously: 
 
(MLE) b0control21.fnal.gov:Thread-31:Jun 15, 2006 08:52:30 PM-&gt;FrontEnd Error: IMU_01 CMX_01 VRB_SVX_02  
(MLW) b0control21.fnal.gov:Thread-31:Jun 15, 2006 08:52:30 PM-&gt;Received error message: errmon Not a state HaltRecoverRun 1 crate/s: b0imu01(81),  in error. [RXPT](MLE) b0control21.fnal.gov:Thread-31:Jun 15, 2006 08:52:30 PM-&gt;Requested Halt-Recover-Run issued (errmon) 
(MLW) b0control21.fnal.gov:Thread-31:Jun 15, 2006 08:52:33 PM-&gt;Inhibit flag SET (MLE) b0svx00:Messenger:Jun 15, 2006 08:52:38 PM-&gt;Silicon Resonator: Sl 16 Ch 9 -&gt; e074: Bytes: 00134 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx01:Messenger:Jun 15, 2006 08:52:37 PM-&gt;Silicon Resonator: Sl 10 Ch 5 -&gt; e130: Bytes: 00076 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 15, 2006 08:52:37 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 15, 2006 08:52:38 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:38 PM-&gt;Runtime Error 1, Event 2424537: See nonsequential event number (old 8, new 0) in L2 message 
(MLE) b0svx02:Messenger:Jun 15, 2006 08:52:37 PM-&gt;Silicon Resonator: Sl 11 Ch 3 -&gt; e203: Bytes: 00118 
 
b0svx02:Messenger:Jun 15, 2006 08:52:37 PM-&gt;SRC Fatal Error:Sl 6 L2A w/o L1A  
 --&gt;   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 L2A w/o L1A  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:39 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 3 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:39 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 4 not ready for readout 
1 crate/s: b0imu01(81),  in error. [RXPT] 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:39 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 5 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:39 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 7 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:39 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 8 not ready for readout 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 15, 2006 08:52:39 PM-&gt;{ 20:52:39, 6/15/2006 }: L2-Pulsar-Linux: Halted while  waiting for SVT data 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:39 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 9 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 11 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 12 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 13 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 15 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 16 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 2, Event 2424537: TDC in slot 17 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 3 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 4 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 5 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:40 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 7 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 8 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 9 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 11 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 12 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 13 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 15 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 16 not ready for readout 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:41 PM-&gt;Runtime Error 3, Event 2424538: TDC in slot 17 not ready for readout 
(RC)   20:52:40 Halt -&gt; HALTED 
(MLE) b0cmx01:Messenger:Jun 15, 2006 08:52:42 PM-&gt;Resynchronizing eventNumber to 2424536 (was 2424538) 
(RC)   20:52:48 Recover -&gt; RECOVERED 
(RC)   20:52:51 Run -&gt; ACTIVE 
(MLE) b0svx01:Messenger:Jun 15, 2006 08:53:19 PM-&gt;Silicon Resonator: Sl 10 Ch 5 -&gt; e130: Bytes: 00076 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 15, 2006 08:53:19 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 15, 2006 08:53:20 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
(MLE) b0svx02:Messenger:Jun 15, 2006 08:53:19 PM-&gt;Silicon Resonator: Sl 11 Ch 3 -&gt; e203: Bytes: 00118 
 
 
(MLE) b0svx00:Messenger:Jun 15, 2006 08:53:21 PM-&gt;Silicon Resonator: Sl 16 Ch 9 -&gt; e074: Bytes: 00134 
unction (S_errno_ETIMEDOUT) 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 15, 2006 08:53:21 PM-&gt;{ 20:53:21, 6/15/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data 
(RC)   20:53:21 Halt -&gt; HALTED 
(MLW) b0control21.fnal.gov:Thread-31:Jun 15, 2006 08:55:33 PM-&gt;Inhibit flag CLEAR (MLW) b0control21.fnal.gov:Thread-31:Jun 15, 2006 08:55:37 PM-&gt;Inhibit flag SET </pre>
<!-- Author: Michael :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->


<!-- Date: Thu Jun 15 21:03:08.634 2006 -->
<pre>More after we tried to recover the run: 
 
RC)   21:01:04 Run -&gt; ACTIVE 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-31:Jun 15, 2006 09:01:05 PM-&gt;FrontEnd Crate Error Condition from: 
VRB_SVX_02 
 
(MLE) b0control21.fnal.gov:Thread-31:Jun 15, 2006 09:01:05 PM-&gt;FrontEnd Error: VRB_SVX_02  
(MLE) b0svx08:Messenger:Jun 15, 2006 09:01:48 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 15, 2006 09:01:49 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
b0svx02:Messenger:Jun 15, 2006 09:01:48 PM-&gt;SRC Fatal Error:Sl 6 2 L1A to Buff  
 --&gt;   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 2 L1A to Buff  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 15, 2006 09:01:52 PM-&gt;{ 21:1:52, 6/15/2006 }: L2-Pulsar-Linux: Halted while  waiting for ShowerMax data 
(RC)   21:01:52 Halt -&gt; HALTED 
</pre>
<!-- Author: Michael :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->


<!-- Date: Thu Jun 15 21:06:21.920 2006 -->
<pre>And again, so we stop the run. The experts want to switch to RedHead anyway. 
 
(RC)   21:04:05 Run -&gt; ACTIVE 
(MLE) b0control21.fnal.gov:Thread-31:Jun 15, 2006 09:04:05 PM-&gt;FrontEnd Crate Error Condition from: 
VRB_SVX_02 
 
(MLE) b0control21.fnal.gov:Thread-31:Jun 15, 2006 09:04:05 PM-&gt;FrontEnd Error: VRB_SVX_02  
(MLE) b0svx04:Messenger:Jun 15, 2006 09:04:25 PM-&gt;Silicon Resonator: Sl 12 Ch 4 -&gt; e444: Bytes: 00128 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 15, 2006 09:04:25 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 15, 2006 09:04:26 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
(MLE) b0svx02:Messenger:Jun 15, 2006 09:04:25 PM-&gt;Silicon Resonator: Sl 12 Ch 4 -&gt; e224: Bytes: 00116 
 
b0svx02:Messenger:Jun 15, 2006 09:04:25 PM-&gt;SRC Fatal Error:Sl 6 2 L1A to Buff  
 --&gt;   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 2 L1A to Buff  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0felix.fnal.gov:TEDCpp:Jun 15, 2006 09:04:29 PM-&gt;{ 21:4:29, 6/15/2006 }: L2-Pulsar-Linux: Halted while  waiting for SVT data 
(MLE) b0svx01:Messenger:Jun 15, 2006 09:04:29 PM-&gt;Silicon Resonator: Sl 20 Ch 3 -&gt; e1a3: Bytes: 00110 
unction (S_errno_ETIMEDOUT) 
(RC)   21:04:29 Halt -&gt; HALTED</pre>
<!-- Author: Michael :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406' target=_top>218406</a>) -->

<!-- Date: Thu Jun 15 21:07:20.731 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406>218406</A>
 Terminated at 2006.06.15 21:06:57
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 21:08:12.039 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218406>218406</A>
TERMINATE: scx errors, so we end the run and start from scratch.
<!-- Author: Michael -->


<!-- Date: Thu Jun 15 21:15:42.163 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218407>218407</A>
 Activated at 2006.06.15 21:15:04
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 21:15:43.290 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218407>218407</A>
ACTIVATE: RunSet: AAA_CURRENT_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Michael -->



<!-- Date: Thu Jun 15 21:19:35.572 2006 -->
<pre>still problems with the SRC: 
 
Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 L2A w/o L1A  
 
.....</pre>
<!-- Author: Michael :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218407' target=_top>218407</a>) -->
<!-- Comment: Thu Jun 15 21;20;04 comment by...Michael -->
<pre>Silicon experts think about it!</pre>

<!-- Comment: Thu Jun 15 21;21;24 comment by...Michael -->
<pre>we will start a new run without silicon</pre>
<!-- Date: Thu Jun 15 21:20:58.496 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218407>218407</A>
 Terminated at 2006.06.15 21:20:37
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 21:21:04.710 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218407>218407</A>
TERMINATE: just 9 events and silicon problems
<!-- Author: Michael -->




<!-- Date: Thu Jun 15 21:23:32.523 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4059&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4059' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4060&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4060' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4061&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4061' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 15 21;24;10 comment by...gianluca -->
hourly ace plots: 20:00->21:00

<!-- Date: Thu Jun 15 21:24:30.798 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218408>218408</A>
 Activated at 2006.06.15 21:24:15
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 21:24:31.660 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218408>218408</A>
ACTIVATE: RunSet: AAA_NOSILICON_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Michael -->
<!-- Comment: Thu Jun 15 21;30;50 comment by...gianluca -->
<pre>we tried to restart a AAA_CURRENT run but the SRC problem was still there so we decide to run without silicon:
experts are going to investigate.

we switched to RH L3 farms following the L3 experts plan: SL farms worked properly for some hours.</pre>

<!-- Comment: Thu Jun 15 22;43;38 comment by...Sebsatian -->
This was not an SRC problem in reality. The CCC lost lock to RF and since the FER code which would give a warning is not in place yet (will be there in the next quiet time) we did not realized this was the problem. We do not know the reason for loosing lock, if we are optimistic: it could have been a real glitch which the CCC protected us against.
<!-- Date: Thu Jun 15 22:11:42.219 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218408>218408</A>
 Terminated at 2006.06.15 22:11:25
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 22:12:12.727 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218408>218408</A>
TERMINATE: Silicon experts want to make some tests. So we stop this run.
<!-- Author: Michael -->



<!-- Date: Thu Jun 15 22:13:36.935 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4062&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4062' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4063&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4063' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4064&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4064' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 15 22;17;12 comment by...gianluca -->
hourly ace plots: 21:00->22:00
<!-- Date: Thu Jun 15 22:15:32.277 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218410>218410</A>
 Activated at 2006.06.15 22:15:11
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 22:15:33.026 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218410>218410</A>
ACTIVATE: RunSet: AAA_CURRENT_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Michael -->






<!-- Date: Thu Jun 15 22:54:45.627 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4058&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4058  width='300' align='top'></a></td><td>Hot tower at eta -2 phi 3L. 
</td></tr></table>
<!-- Author: Michael -->



<!-- Date: Thu Jun 15 23:02:01.188 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-4065&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-4065  width='300' align='top'></a></td><td><h3>The rise of AGIGI2</h3> 
 
<p>This ACNET plot show the history of the losses and AGIGI2 over the last few hours.  AGIGI2 has been increasing monotonically over the last few hours.  By midnight it will probably hit 5.0 which is our TevMon limit. So we will take Silicon out.</p></td></tr></table>
<!-- Author: Ankush -->

<!-- Date: Thu Jun 15 23:04:06.674 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218410>218410</A>
 Terminated at 2006.06.15 23:03:59
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 23:05:32.871 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218410>218410</A>
TERMINATE: experts want to set silicon into standby
<!-- Author: Michael -->



<!-- Date: Thu Jun 15 23:08:32.597 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4066&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4066' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4067&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4067' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4068&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4068' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 15 23;09;07 comment by...gianluca -->
hourly ace plots: 22:00->23:00.
<!-- Date: Thu Jun 15 23:08:42.584 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218411>218411</A>
 Activated at 2006.06.15 23:08:35
<!-- Author: RunControl -->


<!-- Date: Thu Jun 15 23:08:43.374 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218411>218411</A>
ACTIVATE: RunSet: AAA_NOSILICON_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Michael -->



<!-- Date: Thu Jun 15 23:55:05.983 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Fri Jun 16 00:07:32.497 2006 -->
<font color=red>Shift Summary: </font><pre>Started with store 4772, all detectors included. 
Stopped the run to try scientific linux. 
Ran that way for a few hours.  Chris Neu was contacted 
about the TrigMon errors.  Ended the run with errors 
from silicon readout.  Resumed running without 
silicon using redhad configuration.  After much 
pondering, silicon was included in another run 
but removed because the silicon crew wanted to go 
home and didn't want to deal with rising losses. 
Running smoothly since then.  Plan is to keep the 
store until morning. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   218400, 218406, 218408, 218411
Delivered Luminosity     
Acquired Luminosity    1278.0  
Efficiency             
   
</td></tr></table></pre>
<!-- Author: M Jones -->


<!-- Date: Fri Jun 16 00:09:34.738 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4069&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4069' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4070&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4070' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-4071&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-4071' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Fri Jun 16 00;10;12 comment by...gianluca -->
hourly ace plots: 23:00->00:00