<!-- Title: Eve shift. Mon Jul 21, 2003 -->
<!-- SciCo: Colin Gay -->
<!-- DAQAce: Mauro Donega -->
<!-- MonAce: Jason Galyardt -->
<!-- CO: Stefano Torre -->
<!-- OpManager: Nathan Eddy -->
<!-- Notes: -->
<pre>Inherting shiny, brand-new store, only 1/2 hour old. 
Current L=30.5 
Stack at 17.8, rate = 12.4</pre>


<!-- Date: Mon Jul 21 16:08:39.987 2003 -->
(MLE) b0dap73.fnal.gov:Thread-8830:4:01:39 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472' target=_top>166472</a>) -->
<!-- Comment: Mon Jul 21 16;17;17 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-8830:4:07:43 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00 <br>
<b> This type of done time out has no voice alarm</b>
<!-- Comment: Mon Jul 21 16;34;02 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-8830:4:26:01 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00<p>
From here on the run number is 166479<p>
<!-- Comment: Mon Jul 21 17;14;03 comment by...mauro -->
 (MLE) b0dap73.fnal.gov:Thread-11652:5:07:55 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 17;17;11 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:5:11:19 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 17;18;44 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:5:13:05 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 17;56;46 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:5:50:11 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 18;22;05 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:6:15:15 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 18;22;44 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:6:15:47 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 18;26;35 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:6:20:08 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 18;34;49 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:6:28:02 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;05;22 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:6:59:41 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;08;04 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:7:01:31 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;23;51 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:7:17:56 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;24;16 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:7:18:24 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;28;10 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:7:22:13 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;39;25 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:7:33:35 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 19;40;07 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:7:34:12 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 20;19;25 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:8:06:20 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 20;19;40 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:8:09:36 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 20;19;57 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:8:13:04 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 20;23;36 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:8:17:50 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 20;47;52 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:8:41:53 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 21;27;01 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:9:20:51 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 21;30;23 comment by...mauro -->
 (MLE) b0dap73.fnal.gov:Thread-11652:9:24:24 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 21;31;52 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:9:26:06 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 21;39;32 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:9:30:27 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00<br>
as usual without voice alarm and this time not even the orange window...

<!-- Comment: Mon Jul 21 21;55;06 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:9:49:07 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 22;23;42 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:10:17:11 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 22;30;53 comment by...Donatella T. -->
I'll enable the sound alarm for L1 Done Timeout(s) and check about the missing orange window ...
<!-- Comment: Mon Jul 21 23;30;21 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:11:24:07 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 23;36;56 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:11:31:04 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Mon Jul 21 23;51;37 comment by...mauro -->
(MLE) b0dap73.fnal.gov:Thread-11652:11:44:20 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Tue Jul 22 00;10;05 comment by...Dan Ryan -->
b0dap73.fnal.gov:Thread-11652:12:03:44 AM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00
<!-- Comment: Tue Jul 22 00;11;55 comment by...dan Ryan -->
b0dap73.fnal.gov:Thread-11652:12:05:51 AM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00


<!-- Date: Mon Jul 21 16:15:29.364 2003 -->
Trigger inhibit is set from the component TOF. <br> 
Bizarre... we don't have any problem on the ifix and the TOF PC looks OK. No action taken on our side.  
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472' target=_top>166472</a>) -->



<!-- Date: Mon Jul 21 16:18:41.529 2003 -->
<b> Done time out's</b>
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472' target=_top>166472</a>) -->
<!-- Comment: Mon Jul 21 16;18;53 comment by...mauro -->
(MLE) b0cot10:Messenger:4:16:41 PM->Event 523398: Bunchcounters in slot 5 (BC=118) and slot 8 (BC=56) disagree (L2B=1) 


<!-- Comment: Mon Jul 21 16;26;36 comment by...mauro -->
(MLE) b0cot00:Messenger:4:23:24 PM->Event 596953: Bunchcounters in slot 2 (BC=33) and slot 4 (BC=3) disagree (L2B=1)
<!-- Comment: Mon Jul 21 16;26;47 comment by...mauro -->
 (MLE) b0cot10:Messenger:4:24:13 PM->Event 17861: Bunchcounters in slot 5 (BC=112) and slot 7 (BC=18) disagree (L2B=2)<p>

From here on the run number is 166479<p>

<!-- Comment: Mon Jul 21 17;23;37 comment by...mauro -->
(MLE) b0cot10:Messenger:5:21:26 PM->Event 109559: Bunchcounters in slot 5 (BC=33) and slot 7 (BC=71) disagree (L2B=1)
<!-- Comment: Mon Jul 21 17;34;49 comment by...mauro -->
 (MLE) b0cot19:Messenger:5:32:36 PM->Event 276444: Bunchcounters in slot 5 (BC=115) and slot 7 (BC=12) disagree (L2B=0)
<!-- Comment: Mon Jul 21 17;39;07 comment by...mauro -->
(MLE) b0cot09:Messenger:5:37:22 PM->Event 346113: Bunchcounters in slot 2 (BC=59) and slot 4 (BC=3) disagree (L2B=1)
<!-- Comment: Mon Jul 21 17;42;54 comment by...mauro -->
(MLE) b0cot09:Messenger:5:40:46 PM->Event 394394: Bunchcounters in slot 2 (BC=62) and slot 4 (BC=65) disagree (L2B=2)
<!-- Comment: Mon Jul 21 17;47;59 comment by...mauro -->
 (MLE) b0cot09:Messenger:5:45:35 PM->Event 464382: Bunchcounters in slot 2 (BC=56) and slot 4 (BC=112) disagree (L2B=1)
<!-- Comment: Mon Jul 21 18;13;29 comment by...mauro -->
 (MLE) b0cot06:Messenger:6:11:22 PM->Event 834508: Bunchcounters in slot 6 (BC=86) and slot 7 (BC=83) disagree (L2B=2) 
<!-- Comment: Mon Jul 21 18;24;58 comment by...mauro -->
(MLE) b0wcal03:Messenger:6:23:15 PM->Event 988666: Bunchcounters in slot 5 (BC=130) and slot 19 (BC=3) disagree (L2B=2)

<!-- Comment: Mon Jul 21 18;49;52 comment by...mauro -->
(MLE) b0cot09:Messenger:6:47:58 PM->Event 1328504: Bunchcounters in slot 2 (BC=139) and slot 5 (BC=130) disagree (L2B=2)
<!-- Comment: Mon Jul 21 18;52;38 comment by...mauro -->
(MLE) b0cot18:Messenger:6:50:54 PM->Event 1365886: Bunchcounters in slot 4 (BC=18) and slot 7 (BC=71) disagree (L2B=1)
<!-- Comment: Mon Jul 21 18;55;20 comment by...mauro -->
(MLE) b0cot00:Messenger:6:53:39 PM->Event 1401318: Bunchcounters in slot 2 (BC=53) and slot 4 (BC=18) disagree (L2B=1)
<!-- Comment: Mon Jul 21 19;05;49 comment by...mauro -->
(MLE) b0cot19:Messenger:7:04:02 PM->Event 1538161: Bunchcounters in slot 5 (BC=59) and slot 11 (BC=74) disagree (L2B=1)
<!-- Comment: Mon Jul 21 19;18;29 comment by...mauro -->
(MLE) b0cot01:Messenger:7:16:39 PM->Event 1704099: Bunchcounters in slot 2 (BC=115) and slot 4 (BC=133) disagree (L2B=2)
<!-- Comment: Mon Jul 21 19;27;05 comment by...mauro -->
(MLE) b0cot19:Messenger:7:24:11 PM->Event 1793134: Bunchcounters in slot 5 (BC=3) and slot 18 (BC=115) disagree (L2B=1) 
<!-- Comment: Mon Jul 21 19;27;18 comment by...mauro -->
(MLE) b0cot09:Messenger:7:25:05 PM->Event 1802050: Bunchcounters in slot 2 (BC=74) and slot 5 (BC=65) disagree (L2B=1)
<!-- Comment: Mon Jul 21 19;27;28 comment by...mauro -->
(MLE) b0cot09:Messenger:7:25:05 PM->Event 1802050: Bunchcounters in slot 2 (BC=74) and slot 5 (BC=65) disagree (L2B=1)
<!-- Comment: Mon Jul 21 19;28;31 comment by...mauro -->
(MLE) b0cot00:Messenger:7:26:36 PM->Event 1813873: Bunchcounters in slot 2 (BC=9) and slot 4 (BC=68) disagree (L2B=2)
<!-- Comment: Mon Jul 21 19;49;18 comment by...mauro -->
(MLE) b0cot09:Messenger:7:42:02 PM->Event 2008140: Bunchcounters in slot 2 (BC=74) and slot 4 (BC=124) disagree (L2B=2)
<!-- Comment: Mon Jul 21 19;49;41 comment by...mauro -->
(MLE) b0cot10:Messenger:7:47:15 PM->Event 2073692: Bunchcounters in slot 5 (BC=74) and slot 7 (BC=71) disagree (L2B=0)
<!-- Comment: Mon Jul 21 20;45;21 comment by...mauro -->
(MLE) b0cot11:Messenger:8:43:16 PM->Event 2790264: Bunchcounters in slot 4 (BC=115) and slot 6 (BC=62) disagree (L2B=3)
<!-- Comment: Mon Jul 21 21;00;47 comment by...mauro -->
(MLE) b0cot01:Messenger:8:58:56 PM->Event 2995574: Bunchcounters in slot 2 (BC=15) and slot 6 (BC=133) disagree (L2B=3)
<!-- Comment: Mon Jul 21 21;18;46 comment by...mauro -->
(MLE) b0cot09:Messenger:9:14:52 PM->Event 3201172: Bunchcounters in slot 2 (BC=6) and slot 4 (BC=71) disagree (L2B=0)
<!-- Comment: Mon Jul 21 21;48;03 comment by...mauro -->
(MLE) b0cot09:Messenger:9:45:55 PM->Event 3582077: Bunchcounters in slot 2 (BC=71) and slot 5 (BC=80) disagree (L2B=1)
<!-- Comment: Mon Jul 21 22;24;43 comment by...mauro -->
(MLE) b0cot09:Messenger:10:22:51 PM->Event 4041277: Bunchcounters in slot 2 (BC=71) and slot 5 (BC=133) disagree (L2B=0)
<!-- Comment: Mon Jul 21 23;11;17 comment by...mauro -->
(MLE) b0cot00:Messenger:11:09:28 PM->Event 4568904: Bunchcounters in slot 2 (BC=21) and slot 3 (BC=118) disagree (L2B=1)
<!-- Comment: Mon Jul 21 23;39;30 comment by...mauro -->
(MLE) b0cot16:Messenger:11:37:26 PM->Event 4890947: Bunchcounters in slot 3 (BC=109) and slot 18 (BC=3) disagree (L2B=2)
<!-- Comment: Mon Jul 21 23;40;02 comment by...mauro -->
(MLE) b0cot19:Messenger:11:37:59 PM->Event 4894390: Bunchcounters in slot 5 (BC=112) and slot 8 (BC=71) disagree (L2B=2)
<!-- Comment: Mon Jul 21 23;42;48 comment by...mauro -->
(MLE) b0cot00:Messenger:11:41:00 PM->Event 4927195: Bunchcounters in slot 2 (BC=136) and slot 3 (BC=112) disagree (L2B=1)
<!-- Comment: Mon Jul 21 23;45;35 comment by...mauro -->
(MLE) b0cot16:Messenger:11:43:07 PM->Event 4948539: Bunchcounters in slot 3 (BC=118) and slot 13 (BC=3) disagree (L2B=1)
<!-- Comment: Tue Jul 22 00;05;03 comment by...Dan Ryan -->
b0wcal05:Messenger:11:59:38 PM->Runtime Error 1, Event 5132859: Bunch counter mismatch, mismatch count = 1
b0cot10:Messenger:12:01:18 AM->Event 1002187: Bunchcounters in slot 5 (BC=0) and slot 16 (BC=9) disagree (L2B=2) 
<!-- Comment: Tue Jul 22 00;07;16 comment by...Dan -->
b0cot00:Messenger:12:04:59 AM->Event 5189282: Bunchcounters in slot 2 (BC=74) and slot 5 (BC=18) disagree (L2B=0) 
<!-- Comment: Tue Jul 22 00;11;18 comment by...Dan ryan -->
b0cot00:Messenger:12:09:07 AM->Event 5230081: Bunchcounters in slot 2 (BC=112) and slot 5 (BC=53) disagree (L2B=3) 
<!-- Date: Mon Jul 21 16:19:45.836 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-8453&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-8453  width='300' align='top'></td><td>CEM gamma HV, praticularly NE arch, does not seem to be doing well since the series of power failures ~run 1638000.  Plot shows blpc e/p  by arch 2003</a></td></tr></table>
<!-- Author: Larry Nodulman -->
<!-- Comment: Mon Jul 21 16;23;33 comment by...Larry Nodulman -->
OF course overall time dependence has been (cough) fixed and tower gains put
in guarantee that arches average the same up to 163800 (one too many 0 above)


<!-- Date: Mon Jul 21 16:26:01.627 2003 -->
 (MLE) b0cot10:Messenger:4:16:41 PM->Event 523398: Bunchcounters in slot 5 (BC=118) and slot 8 (BC=56) disagree (L2B=1) <br> 
FERMI_TDC_NOT_READY_GLOBAL<br> 
cot10 shepherd manually
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472' target=_top>166472</a>) -->




<!-- Date: Mon Jul 21 16:30:31.649 2003 -->
<pre>Another 250 channels show up as bad in the COT calibration. Experts investigating. Please take one more COT calibration after the current store, then stop taking COT calibrations until further notice. 
 
 
                                     Bill</pre>
<!-- Author: Bill -->



<!-- Date: Mon Jul 21 16:36:11.645 2003 -->
<b>BUSY TIMEOUT</b><p> 
 
Error (31) >>   SCPU b0eb12 (forwarded by FER crate LEVEL1_GLOBAL_00). <br> 
Error (22) >>  SCPU-P7-E-TracerEventId: Event 724959, crate 135, channel 6 has either bad Tracer ID or bad markers around Tracer word.<br> 
 
  
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472' target=_top>166472</a>) -->
<!-- Comment: Mon Jul 21 18;21;32 comment by...mauro -->
(MLE) b0l3gate1.fnal.gov:main:6:16:53 PM->Host b0eb12.fnal.gov, task tRec_0
SCPU-P7-E-VrbHeader: Dump of header words for event 912343 from VRB in slot 14:
0x00000000 0x00001a68 0x00090009 0x03e003e0 0x03e003e0 0x03e003e0 0x03080000 0x00000000
<!-- Date: Mon Jul 21 16:42:27.320 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472">166472</A>
 Terminated at 2003.07.21 16:41:05
<!-- Author: RunControl -->


<!-- Date: Mon Jul 21 16:43:21.982 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472">166472</A>
TERMINATE: L3 frozen
<!-- Author: mauro *2080 -->



<!-- Date: Mon Jul 21 16:54:54.910 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8454&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8454' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8455&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8455' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8456&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8456' width='300'></a></center>
<!-- Author: Jason -->
<!-- Comment: Mon Jul 21 16;56;57 comment by...Jason -->
Hourly ACNet plots; note that LOSTP is ~2 times as large as usual (may be due to recent changes in the beam spot position), and the scale of B0RAT4 has been increased.

<!-- Date: Mon Jul 21 17:00:44.614 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166477">166477</A>
TERMINATE: errlog error
<!-- Author: mauro *2080 -->


<!-- Date: Mon Jul 21 17:06:33.605 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166478">166478</A>
TERMINATE: no data
<!-- Author: mauro *2080 -->


<!-- Date: Mon Jul 21 17:12:21.678 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479">166479</A>
 Activated at 2003.07.21 17:10:57
<!-- Author: RunControl -->


<!-- Date: Mon Jul 21 17:12:47.462 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479">166479</A>
ACTIVATE: physics run
<!-- Author: mauro *2080 -->







<!-- Date: Mon Jul 21 17:28:17.377 2003 -->
<pre>The Level3 Proxy was having trouble receiving  
communications from RunControl.   
 
After cleanup, Ace tried to start a new run, but RunControl  
was blocked from continuing due to the partition being  
spontaneously (and erroneously) close.  Unfortunately,  
this was not so clear from the error "Failed to fetch  
RunNumber".  (When a partition closes, you are locked out  
from fetching run numbers).  This problem is a symptom  
of trouble with the rtserver that runs on b0dau30 under  
the cdfdaq account.    
 
So I restarted rtserver on b0dau30, and the problem  
cleared up.   RunControl had to re-open the partition  
at this point. 
 
We'll try to figure out a way to make the system more  
robust;  this is difficult w.r.t the rtserver, as  
that is a commericial, closed-source product. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->
<!-- Comment: Mon Jul 21 17;56;07 comment by...Jason/Mauro -->
The Ace point of view (the full history):<br>
An auto HRR was issued, but L3 got stuck in the Recovered state (all other clients who were sent the Run transition obeyed it).  We aborted the run, but again, everything responded except L3.  We reset and tried to start a new run, but still L3 was stuck in the Recovered state.  <br><br>

After a little while, the L3 experts (Guillelmo & Alberto B.) showed up and cleaned up L3 (NOTE TO OTHER ACES: L3 must be in the Start state in order to cleanup; if you're having communication problems, you can send an Abort transition from the Ace Control Panel under the L3 Proxy Control).  We still couldn't start a new run due to this L3 communication problem, so we called Bill Badgett over.  <br><br>

Bill "viewed" the partitions and saw that partition 7 (which we thought was the active partition) had been closed.  This led him to suspect the rtserver, since it will close a partion if it doesn't receive an acknowledgement from a certain client after some time period.  So he restarted the rtserver, and L3 was now able to receive RC transition requests.<br><br>
We started a new run from the Start state.  In all, the downtime was 39 minutes.  Ugh.<br><br>

So, the moral of the story is that if RC gets stuck in some transition due to clients not responding to the transition request, one should "view" the partitions to see if the current partition has been closed erroneously (NOTE: if one hits "Select Partition", the current partition is closed no matter what, even if you hit "Cancel").  If the current partition is indeed closed, check to see that the rtserver is running; one can use the unix "netstat" command to see whether there are a number of disconnected sockets (indicating that the rtserver is not communicating properly).  If the rtserver is indeed in a wacky state, it should be killed and restarted.  Then one needs to open the partition again; then Partition->Config->Activate!



<!-- Date: Mon Jul 21 17:58:55.049 2003 -->
L3 filter node 032 turned red
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->
<!-- Comment: Mon Jul 21 18;58;47 comment by...mauro -->
L3 filter node 065 turned red
<!-- Date: Mon Jul 21 18:00:19.837 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8457&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8457' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8458&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8458' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8459&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8459' width='300'></a></center>
<!-- Author: Jason -->
<!-- Comment: Mon Jul 21 18;03;29 comment by...Jason -->
Hourly ACNet plots; note the scale on B0RAT4.  Also, B0ABSM (abort gap pbar losses) are larger than normal by ~30%.








<!-- Comment: Mon Jul 21 18;27;24 comment by...Jason -->
Checking back to July 13, B0ABSM looks the same for the luminosity in this plot.





<!-- Date: Mon Jul 21 18:53:17.740 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8460&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8460' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8461&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8461' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8462&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8462' width='300'></a></center>
<!-- Author: Jason -->
<!-- Comment: Mon Jul 21 18;53;57 comment by...Jason -->
Hourly ACNet plots; note the spike in B0PBSM (middle plot, yellow variable).






<!-- Date: Mon Jul 21 19:12:18.668 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472">166472</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
IMU BMU HV channel bad
</PRE>
<!-- Author: cdfscico -->











<!-- Date: Mon Jul 21 19:34:43.150 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8463&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8463' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8464&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8464' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8465&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8465' width='300'></a></center>
<!-- Author: Jason -->











<!-- Date: Mon Jul 21 20:36:53.879 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8466&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8466' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8467&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8467' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8468&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8468' width='300'></a></center>
<!-- Author: Jason -->
<!-- Comment: Mon Jul 21 20;37;36 comment by...Jason -->
Hourly ACNet plots; note the spike in B0PBSM at 19:45.









<!-- Date: Mon Jul 21 21:34:32.006 2003 -->
IMon: ISL DVDD B0W4L0 was pink again (just barely over the warning level).  This is the third time this store.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->



<!-- Date: Mon Jul 21 21:40:00.285 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8469&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8469' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8470&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8470' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8471&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8471' width='300'></a></center>
<!-- Author: Jason -->




<!-- Date: Mon Jul 21 22:23:07.099 2003 -->
During the shift we've collected few (reformatter) errors on the data flow<br>  
nodes: 32, 50, 65, 115, 244, 253, 267
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->





<!-- Date: Mon Jul 21 22:34:28.912 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8472&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8472' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8473&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8473' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8474&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8474' width='300'></a></center>
<!-- Author: Jason -->


<!-- Date: Mon Jul 21 22:36:07.416 2003 -->
Timeout on global TDC Done <br> 
b0cot10:Messenger:10:31:35 PM->Runtime Error 4, Event 4147270: Timeout on global TDC Done[MLE]<br> 
shepherd manually.
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->


<!-- Date: Mon Jul 21 23:07:06.678 2003 -->
b0pcal08 red in the VxWorks system monitor: Heap Corrupted.<br> 
Halt the run shepherd the crate and recover->run. Back to normal.
<!-- Author: mauro/jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->




<!-- Date: Mon Jul 21 23:34:23.363 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8475&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8475' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8476&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8476' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8477&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8477' width='300'></a></center>
<!-- Author: Jason -->








<!-- Date: Mon Jul 21 23:53:48.137 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030><b>DONE TIME OUT CHALLENGE:</b><br> 
SILICON L1 DTO's vs. REST OF THE DTO's<br> 
...and the winner is SILICON L1 DTO !!! 
(32 to 31)</td></tr></td></table>
<!-- Author: mauro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->

<!-- Date: Mon Jul 21 23:56:32.252 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166472>
166472</A> 
x28A48</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [5,319,391]</TD>                                       
<TD ALIGN=CENTER>15:38:38</TD>                                          
<TD ALIGN=CENTER>16:41:05</TD>                                          
<TD ALIGN=CENTER>00:45:36</TD>                                          
<TD ALIGN=RIGHT>     37,126,678</TD>                                    
<TD ALIGN=RIGHT>       724,966</TD>                                     
<TD ALIGN=RIGHT>     153,704</TD>                                       
<TD ALIGN=RIGHT>   83.284</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479>
166479</A> 
x28A4F</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [5,319,391]</TD>                                       
<TD ALIGN=CENTER>17:10:57</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>06:12:09</TD>                                          
<TD ALIGN=RIGHT>    289,596,592</TD>                                    
<TD ALIGN=RIGHT>     5,066,989</TD>                                     
<TD ALIGN=RIGHT>   1,085,877</TD>                                       
<TD ALIGN=RIGHT>  493.249</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:02</TD>                                                       
<TD ALIGN=CENTER>06:57:46</TD>                                          
<TD ALIGN=RIGHT>     326,723,270</TD>                                   
<TD ALIGN=RIGHT>      5,791,955</TD>                                    
<TD ALIGN=RIGHT>     1,239,581</TD>                                     
<TD ALIGN=RIGHT>  576.533</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Mon Jul 21 23:58:04.412 2003 -->
<font color=red>Shift Summary: </font><pre>SHIFT SUMMARY: 
============== 
16:00 Store 2810 and run 166472 just started. 
      Lum = 32 
 
Had trouble with L3 losing connection to the RT server.  It took 
a while to realize that this was the problem (see log entry 
by Bill B.)  Restarted RT server, and got running again. 
Lost ~40mins of beam. 
 
Low rate of L1Done timeouts due to 2 SRC mode.  These are 
likely L2 timeouts, but show up as L1Done now.  Steve has 
been investigating, but we need to look at the TS firmware 
to make sure we understand. 
 
21:13 Pbar stack lost due to interlock failure.  Still not stacking as of shift end.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   166472, 166473
Delivered Luminosity   652.3  
Acquired Luminosity    535.5  
Efficiency             82.1
   
</td></tr></table></pre>
<!-- Author: Colin -->


<!-- Date: Tue Jul 22 00:03:51.402 2003 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2003.07.21</td><td>23:54:53</td><td>W Inner BLM</td><td>306.70</td><td>RADS</td></tr> 
<tr><td>2003.07.21</td><td>23:54:53</td><td>W Outer BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2003.07.21</td><td>23:54:53</td><td>E Inner BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2003.07.21</td><td>23:54:53</td><td>E Outer BLM</td><td>232.66</td><td>RADS</td></tr> 
</TABLE> 
<strong>Integrated dosage</strong> 
  
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=166479' target=_top>166479</a>) -->


<!-- Date: Tue Jul 22 00:04:55.107 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8478&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8478' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-8479&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-8479' width='300'></a></center>
<!-- Author: Jason -->