<!-- Title: Day shift. Fri Dec 15, 2006 -->
<!-- SciCo: Roberto Carosi -->
<!-- DAQAce: Nagesh Kulkarni -->
<!-- MonAce: Nate Goldschmidt -->
<!-- CO: Chris Neu -->
<!-- OpManager: Tom Wright -->
<!-- Notes: -->
<pre>Continuing store 5127, luminosity 76.5 e30</pre>


<!-- Date: Fri Dec 15 08:22:23.629 2006 -->
Done TO on b0imu01. 
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->


<!-- Date: Fri Dec 15 08:38:34.618 2006 -->
<p>I see no entry here, so I'll make one now.  I just learned 
Mark and Jim are rebooting the cryo CDFSx PCs one by one, so we 
will see various tags come and go.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->
<!-- Comment: Fri Dec 15 08;41;36 comment by...Steve Hahn -->
<p>I said what but I didn't say why: they are seeing problems
accessing the correct "pictures" (*.GRF files) from the fileserver
cdf-fs1.  The reboots are to correct that problem.  Already, I 
see that some of the tags that were not there have returned to
the sign display on VNODE2.</p>


<!-- Date: Fri Dec 15 09:03:52.827 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16157&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16157' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16158&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16158' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16159&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16159' width='300'></a></center>
<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 09:17:30.908 2006 -->
Done TO. for crate/s:  
    b0htdc00   
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->


<!-- Date: Fri Dec 15 09:18:53.641 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16163&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16163' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16164&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16164' width='300'></a></center>
<!-- Author: neu -->
<!-- Comment: Fri Dec 15 09;20;38 comment by...neu -->
<pre>Data link between XTC and the Stereo finder apears to have a problem in finder slot 18.  Instructions on paging experts are not complete - I will email the XFT guys.  </pre>


<!-- Comment: Sat Dec 16 05;42;50 comment by...Nils -->
That looks like a single event error to me.  We need to do something about those not firing TrigMon.
<!-- Date: Fri Dec 15 09:47:11.067 2006 -->
RXPT Error for crate/s:  
    b0fcal00   
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->


<!-- Date: Fri Dec 15 09:53:41.349 2006 -->
EVB2 was red. HRR fixed it.
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->


<!-- Date: Fri Dec 15 09:58:59.928 2006 -->
<p>It looks like rebooting VNODE1 one more time after the reboots 
of the cryo computers has fixed all the problems.  Only some  
heartbeats are still missing; I expect them to come back in the 
next half hour of so.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->


<!-- Date: Fri Dec 15 10:03:39.103 2006 -->
Done timeout for b0cot16. Trying to shepherd crate for second time.
<!-- Author: Nagesh and Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419' target=_top>230419</a>) -->
<!-- Comment: Fri Dec 15 10;07;50 comment by...Nate -->
second shepherding of b0cot16 didn't work. ending run and starting another. L3 was taking a long time in the end transition (over 2.5minutes) Error handler reports a L3 relay failure.
<!-- Comment: Fri Dec 15 10;09;58 comment by...Nate -->
reset DAQ; cleaning up L3 and evb as per instructions <a href="http://www-cdfonline.fnal.gov/ops/evbl3shift/troubleshoot.html#relay">here</a>
<!-- Comment: Fri Dec 15 10;18;46 comment by...Nate -->
AFTER cleaning up L3 and EVB, could not partition. Event builder proxy was red inb the "Component Status" window. Cleaned up EVB -AGAIN- as described <a href="http://www-cdfonline.fnal.gov/ops/evbl3shift/troubleshoot.html#evbdaqmon">here</a>. Managed to partition and coldstart. 
<!-- Comment: Fri Dec 15 10;19;23 comment by...Nate -->
Immeadiate done timeout from b0cot16 again. Paging TDC expert. 
<!-- Comment: Fri Dec 15 11;33;00 comment by...Conor -->
The Level3 'relay failure' was because one of the nodes crashed. This was unrelated to the cot16 Done TOs; it was ending the run that brought up the L3 relay failure. This was what was responsible for the slow transition. Its okay to clean up Level3, but usually its not needed - most of the time you can just try to start a new run and it will work. And by the way, the instructions for L3 relay failure do not say to clean up evb...
<!-- Comment: Fri Dec 15 12;06;59 comment by...Nate -->
<table border=0><td><tr><td bgcolor=#e0e030>Actually, documentation DOES say to clean up L3 AND evb, under "If L3 gets stuck when you are ending the run". </td></tr></td></table>
<!-- Comment: Fri Dec 15 13;31;09 comment by...Conor -->
Yeah, okay, you're right, the instructions do say that if you have to 'manually over-ride the RunControl state manager', you should clean up evb.
<!-- Date: Fri Dec 15 10:04:30.515 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419>230419</A>
 Terminated at 2006.12.15 10:04:17
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 10:05:17.960 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230419>230419</A>
TERMINATE: COT 16 problems
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 10:08:59.158 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230424>230424</A>
TERMINATE: no data; l3 relay failure
<!-- Author: Nagesh -->




<!-- Date: Fri Dec 15 10:12:31.779 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16165&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16165' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16166&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16166' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16167&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16167' width='300'></a></center>
<!-- Author: neu -->
<!-- Comment: Fri Dec 15 10;14;53 comment by...neu -->
<pre>Several ObjectMon plots have red histograms that do not match their references:
 - MET in ObjectMon #29
 - CMU, CMP deltaX, Q/pT, CMX Z stub in ObjectMon #97
 - PHA, WHA corr time in ObjectMon #2

The experts have been emailed per ObjectMon instructions.</pre>

<!-- Date: Fri Dec 15 10:18:45.803 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230430>230430</A>
 Activated at 2006.12.15 10:18:13
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 10:18:46.938 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230430>230430</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Nagesh -->




<!-- Date: Fri Dec 15 10:24:19.682 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230430>230430</A>
 Terminated at 2006.12.15 10:24:05
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 10:24:26.315 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230430>230430</A>
TERMINATE: no data, b0cot16
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 10:25:29.799 2006 -->
Error messages from b0cot16: 
 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:09 AM->TDC evt 4 slot 8 static/fifo: ModID 210/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Invalid SRAM Header evt 4 slot 9: exp/fnd mod 282/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Header Mismatch evt 4 slot 9: static/fifo 0x8D080E76/0x00000000 [004,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC evt 4 slot 9 static/fifo: ModID 282/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Invalid SRAM Header evt 4 slot 10: exp/fnd mod 281/0 l2b 3/0 00000000 
(MLE) b0felix.fnal.gov:TEDCpp:Dec 15, 2006 10:23:09 AM->{ 10:23:9, 12/15/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Header Mismatch evt 4 slot 10: static/fifo 0x8C880E76/0x00000000 [004,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC evt 4 slot 10 static/fifo: ModID 281/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Invalid SRAM Header evt 4 slot 11: exp/fnd mod 280/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Header Mismatch evt 4 slot 11: static/fifo 0x8C080E76/0x00000000 [004,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC evt 4 slot 11 static/fifo: ModID 280/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:10 AM->TDC Invalid SRAM Header evt 4 slot 12: exp/fnd mod 485/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Header Mismatch evt 4 slot 12: static/fifo 0xF2880E76/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC evt 4 slot 12 static/fifo: ModID 485/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Invalid SRAM Header evt 4 slot 13: exp/fnd mod 484/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Header Mismatch evt 4 slot 13: static/fifo 0xF2080E76/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC evt 4 slot 13 static/fifo: ModID 484/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Invalid SRAM Header evt 4 slot 14: exp/fnd mod 483/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Header Mismatch evt 4 slot 14: static/fifo 0xF1881076/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC evt 4 slot 14 static/fifo: ModID 483/0 L2B 2/0 Hits 16/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Invalid SRAM Header evt 4 slot 15: exp/fnd mod 482/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:11 AM->TDC Header Mismatch evt 4 slot 15: static/fifo 0xF1081076/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC evt 4 slot 15 static/fifo: ModID 482/0 L2B 2/0 Hits 16/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC Invalid SRAM Header evt 4 slot 16: exp/fnd mod 481/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC Header Mismatch evt 4 slot 16: static/fifo 0xF0880E76/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC evt 4 slot 16 static/fifo: ModID 481/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC Invalid SRAM Header evt 4 slot 17: exp/fnd mod 480/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC Header Mismatch evt 4 slot 17: static/fifo 0xF0080E76/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC evt 4 slot 17 static/fifo: ModID 480/0 L2B 2/0 Hits 14/0 BC 118/0 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC Invalid SRAM Header evt 4 slot 18: exp/fnd mod 479/0 l2b 3/0 00000000 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC Header Mismatch evt 4 slot 18: static/fifo 0xEF880E76/0x00000000 [000,000] 
(MLE) b0cot16:Messenger:Dec 15, 2006 10:23:12 AM->TDC evt 4 slot 18 static/fifo: ModID 479/0 L2B 2/0 Hits 14/0 BC 118/0
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230430' target=_top>230430</a>) -->


<!-- Date: Fri Dec 15 10:26:37.719 2006 -->
Eric James (TDC expert) called, and said he'd look at the error logs for b0cot16, will call back in a few minutes. He says we'll probably have to powercycle the crate. We're waiting for his call.
<!-- Author: Nate and Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230430' target=_top>230430</a>) -->

<!-- Date: Fri Dec 15 10:34:12.741 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435>230435</A>
 Activated at 2006.12.15 10:33:32
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 10:34:13.623 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435>230435</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 10:37:29.014 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>Running again. Sequence of actions that seemed to fix these problems with b0cot16: end run and reset; powercycle b0cot16 from iFix (RACKS button on Process Systems window); clean up event builder; start a new run.</td></tr></td></table>
<!-- Author: Nate and Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435' target=_top>230435</a>) -->
<!-- Comment: Fri Dec 15 10;40;08 comment by...Nate -->
Eric mentioned that he doesn't know exactly what the problem was in this instance, other than it doesn't appear to have been a TDC problem. However, he's had success with the voodoo of powercycling the cot crate and then cleaning up event builder.


<!-- Date: Fri Dec 15 10:44:34.036 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16168&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16168' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16169&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16169' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16170&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16170' width='300'></a></center>
<!-- Author: Nagesh -->

<!-- Date: Fri Dec 15 10:47:17.367 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230437>230437</A>
ACTIVATE: RunSet: DAQTEST c3po in
<!-- Author: ag -->


<!-- Date: Fri Dec 15 10:47:32.608 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230437>230437</A>
TERMINATE: c3po out
<!-- Author: ag -->



<!-- Date: Fri Dec 15 10:48:41.659 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16174&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16174' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16175&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16175' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16176&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16176' width='300'></a></center>
<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 10:54:19.914 2006 -->
Solenoid alarm. Halted the run. B field inhibit. 
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435' target=_top>230435</a>) -->
<!-- Comment: Fri Dec 15 10;56;38 comment by...Nagesh -->
COT and TOF inhibits.

<!-- Date: Fri Dec 15 10:58:09.947 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435>230435</A>
 Terminated at 2006.12.15 10:57:59
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 10:58:31.579 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435>230435</A>
TERMINATE: Solenoid problems. 
<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 11:15:13.884 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230438>230438</A>
 Activated at 2006.12.15 11:14:16
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 11:15:21.242 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230438>230438</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 11:19:35.398 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230438>230438</A>
 Terminated at 2006.12.15 11:19:21
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 11:20:05.241 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230438>230438</A>
TERMINATE: no data, b0cot16 done timeouts
<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 11:25:05.176 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230445>230445</A>
 Activated at 2006.12.15 11:24:47
<!-- Author: RunControl -->



<!-- Date: Fri Dec 15 11:29:08.773 2006 -->
b0cot16-> Attached TCP/IP interface to wancom unit 0 
 
Attaching interface lo0...done 
 
Unable to add route to 131.225.237.0; errno = 0xffffffff.  
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230445' target=_top>230445</a>) -->
<!-- Comment: Fri Dec 15 11;29;50 comment by...Nate -->
unusual message seen during shepherding

<!-- Date: Fri Dec 15 11:31:31.706 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230445>230445</A>
 Terminated at 2006.12.15 11:30:59
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 11:31:32.713 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230445>230445</A>
TERMINATE: b0cot16 problems
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 11:34:59.789 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230447>230447</A>
 Activated at 2006.12.15 11:34:43
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 11:35:04.668 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230447>230447</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 11:36:16.348 2006 -->
   
  RXPT Error for crate/s:  
    b0cot16     
<!-- Author: nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230447' target=_top>230447</a>) -->
<!-- Comment: Fri Dec 15 11;37;12 comment by...n -->
  RXPT Error for crate/s: 
    b0cot16    Done TO. 1 crate/s: b0cot16   
<!-- Date: Fri Dec 15 11:36:33.070 2006 -->
Run was activated after one more powercycling. But we got cot16 problem again. Run halted now.
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230447' target=_top>230447</a>) -->
<!-- Comment: Fri Dec 15 11;37;17 comment by...Nagesh -->
still problems with COT16 

<!-- Date: Fri Dec 15 11:37:32.648 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230447>230447</A>
 Terminated at 2006.12.15 11:37:21
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 11:38:08.407 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230447>230447</A>
TERMINATE: DONE timeouts are back

<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 11:44:26.068 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230450>230450</A>
 Activated at 2006.12.15 11:43:47
<!-- Author: RunControl -->



<!-- Date: Fri Dec 15 11:44:56.575 2006 -->
Sasha, DAQ expert, says he'll be in the control room in a few minutes.
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230450' target=_top>230450</a>) -->
<!-- Comment: Fri Dec 15 11;46;19 comment by...Nate -->
Sasha is here

<!-- Date: Fri Dec 15 12:04:33.133 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230450>230450</A>
 Terminated at 2006.12.15 12:04:30
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 12:04:50.383 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230450>230450</A>
TERMINATE: end the run to take cot16 out
<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 12:05:12.049 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230450>230450</A>
TERMINATE: end the run to take cot16 out
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 12:08:17.143 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230456>230456</A>
ACTIVATE: RunSet: DAQTEST 
<!-- Author: ag, x2080 -->



<!-- Date: Fri Dec 15 12:17:14.855 2006 -->
Fastime plot of B0ILIM, etc. (in window GxSC), is not updating. E11 window shows the message "FTP timeout ACNET_UTIME"
<!-- Author: Nate -->
<!-- Comment: Fri Dec 15 12;22;21 comment by...Nate -->
read old elog entries which suggest there are "too many clients" requesting these fast-time plot variables. Closed fast-time plot window of B0ILUM, etc. at "old" mon ace station. STILL can't get B0ILUM, etc. to draw at new single-ace station. 

<!-- Date: Fri Dec 15 12:24:28.804 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459>230459</A>
 Activated at 2006.12.15 12:23:46
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 12:24:29.441 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459>230459</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 12:28:05.672 2006 -->
We're back. Sasha, DAQ expert, will make an entry describing the voodoo he used to fix whatever problem(s) we have had.
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459' target=_top>230459</a>) -->


<!-- Date: Fri Dec 15 12:36:05.460 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre> 
DAQ was paged about cot16:  
Done Timeouts with numerours errors in multiple slots... 
 
0. The crate has been power cycled before the page. 
1. put crate in the local readout mode 
2. marked half of the cards offline 
    -&gt; errors persist 
3. the other half  
    -&gt; errors persist 
4, readout the crate cot16 in the separate partition 
    using software event builder 
    -&gt;  no errors 
5. redout using hardware event builder 
    -&gt; no errors 
6. put back into the regualr partition 
    -&gt; no errors 
 
Normal data-taking.  The problem appears to be solved.  
Will leave it in the local readout mode until the end 
 of the store. 
 
TOM, PLEASE, DON'T FORGET TO PUT IT BACK INTO  
THE SPY MODE LATER. 
 
Done. 
 
 </pre></td></tr></td></table>
<!-- Author: Sasha Golossanov :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459' target=_top>230459</a>) -->


<!-- Date: Fri Dec 15 12:52:13.442 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16177&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16177  width='300' align='top'></a></td><td>instability in B0PAGC, correlated w AGIGI2. Seems to have subsided. </td></tr></table>
<!-- Author: Nate -->


<!-- Date: Fri Dec 15 12:52:45.171 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16178&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16178  width='300' align='top'></a></td><td> </td></tr></table>
<!-- Author: neu -->
<!-- Comment: Fri Dec 15 12;54;34 comment by...neu -->
<pre>Looks like one complete CAT TDC is malfunctioning.  Heather G. 
tracked this one down to b0cot16 slot 10 - a TDC in the crate that 
has been giving problems today.  </pre>
<!-- Date: Fri Dec 15 12:53:15.805 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459>230459</A>
 Terminated at 2006.12.15 12:53:08
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 12:53:47.862 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459>230459</A>
TERMINATE: Card 10 of COT_16 possibly taken out of readout - put it back

<!-- Author: Nagesh -->



<!-- Date: Fri Dec 15 12:57:35.967 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230460>230460</A>
 Activated at 2006.12.15 12:57:04
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 12:57:36.688 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230460>230460</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] With TDC in slot 10 of COT_16 back in the readout

<!-- Author: Nagesh -->
<!-- Comment: Fri Dec 15 12;58;39 comment by...twright -->
The crate itself is still in local DONE mode



<!-- Date: Fri Dec 15 13:01:06.039 2006 -->
MCR rebooted some machine, and we aren't updating B0PAGC and B0AAGC. They say it will be back in a few minutes, and it is.
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230460' target=_top>230460</a>) -->

<!-- Date: Fri Dec 15 13:02:22.023 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230460>230460</A>
 Terminated at 2006.12.15 13:01:19
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 13:02:23.046 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230460>230460</A>
TERMINATE: TDC in slot 10 fell out of the readout again somehow

<!-- Author: Nagesh -->


<!-- Date: Fri Dec 15 13:07:04.529 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461>230461</A>
 Activated at 2006.12.15 13:06:53
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 13:07:56.039 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461>230461</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] Restart run control and see if slot 10 TDC will stick in the readout
<!-- Author: Nate x2080 -->
<!-- Comment: Fri Dec 15 13;35;13 comment by...twright -->
This seems to have worked, finally.  The hole in SL5 is gone in the YMon plot now.

<!-- Date: Fri Dec 15 13:22:15.464 2006 -->
b0fib07 turned red in replies and acknowledgements window during data taking. According to procedure, tried to ping fib crate, unsuccesful. Reset and shephearded crate. SiExpert noticed that RC didn't automatically go to halt. Halted run. Shepheard succesful. 
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461' target=_top>230461</a>) -->
<!-- Comment: Fri Dec 15 13;30;51 comment by...Donatella T. -->
there were NO error messages received by the Error Handler from b0fib07 in this particula occasion, so NO actions were taken.




<!-- Date: Fri Dec 15 13:53:11.021 2006 -->
checked hourly plots
<!-- Author: n :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461' target=_top>230461</a>) -->

<!-- Date: Fri Dec 15 13:58:40.934 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435>230435</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=230435 target=_runSummary>230435</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->



<!-- Date: Fri Dec 15 14:38:19.999 2006 -->
Done TO. for crate/s:  
    b0pcal05   
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461' target=_top>230461</a>) -->


<!-- Date: Fri Dec 15 14:53:40.487 2006 -->
GAS alarm; cryo tech silenced
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461' target=_top>230461</a>) -->


<!-- Date: Fri Dec 15 14:58:42.085 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16180&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16180  width='300' align='top'></a></td><td>L1 Dirac CAL TOT occupancies high in wedges 9, 14.  According 
to Vadim this is due to the 2 masked XTCs into the XFT documented on the CO Known problems page.  </td></tr></table>
<!-- Author: neu -->


<!-- Date: Fri Dec 15 15:25:08.734 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>The TOF pager is not available till noon tomorrow.  
Please call at 765-714-7878 in case of any problems with TOF. Thanks! 
 
</pre></td></tr></td></table>
<!-- Author: Niharika -->

<!-- Date: Fri Dec 15 15:28:02.376 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230435>230435</A>
monitors: <tt>Run 230435 was a short run taken during the morning when there were various problems. &nbsp;Several monitors were not running - hence they are <br>
marked with the ambiguous "not yet" in the CO checklist.</tt>
<!-- Author: Consumer Operator( Christopher Neu) -->



<!-- Date: Fri Dec 15 15:29:56.419 2006 -->
Automatic hourly plots seem to have stopped as of 13:00.
<!-- Author: Nate :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230461' target=_top>230461</a>) -->


<!-- Date: Fri Dec 15 15:33:35.909 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16181&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16181' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16182&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16182' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16183&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16183' width='300'></a></center>
<!-- Author: Nate -->

<!-- Date: Fri Dec 15 15:53:30.387 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459>230459</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=230459 target=_runSummary>230459</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->


<!-- Date: Fri Dec 15 15:54:18.457 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230459>230459</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
COT 
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Fri Dec 15 15:55:27.023 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230419 target=_runSummary>230419</A> x38413</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>01:19:02</TD>
<TD ALIGN=CENTER>10:04:17</TD>
<TD ALIGN=CENTER>08:03:36</TD>
<TD ALIGN=RIGHT>   3,189,509</TD>   <TD ALIGN=RIGHT>  3,221.19</TD>
<TD ALIGN=RIGHT>  2,948.69</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230435 target=_runSummary>230435</A> x38423</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>10:33:32</TD>
<TD ALIGN=CENTER>10:57:59</TD>
<TD ALIGN=CENTER>00:17:13</TD>
<TD ALIGN=RIGHT>      73,672</TD>   <TD ALIGN=RIGHT>     92.83</TD>
<TD ALIGN=RIGHT>     64.93</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230447 target=_runSummary>230447</A> x3842F</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>11:34:43</TD>
<TD ALIGN=CENTER>11:37:21</TD>
<TD ALIGN=CENTER>00:00:57</TD>
<TD ALIGN=RIGHT>       3,591</TD>   <TD ALIGN=RIGHT>     10.41</TD>
<TD ALIGN=RIGHT>      3.45</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230459 target=_runSummary>230459</A> x3843B</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>12:23:46</TD>
<TD ALIGN=CENTER>12:53:08</TD>
<TD ALIGN=CENTER>00:27:01</TD>
<TD ALIGN=RIGHT>     130,202</TD>   <TD ALIGN=RIGHT>     98.02</TD>
<TD ALIGN=RIGHT>     88.89</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230460 target=_runSummary>230460</A> x3843C</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>12:57:04</TD>
<TD ALIGN=CENTER>13:01:19</TD>
<TD ALIGN=CENTER>00:04:01</TD>
<TD ALIGN=RIGHT>      15,730</TD>   <TD ALIGN=RIGHT>     15.33</TD>
<TD ALIGN=RIGHT>     12.94</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230461 target=_runSummary>230461</A> x3843D</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>13:06:53</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>02:34:19</TD>
<TD ALIGN=RIGHT>     835,216</TD>   <TD ALIGN=RIGHT>    497.09</TD>
<TD ALIGN=RIGHT>    458.25</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>11:27:09</TD>
<TD ALIGN=RIGHT>     4,247,920</TD> <TD ALIGN=RIGHT>  3,934.87</TD>
<TD ALIGN=RIGHT>  3,577.15</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Fri Dec 15 15:56:45.772 2006 -->
<font color=red>Shift Summary: </font><pre>Store 5127, initial lum 76.5 
 
10:05 terminate run 230419, COT 16 problems. 
      TDC expert paged. After powercycling the 
      crate, everything ok. 
 
10:35 Run 230435 started. 
 
10:54 Solenoid alarm. Run terminated. 
 
11:15 Solenoid ok, but COT 16 problems again. TDC expert 
      paged, then DAQ expert paged. 
11:55 TDC paged again. DAQ expert investigating. 
 
12:28 Run 230459 started, COT16 problem fixed. (but it is 
      not clear what the problem was; aces should make 
      sure that the downtime logger is kept up to date).</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   230419-230461
Delivered Luminosity   73.3 nb-1  
Acquired Luminosity    1106.4 nb-1  
Efficiency             66.1%
   
</td></tr></table></pre>
<!-- Author: Roberto Carosi -->