<!-- Title: Eve shift. Tue Mar 20, 2007 -->
<!-- SciCo: Shinhong Kim -->
<!-- DAQAce: Rob Forrest -->
<!-- MonAce: &nbsp; -->
<!-- CO: Kihyeon Cho -->
<!-- OpManager: Farrukh Azfar -->
<!-- Notes: -->
<pre>Continueing collision run 236965 store 5292 at B0ILUM=69E30.</pre>

<!-- Date: Tue Mar 20 16:11:16.645 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236971>236971</A>
ACTIVATE: RunSet: DAQTEST 
FILAR timing test
<!-- Author: andrew -->



<!-- Date: Tue Mar 20 16:12:29.110 2007 -->
fcall00 timeout. Shepherded successfully.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->

<!-- Date: Tue Mar 20 16:24:21.228 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236972>236972</A>
ACTIVATE: RunSet: DAQTEST 
FILAR timing test
<!-- Author: andrew -->


<!-- Date: Tue Mar 20 16:46:57.597 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965>236965</A>
general: <tt>CO Checklist done.</tt>
<!-- Author: Consumer Operator( Kihyeon Cho) -->


<!-- Date: Tue Mar 20 17:15:47.187 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236985>236985</A>
ACTIVATE: RunSet: DAQTEST 
FILAR timing test
<!-- Author: andrew -->



<!-- Date: Tue Mar 20 18:41:34.246 2007 -->
Done timeout pcal10. Shepherded.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->


<!-- Date: Tue Mar 20 18:43:06.212 2007 -->
The internet connection to the outside world looks to be down.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->


<!-- Date: Tue Mar 20 19:00:49.275 2007 -->
Filesystem is red in ProcMon. It  complains of being slow and timing out. The disks this process checks are not full, so maybe it is realted to the network problems.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->
<!-- Comment: Tue Mar 20 21;08;33 comment by...jj -->
The network problem (disconnect from rest of lab/world) should
not cause Filesystem check to go red.  But this has happened 
twice now so we clearly do not understand the connection.  I
have asked jdl/bb/fc/rlc to investigate.
<!-- Date: Tue Mar 20 19:01:10.312 2007 -->
clc00 done timeout. Auto HRR fixed.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->
<!-- Comment: Tue Mar 20 19;48;39 comment by...rob -->
again
<!-- Date: Tue Mar 20 19:25:15.005 2007 -->
ENCP to TAPE is red in Procmon. 
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->
<!-- Comment: Wed Mar 21 12;45;21 comment by...Willis -->
The "RED" part should be changed. The CSL has about a 3 day local
buffer for data files. Any data transfer or copy problem to the
Enstore does not need immediate shift crew action. The data
transfer system is set up to queue up data files and move them
when the "blockage" is fixed. Because the CSL has a large data
file buffer, problems can be fixed the next day. 
<!-- Date: Tue Mar 20 20:31:55.994 2007 -->
Called Helpdesk 2345 to ask the time estimate for recovering the present network down.  The answer is that there is no time estimate though expert is working on this problem. 
<!-- Author: shinhong kim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->


<!-- Date: Tue Mar 20 20:51:43.971 2007 -->
Network connection has recovered.
<!-- Author: shinhong kim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->

<!-- Date: Tue Mar 20 20:56:03.192 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965>236965</A>
general: <tt>CO Checklist done.</tt>
<!-- Author: Consumer Operator( Kihyeon Cho) -->



<!-- Date: Tue Mar 20 21:03:16.150 2007 -->
Network problem appears to have been uplink between CDF core 
router and FCC (site) hub router. I am attempting to get more 
information on the problem and fix.
<!-- Author: jj -->
<!-- Comment: Tue Mar 20 21;26;16 comment by...jj -->
Note in HELPDESK report query says....<br>
"Replaced failed 10GE transceiver in FCC hub switch."
<!-- Comment: Tue Mar 20 21;42;28 comment by...JJ -->
Transceiver was replaced by Vyto G but David T was on his 
white horse and galloping in as well!  <br>
I also note that Shinhong complained that instructions
for dealing with network problems were not very clear
in documentation for shift crew.  I am working on this 
but there are some notes under "Networking Help and Tools"
linked from ace help page.
<!-- Date: Tue Mar 20 21:07:22.654 2007 -->
Icicle heartbeat stopped. Restarted. Also PSM heartbeat has been down for some time although all other PSM interfaces are responsive.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->


<!-- Date: Tue Mar 20 21:08:21.373 2007 -->
done timeout cot06. Auto HRR fixed.
<!-- Author: rob -->



<!-- Date: Tue Mar 20 21:25:34.060 2007 -->
b0svx02: Sl 6 Resonance Detected -- Same as last night.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->



<!-- Date: Tue Mar 20 21:27:14.540 2007 -->
L2 Timeout errors. No indication of what is wrong yet.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965' target=_top>236965</a>) -->
<!-- Comment: Tue Mar 20 21;34;24 comment by...rob -->
I don't know what is causing these L2TO errors in RC, restarting the run.
<!-- Date: Tue Mar 20 21:32:20.283 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965>236965</A>
 Terminated at 2007.03.20 21:32:03
<!-- Author: RunControl -->


<!-- Date: Tue Mar 20 21:32:21.822 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965>236965</A>
TERMINATE: done
<!-- Author: Ford -->



<!-- Date: Tue Mar 20 21:36:28.128 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237002>237002</A>
 Activated at 2007.03.20 21:36:11
<!-- Author: RunControl -->


<!-- Date: Tue Mar 20 21:36:29.454 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237002>237002</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [5,866,678] 
<!-- Author: rob -->



<!-- Date: Tue Mar 20 21:38:54.251 2007 -->
Same problems, in starting this run. Already paged L2 a few minutes ago. I think this is a problem with the pulsar similar to earlier today.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237002' target=_top>237002</a>) -->
<!-- Comment: Tue Mar 20 21;45;57 comment by...rob -->
Got this error when restarting run: L2PULSAR_NODELOST!

<!-- Comment: Tue Mar 20 21;47;16 comment by...rob -->
Called Vadim's cell, he will take a look at this.

<!-- Comment: Tue Mar 20 22;08;23 comment by...VR -->
Multiple control processes left in by the previous XFT and L2CAL tests today confused the decision node. I restarted everything fresh. Will come up with procedures for tests regarding this thing so willnot happen again.
<!-- Date: Tue Mar 20 21:53:44.774 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237002>237002</A>
 Terminated at 2007.03.20 21:53:38
<!-- Author: RunControl -->


<!-- Date: Tue Mar 20 21:54:01.601 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237002>237002</A>
TERMINATE: done
<!-- Author: rob -->


<!-- Date: Tue Mar 20 22:00:12.423 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003>237003</A>
 Activated at 2007.03.20 21:59:02
<!-- Author: RunControl -->



<!-- Date: Tue Mar 20 22:01:02.761 2007 -->
Ok, it's fixed and we are taking data again.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->


<!-- Date: Tue Mar 20 22:02:58.907 2007 -->
Procmon is clear now that the network has been up for a while.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->

<!-- Date: Tue Mar 20 22:05:07.001 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965>236965</A>
general: <tt>CO Checklist done.</tt>
<!-- Author: Consumer Operator( Kihyeon Cho) -->




<!-- Date: Tue Mar 20 22:32:07.158 2007 -->
b0imu00   done to auto hrr fixed
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->


<!-- Date: Tue Mar 20 23:00:51.977 2007 -->
donetimeout COT06. Auto HRR fixed.
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->

<!-- Date: Tue Mar 20 23:30:41.889 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003>237003</A>
general: <tt>CO Checklist done.</tt>
<!-- Author: Consumer Operator( Kihyeon Cho) -->



<!-- Date: Tue Mar 20 23:30:41.723 2007 -->
clc00 dto. Auto hrr fixed
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->
<!-- Comment: Tue Mar 20 23;39;29 comment by...rob -->
again
<!-- Date: Tue Mar 20 23:30:57.177 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236965>236965</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP 
</PRE>miniplug crate is removed from DAQ.
<!-- Author: cdfscico -->


<!-- Date: Tue Mar 20 23:33:23.491 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=236893>236893</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP 
</PRE>miniplug crate is removed from DAQ.
<!-- Author: cdfscico -->



<!-- Date: Tue Mar 20 23:38:21.556 2007 -->
 b0svx02: Sl 6 Resonance Detected
<!-- Author: rob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->
<!-- Comment: Tue Mar 20 23;44;46 comment by...rob -->
again

<!-- Comment: Tue Mar 20 23;47;47 comment by...Jose -->
This is not any more statistics driven! Something is going on for a luminosity around 45e30...


<!-- Comment: Tue Mar 20 23;56;40 comment by...Rob -->
again
<!-- Date: Tue Mar 20 23:54:25.057 2007 -->
<font color=red>Shift Summary: </font><pre>EVENING SHIFT 3/20/07 
 
16:00 Inherited a collision run 236965 (store 5292) at B0ILUM=69E30. 
 
18:40 Network connection from CDF to outside became down until 20:51. 
 
21:32 L2 timeout error caused DAQ down time of about 30 minutes. 
 
22:00 Started run 237003. 
 
  Tevatron Scedule: This store 5292 will be terminated at 7:30am. ( From Run Coordinater e-log 22:15 ) 
 </pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   236965, 237003
Delivered Luminosity   1565.9 nb-1  
Acquired Luminosity    1327.6 nb-1  
Efficiency             84.8
   
</td></tr></table></pre>
<!-- Author: shinhong kim -->

<!-- Date: Tue Mar 20 23:55:20.267 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=236965 target=_runSummary>236965</A> x39DA5</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [5,866,678]</FONT></TD>
<TD ALIGN=CENTER>15:53:41</TD>
<TD ALIGN=CENTER>21:32:03</TD>
<TD ALIGN=CENTER>05:04:42</TD>
<TD ALIGN=RIGHT>   1,902,548</TD>   <TD ALIGN=RIGHT>  1,202.00</TD>
<TD ALIGN=RIGHT>  1,084.27</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=237002 target=_runSummary>237002</A> x39DCA</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [5,866,678]</FONT></TD>
<TD ALIGN=CENTER>21:36:11</TD>
<TD ALIGN=CENTER>21:53:38</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT>     53.37</TD>
<TD ALIGN=RIGHT>      0.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=237003 target=_runSummary>237003</A> x39DCB</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [5,866,678]</FONT></TD>
<TD ALIGN=CENTER>21:59:02</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>01:40:42</TD>
<TD ALIGN=RIGHT>     657,807</TD>   <TD ALIGN=RIGHT>    322.25</TD>
<TD ALIGN=RIGHT>    278.74</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>06:45:25</TD>
<TD ALIGN=RIGHT>     2,560,355</TD> <TD ALIGN=RIGHT>  1,577.62</TD>
<TD ALIGN=RIGHT>  1,363.00</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Wed Mar 21 00:05:47.642 2007 -->
Done TO for cot01, auto-HRR recovered
<!-- Author: Andy :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=237003' target=_top>237003</a>) -->