<!-- Title: Owl shift. Thu Apr 12, 2007 -->
<!-- SciCo: Frank Chlebana -->
<!-- DAQAce: Gheorghe Lungu/Ji-Eun Jung -->
<!-- MonAce: &nbsp; -->
<!-- CO: Dino Goulianos -->
<!-- OpManager: Sasha Pronko -->
<!-- Notes: -->
<pre>Store ongoing  
 
 L   = 56.5E30 
 RR  = 11.83E10 
 
From MRC: 
Luminosity decay fit projections show that it will still be above 30 µb-1/s at the start of the day shift Thursday 
 
</pre>

<!-- Date: Thu Apr 12 00:24:04.433 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238917>238917</A>
 Terminated at 2007.04.12 00:23:48
<!-- Author: RunControl -->



<!-- Date: Thu Apr 12 00:24:14.408 2007 -->
Done TO pcal02 : shepherd three times,but it didn't work.  
<!-- Author: Gheorghe/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238917' target=_top>238917</a>) -->

<!-- Date: Thu Apr 12 00:24:39.924 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238917>238917</A>
TERMINATE: take slot15 out of pcal02
<!-- Author: Ji-Eun/Jim -->


<!-- Date: Thu Apr 12 00:28:01.418 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238918>238918</A>
 Activated at 2007.04.12 00:27:54
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 00:28:02.985 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238918>238918</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Ji-Eun/Jim -->


<!-- Date: Thu Apr 12 00:30:30.074 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238918>238918</A>
 Terminated at 2007.04.12 00:29:58
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 00:30:31.910 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238918>238918</A>
TERMINATE: remove pcal02
<!-- Author: Ji-Eun/Jim -->



<!-- Date: Thu Apr 12 00:32:27.589 2007 -->
<pre>Getting persistent errors from PCAL02 Slot 15 
 
b0pcal02:Messenger:Apr 12, 2007 12:17:33 AM-&gt;ADMEM_setCafeDelay geo32:/slot=15 VME bus error: old/target/new 0xff/0x2c/ff [4] 
 
Tried sheperding 3 tims followed by a VISIONdemo buss reset. 
 
Problem is still there 
 
Paged ADMEM expert 
 
We removed the card from the current run. 
 
ADMEM Expert calls back and asks for the PCAL02 crate. We remove 
it from the current run and start a new run. 
</pre>
<!-- Author: Chlebana :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238918' target=_top>238918</a>) -->

<!-- Date: Thu Apr 12 00:34:16.721 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238919>238919</A>
 Activated at 2007.04.12 00:34:07
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 00:34:26.055 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238919>238919</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Ji-Eun/Jim -->
<!-- Comment: Thu Apr 12 00;36;02 comment by...Jim/Ji-Eun -->
s02p3 (L3 display) is red

<!-- Comment: Thu Apr 12 00;47;58 comment by...Chlebana -->
<pre>This run has PCAL02 removed because of an ADMEM problem.</pre>
<!-- Comment: Thu Apr 12 00;59;25 comment by...Jim -->
<pre>node s02p03 is ok in the next run (238920); pcal02 good too</pre>
<!-- Date: Thu Apr 12 00:38:48.985 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238919>238919</A>
 Terminated at 2007.04.12 00:38:39
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 00:39:29.452 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238919>238919</A>
TERMINATE: end of run: include pcal02
<!-- Author: Ji-Eun/Jim -->


<!-- Date: Thu Apr 12 00:42:37.012 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
 Activated at 2007.04.12 00:42:19
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 00:43:11.333 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Ji-Eun/Jim -->




<!-- Date: Thu Apr 12 00:49:09.973 2007 -->
<pre>ADMEM Expert (Kwangzoo) called back. He has reconfigured the  
bad ADMEM. We start a new run to include PCAL02</pre>
<!-- Author: Chlebana :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->


<!-- Date: Thu Apr 12 00:49:49.887 2007 -->
We had DQMon alarm(PES and PEM): maybe because we remove pcal02. 
Now, we got a call expert and we will include pcal02 again.  
<!-- Author: Jim/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->


<!-- Date: Thu Apr 12 00:49:52.880 2007 -->
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid S/B3/W9/L3/C0-5 .  
 AUTO HRR will be issued    
<!-- Author: Jim/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->
<!-- Comment: Thu Apr 12 01;07;07 comment by...Jim/Ji-Eun -->
happen again-> auto HRR
<!-- Comment: Thu Apr 12 02;32;13 comment by...Ji-Eun/Jim -->
again- auto HRR
<!-- Date: Thu Apr 12 00:52:42.885 2007 -->
WE got a pcal10:PERML_HEAP_CORRUPT error,and shepherd-> now ok
<!-- Author: Jim/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->


<!-- Date: Thu Apr 12 00:54:50.638 2007 -->
<pre>Done Timeout detected in crates: 
IMU_00 
IMU_01 
CMU_01 
CMU_00 
CMP_00 
MUON_SCINT_00 
CMX_00 
CMX_01 
CLC_00 
HADRON_TIMING_00 
EM_TIMING_00 
 
this happened right after the heap corrupt message from pcal10. The error logger printed out many lines regarding bunch counter mismatches. After HRR things seem fine.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->
<!-- Comment: Thu Apr 12 01;08;01 comment by...Ji-Eun/Jim -->
happen again after SVXMON error -> now ok


<!-- Date: Thu Apr 12 01:08:41.289 2007 -->
imu00 done TO -> now ok, auto HRR
<!-- Author: Jim/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->


<!-- Date: Thu Apr 12 01:20:12.926 2007 -->
<pre>DQMon alarm: WHA seems to have a dead channel in west side wedge eta=-11, phi2R. This was spotted in the previous shift with about 8.5k entries. Right now we're at 6k or so. We'll keep watching it, but it doesn't seem like low stats to me.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->
<!-- Comment: Thu Apr 12 01;54;44 comment by...Jim -->
<pre>this might be low stats effect: as per the CO instructions we should cross check with the Hadron TDC occupancy plot and the wedge seems fine there. Ji-Eun is posting the plots now.</pre>
<!-- Date: Thu Apr 12 01:21:26.179 2007 -->
<pre>ProcMon error: IMON check not ok, after recheck went ok.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->


<!-- Date: Thu Apr 12 01:32:53.683 2007 -->
We had PSM alarm 1RR21E_2: inhibited and Halt. So, we Recover and Run manually.
<!-- Author: Jim/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->
<!-- Comment: Thu Apr 12 02;02;38 comment by...Jim -->
<pre>channel 1 (+12V digital) was low for a minute, then it recovered by  itself</pre>
<!-- Date: Thu Apr 12 01:53:30.861 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238919>238919</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
PCAL PCAL02 ADMEM in slot 15 was removed
</PRE>
<!-- Author: cdfscico -->





<!-- Date: Thu Apr 12 02:02:42.358 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3529&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3529  width='300' align='top'></a></td><td>YMon complains channel 11 -> It is maybe dead channel or low statistics problem.</td></tr></table>
<!-- Author: Ji-Eun -->
<!-- Comment: Thu Apr 12 02;06;27 comment by...Ji-Eun -->
See below. It was not dead channel.
<!-- Comment: Thu Apr 12 02;17;21 comment by...Ji-Eun -->
<pre>DQMon: WHA is red!</pre>
<!-- Date: Thu Apr 12 02:05:28.392 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3530&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3530  width='300' align='top'></a></td><td>So, we look into HTDC slide. But WHA wedge 2 is not zero(not dead), so it is  maybe low statistics.</td></tr></table>
<!-- Author: Ji-Eun -->



<!-- Date: Thu Apr 12 02:14:47.118 2007 -->
Busy TO COT_04 -> checking Ace Control Panel : load error in EVEL1_cal_04 => auto HRR recovered
<!-- Author: Jim/Ji-Eun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->



<!-- Date: Thu Apr 12 02:31:15.503 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3531&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3531  width='300' align='top'></a></td><td>WHAPMT channel phi=12 eta=7 dead; channel phi=9 eta=-7 hot</td></tr></table>
<!-- Author: dino -->
<!-- Comment: Thu Apr 12 02;38;12 comment by...dino -->
I will check this with higher statistics

<!-- Comment: Thu Apr 12 04;05;59 comment by...dino -->
Pattern looks the same with 32000 events. Conclusion: one new dead tower and "temperature" of tower (11,-7) rising.


<!-- Date: Thu Apr 12 06:45:47.908 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3536&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3537  width='300' align='top'></a></td><td>We just got three pinkies nearly all together</td></tr></table>
<!-- Author: Chlebana -->


<!-- Date: Thu Apr 12 06:46:31.303 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3538&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3539  width='300' align='top'></a></td><td>Another</td></tr></table>
<!-- Author: Chlebana -->


<!-- Date: Thu Apr 12 06:47:03.765 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3540&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3541  width='300' align='top'></a></td><td>Another</td></tr></table>
<!-- Author: Chlebana -->

<!-- Date: Thu Apr 12 07:45:28.382 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
cal: <tt>WHAPMT tower (12,7) dead</tt>
<!-- Author: Consumer Operator( Dino Goulianos) -->


<!-- Date: Thu Apr 12 07:45:28.087 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
cal: <tt>WHAPMT tower (12,7) dead</tt>
<!-- Author: Consumer Operator( Dino Goulianos) -->



<!-- Date: Thu Apr 12 07:48:33.578 2007 -->
Done TO COT_03 : auto HRR ok
<!-- Author: Jim/Ji-Eun -->


<!-- Date: Thu Apr 12 07:53:19.436 2007 -->
<pre>after the COT03 done TO recovered, I have noticed in the error logger the following printouts: 
 
b0cot13:Messenger:Apr 12, 2007 07:41:51 AM-&gt;TDC_readXTC2Buffer VME bus error reading XTC2 mezz slot 10 [11] 
 
decided to shepherd COT13, and things seem ok now. after going to halt we got done TO from clc00, and alot of other printouts showed up in the error logger, this time there were TDC header mismatches and bunch counter mismatched from many crates. That stopped after we resumed running.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920' target=_top>238920</a>) -->

<!-- Date: Thu Apr 12 07:55:08.866 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=238917 target=_runSummary>238917</A> x3A545</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>23:23:41</TD>
<TD ALIGN=CENTER>00:23:48</TD>
<TD ALIGN=CENTER>00:47:58</TD>
<TD ALIGN=RIGHT>     301,737</TD>   <TD ALIGN=RIGHT>    207.02</TD>
<TD ALIGN=RIGHT>    164.27</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=238918 target=_runSummary>238918</A> x3A546</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>00:27:54</TD>
<TD ALIGN=CENTER>00:29:58</TD>
<TD ALIGN=CENTER>00:02:01</TD>
<TD ALIGN=RIGHT>       8,368</TD>   <TD ALIGN=RIGHT>      7.90</TD>
<TD ALIGN=RIGHT>      6.24</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=238919 target=_runSummary>238919</A> x3A547</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>00:34:07</TD>
<TD ALIGN=CENTER>00:38:39</TD>
<TD ALIGN=CENTER>00:04:25</TD>
<TD ALIGN=RIGHT>      19,595</TD>   <TD ALIGN=RIGHT>     15.59</TD>
<TD ALIGN=RIGHT>     14.13</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=238920 target=_runSummary>238920</A> x3A548</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>00:42:19</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>06:44:45</TD>
<TD ALIGN=RIGHT>   2,810,129</TD>   <TD ALIGN=RIGHT>  1,161.86</TD>
<TD ALIGN=RIGHT>  1,089.26</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:00</TD>
<TD ALIGN=CENTER>07:39:10</TD>
<TD ALIGN=RIGHT>     3,139,829</TD> <TD ALIGN=RIGHT>  1,392.38</TD>
<TD ALIGN=RIGHT>  1,273.90</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Thu Apr 12 07:58:06.189 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
 Terminated at 2007.04.12 07:57:28
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 07:58:07.943 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
TERMINATE: end of run, csl test
<!-- Author: Ji-Eun/Jim -->



<!-- Date: Thu Apr 12 08:02:00.123 2007 -->
<font color=red>Shift Summary: </font><pre> 
Started shift with store in progress 
 L   = 56.5E30  
 RR  = 11.83E10  
 
Get Error from PCAL02 ADMEM in slot 15. 
Removed card/crate 
Paged expert (Kwangzoo) who reconfigured it 
Put PCAL02 back in run  
Recovered within 28 mins (73 nb lost)  
  Run 238917  Thu Apr 12 00:14     
  Run 238920  Thu Apr 12 00:42 
 
PSM Alarm (1RR21E_2) causes RC to HALT 
Alarm clears by itself, need to click recover run to get running again. 
 
DQMon reports error in WHA 
(second time DQMon was first to identify a problem) 
 
WHAPMT channels phi=12 eta=7 dead; channel phi=9 eta=-7 hot 
Third dead channel in lower left appears in reference plot. 
 
Same three problem towers are seen in run 238877 (run from last shift) 
 
Around 6:35 we got three BIAS pinkies - set to ignore 
 
We had several Time Outs associated with many crates complaining 
about header mismtatches, BC mismatches...  (See ErrorLog) 
 
Runs: 
  238917  - PCAL02 ADMEM Slot 15 gives bus error 
  238918  - run without PCAL02 ADMEM Slot 15 
  238919  - run without PCAL02 
  238920  - ongoing 
 
 
Totals 
Date:	2007.04.12 
Shift:	owl 
Delivered luminosity:	1304.6 nb-1 
Acquired luminosity:	1158.5 nb-1 
Efficiency:	        88.8</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   238917 238918 238919 238920
Delivered Luminosity   1304.6 nb-1  
Acquired Luminosity    	1158.5 nb-1  
Efficiency             	 88.8
   
</td></tr></table></pre>
<!-- Author: Chlebana -->

<!-- Date: Thu Apr 12 08:07:33.578 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238921>238921</A>
 Activated at 2007.04.12 08:06:43
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 08:10:24.210 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238921>238921</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Ji-Eun/Jim -->


<!-- Date: Thu Apr 12 08:12:42.627 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238921>238921</A>
 Terminated at 2007.04.12 08:11:53
<!-- Author: RunControl -->


<!-- Date: Thu Apr 12 08:15:31.743 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=238920>238920</A>
general: <tt>General: aside from a new dead PHA tower (noted by previous CO) everything else looks good.</tt>
<!-- Author: Consumer Operator( Cheng-Ju Lin) -->