<!-- Title: Eve shift. Tue Jul 29, 2008 -->
<!-- SciCo: Fotis Ptochos -->
<!-- DAQAce: Andreas Gessler -->
<!-- MonAce: &nbsp; -->
<!-- CO: Joachim Heuser -->
<!-- OpManager: Peter Dong -->
<!-- Notes: -->
<pre>Take over with store 6324 colliding and luminosity 161.4E30</pre>


<!-- Date: Tue Jul 29 16:12:04.630 2008 -->
b0pcal05 got stuck, thus I shepherded it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->


<!-- Date: Tue Jul 29 16:29:49.391 2008 -->
b0svt03 RXPT error, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->


<!-- Date: Tue Jul 29 16:30:15.211 2008 -->
L2 Decision TO, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 17;41;08 comment by...Andreas -->
Again.
<!-- Comment: Tue Jul 29 21;21;36 comment by...Andreas -->
Again.
<!-- Date: Tue Jul 29 17:26:49.596 2008 -->
b0cot16 Done TO, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 18;36;54 comment by...Andreas -->
Again.
<!-- Date: Tue Jul 29 17:33:26.895 2008 -->
b0cot07 Done TO, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 20;22;04 comment by...Andreas -->
Again.
<!-- Date: Tue Jul 29 17:40:56.876 2008 -->
b0svt00 RXPT error, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->



<!-- Date: Tue Jul 29 17:47:53.991 2008 -->
CMP HV trip. It is the same trip as in the <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=632&frame=2&anchor=14:47:30.280'>previous shift</a>. Turning on the HV again fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 17;54;51 comment by...Andreas -->
Again.


<!-- Comment: Tue Jul 29 18;05;10 comment by...Andreas -->
Again (Bottom A3) - We tried to page the expert.
<!-- Comment: Tue Jul 29 18;17;14 comment by...Fotis -->
According to Xiaojian who answered the page, if the specific channel trips again we should end the run, put the channel on standby(for about an hour) and start another run in this configuration. 
<!-- Date: Tue Jul 29 17:55:55.752 2008 -->
Consumer Error: CER_SVXMON_HALT_RECOVER_RUN_ERROR. Auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->




<!-- Date: Tue Jul 29 18:33:02.154 2008 -->
b0pcal10 Done TO, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->



<!-- Date: Tue Jul 29 18:54:47.483 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3856&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3857  width='300' align='top'></a></td><td>Pinkie from ISL_VBIAS_B3W1L2. Set it to ignore</td></tr></table>
<!-- Author: Fotis -->


<!-- Date: Tue Jul 29 18:57:15.110 2008 -->
b0cot19 Done TO, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 19;00;44 comment by...Andreas -->
Again, but no auto HRR was performed so I did HRR manually.

<!-- Comment: Tue Jul 29 19;01;50 comment by...Andreas -->
Again no auto HRR thus I shepherded it (the crate didn't even turn red).


<!-- Date: Tue Jul 29 19:20:42.610 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3858&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3858  width='300' align='top'></a></td><td>One more pinkie from I_VBIAS_B1W4L1. Set it to ignore</td></tr></table>
<!-- Author: Fotis -->


<!-- Date: Tue Jul 29 19:57:50.125 2008 -->
b0pcal02 Done TO, auto HRR fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 22;50;38 comment by...Andreas -->
Again.
<!-- Date: Tue Jul 29 20:22:17.478 2008 -->
<table><tr><td><a href=/elog/notebooks/2008/Tue_Jul_29_2008_Eve.202213.ps align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3860  align='top'></a></td><td>SVT Failures, red background</td></tr></table>
<!-- Author: Joachim -->
<!-- Comment: Tue Jul 29 20;26;41 comment by...Fotis -->
We paged SVT and Massimo is coming over
<!-- Comment: Tue Jul 29 21;32;08 comment by...Joachim -->
It seems that "Diff # Tracks" increases regularly. While the previous run (in this store) also has 5 different tracks, the long runs for the previous stores had zero! So something might be going on.
<!-- Comment: Tue Jul 29 21;33;49 comment by...Joachim -->
Massimo found some incosistencies in b0svt08. After sheperding this crate the number of "Diff # tracks" has so far not further increased. It is however still too early to draw final conclusions.
<!-- Comment: Tue Jul 29 22;10;43 comment by...Joachim -->
<table border=0><td><tr><td bgcolor=#e0e030>Too bad, now keeps increasing again. Overall rate steady at ~2 per mille.</td></tr></td></table>
<!-- Comment: Tue Jul 29 22;13;20 comment by...Massimo -->
I was blaming b0svt08 because I had noticed that the error LED of the tracer was on. After the reboot it's still on, so I'm still investigating the problem.
<!-- Date: Tue Jul 29 20:23:46.332 2008 -->
We got a storm of pinkies (34 total - 5 on L00 and the rest on SVX ). All VBias type. Small changes from nominal. Paged Silicon and Sergo suggested to wait a bit and then he will instruct us on global reset for the pinkies if everything is ok  
<!-- Author: Fotis :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 20;45;00 comment by...sergo -->
<pre>The appearance of pinkies in SVX and L00 is most probably caused by opening of the deionizer valve. My suggestion is to wait 30 minutes and then do global unmark in IMON and see how many of pinkies return.  Then set returned  pinkies to ignore  - I will look at them later tonight or in the morning. </pre>
<!-- Date: Tue Jul 29 20:24:08.052 2008 -->
CMX HV trip (Southwest, wedges 13-14). Turning on the HV again fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->
<!-- Comment: Tue Jul 29 21;55;00 comment by...Andreas -->
Again - This time southwest, wedges 16-17. Turning on the HV twice fixed it (curiously RC didn't go to HALTED state).
<!-- Date: Tue Jul 29 20:26:16.543 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489>265489</A>
trigger: <tt>SVT issues, not clear how/whether bad</tt>
<!-- Author: Consumer Operator(Joachim Heuser ) -->






<!-- Date: Tue Jul 29 21:24:44.787 2008 -->
Sergo (Silicon Pager Carrier) called back about the many pinkies we got earlier. He instruct me to hit the global unmask button and  then wait till some of the pinkies reappeared. He told me to set ignore to all of them. He thinks the problem appeared due to the opening of a cooling valve that causes some channels that are close to the high limit to warm up and slightly pass the limit.  
The plots below show the pattern observed on all channels. 
<!-- Author: Fotis :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->


<!-- Date: Tue Jul 29 21:31:20.970 2008 -->
<center><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-3869&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-3870' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-3861&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-3862' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-3863&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-3864' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-3865&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-3866' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-3867&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-3868' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-3869&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-3870' ></a></center>
<!-- Author: Fotis -->
<!-- Comment: Tue Jul 29 21;33;01 comment by...Fotis -->
Ok these are some of the channels (20 in total I think) that were set to ignore after the hitting the global unmark button. The pattern was similar for all the other channels



<!-- Date: Tue Jul 29 21:45:07.528 2008 -->
PLUG HV trip in iFix. Restarting the monitoring programm and turning the HV from ON to STANDBY and back again fixed it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->




<!-- Date: Tue Jul 29 22:13:07.834 2008 -->
5 more pinkies from ISL all bias type and with the same pattern as before. Three channels were set to ignore (I_VBIAS_B0W0L1, I_VBIAS_B0W2L2, I_VBIAS_B1W2L1) and two just unmarked (I_VBIAS_B0W3L1 and I_VBIAS1_B2W0L3) 
<br>
<!-- Author: Fotis :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489' target=_top>265489</a>) -->


<!-- Date: Tue Jul 29 22:26:33.234 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489>265489</A>
trigger: <tt>SVT error rate fluctuates around 2 per mille, expert (Massimo) investigating</tt>
<!-- Author: Consumer Operator(Joachim Heuser ) -->


<!-- Date: Tue Jul 29 22:43:44.860 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489>265489</A>
 Terminated at 2008.07.29 22:43:34
<!-- Author: RunControl -->


<!-- Date: Tue Jul 29 22:43:45.022 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489>265489</A>
TERMINATE: Done.
<!-- Author: Andreas -->
<!-- Comment: Tue Jul 29 22;55;58 comment by...Fotis -->
Run was terminated per SVT expert's request to see if the rate of the observed inconsistensies persist in the new run.  
<!-- Comment: Tue Jul 29 23;13;42 comment by...Joachim -->
<table border=0><td><tr><td bgcolor=#e0e030>judging from the small statistics so far, starting a new run did not help</td></tr></td></table>
<!-- Date: Tue Jul 29 22:46:56.007 2008 -->
b0pcal10 Heap Corrupt Error. Shepherded it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265506' target=_top>265506</a>) -->

<!-- Date: Tue Jul 29 22:47:36.986 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265506>265506</A>
 Activated at 2008.07.29 22:46:59
<!-- Author: RunControl -->


<!-- Date: Tue Jul 29 22:47:37.833 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265506>265506</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [9,1115,725] 
<!-- Author: Andreas -->




<!-- Date: Tue Jul 29 22:57:40.275 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265489>265489</A>
trigger: <tt>SVT status unclear</tt>
<!-- Author: Consumer Operator(Joachim Heuser ) -->




<!-- Date: Tue Jul 29 23:20:07.805 2008 -->
The CDF Radition Monitoring application disappeared from CNS52PC thus I restarted it.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265506' target=_top>265506</a>) -->


<!-- Date: Tue Jul 29 23:37:36.064 2008 -->
Halt the run to shepherd crate SVT00 that may cause the track inconsistencies.
<!-- Author: Fotis :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265506' target=_top>265506</a>) -->
<!-- Comment: Tue Jul 29 23;51;12 comment by...Fotis -->
Didn't do it

<!-- Date: Tue Jul 29 23:55:11.137 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=265489 target=_runSummary>265489</A> x40D11</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [9,1115,725]</FONT></TD>
<TD ALIGN=CENTER>14:36:54</TD>
<TD ALIGN=CENTER>22:43:34</TD>
<TD ALIGN=CENTER>07:01:03</TD>
<TD ALIGN=RIGHT>   3,333,641</TD>   <TD ALIGN=RIGHT>  3,842.92</TD>
<TD ALIGN=RIGHT>  3,285.59</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=265506 target=_runSummary>265506</A> x40D22</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [9,1115,725]</FONT></TD>
<TD ALIGN=CENTER>22:46:59</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>01:01:49</TD>
<TD ALIGN=RIGHT>     358,962</TD>   <TD ALIGN=RIGHT>    362.37</TD>
<TD ALIGN=RIGHT>    331.94</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>08:02:52</TD>
<TD ALIGN=RIGHT>     3,692,603</TD> <TD ALIGN=RIGHT>  4,205.29</TD>
<TD ALIGN=RIGHT>  3,617.54</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Jul 29 23:59:49.551 2008 -->
Done TO b0cot16. Auto HRR.
<!-- Author: Andreas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=265506' target=_top>265506</a>) -->


<!-- Date: Wed Jul 30 00:01:29.655 2008 -->
<font color=red>Shift Summary: </font><pre>Rather quiet shift with some problems that addressed with 
no significant downtime 
 
The problems we had: 
-- CMP (Bottom A3) tripped three times in last 20min  
   (17:50-18:10). Under the threat to set it on stanby  
   it settled down.  
 
-- Some inconsistencies (2 per mil level) between number  
   of tracks found in data and simulation for SVT. 
   Massimo advised to shepherd crates SVT00 and SVT08  
   and start a new run. Problem didn't get solved and 
   investigation continues. 
  
-- Many silicon pinkies (35 in total SVX and L00) all 
   together and all of bias type and with small jumps 
   above max limit. Some more (5) of same type from ISL  
   at 22:10. See entry from Sergo 
 
-- CMX (SW wedges 13-14 and 16-17 twice) trip.  
   Reset them ok  
 
-- A Plug HV trip was reset going from ON-Standby-ON   
 
 
It is probably going to be a long store at low luminosity 
as MCR is dealing with a large water leak that does not 
allow for beam transport. The stush remains at 93E10 
while stack and production rate is zero.  
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   265489, 265506
Delivered Luminosity   3371.5 nb-1  
Acquired Luminosity    2971.3 nb-1  
Efficiency             88.1%
   
</td></tr></table></pre>
<!-- Author: Fotis -->