<!-- Title: Owl shift. Sat May 17, 2008 -->
<!-- SciCo: tipton -->
<!-- DAQAce: Chang Seong Moon -->
<!-- MonAce: &nbsp; -->
<!-- CO: Sergei Klimenko -->
<!-- OpManager: &nbsp; -->
<!-- Notes: -->
<pre></pre>


<!-- Date: Sat May 17 00:35:38.832 2008 -->
Shift starts with luminosity of 160E30
<!-- Author: Tipton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->




<!-- Date: Sat May 17 00:38:15.223 2008 -->
<pre>There was some error message in DQMon like below. 
 
SVX 
 
Errors found in Silicon cells by SVXMon. 
Check SVXMon plots and act accordingly. 
 
Co just checked SVX plots of current run. there is 2 new red ladders but not serious. Because current run is started in half hour, we will keep an eye on it.</pre>
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->

<!-- Date: Sat May 17 00:43:46.148 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262420>262420</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
TOF 
RSF 
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Sat May 17 00:45:40.208 2008 -->
Icicle button turns red in the ProcMon. I restart the icicle. It turns green now.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->
<!-- Comment: Sat May 17 04;24;07 comment by...Chang-Seong -->
Again.
<!-- Comment: Sat May 17 07;12;46 comment by...Chang-Seong -->
Again.
<!-- Date: Sat May 17 00:46:14.926 2008 -->
The crate b0pcal11 is not responsive. Shepherd fixed it.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->
<!-- Comment: Sat May 17 05;50;09 comment by...Chang-Seong -->
Again.
<!-- Date: Sat May 17 00:53:16.045 2008 -->
Suddenly, CMU HV East and CMX SW wedge 16-17 trip. CMU is automatically recovered. For CMX, I turn stanby and on wedge 16-17. It works well. We're taking the data.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->


<!-- Date: Sat May 17 01:05:21.600 2008 -->
<pre>FrontEnd Crate Error Condition from: SVT_TRACKING_02 
RXPT Error for crate/s: b0svt02   
 
Auto HRR 
 
</pre>
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->


<!-- Date: Sat May 17 02:03:22.483 2008 -->
<pre> Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid I/B0/W0/L1/C0-3 .  
 AUTO HRR.  </pre>
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->


<!-- Date: Sat May 17 02:48:38.475 2008 -->
Trigger inhibit is set in few second due to TOF HV. However it went away soon.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->
<!-- Comment: Sat May 17 03;44;02 comment by...Chang-Seong -->
Again.
<!-- Date: Sat May 17 03:10:37.705 2008 -->
Done TO. for crate b0pcal00. Auto HRR.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->
<!-- Comment: Sat May 17 03;15;52 comment by...Chang-Seong -->
Again.

<!-- Comment: Sat May 17 03;22;31 comment by...Chang-Seong -->
Again.


<!-- Comment: Sat May 17 03;27;37 comment by...Chang-Seong -->
Again.
<!-- Comment: Sat May 17 03;34;33 comment by...Chang-Seong -->
Again.
<!-- Date: Sat May 17 03:26:25.943 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-2411&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-2411  width='300' align='top'></a></td><td>DAQErrMon shows a large rate (~50/minute!) of DataSyncErr errors. (Typical rate is 1-2 per hour). Expert is paged. </td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Sat May 17 03:33:53.363 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422>262422</A>
 Terminated at 2008.05.17 03:33:48
<!-- Author: RunControl -->


<!-- Date: Sat May 17 03:34:26.873 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422>262422</A>
TERMINATE: Too many error in DAQErrMon
<!-- Author: Chang-Seong -->



<!-- Date: Sat May 17 03:39:57.952 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262423>262423</A>
 Activated at 2008.05.17 03:39:24
<!-- Author: RunControl -->


<!-- Date: Sat May 17 03:39:58.006 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262423>262423</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [7,1083,721] 
<!-- Author: Chang-Seong -->
<!-- Comment: Sat May 17 03;41;55 comment by...Chang-Seong -->
When the new run is started, b0cot04 is stuck. Shepherd fixed it.
<!-- Date: Sat May 17 03:40:04.766 2008 -->
With 47 Data sync errors/minute over the last 5 minutes, we call DAQ/RC pager ( 612-8215) No answer.   But we note on their schedule that Bill is supposed to be carrying the pager and it is also stated that Bill is @CERN this week. 
 
So we page J.D. Lewis...also no answer. 
 
So we end 26422 and start 26423 to see if that fixes the problem.
<!-- Author: Tipton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422' target=_top>262422</a>) -->
<!-- Comment: Sat May 17 04;07;13 comment by...Chang-Seong -->
<pre>Steve Hahn called us and suggested to check out the error handler. It said there is the BunchCounter errors at b0pcal00. So he suggested shepherd it and we did it. As a result, no more errors is not increased in the DAQErrMon. It looks related to Done TO. for crate b0pcal00 and Auto HRR a half hour ago. And we just restarted the new run(262424).</pre>
<!-- Comment: Sat May 17 04;13;23 comment by...Steve Hahn -->
<p>Doing a Google search on "site:www-cdfonline.fnal.gov data sync errors", there
seem to be some old (2003, 2004) ops notes that seem to indicate this is a L2
error, though the origin may still be b0pcal00 since there is bunch counter info
in the trigger data also.  It would be nice if this was marked "L2 data sync error" 
if that were the case.</p>
<!-- Comment: Sat May 17 09;26;02 comment by...Donatella -->
I have the pager this weekend and it did not go off.
I'll ask the day scico to page me now and see if it works.
<!-- Date: Sat May 17 03:52:24.783 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262423>262423</A>
 Terminated at 2008.05.17 03:52:14
<!-- Author: RunControl -->


<!-- Date: Sat May 17 03:54:06.799 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262423>262423</A>
TERMINATE: Fixed b0pcal00 so that Error is gone in the DAQErrMon.
<!-- Author: Chang-Seong -->


<!-- Date: Sat May 17 03:56:45.839 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424>262424</A>
 Activated at 2008.05.17 03:56:39
<!-- Author: RunControl -->


<!-- Date: Sat May 17 03:57:12.941 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424>262424</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [7,1083,721] 
<!-- Author: Chang-Seong -->




<!-- Date: Sat May 17 04:09:27.471 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262422>262422</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
PCAL 
RSF 
</PRE>
Saw data-sync errors of 40/min starting somewhere around event number 10500
<!-- Author: cdfscico -->


<!-- Date: Sat May 17 04:12:21.212 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262423>262423</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
RSF 
</PRE>pcal00 crate caused datasync errors through most of this short run.  Fixed at the end.
<!-- Author: cdfscico -->




<!-- Date: Sat May 17 04:16:49.633 2008 -->
 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid S/B2/W4/L4/C7-13 .  
 AUTO HRR. 
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424' target=_top>262424</a>) -->


<!-- Date: Sat May 17 04:20:03.071 2008 -->
A Pinkie in B1 Bias1
<!-- Author: Tipton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424' target=_top>262424</a>) -->


<!-- Date: Sat May 17 04:22:46.185 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-2412&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-2413  width='300' align='top'></a></td><td>This pinkie isn't very pink</td></tr></table>
<!-- Author: Tipton -->



<!-- Date: Sat May 17 04:38:22.416 2008 -->
We have had ~4 pinkie's.  After the first one, we were no longer able to save history plot.  All pinkie's so far look OK
<!-- Author: Tipton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424' target=_top>262424</a>) -->
<!-- Comment: Sat May 17 04;48;31 comment by...Chang-Seong -->
There were 3 pinkies like below. <br> <br>

SVX VBIAS B2W4L1  <br>
SVX VBIAS B3W2L0   <br>
ISL VBIAS1 B1W0L1  <br> <br>

Pinkie of SVX VBIAS B3W2L0 is gone after unmake. But the others is still pinkie. So set to ignore.  <br>

By the way, we can't save the history plot.


<!-- Date: Sat May 17 05:09:16.049 2008 -->
Done TO. for crate cot07, pcal05. Shepherd fixed it.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424' target=_top>262424</a>) -->


<!-- Date: Sat May 17 05:47:32.125 2008 -->
ISL, L00 HV trip. the problem is caused by crate 7. Auto recovery is failed. Scico paged expert. Miguel called and let me stop the auto reconvery and make the run control halt state and then start the auto recoverry. As a result auto recovery is successful.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424' target=_top>262424</a>) -->
<!-- Comment: Sat May 17 06;16;24 comment by...Miguel M -->
Thank you, Chang. The problem was that Run Control was in Recovered State at the time when the Silicon Auto Recovery was in progress. Please keep in mind that RC must be kept in HALT State  while SAR tries to solve a Silicon HV issue. Do not click on Recover prematurely. The only two other states compatible with the SAR action are IDLE and START. In a word, whenever RC is not in HALT, IDLE or START, then SAR will certainly fail.


<!-- Date: Sat May 17 07:55:38.003 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=262422 target=_runSummary>262422</A> x40116</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [7,1083,721]</FONT></TD>
<TD ALIGN=CENTER>23:55:32</TD>
<TD ALIGN=CENTER>03:33:48</TD>
<TD ALIGN=CENTER>03:21:32</TD>
<TD ALIGN=RIGHT>   1,718,783</TD>   <TD ALIGN=RIGHT>  1,817.89</TD>
<TD ALIGN=RIGHT>  1,670.53</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=262423 target=_runSummary>262423</A> x40117</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [7,1083,721]</FONT></TD>
<TD ALIGN=CENTER>03:39:24</TD>
<TD ALIGN=CENTER>03:52:14</TD>
<TD ALIGN=CENTER>00:11:00</TD>
<TD ALIGN=RIGHT>      45,309</TD>   <TD ALIGN=RIGHT>     89.87</TD>
<TD ALIGN=RIGHT>     74.02</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=262424 target=_runSummary>262424</A> x40118</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [7,1083,721]</FONT></TD>
<TD ALIGN=CENTER>03:56:39</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>03:25:20</TD>
<TD ALIGN=RIGHT>   1,434,623</TD>   <TD ALIGN=RIGHT>  1,398.80</TD>
<TD ALIGN=RIGHT>  1,203.38</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>06:57:54</TD>
<TD ALIGN=RIGHT>     3,198,715</TD> <TD ALIGN=RIGHT>  3,306.56</TD>
<TD ALIGN=RIGHT>  2,947.93</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sat May 17 07:56:07.464 2008 -->
<font color=red>Shift Summary: </font><pre>Shift started with luminosity of 160E30 with run 26422 
 
3:25 After a number of done TO in B0pcal00 we start getting 
DATAsync errors, also for this crate.  We call DAQ papger, no answer (Bill is 
on schedule, also listed as at cern this week), call JDL, no answer, we end 
run 26422, start 26423, observe same problem. 
 
Then call Steve Hahn who suggests we shepard B0pcal00 which fixes the problem. 
 
3:54  We end 26423 and start 26424 
 
 4:40 We can no longer save the pinkie current vs. time  historgram from silicon 
terminal... 
 
5:30  ISL L00 HV trip, we call Miguel Mondragon, fixed 
 
Get Silicon crew to fix process that allows for easy saving of pinkie history 
plots 
 
7:25  Two more pinkies, nothing serious, ignored 
 
 
Date:	2008.05.17 
Shift:	owl 
Delivered luminosity:	3320.8 nb-1 
Acquired luminosity:	2907.2 nb-1 
Efficiency:	87.5% 
 
 
</pre> 
<!-- Author: Tipton -->


<!-- Date: Sat May 17 08:00:47.364 2008 -->
A node b0fib04 is not responsive. I just tried to follow the instruction as vxlogin b0fib04. But login is rejected. So I tried to shepherd it. It works.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=262424' target=_top>262424</a>) -->