<!-- Title: Eve shift. Tue Dec 26, 2006 -->
<!-- SciCo: DongHee Kim -->
<!-- DAQAce: Till Hoffmann -->
<!-- MonAce: Naoki Kimura -->
<!-- CO: Chen Yang -->
<!-- OpManager: Mark Mattson -->
<!-- Notes: -->
<pre>start shift</pre>


<!-- Date: Tue Dec 26 16:17:41.126 2006 -->
Run Coordinator e-log:<br> 
D0 has a bad PDT, so they are requesting a 1.5 hour access after the store. We will approve this request. I will likely run this store until mid-day tomorrow. The end of store time will be set by tomorrow morning.
<!-- Author: m mattson -->


<!-- Date: Tue Dec 26 16:18:48.954 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>It looks like the access to the  
 
  b0spool 
 
file server is very slow. This is were we write out various 
log files. 
 
</pre></td></tr></td></table>
<!-- Author: Chlebana -->


<!-- Date: Tue Dec 26 16:19:08.082 2006 -->
<br>We still have massive problems with CSL, RTserver, Acnetmonitor and what not.. <br> 
Nagesh is still talking to CSL expert Frank Chlebana. 
<!-- Author: Till -->


<!-- Date: Tue Dec 26 16:32:31.643 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>We are first going to try to reboot b0csl20 to see if the access 
to b0spool gets any better. 
 
If not, we need to contact the sys admin to have a look at  
b0spool. 
 
</pre></td></tr></td></table>
<!-- Author: Chlebana -->


<!-- Date: Tue Dec 26 16:43:00.804 2006 -->
page system administrator for b0spool file server problem.  
<!-- Author: DongHee -->


<!-- Date: Tue Dec 26 16:45:04.636 2006 -->
page Opsmanager
<!-- Author: DongHee -->
<!-- Comment: Tue Dec 26 17;03;10 comment by...m mattson -->
No page received; calling control room.
<!-- Date: Tue Dec 26 16:45:48.496 2006 -->
<pre>We had similar problems a little while back and we saw high cpu 
usage on b0spool.  
 
At that time we tried restarting the b0cslxx machines - but the  
problem did not go away. 
 
We also rebooted b0spool.  
 
We could not find a definite source of the problem - but it looked 
like it got better by itself after a while. 
</pre>
<!-- Author: Chlebana -->


<!-- Date: Tue Dec 26 16:50:08.242 2006 -->
So lots of problems here. First started with RT server complaining.  We paged DAQ expert. Bill Badget asked us to page CSL expert. Frank called in. We treid to restart CSL. It did not restart from b0control15. ProcMon complained on RTServer and then on Filesystem. Frank observed CSL is slow connecting to RTServer and also b0spool. ProcMon then showed FileSystem Red. As per CSL expert, there are problems in accessing b0spool from b0gateway and other systems. We emailed to online system administrator expert.
<!-- Author: Nagesh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER= ' target=_top> </a>) -->
<!-- Comment: Tue Dec 26 16;59;47 comment by...Nagesh -->
From the discussion with Dehong (I am not sure if the spelling is correct, sorry), this problem has occurred in November and it took several parties to resolve it. The problem looks serious.
<!-- Date: Tue Dec 26 16:59:29.562 2006 -->
sysadmin calls back and ask to talk with CSL expert(Frank C.). He will talk with Frank by phone. x8746 (Dehong). Frank will call this number and calls back to us.
<!-- Author: DongHee -->




<!-- Date: Tue Dec 26 17:19:14.487 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>I removed the conjobs on the b0cslxx machines under the cslace account to reduce the load on b0spool. 
 
The sysadmin will now reboot b0spool. 
 
</pre></td></tr></td></table>
<!-- Author: Chlebana -->


<!-- Date: Tue Dec 26 17:20:01.012 2006 -->
Frank calls back 'trying to reboot b0spool' and back to us
<!-- Author: DongHee -->


<!-- Date: Tue Dec 26 17:32:04.290 2006 -->
<pre>After b0spool was rebooted the response to the disk areas 
served by b0spool is much better. CPU usage on b0spool also 
looks better. 
 
I am now also able to restat the CSL.... </pre>
<!-- Author: Chlebana -->


<!-- Date: Tue Dec 26 17:33:38.703 2006 -->
We're biasing silicon again.
<!-- Author: Till :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231021' target=_top>231021</a>) -->

<!-- Date: Tue Dec 26 17:35:43.724 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231021>231021</A>
TERMINATE: cleaning up l3 and stuff
<!-- Author: Till -->



<!-- Date: Tue Dec 26 17:41:09.480 2006 -->
PCAL11 is giving us some problems during config.
<!-- Author: Till :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->
<!-- Comment: Tue Dec 26 17;42;47 comment by...Till -->
It turned red and during shepherding it says 
readback SMX slot 4 LUT word ....

<!-- Comment: Tue Dec 26 18;20;35 comment by...T -->
<pre>A second shepherding did the trick.</pre>
<!-- Date: Tue Dec 26 17:44:12.860 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022>231022</A>
 Activated at 2006.12.26 17:43:54
<!-- Author: RunControl -->


<!-- Date: Tue Dec 26 17:44:13.849 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022>231022</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Till -->
<!-- Comment: Tue Dec 26 17;45;29 comment by...Till -->
<b>Hooray!!!! We are back in business.</b>

<!-- Comment: Tue Dec 26 17;47;29 comment by...chlebana -->
<pre>Double Hooray!</pre>

<!-- Comment: Tue Dec 26 17;53;40 comment by...Aces -->
Good job Frank!

<!-- Comment: Tue Dec 26 18;07;01 comment by...Timo -->
Well Done Boys!! I should have had some beer and popcorn while watching how you handle the problems!


<!-- Comment: Tue Dec 26 18;42;56 comment by...Larry   -->
I was!
<!-- Comment: Tue Dec 26 21;23;59 comment by...Nagesh -->
cool.
<!-- Comment: Tue Dec 26 21;25;16 comment by...Nagesh -->
Thanks Frank.
<!-- Date: Tue Dec 26 18:20:02.514 2006 -->
<pre>The RTServer check in ProcMon spews out this error: 
 
2006.12.26 18:13:30 Status: CHECK FAILED  
Tue Dec 26 18:12:00 CST 2006   procAction  rtserver   check   b0server05 
Checking rtserver 
Found 1 copy of the rtserver watcher 'sscheck' script running (1 expected) 
Found 1 copy of the rtserver (rtserver.x) running (1 expected) 
Found 209 rtserver connections (near &gt;235 will degrade performance) 
Found 1 rtserver FIN_WAIT connections (&gt;0 may degrade performance) 
Found 0 rtserver FIN_WAIT connections from b0www (TOMCAT) 
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      
tcp        0  73849 b0server05.fnal.gov:5101    b0server02.fnal.gov:60312   FIN_WAIT1   root       0           
Found 1 potentially stuck ports  
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      
tcp        0  73849 b0server05.fnal.gov:5101    b0server02.fnal.gov:60312   FIN_WAIT1   root       0           
RTserver is not running correctly 
 
We tried to pin down the process that is causing this but were unable to do it. Candidates running on b0server02 are ACNET Monitor and scaler, downtime entries, all via the command start_acnet. 
We do not want to restart that at this time since we are taking good data and do not want to create problems.</pre>
<!-- Author: Naoki/Till :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->



<!-- Date: Tue Dec 26 18:29:07.112 2006 -->
<br>The CDF scrapbook is working fine so far. The plots are good.<br>
<!-- Author: Till :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->

<!-- Date: Tue Dec 26 18:37:14.387 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230996>230996</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=230996 target=_runSummary>230996</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->




<!-- Date: Tue Dec 26 18:43:48.517 2006 -->
<pre>Done TO by pcal02. Did not go to HALT, so I shepherded it a couple times. Then it had a corrupt heap, so it got shepherded again. Is behaving now.</pre>
<!-- Author: Till :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->


<!-- Date: Tue Dec 26 19:06:21.076 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16866&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16866  width='300' align='top'></a></td><td>PES West shows new contiguous dead channels, just short of 10 by two separating hot channels. I post it here as a reference for the next check.</td></tr></table>
<!-- Author: Chun -->


<!-- Date: Tue Dec 26 19:28:46.204 2006 -->
checklist OK. There are several doubtful plots as posted, but since this is only the 30 min. check, watching how they develop will be the action taken.
<!-- Author: Chun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->


<!-- Date: Tue Dec 26 19:29:17.827 2006 -->
Done timeout detected in crate CLC_00 and IMU_00 : aouto HRR
<!-- Author: Naoki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->
<!-- Comment: Tue Dec 26 20;37;11 comment by...Naoki -->
Done time out for crate b0htdc00 : auto HRR
<!-- Date: Tue Dec 26 19:32:52.643 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16869&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16869  width='300' align='top'></a></td><td>This plot is also in doubt, a new "dead" channel (white bar) in eta=6 region? The instructions said this region is in low statistics, and we are doing 30 mins. run check, so it might not be a problem, just post it for reference.</td></tr></table>
<!-- Author: Chun -->
<!-- Comment: Tue Dec 26 22;03;48 comment by...m mattson -->
Sometimes you need 20 or 30K events before all eta 6 channels are occupied.
<!-- Date: Tue Dec 26 19:49:59.886 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16870&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16871  width='300' align='top'></a></td><td>SVX VBias pinkie</td></tr></table>
<!-- Author: Till -->


<!-- Date: Tue Dec 26 20:15:35.173 2006 -->
<pre>I restarted the monitoring cron jobs on the b0cslxx machines</pre>
<!-- Author: chlebana -->


<!-- Date: Tue Dec 26 20:15:52.605 2006 -->
Done TO for crate b0imu00. : aout HRR
<!-- Author: Naoki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->



<!-- Date: Tue Dec 26 20:51:31.492 2006 -->
Scrapbooks of hourly plot look fine.
<!-- Author: Naoki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->


<!-- Date: Tue Dec 26 21:15:20.544 2006 -->
Alarm in ICI hertbeat : restart ICI
<!-- Author: Naoki/Till :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->


<!-- Date: Tue Dec 26 21:20:50.228 2006 -->
Done Timeout detected in crate CLC_00 : it is the 3rd, so I shepherd b0clc_00.
<!-- Author: Naoki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->




<!-- Date: Tue Dec 26 21:26:34.781 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16872&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16872' width='300'></a></center>
<!-- Author: Naoki -->
<!-- Comment: Tue Dec 26 21;29;16 comment by...Naoki -->
<pre>Other hourly plot is in scrapbook.
Is B0AAGC too spiky?</pre>


<!-- Date: Tue Dec 26 21:57:58.554 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16874&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16874  width='300' align='top'></a></td><td>I went through Checklist, PES still shown that "just short of 10" pattern. It didn't meet the Criteria to be marked as bad, but I decided to call the experts to see what I should mark and what we could do to correct this.</td></tr></table>
<!-- Author: Chun -->


<!-- Date: Tue Dec 26 22:12:49.346 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022>231022</A>
smx: <tt>PES west V layer (layer 1) shows new dead channels at Phi Octant 4, between strip 170 and 180, just short of 10 (separated by two hot channels). It doesn't meet the criteria to be marked bad (10 consecutive dead channels), however it is very close to this criteria. We paged the PES experts but it seems we got the wrong page number. I decid to mark it as "Not Yet", leave it to be decided by the experts.</tt>
<!-- Author: Consumer Operator(Chun Yang) -->



<!-- Date: Tue Dec 26 22:13:34.300 2006 -->
Done TO. for crate b0imu00 : HRR
<!-- Author: Naoki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022' target=_top>231022</a>) -->
<!-- Comment: Tue Dec 26 22;53;04 comment by...Naoki -->
Done TO. for crate b0imu00 : HRR

<!-- Comment: Tue Dec 26 22;57;53 comment by...NAoki -->
Done TO. for crate b-cot14 : shepherd 


<!-- Date: Tue Dec 26 23:22:30.003 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16875&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16875  width='300' align='top'></a></td><td>CCR East has developed several dead channels around Phi 14, pretty consistent these days. I post the plot here as a reference.</td></tr></table>
<!-- Author: Chun -->


<!-- Date: Tue Dec 26 23:35:49.578 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16876&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16876' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16877&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16877' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16878&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16878' width='300'></a></center>
<!-- Author: Hourly plot --Naoki -->
<!-- Comment: Tue Dec 26 23;37;07 comment by...Naoki -->
Unfortunately, Scrapbook dose not work.


<!-- Date: Tue Dec 26 23:39:52.674 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16879&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16879' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16880&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16880' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16881&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16881' width='300'></a></center>
<!-- Author: Hourly plot   -- Naoki -->

<!-- Date: Tue Dec 26 23:42:01.349 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=231022>231022</A>
smx: <tt>Same reason as last check, &nbsp;PES West has some dead channels, very close (though not reaching) the bad criteria.</tt>
<!-- Author: Consumer Operator(Chun Yang) -->


<!-- Date: Tue Dec 26 23:56:11.721 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=231022 target=_runSummary>231022</A> x3866E</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>17:43:54</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>05:43:48</TD>
<TD ALIGN=RIGHT>   2,094,107</TD>   <TD ALIGN=RIGHT>  1,545.62</TD>
<TD ALIGN=RIGHT>  1,427.25</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>05:43:48</TD>
<TD ALIGN=RIGHT>     2,094,107</TD> <TD ALIGN=RIGHT>  1,545.62</TD>
<TD ALIGN=RIGHT>  1,427.25</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Dec 26 23:59:30.112 2006 -->
<font color=red>Shift Summary: </font><pre>We had only ~67% efficiency due to CSL problem 
 
~ 17:30  stuck with CSL problem 
 
  17:32  OK after rebooting b0spool. thanks Frank 
 
  17:44 start new run (#231022) 
 
    was quiet 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   231021,231022
Delivered Luminosity   2070.2  
Acquired Luminosity    1386.1  
Efficiency             67%
   
</td></tr></table></pre>
<!-- Author: DongHee Kim -->