<!-- Title: Day shift. Mon Nov 27, 2006 -->
<!-- SciCo: Xin Wu -->
<!-- DAQAce: Ulysses Grundler -->
<!-- MonAce: Long Zhang -->
<!-- CO: Min Jeong Kim -->
<!-- OpManager: Koji Sato -->
<!-- Notes: -->
<pre>Store 5095 colliding. Take over run 228738. L=40E30</pre>


<!-- Date: Mon Nov 27 08:54:38.572 2006 -->
<pre>Done Timeout detected in crates: 
IMU_01 
 
Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 - Timeout on global TDC Done  for crate b0imu01 
 
auto HRR fixed</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 08:55:52.834 2006 -->
<pre>Trigger Inhibit is set from the components: 
 SVX06 Trip  		[006] Silicon Tracker Trip 06 
 
Halted run (didn't halt automatically because was in middle of HRR from IMU done TO)</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 08:57:36.953 2006 -->
<pre>Trigger Inhibit is set from the components: 
 SVX06 Trip  		[006] Silicon Tracker Trip 06 
 ISL02 Trip  		[010] Intermediate Silicon Trip 02 
 SVX HV:iFix  		[128] Silicon Tracker High Voltage from iFix 
 
still halted</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 08;58;51 comment by...Ulysses -->
recovered and back to running


<!-- Date: Mon Nov 27 09:01:20.855 2006 -->
<pre>RTServer and File System went red in ProcMon.  RTServer recovered quickly, File System took a little longer, but everything's running again.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 09:13:00.803 2006 -->
<pre>TevMon giving warnings about stale data, will wait a few minutes and see if it goes away.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 09;34;38 comment by...Ulysses -->
Did not go away, Silicon experts told me to restart it, and it's working now.
<!-- Date: Mon Nov 27 09:18:26.640 2006 -->
<pre>ProcMon giving another error in RTServer, check says: 
 
Found 1 potentially stuck ports  
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      
tcp        0  73848 b0server05.fnal.gov:5101    b0server02.fnal.gov:38173   ESTABLISHED cdfdaq     673529878   
RTserver is not running correctly 
 
will give it a minute to clear.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 09:21:24.916 2006 -->
We had a trip of SVX B5W9L0 on spike in the DVDD current. Recovered fine, but now (and possibly at the same time as the trip happened), both SB5W8L0 and SV5W9L0 show a sharp drop in their AVDD and DVDD currrents.  At the moment, both ladders only return corrupted data. Since both of them share the same FIB board, this points to a problem with this board. Investigating.
<!-- Author: Ulrich -->
<!-- Comment: Mon Nov 27 09;43;06 comment by...Ulrich, Jose, Mark -->
We reset b0fib06 and the crate cpu failed to connect the rtserver the first time around.  After restarting the rtserver, we reset again and saw the following while checking/loading the sequences:
<pre>
Bad Seq Ram Slot 19 IDO e589 Calc 000155aa Expect 0001552a
</pre>
This sequence was rewritten and we successfully took the fib to Halt and then started taking data again.  The currents have recovered (see below), there are no new errors in SVXMon and the data looks fine.
<!-- Date: Mon Nov 27 09:33:52.085 2006 -->
<pre>RTServer was having problems with a stuck port, eventually had to restart.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->



<!-- Date: Mon Nov 27 09:38:57.397 2006 -->
<pre>Another error with RTServer: 
 
Found 1 copy of the rtserver watcher 'sscheck' script running (1 expected) 
Found 1 copy of the rtserver (rtserver.x) running (1 expected) 
Found 211 rtserver connections (near &gt;235 will degrade performance) 
Found 1 rtserver FIN_WAIT connections (&gt;0 may degrade performance) 
Found 0 rtserver FIN_WAIT connections from b0www (TOMCAT) 
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      
tcp        0  73849 b0server05.fnal.gov:5101    b0server02.fnal.gov:38173   FIN_WAIT1   root       0           
Found 1 potentially stuck ports  
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      
tcp        0  73849 b0server05.fnal.gov:5101    b0server02.fnal.gov:38173   FIN_WAIT1   root       0           
RTserver is not running correctly</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 10;43;31 comment by...Ulysses -->
DAQ expert (didn't catch his name) said he'd log in and try and determine the problem
<!-- Comment: Mon Nov 27 10;53;42 comment by...badgett -->
I'm investigating the problem...
<!-- Date: Mon Nov 27 09:47:20.416 2006 -->
<pre>stale data warning from TevMon again</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 09:47:25.090 2006 -->
Looks like the curious behavior of CP2 in 6E is power supply noise.  Seems to be ~10 counts of coherent noise there, code finds but does not fix it.  Data should be marked ok.  Can "liesurely" decide what to do about recently installed (and thus suspect) power supply.
<!-- Author: Larry   :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 09:54:30.687 2006 -->
<pre>SVXMon not sending heart beat to ConsumerErrorReceiver (aka ConsumerMonitor) 
  since 2006-11-27 09:44:27.0 
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon 
  and these processes are very important to data taking! 
Restart SVXMon immediately! 
 
had CO restart SVXMon</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->

<!-- Date: Mon Nov 27 09:54:55.485 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738>228738</A>
cal: <tt>The total number of dead channels in WHAPMTCanvas is same compared to the reference plot. However, there is a change in the channel numbers.<br>
I see the channel West-W22-T6 dead instead of East-W21-T9. However, the tower 6 is a overlapping tower with CHA, so to consider this as a real problem we might need more statistics. At the end of the previous shift, YMon has been restarted so I mark WHA "not yet" in the checklist and will monitor during the whole shift.</tt>
<!-- Author: Consumer Operator( Min Jeong Kim) -->


<!-- Date: Mon Nov 27 09:54:56.787 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738>228738</A>
monitors: <tt>ObjectMon:CMX z(stub) has bad chi2 (7.2, it should be <=3). At the end of shift, will e-mail the experts if it remains.</tt>
<!-- Author: Consumer Operator( Min Jeong Kim) -->



<!-- Date: Mon Nov 27 09:55:47.475 2006 -->
<pre>recieved error from ProcMon about CSL loggers: 
 
2006.11.27 09:50:41 Status: CHECKING 
2006.11.27 09:52:41 Status: CHECK FAILED java.lang.InterruptedException 
 
told this may not be a real error, just one in checking.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 09:57:33.248 2006 -->
<pre>DQMon giving errors about several consumers not updating (this is the second time this has happened).  All but SVXMon have gone back to green.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 10;00;27 comment by...Ulysses -->
SVXMon still red probably because CO is restarting it after an error message told us we should

<!-- Comment: Mon Nov 27 10;08;42 comment by...Min Jeong -->
We restarted SVXMon.


<!-- Date: Mon Nov 27 10:10:06.718 2006 -->
<pre>Getting errors on File Systems: 
 
Error: vxworks is over quota: 
quota: Error while getting quota from b0spool:/vol/data for 2064: Invalid argument 
Error: cdfcalib is over quota: 
quota: Error while getting quota from b0spool:/vol/data for 5574: Invalid argument 
Error: level3 is over quota: 
quota: Error while getting quota from b0spool:/vol/data for 10351: Invalid argument 
Disk systems and quota check failed 
 
RTServer still complaining as well</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 10;15;00 comment by...Ulysses -->
File Systems back to green


<!-- Date: Mon Nov 27 10:19:58.534 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15060&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15060' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15061&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15061' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15062&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15062' width='300'></a></center>
<!-- Author: Long  07:30--09:30 -->


<!-- Date: Mon Nov 27 10:30:18.709 2006 -->
<pre>TevMon failing to receive Acnet messages via RTServer. 
Also got stale data warnings from TevMon, but this cleared quickly.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 10:39:06.586 2006 -->
<pre>Done Timeout detected in crates: 
PCAL_11 
 
Auto HRR got stuck trying to halt b0pcal11, shepherding.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 10;41;28 comment by...Ulysses -->
Attention!!!. FERML_SHEPHERD_FAILED !!!. 
<!-- Comment: Mon Nov 27 10;44;41 comment by...Ulysses -->
Attention!!!. FERML_SHEPHERD_FAILED !!!. 
<!-- Comment: Mon Nov 27 10;50;23 comment by...Ulysses -->
Tried once more: Attention!!!. FERML_SHEPHERD_FAILED !!!. 
<!-- Comment: Mon Nov 27 10;53;44 comment by...Ulysses -->
<pre>b0pcal11 had gotten back to ready state, so I activated that single client and then halted it to bring it in line with the rest of the clients.  Once that was done the auto HRR took over again, recovered and running again.</pre>
<!-- Date: Mon Nov 27 10:40:13.591 2006 -->
<pre>TevMon warning: LOSTP Monitor May not be working.  This cleared right away.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->








<!-- Date: Mon Nov 27 10:55:19.230 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><P> 
<FONT COLOR=RED> 
In the next few minutes, I will be restarting  
the AcnetMonitor on b0server02.   You may see  
ProcMon and/or TevMon errors. 
</FONT> 
<P> 
</td></tr></td></table>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 11;03;54 comment by...Ulysses -->
TevMon did say it was failing to recieve Acnet messages via RTServer at 10:59


<!-- Date: Mon Nov 27 11:04:31.131 2006 -->
<pre>RTServer is back to green now.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 11:05:27.545 2006 -->
In preparation for the automatic hockerization mechanism, I have updated a Silicon power supply monitoring program running on the Silicon iFix PC (PSMon.exe). Please page Silicon Cooling (218-8626) if you encounter any problems with Silicon PS monitoring and iFix.
<!-- Author: Ulrich -->


<!-- Date: Mon Nov 27 11:05:36.581 2006 -->
<pre>FileSystems complaining about disk quotas again, checked again and the warning is gone.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 11:06:27.745 2006 -->
Got an call from MCR about plans for the store: 
- 12:00 Tevetron studies (TEL 2 single bunch measurement) for two       
  hours, transparent to data-taking 
- 14:00 End of store 
<!-- Author: Xin Wu :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 11;27;31 comment by...Xin Wu -->
MCR moved the end of store to 14:30.
<!-- Date: Mon Nov 27 11:10:06.027 2006 -->
<P> 
Have finished restarting AcnetMonitor, AcnetScaler  
and DownTimeCopy on b0server02. 
<P> 
There were at least two problems: 
<UL> 
<LI>The AcnetScaler application had about 70kbytes stuck in  
its network receive queue.   This was causing the ProcMon  
rtserver complaints. 
<LI>The DownTimeCopy application had spawned a root subprocess  
to create the various store summary plots.   This root  
subprocess was in <I>zombie</I> mode, using up all the CPU  
on one of the four (half of two) processors on b0server02. 
</UL> 
Restarting the processes cleared up the root hang and  
rtserver complaints.   From the log files, it's not clear why  
either problem was occurring.
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 11:18:28.792 2006 -->
<pre>Recieved another RTServer error message from ProcMon: a potentially stuck port.  This seems to have gone away quickly.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 11:18:50.826 2006 -->
<pre>Done Timeout detected in crates: 
IMU_00 
 
auto HRR fixed.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->


<!-- Date: Mon Nov 27 11:21:06.577 2006 -->
<pre>TevMon Warning: LOSTP Monitor May not be working. 
Must have been intermittent as the box was green when I looked.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 11;29;07 comment by...Ulysses -->
Got this again, went away quickly.
<!-- Comment: Mon Nov 27 11;36;56 comment by...Ulysses -->
again
<!-- Date: Mon Nov 27 11:22:43.697 2006 -->
<pre>evb2b was red, but didn't get an error message.  Did HRR and everything seems to be fine.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->





<!-- Date: Mon Nov 27 11:45:39.116 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15066&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15066' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15067&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15067' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15068&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15068' width='300'></a></center>
<!-- Author: Long  09:30--11:30 -->


<!-- Date: Mon Nov 27 11:46:01.412 2006 -->
<pre>Another File Systems error about disk quotas: said cdfcalib was over quota, checked again and said cdfdaq and vxworks over quota.  Checked one more time and it said level3 over quota.  Seeing if this can be fixed.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->
<!-- Comment: Mon Nov 27 11;57;38 comment by...Ulysses -->
Finally cleared (when we started new run)
<!-- Date: Mon Nov 27 11:54:33.327 2006 -->
Updated the hardware database to mask the problematic SL6 finder boards.  This requires a new run, to take effect.
<!-- Author: Nils :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738' target=_top>228738</a>) -->

<!-- Date: Mon Nov 27 11:56:07.064 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738>228738</A>
 Terminated at 2006.11.27 11:56:05
<!-- Author: RunControl -->


<!-- Date: Mon Nov 27 11:57:17.567 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738>228738</A>
TERMINATE: new XFT dead wire list 
<!-- Author: Nate -->




<!-- Date: Mon Nov 27 12:16:51.669 2006 -->
<pre>Problems in starting new run.  In particular ccal01 and ccal05 are going unresponsive (giving errors that last data more than 120s ago).  Have tried shepherding these crates, but not having any luck.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228739' target=_top>228739</a>) -->

<!-- Date: Mon Nov 27 12:19:45.826 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228739>228739</A>
 Activated at 2006.11.27 12:19:10
<!-- Author: RunControl -->


<!-- Date: Mon Nov 27 12:19:46.987 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228739>228739</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [1,842,657] 
<!-- Author: Nate -->


<!-- Date: Mon Nov 27 12:31:22.409 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738>228738</A>
cal: <tt>The channel West-W22-T6 of WHAPMTCanvas is O.K. now with more data events. In the previous checklist, WHA was marked with "not yet" due to this channel.</tt>
<!-- Author: Consumer Operator( Min Jeong Kim) -->


<!-- Date: Mon Nov 27 12:31:23.016 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228738>228738</A>
general: <tt>This is checklist at the end of run 228738.</tt>
<!-- Author: Consumer Operator( Min Jeong Kim) -->


<!-- Date: Mon Nov 27 12:35:12.405 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228739>228739</A>
 Terminated at 2006.11.27 12:34:58
<!-- Author: RunControl -->


<!-- Date: Mon Nov 27 12:35:45.969 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228739>228739</A>
TERMINATE: problems with ccal01 and ccal05 starting
<!-- Author: Nate -->


<!-- Date: Mon Nov 27 12:35:46.778 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228739>228739</A>
TERMINATE: problems with ccal01 and ccal05 starting
<!-- Author: Ulysses -->


<!-- Date: Mon Nov 27 12:50:34.331 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228740>228740</A>
TERMINATE: no data taken
<!-- Author: Ulysses -->


<!-- Date: Mon Nov 27 13:01:35.100 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
 Activated at 2006.11.27 13:01:10
<!-- Author: RunControl -->


<!-- Date: Mon Nov 27 13:01:36.919 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [1,842,657] 
<!-- Author: Ulysses -->


<!-- Date: Mon Nov 27 13:01:37.767 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
 Activated at 2006.11.27 13:01:10
<!-- Author: RunControl -->


<!-- Date: Mon Nov 27 13:01:38.113 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [1,842,657] 
<!-- Author: Ulysses -->



<!-- Date: Mon Nov 27 13:06:39.711 2006 -->
<pre>Done Timeout detected in crates: 
IMU_01 
 
Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 - Timeout on global TDC Done  for crate b0imu01    
 
Auto HRR fixed.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 13:10:55.744 2006 -->
<P> 
There's something very wrong with the NFS file server  
for the B0 online Linux cluster  (b0home or b0spool are  
the NFS servers).   During the last config, access to the  
NFS mounted disks was glacially slow.    The CCAL_00...15  
and the PCAL_01,02,04,05,07,08,10,11 crates contain  
SMX cards, whose configuration procedure reads the calibration  
file from the /cdf/onln/data/ disk served by b0spool, and  
this procedure was so slow that all of the crates turned  
red, for unresponsive, in RunControl.    Eventually they  
completed their CONFIG transition after a few minutes &minus;  
our patience was rewared and we began a run successfully. 
<P> 
Have sent urgent e-mail to the Rex sysadmin group. 
 
<P> 
 
  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->
<!-- Comment: Mon Nov 27 13;39;55 comment by...Koji -->
E-mail reply from Jason at 13:33 :
<pre>
Something/Someone currently has the processor pegged on the NFS server b0spool /cdf/onln/spool /cdf/onln/data. I'm looking into this.

Jason Hedden 
</pre>
<!-- Date: Mon Nov 27 13:21:15.891 2006 -->
The funny results from ProcMon check for quotas 
seem to be caused by a slow or non-response from b0spool. 
This may just be heavy usage or may be more serious. 
I have sent a note to sysadmins.  
<!-- Author: Ray Culbertson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 13:22:53.541 2006 -->
<pre>SVXMon not sending heart beat to ConsumerErrorReceiver (aka ConsumerMonitor) 
  since 2006-11-27 13:01:25.0 
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon 
  and these processes are very important to data taking! 
Restart SVXMon immediately! 
 
Having CO restart SVXMon</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 13:25:36.689 2006 -->
<pre>Done Timeout detected in crates: 
HADRON_TIMING_00 
 
auto HRR fixed</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 13:30:56.620 2006 -->
<pre>Another File System warning about disk quotas.  This time it was cdfcalib, but it's gone now.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->
<!-- Comment: Mon Nov 27 13;49;13 comment by...badgett -->
Most likely, this error was due to a timeout when checking 
the quota, not the quota itself.

<!-- Comment: Mon Nov 27 13;56;46 comment by...Ulysses -->
<pre>Got the error again:

Error: cdfdaq is over quota:
quota: Error while getting quota from b0spool:/vol/data for 5430: Invalid argument

cleared again.</pre>
<!-- Date: Mon Nov 27 13:37:47.949 2006 -->
<pre>From  	Jason Hedden &lt;jhedden@fnal.gov&gt;  
Sent  	Monday, November 27, 2006 1:33 pm 
To  	CDF Operations Managers &lt;cdf_ops@fnal.gov&gt; , cdf-rc-support@fnal.gov  
Cc  	Run2 Sys &lt;run2-sys@fnal.gov&gt;  
Subject  	b0spool time outs due to high utilization 
 
Something/Someone currently has the processor pegged on the NFS server 
b0spool /cdf/onln/spool /cdf/onln/data. I'm looking into this. 
 
Jason Hedden 
</pre>
<!-- Author: Jason Hedden :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->





<!-- Date: Mon Nov 27 13:58:27.571 2006 -->
<pre>Done Timeout detected in crates: 
IMU_01 
 
Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 - Timeout on global TDC Done  for crate b0imu01 
 
auto HRR fixed.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:01:52.224 2006 -->
End of store in 30 minuts.
<!-- Author: Xin WU :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:11:11.322 2006 -->
<pre>Error in CSL Loggers: 
 
2006.11.27 14:09:32 Status: CHECK FAILED java.lang.InterruptedException 
 
error went away.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:11:28.590 2006 -->
From  	Jason Hedden <jhedden@fnal.gov>  
Sent  	Monday, November 27, 2006 2:00 pm 
To  	William Badgett <badgett@fnal.gov>  
Subject  	Re: Online Slowage 
 
Hi Bill, 
 
I'm not sure what normal operations really look like, the only Online 
system I've found unusual is b0server03 with 489 files open to the 
b0spool, and idea if this is normal? 
 
Jason  
<!-- Author: Jason Hedden :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:11:44.301 2006 -->
<pre>From  	Jason Hedden &lt;jhedden@fnal.gov&gt;  
Sent  	Monday, November 27, 2006 2:00 pm 
To  	William Badgett &lt;badgett@fnal.gov&gt;  
Subject  	Re: Online Slowage 
 
Hi Bill, 
 
I'm not sure what normal operations really look like, the only Online 
system I've found unusual is b0server03 with 489 files open to the 
b0spool, and idea if this is normal? 
 
Jason 
</pre>
<!-- Author: Jason Hedden :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:12:18.753 2006 -->
<pre>From  	William Badgett &lt;badgett@fnal.gov&gt;  
Sent  	Monday, November 27, 2006 2:08 pm 
To  	cdf_csl_upgrade@fnal.gov  
Subject  	Calibration CSL wreaking havoc? 
 
Dear CSLers, 
 
There are an extraordinary number of csl_logger on 
node b0server03.    Which is very strange, since no one 
is taking calibrations right now. 
 
Also, we are suspecting that the 489 open files on 
b0server03 are owned by csl_logger processes, and 
are causing the NFS server b0spool to be clobbered. 
 
Could you please have a look, as soon as possible, 
before we kill the calibration CSL? 
 
Thanks, 
Bill 
</pre>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:14:58.204 2006 -->
<pre>From  	Jason Hedden &lt;jhedden@fnal.gov&gt;  
Sent  	Monday, November 27, 2006 2:08 pm 
To  	cdf_csl_upgrade@fnal.gov  
Cc  	Run2 Sys &lt;run2-sys@fnal.gov&gt;  
Subject  	b0spool /cdf/onln/spool /cdf/onln/data 
 
It looks like we're receiving a lot of NFS requests from the new CSL 
nodes. Could someone please verify that everything is running as 
expected, or let me know of any testing that might be causing this? 
 
This is causing problems for the entire Online cluster, we need to stop 
whatever is causing this. 
 
Thanks, 
Jason Hedden 
 
 
 
131.225.236.244 b0csl23.fnal.gov                NFSOPS =      19773 (12%) 
131.225.236.247 b0csl26.fnal.gov                NFSOPS =      17789 (11%) 
131.225.236.248 b0csl27.fnal.gov                NFSOPS =      17486 (10%) 
131.225.236.249 b0csl28.fnal.gov                NFSOPS =      16477 (10%) 
131.225.236.243 b0csl22.fnal.gov                NFSOPS =      16431 (10%) 
131.225.236.246 b0csl25.fnal.gov                NFSOPS =      16336 (10%) 
131.225.236.250 b0csl29.fnal.gov                NFSOPS =      11655 ( 7%) 
131.225.236.160 b0csl21.fnal.gov                NFSOPS =       7785 ( 5%) 
131.225.236.241 b0csl20.fnal.gov                NFSOPS =       4870 ( 3%)</pre>
<!-- Author: Jason Hedden :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->

<!-- Date: Mon Nov 27 14:19:49.762 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
 Terminated at 2006.11.27 14:19:33
<!-- Author: RunControl -->


<!-- Date: Mon Nov 27 14:19:50.385 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
TERMINATE: End of store
<!-- Author: Ulysses -->



<!-- Date: Mon Nov 27 14:19:50.656 2006 -->
My XFT tests didn't work out, because TrigMon is down.  I returned the XFT deadwire list back to the original state.  Now everything is as it was yesterday and we should be ready for the next store.  I will try to make another attempt at this tomorrow.
<!-- Author: Nils :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->


<!-- Date: Mon Nov 27 14:21:00.638 2006 -->
Guem Bong is investigated the CSL problem.
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->
<!-- Comment: Mon Nov 27 14;21;17 comment by...bb -->
investigating


<!-- Comment: Mon Nov 27 16;03;31 comment by...Richard, Geumbong -->
The only process, which calls "/cdf/onln/data", is csl_logger for writing out log files. At that time, we expect lack of space for log files and I started deleting old logger files(made before Sep).
It got the speed back after rebooting b0spool(below). 
<!-- Date: Mon Nov 27 14:23:18.559 2006 -->
<pre>TevMon Warning: BOILUM Decreased Suddenly. 
 
Now is blue.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741' target=_top>228741</a>) -->
<!-- Comment: Mon Nov 27 14;28;01 comment by...Ulrich -->
Did you power down the CLC for end of store? Then you don't measure luminosity any more.
<!-- Date: Mon Nov 27 14:27:43.908 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15069&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15069' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15070&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15070' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15071&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15071' width='300'></a></center>
<!-- Author: Long  11:30--13:30 -->



<!-- Date: Mon Nov 27 14:37:58.025 2006 -->
<pre>another File Systems error: 
 
Error: level3 is over quota: 
quota: Error while getting quota from b0spool:/vol/data for 10351: Invalid argument 
 
went away by itself</pre>
<!-- Author: Ulysses -->


<!-- Date: Mon Nov 27 14:41:04.873 2006 -->
Store has been terminated. MCR goes into shot setup right away.
<!-- Author: Xin Wu -->


<!-- Date: Mon Nov 27 14:49:22.913 2006 -->
<P> 
The <i>fer</I> product has arisen to version <i>v5_27</I>  
with the changes: 
<UL> 
<LI>ProcMon updates for the new CSL (Ray Culbertson) 
<LI>DQMon updates from Oriol Salto 
<LI>Finally fix the L2 lumi-enable prescale problem where  
the L2 lumi enable prescale changes were not recorded in  
the run database.   They were applied, though.  Now they  
will be both. 
</UL> 
Restarted DAQ Ace RunControl. 
<P> 
  
<!-- Author: badgett -->


<!-- Date: Mon Nov 27 15:19:59.316 2006 -->
<pre>Continuing to get File Systems errors.</pre>
<!-- Author: Ulysses -->


<!-- Date: Mon Nov 27 15:34:41.655 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>Frank Chlebana called, they are rebooting b0spool, and no one should be running anything until the reboot is complete.</td></tr></td></table>
<!-- Author: Koji -->
<!-- Comment: Mon Nov 27 16;13;58 comment by...koji -->
They finished rebooting during the final proton injection, and we switched to aaa_shotsetup immediately.
<!-- Date: Mon Nov 27 15:34:59.360 2006 -->
<pre>CSL Loggers error in ProcMon: 
 
Mon Nov 27 15:31:34 CST 2006   procAction  csllog   check   b0control12 
Checking CSL Logger Clients 
Found 0 CSL logger client processes on b0csl22, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl23, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl24, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl25, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl26, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl27, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl28, (&gt;0 expected) 
Found 0 CSL logger client processes on b0csl29, (&gt;0 expected) 
CSL Logger check failed 
</pre>
<!-- Author: Ulysses -->
<!-- Comment: Mon Nov 27 16;10;58 comment by...chlebana -->
<pre>We had stopped the CSL processes when trying to determine the 
source of high cpu usage on b0spool...</pre>
<!-- Date: Mon Nov 27 15:39:18.578 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15072&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15072' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15073&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15073' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-15074&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-15074' width='300'></a></center>
<!-- Author: Long  13:30--15:30 -->

<!-- Date: Mon Nov 27 15:40:28.835 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
trigger: <tt>During this run, we had a trouble to run TrigMon due to several reasons (CLS, network, or some other), so </tt>
<!-- Author: Consumer Operator( Min Jeong Kim) -->
<!-- Comment: Mon Nov 27 15;46;48 comment by...Min Jeong -->
(continue) a small number of events in TrigMon histograms.
<!-- Date: Mon Nov 27 15:40:30.225 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228741>228741</A>
general: <tt>This run last for around an hour: a low statistics run to judge some plots.</tt>
<!-- Author: Consumer Operator( Min Jeong Kim) -->




<!-- Date: Mon Nov 27 15:49:47.873 2006 -->
<pre>Loading final protons.</pre>
<!-- Author: Xin Wu -->

<!-- Date: Mon Nov 27 15:56:02.024 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=228738 target=_runSummary>228738</A> x37D82</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [1,842,657]</FONT></TD>
<TD ALIGN=CENTER>22:52:29</TD>
<TD ALIGN=CENTER>11:56:05</TD>
<TD ALIGN=CENTER>11:49:23</TD>
<TD ALIGN=RIGHT>   3,778,119</TD>   <TD ALIGN=RIGHT>  2,349.35</TD>
<TD ALIGN=RIGHT>  2,136.12</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=228741 target=_runSummary>228741</A> x37D85</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [1,842,657]</FONT></TD>
<TD ALIGN=CENTER>13:01:10</TD>
<TD ALIGN=CENTER>14:19:33</TD>
<TD ALIGN=CENTER>01:12:27</TD>
<TD ALIGN=RIGHT>     354,800</TD>   <TD ALIGN=RIGHT>    137.21</TD>
<TD ALIGN=RIGHT>    125.64</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>13:01:51</TD>
<TD ALIGN=RIGHT>     4,132,919</TD> <TD ALIGN=RIGHT>  2,486.56</TD>
<TD ALIGN=RIGHT>  2,261.75</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Mon Nov 27 15:58:24.080 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228751>228751</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_01 [1,842,657] 
<!-- Author: Ulysses -->



<!-- Date: Mon Nov 27 15:59:08.931 2006 -->
Done TO for b0l1cl00 Hrred
<!-- Author: Ulyses && Zeynep :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228751' target=_top>228751</a>) -->


<!-- Date: Mon Nov 27 16:00:35.082 2006 -->
<font color=red>Shift Summary: </font><pre>  - Shift started with store 5095 colliding. Run 228738. L=40E30 
  - Recurrent ProcMon RTSERVER warning and ProcMon FileSystem     
    warning. Probably related to the CSL problem later 
  - SVX tripped once. A FIB crate had to be reset 
  - Changed run to use new XFT dead wire list. 
    Took one hour to restart the run because of failues to  
    configure Calorimeter crates. It was later traced to problem    
    with CSL 
  - Store terminated on 14:30. Shot setup followed 
  - Experts worked on solving the CSL ongoing</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   228738, 228741
Delivered Luminosity   756.3 nb-1  
Acquired Luminosity    547.9 nb-1  
Efficiency             	 72.5
   
</td></tr></table></pre>
<!-- Author: Xin Wu -->


<!-- Date: Mon Nov 27 16:03:14.275 2006 -->
<pre>We were getting sluggish nfs response for the spool and data  
area. Saw that the CPU usage on b0spool (file server) was near 
100%. 
 
At first it looked like the csl machines was causing the high 
usage - but after stopping the processes on these machines we 
still saw high cpu usage on the file server. 
 
We rebooted b0spool and continue to see large cpu usage, although it appears a little better. 
 
We will continue to monitor it...</pre>
<!-- Author: chlebana -->





<!-- Date: Mon Nov 27 16:15:41.970 2006 -->
<pre>Calibration CSL problem: 
As Bill said, many process(78 process) is existing on b0server03. According to process monitor, it seems that old processes from several days ago are remained. After restarting, old process is killed correctly. Now 6 processes are working properly.   
We are investigating this problem.</pre> <font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: tatsuya -->