<!-- Title: Owl shift. Wed Aug  4, 2004 -->
<!-- SciCo: Steve Worm -->
<!-- DAQAce: Else Lytken -->
<!-- MonAce: Masato Aoki -->
<!-- CO: Oleg Suslov -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre></pre>


<!-- Date: Wed Aug 4 00:07:19.007 2004 -->
<pre>Panic message from b0dau35: 
red Wed Aug 4 00:03:57 CDT 2004 - Disk on b0dau35,fnal,gov at PANIC level 
 
red /ora1 (100%) has reached the defined disk space PANIC level (100%) 
 
swap                 2087320   41512 2045808     2%    /tmp 
/dev/vx/dsk/oradg/trigvol 34603008  932477 31566129     3%    /dbs10 
/dev/vx/dsk/oradg/hdwvol 34603008 1955213 30607314     7%    /dbs12 
/dev/vx/dsk/oradg/toolman 34812722 3351122 30478478    10%    /toolman 
/dev/vx/dsk/oradg/oractlvol1  204800   29845  164027    16%    /dbs1 
/dev/vx/dsk/oradg/hdwindvol 34603008 5027221 27727307    16%    /dbs13 
/dev/vx/dsk/oradg/oractlvol2  204800   29837  164034    16%    /dbs2 
/dev/vx/dsk/oradg/oractlvol3  204800   29837  164034    16%    /dbs7 
/dev/vx/dsk/oradg/tempdatvol 34603008 7926776 25008973    25%    /dbs24 
/dev/vx/dsk/oradg/orasysdatvol 2097152  539567 1460242    27%    /dbs5 
/dev/vx/dsk/oradg/mcsvol 71303168 20076176 50426576    29%    /dbs14 
/dev/vx/dsk/oradg/rolltmp 34603008 13989640 20452336    41%    /dbs60 
/dev/vx/dsk/oradg/trigindvol 34603008 13944543 19367317    42%    /dbs11 
/dev/vx/dsk/rootvol  1549594  710596  777015    48%    / 
/dev/vx/dsk/oradg/archlogvol 71303168 36190070 34017108    52%    /dbs8 
/dev/vx/dsk/var      2055705 1056709  937325    53%    /var 
/dev/vx/dsk/oradg/runindvol 35651584 19261375 15365826    56%    /dbs17 
/dev/vx/dsk/oradg/backupvol 286726144 175905024 109955416    62%    /backup 
/dev/vx/dsk/oradg/rundbvol 35651584 22026143 12773856    64%    /dbs16 
/dev/vx/dsk/usr      2981450 2276873  644948    78%    /usr 
/dev/vx/dsk/oradg/redovol1  204800  165014   37304    82%    /dbs26 
/dev/vx/dsk/oradg/calibvol 106954752 87656696 18996588    83%    /dbs3 
/dev/vx/dsk/oradg/orausrvol 4194304 3878426  296205    93%    /dbs6 
/dev/vx/dsk/oradg/mcsindvol 71303168 69740688 1538068    98%    /dbs15 
/dev/vx/dsk/oradg/calibindvol 71303168 69740816 1538004    98%    /dbs4 
/dev/vx/dsk/oradg/orasysvol 6291456 6291456       0   100%    /ora1 
 
 
 
Status unchanged in 15.08 minutes 
Status message received from 131.225.236.185 
</pre>
<!-- Author: Else :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894' target=_top>185894</a>) -->
<!-- Comment: Wed Aug  4 01;20;27 comment by...Dmitry Litvintsev -->
OK, I logged in and deleted directory with core file:

cd /ora1/oracle/v9_2_0/dbs
rm -rf  core_15656

> df -k | grep ora1
/dev/vx/dsk/oradg/orasysvol 6291456 5490384  788910    88%    /ora1

<!-- Date: Wed Aug 4 00:19:26.400 2004 -->
Heartbeat alarm from TOF. Restarted smacs.
<!-- Author: Masato -->


<!-- Date: Wed Aug 4 00:20:03.324 2004 -->
Aces who are leaving report water coming in West doors (by mailboxes).
<!-- Author: Worm -->


<!-- Date: Wed Aug 4 00:23:05.465 2004 -->
A. Sill calls about the problem above... he says it is the Oracle system disk and it is worth paging someone.
<!-- Author: Worm -->
<!-- Comment: Wed Aug  4 00;32;38 comment by...Alan Sill -->
See <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=database&action=view&page=197&frame=2&anchor=16:00:45'>
this link </a> for an example of this same disk space filling up in the past, and what was done by the DBAs about it.

Definitely worth paging!
<!-- Date: Wed Aug 4 00:28:25.713 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0pcal05   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->
<!-- Comment: Wed Aug  4 00;32;23 comment by...Else -->
auto-HRR fixed this again. This error was already common on the evening shift.


<!-- Comment: Wed Aug  4 00;41;46 comment by...Else -->
Again ....

<!-- Comment: Wed Aug  4 01;11;25 comment by...Else -->
One more.


<!-- Comment: Wed Aug  4 01;25;19 comment by...Else -->
Again!

<!-- Comment: Wed Aug  4 01;40;05 comment by...Else -->
Again. Strange, as far as I can tell from the errorfile there is nothing special. But auto-HRR always clears it. 

<!-- Comment: Wed Aug  4 02;01;06 comment by...Else -->
Another one.

<!-- Comment: Wed Aug  4 02;13;01 comment by...Else -->
Again.

<!-- Date: Wed Aug 4 02:18:47.352 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0pcal01   b0pcal02   b0pcal04   b0pcal05   b0pcal07   b0pcal08   b0pcal10   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->
<!-- Comment: Wed Aug  4 02;28;04 comment by...Else -->
<br> This one was also seen on evening shift. <br>And now another pcal05 + pcal11 timeout - <br>
Something may need to be cleaned up before datataking. <br>
The checks in the ace control panel looks ok, except for few FISHY -> retry's in the SCRAMnet check and then this line for <br>c0l3c11: SRI 0x20a0:(Protocol err:BAD)(Message err:BAD). <br> 
<!-- Comment: Wed Aug  4 02;34;07 comment by...Guillelmo -->
Try again, and maybe it will dissapear
<!-- Comment: Wed Aug  4 02;39;38 comment by...Else -->
Yes, now everything is just plain GOOD.
<!-- Date: Wed Aug 4 02:26:49.890 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0pcal05   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->
<!-- Comment: Wed Aug  4 03;44;59 comment by...Else -->
Had <b>two </b> more of these, right after the done timeout for l2de01.
<!-- Comment: Wed Aug  4 03;54;20 comment by...Else -->
One more.
<!-- Comment: Wed Aug  4 04;14;07 comment by...Else -->
Again.
<!-- Comment: Wed Aug  4 04;16;28 comment by...Else -->
and again...
<!-- Comment: Wed Aug  4 04;41;54 comment by...Else -->
Two more. (not at the same time, I am just too lazy to write every single occurance.)
<!-- Comment: Wed Aug  4 05;06;05 comment by...Else -->
Another one ...
<!-- Comment: Wed Aug  4 05;33;12 comment by...Else -->
again.
<!-- Comment: Wed Aug  4 07;13;12 comment by...Else -->
We have had two more of these.
<!-- Date: Wed Aug 4 02:27:49.497 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->
<!-- Comment: Wed Aug  4 02;28;57 comment by...Else -->
Fixed by auto-HRR, too.



<!-- Comment: Wed Aug  4 03;25;55 comment by...Else -->
Again done TO from cot_18.

<!-- Comment: Wed Aug  4 07;05;40 comment by...Else -->
One more.
<!-- Date: Wed Aug 4 03:41:55.897 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de01   
 </PRE>
<!-- Author: ErrorHandler -->
<!-- Comment: Wed Aug  4 03;43;58 comment by...Else -->
Also fixed by auto-HRR. <br>
The error message was: "Internal buffer too small" <br>









<!-- Date: Wed Aug 4 05:40:54.643 2004 -->
Called MCR; they say that despite all the troubles and the planned access at 8am they are still trying to get a shot in on the night shift.
<!-- Author: Worm -->


<!-- Date: Wed Aug 4 06:10:41.590 2004 -->
<pre>Had another sequence of done timeout for COT_18 followed by a done timeout (after the HRR) from b0l2de01. (and with the same "internal buffer too small" error).</pre>
<!-- Author: Else :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894' target=_top>185894</a>) -->


<!-- Date: Wed Aug 4 06:25:28.052 2004 -->
Two items of notes from various other Accelerator Division eLogs..<br><br> 
* TEV access for today at 8 a.m. has been delayed until there 
  is a shot in the TEV<br> 
* Gated PMT in the synclight abort gap monitor has been replaced.<br> New PMT may have to be "calibrated/timed-in" 
<br>
<!-- Author: JJ -->



<!-- Date: Wed Aug 4 07:30:22.404 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->
<!-- Comment: Wed Aug  4 07;32;24 comment by...Else -->
Only problems from one of the crates this time! HRR didn't work in this case but shepherding did.


<!-- Date: Wed Aug 4 07:34:56.520 2004 -->
<pre>Here we go again:  
 
Busy TO. for crate/s:  
    b0pcal05   b0pcal11    
 </pre>
<!-- Author: Else :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894' target=_top>185894</a>) -->

<!-- Date: Wed Aug 4 07:56:01.493 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:05</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Wed Aug 4 08:00:38.904 2004 -->
<font color=red>Shift Summary: </font><pre>        Graveyard Shift Aug 4  
        =====================  
        Worm, Lytken, Aoki, Suslov  
  
 00:00  No beam.  No stacking.  Just rain.    
        TeV fighting a spate of storm-related problems.  
 00:07  b0dau35.fnal.gov:/oral at 100%.  Paged then called D. Litvintsev to fix.  
 00:10  Water coming in west doors.  I put up sign.  
 07:55  Nothing to report.  
  
Techs and MCR folk fight many battles  
------------------------------------- 
*(3am) D:EKIK power supply repairs... stuck kautzky valve at D11... D37,D27,B33 and C17 pumps changed out... Crate problems at C4... 
*(4am) TeV racked in (feeder 23), turn-on in progress.  Glitch.  Trips on 2 of 3 phases on both feeder 23 and 47.  Cryo, Linac, Booster, and TeV RF affected. 
* 11 Tev CC's trip.  
* MV0, DC SEP, MH1, and MV2 all trip.  Bunches of BRF* and LRF* trips.  
* etc etc  
 
 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   none
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             100%
   
</td></tr></table></pre>
<!-- Author: Worm -->

<!-- Date: Wed Aug 4 08:12:42.299 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=185894">185894</A>
RECOVERED: <PRE>   
  Busy TO. for crate/s: 
    b0pcal05   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->