<!-- Title: Owl shift. Thu Nov 20, 2003 -->
<!-- SciCo: Beate Heinemann -->
<!-- DAQAce: Irina Shreyber -->
<!-- MonAce: Jeff Miles -->
<!-- CO: Subir Sarkar -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>- DAQ problem: cannot get a run going right now...Aces are investigating 
- TeV studies ongoing: uncoalesced beam at 980 GeV (TeVPR=15E10) 
- planning to run COSMICS_NOTRACKS_SRCDEBUG withour musc00 and without b0trin00
- if SRC pulls a fatal error, page primary silicon pager and 
    Sal Rappoccio (Silicon SRC pager) 


</pre>

<!-- Date: Thu Nov 20 00:39:25.696 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174504">174504</a>
ACTIVATE: COSMICS_NOTRACKS_SRCDEBUG without musc00
<!-- Author: Irina x2080 -->
<!-- Comment: Thu Nov 20 01;15;43 comment by...Jeff / Irina -->
Getting this run activated required a reboot for b0tsi03 (it was red on vxworks mon and would not transition) and a reboot for b0tsi00, by way of an EVB clean-up (the crate would not respond to a SCRAMNet check, and EVB would not partition transition).  The run was then activated successfully, but within a minute, several new crates became red on vxworks mon and monitoring info was lost on the L3 display (gold subfarms and red L3 state area). The displayed data rate dropped as low as ~1 Hz.  However, the run seemed to continue normally, leading us to suspect that it is a RT Server problem.  Within 2 minutes, the vxworks status had gone back to green and L3 was displaying properly. Events were counting up normally on all displays at the expected rate of ~25 Hz.  This whole cycle was repeated once more, and has apparently returned to "normal".  We will hold off on paging Frank Chlebana unless the problem becomes critical.
<!-- Comment: Thu Nov 20 01;17;26 comment by...Jeff / Irina -->
To clarify:  when the apparent RT problem occurs, practically all the vxmon crates go red eventually, not just the few at first.
<!-- Comment: Thu Nov 20 01;24;09 comment by...Jeff / Irina -->
Pulsar guys (Burkhard and Co.) stop by to ask if RT Server is giving us trouble, confirming our suspicions. They're going home, as it interrupted their work. The run seems to remain well-behaved after the initial glitches, so I think we'll wait to tell Frank until he gets in ~07:00.
<!-- Date: Thu Nov 20 00:40:25.660 2003 -->
Power supply manager on iFix shows two crates in row 1RR12 to be 
unresponsive (the channels are grey; alarm code 5).  On inspection, one is clearly powered off (1RR12D_1). The other is only identified by a serial number (124569), and I can't see an obviously off crate.  Will investigate a little further.
<!-- Author: Jeff Miles -->




<!-- Date: Thu Nov 20 01:24:07.562 2003 -->
<pre>Two CLC alarms which disappeared by themselves. There was no any alarm state on the CLC page (opened via Global Alarms)</pre>
<!-- Author: Irina/Jeff -->
<!-- Comment: Thu Nov 20 03;46;28 comment by...Irina -->
<pre>One more CLC global alarm without any alarm state on the CLC page
</pre>
<!-- Comment: Thu Nov 20 09;14;49 comment by...doug -->
mary convery reported this morning that this is a problem w/
their new monitoring PC.  they're working on it.
<!-- Date: Thu Nov 20 01:58:49.255 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12136&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12136' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12137&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12137' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12138&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12138' width='300'></a></center>
<!-- Author: Jeff Miles -->
<!-- Comment: Thu Nov 20 02;03;13 comment by...Jeff -->
Losses are constant and low; stable RF.



<!-- Date: Thu Nov 20 03:56:58.561 2003 -->
COT global alarm: under the Details page, the COT Low Voltage "Power Level" button was blinking red (indicating a trip), but clicking it revealed no anomolous entries. COT's general "Alarms" list shows a generic POWR_ALR in node COT1.  With telephone guidance from the expert (Aseet), I've reset (toggle off, then back on) channels 6A,6B,7A,7B,2,3 in the HNW section of the COT LV crate in rack 1RR04H. The red Trip LED's associated with each channel immediately went off, and the COT returned to its green state in iFix within a couple minutes.
<!-- Author: Jeff Miles -->
<!-- Comment: Thu Nov 20 06;31;19 comment by...Jeff -->
Incidentally, this is the same problem/symptoms as experienced during the evening shift <a href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=copyentry&page=968&time=19:25:57.991">here</a>.
<!-- Date: Thu Nov 20 04:01:55.492 2003 -->
We've just seen the fourth occurrence this shift of the ghostly CLC alarm; when we catch it, it's in CLC West (central region). As always, it went away within 30 seconds.
<!-- Author: Jeff / Irina -->


<!-- Date: Thu Nov 20 05:08:28.289 2003 -->
Silicon cooling alarm: paged Si cooling expert and Si main pager
<!-- Author: Beate -->
<!-- Comment: Thu Nov 20 05;17;30 comment by...beate -->
Andy Hocker called back and talked to Cryo guy. Si main pager carrier (Stan Forrester) is coming in.

<!-- Comment: Thu Nov 20 05;27;03 comment by...beate -->
Alarms have disappeared: Cryo guy called and says that the flow is back. Andy will give more detailed explanation in the morning. 


<!-- Date: Thu Nov 20 05:59:54.527 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12139&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12139' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12140&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12140' width='300'></a></center>
<!-- Author: sarkar -->
<!-- Comment: Thu Nov 20 06;29;01 comment by...Beate -->
This is to prove that YMon updates itself. The number of events changed and also the entries, e.g. look carefully at first two bins: they are equal in left but not in right plot. There had been a <a href='/cgi/elog/elog.pl?nb=2003&action=view&page=968&frame=2&anchor=22:17:30.115'> comment concerning this</a> in the previous shift.
<!-- Comment: Thu Nov 20 16;05;26 comment by...Dale -->
<pre>Actually this was not the problem - YMon and the other consumers update themself just fine.  The problem is that if you go to a slide with the 'auto update' enabled then all histograms reset in the slide.  I think this has always been the case but it seems that it shouldn't happen.</pre>
<!-- Date: Thu Nov 20 06:23:22.568 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12141&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12141' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12142&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12142' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12143&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12143' width='300'></a></center>
<!-- Author: Jeff Miles -->
<!-- Comment: Thu Nov 20 06;24;55 comment by...Jeff -->
<br>
MCR dumped the p/pbar beam; ramped down magnets.<br>
(Fast time plots taken at 06:20)




<!-- Date: Thu Nov 20 06:48:07.799 2003 -->
We saw another COT LV power supply alarm, beginning just after 06:00.  The symptoms in iFix were identical, with the voltages in the UNW column being approx. +/- 0.3V, down from the desired +/- 3.0V.  I immediately went downstairs to reset the affected channels as <a href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=copyentry&page=969&time=03:56:58.561">before</a>, but the trip LED's were not lit this time. Resetting them anyway did not resolve the problem.  We waited about 25 minutes to account for the time overhead that was involved in the first incident with talking to the expert, etc., and found that at some point, the trip LED's had lit up.  I reset them as before, and the affected voltages returned to nominal within 15 seconds.<br> 
Lesson:  Assuming that one does not need to use the COT crates (we've been doing COSMICS_NOTRACKS all night), this problem can be resolved by just waiting for at least 10 minutes and then resetting the channels.  The COT experts can probably provide a more expedient solution...
<!-- Author: Jeff Miles -->


<!-- Date: Thu Nov 20 06:59:07.721 2003 -->
<pre>Sound's funny (but not for me :( ) :I've lost my Fermi ID around the control room area. If somebody find it, please let me know. 
(my email: shreyber@fnal.gov)</pre>
<!-- Author: Irina -->


<!-- Date: Thu Nov 20 07:11:31.937 2003 -->
<pre>Replaced b0trin00 (old inhibit crate) with  
b0inh00 (new inhibit crate) in the VxMon display. 
Problems with b0inh00 should be considered real. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174054' target=_top>174054</a>) -->


<!-- Date: Thu Nov 20 07:27:31.016 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12144&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12144' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12145&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12145' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12146&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12146' width='300'></a></center>
<!-- Author: Jeff Miles -->
<!-- Comment: Thu Nov 20 07;30;39 comment by...Jeff -->
Beam (TevPR ~ 15E09, TevPB ~ 10E09) back in the machine.<br>
Spike in the losses for both, but still low < 1000 Hz<br>
(Fast Time plots from 07:25)

<!-- Date: Thu Nov 20 07:56:09.457 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Nov 20 08:01:28.577 2003 -->
Integrated number of phantom CLC West <a href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=copyentry&page=968&time=16:27:11.571">alarms</a> this shift == 8.
<!-- Author: Jeff Miles -->

<!-- Date: Thu Nov 20 08:19:12.101 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174504">174504</A>
TERMINATE: End Cosmics to test L3

<!-- Author: kohei x2080 -->



<!-- Date: Thu Nov 20 08:21:10.481 2003 -->
<font color=red>Shift Summary: </font><pre><pre> 
running: 
- Tevatron had protons and pbars in at 980 GeV until about 6.30am 
- we ran COSMICS_NOTRACKS_SRCDEBUG all night (no SRC error though) 
 
events: 
- COT global alarm: With telephone guidance from the expert (Aseet),  
  Mon Ace reset channels 6A,6B,7A,7B,2,3 in the HNW section of the COT  
  LV crate in rack 1RR04H. 
- Silicon cooling alarm: solved by A. Hocker and Cryo guy. The flow had 
  been interrupted and is now back again. 
- several very short CLC alarms (too short to even find out what exactly  
  gives an alarm) 
 
issues/problems: 
- b0musc00 crate cannot be included 
- CMX_NX_1 (needs access)  
- consumers: Stage0, BeamMon and TimeMon disabled 
 
other: 
- if SRC pulls a fatal error, page primary silicon pager and  
  Sal Rappoccio (Silicon SRC pager)  
- if COT HV on, notify Bill Badget & Ron Moore for DSP runs  
- INHIBITS_00 boinh00 is the new Trigger Inhibit crate. b0trin00 crate  
  should not be included 
- previous shift thought YMon doesn't auto-update the plots but we  
  think it does: so we think YMon is okay 
- Irina Shreyber has lost her ID in the control room tonight during  
  this hectic shift: please let her know if you find it 
 
</pre></pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   0
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             0
   
</td></tr></table></pre>
<!-- Author: Beate -->