<!-- Title: Eve shift. Tue Nov 20, 2001 -->
<!-- SciCo: Steve Errede -->
<!-- DAQAce: A. Kovalev/T.Yamashita -->
<!-- MonAce: &nbsp; -->
<!-- CO: Hirokazu Kobayashi -->
<!-- OpManager: Jay Dittman -->
<!-- Notes: -->
<pre>Start Eve Shift - run Cosmic_Notrack w/ entire detector. 
S&S in progress.</pre>


<!-- Date: Tue Nov 20 16:15:44 2001 -->
trying to start cosmics, got a bunch of <br> 
b0fcal00 : Messenger <br> 
-> ADMEM  in slot XX not properly configured. 
  
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132906' target=_top>132906</a>) -->


<!-- Date: Tue Nov 20 16:34:10 2001 -->
got SCPU_CANT_RESET_VRB red screen error.<br> 
suggests to power-cycle offending crate and clean up evb.<br> 
power-cycled the b0eb13.<br>
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132906' target=_top>132906</a>) -->


<!-- Date: Tue Nov 20 16:34:14 2001 -->
<pre>Test run for new fer code that talks to the  
shiftcrew e-log. 
 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132908' target=_top>132908</a>) -->


<!-- Date: Tue Nov 20 16:46:15 2001 -->
We started a cosmic run. 
We got error messages: 
<pre> 
(MLE) b0fcal01:Messenger:4:33:25 PM->ADMEM in slot 16 not properly configured 
(MLE) b0fcal01:Messenger:4:33:25 PM->ADMEM in slot 14 not properly configured 
(MLE) b0fcal01:Messenger:4:33:25 PM->ADMEM in slot 12 not properly configured 
(MLE) b0fcal01:Messenger:4:33:25 PM->ADMEM in slot 10 not properly configured 
(MLE) b0fcal01:Messenger:4:33:25 PM->ADMEM in slot 8 not properly configured 
(MLE) b0fcal01:Messenger:4:33:25 PM->ADMEM in slot 6 not properly configured 
(MLE) b0fcal00:Messenger:4:33:27 PM->ADMEM in slot 6 not properly configured 
(MLE) b0fcal00:Messenger:4:33:27 PM->ADMEM in slot 16 not properly configured 
(MLE) b0fcal00:Messenger:4:33:27 PM->ADMEM in slot 14 not properly configured 
(MLE) b0fcal00:Messenger:4:33:27 PM->ADMEM in slot 12 not properly configured 
(MLE) b0fcal00:Messenger:4:33:27 PM->ADMEM in slot 10 not properly configured 
(MLE) b0fcal00:Messenger:4:33:27 PM->ADMEM in slot 8 not properly configured  
</pre> 
We paged BSC and Koji Terashi came soon. He said we can ignore the messages.
<!-- Author: Aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132907' target=_top>132907</a>) -->


<!-- Date: Tue Nov 20 16:46:18 2001 -->
<pre>CMP HV Status: 
 
We now have permission to raise HV on the section of CMP bottom 
that was locked out until the chambers had flowed Ar/Eth long 
enough that we could be certain of the gas quality.  We have 
raised HV to that section (so that now all CMP HV is up).  A 
simple check with a scope showed signals coming from the chambers 
which had previously had a blockage to gas flow (pinched gas 
tube), so the problem seems fixed.  However, the shift crew  
should be alert for CMP trips.  If the CMP trips, follow the 
posted HV recovery procedures (see flow chart in the control 
room).  Page an expert only if the shift crew's attempts to raise 
HV in a tripped section fail.</pre>
<!-- Author: K. Lannon, H.S. Kim, T. Vickey -->


<!-- Date: Tue Nov 20 17:07:07 2001 -->
got DONE TIMEOUT from fcal00  
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132907' target=_top>132907</a>) -->


<!-- Date: Tue Nov 20 17:13:40 2001 -->
got "data truncated" for cot10<br> 
followed by "Error reading VRB in slot 10 for event 28405" 
(b0eb15)<br> 
will try to restart the run
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132907' target=_top>132907</a>) -->


<!-- Date: Tue Nov 20 17:22:46 2001 -->
We check YMON CMP Bottom Stack plot # 61 to see if 
fix to gas flow problem of CMP stacks # 60 & 61 is 
good. We see hits in these stacks in the current 
cosmic ray run - looks good! Hirokazu put this (and some other) YMON plot(s) in the Consumer Operator E-Logbook.
<!-- Author: S.Errede/H.Kobayashi :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132907' target=_top>132907</a>) -->


<!-- Date: Tue Nov 20 17:44:59 2001 -->
What type of run was 132910 ? 
  
<!-- Author: Observer :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132910' target=_top>132910</a>) -->
<!-- Comment: Tue Nov 20 17;48;05 comment by...aces -->
<pre>it was a nonsense cosmic_notracks without calorimeters and with softEV</pre>


<!-- Date: Tue Nov 20 18:02:53 2001 -->
<pre>CES trip; HV reset</pre>
<!-- Author: aces -->


<!-- Date: Tue Nov 20 18:05:29 2001 -->
softevb continuously dies after about 20k of our nonsense cosmics
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132913' target=_top>132913</a>) -->


<!-- Date: Tue Nov 20 18:12:51 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>Dear ACES 
Please propagate to the next shift crew and notice for yourself  
that Level3 hardware page number has changed to  
6302188906 
 
thanks</td></tr></td></table>
<!-- Author: Arkadiy -->


<!-- Date: Tue Nov 20 18:21:27 2001 -->
CES trips again. Upper quadrant (6:11) of SW calorimeter arch. We page CES HV. LJN responds. He says to reset it again. S&S is over 
so air flow in CH should be nominal (?) Temps around CH look more uniform than e.g. last nite. Rel Humidity ~ 22% inside CH at moment. Re-raised CES HV successfully. We will watch currents.
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132914' target=_top>132914</a>) -->


<!-- Date: Tue Nov 20 18:24:27 2001 -->
CES upper SW (6:11) HV tripped again. We did not see any 
current draw before it tripped. Bring it back to SBY for 
now, and wait a bit.
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132914' target=_top>132914</a>) -->


<!-- Date: Tue Nov 20 18:28:40 2001 -->
Now CMU tripped. Resetting CMU and CES.
<!-- Author: S. Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132914' target=_top>132914</a>) -->


<!-- Date: Tue Nov 20 18:30:51 2001 -->
took l1/l2/cal crates for ~1 hour to track down a bit that was stuck low .3% of the time.  It appears in LOCOS for a given slot having clusters, but doesn't look like it's actually in the DCAS board itself - which means either the backplane daisychain or LOCOS chips are bad/not quite connected.  I will think about this and probally come back later to reflow some pins
<!-- Author: hray -->


<!-- Date: Tue Nov 20 18:40:32 2001 -->
CES HV tripped again. Leaving it at SBY.
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132916' target=_top>132916</a>) -->


<!-- Date: Tue Nov 20 20:19:32 2001 -->
Put CES at full nom. HV. SW (6:11) wedges tripped again. 
Paged LJN. He calls back, we leave these wedges at SBY, 
but raise all other CES HV channels to full nom. HV.
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132922' target=_top>132922</a>) -->


<!-- Date: Tue Nov 20 20:29:54 2001 -->
<pre>Tested new trigger table: Physics_0_02 [2, 215], that is made from code version 4.1.0 and is otherwise identical to previously tested tables. This is what happened: 
o Physics_0_02 [2, 215] crashed: Level3 didn't pass cold start transition. 
o tried old table: Physics_0_02 [2, 212] (didn't crash) 
o tried Physics_0_02 [2, 215] again - worked (strange, isn't it) 
o tried PHYSICS_0_02 [2, 216] same cold-start error 
o tried Physics_0_02 [2, 215] same cold-start error 
 
It's all a mistery and Ilya is coming around to solve it. 
 
Another problem: In all cases we got the error message on startup: "L3 CalibTag not found" 
 
</pre>
<!-- Author: Jonas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132917-132922' target=_top>132917-132922</a>) -->


<!-- Date: Tue Nov 20 20:34:21 2001 -->
We notice 3 alarms in Arch Power Supply Monitoring: 
23E has ch5 -5V CMU reading -10.57v 
16W has ch5 -5V CMU reading 0.00v 
22W has ch4 fan reading 0%. We called Mike Lindgren. 
He knows about all of these. He explicitly checked these 
yesterday - 23E & 16W  ch5 both measure at -5.0V, and 
22W both fans are operating fine.
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132922' target=_top>132922</a>) -->


<!-- Date: Tue Nov 20 20:54:23 2001 -->
<pre>   Level3 have been paged about certain transition failures 
that were observed during new filter tests of Sarah and Co. 
  The failures happened during the ColdStart transitions. 
In 4 or 5 runs one or two processor nodes failed to start 
according to l3proxy error message. 
  Examining the nodes we see that the failure happened  
because of the filter crashes. Those crashes produced core 
files and happened in the very beginning with no events 
going through the system apart from BoR record. Of course, 
such crashes are not reported automatically to the filter  
group because there was no active run and no End transition. 
   By the time we arrived, the filter experts decided to finish 
their tests for today. For the next series of their tests 
it might be useful to have L3 expert present to provide  
access to core, etc. 
</pre>
<!-- Author: Ilya/Arkadiy -->


<!-- Date: Tue Nov 20 21:09:19 2001 -->
we went back to cosmics_notrack and now we <font color=red>always</font> get L3CalibTag load failure. 
<br> 
suggests that we page Farukh... 
  
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132926' target=_top>132926</a>) -->


<!-- Date: Tue Nov 20 21:58:05 2001 -->
<pre>Ilya thinks that the afforementioned problem with L3CalibTag load failure is due to l3manager running out of log disk space. 
Jonas R. is looking into that. 
</pre>
<!-- Author: aces -->


<!-- Date: Tue Nov 20 22:30:18 2001 -->
<pre>We tried to start cosmics after Jonas called and said he cleaned up logs. Stuck on ColdStart. L3proxy looks dead, won't restart either. Paged Arkady, he's looking into it. 
</pre>
<!-- Author: aces -->


<!-- Date: Tue Nov 20 23:19:15 2001 -->
We raise CES SW (6:11) HV from SBY to full. 
Hope it stays up....
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132929' target=_top>132929</a>) -->

<!-- Date: Tue Nov 20 23:20:01 2001 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132929">132929</A>  Activated at 2001.11.20 23:17:01
<!-- Author: Activation Notification -->



<!-- Date: Tue Nov 20 23:20:22 2001 -->
<pre>  Level3 is restored back to normal. The problem was 
with the second Level 3 gateway b0l3pcom2 that had network 
(NFS) problems and required reboot. 
 
  After that RTserver had unclosed connection to l3proxy 
and was refusing to accept any more connection from the 
new instance of it on newly rebooted L3 gateway. After 
waiting for about 15 min hoping that the stale connection 
would time out in RTserver, we asked aces to restart RTserver. 
After that, everything came to normal. 
 
 </pre>
<!-- Author: Ilya/Arkadiy -->

<!-- Date: Tue Nov 20 23:24:34 2001 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132929">132929</A>  Terminated at 2001.11.20 23:21:36
<!-- Author: Termination Notification -->



<!-- Date: Tue Nov 20 23:25:29 2001 -->
CES SW (6:11) tripped again. 
We leave it STANDBY.
<!-- Author: Aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132929' target=_top>132929</a>) -->


<!-- Date: Tue Nov 20 23:32:26 2001 -->
We started up cosmics_notrack, everything looks OK, all ColdStart 
passed without errors.  
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132930' target=_top>132930</a>) -->
<!-- Comment: Tue Nov 20 23;51;12 comment by...hray -->
Run 132920 had a cable pulled on DCAS in slots 15 and 18 for some portion of the time in the l2cl00 crate.  I was checking backplane signals for this HAD bit that sometimes sticks low.


<!-- Date: Tue Nov 20 23:52:50 2001 -->
Endwall Power Supply read-back and software 
problems similar to above reported power supply 
problem - experts please also check on: 
CCUNET1, CCUNEB1, CCUNWT1, CCUNWB1 & CCUNWB2. 
 
Also, End Plug SMD09 apparently has some kind 
of communication error - no response from PS.
<!-- Author: Steve Errede :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=132931' target=_top>132931</a>) -->


<!-- Date: Wed Nov 21 00:01:11 2001 -->
<font color=red>Shift Summary: </font><pre>End Eve Shift. Just started another cosmizx run. 
Had some HV problems w/ CES - see above. Had problems 
with L3Tag log files, this is now fixed. Have fun...</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   132906-132931
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             -
   
</td></tr></table></pre>
<!-- Author: Steve Errede. -->