<!-- Title: Owl shift. Mon Oct  7, 2002 -->
<!-- SciCo: Steve Nahn -->
<!-- DAQAce: Bruce Knuteson -->
<!-- MonAce: John Zhou -->
<!-- CO: The Incomparable JJ! -->
<!-- OpManager: Matthew Jones -->
<!-- Notes: -->
Inherited Store 1830 Run 152579, Stack is at 5.6, Stack' is 10.  Our plan is to hand this run to the Day shift.


<!-- Date: Mon Oct 7 00:21:35 2002 -->
Extended HALT to reset VRB in b0eb15 Slot 14
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 00;26;54 comment by...Knuteson -->
<pre>The sequence of errors here was:

(1) L2 Trigger Supervisor error (white dialog box; no corresponding orange warning screen); upon HRR, received
(2) L1 done from VRB_SVX_02, and simultaneously
(3) scpu_cant_reset_vrb implicating b0eb15, slot 14.

Successfully recovered upon resetting this vrb.
</pre>


<!-- Comment: Mon Oct  7 00;40;31 comment by...Knuteson -->
Above, "L1 done from VRB_SVX_02" should be "Busy timeout from VRB_SVX_02"
<!-- Date: Mon Oct 7 00:27:43 2002 -->
ISL IFIX briefly <FONT COLOR=YELLOW>YELLOW</FONT>, looks like CCE1 Pressure at COT Face is oscillating between 12.9 and 13.0 == High Warning Threshold.  We watch with baited breath. 
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 05;13;13 comment by...SCN -->
Oscillation persists...Experts emailed
<!-- Comment: Mon Oct  7 07;10;15 comment by...Steve Hahn -->
That's "bated", not "baited".
<!-- Comment: Mon Oct  7 08;54;28 comment by...PW -->
Unless he's trying to catch a fish.
<!-- Comment: Wed Oct  9 05;41;43 comment by...SCN -->
With my teeth
<!-- Comment: Wed Oct  9 18;10;40 comment by...hk -->
<pre>Stop having fun!</pre>
<!-- Date: Mon Oct 7 01:10:34 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8528&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8528' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8529&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8529' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8530&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8530' width='300'></a></center>
<!-- Author: JZ -->


<!-- Date: Mon Oct 7 01:16:30 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8531&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8531  width='300' align='top'></td><td>x</a></td></tr></table>
<!-- Author: JZ -->
<!-- Comment: Mon Oct  7 01;17;27 comment by...JZ -->
The previous entry didn't plot the losses. Here they are.


<!-- Date: Mon Oct 7 01:36:51 2002 -->
<h3>Case Study on Vrb Corruption</h3> 
By popular demand and in an attempt to provide more information to the inquisitive aces...we have had several cases of "Vrb Corruption" causing BTOs and requiring HRRs.  One such example<pre>28'28" b0l3gate1.fnal.gov:main:12:28:28 AM->Host b0eb13.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 1600167, crate 54, channel 5 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 5, should be 7. 
Tracer word = 0xa4550036 (I think). 
Channel data = 0x00000103 0xa4000036 0x43075444 0x004f0006 0x00140000 0x000000e6 0x00000003 0x0000000f ... 
... 0x00000b20 0x00000000 0x0000b419 0x00120000 0xaa00aaaa 0x55aa5555 0xa4550036 0x80070000 -->   
</pre> 
The theory here is that the EVB system gets the Event Number from the TSI, and checks it against what the Tracers put in the datastream which is just the bottom 4 bits.  These bits are 
located after the "Tracer Marker Words" 0xaaaaaaaa 0x55555555 which are tacked on to the end of the data stream.  In the 
above case, it seems the 3rd byte is shifted to the next 32 bit word <pre> 
0x00120000 0xaa00aaaa 0x55aa5555 0xa4550036 0x80070000 --should be 
0x00000000 0xaaaaaaaa 0x55555555 0xa4075036 0x80..0000 
</pre> so that in particular in the 4th word (the "tracer word" 0x55 becomes 0x07 
which would give the correct event number. This shift to me suggests a hardware or firmware failure in the tracer or Vrb or Vtm.<P> 
Glancing through the log, this seems not to be the only  
failure mode.  In many cases the "tracer markers" looks nothing 
like the a's and 5's <pre> 
9'12" b0l3gate1.fnal.gov:main:10:39:10 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 366254, crate 36, channel 3 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 10, should be 14. 
Tracer word = 0x000a0024 (I think). 
Channel data = 0x00000033 0x00080024 0x00410004 0x00080002 0x00000000 0x00000011 0x00000001 0x00d40000 ... 
... 0x00000000 0x00000000 0x00000000 0x00000000 0x02020020 0x01100015 0x000a0024 0x00000000 --></pre> although the byte counts do add up.<pre> (MLE) b0l3gate1.fnal.gov:main:10:39:10 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-VrbHeader: Dump of header words for event 366254 from VRB in slot 12: 
0x00000000 0x000023a0 0x001b009b 0x02ec07fc 0x084402ec 0x06e40670 0x00000000 0x00000000</pre> In the next event however, you get header corruption in that the total byte count is not a multiple of 8 bytes, and from the looks of the header  
you are not actually looking at a header but rather something in 
the middle of the data stream. 
<pre>39'12" b0l3gate1.fnal.gov:main:10:39:10 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-BadVrbByteCount: Event 366255 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 10. 
Count = 393233. -->   
(MLE) b0l3gate1.fnal.gov:main:10:39:10 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-VrbHeader: Dump of header words for event 366255 from VRB in slot 10: 
0x0535001a 0x05360011 0x0539000e 0x053a0012 0x0542002f 0x0546001e 0x048f0012 0x04930043 
</pre> 
I think this is just some (Channel Data) pattern in the middle of the data stream. This suggests to me that the beginning of the Vrb read from the fifo in the previous event is ok, but in  
fact you didn't get that whole event (bytes requested != bytes received), so  when you look for the  
tracer word as an offset from the end of the data, you  
are not looking in the correct place.  Then, when you go back to read the next event, you have the fragment of the event  
left behind to read out first, not the header.  Since it is 
a fifo, once you lose track of event boundaries, you're sunk.  Something to note is that we are also getting  (very few) RF errors both  
of the Silicon variety and the Non-Silicon variety.  The Non-Silicon types (RF Err Code #9) might be useful to pull 
off the L3 nodes and look at in detail since there you get the entire unreformatted event.  However, if the problem again is because of the Event before, you won't learn much.
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 08;53;46 comment by...patrick -->
The b0eb13 error was extensively investigated over
labor day weekend. Everything that could be swapped
was swapped, except VME crates. But including cables.
Most all swaps made the frequency of the errors increase.
On the other errors, yes words are added or subtracted
somewhere in the middle of a fragment. The bad events
don't go to level 3 so can't be looked at in detail
to see where it goes bad.

<!-- Comment: Wed Oct  9 05;44;09 comment by...SCN -->
But RF error'd events can be extracted from L3, and sometimes
this is helpful in understanding the ones that don't even
get there.  But anyway, this whole entry was just because my
Ace requested it-I am pretty confident the people doing the
investigation knew all this already.
<!-- Date: Mon Oct 7 01:39:43 2002 -->
PSM alarm for BSU NW 1-paging expert.  One Channel had been reading high, now no channels read anything  and voltage status page is Brown.
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 01;49;00 comment by...SCN -->
Now TSU and BSU turn red, "Cannot Read".
Oleg is coming in 10 minutes.<P>
Also, a CMX "trip" occurred.  Wedge 8 and 9 SE tripped. Resetting them...

<!-- Comment: Mon Oct  7 01;54;10 comment by...JZ -->
They are back now. This happened twice in 5 minutes in the previous shift.


<!-- Comment: Mon Oct  7 02;10;33 comment by...SCN -->
PSM status for BSU NW 1 magically returns to <FONT COLOR=GREEN>GREEN</FONT> as Oleg comes in and resets the 
TSU/BSU CCU PSs to regain communication-BSU and TSU then also go <FONT COLOR=GREEN>GREEN</FONT>.  The only issue now is that VNODE2 where the GLobal IFIX display is seems to be kinda pokey,
but hey, we can live with that.  Oleg was convinced to take the 
opportunity to teach the Aces how to perform this reset so that next time he can stay in bed.
<!-- Date: Mon Oct 7 02:07:44 2002 -->
North-West side of muon scintillators had voltages=0. Fixed.
<!-- Author: Oleg Poukhov -->



<!-- Date: Mon Oct 7 02:18:06 2002 -->
CMX trip on SE Wedges 8,9, 10,11.  Spike in Pbars as well.  Will 
reset and call Phil Schlabach if it happens again within a time period dictated by a balance between reluctance to drag people out of bed and tolerance of annoying repetitive problems.
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 02;27;32 comment by...JZ -->
Wedge 8 and 9 triggered again. Experts are paged.
<!-- Date: Mon Oct 7 02:22:40 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8532&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8532' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8533&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8533' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8534&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8534' width='300'></a></center>
<!-- Author: JZ -->


<!-- Date: Mon Oct 7 02:25:13 2002 -->
<h3>Theory on DPS</h3> 
During these trigger inhibits we've noticed that the  
prescales have been changing (there is some reluctance to 
go to PAUSE for inhibits since it can affect the Prescale  
process).  We were wondering if perhaps when the Scalers see 
the L1A rate go to zero (due to PAUSE or Inhibit) it then tries 
to adjust prescales to account for "additional available bandwidth" and then when the L1As are reestablished it  
prescales everything back to a reasonable value?  Perhaps 
the prescale program should distinguish continuous rate changes from discontinuous drops to 0 rate?
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 10;12;06 comment by...Greg -->
<pre>We have suggested disabling DPS during Pause until we 
can get this sorted out.

</pre>
<!-- Comment: Mon Oct  7 10;13;27 comment by...Greg -->
<pre>&lt;pre&gt;We have suggested disabling DPS during Pause until we 
can get this sorted out.

&lt;/pre&gt;</pre>
<!-- Date: Mon Oct 7 02:39:40 2002 -->
<h3>CMX trips</h3> Coincident with Pbar spikes<ul> 
<li> Called MCR to complain about Pbar spikes, they responded they'll look into it and probably change chromaticities or tunes or something 
<li> Paged CMX pager, who is on the phone currently with DAQ Ace 
and has a plan but is checking things and will call us back 
<li> Talked to Phil S., who passed the baton to the CMX pager and is back in the land of nod.  He said we can call him back if we feel it is necessary 
</ul> 
Stay tuned...
<!-- Author: SCN -->


<!-- Date: Mon Oct 7 02:40:17 2002 -->
Seems that the wedges in CMX HV trip associated with PB losses are most likely: SE 8-9, 10-11, and NE 2-3.
<!-- Author: JZ -->
<!-- Comment: Mon Oct  7 02;51;20 comment by...JZ -->
The CMX expert called in and adviced us to turn off these wedges. If there are more wedges caused by beam activities, we'll disable them too.

<!-- Comment: Mon Oct  7 03;10;48 comment by...JZ -->
Duely, we turned off SE 12-13 after another pb loss causing HV trip.
<!-- Date: Mon Oct 7 02:58:49 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152579">152579</A>
 Terminated at 2002.10.07 02:57:32
<!-- processed by 131.225.236.229 -->
<!-- Author: RunControl -->



<!-- Date: Mon Oct 7 02:58:50 2002 -->
Stopping the run and restarting to make a somewhat clear 
boundary between "Good for CMX" and "Bad for CMX" runs.
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 03;08;29 comment by...SCN -->
Also iterating with MCR on improving the spike situation.
<!-- Date: Mon Oct 7 02:59:39 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152579">152579</A>
TERMINATE: Ending run due to consistent pbar losses causing CMX trips.  This run will be marked "good" for CMX.  We are turning off some CMX wedges.  The next run is likely to be marked "bad" for CMX.
<!-- processed by 131.225.127.7 -->
<!-- Author: Knuteson -->


<!-- Date: Mon Oct 7 03:08:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152580">152580</A>
 Activated at 2002.10.07 03:05:34
<!-- processed by 131.225.236.229 -->
<!-- Author: RunControl -->


<!-- Date: Mon Oct 7 03:08:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152580">152580</A>
ACTIVATE: Restarting physics run with CMX SE 8-12 and NE 2-3 turned off.
<!-- processed by 131.225.236.229 -->
<!-- Author: Knuteson -->




<!-- Date: Mon Oct 7 03:43:27 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152580">152580</A>
 Terminated at 2002.10.07 03:42:37
<!-- processed by 131.154.11.41 -->
<!-- Author: RunControl -->


<!-- Date: Mon Oct 7 03:44:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152580">152580</A>
TERMINATE: Ending physics run.  MCR claims to have reduced pbar losses.  The next run will begin with all CMX wedge HVs on.
<!-- processed by 131.225.127.7 -->
<!-- Author: Knuteson -->



<!-- Date: Mon Oct 7 03:45:29 2002 -->
MCR calls (several times actually) and has made best efforts to 
get losses down, although some amount of spikes persist.  So, we 
are giving it a go, with a new run with all CMX turned on, and a  
policy of disabling wedges as they trip.  Status of next run vis a vis CMX will be an interesting quandry.
<!-- Author: SCN -->

<!-- Date: Mon Oct 7 03:49:26 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581">152581</A>
 Activated at 2002.10.07 03:48:36
<!-- processed by 131.225.127.7 -->
<!-- Author: RunControl -->


<!-- Date: Mon Oct 7 03:50:02 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581">152581</A>
ACTIVATE: Begin physics run.
<!-- processed by 131.225.127.7 -->
<!-- Author: Knuteson -->



<!-- Date: Mon Oct 7 03:50:44 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8535&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8535' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8536&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8536' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8537&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8537' width='300'></a></center>
<!-- Author: JZ -->
<!-- Comment: Mon Oct  7 03;53;42 comment by...JZ -->
Many of the pbar loss caused CMX trip (we turned a few wedges off after 5-6 trips). We have turned all the HV of CMX on for the new run.

<!-- Date: Mon Oct 7 03:57:53 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152580">152580</A>
RUNSTATUS: Marking run bad for CMX due to disabled Wedges SE 8-13 and NE 2,3
<!-- processed by 131.225.127.7 -->
<!-- Author: cdfscico -->



<!-- Date: Mon Oct 7 04:00:08 2002 -->
goodrun GUI goes into "RTserver refusal" mode when I try to add a comment<pre>Attempting connection to <tcp:_node:5101> RTserver. 
WARNING: lost connection: Connection refused 
Attempting connection to <tcp:_node:5101> RTserver. 
WARNING: lost connection: Connection refused 
Attempting connection to <tcp:_node:5101> RTserver. 
WARNING: lost connection: Connection refused 
Attempting connection to <tcp:_node:5101> RTserver. 
WARNING: lost connection: Connection refused 
Attempting connection to <tcp:_node:5101> RTserver. 
WARNING: lost connection: Connection refused 
... 
</pre> 
I put it out of its misery.
<!-- Author: SCN -->


<!-- Date: Mon Oct 7 04:05:23 2002 -->
CMX SE 8,9,10,11 tripped with Pbar spike of ~ 15 kHz.  Wedges  
HV brought down.
<!-- Author: SCN -->


<!-- Date: Mon Oct 7 04:16:19 2002 -->
MCR called -- he has noticed the continued pbar loss spikes.  A knob has been dialed -- we'll see if this helps.
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581' target=_top>152581</a>) -->
<!-- Comment: Mon Oct  7 05;06;38 comment by...SCN -->
We were looking for a 30 minute period with no spikes above 10 kHz, and just as we were about to say it was good (as we were looking at the monitor), we suddenly got more spikes above 12.5 kHz (which seems to be the threshold for trips), more CMX trips, etc.  We will continue on our program of turning off Wedges.  Currently down SE 8-13 NE 2,3

<!-- Comment: Mon Oct  7 05;08;33 comment by...SCN -->
By the way, it <i>was</i> looking ok there for a bit, until we 
decided it was judgement time, and then it all went to hell...


<!-- Date: Mon Oct 7 05:10:44 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8538&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8538' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8539&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8539' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8540&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8540' width='300'></a></center>
<!-- Author: JZ -->



<!-- Date: Mon Oct 7 05:33:57 2002 -->
We tell MCR that they <a href='http://www-bd.fnal.gov/cgi-mcr/elog.pl?nb=2002&action=xe&page=758&time=05:27:23'>lost an RF station</a> (<B>Yea TevMon!</B>) and they confirm it, RF Station 7.  Everything else "looks ok" 
according to MCR, so we go into "high alert" mode on the ACNET beam stuff and try to lower our heart rates.
<!-- Author: SCN -->
<!-- Comment: Mon Oct  7 05;40;49 comment by...SCN -->
High alert means follow <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2002&action=view&page=700&frame=2&anchor=01:09:36'>
these procedures</a>. 


<!-- Date: Mon Oct 7 06:02:19 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8542&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8542' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8543&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8543' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8544&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8544' width='300'></a></center>
<!-- Author: JZ -->


<!-- Date: Mon Oct 7 06:08:40 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8545&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8545  width='300' align='top'></td><td>x</a></td></tr></table>
<!-- Author: JZ -->


<!-- Date: Mon Oct 7 06:14:39 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8546&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8546  width='300' align='top'></td><td>This the SITEST3 plot Steve wants. Look at the effect  of the loss of an RF station.</a></td></tr></table>
<!-- Author: JZ -->


<!-- Date: Mon Oct 7 06:14:55 2002 -->
Somewhere in here we also kicked the DPS mechanism by restarting the GUI and that worked.  Aces should note that seeing <pre> 
dpsEnabled: true</pre> is not enough, one must look for the timestamp. <BR> A useful command: <i> 
tail -f < scalerGui logfile> | grep -e "TimeStamp" </i>
<!-- Author: SCN -->


<!-- Date: Mon Oct 7 06:17:40 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8547&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8547  width='300' align='top'></td><td>Losses during the past hour</a></td></tr></table>
<!-- Author: JZ -->


<!-- Date: Mon Oct 7 06:34:54 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8548&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8548  width='300' align='top'></td><td>This is the SITEST2, illustrating the RF station loss.</a></td></tr></table>
<!-- Author: JZ -->
<!-- Comment: Mon Oct  7 07;12;23 comment by...JZ -->
A little sleepy. The next entry is what I intend.
<!-- Date: Mon Oct 7 07:09:00 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8553&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8553  width='300' align='top'></td><td>x</a></td></tr></table>
<!-- Author: JZ -->



<!-- Date: Mon Oct 7 07:10:53 2002 -->
ProcMon error with no apparent cause.
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581' target=_top>152581</a>) -->


<!-- Date: Mon Oct 7 07:11:06 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8554&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8554' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8555&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8555' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-8556&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-8556' width='300'></a></center>
<!-- Author: JZ -->



<!-- Date: Mon Oct 7 07:17:03 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-8558&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-8558  width='300' align='top'></td><td>SI-TEST3</a></td></tr></table>
<!-- Author: JZ -->

<!-- Date: Mon Oct 7 07:24:45 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581">152581</A>
 Terminated at 2002.10.07 07:24:11
<!-- processed by 131.225.236.229 -->
<!-- Author: RunControl -->



<!-- Date: Mon Oct 7 07:25:31 2002 -->
PBar spikes seem to have diminished, so we start a new run (again) with all CMX back on.  152581 will be marked "Bad for CMX"
<!-- Author: SCN :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581' target=_top>152581</a>) -->

<!-- Date: Mon Oct 7 07:26:39 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581">152581</A>
TERMINATE: Ending physics run.  (This run had pieces of CMX off.  pbar losses have been stable recently.  The next run will have CMX on, at least to start.)
<!-- processed by 131.225.234.114 -->
<!-- Author: Knuteson -->


<!-- Date: Mon Oct 7 07:31:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152582">152582</A>
 Activated at 2002.10.07 07:30:08
<!-- processed by 131.225.236.229 -->
<!-- Author: RunControl -->
<!-- Comment: Mon Oct  7 07;45;05 comment by...SCN -->
More Wedges failed to heed our warning, to their demise.
Now disabled SE 8-13, NE 2-4 
<!-- Date: Mon Oct 7 07:31:25 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152582">152582</A>
ACTIVATE: Begin physics run, all of CMX on (initially, at least).
<!-- processed by 131.225.236.229 -->
<!-- Author: Knuteson -->
<!-- Comment: Mon Oct  7 07;34;59 comment by...SCN -->
<h3>Damn!</h3>  A Pbar spike trips CMX SE 8,11-we recover (this time).  <P>  JJ likens it to watching your
favorite team blow it just when you tune in.  General concurrance
is rampant in the Control Room, with the notable exception of
the skeptical Ace

<!-- Comment: Mon Oct  7 07;39;09 comment by...SCN -->
Again, same ones, who didn't listen to our threats about the consequences of misbehaviour.  We must punish them by turning them off, as fair warning to the rest of the Wedges.

<!-- Comment: Mon Oct  7 07;41;06 comment by...SCN -->
We also notice that DPS TimeStamp in the log file ceases
when we hit the trigger inhibit.  Could it be that the algorithm 
turned itself off when it didn't understand the rates?


<!-- Date: Mon Oct 7 07:56:08 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152579>
152579</A> 
x25403</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,319]</TD>                                       
<TD ALIGN=CENTER>22:13:47</TD>                                          
<TD ALIGN=CENTER>02:57:32</TD>                                          
<TD ALIGN=CENTER>03:55:39</TD>                                          
<TD ALIGN=RIGHT>    110,466,383</TD>                                    
<TD ALIGN=RIGHT>     2,847,447</TD>                                     
<TD ALIGN=RIGHT>     554,236</TD>                                       
<TD ALIGN=RIGHT>  255.625</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152580>
152580</A> 
x25404</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,319]</TD>                                       
<TD ALIGN=CENTER>03:05:34</TD>                                          
<TD ALIGN=CENTER>03:42:37</TD>                                          
<TD ALIGN=CENTER>00:36:40</TD>                                          
<TD ALIGN=RIGHT>     10,828,912</TD>                                    
<TD ALIGN=RIGHT>       318,853</TD>                                     
<TD ALIGN=RIGHT>      66,297</TD>                                       
<TD ALIGN=RIGHT>   31.723</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152581>
152581</A> 
x25405</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,319]</TD>                                       
<TD ALIGN=CENTER>03:48:36</TD>                                          
<TD ALIGN=CENTER>07:24:11</TD>                                          
<TD ALIGN=CENTER>03:28:59</TD>                                          
<TD ALIGN=RIGHT>     80,218,709</TD>                                    
<TD ALIGN=RIGHT>     1,659,746</TD>                                     
<TD ALIGN=RIGHT>     375,361</TD>                                       
<TD ALIGN=RIGHT>  163.412</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=152582>
152582</A> 
x25406</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,319]</TD>                                       
<TD ALIGN=CENTER>07:30:08</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>00:17:55</TD>                                          
<TD ALIGN=RIGHT>      6,197,123</TD>                                    
<TD ALIGN=RIGHT>       129,006</TD>                                     
<TD ALIGN=RIGHT>      31,090</TD>                                       
<TD ALIGN=RIGHT>   13.418</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>08:19:14</TD>                                          
<TD ALIGN=RIGHT>     207,711,127</TD>                                   
<TD ALIGN=RIGHT>      4,955,052</TD>                                    
<TD ALIGN=RIGHT>     1,026,984</TD>                                     
<TD ALIGN=RIGHT>  464.180</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- processed by 131.225.236.42 -->
<!-- Author: End of Shift Report -->



<!-- Date: Mon Oct 7 07:59:46 2002 -->
<font color=red>Shift Summary: </font>    Other than lots of PBar spikes tripping CMX East  
    and a lost RF station, things went great. 
    <ul> 
      <li> Inherited Store 1830 Run 152579 from Evening 
      <li> 1:40 <a href= 
	  '/cgi/elog/elog.pl?nb=2002&action=view&page=837&frame=2&anchor=01:39:43'> 
	  Muon Power Supply Problems</a> fixed by Oleg  
      <li> 1:40 onward CMX trips concurrent with Pbar spikes.  We disable some 
	wedges and talk to MCR, eventually attempt to  
	reenable all wedges and run again.  This fails, spikes keep coming,  
	and there seems to be a threshold of ~12 kHz for CMX trips.   
	We turn off wedges when they trip.  Runs w/o all CMX on are declared "Bad for CMX" 
      <li> 5:25 RF Station 7 lost, TevMon moans, we tell MCR ("Oh really?") and  
	then go into High Alert Si Safety mode 
      <li> 7:30 PBar spikes seem diminished, reattempt running with all CMX on, but they keep 
	tripping. 
      <li> Notice that we need to kick the DPS program at every trigger inhibit 
      <li> 7:50 MCR plan: Stack to 120 and Shoot (but haven't heard from Run Coordinator yet) 
    </ul>   
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   152579-152582
Delivered Luminosity   411.5  
Acquired Luminosity    349.8  
Efficiency             85.0
   
</td></tr></table></pre>
<!-- Author: SCN -->


<!-- Date: Mon Oct 7 08:27:40 2002 -->
CMX wedges 0-1 tripped.  Turned the voltage back on.<font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: natalia -->