<!-- Title: Eve shift. Sun Jul  8, 2001 -->
<!-- SciCo: Henry Frisch  -->
<!-- DAQAce: Anyes Tafford -->
<!-- MonAce: Victoria Martin -->
<!-- CO: Fumi Ukegawa -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre>Inherit store 567, lum =1.3E30. Everything booked (!). (Some L2 cards are pulled out). 49 wedges of Silicon are in. 
 
Plan: run with default (4stream) table Physcs_0_00[2,161]; inquire if we should take some min bias and/or 5-stream data.</pre>


<!-- Date: Sun Jul 8 16:22:12 2001 -->
<br> Started new run 119636 at the begining of the shift  
<br> (RC java out of memory on the previous shift forced to end ongoing run 119635) <br>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119636' target=_top>119636</a>) -->
<!-- Comment: Mon Jul  9 06;58;13 comment by...W.Badgett -->
<pre>Did you try rejoining the orphaned partition?
</pre>
<!-- Comment: Mon Jul  9 08;21;33 comment by...hray -->
again, no because I screwed up and jumped to choose a partition.  Anyes tried to stop me, but it was too late.  
<!-- Date: Sun Jul 8 16:41:23 2001 -->
<pre>TOF alarm at the flying time (16.29).  I went to check the display downstairs, all of the channels had the actual voltage the same as the requested voltages, so the TOF didn't trip. 
The TOF is now green, probably just a fake alarm.</pre>
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119636' target=_top>119636</a>) -->


<!-- Date: Sun Jul 8 16:58:38 2001 -->
<pre>These are the integrated silicon wedges: 

SB0W0   SB1W0   SB2W2  SB3W0  SB4W1  SB5W2  IB0W0
SB0W2	SB1W2   SB2W3  SB3W2  SB4W2  SB5W3  IB0W1
SB0W3	SB1W3   SB2W5  SB3W3  SB4W3	    IB5W0
SB0W4	SB1W4   SB2W6			    IB5W1
SB0W6	SB1W5   SB2W7			    IB5W5
SB0W7	SB1W6   SB2W8
SB0W8	SB1W7   SB2W9
SB0W9	SB1W8
	SB1W9
</pre>
<!-- Author: Victoria -->


<!-- Date: Sun Jul 8 17:24:56 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1627&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1627  width='300' align='top'></td><td>Following the observations of the owl shift consumer. CES in wedge East 02. Here is what YMON gives you for wires and strips. We need CES for electrons.</a></td></tr></table>
<!-- Author: Fumi Ukegawa -->


<!-- Date: Sun Jul 8 17:25:41 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1628&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1628  width='300' align='top'></td><td>Wires.</a></td></tr></table>
<!-- Author: Fumi -->


<!-- Date: Sun Jul 8 18:00:58 2001 -->
<ul>  
<li> Savario noticed that in wedge SB0W3 layer 4 was in standby (I presume since we integrated last night in run 119619).  
This mean the ladder was readout but not data. 
<li> End run 119636 with 400k evts to put this ladder on. <br></ul>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119636' target=_top>119636</a>) -->


<!-- Date: Sun Jul 8 18:02:37 2001 -->
COT SL8 S9a trips between runs.
<!-- Author: Victoria -->


<!-- Date: Sun Jul 8 18:03:05 2001 -->
<br> While getting ready to restart run, CMX voltages were readjusted<br>  
<!-- Author: Anyes -->


<!-- Date: Sun Jul 8 18:06:39 2001 -->
Started run 119637.
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119637' target=_top>119637</a>) -->


<!-- Date: Sun Jul 8 18:17:33 2001 -->
<pre>Another fake TOF alarm at 6.08. 
All the voltages (from the PC on the 1st floor) are ok. 
I paged the TOF expert at the last alarm and he said that if all the voltages are okay, there is no problem.</pre>
<!-- Author: Victoria  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119637' target=_top>119637</a>) -->


<!-- Date: Sun Jul 8 18:19:20 2001 -->
Monitoring on L3 subfarm 1 (counting from 0) is dead.
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119637' target=_top>119637</a>) -->


<!-- Date: Sun Jul 8 18:26:00 2001 -->
<br>Reformatter errors:<pre>  
 Error during scanning MINI structure! 
 Last Unit prob. worked on (all counts from 0): 
  SCPU: 2 
  VRB : 2 
  LINK: 0 
  MINI: 0     </pre> 
Using the tool, I see this is FIB_44, channel 6.<br> 
The rate seems to be large about 10%
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119637' target=_top>119637</a>) -->
<!-- Comment: Sun Jul  8 18;30;56 comment by...V -->
Using hdump Anyes finds that FIB_44, channel 6 is FIB_ISL_03. 


<!-- Date: Sun Jul 8 18:47:37 2001 -->
<pre>We are getting no events written out by the CSL. 
L1 is talking events at 66Hz, L3 numbers are increasing in the L3 display.   
We have had no more refomators errors since the flying wire time (18.29) when Anyes reset the L3 display (ie none of the L3 names have turned red since them). 
There are no files on the look area. 
Tony is coming over...</pre>
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119637' target=_top>119637</a>) -->
<!-- Comment: Sun Jul  8 19;56;09 comment by...V -->
<pre>We never had a CSL problem.  It was a silicon reformator error, which was rejecting all the events, so no events ever got to the CSL.</pre>
<!-- Date: Sun Jul 8 18:51:17 2001 -->
<pre> 
   ~18:00pm I had a chance to decrease the HV settings  
   of wedges 7,8 West side of the CMX. We had few 
   channel that were hot. See CMX E-log book for  
   details. </pre>
<!-- Author: Abraham -->


<!-- Date: Sun Jul 8 18:51:25 2001 -->
<pre>We thought we were taking data (looked like it from L3 and RC displays!), but found that no events were coming out of the CSL. Page CSL- Tony here investigating</pre>
<!-- Author: Henry :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119636' target=_top>119636</a>) -->
<!-- Comment: Mon Jul  9 09;28;01 comment by...Henry -->
As Tony noted above- not a CSL problem. See notes below...
<!-- Date: Sun Jul 8 19:08:22 2001 -->
<pre>We end the run (119637).</pre>
<!-- Author: Henry -->


<!-- Date: Sun Jul 8 19:13:11 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1629&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1629  width='300' align='top'></td><td>L3 display from one node, this is a fairly typical of the ones I looked at<br> Huge number of events rejected because of reformatter errors.</a></td></tr></table>
<!-- Author: Victoria -->


<!-- Date: Sun Jul 8 19:27:46 2001 -->
Clean up EVB and L3.  b0eb14 didn't recover.  I went to the 1st floor and pressed the reset button.  Cleanup EVB again, everything is ok now!
<!-- Author: Victoria -->


<!-- Date: Sun Jul 8 19:34:57 2001 -->
Silicon was giving 80% reformatter errors, coming from  
FIB_ISL_03 clot 12 chan 6, wihch is an ISL wedge which  
is not powered. Found it active in the hardware database, 
so I turned it off.  
 
It's a complete mystery how it managed to get turned on in DB.
<!-- Author: Lester -->
<!-- Comment: Mon Jul  9 08;26;59 comment by...SCN -->
This is the <B>SOC</B> ladder!
<!-- Comment: Mon Jul  9 09;29;04 comment by...Henry -->
What's an SOC ladder?
<!-- Comment: Mon Jul  9 16;18;49 comment by...Alan -->
Silicon Outside the COT.
<!-- Date: Sun Jul 8 19:43:27 2001 -->
Started run 119638.
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119638' target=_top>119638</a>) -->



<!-- Date: Sun Jul 8 20:06:12 2001 -->
<ul> 
New toy to check which silicon is integrated (works on b0dap23!)
<li> setup perl v5_005a 
<li> ./cratematch.pl 
<li> at the prompt: <br>Please enter a list of fib crates separated by space, e.g. b0fib01
<br>cratematch> 
<li> type: b0fib00 b0fib02 b0fib03 b0fib04 b0fib07  
<br> The script is a bit fussy with spaces, so make sure you don't have any space before the first fib.
<br> What need to be check is the inconsistencies at the end. 
<li> At the moment we get: 
<br>  The following wedges should be biased and are not: 
<br> SB0W7L4 
<br> SB1W4L3        
<li> This is "normal", because of setting on power supply. 
<li> This should be used when we intergrate the Si, to make sure
that the correct wedge got power up, <br> in order to avoid runing with ladders which are only in StandBy!
</ul>
<!-- Author: Anyes -->


<!-- Date: Sun Jul 8 20:22:02 2001 -->
<pre>A brief summary of what happened, and what we (think we) learned from it: 
	
1. It looks like somehow in the middle of run 119637 FIB_ISL_03clot 12 chan 6, an ISL wedge which is not powered, became active in the hardware database.  

	2. This caused a very high percentage of reformatter errors in L3, eventually 100%, so no events were being written out. 

	3. L1 triggers were happily running, RC was fine- but the consumers weren't getting any events, there were reformatter errors on the monitor over the Ace desk, and the L3 display was all light blue (no flickering of darker blue).  

	4. The average rates in various streams were ok- only the instantaneous rates went to zero. 
 
It took us a while to catch on that even though it looked as if we were running fine, in fact we weren't taking ANY  data (it's been steady running, and the luminosity isn't high). 
 
I'll put some suggestions based on what we learned in the next message so we can hilite them.</pre>
<!-- Author: Henry -->
<!-- Comment: Mon Jul  9 07;10;07 comment by...W.Badgett -->
<pre>FIB_ISL_03, FIB_44  (Slot 12), channel 6 was put online 
by Petra at 2001-07-05 21:59.     Presumably this should 
have been noticed long before now.  (??) Why wasn't it?
(It was not put online during run 119637)

History since April:
                2001-07-08 19:27  Put offline  LMILLER
                2001-07-05 21:59  Put online   PETRA
                2001-07-05 13:25  Put offline  PETRA
                2001-07-02 18:35  Put online   PETRA
               </pre>
<!-- Date: Sun Jul 8 20:38:21 2001 -->
CPR trip SE wedges 12-17 at event 210972.  HV reset ok.
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119638' target=_top>119638</a>) -->


<!-- Date: Sun Jul 8 20:44:00 2001 -->


<pre> Some suggestions and comments from the above (at the risk of generating acerbic messages- we're trying to be helpful): 
	 
1. We need an alarm (e.g. a red window, or something more obvious than at present) if the reformatter errors averaged over some (adjustable) length of time exceed some (adjustable) threshold. We cannot afford to lose the deadtime and events from any substantial rate of reformatter errors. At present these give error messages, but are prescaled by a large number; we had a situation where it got suddenly worse, and we didn't catch it immediately even though we knew something was wrong.  The loss 
due to reformatter errors needs to be made very small.
 
	2. The deadtime that is quoted doesn't take into account reformatter errors (or any other L3 rejection factor- it's not
that it's calculated incorrectly- it's just it's the deadtime
up until the input to L3)- the CSL display claimed we were 98.8% live, whereas in fact were effectively 100% dead. The run db doesn't track this either. The real livetime, including the
L3 rejection factor, is consequently quite different from what
we display as livetime- we can be 100% dead and claim to be 100% live.  This seems very serious. 
 
	3. We do not have (or don't know of) a display running in B0 that tells us how many events have been written out that is available after a run is ended. During the run we can look at the CSL display and it tells us, but the run db doesn't (we think) have anything. Thus one could run the way we were at the end of 199637, have many triggers and nb-1 in the run db, livetimes close to 100%, and no events written out. In terms of calculating cross-sections offline from our data it seems as if there's a leak in our (easily accessed) accounting 
somewhere. The CSL numbers for the various streams, or at least the total events written out, should be in the rundb? 
Again, this seems very serious: we should be able to account for every event that comes through the pipeline starting at L1.
(In a worst case, one can imagine L3 reformatter errors, for example, that are topology related- e.g. more often to occur in 
top events, say, than in other events.). </pre>
<!-- Author: Henry -->
<!-- Comment: Sun Jul  8 22;26;18 comment by...Henry -->
On further thought, I've realized that we really should KEEP
events with reformatter errors, rather than throw them away.
They would have to have their own path of some kind, but if
we cannot guarantee that an error is not 
correlated with physics (e.g., occupancy in the SVX), we should
keep the event for study rather than forever ditch it. (This is true in general for ALL trigger system errors- it is always
more conservative to accept the event on error rather than reject it.) 
<!-- Comment: Sun Jul  8 22;40;19 comment by...patrick -->
<ul>
<li>The reformatter messages contain a reject faction, for
example this is one from near the end of run 119637:<br>
<pre>
RAWREF Error - UTILSILDEC - with code 38 -- START
 Line -  302 in: src/library/rr_rawevt.c
 Buffer position: 13599
 Error count: 1000 of 1000
 Reject rate: 80.906 percent

 Detector ID not known (yet?).
 FIB id found:
  0xc2c2c4c4
</pre>
<p>
This suggests that 80% of the events were being rejected.
There were many of these in the run control message windows.
We are working on making a more succinct version of the
Level 3 reformatter message for the message window,
and will implement a pop-up window if the rate is high.
<li>A summary of events seen by various parts of the
system may be found
<a href="/java/daqmon/servlet/DaqMonServlet3">here</a>.
The level 3 numbers will not be correct if some portion of
the Level 3 display is orange.
<li>All messages from the run control window go to log files:
$ERRMON_DIR/../log/daq/errorfilennnnnn.log where nnnnnn is
the run number. At the end of the file is a summary of
events from every component in the system.
Except that the Level 3 end of run summary is not yet
implemented.
<li>Changes to the hardware database during a run only have
any effect if one selects "Refresh Hardware Database" and
redownloads a part of the system during the run.
The 100% errors started only 8 minutes into the run.
There must be some other explanation for what happened.

</ul>

<!-- Comment: Mon Jul  9 07;14;43 comment by...W.Badgett -->
<pre>About HDWDB:  see my previous comment above.   Change was 
done last week.</pre>
<!-- Comment: Mon Jul  9 09;22;05 comment by...SCN -->
Here's my guess:<ul>
<li> This Si Channel corresponds to the SOC (<B>S</B>ilicon <B>O</B>utside <B>C</B>OT), which has been
turned on and off a bit as of late to do "special studies".
I believe recently there was a decision made to include it
as part of normal running, thus Petra turned on the PS
and turned on the bit in the HDWDB on the 5th.  And it
behaved fine, read out fine, etc, so noone "noticed" anything
since there was nothing to notice.
<li> Looking at the PS entry 3 days later, I see <pre>
Sun Jul 08 17:56:18 CDT 2001     success: Voltage changed for SB0W3 L4 to 45.0 V
Sun Jul 08 18:12:28 CDT 2001     success: Power Supply SVX B1 W7 OFF
Sun Jul 08 21:16:18 CDT 2001     success: Power OFF
</pre>

The SOC is using PS for <pre>SVX B1 W7</pre> so it looks like
someone just turned it off at around 18:12 (it is <B>NOT</B>
a trip), which fits in
pretty well with when the reformatter errors started occuring.
<li>
In the Silicon <a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=50&frame=2&anchor=18:12:03'>
log book</a>,  it is clear that the SOC was mistaken 
for the real SVX B1 W7 L1 and this the PS was turned off,
and everybody got confused.
</ul>
I guess the only good thing I can say about this debacle is that
I was able to reconstruct what happened from logs and notebook entries.  It looks like
pilot error, so we'll try to straighten out the pilots.  Better
debugging (what Jim said) would help too of course.
<!-- Date: Sun Jul 8 20:56:08 2001 -->
<br> Run out pof memory, so stoped taking events!!! 
<br> end run 119638 <br>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119638' target=_top>119638</a>) -->


<!-- Date: Sun Jul 8 21:03:28 2001 -->
COT, CMP, CMX CES, CPR trip- they have lost the beam. 
<!-- Author: Henry -->


<!-- Date: Sun Jul 8 21:08:11 2001 -->
<pre>Wayne in Cryo had asked earlier about pumping the Watt can; 
I had put him off, but now tell him to go ahead. We run down 
COT and TOF. Page SVX to ask them what they want to do. MCR 
is figuring out what happened- don't know their plans yet.</pre>
<!-- Author: Henry -->


<!-- Date: Sun Jul 8 21:16:20 2001 -->
Auke-Pieter would like us to turn off the SVX voltage.  
Lester helps us by phone.
<!-- Author: Henry -->
<!-- Comment: Sun Jul  8 21;18;36 comment by...V -->
It's off (there were no problems).


<!-- Date: Sun Jul 8 21:22:06 2001 -->
<pre>Silicon got a tiny amount of radiation when the beam was lost: 
 
W Inner  0.064 
W Outer  0.040 
E Inner  0.043 
E Outer  0.055 
 
We got no rads during the rest of this store.</pre>
<!-- Author: Victoria -->


<!-- Date: Sun Jul 8 21:58:56 2001 -->
<center><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-1630&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-1630' width='300'></a></center>
<!-- Author: Victoria -->
<!-- Comment: Sun Jul  8 22;01;14 comment by...V -->
Luminosity summary for store 567.
Got 150 nb-1, took 100 nb-1, so overall data talking efficiency was 67%.




<!-- Date: Sun Jul 8 23:02:36 2001 -->
<ul> 
<br> CALIBRATION TIME  
<li> QIE calib 119639 
<li> SMQIE calib 119640 
<li> COT calib 119641 
<li> Muon  calib 119642 (error imu01 bunchcounter slot 11 & 19 disagree) 
<br> Do another calib 119643  
<li> TOF calib 119644 
<li> COT again (same reseason below) 119645 
<li> Muon again (forgot to change it to generic calib) 119646 
 
<li> CAL LED calib 119467 (error starting consumer LED_0_Consumer) 
<br> do another one 119468 
<li> XEF xenon OFF calib 119649  
<li> XEF xenon ON calib couldn't do it due to the PC pb (see below) 
<br> 
</ul>
<!-- Author: Anyes -->
<!-- Comment: Sun Jul  8 23;31;54 comment by...Anyes -->
<br> LED Calib never appear into DB so did another one 119650
which worked.<br>
<!-- Date: Sun Jul 8 23:06:23 2001 -->
<pre>Tried to swtich on the Xenon HV on the pisabox on the 1st floor. 
It didn't want to start IFIX, as there was not enough free disk space.  It keeps giving me lots of errors essentially telling me this!
 
At the advice of Steve Hahn (over the phone) I log out the pc and log back in.  Again I can't start IFIX for the same reason, so I give up. 
 
The two processes that are supposed to be running on the pisabox (gamma + pb(?)) are running.
</pre>
<!-- Author: Victoria -->
<!-- Comment: Sun Jul  8 23;18;10 comment by...V -->
<table border=0><td><tr><td bgcolor=#e0e030>Someone should clean up the disk on the pisabox!!!</td></tr></td></table>
<!-- Comment: Mon Jul  9 00;51;47 comment by...ACE -->
Every time I was there, the first thing I would do is to kill these processes(gamma+pb(?)), as it is said in the instruction pasted beside the PC. After I've finished the calibration, I just turn off the flasher HV and logout the pisabox account. Then login to make some processes restarted:whatever. 




<!-- Date: Sun Jul 8 23:07:35 2001 -->
Rebooted SVT crates, to fix a bug I introduced on Friday into 
the Hit Finder coldstart code.  The new code was cleverly 
trying to enable/disable Hit Finder channels automatically, 
depending on which FIB channels are enabled/disabled. 
Unfortunately, I made a dumb mistake in a simple computation 
of which bits to flip, resulting in SVT finding essentially 
no tracks over the weekend.  Let's hope we do better tonight! 
<p> 
Marco Rescigno currently has the SVT pager.  If this fix gives 
trouble tonight, you may want to call me at 773-263-5249 
before you wake him up. 
<p>  
<!-- Author: Ashmanskas -->



<!-- Date: Sun Jul 8 23:27:51 2001 -->
<pre>Aseet is working on COT - ignore  alarms until he says the  
coast is clear</pre>
<!-- Author: Henry -->



<!-- Date: Sun Jul 8 23:40:47 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1631&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1631  width='300' align='top'></td><td>Lumminosity Summary for our shift, until the beam disapeared.</a></td></tr></table>
<!-- Author: Victoria -->


<!-- Date: Sun Jul 8 23:49:24 2001 -->
<br> Rick Run 119650 and live it running for a while since this is about the only usefull thing that can be done right now <br>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=119650' target=_top>119650</a>) -->
<!-- Comment: Mon Jul  9 08;57;15 comment by...R.J. Tesarek -->
This is really run 119651.  Run 119650 is an LED calibration.
<!-- Date: Mon Jul 9 00:02:32 2001 -->
<font color=red>Shift Summary: </font><pre>We took data until the beam was lost in a big quench. 
We had a problem in the Silicon that cause L3 reformatter 
errors, and so ran for a while with no output. 
We then took calibrations, while folks pumped on their 
Watt can and Aseet worked on the COT 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   119636-638
Delivered Luminosity   22 nb-1  
Acquired Luminosity    15.5 nb-1  
Efficiency             2/3
   
</td></tr></table></pre>
<!-- Author: Henry -->