<!-- Title: Eve shift. Sun Sep  8, 2002 -->
<!-- SciCo: Giorgio Apollinari -->
<!-- DAQAce: Bruce Knuteson -->
<!-- MonAce: Adam Gibson -->
<!-- CO: Stan Lai -->
<!-- OpManager: Eric James -->
<!-- Notes: -->
<pre>Inheriting store 1737 and run 151140.</pre>

<!-- Date: Sun Sep 8 16:20:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151140">151140</A>
 Terminated at 2002.09.08 16:20:42
<!-- processed by 131.225.236.161 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 16:20:22 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151140">151140</A>
TERMINATE: Ending run due to done timeouts from COT crates.
<!-- processed by 131.225.235.217 -->
<!-- Author: Knuteson -->



<!-- Date: Sun Sep 8 16:24:55 2002 -->
<pre>COT expert (Ting) was paged, because we get the voice alarm \ 
"COT alarm" from time to time in the last 1 hour. 
 
 The high voltage is fine in the "High Voltage Summary" page, 
and on the top-righmost screen on the Monitoring ACE's side 
which displays all COT channels to be written in green color. 
 The "COT High Voltage" popup window though shows that 
almost all channels read about 2 Volts lower than their setting 
voltage... 
 
 Expert says it's an iFix problem, and they'll work on it tomorrow.</pre>
<!-- Author: Kordas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151140' target=_top>151140</a>) -->
<!-- Comment: Mon Sep  9 10;30;07 comment by...Ting Miao -->
The CAMAC ADCs monitoring low voltages and temperatures were responsible for the alarms. We will watch it closely. If this happens more, we will try something to fix it -- may have to replace the ADCs. 
<!-- Date: Sun Sep 8 16:31:45 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151141">151141</A>
 Activated at 2002.09.08 16:32:46
<!-- processed by 63.215.233.57 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 16:32:15 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151141">151141</A>
ACTIVATE: Restarting physics run
<!-- processed by 131.225.236.161 -->
<!-- Author: Knuteson -->



<!-- Date: Sun Sep 8 17:13:29 2002 -->
MCR and Eric called in. The plan is to keep this store until 4:00-5:00 AM, then prepare the Collision Hall for access and fix the TEL. FOr CDF the plan is therefore to keep taking data and let the expert crew work on SVX problem until they feel exhausted (~midnight). After that happen we will run the two pending requests of tests with beam.
<!-- Author: Giorgio -->

<!-- Date: Sun Sep 8 18:05:12 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151141">151141</A>
 Terminated at 2002.09.08 18:06:11
<!-- processed by 131.225.236.229 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 18:05:34 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151141">151141</A>
TERMINATE: Ending run at Gino's request.
<!-- processed by 131.225.236.229 -->
<!-- Author: Knuteson -->



<!-- Date: Sun Sep 8 18:05:34 2002 -->
Stopping and starting new run on SVX request.
<!-- Author: Giorgio -->

<!-- Date: Sun Sep 8 18:09:50 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142">151142</A>
 Activated at 2002.09.08 18:10:46
<!-- processed by 131.225.236.161 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 18:10:36 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142">151142</A>
ACTIVATE: Restarting physics run.
<!-- processed by 131.225.236.161 -->
<!-- Author: Knuteson -->



<!-- Date: Sun Sep 8 19:39:33 2002 -->
<pre>EVB TracerEventId error seen (SCPU = b0eb11).  Seemingly okay after HRR.</pre>
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142' target=_top>151142</a>) -->


<!-- Date: Sun Sep 8 19:41:40 2002 -->
<pre>Subfarms C11, C14, and C15 died soon after the cleanup of EVB and L3 toward the beginning of this shift, around 5pm.  Have been running without them.</pre>
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142' target=_top>151142</a>) -->


<!-- Date: Sun Sep 8 19:49:23 2002 -->
<pre>EVB VrbVadByteCount error (SCPU=b0eb15, slot=10), recovered with HRR.</pre>
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142' target=_top>151142</a>) -->


<!-- Date: Sun Sep 8 20:10:06 2002 -->
<pre>Busy timeout caused by EVB TracerEventId error (SCPU=b0eb15).</pre>
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142' target=_top>151142</a>) -->


<!-- Date: Sun Sep 8 20:11:48 2002 -->
<pre>CMUP and CMX tripped, correlated with enhanced pbar losses.  CMUP and CMX high voltages brought back up; now seemingly okay.</pre>
<!-- Author: Knuteson :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142' target=_top>151142</a>) -->


<!-- Date: Sun Sep 8 20:25:10 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-7402&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-7402' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-7401&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-7401' width='300'></a></center>
<!-- Author: Adam -->
<!-- Comment: Sun Sep  8 20;30;47 comment by...Adam -->
<pre>Plots of the losses apparently responsible for the CMX and CMP trips.  LOSTPB (regular pbar losses) jumped from ~1800 HZ to ~4000 HZ.  B0ABSM (pbar losses in the abort gap - the TEL is off) jumped from ~18,000 HZ to ~40,000 HZ.  B0PBSM (proton losses in the abort gap) jumped from ~15,000 - 50,000 Hz to as high as ~450,000 Hz.  LOSTP (the normal proton losses) didn't really move.  Now, all four variables seem back to normal.  The MCR apparently saw the same jump in losses - and has no immediate explanation.  </pre>

<!-- Date: Sun Sep 8 21:10:15 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142">151142</A>
 Terminated at 2002.09.08 21:10:59
<!-- processed by 216.214.203.252 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 21:11:15 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142">151142</A>
TERMINATE: SM_VRB_READ, SCPU_CANT_RESET_VRB, and SM_SCPU_TIMEOUT errors.
<!-- processed by 131.225.236.229 -->
<!-- Author: Knuteson -->
<!-- Comment: Sun Sep  8 21;15;27 comment by...Frank Chlebana -->
<pre>

SCPU-P0-E-CantResetVrb: Reset of VRB in slot 10 failed.
VRB_BUS_ERROR: Alignment or VME bus error.[MLE]

(RC)   21:1:2 Recover -&gt; RECOVERED
1'3" b0l3gate1.fnal.gov:main:9:01:01 PM-&gt;Host b0eb15.fnal.gov, task tRec_0
SCPU-P0-E-CantResetVrb: Reset of VRB in slot 10 failed.
VRB_BUS_ERROR: Alignment or VME bus error. --&gt;  
 Additional Information: 
 
 Attention!!!. Event Builder SCPU_CANT_RESET_VRB Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Abort + Reset 
 - Identify the BAD VRB CRATE ( b0ebXX ) in the error message text 
 - Go to SCPU rack: 
 - turn power off for 30s, then back on 
   ONLY FOR THE BAD VRB CRATE b0ebXX 
 - Wait for 1 min, run "Check SM/SCPU" from EVB/L3 Control Panel
 - When all SCPU's are OK, run "Clean up EVB"                       
 - Start run again. If the problem persists, please page L3 pager. 
   
 &gt;&gt;&gt;&gt; IF the BAD VRB crate corresponds to b0eb19 &lt;&lt;&lt;&lt; 
 &gt;&gt;&gt;&gt; please page SVX expert and get OK BEFORE doing anything &lt;&lt;&lt;&lt;  </pre>


<!-- Comment: Sun Sep  8 21;18;12 comment by...Frank Chlebana -->
<pre>At the end of teh error log file:

SM-P0-E-ScpuDropout: Lost LAN connection to SCPU b0eb15 during End Run (reading reply).[MLE]</pre>
<!-- Comment: Sun Sep  8 21;59;09 comment by...patrick -->
The lost LAN connection is what would be
expected if the crate CPU was reset
rather than just the VRB.

<!-- Date: Sun Sep 8 21:16:26 2002 -->
<pre>SCPU_CANT_RESET_VRB error: 
 
Here are the diagnostics: 
 
crate: b0vrb00 slot 10 OK; ID=3 
crate: b0vrb00 slot 12 OK; ID=3 
crate: b0vrb01 slot 10 OK; ID=3 
crate: b0vrb01 slot 12 OK; ID=3 
crate: b0vrb01 slot 14 OK; ID=3 
crate: b0vrb02 slot 10 OK; ID=3 
crate: b0vrb03 slot 10 OK; ID=3 
FISION: Input/output error (0x5): VME BERR received! 
crate: b0vrb04 slot 10 not found; try D0 mode 
FISION: Input/output error (0x5): VME BERR received! 
VRB not found at D0 address either 
crate: b0vrb04 slot 12 OK; ID=3 
crate: b0vrb04 slot 14 OK; ID=3 
crate: b0vrb05 slot 10 OK; ID=3 
crate: b0vrb05 slot 12 OK; ID=3 
 
after resetting crate: 
 
crate: b0vrb00 slot 10 OK; ID=3 
crate: b0vrb00 slot 12 OK; ID=3 
crate: b0vrb01 slot 10 OK; ID=3 
crate: b0vrb01 slot 12 OK; ID=3 
crate: b0vrb01 slot 14 OK; ID=3 
crate: b0vrb02 slot 10 OK; ID=3 
crate: b0vrb03 slot 10 OK; ID=3 
crate: b0vrb04 slot 10 OK; ID=3 
crate: b0vrb04 slot 12 OK; ID=3 
crate: b0vrb04 slot 14 OK; ID=3 
crate: b0vrb05 slot 10 OK; ID=3 
crate: b0vrb05 slot 12 OK; ID=3 
 
Continued getting errors -- had to power cycle crate.</pre>
<!-- Author: Knuteson -->
<!-- Comment: Sun Sep  8 21;56;56 comment by...patrick -->
Before the 2nd vrbscan, was the crate CPU reset,
or just the VRB? The intention is you should just
reset the VRB and see if that gets rid of the
bus error.
<!-- Comment: Sun Sep  8 23;03;54 comment by...Frank Chlebana -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Just taled to the daq ace.

The reset button of the crate controller was pressed, not the 
reset button on the VRB.

If you get a bus error again please try pressing the reset 
button on the VRB card in the slot corresponding to the card
that has the bus error.

The reset button is recessed and you will need a pen to press it.

In this case it would have been the VRB card in slot 10.



</pre></td></tr></td></table>
<!-- Date: Sun Sep 8 21:20:48 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151143">151143</A>
 Activated at 2002.09.08 21:20:54
<!-- processed by 208.19.217.119 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 21:20:49 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151143">151143</A>
ACTIVATE: Restarting physics run.
<!-- processed by 208.19.217.119 -->
<!-- Author: Knuteson -->




<!-- Date: Sun Sep 8 22:39:38 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151143">151143</A>
 Terminated at 2002.09.08 22:40:14
<!-- processed by 131.225.236.161 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 22:40:20 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151143">151143</A>
TERMINATE: Ending physics run in order to test ToF trigger board.
<!-- processed by 131.225.236.229 -->
<!-- Author: Knuteson -->


<!-- Date: Sun Sep 8 22:40:21 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151143">151143</A>
TERMINATE: Ending physics run in order to test ToF trigger board.
<!-- processed by 131.225.236.229 -->
<!-- Author: Knuteson -->



<!-- Date: Sun Sep 8 22:41:35 2002 -->
Ending run 151143 to try including the TOTRIB (TOF) in the read-out.
<!-- Author: APollinari -->

<!-- Date: Sun Sep 8 22:47:54 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151144">151144</A>
 Activated at 2002.09.08 22:47:55
<!-- processed by 131.225.127.7 -->
<!-- Author: RunControl -->


<!-- Date: Sun Sep 8 22:47:54 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151144">151144</A>
ACTIVATE: Restarting physics run with ToF trigger board.
<!-- processed by 131.225.127.7 -->
<!-- Author: Knuteson -->




<!-- Date: Sun Sep 8 23:09:17 2002 -->
<pre>We have received occasional done timeouts this shift, most from COT, plus one from WCAL_05.  These have been recoverable with HRRs.</pre>
<!-- Author: Knuteson -->

<!-- Date: Sun Sep 8 23:54:42 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151140>
151140</A> 
x24E64</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,315]</TD>                                       
<TD ALIGN=CENTER>14:49:40</TD>                                          
<TD ALIGN=CENTER>16:20:42</TD>                                          
<TD ALIGN=CENTER>01:19:33</TD>                                          
<TD ALIGN=RIGHT>     32,059,655</TD>                                    
<TD ALIGN=RIGHT>       724,191</TD>                                     
<TD ALIGN=RIGHT>     257,689</TD>                                       
<TD ALIGN=RIGHT>   61.656</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151141>
151141</A> 
x24E65</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,315]</TD>                                       
<TD ALIGN=CENTER>16:32:46</TD>                                          
<TD ALIGN=CENTER>18:06:11</TD>                                          
<TD ALIGN=CENTER>01:32:11</TD>                                          
<TD ALIGN=RIGHT>     43,686,869</TD>                                    
<TD ALIGN=RIGHT>       765,123</TD>                                     
<TD ALIGN=RIGHT>     280,549</TD>                                       
<TD ALIGN=RIGHT>   64.252</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151142>
151142</A> 
x24E66</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,315]</TD>                                       
<TD ALIGN=CENTER>18:10:46</TD>                                          
<TD ALIGN=CENTER>21:10:59</TD>                                          
<TD ALIGN=CENTER>02:41:34</TD>                                          
<TD ALIGN=RIGHT>     67,211,540</TD>                                    
<TD ALIGN=RIGHT>     1,201,502</TD>                                     
<TD ALIGN=RIGHT>     447,731</TD>                                       
<TD ALIGN=RIGHT>   99.637</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151143>
151143</A> 
x24E67</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,315]</TD>                                       
<TD ALIGN=CENTER>21:20:54</TD>                                          
<TD ALIGN=CENTER>22:40:14</TD>                                          
<TD ALIGN=CENTER>01:18:10</TD>                                          
<TD ALIGN=RIGHT>     32,552,150</TD>                                    
<TD ALIGN=RIGHT>       514,436</TD>                                     
<TD ALIGN=RIGHT>     196,833</TD>                                       
<TD ALIGN=RIGHT>   42.284</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151144>
151144</A> 
x24E68</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [5,151,315]</TD>                                       
<TD ALIGN=CENTER>22:47:55</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>01:03:45</TD>                                          
<TD ALIGN=RIGHT>     29,118,659</TD>                                    
<TD ALIGN=RIGHT>       397,000</TD>                                     
<TD ALIGN=RIGHT>     156,722</TD>                                       
<TD ALIGN=RIGHT>   32.779</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:01</TD>                                                       
<TD ALIGN=CENTER>07:55:15</TD>                                          
<TD ALIGN=RIGHT>     204,628,873</TD>                                   
<TD ALIGN=RIGHT>      3,602,252</TD>                                    
<TD ALIGN=RIGHT>     1,339,524</TD>                                     
<TD ALIGN=RIGHT>  300.609</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- processed by 131.225.127.7 -->
<!-- Author: End of Shift Report -->



<!-- Date: Sun Sep 8 23:55:22 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-7403&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-7403' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-7404&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-7404' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-7405&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-7405' width='300'></a></center>
<!-- Author: Adam -->
<!-- Comment: Sun Sep  8 23;56;38 comment by...Adam -->
<pre>The silicon's been off all night - but here are the usual silicon plots, since they give a nice idea of beam conditions.  Aside from TEL being off, and the spike at 8 pm, things look pretty smooth.  I'm not sure if the RF dip at 7:50 or so is related to the spike in losses at 8 pm.</pre>


<!-- Date: Mon Sep 9 00:03:27 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-7406&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-7406  width='300' align='top'></td><td>Shift luminosity summary plot.</a></td></tr></table>
<!-- Author: Adam -->


<!-- Date: Mon Sep 9 00:04:05 2002 -->
<font color=red>Shift Summary: </font><pre>16:00  Inheriting store 1737 and run 151140. 
 
16:20  Terminating run 141150 for COT crates "Done Time-out". 
 
17:13  Communication with MCR. Store to be kept until 4:00-5:00 AM, then prepare Collision Hall for Access and fix TEL. For CDF plan is to run until ~midnight for Silicon debugging, then switch to the two pending requests of tests with beam. 
 
18:05  Stop and starting 151141 on SVX request. We are still running the [5,151,315] table with the Dynamic prescaling disabled. 
 
20:10  Some LOSTP spikes cause CMX and CMP to trip. Reset and resume data-taking. 
 
22:40  Ending run 151143 to try including the TOTRIB (TOF) in the readout.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   151140-151144
Delivered Luminosity   253.1  
Acquired Luminosity    293.1  
Efficiency             86.4%
   
</td></tr></table></pre>
<!-- Author: Giorgio Apollinari -->
<!-- Comment: Mon Sep  9 00;06;10 comment by...Apollinari -->
Delivered Luminosity 293.1
Acquired Luminosity 253.1

<!-- Date: Mon Sep 9 00:12:46 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151144">151144</A>
 Terminated at 2002.09.09 00:11:50
<!-- processed by 131.225.236.168 -->
<!-- Author: RunControl -->


<!-- Date: Mon Sep 9 00:12:46 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=151144">151144</A>
TERMINATE: Ending run for Koji's test
<!-- processed by 131.225.236.168 -->
<!-- Author: DAQ ace 2080 -->