<!-- Title: Eve shift. Sun Jun 16, 2002 -->
<!-- SciCo: Joey Huston -->
<!-- DAQAce: Takashi Ogawa -->
<!-- MonAce: Kathy Copic -->
<!-- CO: Daniele Depedis -->
<!-- OpManager: Dave Ambrose -->
<!-- Notes: -->
<pre>Shot setup may occur during shift. </pre>

<!-- Date: Sun Jun 16 16:24:29 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146669">146669</A>
 Terminated at 2002.06.16 16:24:08
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 16:25:25 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146669">146669</A>
TERMINATE: IMU_00 giving done timeouts, HRR did not solve
<!-- Author: ACE 2080 -->


<!-- Date: Sun Jun 16 16:54:30 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146672">146672</A>
 Activated at 2002.06.16 16:53:01
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 16:54:57 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146672">146672</A>
ACTIVATE: Max G: Cosmic ray run with only one cem crate in, SUMET2.5 trigger table.
Test for EMTiming project
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 17:07:50 2002 -->
Cosmic ray run with only ccal15 and htdc crates left. 
Study for EMTiming project
<!-- Author: Max G :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146672' target=_top>146672</a>) -->

<!-- Date: Sun Jun 16 17:26:31 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146672">146672</A>
 Terminated at 2002.06.16 17:25:45
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 17:34:53 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146673">146673</A>
ACTIVATE: Max G
CEM LED calibration run in D-mode. For EMTiming project
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 17:44:29 2002 -->
CEM LED run in D-mode. EMTiming project study
<!-- Author: Max G :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146673' target=_top>146673</a>) -->

<!-- Date: Sun Jun 16 17:58:54 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146674">146674</A>
 Activated at 2002.06.16 17:57:24
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 17:58:54 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146674">146674</A>
ACTIVATE: Back to Cosmic run
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 18:07:36 2002 -->
TSU and BSU are removed from imu00 but there is still L2 timeout problem.   
<!-- Author: takashi -->

<!-- Date: Sun Jun 16 18:16:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146674">146674</A>
 Terminated at 2002.06.16 18:14:12
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 18:16:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146674">146674</A>
TERMINATE: Problem in imu00.
<!-- Author: ACE 2080 -->


<!-- Date: Sun Jun 16 18:48:34 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146675">146675</A>
 Activated at 2002.06.16 18:47:39
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 18:50:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146675">146675</A>
ACTIVATE: I remove imu crates because BMU may have trouble and we can't catch the experts.

<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 19:07:15 2002 -->
<pre>We had a BMU trip around 18:15, when the last run was terminated.  #144 had tripped, the left-hand side/outside was pink, indicating Hardware Over/Under Voltage, and the inside/right-hand side was red, indicating a Hardware Trip.  
 
I followed the instructions on the Muon trip webpage: I tried to get the whole detector back to Standby, and this reset the trouble chamber, but it tripped again immidiately. Then tried to reset it individually, but it tripped again. I waited in between tries and tried a total of 5-6 times...  
 
Finally, we got ahold of an expert on the phone, Camille, and she advised me to turn the channel off, so I did. Everything reverted to normal after that. (yay!) I would have turned off the channel sooner, but we were just running Cosmics and the webpage said to page an expert for Over/Under Voltage pink trips.</pre>
<!-- Author: k copic -->

<!-- Date: Sun Jun 16 19:09:10 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146675">146675</A>
 Terminated at 2002.06.16 19:08:40
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 19:10:59 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146675">146675</A>
TERMINATE: large dead time because of too high trigger rate
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 19:11:36 2002 -->
Turned COT to Standby, anticipating shot setup this evening, on instructions from Dave Ambrose and Aseet (on the phone).
<!-- Author: k copic -->




<!-- Date: Sun Jun 16 19:28:02 2002 -->
Daniele is working on TOF, Fiodar is working on Scintillators ... so we are ignoring TOF and CSP/CSW/MSK alarms.
<!-- Author: k copic -->


<!-- Date: Sun Jun 16 19:35:23 2002 -->
BSU/TSU/CSK heartbeats have now been gone for 35 minutes... CSP/CSW/MSK for 10 minutes. Fiodar came upsatirs and said this was fine. He is working on those systems now. 
 
<!-- Author: k copic -->

<!-- Date: Sun Jun 16 19:45:51 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146676">146676</A>
 Activated at 2002.06.16 19:44:09
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 19:49:23 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146676">146676</A>
 Terminated at 2002.06.16 19:46:33
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 19:49:23 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146676">146676</A>
TERMINATE: imu crate with BMU (without BSU/TSU) failed.
<!-- Author: ACE 2080 -->


<!-- Date: Sun Jun 16 19:55:46 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146677">146677</A>
 Activated at 2002.06.16 19:54:49
<!-- Author: RunControl -->


<!-- Date: Sun Jun 16 19:56:10 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146677">146677</A>
ACTIVATE: remove imu crates again
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 20:19:19 2002 -->
Just called MCR to check on shot setup schedule. Was told to expect midnight at best. 
<!-- Author: huston -->


<!-- Date: Sun Jun 16 21:25:17 2002 -->
MCR called and said that they were beginning a 36X0 proton only store. Shot setup still looks like midnight. 
<!-- Author: huston -->


<!-- Date: Sun Jun 16 22:08:21 2002 -->
Final proton bunches are loaded.  
<!-- Author: k copic -->


<!-- Date: Sun Jun 16 22:10:05 2002 -->
<pre>SVRAD totals 
 
W Inner BLM 1.30 
W Outer BLM .283 
E Inner BLM 4.16 
E Outer BLM 4.23</pre>
<!-- Author: k copic -->

<!-- Date: Sun Jun 16 22:11:36 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146677">146677</A>
 Terminated at 2002.06.16 22:10:17
<!-- Author: RunControl -->



<!-- Date: Sun Jun 16 22:17:26 2002 -->
Talked to Dave Ambrose on the phone -- swtiching to Default Physics Trigger Table on his OK. 
<!-- Author: aces -->


<!-- Date: Sun Jun 16 22:23:14 2002 -->
Some heartbeat alarms occurred and then disappeared, twice, for CMU/CMP/CMX/BMU/CES/CPR at the end of the proton load.
<!-- Author: k copic -->


<!-- Date: Sun Jun 16 22:23:39 2002 -->
Heartbeat alarm for COT, appears then disappears.
<!-- Author: k copic -->


<!-- Date: Sun Jun 16 22:25:05 2002 -->
Switching from Cosmics to the Default Physics Trigger, we got the message to Clean up EVB. We ended the run, reset, tried again, and got the same error, so we did a Clean up of EVB. 
<!-- Author: aces -->


<!-- Date: Sun Jun 16 22:25:34 2002 -->
Hint for the shift crew regarding <b>IMU crate problems</b>: because of the way most TDC crates are read out, <b>one cannot simply remove a bunch of TDC's from the readout</b>; this will cause immediate 
"timeout on global TDC done" when the run is started. To run with 
TDC's removed from the readout, an expert (e.g. muon TDC on call person) needs to change a db parameter.  
<!-- Author: Arnd -->
<!-- Comment: Sun Jun 16 22;44;52 comment by...aces -->
OK thanks .. we won't try that, then. :)
<!-- Date: Sun Jun 16 22:27:01 2002 -->
Switching from Cosmics to the Default Physics Trigger, we got the message to Clean up EVB. We ended the run, reset, tried again, and got the same error, so we did a Clean up of EVB. 
<!-- Author: aces -->


<!-- Date: Sun Jun 16 22:29:06 2002 -->
Another round of heartbeat alarms: COT, CMU, CMP, CMX, BMU, CES, CPR, CCR all at once, then all recover. 
<!-- Author: k copic -->
<!-- Comment: Sun Jun 16 22;37;11 comment by...k copic -->
and another round, just a few minutes later, coinciding with a TOF alarm due to Daniele working downstairs.

<!-- Date: Sun Jun 16 22:38:40 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146678">146678</A>
 Activated at 2002.06.16 22:37:53
<!-- Author: RunControl -->



<!-- Date: Sun Jun 16 22:49:59 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146678">146678</A>
 Terminated at 2002.06.16 22:48:56
<!-- Author: RunControl -->



<!-- Date: Sun Jun 16 22:52:11 2002 -->
We wanted to try and add the IMU cartes again, so we ended the run and got a RXPT loop error after hitting end. (?)
<!-- Author: aces -->


<!-- Date: Sun Jun 16 22:55:31 2002 -->
Trying to run with IMU00 and IMU01 included, without TSU and BSU cards, since those systems are still down.
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146679' target=_top>146679</a>) -->


<!-- Date: Sun Jun 16 23:08:41 2002 -->
We're leaving the problem of the IMU crates for the next shifts; we paged the IMU pager twice and the Muon TDC pager once to try and figure out what's happening, but no one called back. Now we're running without them. 
 
It seems like the problem might have to do with the part of the BMU that's shut off - the BMU TDCs are the only cards left in these crates. Or maybe there was a problem with the TDCs that caused the trip?
<!-- Author: k copic -->

<!-- Date: Sun Jun 16 23:10:46 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146680">146680</A>
ACTIVATE: no imu crates
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 23:18:06 2002 -->
L2 decision timeout error occurs. 
Then I type "go" and press "enter" in alpha minicom and recover  
it.  
<!-- Author: takashi -->

<!-- Date: Sun Jun 16 23:38:51 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146680">146680</A>
TERMINATE: L2 decision timeouts
<!-- Author: ACE 2080 -->



<!-- Date: Sun Jun 16 23:47:10 2002 -->
<pre>Final proton bunches are being loaded.</pre>
<!-- Author: k copic -->


<!-- Date: Sun Jun 16 23:51:01 2002 -->
<pre>SVRAD totals: 
 
W Inner BLM 2.61 
W Outer BLM .446 
E Inner BLM 7.38 
E Outer BLM 7.30</pre>
<!-- Author: k copic -->

<!-- Date: Sun Jun 16 23:55:35 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:02</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Jun 16 23:56:28 2002 -->
b0dap58 crashed -- the mouse moved but no keyboard response, and the mouse didn't move correctly (it would scroll crazily!) We could ping it but ending processes didn't seem to help, so we eventually rebooted it manually and restarted the DAQ voice alarms.
<!-- Author: k copic -->


<!-- Date: Mon Jun 17 00:02:38 2002 -->
<font color=red>Shift Summary: </font><pre>EM timing tests (Run 2b related) early in shift. 36X0 proton only stores later in shift. Problems relating to IMU crates during shift.  Now running the default physics trigger table. Expect shot setup to occur after midnight. </pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   146672-146680
Delivered Luminosity     
Acquired Luminosity      
Efficiency             
   
</td></tr></table></pre>
<!-- Author: huston -->


<!-- Date: Mon Jun 17 00:08:06 2002 -->
b0dap58 crashed -- the mouse moved but no keyboard response, and the mouse didn't move correctly (it would scroll crazily!) We could ping it but ending processes didn't seem to help, so we eventually rebooted it manually and restarted the DAQ voice alarms.
<!-- Author: k copic -->