<!-- Title: Owl shift. Sun Feb  9, 2003 -->
<!-- SciCo: Willis Sakumoto -->
<!-- DAQAce: Kristian Hahn -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: Matthew Jones -->
<!-- Notes: -->



<!-- Date: Sun Feb 9 00:10:08 2003 -->
Running with shot setup configuration
<!-- Author: Willis -->


<!-- Date: Sun Feb 9 00:14:48 2003 -->
Heartbeat alarms from CLC, MNP, BSC, RPS.  They had been powered down by the previous shift.  The BigBrother browser was dead upon the start of shift.  It now shows purple "No Report" icons for cpu,disk,messages and procs for b0dap52, the machine that it's running on.  It claims dap52 has recently been rebooted.
<!-- Author: Kristian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158474' target=_top>158474</a>) -->
<!-- Comment: Sun Feb  9 00;31;33 comment by...Kristian -->
I cleared the "Error opening log files" messages on CLC1, as per Andrea's instructions and the heartbeats were restored.  The timing of these errors fits the pattern he described ... Sunday owl shift around midnight.  I emailed him as was requested.


<!-- Date: Sun Feb 9 00:35:50 2003 -->
Just noticed that ICI shows a grey heartbeat box on the Global Alarms page.  I assume this isn't really a problem since no voice alarms were activated and the Icicle page shows a valid hearbeat.
<!-- Author: Kristian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158474' target=_top>158474</a>) -->


<!-- Date: Sun Feb 9 01:59:01 2003 -->
A number of L3 nodes have their node number shown in red on the Level3Display.  The errors in the messages window are all of the following type <br><br> 
 
ERROR ...<br> 
in exe_exchange()[l3_ana_exc.c 659]<br> 
info: FILTER ERROR [1]Severe!  ERRROR in CT_TrackingModule-globalCT_HL2<br> 
 
 
 
<!-- Author: Kristian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158474' target=_top>158474</a>) -->


<!-- Date: Sun Feb 9 01:59:45 2003 -->
MCR called.  No beam for about 1/2 hr.  We stop the run and take calibrations.
<!-- Author: Kristian -->

<!-- Date: Sun Feb 9 02:00:45 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158474">158474</A>
TERMINATE: ending inherited shot-setup run
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:07:37 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158475">158475</A>
ACTIVATE: Cal QIE calibration run begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:07:38 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158475">158475</A>
TERMINATE: cal QIE calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:13:25 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158476">158476</A>
ACTIVATE: CLC QIE calibration run begins

<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:13:26 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158476">158476</A>
TERMINATE: CLC QIE calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:13:27 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158477">158477</A>
ACTIVATE: BSC QIE calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:15:36 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158477">158477</A>
TERMINATE: BSC QIE calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:15:37 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158478">158478</A>
ACTIVATE: SMQIE calibration begins
<!-- Author: kristian 2080 -->
<!-- Comment: Sun Feb  9 02;58;48 comment by...Willis -->
The session log says this is a Roman pot calibration
<!-- Date: Sun Feb 9 02:17:11 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158478">158478</A>
TERMINATE: SMQIE calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:17:12 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158479">158479</A>
ACTIVATE: LED calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:18:55 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158479">158479</A>
TERMINATE: LED calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:21:03 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158480">158480</A>
ACTIVATE: TOF QIE calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:21:04 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158480">158480</A>
TERMINATE: TOF QIE calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:22:48 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158481">158481</A>
ACTIVATE: COT calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:22:49 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158481">158481</A>
TERMINATE: COT calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:25:00 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158482">158482</A>
ACTIVATE: Muon calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:27:53 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158482">158482</A>
TERMINATE: Muon calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:28:28 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158483">158483</A>
ACTIVATE: TOF TAC calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 02:37:17 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158483">158483</A>
TERMINATE: TOF TAC calibration ends
<!-- Author: kristian 2080 -->



<!-- Date: Sun Feb 9 02:51:51 2003 -->
I tried to initiate a XEF calibration run but had problems with the Xeon gui on the Pisabox.  Rather than responding to clicks, the gui would highlight the buttons and I could drag them across the screen.  Also the "muon" folder that the instructions list did not appear when I followed "open" from the "Files" menu.  I might help to include in the instructions the full path to the muon folder.
<!-- Author: Kristian -->
<!-- Comment: Sun Feb  9 10;07;34 comment by...Steve Hahn -->
This sounds like someone left the Intellution Workspace window in configure mode instead of run mode.  This should never be the case unless experts are editing pictures.  You can check this by looking under the "Workspace" pulldown menu at the top of the window: it should say "Switch to Configure" (do not select this), if it says "Switch to Run" do select it.  But, in theory, this is something you should never have to do.
<!-- Date: Sun Feb 9 02:57:31 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158484">158484</A>
ACTIVATE: Plug Laser (normal mode) calibration begins
<!-- Author: kristian 2080 -->



<!-- Date: Sun Feb 9 03:04:48 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158484">158484</A>
TERMINATE: plug laser (normal mode) callibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 03:05:16 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158485">158485</A>
ACTIVATE: Plug Laser (Myron Mode) calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 03:08:22 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158485">158485</A>
TERMINATE: Plug Laser (Myron Mode) calibration ends
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 03:12:21 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158486">158486</A>
ACTIVATE: SMX QIE calibration begins
<!-- Author: kristian 2080 -->


<!-- Date: Sun Feb 9 03:12:22 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158486">158486</A>
TERMINATE: SMX QIE calibration ends
<!-- Author: kristian 2080 -->



<!-- Date: Sun Feb 9 03:19:15 2003 -->
I received the following message while partitioning for the impending Rick's Run : <br> 
 
2003.02.09 03:13:30 Strange (but not necessarily fatal) Run Configuration selections:<br> 
Using L2 Processors without crate SVT_FITTER_06<br> 
Using L2 Processors without crate SVT_FANOUT_07<br> 
Using L2 Processors without crate XFT_XTRP_00<br> 
Missing crates requires L2AutoAccept non-zero<br> 
Overriding Trigger Table with auto L2 accept, prescale 1<br>  
<!-- Author: Kristian -->
<!-- Comment: Sun Feb  9 10;05;28 comment by...Greg Feild -->
<pre>Sorry about these messages.  They can be ignored.  The 
Ricks' Run configuration is in a transitional phase as 
we go to Readout Lists for all run types and trigger tables.
</pre>
<!-- Date: Sun Feb 9 03:25:29 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158487">158487</A>
TERMINATE: Aborting the Rick's Run as the MCR called and expects beam again soon.
<!-- Author: kristian 2080 -->



<!-- Date: Sun Feb 9 03:27:38 2003 -->
Received the following message while partitioning for the shot-setup run ... <br> 
 
Crate HADRON_TIMING_00 is missing card TDC_EM_09 in slot 3<br>
<!-- Author: Kristian -->
<!-- Comment: Sun Feb  9 04;32;35 comment by...slava77 -->
You can edit the run configuration to include the EM_TDC_09 card. Go to "Modify run parameters" select HADRON_TIMING_00 crate, click "Edit", then add EM_TDC_09 card. Then save the run configuration.
<!-- Date: Sun Feb 9 03:32:42 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158488">158488</A>
ACTIVATE: returning to shot-setup running
<!-- Author: kristian 2080 -->



<!-- Date: Sun Feb 9 04:03:02 2003 -->
Done timeout from CCAL07.  HRR ok.  Here's a snippet from the errorlog <br><br> 
 
39'28" b0dau32.fnal.gov:csl_mon_send:3:39:28 AM->CSL received a "BAD EVENT" in partition 2.[MLW]<br> 
47'35" b0dau32.fnal.gov:csl_mon_send:3:47:35 AM->CSL received a "BAD EVENT" in partition 2.[MLW]<br> 
(MLE) b0ccal07:Messenger:3:54:34 AM->SMXR Not Responding event 4490, slot 4, bunch ctr 80, L2buffer 0, header 0x3bfe450 
54'39" 1 crate/s: b0ccal07(7),  Done TO.[RXPT]<br> 
(MLE) b0tsi02:Messenger:3:54:38 AM->DONE timeout<br> 
(MLE) b0dap59.fnal.gov:AWT-EventQueue-0:3:54:38 AM->Done Timeout: CCAL_07 <br>  
<!-- Author: Kristian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158488' target=_top>158488</a>) -->



<!-- Date: Sun Feb 9 05:26:55 2003 -->
CCU machine crashed ~4:35.  THe Blue Screen 'o Death reported the follwing error:<br> 
IRQL_NOT_LESS_OR_EQUAL ... ntoskrnl.exe<br><br> 
 
I rebooted the machine and Ifix refilled the database.  CSP,BSU,TSU,CSX,CSW went green on the global alarms, but they still have heartbeat problems. 
<br><br> 
CLC,BSC,RPS went red in the alarms page.  Everything was ok on the 3rd floor.  By the time I got back to the control room they went green again.  Now Plug is red.  Willis thinks this could be scheduled virus checking<br> 
  
<!-- Author: Kristian -->
<!-- Comment: Sun Feb  9 10;45;45 comment by...Steve Hahn -->
If an iFIX node like this crashes (blue screen of death), it's a good idea to call the system experts (muon scintillator, in this case), if for no other reason than to make them aware that they have severe errors like this.
<!-- Date: Sun Feb 9 05:27:25 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030>Added EM_TDC_09 card into HADRON_TIMING (b0htdc00) in the plug laser runconfiguration. To have the plug laser runs with EMT (EM timing) data. As long as this works we will not need a special run with EMT included.<br> 
Added to BOTHPLUGS_NORMAL and BOTHPLUGS_MM<br> 
In case it gives problems, pls email slava77@fnal.gov .. and drop the abovementioned card from the HADRON_TIMING crate from the plug laser runconfiguration.</td></tr></td></table>
<!-- Author: slava77 -->


<!-- Date: Sun Feb 9 05:36:27 2003 -->
Plug is back.  Willis confirmed the virus scan.
<!-- Author: Kristian -->


<!-- Date: Sun Feb 9 06:54:30 2003 -->
Converter nodes c03 & c04, as well as output node u02, have turned gold on the L3 display.  I'll end this run and start another.
<!-- Author: Kristian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158488' target=_top>158488</a>) -->

<!-- Date: Sun Feb 9 06:55:52 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158488">158488</A>
TERMINATE: ending run
<!-- Author: kristian 2080 -->



<!-- Date: Sun Feb 9 07:12:58 2003 -->
Before partitioning for the next run, I tried to clean up L3.  The cleanup appears to have hung for b0l3c02.  All partitions appear red in the L3 Display.  ProcMon issued an error saying that Soft EVB proxy has been disabled.  We'll page L3.
<!-- Author: Kristian -->
<!-- Comment: Sun Feb  9 07;36;21 comment by...Kristian -->
The clean up actually progressed past b0l3c02 and is now stuck on b0l3c08 for a longer while.  Here's the output from the error log <br><br> 

6:56:02 AM->Error on L3 node b0l3gate2 (partition 2) Sun Feb 9 06:56:02 2003 l3proxy <br>
  in End (L3ProxyPartition.cc:2792)<br>
  @L3_RELAY_FAILURE Level3 Relay failure detected!<br>
The following subfarms need to be removed from the next runs:<br>
      3  4  <br>
The following Level3 converter nodes are marked bad for now:<br>
      b0l3c03  b0l3c04  <br>
The full list of Level3 nodes giving Relay trouble is printed below<br><br>

      b0l3u02<br><br>


<!-- Date: Sun Feb 9 07:56:58 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Feb 9 07:58:37 2003 -->
<font color=red>Shift Summary: </font><pre>0000-0800 Shift Summary 
----------------------- 
 1. 0000: Running with shot setup configuration 
 2. 0157: MCR calls -- no beam for a while. Quiet time 
          calibrations done, except for XEF (HV setup problems). 
 3. 0320: MCR calls -- back to TeV studies and running with shot 
          setup run configuration. CCAL07 DTO (smx slot 4): 0355 
 4. 0420: iFix - all Muon scintillation counters go red. The 
          monitor PC crashed (BSOD) 
 5. 0430: Lots of iFix "heartbeat" alarms -- probably virus scan 
          going from one monitor PC to another. Confirmed for plug HV 
 6. 0710: L3 display goes all red. L3 cleanup hangs.  
</pre> 
<!-- Author: willis -->