<!-- Title: Day shift. Tue Sep 25, 2007 -->
<!-- SciCo: Jean slaughter -->
<!-- DAQAce: Glenn Strycker -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: &nbsp; -->
<!-- Notes: -->
<pre>Another day on shutdown shift</pre>


<!-- Date: Tue Sep 25 08:16:05.797 2007 -->
Dervin going to cut the first DIRAC-DCAS cable.
<!-- Author: Laura -->


<!-- Date: Tue Sep 25 08:17:28.241 2007 -->
<pre>  - When I came in today there was a Busy Error and a ProcMon error that 
    occurred overnight.  However, these errors occurred at 11:00pm 
    and 3:00am, so things appear to be running smoothly for quite a while. 
 
  - I got a second busy TO at 7:58am.  "Check Timeout" revealed the problem to be a 
    "too large event fragment in scpu b0evme08".  Auto-HRR recovered the system. 
 
  - at 8:05 we got a "node-unreachable" audio alarm and the heartbeats for CLC systems 
    all went off.  I wonder if this is related to the CDFONPROD upgrade? 
    CES, CMU, CMP, CMX, BMU also lost heartbeat (pink state).</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 08:19:03.183 2007 -->
Many errors just occurred!  Laura says that they are related to the crate power cycle she's doing, so I won't list them all.
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 08:24:13.568 2007 -->
RTSERVER just went down -- this may be the start of the CDFONPROD DB upgrade that was scheduled for 8am today (see yesterday's elog note).  I'm ending the COSMICS run and will attempt to reset RTSERVER.
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 08:26:15.579 2007 -->
Here's the RTSERVER error: 
 
<pre> 
Found 1 potentially stuck ports  
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      
tcp        0    272 b0server04.fnal.gov:5101    b0l1cl00.fnal.gov:1729      ESTABLISHED cdfdaq     66319482 
</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 08;28;28 comment by...G.S. -->
Oh, wait, this is the crate on which Laura is working, I believe.  We'll wait until she's done before worrying about things.


<!-- Date: Tue Sep 25 08:37:47.770 2007 -->
Ray Culbertson came in and helped us with the heartbeat problems -- I need to reboot VOLTMAN.
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 08;59;27 comment by...Glenn Strycker -->
<pre>Talked with Steve Hahn -- he said to check the CLC1 and Muon3 systems.  I did and noticed errors that had shutdown the voltage monitors on those systems.  I closed the error messages and ended the programs.  Restarted.  Rebooted VOLTMAN again (it was showing errors).  I also hit "restart" for ICICLE on the ProcMon.  Hmmm... I still seem to have the heartbeat errors.  I definitely fixed something, but there must be more to do.</pre>
<!-- Date: Tue Sep 25 08:49:42.141 2007 -->
Bob Wagner said that Bill Noe will be checking the SUVA for COT.  bob will disable the alarms associated with the power supplies he will turn off.  No COT calibrations, but we should pay attention to any alarms that do sound and page Bob if needed.
<!-- Author: Jean slaughter -->



<!-- Date: Tue Sep 25 09:00:52.193 2007 -->
<pre>Laura is halfway done with her work -- she asked me to start a COSMICS run to test things.  I have to re-shepherd crate b0l1pu03 (unresponsive) and then I'll start the run.</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 09;01;44 comment by...G.S. -->
Hmmm... I can't seem to reset RunControl.  It's stuck.  I'm closing and restarting.


<!-- Comment: Tue Sep 25 09;04;22 comment by...G.S. -->
Now it won't start!  This must be CDFONPROD DB upgrade related.  We won't be able to do this test for a few hours then.
<!-- Date: Tue Sep 25 09:04:15.266 2007 -->
<table border=0><td><tr><td bgcolor=#e0e030>This morning Dervin cut the first cable, we checked everything was ok, then he cut additional 15 cables, (phi=0, eta=0..23, phi=1, eta=23..16). As soon as run control came  back we'll have cosmics, so that Giorgio can monitor the system. </td></tr></td></table>
<!-- Author: Laura -->
<!-- Comment: Tue Sep 25 13;07;39 comment by...Laura -->
No problems found during the 1 hour cosmic run after cables cut
<!-- Date: Tue Sep 25 09:06:51.620 2007 -->
PTM went out temporarily (global alarm showed red), but came back.
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 09:10:14.491 2007 -->
<pre>VOLTMAN is restarted and working, but showing MANY purple (unresponding) crates in the detector display.  It looks like I need to reset some other system, so I'll browse the Ace help pages until Steve arrives and see if I can figure anything out.</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 09:24:03.105 2007 -->
<pre>I just saw something important taped to the lower monitor on the HV monitor... a sign that says if we get heartbeat errors to check the "iFix background server" running on that computer.  There may be an error message dialog box that needs to be closed (there was, so I closed it).  That's all the sign said -- I tried that, and some of the heartbeat errors went away after SEVERAL minutes.  I guess it takes awhile, so note to other Aces to be patient if you need to do this step. 
 
Unfortunately CES,CMU,CMP,CMX,BMU are still showing pink heartbeats, and ICICLE is not working.  VOLTMAN still shows purple crates as well.</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 11;50;55 comment by...G.S. -->
Correction -- that sign below the monitor only applies to HBM.  But I did see a dialog box, so maybe the error was related.
<!-- Date: Tue Sep 25 09:40:48.690 2007 -->
<pre>Another error on Muon3 -- said "unallocateable memory" error.  I closed the dialog box and the program instantly closed, too.  Restarted it.  Let's see if CMU et al come back.</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 09:56:20.761 2007 -->
no surveyers yet - 
<!-- Author: Jean Slaughter -->


<!-- Date: Tue Sep 25 10:09:59.618 2007 -->
<pre>Steve is here, answered my questions.  Evidently ICICLE is not related to the heartbeat problems, but it probably IS related to the CDFONPROD DB upgrade, so I'll ignore it until our DAQ system is back up. 
 
We found that Muon3 is giving an error and we will try rebooting it to fix the memory issues.  This fix is in progress, so I may add more details to this post later if we discover any new problems.</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 10;25;54 comment by...G.S. -->
<pre>Before we rebooted, Steve noticed that Muon3 didn't have certain windows loaded that usually are loaded.  After the reboot, these windows (the iFix workspace for example) loaded just fine.  It seems things are on their way to getting better.

We also noticed a system scan initializing when we rebooted.  We conjecture that perhaps this automatic scan is what triggered the system to crap out in the first place.  We decided to allow the scan to continue, so hopefully it won't need to be run again anytime soon.</pre>

<!-- Comment: Tue Sep 25 10;30;19 comment by...G.S. -->
2 new errors from the Muon3 CAENSY127Detectors program running on the iFix workspace:

Error Number: -2147220484
Write Value
Data source is undefined: FIX32.CDFS2.TRGR_INHIBIT_CES_HV

Error Number: 1914
Vnode1 and 2 are down

Steve explained to me that these are tag errors and may usually be ignored.


<!-- Comment: Tue Sep 25 10;40;31 comment by...Steve Hahn -->
<p>Just a little clarification on what Glenn said above.  MUON3
was in bad shape before we rebooted.  Windows could not be 
displayed.  While trying to use the iFIX Workspace window we got
the error dialog message: "Error occurred in file 
d:\fix40\dynamic libraries\scripting\scriptingmgr.cpp" at line
11063.  VBA failed to initialize.  There may be a problem with
iFIX installation."  I think this is probably wrong; I
conjecture it was not an iFIX problem as such, but system 
resources being consumed.  When the PC was rebooted, it 
immediately started a Symantec AntiVirus scheduled system scan.
I suspect it had done this before rebooting, which in turn
caused the various problems we saw.</p>
<!-- Date: Tue Sep 25 10:30:49.453 2007 -->
Surveyers have arrived - 
<!-- Author: Jean Slaughter -->



<!-- Date: Tue Sep 25 10:46:32.267 2007 -->
<pre>I fixed the Firefox problem on b0desktop09.  It was giving an error about "Firefox already running, please close existing application or reboot" whenever we tried to open a browser.  I've seen this problem before -- firefox makes a dummy file when it loads, and occasionally this file is not delete upon exit (or if the computer is restarted with firefox running). 
 
I renamed the ".mozilla" directory to ".mozilla_old" in the home folder and tried to load Firefox, and it loaded fine.  Of course I had to then change all of the settings, add back in URLs, etc.</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 10;55;34 comment by...G.S. -->
I fixed the Firefox problem on the Consumer Operator main computer, b0control22
<!-- Date: Tue Sep 25 10:52:02.347 2007 -->
<p>I just rebooted SOLENOID2 to get up the correct solenoid 
picture.  Just for yucks, I tried restarting ICICLE.  To my 
surprise, it appears to be working, which would imply the 
CDFONPRD Oracle DB is available.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250123' target=_top>250123</a>) -->



<!-- Date: Tue Sep 25 11:02:30.795 2007 -->
<pre>Since the work on the database seems to be completed and ICICLE is back up, I thought I would re-enable ICICLE in ProcMon and try starting RunControl back up.  RunControl DID load properly, which it hadn't during the DB upgrade.  However, after I loaded the COSMICS mode and hit "partition", the program stayed in the "partition" state and will not procede to "Idle".  The only green crate in the partition is "errlog" and the rest are yellow, so it is unclear which crate is causing RunControl to stop. 
 
Even though the ICICLE is back up, the heartbeat remains purple on the Global Alarms window.  Perhaps we're still having a few communications problems?</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 11;12;32 comment by...G.S. -->
<pre>It's working now.  Steve stepped in the room and magic happened.  He didn't even have to do anything -- his mere presence fixed the problem.</pre>

<!-- Date: Tue Sep 25 11:14:38.130 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250124>250124</A>
 Activated at 2007.09.25 11:14:08
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 11:14:39.359 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250124>250124</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639]
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 11:19:01.006 2007 -->
<pre>Laura wants to test her board changes, so we're going to run this COSMICS run for at least an hour.  She might work on another crate after that, but otherwise I'll start my calibrations. 
 
Note to Aces: it took a very long time for RunControl to get through partitioning, config, and activate.  Jean suggested that perhaps things will run slower today since various upgrades.</pre>
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 12:08:09.344 2007 -->
<pre>GAM went red on the HV monitor, but it remains green on the Global Alarms window.  I think it may just be being read (just like the CEM,CHA,WHA flash off when being read).  As long as nothing else happens I won't worry about it.</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 12;08;54 comment by...G.S. -->
EVERYTHING JUST WENT RED -- all HV displays and all global alarms.  Something needs reset... but RTSERVER is ok for now... hmmm.

<!-- Comment: Tue Sep 25 12;13;22 comment by...Glenn Strycker -->
<pre>Someone is fooling around with the network boxes or Internet, because multiple computers in this room have Microsoft WinXP "network connection unplugged" boxes popping up.  This includes VNODE1, Muon3, CLC1.

Jean just came in and reported that there was a loose power cable.  JJ is working on it.</pre>

<!-- Comment: Tue Sep 25 12;20;43 comment by...jj -->
<b>I was connecting power cord for new nameserver ( fnsrvcdf2 )
and in process bumped loose power cord on s-cdf-online-2 which
I believe is network switch for all the iFix nodes.  Sorry
about that....</b>
<br>Note that I added the new nameserver to the Matrix UPS
that handles network equipment and the load went from 24.4%
to 28%.  

<!-- Comment: Tue Sep 25 12;23;55 comment by...jj -->
On Wednesday morning, DataComm will switch to a new physical
box for the nameserver fnsrvcdf2.  The new box is Linux while
the old box was Windows based.  The name and IP address will
remain the same so in principle the switch should be invisible.



<!-- Date: Tue Sep 25 12:29:15.048 2007 -->
<pre>We got an error on VOLTMON: 
 
An unhandled exception has occurred in your application... details: 
 
See the end of this message for details on invoking  
just-in-time (JIT) debugging instead of this dialog box. 
 
************** Exception Text ************** 
System.Runtime.InteropServices.COMException (0x800A0BE3): Disk or network error. 
   at dao.DBEngineClass.OpenDatabase(String Name, Object Options, Object ReadOnly, Object Connect) 
   at Project1.IOCode.DBWrite() 
   at Project1.Module1.ChannelCheckNEW(PowerSupplyA BoxName) 
   at Project1.IOCode.GetVolt(PowerSupplyA BoxElement, Int16& isend, Int16& ierror) 
   at Project1.Form1.Command13_Click(Object eventSender, EventArgs eventArgs) 
   at System.Windows.Forms.Control.OnClick(EventArgs e) 
   at System.Windows.Forms.Button.OnClick(EventArgs e) 
   at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent) 
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks) 
   at System.Windows.Forms.Control.WndProc(Message& m) 
   at System.Windows.Forms.ButtonBase.WndProc(Message& m) 
   at System.Windows.Forms.Button.WndProc(Message& m) 
   at System.Windows.Forms.ControlNativeWindow.OnMessage(Message& m) 
   at System.Windows.Forms.ControlNativeWindow.WndProc(Message& m) 
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam) 
. . . 
 
 
I won't quote the whole thing.  I'll try closing the application and reloading, and if that doesn't work we'll reboot. 
</pre>
<!-- Author: Glenn Strycker -->

<!-- Date: Tue Sep 25 12:39:14.855 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250124>250124</A>
 Terminated at 2007.09.25 12:38:57
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 12:42:47.840 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250124>250124</A>
TERMINATE: It's been an hour, so I'm ending this run.  Also, we just had ICICLE go down, so I want RunControl in start, not active.
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 12:58:59.299 2007 -->
<pre>I tried to restart the software on CLC1 since we had heartbeat alarms on all CLC systems.  Unfortunately I had to reboot, and worse yet, it didn't like a soft-reboot and I had to manually shut the computer off downstairs and power it back up. 
 
This didn't really fix anything -- the heartbeat problems are still there, as are the red global errors for PEM, PHA, and PSH.  HBM is also down.  At least GAM is back up, and the network connections are restored to all computers. 
 
I'm beginning to wonder if there is a particular order I should boot up systems after a big failure (like the network failure a half-hour ago).  The inter-dependence of the systems seems to require certain systems to be up before others.</pre>
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 13:14:20.070 2007 -->
<pre>Steve came and helped get rid of lots of errors, appearantly still caused by the power-cord/network failure a half-hour ago.  We restarted the Plug HV computer, restarted ICICLE, and clicked error messages closed on several systems.  Everything is now green except 
 
- PSM (known) 
- SVX (silicon related, so ignore) 
- ICICLE heartbeat is purple (I'm currently working on fixing that)</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 13:20:53.141 2007 -->
<pre>With all of the problems occurring today, I'm skipping calibrations for now.  I want to run L2_TORTURE for an hour until I'm positive that the system is stable -- plus I'm hoping that we can solve the ICICLE heartbeat in the meantime.</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 13:23:25.043 2007 -->
ICICLE heartbeat is back!  Yay!  I'll just do the L2_TORTURE for a half-hour, then start on calibrations.
<!-- Author: Glenn Strycker -->

<!-- Date: Tue Sep 25 13:26:02.701 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250125>250125</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,952,639] 
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 13:31:14.488 2007 -->
<pre>CRYO ALARM -- The LL-ADSBR is rapidly falling and approaching 0.  It should be 50%.  We called the Cryo guys, and they said someone is working on it, so it's safe to ignore.  I silenced the alarm.</pre>
<!-- Author: Glenn Strycker -->

<!-- Date: Tue Sep 25 13:42:04.557 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250125>250125</A>
TERMINATE: Laura came by and wants to change something with the crates, and then wants to test with COSMICS mode.
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 13:48:48.454 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250128>250128</A>
 Activated at 2007.09.25 13:47:01
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 13:48:49.004 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250128>250128</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 13:50:59.799 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250128>250128</A>
 Terminated at 2007.09.25 13:49:54
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 13:51:00.785 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250128>250128</A>
TERMINATE: end 
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 13:52:22.784 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250130>250130</A>
 Activated at 2007.09.25 13:52:14
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 13:53:34.590 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250130>250130</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] no L3
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 13:53:34.878 2007 -->
In order to accommodate the surveyers, steve Hahn is unplugging B0 WCAL7 = C0NEB-3
<!-- Author: slaughter :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250130' target=_top>250130</a>) -->
<!-- Comment: Tue Sep 25 13;55;48 comment by...G.S. -->
We removed the crate from our COSMICS runs and restarted the run.

<!-- Comment: Tue Sep 25 14;36;14 comment by...Phil -->
I don't understand this. Someone clarify it.
<!-- Comment: Wed Sep 26 08;40;41 comment by...Steve Hahn -->
OK, Phil.  Turns out the plug into the 400 Hz power strip
located in the LNE corner of the wall blocked the mounting of
the surveyor's extension on top of the CDF elevation target in
that corner.  The plug had to be moved over to the adjacent
receptacle; this was the least invasive method I could concoct.
<!-- Date: Tue Sep 25 13:59:20.096 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250131>250131</A>
 Activated at 2007.09.25 13:59:14
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 13:59:59.390 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250131>250131</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] software event builder
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 14:01:27.014 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250131>250131</A>
 Terminated at 2007.09.25 14:01:15
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 14:19:30.507 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250133>250133</A>
 Activated at 2007.09.25 14:19:08
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 14:19:31.520 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250133>250133</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 14:26:34.776 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250133>250133</A>
 Terminated at 2007.09.25 14:26:16
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 14:26:49.427 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250133>250133</A>
TERMINATE: Eiko wants the COT crates, so I'm ending the run and restarting COSMICS mode without them.
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 14:27:32.645 2007 -->
Eiko is taking the COT crates and the COT pulser crate to do a special calibration from her office.
<!-- Author: slaughter :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250133' target=_top>250133</a>) -->

<!-- Date: Tue Sep 25 14:31:32.367 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250135>250135</A>
ACTIVATE: RunSet: COT_ACE 
<!-- Author: Eiko X 4953 -->


<!-- Date: Tue Sep 25 14:31:33.681 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250134>250134</A>
 Activated at 2007.09.25 14:30:58
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 14:31:34.750 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250134>250134</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 14:31:35.797 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250135>250135</A>
TERMINATE: Check of COT ASDQ after turning off and on LV

<!-- Author: Eiko X 4953 -->



<!-- Date: Tue Sep 25 14:36:37.229 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250136>250136</A>
ACTIVATE: RunSet: COTCALIB_ACE after the LV off and on this morning

<!-- Author: Eiko X 4953 -->


<!-- Date: Tue Sep 25 14:37:33.826 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250136>250136</A>
CALIBRATION: <U><B>cot</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cot target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->



<!-- Date: Tue Sep 25 14:40:08.799 2007 -->
<pre>Performed ASDQ and COT calibrations since this morning Bob had  
turned on and off LV power supplies for the ASDQ. Two cells which  
were hot yesterday (SL7 160,161) have become quieter (occupancy  
1.03 and 1.09). The number of bad channels in COT calibration also went down to 402 now. </pre>
<!-- Author: Shin-Shan Yu (Eiko) -->


<!-- Date: Tue Sep 25 14:54:55.467 2007 -->
Shift summary 
<UL> 
<li> CDFONPROD database got updated  
<li>Laura did L2 work 
<li>Eiko did special COT calibrations 
<li> the surveyers are here 
<li> we had two film crew tours 
<li> Bill Noe did SUVA work for the COT 
<li> A power cord was accidentally knocked for the network switch for theipix computers which caused problems to get everything back  
<li> a power cord is unplugged for B0WCAL7 for the surveyers 
</ul>
<!-- Author: slaughter :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250134' target=_top>250134</a>) -->


<!-- Date: Tue Sep 25 15:11:50.323 2007 -->
Phil and Francesco are taking over.
<!-- Author: Phil -->


<!-- Date: Tue Sep 25 15:18:18.807 2007 -->
Surveyoring done for the day. (Or at least Peter says the hall is empty.) They said they removed the duct tape from the tunnel door.
<!-- Author: Phil -->
<!-- Comment: Tue Sep 25 15;19;41 comment by...Phil -->
Muon scintillator are going in.

<!-- Comment: Wed Sep 26 07;51;57 comment by...dee -->
Dectector elevation survey was finished.
<!-- Date: Tue Sep 25 15:30:55.398 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250134>250134</A>
 Terminated at 2007.09.25 15:30:45
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 15:32:28.928 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250134>250134</A>
TERMINATE: done; tests L2 Decision Node
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 15:37:41.580 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250147>250147</A>
 Activated at 2007.09.25 15:34:37
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 15:37:42.300 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250147>250147</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] L2 Decision node testing
<!-- Author: Glenn Strycker -->
<!-- Comment: Tue Sep 25 15;41;51 comment by...Francesco -->
Of course it's not Glenn, it's me that I forgot to change the client name.
<!-- Date: Tue Sep 25 15:37:43.749 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250147>250147</A>
 Terminated at 2007.09.25 15:36:42
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 15:37:44.749 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250147>250147</A>
TERMINATE: end 
<!-- Author: Glenn Strycker -->


<!-- Date: Tue Sep 25 15:40:13.550 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250149>250149</A>
 Activated at 2007.09.25 15:40:01
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 15:40:14.866 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250149>250149</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->



<!-- Date: Tue Sep 25 15:41:40.220 2007 -->
Gene and Laura are doing some L2 testing.
<!-- Author: Phil -->


<!-- Date: Tue Sep 25 15:42:56.182 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250149>250149</A>
 Terminated at 2007.09.25 15:42:37
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 15:55:24.055 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:00</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Sep 25 16:02:58.201 2007 -->
Gamma node pc was unreachable for a minute or so. Now the values are back in the Gamma window and the global allarm is off, but in the HV summary it's still red.
<!-- Author: Francesco -->

<!-- Date: Tue Sep 25 16:05:04.698 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250160>250160</A>
 Activated at 2007.09.25 16:04:30
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:05:05.939 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250160>250160</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Francesco Crescioli -->


<!-- Date: Tue Sep 25 16:12:28.323 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250160>250160</A>
 Terminated at 2007.09.25 16:06:21
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:12:29.284 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250160>250160</A>
TERMINATE: more testing
<!-- Author: Francesco Crescioli -->


<!-- Date: Tue Sep 25 16:12:30.024 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250161>250161</A>
 Activated at 2007.09.25 16:08:43
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:12:31.114 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250161>250161</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] l2 test
<!-- Author: Francesco Crescioli -->


<!-- Date: Tue Sep 25 16:12:32.714 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250161>250161</A>
 Terminated at 2007.09.25 16:09:10
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:12:33.152 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250161>250161</A>
TERMINATE: more tests
<!-- Author: Francesco Crescioli -->


<!-- Date: Tue Sep 25 16:12:34.329 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250162>250162</A>
TERMINATE: restart rc

<!-- Author: Francesco Crescioli -->


<!-- Date: Tue Sep 25 16:14:35.667 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250163>250163</A>
 Activated at 2007.09.25 16:13:52
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:15:06.518 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250163>250163</A>
 Terminated at 2007.09.25 16:14:47
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:20:14.303 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250164>250164</A>
 Activated at 2007.09.25 16:19:46
<!-- Author: RunControl -->


<!-- Date: Tue Sep 25 16:20:16.447 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250164>250164</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] tests l2
<!-- Author: Francesco Crescioli -->



<!-- Date: Tue Sep 25 16:32:56.015 2007 -->
See Jean's summary above <font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Phil Schlabach -->