<!-- Title: Day shift. Tue Sep 30, 2003 -->
<!-- SciCo: R. Blair -->
<!-- DAQAce: Monica D'Onofrio -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: &nbsp; -->
<!-- Notes: -->
<pre>Water is still off - real recovery probably awaits later today when all the electronics can be powered up.</pre>


<!-- Date: Tue Sep 30 08:37:31.313 2003 -->
Dervin says the "first four" relay racks downstairs are powered up.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 08:39:36.560 2003 -->
Monica is on call (she was not feeling that well).  There is not a lot for an Ace to do just now, but we may have to rethink this later in the shift.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 08:54:27.120 2003 -->
We tried to run consumers from CSL partition 14. 
 
However, we guess CSL is still off. At least that is what 
Procmon shows. So we wait until things stabilize more. 
 
We had to bring up b0dap51 from which we run the consumers. 
b0dap54 was up already. 
  
<!-- Author: Wolfgang and Kohei -->
<!-- Comment: Tue Sep 30 09;51;59 comment by...Wolfgang -->
I talked to Tony Vaiciulis. He said they will start the CSL
up in the late afternoon.
<!-- Date: Tue Sep 30 08:55:33.675 2003 -->
Steve Hahn is turning on the motor generators and will proceed to turn on the racks in the hall.
<!-- Author: R. Blair -->

<!-- Date: Tue Sep 30 09:08:35.642 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169937">169937</A>
TERMINATE: testing trigger - fred and scalers seemto be asserting error.
<!-- Author: greg 2080 -->


<!-- Date: Tue Sep 30 09:15:21.054 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169938">169938</A>
TERMINATE: still errors
<!-- Author: greg 2080 -->



<!-- Date: Tue Sep 30 09:38:39.038 2003 -->
<pre>It seems the cards are not being configured, 
Both crates are asserting DONE and ERRROR. 
Also, can't seem to read the TRACER with the GUI. 
Pagng DAQ.</pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169946' target=_top>169946</a>) -->


<!-- Date: Tue Sep 30 09:42:56.488 2003 -->
Greg is having trouble getting RC going - page DAQ 
<!-- Author: R. Blair -->

<!-- Date: Tue Sep 30 09:47:27.959 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169948">169948</A>
ACTIVATE: calib continuos with no crates - working!
<!-- Author: greg 2080 -->



<!-- Date: Tue Sep 30 09:47:29.593 2003 -->
Bill Badgett is coming in.  We are starting with only two crates to bootstrap things for when the trigger room power supplies are burned in.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 09:54:44.523 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169950">169950</A>
ACTIVATE: calib continuos with cot
<!-- Author: greg 2080 -->


<!-- Date: Tue Sep 30 09:56:33.719 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169950">169950</A>
TERMINATE: done timeout every  1 events.
<!-- Author: greg 2080 -->



<!-- Date: Tue Sep 30 09:56:34.320 2003 -->
<pre>Frank Chlebana reported that /data1 was filled up to 100%. 
It turns out, the cron job runing on b0dap30 which copies 
the files off /data1 disappeared after the system was upgraded 
(of course) and we forgot to restart it again. 
 
I have now started it manually and will later submit it to 
cron.</pre>
<!-- Author: Wolfgang -->


<!-- Date: Tue Sep 30 09:56:36.642 2003 -->
Muon HV consern: 
 
Muon3 down in the counting room does not appear to be operational as far as getting HVMON++ Ground Control with iFIX to run as expected.  The problem is almost certainly related to the power-outage.  Experts will be notified.
<!-- Author: Dale Stentz -->
<!-- Comment: Tue Sep 30 11;13;15 comment by...Phil Schlabach -->
HVMON++ looks fine now.
<!-- Date: Tue Sep 30 09:56:54.766 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169951">169951</A>
ACTIVATE: trying again
<!-- Author: greg 2080 -->


<!-- Date: Tue Sep 30 09:58:13.161 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169951">169951</A>
TERMINATE: DTO every 4 calib events (or every one L2A).
<!-- Author: greg 2080 -->



<!-- Date: Tue Sep 30 10:12:45.513 2003 -->
B0dau30 is having disk problems - this affects user areas on the local systems.  The disk is being restored.
<!-- Author: R. Blair -->

<!-- Date: Tue Sep 30 10:55:09.628 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->
<!-- Comment: Tue Sep 30 11;02;26 comment by...R. Blair -->
This appears to have been a consequence of the reboot.
<!-- Date: Tue Sep 30 11:01:32.767 2003 -->
Disk has been fixed in b0dau30 - web server has been rebooted.
<!-- Author: R. Blair -->



<!-- Date: Tue Sep 30 11:03:28.466 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030>b0dau30 has lost a disk, was down few times during last hour 
It's up now</td></tr></td></table>
<!-- Author: Sergei -->


<!-- Date: Tue Sep 30 11:04:18.297 2003 -->
cdf online web-server was rebooted
<!-- Author: Sergei -->
<!-- Comment: Tue Sep 30 11;14;26 comment by...Phil Schlabach -->
HVMON++ is running. All MUON chamber HV set to "OFF" as I don't know what the individual detectors want.


<!-- Comment: Tue Sep 30 11;16;20 comment by...ps -->
oops. I wanted an entry not a comment.
<!-- Date: Tue Sep 30 11:15:29.116 2003 -->
HVMON++ is running. All MUON chamber HV set to "OFF" as I don't know what the individual detectors want. 
<!-- Author: Phil Schlabach -->
<!-- Comment: Tue Sep 30 11;59;03 comment by...hyunsoo -->
CMU off till further notice.
<!-- Date: Tue Sep 30 11:22:51.587 2003 -->
<p>Took a while, but all 400 Hz and 60 Hz is on to the collision 
hall racks except for SVX and alcove racks.  First, found the  
400 Hz breakers off (not tripped) in PHP-B0-2 in the pit.  Then 
found various breakers off in B0-400-4 and B0-400-6B in the  
collision hall itself.  Presumably, these are remnants of the 
breaker panel work done yesterday by the electricians.  Lastly, 
I confused myself by turning off the output breakers of the  
MGs and then forgetting I'd done this.  Now, all power is back 
on including the SE arch.</p> 
 
<p>Also, Del Allspach is working on restoring argon-CO2 for  
the CPR work Karen Byrum and Steve Kuhlmann are doing on 
the interior of the SE arch.</p>
<!-- Author: Steve Hahn -->


<!-- Date: Tue Sep 30 11:42:10.534 2003 -->
<pre>Trying to reproduce Greg's problems. 
 
No sucess so far. 
</pre>
<!-- Author: W.Badgett -->



<!-- Date: Tue Sep 30 13:19:24.029 2003 -->
The chilled water system is back online.  There is no problem with powering crates up now.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 13:21:52.963 2003 -->
The HVAC work on the third floor is done - we are now ready to bring L3 and the EVB up.  Rob tells me that this is happening and should be done soon. 
 
The power supplies are almost all swapped in the trigger room.    In about an hour Greg will come in to plug the cards back in.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 13:29:58.256 2003 -->
<pre>Level3 will stay off untill tomorrow  
when all air conditioning problems are fixed </pre>
<!-- Author: Arkadiy -->
<!-- Comment: Tue Sep 30 14;12;37 comment by...R. Blair -->
Rob Roser says Arkadiy needs to talk to Guillelmo - the work is done
<!-- Date: Tue Sep 30 13:45:33.439 2003 -->
Doug Glenzinski called to say they are turning on a pump that on occassion when turned on trips the vesda system.  He can be reached at 6787.  We hope it won't happen this time...  
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 14:02:47.308 2003 -->
JC is restarting iFix.
<!-- Author: R.Blair -->



<!-- Date: Tue Sep 30 14:28:08.324 2003 -->
deionizer bottles alarm went off in iFix for about 30 sec.  We left a message on Rob's cell.  It healed itself it was low and then went back up to 17.6 well above th 12 limit.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 14:29:10.540 2003 -->
Greg is going to be reseating the trigger cards so PS alarms will likely be a ringing.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 14:44:57.685 2003 -->
<pre>CPR/CCR ACCESS REPORT: 
 
Working on gas flow problems on CCR on southeast arch, which 
have caused 4 dead chambers.  We have found the blocked places,  
and have flowed gas thru 3 of the 4 chambers.  The 4th, 12E,  
seems more difficult.  We're going to talk to other experts 
and consult examples before trying again Weds. </pre>
<!-- Author: Karen Byrum, Steve Kuhlmann -->


<!-- Date: Tue Sep 30 15:03:14.056 2003 -->
Mike L. calls from the hall and instructs us to call cryo and have them disable the vesda so the pump (which is putting out oil) does not trip it.  We call cryo and they suggest we try Bill Noie at his office - I do and then try his SRP and LRP - no luck.  I move on to Stefano.  Eventually Dervin and Dee explain that we are responsible for VESDA and Dee disables it (note she/we have to call security dispatch and there is a signoff procedure).  We are now responsible to follow through and re-enable'ing when the work that causes the problem is completed.  Dee left a voicemail on Mike's phone reminding him.  The next shift likely will need to finish this up (check that it gets re-enabled).  
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 15:04:42.651 2003 -->
We get yet another deionizer bottles water error.  Again it goes away after a short stint below the limit.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 15:07:54.834 2003 -->
Rob Roser breezes by.  Deionizer bottle is a known problem - they are working on it.
<!-- Author: R. Blair -->


<!-- Date: Tue Sep 30 15:30:04.660 2003 -->
Water alarm - this time pump discharge pressure went down to 20 psi (trip at 40).  After a minute it goes back up to 207 psi.  I tried to get Rob since he was the most likely conspirator.  I fail to get him via pager or cell.
<!-- Author: R. Blair -->

<!-- Date: Tue Sep 30 15:55:08.943 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Sep 30 16:03:11.709 2003 -->
<pre>b0imu00 gives lots of VME bus errors. 
 
Please fix. 
</pre>
<!-- Author: W.Badgett, G.Feild -->

<!-- Date: Tue Sep 30 16:04:42.644 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169960">169960</A>
ACTIVATE: running without L2 to check the other trigger cards
<!-- Author: Peter Wittich x2080 -->



<!-- Date: Tue Sep 30 16:05:35.704 2003 -->
<font color=red>Shift Summary: </font><pre>Most things back up.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   NA
Delivered Luminosity     
Acquired Luminosity      
Efficiency             
   
</td></tr></table></pre>
<!-- Author: R. Blair -->

<!-- Date: Tue Sep 30 16:06:53.668 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=169961">169961</A>
ACTIVATE: argh!
<!-- Author: Peter Wittich x2080 -->