<!-- Title: Owl shift. Thu Nov  6, 2003 -->
<!-- SciCo: Giovanni Punzi -->
<!-- DAQAce: Helen Hayward -->
<!-- MonAce: Whitehouse/Pronko -->
<!-- CO: &nbsp; -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>No beam. 
 
Plan: 
-Complete 2 hours of High-rate cosmics (no Si) 
-L2 torture for 3hours (no Si) 
-Then Cosmics with everything, including Si. 
 
Notes on Alarms:  
 
ODH and Flam. Gas alarms disabled: cryo tech downstairs watching  monitor: 
 will announce an alarm over the intercom if necessary  
 
RTserver intermittantly glitches: if  Silicon Cooling Alarm  
      sounds, hit Silicon Rack Power Crash Button  
 
</pre>


<!-- Date: Thu Nov 6 00:29:08.719 2003 -->
The PSM on the Global alarm is red.  It is due to crate b0fib00.  According to <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=172&frame=2&anchor=15:43:49'> 
entry in silicon log </a>, experts turned the power off for this crate at about 3-4pm today.
<!-- Author: Helen -->


<!-- Date: Thu Nov 6 00:44:05.747 2003 -->
b0tsi03 is red in VxWorks.  Previous Ace reported it was red all his shift.
<!-- Author: Helen -->
<!-- Comment: Thu Nov  6 01;48;27 comment by...Helen -->
WE reset this crate and it is now green again
<!-- Date: Thu Nov 6 00:45:03.792 2003 -->
We noticed b0dau35.fnal.gov - disk has been signaling red in Big Brother roughly every 15 minutes since about 3pm.  The red state lasts approx. 5 minutes.  It looks like the /usr volume is full.  We'll keep an eye on things...
<!-- Author: Ben -->


<!-- Date: Thu Nov 6 01:08:50.328 2003 -->
L3 nodes are green all the time. L3 experts asked to look at CSL rate. The CSL rate is fluctuating from 16MB/s up to 24MB/s (coming from stream "i"). 
<!-- Author: Alex :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173350' target=_top>173350</a>) -->

<!-- Date: Thu Nov 6 01:24:10.766 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173350">173350</A>
 Terminated at 2003.11.06 01:21:51
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 01:24:11.826 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173350">173350</A>
TERMINATE: Ending Cosmics run to start a L2_TORTURE
<!-- Author: Ben W x2080 -->



<!-- Date: Thu Nov 6 01:24:12.567 2003 -->
<pre>End High-rate cosmics and switch to L2-torture. 
L3 experts will be fiddling with L3 nodes during this run. 
We will watch and see what happens...</pre>
<!-- Author: Giovanni -->
<!-- Comment: Thu Nov  6 01;55;52 comment by...Alberto -->
L3 night test. Today, we tested a new version of the L3 Display
(and tomorrow morning a new Ace Control Panel For L3/EVB will be 
available). I used node 77 to test how the system reports errors
that I purposely produced. The debut of the new and improved Display is put off a couple days...
There shouldn't be any problem for the shift crew, as a consequence of my test. Node 77 is working and all the processes I 
worked with will restart with the next run. 
<!-- Date: Thu Nov 6 01:31:47.113 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-11899&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-11899  width='300' align='top'></td><td>YMON #82 TOF Summary</a></td></tr></table>
<!-- Author: Ben -->
<!-- Comment: Thu Nov  6 01;33;03 comment by...Ben -->
<pre>TOF Summary for Cosmics Run 173350 per request</pre>

<!-- Date: Thu Nov 6 01:34:30.153 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351">173351</A>
ACTIVATE: Start L2TORTURE[13,282,370] w/o silicon and fcal00
<!-- Author: Ben W x2080 -->





<!-- Date: Thu Nov 6 02:03:19.897 2003 -->
<pre>L3 experts have finished with their tests.</pre>
<!-- Author: Giovanni :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351' target=_top>173351</a>) -->


<!-- Date: Thu Nov 6 02:29:52.084 2003 -->
we have seen 2 timeouts from CCAL_11 during this run.  Auto HHR seems to pasify it, but we'll watch for other trouble...
<!-- Author: Ben :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351' target=_top>173351</a>) -->
<!-- Comment: Thu Nov  6 02;55;53 comment by...Ben -->
3 times

<!-- Comment: Thu Nov  6 03;57;26 comment by...Helen -->
5 times:  Always slot 4?<br>
<br>
From error logger:<br>
<br>
(MLE) b0ccal11:Messenger:1:41:00 AM->SMXR Not Responding event 147007, slot 4, bunch ctr 86, L2buffer 0, header 0x3bfe456<br>
<br>
This causes a done timeout error, which is always solved by an auto-HRR.<br>
<br>
There appeared to be an abnormal number of entries in the error logger concerning b0svt06.  We rebooted/reset/shepherded this crate and it appears to have solved this.<br>
<br>
(MLE) b0svt06:Messenger:1:41:18 AM->FER_E_SVT, GB 13.0 nw=29 (mod7=1) ee=600747 event=147646<br>
(MLE) b0svt06:Messenger:1:42:05 AM->FER_E_SVT, GB 13.0 nw=29 (mod7=1) ee=600744 event=160685<br>
(MLE) b0svt06:Messenger:1:42:27 AM->FER_E_SVT, GB 13.0 nw=29 (mod7=1) ee=60074a event=166811<br>



<!-- Date: Thu Nov 6 04:02:56.539 2003 -->
How bizarre!  The AcceleratorStatus panel in rc has gone blank!  This is a new one on me!
<!-- Author: Helen :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351' target=_top>173351</a>) -->


<!-- Date: Thu Nov 6 04:06:03.179 2003 -->
Another done timout for CCAL_11.  Again solved by auto-HRR:<br 
<br> 
(MLE) b0ccal11:Messenger:4:02:14 AM->SMXR Not Responding event 2458361, slot 4, bunch ctr 83, L2buffer 0, header 0x3bfe453<br> 
<br> 
  
<!-- Author: Helen :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351' target=_top>173351</a>) -->


<!-- Date: Thu Nov 6 04:31:53.047 2003 -->
<pre>The goodrun GUI window freezes when I hit the "search" button. 
 It then stays blank until I kill it. 
 
This has happened on occasions in the past few days, but now it 
 seems to have turned into a permanent problem. Killing all 
 processes and restarting does not restore normal behavior.</pre>
<!-- Author: Giovanni -->


<!-- Date: Thu Nov 6 04:41:09.871 2003 -->
(MLE) b0musc00:Messenger:4:34:18 AM->VISIONread/write failed in slot 10<br> 
<br> 
Had the above error with b0musc00.  Had to reboot/reset/shepherd twice to solve this one.<br>
<!-- Author: Helen :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351' target=_top>173351</a>) -->

<!-- Date: Thu Nov 6 04:50:06.998 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351">173351</A>
TERMINATE: End run to begin cosmics with silicon
<!-- Author: Ben W x2080 -->



<!-- Date: Thu Nov 6 04:51:03.849 2003 -->
I've had 2 more auto-HRR due to done-timouts with CCAL_11.  The problem with b0musc00 also reoccurred.  And again a reboot/reset/shepherd was required.
<!-- Author: Helen :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173351' target=_top>173351</a>) -->

<!-- Date: Thu Nov 6 05:30:02.851 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173353">173353</A>
 Activated at 2003.11.06 05:27:01
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 05:30:04.587 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173353">173353</A>
 Terminated at 2003.11.06 05:27:21
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 05:41:05.291 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173355">173355</A>
TERMINATE: b0fib02 does not want to configure
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 05:44:23.713 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173356">173356</A>
 Activated at 2003.11.06 05:42:31
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 05:46:24.225 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173356">173356</A>
ACTIVATE: begin COSMIC.SILICON.TAPE
<!-- Author: Helen x2080 -->
<!-- Comment: Thu Nov  6 05;48;45 comment by...Helen -->
with b0fcal00, b0fib00 (power off), and b0fib02(does not config) removed.
<!-- Date: Thu Nov 6 05:46:25.531 2003 -->
When i first tried to start a COSMIC.SILICON.TAPE run, i got a message from run control telling me that the CONSUMER MONITOR PROXY was running and i should start it.  Someone had greyed out both the CONSUMER MONITOR PROXY and DBBROKER PROXY in ProcMon.  I enabled these in ProcMon and they turned red.  After restarting both these processes they now show green.  I had to restart runcontrol to stop the message appearing.
<!-- Author: Helen -->

<!-- Date: Thu Nov 6 05:46:54.010 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173356">173356</A>
 Terminated at 2003.11.06 05:45:24
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 05:47:57.654 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173356">173356</A>
TERMINATE: end run due to no events being logged.  due to missing b0fib00 and b0fib02?
<!-- Author: Helen x2080 -->



<!-- Date: Thu Nov 6 05:50:55.451 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173357">173357</A>
 Activated at 2003.11.06 05:49:17
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 05:50:56.872 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173357">173357</A>
ACTIVATE: try COSMIC.SILICON.TAPE again.
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 05:53:07.226 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173357">173357</A>
 Terminated at 2003.11.06 05:50:19
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 05:53:09.904 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173357">173357</A>
TERMINATE: still no events
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 05:59:34.890 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173358">173358</A>
TERMINATE: tried including b0fib02 again..... no success
<!-- Author: Helen x2080 -->
<!-- Comment: Thu Nov  6 06;35;51 comment by...Helen -->
I have tried both reboot/reset/shepherd and using vxlogin->reboot with no success.
<!-- Comment: Thu Nov  6 08;35;50 comment by...SCN -->
From the error logger:
<pre>
(MLE) b0fib02:Messenger:5:54:07 AM->Error Initializing HDI Slot 13 Chan 0: SVX B1W6
(MLE) b0fib02:Messenger:5:54:07 AM->Error Initializing HDI Slot 13 Chan 1: SVX B1W6
(MLE) b0fib02:Messenger:5:54:08 AM->Error Initializing HDI Slot 13 Chan 2: SVX B1W6
(MLE) b0fib02:Messenger:5:54:08 AM->Error Initializing HDI Slot 13 Chan 3: SVX B1W6
(MLE) b0fib02:Messenger:5:54:08 AM->Statechange failed
(MLE) b0fib02:Messenger:5:55:21 AM->Error Initializing HDI Slot 13 Chan 0: SVX B1W6
(MLE) b0fib02:Messenger:5:55:21 AM->Error Initializing HDI Slot 13 Chan 1: SVX B1W6
(MLE) b0fib02:Messenger:5:55:21 AM->Error Initializing HDI Slot 13 Chan 2: SVX B1W6
(MLE) b0fib02:Messenger:5:55:21 AM->Error Initializing HDI Slot 13 Chan 3: SVX B1W6
</pre>
This suggests that this wedge tripped off, or was left off after
some work done on it last night.  Rebooting the crate never helps turn a wedge back on.
<!-- Date: Thu Nov 6 06:01:44.123 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173359">173359</A>
 Activated at 2003.11.06 06:00:04
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:01:45.229 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173359">173359</A>
ACTIVATE: trying an ACE_COSMICS_ALL
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 06:10:11.619 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173359">173359</A>
 Terminated at 2003.11.06 06:07:16
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:10:12.313 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173359">173359</A>
TERMINATE: Run Control Lies!!!!!  According to rc, this run has No, L2 or L3 events! dispite L3 processing, stuff, the trigger rate monitor showing events and even the event display is showing stuff!
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 06:13:50.835 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173360">173360</A>
 Activated at 2003.11.06 06:12:31
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:14:29.781 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173360">173360</A>
ACTIVATE: I restarted rc, ACE_COSMICS_ALL, and this time rc is telling the truth.
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 06:14:30.603 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173360">173360</A>
 Terminated at 2003.11.06 06:13:15
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:15:59.690 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173360">173360</A>
TERMINATE: end run to try cosmic with silicon again
<!-- Author: Helen x2080 -->



<!-- Date: Thu Nov 6 06:19:30.275 2003 -->
<pre>Turning off CMX HV for wedge 6-7 west.  The reason for this is that CMX workers yesterday found a signal cable when removed and with 6-7 not on caused oscilations* to stop.  (When put back osciliations in wedges 10 and 11 came back.)  Thus it be good to see the effect of turning off the HV the double check this.  (Note of course that the signal cable which is in wedge 10 that is suspected to cause oscilations is still removed.)  Further details to follow in the CMX e-log. 
 
osciliations = hot spots or super large number of hits in the CMX.</pre>
<!-- Author: Dale Stentz -->
<!-- Comment: Thu Nov  6 06;27;11 comment by...Dale Stentz -->
<pre>my bad.  west wedges 7-8 are turned off.</pre>
<!-- Date: Thu Nov 6 06:22:47.350 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173362">173362</A>
 Activated at 2003.11.06 06:21:08
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:22:52.952 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173362">173362</A>
ACTIVATE: COSMICS.SILICON.TAPE (fcal00, b0fib00, b0fib02 removed)
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 06:24:58.582 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173362">173362</A>
 Terminated at 2003.11.06 06:23:17
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:25:00.562 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173362">173362</A>
TERMINATE: genuinely no events! ->end run
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 06:27:11.484 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173363">173363</A>
 Activated at 2003.11.06 06:25:54
<!-- Author: RunControl -->



<!-- Date: Thu Nov 6 06:30:58.260 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173363">173363</A>
 Terminated at 2003.11.06 06:28:16
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 06:30:59.766 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173363">173363</A>
ACTIVATE: cosmics with silicon
<!-- Author: Helen x2080 -->


<!-- Date: Thu Nov 6 06:31:00.488 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173363">173363</A>
TERMINATE: still nothing triggers
<!-- Author: Helen x2080 -->
<!-- Comment: Thu Nov  6 07;23;48 comment by...Helen -->
In RunControl "InhibitStatus Set" is apparant, however no red lights are present in the trigger inhibit board.
<!-- Date: Thu Nov 6 06:33:29.136 2003 -->
WE are giving up with Comics with silicon..... nothing triggers.  In order to configure the crates going you have to remove b0fib00 and b0fib02..... But then when the run is going...... nothing happens!<br> 
<br>We shall return to L2 Torture, without silicon and without fcal00
<!-- Author: Helen -->

<!-- Date: Thu Nov 6 06:34:44.582 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173364">173364</A>
ACTIVATE: L2_TORTURE with fcal00 removed.
<!-- Author: Helen x2080 -->




<!-- Date: Thu Nov 6 06:49:14.948 2003 -->
@ 6:38 I turned on CMX 7-8 west but turned off 9-10.   
 
It seems that in both cases there are strong osciliation problems still in the CMX top SW.  CMX experts will tackel the problem later today.
<!-- Author: Dale Stentz :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173364' target=_top>173364</a>) -->
<!-- Comment: Thu Nov  6 07;27;36 comment by...Dale Stentz -->
All of CMX HV is up (started @ 7:10).  Detials about CMX work durning access can be found <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=5&frame=2&anchor=07:24:51'>
here in the CMX e-log. </a>
<!-- Comment: Thu Nov  6 07;39;33 comment by...Ingin Zaw, John Strologas, Dale Stentz -->
<pre>One detail that we should note is that wedge 21 on the NW CMX triped yesterday in the afternoon (about 4:30-5:30 timeframe).  It came back up but drew a large amount of current.  While inspecting it we "heard" and saw the problem.  The bottom (inside facing the CMP wall) cover was smashed in and was making contact with the anode pins on the bottom of the chambers.  We fixed it and also put on plastic protectors to help protect the pins and prevent the problem from causing a trip again.  

Needless the say CMXers encourages everyone to be careful and mindful of other detectors.  As the support for the covers are not very strong it is very possible that a person pushed it in while while moving through the crack between the CMP wall and the arch and didn't notice it.</pre>
<!-- Date: Thu Nov 6 07:11:12.693 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173364">173364</A>
TERMINATE: end run 
<!-- Author: Helen x2080 -->



<!-- Date: Thu Nov 6 07:14:37.146 2003 -->
b0dau35 is still havng memory problems.<a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=927&frame=2&anchor=00:45:03.792'> 
noticed earlier here </a>  
<!-- Author: Helen -->

<!-- Date: Thu Nov 6 07:16:45.942 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173365">173365</A>
 Activated at 2003.11.06 07:15:12
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 07:22:14.250 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173365">173365</A>
 Terminated at 2003.11.06 07:20:43
<!-- Author: RunControl -->




<!-- Date: Thu Nov 6 07:30:05.593 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173366">173366</A>
ACTIVATE: L2_Torture without fcal_00
<!-- Author: Helen x2080 -->




<!-- Date: Thu Nov 6 07:49:10.481 2003 -->
Most processes in ProcMon turned red.  They all turned green again after a few minutes.  I do not know whether this is connected to problems with the rtserver.... but run control never lost connection with the rtserver.... or at least no error messages came up in the xterm associated with the runcontrol.
<!-- Author: Helen -->
<!-- Comment: Thu Nov  6 10;21;45 comment by...Doug -->
<pre>RTserver has not glitched or rebooted since ~1600 yesterday,
so this problem was not caused by RTserver.</pre>
<!-- Date: Thu Nov 6 07:56:29.446 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Nov 6 08:01:26.098 2003 -->
<font color=red>Shift Summary: </font><pre>No beam. 
 
-Completed a 2h Hi-rate Cosmics run. 
-Took 3h L2_torture. 
  L3 expert did a brief test of node monitoring during it. 
-Tried to run cosmics with Silicon but failed, revert to L2_torture at 6:30. 
 
- CCAL_11 slot4 timeout every ~1/2 hour when in L2_torture, recovered by auto-HRR 
 
- b0fib02 does not configure, had to remove it. b0fib00 and b0fcal00 are off. 
 
- Trying to run cosmics w/Si yields zero rate. Cosmics w/o Si works. 
 
- Strange behaviors from RC display:  
reports zero events while it is not true, accelerator status panel is blank. 
 
- goodrun GUI does not work: window freezes when hitting search button. 
 
- Dale tried to get rid of CMX oscillation by turning off and on various wedges 
 but could not fix the problem. 
 
 
</pre> 
<!-- Author: Giovanni -->



<!-- Date: Thu Nov 6 08:36:24.240 2003 -->
<pre>The three nodes in the new rack have been up since we moved  
them to a different power circuit.  
 
dap78 uptime 
  8:32am  up 16:31,  4 users,  load average: 0.37, 0.31, 0.28 
dap33 uptime 
  8:33am  up 16:30,  1 user,  load average: 0.13, 0.14, 0.10 
dap34 uptime 
  8:33am  up 16:31,  6 users,  load average: 0.00, 0.00, 0.00 
 
</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Frank Chlebana -->

<!-- Date: Thu Nov 6 08:36:50.692 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173366">173366</A>
TERMINATE: end L2 torture so L3 expert can test ACE control panel
<!-- Author: Andrei/Jeff/Irina x2080 -->


<!-- Date: Thu Nov 6 08:59:05.350 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173367">173367</A>
 Activated at 2003.11.06 08:57:20
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 08:59:06.139 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173367">173367</A>
ACTIVATE: L3 upgrade test
<!-- Author: Andrei/Jeff/Irina x2080 -->


<!-- Date: Thu Nov 6 09:00:16.662 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173367">173367</A>
 Terminated at 2003.11.06 08:58:57
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 09:02:25.063 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173367">173367</A>
TERMINATE: end test (successful); trigger [1,276,406]
<!-- Author: Andrei/Jeff/Irina x2080 -->


<!-- Date: Thu Nov 6 09:04:32.502 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173368">173368</A>
 Activated at 2003.11.06 09:02:52
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 09:06:41.746 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173368">173368</A>
 Terminated at 2003.11.06 09:04:13
<!-- Author: RunControl -->


<!-- Date: Thu Nov 6 09:10:56.451 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=173369">173369</A>
 Activated at 2003.11.06 09:09:19
<!-- Author: RunControl -->