<!-- Title: Eve shift. Tue Jul 24, 2007 -->
<!-- SciCo: Pekka Sinervo -->
<!-- DAQAce: Sasha Paramonov -->
<!-- MonAce: &nbsp; -->
<!-- CO: Oriol Salto -->
<!-- OpManager: Max Goncharov -->
<!-- Notes: -->
<pre>Currently waiting for next store, scheduled for mid to late evening.  All systems OK, no requests.</pre>

<!-- Date: Tue Jul 24 16:13:04.802 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245698>245698</A>
TERMINATE: going to cosmics
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 16:18:29.650 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245699>245699</A>
 Activated at 2007.07.24 16:18:23
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:18:30.234 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245699>245699</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,927,674] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 16:20:08.187 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245699>245699</A>
 Terminated at 2007.07.24 16:20:03
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:34:08.584 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245700>245700</A>
 Activated at 2007.07.24 16:33:54
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:34:09.199 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245700>245700</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,927,674] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 16:35:25.510 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245700>245700</A>
 Terminated at 2007.07.24 16:35:11
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:35:50.772 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245700>245700</A>
TERMINATE: restarting cosmics again. testing csl
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 16:41:55.888 2007 -->
<pre>While Ending a a cosmics run we got: 
 
(MLE) b0eproxy02.fnal.gov:main:Jul 24, 2007 04:35:41 PM->Error on L3 node b0eproxy02.fnal.gov (partition 0) Tue Jul 24 16:35:36 2007 l3proxy  
  in End (L3ProxyPartition.cc:2269) 
  @L3_TRANS_FAIL - transition actions failed:      b0l3s09p12 
 
(MLE) b0eproxy02.fnal.gov:main:Jul 24, 2007 04:38:33 PM->Error on L3 node b0eproxy02.fnal.gov (partition 0) Tue Jul 24 16:38:33 2007 l3proxy  
  in End (L3ProxyPartition.cc:2287) 
  @L3_TRANS_FAIL - partition End timeout:     b0l3s09p12</pre>
<!-- Author: Sasha -->

<!-- Date: Tue Jul 24 16:46:28.111 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245701>245701</A>
 Activated at 2007.07.24 16:46:12
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:46:29.059 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245701>245701</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,927,674] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 16:48:34.981 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245701>245701</A>
 Terminated at 2007.07.24 16:48:17
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:49:43.078 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245701>245701</A>
TERMINATE: excluding b0l2pu02
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 16:53:15.901 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245702>245702</A>
 Activated at 2007.07.24 16:51:26
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 16:53:16.936 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245702>245702</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,927,674] 
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 17:02:11.357 2007 -->
<pre>Putting in errors from a filter log to help diagnose current problems: 
 
 
ERROR--> hostname: b0l3s01p02 
         prg name: l3_node 
         function: l3_msgque_receive() 
         datetime: Tue Jul 24 16:35:42 2007 
         src file: /home/l3proxy/commissioning/level3_converter/control/src/library/l3_msgque.c 
         src line: 677 
         src info: msgrcv() -> error 
         sys perr: Identifier removed 
  Priority 0 (lowest = 0) 
Chain destroy method 
 
ERROR--> hostname: b0l3s01p02 
         prg name: l3_node 
         function: exe_exchange() 
         datetime: Tue Jul 24 16:35:42 2007 
         src file: /home/l3proxy/commissioning/level3_converter/control/src/library/l3_ana_exe.c 
         src line: 548 
         src info: Error receiving shmbuf from filter 
 
ERROR--> hostname: b0l3s01p02 
         prg name: l3_node 
         function: l3_ana_exe_end() 
         datetime: Tue Jul 24 16:35:42 2007 
         src file: /home/l3proxy/commissioning/level3_converter/control/src/library/l3_ana_exe.c 
         src line: 944 
         src info: Error exchanging buffer to filter 
</pre>
<!-- Author: khaldoun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245702' target=_top>245702</a>) -->
<!-- Comment: Tue Jul 24 17;23;41 comment by...khaldoun -->
<pre>Correction, this is the l3_node log....</pre>
<!-- Date: Tue Jul 24 17:03:57.079 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245702>245702</A>
 Terminated at 2007.07.24 17:03:38
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 17:03:58.704 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245702>245702</A>
TERMINATE: L3/csl issues
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 17:11:45.030 2007 -->
We are going to run some L2 torture to test the abort functionality.  Will immediately give back system to L3/CSL upon request.
<!-- Author: Heather -->
<!-- Comment: Tue Jul 24 17;14;39 comment by...Heather and Kevin -->
Never got the system.  



<!-- Comment: Tue Jul 24 18;05;17 comment by...Heather and Kevin -->
We returned b0l2pu02 back to normal configuration.
<!-- Date: Tue Jul 24 17:29:03.056 2007 -->
<pre>(MLE) b0eproxy02.fnal.gov:main:Jul 24, 2007 05:24:51 PM->Error on L3 node b0eproxy02.fnal.gov (partition 0) Tue Jul 24 17:24:51 2007 l3proxy  
  in Initialize (L3ProxyPartition.cc:1005) 
  @L3_TRANS_FAIL - initialization timeout (will clean up):      b0l3s08p20   b0l3s18p14</pre>
<!-- Author: Sasha -->


<!-- Date: Tue Jul 24 17:31:17.042 2007 -->
<pre>Strange thing on several L3 nodes: after clean-up, some defunct l3_node processes still hanging around.  We get a L3_TRANS_FAIL message from those nodes in the RC error log, and when I do top on those nodes I see things like: 
 
32001 l3proxy   15   0  5872    8     0 S     0.0  0.0   0:00   2 l3_node 
32002 l3proxy   15   0     0    0     0 Z     0.0  0.0   0:00   2 l3_node <defunct> 
  588 l3proxy   25   0     0    0     0 Z     0.0  0.0   0:23   2 l3_node <defunct> 
19847 l3proxy   15   0   544    8     0 S     0.0  0.0   0:00   0 l3_node 
19848 l3proxy   15   0     0    0     0 Z     0.0  0.0   0:00   0 l3_node <defunct> 
 6626 l3proxy   15   0  6292    8     0 S     0.0  0.0   0:00   0 l3_node 
16775 l3proxy   15   0 32492  28M 26780 S     0.0  1.4   0:00   1 l3_node 
16777 l3proxy   15   0     0    0     0 Z     0.0  0.0   0:00   2 l3_node <defunct> 
17699 l3proxy   25   0     0    0     0 Z     0.0  0.0   0:28   2 l3_node <defunct> 
17701 l3proxy   15   0     0    0     0 Z     0.0  0.0   0:00   1 l3_node <defunct> 
17287 l3proxy   15   0  1160 1160   704 S     0.0  0.0   0:27   0 RemoteServer 
</pre>
<!-- Author: khaldoun -->
<!-- Comment: Tue Jul 24 17;48;18 comment by...khaldoun -->
<pre>I logged in as root with a script to all the nodes in the L3 farm and force killed at l3_node processes.   

Although I have never seen this before, it does look like the L3 Clean Up failed to clean up.  In other words, l3_node processes, which should be killed by L3 Clean Up, were surviving this process.  This was also likely the source of the communication failure from the log file I posted above.  

Now we are able to partition, config and activate.  We are going to end the run, reset and try a Shot Setup run to make sure that things are OK.</pre>
<!-- Date: Tue Jul 24 17:44:44.252 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245703>245703</A>
 Activated at 2007.07.24 17:44:34
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 17:44:45.795 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245703>245703</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,927,674] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 17:46:12.030 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245703>245703</A>
 Terminated at 2007.07.24 17:45:55
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 17:46:13.438 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245703>245703</A>
TERMINATE: more L3 testing
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 17:53:32.595 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245704>245704</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:00:13.231 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245704>245704</A>
TERMINATE: testing L3
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:05:16.170 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245705>245705</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:05:17.830 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245705>245705</A>
TERMINATE: more tests
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 18:14:53.318 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245706>245706</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:15:49.508 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245706>245706</A>
TERMINATE: L3 test
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:43:50.586 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245707>245707</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:53:01.193 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245708>245708</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->


<!-- Date: Tue Jul 24 18:53:22.333 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245708>245708</A>
TERMINATE: more L3 tests
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 18:59:39.549 2007 -->
Started loading protons for next store.
<!-- Author: Pekka -->
<!-- Comment: Tue Jul 24 19;12;54 comment by...Pekka -->
Loaded 11205e09 @ 2.877nsec avg.
<!-- Date: Tue Jul 24 19:01:46.481 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245709>245709</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 19:09:23.968 2007 -->
COT2 PC was not functioning properly - iFix freezed. We rebooted the PC.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->

<!-- Date: Tue Jul 24 19:10:16.471 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710>245710</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->




<!-- Date: Tue Jul 24 19:13:24.991 2007 -->
Starting to load pbars.
<!-- Author: Pekka -->
<!-- Comment: Tue Jul 24 19;14;15 comment by...Pekka -->
Silicon and OPS Managers paged.

<!-- Comment: Tue Jul 24 19;19;02 comment by...Pekka -->
Immediately after posting this, MCR decided to dump the proton load as they weren't happy with the lifetime.  So now back to checking tunes, etc.


<!-- Date: Tue Jul 24 19:21:47.447 2007 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Results and conclusions from the past couple hours of testing: 
 
We see a consistent failure pattern in L3 where 60% of the nodes in the farm turn grey instead of blue at Config, and are subsequently excluded from the run.   
 
This problem actually starts during the END transition of the previous run  None of these nodes get reported as Relay Failures, but the L3 Proxy log says at END: 
 
        L3ParaRelay::send WARNING! Removing a command for the broken node from the list.b0l3s01p03 
 
followed by (at the next PARTITION): 
 
        L3ProxyFarmHandler::verifyRelay: relay detects 216  broken nodes 
        ... 
        L3ProxySubfarm::RemoveProcessor: Deleting b0l3s15p16 
        ... 
 
These are always the same exact nodes. 
 
For the moment we have not found the source of the failure, but we have found a fix temporarily.  Cleaning up L3 gets things back to normal. 
 
As much as I might dislike this solution, until I find the source of the problem, I can only recommend the following: 
 
Clean up L3 between each run.</pre></td></tr></td></table>
<!-- Author: khaldoun -->


<!-- Date: Tue Jul 24 19:33:46.659 2007 -->
L2 timeout. HRRed
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->


<!-- Date: Tue Jul 24 19:34:26.210 2007 -->
Protons now being loaded.
<!-- Author: Pekka :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->


<!-- Date: Tue Jul 24 19:34:54.315 2007 -->
<pre>Attention!!! L2 Decision Timeout ERROR !!.  
Fixed by an HRR</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->
<!-- Comment: Tue Jul 24 19;39;46 comment by...Sasha -->
again


<!-- Date: Tue Jul 24 19:41:37.829 2007 -->
Protons Loaded 11272e09 @ 2.83nsec avg.
<!-- Author: Pekka :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->


<!-- Date: Tue Jul 24 19:53:08.698 2007 -->
<pre>(MLE) b0control21.fnal.gov:Thread-25:Jul 24, 2007 07:51:56 PM->Busy Timeout: XFT_FINDER_06 
 
Fixed by an HRR</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->


<!-- Date: Tue Jul 24 20:25:50.017 2007 -->
<pre>Busy timeout. It was fixed by an auto HRR. 
 
 
 
MLE) b0control21.fnal.gov:Thread-25:Jul 24, 2007 07:51:56 PM->Busy Timeout: XFT_FINDER_06</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->


<!-- Date: Tue Jul 24 20:28:01.439 2007 -->
<pre> 
Max has approved the tracklist test, the instructions are below: 
 
1) data is good 
2) shift crew just need to run AAA_SHOTSETUP_L2UP and AAA_CURRENT_L2UP 
3) If problems occur ( ie frequent time outs/l2ctrla goes red ) just end run and restart with 
AAA_CURRENT 
 
I will be keeping an eye on it also 
Thanks</pre>
<!-- Author: gene -->


<!-- Date: Tue Jul 24 20:31:11.804 2007 -->
<pre>L2 Decision Timeout.[RXPT] 
(MLE) b0control21.fnal.gov:Thread-25:Jul 24, 2007 08:30:05 PM->  Level 2 Decision Timeout 
 
(MLW) b0control21.fnal.gov:Thread-25:Jul 24, 2007 08:30:05 PM->Received error message: errmon Not a state HaltRecoverRun L2 Decision Timeout.[RXPT](MLE) b0control21.fnal.gov:Thread-25:Jul 24, 2007 08:30:05 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 24, 2007 08:30:15 PM->{ 20:30:15, 7/24/2007 }: L2-Pulsar-Linux: Halted while  processing happily 
 
Fixed by an HRR.</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->
<!-- Comment: Tue Jul 24 20;32;46 comment by...Sasha -->
Same thing.

<!-- Comment: Tue Jul 24 20;34;32 comment by...Sasha -->
again

<!-- Comment: Tue Jul 24 20;36;32 comment by...Sasha -->
1 more


<!-- Comment: Tue Jul 24 20;38;24 comment by...Sasha -->
more
<!-- Date: Tue Jul 24 20:37:25.300 2007 -->
<br> Done TO. for crate/s: b0cmu00  
<br> Fixed by an HRR.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->



<!-- Date: Tue Jul 24 20:40:43.782 2007 -->
Final protons have been loaded again.  11130e09 @ 2.814nsec avg
<!-- Author: Pekka :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710' target=_top>245710</a>) -->

<!-- Date: Tue Jul 24 20:43:53.875 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245710>245710</A>
TERMINATE: Switching to shotsetup L2UP
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 20:49:32.457 2007 -->
CSL restarted.
<!-- Author: Sasha -->


<!-- Date: Tue Jul 24 20:57:03.191 2007 -->
L3 got stuck on "L3 cleanup". BOttom of the l3 display is red.. L3 cleanup is not finished.
<!-- Author: Sasha -->
<!-- Comment: Tue Jul 24 21;10;18 comment by...Sasha -->
Clean up went through but it skipped c15. We are running now.
<!-- Date: Tue Jul 24 21:01:05.209 2007 -->
Finally starting to load pbars.  
<!-- Author: Pekka -->
<!-- Comment: Tue Jul 24 21;22;31 comment by...Pekka -->
Pbars loaded 159E10.
<!-- Date: Tue Jul 24 21:07:14.436 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245711>245711</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->




<!-- Date: Tue Jul 24 21:12:51.944 2007 -->
<pre>Busy timeout: 
 
MLE) b0control21.fnal.gov:Thread-25:Jul 24, 2007 09:11:55 PM->Busy Timeout: XFT_FINDER_06 </pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245711' target=_top>245711</a>) -->
<!-- Comment: Tue Jul 24 21;13;06 comment by...Sasha -->
fixed by an HRR

<!-- Date: Tue Jul 24 21:22:31.537 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245711>245711</A>
TERMINATE: new store
<!-- Author: Alexander Paramonov -->




<!-- Date: Tue Jul 24 21:27:22.442 2007 -->
Integrated dimond alarm went off. Silences. No LEDs are on.<br> 
SVRAD2 took 24 rads.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245711' target=_top>245711</a>) -->


<!-- Date: Tue Jul 24 21:41:13.107 2007 -->
Scraping over, LOSTP <1KHz, LOSTPB <200Hz.  L = 145.6E30.  Looks good fro Silicon -- bringing up HVs, and prepare for data taking.
<!-- Author: Pekka -->
<!-- Comment: Tue Jul 24 21;56;23 comment by...Pekka -->
Now running, trigger rates look reasonable. Having some crate timeouts though...
<!-- Date: Tue Jul 24 21:46:26.893 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712>245712</A>
 Activated at 2007.07.24 21:45:51
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 21:46:27.156 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712>245712</A>
 Activated at 2007.07.24 21:45:52
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 21:46:28.140 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712>245712</A>
ACTIVATE: RunSet: AAA_CURRENT_L2UP  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 21:50:49.662 2007 -->
<br>Done Timeout detected in crates: LEVEL2_PULSAR_02 
<br>I had to shephered the crate.  
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712' target=_top>245712</a>) -->



<!-- Date: Tue Jul 24 21:56:59.787 2007 -->
<pre>L2 Decision Timeout ERROR !!. 
 
followed by  
 
Done Timeout detected in crates:LEVEL2_PULSAR_02. 
 
b0l2pu02 was shepherded again... 
</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712' target=_top>245712</a>) -->
<!-- Comment: Tue Jul 24 22;13;34 comment by...Sasha -->
again
<!-- Date: Tue Jul 24 21:58:09.188 2007 -->
L2 Decision Timeout ERROR !!. again!
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712' target=_top>245712</a>) -->


<!-- Date: Tue Jul 24 22:10:39.938 2007 -->
There's a few uncommented auto-entries in E8 Downtime logger left from day shift...
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712' target=_top>245712</a>) -->



<!-- Date: Tue Jul 24 22:16:11.221 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6024&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6024' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6025&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6025' width='300'></a></center>
<!-- Author: Sasha -->

<!-- Date: Tue Jul 24 22:19:06.904 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712>245712</A>
 Terminated at 2007.07.24 22:18:21
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 22:19:07.588 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712>245712</A>
TERMINATE: L2pu02 problems..
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 22:29:23.411 2007 -->
we called Gene. He suggested to start a new run with AAA_CURRENT
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245713' target=_top>245713</a>) -->

<!-- Date: Tue Jul 24 22:32:42.596 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245713>245713</A>
 Activated at 2007.07.24 22:32:36
<!-- Author: RunControl -->


<!-- Date: Tue Jul 24 22:33:14.420 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245713>245713</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [10,929,698] 
<!-- Author: Alexander Paramonov -->



<!-- Date: Tue Jul 24 22:38:33.653 2007 -->
<br>Trigger Inhibit is set from the components: VME Power  		 
<br>[147] VME Crate Power Supplies from iFix 
<br>The error cleared on its own.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245713' target=_top>245713</a>) -->
<!-- Comment: Tue Jul 24 22;41;10 comment by...Sasha -->
Crate 1RR18I_2

<!-- Date: Tue Jul 24 23:41:58.990 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245712>245712</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
ISL No cooling on one side
L00 No cooling
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Tue Jul 24 23:56:04.462 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=245712 target=_runSummary>245712</A> x3BFD0</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [10,929,698]</FONT></TD>
<TD ALIGN=CENTER>21:45:52</TD>
<TD ALIGN=CENTER>22:18:21</TD>
<TD ALIGN=CENTER>00:23:10</TD>
<TD ALIGN=RIGHT>     179,754</TD>   <TD ALIGN=RIGHT>    268.64</TD>
<TD ALIGN=RIGHT>    187.95</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=245713 target=_runSummary>245713</A> x3BFD1</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [10,929,698]</FONT></TD>
<TD ALIGN=CENTER>22:32:36</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>01:16:01</TD>
<TD ALIGN=RIGHT>     614,994</TD>   <TD ALIGN=RIGHT>    544.59</TD>
<TD ALIGN=RIGHT>    502.38</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>01:39:11</TD>
<TD ALIGN=RIGHT>       794,748</TD> <TD ALIGN=RIGHT>    813.24</TD>
<TD ALIGN=RIGHT>    690.34</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Wed Jul 25 00:04:23.180 2007 -->
<font color=red>Shift Summary: </font><pre>Shift is reasonably productive, with progress made on some outstanding problems, and the successful injection of store 5577. 
 
 1)  L3 and CSL folks spend several hours and determine that the earlier problems appear to be due, at least in part, to a failure of L3 to properly clean up after the run is stopped.  This apparently ends up leaving >200 nodes taken out of service and leaving the CSL quite confused.  A work-around was identified, involving cleaning up the L3 Farm by hand before starting the next run. 
 
 2)  Further testing of the L2 upgrades was performed, with the most recent configuration provided by Gene tested at higher luminosity.  In this case, we saw consistent timeouts of a specific crate emerge as a problem to be resolved.  We backed out the latest version. 
 
 3)  Store 5577 came in smoothly once the accelerator was able to get a decent proton store, and we started with a clean store with initial L of 145E30, limited by the pbars available.  We were able to take data uneventfully through the last 1.5 hours of the shift (once the L2 upgrade configuration was backed out). 
 
ACEs to note that the L3 Farm must be cleaned up by hand before one begins setting up the next run till further notice from the L3 team.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   245712-713
Delivered Luminosity   998 nb-1  
Acquired Luminosity    721 nb-1  
Efficiency             72%
   
</td></tr></table></pre>
<!-- Author: Pekka -->