<!-- Title: Owl shift. Thu Jun 29, 2006 -->
<!-- SciCo: Tom Junk -->
<!-- DAQAce: Ed Rogers -->
<!-- MonAce: Gianluca De Lorenzo -->
<!-- CO: Aurore Savoy-Navarro -->
<!-- OpManager: Bob Wagner -->
<!-- Notes: -->
<pre>Shot setup</pre>


<!-- Date: Thu Jun 29 00:09:44.667 2006 -->
Final protons being loaded  
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219262' target=_top>219262</a>) -->

<!-- Date: Thu Jun 29 00:14:28.951 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219263>219263</A>
 Activated at 2006.06.29 00:14:06
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 00:15:36.586 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219263>219263</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,598] Test of Trigger TagSet fails.  Switching to Physics Mode.
<!-- Author: Ed -->


<!-- Date: Thu Jun 29 00:15:37.477 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219263>219263</A>
 Terminated at 2006.06.29 00:15:31
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 00:16:00.656 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219263>219263</A>
TERMINATE: Failed test.  See above.
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 00:18:43.084 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>CMX INSTRUCTIONS FOR THE SHIFT CREW 
<br> 
------------------------------------------------------------------------ 
<br> 
CMX NE 18-19 is known to be unstable lately.  If it keeps tripping, and if data is taken, please turn this iFix channel off (no need to page the CMX expert).  The CMX should be marked good even with CMX NE 18-19 off, because it corresponds to only 10 degrees of the CMX system (less than a wedge).</td></tr></td></table>
<!-- Author: John Strologas -->

<!-- Date: Thu Jun 29 00:22:40.583 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219264>219264</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 00:32:12.265 2006 -->
pbars started.  Paged silicon  
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219264' target=_top>219264</a>) -->


<!-- Date: Thu Jun 29 00:57:26.478 2006 -->
pbars all loaded. 
 
pr= 9184E9 
pb=1539E9
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219264' target=_top>219264</a>) -->

<!-- Date: Thu Jun 29 00:58:33.892 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219264>219264</A>
TERMINATE: Here comes the science.
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 01:15:07.156 2006 -->
Store #4804 - scraping complete  
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265' target=_top>219265</a>) -->


<!-- Date: Thu Jun 29 01:21:27.873 2006 -->
clock errors before resetting: 
742 
1113 
BC LOC
<!-- Author: gianluca -->

<!-- Date: Thu Jun 29 01:22:52.509 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265>219265</A>
 Activated at 2006.06.29 01:22:34
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 01:22:53.909 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265>219265</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 01:25:07.022 2006 -->
Taking data.  Lumi: 123.7E30, protons: 8745E9, pbars: 1501E9
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265' target=_top>219265</a>) -->


<!-- Date: Thu Jun 29 01:25:18.630 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5241&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5241  width='300' align='top'></a></td><td>Losses and Abort Gap Counters during the shot</td></tr></table>
<!-- Author: gianluca -->


<!-- Date: Thu Jun 29 01:25:59.325 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5242&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5242  width='300' align='top'></a></td><td>SVRAD rates during the shot</td></tr></table>
<!-- Author: gianluca -->

<!-- Date: Thu Jun 29 01:43:44.086 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265>219265</A>
 Terminated at 2006.06.29 01:43:41
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 01:44:38.992 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265>219265</A>
TERMINATE: Event Builder errors require EVB cleanup
<!-- Author: Ed -->


<!-- Date: Thu Jun 29 02:01:12.170 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266>219266</A>
 Activated at 2006.06.29 02:01:03
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 02:03:24.561 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266>219266</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 02:09:33.534 2006 -->
SCPU for EVB2B got hung up, requiring (a) a reset of b0evme22 in 1RR21H, (b) a clean-up of EVB, and (c) that an L3 expert get paged.  Connor responded promptly, and the instructions for the reset (found <a href="http://www-cdfonline.fnal.gov/ops/evbl3shift/evbl3shift.html#scpu">here</a>) were quite clear. 
 
Back up and running now.
<!-- Author: Ed :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266' target=_top>219266</a>) -->
<!-- Comment: Thu Jun 29 02;15;59 comment by...Conor -->
Good work, Ed. For scpu 22, since that is in a silicon crate, it is particularly important that you reset the evb spu, and not just the board in slot 0, since that would cause considerable problems with Silicon. That's why we like to be paged about this. In this case, it clearly went fine.


<!-- Comment: Thu Jun 29 02;28;27 comment by...Ulrich -->
I was in the control room at the time. There is indeed a risk of tripping parts of the silicon by resetting the wrong CPU, especially in b0svx02 and b0svx06, where the Silicon Readout Controllers (SRC) are situated.
<!-- Date: Thu Jun 29 02:17:22.232 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5243&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5243' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5244&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5244' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5245&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5245' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 29 02;18;00 comment by...gianluca -->
hourly ace plots: 01:00 -> 02:00


<!-- Date: Thu Jun 29 02:18:51.947 2006 -->
<pre>EVB2B problem seems to have returned: 
 
 
 Attention !!!. FERML_TM2_HEVB2B_EVENTS_PENDING_TIMEOUT !!!  
 - AUTO HRR will be issued.  
  
 - If problem persists, you may have to clean up the Event Builder. </pre>
<!-- Author: Ed :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266' target=_top>219266</a>) -->
<!-- Comment: Thu Jun 29 02;22;17 comment by...Ed -->
<pre>Busy Condition in VRB/EVB forwarded by front-end crate(s)
Busy conditions are produced by the VRB and are forwarded
 by the front end crates to the busy crosspoint.  Busy
 conditions do not represent a problem with the front end
 crates themselves, but with the EVB system.</pre>

<!-- Comment: Thu Jun 29 02;33;43 comment by...Ed -->
<pre>EVB needed a little TLC.  With a clean-up and restart, everything seems to be running smoothly again.</pre>
<!-- Date: Thu Jun 29 02:23:21.052 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266>219266</A>
 Terminated at 2006.06.29 02:22:27
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 02:23:22.893 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266>219266</A>
TERMINATE: evb2b may need a cleanup
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 02:30:34.022 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219267>219267</A>
 Activated at 2006.06.29 02:30:28
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 02:30:53.114 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219267>219267</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 02:30:54.074 2006 -->
<pre>error from iFix:  "A network transaction is in progress".</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219267' target=_top>219267</a>) -->
<!-- Comment: Thu Jun 29 02;33;53 comment by...gianluca -->
<pre>errors were from TOF1 PC.  TOF HV looks good, anyway.</pre>

<!-- Date: Thu Jun 29 02:52:33.449 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219267>219267</A>
 Terminated at 2006.06.29 02:52:30
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 02:54:11.479 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219267>219267</A>
TERMINATE: Ending run because (a) CMX has tripped causing Halt, and (b) L3 Display has gone RED, which can only be remedied with a new run, according to the webpage
<!-- Author: Ed -->
<!-- Comment: Thu Jun 29 03;01;48 comment by...Ed -->
<pre>Well, cleaning up the L3 farm went smoothly, but was perhaps unnecessary. (?)  It didn't seem to fix the L3 Display's redness problem.  Closing it and re-opening it, however, worked like a charm.</pre>
<!-- Date: Thu Jun 29 02:57:54.366 2006 -->
Ulrich called to say he was having troubles connecting from offsite to computers onsite.  I'm having no network difficulties now, though.
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->

<!-- Date: Thu Jun 29 02:59:35.922 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
 Activated at 2006.06.29 02:58:57
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 02:59:36.330 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 03:04:38.067 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219265>219265</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP Ongoing
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Jun 29 03:05:20.544 2006 -->
<pre>CMX trip: NE 18 and 19.   trip number 1 since the beginning of the store.</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->

<!-- Date: Thu Jun 29 03:08:32.633 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219266>219266</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP ongoing
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Jun 29 03:09:12.433 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219267>219267</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP ongoing
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Jun 29 03:19:57.856 2006 -->
<pre>L3 partition boxes issue: 
the partition boxes went red for the second time.  we simply fixed the problem killing the L3 display and starting it again.   
 
Do the SciLinux farms have any communication problem with RTServer?</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->
<!-- Comment: Thu Jun 29 03;24;12 comment by...Conor -->
The level3 proxy does appear to be running fine. Can you use ProcMon to check the RTserver?

<!-- Comment: Thu Jun 29 03;37;05 comment by...Conor -->
<pre>Yes, from the rtserver log:
"Thu Jun 29 03:31:17 2006: Closing connection on fd 15 from b0eproxy02.fnal.gov
************************* Socket 15 appears stuck; sendq: 73848; configure for reset"
So the L3 display turned red because the connection to RTserver was temporarily closed. I will ask Bill Badgett about this tomorrow. For the rest of the night, if you see the L3 display turn red, please check the RTserver and use the Level3 ACE Control Panel to 'check l3proxy'. Only proceed with the level3 cleanup if the level3 proxy check says that the proxy is not running. Call me if its not clear.</pre>

<!-- Comment: Thu Jun 29 08;45;40 comment by...Sasha Golossanov -->
<pre>
sometimes rtserver indeed resets stuck connections.  it can take a
few minutes.  it's not clear why a connection gets stuck, but it
appears that L3 monitoring program doesn't update the state of
those boxes unless something get's reset or the program get's
re-started all together.     

perhaps, as a quick fix, the instructions on the L3 monitor page can be updated to include the check of the evb proxy state and it's connection to the rtserver.

</pre>
<!-- Date: Thu Jun 29 03:52:57.603 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
muons: <tt>Still low statistices on CMP trigger plot</tt>
<!-- Author: Consumer Operator( Aurore Savoy-Navarro) -->


<!-- Date: Thu Jun 29 03:52:59.745 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
trigger: <tt>L2_Table_Beam, SVT Efficiencies, below ! per mil, thus OK.</tt>
<!-- Author: Consumer Operator( Aurore Savoy-Navarro) -->


<!-- Date: Thu Jun 29 03:53:00.409 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
monitors: <tt>Slightly more errors 6/302 wrt 2/397 in the DAQErrMon Summqry.</tt>
<!-- Author: Consumer Operator( Aurore Savoy-Navarro) -->


<!-- Date: Thu Jun 29 03:53:01.573 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
general: <tt>To do the checklist we had to wait to get stable running conditions. That's why this is the first checklist of our owl shift. One may note still a rather low statistics but apart from the MNP which is still marked as BAD the rest looks fine so far.</tt>
<!-- Author: Consumer Operator( Aurore Savoy-Navarro) -->



<!-- Date: Thu Jun 29 04:06:55.955 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5246&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5246' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5247&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5247' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5248&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5248' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 29 04;08;12 comment by...gianluca -->
<pre>more-or-less-hourly ace plots: 2:00 -&gt; 4:00</pre>


<!-- Date: Thu Jun 29 04:18:33.041 2006 -->
Lum: 91E30, protons: 8581E9, pbars: 1361E9
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->


<!-- Date: Thu Jun 29 04:51:29.742 2006 -->
The L3 monitoring seems to be rather spotty today.  All gray on the screen, but a moment later the nodes started showing activity.
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->
<!-- Comment: Thu Jun 29 06;52;13 comment by...Conor -->
THe display clears itself every 10 mins. You probably noticed it in the middle of a refresh. Nothing to worry about.
<!-- Date: Thu Jun 29 05:03:43.928 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5249&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5249' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5250&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5250' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5251&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5251' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 29 05;04;09 comment by...gianluca -->
<pre>ace plots: 04:00 -&gt; 05:00</pre>


<!-- Date: Thu Jun 29 05:16:34.866 2006 -->
<pre>CMX trip: NE 18 and 19.   trip number 2 since the beginning of the store.</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->


<!-- Date: Thu Jun 29 05:22:38.951 2006 -->
(MLW) b0control21.fnal.gov:Thread-33:Jun 29, 2006 05:21:54 AM->Inhibit flag CLEAR (MLE) b0control21.fnal.gov:Thread-33:Jun 29, 2006 05:22:05 AM->Busy Timeout: TRIGGER_SCALERS_00 LEVEL2_PULSAR_01 
<!-- Author: Ed :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268' target=_top>219268</a>) -->
<!-- Comment: Thu Jun 29 05;22;53 comment by...Ed -->
<pre>Busy Condition in VRB/EVB forwarded by front-end crate(s)
Busy conditions are produced by the VRB and are forwarded
 by the front end crates to the busy crosspoint.  Busy
 conditions do not represent a problem with the front end
 crates themselves, but with the EVB system.</pre>

<!-- Comment: Thu Jun 29 05;29;31 comment by...Ed -->
<pre>We had to start a new run after strange Busy Timeouts for TRIGGER_SCALERS_00.  Perhaps unrelated, but deadtime had been creeping up.  (15 - 20%)</pre>
<!-- Date: Thu Jun 29 05:24:58.247 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
 Terminated at 2006.06.29 05:24:46
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 05:26:53.967 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
TERMINATE: Restarting run due to reports from RC of TRIGGER_SCALARS_00 giving busy time-outs.  Requires new run (no shepherding allowed).
<!-- Author: Ed -->


<!-- Date: Thu Jun 29 05:29:10.717 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219269>219269</A>
 Activated at 2006.06.29 05:28:35
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 05:29:11.287 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219269>219269</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->


<!-- Date: Thu Jun 29 05:29:31.879 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219225>219225</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP ongoing
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Jun 29 05:39:49.353 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
trigger: <tt>Diff parameters in SVT efficiencies in L2-Table-Beam: stays below !/!000 thus this is OK</tt>
<!-- Author: Consumer Operator( Aurore Savoy-Navarro) -->


<!-- Date: Thu Jun 29 05:39:50.283 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
general: <tt>MNP marked BAD as usual, the rest is stable too...</tt>
<!-- Author: Consumer Operator( Aurore Savoy-Navarro) -->



<!-- Date: Thu Jun 29 05:42:35.284 2006 -->
N.B. The change in run number occured while I was doing the checklist, thus I finished the checklist first and then changed the run number.
<!-- Author: aurore :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219269' target=_top>219269</a>) -->

<!-- Date: Thu Jun 29 05:51:00.961 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219269>219269</A>
 Terminated at 2006.06.29 05:50:48
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 05:51:01.926 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219269>219269</A>
TERMINATE: EVB needs cleanup
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 05:52:28.541 2006 -->
<br /><br /><b>List of problems that have temporarily reared their head in the past 10 minutes, before slinking back into the darkness:</b><br /> 
<ul> 
<li>Everything on b0control16 froze: which includes displays for CSL, DAQ Monitor, VxWorks System Monitor. 
<li>Lost COT HeartBeat 
<li>EVB Proxy is not running for Partition 0 
<li>RTServer had stuck ports 
<li>Deadtime over 20%, due to Readout and TSI 
<li>CMX HV Trips 
</ul>
<!-- Author: Ed :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219269' target=_top>219269</a>) -->

<!-- Date: Thu Jun 29 05:58:42.808 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270>219270</A>
 Activated at 2006.06.29 05:57:20
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 05:58:43.298 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270>219270</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Ed -->


<!-- Date: Thu Jun 29 05:58:44.580 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270>219270</A>
 Terminated at 2006.06.29 05:58:07
<!-- Author: RunControl -->


<!-- Date: Thu Jun 29 05:58:45.858 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270>219270</A>
TERMINATE: Solenoid powered down.  CRYO is working on it.
<!-- Author: Ed -->



<!-- Date: Thu Jun 29 06:01:41.601 2006 -->
Just when you thought this shift couldn't get any more exciting, the solenoid has tripped, and the status is being investigated as the magnetic field is decreasing rapidly.  Run has been ended, and HV systems have been set to standby.
<!-- Author: Ed :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270' target=_top>219270</a>) -->


<!-- Date: Thu Jun 29 06:01:41.168 2006 -->
<pre>Serious problem with the magnet (Helium FIMF below the lower limit and magnet HV trip).  We ended the run and put all the HVs in the "End Of Store" configuration.</pre>
<!-- Author: gianluca :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270' target=_top>219270</a>) -->


<!-- Date: Thu Jun 29 06:05:18.537 2006 -->
Solenoid trip.  Low Helium.  Called cryo -- they are working on it.  The solenoid is ramping back up now.  We had hit the "end of store" button for the HV to put it in a safe state.
<!-- Author: Tom Junk :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219270' target=_top>219270</a>) -->


<!-- Date: Thu Jun 29 06:06:06.196 2006 -->
Bob Wagner says when ramping COT back up, to hold volts at standby for 10 minutes.  Instructions on COT web page.
<!-- Author: Tom Junk -->


<!-- Date: Thu Jun 29 06:17:30.857 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5252&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5252' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5253&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5253' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5254&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5254' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 29 06;18;15 comment by...gianluca -->
ace plots: 5:00 -> 6:00


<!-- Date: Thu Jun 29 06:25:23.406 2006 -->
Solenoid holding at 13304 Gauss.  Cryo says rampup is paused there until a warm bubble from the trip propagates through the system and it settles down.
<!-- Author: Tom Junk -->



<!-- Date: Thu Jun 29 07:04:58.441 2006 -->
GAP triggers were very hot on first two runs this store.  Must be a BSC problem.  Gap+muon should not be 600Hz in L2!
<!-- Author: jdl -->

<!-- Date: Thu Jun 29 07:10:21.092 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219232>219232</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP Ongoing
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Jun 29 07:10:50.294 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219237>219237</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP ongoing
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Jun 29 07:13:31.404 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5255&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5255' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5256&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5256' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5257&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5257' width='300'></a></center>
<!-- Author: gianluca -->
<!-- Comment: Thu Jun 29 07;14;53 comment by...gianluca -->
ace plots 6:00 -> 7:00


<!-- Date: Thu Jun 29 07:30:06.721 2006 -->
Jonathan called about our high deadtime in 
the runs this store -- 219268 and 219269 are examples. 
He says it's the triggers with BSC in them. 
Here from 219268's XMON summary are the four with the 
highest zScore 
 
L2  	5 L2_AUTO_L1_TWO_GAP_&_CMU1.5_PT1.5_v1 	71.09 
L1 	11 L1_TWO_GAP_&_CMU1.5_PT1.5_v1 	71.09 
L1 	43 L1_CMUP6_PT4_v2 	12.28 
L2 	93 L2_PS250_L1_CMUP6_PT4_v1 	12.08  
<!-- Author: Tom Junk -->
<!-- Comment: Thu Jun 29 07;46;25 comment by...Tom -->
I paged Koji shortly after Jonathan's call.  Just getting to make the logbook entry now.
<!-- Comment: Thu Jun 29 07;52;11 comment by...Tom -->
Koji investigated -- it looks like the it might be the muon triggers and not the BSC part.  I see that CMUP6_PT4v2 just recently got hot (it's not as hot as the gap ones, but the pt cut is harder).  Will investigate.
<!-- Date: Thu Jun 29 07:39:55.284 2006 -->
Second solenoid trip.  Cryo says it was inevitable -- they had to switch dry engines.
<!-- Author: Tom Junk -->



<!-- Date: Thu Jun 29 07:53:08.364 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219268>219268</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
MNP ongoing
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Jun 29 07:55:15.384 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219265 target=_runSummary>219265</A> x35881</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,638]</FONT></TD>
<TD ALIGN=CENTER>01:22:34</TD>
<TD ALIGN=CENTER>01:43:41</TD>
<TD ALIGN=CENTER>00:17:14</TD>
<TD ALIGN=RIGHT>      81,680</TD>   <TD ALIGN=RIGHT>    155.26</TD>
<TD ALIGN=RIGHT>    125.34</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219266 target=_runSummary>219266</A> x35882</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,638]</FONT></TD>
<TD ALIGN=CENTER>02:01:03</TD>
<TD ALIGN=CENTER>02:22:27</TD>
<TD ALIGN=CENTER>00:15:45</TD>
<TD ALIGN=RIGHT>      68,443</TD>   <TD ALIGN=RIGHT>    142.84</TD>
<TD ALIGN=RIGHT>    105.95</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219267 target=_runSummary>219267</A> x35883</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,638]</FONT></TD>
<TD ALIGN=CENTER>02:30:28</TD>
<TD ALIGN=CENTER>02:52:30</TD>
<TD ALIGN=CENTER>00:19:28</TD>
<TD ALIGN=RIGHT>      79,970</TD>   <TD ALIGN=RIGHT>    141.79</TD>
<TD ALIGN=RIGHT>    122.57</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219268 target=_runSummary>219268</A> x35884</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,638]</FONT></TD>
<TD ALIGN=CENTER>02:58:57</TD>
<TD ALIGN=CENTER>05:24:46</TD>
<TD ALIGN=CENTER>02:01:21</TD>
<TD ALIGN=RIGHT>     464,485</TD>   <TD ALIGN=RIGHT>    807.80</TD>
<TD ALIGN=RIGHT>    673.53</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219269 target=_runSummary>219269</A> x35885</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,638]</FONT></TD>
<TD ALIGN=CENTER>05:28:35</TD>
<TD ALIGN=CENTER>05:50:48</TD>
<TD ALIGN=CENTER>00:13:04</TD>
<TD ALIGN=RIGHT>      45,963</TD>   <TD ALIGN=RIGHT>    110.18</TD>
<TD ALIGN=RIGHT>     64.05</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>03:06:54</TD>
<TD ALIGN=RIGHT>       740,541</TD> <TD ALIGN=RIGHT>  1,357.87</TD>
<TD ALIGN=RIGHT>  1,091.44</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Jun 29 07:59:56.671 2006 -->
<font color=red>Shift Summary: </font><pre>Many L3 problems, but some of them were just monitoring.  The L3 monitor screen can look quite alarming and everything is fine.  Restarting the monitor display should be the first to try. 
 
Solenoid trip close to the end of the trip due to low helium.  Cryo is switching dry engines, and induced a second trip.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   219265,6,7,8,9
Delivered Luminosity   1652.7  
Acquired Luminosity    1091.4  
Efficiency             66%
   
</td></tr></table></pre>
<!-- Author: Tom Junk -->