<!-- Title: Day shift. Wed Apr 18, 2007 -->
<!-- SciCo: Robert Blair -->
<!-- DAQAce: Shih-Chieh Hsu -->
<!-- MonAce: &nbsp; -->
<!-- CO: Mike Albrow -->
<!-- OpManager: Sasha -->
<!-- Notes: -->
<pre>Store has been dropped - shot setup is starting immediately.</pre>

<!-- Date: Wed Apr 18 08:16:07.691 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239473>239473</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,792,632] 
<!-- Author: Shih-Chieh -->



<!-- Date: Wed Apr 18 08:20:35.749 2007 -->
MCR calls and tells us that they intend to begin injecting beam.
<!-- Author: reb -->

<!-- Date: Wed Apr 18 08:25:15.906 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239473>239473</A>
 Terminated at 2007.04.18 08:24:59
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 08:26:15.554 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239473>239473</A>
TERMINATE: L1A 47.8Hz
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:28:36.949 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239474>239474</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Wed Apr 18 08:28:37.093 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239474>239474</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:30:44.750 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239475>239475</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:34:25.513 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239476>239476</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:35:39.616 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239476>239476</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Wed Apr 18 08:35:40.357 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239475>239475</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Wed Apr 18 08:37:00.577 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239477>239477</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Shih-Chieh -->
<!-- Comment: Wed Apr 18 08;37;32 comment by...Shih-Chieh -->
COT_19 Done timeout during calibration
 

<!-- Comment: Wed Apr 18 10;12;09 comment by...Shin-Shan Yu (Eiko) -->
<pre>When a done TO occurrs after the calibration is activated, please 
do not HRR. Abort the run, reset RC, before you partition, shepherd
the crate or do a vxboot. Then start a new run again. 
You may find more details in the COT ACE help homepage.

We are working on this problem now and for the time being, this is 
the solution. 

Sorry about the confusing buttons (HRR, etc), I will ask the RC 
experts to remove the unnecessary buttons in this run 
configuration.</pre>
<!-- Date: Wed Apr 18 08:40:09.545 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239477>239477</A>
TERMINATE: More information from Error handler:
Code: 0x60f0800000011  Mnemonic: FERML_DONE_TIMEOUT_CRATES
Node Name: b0control21.fnal.gov; Node Id: 131.225.236.159
Process: Thread-47462; PID: -1; Time: 8:36:46 AM
Code Line: 454; Routine: rc.DebugLevel.sendMerlin
Parameters: Done Timeout: COT_19  

The run is frozen at 878(L2A) and 881 Calib Events. I hit HRR but it ends the calibration
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:45:37.979 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239478>239478</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:45:38.086 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239477>239477</A>
CALIBRATION: <U><B>cot</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->


<!-- Date: Wed Apr 18 08:45:39.856 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239478>239478</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Wed Apr 18 08:48:19.216 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239479>239479</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Shih-Chieh -->
<!-- Comment: Wed Apr 18 08;48;54 comment by...Shih-Chieh -->
<pre>Error in Partition&Configuration
Strange (but not necessarily fatal) Run Configuration selections:
Crate IMU_00 is missing card TDC_BMU_03 in slot 14
</pre>

<!-- Date: Wed Apr 18 08:51:01.311 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239480>239480</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 08:51:17.837 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239356>239356</A>
general: <tt>OK</tt>
<!-- Author: Consumer Operator( Mike Albrow) -->



<!-- Date: Wed Apr 18 08:53:44.496 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3740&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3740  width='300' align='top'></a></td><td>Unclear from reference plot if this is reallyt bad.</td></tr></table>
<!-- Author: Mike Albrow -->
<!-- Comment: Wed Apr 18 09;55;52 comment by...kchung -->
<pre>PHA 12W ch 3 and WHA 12E ch 2 can be found in CO Fixlist.</pre>
<!-- Date: Wed Apr 18 08:53:57.029 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239481>239481</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [3,873,681] l2cal test 
<!-- Author: gene -->


<!-- Date: Wed Apr 18 08:54:43.711 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239481>239481</A>
TERMINATE: end to include new l2cal unpacking
<!-- Author: gene -->



<!-- Date: Wed Apr 18 08:54:51.325 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3741&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3741  width='300' align='top'></a></td><td>Not too good (?)</td></tr></table>
<!-- Author: Mike Albrow -->

<!-- Date: Wed Apr 18 08:56:37.038 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239480>239480</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Wed Apr 18 09:03:09.425 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239482>239482</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [3,873,681] includes new unpacking and running over the new jet90 trigger and a new electron trigger
<!-- Author: gene -->



<!-- Date: Wed Apr 18 09:06:39.425 2007 -->
Injecting final protons.
<!-- Author: reb -->

<!-- Date: Wed Apr 18 09:09:24.783 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239482>239482</A>
TERMINATE: end run due to time outs in l2pu02, something might be slow in the code or maybe sending junk. Testing continues...
<!-- Author: gene -->


<!-- Date: Wed Apr 18 09:14:11.141 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239483>239483</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [3,873,681] new jet90
<!-- Author: gene -->


<!-- Date: Wed Apr 18 09:18:36.929 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239483>239483</A>
TERMINATE: more time outs 
<!-- Author: gene -->


<!-- Date: Wed Apr 18 09:25:44.946 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239484>239484</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [3,873,681] back to old triggers and no unpacking
<!-- Author: gene -->


<!-- Date: Wed Apr 18 09:25:45.138 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239484>239484</A>
TERMINATE: even with old triggers and no new unpacking l2pu02 is giving time outs ?, this is odd as it ran smoothly with this configuration half an hour ago 
<!-- Author: gene -->



<!-- Date: Wed Apr 18 09:29:05.025 2007 -->
injecting pbars
<!-- Author: reb -->

<!-- Date: Wed Apr 18 09:37:03.629 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239485>239485</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [3,873,681] 
Loading pbar. 
<!-- Author: ACE -->



<!-- Date: Wed Apr 18 09:37:51.225 2007 -->
SVT_TRACKING_00 done time out. HRR works.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239485' target=_top>239485</a>) -->


<!-- Date: Wed Apr 18 09:39:43.166 2007 -->
LEVEL2_PULSAR_02 DT. Autho HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239485' target=_top>239485</a>) -->
<!-- Comment: Wed Apr 18 09;50;01 comment by...Shih-Chieh -->
again



<!-- Date: Wed Apr 18 09:59:25.525 2007 -->
Procmon needs to be restarted because rtserver has lost communication.  We have Jane in the control room to walk us through this.
<!-- Author: reb -->

<!-- Date: Wed Apr 18 10:01:05.265 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239485>239485</A>
TERMINATE: Powwer cycle xft06
<!-- Author: ACE -->



<!-- Date: Wed Apr 18 10:02:38.468 2007 -->
RTServer turns "GRAY" in "ProcMon". Jane came to control room and instruction to restart ProcMon.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239485' target=_top>239485</a>) -->
<!-- Comment: Wed Apr 18 11;24;00 comment by...Shih-Chieh -->
<table border=0><td><tr><td bgcolor=#e0e030>The instruction of 
<a href=
http://www-cdfonline.fnal.gov/daq/online/check.html
"> start up ProcMon  <a> needs to up updated. 

It would be nice to add the case of "RTSERVER" turns "Gray" into 
the RTServer trouble shooting <a href="http://www-cdfonline.fnal.gov/ops/ace2help/procmon_smartsockets.html">page<a> .</td></tr></td></table>
<!-- Date: Wed Apr 18 10:02:52.975 2007 -->
pbars are in
<!-- Author: reb -->



<!-- Date: Wed Apr 18 10:21:39.010 2007 -->
Scraping complete at 10:20. Store # 5358
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486' target=_top>239486</a>) -->


<!-- Date: Wed Apr 18 10:22:10.069 2007 -->
replaced transition modules in the b0xft06 crate: slot 4 (working) -> slot 3 (not working), new TM into slot 4. 
<!-- Author: Andrew :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486' target=_top>239486</a>) -->


<!-- Date: Wed Apr 18 10:22:18.573 2007 -->
<TABLE bgcolor=#add8e6>
<TR><TH>Date</th><th>Time</th><th colspan=3>Quantity</th></tr>
<tr><td>2007.04.18</td><td>10:18:01</td><td>TEVpr</td><td>9703.477</td><td>E09</td></tr>
<tr><td>2007.04.18</td><td>10:18:01</td><td>TEVpb</td><td>2259.076</td><td>E09</td></tr>

<tr><td>2007.04.18</td><td>10:21:23</td><td>C:B0ILUM</td><td>212.266</td><td>E30</td></tr>
<tr><td>2007.04.18</td><td>10:21:23</td><td>C:B0LLUM</td><td>0.000</td><td>E30</td></tr>
<tr><td>2007.04.18</td><td>10:21:23</td><td>C:B0PLOS</td><td>254.042</td><td>Hz</td></tr>

<tr><td>2007.04.18</td><td>10:21:23</td><td>C:B0ALOS</td><td>11.547</td><td>Hz</td></tr>
</TABLE>
Luminosity and losses
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486' target=_top>239486</a>) -->

<!-- Date: Wed Apr 18 10:26:19.955 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486>239486</A>
 Activated at 2007.04.18 10:26:08
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 10:26:20.824 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486>239486</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: ACE -->


<!-- Date: Wed Apr 18 10:30:15.221 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486>239486</A>
 Terminated at 2007.04.18 10:29:59
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 10:30:58.756 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486>239486</A>
TERMINATE: restart due to SVX problem
<!-- Author: ACE -->
<!-- Comment: Wed Apr 18 10;48;00 comment by...Ignacio -->
The SVX data header gets sometimes corrupted somewhere between our VRB electronics and the event builder. This is a low frequency failure that shows clearly in SVXMon fairly early into the run, waiting 5 minutes is enough. The SVX cell-id  slide (numeber 2 in SVXMon) is completely yellow (and a bit of blue: the ignored channels). The required action is to start a new run.   
<!-- Date: Wed Apr 18 10:34:14.280 2007 -->
No svxmon data - end and restart run.
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->

<!-- Date: Wed Apr 18 10:34:46.712 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487>239487</A>
 Activated at 2007.04.18 10:34:15
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 10:34:47.651 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487>239487</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: ACE -->



<!-- Date: Wed Apr 18 10:41:21.005 2007 -->
TOF HV Inihibit [132] Time of Flight High Voltage from iFix but recovers immediately.   
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->


<!-- Date: Wed Apr 18 10:42:01.672 2007 -->
SVT_PULSAR_09 DT. Do HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->


<!-- Date: Wed Apr 18 10:43:00.184 2007 -->
Node b0pcal10 in Error "Heap is Corrupt". Shepherd and HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->


<!-- Date: Wed Apr 18 10:46:07.718 2007 -->
COT_02 DT. Auto HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->



<!-- Date: Wed Apr 18 11:05:28.432 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-3742&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-3742' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-3743&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-3743' width='300'></a></center>
<!-- Author: Shih-Chieh -->
<!-- Comment: Wed Apr 18 11;06;02 comment by...Shih-Chieh -->
loss/Abgap & Svt integration for startup of store 5358


<!-- Date: Wed Apr 18 11:07:21.845 2007 -->
10:50 pcal11 DT but Auto HRR failed. Shepherd & HRR works.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->

<!-- Date: Wed Apr 18 11:15:04.404 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487>239487</A>
general: <tt>All OK</tt>
<!-- Author: Consumer Operator( Mike Albrow) -->




<!-- Date: Wed Apr 18 11:51:38.816 2007 -->
Central arch SE top 2 channel 0 digital 5V was zero.  Cyro recycled power on that crate and it recovered.
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487' target=_top>239487</a>) -->

<!-- Date: Wed Apr 18 11:52:06.926 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487>239487</A>
 Terminated at 2007.04.18 11:51:52
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 11:52:28.234 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239487>239487</A>
TERMINATE: L2 TO after pb0pcal11 crate Power cycle. 
<!-- Author: Shih-Chieh -->
<!-- Comment: Wed Apr 18 12;12;03 comment by...Shih-Chieh -->
Typo. It should be b0ccal11
<!-- Date: Wed Apr 18 11:55:57.035 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239496>239496</A>
 Activated at 2007.04.18 11:55:40
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 11:55:58.103 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239496>239496</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 11:56:42.338 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239496>239496</A>
 Terminated at 2007.04.18 11:56:36
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 11:58:07.451 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239496>239496</A>
TERMINATE: L2 TO again
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:01:22.076 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239497>239497</A>
 Activated at 2007.04.18 12:00:32
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:01:23.466 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239497>239497</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:06:52.806 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239497>239497</A>
 Terminated at 2007.04.18 12:06:48
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:07:28.107 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239497>239497</A>
TERMINATE: end because we don't get SMXR data
<!-- Author: Shih-Chieh -->



<!-- Date: Wed Apr 18 12:07:43.035 2007 -->
starting a short run to diagnose the L2 timeout problem.
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239497' target=_top>239497</a>) -->

<!-- Date: Wed Apr 18 12:11:25.535 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239498>239498</A>
 Activated at 2007.04.18 12:11:05
<!-- Author: RunControl -->



<!-- Date: Wed Apr 18 12:13:36.986 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239498>239498</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] This run is ignoring smx
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:14:53.622 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239498>239498</A>
 Terminated at 2007.04.18 12:14:28
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:14:54.585 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239498>239498</A>
TERMINATE: same problem
<!-- Author: Shih-Chieh -->
<!-- Comment: Wed Apr 18 12;18;46 comment by...gene -->
<pre>Ignored reces to which crate/slot fails, turns out ccal11 slot 5 is the problem.
ended run and excluded this board from the crate. Now restarting run to see if it runs. Experts are being paged</pre>

<!-- Date: Wed Apr 18 12:18:53.001 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239499>239499</A>
 Activated at 2007.04.18 12:18:50
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:20:37.706 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239499>239499</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] Slot 5 ccall 11 is excluded
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:20:38.440 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239499>239499</A>
 Terminated at 2007.04.18 12:20:07
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:20:56.039 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239499>239499</A>
TERMINATE: ending run due to l2 time outs coming from reces again
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:21:43.468 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239500>239500</A>
TERMINATE: test
<!-- Author: Shih-Chieh -->



<!-- Date: Wed Apr 18 12:22:51.667 2007 -->
Problem traced to an SMXR problem - perhaps related to the power cycling of the crate.  We tried the pager and couldn't get Karen - tired her office - no luck.  We finally got Mike Lindgren's cell - he will come in.
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239499' target=_top>239499</a>) -->
<!-- Comment: Wed Apr 18 12;36;47 comment by...byrum -->
I was on the phone and couldn't call back till now.  I suggest do whatever Mike says. 
<!-- Date: Wed Apr 18 12:24:22.819 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239501>239501</A>
 Activated at 2007.04.18 12:24:05
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:24:23.061 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239501>239501</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] l2 ignoring reces
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:26:37.965 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239501>239501</A>
 Terminated at 2007.04.18 12:25:18
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:26:38.779 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239501>239501</A>
TERMINATE: still more problems
<!-- Author: Shih-Chieh -->



<!-- Date: Wed Apr 18 12:29:31.212 2007 -->
Mike recommends that we try cycling power again on the crate that failed (SE top 2).
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239501' target=_top>239501</a>) -->

<!-- Date: Wed Apr 18 12:33:30.687 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239502>239502</A>
 Activated at 2007.04.18 12:32:52
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:33:31.628 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239502>239502</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 12:34:22.224 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239502>239502</A>
 Terminated at 2007.04.18 12:33:59
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:34:23.419 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239502>239502</A>
TERMINATE: same problem
<!-- Author: Shih-Chieh -->




<!-- Date: Wed Apr 18 12:38:51.698 2007 -->
Cycling 400Hz power on the SE arch - this is what Mike and Karen tell us is needed to restore operarionson.
<!-- Author: reb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239502' target=_top>239502</a>) -->

<!-- Date: Wed Apr 18 12:49:10.204 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503>239503</A>
 Activated at 2007.04.18 12:48:28
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 12:49:11.096 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503>239503</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Shih-Chieh -->



<!-- Date: Wed Apr 18 12:50:21.521 2007 -->
Eventually the power down of the arch 400Hz left us in a consistent state with frontend all working.
<!-- Author: reeb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503' target=_top>239503</a>) -->
<!-- Comment: Wed Apr 18 13;05;50 comment by...reb -->
powerdown and up ;->

<!-- Comment: Wed Apr 18 13;31;08 comment by...Shih-Chieh -->
<table border=0><td><tr><td bgcolor=#e0e030>It's my bad to not look for instructions from Ace help.
The persistent "L2 done timeout" could be triggered by SMXR card and is documented in 
<a href="http://www-cdfonline.fnal.gov/ops/ace2help/smxr/notresponding.html">SMXR  trouble shooting</a>.
<br>The correct debugging procedure should be: 
<br>1. Power Cycle Crate 
<br>2. Diagonize L2 Done timeout source from "Error Handler" 
<br>3. If L2DT comes from SMX, Power Cycly whole rack.</td></tr></td></table>
<!-- Comment: Thu Apr 19 00;24;13 comment by...Shih-Chieh -->
The similar situation happened on <a href="https://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=Thu_Dec_14_2006_Owl&anchor=03:12:14.765">Owl shift Dec 14 2006</a>
<!-- Comment: Thu Apr 19 09;59;10 comment by...gregf -->
<pre>One probably shouldn't power cycle the arch without
talking with Karen or Mike!  Thnaks.</pre>
<!-- Date: Wed Apr 18 13:17:02.553 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503>239503</A>
general: <tt>All OK</tt>
<!-- Author: Consumer Operator( Mike Albrow) -->


<!-- Date: Wed Apr 18 13:30:53.049 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486>239486</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=239486 target=_runSummary>239486</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->




<!-- Date: Wed Apr 18 13:47:17.279 2007 -->
b0fib07 shephered sa requested by Silicon SPL to solve the discripency of SVXMon ISL plots to rererence.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503' target=_top>239503</a>) -->
<!-- Comment: Wed Apr 18 14;15;00 comment by...Jose/Ignacio -->
Dqmon noticed that one DAQ ISL ladder was not giving back data.
We tried to reset the individual VRB board but L3 came to a strange situation so we had to ask for a new run. Now the ladder is back.

<!-- Date: Wed Apr 18 13:49:19.929 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239486>239486</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVX svxmon shows VRB and event builder not talking 
ISL 
L00 
SVT 
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Wed Apr 18 13:55:12.762 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503>239503</A>
 Terminated at 2007.04.18 13:55:00
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 13:55:42.244 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503>239503</A>
TERMINATE: L3 is not output data after HRR due to Silicon SPL requests
<!-- Author: Shih-Chieh -->


<!-- Date: Wed Apr 18 14:01:29.525 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504>239504</A>
 Activated at 2007.04.18 14:00:38
<!-- Author: RunControl -->


<!-- Date: Wed Apr 18 14:01:30.117 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504>239504</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_02 [3,873,681] 
<!-- Author: Shih-Chieh -->



<!-- Date: Wed Apr 18 14:05:28.187 2007 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre> 
I have moved UPSLUMIMAX from 90 to 110 E30.  
This means the UPS trigger should kick in @ 110 E30. 
There will be a slightly larger (~10 kHz) jump in L1 rate at that time. 
Be on the lookout for any suspicious trigger behaviour. 
</pre></td></tr></td></table>
<!-- Author: Dave Waters :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->


<!-- Date: Wed Apr 18 14:08:01.208 2007 -->
"CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  Stuck Cellid S/B2/W7/L1/C0-2 ". auto HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->



<!-- Date: Wed Apr 18 14:24:34.971 2007 -->
Busy Timeout. auto HRR works.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->


<!-- Date: Wed Apr 18 14:35:54.922 2007 -->
b0pcal05 DT. Shepherd and HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->


<!-- Date: Wed Apr 18 14:45:41.737 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-3744&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-3744' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-3745&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-3745' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-3747&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-3747' width='300'></a></center>
<!-- Author: Shih-Chieh -->
<!-- Comment: Wed Apr 18 14;48;55 comment by...Shih-Chieh -->
Auto scrape book doesn't work. Had been paged Steve to solve this problem. I just did eye watch to make sure fast time hourly plot is fine and worked on the SMXR crate power cycle issue. Here's the three ACNet plots since start of the store 5358.


<!-- Date: Wed Apr 18 15:06:30.118 2007 -->
"CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  Stuck Cellid S/B3/W9/L3/C0-5". Auto HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->
<!-- Comment: Wed Apr 18 15;20;08 comment by...Shih-Chieh -->
again
<!-- Comment: Wed Apr 18 15;30;07 comment by...Shih-Chieh -->
again
<!-- Comment: Wed Apr 18 15;59;44 comment by...Shih-Chieh -->
again
<!-- Date: Wed Apr 18 15:10:39.941 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239503>239503</A>
general: <tt>All OK</tt>
<!-- Author: Consumer Operator( Mike Albrow) -->



<!-- Date: Wed Apr 18 15:14:28.258 2007 -->
<p>Restarted the CDF sequencer which went down 1842 last night. 
Ace complained that CopyToFile shortcut on desktop was not 
working, which surprised me since I've never used this (I'd 
thought it was only to put plots in the AD e-logs, but it does 
copy the files to C:\Graphics_Storage).  Apparently, the reason 
it is not working is because the PrintKey2000 program which is 
uses is not working on CNS52PC; it is working on all the other 
ACnet PCs.  I think a reboot is in order, but I'll hold off on 
that till betweeen stores.</p> 
 
<p>Also, Bob Blair pointed out the Global Alarm for CSP actually 
says "CES alarm".  I'll check with J.C. Yun on this one.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->
<!-- Comment: Wed Apr 18 15;32;12 comment by...reb -->
Actually the CSP says CPR.
<!-- Date: Wed Apr 18 15:21:18.308 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-3748&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-3749  width='300' align='top'></a></td><td>Unmark on Bias B0W5L3 doesn't work - need to ignore.</td></tr></table>
<!-- Author: reb -->




<!-- Date: Wed Apr 18 15:36:41.466 2007 -->
FrontEnd Crate Error Condition from LEVEL2_PULSAR_02. Auto HRR works.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->


<!-- Date: Wed Apr 18 15:45:52.901 2007 -->
b0cot16 DT. Spherphed and HRR.
<!-- Author: Shih-Chieh :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=239504' target=_top>239504</a>) -->

<!-- Date: Wed Apr 18 15:55:29.851 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=239486 target=_runSummary>239486</A> x3A77E</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>10:26:08</TD>
<TD ALIGN=CENTER>10:29:59</TD>
<TD ALIGN=CENTER>00:02:59</TD>
<TD ALIGN=RIGHT>      19,858</TD>   <TD ALIGN=RIGHT>     49.60</TD>
<TD ALIGN=RIGHT>     33.96</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=239487 target=_runSummary>239487</A> x3A77F</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>10:34:15</TD>
<TD ALIGN=CENTER>11:51:52</TD>
<TD ALIGN=CENTER>00:55:28</TD>
<TD ALIGN=RIGHT>     422,252</TD>   <TD ALIGN=RIGHT>    848.38</TD>
<TD ALIGN=RIGHT>    612.49</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=239499 target=_runSummary>239499</A> x3A78B</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>12:18:50</TD>
<TD ALIGN=CENTER>12:20:07</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT>     16.46</TD>
<TD ALIGN=RIGHT>      0.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=239503 target=_runSummary>239503</A> x3A78F</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>12:48:28</TD>
<TD ALIGN=CENTER>13:55:00</TD>
<TD ALIGN=CENTER>00:58:04</TD>
<TD ALIGN=RIGHT>     405,543</TD>   <TD ALIGN=RIGHT>    540.72</TD>
<TD ALIGN=RIGHT>    470.12</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=239504 target=_runSummary>239504</A> x3A790</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [3,873,681]</FONT></TD>
<TD ALIGN=CENTER>14:00:38</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>01:42:38</TD>
<TD ALIGN=RIGHT>     755,330</TD>   <TD ALIGN=RIGHT>    783.32</TD>
<TD ALIGN=RIGHT>    708.68</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>03:39:10</TD>
<TD ALIGN=RIGHT>     1,602,983</TD> <TD ALIGN=RIGHT>  2,238.48</TD>
<TD ALIGN=RIGHT>  1,825.25</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Wed Apr 18 16:05:46.931 2007 -->
<font color=red>Shift Summary: </font><pre>Day shift 4/18/2007 
------------------ 
o Store dropped at start of shift - setup to proceed immediately 
o Store 5358 luminosity 212E30 in at 10:22 
   - first run needed to be restarted because 
     svxmon not getting data 
o PSM alarm - Central arch SE top 2 digital 5V failed. 
   - Power cycling the crate brought it back to 4.8V 
o L2 timeouts - the real problem was SMD to SMXR 
  communication broken by crate 
  power cycle - resolution was to power cycle the SE arch 400Hz 
   - took some time to locate an expert to walk us through this 
o Si folks try to resolve ISL problem which causes an L3  
  problem and requires starting a new run 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   239486 239487 239499 239503 239504  
Delivered Luminosity   2927.0 nb-1  
Acquired Luminosity    1861.6 nb-1  
Efficiency             63.6
   
</td></tr></table></pre>
<!-- Author: reb -->