<!-- Title: Day shift. Tue Jun 17, 2008 -->
<!-- SciCo: Paul Shepard -->
<!-- DAQAce: Chang-Seong Moon -->
<!-- MonAce: David Cox -->
<!-- CO: Veronica Sorin -->
<!-- OpManager: Massimo Casarsa -->
<!-- Notes: -->
<pre>We are between stores.  Shot setup for store 6228 in progress</pre>

<!-- Date: Tue Jun 17 08:14:15.845 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263581>263581</A>
 Activated at 2008.06.17 08:14:02
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 08:14:16.878 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263581>263581</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,1023,706] Quiet Time
<!-- Author: jacob/David -->


<!-- Date: Tue Jun 17 08:24:39.855 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263581>263581</A>
 Terminated at 2008.06.17 08:23:59
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 08:28:39.504 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263583>263583</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:28:48.351 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263583>263583</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Tue Jun 17 08;31;06 comment by...Veronica -->
Calibration OK.
<!-- Date: Tue Jun 17 08:29:16.376 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263583>263583</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->



<!-- Date: Tue Jun 17 08:31:38.091 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263584>263584</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:31:59.015 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263584>263584</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:34:10.532 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263585>263585</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:35:35.806 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263585>263585</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:36:01.975 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263585>263585</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Tue Jun 17 08;36;59 comment by...Veronica -->
Calibration ok.
<!-- Date: Tue Jun 17 08:36:48.809 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263584>263584</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Tue Jun 17 08;37;50 comment by...Veronica -->
Calibration ok.
<!-- Date: Tue Jun 17 08:37:37.769 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263586>263586</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:37:38.014 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263586>263586</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:37:50.902 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263586>263586</A>
CALIBRATION: <U><B>cot</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cot target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Tue Jun 17 08;38;29 comment by...Veronica -->
Calibration OK.

<!-- Date: Tue Jun 17 08:40:55.103 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263587>263587</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:42:14.150 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263587>263587</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:42:15.869 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263587>263587</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Tue Jun 17 08;44;22 comment by...Veronica -->
Calibration Ok.

<!-- Date: Tue Jun 17 08:45:11.897 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263588>263588</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:45:43.239 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263588>263588</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:47:17.736 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263589>263589</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:53:23.459 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263589>263589</A>
TERMINATE: Done
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 08:53:37.398 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263589>263589</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Tue Jun 17 08;54;33 comment by...Veronica -->
Calibration OK.

<!-- Date: Tue Jun 17 08:56:41.819 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263590>263590</A>
 Activated at 2008.06.17 08:56:34
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 08:56:42.547 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263590>263590</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,1019,706] 
<!-- Author: Chang-Seong/David -->
<!-- Comment: Tue Jun 17 09;03;59 comment by...Chang-Seong/David -->
<pre>The rate of L1_TWO_CJET0.5{3] Trigger is 775.4 Hz.

The rate of L1_TWO_JET1.5[2] Trigger is 150.2 Hz.

The rate of MB_XING{1] Trigger is 1,715,266.0 Hz.</pre>

<!-- Date: Tue Jun 17 09:05:04.399 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263590>263590</A>
 Terminated at 2008.06.17 09:04:09
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 09:05:05.927 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263590>263590</A>
TERMINATE: Preparing for Shot Setup run
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 09:07:54.187 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263591>263591</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_5_01 [7,1088,721] 
<!-- Author: Chang-Seong/David -->



<!-- Date: Tue Jun 17 09:08:16.056 2008 -->
Injecting p bars.  Si and OPS manager paged.
<!-- Author: shepard :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263591' target=_top>263591</a>) -->

<!-- Date: Tue Jun 17 09:09:40.015 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263591>263591</A>
TERMINATE: Going to Cosmics for Jonathan
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 09:15:33.687 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263593>263593</A>
 Terminated at 2008.06.17 09:15:29
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 09:17:00.639 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263593>263593</A>
TERMINATE: Going to Shotsetup
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 09:19:11.209 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263594>263594</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_5_01 [7,1088,721] 
<!-- Author: Chang-Seong/David -->



<!-- Date: Tue Jun 17 09:22:22.907 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3058&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3058  width='300' align='top'></a></td><td>Scalarmon display</td></tr></table>
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 09:32:14.710 2008 -->
Scraping has begun.
<!-- Author: Shepard :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263594' target=_top>263594</a>) -->


<!-- Date: Tue Jun 17 09:33:25.425 2008 -->
  Done TO. for crate/s: b0pcal11. Sheperded.   
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263594' target=_top>263594</a>) -->

<!-- Date: Tue Jun 17 09:34:35.713 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263594>263594</A>
TERMINATE: Scraping begun.
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 09:54:48.227 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595>263595</A>
 Activated at 2008.06.17 09:54:24
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 09:54:49.609 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595>263595</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [7,1088,721] 
<!-- Author: Chang-Seong/David -->



<!-- Date: Tue Jun 17 09:56:17.596 2008 -->
MCR calls at 09:42 to report scraping complete.  We turn CLC and call MCR back at 09:45
<!-- Author: Shepard :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595' target=_top>263595</a>) -->


<!-- Date: Tue Jun 17 09:59:11.381 2008 -->
Done Timeout detected in crates: b0wcal06. 
Auto HRR fails. 
Repeated HRR fails. 
Sheperding fails.  
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595' target=_top>263595</a>) -->


<!-- Date: Tue Jun 17 09:59:16.891 2008 -->
Error found by Pulsa Monitor. There were saveral red spot. We page the expert. He let us whet crate make the problem. We shepherd wcal04, wcal06, ccal08, ccal10, ccal12, ccal13.
<!-- Author: Chang-Seong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595' target=_top>263595</a>) -->

<!-- Date: Tue Jun 17 10:08:26.854 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595>263595</A>
 Terminated at 2008.06.17 10:08:16
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 10:08:51.607 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263595>263595</A>
TERMINATE: For problem wcal06
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 10:12:19.849 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263596>263596</A>
 Activated at 2008.06.17 10:12:08
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 10:12:23.707 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263596>263596</A>
 Terminated at 2008.06.17 10:12:18
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 10:12:58.344 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263596>263596</A>
TERMINATE: aborted
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 10:12:59.237 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263596>263596</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [7,1088,721] 
<!-- Author: Chang-Seong/David -->



<!-- Date: Tue Jun 17 10:20:03.229 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3064&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3064  width='300' align='top'></a></td><td>SVRAD from Shotsetup 
AbortGap from Shotsetup</td></tr></table>
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 10:20:58.766 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3065&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3065  width='300' align='top'></a></td><td>AbortGap</td></tr></table>
<!-- Author: Chang-Seong/David -->


<!-- Date: Tue Jun 17 10:45:31.078 2008 -->
We have been unable to get a run going.  First, ADMEM board for TOF was not working.  It has been dropped (6 channels of TOF), 2nd, Si crate was not working.  Turned out to be connection with RTF server. 3rd, event builder is hosed.  Trying to do clean up.  Have paged evb expert.
<!-- Author: shepard -->
<!-- Comment: Tue Jun 17 11;18;44 comment by...shepard -->
This collection of problems has cost approximately 1 pb^-1 of beam.
Running now without TOF ADMEM board.
<!-- Date: Tue Jun 17 10:50:53.370 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
 Activated at 2008.06.17 10:50:35
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 10:51:04.337 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [7,1088,721] 
<!-- Author: Chang-Seong -->



<!-- Date: Tue Jun 17 10:51:07.076 2008 -->
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! b0pcal02 Sheperded.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 10:53:11.800 2008 -->
TOF bad in DQMon (probably because b0wcal06 slot 8 is disabled in database)
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 10:54:26.278 2008 -->
 Done TO. for crate/s: b0pcal10 Sheperded.  
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 10:57:25.207 2008 -->
PEM HV turned to yellow. We keep taking data.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 11:10:55.796 2008 -->
b0pcal11 in error. Sheperded.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 11:17:37.821 2008 -->
pcal02 isn't responsive. Shepherd fixed it.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->



<!-- Date: Tue Jun 17 11:22:26.955 2008 -->
PreFred counts are are still very high.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 11:37:04.983 2008 -->
bocot06 unresponsive. Shepherding successful
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 11:38:49.915 2008 -->
Still seeing very low rate of errors from L2 clustering pass 15 (tau's).  Examining a little further I find that we are very occasionally getting one more tau cluster than the simulation expects, so this is not biasing the trigger at all.  I will look a little deeper to see if there have been recent changes to the trigger library.
<!-- Author: Dan Krop :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->
<!-- Comment: Tue Jun 17 15;05;34 comment by...Dan -->
pass 15 is actually cone 0.4 jets, not taus.
<!-- Date: Tue Jun 17 11:43:25.906 2008 -->
Here is the sad saga of the start of this super store (S5): 
<ul> 
<li>9:42 Problems partitioning.  Ace tries shepherding.  Error is 
<tt>AM->ADMEM_setCafeDelay geo32:/slot=8 VME bus error</tt>.  There are problems at each transition and the done time-outs after activate.  Shepherding is attempted repeatedly including under direction of the pager carrier. 
<li>9:54 Activate (for naught) 
<li>10:08 Expert on her way.  End to attempt to take out WCAL06 and attempt to redownload Admem before she arrives. 
<li>10:10  Doh!  Slot 8 might be a TOF admem 
<li>10:11  It is.  Try to a run with that card excluded.  
<li>10:14  Can't partition.  b0svx00 is not responsive.  Try to connect to serial port.  It's dead.  Sasha goes downstairs to reboot it. 
<li>10:18 Still can't partition.  RTServer port is stuck for b0svx00.  Restart it.   
<li>10:30 Still can't parition EVB is stuck.  Check EVB.  Proxies are dead.  Cleanup.  Still bad.  Can't connect to b0evme17.  That is the SCPU in the svx00 crate.  We reset it. 
<li>10:35 Partition still failing in EVB.  Check fails for SCPU.  Call Conor.  While he is on the phone, the SCPU wakes up and the check passes. 
<li> 10:48 Partition is successful. 
</ul> 
I have marked the WCAL06, slot 8 offline.  That diables 9 PMTs for TOF.  TOF data is bad (per Bruno) for this store.  After the store, Elisa needs to reload the Admem.  Somebody with HWDB access should then mark the card back online.   
<p>Another problem that I noticed is that the instructions for loading the Admem are on Mark Matson's <a href=http://www-cdf.fnal.gov/internal/people/links/MarkMattson/admem_expert/welcome.html>private web page</a>.  They should be on the online web server linked from the calorimeter operations page which also should be on the online web server.  We need to have access to documentation even if links out of B0 are down. 
<p>Downtime: 27 min Admem, 19 min Front-end CPU, 18 min EVB. 
<p>Thanks to Sasha, Donatella and Elisa for working through this mess.
<!-- Author: jdl -->
<!-- Comment: Tue Jun 17 12;40;31 comment by...Elisa (cal admem) -->
<pre>Procedures for reconfiguring TOF ADMEM is slightly different (but similar) to reconfiguring CAL ADMEM. Since we aren't sure how long a reconfiguration takes for TOF, decided with scico and ops manager that we should wait till end of store to reconfigure. Bruno (TOF) will do the reconfig, since it is a TOF board and slightly different than calorimeter, and I will assist, since it is similar.</pre>
<!-- Date: Tue Jun 17 11:54:59.097 2008 -->
  Done TO. for crate/s: b0pcal02. Sheperding successful.   
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->

<!-- Date: Tue Jun 17 11:56:24.980 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
central: <tt>TOF marked bad (admem board out of RC configuration) see comments in the e-log.</tt>
<!-- Author: Consumer Operator( Veronica Sorin) -->


<!-- Date: Tue Jun 17 11:56:25.231 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
general: <tt>Overall fine. BMU and TOF marked bad. The BMU is the same problem as previous rus, for TOF is due to the admem we took out the run.</tt>
<!-- Author: Consumer Operator( Veronica Sorin) -->


<!-- Date: Tue Jun 17 11:56:26.649 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
muons: <tt>BMU marked bad, due to west channels off (see e-log, same problem as yesterday).</tt>
<!-- Author: Consumer Operator( Veronica Sorin) -->



<!-- Date: Tue Jun 17 12:00:20.115 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3068&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3069  width='300' align='top'></a></td><td>TOF showing channels off. See comments above.</td></tr></table>
<!-- Author: Veronica -->


<!-- Date: Tue Jun 17 12:08:08.407 2008 -->
  Done TO. for crate/s: b0pcal10. Sheperding successful
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->



<!-- Date: Tue Jun 17 12:41:44.190 2008 -->
We have a plan for reloading the TOF ADMEM card.  When we know approximately when the store will end, we will page both the TOF and ADMEM pager carriers and let them know (let's say one to two hours in advance).  Then we will page them again when the store actually ends so they can come in and reload the TOF ADMEM card.  If the store goes down suddenly in an unexpected way, we will also page the experts and execute the reload.  These instructions have been posted to the white board for the next shifts.
<!-- Author: Shepard :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 13:12:09.074 2008 -->
  Done TO. for crate/s: b0imu00. Auto HRR.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->
<!-- Comment: Tue Jun 17 13;27;30 comment by...Chang-Seong/David -->
Again. With error.
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done  for crate b0imu00 
AUTO HRR
<!-- Comment: Tue Jun 17 14;33;49 comment by...Chang-Seong/David -->
Again. No FERML_TDC_NOT_READY_GLOBAL message.
<!-- Date: Tue Jun 17 13:14:36.748 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3070&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3071  width='300' align='top'></a></td><td>This one has been creeping up.</td></tr></table>
<!-- Author: shepard -->
<!-- Comment: Tue Jun 17 13;16;42 comment by...shepard -->
It has gone off again so I have marked it ignore.  This time it stays red


<!-- Date: Tue Jun 17 13:21:18.363 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-3072&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-3073  width='300' align='top'></a></td><td>Mazel Tov Grandma and Grandpa Hahn, here is Sadie Camille</td></tr></table>
<!-- Author: Larry N -->


<!-- Date: Tue Jun 17 13:46:49.033 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
central: <tt>TOF bad due to the admem that was taken of the configuration (it needs to be reloaded, see comments in the e-log)</tt>
<!-- Author: Consumer Operator( Veronica Sorin) -->


<!-- Date: Tue Jun 17 13:46:50.376 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
muons: <tt>BMU bad due to channels off , west side (check comments in e-log about bmu).</tt>
<!-- Author: Consumer Operator( Veronica Sorin) -->


<!-- Date: Tue Jun 17 13:46:51.824 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
general: <tt>Overall fine, BMU and TOF marked bad for the run.</tt>
<!-- Author: Consumer Operator( Veronica Sorin) -->



<!-- Date: Tue Jun 17 13:47:28.513 2008 -->
Silicon problem. Silicon recovery successful.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->


<!-- Date: Tue Jun 17 13:49:49.168 2008 -->
D0 has requested 1-1.5 hours access after the store.
<!-- Author: Phil -->
<!-- Comment: Tue Jun 17 14;47;01 comment by...Sebastian -->
The estimated time for Store termination is between 2200 to 0000hrs - from runco.
<!-- Comment: Tue Jun 17 15;49;50 comment by...Massimo -->
We also will have a 1 h access to work on BMU.
<!-- Date: Tue Jun 17 14:43:21.607 2008 -->
  Done TO. for crate/s: b0cot16. Auto HRR.
<!-- Author: Chang-Seong/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->




<!-- Date: Tue Jun 17 15:19:11.680 2008 -->
<b>Prefred Scalers</b><p> 
With Carla's help, we have gone a long way toward unravelling the mystery of the PreFred Scalers.  The symptom that we observe is that the unused Fred Inputs are all counting at a high rate.  The counts are synchronous.  The Inputs are actually the inputs to the Fred RAMs and are the output of the Crossbow chips that take the 64 Prefred output and map them to the Fred inputs.  In the past, we believe that these scalers read 0.  We do not find any other strange behavior, and all of the Trigmon checks for L1 triggers look OK.  Nevertheless, new unexplained behavior is a source of concern. 
<p> 
Carla has checks in Trigmon that compare the trigger info at the output of Prefred with the inputs to Fred and also counts the number of times each of these is true in the analyzed events.  The L1 info is found in the output files: <tt>/cdf/onln/data/cdfdaq/consumer/results/TrigMon/L1TriggerSummary*.dat</tt> where the star can be replaced by a run number.  We see that there are counts there in Fred for the unused lines.  We also see that there are the same number of counts for the single High-pt CMU track bit(i.e. 4 GeV) for the Muon Match box.  That is Prefred bit 29.  This same behavior is observed in the Trigmon output for many runs.  Carla has spot-checked runs in the past.  The same effect is seen in the PHYSICS_5_01_v5 table in May and even in a run in January with the v3 table.   
<p> 
So, the effect is not new.  It just appears that since there have not been any hot trigger problems, nobody has looked at Prefred rates in ScalerMon with beam for quite a while.  Donatella pulled up the code that turns the Trigger DB information into the 32 32-bit words that are used to configure the Crossbows on the board.  We will try to understand the code.  It can be found in the online CVS at <tt>/TriggerDB/sources/triggertable/run/Fred.java</tt> 
Looking at the actual data that is written in the config message, nothing is obvious.   
<p> 
I do know that in run 161011 for which I have a screen capture of ScalerMon, there was no such problem.  In that run, we had a L1_CMU0_PT4 trigger.  Now we need to isolate the problem to some time in between.
<!-- Author: jdl -->


<!-- Date: Tue Jun 17 15:55:28.482 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=263597 target=_runSummary>263597</A> x405AD</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [7,1088,721]</FONT></TD>
<TD ALIGN=CENTER>10:50:35</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>04:27:15</TD>
<TD ALIGN=RIGHT>   2,203,149</TD>   <TD ALIGN=RIGHT>  2,987.32</TD>
<TD ALIGN=RIGHT>  2,621.14</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>04:27:15</TD>
<TD ALIGN=RIGHT>     2,203,149</TD> <TD ALIGN=RIGHT>  2,987.32</TD>
<TD ALIGN=RIGHT>  2,621.14</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Jun 17 15:55:34.405 2008 -->
<font color=red>Shift Summary: </font><pre>08:00  Store 6227 was terminated normally at 07:30 hours.  We are in shot setup for store 6228.  
 
11:00 Lost the first hour of the store (~1 pb^-1) due to an assortment of problems. 1st, TOF ADMEM board caused problems.  We have dropped this board and will reload it between stores.  RTF server had to be rebooted to make proper connections with an Si crate.  Event builder crashed and had to be cleaned up and restarted. 
 
15:20 After the poor startup, the shift has been quiet. We are currently at 62% efficiency for run 263597.  Store will end between 2200 and 0000 hours.  Plan to do ADMEM TOF reload at that time.  See log or white board for instruction on who to call when. When this run ends mark BMU and TOF bad.  We expect to do an access as well to bix BMU. 
 
Begin shift or beam at 09:43:54 
End shift or beam at 15:51:55 
Delivered luminosity: 3963.49 
Acquired luminosity: 2610.24 
Totals 
Date:	2008.06.17 
Shift:	day 
Delivered luminosity:	3963.5 nb-1 
Acquired luminosity:	2610.2 nb-1 
Efficiency:	65.9 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   263597
Delivered Luminosity   3963.5  
Acquired Luminosity    2610.2  
Efficiency             65.9
   
</td></tr></table></pre>
<!-- Author: shepard -->

<!-- Date: Tue Jun 17 15:56:55.233 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263059>263059</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=263059 target=_runSummary>263059</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->


<!-- Date: Tue Jun 17 16:02:30.921 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597>263597</A>
general: <tt>Everything is fine except the same persisting issues on BMU west &nbsp;dead channels &nbsp;and TOF off-channels. </tt>
<!-- Author: Consumer Operator( Zeynep Unalan) -->



<!-- Date: Tue Jun 17 16:04:21.670 2008 -->
TOF HV Trip (also detected by iFix: red bards and missing left bar). Trigger ihibit set but immediately cleared itself w/o any action was taken.
<!-- Author: Simone :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=263597' target=_top>263597</a>) -->