<!-- Title: Day shift. Mon Sep 29, 2008 -->
<!-- SciCo: Andreas Warburton -->
<!-- DAQAce: David Mietlicki -->
<!-- MonAce: &nbsp; -->
<!-- CO: Javier Cuevas -->
<!-- OpManager: Jen Pursley -->
<!-- Notes: -->
<pre>Store 6459 nearing its end; Run 267449 in progress; RR 268+ E10</pre>

<!-- Date: Mon Sep 29 08:22:44.174 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267449>267449</A>
 Terminated at 2008.09.29 08:22:38
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 08:22:57.418 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267449>267449</A>
TERMINATE: End of store
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 08;23;40 comment by...Dave -->
Leaving CLC on for studies
<!-- Comment: Mon Sep 29 08;26;02 comment by...Dave -->
L3 Relay Failure Error while ending run, L3 recovered by itself and run ended.
<!-- Date: Mon Sep 29 08:23:26.252 2008 -->
MCR called to say they're ready for EOS studies.  They ask that we bring down our HV, except for CLC.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267449' target=_top>267449</a>) -->




<!-- Date: Mon Sep 29 08:33:56.268 2008 -->
At End of Store state, I am getting a warning (yellow in iFix) from the PHA HV.  This is the same crate (crate 5) that gave us a warning last week when HV was on, but it is now also giving a waning in standby.  I checked with Jen, she said to just make a note in the elog, since the crate is a known problem.
<!-- Author: Dave -->

<!-- Date: Mon Sep 29 08:37:19.222 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267450>267450</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 08:38:22.347 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267450>267450</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 08;49;15 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 08:38:23.179 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267450>267450</A>
TERMINATE: Done
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 08:38:23.566 2008 -->
With Jen's approval, I will run non-quiet time calibrations (except CEM LED and CEM XEF, which require CAL QIE first) while studies are conducted, then will run AAA_SHOTSETUP until beam is dumped.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267450' target=_top>267450</a>) -->

<!-- Date: Mon Sep 29 08:42:45.546 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267451>267451</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 08:42:46.952 2008 -->
b0pcal11 got stuck during "Setup" transition, had to shepherd
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267451' target=_top>267451</a>) -->

<!-- Date: Mon Sep 29 08:44:09.070 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267451>267451</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 08:46:51.559 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267452>267452</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 08;47;03 comment by...Dave -->
Done

<!-- Date: Mon Sep 29 08:48:59.580 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267453>267453</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 08:49:15.679 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267453>267453</A>
TERMINATE: Done
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 08:50:54.898 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267451>267451</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 08;59;49 comment by...javier cuevas -->
smxqie_3.gif  tell us to page expert
<!-- Comment: Mon Sep 29 09;04;42 comment by...A. Warburton -->
Expert has been paged.
<!-- Comment: Mon Sep 29 09;11;07 comment by...A. Warburton -->
Mike Lindgren responded.  One channel in pcal07 is the culprit.  Recommends (1) run calibration again, see if that fixes it; if not, (2) ask cryo tech to power cycle pcal07, recalibrate, see if that fixes it; if not, (3) ask cryo tech to cycle 400Hz of the entire east plug, recalibrate.
<!-- Comment: Mon Sep 29 09;26;50 comment by...A. Warburton -->
Second calibration resulted in the same problem.  We asked the Cryo Tech to power cycle b0pcal07.
<!-- Date: Mon Sep 29 08:51:23.708 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267454>267454</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 08:51:58.948 2008 -->
Done TO in wcal01, but it turned back to green on its own.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267454' target=_top>267454</a>) -->
<!-- Comment: Mon Sep 29 08;53;31 comment by...Dave -->
Turned back to red, then failed two HRRs.  I halted and then shepherded it.
<!-- Comment: Mon Sep 29 08;55;27 comment by...Dave -->
First shepherd failed, I am trying to shepherd again.
<!-- Comment: Mon Sep 29 09;01;01 comment by...Dave -->
Second shepherd also failed.  We paged expert, while we are waiting I will abort the run and get ready for quiet time calibrations.
<!-- Comment: Mon Sep 29 09;17;21 comment by...Dave -->
Shalhout called and said that since b0wcal01 made it through other calibrations ok, he doesn't think it's a problem, but to try the TOFTAC calibration again if we get the chance
<!-- Date: Mon Sep 29 08:53:23.210 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267452>267452</A>
CALIBRATION: <U><B>cot</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cot target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;01;52 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 08:53:25.389 2008 -->
MCR called to say that their studies are complete.  They are proceeding to shot setup and intend to give us a courtesy call prior to injection to help us get in our quiet-time calib's.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267454' target=_top>267454</a>) -->




<!-- Date: Mon Sep 29 09:01:01.996 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267454>267454</A>
CALIBRATION: <U><B>toftac</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->



<!-- Date: Mon Sep 29 09:01:27.586 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267454>267454</A>
TERMINATE: Did not finish due to b0wcal01 problem
<!-- Author: Dave -->




<!-- Date: Mon Sep 29 09:02:09.794 2008 -->
Paged WHA readout expert at Ace request.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267454' target=_top>267454</a>) -->
<!-- Comment: Mon Sep 29 09;05;40 comment by...A. Warburton -->
ADMEM expert responded.
<!-- Date: Mon Sep 29 09:02:59.276 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267455>267455</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:03:49.495 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267455>267455</A>
TERMINATE: Done
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 09:05:04.810 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267455>267455</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;12;46 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 09:07:39.323 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267456>267456</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:08:32.318 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267456>267456</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:08:33.707 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267456>267456</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Mon Sep 29 09:09:47.220 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267457>267457</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:10:20.408 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267457>267457</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:10:53.139 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267457>267457</A>
CALIBRATION: <U><B>clcqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#clcqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;14;31 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 09:13:36.117 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267458>267458</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:14:21.133 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267458>267458</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:14:22.239 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267458>267458</A>
CALIBRATION: <U><B>bscqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#bscqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;15;37 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 09:14:57.893 2008 -->
MCR called to warn that they will inject beam in ~1 minute.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267458' target=_top>267458</a>) -->



<!-- Date: Mon Sep 29 09:17:16.644 2008 -->
Bill Badgett will be power cycling some pcal crates; will coordinate with the Ace's current activities to avoid any problems.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267459' target=_top>267459</a>) -->
<!-- Comment: Mon Sep 29 09;19;53 comment by...badgett -->
Just rebooting, not power cycling
<!-- Date: Mon Sep 29 09:17:30.986 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267459>267459</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:18:25.434 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267459>267459</A>
TERMINATE: Done
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 09:18:32.444 2008 -->
I would like to re-disable the Level 2 memory  
cache on the B0PCAL01,02,04,05,07,08,10,11 crate CPUs. 
Over the last month they have been running in "enable L2  
cache" mode in order to test the rate of CPU hang-ups  
with an entirely MVME2400 ensemble of crate CPUs.    
Disabling the L2 memory cache on the PCAL MVME2400s reduces  
the rate of CPU hang-ups by a modest 35% or so.  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267459' target=_top>267459</a>) -->


<!-- Date: Mon Sep 29 09:21:59.857 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267460>267460</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 09;23;04 comment by...Dave -->
b0wcal01 now is not having any problems
<!-- Comment: Mon Sep 29 09;27;02 comment by...Dave -->
Had to restart due to Bill's work
<!-- Date: Mon Sep 29 09:22:23.104 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267459>267459</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;24;20 comment by...javier cuevas -->
now smxqie_1.gif empty smxqie_3.gif ok but smxqie_4.gif is BAD and tell us to page expert

<!-- Date: Mon Sep 29 09:26:13.187 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267461>267461</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Dave -->




<!-- Date: Mon Sep 29 09:29:52.413 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267460>267460</A>
CALIBRATION: <U><B>toftac</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->



<!-- Date: Mon Sep 29 09:30:06.735 2008 -->
Done TO in b0wcal04, fixed by HRR
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267461' target=_top>267461</a>) -->

<!-- Date: Mon Sep 29 09:32:47.676 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267461>267461</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:34:25.045 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267463>267463</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:35:25.722 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267463>267463</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:35:41.378 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267449>267449</A>
pr: <tt>phi wedges 7 and 8 blank (west part) </tt>
<!-- Author: Consumer Operator( javier cuevas) -->


<!-- Date: Mon Sep 29 09:35:42.444 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267449>267449</A>
general: <tt>check done at the end of run 267449</tt>
<!-- Author: Consumer Operator( javier cuevas) -->


<!-- Date: Mon Sep 29 09:36:37.871 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267464>267464</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:38:07.697 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267464>267464</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:38:40.264 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267464>267464</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;42;04 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 09:40:41.864 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267463>267463</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;43;55 comment by...javier cuevas -->
the two plots looks ok, and no warning about anything BAD.
<!-- Comment: Mon Sep 29 09;44;24 comment by...lindgren -->
<pre>looks good</pre>
<!-- Comment: Mon Sep 29 09;45;16 comment by...Dave -->
Power cycling of b-pcal07 appears to have fixed the problem.
<!-- Date: Mon Sep 29 09:41:44.531 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267465>267465</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 09:43:38.940 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267465>267465</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:43:39.392 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267465>267465</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;49;46 comment by...javier cuevas -->
topright plot (PMT counts) shows varios rows of empty bins
both in east and west sides
<!-- Date: Mon Sep 29 09:46:19.746 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267466>267466</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 09;47;14 comment by...Dave -->
TOF TAC calibration results did not appear, so I am running it one more time.


<!-- Date: Mon Sep 29 09:52:15.150 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267466>267466</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 09:52:46.048 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267466>267466</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 09;59;14 comment by...javier cuevas -->
ok
<!-- Date: Mon Sep 29 09:54:23.739 2008 -->
We tried to turn on CMX SW Wedge 15, which was giving problems last shift with trips.  It turned on and stayed on without a problem so we will include it at start of next store.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267466' target=_top>267466</a>) -->
<!-- Comment: Mon Sep 29 10;58;39 comment by...John Strologas -->
Good job!
<!-- Date: Mon Sep 29 10:07:11.925 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267468>267468</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: TEST_FUPS [1,1131,706] 
<!-- Author: donatella -->
<!-- Comment: Mon Sep 29 10;40;15 comment by...Donatella -->
took two quick FUPS test runs with Sparky. Restored everything for regular data taking.
<!-- Date: Mon Sep 29 10:08:38.960 2008 -->
Shot setup is underway, and Donatella is running some more FUPS table tests.
<!-- Author: Jen -->


<!-- Date: Mon Sep 29 10:28:45.471 2008 -->
Final protons loading.
<!-- Author: A. Warburton -->


<!-- Date: Mon Sep 29 10:44:03.701 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267470>267470</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_5_01 [10,1139,728] 
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 10:46:19.900 2008 -->
Antiprotons being injected; will contact Silicon.
<!-- Author: A. Warburton -->
<!-- Comment: Mon Sep 29 10;48;48 comment by...A. Warburton -->
Paged ops manager.


<!-- Date: Mon Sep 29 10:54:25.731 2008 -->
Carla recommends we page Level 2; Run 267470 is showing bad.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267470' target=_top>267470</a>) -->
<!-- Comment: Mon Sep 29 10;56;11 comment by...A. Warburton -->
L2 paged.

<!-- Comment: Mon Sep 29 10;58;21 comment by...A. Warburton -->
Charles Cox (L2 expert) is attending in the control room.



<!-- Comment: Mon Sep 29 11;11;57 comment by...David Cox (L2 Expert) -->
Actually David Cox.
<!-- Comment: Mon Sep 29 11;12;10 comment by...Charles -->
Actually that's David Cox.
<!-- Comment: Mon Sep 29 11;42;17 comment by...Laura  -->
During shot setup L2 cluster monitoring was indicating several mismatches for both pass12 and pass10. No errors coming from the hardware, it was a software item. 
The problem disappeared starting a new physics run.
I suspect sometime TrigMon fails in shot setup configuration.
<!-- Date: Mon Sep 29 11:06:13.781 2008 -->
Antiproton bunches have been loaded.
<!-- Author: A. Warburton -->


<!-- Date: Mon Sep 29 11:11:27.271 2008 -->
Busy TO, fixed by auto HRR
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267470' target=_top>267470</a>) -->




<!-- Date: Mon Sep 29 11:26:39.413 2008 -->
Store 6460, scraping complete at 11:24 CT.  TevPR: 9680E09; TevPB: 2200E09; B0ILUM: 246E30.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471' target=_top>267471</a>) -->


<!-- Date: Mon Sep 29 11:27:15.271 2008 -->
Heartbeat alarm from Icicle, but Icicle is still updating, so I will not restart it yet.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471' target=_top>267471</a>) -->
<!-- Comment: Mon Sep 29 12;01;51 comment by...Dave -->
Alarm has cleared
<!-- Date: Mon Sep 29 11:29:30.106 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471>267471</A>
 Activated at 2008.09.29 11:29:17
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 11:29:31.566 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471>267471</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [10,1139,728] 
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 11:29:44.278 2008 -->
IMON turned red in ProcMon, but immediately turned back to green upon a second check
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471' target=_top>267471</a>) -->


<!-- Date: Mon Sep 29 11:32:49.331 2008 -->
<center><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-5048&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-5048' width='300'></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-5049&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-5049' width='300'></a></center>
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 11;33;02 comment by...Dave -->
Store 6460 shot setup



<!-- Date: Mon Sep 29 11:44:21.651 2008 -->
L2 Decision TO, fixed by auto HRR
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471' target=_top>267471</a>) -->


<!-- Date: Mon Sep 29 11:50:54.844 2008 -->
At request of jdl, I will halt run and shepherd b0l2pu00,03,04,05.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471' target=_top>267471</a>) -->


<!-- Date: Mon Sep 29 12:00:37.121 2008 -->
Busy TO, auto HRR issued.  b0l2pu05 got stuck duing Halt transition, had to be shepherded.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471' target=_top>267471</a>) -->


<!-- Date: Mon Sep 29 12:32:10.203 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471>267471</A>
 Terminated at 2008.09.29 12:32:04
<!-- Author: RunControl -->
<!-- Comment: Mon Sep 29 12;33;48 comment by...Dave -->
Ended run at Bill Badgett's request to attempt to deal with L2 cal problems.


<!-- Date: Mon Sep 29 12:36:06.436 2008 -->
Backing out recent TL1D pulsar readout code changes  
to see if that matteres.    This goes from v6_49  
to v6_46 of the fer readout code.   Starting a new  
run.
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->

<!-- Date: Mon Sep 29 12:37:16.533 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472>267472</A>
 Activated at 2008.09.29 12:37:01
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 12:37:17.095 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472>267472</A>
 Activated at 2008.09.29 12:37:02
<!-- Author: RunControl -->
<!-- Comment: Mon Sep 29 12;37;51 comment by...Dave -->
Error in b0pcal06 during Config transition, had to be shepherded
<!-- Date: Mon Sep 29 12:37:18.519 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472>267472</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [10,1139,728] 
<!-- Author: Dave -->




<!-- Date: Mon Sep 29 12:48:52.767 2008 -->
Still saw errors from b0l2pu05.  Tried power cycling.
<!-- Author: jdl :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->


<!-- Date: Mon Sep 29 12:49:12.769 2008 -->
<pre>Mixed bag on b0l2pu05 reboot: 
 
Message from RTserver: Connection established.Start subscribing to subject &lt;/_b0l2pu05_517176848&gt;. 
Subscribed to &gt;&gt;&gt;/setPartition/frontEnd/level2/pulsar/05&lt;&lt;&lt; 
message length corrupt deadbeef 
Connection 127.0.0.1:1029 aborted, status is -1 
value = 0 = 0x0 
FER_setTaskPriority("rtlm_session",189); 
*** FER_setTaskPriority: failed to find task rtlm_session [S_taskLib_NAME_NOT_FOUND] *** 
value = 1 = 0x1 
 
For rtlm_session task not to exist is very unusual... 
</pre>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->


<!-- Date: Mon Sep 29 12:50:07.318 2008 -->
Sporadic rtserver errors on b0l2pu05: 
 
b0l2pu05-> WARNING: Lost connection to RTlm: error code = 514. 
Attempting to reconnect to RTlm. 
Connecting to <tcp:localhost> RTlm. 
Using tcp protocol. 
WARNING: Could only recv -1/8 first bytes from connection client:tcp:localhost:5111. 
Error was <timeout reached>. 
Could not connect to <tcp:localhost> RTlm. 
Skipping starting <start_never:tcp:localhost> RTlm. 
Cannot reconnect to RTlm. 
Cannot continue. 
Found CPU Type 80020 with bus speed 30.000000 ns 
received connection from 127.0.0.1:1036, socket is 4 
msg_recv: bad size -1 reading length from socket 4, data 0, type -10 
msg_recv: Connection reset by peer 
error -1 receiving Connection Init message from 127.0.0.1:1036 
Pipe device /pipe/FER_closePipe already exists, skipping pipeDevCreate 
UNIVERSE_U2SPEC addr   0x830004FC 
UNIVERSE_U2SPEC data   0x00000000 
Found Tracer in slot 2, mapped TRACER_vmeAddr 0xEFF00000 
Connecting to <tcp:localhost> RTlm. 
Using tcp protocol. 
received connection from 127.0.0.1:1037, socket is 20 
Message from RTlm: Connection established. 
WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING: Internal validation failed, file: ipcmsg.c, line 1314Message from RTlm: License for <RTclient> granted (132 licenses left). 
Connecting to project <cdf_daq> on <tcp:b0server05> RTserver. 
Using tcp protocol. 
Message from RTserver: Connection established.Start subscribing to subject </_b0l2pu05_514222448>. 
Subscribed to >>>/setPartition/frontEnd/level2/pulsar/05<<<  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->
<!-- Comment: Mon Sep 29 12;50;33 comment by...badgett -->
<pre>b0l2pu05-&gt; WARNING: Lost connection to RTlm: error code = 514.
Attempting to reconnect to RTlm.
Connecting to &lt;tcp:localhost&gt; RTlm.
Using tcp protocol.
WARNING: Could only recv -1/8 first bytes from connection client:tcp:localhost:5111.
Error was &lt;timeout reached&gt;.
Could not connect to &lt;tcp:localhost&gt; RTlm.
Skipping starting &lt;start_never:tcp:localhost&gt; RTlm.
Cannot reconnect to RTlm.
Cannot continue.
Found CPU Type 80020 with bus speed 30.000000 ns
received connection from 127.0.0.1:1036, socket is 4
msg_recv: bad size -1 reading length from socket 4, data 0, type -10
msg_recv: Connection reset by peer
error -1 receiving Connection Init message from 127.0.0.1:1036
Pipe device /pipe/FER_closePipe already exists, skipping pipeDevCreate
UNIVERSE_U2SPEC addr   0x830004FC
UNIVERSE_U2SPEC data   0x00000000
Found Tracer in slot 2, mapped TRACER_vmeAddr 0xEFF00000
Connecting to &lt;tcp:localhost&gt; RTlm.
Using tcp protocol.
received connection from 127.0.0.1:1037, socket is 20
Message from RTlm: Connection established.
WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING: Internal validation failed, file: ipcmsg.c, line 1314Message from RTlm: License for &lt;RTclient&gt; granted (132 licenses left).
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0server05&gt; RTserver.
Using tcp protocol.
Message from RTserver: Connection established.Start subscribing to subject &lt;/_b0l2pu05_514222448&gt;.
Subscribed to &gt;&gt;&gt;/setPartition/frontEnd/level2/pulsar/05&lt;&lt;&lt;
</pre>


<!-- Date: Mon Sep 29 12:53:06.968 2008 -->
<pre>Now there are two rtlm_session tasks on b0l2pu05: 
 
b0l2pu05-&gt; i 
 
  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY 
---------- ------------ -------- --- ---------- -------- -------- ------- ----- 
tExcTask   excTask      1effb050   0 PEND         2e76c8 1effaf30  3d0001     0 
tNetTask   netTask      1edb9010   1 PEND         2da0a0 1edb8f20       0     0 
tPortmapd  portmapd     1ed95810   1 PEND         2da0a0 1ed955b0  3d0002     0 
tShell     shell        1ed8a600   5 READY        2df9fc 1ed8a1c0   30065     0 
tRlogind   rlogind      1ed923c0  15 PEND         2da0a0 1ed91f80       0     0 
t1         VISIONserver 1ed7e220 100 PEND         2da0a0 1ed7e050       0     0 
t2         ROBINserver  1ed79060 100 PEND         2da0a0 1ed78e90       0     0 
Readout    FER_readout2 1ea98b90 199 READY        2da0a0 1ea988b0  3d0002     0 
Watchdog   FER_watchdog 1ed84e90 200 PEND+T       2e76c8 1ed84d30  3d0004    39 
Messenger  FER_messenge 1ea66970 200 PEND+T       2da0a0 1ea666e0  3d0002     1 
rtlm_main  rtlm_main    1ed73ea0 220 PEND         2da0a0 1ed73cc0       0     0 
Mon_III    FER_monitorI 1e8ddcb0 220 DELAY        2df320 1e8ddbe0       0    19 
rtlm_sessiortlm_session 1ebf3570 225 PEND         2da0a0 1ebf32c0       0     0 
rtlm_sessiortlm_session 1e8f0ee0 225 PEND         2da0a0 1e8f0c30       0     0 
tLogTask   logTask      1eff8450 250 PEND         2e76c8 1eff8340       0     0 
value = 0 = 0x0 
</pre>
<!-- Author: bb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->

<!-- Date: Mon Sep 29 12:53:51.911 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472>267472</A>
 Terminated at 2008.09.29 12:53:42
<!-- Author: RunControl -->
<!-- Comment: Mon Sep 29 13;01;09 comment by...Dave -->
Starting another new run to try to clear up L2 cal problems
<!-- Date: Mon Sep 29 12:54:26.325 2008 -->
<pre>At end of run got errors: 
 
b0l2pu05-&gt; TRACER_enableDone: Front End Fatal Error (-1): Not setting Tracer done 
Nonsequential event number, prev evt: 11, 1 try: 14, 2 try: 14 
Nonsequential event number, prev evt: 11, 2 try: 14, 3 try: 14 
Nonsequential event number, prev evt: 11, 3 try: 14, 4 try: 14 
Nonsequential event number, prev evt: 11, 4 try: 14, 5 try: 14 
Nonsequential event number, prev evt: 11, 5 try: 14, 6 try: 14 
Nonsequential event number, prev evt: 11, 6 try: 14, 7 try: 14 
Nonsequential event number, prev evt: 11, 7 try: 14, 8 try: 14 
Nonsequential event number, prev evt: 11, 8 try: 14, 9 try: 14 
Nonsequential event number, prev evt: 11, 9 try: 14, 10 try: 14 
Runtime Error 1, Event 92032: EventID not sequential: old=11 , new=14 
 
 
Yikes!!!!</pre>
<!-- Author: bb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->


<!-- Date: Mon Sep 29 12:55:35.537 2008 -->
restarted rtserver and then rebooted b0l2pu05;  worth  
a try , but not clear if it will help...  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->


<!-- Date: Mon Sep 29 12:56:51.766 2008 -->
<pre>At end of b0l2pu05 reboot: 
 
Using tcp protocol. 
received connection from 127.0.0.1:1025, socket is 17 
Message from RTlm: Connection established. 
WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING: Internal validation failed, file: ipcmsg.c, line 1314Message from RTlm: License for &lt;RTclient&gt; granted (132 licenses left). 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0server05&gt; RTserver. 
Using tcp protocol. 
Message from RTserver: Connection established.Start subscribing to subject &lt;/_b0l2pu05_517176848&gt;. 
Subscribed to &gt;&gt;&gt;/setPartition/frontEnd/level2/pulsar/05&lt;&lt;&lt; 
value = 0 = 0x0 
FER_setTaskPriority("rtlm_session",189); 
FER_setTaskPriority: set task rtlm_session priority to 189 
value = 0 = 0x0 
 
 
Done executing startup scr 
ipt boot/front-end/startup.5500.cdf 
b0l2pu05-&gt; Sent ack to Run Control: 0 IDLE SUCCESS  
 
 
Note that there is no "rtlm_session" error this time. 
Good sign? hmm.</pre>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472' target=_top>267472</a>) -->

<!-- Date: Mon Sep 29 12:59:44.873 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473>267473</A>
 Activated at 2008.09.29 12:59:31
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 12:59:45.499 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473>267473</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [10,1139,728] 
<!-- Author: Dave -->




<!-- Date: Mon Sep 29 13:09:19.343 2008 -->
The L1TriggerMonitor and 125MeV Occupancy TrigMon plots  
seem to be back in order for this run.     We suspect  
that the data might be OK.   Trigger team will investigate  
offline.
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473' target=_top>267473</a>) -->


<!-- Date: Mon Sep 29 13:21:57.317 2008 -->
L2 Cal problems are resolved.  The original problem seems to be a problem with the fer code that makes TC2D in b0l2pu00, 03 and 05.  The pulsar fer code was updated to fix the TL1D readout in bol2pu07.  The code is common to the crates, howeever, so something got hammered in the TC2D construction.  Bill has backed out to the previous version of fer.  There also developed an rtserver problem with b0l2pu05.  That actually showed up when we did the first shepherding of the crate since that crate would perform a little differently than the others.  In all cases, it would lose the rtserver connection and when the connection was re-established, the TC2D readout was lost.  Restarting the rtserver, shepherding and starting a new run fixed that. 
<p> 
All indications from L1 and L2 occupancies and trigger rates are that the trigger was working correctly for runs 267471 and 72.  However, the calorimeter trigger monitorring was screwed up.  I don't know if anybody uses TC2D for physics analysis.  That is bad for the two runs.  We'll have to do a little more investigation to determine the quality of the runs (850/nb).  In the meantime, the scico should mark the runs bad for L2 and RSF. 
<p> 
Thanks to all investigators: Bill, Olga, Tom, Gene, Laura, Dan, Charles, and of course Carla who blew the whistle.  Don't leave us!
<!-- Author: jdl -->

<!-- Date: Mon Sep 29 13:35:21.961 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267471>267471</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L2T 
RSF 
</PRE>
<!-- Author: cdfscico -->
<!-- Comment: Tue Sep 30 11;02;42 comment by...A. Warburton -->
Marked good per request by Dan Krop.
<!-- Date: Mon Sep 29 13:36:29.792 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267472>267472</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L2T 
RSF 
</PRE>
<!-- Author: cdfscico -->
<!-- Comment: Tue Sep 30 11;02;56 comment by...A. Warburton -->
Marked good per request by Dan Krop.
<!-- Date: Mon Sep 29 13:50:40.819 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473>267473</A>
general: <tt>looks ok.</tt>
<!-- Author: Consumer Operator( javier cuevas) -->



<!-- Date: Mon Sep 29 13:57:23.769 2008 -->
Done TO for b0plca11.  Two HRRs did not fix the problem so I halted the run and shepherded it, which solved the problem.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473' target=_top>267473</a>) -->


<!-- Date: Mon Sep 29 14:16:12.951 2008 -->
Disk Servers turned red in ProcMon.  Following instructions, I first waited a few minutes and then checked again.  The check was ok and Disk Servers is now back to green.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473' target=_top>267473</a>) -->


<!-- Date: Mon Sep 29 14:18:36.744 2008 -->
IMON also turned red in ProcMon, but turned back to green immediately after performing a second check.
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473' target=_top>267473</a>) -->

<!-- Date: Mon Sep 29 14:41:38.441 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473>267473</A>
 Terminated at 2008.09.29 14:41:34
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 14:41:53.756 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267473>267473</A>
TERMINATE: Quench
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 14:43:08.780 2008 -->
Tev quench, B4; early indication is a power interruption.
<!-- Author: A. Warburton :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267474' target=_top>267474</a>) -->


<!-- Date: Mon Sep 29 14:44:10.960 2008 -->
Silicon Coldstart is ok
<!-- Author: Dave :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267474' target=_top>267474</a>) -->


<!-- Date: Mon Sep 29 14:46:11.311 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-5050&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-5050  width='300' align='top'></a></td><td>Store 640 abort</td></tr></table>
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 14;46;27 comment by...Dave -->
That should be store 6460

<!-- Date: Mon Sep 29 14:51:27.774 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267475>267475</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 14:51:58.084 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267475>267475</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 14:52:41.890 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267475>267475</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;07;18 comment by...j. cuevas -->
ok
<!-- Date: Mon Sep 29 14:53:12.653 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267476>267476</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 14:54:07.380 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267476>267476</A>
CALIBRATION: <U><B>clcqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#clcqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;08;49 comment by...j. cuevas -->
ok
<!-- Date: Mon Sep 29 14:54:08.518 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267476>267476</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 14:56:57.398 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267477>267477</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 14:57:18.032 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267477>267477</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 14:57:19.027 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267477>267477</A>
CALIBRATION: <U><B>bscqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#bscqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;09;23 comment by...j. cuevas -->
ok

<!-- Date: Mon Sep 29 15:11:37.841 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267479>267479</A>
 Activated at 2008.09.29 15:11:30
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 15:11:46.654 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267479>267479</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,1130,706] 
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 15;12;06 comment by...Dave -->
Frank Chlebana is trying to add some L3 nodes back in

<!-- Date: Mon Sep 29 15:12:41.501 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267479>267479</A>
 Terminated at 2008.09.29 15:12:33
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 15:14:40.959 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267480>267480</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:15:09.093 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267480>267480</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:15:10.258 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267480>267480</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;16;42 comment by...j. cuevas -->
ok


<!-- Date: Mon Sep 29 15:17:12.254 2008 -->
<pre>Added back in a total of 11 recovered L3 nodes. 
Took a test run without problems. 
Some of the recovered nodes were returned from repair by D1. 
 
ooops... s03p21 just gave a relay error...  
so one did not survive very long.</pre>
<!-- Author: chlebana -->

<!-- Date: Mon Sep 29 15:18:58.648 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267481>267481</A>
ACTIVATE: RunSet: BOTHPLUGS_NORMAL 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:25:21.865 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267481>267481</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:27:10.466 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267482>267482</A>
ACTIVATE: RunSet: BOTHPLUGS_MM_C 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:28:55.751 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267482>267482</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:30:06.983 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267482>267482</A>
CALIBRATION: <U><B>pluglaser</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#pluglaser target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;33;35 comment by...j. cuevas -->
ok
<!-- Date: Mon Sep 29 15:30:07.648 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267483>267483</A>
ACTIVATE: RunSet: BOTHPLUGS_MM 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:31:38.421 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267483>267483</A>
TERMINATE: Done
<!-- Author: Dave -->



<!-- Date: Mon Sep 29 15:34:19.084 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267484>267484</A>
 Activated at 2008.09.29 15:34:08
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 15:34:20.673 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267484>267484</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,1134,706] 
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 15;34;38 comment by...Dave -->
No beam

<!-- Date: Mon Sep 29 15:43:41.118 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267484>267484</A>
 Terminated at 2008.09.29 15:43:34
<!-- Author: RunControl -->


<!-- Date: Mon Sep 29 15:43:42.072 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267484>267484</A>
TERMINATE: Done
<!-- Author: Dave -->
<!-- Comment: Mon Sep 29 15;44;36 comment by...Dave -->
L1A rate 53.3 Hz

<!-- Date: Mon Sep 29 15:45:38.102 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267485>267485</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:46:28.982 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267485>267485</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:48:52.279 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267486>267486</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:49:29.676 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267486>267486</A>
TERMINATE: Done
<!-- Author: Dave -->


<!-- Date: Mon Sep 29 15:49:30.711 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267486>267486</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;52;29 comment by...j. cuevas -->
ok
<!-- Date: Mon Sep 29 15:50:08.231 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267485>267485</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;52;54 comment by...j. cuevas -->
ok
<!-- Date: Mon Sep 29 15:51:40.215 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267487>267487</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Yoshikazu -->




<!-- Date: Mon Sep 29 15:54:31.995 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267487>267487</A>
CALIBRATION: <U><B>cot</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cot target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 15;55;35 comment by...j. cuevas -->
ok
<!-- Date: Mon Sep 29 15:54:32.353 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267488>267488</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Yoshikazu -->


<!-- Date: Mon Sep 29 15:55:10.285 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=267449 target=_runSummary>267449</A> x414B9</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [10,1139,728]</FONT></TD>
<TD ALIGN=CENTER>06:07:30</TD>
<TD ALIGN=CENTER>08:22:38</TD>
<TD ALIGN=CENTER>02:07:37</TD>
<TD ALIGN=RIGHT>     683,484</TD>   <TD ALIGN=RIGHT>    612.57</TD>
<TD ALIGN=RIGHT>    574.30</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=267471 target=_runSummary>267471</A> x414CF</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [10,1139,728]</FONT></TD>
<TD ALIGN=CENTER>11:29:17</TD>
<TD ALIGN=CENTER>12:32:04</TD>
<TD ALIGN=CENTER>00:52:50</TD>
<TD ALIGN=RIGHT>     479,177</TD>   <TD ALIGN=RIGHT>    833.65</TD>
<TD ALIGN=RIGHT>    690.56</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=267472 target=_runSummary>267472</A> x414D0</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [10,1139,728]</FONT></TD>
<TD ALIGN=CENTER>12:37:02</TD>
<TD ALIGN=CENTER>12:53:42</TD>
<TD ALIGN=CENTER>00:13:15</TD>
<TD ALIGN=RIGHT>      93,806</TD>   <TD ALIGN=RIGHT>    194.54</TD>
<TD ALIGN=RIGHT>    150.94</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=267473 target=_runSummary>267473</A> x414D1</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [10,1139,728]</FONT></TD>
<TD ALIGN=CENTER>12:59:31</TD>
<TD ALIGN=CENTER>14:41:34</TD>
<TD ALIGN=CENTER>01:35:11</TD>
<TD ALIGN=RIGHT>     852,551</TD>   <TD ALIGN=RIGHT>  1,005.95</TD>
<TD ALIGN=RIGHT>    939.15</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:00</TD>
<TD ALIGN=CENTER>04:48:54</TD>
<TD ALIGN=RIGHT>     2,109,018</TD> <TD ALIGN=RIGHT>  2,646.72</TD>
<TD ALIGN=RIGHT>  2,354.96</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Mon Sep 29 15:55:55.877 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267488>267488</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Mon Sep 29 16;25;07 comment by...Chunlei -->
checked, but seems many new dead channels in the west tower. According to  the prevous elogs, seems it has been there for sometime.
<!-- Date: Mon Sep 29 15:55:58.862 2008 -->
<font color=red>Shift Summary: </font><pre>08:23: Store 6459 ended for physics. 
 
08:50: SMX QIE calibration failed; power cycle b0pcal07 fixed it. 
 
08:51: Done TO in b0wcal01; two HRRs and two shepherds did not fix; expert Shalhout advised that subsequent calibrations ok. 
 
08:53: AD EoS studies complete. 
 
09:18: Bill Badgett re-disabled the Level 2 memory cache on several b0pcalxx crate CPUs. 
 
09:54: Test HV ramping of CMX SW Wedge 15 succeeded (had been tripping on earlier shifts). 
 
10:40: Donatella took two quick FUPS table test runs. 
 
10:54: Carla G.-P. alerts us to a L2 Cal problem.  Several experts investigated.  Culprit was fer code confounded by an rtserver problem.  See JDL's elog entry for the full glory. 
 
11:24: Store 6460 began with B0ILUM of 246E30.  First two runs of the store had L2T and RSF bits set bad due to L2 problem above. 
 
14:42: Tev quench at B4 due to power interruption. 
 
15:17: Frank Chlebana added in 10 recovered L3 nodes. 
 
Shift ended with AD still investigating their power situation. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   267449, 267471, 267472, 267473
Delivered Luminosity   2403.4 nb-1  
Acquired Luminosity    1857.9 nb-1  
Efficiency             77.3
   
</td></tr></table></pre>
<!-- Author: A. Warburton -->

<!-- Date: Mon Sep 29 15:58:25.956 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267489>267489</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Yoshikazu -->


<!-- Date: Mon Sep 29 16:00:27.267 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=267490>267490</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Yoshikazu -->



<!-- Date: Mon Sep 29 16:01:20.052 2008 -->
<font color=red>Shift Summary: </font><pre>Formal end of shift summary</pre> 
<!-- Author: P Derwent -->