<!-- Title: Owl shift. Sat Sep  9, 2006 -->
<!-- SciCo: Robin Erbacher -->
<!-- DAQAce: Roberto Martinez Ballarin -->
<!-- MonAce: Shalhout Shalhout -->
<!-- CO: James Bellinger -->
<!-- OpManager: Koji Sato -->
<!-- Notes: -->
<pre>Shot setup in 10 minutes.  Yay!</pre>

<!-- Date: Sat Sep 9 00:12:38.220 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222776>222776</A>
 Activated at 2006.09.09 00:10:49
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 00:12:39.547 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222776>222776</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,639] 
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 00:24:47.238 2006 -->
MCR Called, going into shot setup.  Aces began checklist. 
<!-- Author: rde :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222776' target=_top>222776</a>) -->


<!-- Date: Sat Sep 9 01:00:12.374 2006 -->
MCR still testing small amts of protons in the buckets-- no real proton loading happening yet.  
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222776' target=_top>222776</a>) -->

<!-- Date: Sat Sep 9 01:11:41.629 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222776>222776</A>
 Terminated at 2006.09.09 01:11:30
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 01:11:54.577 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222776>222776</A>
TERMINATE: final protons are going in
<!-- Author: Roberto -->
<!-- Comment: Sat Sep  9 01;16;29 comment by...RDE -->
Final protons being loaded. 

<!-- Date: Sat Sep 9 01:17:34.328 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222777>222777</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_00 [4,820,642] 
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 01:18:25.249 2006 -->
<pre> Done TO. for crate/s:  
    b0l1cl02    
 
Auto HRR</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222777' target=_top>222777</a>) -->
<!-- Comment: Sat Sep  9 01;18;51 comment by...Shalhout -->
<pre>once again</pre>

<!-- Date: Sat Sep 9 01:19:05.872 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
clc: <tt>222728 ~10 min run.  Checked long after, so HV status not known.  </tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:06.834 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
fwd: <tt>Short run. Checked later, so no HV info.</tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:07.721 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
si: <tt>Short run checked later, so no HV info.  Agrees with reference pretty <br>
much.  Not sure what level of "agreement" is required. </tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:08.620 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
central: <tt>HV unknown.  COT 1d occupancy for SL5 seems to have gap about 10 cells <br>
around number 100.</tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:09.458 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
cal: <tt>HV unknown. One white cell in tower 2 East.  I assume that means that <br>
not both PMTs are dead, as the criteria require for bad. 4 new white <br>
cells in PHA -11 West and 11 East, but not adjacent.  WHFA PMT shows <br>
many white cells, but only 265 events.</tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:10.092 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
smx: <tt>HV unknown. CES strip plots have many white cells, but only 265 events. <br>
PES East probably has more than 9 consecutive white cells, but only 265 <br>
events.  Probably PES West also.  PRT is okay. </tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:11.550 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
muons: <tt>short run checked later so no HV info</tt>
<!-- Author: Consumer Operator( richard lander) -->


<!-- Date: Sat Sep 9 01:19:13.150 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728>222728</A>
trigger: <tt>No HV info.  Low statistics</tt>
<!-- Author: Consumer Operator( richard lander) -->



<!-- Date: Sat Sep 9 01:24:47.180 2006 -->
Injecting pbars.
<!-- Author: RDE :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222777' target=_top>222777</a>) -->
<!-- Comment: Sat Sep  9 01;31;12 comment by...rde -->
Stash at ~317.  Stack at ~147.  They will shoot just from the stash due to upper limit of 350 (according to MCR elog entry earlier).


<!-- Date: Sat Sep 9 01:53:15.393 2006 -->
Pbars look loaded.  Paged silicon.
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222777' target=_top>222777</a>) -->

<!-- Date: Sat Sep 9 02:02:34.685 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222777>222777</A>
TERMINATE: start of scraping
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 02:05:45.475 2006 -->
Silicon expert monitoring from home, will call back with "ok" when ready.
<!-- Author: rde :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778' target=_top>222778</a>) -->


<!-- Date: Sat Sep 9 02:14:38.424 2006 -->
Dick Lander checked *short* run 222728 which was left over from day shift.  It was tough to tell things due to low stats.  COT is marked bad by CO but likely due to known COT TDC problems that Eiko noted earlier, so we (Robin/Todd) marked it good anyway as has been customary today while she's monitoring things.  Run 222729 looks also short and hasn't been checked by a CO yet, so I won't do good run checklist until it has been marked at least.  
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222728' target=_top>222728</a>) -->


<!-- Date: Sat Sep 9 02:20:59.605 2006 -->
On step one! 
 
MCR called, scraping done.  Ramping CLC. 
<!-- Author: rde :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778' target=_top>222778</a>) -->
<!-- Comment: Sat Sep  9 02;23;20 comment by...Robin -->
Initial lumi at 209.73, ACEs working on prescale settings.  Called MCR and told them we are up.  


<!-- Comment: Sat Sep  9 02;47;35 comment by...Shalhout -->
<pre>prescale for L1 bit 45 set to zero;

dynamic prescales disabled;

L2_CMU_*(53,54,55) and L2_CMU1.5_PT2....PS2(43) set to 200.</pre>
<!-- Comment: Sat Sep  9 02;50;17 comment by...rde -->
Silicon expert ok'd ramping silicon.  Ramped everything.  Troubles starting a physics run, see below.
<!-- Date: Sat Sep 9 02:28:01.563 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10797&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10797  width='300' align='top'></a></td><td>losses</td></tr></table>
<!-- Author: Shalhout -->

<!-- Date: Sat Sep 9 02:33:55.112 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778>222778</A>
 Activated at 2006.09.09 02:32:56
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 02:33:56.524 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778>222778</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [6,824,646] 
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 02:36:53.481 2006 -->
<pre>Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Pipeline out of synch in 115 silicon readout chips .  
 AUTO HRR will be issued   
 
Done Timeout detected in crates: 
LEVEL1_CAL_02 
 
the auto HRR took care of it</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778' target=_top>222778</a>) -->


<!-- Date: Sat Sep 9 02:37:33.163 2006 -->
<pre>Done Timeout detected in crates: 
LEVEL1_CAL_02 
 
auto HRR</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778' target=_top>222778</a>) -->


<!-- Date: Sat Sep 9 02:40:35.155 2006 -->
<pre>clock errors 371, 742, 1113 were cleared</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778' target=_top>222778</a>) -->


<!-- Date: Sat Sep 9 02:41:42.373 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10798&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10798  width='300' align='top'></a></td><td>pink cell unmarked to 'green'</td></tr></table>
<!-- Author: Shalhout -->


<!-- Date: Sat Sep 9 02:48:40.630 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222729>222729</A>
monitors: <tt>Low statistics run, so can't check all tests</tt>
<!-- Author: Consumer Operator( James Bellinger) -->


<!-- Date: Sat Sep 9 02:50:17.184 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778>222778</A>
 Terminated at 2006.09.09 02:49:41
<!-- Author: RunControl -->



<!-- Date: Sat Sep 9 02:50:32.046 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778>222778</A>
TERMINATE: reseated b0l1cl02.
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 02:52:34.784 2006 -->
ACEs couldn't get L1Cl02 to cooperate after numerous HRRs (see their entry).  I called Vadim per white board instructions.  He doesn't know what's going on with that board-- Carla reseated it earlier and it was ok.  I went over and reseated it (Koji already knew where it was) and reconnected the trigger cables (no bent pins).  Koji/ACEs shepharded crate, then had DTO, then HRR worked finally.  Koji still on with Vadim-- report to come.
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222778' target=_top>222778</a>) -->
<!-- Comment: Sat Sep  9 02;53;20 comment by...rde -->
seem to be taking data now

<!-- Comment: Sat Sep  9 03;01;50 comment by...Robin -->
Turns out Carla didn't just reseat board earlier, it was replaced, so likely not a board problem.  Vadim and I suspect something else with the crate or cable infrastructure-- bent backplane pin unlikely because messing with the board/cables got it to work for now, and usually when a pin goes, its gone.  Vadim says to call back if things get really bad again.  He will come in and inspect things tomorrow morning though doubts he can do much as he is not the expert.  I think its possible its a flakey connector or something flakey since it seems intermittent. Needs more debugging.
<!-- Comment: Sat Sep  9 03;42;32 comment by...Koji -->
<pre>This is the third time during the last two days that we had a problem with slot 8 in b0l1cl02.
The symptom is that we get Done Time out due to bunch mismatch error from this crate, and HRR doesn't work.
 - 1st incident was on Sep 7 evening shift, when the board seemed to have gone bad and we replaced the board with a spare (with Carla.  In this incident, reseating the original board (that was taken out) actually did not work.
 - 2nd incident was on previous (evening) shift.  It happened while we were running a shot setup run.  This time, reseating the board worked and this crate stopped bothering us during L2 torture and cosmic runs until this store came in.

Vadim will come in to take a look at this board tomorrow, and I will leave an recovery procedure intruction which shift crew can follow until then.
</pre>
<!-- Date: Sat Sep 9 02:54:33.164 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779>222779</A>
 Activated at 2006.09.09 02:53:46
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 02:56:06.721 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779>222779</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [6,824,646] 

prescale back to zero
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 02:56:06.594 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10799&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10799' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10800&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10800' width='300'></a></center>
<!-- Author: Shalhout (shot setup) -->



<!-- Date: Sat Sep 9 03:04:19.832 2006 -->
An SVXMon Chip Status Map has gone almost solid yellow a couple of times:  Messages have included 
%ERLOG-e Unknown Chip Error:  SIXDMon Silicon/S/B4/W10/L3 
 
%ERLOG-e Pipeline Out of Sync: Pipeline cell is 17, expected 25. This  
         chip is out of sync for 2 consecutive monitor events.  SIXDMon  
         Silicon/S/B4/W6/L3/C7() 09-Sep-2006 03:01:19 CDT  
 
%ERLOG-S No Data: No data in 101 events processed  SIXDMon  
         Silicon/I/B1/W4/L2/C13() 09-Sep-2006 03:02:48 CDT  
         run = 222779 event = 433554 
%ERLOG-S No Data: No data in 101 events processed  SIXDMon  
         Silicon/I/B1/W4/L2/C14() 09-Sep-2006 03:02:48 CDT  
         run = 222779 event = 433554  
<!-- Author: James Bellinger :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779' target=_top>222779</a>) -->
<!-- Comment: Sat Sep  9 03;20;26 comment by...rde -->
Paged silicon: see expert comment after below plot
<!-- Comment: Sat Sep  9 03;34;23 comment by...Ignacio (Silicon MPC) -->
I have checked the data for these ladders and they are OK  (meaning as they should be).  SB4W6L3 and IB1W4L2 have unfixable bit errors (known problem) that may confuse SVXMon. 
<!-- Date: Sat Sep 9 03:05:11.188 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10801&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10801  width='300' align='top'></a></td><td>silicon will be paged</td></tr></table>
<!-- Author: Shalhout -->
<!-- Comment: Sat Sep  9 03;13;55 comment by...Ignacio (Silicon MPC) -->
This ISL particular ladder tends to require more statistics to 
wake up. It should become green in half an hour. 
<!-- Comment: Sat Sep  9 04;14;55 comment by...Shalhout -->
<pre>it is now green</pre>
<!-- Date: Sat Sep 9 03:12:55.093 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779>222779</A>
 Terminated at 2006.09.09 03:12:39
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 03:13:32.089 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779>222779</A>
TERMINATE: ending for level3 problem
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 03:18:12.847 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
 Activated at 2006.09.09 03:18:05
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 03:18:57.927 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [6,824,646] 

L1 bit 45 set to zero
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 03:19:59.582 2006 -->
<pre>Level 3 partition boxes (bottom of L3 display) are RED 
 
was a L3 display problem; restarted from DAQMon, it is ok now.</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->
<!-- Comment: Sat Sep  9 03;32;06 comment by...Robin -->
Lost some time because accidentally stopped the run to check RTserver, proxy, etc. in haste to debug.  Oops. 



<!-- Date: Sat Sep 9 03:52:20.832 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10802&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10802  width='300' align='top'></a></td><td>TrigMon XFT Finder Canvas asks us to page the expert.  Chris was working on this earlier today.</td></tr></table>
<!-- Author: James Bellinger -->
<!-- Comment: Sat Sep  9 04;08;52 comment by...rde -->
Nils pointed out this is in the <a href="http://www-cdfonline.fnal.gov/cgi/elog/fixlist.pl">known problems</a> list, and indeed it is.  It seems to appear at high lumi, and they will investigate when they can, possibly an XTC problem.  We'll make sure we check all of the known problems plots from here on. 
<!-- Date: Sat Sep 9 04:03:23.583 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779>222779</A>
si: <tt>Yellow ladder in ISL1 Chip Status Map, but expert said probably not enough statistics</tt>
<!-- Author: Consumer Operator( James Bellinger) -->


<!-- Date: Sat Sep 9 04:03:24.215 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222779>222779</A>
trigger: <tt>XFT FINDER Canvas SL3-4 alarms for COT L6 TrigSim vs Data discrepancy.  Expert paged.</tt>
<!-- Author: Consumer Operator( James Bellinger) -->



<!-- Date: Sat Sep 9 04:06:15.084 2006 -->
Got Done time out from b0l1cl02 and the second HRR worked this time.  
<!-- Author: Roberto, Koji :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->
<!-- Comment: Sat Sep  9 04;07;17 comment by...Roberto -->
<pre>(MLE) b0felix.fnal.gov:TEDCpp:Sep 9, 2006 03:57:34 AM->{ 3:57:34, 9/9/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   3:57:34 Halt -> HALTED
(RC)   3:57:41 Recover -> RECOVERED
(MLE) b0l1cl02:Messenger:Sep 9, 2006 03:57:43 AM->Event 1481662: Bunchcounters in slot 8 (BC=71) and slot 5 (BC=73) disagree (L2B=0) 
(MLE) b0l1cl02:Messenger:Sep 9, 2006 03:57:43 AM->Runtime Error 3, Event 1481662: Bunch counter mismatch, mismatch count = 1
(RC)   3:57:44 Run -> ACTIVE
(MLE) b0control21.fnal.gov:Thread-33:Sep 9, 2006 03:57:46 AM->Done Timeout: LEVEL1_CAL_02 
(MLE) b0svx08:Messenger:Sep 9, 2006 03:58:00 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780

(MLE) b0svx07:Messenger:Sep 9, 2006 03:58:01 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780

(MLE) b0felix.fnal.gov:TEDCpp:Sep 9, 2006 03:58:01 AM->{ 3:58:1, 9/9/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   3:58:01 Halt -> HALTED
(RC)   3:58:09 Recover -> RECOVERED
(RC)   3:58:11 Run -> ACTIVE
</pre>



<!-- Date: Sat Sep 9 04:09:26.749 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10803&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10803' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10804&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10804' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10805&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10805' width='300'></a></center>
<!-- Author: Shalhout  (hourly plots) -->



<!-- Date: Sat Sep 9 04:28:34.975 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>If you have a problem with b0l1cl02,  
 - Try HRR 3 times. 
 - Shepard it twice. 
 - If the error is from slot 8 (in error logger) try reseating the board in slot 8. 
 - If reseating works, start a new run. 
 - If the error is not on slot 8 or reseating on slot 8 doesn't work, page Vadim at 267-496-0160. 
  
Your Scico knows how to reseat the board.  (Scicos, pleas pass on the crate location to the next shift). 
</pre></td></tr></td></table>
<!-- Author: Koji -->
<!-- Comment: Sat Sep  9 05;24;20 comment by...Koji -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Ending the run must happen before the reseating, then you can start a new run after reseating happens.  
If the new run goes well, you can keep going.  If not, make the call.
Sorry for the confusion.</pre></td></tr></td></table>
<!-- Date: Sat Sep 9 05:07:42.748 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10806&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10806' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10807&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10807' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10808&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10808' width='300'></a></center>
<!-- Author: Shalhout (hourly plots) -->


<!-- Date: Sat Sep 9 05:17:39.885 2006 -->
<pre>b0l1cl02 error. 
 
Shepherd was successful</pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->
<!-- Comment: Sat Sep  9 05;26;49 comment by...rde -->
HRR performed first, transition failed, then he shepherded it.


<!-- Date: Sat Sep 9 05:29:57.043 2006 -->
<pre>done time out for ccrate b0htdc00. 
 
HRR fixed the problem</pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->


<!-- Date: Sat Sep 9 05:30:40.543 2006 -->
<pre>PhysMon ERROR 
 
Run 222779 has been found BAD</pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->
<!-- Comment: Sat Sep  9 05;37;36 comment by...rOBERTO -->
<pre>PhysMon Info

Run 222779 BAD

W     BAD
J     GOOD
COT   BAD
CAL   BAD
MUON  BAD</pre>

<!-- Comment: Sat Sep  9 05;47;57 comment by...Roberto -->
<pre>expert emailed:  cwolfe@fnal.gov</pre>

<!-- Comment: Sat Sep  9 05;49;01 comment by...Robin -->
I'm worried about this: there may be good data here and I'm wondering why it may get thrown out.  Not sure what the problem is.


<!-- Date: Sat Sep 9 05:54:09.194 2006 -->
<pre>DTO in b0pcal11  
 
Shepherding successful</pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->


<!-- Date: Sat Sep 9 05:58:23.246 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10809&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10809  width='300' align='top'></a></td><td>Spike in CMP Stub phi (YMON shows some increase in CMP part N, stack 5)</td></tr></table>
<!-- Author: James Bellinger -->

<!-- Date: Sat Sep 9 06:00:28.956 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
monitors: <tt>ObjectMon hot CMP phi (in YMON Part N stack5?); also CMU/CMP deltaX, charge/Pt out of tolerance.</tt>
<!-- Author: Consumer Operator( James Bellinger) -->



<!-- Date: Sat Sep 9 06:06:00.511 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10810&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10810' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10811&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10811' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10812&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10812' width='300'></a></center>
<!-- Author: Shalhout  (hourly plots) -->


<!-- Date: Sat Sep 9 06:22:30.231 2006 -->
<pre>Crate 7 communictaion failure alarm, it was successfully Hockerized.  When I came back up to IFIX, it crashed! I restarted IFIX on VNODE1 and brought the HV back up.  We are back to taking data.</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->


<!-- Date: Sat Sep 9 06:37:27.550 2006 -->
Paged silicon.  ACEs saw that some SVX and ISL wedges were yellow (no bias) and remained yellow even after recovering the IFIX problems.  I watched it for a while and they didn't seem to recover.  ACEs are instructed only to deal with pink problems, so Roberto is now speaking to Ignacio (silicon expert) about it.
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->
<!-- Comment: Sat Sep  9 07;23;24 comment by...Shalhout -->
<pre>Hockerized Crate 7 again at expert's request; He is on his way here.</pre>
<!-- Comment: Sat Sep  9 07;35;36 comment by...Shalhout -->
<pre>Also when talking to the expert, IMON jammed and had to be restarted.</pre>
<!-- Date: Sat Sep 9 06:38:31.243 2006 -->
<pre>  Done TO. for crate/s:  
    b0fcal00    
HRR's failed 
 
sheparded fcal00</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->


<!-- Date: Sat Sep 9 06:54:02.029 2006 -->
<pre>TevMon warning:  TevMon failing to receive Acnet messages via RTsertver; 
 
A check of RTserver says it's ok 
 
and the TevMon webpage is updating</pre>
<!-- Author: Shalhout :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->



<!-- Date: Sat Sep 9 07:23:45.064 2006 -->
DTO for ccal04. Left it in halt state for a while since ACEs are on the phone with Ignacio, the silicon MPC, trying to figure out why many boxes in B0, B1, B2 SVX wedges are yellow (bias problem) since after the IFIX crash and crate 7 communication problem (ISL/SVX?).
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->


<!-- Date: Sat Sep 9 07:27:21.778 2006 -->
<pre>SVXMon not sending heart beat to ConsumerErrorReceiver (aka ConsumerMonitor) 
  since 2006-09-09 07:02:12.0 
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon 
  and these processes are very important to data taking! 
Restart SVXMon immediately!</pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->


<!-- Date: Sat Sep 9 07:28:55.754 2006 -->
<pre>DTO  b0ccal04 . 
 
Shepherd was successful.  </pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780' target=_top>222780</a>) -->

<!-- Date: Sat Sep 9 07:33:49.830 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
muons: <tt>Central Muon Slide shows hot CMP (part N) stack 5 toastier than usual</tt>
<!-- Author: Consumer Operator( James Bellinger) -->


<!-- Date: Sat Sep 9 07:33:50.017 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
trigger: <tt>TrigMon:: &nbsp;1.3.4: 125MeV_DCAS_Occupancy (CalTriggerMonitor) shows eta 6/phi 16 with higher than usual rate<br>
<br>
TrigMon:: &nbsp;1.5.1.2: XFT FINDER Canvas SL3-4 (XFTTriggerMonitor2) shows COT SL6 TrigSim vs Data Discrepancy with O(547) errors out of 9473 events. &nbsp;This is greater than 5% and almost equal to the 10,000 events demanded before paging.</tt>
<!-- Author: Consumer Operator( James Bellinger) -->


<!-- Date: Sat Sep 9 07:33:51.081 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
monitors: <tt>Stage0 Used hit times plot is still missing<br>
<br>
Ignoring SiliMon<br>
<br>
ObjectMon: &nbsp;JetMonitor multiplicity differs from reference: &nbsp;new table?<br>
MetMonitor: &nbsp;Met peak shifted<br>
MuonMonitor: CMP Stub phi has a spike, charge/Pt has high secondary wings</tt>
<!-- Author: Consumer Operator( James Bellinger) -->



<!-- Date: Sat Sep 9 07:41:53.747 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
 Terminated at 2006.09.09 07:40:53
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 07:41:54.725 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222780>222780</A>
TERMINATE: ending run
<!-- Author: Roberto -->
<!-- Comment: Sat Sep  9 08;00;48 comment by...rde -->
From the CO and SciCo checks of plots, etc, every component on this run 222780 should be good.  Only problem might be that the last part of the run has bad SVX portions-- we are still determining whether it is a real voltage bias problem or just a monitoring problem.  Brian says he'll mark the good run for this pending the outcome of the silicon SPL investigations. 
<!-- Date: Sat Sep 9 07:48:01.066 2006 -->
Problems with restarting SVXMon:  complains that the port is already in use.  Stopping all monitors; will restart them shortly.
<!-- Author: James Bellinger :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222781' target=_top>222781</a>) -->

<!-- Date: Sat Sep 9 07:50:12.593 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222782>222782</A>
 Activated at 2006.09.09 07:50:09
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 07:50:41.668 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222782>222782</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_4_00 [6,824,646] 
<!-- Author: Roberto -->



<!-- Date: Sat Sep 9 07:54:29.825 2006 -->
ACEs and Ignacio (by phone) were stumped.  Ignacio came in, is on the phone with Ulrich.  We ended the run, pulled silicon from the partition, and started a new run.  
<!-- Author: Robin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222782' target=_top>222782</a>) -->

<!-- Date: Sat Sep 9 07:55:23.587 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222777 target=_runSummary>222777</A> x36639</TD>
<TD>DAQTEST</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,820,642]</FONT></TD>
<TD ALIGN=CENTER>01:16:49</TD>
<TD ALIGN=CENTER>02:01:41</TD>
<TD ALIGN=CENTER>00:43:36</TD>
<TD ALIGN=RIGHT>      24,299</TD>   <TD ALIGN=RIGHT>      0.00</TD>
<TD ALIGN=RIGHT>      0.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222778 target=_runSummary>222778</A> x3663A</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [6,824,646]</FONT></TD>
<TD ALIGN=CENTER>02:32:56</TD>
<TD ALIGN=CENTER>02:49:41</TD>
<TD ALIGN=CENTER>00:05:49</TD>
<TD ALIGN=RIGHT>      53,458</TD>   <TD ALIGN=RIGHT>    202.55</TD>
<TD ALIGN=RIGHT>     68.17</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222779 target=_runSummary>222779</A> x3663B</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [6,824,646]</FONT></TD>
<TD ALIGN=CENTER>02:53:46</TD>
<TD ALIGN=CENTER>03:12:39</TD>
<TD ALIGN=CENTER>00:16:28</TD>
<TD ALIGN=RIGHT>     130,035</TD>   <TD ALIGN=RIGHT>    213.61</TD>
<TD ALIGN=RIGHT>    183.30</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222780 target=_runSummary>222780</A> x3663C</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [6,824,646]</FONT></TD>
<TD ALIGN=CENTER>03:18:05</TD>
<TD ALIGN=CENTER>07:40:53</TD>
<TD ALIGN=CENTER>03:05:19</TD>
<TD ALIGN=RIGHT>   1,322,375</TD>   <TD ALIGN=RIGHT>  2,139.16</TD>
<TD ALIGN=RIGHT>  1,562.89</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222781 target=_runSummary>222781</A> x3663D</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,820,642]</FONT></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>07:47:03</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222782 target=_runSummary>222782</A> x3663E</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [6,824,646]</FONT></TD>
<TD ALIGN=CENTER>07:50:09</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:03:56</TD>
<TD ALIGN=RIGHT>      11,296</TD>   <TD ALIGN=RIGHT>     27.96</TD>
<TD ALIGN=RIGHT>     25.24</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:00</TD>
<TD ALIGN=CENTER>04:15:08</TD>
<TD ALIGN=RIGHT>     1,541,463</TD> <TD ALIGN=RIGHT>  2,583.29</TD>
<TD ALIGN=RIGHT>  1,839.61</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sat Sep 9 07:57:22.083 2006 -->
<pre>RXPT error for crate b0fcal01 
HRR done</pre>
<!-- Author: Roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222782' target=_top>222782</a>) -->


<!-- Date: Sat Sep 9 07:58:53.475 2006 -->
<pre>Very bad luck this night.  Hope day shift is smooth.  Here is a recap: 
 
00:15 Shot setup beginning. 
 
01:10 Final protons loading.  Pbars in RR at 317.  They won't put stack into RR at this point since they have an upper limit of 350, and will shoot just from the RR.  Stack is at 147, fyi. 
 
01:30 Unstashing pbars. 
 
01:53 Pbars loaded, preparing to ramp.  Paged silicon. 
 
02:00 Silicon expert monitoring from home, will call to indicate an ok to ramp. Scraping--ACEs restarting scalermon.   
 
Moving to physics running was delayed more than 20 minutes due to problems with L1cl02 crate (as occurred earlier).  HRRs didn't work, paged Vadim, he doesn't know what the problem is (Carla swapped this board already).  I reseated board and trigger cables and we were able to shephard/HRR out into running.  Shortly into the run, though, L3 turned all red. Was just a display problem, restarted DAQmon and all ok. 
 
03:20 Finally running, hopefully smoothly, Run 222780.  Run 222779 ok, though, the L3 problem wasn't really a problem but run was ended during checkout. 
 
04:06 Another DTO from b0l1cl02, 2 HRRs fixed it. 
 
 
04:30 Plan for dealing with b0l1cl02 DTOs posted by Koji in owl shift log book on 09/09/06: &lt;a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=660&frame=2&anchor=04:28:34.975'&gt; 
here &lt;/a&gt;. 
 
ACE tries HRR 3 times.  Try Shepherding crate twice.  If error logger says its from slot 8, speak to SciCo or trigger expert to try reseating the card in that slot. 
Note: this requires powering off the crate while reseating, so end run! 
SciCo's will pass on how to reseat this card.  This is a kludge and perhaps a coincidence that messing with it does the trick, but Vadim will come in to look. 
 
05:15 DTO for b0l1cl02, HRR/shepherd successful.  DTO b0htdc00, HHR worked. 
 
05:30 PhysMon claims 222779 is bad for W's, COT, CAL, muon.  I find it hard to believe and don't want to lose these data unnecessarily.  ACE emailed cwolfe (physmon webpage) 
 
06:20 Silicon alarms: crate 7 communication failure, hockerizing worked.  IFIX crashed.  HV came back up, and took data.  Then ACEs noticed that three large blocks of chips were yellow for SVX, see entry.  Expert paged, stopped data taking for too long.  Expert came in, and Ulrich working with him.  Started new run sans silicon. 
 
Who said owl shift is smooth?? 
 
</pre>
<!-- Author: Robin -->



<!-- Date: Sat Sep 9 08:03:11.004 2006 -->
<font color=red>Shift Summary: </font><pre>See above summary.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   222776-222782
Delivered Luminosity   2873.9 nb-1  
Acquired Luminosity    1868.8 nb-1  
Efficiency             65% ugh
   
</td></tr></table></pre>
<!-- Author: Robin Erbacher -->

<!-- Date: Sat Sep 9 08:07:11.706 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222782>222782</A>
 Terminated at 2006.09.09 08:06:40
<!-- Author: RunControl -->


<!-- Date: Sat Sep 9 08:07:13.586 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222782>222782</A>
TERMINATE: to take back silicon
<!-- Author: Silvia -->