<!-- Title: Owl shift. Thu Dec 14, 2006 -->
<!-- SciCo: Teruki Kamon -->
<!-- DAQAce: Long Zhan -->
<!-- MonAce: Ulysses Grundler -->
<!-- CO: Victoria Martin -->
<!-- OpManager: Tom Wright -->
<!-- Notes: -->
<pre>Shot setup in progress 
-- MCR is doing a tune up now. store and collider run to follow.</pre>

<!-- Date: Thu Dec 14 01:07:10.620 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230235>230235</A>
 Terminated at 2006.12.14 01:06:23
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 01:07:11.902 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230235>230235</A>
TERMINATE: need to clean EVB
<!-- Author: Timo -->



<!-- Date: Thu Dec 14 01:08:13.587 2006 -->
 Attention !!!. FERML_TM2_HEVB2B_EVENTS_PENDING_TIMEOUT !!!  
 - HRR should help.  
  
 - If problem persists, please use the DiagnoseTO button   
   on the EVB Ace Control Panel.
<!-- Author: Long :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230235' target=_top>230235</a>) -->

<!-- Date: Thu Dec 14 01:11:45.291 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230248>230248</A>
 Activated at 2006.12.14 01:10:30
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 01:11:46.680 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230248>230248</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,639] 
<!-- Author: Timo -->


<!-- Date: Thu Dec 14 01:11:47.184 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230248>230248</A>
 Terminated at 2006.12.14 01:11:03
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 01:11:48.967 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230248>230248</A>
TERMINATE: switch to shot setup mode
<!-- Author: Timo -->



<!-- Date: Thu Dec 14 01:13:05.761 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>0:16 - MCR called. HEP shot setup begins. 
1:12 - Final protons, loaded</td></tr></td></table>
<!-- Author: Teruki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230248' target=_top>230248</a>) -->

<!-- Date: Thu Dec 14 01:17:20.401 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230249>230249</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Timo -->



<!-- Date: Thu Dec 14 01:47:58.369 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>Pbars, loaded.</td></tr></td></table>
<!-- Author: Teruki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230249' target=_top>230249</a>) -->

<!-- Date: Thu Dec 14 01:54:26.398 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230249>230249</A>
TERMINATE: Terminate shot setup
<!-- Author: Long -->


<!-- Date: Thu Dec 14 02:02:18.961 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230250>230250</A>
TERMINATE: done
<!-- Author: Long -->



<!-- Date: Thu Dec 14 02:06:27.260 2006 -->
<pre>*** Node b0pcal07 in Error *** 
Last data more than 120s ago!  Node is not responsive 
 
when partitioning for AAA_CURRENT 
 
shepherded.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230250' target=_top>230250</a>) -->


<!-- Date: Thu Dec 14 02:09:19.110 2006 -->
<pre>*** Node b0pcal11 in Error *** 
Last data more than 120s ago!  Node is not responsive 
 
while trying to config. 
 
shepherded</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230251' target=_top>230251</a>) -->


<!-- Date: Thu Dec 14 02:19:09.734 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16105&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16106  width='300' align='top'></a></td><td>Pinkies - AVDD ISL B2W1L3</td></tr></table>
<!-- Author: Teruki -->
<!-- Comment: Thu Dec 14 02;20;34 comment by...Mark -->
I am looking into this.  


<!-- Date: Thu Dec 14 02:23:43.497 2006 -->
<pre>2:12 - MCR called. Scraping is done. 
       LOSTP and LOSPB are well below 30 kHz. 
 
2:15 - Called MCR. CLC HV is up. 
       Lum = 204E30 
 
 
Meantime, 5-6 pinkies on ISL B2W1L3 (AVDD).  Unmasked. 
Goes to yellow.</pre>
<!-- Author: Teruki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230251' target=_top>230251</a>) -->

<!-- Date: Thu Dec 14 02:39:11.679 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230251>230251</A>
TERMINATE: run without silicon
<!-- Author: Long -->


<!-- Date: Thu Dec 14 02:41:59.978 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
 Activated at 2006.12.14 02:41:38
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 02:42:00.285 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->



<!-- Date: Thu Dec 14 02:54:50.347 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-16107&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-16108  width='300' align='top'></a></td><td>Pinkies - ISL B2W1L3 (AVDD)</td></tr></table>
<!-- Author: Teruki -->


<!-- Date: Thu Dec 14 02:57:35.314 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16109&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16109' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16110&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16110' width='300'></a></center>
<!-- Author: Ulysses (shot) -->


<!-- Date: Thu Dec 14 03:12:14.765 2006 -->
Can someone please make an entry describing what problems you are having and who you have paged about it?  That information needs to be here.  
<!-- Author: twright -->


<!-- Date: Thu Dec 14 03:12:30.776 2006 -->
YMON is not working.  Kaori is investigating...
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->
<!-- Comment: Thu Dec 14 03;32;39 comment by...kaori -->
YMon has been having a problem to do with COSlide # of parameters
in SMX in the version 6.1.4ak (default cdfsoftb0 version).  Some
reason, when I switched back to the older version 6.1.4ag (which supposed to work) yesterday,
I must not copy the tcl correctly.  YMon should run o.k., now.
<!-- Date: Thu Dec 14 03:15:02.602 2006 -->
<pre>Attention!!! L2 Decision Timeout ERROR !!.  
 If L2 Timeouts persist, please page L2 
 
HRR fixed</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->


<!-- Date: Thu Dec 14 03:15:30.380 2006 -->
<pre>L2 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!!</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->


<!-- Date: Thu Dec 14 03:16:49.441 2006 -->
<pre>Trigger Inhibit is set from the components: 
 CES LV:iFix  		[136] Central EM Strip Low Voltage from iFix 
</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->
<!-- Comment: Thu Dec 14 03;51;43 comment by...byrum -->
Several wedges are reading funny value and causing a CES inhibit - plus one arch is displaying purple for CES PSM in IFIX.   Instructions for fixing IFIX PSM problems are linked off of monitoring ACE help page.   Specifically, see http://www.hep.anl.gov/byrum/pagerhelp/psm.htm    This has a picture of the BNB power supply box located on the first floor in Rack 1RR06F which you need to powercycle.    
<!-- Comment: Thu Dec 14 03;56;41 comment by...byrum -->
There seems to be a problem with the link to the ops page to powercycle the BNB power supply - http://www-cdfonline.fnal.gov/opshelp/powercycle.html
<!-- Comment: Thu Dec 14 04;08;43 comment by...byrum -->
Aces say that powercycling the BNB power supply did not fix the IFIX readout problem - CES wedges still purple and 3 wedges sill red (reading -17V instead of -5V) causing a trigger inhibit.   This really looks to be a readout communications problem - I would figure out a way to ignor the trigger inhibit to take data.   
<!-- Comment: Thu Dec 14 04;41;43 comment by...Ulysses -->
<pre>We have also talked to Steve Hahn.  He was unsure what the problem might be and suggested trying to powercycle the NE arch to see if that would fix things.  It does not seem to have.  We disabled the trigger inhibit to try and take data since it seems to be just a communication problem.  When trying to run we keep getting L2 T0s, possibly due to this problem because it says it is waiting for ShowerMax data.</pre>
<!-- Date: Thu Dec 14 03:17:19.674 2006 -->
<pre>Without YMON I can't check the status of all the systems. 
 
However everything I can check: Triggers, DAQ and ObjectMon looks fine so far. 
</pre>
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->
<!-- Comment: Thu Dec 14 03;24;41 comment by...kaori -->
YMon is fixed.  (running with 6.1.4ag).  
<!-- Date: Thu Dec 14 03:21:02.204 2006 -->
Jennifer, the Silicon Cooling pager carrier was paged and called in for "ISL flow problem". Turns out, all the ACE needed to do was move the iFix alarm page away from the status pannel to see that the ISL conductivity was already ignored. NO problems, my ISL is a happy camper. Nice and cool.
<!-- Author: Jennifer :-) :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->


<!-- Date: Thu Dec 14 03:24:47.904 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
 Terminated at 2006.12.14 03:24:42
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 03:25:01.218 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
TERMINATE: putting in silicon
<!-- Author: Long -->



<!-- Date: Thu Dec 14 03:25:33.051 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>PLEASE refer <a href="https://www-cdfonline.fnal.gov/internal/ops/svxii/svx_for_aces/recover_link.html">HERE</a> when understanding what kind of silicon problem you are having.</td></tr></td></table>
<!-- Author: Jennifer :-) for the Sillys :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252' target=_top>230252</a>) -->
<!-- Comment: Thu Dec 14 03;55;16 comment by...Ulrich (from Europe) -->
Checking the log files it seems that there was a problem with a part of the Silicon PS GUI (could somebody please comment in the e-log what happened?). This causes an iFix alarm in SVX, ISL, and L00, but will not show up in the alarm list (maybe we should add it there). My guess is that the Aces searched for the offending alarm, found ISL-IACONDUCT red and didn't verify that it is not silenced.
<!-- Date: Thu Dec 14 03:48:40.614 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
fwd: <tt>No YMON plots available for this run at all!<br>
I've marked the catagories that rely on YMON as "Not Yet".</tt>
<!-- Author: Consumer Operator( Victoria Martin) -->


<!-- Date: Thu Dec 14 03:48:41.305 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
monitors: <tt>YMON wasn't running for the entirety of this run. &nbsp;Kaori claims it's fixed now, we have to wait for the next run to see.</tt>
<!-- Author: Consumer Operator( Victoria Martin) -->
<!-- Comment: Thu Dec 14 06;01;02 comment by...Victoria -->
run 230263: YMON is working now!  Thanks Kaori.
<!-- Date: Thu Dec 14 03:48:43.918 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
general: <tt>No YMON plots available for this run at all!<br>
I've marked the catagories that rely on YMON as "Not Yet".</tt>
<!-- Author: Consumer Operator( Victoria Martin) -->




<!-- Date: Thu Dec 14 03:52:16.831 2006 -->
<pre>Below are events between 2 am and 4 am. 
 
2:12 - MCR called. Scraping is done.  
       LOSTP and LOSPB are well below 30 kHz.  
2:15 - Called MCR. CLC HV is up.  
       Lum = 204E30  
 
       Silicon HV trips - Paged silicon. 
 
2:42 - Run 230252 (without silicon) 
 
2:40 - Paged silicon cooling. 
2:49 - Silicon cooling expert is on the way 
          It tunrs out that all we needed to do was to move  
          the iFix alarm page away from the status pannel  
          to see that the ISL conductivity. 
3:02 - Paged online monitor - We couldn't start YMON. 
3:23 - PSM alarm (East Central 00E-05E; 18E-23E).  
       Paged CES,CCR,CPR,PES,PPR FE electronics. 
3:41 - L00, ISL, SVX alarms. Recovered smoothly. 
3:50 - ACEs are working on PSM.</pre>
<!-- Author: Teruki -->
<!-- Comment: Thu Dec 14 04;02;56 comment by...Ulrich (from Europe) -->
I'm confused. I don't see trips around 2:20. There was an issue with the power supply GUI between 2:23 and 2:49 (when the GUI was finally restarted), and after that a couple of wedges were still off (problem with CAEN crate #8?). The there were problems powering up one L00 wedge during recovery. 

<!-- Comment: Thu Dec 14 04;38;44 comment by...Mark -->
For the silicon, the timeline as best as I can remember:
<pre>
2:20 - The ACE tries to bias the silicon, nothing happens.  I call in to find out what is going on.
2:24 - IFIX starts giving error messages that there is a power supply logging problem.
2:35? - We reboot svxinter, but it gives communication errors after the reboot
~2:45 - Jose, Jennifer and Mark leave for the lab.
2:50 - svxinter is now able to communicate with the ps crates
3:15-3:20 - We are able turn SVX/ISL/L00 on
</pre>
<!-- Comment: Thu Dec 14 05;59;33 comment by...Teuki -->
The words "HV trip" at around 2:20 was due to my confusion
and not accurately described. Mark described.
<!-- Date: Thu Dec 14 04:06:51.970 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230255>230255</A>
 Activated at 2006.12.14 04:06:42
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:06:52.210 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230255>230255</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->



<!-- Date: Thu Dec 14 04:14:03.584 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230255>230255</A>
 Terminated at 2006.12.14 04:13:07
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:16:21.539 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230255>230255</A>
TERMINATE: need to fix the L2 problem
<!-- Author: Long -->


<!-- Date: Thu Dec 14 04:18:28.628 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230256>230256</A>
 Activated at 2006.12.14 04:18:24
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:20:05.898 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230256>230256</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 04:24:14.379 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230256>230256</A>
 Terminated at 2006.12.14 04:23:28
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:24:15.575 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230256>230256</A>
TERMINATE: need to fix the L2 problem
<!-- Author: Long -->



<!-- Date: Thu Dec 14 04:27:42.794 2006 -->
Concerning the silicon problems at the beginning of the store, we were unable to bias the Silicon due to a communication problem between svxinter and the silicon crates.  We tried rebooting svxinter, which did not appear to help, and then the problem fixed itself after 30 minutes.  We are still trying to find out why this happened, but the silicon is currently biased and ready for data taking whenever the other issues are fixed.
<!-- Author: The Silis -->

<!-- Date: Thu Dec 14 04:34:00.452 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230257>230257</A>
 Activated at 2006.12.14 04:33:28
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:34:01.608 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230257>230257</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->




<!-- Date: Thu Dec 14 04:50:06.942 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230257>230257</A>
 Terminated at 2006.12.14 04:50:00
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:51:28.092 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230257>230257</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 04:59:03.085 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230258>230258</A>
 Activated at 2006.12.14 04:58:55
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 04:59:51.195 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230258>230258</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->
<!-- Comment: Thu Dec 14 05;09;50 comment by...Wojtek -->
<pre>I directed daqace to try a run with 'ignore RESCES' checked in the runconfig.
This will cause L2 not to wait for the showermax data - instead one should see done TO for ccal (etc) crates which is exactly what happened. 
ShowerMax ps problem must be the root cause. -You should be able to run normally when this problem is fixed.
I directed the ace to reopen RunConfig after this check was done.
 </pre>
<!-- Date: Thu Dec 14 05:03:24.594 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230258>230258</A>
 Terminated at 2006.12.14 05:02:18
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 05:03:25.530 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230258>230258</A>
TERMINATE: ending
<!-- Author: Long -->
<!-- Comment: Thu Dec 14 05;29;41 comment by...Wojtek -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Just a reminder: make sure 'ignore RECES' box is NOT checked in the expert panel of runconfig before  partition/config on the next run </pre></td></tr></td></table>

<!-- Comment: Thu Dec 14 05;37;17 comment by...Long -->
It is not checked. 
<!-- Date: Thu Dec 14 05:37:17.547 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230262>230262</A>
 Activated at 2006.12.14 05:36:59
<!-- Author: RunControl -->



<!-- Date: Thu Dec 14 05:37:42.252 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230262>230262</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->



<!-- Date: Thu Dec 14 05:37:42.434 2006 -->
Strange (but not necessarily fatal) Run Configuration selections: 
Crate CCAL_08 missing from run 
Crate CCAL_09 missing from run 
Crate CCAL_14 missing from run 
Crate CCAL_15 missing from run
<!-- Author: Long :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230262' target=_top>230262</a>) -->
<!-- Comment: Thu Dec 14 05;41;00 comment by...Wojtek -->
<pre>now you'll get l2 decision to for sure
L2 processor will be waiting for showermax (=reces) data</pre>
<!-- Date: Thu Dec 14 05:39:15.289 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230262>230262</A>
 Terminated at 2006.12.14 05:38:23
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 05:39:16.400 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230262>230262</A>
TERMINATE: done
<!-- Author: Long -->



<!-- Date: Thu Dec 14 05:47:19.316 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
 Activated at 2006.12.14 05:47:01
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 05:47:20.474 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->
<!-- Comment: Thu Dec 14 05;49;18 comment by...Wojtek -->
<pre>now L2 is ignoring RECES 
I understand some ccal crates are excluded</pre>

<!-- Comment: Thu Dec 14 05;50;29 comment by...Long -->
CCAL_08; CCAL_09; CCAL_14; CCAL_15 are missing from this run!!! but we have no other choice......


<!-- Date: Thu Dec 14 05:54:54.066 2006 -->
<pre>Getting numerous DQMon alarms about Wrong trigger table setting in YMon.  Have to do with ccal crates not in partition and ignoreReces set?  Have disabled.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->


<!-- Date: Thu Dec 14 05:57:09.439 2006 -->
<pre>DQMon is also complaining about CEM, CHA, and CES.  Of course, these are all affected by having the ccal crates out.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->


<!-- Date: Thu Dec 14 05:59:14.512 2006 -->
<pre>*** Node b0pcal02 in Error *** 
Last data more than 120s ago!  Node is not responsive 
 
Halted run, shepherding.</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->
<!-- Comment: Thu Dec 14 06;06;09 comment by...Ulysses -->
<pre>shepherding has not worked.</pre>
<!-- Comment: Thu Dec 14 06;16;10 comment by...Ulysses -->
I am unable to ping this crate, either.
<!-- Comment: Thu Dec 14 06;41;24 comment by...Ulysses -->
<pre>Also had trigger inhibit "VME Power:iFix", admem expert had me powercycle crate and start a new run.  Did this, the inhibit cleared and was able to ping pcal02.  When run activated got:

Done Timeout detected in crates:
PCAL_02

though pcal02 did not go red.  Tried HRR a few times, but still got this message and L2 or L3 trigger rates had dropped to 0Hz.
Expert said to page showermax</pre>
<!-- Comment: Thu Dec 14 06;53;33 comment by...byrum -->
Powercycle pcal02, start a new run - if this doesn't work, try to powercycle the west plug, start a new run.   If you are still getting IFIX problems from PSM alarms from this crate - I suggest you try to restart IFIX.   Perhaps you should try to restart IFIX anyway to clear up the central PSM problems too.
<!-- Comment: Thu Dec 14 06;55;18 comment by...Ulysses -->
<pre>ShowerMax expert (Karen, I think) said to try powercycling once more, and if that doesn't work try powercycling the 400Hz on the west plug.  Tried to powercycle and the VME Power:iFix trigger inhibit has come back.  I guess we'll try the 400Hz.</pre>
<!-- Comment: Thu Dec 14 07;01;00 comment by...Ulysses -->
<pre>Didn't find anybody in cryo, but the inhibit cleared.</pre>
<!-- Date: Thu Dec 14 05:59:34.148 2006 -->
<pre>Trigger Inhibit is set from the components: 
 VME Power:iFix  	[147] VME Crate Power Supplies from iFix 
</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->



<!-- Date: Thu Dec 14 06:05:06.102 2006 -->
There is a document off of the monitoring ace page on "Voltman Web page and  
Power Supply Monitoring Guide".  The latter is at http://www-cdfonline.fnal.gov/ops/mcs/cdf5749_PSM_Alarms.pdf 
It looks like it tells you how to mask off an alarm in PSM.  Perhaps if you do this, you can leave the  ccal crates 8,9,14 &15 in the run and see if there really is a hardware problem. 
 
<!-- Author: byrum :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->
<!-- Comment: Thu Dec 14 06;27;36 comment by...Long -->
We tried to "mask" the alarm. It does not work.
<!-- Date: Thu Dec 14 06:06:42.870 2006 -->
<pre>SVXMon not connected to ConsumerErrorReceiver (aka ConsumerMonitor) 
  since 2006-12-14 06:04:28.0 
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon 
  and these processes are very important to data taking! 
Restart SVXMon immediately!</pre>
<!-- Author: Ulysses :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->


<!-- Date: Thu Dec 14 06:08:28.663 2006 -->
<pre>Time line: 
 
4:00 - ACEs are working on PSM. Power-cycling doesn't help. 
4:09 - Called Steve Hahn 
4:27 - According to Steve: very odd situation. 
       ACEs try to do power-cycling entire East arch. 
       No help. 
 
4:34 - Run 230257 to see if PSM alarm is just a communication problem. 
       L2 TimeOut.  
4:37 - Paged L2. 
           
       This seems to be due to ShowerMax crate problem. 
 
5:05 - Paged ShowerMax expert, again. 
 
5:06 - ISL alarm. Silicon experts are investigating. Recovered 
5:11 - L00 alarm. Silicon experts are investigating. Recovered. 
5:20 - Silicon experts left w/o seeing the collision data. 
 
5:35 - Paged iFIX expert. 
5:51 - Called Steve Hahn. 
6:06 - ACEs finished conversation with Steve, and follow his 
       instruction.</pre>
<!-- Author: Teruki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263' target=_top>230263</a>) -->


<!-- Date: Thu Dec 14 06:24:54.144 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230252>230252</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=230252 target=_runSummary>230252</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 06:26:05.978 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
 Terminated at 2006.12.14 06:25:48
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 06:26:34.727 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
TERMINATE: restart
<!-- Author: Long -->



<!-- Date: Thu Dec 14 06:29:32.767 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230264>230264</A>
 Activated at 2006.12.14 06:29:20
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 06:29:33.132 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230264>230264</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->



<!-- Date: Thu Dec 14 06:30:51.729 2006 -->
Ymon crashed followed by XMon. Restarted and keeping an eye on 
them.
<!-- Author: Satyajit (CO) :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230264' target=_top>230264</a>) -->
<!-- Comment: Thu Dec 14 07;05;36 comment by...Satyajit -->
YMon crashed again.
<!-- Date: Thu Dec 14 06:44:44.346 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230264>230264</A>
 Terminated at 2006.12.14 06:44:25
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 06:45:03.046 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230264>230264</A>
TERMINATE: ending
<!-- Author: Long -->



<!-- Date: Thu Dec 14 06:50:38.336 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16111&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16111' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16112&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16112' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-16113&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-16113' width='300'></a></center>
<!-- Author: Long  0000--0400 -->




<!-- Date: Thu Dec 14 07:02:26.012 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230265>230265</A>
 Activated at 2006.12.14 07:02:09
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:02:27.730 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230265>230265</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:03:34.195 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230265>230265</A>
 Terminated at 2006.12.14 07:03:17
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:03:35.520 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230265>230265</A>
TERMINATE: ending
<!-- Author: Long -->



<!-- Date: Thu Dec 14 07:07:19.233 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230266>230266</A>
 Activated at 2006.12.14 07:07:10
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:07:20.825 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230266>230266</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:09:53.708 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230266>230266</A>
 Terminated at 2006.12.14 07:08:57
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:09:54.859 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230266>230266</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:12:35.660 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230267>230267</A>
 Activated at 2006.12.14 07:12:14
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:12:36.005 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230267>230267</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:15:09.045 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230267>230267</A>
 Terminated at 2006.12.14 07:14:33
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:15:10.107 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230267>230267</A>
TERMINATE: ending

<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:19:23.627 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230268>230268</A>
 Activated at 2006.12.14 07:18:55
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:19:25.757 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230268>230268</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:19:46.040 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230268>230268</A>
 Terminated at 2006.12.14 07:19:41
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:20:23.332 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230268>230268</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:25:15.733 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230269>230269</A>
 Activated at 2006.12.14 07:24:56
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:25:16.814 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230269>230269</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:30:27.844 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230269>230269</A>
 Terminated at 2006.12.14 07:30:14
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:30:28.781 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230269>230269</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:30:56.535 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230269>230269</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 07:42:53.620 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
 Activated at 2006.12.14 07:42:34
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 07:42:54.789 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
Strange (but not necessarily fatal) Run Configuration selections:
Crate CCAL_08 missing from run
Crate CCAL_09 missing from run
Crate CCAL_14 missing from run
Crate CCAL_15 missing from run

<!-- Author: Long -->



<!-- Date: Thu Dec 14 07:48:08.048 2006 -->
I can talk to all the showermax boards using standalone pedcal, so I don't understand why we can't take data with them included.  Dervin thinks the power supply problem may just be a monitoring problem.  But when we try to run with those ccal crates, we get L2 timeouts.
<!-- Author: convery :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270' target=_top>230270</a>) -->

<!-- Date: Thu Dec 14 07:55:41.461 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230252 target=_runSummary>230252</A> x3836C</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>02:41:38</TD>
<TD ALIGN=CENTER>03:24:42</TD>
<TD ALIGN=CENTER>00:30:01</TD>
<TD ALIGN=RIGHT>     193,851</TD>   <TD ALIGN=RIGHT>    445.68</TD>
<TD ALIGN=RIGHT>    314.46</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230255 target=_runSummary>230255</A> x3836F</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>04:06:42</TD>
<TD ALIGN=CENTER>04:13:07</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT>     58.69</TD>
<TD ALIGN=RIGHT>      0.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230263 target=_runSummary>230263</A> x38377</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>05:47:01</TD>
<TD ALIGN=CENTER>06:25:48</TD>
<TD ALIGN=CENTER>00:11:05</TD>
<TD ALIGN=RIGHT>      27,470</TD>   <TD ALIGN=RIGHT>    263.29</TD>
<TD ALIGN=RIGHT>     76.48</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230265 target=_runSummary>230265</A> x38379</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>07:02:09</TD>
<TD ALIGN=CENTER>07:03:17</TD>
<TD ALIGN=CENTER>00:01:05</TD>
<TD ALIGN=RIGHT>       5,657</TD>   <TD ALIGN=RIGHT>      7.23</TD>
<TD ALIGN=RIGHT>      6.07</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230269 target=_runSummary>230269</A> x3837D</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>07:24:56</TD>
<TD ALIGN=CENTER>07:30:14</TD>
<TD ALIGN=CENTER>00:05:14</TD>
<TD ALIGN=RIGHT>      22,103</TD>   <TD ALIGN=RIGHT>     31.03</TD>
<TD ALIGN=RIGHT>     29.52</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=230270 target=_runSummary>230270</A> x3837E</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,663]</FONT></TD>
<TD ALIGN=CENTER>07:42:34</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:11:36</TD>
<TD ALIGN=RIGHT>      60,195</TD>   <TD ALIGN=RIGHT>     69.94</TD>
<TD ALIGN=RIGHT>     64.48</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:00</TD>
<TD ALIGN=CENTER>00:59:03</TD>
<TD ALIGN=RIGHT>       309,276</TD> <TD ALIGN=RIGHT>    875.86</TD>
<TD ALIGN=RIGHT>    491.01</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Dec 14 08:00:10.000 2006 -->
<font color=red>Shift Summary: </font><pre>	OWL SHIFT SciCO Teruki Kamon 
 
Start Shift: 
===========  
 Store 51xx colliding 
 Lumi at yy.yE30 [36(p)x36(ap) bunches] 
 
 
Run Plan:Take as much data as possible 
 
Run Summary: 
============ 
230252 - w/o silicon 
 
 
Luminosity Summary (OWL) 
======================== 
Begin shift or beam at 02:13:31 
End shift or beam at 07:51:47 
Delivered luminosity: 2695.57 
Acquired luminosity: 476.57 
Totals 
Date:	2006.12.14 
Shift:	owl 
Delivered luminosity:	2695.6 nb-1 
Acquired luminosity:	476.6 nb-1 
Efficiency:	17.7% 
 
 
Highlights: 
=========== 
 0:00 - Shot setup in progress 
*0:16 - MCR called. HEP shot setup - start injecting beams 
 1:12 - Final protons, loaded. Pagedn Ops manager. 
 1:47 - Pbars, loaded. Paged silicon. 
*2:12 - MCR called. Scraping is done.  
        LOSTP and LOSPB are well below 30 kHz.  
 2:15 - Called MCR. CLC HV is up.  
        Lum = 204E30  
 
*       Unable to bias silicon - Paged silicon. 
 
 2:42 - Run 230252 (without silicon) 
  
 2:40 - Paged silicon cooling. 
 2:49 - Silicon cooling expert is on the way 
          It tunrs out that all we needed to do was to move  
          the iFix alarm page away from the status pannel  
          to see that the ISL conductivity. 
 
Concerning the silicon problems at the beginning of the store,  
we were unable to bias the Silicon due to a communication problem 
between svxinter and the silicon crates. We tried rebooting  
svxinter, which did not appear to help, and then the problem  
fixed itself after 30 minutes. We are still trying to find out  
why this happened, but the silicon is currently biased and ready 
for data taking whenever the other issues are fixed.  
 
3:02 - Paged online monitor - We couldn't start YMON. 
          Kaori thinks it is fixed.  We need to start a run. 
 
*3:23 - PSM low voltage alarm (East Central 00E-05E; 18E-23E).  
        Paged CES,CCR,CPR,PES,PPR FE electronics. 
        Power cycling.. 
 3:41 - L00, ISL, SVX alarms. Recovered smoothly. 
 
 4:00 - ACEs are working on PSM. Power-cycling doesn't help. 
 4:09 - Called Steve Hahn 
 4:27 - According to Steve: very odd situation. 
        ACEs try to do power-cycling entire East arch. 
        No help. 
 
 4:34 - Run 230257 to see if PSM alarm is just a communication problem. 
        L2 TimeOut.  
 4:37 - Paged L2. 
           
        This seems to be due to ShowerMax crate problem. 
 
 5:05 - Paged ShowerMax expert, again. 
 
 5:06 - ISL alarm. Silicon experts are investigating. Recovered 
 5:11 - L00 alarm. Silicon experts are investigating. Recovered. 
 5:20 - Silicon experts left w/o seeing the collision data. 
 
 5:35 - Paged iFIX expert. 
 5:51 - Called Steve Hahn. 
 6:06 - ACEs finished conversation with Steve, and follow his 
        instruction. 
 
 6:16 - Plug Power Supply Monitor Alarm (EPSW_1) 
        Paged Calorimeter Electronics expert. 
 6:37 - Paged ShowerMax expert. 
 6:48 - Paged Dervin and Roberto --&gt; No response 
 6:52 - Paged Hardware Power Suplies expert.--&gt; No response 
 7:00 - Steve Hahn is here and looking the problem with ACEs. 
 7:25 - Dervin is also here. 
*7:33 - Called Cryo - power-cycling 400 Hz power (NE arch) 
 7:59 - End of shift 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   230252
Delivered Luminosity   2695.57  
Acquired Luminosity    476.57  
Efficiency             17.7%
   
</td></tr></table></pre>
<!-- Author: Teruki -->

<!-- Date: Thu Dec 14 08:02:55.122 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
 Terminated at 2006.12.14 08:02:33
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 08:02:56.375 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 08:03:04.976 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 08:03:04.675 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 08:03:39.965 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230270>230270</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 08:07:55.428 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230271>230271</A>
 Activated at 2006.12.14 08:07:14
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 08:07:56.674 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230271>230271</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->
<!-- Comment: Thu Dec 14 08;20;03 comment by...wojtek -->
<pre>MLE) b0ccal15:Messenger:Dec 14, 2006 08:07:41 AM-&gt;SMXR Not Responding event 2, slot 5, bunch ctr 74, L2buffer 0, header 0x3df654a
(MLW) b0ccal15:Messenger:Dec 14, 2006 08:07:41 AM-&gt;Runtime Error 3, Event 2: ShowerMax FE board has stopped sending data, slot 5, header 0x0x3df654a</pre>
<!-- Date: Thu Dec 14 08:11:26.261 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230271>230271</A>
 Terminated at 2006.12.14 08:10:44
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 08:11:27.331 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230271>230271</A>
TERMINATE: ending
<!-- Author: Long -->


<!-- Date: Thu Dec 14 08:14:13.467 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230272>230272</A>
 Activated at 2006.12.14 08:13:52
<!-- Author: RunControl -->
<!-- Comment: Thu Dec 14 08;17;09 comment by...wojtek -->
<pre>can you exclude the cal crates that give you problem one by one?
what I mean 
include all
record several done TO
exclude the crates that give problem
repeat</pre>
<!-- Comment: Thu Dec 14 08;18;28 comment by...wojtek -->
<pre>AND PLEASE RECORD THE CRATES THAT GIVE DONETO</pre>
<!-- Date: Thu Dec 14 08:15:16.637 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230272>230272</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Long -->


<!-- Date: Thu Dec 14 08:16:43.002 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230272>230272</A>
 Terminated at 2006.12.14 08:16:08
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 08:16:46.409 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230272>230272</A>
TERMINATE: end to exclude ccal15
<!-- Author: Tatiana -->



<!-- Date: Thu Dec 14 08:17:07.041 2006 -->
Mike Lindgren reports that we had a very similar problem Nov 12 when he worked on the same SMD ps on the south side as he did yesterday.  Again we had a 
<!-- Author: convery, lindgren -->



<!-- Date: Thu Dec 14 08:19:02.638 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
central: <tt>On COT slide #2 occupancy plots, SL7 and SL8 are RED, although no<br>
gap >8 cells is visible.</tt>
<!-- Author: Consumer Operator( Satyajit Behari) -->


<!-- Date: Thu Dec 14 08:19:03.542 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
cal: <tt>A quarter of the central calorimeter system is OFF. WHA Phi 5L, Tower -6<br>
show a dead channel.</tt>
<!-- Author: Consumer Operator( Satyajit Behari) -->


<!-- Date: Thu Dec 14 08:19:05.279 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
smx: <tt>A quarter of the central calorimeter system is OFF.</tt>
<!-- Author: Consumer Operator( Satyajit Behari) -->


<!-- Date: Thu Dec 14 08:19:06.233 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
monitors: <tt>Many low stats plots, mainly due to the calorimeter problem.</tt>
<!-- Author: Consumer Operator( Satyajit Behari) -->


<!-- Date: Thu Dec 14 08:19:07.699 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230263>230263</A>
trigger: <tt>Mostly problems pertaining to the central calorimeter. CES low trigger<br>
efficiency is 0. Pulsar monitor shows RED status of Muon, CES and TL2D.</tt>
<!-- Author: Consumer Operator( Satyajit Behari) -->



<!-- Date: Thu Dec 14 08:21:19.693 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230273>230273</A>
 Activated at 2006.12.14 08:20:19
<!-- Author: RunControl -->


<!-- Date: Thu Dec 14 08:21:20.921 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=230273>230273</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,663] 
<!-- Author: Tatiana -->