<!-- Title: Day shift. Sat Dec  8, 2007 -->
<!-- SciCo: Ron Moore -->
<!-- DAQAce: Pedro Ttito-Guzman -->
<!-- MonAce: &nbsp; -->
<!-- CO: Maurizio Iori -->
<!-- OpManager: Max Goncharov -->
<!-- Notes: -->
Taking data HEP store 5787, CDF lumi = 34 ub^-1/s.  Waiting for plan from MCR.


<!-- Date: Sat Dec 8 08:43:37.387 2007 -->
<b>MCR Plan</b><br> 
Terminate store 5787 at 12:00 and immediately start next HEP shot-setup.
<!-- Author: ronmoore -->


<!-- Date: Sat Dec 8 08:48:48.468 2007 -->
Jim Loskot called from downstairs.  He responded to an NWX gas alarm - his comment <a href= 
'/cgi/elog/det-system-elog.pl?nb=cryo-gas&action=view&page=2502&frame=2&anchor=08:43:57'> 
here</a>.  
<!-- Author: ronmoore -->


<!-- Date: Sat Dec 8 09:19:59.481 2007 -->
Paging CES electronics to follow-up on possible problem seen on owl shift - <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=958&frame=2&anchor=05:12:55.597'> 
link here</a>. 
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 09;34;22 comment by...ronmoore -->
No response.  Trying calorimeter SPL pager.


<!-- Comment: Sat Dec  8 09;54;13 comment by...ronmoore -->
No response from SPL pager either!
<!-- Comment: Sat Dec  8 09;54;35 comment by...ronmoore -->
As a sanity check, I paged myself and received the page < 1 min.
<!-- Comment: Sat Dec  8 14;33;57 comment by...ronmoore -->
Note that Larry called in around 14:30 after reading my note regarding the CES problem - he says it is fine - no problem.  He never did receive a page (calorimeter SPL).
<!-- Date: Sat Dec 8 09:53:35.444 2007 -->
We are gettings PSM alarms from 1RR18E_2 fans.  I went downstairs to check, saw nothing obvious amiss.  Then the alarm disappeared briefly, but now it is back.  Paging electronics infrastructure.  (FYI - this is the CMP TDC and muon trigger rack.)
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 09;57;00 comment by...ronmoore -->
OK, now it looks like we lost the +5V for the muon trigger crate - we're seeing front-end errors.  I asked the ace to stop the run.

<!-- Comment: Sat Dec  8 10;02;46 comment by...ronmoore -->
I talked to Dervin - he is busy at the moment, but he will contact Roberto to come in and replace the supply.

<!-- Comment: Sat Dec  8 10;03;45 comment by...ronmoore -->
Indeed, the b0mutr00 crate CPU is no longer pingable.

<!-- Comment: Sat Dec  8 10;11;07 comment by...ronmoore -->
Dervin called back.  Roberto will be here in ~30 min to replace the supply.  I switched off the supply at Dervin's request.
<!-- Comment: Sat Dec  8 10;50;19 comment by...ronmoore -->
Roberto is here.
<!-- Date: Sat Dec 8 10:05:55.813 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254924>254924</A>
 Terminated at 2007.12.08 10:05:05
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 10:05:56.201 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254924>254924</A>
TERMINATE: end run
<!-- Author: Pedro -->


<!-- Date: Sat Dec 8 10:06:57.089 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254924>254924</A>
TERMINATE: end run
<!-- Author: Pedro -->




<!-- Date: Sat Dec 8 10:11:59.872 2007 -->
Since we're not taking data at the moment, I asked Pedro to take the Silicon HV to standby.  I will call Max to see what he would like us to do.
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 10;21;49 comment by...ronmoore -->
Max is coming in.  We will bring all HV to stand-by as if end-of-store, but then bring CLC HV back up so MCR can keep track of our luminosity.  For now we do nothing useful.


<!-- Comment: Sat Dec  8 10;36;59 comment by...JJ -->
<pre>It would be cleaner to take each system to EOS state  with individual
control buttons and just leave CLC on.</pre>
<!-- Date: Sat Dec 8 10:31:46.139 2007 -->
<pre>we got this chain of errors: 
 
FrontEnd Crate Error Condition from: 
MUON_TRIGGER_00***  
Node b0mutr00 in Error *** 
Last data more than 120s ago!  Node is not responsive. 
Trigger Inhibit is set from the components: 
 VME Power  		[147] VME Crate Power Supplies from iFix 
 
 
 
 
</pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254924' target=_top>254924</a>) -->




<!-- Date: Sat Dec 8 10:52:17.779 2007 -->
Several minutes ago, we lost power supply monitoring for the west plug.  Roberto will also investigate that while he's here.
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 11;30;36 comment by...ronmoore -->
Roberto also recovered the west plug power supply monitoring.
<!-- Date: Sat Dec 8 11:25:57.155 2007 -->
Roberto has replaced the muon trigger crate power supply.  We are ramping HV to resume data-taking.
<!-- Author: ronmoore -->

<!-- Date: Sat Dec 8 11:28:05.513 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928>254928</A>
 Activated at 2007.12.08 11:27:21
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 11:29:26.108 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928>254928</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [2,1010,710] 
<!-- Author: Pedro -->
<!-- Comment: Sat Dec  8 11;30;08 comment by...ronmoore -->
Hooray, we are back!


<!-- Comment: Sat Dec  8 11;34;16 comment by...Pedro -->
Done TO for crate b0svt09, b0svt04. Manual HRR recovered it.


<!-- Date: Sat Dec 8 11:35:42.515 2007 -->
 If problem persists, please use the DiagnoseTO button  
 on the EVB Ace Control Panel. Auto-HRR recovered it.
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928' target=_top>254928</a>) -->
<!-- Comment: Sat Dec  8 11;36;13 comment by...Pedro -->
Again

<!-- Comment: Sat Dec  8 11;37;10 comment by...Pedro -->
Again

<!-- Comment: Sat Dec  8 11;37;50 comment by...Pedro -->
Again


<!-- Comment: Sat Dec  8 11;44;09 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 11;46;28 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 11;48;45 comment by...ronmoore -->
LOAD_ERRORs from b0mutr00...
<!-- Comment: Sat Dec  8 11;54;24 comment by...Pedro -->
I had three more BUSY TIMEOUT !
<!-- Comment: Sat Dec  8 12;14;35 comment by...Pedro -->
After shepherded b0mutr00, I had BUSY TIMEOUT again.
<!-- Comment: Sat Dec  8 12;14;49 comment by...Pedro -->
again
<!-- Comment: Sat Dec  8 12;15;52 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 12;16;05 comment by...Pedro -->
again
<!-- Comment: Sat Dec  8 12;22;28 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 12;22;47 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 12;28;45 comment by...Pedro -->
I shepherded again, now is fine.
<!-- Comment: Sat Dec  8 13;15;58 comment by...Pedro -->
We got Again Busy Timeout.
<!-- Comment: Sat Dec  8 13;18;10 comment by...Pedro -->
Again
<!-- Date: Sat Dec 8 11:43:00.078 2007 -->
PHA is red in DQMon.
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928' target=_top>254928</a>) -->






<!-- Date: Sat Dec 8 12:03:13.369 2007 -->
I talked to Eric James regarding the EVB complaints about b0mutr00.  He suggested halting the run and power-cycling the crate.  We will try that now.
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 12;08;38 comment by...Pedro -->
We successfully shepherded again b0mutr00. 


<!-- Date: Sat Dec 8 12:10:24.435 2007 -->
The linac is having problems (LRF4 power amplifier), so we'll be keeping the current store longer than planned.  No estimate at this time.
<!-- Author: ronmoore -->


<!-- Date: Sat Dec 8 12:10:51.368 2007 -->
L2 Decision TO. HRR recovered it.
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928' target=_top>254928</a>) -->






<!-- Date: Sat Dec 8 12:21:30.675 2007 -->
RXPT Error for create b0imu01. I fixed it, now is working again.
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928' target=_top>254928</a>) -->
<!-- Comment: Sat Dec  8 12;29;00 comment by...ronmoore -->
I think the original problem was a HV trip of BMU SE 156 that happened just before the RXPT error.
<!-- Date: Sat Dec 8 12:26:49.237 2007 -->
I paged DAQ regarding the apparent data corruption from the muon trigger crate.  Bill Badgett responded and suggested sheperding the crate for now.  He will investigate more from home.
<!-- Author: ronmoore :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928' target=_top>254928</a>) -->
<!-- Comment: Sat Dec  8 12;34;25 comment by...Pedro -->
We end the run
<!-- Date: Sat Dec 8 12:30:53.392 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928>254928</A>
 Terminated at 2007.12.08 12:30:21
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 12:30:55.507 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928>254928</A>
TERMINATE: end run
<!-- Author: Pedro -->
<!-- Comment: Sat Dec  8 12;35;03 comment by...ronmoore -->
Bill Badgett requested us to stop the run so that he can run a data transfer test on the muon trigger crate.
<!-- Comment: Sat Dec  8 12;36;59 comment by...Pedro -->
Ricardo silicon expert said that the silicon still on.
<!-- Date: Sat Dec 8 12:33:22.140 2007 -->
<pre>--------------------- End of Report ----------------------------------- 
2007/12/08, 11:57:15 END: Diagnose TimeOuts 
2007/12/08, 12:15:15 BEGIN: Diagnose TimeOuts 
--- Report of LoadErrors and byte count errors within the last 20min--- 
LoadError at 2007/12/08 11:35:57 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:37:24 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:42:56 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:44:42 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:47:46 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:50:15 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:52:09 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:54:28 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 11:56:05 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 12:11:27 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 12:13:57 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 12:14:51 from MUON_TRIGGER_00. 
Total errors: 12. 
---------------------- End of Report ----------------------------------- 
2007/12/08, 12:15:52 END: Diagnose TimeOuts</pre>
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 12;34;04 comment by...ronmoore -->
EVB Busy Timeout debugging info for Bill Badgett.




<!-- Date: Sat Dec 8 12:36:30.542 2007 -->
<P> 
Ran busy and data transfer tests between b0mutr00  
and the b0vrb16 VRB crate.   So far, no errors. 
Will continue testing for a bit. 
<P>  
<!-- Author: badgett -->


<!-- Date: Sat Dec 8 12:43:18.588 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254930>254930</A>
ACTIVATE: RunSet: COLDSTART  Trigger: COSMICS [15,954,639] 
<!-- Author: Ricardo -->



<!-- Date: Sat Dec 8 12:48:23.504 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7248&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7248  width='300' align='top'></a></td><td>  </td></tr></table>
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 12;51;35 comment by...ronmoore -->
There was a separator spark (C49 vertical #2 minus plate) at 12:24.  We saw a small spike in losses and a small drop in luminosity.

<!-- Comment: Sat Dec  8 12;52;09 comment by...ronmoore -->
The BMU SE 156 HV trip happened a few minutes before the spark.


<!-- Date: Sat Dec 8 12:52:31.056 2007 -->
<P> 
There are sporadic data transfer timeouts in multiple  
channels in the VRB in crate b0vrb16.    The channel 0  
is the most consistently bad, but others fail, too.   
Suspect a bad VRB. 
<P>  
<!-- Author: badgett -->
<!-- Comment: Sat Dec  8 12;54;39 comment by...ronmoore -->
Bill called back and is coming in.  In the meantime, he suggested I try cycling the power on VRB16 or try reseating it.

<!-- Comment: Sat Dec  8 13;01;17 comment by...ronmoore -->
I reseated the card.  The Ace is cleaning up the EVB.

<!-- Comment: Sat Dec  8 13;10;04 comment by...ronmoore -->
To be clear, I reseated the card within its 6->9U adapter.
<!-- Date: Sat Dec 8 13:05:31.595 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254924>254924</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=254924 target=_runSummary>254924</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 13:08:15.874 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254931>254931</A>
 Activated at 2007.12.08 13:06:33
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 13:08:16.332 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254931>254931</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [2,1010,710] 
<!-- Author: Pedro -->
<!-- Comment: Sat Dec  8 13;10;43 comment by...Pedro -->
We are taking date again



<!-- Date: Sat Dec 8 13:19:13.106 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254931>254931</A>
 Terminated at 2007.12.08 13:18:49
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 13:19:14.556 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254931>254931</A>
TERMINATE: end run
<!-- Author: Pedro -->



<!-- Date: Sat Dec 8 13:19:14.223 2007 -->
Bill Badgett is here for the MUON_TRIGGER_00 / VRB 16 problem.
<!-- Author: ronmoore -->


<!-- Date: Sat Dec 8 13:58:07.875 2007 -->
Bill can't find a quick solution, so we're bringing the HV to standby except CLC.
<!-- Author: ronmoore -->


<!-- Date: Sat Dec 8 14:12:12.352 2007 -->
<P> 
Tried swapping VRB_CDF_16 in crate b0vrb16, slot 10, but  
no change.   Tried swapping the auxillary TART card in  
the same slot, but no change. 
<P>  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254932' target=_top>254932</a>) -->

<!-- Date: Sat Dec 8 14:12:45.846 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254932>254932</A>
 Activated at 2007.12.08 14:12:37
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 14:14:21.667 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254932>254932</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [2,1010,710] 
<!-- Author: Pedro -->


<!-- Date: Sat Dec 8 14:14:22.801 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254932>254932</A>
 Terminated at 2007.12.08 14:12:59
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 14:14:23.360 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254932>254932</A>
TERMINATE: no data
<!-- Author: Pedro -->
<!-- Comment: Sat Dec  8 14;20;34 comment by...Pedro -->
we are bringing HV on.
<!-- Date: Sat Dec 8 14:16:59.021 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254933>254933</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_5_01 [2,1010,710] 
<!-- Author: Pedro -->



<!-- Date: Sat Dec 8 14:17:28.942 2007 -->
<P> 
Ron found that the TXPT cable into b0mutr00's Tracer  
card was loose.    He reseated it... 
<P>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254933' target=_top>254933</a>) -->


<!-- Date: Sat Dec 8 14:27:53.167 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934>254934</A>
 Activated at 2007.12.08 14:27:16
<!-- Author: RunControl -->


<!-- Date: Sat Dec 8 14:27:54.561 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934>254934</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [2,1010,710] 
<!-- Author: Pedro -->
<!-- Comment: Sat Dec  8 14;29;45 comment by...Pedro -->
We are taking date again.
<!-- Comment: Sat Dec  8 14;31;56 comment by...Pedro -->
Busy Timeout again. We shepherded crate b0pcal07.
<!-- Comment: Sat Dec  8 14;34;08 comment by...Pedro -->
ISU is red in DQMon.
<!-- Comment: Sat Dec  8 14;36;02 comment by...Pedro -->
Now ISU is green.
<!-- Comment: Sat Dec  8 14;39;22 comment by...Pedro -->
Busy Timeout again.
<!-- Comment: Sat Dec  8 14;40;38 comment by...Pedro -->
Done for crate b0cot14. Manual HRR fixec it.
<!-- Comment: Sat Dec  8 14;41;34 comment by...Pedro -->
Busy Timeout again.
<!-- Comment: Sat Dec  8 14;42;00 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 14;43;03 comment by...Pedro -->
Done TO for crate b0cot14. Auto-HRR fixed it.
<!-- Comment: Sat Dec  8 14;45;02 comment by...Pedro -->
Busy Timeout again.
<!-- Comment: Sat Dec  8 14;46;09 comment by...Pedro -->
Done TO for crate b0cot14. Auto-HRR fixed it
<!-- Comment: Sat Dec  8 14;46;49 comment by...Pedro -->
Busy Timeout again.
<!-- Comment: Sat Dec  8 14;49;07 comment by...Pedro -->
Again
<!-- Comment: Sat Dec  8 14;49;47 comment by...Pedro -->
Done TO for crate b0cot14. Auto-HRR fixed it
<!-- Comment: Sat Dec  8 14;50;53 comment by...Pedro -->
PHA is red in DQMon.
<!-- Date: Sat Dec 8 14:29:30.605 2007 -->
Prior to Ron's finding the TXPT cable loose on b0mutr00,  
I had swapped the VRB inputs at the back of the crate between  
the adjacent b0mutr00 and b0svt07 to see if the problem  
follows the VRB input or stays with the crate.  We'll see...
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->




<!-- Date: Sat Dec 8 14:32:50.717 2007 -->
We just got a busy timeout again, with the EVB timeout  
diagnosis: 
 
<PRE> 
2007/12/08, 14:29:40 BEGIN: Diagnose TimeOuts 
--- Report of LoadErrors and byte count errors within the last 20min--- 
LoadError at 2007/12/08 14:29:02 from MUON_TRIGGER_00. 
Total errors: 1. 
</PRE> 
 
Will wait for another one to confirm... 
<P>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->






<!-- Date: Sat Dec 8 14:39:59.925 2007 -->
Another busy timeout, with report: 
 
<PRE> 
 
2007/12/08, 14:37:48 BEGIN: Diagnose TimeOuts 
--- Report of LoadErrors and byte count errors within the last 20min--- 
LoadError at 2007/12/08 14:29:02 from MUON_TRIGGER_00. 
LoadError at 2007/12/08 14:37:40 from MUON_TRIGGER_00. 
Total errors: 2. 
 
</PRE> 
 
So presumably the problem is really in b0mutr00.   The  
timeouts in the vrbtest program may have been a red  
herring. 
<P> 
We may need the EVB experts to give us more information  
and/or a raw dump of the problem events. 
<P>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->






<!-- Date: Sat Dec 8 14:43:32.862 2007 -->
<pre>Per Bill's request, paging L3/EVB Farm Support.</pre>
<!-- Author: ronmoore -->
<!-- Comment: Sat Dec  8 14;49;21 comment by...badgett -->
Khaldoun called back and I asked him for event fragment 
files from the MUON_TRIGGER_00 errors.   Also, more information 
as to the exact nature of the error would be nice...




<!-- Date: Sat Dec 8 14:52:28.024 2007 -->
A hopefully fuller error from SCPU 16 (which should be the one that is receiving from the Muon Trigger crate according to: http://www-cdfonline.fnal.gov/java/cdfdb/servlet/VRB): 
 
2007/12/08-14:45:58 (2/16) :scpu::Loader: data rate=720bytes/sec, evt rate=0 Hz. 
2007/12/08-14:45:59 (2/16) :Warning at event 39427, vrb slot 10, channel 8, byte count 8224: Third trailer word  
not equal 0x80000000. 
2007/12/08-14:45:59 (2/16) :LoadError at event 39427, vrb slot 10, channel 8, byte count 8224: Tracer words inco 
nsistant. 
2007/12/08-14:45:59 (2/16) :LoadError at event 39427, vrb slot 10, channel 8, byte count 8224: First trailer wor 
d not equal 0xaaaaaaaa. 
2007/12/08-14:45:59 (2/16) :LoadError at event 39427, vrb slot 10, channel 8, byte count 8224: Second trailer wo 
rd not equal 0x55555555. 
2007/12/08-14:45:59 (2/16) :EVB error caused by SCPU.  LoadBad - event failed inspection. 
 
 
I am also copying a raw event to b0gateway:~/khaldoun/20071207/
<!-- Author: khaldoun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->
<!-- Comment: Sat Dec  8 14;54;18 comment by...khaldoun -->
<pre>That should have been plain text:

A hopefully fuller error from SCPU 16 (which should be the one that is receiving from the Muon Trigger crate according to: http://www-cdfonline.fnal.gov/java/cdfdb/servlet/VRB): 

2007/12/08-14:47:28 (2/16) :scpu::Loader: data rate=986922bytes/sec, evt rate=92 Hz.
2007/12/08-14:47:57 (2/16) :Warning at event 44941, vrb slot 10, channel 8, byte count 8384: Third trailer word not equal 0x80000000.
2007/12/08-14:47:57 (2/16) :LoadError at event 44941, vrb slot 10, channel 8, byte count 8384: Tracer words inconsistant.
2007/12/08-14:47:57 (2/16) :LoadError at event 44941, vrb slot 10, channel 8, byte count 8384: Word count inconsistant with byte count.
2007/12/08-14:47:57 (2/16) :LoadError at event 44941, vrb slot 10, channel 8, byte count 8384: First trailer word not equal 0xaaaaaaaa.
2007/12/08-14:47:57 (2/16) :LoadError at event 44941, vrb slot 10, channel 8, byte count 8384: Second trailer word not equal 0x55555555.
2007/12/08-14:47:57 (2/16) :EVB error caused by SCPU.  LoadBad - event failed inspection.

I am also copying a raw event to b0gateway:~/khaldoun/20071207/
</pre>


<!-- Date: Sat Dec 8 14:59:22.613 2007 -->
<P> 
More observations about b0mutr00: 
<UL> 
<LI>The ErrorDisable flag for the tracer was set to true,  
which is not normally a good thing 
<LI>I have set this flag to false, and now we are getting  
the error line asserted from this crate instead of  
busy timeouts. 
<LI>I remember that in the past this crate is affected  
by TDC communication problems upstream. 
<LI>Tried shepherding all muon TDC crates, but still  
have problems. 
</UL> 
Hmm. 
<P>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->


<!-- Date: Sat Dec 8 15:28:39.023 2007 -->
Replaced the Tracer in b0mutr00;  also reënabled the  
ErrorDisable flag to true out of frustration. 
<P>  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->
<!-- Comment: Sat Dec  8 15;30;13 comment by...badgett -->
<P>
Old tracer serial number: 0101 <P>
New tracer serial number: 0158 <P>
<P>

<!-- Date: Sat Dec 8 15:30:40.071 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928>254928</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=254928 target=_runSummary>254928</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->



<!-- Date: Sat Dec 8 15:33:01.605 2007 -->
Watching the b0mutr00 front-end CPU console this whole time,  
I did not see any error messages or anything unusual.
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->

<!-- Date: Sat Dec 8 15:48:19.315 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254924>254924</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
Number of bad channels for "TOF East Ave Time" plot is 9.
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Sat Dec 8 15:53:56.426 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254928>254928</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T = corrupted data from muon trigger crate?
TOF = number of bad channels for "TOF East Ave Time" plot is 9.
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Sat Dec 8 15:54:35.475 2007 -->
I had two times this message: 
 
SVXMon not connected to ConsumerErrorReceiver (aka ConsumerMonitor) 
  since 2007-12-08 15:52:00.0 
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon 
  and these processes are very important to data taking! 
Restart SVXMon immediately!
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254934' target=_top>254934</a>) -->

<!-- Date: Sat Dec 8 15:56:02.872 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254924 target=_runSummary>254924</A> x3E3CC</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1010,710]</FONT></TD>
<TD ALIGN=CENTER>15:14:27</TD>
<TD ALIGN=CENTER>10:05:05</TD>
<TD ALIGN=CENTER>16:22:54</TD>
<TD ALIGN=RIGHT>   4,652,367</TD>   <TD ALIGN=RIGHT>  3,619.06</TD>
<TD ALIGN=RIGHT>  3,084.37</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254928 target=_runSummary>254928</A> x3E3D0</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1010,710]</FONT></TD>
<TD ALIGN=CENTER>11:27:21</TD>
<TD ALIGN=CENTER>12:30:21</TD>
<TD ALIGN=CENTER>00:27:35</TD>
<TD ALIGN=RIGHT>      46,263</TD>   <TD ALIGN=RIGHT>    109.32</TD>
<TD ALIGN=RIGHT>     47.33</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254931 target=_runSummary>254931</A> x3E3D3</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1010,710]</FONT></TD>
<TD ALIGN=CENTER>13:06:33</TD>
<TD ALIGN=CENTER>13:18:49</TD>
<TD ALIGN=CENTER>00:09:31</TD>
<TD ALIGN=RIGHT>      14,489</TD>   <TD ALIGN=RIGHT>     19.42</TD>
<TD ALIGN=RIGHT>     14.21</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254934 target=_runSummary>254934</A> x3E3D6</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1010,710]</FONT></TD>
<TD ALIGN=CENTER>14:27:16</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:36:20</TD>
<TD ALIGN=RIGHT>      83,255</TD>   <TD ALIGN=RIGHT>    121.81</TD>
<TD ALIGN=RIGHT>     50.15</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>17:36:21</TD>
<TD ALIGN=RIGHT>     4,796,374</TD> <TD ALIGN=RIGHT>  3,869.60</TD>
<TD ALIGN=RIGHT>  3,196.05</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sat Dec 8 15:57:39.677 2007 -->
<TABLE bgcolor=#cdc9a5> 
<tr><th colspan=2>Totals</th></tr> 
<tr><td>Date:</td><td>2007.12.08</td> 
<tr><td>Shift:</td><td>day</td> 
<tr><td>Delivered luminosity:</td><td> 785.4 nb<sup>-1</sup></td></tr> 
 
<tr><td>Acquired luminosity:</td><td> 309.5 nb<sup>-1</sup></td></tr> 
<tr><td>Efficiency:</td><td> 39.4</td></tr> 
</TABLE>
<!-- Author: ronmoore -->


<!-- Date: Sat Dec 8 16:04:07.312 2007 -->
<pre>Start shift with HEP store 5787, CDF lumi = 34 ub^-1/s, taking data smoothly. 
 
* 09:50 - PSM fan alarm for 1RR18E_2.  Lost the +5V for the muon trigger 
crate soon afterwards.  Stopped data-taking.  Talked to Dervin Allen; Roberto 
Davila coming in to replace the supply. 
 
* Roberto replaced the b0mutr00 power supply and recovered west plug power 
supply monitoring that had failed just as he arrived. 
 
* Event builder complains about corrupt data from b0mutr00.  Eric James 
suggested power cycling b0mutr00, but problem persists. 
 
* Linac problems are forcing store 5787 to run longer than scheduled. 
 
* Bill Badgett investigating muon trigger crate problem. 
 
* 12:24 - A Tevatron separator spark caused a small spike in losses and small drop in luminosity. 
 
* One silicon power supply #12 auto-hockerization. 
 
* After a few false starts, Bill eventually replaced the Tracer in the muon 
trigger crate - no Busy Timeouts since then. 
 
* Linac problems have forced store 5787 to linger on.  No plan yet from MCR. 
 
* Bill Badgett does not have the DAQ pager with him - if needed, call him @ 
630 699 8822 until 23:00 this evening. 
 
End shift still with HEP store 5787, CDF lumi = 22 ub^-1/s, taking data. 
</pre>
<!-- Author: romoore -->