<!-- Title: 09:46:31  Fri Sep 10 2004 -->
<!-- Crewchief: Aces -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>error messages</pre>

<!-- Date: Fri Sep 10 10:39:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187118">187118</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 10 10:40:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187118">187118</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal15 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 10 11:42:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187120">187120</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 10 11:42:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187120">187120</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 10 11:42:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187120">187120</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 13 15:49:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187129">187129</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 14:59:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187151">187151</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 16:06:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187154">187154</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 16:06:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187154">187154</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 17:29:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187157">187157</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 17:29:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187157">187157</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 17:29:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187157">187157</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 17:42:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187158">187158</A>
Abort: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 17:52:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187159">187159</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 17:52:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187159">187159</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 18:27:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187160">187160</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 18:29:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187160">187160</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 19:44:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187162">187162</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 19:44:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187162">187162</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 23:21:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187169">187169</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 23:21:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187169">187169</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 23:23:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187169">187169</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal03 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 14 23:23:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187169">187169</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 10:25:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187175">187175</A>
IDLE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0wcal01    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 16:38:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187193">187193</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0wcal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 16:40:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187193">187193</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0wcal01 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:06:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:06:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:06:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0clc00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:06:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:06:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0clc00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187198">187198</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0clc00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:21:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:21:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:21:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:22:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:23:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:23:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:23:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:23:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187199">187199</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:36:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187200">187200</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:36:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187200">187200</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:41:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187201">187201</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:41:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187201">187201</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:41:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187201">187201</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:45:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187202">187202</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Sep 15 18:45:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187202">187202</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 08:36:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187210">187210</A>
HRR: <PRE>   
  RXPT Error for crate/s: 
    b0imu00   b0cmx00    busy. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 08:37:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187210">187210</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 09:22:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187211">187211</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0wcal06   b0wcal07   b0htdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 09:23:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187211">187211</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 2 Event 2147 Run 187211 Bank HATD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 09:23:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187211">187211</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 2145 Run 187211 Bank HATD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 09:23:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187211">187211</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0wcal06   b0wcal07   b0htdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:58:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:58:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0l2de00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:58:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:58:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:58:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:59:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 13:59:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187217">187217</A>
START: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 18:56:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187233">187233</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal14   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 18:57:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187233">187233</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal14 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 16 18:58:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187233">187233</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 08:43:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187245">187245</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 08:44:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187245">187245</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 08:53:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187246">187246</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 08:55:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187246">187246</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 09:48:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187249">187249</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 09:48:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187249">187249</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 10:43:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187251">187251</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 17 14:41:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187272">187272</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
RECOVERED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cot03    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot03 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0mutr00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187296">187296</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187297">187297</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187297">187297</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187297">187297</A>
End: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cot05    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187297">187297</A>
End: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 98.27586 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187297">187297</A>
End: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 98.27586 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 09:28:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187298">187298</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 11:12:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187299">187299</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 11:12:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187299">187299</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 14:13:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187299">187299</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmu01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 14:13:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187299">187299</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmu01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 14:13:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187299">187299</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cmu01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 18 14:13:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187299">187299</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cmu01    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:25:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187309">187309</A>
CALIBRATING: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 50.0 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:25:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187310">187310</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:25:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187310">187310</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:26:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187310">187310</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:27:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187310">187310</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0mutr00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:29:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187310">187310</A>
End: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:46:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187311">187311</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 08:48:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187311">187311</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 14:10:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 14:10:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal04 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 14:10:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate CCAL_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 14:10:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 14:10:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 16:08:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate CCAL_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 16:08:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 16:08:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 16:08:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0ccal00   b0ccal01   b0ccal02   b0ccal03   b0ccal06   b0ccal07   b0ccal08   b0ccal09   b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 16:10:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 16:11:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187312">187312</A>
RECOVERED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 18:44:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187316">187316</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate CCAL_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 18:53:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187316">187316</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate WCAL_03). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 21:07:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187316">187316</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate LEVEL2_CAL_04). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 21:07:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187316">187316</A>
RECOVERED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 19 21:08:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187316">187316</A>
End: <PRE>   
  Busy TO. for crate/s: 
    b0ccal00   b0ccal01   b0ccal02   b0ccal03   b0ccal06   b0ccal07   b0ccal08   b0ccal09   b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 17:57:02 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187353">187353</A>
Config: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0imu00 
 - Please SHEPHERD crate b0imu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:00:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187353">187353</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:09:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187354">187354</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:10:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187354">187354</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:29:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187358">187358</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:29:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187358">187358</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:31:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187358">187358</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:31:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187358">187358</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:31:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187358">187358</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:34:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187359">187359</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:34:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187359">187359</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:34:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187359">187359</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:40:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187360">187360</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:40:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187360">187360</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:41:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187360">187360</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:41:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187360">187360</A>
End: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:41:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187360">187360</A>
End: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:47:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187361">187361</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:47:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187361">187361</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:48:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187361">187361</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 20 18:53:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187362">187362</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 10:59:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187379">187379</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 16:52:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187401">187401</A>
Config: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 18:28:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187402">187402</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 18:30:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187402">187402</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 18:30:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187402">187402</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 18:30:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187402">187402</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 21:08:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187406">187406</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 22:51:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187409">187409</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00    Done TO. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 22:51:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187409">187409</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 22:52:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187409">187409</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0mutr00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 22:52:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187409">187409</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00    Done TO. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 22:52:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187409">187409</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 21 22:54:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187409">187409</A>
Halt: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 09:58:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187456">187456</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 10:05:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187457">187457</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 10:05:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187457">187457</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 10:08:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187457">187457</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0etdc00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 16:49:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187479">187479</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 19:42:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187491">187491</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 19:43:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187491">187491</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 23 19:43:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187491">187491</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal02    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 10:31:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187505">187505</A>
Config: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@e63606
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096039615
  TimeNanoSeconds:  666666640
  SeqCSR0:          0x6d27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 10:41:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187506">187506</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@ed9f47
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096040118
  TimeNanoSeconds:  699999972
  SeqCSR0:          0x6d27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 10:41:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187506">187506</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1cee361
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096040118
  TimeNanoSeconds:  699999972
  SeqCSR0:          0x6d27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 11:07:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187507">187507</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@199f8e6
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096041652
  TimeNanoSeconds:  183333326
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 11:07:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187507">187507</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@12b9f14
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096041652
  TimeNanoSeconds:  183333326
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal10   b0ccal11   b0ccal12   b0ccal13   b0l2cl02   b0l2cl05    busy. 1 crate/s: b0l1cl04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@397d4e
  Status:           0x1740
  Fatal Errors:
    SEQ_CSR1_ERR_1_SYNC_MISSING
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096056414
  TimeNanoSeconds:  266666656
  SeqCSR0:          0xfd27
  SeqCSR1:          0xbe
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
HALTED: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1bb55d3
  Status:           0x1740
  Fatal Errors:
    SEQ_CSR1_ERR_1_SYNC_MISSING
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096056414
  TimeNanoSeconds:  266666656
  SeqCSR0:          0xfd27
  SeqCSR1:          0xbe
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0ccal10   b0ccal11   b0ccal12   b0ccal13   b0l2cl02   b0l2cl05    busy. 1 crate/s: b0l1cl04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   b0l1cl03   b0l1cl04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   b0l1cl03   b0l1cl04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:08:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
HALTED: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:10:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
HALTED: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:23:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:23:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187519">187519</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:46:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:46:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:48:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:48:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:48:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal04 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:48:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal04 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:50:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:50:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:50:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
IDLE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@d87431
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096058951
  TimeNanoSeconds:  149999994
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 15:50:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187528">187528</A>
IDLE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@17ea0fa
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096058951
  TimeNanoSeconds:  149999994
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 18:16:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187533">187533</A>
Config: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1c20d4
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096067297
  TimeNanoSeconds:  583333310
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 18:16:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187533">187533</A>
Config: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1022b73
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096067297
  TimeNanoSeconds:  583333310
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 22:00:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187540">187540</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 22:00:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187553">187553</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 22:00:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187554">187554</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@2146ca
  Status:           0x1700
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096075564
  TimeNanoSeconds:  149999994
  SeqCSR0:          0xfd27
  SeqCSR1:          0xbc
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 22:00:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187554">187554</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal11   b0ccal12   b0l2cl00   b0l2cl01   b0l2cl02   b0l2cl04   b0l2de00    busy. 1 crate/s: b0l1cl02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Sep 24 22:00:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187555">187555</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1c21ab9
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096076521
  TimeNanoSeconds:  16666666
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 06:23:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187558">187558</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 06:23:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187558">187558</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 06:23:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187558">187558</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 09:00:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187559">187559</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@15828f
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096093741
  TimeNanoSeconds:  66666664
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 09:00:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187560">187560</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 09:00:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187560">187560</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 09:00:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187560">187560</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@169f3b6
  Status:           0x1700
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096119714
  TimeNanoSeconds:  0
  SeqCSR0:          0xfd27
  SeqCSR1:          0xbc
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 09:00:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187560">187560</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal10   b0ccal12   b0ccal13   b0l2cl00   b0l2cl01   b0l2cl02   b0l2cl03   b0l2cl04   b0l2cl05   b0l2de00    busy. 1 crate/s: b0l1cl03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 09:00:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187560">187560</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl01   b0l1cl04   b0l1gl00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187561">187561</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@556efd
  Status:           0x1300
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096127664
  TimeNanoSeconds:  33333332
  SeqCSR0:          0xfd27
  SeqCSR1:          0x9c
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187561">187561</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187561">187561</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187561">187561</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   b0l1cl01   b0l1cl02   b0l1cl04   b0l1gl00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187562">187562</A>
Config: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1e9753f
  Status:           0x1300
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096128379
  TimeNanoSeconds:  83333330
  SeqCSR0:          0xfd27
  SeqCSR1:          0x9c
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187562">187562</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   b0l1cl03   b0l1cl04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 12:18:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187562">187562</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 18:30:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187563">187563</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@90579
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096150019
  TimeNanoSeconds:  183333326
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Sep 25 18:30:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187565">187565</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 26 10:17:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187569">187569</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@f77193
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096174541
  TimeNanoSeconds:  16666666
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 26 12:28:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187570">187570</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@d23cdf
  Status:           0x200
  Fatal Errors:
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096217007
  TimeNanoSeconds:  0
  SeqCSR0:          0xfd27
  SeqCSR1:          0x14
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Sep 26 21:07:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187573">187573</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 27 07:57:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187575">187575</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 27 07:57:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187575">187575</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Sep 27 07:57:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187575">187575</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 07:49:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187588">187588</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 07:49:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187588">187588</A>
ACTIVE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1c55cdc
  Status:           0x1740
  Fatal Errors:
    SEQ_CSR1_ERR_1_SYNC_MISSING
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096348962
  TimeNanoSeconds:  249999990
  SeqCSR0:          0xfd27
  SeqCSR1:          0xbe
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 07:50:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187588">187588</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 07:50:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187588">187588</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal10   b0ccal11   b0ccal12   b0ccal13   b0l1cl00   b0l1cl01   b0l1cl02   b0l1cl03   b0l1cl04   b0l1cl05   b0l2cl00   b0l2cl01   b0l2cl02   b0l2cl03   b0l2cl04   b0l2cl05    busy. 1 crate/s: b0l1cl05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 07:50:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187588">187588</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 09:40:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187593">187593</A>
IDLE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@192db17
  Status:           0x1300
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096382411
  TimeNanoSeconds:  949999962
  SeqCSR0:          0xfd27
  SeqCSR1:          0x9c
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:19:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187598">187598</A>
End: <PRE>   
  Done TO. for crate/s: 
    b0cot06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:24:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187599">187599</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:26:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187599">187599</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:26:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187599">187599</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal04 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:26:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187599">187599</A>
End: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:32:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187600">187600</A>
End: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:54:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187601">187601</A>
HALTED: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:54:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187601">187601</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:54:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187601">187601</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0etdc00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:55:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187601">187601</A>
End: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0etdc00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 10:55:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187601">187601</A>
End: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 11:01:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187602">187602</A>
Config: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@197522
  Status:           0x300
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
  Time: :           1096387272
  TimeNanoSeconds:  833333300
  SeqCSR0:          0xfd27
  SeqCSR1:          0x1c
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 11:11:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187602">187602</A>
Config: <PRE>   Attention !!!.  L3CalibTag not found  Error !!!. 
 We RECOMMEND the following steps: 
 - Please page L3 Filter software expert immediately. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 12:30:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187603">187603</A>
READY: <PRE>   Attention !!!.  L3CalibTag not found  Error !!!. 
 We RECOMMEND the following steps: 
 - Please page L3 Filter software expert immediately. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 13:16:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187603">187603</A>
IDLE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@9fbddf
  Status:           0x1300
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1096395370
  TimeNanoSeconds:  16666666
  SeqCSR0:          0xfd27
  SeqCSR1:          0x9c
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 13:33:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187604">187604</A>
IDLE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 15:13:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187608">187608</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 15:13:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187608">187608</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal13    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 15:13:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187608">187608</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 16:56:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187612">187612</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal14   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 16:58:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187612">187612</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal14 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 17:27:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187613">187613</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 18:02:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187619">187619</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 19:35:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187634">187634</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 19:35:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187634">187634</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 19:42:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187637">187637</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 19:42:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187637">187637</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0cot06. 
 - please SHEPHERD the offending TDC crate (b0cot06) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 20:43:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187638">187638</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Sep 28 20:45:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187638">187638</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 08:12:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187663">187663</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 08:12:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187663">187663</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 08:12:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187663">187663</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 10:45:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187672">187672</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187674">187674</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187674">187674</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0imu00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187674">187674</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187674">187674</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187679">187679</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187679">187679</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187680">187680</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187681">187681</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:34:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187681">187681</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 20:53:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187681">187681</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Sep 30 21:20:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187681">187681</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 1 12:00:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187684">187684</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 1 12:00:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187684">187684</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 1 12:06:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187685">187685</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 1 12:20:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187686">187686</A>
Config: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 1 12:20:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187686">187686</A>
Config: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 1 18:30:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187699">187699</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 08:03:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187702">187702</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 08:05:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187702">187702</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 08:12:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187702">187702</A>
Partition: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 09:51:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187703">187703</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 09:51:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187703">187703</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 09:53:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187703">187703</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:01:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187708">187708</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:01:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187708">187708</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:09:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187709">187709</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:11:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187709">187709</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:36:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187711">187711</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:36:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187711">187711</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 2 15:59:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187713">187713</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 12:09:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187738">187738</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187739">187739</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187739">187739</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187740">187740</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187740">187740</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187741">187741</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187741">187741</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187742">187742</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187674">187674</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 3 17:07:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187674">187674</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 08:01:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187752">187752</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 08:15:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187754">187754</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 08:15:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187754">187754</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 10:39:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187754">187754</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0wcal06 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:20:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187769">187769</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0xft02 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:20:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187769">187769</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0xft00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:20:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187769">187769</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cmx01 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:21:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187769">187769</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:22:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187769">187769</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0xft04 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:31:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:31:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:33:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0etdc00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:33:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot09   b0cot11   b0cot17   b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:33:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_18). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:33:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot09   b0cot11   b0cot17   b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:35:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot17 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:35:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot11 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:35:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot09 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:36:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot18 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:43:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0etdc00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 11:43:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187772">187772</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 12:00:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187775">187775</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 12:00:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187775">187775</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 4 14:35:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187775">187775</A>
End: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_19). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 08:12:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187791">187791</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 08:12:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187791">187791</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 08:12:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187791">187791</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 08:12:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187791">187791</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 08:12:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187791">187791</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:28:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187805">187805</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:30:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187805">187805</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal06 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:37:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187806">187806</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:38:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187806">187806</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:40:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187806">187806</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:42:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187806">187806</A>
IDLE: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@1f2f60d
  Status:           0x1740
  Fatal Errors:
    SEQ_CSR1_ERR_1_SYNC_MISSING
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1097005151
  TimeNanoSeconds:  433333316
  SeqCSR0:          0xfd27
  SeqCSR1:          0xfe
  PccDelay:         0xffb0
  PccCSR:           0xff00 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:53:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187807">187807</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal08   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 14:57:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187807">187807</A>
Recover: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:38:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187809">187809</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:38:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187809">187809</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 5 15:59:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187811">187811</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 6 11:31:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187831">187831</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 6 11:33:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187831">187831</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 14:21:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187875">187875</A>
HRR: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0l2de00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 15:12:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187879">187879</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 15:12:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187879">187879</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 15:41:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187880">187880</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 16:06:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187885">187885</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 16:06:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187885">187885</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 16:06:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187885">187885</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 16:08:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187885">187885</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 17:48:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187887">187887</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 19:07:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187887">187887</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 21:28:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187894">187894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 21:28:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187894">187894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 21:30:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187894">187894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 21:31:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187894">187894</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 21:42:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187894">187894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 7 21:57:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187894">187894</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 8 08:00:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187896">187896</A>
Abort: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 8 19:00:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187945">187945</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 8 22:32:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187949">187949</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 9 10:26:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187958">187958</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0etdc00    Done TO. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 9 10:28:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187958">187958</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0etdc00    Done TO. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 10 07:56:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187965">187965</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 10 11:25:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187967">187967</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 10 14:04:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187978">187978</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 10 20:14:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187987">187987</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 11 08:05:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187989">187989</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 11 09:18:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187990">187990</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 11 09:40:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187990">187990</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 11 09:50:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187990">187990</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 11 09:55:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187990">187990</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 11 21:24:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=187991">187991</A>
Run: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 12 13:47:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188026">188026</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 12 13:47:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188026">188026</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 12 13:47:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188026">188026</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 12 13:47:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188026">188026</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 12 21:38:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188050">188050</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 07:43:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188051">188051</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 07:43:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188051">188051</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 08:24:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188051">188051</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 08:48:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188051">188051</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 09:18:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188052">188052</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 09:50:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188055">188055</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:33:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188055">188055</A>
HRRinProgress: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb15
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:33:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188055">188055</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0ccal12    Done TO. 1 crate/s: b0ccal12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:33:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188055">188055</A>
HRR: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate PCAL_03). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:35:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188055">188055</A>
Abort: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:45:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188056">188056</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:50:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:52:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:52:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:52:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal06 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:52:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:52:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal06    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:52:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal06    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:53:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:53:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:53:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal06 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:54:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:55:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 10:55:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188057">188057</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 12:56:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188060">188060</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 15:05:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188065">188065</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0ccal09   b0ccal11   b0ccal13    Done TO. 4 crate/s: b0ccal10   b0ccal12   b0ccal14   b0ccal15    busy. 1 crate/s: b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 15:05:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188065">188065</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal09   b0ccal11   b0ccal13    Done TO. 4 crate/s: b0ccal10   b0ccal12   b0ccal14   b0ccal15    busy. 1 crate/s: b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 15:06:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188065">188065</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal09   b0ccal11   b0ccal13    Done TO. 4 crate/s: b0ccal10   b0ccal12   b0ccal14   b0ccal15    busy. 1 crate/s: b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 19:13:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188097">188097</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 19:13:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188097">188097</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal11    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 19:21:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188098">188098</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 13 19:28:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188098">188098</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:01:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188102">188102</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:03:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188102">188102</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:03:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188102">188102</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:22:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188103">188103</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:35:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188104">188104</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:36:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188104">188104</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:36:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188104">188104</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:36:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188104">188104</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal13    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:38:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188104">188104</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 09:48:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188106">188106</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 11:20:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188107">188107</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 11:20:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188107">188107</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 11:20:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188107">188107</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 11:20:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188107">188107</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 12:33:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188109">188109</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:16:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188111">188111</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0xtrp00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:27:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:27:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:27:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:27:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:28:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:28:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:28:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:28:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:28:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:29:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal05    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 14 13:29:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188110">188110</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal05 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 07:56:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188168">188168</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 07:56:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188168">188168</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 07:56:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188168">188168</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:17:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188170">188170</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:26:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188172">188172</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:28:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188172">188172</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal11 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:28:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188172">188172</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:34:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188173">188173</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:36:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188173">188173</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 08:42:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188173">188173</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 09:22:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188175">188175</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 09:23:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188175">188175</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 15:20:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188189">188189</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 15:44:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188189">188189</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 15:45:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188189">188189</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 15 15:46:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188189">188189</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal11 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 09:01:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188241">188241</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal15    Done TO. 2 crate/s: b0ccal14   b0ccal15    busy. 1 crate/s: b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 12:38:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188248">188248</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 13:28:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188249">188249</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0wcal07    Done TO. 1 crate/s: b0wcal07    busy. 1 crate/s: b0wcal07   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 16:24:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188250">188250</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 17:05:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188251">188251</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 17:06:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188251">188251</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 21:50:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188252">188252</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 18 23:00:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188252">188252</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 08:34:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188253">188253</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 08:35:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188253">188253</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb15
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 08:55:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188254">188254</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 15:55:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188283">188283</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 15:57:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188283">188283</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:23:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188285">188285</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:24:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188285">188285</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:24:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188285">188285</A>
HRRinProgress: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal11    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:24:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188285">188285</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:41:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188287">188287</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:41:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188287">188287</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:42:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188287">188287</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:42:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188287">188287</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:57:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188289">188289</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 16:59:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188289">188289</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 17:07:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188290">188290</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 17:07:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188290">188290</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 17:12:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188291">188291</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 18:34:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188293">188293</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 18:34:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188293">188293</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 19:17:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188293">188293</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 19:19:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188293">188293</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 20:05:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188293">188293</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb15
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 20:05:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188293">188293</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate CCAL_15). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 20:11:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188294">188294</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 19 20:17:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188295">188295</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 20 10:27:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188314">188314</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 20 10:27:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188314">188314</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 20 12:46:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188320">188320</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0ccal00   b0ccal01    Done TO. 1 crate/s: b0ccal00    busy. 1 crate/s: b0ccal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 20 12:46:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188320">188320</A>
HRR: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal00   b0ccal01    Done TO. 1 crate/s: b0ccal00    busy. 1 crate/s: b0ccal01    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 20 17:18:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188328">188328</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate CCAL_15). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Oct 20 19:35:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188329">188329</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0pcal10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 21 12:11:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188349">188349</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal08   b0ccal09   b0ccal14   b0pcal10   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 21 16:44:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188366">188366</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   b0ccal14   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 21 19:29:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188371">188371</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Oct 21 20:13:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188375">188375</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 10:58:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188402">188402</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 12:58:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188424">188424</A>
Config: <PRE>   </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 13:15:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188425">188425</A>
Recover: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 13:16:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188425">188425</A>
End: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 15:55:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188438">188438</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_03). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 16:11:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188439">188439</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 16:12:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188439">188439</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 18:33:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188448">188448</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 18:33:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188448">188448</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 18:37:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188448">188448</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 18:46:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188448">188448</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 18:53:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188448">188448</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 18:59:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188448">188448</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 19:07:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188454">188454</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 19:15:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188454">188454</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 19:21:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188454">188454</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 19:28:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188454">188454</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 19:35:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188454">188454</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 19:54:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:18:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:24:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:25:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:25:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:25:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:25:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:26:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:26:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:27:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
End: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:27:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb23 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 22 20:27:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188461">188461</A>
START: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0l1cl00   b0l1cl02   b0l1cl05    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 23 12:02:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188479">188479</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 23 22:06:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188481">188481</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 23 22:06:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188481">188481</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 23 22:54:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188482">188482</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 23 22:54:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188482">188482</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 23 22:54:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188482">188482</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 24 12:48:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188484">188484</A>
HRRinProgress: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 24 12:49:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188484">188484</A>
Recover: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 24 12:49:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188484">188484</A>
Recover: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Oct 24 13:07:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188485">188485</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 15:23:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188510">188510</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 17:14:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188516">188516</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 17:14:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188516">188516</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 17:17:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188516">188516</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 17:17:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188516">188516</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 17:46:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188516">188516</A>
Partition: <PRE> 
 Attention !!!. Event Builder EVB_PROXY_SM_TIMEOUT Error !!!. 
 Event Builder proxy lost connection to Scanner Manager. 
 We RECOMMEND the following steps:
 - End of run, Reset
 - Clean up EVB 			    
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 17:51:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188517">188517</A>
CALIBRATING: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:26:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188522">188522</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:32:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:32:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:32:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:32:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:32:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:33:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:34:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:34:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188523">188523</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0etdc00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:52:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188524">188524</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:52:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188524">188524</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:54:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188524">188524</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 20:54:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188524">188524</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Oct 25 22:20:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188526">188526</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 16:46:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188567">188567</A>
Abort: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 16:46:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188567">188567</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:00:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188571">188571</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 100.0 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:03:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188571">188571</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal14 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:03:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188571">188571</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal15 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:03:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188571">188571</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0etdc00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:03:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188571">188571</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal10 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:03:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188571">188571</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal11 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:35:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188574">188574</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal11 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:37:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188574">188574</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:44:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188575">188575</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:45:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188575">188575</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 17:45:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188575">188575</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 18:26:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188580">188580</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 18:26:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188580">188580</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 18:33:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188581">188581</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 18:38:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188582">188582</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 18:38:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188582">188582</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:08:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:08:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:08:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:15:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:15:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:15:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:19:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 19:50:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188583">188583</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Oct 26 23:14:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188584">188584</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   b0ccal05   b0ccal07   b0ccal08   b0ccal10   b0ccal12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 12:44:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188717">188717</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 16:28:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188734">188734</A>
READY: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:36:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188742">188742</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:36:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188742">188742</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:38:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188742">188742</A>
Recover: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:38:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188742">188742</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
RECOVERED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
HRRinProgress: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0xft05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
START: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0xft05    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 18:46:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188743">188743</A>
-: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 19:27:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188748">188748</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 19:27:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188748">188748</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 19:32:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188748">188748</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 19:32:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188748">188748</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Oct 29 19:48:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188750">188750</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 15:55:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:05:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:05:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:05:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:05:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:05:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:23:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:25:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 16:52:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 17:07:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 17:19:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 17:37:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 17:48:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 17:50:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:00:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:08:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:17:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:19:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:37:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:38:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_03). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:46:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 18:57:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 19:03:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 19:23:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 19:23:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 19:31:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 19:34:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 19:44:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:01:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:02:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:08:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:11:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:13:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:25:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_11). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:25:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_11). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:37:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 20:59:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Oct 30 21:42:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188759">188759</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 09:52:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188767">188767</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 09:54:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188767">188767</A>
Recover: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0xft00   b0xft02   b0xft04    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 09:54:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188767">188767</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 16:03:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188779">188779</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 16:03:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188779">188779</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 16:06:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188779">188779</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal09 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 16:08:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188779">188779</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal04   b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 17:45:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188785">188785</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 17:45:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188785">188785</A>
HALTED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 17:45:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188785">188785</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 17:49:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188785">188785</A>
START: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 18:46:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188787">188787</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 18:59:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188788">188788</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 18:59:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188788">188788</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 18:59:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188788">188788</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 18:59:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188788">188788</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 18:59:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188788">188788</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:03:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188788">188788</A>
START: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:14:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188789">188789</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:14:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188789">188789</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:31:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188791">188791</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:32:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188791">188791</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:32:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188791">188791</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 1 19:32:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188791">188791</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:35:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:38:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:40:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:44:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:44:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:44:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:47:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:49:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:49:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:53:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:54:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:54:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:56:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:56:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:56:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 18:56:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188826">188826</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 19:13:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188831">188831</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0pcal08   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 19:14:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188831">188831</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal08    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 19:24:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188831">188831</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 19:29:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188831">188831</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 19:32:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188831">188831</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 19:32:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188831">188831</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 20:06:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188846">188846</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TS_LEVEL2_ERROR !!! 
 
 Level 2 Error  
 AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 20:07:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188846">188846</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 21:04:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188856">188856</A>
HRR: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate PCAL_03). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 2 21:11:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188856">188856</A>
Abort: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 10:43:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188865">188865</A>
Partition: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 10:52:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188865">188865</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb23 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 11:09:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188868">188868</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 11:23:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188869">188869</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:21:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0ccal01   b0ccal02   b0ccal06   b0ccal08   b0ccal09   b0ccal10   b0ccal11   b0ccal13   b0ccal14   b0ccal15   b0fcal00   b0pcal06   b0pcal09   b0svt00   b0svt01   b0svt07   b0svt02   b0svt06   b0svt03   b0svt05   b0svt04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:22:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
HRR: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:22:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
HRR: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:22:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
HRR: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:22:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
HRR: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:22:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
HRR: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:22:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
HRR: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:23:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188880">188880</A>
START: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:41:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188883">188883</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   b0ccal06   b0pcal02   b0pcal03   b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:41:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188883">188883</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   b0ccal02   b0pcal02   b0pcal03   b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:44:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188884">188884</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   b0pcal02   b0pcal03   b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 12:45:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188884">188884</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   b0ccal06   b0pcal02   b0pcal03   b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 14:47:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188893">188893</A>
START: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal07 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 14:54:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188894">188894</A>
Partition: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 14:54:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188894">188894</A>
Partition: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 14:57:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188895">188895</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0ccal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 14:57:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188895">188895</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal05    busy. 1 crate/s: b0ccal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:06:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188896">188896</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0ccal02   b0ccal04    busy. 1 crate/s: b0ccal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:09:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:10:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal00   b0ccal01   b0ccal03   b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:10:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:11:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:12:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:12:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
HALTED: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:13:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:13:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:14:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal06 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 15:14:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188897">188897</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 19:49:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188917">188917</A>
ACTIVE: <PRE>  
 Attention!!!. CSL_LOGGER_LAST_RUN Error !!!.  
 Events in this run will NOT be written to disk !!!.  
 We STRONGLY RECOMMEND the following steps: 
 - END-OF-RUN 
 - Reset Transition 
 - Partition Transition 			    
 Re-starting of CSL should NOT be necessary in this case. 
 
 >>>> Ignore this message if RC is NOT in ACTIVE State. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 20:42:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188920">188920</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2cl03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 3 21:58:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188921">188921</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 23 Run 188921 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 10:57:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188934">188934</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 10:57:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188934">188934</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 11:03:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188934">188934</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 12:37:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188951">188951</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 12:37:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188951">188951</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 12:38:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188951">188951</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 12:41:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188951">188951</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 12:41:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188951">188951</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 12:42:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188951">188951</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 14:00:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188953">188953</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 14:00:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188953">188953</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 14:01:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188953">188953</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 14:02:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188953">188953</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 14:03:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188953">188953</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 14:05:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188953">188953</A>
IDLE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 15:25:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188956">188956</A>
START: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 17:08:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188959">188959</A>
Recover: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 17:08:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188959">188959</A>
HALTED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 17:11:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188959">188959</A>
START: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 17:41:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188962">188962</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 17:43:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188962">188962</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 20:53:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188978">188978</A>
START: <PRE>  
 Attention!!!. CSL_LOGGER_LAST_RUN Error !!!.  
 Events in this run will NOT be written to disk !!!.  
 We STRONGLY RECOMMEND the following steps: 
 - END-OF-RUN 
 - Reset Transition 
 - Partition Transition 			    
 Re-starting of CSL should NOT be necessary in this case. 
 
 >>>> Ignore this message if RC is NOT in ACTIVE State. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 21:21:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188981">188981</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 21:26:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188982">188982</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 21:50:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188983">188983</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb12
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb12 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 4 22:24:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188986">188986</A>
START: <PRE>  
 Attention!!!. CSL_LOGGER_LAST_RUN Error !!!.  
 Events in this run will NOT be written to disk !!!.  
 We STRONGLY RECOMMEND the following steps: 
 - END-OF-RUN 
 - Reset Transition 
 - Partition Transition 			    
 Re-starting of CSL should NOT be necessary in this case. 
 
 >>>> Ignore this message if RC is NOT in ACTIVE State. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 08:55:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188996">188996</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 08:56:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188996">188996</A>
START: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 08:56:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188996">188996</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 08:59:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188996">188996</A>
START: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:03:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:03:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:04:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:04:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:05:02 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0xft00   b0xft02   b0xft04    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:05:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:05:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188997">188997</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:11:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188998">188998</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:11:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188998">188998</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:11:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188998">188998</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:19:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188999">188999</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:20:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188999">188999</A>
RECOVERED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:20:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188999">188999</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:37:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=188999">188999</A>
START: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:51:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189000">189000</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:51:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189000">189000</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 09:52:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189000">189000</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:03:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189009">189009</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:03:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189009">189009</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:03:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189009">189009</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb23 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:10:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189012">189012</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:10:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189012">189012</A>
HALTED: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:11:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189012">189012</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:11:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189012">189012</A>
HALTED: <PRE>    Attention!!!.   L1 Done Timeout LOOP!!!. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:11:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189012">189012</A>
HALTED: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:19:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189012">189012</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:36:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189017">189017</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:47:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189017">189017</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 11:47:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189017">189017</A>
HALTED: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 12:09:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189021">189021</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:51:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
HRR: <PRE>   
  RXPT Error for crate/s: 
    b0svx06   b0pcal08    busy. 1 crate/s: b0pcal08   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:51:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
HRR: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   b0pcal08    busy. 1 crate/s: b0pcal08   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:51:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:53:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
RECOVERED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal08 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:55:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:55:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 13:56:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189026">189026</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0pcal02   b0pcal09   b0pcal10    busy. 1 crate/s: b0pcal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:12:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189036">189036</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0svt01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:13:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189036">189036</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0svt00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:13:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189036">189036</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0svt02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:14:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189036">189036</A>
READY: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0svt03 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:38:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:38:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:38:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:38:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:39:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:39:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0xft00   b0xft02   b0xft04    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:39:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189039">189039</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:46:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189040">189040</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0xft00   b0xft02   b0xft04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 5 15:47:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189040">189040</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 8 08:26:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189134">189134</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal15    Done TO. 1 crate/s: b0ccal15    busy. 1 crate/s: b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 8 08:26:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189134">189134</A>
HRR: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal15    Done TO. 1 crate/s: b0ccal15    busy. 1 crate/s: b0ccal15    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 10 14:03:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189312">189312</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal04 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 11 08:12:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189366">189366</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:08:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal07   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:08:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal07    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:09:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal07   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:47:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:47:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:48:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:48:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 08:48:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189440">189440</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:10:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:10:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:10:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:12:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:12:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:13:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
End: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 09:13:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189441">189441</A>
End: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 10:01:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189450">189450</A>
HRR: <PRE>   
  RXPT Error for crate/s: 
    b0wcal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 10:04:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189450">189450</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 10:07:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189450">189450</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:05:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189458">189458</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:07:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189458">189458</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:38:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189464">189464</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:39:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189464">189464</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:40:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189464">189464</A>
HRR: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0l2pu00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:41:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189464">189464</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 11:42:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189464">189464</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 12:06:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189467">189467</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 12:45:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189486">189486</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0imu01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 15:50:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189497">189497</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 15:54:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189500">189500</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 18:00:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189512">189512</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 12 18:06:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189512">189512</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate CMP_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 12:46:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189525">189525</A>
IDLE: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb11 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 15:06:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189526">189526</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 15:21:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189526">189526</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0pcal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 15:22:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189526">189526</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 20:17:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189532">189532</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 20:17:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189532">189532</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 20:17:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189532">189532</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal06    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 22:46:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189532">189532</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 13 22:46:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189532">189532</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 08:15:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189535">189535</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 09:47:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189536">189536</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CMP_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 09:48:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189536">189536</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:32:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:32:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:35:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
Recover: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:39:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmp00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:39:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:39:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmp00    Done TO. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:40:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cmp00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:41:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cmp00    Done TO. 1 crate/s: b0mutr00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:41:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0mutr00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:42:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 18:42:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189558">189558</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0mutr00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:18:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot01   b0cot06   b0cot14   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:19:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:19:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:20:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot06 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:22:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot14 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:22:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
HRR: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot06 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:33:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 20:33:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:03:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:04:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:04:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:10:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:11:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal05 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:16:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:19:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:19:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:20:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:20:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal02    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 14 21:22:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189562">189562</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 11:34:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189576">189576</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_11). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 18:24:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189615">189615</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 18:25:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189615">189615</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 18:25:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189615">189615</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 18:29:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189615">189615</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:04:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:06:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal05 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:06:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CMP_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:07:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:09:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:11:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 19:11:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189617">189617</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Nov 15 20:18:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189619">189619</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 16:10:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189694">189694</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 16:11:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189694">189694</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 18:00:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189713">189713</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 18:02:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189713">189713</A>
-: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 18:50:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189714">189714</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 18:51:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189714">189714</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 20:13:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189715">189715</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 20:14:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189715">189715</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 21:58:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 22:21:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 23:03:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 23:03:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 23:06:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0l2pu00 
 - Please SHEPHERD crate b0l2pu00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 23:07:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 16 23:18:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189717">189717</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 12:56:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189741">189741</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal00   b0ccal12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 16:53:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189743">189743</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 18:08:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189754">189754</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 18:08:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189754">189754</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 18:09:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189754">189754</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cot03    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 18:09:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189754">189754</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 18:10:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189754">189754</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot03 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 18:10:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189754">189754</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:16:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189757">189757</A>
CALIBRATING: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:54:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189760">189760</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:54:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189760">189760</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:55:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189760">189760</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0etdc00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:55:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189760">189760</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0etdc00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:56:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189760">189760</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0etdc00. 
 - please SHEPHERD the offending TDC crate (b0etdc00) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Nov 17 21:57:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189760">189760</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:09:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done     
 - More > 1 crate in Done TO.     
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:09:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:09:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0etdc00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:11:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 2.1276596 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:11:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0etdc00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:11:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 2.1276596 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 18 22:11:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189856">189856</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0etdc00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 19 23:30:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189931">189931</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 19 23:31:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189931">189931</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 20 08:25:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189934">189934</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13 (forwarded by FER crate CCAL_14). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Nov 20 16:30:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=189945">189945</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Nov 21 07:55:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190005">190005</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 23 09:39:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190098">190098</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 25 08:56:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190208">190208</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Nov 25 13:03:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190208">190208</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 26 08:10:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190222">190222</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb11 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Nov 26 09:49:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190225">190225</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal00   b0ccal01   b0ccal02   b0ccal03   b0cot00   b0cot01   b0cot02   b0cot03   b0cmp00   b0musc00   b0cmx00   b0cmx01   b0svt00   b0svt01   b0svt07   b0svt02   b0svt06   b0svt03   b0svt05   b0svt04   b0l1cl00   b0l1cl01   b0l1cl02   b0l1cl03   b0puls00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 09:27:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190336">190336</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 09:29:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190336">190336</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 09:33:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190336">190336</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 09:33:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190336">190336</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 09:38:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190336">190336</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 10:04:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190337">190337</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 10:23:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190338">190338</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 10:45:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190339">190339</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal12    Done TO. 3 crate/s: b0ccal10   b0ccal11   b0ccal12    busy. 1 crate/s: b0ccal10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 12:12:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190355">190355</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 12:12:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190355">190355</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 13:03:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190358">190358</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 13:04:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190358">190358</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal11 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 13:19:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190358">190358</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 13:33:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190358">190358</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 13:34:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190358">190358</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Nov 30 13:35:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190358">190358</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 1 14:35:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190409">190409</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 1 16:32:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190423">190423</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 1 16:32:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190423">190423</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 1 19:10:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190429">190429</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 2 09:13:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190438">190438</A>
Config: <PRE>   Attention !!!.  L3CalibTag not found  Error !!!. 
 We RECOMMEND the following steps: 
 - Please page L3 Filter software expert immediately. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 2 22:28:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190474">190474</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 4 21:22:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190571">190571</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 5 10:45:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190578">190578</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 09:21:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190622">190622</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 09:46:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190622">190622</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot19   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 09:50:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190622">190622</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal11 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 10:39:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190634">190634</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 10:45:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190634">190634</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 10:45:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190634">190634</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 13:26:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190639">190639</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 13:34:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190639">190639</A>
Partition: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 13:34:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190639">190639</A>
Partition: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal01 
 - Please SHEPHERD crate b0pcal01 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 21:58:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190663">190663</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 6 22:03:35 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190663">190663</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 03:11:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190668">190668</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 06:14:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190672">190672</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 06:46:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190673">190673</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate CCAL_14). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 06:48:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190673">190673</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 09:45:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190681">190681</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0cot07. 
 - please SHEPHERD the offending TDC crate (b0cot07) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 09:45:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190681">190681</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13 (forwarded by FER crate IMU_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 10:56:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190683">190683</A>
READY: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot08 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 14:55:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190695">190695</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 7 14:55:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190695">190695</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 00:49:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190727">190727</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal07   b0pcal08   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 00:49:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190727">190727</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:15:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:25:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:29:48 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:35:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:38:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:39:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 01:44:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190730">190730</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:33:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:35:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:38:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:41:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:45:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:45:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:49:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 02:58:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 03:00:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190731">190731</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:04:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:08:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:13:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:16:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:21:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:25:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:38:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:42:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:54:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 04:56:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 05:09:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 05:35:38 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190735">190735</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 07:11:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 07:13:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 07:13:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 07:21:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 07:56:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 08:27:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 08:38:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 08:38:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 09:21:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190736">190736</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 11:08:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190738">190738</A>
HRR: <PRE>   
  RXPT Error for crate/s: 
    b0clc00    busy. 1 crate/s: b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 8 12:42:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190751">190751</A>
Config: <PRE>  
 Attention!!!. CSL_LOGGER_CURRENT_RUN Error !!!. 
 Events in this run will NOT be written to disk !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - END-OF-RUN 
 - Reset Transition 
 - Partition Transition 			    
 Re-starting of CSL should NOT be necessary in this case. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 9 00:43:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190768">190768</A>
CALIBRATING: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0fcal00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 10 01:36:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190863">190863</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 10 16:15:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190891">190891</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 10 16:16:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190891">190891</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 10 16:17:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190891">190891</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 10 16:21:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190891">190891</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 02:11:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190909">190909</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 2 Event 62 Run 190909 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 02:19:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190910">190910</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 47 Event 4 Run 190910 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 02:19:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190910">190910</A>
End: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 46 Event 3 Run 190910 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 02:37:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190913">190913</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 76 Event 10 Run 190913 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 04:34:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190920">190920</A>
End: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 16 Event 40 Run 190920 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 05:55:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190921">190921</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 06:31:02 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190921">190921</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 07:23:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190922">190922</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 07:47:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190922">190922</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 11 19:24:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190932">190932</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 12 23:58:04 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190982">190982</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_11). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 00:29:08 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=190982">190982</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_07). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 18:07:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191022">191022</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 20:04:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191041">191041</A>
IDLE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal05   b0pcal10   b0pcal11    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 20:32:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191044">191044</A>
Config: <PRE>   </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 20:45:19 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191045">191045</A>
Config: <PRE>   </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 21:41:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191047">191047</A>
CALIBRATING: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal00 
 - Please SHEPHERD crate b0pcal00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 13 23:36:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191057">191057</A>
DONE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal00 
 - Please SHEPHERD crate b0pcal00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 14 16:22:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191089">191089</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 14 16:22:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191089">191089</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 14 16:24:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191089">191089</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 14 19:14:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191098">191098</A>
PAUSED: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 2.3123732 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 10:59:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191208">191208</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB BadChannelCounts error in 
 SCPU = b0eb25, slot = 10. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb25 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 12:01:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191208">191208</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal08 
 - Please SHEPHERD crate b0pcal08 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 12:55:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191208">191208</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 13:24:21 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191209">191209</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 13:30:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191209">191209</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0fcal00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 13:30:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191209">191209</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_18). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 13:31:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191209">191209</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot00   b0cot01   b0cot02   b0cot04   b0cot05   b0cot06   b0cot07   b0cot08   b0cot09   b0cot10   b0cot11   b0cot12   b0cot13   b0cot14   b0cot15   b0cot16   b0cot17   b0cot18   b0cot19   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 13:58:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191210">191210</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0cot15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 16 14:00:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191210">191210</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cot15    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 02:00:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191281">191281</A>
START: <PRE> 
 Attention !!!. Event Builder EVB_PROXY_SM_TIMEOUT Error !!!. 
 Event Builder proxy lost connection to Scanner Manager. 
 We RECOMMEND the following steps:
 - End of run, Reset
 - Clean up EVB 			    
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 03:52:17 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191284">191284</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 04:29:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191286">191286</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 04:29:09 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191286">191286</A>
HALTED: <PRE> 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done for crate b0cot13. 
 - please SHEPHERD the offending TDC crate (b0cot13) 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 06:26:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191286">191286</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 07:06:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191286">191286</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 08:13:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191287">191287</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 08:13:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191287">191287</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 11:37:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191288">191288</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 11:39:34 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191288">191288</A>
Abort: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 11:56:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191289">191289</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 12:43:32 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 12:49:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:04:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:05:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:06:20 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:19:10 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
RECOVERED: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:27:27 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:27:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B1/W4/L2/C4-7 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:41:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 13:44:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 15:31:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 16:40:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191290">191290</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.20268 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 18:19:13 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191303">191303</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate XFT_LINKER_05). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 18:51:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191303">191303</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 19:50:12 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191303">191303</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 21:01:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191304">191304</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal08 
 - Please SHEPHERD crate b0pcal08 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 17 21:01:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191304">191304</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal08 
 - Please SHEPHERD crate b0pcal08 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 04:07:40 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191341">191341</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_03). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 05:03:56 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191341">191341</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 05:18:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191341">191341</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 05:50:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191342">191342</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 09:12:49 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191348">191348</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal00   b0ccal06   b0ccal10   b0ccal11   b0ccal13   b0pcal04   b0pcal07   b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 09:16:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191348">191348</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 19 15:14:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191349">191349</A>
START: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@73ba17
  Status:           0x18100
  Fatal Errors:
    SEQ_CSR1_ERR_3_SEQ_HOLD
    PCC_CSR_6_SYNC_TIMING_ERROR
    PCC_CSR_7_SYNC_MISSING_ERROR
  Time: :           1103490760
  TimeNanoSeconds:  16666666
  SeqCSR0:          0xed27
  SeqCSR1:          0xc
  PccDelay:         0xffb0
  PccCSR:           0xffc3 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 20 14:10:11 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191389">191389</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 21 16:30:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191472">191472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 21 16:32:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191472">191472</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 21 19:50:15 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191486">191486</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 2 Event 8 Run 191486 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 21 19:51:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191486">191486</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 24 Event 42 Run 191486 Bank COTD_StorableBank missing from event record 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 21 23:07:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191489">191489</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 21 23:08:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191489">191489</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 23 08:51:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191534">191534</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 99.82309 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 23 17:17:58 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191544">191544</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Dec 23 18:16:39 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.60428 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:19:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 44.020027 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:33:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 3519 Run 191545 Bank HATD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:33:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb15
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:33:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb15
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:33:46 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate TRIGGER_SCALERS_00). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:36:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 48.939796 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:38:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb25, slot = 11. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb25 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:38:30 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 24 09:40:18 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191545">191545</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 38.326935 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 00:14:47 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191558">191558</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 00:58:51 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.12407 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:01:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 99.83735 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:54:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:56:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:56:59 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:59:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:59:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:59:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:59:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:59:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 01:59:36 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 02:01:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 02:01:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191560">191560</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 04:01:07 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191565">191565</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 04:57:01 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191565">191565</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 06:32:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191567">191567</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 06:32:26 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191567">191567</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 06:39:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191567">191567</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb25, slot = 17. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb25 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 06:39:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191567">191567</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 07:40:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191567">191567</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 07:40:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191567">191567</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 10:12:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191568">191568</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 99.89353 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 10:32:03 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191568">191568</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb22, slot = 16. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb22 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 10:39:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191568">191568</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 12:27:23 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191568">191568</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 12:47:29 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191568">191568</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 12:57:25 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191568">191568</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 13:51:37 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191570">191570</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal10 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Dec 25 14:24:43 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191570">191570</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 26 00:47:16 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191586">191586</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate LEVEL2_CAL_04). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 26 14:54:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191588">191588</A>
Config: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal14 
 - Please SHEPHERD crate b0ccal14 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 26 14:54:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191588">191588</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 2 Event 21 Run 191588 Bank COTD_StorableBank missing from event record, nErrors 2 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Dec 26 14:54:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191588">191588</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 20 Run 191588 Bank COTD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 00:43:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191592">191592</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.147644 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 08:36:28 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191593">191593</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 09:44:53 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191593">191593</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.060715 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 09:46:42 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191593">191593</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.060715 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 13:35:22 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191596">191596</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.063644 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 15:23:55 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191596">191596</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B0/W2/L4/C4-7 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 20:57:52 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191626">191626</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0wcal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Dec 27 20:59:57 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191626">191626</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0wcal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 28 09:02:54 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191636">191636</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 28 09:28:00 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191636">191636</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Dec 28 09:32:45 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191636">191636</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate MUON_TRIGGER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Dec 29 01:13:14 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191640">191640</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 01:20:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal11 
 - Please SHEPHERD crate b0pcal11 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 01:20:24 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal11 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 01:47:41 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W5/L2/C0-4 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 02:27:05 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb20. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 02:39:31 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0fcal00 
 - Please SHEPHERD crate b0fcal00 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 02:45:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 02:45:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 02:45:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 02:45:44 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 03:37:33 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191727">191727</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 06:36:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191755">191755</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 47 Run 191755 Bank COTD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 06:36:50 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191755">191755</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 47 Run 191755 Bank COTD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 06:36:50 2004 -->
Message-Id: <200412311236.iBVCaYk27069@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191755">191755</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 26 Event 80 Run 191755 Bank COTD_StorableBank missing from event record, nErrors 26 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 06:36:50 2004 -->
Message-Id: <200412311236.iBVCaYk27069@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191755">191755</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 26 Event 80 Run 191755 Bank COTD_StorableBank missing from event record, nErrors 26 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 23:54:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191766">191766</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Dec 31 23:54:06 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191766">191766</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 00:05:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191766">191766</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 00:06:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191766">191766</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 00:27:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191766">191766</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0cot13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 01:32:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191766">191766</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 57.59784 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 05:06:48 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191768">191768</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 57.92261 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 05:29:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191768">191768</A>
HRR: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 06:33:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 06:50:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 43.630997 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 07:31:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
HALTED: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 08:38:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 11:03:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 11:06:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.72324 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 11:09:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 11:26:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191769">191769</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 12:47:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191775">191775</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal13 
 - Please SHEPHERD crate b0ccal13 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 13:48:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191776">191776</A>
Config: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal13 
 - Please SHEPHERD crate b0ccal13 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 1 15:10:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191777">191777</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 03:09:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191779">191779</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 28.748125 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 03:21:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191779">191779</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 03:42:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191782">191782</A>
ACTIVE: <PRE>   Attention !!!.  L3CalibTag not found  Error !!!. 
 We RECOMMEND the following steps: 
 - Please page L3 Filter software expert immediately. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 03:56:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191782">191782</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 03:56:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191782">191782</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 06:06:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191783">191783</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 11:16:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191783">191783</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 53.665073 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 11:16:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191783">191783</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 53.665073 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 11:55:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191783">191783</A>
RECOVERED: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W5/L2/C0-4 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 13:10:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191783">191783</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 95.94866 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 14:47:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191783">191783</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate CCAL_08). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 2 15:42:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=191786">191786</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 6 17:05:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192063">192063</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 6 17:10:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192063">192063</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 6 17:30:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192063">192063</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 20.274067 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 6 18:08:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192065">192065</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 91.18034 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 00:17:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 00:19:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
RECOVERED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 00:19:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 00:59:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 01:38:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 02:19:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 02:59:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 03:27:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 04:22:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 04:52:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192069">192069</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate MUON_TRIGGER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 17:07:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192125">192125</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 7 17:33:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192130">192130</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 8 00:30:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192159">192159</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 8 03:09:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192161">192161</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal11    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 00:26:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 00:31:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 00:32:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 00:38:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 01:50:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 02:09:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 02:15:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 03:01:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 03:20:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 9 05:20:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192171">192171</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 10 16:22:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192214">192214</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 10 16:52:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192214">192214</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 10 17:17:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192214">192214</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 10 18:40:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192214">192214</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate MUON_TRIGGER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 01:19:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192216">192216</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb20, slot = 8. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb20 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 01:39:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192216">192216</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 03:08:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192216">192216</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B0/W2/L4/C8-11 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 04:10:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192216">192216</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 16:45:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192230">192230</A>
Recover: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 16:49:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192230">192230</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 16:51:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192230">192230</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 17:04:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192230">192230</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 17:28:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192232">192232</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 17:28:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192232">192232</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 18:01:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192232">192232</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 11 20:22:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192232">192232</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 12 15:54:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192244">192244</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 01:07:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192275">192275</A>
End: <PRE> 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely
 to have died.
   Do not panic! Level3 will try to repair itself. We recommend:
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS   
     - exclude all affected subfarms from your run configuration
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!)
     - start another run and page L3 farm pager at closest day/evening shift
     - if >2 subfarms affected page L3 farm IMMEDIATELY
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 02:07:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192278">192278</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 05:20:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192280">192280</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 2 Event 162 Run 192280 Bank TCMD_StorableBank missing from event record, nErrors 2 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 05:20:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192280">192280</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 161 Run 192280 Bank TCMD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:34:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:38:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:44:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:47:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:48:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HRR: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal11    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:52:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 08:56:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 24.328861 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:04:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:19:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:25:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:38:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:38:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:45:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 09:57:22 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:07:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:18:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:28:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:32:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:35:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:42:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:51:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0fcal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 10:59:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:05:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:07:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:08:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:13:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:14:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:23:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb22. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:30:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:33:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:39:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:42:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 11:58:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:03:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:07:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:07:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:08:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:11:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:45:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB BadChannelCounts error in 
 SCPU = b0eb15, slot = 14. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:45:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:49:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:56:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 12:56:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:02:22 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:07:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HRRinProgress: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:14:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:18:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:21:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:29:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:32:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:36:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:37:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:43:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:45:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:47:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 13:51:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 14:13:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 14:33:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 14:35:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 14:43:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 14:59:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 15:00:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 15:19:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 15:57:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 13 16:11:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192282">192282</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 03:56:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192343">192343</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 04:09:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192343">192343</A>
ACTIVE: <PRE>   
 
    L3 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 04:47:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192344">192344</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 04:59:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192344">192344</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 05:44:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192344">192344</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 05:46:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192344">192344</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 05:57:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192344">192344</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 06:42:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 06:42:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 07:32:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
Recover: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 07:32:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 07:32:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 07:34:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 15 07:34:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192345">192345</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 01:07:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192347">192347</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 1.0638298 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:03:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
HALTED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:16:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 44.50618 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:16:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 44.50618 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:35:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb17. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:49:17 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:50:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B1/W1/L2/C12-15 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:50:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B1/W1/L2/C12-15 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 02:51:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 05:58:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
HALTED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 06:15:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 06:36:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B1/W1/L2/C12-15 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 06:38:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B1/W1/L2/C12-15 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 06:42:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid I/B1/W1/L2/C12-15 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 07:13:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 07:17:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 16 07:48:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192348">192348</A>
HALTED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 16:45:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192365">192365</A>
Recover: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx06: Sl 6 L2A w/o L1A 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 16:57:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192365">192365</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 23.215618 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 17:28:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192366">192366</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 17:35:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192366">192366</A>
IDLE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 17:36:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192366">192366</A>
IDLE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02    busy. 1 crate/s: b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 17:41:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192367">192367</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:16:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192368">192368</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:16:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192368">192368</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:20:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192368">192368</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb19. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:29:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192368">192368</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:44:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192369">192369</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:47:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192369">192369</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 18:54:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192369">192369</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 19:47:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192371">192371</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 19:50:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192371">192371</A>
RECOVERED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal01 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 19:57:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192372">192372</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 22:32:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192374">192374</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 22:46:48 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192374">192374</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal11 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 17 23:52:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192374">192374</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate WCAL_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 00:56:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:08:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:12:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:31:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:37:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:42:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:54:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 01:54:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 02:02:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 02:57:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 03:13:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 03:32:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 03:33:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 04:15:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 04:20:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0cot03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 04:44:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 04:58:48 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 05:05:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 05:29:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 06:07:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 06:19:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HRRinProgress: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 06:21:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0svx06    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 06:38:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 06:48:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 06:53:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 07:01:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 07:28:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192376">192376</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 18 23:26:22 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192424">192424</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 61.005733 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 19 16:42:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192439">192439</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 19 17:42:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192441">192441</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 04:47:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192500">192500</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   b0l1cl01   b0l1cl03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 04:48:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192500">192500</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0l1cl00   b0l1cl01   b0l1cl03    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 04:49:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192500">192500</A>
End: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 04:49:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192500">192500</A>
End: <PRE>   
  RXPT Error for crate/s: 
    b0l1cl00   b0l1cl01   b0l1cl03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 05:20:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192503">192503</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cmx01    Done TO. 1 crate/s: b0cot07   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 05:33:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 05:35:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 32.616634 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 05:37:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 31.308416 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 06:33:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate FIB_ISL_07). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 06:33:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 06:40:48 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 06:51:37 2005 -->
Message-Id: <200501201250.j0KCogc02299@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13 (forwarded by FER crate CCAL_05). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 07:02:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0svt07   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 07:35:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_18). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 07:35:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot19   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 07:35:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192504">192504</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot19   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 08:12:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb17, slot = 16. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb17 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 08:12:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb17, slot = 16. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb17 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 08:12:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb17. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 08:17:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 08:21:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
RECOVERED: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 08:37:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 09:36:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 09:50:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 11:13:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
HALTED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 11:16:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 20 11:42:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192505">192505</A>
Recover: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 21 02:34:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192565">192565</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot00   b0cot01   b0cot02   b0cot05   b0cot06   b0cot07   b0cot08   b0cot09   b0cot10   b0cot12   b0cot13   b0cot14   b0cot15   b0cot16   b0cot17   b0cot18   b0cot19   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 21 03:45:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192568">192568</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 21 07:50:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192575">192575</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 23 07:30:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192753">192753</A>
PAUSED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 24 00:55:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192791">192791</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 1.1494253 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 24 01:31:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192792">192792</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 24 03:07:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192794">192794</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 03:02:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192863">192863</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 03:47:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192863">192863</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 03:52:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192863">192863</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 04:00:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192863">192863</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 04:02:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192863">192863</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:03:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192864">192864</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:05:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192864">192864</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:10:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192864">192864</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:54:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:54:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:56:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal13    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:56:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:57:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal13 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:57:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 05:58:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 06:03:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 06:03:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 06:06:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal13   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 06:06:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192866">192866</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal13    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:26:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:31:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb22. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:35:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:35:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0svx06    busy. 1 crate/s: b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:35:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:37:31 2005 -->
Message-Id: <200501251337.j0PDbQ527755@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:37:31 2005 -->
Message-Id: <200501251337.j0PDbQ527755@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 07:39:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192868">192868</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 17:12:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192878">192878</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 17:16:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192878">192878</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 17:53:14 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192888">192888</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 18:33:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 18:39:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HALTED: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W3/L3/C0-5 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 18:42:14 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HALTED: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W3/L3/C0-5 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 18:45:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HRR: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W3/L3/C0-5 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 20:05:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 20:15:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0fcal01 
 - Please SHEPHERD crate b0fcal01 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 21:20:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 21:28:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HRRinProgress: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 21:42:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 22:45:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HRR: <PRE>   
  Busy TO. for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Jan 25 23:07:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192892">192892</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 11:50:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192915">192915</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 20:30:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192953">192953</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 21:41:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 21:43:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 21:51:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 21:53:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0l2pu00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 21:57:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 21:58:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 22:06:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 22:16:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0l2pu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 22:18:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
Halt: <PRE>   
  RXPT Error for crate/s: 
    b0ccal00   b0ccal01   b0ccal02   b0ccal03   b0ccal04   b0ccal05   b0ccal06   b0ccal07   b0ccal08   b0ccal09   b0ccal10   b0ccal11   b0ccal12   b0ccal13   b0ccal14   b0ccal15   b0wcal01   b0wcal02   b0wcal03   b0wcal04   b0wcal05   b0wcal06   b0wcal07   b0fcal00   b0fcal01   b0wcal00   b0pcal01   b0pcal02   b0pcal03   b0pcal04   b0pcal05   b0pcal00   b0pcal06   b0pcal07   b0pcal08   b0pcal09   b0pcal10   b0pcal11   b0cot00   b0cot01   b0cot02   b0cot03   b0cot04   b0cot05   b0cot06   b0cot07   b0cot08   b0cot09   b0cot10   b0cot11   b0cot12   b0cot13   b0cot14   b0cot15   b0cot16   b0cot17   b0cot18   b0cot19   b0imu00   b0cmx01   b0l1cl00   b0l1cl01   b0l1cl02   b0l1cl03   b0l1gl00   b0l2cl04    busy. 1 crate/s: b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 22:18:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192961">192961</A>
Halt: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0ccal00   b0ccal01   b0ccal02   b0ccal03   b0ccal04   b0ccal05   b0ccal06   b0ccal07   b0ccal08   b0ccal09   b0ccal10   b0ccal11   b0ccal12   b0ccal13   b0ccal14   b0ccal15   b0wcal01   b0wcal02   b0wcal03   b0wcal04   b0wcal05   b0wcal06   b0wcal07   b0fcal00   b0fcal01   b0wcal00   b0pcal01   b0pcal02   b0pcal03   b0pcal04   b0pcal05   b0pcal00   b0pcal06   b0pcal07   b0pcal08   b0pcal09   b0pcal10   b0pcal11   b0cot00   b0cot01   b0cot02   b0cot03   b0cot04   b0cot05   b0cot06   b0cot07   b0cot08   b0cot09   b0cot10   b0cot11   b0cot12   b0cot13   b0cot14   b0cot15   b0cot16   b0cot17   b0cot18   b0cot19   b0imu00   b0cmx01   b0l1cl00   b0l1cl01   b0l1cl02   b0l1cl03   b0l1gl00   b0l2cl04    busy. 1 crate/s: b0mutr00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Jan 26 23:10:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192967">192967</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal05 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:11:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192971">192971</A>
START: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@167a465
  Status:           0x1030
  Fatal Errors:
    OUTPUT_DISABLED
    SEQ_CSR1_ERR_0_SEQ_HALTED
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1106805413
  TimeNanoSeconds:  499999980
  SeqCSR0:          0x21
  SeqCSR1:          0xc1
  PccDelay:         0xffb0
  PccCSR:           0xff03 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:21:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192972">192972</A>
RECOVERED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:21:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192972">192972</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0mutr00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:22:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192972">192972</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0mutr00    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:40:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192973">192973</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 99.86317 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:44:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192973">192973</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 99.86317 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 02:50:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192973">192973</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 99.51872 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 03:52:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192973">192973</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 03:52:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192973">192973</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0ccal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 03:55:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192973">192973</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal04 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 04:41:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192977">192977</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 04:45:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192977">192977</A>
RECOVERED: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 04:46:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192977">192977</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal04    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 04:46:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192977">192977</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 06:39:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192980">192980</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CCAL_09). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 06:39:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192980">192980</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb16
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb16 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 07:17:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192982">192982</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 07:40:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192984">192984</A>
RECOVERED: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Pipeline out of synch in 152 silicon readout chips . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 07:53:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192984">192984</A>
RECOVERED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb17. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 11:50:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192988">192988</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 16:28:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192988">192988</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 18:00:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192990">192990</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 19:04:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192990">192990</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 19:09:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192990">192990</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0pcal07   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:16:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:18:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:18:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:19:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
End: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cot05   b0cot12   b0tsi03    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:19:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
End: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 36.74286 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:19:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
End: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:19:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
End: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:20:14 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
End: <PRE>   
  Busy TO. for crate/s: 
    b0cot05   b0cot12   b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:20:14 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192992">192992</A>
IDLE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 22:48:17 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192993">192993</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W5/L4/C0-6 . 
 HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Jan 27 23:07:48 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=192993">192993</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 28 12:03:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193006">193006</A>
RECOVERED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 28 16:19:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193008">193008</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 0 Run 193008 Bank TCMD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 28 20:22:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193013">193013</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 28 20:24:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193013">193013</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 28 20:24:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193013">193013</A>
HALTED: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Jan 28 20:26:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193013">193013</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 10:39:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193013">193013</A>
Recover: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 12:08:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193013">193013</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 100.13822 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 20:48:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 20:53:45 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 21:11:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 22:01:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal02 
 - Please SHEPHERD crate b0pcal02 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 22:07:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 22:07:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal02 
 - Please SHEPHERD crate b0pcal02 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 22:28:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 22:32:17 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 22:36:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193029">193029</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Jan 29 23:13:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193030">193030</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb18. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 10:02:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193031">193031</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate XFT_LINKER_05). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 16:33:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 16:41:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb17. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 16:41:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 18:51:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 19:21:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 19:28:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 19:28:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 2.3099532 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 19:53:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 22:45:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal07 
 - Please SHEPHERD crate b0ccal07 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 22:45:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal07 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 23:02:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal07 
 - Please SHEPHERD crate b0ccal07 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Jan 30 23:58:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 84.84042 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 01:39:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 03:27:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193032">193032</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 16:32:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
Recover: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13 (forwarded by FER crate CCAL_05). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 17:22:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 17:40:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 17:41:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
HALTED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 17:41:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
HALTED: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 18:30:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 21:29:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 22:28:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 22:32:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal05 
 - Please SHEPHERD crate b0ccal05 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Jan 31 22:55:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193051">193051</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 1 11:01:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193067">193067</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 1 23:45:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193104">193104</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb20. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 01:56:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb24, slot = 10. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb24 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 01:56:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 04:18:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb18. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:15:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:15:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:15:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:15:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 0 Run 193105 Bank CMPD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:25:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:25:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:25:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:30:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:30:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:56:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb23, slot = 9. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb23 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 05:56:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 06:11:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal08 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 06:52:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal05 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 06:52:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal05   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 06:52:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193105">193105</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb19. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 07:40:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193106">193106</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 11:55:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193110">193110</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 17:37:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193112">193112</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 18:15:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193112">193112</A>
HRRinProgress: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 19:22:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193112">193112</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 20:47:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193112">193112</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 22:54:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193118">193118</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 2 23:12:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193118">193118</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 3 10:41:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193148">193148</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 3 13:35:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193163">193163</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 1.6393442 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 3 14:26:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193164">193164</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 25.242016 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 16:08:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 16:27:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 17:56:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 18:04:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 5 Event 0 Run 193247 Bank HATD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 18:25:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
RECOVERED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CMU_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 18:33:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
HRR: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 19:09:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 19:09:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 20:02:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193247">193247</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 4 23:26:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193271">193271</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal11 
 - Please SHEPHERD crate b0pcal11 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 5 23:28:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193369">193369</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 5 23:28:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193369">193369</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Feb 6 00:02:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193371">193371</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 2 Event 0 Run 193371 Bank COTD_StorableBank missing from event record, nErrors 2 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Feb 6 00:02:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193371">193371</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 0 Run 193371 Bank COTD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Feb 6 00:02:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193371">193371</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 14 Event 0 Run 193371 Bank COTD_StorableBank missing from event record, nErrors 14 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 00:28:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193396">193396</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb14
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb14 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 00:28:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193396">193396</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 00:38:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193396">193396</A>
HALTED: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 5.6732125 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 00:38:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193396">193396</A>
Recover: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 01:28:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 02:03:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 02:13:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 02:13:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 03:28:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 03:53:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 04:36:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 05:20:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 05:54:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb22. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 06:04:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 06:05:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 06:17:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 06:19:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 07:05:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 07:06:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 07:28:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 08:13:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 08:13:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
START: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 66.666664 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 7 08:13:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193397">193397</A>
START: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 13:00:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193462">193462</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_15). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:08:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:08:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:14:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:24:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
Recover: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate PCAL_02). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:25:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:28:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb14
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb14 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:28:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:28:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:28:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:51:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
RECOVERED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0svx06    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:51:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:51:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:53:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb15, slot = 12. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 16:57:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
HALTED: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 17:00:06 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot02   b0cot03   b0cot05   b0cot06   b0cot07   b0cot08   b0cot11   b0cot12   b0cot14   b0cot15   b0cot16   b0cot17   b0cot18   b0cot19   b0imu01   b0cmp00   b0musc00   b0cmx01   b0htdc00   b0etdc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 17:01:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193464">193464</A>
End: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 18:17:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193466">193466</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 18:19:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193466">193466</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 18:39:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193467">193467</A>
HRRinProgress: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb20. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 18:56:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193468">193468</A>
HRRinProgress: <PRE>   
  Done TO. for crate/s: 
    b0pcal04   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 19:03:22 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193468">193468</A>
HRRinProgress: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb16
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb16 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 19:03:22 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193468">193468</A>
Recover: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_16). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 19:13:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193468">193468</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 19:35:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:20:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 0 Run 193469 Bank CLTD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:20:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:26:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 5 Event 0 Run 193469 Bank CLTD_StorableBank missing from event record, nErrors 2 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:26:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:26:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 6 Event 0 Run 193469 Bank CLAD_StorableBank missing from event record, nErrors 2 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:38:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:38:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 18 Event 0 Run 193469 Bank MPAD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:38:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 17 Event 0 Run 193469 Bank EMTD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:41:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:51:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:55:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 20:57:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193469">193469</A>
Recover: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 21:45:22 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193470">193470</A>
RECOVERED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 22:04:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193470">193470</A>
HALTED: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 36 Event 0 Run 193470 Bank MPAD_StorableBank missing from event record, nErrors 1 
 
 
  HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 22:22:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193470">193470</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 8 23:39:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193470">193470</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 00:32:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 00:48:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 00:59:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 01:13:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 01:16:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 01:57:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:09:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:14:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:24:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:32:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:32:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal01 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:49:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:57:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 02:59:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:03:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:03:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:26:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:32:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:35:04 2005 -->
Message-Id: <200502090934.j199YbC11405@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot05 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:39:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 03:53:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193472">193472</A>
Partition: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 04:20:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193473">193473</A>
START: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 04:35:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193475">193475</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 13:54:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193487">193487</A>
START: <PRE>   Attention!!!   Clock Error !!!  
  rc.vxmon.ClockStatus@cf8ed6
  Status:           0x1760
  Fatal Errors:
    SEQ_CSR1_ERR_0_SEQ_HALTED
    SEQ_CSR1_ERR_1_SYNC_MISSING
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1107978839
  TimeNanoSeconds:  633333308
  SeqCSR0:          0xe3f
  SeqCSR1:          0xfb
  PccDelay:         0xff05
  PccCSR:           0xff03 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 9 20:42:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193526">193526</A>
HALTED: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Pipeline out of synch in 150 silicon readout chips . 
 HRR is STRONGLY RECOMMENDED. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 00:26:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193889">193889</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 00:26:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193889">193889</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 08:34:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 08:34:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 08:59:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate WCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 08:59:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:22:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
HRRinProgress: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:24:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0fcal00 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:42:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:50:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:53:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:54:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 09:56:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 10:23:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 10:29:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 10:36:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 10:51:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B1/W5/L2/C0-4 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 10:52:30 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 10:57:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_02). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:08:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:11:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:13:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 160 Event 0 Run 193892 Bank EMTD_StorableBank missing from event record, nErrors 22 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:17:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:18:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:24:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:28:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:32:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:49:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 11:54:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193892">193892</A>
Recover: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:17:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:19:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
READY: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0svt01 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:26:59 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:28:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb14
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb14 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:28:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:37:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 12:54:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 (forwarded by FER crate COT_16). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 13:12:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 13:31:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 13:43:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 13:50:17 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 13:52:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 13:57:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 14:13:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 14:25:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 14:34:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 15:03:40 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 15:34:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 15:36:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 240 Event 0 Run 193901 Bank SL1D_StorableBank missing from event record, nErrors 15 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 15:40:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 15:43:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 15:46:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193901">193901</A>
IDLE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 14 16:00:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193908">193908</A>
Config: <PRE> 
 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 15 09:16:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193914">193914</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 15 09:17:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193914">193914</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 15 09:24:48 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193914">193914</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B4/W6/L3/C0-5 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 15 09:50:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193914">193914</A>
RECOVERED: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 15 11:34:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193919">193919</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 15 15:00:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193942">193942</A>
Partition: <PRE> 
 Attention !!!. Event Builder EVB_PROXY_SM_TIMEOUT Error !!!. 
 Event Builder proxy lost connection to Scanner Manager. 
 We RECOMMEND the following steps:
 - End of run, Reset
 - Clean up EVB 			    
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:07:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193986">193986</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_15). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:17:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:34:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:36:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:42:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_02). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:50:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:50:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 76 Event 0 Run 193987 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 08:54:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:12:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:16:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:19:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:33:23 2005 -->
Message-Id: <200502161533.j1GFXCL01431@b0dau35.fnal.gov>
Content-Type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:33:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
START: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:33:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
START: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:33:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
START: <PRE> 
 Attention!!!. SCPU_BUFFER_OVERFLOW Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:33:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193987">193987</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:40:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193988">193988</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:43:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193988">193988</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:45:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193988">193988</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B4/W6/L3/C0-5 . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:58:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193988">193988</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 09:59:04 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193988">193988</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx06: Sl 6 Too Many L1A 2 L1A to Buff 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 10:12:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193988">193988</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 10:45:11 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 10:50:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 10:52:12 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 10:57:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 11:32:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 11:32:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
START: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 11:34:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
START: <PRE>   
  Busy TO. for crate/s: 
    b0tsi03   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 11:34:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193989">193989</A>
START: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 12:05:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 12:05:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 12:05:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal09   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 12:18:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 12:30:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_15). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 13:42:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 16 14:43:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193990">193990</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb15 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb16 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb13 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb12 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb11 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb14 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:39:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194012">194012</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb24 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 09:50:17 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194013">194013</A>
START: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0tsi03    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 12:59:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194022">194022</A>
HALTED: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 96.30479 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 13:01:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194022">194022</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 95.53849 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 13:02:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194022">194022</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 93.62949 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 13:37:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 13:47:56 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 13:51:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb11, slot = 14. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:06:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:06:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:07:42 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 37 Event 0 Run 194023 Bank CLTD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:10:36 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:26:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 65 Event 0 Run 194023 Bank EMTD_StorableBank missing from event record, nErrors 13 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:26:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:34:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate TEST_SVX_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:38:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
RECOVERED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:40:21 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:43:49 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:47:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
HRR: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:50:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BUFFER_OVERFLOW Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb15. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:51:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0cot08   b0cot18    
    
  >>        and take the appropriate actions.        <<
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0cot08   b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0cot08   b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB BufferOverflow error in 
 SCPU = b0eb15, slot = 14 would not fit in remaining buffer space. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 received an event that is too large.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB BufferOverflow error in 
 SCPU = b0eb15, slot = 14 would not fit in remaining buffer space. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 received an event that is too large.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 14:52:27 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB BufferOverflow error in 
 SCPU = b0eb15, slot = 14 would not fit in remaining buffer space. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb15 received an event that is too large.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 15:10:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0cot08   b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 15:10:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0cot08   b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 15:11:06 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194023">194023</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 15:45:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194025">194025</A>
HALTED: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0svx06    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 17 15:45:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194025">194025</A>
HALTED: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb11, slot = 14. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 08:09:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194263">194263</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0imu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 08:23:26 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194263">194263</A>
Recover: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0imu00    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 09:58:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194264">194264</A>
HALTED: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal04 
 - Please SHEPHERD crate b0pcal04 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 09:59:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194264">194264</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal04 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 10:37:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194265">194265</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 10:37:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194265">194265</A>
Recover: <PRE>   
  Done TO. for crate/s: 
    b0l2de00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 11:09:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194265">194265</A>
HRR: <PRE>   
  Done TO. for crate/s: 
    b0pcal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 14:54:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194266">194266</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 15:00:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194266">194266</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb20. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 15:00:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194266">194266</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb20, slot = 16. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb20 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 21 19:15:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194282">194282</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 06:38:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194315">194315</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 08:21:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194316">194316</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 59.125957 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 08:25:51 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194316">194316</A>
START: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 08:25:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194316">194316</A>
START: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 08:25:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194316">194316</A>
START: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 08:25:54 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194316">194316</A>
START: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:01:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 22.742334 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:01:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot18   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:01:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 54.5934 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:17:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot19   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:19:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:22:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 27.491371 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:52:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
HRR: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB VrbBadByteCount error in 
 SCPU = b0eb25, slot = 12. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb25 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 09:52:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
HALTED: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:05:18 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0ccal01   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:17:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0pcal10 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:17:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0pcal10    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:17:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:17:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:17:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0pcal10   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 10:19:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 11:42:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 11:53:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 11:57:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 12:19:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194318">194318</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 12:59:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194319">194319</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 13:02:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194319">194319</A>
RECOVERED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 13:05:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194319">194319</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 13:05:09 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194319">194319</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 13:15:44 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 23.068962 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 14:23:01 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 14:25:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:15:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0clc00   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:16:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:16:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 1.306895 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:16:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:16:32 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:17:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:17:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:28:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 100.0 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:29:17 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:29:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:29:23 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!. 
 We STRONGLY RECOMMEND the following steps: 
 - Try Halt-Recover-Run.
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB.
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:29:52 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Feb 22 15:48:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194320">194320</A>
IDLE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0fcal01 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 23 09:27:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194328">194328</A>
RECOVERED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx06: Sl 6 L2A w/o L1A 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Wed Feb 23 09:29:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194328">194328</A>
HALTED: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Gap Error 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 08:06:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194383">194383</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 08:11:24 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194383">194383</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb13 (forwarded by FER crate CCAL_01). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 09:39:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 09:39:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 09:39:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 10:05:47 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot15   b0cot17   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 10:35:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 11:33:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx06: Sl 6 2 L1A to Buff 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 11:33:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 11:33:29 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 11:37:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>  
  Attention!!!.   RXPT Errors LOOP!!!. 
  
  >> Please Check the crate/s in the following list: << 
  
   b0svx06    
    
  >>        and take the appropriate actions.        <<

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  >>  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                >>
  >>  ACES ......  YOU HAVE CONTROL of RC NOW.                    >>
  >>  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              >>
  >>  OF SMOOTH RUNNING                                                     >>
  >>  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            >>
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 12:39:31 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot14   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 13:40:25 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 14:01:08 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 14:28:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 14:34:13 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot11   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 15:09:28 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 15:13:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
RECOVERED: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Feb 24 16:03:19 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194386">194386</A>
HALTED: <PRE>   
  Busy TO. for crate/s: 
    b0svx06   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 25 02:51:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194455">194455</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot12   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Feb 25 07:20:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194460">194460</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot13   b0cot15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 03:30:55 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194498">194498</A>
ACTIVE: <PRE> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Pipeline out of synch in 152 silicon readout chips . 
 AUTO HRR will be issued  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:31:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0imu00   b0imu01   b0cmu01   b0musc00   b0cmx01    Done TO. 1 crate/s: b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:31:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0imu00   b0imu01   b0cmu01   b0musc00   b0cmx01    Done TO. 1 crate/s: b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:33:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
   L1 Done Timeout.[RXPT] 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:33:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected 
 
 - HALT the run     
 - record any relevant information in e-log     
 - RECOVER - RUN     
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:33:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
   L1 Done Timeout.[RXPT] 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:33:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:33:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>   
  RXPT Error for crate/s: 
    b0imu00   b0imu01   b0cmu01   b0musc00   b0cmx01    Done TO. 1 crate/s: b0svx02   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sat Feb 26 04:33:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194502">194502</A>
ACTIVE: <PRE>  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 28 00:25:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194562">194562</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb25. 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Mon Feb 28 00:25:05 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194562">194562</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB BadChannelCounts error in 
 SCPU = b0eb25, slot = 11. 
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb25 reports problems with a VRB header.  
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 00:20:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 00:20:20 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
ACTIVE: <PRE> 
 Attention!!!. L3_REF_ERROR_HIGH_RATE Error  
 
 L3 Instantaneous error rate is 6.229037 per cent. 
 
 Level3 Reformatter detects abnormal amount of corrupted data!
 We recommend:
    - use reformatter decoder and find the offending crate
    - try HRR, if no effect try End->ColdStart->Activate
    - call appropriate FE expert (SVX pager for FIB crates)
    - if FE expert is not available, try L3 pager
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 00:30:50 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot17   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 04:33:33 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal10 
 - Please SHEPHERD crate b0pcal10 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 05:43:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot13   b0cot14   b0cot15   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 05:43:53 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
ACTIVE: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal06 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 1 05:44:02 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194590">194590</A>
ACTIVE: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal06 
 - Please SHEPHERD crate b0ccal06 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:08:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194732">194732</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:19:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194742">194742</A>
HALTED: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb12 (forwarded by FER crate CCAL_08). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:19:00 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194742">194742</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot03   b0cot08   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:19:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194742">194742</A>
HALTED: <PRE>   
  Done TO. for crate/s: 
    b0cot03   b0cot08   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:19:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194742">194742</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot03 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:19:57 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194742">194742</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot08 
 an additional AUTO HRR will be issued.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 01:45:06 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194742">194742</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 04:55:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
Config: <PRE> 
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0ccal02 
 - Please SHEPHERD crate b0ccal02 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 04:57:38 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
READY: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0ccal02 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:10:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
ACTIVE: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:10:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
ACTIVE: <PRE>    Attention!!!.   L2 Decision Timeout LOOP!!!. 
  
  Additional Information: 
  If L2 Timeouts persist, please page L2. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:10:03 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
HALTED: <PRE>  Attention!!! L2 Decision Timeout ERROR !!. 
 
 If L2 Timeouts persist, please page L2 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:12:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
ACTIVE: <PRE>   
 
    L2 trigger rate has dropped to 0 Hz!!! 
     Events are not being logged !!! </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:12:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:12:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:25:58 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 05:37:39 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194744">194744</A>
START: <PRE> 
 Attention!!!. SM_SCPU_DROPOUT Error !!! 
 Hardware EVB problem: SCPU b0eb11 was rebooted or hung. 
 - Check that L3 is in run configuration  
 - End run and Reset  
 - Clean up EVB 
 - Check SM/SCPU from EVB/L3 Control Pane 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 07:01:46 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194746">194746</A>
Activate: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Thu Mar 3 17:56:41 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194810">194810</A>
ACTIVE: <PRE>   
  Done TO. for crate/s: 
    b0cot16   
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Fri Mar 4 01:44:16 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194811">194811</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Mar 6 00:56:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194859">194859</A>
START: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Mar 6 00:56:07 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194859">194859</A>
START: <PRE> 
 Attention!!!. SM_SCPU_UNRESPONSIVE Error !!!. 
 One or more SCPUs failed to respond to Delete OR Define Partition.
 We STRONGLY RECOMMEND the following steps: 
 - End the run, Reset 
 - Clean up EVB 			    
 - Start run again. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Mar 6 02:43:43 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194861">194861</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate SVT_TRACKING_05). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Mar 6 02:44:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194861">194861</A>
ACTIVE: <PRE> 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 1 Event 220204 Run 194861 Bank CMPD_StorableBank missing from event record, nErrors 1 

 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Sun Mar 6 02:44:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194861">194861</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb11
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb11 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 8 03:52:37 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194911">194911</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb23. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 8 07:38:15 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194911">194911</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb19. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs.  </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 8 12:55:10 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194939">194939</A>
ACTIVE: <PRE>  Additional Information: 
 
 This BUSY TIMEOUT was caused by EVB TracerEventId error in 
 SCPU = b0eb14
 
 Hardware EVB has detected a problem with data quality. 
 SCPU b0eb14 reports event ID or control sequence mismatch. 
 Datataking is paused; events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 8 12:55:34 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194939">194939</A>
ACTIVE: <PRE> 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb14 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
 </PRE>
<!-- Author: ErrorHandler -->


<!-- Date: Tue Mar 8 12:56:35 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=194939">194939</A>
HALTED: <PRE> 
 Attention!!!. FERML_SHEPHERD_SUCCESS !!! 
 
 Successfully Shepherded b0cot12 
 Autorecoveries are OFF: please DO an additional MANUAL HRR.  
 </PRE>
<!-- Author: ErrorHandler -->