<!-- Title: Owl shift. Mon Jul  3, 2006 -->
<!-- SciCo: Craig Blocker -->
<!-- DAQAce: Niharika Ranjan -->
<!-- MonAce: Sarah Malik -->
<!-- CO: Bob Miller -->
<!-- OpManager: Mary Convery -->
<!-- Notes: -->
<pre>We wait for Tevatron to recover from quench so 
they can try the shot again.</pre>


<!-- Date: Mon Jul 3 00:20:21.903 2006 -->
<pre>The event display is running again. For some reason, it only works if you run it locally with ; 
"start_local_evd.sh"     script. 
 
</pre>
<!-- Author: sarah :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219388' target=_top>219388</a>) -->


<!-- Date: Mon Jul 3 00:23:13.651 2006 -->
Could you please perform the Silicon post-quench checkout?
<!-- Author: Ulrich (Silicon Main Pager) -->

<!-- Date: Mon Jul 3 00:26:26.802 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219388>219388</A>
 Terminated at 2006.07.03 00:26:09
<!-- Author: RunControl -->


<!-- Date: Mon Jul 3 00:28:05.511 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219388>219388</A>
TERMINATE: Performing post quench COLDSTART
<!-- Author: Niharika -->



<!-- Date: Mon Jul 3 00:29:53.476 2006 -->
<pre>COLDSTART config successful</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219389' target=_top>219389</a>) -->
<!-- Comment: Mon Jul  3 00;31;10 comment by...Ulrich -->
Thanks a lot.
<!-- Date: Mon Jul 3 00:31:07.073 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5603&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5603  width='300' align='top'></a></td><td>Post quench plot</td></tr></table>
<!-- Author: sarah -->


<!-- Date: Mon Jul 3 00:34:36.663 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219389>219389</A>
TERMINATE: Finished post quench COLDSTART
<!-- Author: Niharika -->



<!-- Date: Mon Jul 3 00:34:36.089 2006 -->
<pre>There was one pinky , B5W5L4 which was noted in the previous shift and has been there for a while, when I clicked unmark, it turned yellow, but it is back again. It's still flat though. 
</pre>
<!-- Author: sarah :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219389' target=_top>219389</a>) -->

<!-- Date: Mon Jul 3 00:36:23.367 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390>219390</A>
 Activated at 2006.07.03 00:36:14
<!-- Author: RunControl -->


<!-- Date: Mon Jul 3 00:36:24.175 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390>219390</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,639] 
<!-- Author: Niharika -->



<!-- Date: Mon Jul 3 00:38:49.851 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5604&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5604  width='300' align='top'></a></td><td>Post quench check for pinkies</td></tr></table>
<!-- Author: sarah -->


<!-- Date: Mon Jul 3 00:48:12.925 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Issues from previous shift(s): 
 
- PTM(Plug Temperature Monitor) heartbeat alarm is still there ( it was noted in elog in previous shift. It does not require paging, though if it stays like that when we start taking data or ramping up voltage, we will page.) 
 
- Pinky at B5W5L4 which does not turn yellow. Silicon know about this and it does not require any action as long as it is flat. 
 
- If CMX wedges 18 or 19 trip, we can turn the high voltage off for those 2 wedges. If any wedges other than 18/19 trip, page CMX. 
 
 
</pre></td></tr></td></table>
<!-- Author: Sarah -->
<!-- Comment: Mon Jul  3 00;53;49 comment by...Sarah -->
<table border=0><td><tr><td bgcolor=#e0e030>Thats NE CMX wedges 18/19</td></tr></td></table>


<!-- Date: Mon Jul 3 01:12:05.806 2006 -->
<pre>We have L2 decision TO every 2 mins.. 
We are running in cosmics mode so probably don't need to worry too much..  
Here are the messages from the error handler: 
 
(MLW) b0control21.fnal.gov:Thread-41:Jul 3, 2006 01:09:58 AM-&gt;Received error message: errmon Not a state HaltRecoverRun L2 Decision Timeout.[RXPT](MLE) b0control21.fnal.gov:Thread-41:Jul 3, 2006 01:09:58 AM-&gt;Requested Halt-Recover-Run issued (errmon) 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 3, 2006 01:10:06 AM-&gt;{ 1:10:6, 7/3/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data 
(RC)   1:10:6 Halt -&gt; HALTED 
(RC)   1:10:13 Recover -&gt; RECOVERED 
(RC)   1:10:16 Run -&gt; ACTIVE 
</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 01:19:01.089 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5605&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5605' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5606&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5606' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5607&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5607' width='300'></a></center>
<!-- Author: Sarah -->


<!-- Date: Mon Jul 3 03:17:13.253 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@31824a 
  Status:           0x18300 
  Fatal Errors: 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    PCC_CSR_6_SYNC_TIMING_ERROR 
    PCC_CSR_7_SYNC_MISSING_ERROR 
  Time: :           1151914605 
  TimeNanoSeconds:  0 
  SeqCSR0:          0xed27 
  SeqCSR1:          0x1c 
  PccDelay:         0xffb0 
  PccCSR:           0xffc3</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 03;18;17 comment by...Niharika -->
<pre>accompanied by:

RXPT Error for crate/s: 
    b0svt09   b0svt02   b0svt06   b0svt05   b0tsi03   </pre>

<!-- Comment: Mon Jul  3 03;18;44 comment by...Niharika -->
<pre>Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done  for crate b0cot06   
 
 AUTO HRR will be issued.  
 
 If HRR does NOT help, 
 please SHEPHERD the offending TDC crate.  </pre>


<!-- Date: Mon Jul 3 03:19:17.908 2006 -->
<pre>Again 
 
*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@1b6a606 
  Status:           0x18300 
  Fatal Errors: 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    PCC_CSR_6_SYNC_TIMING_ERROR 
    PCC_CSR_7_SYNC_MISSING_ERROR 
  Time: :           1151914727 
  TimeNanoSeconds:  0 
  SeqCSR0:          0xed27 
  SeqCSR1:          0x1c 
  PccDelay:         0xffb0 
  PccCSR:           0xffc3</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 03:21:42.797 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@f0212d 
  Status:           0x18300 
  Fatal Errors: 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    PCC_CSR_6_SYNC_TIMING_ERROR 
    PCC_CSR_7_SYNC_MISSING_ERROR 
  Time: :           1151914849 
  TimeNanoSeconds:  0 
  SeqCSR0:          0xed27 
  SeqCSR1:          0x1c 
  PccDelay:         0xffb0 
  PccCSR:           0xffc3</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 03:24:39.980 2006 -->
<pre>SYNC MISSING and SYNC TIMING errors on the clock</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 03;25;11 comment by...Niharika -->
<pre>I have not cleared them.. waiting for Si</pre>


<!-- Date: Mon Jul 3 03:27:46.280 2006 -->
<pre>FATAL CLOCK ERROR again</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 03;29;24 comment by...Niharika -->
<pre>again</pre>


<!-- Comment: Mon Jul  3 03;34;04 comment by...Niharika -->
<pre>Fatal clock errors every ~ 2 mins</pre>
<!-- Date: Mon Jul 3 03:30:57.078 2006 -->
<pre>Silicon clock error caused a large part of the silicon detector to trip. Silicon have been paged. Ulrich is on his way. DAQ ace had a silicon clock error, which was followed by silicon alarm, silicon PS GUI froze for a minute or so(enough to really scare us). Lots of red and black cells and pinkies, looking scary. </pre>
<!-- Author: sarah :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->



<!-- Date: Mon Jul 3 03:47:45.711 2006 -->
<pre>Cleared the clock errors</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 03:55:25.519 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@1685b65 
  Status:           0x300 
  Fatal Errors: 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
  Time: :           1151916805 
  TimeNanoSeconds:  449999982 
  SeqCSR0:          0xed27 
  SeqCSR1:          0x1c 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 03;55;57 comment by...Niharika -->
<pre>Again at 3:55:52 AM</pre>


<!-- Date: Mon Jul 3 03:57:42.805 2006 -->
<pre>SEQ HOLD and ST CLK PARITY leds are on!</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 04;03;45 comment by...Niharika -->
<pre>BC PH 743 AND 1113  LEDs are lit. Which we guess should happen only if we have colliding beam ? </pre>
<!-- Comment: Mon Jul  3 04;05;12 comment by...niharika -->
<pre>BC LOC LED is on too</pre>
<!-- Date: Mon Jul 3 04:02:11.854 2006 -->
Shift crew, please contact MCR to see if the clock problems could be on their side, otherwise we probably need to contact Steve Hahn.
<!-- Author: convery -->




<!-- Date: Mon Jul 3 04:05:37.271 2006 -->
<pre>Clock ERRORS: 
------------------------------------------------------- 
Column	Data 
FRONTENDTIME	1151917304 
CREATETIME	2006.07.03 04:01:44 
STATUS	0x0300 
 
SEQ_CSR1_ERR_3_SEQ_HOLD 
SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
 
TIMENANOSECONDS	549999978 
SEQCSR0	0xed27 
 
0x0001 SET_ERRORS       true 
0x0002 CLOCK_ON         true 
0x0004 OUTPUT_ENABLE    true 
0x0008 STEP_MODE        false 
0x0010 FREE_RUN_SYNC    false 
0x0020 STEP_PULSE       true 
0x0040 CLEAR_ERRORS     false 
0x0080 RESET            false 
0x0100 GROUP_A_ON       true 
0x0200 GROUP_B_ON       false 
0x0400 SEL_GRP_AB       true 
0x0800 MCLK_DIS         true 
 
SEQCSR1	0x001c 
 
0x0001 SEQ_HALTED       false 
0x0002 SYNC_MISSING     false 
0x0004 BC_PHASE         true 
0x0008 SEQUENCER_HOLD   true 
0x0010 CLOCK_PARITY     true 
0x0020 PCLOCK_MISSING   false 
0x0040 MCLOCK_MISSING   false 
 
PCCDELAY	0xffb0 
PCCCSR	0xff03 
 
0x0001 SYNC_ENABLE      true 
0x0002 RF_ENABLE        true 
0x0008 SET_ERR          false 
0x0010 RF_LOCK_ERR      false 
0x0020 RF_MISSING_ERR   false 
0x0040 SYNC_TIMING_ERR  false 
0x0080 SYNC_MISSING_ERR false 
 
ELOG_URL	nb=2006&action=view&page=Mon_Jul__3_2006_Owl&anchor=03:57:42.805 
 
---------------------------------------------------------------</pre>
<!-- Author: sarah :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 04:06:05.689 2006 -->
<pre>Latest: 
 
*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@1e0e5d8 
  Status:           0x300 
  Fatal Errors: 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
  Time: :           1151917535 
  TimeNanoSeconds:  0 
  SeqCSR0:          0xed27 
  SeqCSR1:          0x1c 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 04:12:02.899 2006 -->
<pre>We talked to Steve. He asked us to clear the errors again. However the BC PH, SEQ HOLD and ST CLK PARITY LEDs are still lit. Steve said he will try resetting the clock from home. </pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 04:14:05.809 2006 -->
<pre>CMX wedges NE 18/19 tripped. Turned them to standby. 
</pre>
<!-- Author: sarah -->


<!-- Date: Mon Jul 3 04:14:13.468 2006 -->
<pre>Latest: 
 
*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@bcedef 
  Status:           0x300 
  Fatal Errors: 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
  Time: :           1151918022 
  TimeNanoSeconds:  0 
  SeqCSR0:          0xed27 
  SeqCSR1:          0x1c 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->


<!-- Date: Mon Jul 3 04:16:16.879 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@17581b8 
  Status:           0x1330 
  Fatal Errors: 
    OUTPUT_DISABLED 
    SEQ_CSR1_ERR_0_SEQ_HALTED 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY 
  Time: :           1151918145 
  TimeNanoSeconds:  0 
  SeqCSR0:          0x7021 
  SeqCSR1:          0xdd 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 04;18;10 comment by...niharika -->
<pre>again</pre>

<!-- Date: Mon Jul 3 04:20:18.768 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390>219390</A>
 Terminated at 2006.07.03 04:20:07
<!-- Author: RunControl -->



<!-- Date: Mon Jul 3 04:21:24.193 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5608&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5608' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5609&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5609' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5610&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5610' width='300'></a></center>
<!-- Author: Sarah -->


<!-- Date: Mon Jul 3 04:24:46.540 2006 -->
<pre>Steve reset the clock. 
BC PH 742 and 1113 are lit. 
SEQ HOLD and ST CLK PARITY LEDs are off.  
</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 04;26;45 comment by...Craig Blocker -->
Steve Hahn said he got from memory errors when he reset things,
so he is worried that the clock is not right yet.  He is
trying to contact Steve Chappa.
<!-- Date: Mon Jul 3 04:26:35.375 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395>219395</A>
 Activated at 2006.07.03 04:26:21
<!-- Author: RunControl -->


<!-- Date: Mon Jul 3 04:26:36.640 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395>219395</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,639] 
<!-- Author: Niharika -->
<!-- Comment: Mon Jul  3 04;28;02 comment by...niharika -->
<pre>seems to be running fine as of now. </pre>

<!-- Comment: Mon Jul  3 04;28;27 comment by...niharika -->
<pre>too soon...
L2 decision TO. auto HRR issued</pre>


<!-- Date: Mon Jul 3 04:32:22.510 2006 -->
<p>Used the clock GUI to completely reset the clock.  Now that I am done, it appears 
there are no errors.  However, on downloading the static memory, would get a  
variable number of errors in the clock GUI itself.  Here are the similar errors in the  
log in the X-window the clock GUI is run from:</p> 
 
<pre> 
Error !!! To write 4096 words to STATIC_MEM Memory. 8192 words are written. 
The start address is a000 
Reading SMem to check with file: 
****** void (seq.)readStaticMemory 
*** void (seq.)readStaticMemory data[0]=975 
*** void (seq.)readStaticMemory data[7]=67 
**** void (seq.)readStaticMemory data[1112]=526 
getSMemory.length =4096 
getSMemory[0]=3cf 
getSMemory[7]=43 
OK! getSMemory[i] != rshort[i], i=4095 
getSMemory[0]=3cf, getSMemory[1112]=20e 
Reading SMem to check with file: 
****** void (seq.)readNextMemory 
*** void (seq.)readStaticMemory data[0]=1 
*** void (seq.)readStaticMemory data[7]=16392 
getNAMemory.length =4096 
getNAMemory[0]=1 
getNAMemory[7]=4008 
Too many errors! Stop action!, i=770 
</pre> 
 
<p>I have advised Craig to try starting a new run, but I believe if these errors are 
correct, the clock is in an undefined state.  I'll get Steve Chappa and see if he  
understands this.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219390' target=_top>219390</a>) -->
<!-- Comment: Mon Jul  3 04;59;24 comment by...Steve Hahn -->
<p>Here are the messages seen in the clock GUI:</p>

<pre>
Action: Seq_Reset 
Load Next Address and Static Memory ...Reading file:STCDF04B.INV is O'K ...Loading Next Address & Static Memory is O'K. Total NA & ST words:4096, 4096
Read Static Memory ... varify 
 Varify Static memory error! word= 0, Memory=204, File=3cf
 Varify Static memory error! word= 1, Memory=0, File=3e8
 Varify Static memory error! word= 2, Memory=0, File=200
 Varify Static memory error! word= 3, Memory=204, File=0
 Varify Static memory error! word= 5, Memory=2000, File=0
 Varify Static memory error! word= 6, Memory=0, File=2
 Varify Static memory error! word= 7, Memory=0, File=43

 Too many errors! Stop action! 
Load Static Memory done. No Errors. First word=204
Error accessing Seq module(read Stat_Mem)!
</pre>
<!-- Date: Mon Jul 3 04:39:10.431 2006 -->
<pre>We have L2 decision TO errors approximately every 2 mins 
 
Messages from the error handler: 
 
(MLW) b0control21.fnal.gov:Thread-41:Jul 3, 2006 04:37:41 AM-&gt;Received error message: errmon Not a state HaltRecoverRun L2 Decision Timeout.[RXPT](MLE) b0control21.fnal.gov:Thread-41:Jul 3, 2006 04:37:41 AM-&gt;Requested Halt-Recover-Run issued (errmon) 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 3, 2006 04:37:50 AM-&gt;{ 4:37:50, 7/3/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data 
(RC)   4:37:50 Halt -&gt; HALTED 
(RC)   4:37:58 Recover -&gt; RECOVERED 
(RC)   4:38:00 Run -&gt; ACTIVE</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->


<!-- Date: Mon Jul 3 04:52:50.593 2006 -->
<pre>Paused the run on Steve's request</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->


<!-- Date: Mon Jul 3 04:59:11.251 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@1f44d34 
  Status:           0x1330 
  Fatal Errors: 
    OUTPUT_DISABLED 
    SEQ_CSR1_ERR_0_SEQ_HALTED 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY 
  Time: :           1151920712 
  TimeNanoSeconds:  16666666 
  SeqCSR0:          0x7021 
  SeqCSR1:          0xdd 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->
<!-- Comment: Mon Jul  3 05;01;05 comment by...niharika -->
<pre>*** Fatal Clock Error ***
rc.vxmon.ClockStatus@206eda
  Status:           0x1760
  Fatal Errors:
    SEQ_CSR1_ERR_0_SEQ_HALTED
    SEQ_CSR1_ERR_1_SYNC_MISSING
    SEQ_CSR1_ERR_3_SEQ_HOLD
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY
    SEQ_CSR1_ERR_5_PCLK_MISSING
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY
  Time: :           1151920837
  TimeNanoSeconds:  16666666
  SeqCSR0:          0xf023
  SeqCSR1:          0xfe
  PccDelay:         0xffb0
  PccCSR:           0xff03</pre>


<!-- Date: Mon Jul 3 05:06:23.140 2006 -->
<pre>Run is *paused* 
We are getting fatal clock errors every 2 mins approximately</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->


<!-- Date: Mon Jul 3 05:11:28.535 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@70160c 
  Status:           0x1330 
  Fatal Errors: 
    OUTPUT_DISABLED 
    SEQ_CSR1_ERR_0_SEQ_HALTED 
    SEQ_CSR1_ERR_3_SEQ_HOLD 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY 
  Time: :           1151921445 
  TimeNanoSeconds:  0 
  SeqCSR0:          0x7021 
  SeqCSR1:          0xdd 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->


<!-- Date: Mon Jul 3 05:15:16.154 2006 -->
<pre>*** Fatal Clock Error *** 
rc.vxmon.ClockStatus@f4cff3 
  Status:           0x1200 
  Fatal Errors: 
    SEQ_CSR1_ERR_4_STAT_CLK_PARITY 
    SEQ_CSR1_ERR_7_DYNA_CLK_PARITY 
  Time: :           1151921689 
  TimeNanoSeconds:  0 
  SeqCSR0:          0x6b27 
  SeqCSR1:          0x94 
  PccDelay:         0xffb0 
  PccCSR:           0xff03</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->


<!-- Date: Mon Jul 3 05:26:31.597 2006 -->
<pre>Pressed the RST button on the clock crate on Steve's request.  
Got the following error after a minute. 
 
** Node b0clk00 in Error *** 
Last data more than 120s ago!  Node is not responsive</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->
<!-- Comment: Mon Jul  3 05;26;58 comment by...niharika -->
<pre>again

*** Node b0clk00 in Error ***
Last data more than 120s ago!  Node is not responsive</pre>

<!-- Comment: Mon Jul  3 05;30;47 comment by...niharika -->
<pre>When I pushed the RST button, I observed the following:

BFL, CPU, PMC LEDs were lit and then were off after a few seconds. However BFL LED did flash for a longer time (few more seconds)

ST CLK PARITY LED is on
BC PH 742 and 1113 LEDs are on</pre>


<!-- Date: Mon Jul 3 05:32:23.570 2006 -->
<p>Short entry on clock (non-) progress.  After talking with Steve Chappa, I asked 
the shift to pause the run while I tried the memory verify function in the clock GUI. 
We were doing this because any bad parity bit in the clock should have caused a 
parity error in the clock, yet we were seeing none.  However, found the verify funtion 
did not work (read error in slave condition) and, worse yet, killed the master clock. 
Since this point, have tried resetting the clock with the clock GUI three times (the last 
after the ace reset the MVME processor), but found the clock with STAT_PARITY and 
DYN_PARITY errors after each time.  Steve Chappa and I are on our ways in.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->


<!-- Date: Mon Jul 3 05:35:28.903 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5611&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5611  width='300' align='top'></a></td><td>plot</td></tr></table>
<!-- Author: Sarah -->
<!-- Comment: Mon Jul  3 05;38;51 comment by...Sarah -->
<pre>Got this sudden jump for B0AAGC. Managed to catch it in the fast time plotter but can't see it when I try to plot in D44. I presume it was momentary and hence missed by the lumberjack plotter. Not sure why we should get this peak when there is no beam in the machine. Perhaps it is a fallout from the recent clock errors? </pre>


<!-- Comment: Tue Jul  4 11;10;26 comment by...Peter Dong -->
The abort gap counters exhibit this behavior periodically.  It has to do with the wraparound of the scaler and is nothing to worry about.
<!-- Date: Mon Jul 3 05:39:00.275 2006 -->
<pre>*** Node b0clk00 in Error *** 
Last data more than 120s ago!  Node is not responsive 
</pre>
<!-- Author: niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395' target=_top>219395</a>) -->
<!-- Comment: Mon Jul  3 05;50;57 comment by...niharika -->
<pre>again</pre>

<!-- Date: Mon Jul 3 06:20:01.826 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219395>219395</A>
 Terminated at 2006.07.03 06:19:53
<!-- Author: RunControl -->



<!-- Date: Mon Jul 3 07:33:02.054 2006 -->
<pre>We got RTSERVER error in ProcMon. 
Experts are probably working on it according to Steve.. so this is not totally unexpected. </pre>
<!-- Author: Niharika -->


<!-- Date: Mon Jul 3 07:35:37.204 2006 -->
<P> 
I restarted the rtserver so that b0clk00 could reconnect 
to it.   You probably saw some warning from ProcMon and/or  
RunControl during this procedure. 
<P> 
There was a sticky port left on the rtserver side when the  
clock crate was abruptly power cycled without going thru  
the "vxshutdown" procedure. 
<P> 
Anyhow, the rtserver connection is re&euml;stablished from  
b0clk00, but the problem with loading the Clock's static  
memory look-up table is still there... 
<P>
<!-- Author: badgett -->
<!-- Comment: Mon Jul  3 07;36;39 comment by...badgett -->
<pre>Steve:  If you need to power cycle the clock crate again, 
i suggest doing:

setup fer
vxshutdown b0clk00

first.</pre>

<!-- Date: Mon Jul 3 07:55:14.816 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Mon Jul 3 08:00:07.011 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219416>219416</A>
 Activated at 2006.07.03 07:58:56
<!-- Author: RunControl -->


<!-- Date: Mon Jul 3 08:00:08.992 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219416>219416</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,639] 
<!-- Author: Niharika -->



<!-- Date: Mon Jul 3 08:00:26.679 2006 -->
<font color=red>Shift Summary: </font><pre>At end of evening shift, protons in Tevatron were lost 
during shot.  pbars survived, so MCR said they would go 
into shot setup in a couple of hours.  However, they 
had problems and still haven't started shot setup. 
In the meantime, they rebooted their LLRF crate, which 
upset our clock, which in turn upset the silicon system. 
The silicon and clock experts were contacted and 
eventually all came in.  At the end of shift, the clock 
people think they have it working again.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             na
   
</td></tr></table></pre>
<!-- Author: Craig Blocker -->
<!-- Comment: Mon Jul  3 09;07;10 comment by...Steve Hahn -->
<p>So, a postmortem on the clock.  After Bill got the MVME up
and running again (and saw the same clock static memory errors
we did), we also confirmed the the clock GUI was still
complaining about the same verify errors.  Steve Chappa was about
to get the spare sequencer module when I suggested we try just
reseating the sequencer module.  Turns out this can be done live.
After resetting the clock again with the clock GUI, we had no
errors!  Thinking this was too good to be true, Steve C. tried
reseating one more time, and again after resetting the clock,
there were no errors.  Unsatisfying, but we are back up and
running.</p>