<!-- Title: Eve shift. Wed Feb 14, 2007 -->
<!-- SciCo: Willis Sakumoto -->
<!-- DAQAce: Gheorghe Lungu -->
<!-- MonAce: &nbsp; -->
<!-- CO: Monica D'Onofrio -->
<!-- OpManager: Farrukh Azfar -->
<!-- Notes: -->
<pre>Store 5228 continues: Lum=134e30, run 234302</pre>


<!-- Date: Wed Feb 14 16:10:43.339 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1937&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1937' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1938&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1938' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1939&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1939' width='300'></a></center>
<!-- Author: Jim (1500-1600) -->
<!-- Comment: Wed Feb 14 16;14;29 comment by...Jim -->
<pre>it looks like the sequencer doesn't work or at least it didn't work at the beginning of the store. I am pasting the hourly plots between 15:20 and 16:20. The proton losses were high, and also the inst. luminosity is dropping faster at the beginning of the interval. But I believe these are the reasons why the COT and Si was put to standby.
After those incidents, the plots look fine.</pre>
<!-- Comment: Wed Feb 14 16;15;15 comment by...Jim -->
<pre>make that time interval 14:20 - 15:20.</pre>
<!-- Comment: Wed Feb 14 16;31;40 comment by...ronmoore -->
For those interested in the gory details about what happened, see   <a href=http://www-bd.fnal.gov/cgi-mach/machlog.pl?nb=tev07&action=view&page=52&anchor=161325&hilite=16:13:25- target=_top> this entry</a> in the TeV eLog.
<!-- Date: Wed Feb 14 16:12:58.168 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1940&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1940  width='300' align='top'></a></td><td>L2_two_track_D100 shows some very high rate for 2 points: 2 points only... related to SVT problem with fitting the beamline?</td></tr></table>
<!-- Author: regis -->




<!-- Date: Wed Feb 14 16:16:15.948 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1941&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1941  width='300' align='top'></a></td><td>SVX monitor canvas empty in ObJMon: related to SVT having problems fitting the beamline?</td></tr></table>
<!-- Author: regis -->

<!-- Date: Wed Feb 14 16:17:15.168 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234302>234302</A>
general: <tt>Checklist done: some problems found with L1 xtrp and with SVT</tt>
<!-- Author: Consumer Operator(Regis Lefevre) -->


<!-- Date: Wed Feb 14 16:27:49.282 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313>234313</A>
 Terminated at 2007.02.14 16:27:46
<!-- Author: RunControl -->


<!-- Date: Wed Feb 14 16:28:31.780 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313>234313</A>
TERMINATE: svt not fitting the beam
<!-- Author: Jim -->



<!-- Date: Wed Feb 14 16:30:20.032 2007 -->
<pre>Jose, silicon expert, restarted IMON.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313' target=_top>234313</a>) -->


<!-- Date: Wed Feb 14 16:34:39.165 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
 Activated at 2007.02.14 16:34:31
<!-- Author: RunControl -->


<!-- Date: Wed Feb 14 16:34:40.261 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,672] 
<!-- Author: Jim -->



<!-- Date: Wed Feb 14 16:39:06.308 2007 -->
<pre>Attention!!! L2 Decision Timeout ERROR !!.  
  
 If L2 Timeouts persist, please page L2  
 
autoHRR ok</pre>
<!-- Author: Jim  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314' target=_top>234314</a>) -->


<!-- Date: Wed Feb 14 16:42:19.523 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1942&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1942' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1943&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1943' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1944&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1944' width='300'></a></center>
<!-- Author: Jim (1500-1600) -->
<!-- Comment: Wed Feb 14 16;43;28 comment by...Jim -->
<pre>hourly plots ok</pre>


<!-- Comment: Wed Feb 14 17;02;11 comment by...Jim -->
<pre>looks like the sequencer is ok, and the scrapbook gets plots. If I don't paste any plots it will be because of that. Also my comments will go in the scrapbook, and not here.</pre>
<!-- Date: Wed Feb 14 16:59:40.507 2007 -->
<pre>TOF HV trip. west channels 216->290 and east channels 0-73; recovered, now ok</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314' target=_top>234314</a>) -->



<!-- Date: Wed Feb 14 17:02:43.993 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1945&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1946  width='300' align='top'></a></td><td>B4W0L0 DVDD ISL</td></tr></table>
<!-- Author: Willis -->

<!-- Date: Wed Feb 14 17:18:23.249 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313>234313</A>
central: <tt>TOF West Side Average Time: one channel ~ 1/4 of the pedestal: red plot  in YMon. Posted on the e-log.  </tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 17:18:24.583 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313>234313</A>
trigger: <tt>L2: XTRPMon Summary still showing high error rate in Segment number 1,3,5 - Board 11 (top left plot) - pasted in the elog.<br>
SVT: Not fitting the beam</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 17:18:25.089 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313>234313</A>
monitors: <tt>XMon: HighBjet, MET+bjet,Photon_Bjet, Zbb triggers present an high rate: probably related to SVT problem (?). Posted in the elog. <br>
SVX Monitor empty: related to SVT problem (?)</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 17:18:26.827 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234313>234313</A>
general: <tt>End-Run check list done.</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->



<!-- Date: Wed Feb 14 17:20:55.683 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1947&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1947  width='300' align='top'></a></td><td>YMon: gap in distribution of TOFAveChargeTimeCanvas   </td></tr></table>
<!-- Author: Monica -->


<!-- Date: Wed Feb 14 17:21:41.358 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1948&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1948  width='300' align='top'></a></td><td>XTRPMon: Summary still showing high error rate in Segment number 1,3,5 - Board 11 (top left plot)</td></tr></table>
<!-- Author: Monica -->
<!-- Comment: Wed Feb 14 22;14;02 comment by...KPitts -->
This error is an intermittent problem between the SLAM and XTRP.  It seems to appear every couple of months or so.   Its affect on the trigger is microscopic. 
<!-- Date: Wed Feb 14 17:24:40.821 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1949&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1949' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1950&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1950' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1951&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1951' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-1952&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-1952' width='300'></a></center>
<!-- Author: Monica -->
<!-- Comment: Wed Feb 14 17;25;15 comment by...Monica -->
High rate triggers from XMon: probably related to SVT problems in fitting beamline... 

<!-- Date: Wed Feb 14 18:07:54.586 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
 Terminated at 2007.02.14 18:07:40
<!-- Author: RunControl -->


<!-- Date: Wed Feb 14 18:08:02.685 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
TERMINATE: end run svt problems
<!-- Author: Jim -->


<!-- Date: Wed Feb 14 18:17:19.072 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317>234317</A>
 Activated at 2007.02.14 18:17:08
<!-- Author: RunControl -->


<!-- Date: Wed Feb 14 18:17:20.885 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317>234317</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_01 [2,852,672] 
<!-- Author: Jim -->



<!-- Date: Wed Feb 14 18:30:27.749 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1953&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1954  width='300' align='top'></a></td><td>AVDD SVX B1W8L4</td></tr></table>
<!-- Author: Willis -->
<!-- Comment: Wed Feb 14 18;44;54 comment by...Willis -->
This had to be HRR'd 
<!-- Date: Wed Feb 14 18:40:36.774 2007 -->
<pre>During run 234317, I got a done TO from HTDC00, then FrontEnd Crate Error condition from: CCAL_00 to CCAL_15, VRB_ISL_06, WCAL_00 to WCAL_07, RXPT Channel 24 Local: 24, RXPT Channel 25 Local: 25, FCAL_00, FCAL_01, TEST_CAL_01, TEST_SVT_01, RXPT Channel 31 Local: 31, TEST_SVX_01, PCAL_00 to PCAL_11, RXPT Channel 39 Local: 39, RXPT Channel 46 Local: 46, RXPT Channel 47 Local: 47, 
COT_00 to COT_19, RXPT Channel 68 Local: 68, TEST_LEVEL2_02, RXPT Channel 70 Local: 70 to RXPT Channel 79 Local: 79, IMU_00, IMU_01, CMU_01, CMU_00, CMP_00, MUON_SCINT_00, CMX_00, CMX_01, RXPT Channel 88 Local: 88 to RXPT Channel 95 Local: 95, XFT_XTRP_00, XFT_FINDER_00, XFT_LINKER_01, XFT_FINDER_02, XFT_LINKER_03, XFT_FINDER_04, XFT_LINKER_05, XFT_FINDER_06, MUON_TRIGGER_00, LEVEL2_PULSAR_05, XFT_FINDER_07, C3PO_00 
RXPT Channel 108 Local: 108 
RXPT Channel 109 Local: 109 
RXPT Channel 110 Local: 110 
RXPT Channel 111 Local: 111 
VRB_SVX_02 
RXPT Channel 113 Local: 113 
RXPT Channel 114 Local: 114 
SVT_PULSAR_09 
TEST_SVT_02 
SVT_PULSAR_08 
CLC_00 
CAL_PULSER_01 
SVT_TRACKING_00 to SVT_TRACKING_05 
SVT_FANOUT_07 
SVT_FITTER_06 
LEVEL1_CAL_00 to LEVEL1_CAL_05
LEVEL1_GLOBAL_00 
TS_CRATE_00 
RXPT Channel 136 Local: 136 
RETURN_XPT_00 
TRIGGER_SCALERS_00 
COT_PULSER_00 
RXPT Channel 140 Local: 140 
TEST_MUON_01 
TEST_XFT_01 
TEST_XFT_02 
LEVEL2_CAL_00 to LEVEL2_CAL_05
LEVEL2_PULSAR_00 to LEVEL2_PULSAR_04
EM_TIMING_00 
RXPT Channel 155 Local: 155 
RXPT Channel 158 Local: 158 
RXPT Channel 159 Local: 159 
 
Then an autoHRR was issued, but the RC was stuck between Active and Halted. Right after that I got a message saying that the Error Handler was missing, even though it was open. I have closed the Error Handler window, started a new one, and tried to take it through the transitions manually. It didn't go beyond IDLE. Tried to shepherd it, and it kept trying to partition it without success. 
 
At this point, RC was still stuck, so I have manually overridden the conditions until it got to active. Error Handler window was empty and yellow in Replies and Acknowledgments window. 
 
Then I got error from ProcMon: RTserver not running. Looking at the error message, there was 1 stuck port: b0control21 (the DAQ ace pc). However, the pc seemed to work fine, nothing stuck except for the Error Handler issues. At this point we paged the DAQ expert, and M. Datta answered and assisted me. The RTserver problem cleared after a while, and she suggested to wait like 5 minutes and force ProcMon check again when similar problems with RTserver occurs. 
Still, the Error Handler couldn't be fixed while she was on the phone. 
We had to end the run anyway when the SVT problems got fixed, so we proceeded. Error Handler was still sutck, manually overide condition, got to idle, reset, got to start. Restart ErrorHandler, shepherd it, then started a fresh run and everything is back to normal. 
</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->


<!-- Date: Wed Feb 14 18:41:39.494 2007 -->
<pre>Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Pipeline out of synch in 101 silicon readout chips .  
 AUTO HRR will be issued   
 
 
autoHRR ok.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->


<!-- Date: Wed Feb 14 18:47:41.663 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
cal: <tt>CEMPMT: high occupancy for tower eta 0, phi wedge 0: checked on the high threshold plot is ok. </tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 18:47:42.200 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
trigger: <tt>SVT: to be marked bad (waiting for expert to confirm)</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 18:47:43.627 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
monitors: <tt>XMon: all trigger path involving CMX show an high rate.<br>
 <br>
ObjectMon: still 2 red distributions (Met and Charge/pT): ok</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 18:47:44.456 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234314>234314</A>
general: <tt>Check list done ok: to be marked not good for SVT.</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->



<!-- Date: Wed Feb 14 18:50:51.195 2007 -->
<table border=0><td><tr><td bgcolor=#e0e030><h2>Today SVT Problem</h2> 
SVT was not fitting the beam for runs 234302,234313,234314<br> 
The reason for this is that b0svt07 was freezing all the crates avoiding the SPYBUFFER to be filled with the tracks. In this way the beam fitter was not finding enough tracks to perform the fit of the beam.<br> 
The reason that caused the b0svt07 to freeze is not clear.<br> 
<h4> The plot 2.3.4: L2_Table_Beam has to be considered bad as soon as it become RED, indipendently on what is written in the table. There is a bug and we need to fix it.</h4></td></tr></td></table>
<!-- Author: Stefano & Marco -->

<!-- Date: Wed Feb 14 19:28:52.625 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317>234317</A>
monitors: <tt>XMon: trigger paths involving CMX still at slightly high rate. <br>
Among them, L2_CMX6_PT15_LUMI_200_v1, L2_CMX6_PT15_PS10_v1, L2_CMX6_PT8_&_TRK5_LUMI_200_v1 (zScore ~ 3.0-3.5).</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 19:28:53.690 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317>234317</A>
general: <tt>Check List done: everything ok so far </tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->



<!-- Date: Wed Feb 14 19:46:22.063 2007 -->
SVX Trip: B2 W9 VBIAS4 and AVDD0 overcurrent. Reset it successfully.
<!-- Author: Jim, Willis :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->
<!-- Comment: Wed Feb 14 20;02;01 comment by...Jose, Ignacio, Ulrich -->
The bias current for SVX B2W9L4 has been high and erratic for a long time. Recently it had started growing again, finally reaching the trip limit of 1000 &mu;A. The trip on AVDD was just a consequence of power-cycling the wedge. There seems to be no problem with the data coming from that ladder.  When the ladder tripped the second time, we have bumped up the trip limit to 1500 &mu;A. Please keep an eye on the bias current for the ladder and page Silicon if it reaches 1400 &mu;A.


<!-- Date: Wed Feb 14 20:09:06.809 2007 -->
<pre>got Done TO from COT_07, then again that long list of front end crates in error condition. The active in RC was red, and so I hit halt, but RC got stuck in transition. In this time I have shepherded crate COT_07 (even though it wasn't red), which was ok because it brought the crate to halted like the rest of them. 
  
Since RC was stuck, I have overridden the condition of RC manually until active. The reason for that was errlog was yellow and the Error Handler seems not updating again. Restarted it, but no good: it's waiting for configuration message.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->
<!-- Comment: Wed Feb 14 20;15;25 comment by...Jim -->
<pre>shepherded errlog, it was ok, and now running.

Hmm, this problem seemed very similar to the one I got before involving htdc00 done TO. It got errlog stuck and the RC along with it. However, this time I was quicker in getting back to active, and perhaps shepherding COT_07 wasn't necessary. 
Something is funny with these error messages and their handling by the Error Handler.</pre>

<!-- Comment: Wed Feb 14 20;20;07 comment by...Donatella T. -->
<pre>the Error Handler is crashing in a TipcSrv loop:

007.02.14 17:37:21     at com.smartsockets.TipcSrvImpl.process(TipcSrvImpl.java:901)
2007.02.14 17:37:21     at com.smartsockets.TipcConnClientImpl.mainLoop(TipcConnClientImpl.java:1216)
2007.02.14 17:37:21     at com.smartsockets.TipcSrvImpl.mainLoop(TipcSrvImpl.java:745)
2007.02.14 17:37:21     at rc.eh.EhAnalizer.loop(EhAnalizer.java:10796)
2007.02.14 17:37:21     at rc.eh.EhAnalizer.main(EhAnalizer.java:13660)



is the RTserver up and running fine?
could you please check the RTServer and perhaps start it again...
or has anything else changed in RC? 

</pre>

<!-- Comment: Wed Feb 14 20;32;01 comment by...Jim -->
<pre>RTserver is running fine. I don't know if RC has been changed.</pre>

<!-- Comment: Wed Feb 14 20;45;27 comment by...Donatella T. -->
the calls where the EH is crashing are related to decoding RXPT registers for Done To/Busy/ or Error
eg.
      status.rxpt_DtimedOut = getRXPTCrateList(dSreg, false);


Something must have changed in the RXTP board.

Could you please contact Bill B. about  this.



<!-- Comment: Wed Feb 14 20;57;53 comment by...Willis -->
Help desk call put out to Bill Badgett. Like our other 
attempts at help desk requests, we strike out.
<!-- Comment: Wed Feb 14 21;59;19 comment by...Willis -->
Note: Just figuratively referring to HelpDesk here...
<!-- Date: Wed Feb 14 20:54:44.249 2007 -->
<pre>heart beat alarm from ICICLE. Restarted the program.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->
<!-- Comment: Wed Feb 14 21;13;49 comment by...Jim -->
<pre>ProcMon error: ICICLE not updating. Restarted ICICLE (again).

Do I do anything wrong with this guy (ICICLE)?!</pre>


<!-- Date: Wed Feb 14 21:50:19.276 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-1955&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-1956  width='300' align='top'></a></td><td>B1W1L4 AVDD SVX: HRR did not help. Contacted Ulrich, who 
advised setting it to IGNORE.</td></tr></table>
<!-- Author: Willis -->
<!-- Comment: Wed Feb 14 21;58;39 comment by...Ulrich -->
We have had the same ladder showing something very similar last Tuesday. Apparently one of the chips drops out with a dead front-end. Last week, HRRs would not help in bringing back the chip, but it came back by itself after 10 hours or so. So there is not much we can do...


<!-- Date: Wed Feb 14 22:10:44.099 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317>234317</A>
smx: <tt>CCR has zero occupancy on East Pad, both per event and per hit:<br>
[ (8-9, wedge 5), (9,wedge 6), (6-7-8,wedge 14)]. Marked "not yet", waiting for more statistics.</tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->


<!-- Date: Wed Feb 14 22:10:45.352 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317>234317</A>
general: <tt>Check list done. </tt>
<!-- Author: Consumer Operator( Monica DOnofrio) -->




<!-- Date: Wed Feb 14 22:22:34.801 2007 -->
<pre>done TO from HTDC_00; autoHRR doesn't work, HRR doesn't work either because the "errlog" gets stuck (yellow). ProcMon happened to check RTserver during this, and it gave an error: "stuck port: b0control21", which probably is related to EH (not confirmed yet by experts). 
 
I have previously disabled the autoHRR function in the ErrorHandler in the hope that RC won't get stuck again if I issue HRR manually, but it didn't happen my way. So I enabled the autoHRR back. 
 
The only way around it so far: 
 
1) when the HRR is issued (automatically or not) if the RC gets stuck in transitions, just "manually overide the state manager condition"  (from Parameters tab in RC) until Active. 
2) click on the yellow "errlog" in Replies and Ack. window, get the LocalClientController for that and in there do: 
  - reset, partition, config, activate waiting for it to get through partitions 
3) start EH from RC (under Partitioning tab) 
 
Steps 2 & 3 might work in either order, not sure though. Anyway, do it so that errlog gets green and EH works. 
 
Until this gets fixed it will be a pain to recover any error.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->
<!-- Comment: Wed Feb 14 22;33;40 comment by...Ulrich -->
Be warned that you can screw up the DAQ by overriding state transitions for anything else that "errlog"! So please do not blindly do this.

<!-- Comment: Wed Feb 14 22;40;12 comment by...Jim -->
<pre>again, same HTDC_00 done TO. 

I've started step2 before step3 and it wasn't good: errlog turned red, got a RC message saying that EH is missing. Started EH and then took errlog back to active. However, EH GUI is not listening. Repeated the procedure on errlog, but the same so far. I will wait a little bit.</pre>

<!-- Comment: Wed Feb 14 22;42;00 comment by...Donatella T. -->
the problem will occur with RXPT generated errors ie. Done TO/Busy/Error  since it looks like the  decoding of the registers is wrong. This has been working til 17:37 pm, not to mention the past five years. I need some input from Bill in order to make any changes to the error handler, if that's the case.

<!-- Comment: Wed Feb 14 22;52;05 comment by...jdl -->
Can we back out by setting up an earlier version of fer?

<!-- Comment: Wed Feb 14 22;57;18 comment by...Willis -->
We can try backing out, does anyone know the previous version?

<!-- Comment: Wed Feb 14 22;57;28 comment by...Jim -->
<pre>again, same done TO from HTDC_00. Tried to get errlog in sync with the rest of the crates without the overriding, but I couldn't. So I have overridden the conditions again. 

I have to say that I am watching the xterm where probably RC is running, and issue the overriding after I hit halt, recover or run in RC and after I see the line :

Success seen in 134/135 clients.

which basically means all less errlog. That should be fine, I guess.</pre>

<!-- Comment: Wed Feb 14 22;59;35 comment by...Donatella T. -->
 I'm not sure wich one to use and if we'll need to reboot crates or else...

<!-- Comment: Wed Feb 14 23;04;27 comment by...Willis -->
Perhaps <a href=
'/cgi/elog/elog.pl?nb=2007&action=view&page=134&frame=2&anchor=13:23:40.799'>
this</a> may explain part of our problems.


<!-- Comment: Wed Feb 14 23;06;40 comment by...Jim -->
<pre>also, does the changing the version of FER code imply restart of all the other applications: L3 display, scalerMon, VxMon, etc?</pre>

<!-- Comment: Wed Feb 14 23;11;57 comment by...jdl -->
Last working version is <a href=
'/cgi/elog/elog.pl?nb=2007&action=view&page=131&frame=2&anchor=15:38:04.096'>
here</a>.  However, this would require rebooting at least the RXPT crate.  This is too dangerous to do in a seat of the pants way.  We'll lose more time from trying to reboot during the run than from the nasty HRR sequence.  The best thing to do is call all of Bill's phone numbers periodically until he answers one. 

<!-- Comment: Wed Feb 14 23;14;28 comment by...Willis -->
Good plan - we&#39;ll follow it, and I&#39;ll pass it on
to the owl SciCo.


<!-- Date: Wed Feb 14 23:30:15.828 2007 -->
<pre>we got some "Silicon alarm" coming from Ifix which I believe was fake since silicon is running ok, no trigger inhibit set, and the web page for Silicon warnings & alarms didn't record anythig too.  
 
More than that when I was looking thru the Racks menu in Process Systems Alarms picture, Ifix got stuck and got the error message that "Proficy HMI/SCADA Workspace will close..." and asked me whether I should send a message to Microsoft (which I did), then Ifix died on VNODE1.  
 
Restarted it, got an BSU alarm which also was fake since BSU was good and no inhibit set. Now we're good.</pre>
<!-- Author: Jim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=234317' target=_top>234317</a>) -->

<!-- Date: Wed Feb 14 23:55:21.863 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=234313 target=_runSummary>234313</A> x39349</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,672]</FONT></TD>
<TD ALIGN=CENTER>15:42:03</TD>
<TD ALIGN=CENTER>16:27:46</TD>
<TD ALIGN=CENTER>00:42:03</TD>
<TD ALIGN=RIGHT>     305,056</TD>   <TD ALIGN=RIGHT>    367.38</TD>
<TD ALIGN=RIGHT>    335.53</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=234314 target=_runSummary>234314</A> x3934A</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,672]</FONT></TD>
<TD ALIGN=CENTER>16:34:31</TD>
<TD ALIGN=CENTER>18:07:40</TD>
<TD ALIGN=CENTER>01:13:02</TD>
<TD ALIGN=RIGHT>     537,358</TD>   <TD ALIGN=RIGHT>    661.64</TD>
<TD ALIGN=RIGHT>    517.04</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=234317 target=_runSummary>234317</A> x3934D</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_01 [2,852,672]</FONT></TD>
<TD ALIGN=CENTER>18:17:08</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>05:08:16</TD>
<TD ALIGN=RIGHT>   1,944,178</TD>   <TD ALIGN=RIGHT>  1,778.79</TD>
<TD ALIGN=RIGHT>  1,622.60</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>07:03:21</TD>
<TD ALIGN=RIGHT>     2,786,592</TD> <TD ALIGN=RIGHT>  2,807.81</TD>
<TD ALIGN=RIGHT>  2,475.17</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Wed Feb 14 23:57:21.968 2007 -->
<font color=red>Shift Summary: </font><pre>1600: Start of shift. Store 5228 spinning, lum 134e30, run 234302 
1800: Rough running: SVT trigger crate b0svt07 is having problems 
1816: SVT'ers mitigated the b0svt07 problem. Run 234317 is good. 
1840: DTO's are hanging RC and crashing the error handler. Appears 
      related to recent RXPT code changes. APB for Bill Badgett! 
2200: Not so smooth running tonight, but taking data 
2345: Running a bit more smoothly now. So far, 2.30 pb-1 recorded 
      at 84% efficiency, but about 20% is marked bad (SVT/L2)</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   234313, 234314, 234317
Delivered Luminosity   2762.6 nb-1  
Acquired Luminosity    2331.9 nb-1  
Efficiency             84.4
   
</td></tr></table></pre>
<!-- Author: Willis -->