<!-- Title: Day shift. Wed May  5, 2004 -->
<!-- SciCo: Ken Bloom -->
<!-- DAQAce: Ingyin Zaw -->
<!-- MonAce: Antonio Boveia -->
<!-- CO: Roberto Rossin -->
<!-- OpManager: Mary Convery -->
<!-- Notes: -->
<pre>Store 3469 continues, lumi = 35.2E30, stack = 49.8E10. 
Silicon is in, expect to run all day.</pre>


<!-- Date: Wed May 5 08:35:35.220 2004 -->
When I came in, I noticed SVX pinkies for SB0W0L2 (AVDD), SB5W6L2(AVDD), and S1W4L3 (BIAS.)  Dae Jung told me that whichever expert brought the silicon up last night (Reid?) told him to ignore them for now.  I unmarked e002 (now yellow) and e143 (now green.)  The e562 AVDD current is only a few mA below the threshold.
<!-- Author: boveia -->


<!-- Date: Wed May 5 08:58:02.634 2004 -->
ObjectMon was crashed previous owl shift. 
We put back it as an old version 5.3.1l. 
 start script(startMon.sh) did not work for this version.  
Consumer Operator should type command by hand, not use script. 
 
% ssh -l cdfdaq b0dap82 
% runObjectMon 0   
<!-- Author: ryo and kohei and kaori -->


<!-- Date: Wed May 5 09:04:54.979 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6464&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6464' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6465&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6465' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6466&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6466' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6467&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6467' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 09;15;12 comment by...boveia -->
Hourly plots, 9am - 10am.  A largish spike in LOSTP just before 9:00 caused a TevMon warning.  Proton abort gap losses increased ~2 kHz over the hour (B0PAGC now at ~9.4 kHz.)
<!-- Date: Wed May 5 09:07:33.865 2004 -->
Got a Busy Timeout from b0svx06, recovered by auto HRR.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->

<!-- Date: Wed May 5 09:10:48.262 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409">182409</a>
smx: <tt>West side of PPR shows 2 wedges with high Average RAW ADC Count. These are Wedges 21 and 22. The occupancy is not affected, I'll mark as GOOD.</tt>
<!-- Author: Consumer Operator( roberto rossin) -->




<!-- Date: Wed May 5 09:47:57.299 2004 -->
Second Busy Timeout from b0svx06. Auto HRR fixed it.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->


<!-- Date: Wed May 5 10:05:08.545 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6473&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6473' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6474&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6474' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6475&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6475' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6476&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6476' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 10;08;14 comment by...boveia -->
Hourly plots, 9am - 10am.  Proton abort gap losses may become a problem later this shift.  B0PAGC is at 11.3 kHz and climbing ~2 kHz/hr.
<!-- Date: Wed May 5 10:07:04.430 2004 -->
made cdfsoftb0 release 5.3.1m (built on b0dap35) current. 
Except ObjectMon, all other Consumer Monitors are running with 5.3.1m
<!-- Author: DonatellaT, Kaori,Kohei,Ryo -->
<!-- Comment: Wed May  5 10;33;44 comment by...kaori -->
<pre>minor correction to above:
All the monitors are running with the new default version 5.3.1m
including ObjectMon.  So, CO person can start any monitors using
the usual startup script 'startMon.sh'.  SiliMon and ObjectMon
crash if TrackMonitor and JpsiMonitor (respectively) are on.
So, we turned them of in tcl in /data1 area.</pre>
<!-- Date: Wed May 5 10:22:16.235 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409">182409</a>
trigger: <tt>Canvas #8, L2 ISOlist, E1Sum1 plot shows about 3% of errors. There are no directions on how to evaluate this error rate. If we have not to check it please write it down explicitly.<br>
<br>
Canvas #115 TL2DMonitor has 1 plot missing, the XFLD vs. XTRP errors plot. If this is normal I suggest to update the reference plot.</tt>
<!-- Author: Consumer Operator( roberto rossin) -->
<!-- Comment: Wed May  5 23;28;26 comment by...Cheng-Ju -->
Yes, we are aware of the isolist ETSum1 error.  This problem
comes and goes.  I'll notify the Isolist expert.

As for TrigMon Canvas#115, we have temporarily disabled the
plot XFLD vs XTRP.  We'll re-enable the plot when we have the
updated TL2DMonitor code that properly accounts for the new
XTRP firmware running in TrigMon.
<!-- Date: Wed May 5 10:28:24.831 2004 -->
Done Timeout from b0cot13. Auto HRR fixed the problem.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->



<!-- Date: Wed May 5 10:43:14.772 2004 -->
<pre>SCPU_BAD_BYTE_COUNT error from b0eb24, recovered by auto HRR. 
 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb24. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->


<!-- Date: Wed May 5 10:51:33.578 2004 -->
From the run coordinator's elog:  The present store has good enough luminosity and lifetime so that I would like to keep it in through tomorrow's MI access. We would maintain the pbar stack through the access period, then shoot again once the access is over and the MI is running again. If this store falls out prematurely, we would put in another and keep it in during the access. If we're not able to have a store in during the MI access, we will probably access the Tev to repair the C23 turbo station and do cryo maintenance. 
<p> 
The MI access they refer to is supposed to start at 7 AM tomorrow and last six hours.  Also, notes from this morning's accelerator meeting say that last night's shot was the best since the March shutdown and the 7th best ever, by some metric that they don't specify.
<!-- Author: Ken Bloom -->
<!-- Comment: Wed May  5 11;21;26 comment by...ronmoore & paul derwent -->
Initial luminosity...plain and simple.
<!-- Date: Wed May 5 10:52:28.147 2004 -->
Got: 
 
Attention !!!. FERML_HEAP_CORRUPT ERROR !!! 
 
 - Heap Corrupt for crate b0pcal07. 
 - Please SHEPHERD crate b0pcal07. 
 
I halted, shepherded the crate, recovered, and ran. Seems to be OK now.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->


<!-- Date: Wed May 5 11:00:53.915 2004 -->
<pre>Got 3 trigger inhibits from CENTRAL HV in a row. They cleared themselves within a couple of seconds. I paused the run each time to investigate but they cleared too fast to see what was going on. 
 
Also got SCPU_BAD_VRB_BYTE_COUNT Error from b0eb11 after the 3 trigger inhibits. Auto HRR fixed the problem. 
 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 
</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->
<!-- Comment: Wed May  5 11;01;28 comment by...Ingyin -->
Got another trigger inhibit from CENTRAL HV. Again, cleared itself very quickly.


<!-- Date: Wed May 5 11:06:01.385 2004 -->
Gamma HV alarm, happened twice without time to determine the source.  Apparently the cryo tech silenced the CSX NE HV alarm.  This voltage is very close to the 2100 V high threshold and is crossing it on occasion (e.g. 2103 V.)<br> 
Though silenced, this happened another two times and kept setting trigger inhibits.  We are paging the CSX HV expert.<br>  
<!-- Author: boveia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->
<!-- Comment: Wed May  5 11;07;56 comment by...boveia -->
Another trigger inhibit.  Expert (Oleg) is investigating.
<!-- Date: Wed May 5 11:07:38.199 2004 -->
Another trigger inhibit from CENTRAL HV. This time, it is not clearing.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->
<!-- Comment: Wed May  5 11;08;30 comment by...Ingyin -->
Cleared after ~1 minute.

<!-- Comment: Wed May  5 11;09;10 comment by...boveia -->
It cleared, but the voltage jumped up higher (~2200 V) and so took longer to fall.

<!-- Comment: Wed May  5 11;10;46 comment by...Ingyin -->
Inhibit is set again. Getting GAM trips from CSX NE. We paged CSX and Oleg is looking into it. I paused the run.

<!-- Comment: Wed May  5 11;12;43 comment by...Ingyin -->
Cleared and running again.

<!-- Comment: Wed May  5 11;13;46 comment by...Ingyin -->
Got another inhibit which cleared immediately.

<!-- Comment: Wed May  5 11;15;50 comment by...Ingyin -->
Yet another one which cleared immediately.

<!-- Comment: Wed May  5 11;16;32 comment by...boveia -->
A couple more trigger inhibits, apparently because the supply voltage is now dropping below the low threshold.  Either way, the voltage is not stable.



<!-- Comment: Wed May  5 11;44;28 comment by...boveia -->
The CSX NE voltage is now stable within ~2V, near the 2050 V low threshold.  The alarm has been unsilenced.
<!-- Date: Wed May 5 11:22:22.205 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6477&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6477' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6478&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6478' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6479&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6479' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6480&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6480' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 11;24;19 comment by...boveia -->
Hourly plots, 10am - 11am.  Proton abort gap losses are at 14.0 kHz and rising at ~2.5 kHz an hour.  Reid stopped by for a look.
<!-- Date: Wed May 5 11:23:46.329 2004 -->
<pre>Got: 
 
Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
Hardware EVB has detected a problem with data quality in 
SCPU b0eb24. 
Datataking is paused: events are not being processed. 
- HRR should help. 
 
HRR did help.</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->
<!-- Comment: Wed May  5 11;30;05 comment by...Ingyin -->
Same error from b0eb24 again. Auto HRR fixed.
<!-- Date: Wed May 5 11:25:08.491 2004 -->
VOI (voice scheduler) is red in iFix, meaning it may have died.
<!-- Author: boveia -->
<!-- Comment: Wed May  5 11;26;37 comment by...boveia -->
The time-based entries window of the voice scheduler shows a good heartbeat, and the VOI box is now green again.


<!-- Comment: Wed May  5 11;41;41 comment by...boveia -->
I accidently close the voice scheduler window.  I restarted it by opening "voice_scheduler_vnode1.evs".  BSU 'alarmed' as I opened it, but that appears to be a false alarm (for instance, the alarm count did not increase from zero), and it allowed me to verify the voice scheduler was working again.



<!-- Comment: Wed May  5 13;04;00 comment by...boveia -->
JC came by to check that I had restarted the scheduler correctly.
<!-- Date: Wed May 5 12:05:37.341 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6481&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6481' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6482&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6482' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6483&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6483' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6484&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6484' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 12;06;38 comment by...boveia -->
Hourly plots, 11am - 12pm.  Abort gap losses at 15.5 kHz and rising 1.5kHz/hr.

<!-- Comment: Wed May  5 12;07;43 comment by...boveia -->
actually more like 2.5 kHz/hr.


<!-- Date: Wed May 5 12:11:21.474 2004 -->
At the halfway point for the shift, lumi = 28.0E30, stack = 51.6E10, things going generally smoothly but for the CSX voltages (now OK, it seems) and the rising (but not yet problematic) abort-gap losses.  Outside temperature 62 F and rising; sunny.
<!-- Author: Ken Bloom -->


<!-- Date: Wed May 5 12:21:29.255 2004 -->
<pre>Simultaneously got an SCPU_TRACER_EVENT_ID error from b0eb14 and a Done Timout from b0cot14. An auto HRR fixed both. 
 
Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
Hardware EVB has detected a problem with data quality in 
SCPU b0eb14 (forwarded by FER crate MUON_TRIGGER_00). 
Datataking is paused: events are not being processed. 
 
AUTO HRR will be issued. 
 
Done Timeout: COT_14</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->


<!-- Date: Wed May 5 12:39:08.446 2004 -->
<pre>Gon CER_SVXMON_HALT_RECOVER_RUN_ERROR. Fixed by auto HRR. 
 
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
Stuck Cellid I/B3/W0/L0/C0-3 . 
AUTO HRR will be issued</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->


<!-- Date: Wed May 5 12:54:19.319 2004 -->
Willis has fixed the channel mentioned <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=299&frame=2&anchor=05:13:47.764'> 
here</a>.
<!-- Author: boveia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->
<!-- Comment: Wed May  5 13;03;06 comment by...boveia -->
Apparently this was a power supply voltage readout problem that did not affect plug data-taking.


<!-- Comment: Wed May  5 13;05;32 comment by...Ken Bloom -->
Note that CO had (mistakenly) been calling this run bad due to this one channel.  We all agree that it is in fact good.


<!-- Date: Wed May 5 13:07:34.085 2004 -->
Only content in comment. 
<!-- Author: Willis -->
<!-- Comment: Wed May  5 13;09;13 comment by...Willis -->
I fixed the PHA yellow box in the slow controls HV summary page
due to Crate 10 slot 3 channel 18 (PHA east, wedge 12 tower 20)
The data taking set voltage (V0) in plug CAEN crate 10 for this 
channel got corrupted to some huge value, so I reset the value
and it is now okay. The actual voltage to this tower was not
affected, so the data is ok.


<!-- Date: Wed May 5 13:10:09.875 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6485&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6485' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6486&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6486' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6487&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6487' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6488&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6488' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 13;13;18 comment by...boveia -->
Hourly plots, 12pm - 1pm.  Proton abort gap losses are at 17.2 kHz and should cross the 18kHz warning threshold in the next hour.  I continue to twiddle my thumbs.


<!-- Date: Wed May 5 13:18:11.631 2004 -->
<pre>Simultaneously got an SCPU_TRACER_EVENT_ID error from b0eb14 and a Done Timeout from b0cot14. Auto HRR fixed both problems. 
 
Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
Hardware EVB has detected a problem with data quality in 
SCPU b0eb14 (forwarded by FER crate MUON_TRIGGER_00). 
Datataking is paused: events are not being processed. 
- HRR should help. 
 
</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409' target=_top>182409</a>) -->


<!-- Date: Wed May 5 13:39:41.909 2004 -->
Abort-gap losses are starting to get towards our limits.  We inform MCR, they will start looking at it.
<!-- Author: Ken Bloom -->

<!-- Date: Wed May 5 13:42:17.797 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409">182409</a>
cal: <tt>The monitor of the HV of PHA showed yellow bars instead of green bars. This doesn't mean the HV was not supplied (actually it was),my undestanding is that it was a readout problem from the CAEN power supply. In fact the occupancy of PHA was OK, no one channel was missing. During the first checks the CAL HV has been marked as bad while instead it has to be marked as good.</tt>
<!-- Author: Consumer Operator( roberto rossin) -->


<!-- Date: Wed May 5 13:42:22.901 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409">182409</a>
 Terminated at 2004.05.05 13:42:17
<!-- Author: RunControl -->


<!-- Date: Wed May 5 13:42:44.801 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409">182409</a>
TERMINATE: Ending to do SVT Road Warrior tests.
<!-- Author: Ingyin x2080 -->



<!-- Date: Wed May 5 13:43:23.122 2004 -->
We're stopping the run for SVT road warrior tests.  We'll also take this opportunity to includ the Pulsar crate in the readout.
<!-- Author: Ken Bloom -->

<!-- Date: Wed May 5 13:50:53.078 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</a>
 Activated at 2004.05.05 13:50:32
<!-- Author: RunControl -->


<!-- Date: Wed May 5 13:50:54.515 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</a>
 Activated at 2004.05.05 13:50:36
<!-- Author: RunControl -->



<!-- Date: Wed May 5 13:50:55.428 2004 -->
The red phone rang again!
<!-- Author: Ken Bloom -->

<!-- Date: Wed May 5 13:52:17.220 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</a>
ACTIVATE: AAA_CURRENT, PHYSICS_2_04[2,464,454] with LEVEL2_PULSAR_00 crate. SVT Road Warrior test.
<!-- Author: Ingyin x2080 -->



<!-- Date: Wed May 5 13:57:07.194 2004 -->
<pre>We got an SRC Fatality Error upon activation. HRRed out of it. 
 
From the error log: 
(RC)   13:50:27 Config -> READY 
CT: 2004.05.05 13:50:45 
 
50'45" 1 crate/s: b0svx02(112),  in error.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-26:1:50:45 PM->FrontEnd Crate Error Condition from: 
VRB_SVX_02 
 
(MLE) b0dap73.fnal.gov:Thread-26:1:50:45 PM->FrontEnd Error: VRB_SVX_02 
(MLE) b0tsi00:Messenger:1:50:46 PM->TS L2 FSM failed to start (State 14) 
(MLE) b0tsi00:Messenger:1:50:46 PM->TS Readout FSM failed to start (State 20) 
(MLE) b0tsi00:Messenger:1:50:46 PM->TS failed when trying to Run (State 2) 
(RC)   13:50:46 Activate -> ACTIVE 
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx02: Sl 5 L2A w/o L1A 
 
 - HALT the run 
 - record any relevant information in e-log 
 - RECOVER - RUN 
[MLE] 
(RC)   13:50:56 Halt -> HALTED 
CT: 2004.05.05 13:51:01 
51'1" 1 crate/s: b0svx02(112),  in error.[RXPT] 
(RC)   13:51:02 Recover -> RECOVERED 
(RC)   13:51:04 Run -> ACTIVE 
</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->


<!-- Date: Wed May 5 14:02:17.970 2004 -->
Another trigger inhibit from CENTRAL HV. It's from GAM: CSX NE again. The voltage seems to be dipping just below the lower limit. It cleared itself very quickly.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->
<!-- Comment: Wed May  5 14;04;16 comment by...Ingyin -->
Happened again.

<!-- Comment: Wed May  5 14;04;36 comment by...Ingyin -->
And again.


<!-- Comment: Wed May  5 14;08;30 comment by...Ingyin -->
And again. It was slightly longer this time. I paused and resumed. We paged Oleg. Happened four more times as I was typing this. Cleared itself quickly.
<!-- Comment: Wed May  5 14;12;47 comment by...boveia -->
These inhibits are caused by CSX NE HV hovering around the 2050 V low threshold.  We've masked off the inhibit to keep taking data while we wait for the expert.
<!-- Comment: Wed May  5 14;12;57 comment by...Ingyin -->
Three more times. I am masking the trigger inhibit for now.
<!-- Comment: Wed May  5 14;52;35 comment by...Ingyin -->
I have unmasked the trigger inhibit because Oleg has fixed the problem, as described <a href=
'/cgi/elog/elog.pl?nb=2004&action=view&page=300&frame=2&anchor=14:39:00.610'>
here </a>. 

<!-- Date: Wed May 5 14:07:09.689 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6489&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6489' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6490&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6490' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6491&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6491' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6492&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6492' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 14;10;29 comment by...boveia -->
Hourly plots, 1pm - 2pm.  B0PAGC is at 19 kHz and still climbing.  We may reach the 20 kHz hard limit in the next hour.  Ken called MCR during the past hour to complain.  
<!-- Date: Wed May 5 14:09:48.647 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409">182409</a>
monitors: <tt>Stage0_ gives this evaluation.<br>
***************************<br>
 182409 Stage0_ 103 CANNOTTELL<br>
(Errors) Insufficient_tracks:_DB_Copied<br>
*********************************<br>
I mark as good anyway and I send an email to the expert. I restart the Stage0.</tt>
<!-- Author: Consumer Operator( roberto rossin) -->
<!-- Comment: Wed May  5 15;51;49 comment by...Ingyin -->
See entry <a href=
'/cgi/elog/elog.pl?nb=2004&action=view&page=300&frame=2&anchor=15:38:52.472'>
below </a> for the explanation
<!-- Date: Wed May 5 14:18:22.519 2004 -->
Proton abort gap losses have reached 20 kHz. I halted the run. Antonio has turned silicon to standby.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->


<!-- Date: Wed May 5 14:24:39.972 2004 -->
Run 182410 was a test run for the SVT Road Warrior. 
It should be marked "bad" temporarily. In next few days 
we will decide if it can turn "good". 
Road Warrior test affected SVX wedge 11 only (all barrels).
<!-- Author: Roberto Carosi :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->


<!-- Date: Wed May 5 14:27:26.662 2004 -->
While we are in halt, Oleg is rebooting the muon HV PC.
<!-- Author: boveia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->


<!-- Date: Wed May 5 14:39:00.610 2004 -->
Rebooting the muon HV PC may have fixed the CSX HV problem we've been having.  Meanwhile, MCR has done some tuning to bring B0PAGC down to 15.5 kHz.  Silicon is back on and we've resumed the SVT testing.
<!-- Author: boveia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->


<!-- Date: Wed May 5 14:49:16.424 2004 -->
We got "Ack net may not be sending updates". ACNET JAVA, SVT SPY MONITOR, and DOWNTIME ENTRIES went red, indicating that b0dap85 had crashed. I rebooted b0dap85. We restarted Acnet, SVT Spy Manager and SVT Spy Monitor according to instructions on the Important DAQ Processes page (after checking with Roberto that restarting SVT SPY MONITOR does not affect the SVT road warrior tests). Everything seems fine now.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->
<!-- Comment: Wed May  5 14;56;37 comment by...boveia -->
Maybe just a coincidence -- MCR tuning and b0dap85 hanging occurred almost simultaneously.
<!-- Date: Wed May 5 14:54:42.458 2004 -->
Got SCPU_BAD_VRB_BYTE_COUNT error, fixed by auto HRR. 
 
Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
Hardware EVB has detected a problem with data quality in 
SCPU b0eb19. 
Datataking is paused: events are not being processed. 
 
AUTO HRR will be issued.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->



<!-- Date: Wed May 5 15:01:36.993 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6493&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6493' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6494&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6494' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6495&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6495' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6496&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6496' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 15;04;11 comment by...boveia -->
Hourly plots, 2pm - 3pm.  Proton abort gap losses reached 20 kHz around 2:15, so I put silicon to standby and MCR did some tuning.  B0PAGC has been stable at around 15 kHz ever since.


<!-- Date: Wed May 5 15:22:43.331 2004 -->
<pre>TrigMon crashed. I restarted it.</pre>
<!-- Author: roberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410' target=_top>182410</a>) -->

<!-- Date: Wed May 5 15:23:23.208 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</A>
 Terminated at 2004.05.05 15:22:57
<!-- Author: RunControl -->


<!-- Date: Wed May 5 15:23:59.936 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</A>
TERMINATE: Done with SVT road warrior test.
<!-- Author: Ingyin x2080 -->



<!-- Date: Wed May 5 15:24:04.800 2004 -->
Road Warrior tests concluded; we end the run, will restart with Pulsar crate still in the readout.
<!-- Author: Ken Bloom -->

<!-- Date: Wed May 5 15:29:24.891 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182411">182411</A>
 Activated at 2004.05.05 15:29:15
<!-- Author: RunControl -->


<!-- Date: Wed May 5 15:31:04.199 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182411">182411</A>
ACTIVATE: AAA_CURRENT, PHYSICS_2_04[2,464,454] with LEVEL2_PULSAR_00. SVT is back to the normal configuration.
<!-- Author: Ingyin x2080 -->



<!-- Date: Wed May 5 15:34:10.900 2004 -->
<pre>Got CER_SVXMON_HALT_RECOVER_RUN_ERROR, fixed by HRR. 
 
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
Pipeline out of synch in 119 silicon readout chips . 
AUTO HRR will be issued 
 
</pre>
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182411' target=_top>182411</a>) -->


<!-- Date: Wed May 5 15:36:36.479 2004 -->
L3 node 222 is purple. The error message says "Node Stopping" and "Shutdown received from faucet" at 15:23:23. I will continue data taking as it is for now.
<!-- Author: Ingyin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182411' target=_top>182411</a>) -->
<!-- Comment: Wed May  5 15;56;51 comment by...Ingyin -->
Reset Level 3 Display from the File menu and the node is ok now.
<!-- Comment: Wed May  5 15;57;56 comment by...Ingyin -->
By "ok", I mean light and dark blue as it should be.
<!-- Date: Wed May 5 15:38:51.631 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</A>
trigger: <tt>This is a SVT test run. Expert told us to mark it as bad.</tt>
<!-- Author: Consumer Operator( roberto rossin) -->


<!-- Date: Wed May 5 15:38:52.472 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</A>
monitors: <tt>Stage0 summary gives again the problem saw in previous run:<br>
***********************************8<br>
 182410 Stage0_ 331 CANNOTTELL<br>
(Errors) Insufficient_tracks:_DB_Copied<br>
************************************<br>
This is related to the restart of the Stage0 monitor immediately after the run started. Data are good, previous run has been checked. I mark this run as Good regarding Stage0.</tt>
<!-- Author: Consumer Operator( roberto rossin) -->
<!-- Comment: Wed May  5 15;48;30 comment by...Ingyin -->
Kevin Burkett helped me figure out the problem. Stage0 only makes this entry once per run. Apparently, Stage0 was restarted very early during these 2 runs, which is when Stage0 made the entry for these runs. At the time, there were insufficient tracks. Over time, there were sufficient tracks but Stage0 does not make this entry again. All the tables, the L3_PHYSICS_COT valid set, and PROD_PHYSICS_COT used set were made for this run, so it is OK. 
<!-- Date: Wed May 5 15:45:02.332 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410">182410</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVT -- Road Warrior tests; experts may change to status=good after futher study
</PRE>
<!-- Author: cdfscico -->





<!-- Date: Wed May 5 15:53:53.057 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6497&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6497' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6498&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6498' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6499&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6499' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-6500&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-6500' width='300'></a></center>
<!-- Author: boveia -->
<!-- Comment: Wed May  5 15;54;42 comment by...boveia -->
Hourly plots, 3pm - 3:52 pm.  Losses are approximately stable.


<!-- Date: Wed May 5 15:54:48.641 2004 -->
George Wyatt reports that he has installed a sono alarm on the fan in rack 3RR17E in the 3rd floor computing room.  There is a note attached to page ops manager if the alarm goes off.  Also, the alarm can be disabled by pulling the plug in the back of the crate (labelled as such).
<!-- Author: convery -->

<!-- Date: Wed May 5 15:55:32.063 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182409>
182409</A> 
x2C889</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_04 [2,464,454]</TD>                                       
<TD ALIGN=CENTER>03:07:05</TD>                                          
<TD ALIGN=CENTER>13:42:17</TD>                                          
<TD ALIGN=CENTER>09:20:12</TD>                                          
<TD ALIGN=RIGHT>    577,633,075</TD>                                    
<TD ALIGN=RIGHT>     9,006,905</TD>                                     
<TD ALIGN=RIGHT>   1,860,284</TD>                                       
<TD ALIGN=RIGHT> 1194.777</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182410>
182410</A> 
x2C88A</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_04 [2,464,454]</TD>                                       
<TD ALIGN=CENTER>13:50:36</TD>                                          
<TD ALIGN=CENTER>15:22:57</TD>                                          
<TD ALIGN=CENTER>01:13:14</TD>                                          
<TD ALIGN=RIGHT>     66,515,298</TD>                                    
<TD ALIGN=RIGHT>       884,258</TD>                                     
<TD ALIGN=RIGHT>     199,972</TD>                                       
<TD ALIGN=RIGHT>  107.897</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=182411>
182411</A> 
x2C88B</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_04 [2,464,454]</TD>                                       
<TD ALIGN=CENTER>15:29:15</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>00:23:42</TD>                                          
<TD ALIGN=RIGHT>     18,742,029</TD>                                    
<TD ALIGN=RIGHT>       237,755</TD>                                     
<TD ALIGN=RIGHT>      60,347</TD>                                       
<TD ALIGN=RIGHT>   32.780</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:03</TD>                                                       
<TD ALIGN=CENTER>10:57:09</TD>                                          
<TD ALIGN=RIGHT>     662,890,402</TD>                                   
<TD ALIGN=RIGHT>     10,128,918</TD>                                    
<TD ALIGN=RIGHT>     2,120,603</TD>                                     
<TD ALIGN=RIGHT> 1335.455</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Wed May 5 15:57:32.718 2004 -->
<font color=red>Shift Summary: </font><pre>Kept store 3469 for entire shift.  Running was mostly smooth, but for some trouble with CSX voltages, and steadily rising abort-gap losses.  At 13:40 we stopped the run to start Road Warrior tests.  Then losses became sufficiently high that we turned off the silicon and paused the run.  MCR tuned; meanwhile, rebooted muon HV PC, which fixed the CSX problems.  This allowed us to get the silicon back.  SVT tests finished at 15:30, back to normal running. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   182408-182410
Delivered Luminosity   818.6 nb-1  
Acquired Luminosity    725.0 nb-1  
Efficiency             88.6
   
</td></tr></table></pre>
<!-- Author: Ken Bloom -->