<!-- Title: Day shift. Mon Apr  4, 2005 -->
<!-- SciCo: Colin Gay -->
<!-- DAQAce: Veronica Sorin -->
<!-- MonAce: Oriol Salto -->
<!-- CO: Michele Gallinaro -->
<!-- OpManager: Sungwon Lee -->
<!-- Notes: -->
<pre>Starting shot setup</pre>


<!-- Date: Mon Apr 4 08:44:05.665 2005 -->
I restarted the HV monitoring on Muon3 (HVMON++).  As it turns out, when it was rebooted  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=279&frame=2&anchor=19:09:10.744'> 
yesterday</a>, CPR and CCR (these are the old school detectors which no longer exist).  As far as I can tell this seems to have 0 effect in the control room but just to be on the safe side we ended and started with CPR and CCR marked as "DO NOT USE". 
<br><Br> 
In a related note I modified the HV settings to ramp the HV for CMX faster.  I'll make a quick test to make sure this doesn't cause any problems.  
<!-- Author: Dale -->
<!-- Comment: Mon Apr  4 08;59;31 comment by...Dale -->
Restarted HVMON++ again - I didn't save the settings correctly on my 1st attempt.
<!-- Comment: Mon Apr  4 09;37;17 comment by...Dale -->
What I did:<br><br>
I modified the HV settings such that CMX will ramp up HV 2 times faster than it did (now 100 V/s) and ramp down HV 3 times as fast (now 300 V/s).  This is a poor man's way of speeding up the trip recovery times.<br><br>
In other news I changed the time on the muon3 pc such that it was moved ahead an hour to the correct time.  I note that the normally checked box for "automatically adjust for DLS time" was not checked (hence why it needed to be mannually changed).  I left it unchecked for fear that somebody had done this on purpose for some "highly important" reason.
<!-- Date: Mon Apr 4 08:57:25.876 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196142">196142</a>
 Terminated at 2005.04.04 08:56:56
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 08:57:26.133 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196142">196142</a>
TERMINATE: ending to see if you can include fcal01 now , no slot 10
<!-- Author: Peter x2080 -->



<!-- Date: Mon Apr 4 09:04:36.139 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196143">196143</a>
TERMINATE: problems with l2pu00, trying again
<!-- Author: Veronica x2080 -->


<!-- Date: Mon Apr 4 09:07:46.011 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196144">196144</a>
 Activated at 2005.04.04 09:07:39
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 09:08:04.746 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196144">196144</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [14,503,538] 
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 09:08:34.352 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5628&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5628  width='300' align='top'></a></td></tr></table>
cal_qie bad pedestals: rms -under 
<!-- Author: M. Gallinaro -->
<!-- Comment: Mon Apr  4 09;29;29 comment by...Steve Hahn -->
<p>It appears to me that there was some beam in the TeV while this QIE run 
was done, thus producing bad means and sigmas.</p>
<!-- Comment: Thu Apr  7 12;40;33 comment by...Steve Hahn -->
<p>Since Peter Wanger asked, I looked at ACnet in this period.  C:LOSTP C:LOSTPB
and T:IBEAM clearly show there was some activity starting at 0824 and continuing
intermittently (though more on than off) till 1000 or after.</p>
<!-- Date: Mon Apr 4 09:09:08.806 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196144">196144</a>
 Terminated at 2005.04.04 09:08:59
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 09:09:30.995 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196144">196144</a>
TERMINATE: abort, evb problems, cleaning evb
<!-- Author: Veronica x2080 -->


<!-- Date: Mon Apr 4 09:22:54.414 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196145">196145</a>
 Activated at 2005.04.04 09:22:48
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 09:23:09.320 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196145">196145</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [14,503,538] 
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 09:24:59.223 2005 -->
<pre>I am trying to start a run with fcal01 in, and actually is not this one the one giving problems, I got: 
 
b0l2pu00 problems, had to shepherded it. 
 
EVB problems, had to clean it, ok now. 
 
Then: 
 
Memory Exhausted for crtae b0xtr00, shepherded and ok now. 
</pre>
<!-- Author: Veronica -->

<!-- Date: Mon Apr 4 09:30:11.253 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196145">196145</a>
 Terminated at 2005.04.04 09:29:54
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 09:31:28.018 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196145">196145</a>
TERMINATE: ending, fcal01 did not give problems, however it is not going to be included at the beginning of the run.
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 09:49:59.572 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196146">196146</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,561,539] testing for pulsar (just a change on a debugging feature)
<!-- Author: Veronica x2080 -->


<!-- Date: Mon Apr 4 09:50:30.174 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196146">196146</a>
TERMINATE: ending 
<!-- Author: Veronica x2080 -->


<!-- Date: Mon Apr 4 09:52:39.127 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196147">196147</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,561,539] 
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 09:57:05.896 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5629&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5629' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5630&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5630' width='300'></a></center>
<!-- Author: jdl -->
<!-- Comment: Mon Apr  4 10;02;37 comment by...jdl -->
The left plot is the number of clusters per event in the test run (196063) where the forward jet seed was set to 127.875.  The right is the following run with standard clustering.  While root does not show overflows, the shape is not entirely unreasonable.  However, I have a little puzzled.  Since Trigmon sees a random sample of L1A events, I'd expect more events with 0 clusters in the test run.
<!-- Date: Mon Apr 4 10:00:55.480 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196148">196148</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Veronica x2080 -->


<!-- Date: Mon Apr 4 10:02:17.355 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196148">196148</a>
TERMINATE: ending test of L2 torture , new L2 tag
<!-- Author: Veronica x2080 -->




<!-- Date: Mon Apr 4 10:03:38.635 2005 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2005.04.04</td><td>10:00:49</td><td>W Inner BLM</td><td>0.35</td><td>RADS</td></tr> 
<tr><td>2005.04.04</td><td>10:00:49</td><td>W Outer BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2005.04.04</td><td>10:00:49</td><td>E Inner BLM</td><td>0.04</td><td>RADS</td></tr> 
<tr><td>2005.04.04</td><td>10:00:49</td><td>E Outer BLM</td><td>0.13</td><td>RADS</td></tr> 
</table> 
<strong>Integrated dosage</strong><br>
<!-- Author: Oriol -->


<!-- Date: Mon Apr 4 10:09:18.641 2005 -->
Got Heap Corrupt for b0cot06 on config, shpeherded it
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196149' target=_top>196149</a>) -->

<!-- Date: Mon Apr 4 10:09:54.094 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196149">196149</a>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_01 [2,618,559] 
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 10:10:04.717 2005 -->
Final protons loaded.  Total 9370 E9
<!-- Author: Colin -->


<!-- Date: Mon Apr 4 10:12:09.875 2005 -->
<P> 
Have reënabled the l2LinuxValid check in <BR> 
the RunSet.validate() method that is run at Partition<BR>  
and Config transitions.   This check requires that <BR> 
the trigger table be successfully compiled for the <BR> 
Pulsar linux executable.    Note that the latest COSMICS <BR> 
version has not been tested as to this check.<BR> 
<P> 
  
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196149' target=_top>196149</a>) -->
<!-- Comment: Tue Apr  5 09;26;57 comment by...WB -->
Correction:  COSMICS tables pass the check, but have 
not been tested in the system.
<!-- Date: Mon Apr 4 10:38:40.022 2005 -->
Pbars loaded.  Total=1600 E9
<!-- Author: Colin -->

<!-- Date: Mon Apr 4 10:42:01.462 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196149">196149</a>
TERMINATE: ending , goto current config
<!-- Author: Veronica x2080 -->


<!-- Date: Mon Apr 4 10:48:20.568 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196151">196151</a>
TERMINATE: go to highlum
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 10:53:42.133 2005 -->
got HEAP CORRUPT for b0cot04 on config, shepherded it, ok now.
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->


<!-- Date: Mon Apr 4 10:56:01.781 2005 -->
<pre>Store 4075 Scraping complete 
 
Protons: 8460 E9 
Pbars:   1470 E9</pre>
<!-- Author: Colin -->


<!-- Date: Mon Apr 4 10:57:34.595 2005 -->
Initial L=118 E30
<!-- Author: Colin -->


<!-- Date: Mon Apr 4 11:02:57.857 2005 -->
CLC tripped off.  Fixed now.
<!-- Author: Colin -->
<!-- Comment: Mon Apr  4 11;48;12 comment by...convery -->
What exactly happened?  Did the monitoring/control program crash?  thanks
<!-- Comment: Mon Apr  4 16;01;53 comment by...Oriol -->
Just after clicking "On step 2", we realized that the CLC turned OFF (spontaneously(?)). Then COT complained (alarm) but I saw no trip.  Downstairs the CLC monitoring was working but showed the HV OFF. Recovered CLC and COT continued ramping up.<br>
Sorry for the delay in the answear.
<!-- Date: Mon Apr 4 11:04:26.371 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152">196152</a>
 Activated at 2005.04.04 11:04:16
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 11:04:33.267 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152">196152</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_HIGHLUM_3_01 [1,617,558] 
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 11:14:41.684 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5631&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5631' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5632&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5632' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Mon Apr  4 11;15;31 comment by...Oriol -->
<b>Loss/AbGaps</b> and <b>SVRAD</b> during shot setup.


<!-- Date: Mon Apr 4 11:17:30.042 2005 -->
<pre>Although we are running, it is happening that on L3 display, nodes get green, output ones (although at some point converters too), what the experts are saying here, is that LumMon was taking too many events. We have restarted .</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->
<!-- Comment: Mon Apr  4 11;32;27 comment by...jdl -->
Lummon rate was a red herring.  The CSL was clogging up.  Expert is investigating.  The effect appears the same as when we got close to  the maximum logger rate before implementing parallel loggers.  There are bursts of high rate that clog the output.  The problem then clears itself after a minute or two. 
<!-- Date: Mon Apr 4 11:29:56.196 2005 -->
<pre>Due to the problems we are seeing with CSL, we have disabled Dynamic Precale, and we have put the ones that have already changed by now, to the maximum again: 
L1 : TWO_TRK10_DPHI20_DPS 
L2: 
CEM12_ISO_DPS 
CEM8_PT8_CES3_DPS 
DPS_L1_CMUP6_PT4 
2_CMU1.5_PT1.5_DPHI120_OPPQ_DPS 
TWO_TRK2_D100_&_MET15_DPS</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->



<!-- Date: Mon Apr 4 11:41:43.575 2005 -->
<br> 
SB3W3 lost HV communication, we reset the corresponding crate 
and now the Si is fine.<br>
<!-- Author: Sebastian -->


<!-- Date: Mon Apr 4 11:45:26.031 2005 -->
<pre>L2 Decision TO 
 
HRRed</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->
<!-- Comment: Mon Apr  4 11;53;33 comment by...Veronica -->
again


<!-- Comment: Mon Apr  4 12;12;23 comment by...Veronica -->
again
<!-- Date: Mon Apr 4 11:55:47.272 2005 -->
Am working on DownTimeEntries program now, so  
ignore ProcMon warnings. 
  
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->



<!-- Date: Mon Apr 4 12:13:46.391 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5633&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5633' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5634&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5634' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5635&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5635' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Mon Apr  4 12;14;34 comment by...Oriol -->
Hourly plots: <b>1110 - 1210</b>.


<!-- Date: Mon Apr 4 12:22:23.021 2005 -->
<b>PSM</b> warning. It lost the communication with SVX_SET_1.<br> 
Went away in ~1 minute.<br>
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->


<!-- Date: Mon Apr 4 12:23:33.504 2005 -->
Got a <b>COT</b> heartbeat alarm that went away in few seconds.<br> 
Everything looks fine in the COT PC.<br>
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->


<!-- Date: Mon Apr 4 12:33:04.819 2005 -->
Done TO for b0ccal00 
 
HRRed
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->


<!-- Date: Mon Apr 4 12:37:24.929 2005 -->
<pre>We are seeing a problem, but we are not sure what is really causing this: 
L3 is turning all GOLD, and partition state/transition part of the display is RED, it stays in that state for seconds and then it recovers itself. Also, we have got a few TevMon failing to receive Acnet messages via RTServer, when looking at process, we saw: 
 
tcp        0  76745 b0dap33.fnal.gov:5101   b0dap12.fnal.gov:43811  FIN_WAIT1 
 
Any suggestion? We are still running, but this issue with L3 happened already twice in the last half hour. 
 
</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->
<!-- Comment: Mon Apr  4 12;41;38 comment by...Veronica -->
Forgot to mention that we restarted TevMon.
<!-- Comment: Mon Apr  4 13;07;08 comment by...Veronica -->
There was a process running on b0dap12 , we asked the user to kill it, let's see now if the problem re appears or not.
<!-- Date: Mon Apr 4 12:40:35.098 2005 -->
ENABLING Dynamic Prescale, let's see what happens now with CSL.
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152' target=_top>196152</a>) -->



<!-- Date: Mon Apr 4 12:44:19.444 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5636&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5636  width='300' align='top'></a></td><td>TPOT positions since yesterday 1600</td></tr></table>
<!-- Author: Oriol -->

<!-- Date: Mon Apr 4 12:56:12.802 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152">196152</a>
 Terminated at 2005.04.04 12:55:34
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 12:56:45.026 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152">196152</a>
TERMINATE: going to the physics table
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 13:01:43.791 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5637&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5637  width='300' align='top'></a></td><td>COT SL4 occupancy: big spike at phi~50.</td></tr></table>
<!-- Author: M. Gallinaro -->
<!-- Comment: Mon Apr  4 14;31;59 comment by...Colin -->
Morris says this is a known issue.
<!-- Date: Mon Apr 4 13:03:05.450 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153">196153</a>
 Activated at 2005.04.04 13:02:05
<!-- Author: RunControl -->


<!-- Date: Mon Apr 4 13:03:07.540 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153">196153</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_01 [2,618,559] 
<!-- Author: Veronica x2080 -->



<!-- Date: Mon Apr 4 13:05:32.572 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5640&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5640  width='300' align='top'></a></td><td>Pulsar: CES alarm. Expert says it is ok. 
Included for elog documentation.</td></tr></table>
<!-- Author: M. Gallinaro -->



<!-- Date: Mon Apr 4 13:10:31.072 2005 -->
Got three pink silicon cells when changed the run:<br> 
<b>SVX AVDD B3W9L2</b><br> 
<b>SVX AVDD B5W7L2</b><br> 
<b>SVX AVDD B5W3L2</b><br> 
The histories of all of them are very flat. Unmarked all three and they returned to green.<br>
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->
<!-- Comment: Mon Apr  4 14;15;13 comment by...Oriol -->
Two of them came back to pink 1 hour and half afterwards:<br>
<b>SVX AVDD B3W9L2</b><br>
<b>SVX AVDD B5W7L2</b><br>
Unmarked them. Both came back to green.
<!-- Date: Mon Apr 4 13:17:28.849 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5641&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5641' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5642&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5642' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5643&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5643' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Mon Apr  4 13;18;44 comment by...Oriol -->
Hourly plots: <b>1210 - 1310</b>.<br>
Spike in <b>LOSTPB</b> of less than 4kHz, safe.


<!-- Date: Mon Apr 4 13:23:17.143 2005 -->
Got: 
 
Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb18.  
 Datataking is paused: events are not being processed.  
 
HRRed
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->
<!-- Comment: Mon Apr  4 13;24;50 comment by...Veronica -->
More on this:<br>
19'11" b0l3pcom1.fnal.gov:main:Apr 4, 2005 1:19:11 PM->Host b0eb18.fnal.gov, task tRec_0
SCPU-P0-E-BadVrbByteCount: Event 305290 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 20.
Count = 35860. -->  <br>
(MLE) b0l3pcom1.fnal.gov:main:Apr 4, 2005 1:19:11 PM->Host b0eb18.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 305290 from VRB in slot 20:
0x8c148b14 0x7002711a 0x72f38a14 0x7cf37d0a 0x7e038914 0x1004112b 0x12071302 0x140c15f3 <br>


<!-- Date: Mon Apr 4 13:33:18.142 2005 -->
L2 Decision TO 
 
HRRed
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->
<!-- Comment: Mon Apr  4 13;35;54 comment by...Veronica -->
again


<!-- Comment: Mon Apr  4 13;57;10 comment by...Veronica -->
again
<!-- Comment: Mon Apr  4 14;24;30 comment by...Veronica -->
again
<!-- Comment: Mon Apr  4 14;30;22 comment by...Veronica -->
again
<!-- Comment: Mon Apr  4 14;35;15 comment by...Veronica -->
again
<!-- Comment: Mon Apr  4 14;35;48 comment by...Veronica -->
again...too close to the last one this time
<!-- Comment: Mon Apr  4 14;51;54 comment by...Veronica -->
again
<!-- Comment: Mon Apr  4 15;14;28 comment by...Veronica -->
again
<!-- Date: Mon Apr 4 13:45:50.897 2005 -->
<pre>I am repopulating the store summary web pages -- you  
will not see store summaries until the job is finished,  
perhaps 20 to 30 minutes. 
</pre>
<!-- Author: W.Badgett -->
<!-- Comment: Tue Apr  5 09;30;26 comment by...WB -->
Turned out to take several hours!   Good thing we don't 
do this very often...
<!-- Date: Mon Apr 4 14:12:12.291 2005 -->
<pre>Got DOne TO for b0pcal07, 
 
HRR did not help, put run on halt, shepherded it,recover, run,  ok now</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=195153' target=_top>196153</a>) -->



<!-- Date: Mon Apr 4 14:18:18.738 2005 -->
<pre> Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb19.  
 Datataking is paused: events are not being processed.  
 
HRRed 
 
</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->
<!-- Comment: Mon Apr  4 14;18;56 comment by...Veronica -->
<br>15'39" b0l3pcom1.fnal.gov:main:Apr 4, 2005 2:15:39 PM->Host b0eb19.fnal.gov, task tRec_0
SCPU-P0-E-BadVrbByteCount: Event 1294660 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 12.
Count = 4161. -->  <br>
(MLE) b0l3pcom1.fnal.gov:main:Apr 4, 2005 2:15:39 PM->Host b0eb19.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 1294660 from VRB in slot 12:
0x10410410 0x009e0084 0x00ea0112 0x00000178 0x0086009a 0x00c000ca 0xe22035fc 0x83050004 <br>


<!-- Date: Mon Apr 4 14:19:50.087 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5644&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5644' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5645&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5645' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5646&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5646' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Mon Apr  4 14;20;25 comment by...Oriol -->
Hourly plots: <b>1310 - 1410</b>.



<!-- Date: Mon Apr 4 14:29:58.932 2005 -->
<pre>Attention!!!. SCPU_BAD_CHANNEL_COUNTS Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb21.  
 Datataking is paused: events are not being processed.  
 
HRRed</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=195153' target=_top>195153</a>) -->
<!-- Comment: Mon Apr  4 14;31;23 comment by...Veronica -->
(MLE) b0l3pcom1.fnal.gov:main:Apr 4, 2005 2:29:05 PM->Host b0eb21.fnal.gov, task tRec_0
SCPU-P0-E-EventIdMismatch: VRB ID for event 1523439 is 0 instead of 15 (slot 16).<br>
(MLE) b0l3pcom1.fnal.gov:main:Apr 4, 2005 2:29:05 PM->Host b0eb21.fnal.gov, task tRec_0
SCPU-P0-E-EventIdMismatch: VRB ID for event 1523440 is 1 instead of 0 (slot 16).<br>



<!-- Date: Mon Apr 4 14:35:06.461 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5647&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5647  width='300' align='top'></a></td><td>XFT occupancy: SL8 spike at phi~120.</td></tr></table>
<!-- Author: M. Gallinaro -->




<!-- Date: Mon Apr 4 14:46:11.131 2005 -->
Regeneration of tev summary plots is taking  
longer than expected -- don't expect them  
for a couple hours.  
<!-- Author: W.Badgett -->



<!-- Date: Mon Apr 4 15:13:05.172 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5648&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5648' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5649&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5649' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5650&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5650' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Mon Apr  4 15;13;38 comment by...Oriol -->
Hourly plots: <b>1410 - 1510</b>.



<!-- Date: Mon Apr 4 15:26:00.453 2005 -->
Got another <b>COT</b> heartbeat alarm. As the previous one (<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=281&frame=2&anchor=12:23:33.504'> 
here</a>) it desappeared in few seconds. COT PC looks fine.
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=195153' target=_top>195153</a>) -->
<!-- Comment: Mon Apr  4 15;28;09 comment by...Oriol -->
again


<!-- Date: Mon Apr 4 15:29:18.175 2005 -->
RTServer: <br>
 
again L3 went gold, and partition RED, and TevMOn failing to receive Acnet messages, so, we checked processes, we got: <br>
tcp        0  76001 b0dap33.fnal.gov:5101   b0dap85.fnal.gov:41232  FIN_WAIT1 <br>
 
On b0dpa85, we have: <br>
9647 cdf_svt   20   0 27676  13M  2720 R    57.3  1.3 58875m SvtHistogramPro <br>
  
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->


<!-- Date: Mon Apr 4 15:32:31.294 2005 -->
<pre>Got L2 Decision TO, auto HRR, but when it was trying to recover, many crates took time to go to recover, meaning yellow and on halted stattus, and we heard a complain also from TevMOn, about RTServer. The crates that were on halt, were b0fib0*, which recovered by themselves in seconds, but the b0tsi00 and 02 did not, so I send a recover message through LocalClientController, and we are back again.</pre>
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->


<!-- Date: Mon Apr 4 15:34:08.349 2005 -->
And again TevMon complaining. 
 
COT trip, so I am just checking now that there are no bad connections, so I do not know what happened at the exact moment TevMon complained.
<!-- Author: Veronica :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->


<!-- Date: Mon Apr 4 15:40:27.275 2005 -->
<b>COT trip</b> on layer 6. No beam activity related.<br> 
Called the expert in case it was related with the several heartbeats we has from the COT. He is coming.
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153' target=_top>196153</a>) -->

<!-- Date: Mon Apr 4 15:55:57.152 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196152>
196152</A> 
x2FE38</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_HIGHLUM_3_01 [1,617,558]</TD>                               
<TD ALIGN=CENTER>11:04:16</TD>                                          
<TD ALIGN=CENTER>12:55:34</TD>                                          
<TD ALIGN=CENTER>01:42:21</TD>                                          
<TD ALIGN=RIGHT>     50,091,112</TD>                                    
<TD ALIGN=RIGHT>     1,676,223</TD>                                     
<TD ALIGN=RIGHT>     416,335</TD>                                       
<TD ALIGN=RIGHT>  598.938</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196153>
196153</A> 
x2FE39</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_01 [2,618,559]</TD>                                       
<TD ALIGN=CENTER>13:02:05</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>02:27:26</TD>                                          
<TD ALIGN=RIGHT>    183,115,678</TD>                                    
<TD ALIGN=RIGHT>     2,887,225</TD>                                     
<TD ALIGN=RIGHT>     702,302</TD>                                       
<TD ALIGN=RIGHT>  658.760</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:07</TD>                                                       
<TD ALIGN=CENTER>04:09:47</TD>                                          
<TD ALIGN=RIGHT>     233,206,790</TD>                                   
<TD ALIGN=RIGHT>      4,563,448</TD>                                    
<TD ALIGN=RIGHT>     1,118,637</TD>                                     
<TD ALIGN=RIGHT> 1257.698</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Mon Apr 4 16:01:56.506 2005 -->
<font color=red>Shift Summary: </font><pre>Relatively smooth shift.  Plan for next crew is to continue taking data ...</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   196153
Delivered Luminosity   1517.8 nb-1  
Acquired Luminosity    1282.8 nb-1  
Efficiency              84.5
   
</td></tr></table></pre>
<!-- Author: Colin -->


<!-- Date: Mon Apr 4 16:06:33.768 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030>Trigger TABLES and RUN CONFIGURATIONS:<br> 
LINUX VALIDATION is working now on RC, that means that all configurations using the decoupled tables (the ones used for L2_TORTURE, COSMICS, calibrations, like SUMET, ) have to be changed to  
use the latest version of the tables (latest L2TAG).<br> 
</td></tr></td></table>
<!-- Author: Veronica -->
<!-- Comment: Mon Apr  4 16;44;00 comment by...Veronica -->
<table border=0><td><tr><td bgcolor=#e0e030>Latest tables to be used (L3Tag the same as before):<br>
COSMICS[15,619,xxx] (although not tested yet)<br>
COSMICS_NOTRACKS[9,620,xxx]<br>
MINIMUM_BIAS_1[12,621,xxx] (although not tested yet) <br>
SUMET5[5,622,xxx]<br>
L2_TORTURE[16,623,xxx]<br>
RANDOM[4,624,xxx]  (although not tested yet)<br>
ZERO_BIAS_90HZ[4,625,xxx]  (although not tested yet) <br></td></tr></td></table>