<!-- Title: Eve shift. Wed Jun 11, 2003 -->
<!-- SciCo: Simone Dell'Agnello -->
<!-- DAQAce: Gilles DeLentdecker -->
<!-- MonAce: Maria Spiropulu -->
<!-- CO: Roberto Rossin -->
<!-- OpManager: Nathan Eddy -->
<!-- Notes: -->
<pre>Found on-going store (=2673).</pre>


<!-- Date: Wed Jun 11 16:08:59.916 2003 -->
Collisions w/Lum=30E30
<!-- Author: Simone D. -->
<!-- Comment: Wed Jun 11 16;11;05 comment by...Simone -->
Physics run=164509.


<!-- Date: Wed Jun 11 16:12:43.666 2003 -->
multiplte DTO's: 
Done Timeout: COT_00 COT_02 COT_03 COT_10 COT_11 COT_13 
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164509' target=_top>164509</a>) -->

<!-- Date: Wed Jun 11 16:19:03.415 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164509">164509</A>
 Terminated at 2003.06.11 16:17:19
<!-- Author: RunControl -->


<!-- Date: Wed Jun 11 16:19:04.606 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164509">164509</A>
TERMINATE: Stop physics run
<!-- Author: Unki/Jaison x2080 -->
<!-- Comment: Wed Jun 11 18;49;19 comment by...SCN -->
<h3>Summation of 2 SRC mode and problems</h3>
Two SRC mode makes use of the Calibration Enable message sent from
the TS (TAXI Data = 000000001) as a "special L2A" to the second SRC, to insure that it gets L2 decisions in L1A order (otherwise,
L2R can "jump over" L2A).  It had been <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=tsi&action=view&page=10&frame=2&anchor=14:19:31'>
tested</a> (several entries) previously with L2 torture and all
types of readout including TDCs, and was successful, however, in
runs <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=409&frame=2&anchor=05:51:49.056'>
163031 and 163032</a> there were synchronization issues in the 2nd SRC which led us back to the drawing board.  Now the 
SRC related issues<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=158&frame=2&anchor=18:36:40'>seem to be gone</a> but the TDCs and perhaps SVT seem to be responding to
the special TAXI word in a way they didn't during the torture tests (it is hard to say if it happened in the data run, since
other errors were occuring at higher rate).<P>
Next step (already underway) is to query TDC/SVT people to see if the
response to the "express L2A" can be simply disabled by VME.
Only when that is answered will we reattempt this with any beam time.

<!-- Date: Wed Jun 11 16:28:03.955 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164510">164510</A>
 Activated at 2003.06.11 16:25:58
<!-- Author: RunControl -->


<!-- Date: Wed Jun 11 16:28:04.434 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164510">164510</A>
ACTIVATE: Start new physics run w. new RC
<!-- Author: Gilles X2080 -->



<!-- Date: Wed Jun 11 16:34:29.462 2003 -->
Stuck Cellid S/B0/W3/L4/C0-6
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164510' target=_top>164510</a>) -->
<!-- Comment: Wed Jun 11 16;35;16 comment by...Gilles -->
accompagnied with  Level 2 Decision Timeout

<!-- Date: Wed Jun 11 16:47:44.059 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164510">164510</A>
 Terminated at 2003.06.11 16:46:31
<!-- Author: RunControl -->
<!-- Comment: Wed Jun 11 17;01;17 comment by...rossin -->
we saw an error in this (short) run. after 90 events collected in the plot.
the xtrd expert has been paged.
we'll wait for more statistics.<p>

<a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6193&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6193'
                                                              width='300'></a>
<!-- Date: Wed Jun 11 16:47:57.501 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164510">164510</A>
TERMINATE: Abort run because of L2 DTO's.
<!-- Author: Gilles X2080 -->



<!-- Date: Wed Jun 11 16:50:45.973 2003 -->
D0 called: inserting their roman pots.
<!-- Author: Simone D. -->

<!-- Date: Wed Jun 11 16:52:17.681 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511">164511</A>
 Activated at 2003.06.11 16:50:44
<!-- Author: RunControl -->


<!-- Date: Wed Jun 11 16:52:18.584 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511">164511</A>
ACTIVATE: start new physics run
<!-- Author: Gilles X2080 -->



<!-- Date: Wed Jun 11 16:59:52.251 2003 -->
b0l2de00:Messenger:4:57:02 PM->CListMonitor: Sync error in L2 pipeline, event 57784
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511' target=_top>164511</a>) -->


<!-- Date: Wed Jun 11 17:03:57.127 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164509">164509</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVT: missing data in SVTMON comsumer; multiple DONE TIMEOUT from several TDC COT
crates. This run was taken on partition 7, with two SRCs.
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Wed Jun 11 17:03:59.933 2003 -->
Stuck Cellid S/B0/W3/L4/C0-6
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511' target=_top>164511</a>) -->
<!-- Comment: Wed Jun 11 18;23;46 comment by...Gilles -->
Stuck Cellid in 5 events in Silicon/S/B0/W3/L4/C0-6
<!-- Comment: Wed Jun 11 18;53;13 comment by...Gilles -->
Stuck Cellid S/B0/W3/L4/C0-6
<!-- Comment: Wed Jun 11 19;13;08 comment by...Gilles -->
<pre> Stuck Cellid in 155 events in Silicon/S/B0/W3/L4/C0-6 .[MLE]</pre>
<!-- Comment: Wed Jun 11 19;36;00 comment by...rainer -->
See comment <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=483&frame=2&anchor=18:05:36.405'>
here. </a> 

<!-- Date: Wed Jun 11 17:20:06.202 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6194&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6194' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6195&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6195' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6196&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6196' width='300'></a></center>
<!-- Author: smaria -->


<!-- Date: Wed Jun 11 17:38:17.094 2003 -->
DTO's parade: <br> 
b0l2de00:Messenger:5:30:12 PM->SvtMonitor: No data sent from client, event 441142
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511' target=_top>164511</a>) -->
<!-- Comment: Wed Jun 11 17;50;00 comment by...Gilles -->
b0l2de00:Messenger:5:44:05 PM->CListMonitor: Sync error in L2 pipeline, event 577828

<!-- Comment: Wed Jun 11 17;50;28 comment by...Gilles -->
<pre>b0l2de00:Messenger:5:44:50 PM->SvtMonitor: No data sent from client, event 577830</pre>

<!-- Comment: Wed Jun 11 17;56;11 comment by...Gilles -->
<pre>b0cot18:Messenger:5:46:41 PM->Runtime Error 1, Event 588010: See nonsequential event number (old 9, new 8) in L2 message. 
crate successfully shephered.</pre>

<!-- Comment: Wed Jun 11 18;05;54 comment by...Gilles -->
b0cot19:Messenger:6:03:54 PM->Event 751700: Bunchcounters in slot 5 (BC=12) and slot 20 (BC=15) disagree (L2B=3)


<!-- Comment: Wed Jun 11 18;48;33 comment by...Gilles -->
<pre>b0cot19:Messenger:6:46:55 PM->Event 1187799: Bunchcounters in slot 5 (BC=74) and slot 6 (BC=121) disagree (L2B=1)</pre>
<!-- Comment: Wed Jun 11 19;01;24 comment by...Gilles -->
<pre>b0cot18:Messenger:6:59:49 PM->Event 722347: Bunchcounters in slot 4 (BC=80) and slot 6 (BC=30) disagree (L2B=0)</pre>
<!-- Comment: Wed Jun 11 19;40;11 comment by...Gilles -->
<pre>b0cot18:Messenger:7:35:59 PM->Runtime Error 2, Event 1061404: See nonsequential event number (old 6, new 4) in L2 message</pre>
<!-- Date: Wed Jun 11 18:09:07.191 2003 -->
RMS LOSTP went pink a few seconds.
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511' target=_top>164511</a>) -->



<!-- Date: Wed Jun 11 18:35:14.426 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6197&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6197' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6198&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6198' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6199&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6199' width='300'></a></center>
<!-- Author: smaria -->







<!-- Date: Wed Jun 11 19:23:17.065 2003 -->
icicle.exe: application error. The instruction at "0x7800cb04" referenced memory at "0x78c7f9d2". The memory could not be "reda"; 
terminate application and restart. Icicle Heartbeat in the meantime. 
<!-- Author: smaria -->
<!-- Comment: Wed Jun 11 19;24;04 comment by...smaria -->
restart icicle




<!-- Date: Wed Jun 11 19:58:56.954 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6200&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6200' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6201&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6201' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6202&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6202' width='300'></a></center>
<!-- Author: smaria -->

<!-- Date: Wed Jun 11 20:13:45.439 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511">164511</A>
 Terminated at 2003.06.11 20:12:43
<!-- Author: RunControl -->


<!-- Date: Wed Jun 11 20:14:37.381 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511">164511</A>
TERMINATE: stop to refresh Hard. DB.
<!-- Author: Gilles X2080 -->
<!-- Comment: Wed Jun 11 20;19;42 comment by...Simone D. -->
The SVX expert found the reason why two SVT wedges were missing.
Consulted also w/SVT pager and decided (1) to end the run and
include the missing wedges, (2) declare SVT data good for this
run because these two wedges were missing in the data but also
in the hardware DB, so everything should be correctly accounted for.
<!-- Comment: Wed Jun 11 20;31;40 comment by...rainer -->
A FIB excluded during today's SRC2 study  was left out of the hardware database by accident after the test. It was successfully included back. See Silicon elog <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=158&frame=2&anchor=19:47:47'>
here </a>  and 
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=158&frame=2&anchor=19:47:47'>
here </a>.
<!-- Date: Wed Jun 11 20:18:56.491 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512">164512</A>
 Activated at 2003.06.11 20:17:31
<!-- Author: RunControl -->


<!-- Date: Wed Jun 11 20:18:57.113 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512">164512</A>
ACTIVATE: Start new physics run w/. new Hard. DB.
<!-- Author: Gilles X2080 -->




<!-- Date: Wed Jun 11 20:25:56.788 2003 -->
 b0l2de00:Messenger:8:23:49 PM->Unknown: ModDone error, event 41328
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->
<!-- Comment: Wed Jun 11 20;44;26 comment by...Gilles -->
b0l2de00:Messenger:8:42:21 PM->Slot12: ModDone error, event 163064


<!-- Comment: Wed Jun 11 20;52;11 comment by...Gilles -->
b0l2de00:Messenger:8:49:33 PM->Slot12: ModDone error, event 209619



<!-- Comment: Wed Jun 11 21;07;08 comment by...Gilles -->
Level 2 Decision Timeout


<!-- Date: Wed Jun 11 21:20:02.390 2003 -->
9:14:13 PM : ConsumerMonitor status not updated
<!-- Author: Gilles :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->
<!-- Comment: Wed Jun 11 21;20;54 comment by...Gilles -->
However SVXMON is running (same problem as on June 4)

<!-- Comment: Wed Jun 11 21;21;57 comment by...Gilles -->
It appeared 4 times already.

<!-- Comment: Wed Jun 11 21;24;51 comment by...smaria/gilles -->
one more


<!-- Date: Wed Jun 11 21:29:51.895 2003 -->
ConsumerMonitor Proxy was dead -- resterted it-- restarted also  
SVXMon  
<!-- Author: gilles/maria/BB :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->


<!-- Date: Wed Jun 11 21:31:17.316 2003 -->
<pre>for the past huge number of L2 Timeouts: here is the error on the  
L2 alpha minicom 
____________________________________________ 
 
Read L2 prescales 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 14079832 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 14531563 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 14803189 
Slot12 saw 3 sequential errors at Jun 11 20:22:09 2003 
FaultManager: Fatal error, sending L2 timeout (L2A 379563) 
 
 
In HALT state after 14803189 events 
 
Running Started 
 
Read L2 prescales 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 16252909 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 16419627 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 17049795 
Slot12 saw 3 sequential errors at Jun 11 20:31:24 2003 
FaultManager: Fatal error, sending L2 timeout (L2A 437140) 
 
 
In HALT state after 17049795 events 
 
Running Started 
 
Read L2 prescales 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 17206110 
 
Error: Startload is still low. ModDone bits 0x4007fdff for event 17716349 
</pre>
<!-- Author: gilles/maria/BB :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->
<!-- Comment: Wed Jun 11 22;44;45 comment by...maria -->
<pre>Gilles is sick; scico dismisses him</pre>
<!-- Date: Wed Jun 11 21:31:30.468 2003 -->
<pre>The partition 7 closed due to rtserver  
misbehaviour.   When you restart the run, 
please go back to "START" state and close  
and reopen your partition. 
 
This closure caused the COnsumerMonitor to die,  
and subsequent complaints from RC. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->


<!-- Date: Wed Jun 11 21:32:11.944 2003 -->
Paging L2 Decision for the continuous L2 Decision Timeouts
<!-- Author: gilles/maria/bb :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->
<!-- Comment: Wed Jun 11 21;49;52 comment by...smaria -->
It might be SVT (L2 expert will make a comment) the reason
<!-- Comment: Wed Jun 11 21;50;50 comment by...smaria -->
<pre>Page SVT : Marco said to stop automatic HRR's and catch the timeout next time so he can run a diagnostic- the cause maybe in the svx-- svx expert here</pre>
<!-- Comment: Wed Jun 11 22;07;05 comment by...Stephen -->
<pre>The L2 error is the board in slot 12 not sending any data.
That board is the SVT interface board. In the past this has
always been problems upstream of the L2 crate. Usually it
was part of the SVX failing to send data to SVT.</pre>
<!-- Comment: Wed Jun 11 22;26;01 comment by...SCN -->
As I understand it, Marco has  used Bill's whodunit script and
the culprit is not SVX.  I don't think the situation Stephen is
referring to has happened this calender year.  I suggest you page Greg, who has may have a better idea, and also Taka, who 
was doing some Ghostbusting earlier today.  This sounds very much
like what was happenning <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=486&frame=2&anchor=16:47:57.501'>
at the beginning of the shift</a>. 

<!-- Date: Wed Jun 11 21:48:55.867 2003 -->
<pre>b0dap73.fnal.gov:Thread-66:9:43:15 PM-&gt;Done Timeout: COT_18  
43'19" 1 crate/s: b0cot18(66),  Done TO.[RXPT] 
 
global tdc done timeout  
need to shepher cot18/OK</pre>
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->




<!-- Date: Wed Jun 11 21:54:50.161 2003 -->
Halt run to powercycle an SVX ladder /  
after recovery and SVX inhibit cleared we get  
 
SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/I/B0/W0/L0/C8-11 . 
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->


<!-- Date: Wed Jun 11 21:56:24.515 2003 -->
@the next L2 timeout there is no automatic HRR: Marco runs his diagnostic and sees no problem with SVT
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->



<!-- Date: Wed Jun 11 22:07:54.059 2003 -->
<pre>b0l3gate1.fnal.gov:main:10:04:47 PM-&gt;Host b0eb14.fnal.gov, task tRec_0 
SCPU-P7-E-VrbHeader: Dump of header words for event 598710 from VRB in slot 10: 
0x00000000 0x00002db0 0x00080008 0x020801dc 0x040c022c 0x0214028c 0x09680430 0x0 
e4002e8 
</pre>
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->


<!-- Date: Wed Jun 11 22:08:47.412 2003 -->
b0dap73.fnal.gov:Thread-66:10:02:38 PM->Done Timeout: COT_18  
CT: 6/11/2003, 22:2'42"  
2'42" 1 crate/s: b0cot18(66),  Done TO.[RXPT] 
 
shepher cot18 again
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164748' target=_top>164748</a>) -->


<!-- Date: Wed Jun 11 22:14:14.749 2003 -->
Again (more or less every 2 mins) 
 
b0dap73.fnal.gov:Thread-66:10:09:23 PM->  Level 2 Decision Timeout 
 
will page Steve Nahn
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->
<!-- Comment: Wed Jun 11 22;24;23 comment by...smaria -->
Steve N: there is no evidence that the SVX is causing this;
reminds us that in the beginning of this run Taka cured similar problem with coldstart
<!-- Date: Wed Jun 11 22:21:05.216 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512">164512</A>
 Terminated at 2003.06.11 22:18:41
<!-- Author: RunControl -->
<!-- Comment: Wed Jun 11 22;42;10 comment by...smaria -->
<pre>We have to terminate the run  to cure more than one problems:

1. L2 DONE timeout every minute
2. re-select partition
3. run with the dynamic prescales enabled
(DPS were DISABLED since the beginning of this store)</pre>
<!-- Date: Wed Jun 11 22:21:06.823 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512">164512</A>
TERMINATE: end run
<!-- Author: Gilles X2080 -->
<!-- Comment: Wed Jun 11 22;25;01 comment by...smaria -->
and we have to go to Start close and open partition again (see W.B. above)



<!-- Date: Wed Jun 11 22:28:03.016 2003 -->
Somehow we are running w/dynamic prescales (DPS) disabled. L1/2/3 rate are consequently low. I remember this being the case since the beginning of the shift. DPS must have been disabled sometimes 
during the two SRCs affair early this afternoon (just a guess). 
To correct this we're going to restart a new run (and it will have to be from scratch - see Bill B. earlier note in the e-log).
<!-- Author: Simone D. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512' target=_top>164512</a>) -->
<!-- Comment: Wed Jun 11 22;32;21 comment by...SCN -->
I seem to recall someone asking how long we wait after
disabling  "2 SRC Operations" to see if the TDC problems 
were gone, and the response was that we needed to wait at least
until after the Prescales changed, so it wasn't (to my knowledge)
done on purpose then.  Perhaps with the L2 Done Timeouts?
<!-- Date: Wed Jun 11 22:30:45.502 2003 -->
<pre>close partion 7 
select partition 7 
partition new run 
errlog does not partition 
quit errorhandler 
restart errorhandler 
partion errlog from RC local client 
L3 partitioning</pre>
<!-- Author: smaria -->


<!-- Date: Wed Jun 11 22:36:38.268 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164517">164517</A>
 Activated at 2003.06.11 22:33:43
<!-- Author: RunControl -->
<!-- Comment: Wed Jun 11 22;46;18 comment by...maria -->
Jonathan called to ask why we had to stop the run
<!-- Date: Wed Jun 11 22:36:39.066 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164517">164517</A>
ACTIVATE: new run
<!-- Author: Gilles X2080 -->
<!-- Comment: Wed Jun 11 22;39;11 comment by...smaria -->
With Prescales Enabled Now



<!-- Date: Wed Jun 11 22:43:10.980 2003 -->
<pre>(MLE) b0cot10:Messenger:10:41:26 PM-&gt;Event 52737: Bunchcounters in slot 5 (BC=62 
) and slot 11 (BC=74) disagree (L2B=1)  
(MLE) b0cot10:Messenger:10:41:26 PM-&gt;Runtime Error 1, Event 52737: Bunch counter 
 mismatch, mismatch count = 1 
</pre>
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164517' target=_top>164517</a>) -->




<!-- Date: Wed Jun 11 23:22:56.151 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6204&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6204' width='300'></a></center>
<!-- Author: smaria -->
<!-- Comment: Wed Jun 11 23;24;23 comment by...smaria -->
TEVMON ALARM (spike on C:B0PAGC): check abort gap behavior


<!-- Comment: Wed Jun 11 23;31;31 comment by...smaria -->
have SVX on standby
<!-- Comment: Wed Jun 11 23;32;24 comment by...smaria -->
svx expert here  
<!-- Comment: Thu Jun 12 00;05;43 comment by...smaria -->
<pre>&lt;a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=158&frame=2&anchor=23:49:36'&gt;
for the diagnosisetc see silicon log&lt;/a&gt; 
</pre>
<!-- Comment: Thu Jun 12 01;14;54 comment by...rainer -->
There was a spike in the abort gap losses which triggered a TEVMon alarm requesting the DAQ Ace to turn the the silicon to standby. The TEVMon alarm came back into normal values about 6 minutes later (see <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=158&frame=2&anchor=23:49:36'>
see silicon log</a>) , but we could not produce the corresponding (above) plot for a while and saw other rates increasing. MCR claimed they did not see any change. We paged Steve Nahn and discussed with him and eventually turned silicon back on.
<!-- Date: Wed Jun 11 23:31:00.652 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6206&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6206' width='300'></a></center>
<!-- Author: smaria -->
<!-- Comment: Wed Jun 11 23;32;44 comment by...smaria -->
Beam Halo plot


<!-- Comment: Wed Jun 11 23;42;24 comment by...smaria -->
called MCR -- nothing to worry-- call SCN; it looks like a glitch;  will bring up SVX
<!-- Date: Wed Jun 11 23:39:43.963 2003 -->
<pre>bring up SVX; @recover b0trin00 is hung; shephered it succesfully;  
 
immediately then:  twice done timeout from cot18 
 
b0cot18:Messenger:11:36:11 PM-&gt;Runtime Error 4, Event 5: TDC in slot 20 not ready for readout 
 
recovered after shepherding twice b0cot18</pre>
<!-- Author: smaria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164517' target=_top>164517</a>) -->



<!-- Date: Wed Jun 11 23:52:40.786 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6207&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6207' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6208&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6208' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6209&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6209' width='300'></a></center>
<!-- Author: smaria -->

<!-- Date: Wed Jun 11 23:56:55.609 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164509>
164509</A> 
x2829D</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [3,298,381]</TD>                                       
<TD ALIGN=CENTER>15:33:09</TD>                                          
<TD ALIGN=CENTER>16:17:19</TD>                                          
<TD ALIGN=CENTER>00:26:27</TD>                                          
<TD ALIGN=RIGHT>     22,610,169</TD>                                    
<TD ALIGN=RIGHT>       355,324</TD>                                     
<TD ALIGN=RIGHT>      75,082</TD>                                       
<TD ALIGN=RIGHT>   48.230</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164510>
164510</A> 
x2829E</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [3,298,381]</TD>                                       
<TD ALIGN=CENTER>16:25:58</TD>                                          
<TD ALIGN=CENTER>16:46:31</TD>                                          
<TD ALIGN=CENTER>00:10:46</TD>                                          
<TD ALIGN=RIGHT>      8,359,539</TD>                                    
<TD ALIGN=RIGHT>       132,317</TD>                                     
<TD ALIGN=RIGHT>      32,226</TD>                                       
<TD ALIGN=RIGHT>   18.368</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164511>
164511</A> 
x2829F</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [3,298,381]</TD>                                       
<TD ALIGN=CENTER>16:50:44</TD>                                          
<TD ALIGN=CENTER>20:12:43</TD>                                          
<TD ALIGN=CENTER>03:08:53</TD>                                          
<TD ALIGN=RIGHT>    140,516,041</TD>                                    
<TD ALIGN=RIGHT>     1,963,048</TD>                                     
<TD ALIGN=RIGHT>     508,693</TD>                                       
<TD ALIGN=RIGHT>  261.643</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164512>
164512</A> 
x282A0</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [3,298,381]</TD>                                       
<TD ALIGN=CENTER>20:17:31</TD>                                          
<TD ALIGN=CENTER>22:18:41</TD>                                          
<TD ALIGN=CENTER>01:42:57</TD>                                          
<TD ALIGN=RIGHT>     26,029,988</TD>                                    
<TD ALIGN=RIGHT>       668,773</TD>                                     
<TD ALIGN=RIGHT>     182,637</TD>                                       
<TD ALIGN=RIGHT>  112.863</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=164517>
164517</A> 
x282A5</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [3,298,381]</TD>                                       
<TD ALIGN=CENTER>22:33:43</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>00:44:43</TD>                                          
<TD ALIGN=RIGHT>     28,224,569</TD>                                    
<TD ALIGN=RIGHT>       394,951</TD>                                     
<TD ALIGN=RIGHT>     101,858</TD>                                       
<TD ALIGN=RIGHT>   42.584</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:01</TD>                                                       
<TD ALIGN=CENTER>06:13:48</TD>                                          
<TD ALIGN=RIGHT>     225,740,306</TD>                                   
<TD ALIGN=RIGHT>      3,514,413</TD>                                    
<TD ALIGN=RIGHT>       900,496</TD>                                     
<TD ALIGN=RIGHT>  483.687</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Jun 12 00:01:08.105 2003 -->
<font color=red>Shift Summary: </font><pre>Eve shift : June 11, 2003 (Wednesday) 
 
16:00   On-going store (=2673), run=164509,Lum=30E30. 
 
16:19   Multiplte DTO's: Done Timeout from COT crates. Stop 
	run, restart rc (see elog entries from rc experts). 
 
17:30   Run 164509 (48 nb-1) marked Bad, explanation: 
	SVT: missing data in SVTMON comsumer; multiple DONE 
	TIMEOUT from several TDC COT crates. This run was 
	taken on partition 7, w/2 SRCs (elog for details) 
 
20:19   The SVX expert found the reason why two SVT wedges 
	were missing. Consulted also w/SVT pager and decided 
	(1) to end the run and include the missing wedges, 
	(2) declare SVT data good for this run because these 
	two wedges were missing also in hardware DB (so 
	everything should be correctly accounted for). 
 
21:20   ConsumerMonitor status not updated. However SVXMON 
	is running (same as on June 4). Got help from Badgett 
 
21:45   Bad ladder which SVX expert requested to be 
	power-cycled: failed, so ladder lost until access. 
 
21:30-22:15 L2 Decision Timeouts. Paged L2 person. Paged 
	SVT. SVX is the culprit ? Paged Steve Nahn. 
 
22:20   Somehow we are running w/dynamic prescales (DPS) 
	disabled !! L1/2/3 rate are consequently low. 
	Restarted a new run w/DPS on. 
 
23:10   TEVMON ALARM (spike on C:B0PAGC, abort gap). 
	SVX set to stdby and investigatING. MCR says no 
	indication of problems, maybe fake. S. Nahn: 
	maybe a glitch. Restart run. 
 
24:00	Bad SVX ladder is back. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   164509/10/11/17
Delivered Luminosity   598.61 nb-1  
Acquired Luminosity    458.81 nb-1  
Efficiency             76.65 %
   
</td></tr></table></pre>
<!-- Author: Simone Dell&#39;Agnello -->
<!-- Comment: Thu Jun 12 01;23;17 comment by...rainer -->
According to Steve's summary of SRC2 activities in the <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=158&frame=2&anchor=18:36:40'>
Sili elog</a>, and  comment in Main<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=486&frame=2&anchor=22:28:03.016'>
elog </a> 
  we did <b> not </b> run in SRC2 mode but simply remained in partition 7 instead of 0.
<!-- Date: Thu Jun 12 00:25:17.811 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-6210&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-6210' width='300'></a></center><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: smaria -->