<!-- Title: Eve shift. Wed Apr 20, 2005 -->
<!-- SciCo: Stefano Belforte -->
<!-- DAQAce: Sofia Vallecorsa -->
<!-- MonAce: Daniel Sherman, Jiyong Han (trainee) -->
<!-- CO: Ulrich Husemann -->
<!-- OpManager: Aron Soha -->
<!-- Notes: -->
<pre>runnign without silicon,experts at work. Also w/o 2 level 3 subfarms.</pre>


<!-- Date: Wed Apr 20 16:16:27.733 2005 -->
Done TO for crate b0clc00. auto HRR
<!-- Author: sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798' target=_top>196798</a>) -->
<!-- Comment: Wed Apr 20 16;46;30 comment by...sofia -->
again
<!-- Comment: Wed Apr 20 17;15;45 comment by...sofia -->
again
<!-- Comment: Wed Apr 20 19;47;48 comment by...sofia -->
again
<!-- Date: Wed Apr 20 16:27:31.050 2005 -->
Flying wires
<!-- Author: sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798' target=_top>196798</a>) -->
<!-- Comment: Wed Apr 20 17;27;38 comment by...sofia -->
again
<!-- Comment: Wed Apr 20 20;27;15 comment by...sofia -->
again
<!-- Comment: Wed Apr 20 22;27;24 comment by...sofia& Jiyeon -->
again
<!-- Comment: Wed Apr 20 23;28;08 comment by...Jiyeon & Sofia -->
again
<!-- Date: Wed Apr 20 16:57:51.265 2005 -->
THe missing L3 subfarms are 4 and 9. We do not know at this tpoint exactely why they are out and wether they should stay out.
<!-- Author: Stefano -->
<!-- Comment: Wed Apr 20 16;58;52 comment by...Stefano -->
Luminosity is now 58.2E30, of course w/o silicon L2 rate is nothing to worry about, we do not miss the two L3 subfarms... at least not until silicon is back in


<!-- Comment: Wed Apr 20 17;26;30 comment by...Stefano -->
Sci.Co. called Arkadiy at his office. Arkadiy says those two sub-farms should be OK and can be included in next run, just be careful to do L3 clean up before.
<!-- Date: Wed Apr 20 17:10:01.777 2005 -->
Consumer monitors SiliMon and SVXMon restarted. SiliMon was restarted by mistake, sorry! The problem is with SVXMon: We are in a no-silicon run. Although SVXMon seemed to be running,  
it was recognized neither by its corresponding display nor by the status website. I was not able to find in the documentation whether this is expected in no-silicon runs. 
<!-- Author: Ulrich H. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798' target=_top>196798</a>) -->
<!-- Comment: Thu Apr 21 07;41;44 comment by...ryo -->
If you are in a no-silicon run, SVXMon should not be running.
<!-- Date: Wed Apr 20 17:10:38.513 2005 -->
Silicon people and Rob Roser still attending the thing. The cause of the problem has not been found yet.  
<!-- Author: Stefano -->
<!-- Comment: Wed Apr 20 18;48;04 comment by...Anna Maria Zanettis -->
six plus silison people still looking puzzled at each other in control room
<!-- Date: Wed Apr 20 17:43:02.880 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798">196798</A>
 Terminated at 2005.04.20 17:42:55
<!-- Author: RunControl -->


<!-- Date: Wed Apr 20 17:43:41.619 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798">196798</A>
TERMINATE: Inhibit for VME. put L3 subfarm back
<!-- Author: John Paul x2080 -->



<!-- Date: Wed Apr 20 17:43:41.343 2005 -->
Done TO for crate b0cot09. Auto HRR fixed it  
<!-- Author: sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798' target=_top>196798</a>) -->


<!-- Date: Wed Apr 20 17:46:31.178 2005 -->
Cleaning up L3
<!-- Author: sofia & Jiyeon -->


<!-- Date: Wed Apr 20 17:51:38.808 2005 -->
While configuring we got heap corrupt error for crates b0cot04, b0cot07. we shepherded them
<!-- Author: sofia & Jiyeon -->


<!-- Date: Wed Apr 20 17:53:07.299 2005 -->
Procmon error conerning RTserver. 
Second checked cleared it
<!-- Author: sofia & Jiyeon -->

<!-- Date: Wed Apr 20 18:01:41.320 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196799">196799</A>
 Activated at 2005.04.20 18:01:27
<!-- Author: RunControl -->


<!-- Date: Wed Apr 20 18:01:42.119 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196799">196799</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_3_02 [1,639,563] 
<!-- Author: John Paul x2080 -->



<!-- Date: Wed Apr 20 18:10:52.212 2005 -->
<p>As I was about to walk out of the building, I saw that the  
status panel was cycling through ALL the alarms (including, for 
example, flammable gas alarm).  I went and talked to Jim Humbert 
and Bruce Vollmer (Bruce was coming on shift after Jim), and 
found out that the control room had called them, and they, in 
turn, had put in a call for Mark Knapp.</p> 
 
<p>I went upstairs to check the status of the marquee picture on 
VNODE2, only to find Aron talking to Mark.  Mark had Aron press 
the ACK button on the marquee picture, at which point all the  
alarms went away, and the status panel returned to the nominal 
display.</p> 
 
<p>I went back to the cryo area and asked Jim and Bruce if there 
had been any alarms earlier which might have provoked this 
behavior.  There were silicon alarms, but this should not have 
produced any alarms on the status panel.  My best guess right 
now is that this is similar to the "?E1212" problem we saw  
before where tag values would disappear from their local  
database momentarily.  If this happened now, the display would 
continue showing the alarms until they were acknowledged (as 
Aron did).</p>
<!-- Author: Steve Hahn -->
<!-- Comment: Wed Apr 20 20;17;36 comment by...Aron -->
This sort of "false alarm" happened again about 30 minutes later.
I hit ACK again on the "marquee" picture.
<!-- Date: Wed Apr 20 18:11:06.085 2005 -->
COT_04 timed out 4 times in a raw and did not come back after sheprding. Paged DAQ and TDC. DAQ pager (Ray) calls while 5th sheparding works.
<!-- Author: Stefano :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798' target=_top>196798</a>) -->


<!-- Date: Wed Apr 20 18:16:46.894 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6667&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6667' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6668&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6668' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6669&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6669' width='300'></a></center>
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 18;17;25 comment by...Daniel -->
Hourly plots, 1600-1815


<!-- Date: Wed Apr 20 18:19:30.167 2005 -->
Daed time 100%, L3 is not rejecting evnets, not is L2. 
Paged CSL and L3 trigger. CSL answered immediately and is coming
<!-- Author: Stefano -->
<!-- Comment: Wed Apr 20 18;21;07 comment by...stefano -->
L3 pager that since L3 display is green (all nodes waiting to write output), it must be CSL problem
<!-- Date: Wed Apr 20 18:20:24.131 2005 -->
We got deadtime error message, deadtime is around 99%. 
We checked CSL display, there's no events in receiver. 
We called csl expert.
<!-- Author: Jiyeon & Sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196799' target=_top>196799</a>) -->
<!-- Comment: Wed Apr 20 18;21;34 comment by...s -->
CLS expert (Jedong Lee) arrives


<!-- Date: Wed Apr 20 18:29:26.408 2005 -->
Problem seems that too many events are marked as error in L3 and CSLchckes on writing stream D. L3 paged again.
<!-- Author: Stefano -->
<!-- Comment: Wed Apr 20 18;30;56 comment by...Stefano -->
We tred pasue/resume the run to drain pipeline. Pipeline was drained but it did not help (this in case the error events where the remnants of the COT crate troubles)


<!-- Comment: Wed Apr 20 18;33;39 comment by...s -->
L3 expert arrives
<!-- Comment: Wed Apr 20 18;41;09 comment by...s -->
Still no clue. Scaler Monitor claimes L2 is nto rejecting any event, do not know if this is a L2 problem or a Monitor problems
<!-- Comment: Wed Apr 20 18;42;54 comment by...S -->
We try to end the run and start a new one. Also Eric James is here. Also Op.Man.
<!-- Comment: Wed Apr 20 18;46;17 comment by...s -->
Burkard Reisert (L2 Pulsar expert) is also here
<!-- Comment: Wed Apr 20 18;47;22 comment by...s -->
Starting new run delayed by TOF trip
<!-- Comment: Wed Apr 20 18;48;29 comment by...Tom Wright -->
The rate-limits at L2 don't seem to be working, all those triggers look to be unprescaled in RunSummary.

<!-- Comment: Wed Apr 20 18;51;19 comment by...S -->
New run, same trouble. L3 experts finds out that L3 is now accepting all event.
<!-- Comment: Wed Apr 20 18;51;54 comment by...S -->
Experts debating if it is L2 or L3 who is not rejecting
<!-- Comment: Wed Apr 20 18;56;10 comment by...s -->
L3 expert asks to page L3 executable. Pager number is crossed on pager sheet. Sci.CO. pages nevertheless and calls Veonique Boisvert  in her office, she is coming in.
<!-- Comment: Wed Apr 20 19;06;38 comment by...S -->
Experts ask for BIll Badget. Called office/home/cell. Left a mesage on the answering machine of all of them.
<!-- Date: Wed Apr 20 18:32:45.983 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-6671&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-6671  width='300' align='top'></a></td><td>There are still hot stacks in CMP, see the previous 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=329&frame=2&anchor=11:11:06.513'> 
discussion</a>.</td></tr></table>
<!-- Author: Ulrich H. -->



<!-- Date: Wed Apr 20 18:42:02.285 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196799">196799</A>
 Terminated at 2005.04.20 18:41:28
<!-- Author: RunControl -->


<!-- Date: Wed Apr 20 18:42:06.777 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196799">196799</A>
TERMINATE: dead time prob
<!-- Author: John Paul x2080 -->







<!-- Date: Wed Apr 20 18:49:07.825 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800">196800</A>
 Activated at 2005.04.20 18:48:56
<!-- Author: RunControl -->


<!-- Date: Wed Apr 20 18:49:15.776 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800">196800</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_3_02 [1,639,563] 
<!-- Author: John Paul x2080 -->
<!-- Comment: Wed Apr 20 19;04;02 comment by...Tom Wright -->
In ScalerMon I see that L2_INCLUSIVE_NOTRACK_1HZ isn't being prescaled at all and is trying to pass events at the L1 accept rate.

<!-- Date: Wed Apr 20 18:50:57.635 2005 -->
Tof trip in crate 1.  Turned back on from TOF PC.
<!-- Author: Daniel :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->







<!-- Date: Wed Apr 20 19:16:12.744 2005 -->
<pre>L2_INCLUSIVE_NOTRACK_1HZ    
L2_INCLUSIVE_TRACK_1HZ 
seem to have a role in the problem, they have a very high rate ~6kHz</pre>
<!-- Author: sofia & Jiyeon :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->
<!-- Comment: Wed Apr 20 19;21;42 comment by...s -->
Burkard sets prescale to 0 for L2_INCLUSIVE_NOTRACK_1HZ  and we are OK now. Need to find out why it was not obeying Rate Limit to 1Hz.


<!-- Comment: Wed Apr 20 19;34;09 comment by...s -->
Now it looks like many L2 triggers that should be rate limited to 1Hz are running freely. The inclusive notrack jumps to 6KHz when left free, the others are not such killer.
<!-- Date: Wed Apr 20 19:24:19.308 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6672&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6672' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6673&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6673' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6674&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6674' width='300'></a></center>
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 19;25;06 comment by...Daniel -->
Hourly plots, 18:15-19:15
<!-- Date: Wed Apr 20 19:24:43.514 2005 -->
Update on silicon status and silicon cooling: 
After the cooling tripped due to work on the ISL chiller vaccum pumps we found that we could not power SVX B1 and B4 due to increasing port card temperatures.  The hypothesis is that bubbles were introduced in the system.  We tried flushing them out but were unsuccessful so we will need an access tomorrow. 
<br> <br> 
We are currently keeping the silicon off until this problem can be fixed.  Additional details are in the <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=245&frame=2&anchor=18:18:20'> 
silicon elog</a>.  
<br>  
<!-- Author: silcon and silicon cooling group + Roser -->



<!-- Date: Wed Apr 20 19:31:35.705 2005 -->
timeout in b0clc00, it go back soon
<!-- Author: Jiyeon & Sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->



<!-- Date: Wed Apr 20 19:35:02.392 2005 -->
It looks like some (not all) of the L2 triggers that are rate limited (like to 1Hz) are not being rate limited, but not all of them, so another one is L2_RL1Hz_L1_MB_CLC for example is running at 36Hz. What could be causing this?
<!-- Author: Veronique for shift crew :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->
<!-- Comment: Wed Apr 20 19;50;48 comment by...Ted -->
I don't think Daniel has changed anything on the L2 code side, at least I didn't hear anything from him. May be Bill Badgett can help to understand the problem.
<!-- Comment: Wed Apr 20 20;33;41 comment by...Ted -->
Bill is looking into the problem now from home. we will hear from him soon.
<!-- Date: Wed Apr 20 19:45:47.116 2005 -->
L2_RL0.5_L1_TOF_14_MIP_&_CLC  33.7 Hz 
L2_RL1HZ_L1_MB_CLC            49.6 Hz 
L2_RL2HZ_L1_TRK_10             5.3 HZ 
 
all supposed to be rate limited. 
Other rate-limited triggers are fine like: 
L2_RL1.3HZ_L1_MB_CLC
<!-- Author: sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->




<!-- Date: Wed Apr 20 19:56:38.070 2005 -->
DAniel Whiteson cvalls back and says he'll be in 10min in a place with connection so he can  look at things
<!-- Author: Stefano -->
<!-- Comment: Wed Apr 20 20;08;46 comment by...s -->
Ray Culberstoj (DAQ pager) sugest we call trigger table experts to help debugging this RateLimit thing. Sci.CO. pages Kirsten Tollefson (15min ago). No answer yes
<!-- Comment: Wed Apr 20 20;16;12 comment by...s -->
Kirsten calls back from her car, while driving home. She has no clear clue but similar problems (with prescale, not rate limit) happened in the last weeks and either went away or had been fixed by Bill Badgett's with some magic. She wil be home in ~20min and look from there.
<!-- Date: Wed Apr 20 20:02:58.961 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800">196800</A>
 Terminated at 2005.04.20 20:02:47
<!-- Author: RunControl -->


<!-- Date: Wed Apr 20 20:03:48.121 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800">196800</A>
TERMINATE:    ..
<!-- Author: John Paul x2080 -->



<!-- Date: Wed Apr 20 20:09:58.914 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801">196801</A>
 Activated at 2005.04.20 20:09:49
<!-- Author: RunControl -->



<!-- Date: Wed Apr 20 20:10:00.676 2005 -->
b0clc00,  Done TO. This caused a very high dead time from L2readout. 
 
Tried to HRR, then to shepherd the crate, nothing worked. 
Started a new run
<!-- Author: sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->

<!-- Date: Wed Apr 20 20:10:44.810 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801">196801</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_3_02 [1,639,563] 
<!-- Author: John Paul x2080 -->



<!-- Date: Wed Apr 20 20:11:19.622 2005 -->
YMon restarted (was hanging)
<!-- Author: Ulrich H. -->




<!-- Date: Wed Apr 20 20:19:37.235 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6675&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6675' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6676&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6676' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6677&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6677' width='300'></a></center>
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 20;20;04 comment by...Daniel -->
Hourly plots 19:15-20:15


<!-- Date: Wed Apr 20 20:21:25.915 2005 -->
It looks like trigger rate problems have been solved, by starting a new run and killing and restarting ScalerMonintor while in the READY state. 
<!-- Author: sofia & Jiyeon :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801' target=_top>196801</a>) -->

<!-- Date: Wed Apr 20 20:35:19.823 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030>Summary of the high-dead-time problems: 
<ul> 
<li>Run 196798 was fine 
<li>We got VME Inhibit from low voltage, because of work on silicon 
<li>We decided to use the time to stop the run and start a new one to include missgin L3 subfarms 
<li>At the start of the run COT_04 timed out and needed sheperding 4 or 5 times, until it worked. Then we noticed the high dead time. 
<li>This was due to L2 monitor triggers running beserk and flooding Stream D 
<li>Later inspection of L2 rates from DBSummary tells that 
L2_INCLUSIVE_NOTRACK_1HZ_v1 had been running at 60Hz all the time 
(see plot blow) 
<li>attempts to fix this by rstarting the run or restarting scaler monitor did not help 
<li> when we did: stop the run, stop scaler monitor, restart scaler monitor, restart the run, then it became OK 
</ul></td></tr></td></table>
<!-- Author: Stefano -->
<!-- Comment: Wed Apr 20 20;49;10 comment by...W.Badgett -->
<P>
<TABLE BGCOLOR=RED>
<TR><TD>
<FONT COLOR=YELLOW>
ScalerMonitor has no control over the L2 real-time <BR>
rate limited triggers, and was not at fault here;  see <BR>
my entry below for more info.<BR>
</FONT></TD></TR></TABLE> 



<!-- Comment: Wed Apr 20 20;59;28 comment by...Stefano -->
<pre>List of "rate limited" triggers and rates from the first run with troubles (196799). Run 196798 was fine and was ended because of VME troubles. From RUn Sumamry DB
L2_RL0.5_L1_TOF_14_MIP_&_CLC [1]  88.57
L2_RL1.3HZ_L1_MB_CLC [2]         116.97
L2_RL1HZ_L1_MB_CLC [4]           116.97
L2_RL2HZ_L1_BMU10_BSU_PT11 [1]     8.46
L2_RL2HZ_L1_TRK10 [1]             13.05
L2_INCLUSIVE_NOTRACK_1HZ [1]     6,660.58
L2_INCLUSIVE_TRACK_1HZ [1]       6,432.86

I do not know why for latter two the table says 6KHz, but the plot 60 Hz. This factor 100 discrepancy between rate in the 3rd column of web page and plt is there also for all the other triggers in this list and apparently for all L2 triggers there (at least the five or six ones I looked at). In the previous (good) run 196798 there is no factor 100 discrepancy.</pre>
<!-- Comment: Wed Apr 20 21;15;30 comment by...S -->
Also in the next run  196800 there is not this factor 100 between web page and plots. In run 196800 the L2_INLCLUSIVE_NOTRACK_1HZ trigger runs at 106Hz, the other RL triggers run not as high as before as well.
<!-- Date: Wed Apr 20 20:36:25.218 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-6678&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-6678  width='300' align='top'></a></td><td>the worst offending L2 trigger in our time of troubles</td></tr></table>
<!-- Author: Stefano -->


<!-- Date: Wed Apr 20 20:43:41.652 2005 -->
Kirsten calls back and talks to Ted Liu about this. Bill Badget also called in 10 min ago, was told what we knw and could not come up with an explanation. Bill is looking at logs from home, Ted and Vadim are doing it from here.
<!-- Author: Stefano -->



<!-- Date: Wed Apr 20 20:45:52.602 2005 -->
<pre>From the RunSummary page, ScalerMonitor did not change  
the prescale of Level 2 trigger L2_INCLUSIVE_NOTRACK_1HZ_v1  
(bit 52) at all during run 196800, but there were several  
changes done by hand.     Note that the L2 rate limited  
triggers always have a very large prescale number  
specified in the trigger table, which in reality represents  
the maximum allowed rate of that trigger.   The mapping  
from the rate limit to the prescale value is: 
 
   int prescale = (int) (1./rate_limit/(2.E-9)); 
 
where "rate_limit" is in Hz and "prescale" is the value  
downloaded to the L2 system, and specified in the prescale  
changer window.   So you must use caution  
when changing one of these rate limited triggers! 
 
The ScalerMonitor stopping and restarting most likely  
had no effect on the rate limited triggers;  ScalerMonitor  
never alters these triggers at all.   Restarting  
the run presumably reset something, probably in the  
pulsar system, which allowed the rate limited triggers  
to work again.   Perhaps some internal clock  
resynchronization was done during the run?  Or a resynch  
cleared the problem at Config transition? 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800' target=_top>196800</a>) -->
<!-- Comment: Wed Apr 20 21;06;37 comment by...Stefano -->
Yes in run 196800 we changed by hand prescales for L2_INCLUSIVE_NOTRACK_1HZ_v1 a few times trying to make it stop from hurting us.
<!-- Comment: Wed Apr 20 21;54;46 comment by...Tom Wright -->
One other piece of information is that L2 triggers with fixed prescales behaved as expected, so it doesn't look like there was a problem initializing the prescale array in the control node.

<!-- Comment: Wed Apr 20 22;04;58 comment by...Vadim, Burkard and Ted -->
<pre>We have spent a few hours looking into the problem. we looked at the log file from the L2 control node as well as the decision node, and couldn't see anything wrong there. All L2 code libraries were loaded properly for the runs which had trouble.

We also checked the decision node to make sure it got the correct prescales at the beginning of run (including rate limited ones which is 0.5B) and also checked that the prescale message for the bad run was identical to the same message for the current good run. 
 
do not quite understand what you mean by "internal clock resynchoronization". The only clock used for the rate limited triggers is the decision CPU clock. One could imagine that the CPU clock somehow decided to go crazy (has to slow down by A HUGE factor) and caused the problem. But that doesn't seem to explain why in Run 196800 only some rate limited triggers are bad and others are ok, and the overall L2 latency looks reasonable.

Here is a summary:

(1) Run196798 was fine. the run ended normally;

(2) Run196799 when problem started: all L2 rate limited triggers were free running;

(3) Started a new run 196800: still bad for some rate limited triggers, but some are ok. for example, L2_Inclusive_Track_1Hz, L2_RL1.3HZ_L1_MB_CLC etc are ok.

(4) then ended the run, and killed ScalerMon and restarted it,
and started a new run196801 again. this time everything is ok and it has been running for a while.


Anyway, this is very strange! the fact that killing and restarting the ScalerMon made things working could be just a coincidence...  </pre>
<!-- Comment: Thu Apr 21 01;12;18 comment by...W.Badgett -->
Yes, it was just a co&iuml;ncidence...
<!-- Date: Wed Apr 20 21:04:28.379 2005 -->
Time out in b0cot09, run halted and recovered.  
<!-- Author: Jiyeon & Sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801' target=_top>196801</a>) -->




<!-- Date: Wed Apr 20 21:16:48.378 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6679&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6679' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6680&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6680' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6681&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6681' width='300'></a></center>
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 21;17;08 comment by...Daniel -->
Hourly plots 20:15-21:15


<!-- Date: Wed Apr 20 21:26:38.764 2005 -->
I accidentally killed the ICICLE PC.  Oops.  Restarted fine.
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 21;47;39 comment by...Daniel -->
Heartbeat alarm in Icicle now.  Restarted it.
<!-- Comment: Wed Apr 20 22;08;07 comment by...Daniel -->
Alarm cleared.
<!-- Date: Wed Apr 20 21:34:56.996 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798">196798</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L2T 
SVX 
ISL 
L00 
SVT 
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Wed Apr 20 21:38:49.857 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196797">196797</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVT 
</PRE>
<!-- Author: cdfscico -->




<!-- Date: Wed Apr 20 21:50:34.289 2005 -->
D0 roman pots go in
<!-- Author: Stefano :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801' target=_top>196801</a>) -->





<!-- Date: Wed Apr 20 22:17:28.255 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6682&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6682' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6683&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6683' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6684&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6684' width='300'></a></center>
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 22;17;56 comment by...Daniel -->
Hourly plots, 21:15-22:15

<!-- Date: Wed Apr 20 22:23:09.305 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800">196800</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVX 
ISL 
L00 
SVT 
</PRE>
<!-- Author: cdfscico -->




<!-- Date: Wed Apr 20 22:39:36.710 2005 -->
Done timeout in b0cot09, HRR solved the problem
<!-- Author: Jiyeon & Sofia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801' target=_top>196801</a>) -->


<!-- Date: Wed Apr 20 23:21:08.974 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6685&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6685' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6686&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6686' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-6687&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-6687' width='300'></a></center>
<!-- Author: Daniel -->
<!-- Comment: Wed Apr 20 23;21;37 comment by...Daniel -->
Hourly plots, 22:15-23:15



<!-- Date: Wed Apr 20 23:48:31.749 2005 -->
<pre>MCR plan: 
 
We hope to get back in the pbar-making business sometime tonight. If the store falls out, fix and recover the Tevatron and turn it over to studiers. - J. Morgan  </pre>
<!-- Author: Stefano -->

<!-- Date: Wed Apr 20 23:56:01.102 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196798>
196798</A> 
x300BE</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_02 [1,639,563]</TD>                                       
<TD ALIGN=CENTER>14:39:28</TD>                                          
<TD ALIGN=CENTER>17:42:55</TD>                                          
<TD ALIGN=CENTER>02:56:05</TD>                                          
<TD ALIGN=RIGHT>    214,531,746</TD>                                    
<TD ALIGN=RIGHT>     1,873,362</TD>                                     
<TD ALIGN=RIGHT>     488,469</TD>                                       
<TD ALIGN=RIGHT>  654.752</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196799>
196799</A> 
x300BF</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_02 [1,639,563]</TD>                                       
<TD ALIGN=CENTER>18:01:27</TD>                                          
<TD ALIGN=CENTER>18:41:28</TD>                                          
<TD ALIGN=CENTER>00:00:17</TD>                                          
<TD ALIGN=RIGHT>        208,877</TD>                                    
<TD ALIGN=RIGHT>       208,859</TD>                                     
<TD ALIGN=RIGHT>     208,848</TD>                                       
<TD ALIGN=RIGHT>    0.906</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>0</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196800>
196800</A> 
x300C0</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_02 [1,639,563]</TD>                                       
<TD ALIGN=CENTER>18:48:56</TD>                                          
<TD ALIGN=CENTER>20:02:47</TD>                                          
<TD ALIGN=CENTER>00:26:46</TD>                                          
<TD ALIGN=RIGHT>     28,989,062</TD>                                    
<TD ALIGN=RIGHT>       734,195</TD>                                     
<TD ALIGN=RIGHT>     454,169</TD>                                       
<TD ALIGN=RIGHT>   79.031</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196801>
196801</A> 
x300C1</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_02 [1,639,563]</TD>                                       
<TD ALIGN=CENTER>20:09:49</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>03:37:12</TD>                                          
<TD ALIGN=RIGHT>    331,377,238</TD>                                    
<TD ALIGN=RIGHT>     2,653,780</TD>                                     
<TD ALIGN=RIGHT>     510,800</TD>                                       
<TD ALIGN=RIGHT>  556.740</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:03</TD>                                                       
<TD ALIGN=CENTER>07:00:22</TD>                                          
<TD ALIGN=RIGHT>     575,106,923</TD>                                   
<TD ALIGN=RIGHT>      5,470,196</TD>                                    
<TD ALIGN=RIGHT>     1,662,286</TD>                                     
<TD ALIGN=RIGHT> 1291.429</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Apr 21 00:03:55.820 2005 -->
<font color=red>Shift Summary: </font><pre>Evening shift summary 
 
Inherit from day: run 196798 Store 4105 Lum 6.1E30 
Start running w/o silicon and w/o two L3 sub-farms (4 and 9) 
 
Evenutally silicon is declared off until an acces (air bubbles in 
cooling liquid pipes). 
 
When starting a new run to reinsert L3 subfarms we had Rate Limits 
not being applied correctely anymore, L2 running beserk in the 
KHz range, L3 and CSL choking and 99.9% deadtime, and it took 
all experts we could rally (7 probably), 2 hours, and some combinatin 
of luck and serendipity to get out of that and into business. 
Eventually exact reason was not found, although the guilty party 
seems to be L2 decision P2 who got confused about how to impose 
rate limites. Wethet it is an internal problem or an effect of 
external causes, or something else, we do not know. 
 
NOTE: if this happens again, try to get deadtime under control 
by forcing rate-limited triggers to prescale 0, and call L2 pager 
so that they can look at it. 
 
Alarm panel is displaying wrong information, ignore it. Mark Knapp 
knows and will be working to fix it. 
 
Sci.Co.'s please note that runs w/o silicon and w/o SVT do not 
have to be marked bad for L2 because of that. Bad silicon + bad SVT 
is enough. Unfortunately CO instructions do not say this clearly. 
 
Stack i very low and staking is halted due to repair in debuncher, 
no schedule for new store until tomorrow. 
 
Runs: 196798, 196799, 196800, 96801 
Delivered luminosity:	1369.8 nb-1 
Acquired luminosity:	966.3 nb-1 
Efficiency:	70.5</pre> 
<!-- Author: Stefano -->


<!-- Date: Thu Apr 21 00:09:40.279 2005 -->
<pre>Trigger inhibit: from CPR, CCR, CLC, RP, MP, BSC. In HV summary SAR of these systems</pre>
<!-- Author: Cristobal :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196901' target=_top>196901</a>) -->