<!-- Title: Eve shift. Thu Jan 22, 2004 -->
<!-- SciCo: Cigdem Issever -->
<!-- DAQAce: Irina Shreyber, Jan Ehlers  -->
<!-- MonAce: Anadi Canepa, Jeffrey Miles -->
<!-- CO: Stanislav Tokar -->
<!-- OpManager: Mary Convery -->
<!-- Notes: -->
<pre>Taking Data,  
Store 3185 
TeVPR 710 
TeVPB  63 
B0LUM 18.00 E30
 
AbortGap Losses: 9320Hz</pre>


<!-- Date: Thu Jan 22 16:06:04.375 2004 -->
<pre>Restarted SpyAlpha after small update... 
</pre>
<!-- Author: W.Badgett -->
<!-- Comment: Thu Jan 22 16;08;44 comment by...WB -->
...please don't start minicom while SpyAlpha is running



<!-- Comment: Thu Jan 22 16;35;33 comment by...Irina -->
<pre>Is it normal that SpyAlpha is disabled(grey) on ProcMon?</pre>
<!-- Comment: Thu Jan 22 22;15;25 comment by...W.Badgett -->
<pre>No, that's not normal while SpyAlpha is running -- probably 
was disabled while minicom was running in lieu of SpyAlpha 
earlier today.

SpyAlpha has been running for six hours without problem, so 
it should be safe to re-enable the ProcMon button.
</pre>
<!-- Date: Thu Jan 22 16:13:27.540 2004 -->
We unmarked both AVDD and DVDD B2W2L4 (there was a spike in DVDD current: 247.6 and a drop in AVDD current: 1006.9)  
<!-- Author: Anadi & Eli -->


<!-- Date: Thu Jan 22 16:19:35.912 2004 -->
<pre>BUSY TIMEOUT from crate B0SVX06</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305' target=_top>178305</a>) -->



<!-- Date: Thu Jan 22 17:02:42.360 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1686&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1686' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1687&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1687' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1688&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1688' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1689&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1689' width='300'></a></center>
<!-- Author: anadi & jeff -->
<!-- Comment: Thu Jan 22 17;04;25 comment by...anadi & jeff -->
hourly plots: 15:00 to 16:50. Drop in proton abort gap losses are due to insertion of collimator at E0


<!-- Comment: Thu Jan 22 17;19;05 comment by...jeff -->
for clarification:  the big collimator tuning happened today at noon;  a plot by R. Tesarek shows that the E0 collimator (T:E03UV) was moved in an additional ~0.04 inches at 16:30 (as compared to the 0.5 inches at noon)
<!-- Date: Thu Jan 22 17:06:26.411 2004 -->
<pre>HIGH DEADTIME -> close to 100% due to L@ trigger -> HRR solved problem</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305' target=_top>178305</a>) -->
<!-- Comment: Thu Jan 22 17;40;21 comment by...I -->
<pre>L@ means L2</pre>


<!-- Date: Thu Jan 22 17:45:40.998 2004 -->
<pre>RXPT error for crate B0SVX06 -> HHR -> solved</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305' target=_top>178305</a>) -->


<!-- Date: Thu Jan 22 17:55:57.667 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1691&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1691  width='300' align='top'></a></td><td>Run 178305: New red channels in SVX L3.</td></tr></table>
<!-- Author: Stano -->


<!-- Date: Thu Jan 22 18:09:50.977 2004 -->
again RXPT error for crate B0SVX06 ->HRR -> solved
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305' target=_top>178305</a>) -->


<!-- Date: Thu Jan 22 18:10:29.366 2004 -->
It looks that the problem with the SVX L3 is not a serious one - the red ladders disappear from time to time.
<!-- Author: Stano :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305' target=_top>178305</a>) -->


<!-- Date: Thu Jan 22 18:19:07.615 2004 -->
<table border=0><td><tr><td bgcolor=#e0e030>Cyro Technician, Jim called from downstairs.  
He found that rack 1RR 21I has a  single fan failure, the fault indicated is a bad electrical connection. It is the upper fan bag. He had a look, but did not see anything obvious and does not 
want to touch the rack, while we are running.  
 
After the store is dumped next Cyro Technician or Jim will have a closer look at the rack. Please pass this information to the next 
shift crews. </td></tr></td></table>
<!-- Author: cigdem -->
<!-- Comment: Thu Jan 22 19;23;53 comment by...Jeff -->
From talking to Jim: This is a single fan which has just slowed down below some threshold (not even stopped) and is one of ~18 individual fans, the rest of which are running fine.  This rack contains EVB crate 24 and is very well air-conditioned -- there is no chance that this could possibly affect the crate performance during this store.  Once the store is dumped, the Cryo Tech on shift will examine the fan unit for a bad connection.
<!-- Comment: Thu Jan 22 19;45;15 comment by...Jeff -->
...and if one were really concerned about the rack temperature, iFix monitors the temperatures for each rack on the page in Process Systems: RACKS --> First Floor Racks
<!-- Comment: Thu Jan 22 21;15;05 comment by...rainer -->
b0evb24 = b0svx07.
<!-- Date: Thu Jan 22 18:22:12.497 2004 -->
<pre> Attention!!!. SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb14 (forwarded by FER crate CMP_00).  
 Datataking is paused: events are not being processed.  
 - HRR should help. 
 
HELPED ;-)</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305' target=_top>178305</a>) -->


<!-- Date: Thu Jan 22 18:26:08.226 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1692&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1692' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1693&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1693' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1694&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1694' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1695&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1695' width='300'></a></center>
<!-- Author: Anadi & Jeff -->
<!-- Comment: Thu Jan 22 18;26;54 comment by...j & a -->
Hourly Plots, 16:50 - 18:00


<!-- Date: Thu Jan 22 18:51:32.892 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1696&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1696  width='300' align='top'></a></td><td>History plot for B2W2L4</td></tr></table>
<!-- Author: I -->
<!-- Comment: Thu Jan 22 18;57;38 comment by...Rainer -->
SB2W2L4 (a ladder which broke after the shutdown we thought we had salvaged) refuses to recover from trip. I dropped the ladder from the hardware database for this run. Needs quiet time investigation what's wrong with it.
<!-- Comment: Thu Jan 22 19;02;06 comment by...cigdem  -->
<pre>The whole wedge tripped and as we tried to turned them on again, after resetting the trips, we couldn't and PS Gui was complaining, that the handshake was missing. 

We paged Silicon (Stan) and I got impatient and paged after 1 or 2min also Rainer. Stan is coming in and Rainer worked with the 
Aces on the phone and took layer 4 out hardware db. We are taking data again. </pre>
<!-- Date: Thu Jan 22 18:53:02.483 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305">178305</A>
 Terminated at 2004.01.22 18:51:59
<!-- Author: RunControl -->


<!-- Date: Thu Jan 22 18:53:03.093 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305">178305</A>
TERMINATE: end run due to Si trip
<!-- Author: Jan/Irina x2080 -->


<!-- Date: Thu Jan 22 18:57:21.909 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307">178307</A>
 Activated at 2004.01.22 18:57:02
<!-- Author: RunControl -->



<!-- Date: Thu Jan 22 18:58:30.304 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307">178307</A>
ACTIVATE: same run conditions as before - PHYSICS_2_01 [3,387,419]
<!-- Author: Jan/Irina x2080 -->




<!-- Date: Thu Jan 22 19:12:40.428 2004 -->
<pre>DECISION TIMEOUT: 
 
(MLE) b0dap10.fnal.gov:SpyAlpha:6:59:24 PM->TIMEOUT: Mod done not set for slot 12 (SVTList)(0x0007fdff, cnt 2) 
(MLE) b0dap10.fnal.gov:SpyAlpha:7:00:00 PM->TIMEOUT: Mod done not set for slot 12 (SVTList)(0x0007fdff, cnt 3) 
(MLE) b0dap73.fnal.gov:Thread-30:6:59:51 PM->  Level 2 Decision Timeout 
 
MLE) b0l2de00:Messenger:6:59:57 PM->Slot12: ModDone error, event 15511 
(MLE) b0l2de00:Messenger:7:00:05 PM->Slot12: ModDone error, event 15511 
(MLE) b0l2de00:Messenger:7:00:13 PM->Slot12: ModDone error, event 15511 
(RC)   19:0:22 Halt -> HALTED 
(RC)   19:0:28 Recover -> RECOVERED 
(RC)   19:0:31 Run -> ACTIVE</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307' target=_top>178307</a>) -->
<!-- Comment: Thu Jan 22 19;14;34 comment by...Irina & Jan -->
AUTOMATIC HRR DIDN'T WORK! -> BY HAND -> FINE
Alredy second time that automatic HRR doesn't work. Same incident already in last run!


<!-- Comment: Thu Jan 22 19;25;26 comment by...cigdem -->
<table border=0><td><tr><td bgcolor=#e0e030>trying to get hold of DAQ Expert-Frank, because HRR is not working.
      Frank said, we do have to restart the error logger if we have a chance. I am worried that silicon is unprotected, since I don't know if the Aces will get informed when Silicon is in a high occupancy state. Or if the only way we can tell is the SVXMON red ladder status.  </td></tr></td></table>
<!-- Comment: Thu Jan 22 19;33;43 comment by...cigdem  -->
<pre>Stan told us, that we should get an error window telling us 
that the silicon is in a high occupancy status asking us to do a manual HRR. </pre>
<!-- Comment: Thu Jan 22 19;40;50 comment by...Donatella T. -->
it is not necessary to re-start Error logger. Clicking on <Clear> button at the top of the GUI  resets error conditions and usually gets things going again. I'm looking at the error handler session logfiles to see why it did not issue the auto-HRR, if this is the case.
<!-- Comment: Thu Jan 22 19;43;38 comment by...Donatella T. -->
oops the last comment should read Click on the Clear button at the top of the EH GUI...
<!-- Comment: Thu Jan 22 19;55;46 comment by...Donatella T. -->
<table border=0><td><tr><td bgcolor=#e0e030>the reason why the Auto-HRR was not issued is that Error Handler expects an RXPT message (in this case L2 Decision Timeout) to issue the Auto-HRR command. This RXPT message is missing here and in run 178306 as well, therefore no Auto-HRR was issued. Did something change in the Return Cross Point ? </td></tr></td></table>
<!-- Comment: Thu Jan 22 20;06;46 comment by...cigdem -->
<pre>I contacted Frank and he tells me that according to his knowledge, return cross point did not change. To be sure, 
he will write Bill. Meanwhile, if we do have to stop the 
run, we shall restart the RC.</pre>
<!-- Date: Thu Jan 22 19:21:56.425 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1701&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1701' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1702&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1702' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1703&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1703' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1704&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1704' width='300'></a></center>
<!-- Author: Anadi and Jeff -->
<!-- Comment: Thu Jan 22 19;22;27 comment by...Anadi and Jeff -->
Hourly plots: 18:00-19:00










<!-- Date: Thu Jan 22 20:09:30.480 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1712&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1712' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1713&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1713' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1714&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1714' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1715&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1715' width='300'></a></center>
<!-- Author: Anadi & Jeff -->
<!-- Comment: Thu Jan 22 20;10;01 comment by...Anadi & Jeff -->
Hourly plots: 19:00 - 20:00 


<!-- Date: Thu Jan 22 20:25:21.436 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1716&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1716  width='300' align='top'></a></td><td>Run 178307: SVX Chip Status Map: yellow bars in L4 are known problem. Silicon Experts had to take out e224.  
This ladder has to be marked in SVXMON.</td></tr></table>
<!-- Author: Cigdem & Stano -->
<!-- Comment: Thu Jan 22 21;33;27 comment by...Rainer -->
next restart of SVXMon will do this for you.
<!-- Date: Thu Jan 22 20:27:21.700 2004 -->
<pre>Scaler Monitor Error: 
HIGH DEADTIME (L2) -> HRR -> Fine</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307' target=_top>178307</a>) -->

<!-- Date: Thu Jan 22 20:32:38.635 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307">178307</A>
 Terminated at 2004.01.22 20:32:18
<!-- Author: RunControl -->


<!-- Date: Thu Jan 22 20:33:16.899 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307">178307</A>
TERMINATE: abort run since HALT transition took over one minute!
<!-- Author: Jan/Irina x2080 -->


<!-- Date: Thu Jan 22 20:44:17.079 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178309">178309</A>
TERMINATE: couldn't config because of svx04
<!-- Author: Irina & Jan x2080 -->
<!-- Comment: Thu Jan 22 21;29;41 comment by...rainer -->
turned red in vxworks and yellow.
<!-- Comment: Thu Jan 22 21;40;09 comment by...rainer -->
another <b> suggestion for spare time reading  </b> for these long and boring owl shifts to be found 
<a href='http://www-cdf.fnal.gov/internal/silicon/silicon_commission/daq/sidaq_sop.html#sidaq_noMess'>  here </a>
<!-- Date: Thu Jan 22 20:47:12.655 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310">178310</A>
 Activated at 2004.01.22 20:46:53
<!-- Author: RunControl -->


<!-- Date: Thu Jan 22 20:47:58.323 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310">178310</A>
ACTIVATE: aaa_current , tr.tablephysics_2_01[3,387,419]
<!-- Author: Irina & Jan x2080 -->



<!-- Date: Thu Jan 22 20:51:28.085 2004 -->
<pre>AGAIN!!! 
L2 Timeout without pop-up window!!! 
Automatic HRR doesn't work!! 
HRR manually -> get stuck in HALT transition 
-> ABORT run 
Frank recommended to restart run contron 
-> start run -> SVX04 coundn't configure (errorlog: Statechange failed) -> abort & restart -> FINE ;-) 
</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310' target=_top>178310</a>) -->
<!-- Comment: Thu Jan 22 21;09;50 comment by...Irina & Jan -->
<pre>MLE) b0svx06:Messenger:8:43:21 PM->Silicon Timeout:BUSY- Slots:  20:f840

(MLE) b0svx06:Messenger:8:43:21 PM->Silicon Timeout:!DONE- Slots:  20:f840
(MLE) b0svx02:Messenger:8:43:21 PM->Silicon Timeout:BUSY- Slots:  16:e260 18:e280 20:e2a0

(MLE) b0svx02:Messenger:8:43:21 PM->Silicon Timeout:!DONE- Slots:  16:e260 18:e280 20:e2a0
(MLE) b0svx04:Messenger:8:43:21 PM->Statechange failed
(RC)   20:43:41 Abort -> IDLE
(RC)   20:46:56 Config -> READY
(RC)   20:47:9 Activate -> ACTIVE</pre>
<!-- Comment: Thu Jan 22 21;25;40 comment by...Rainer -->
this is <b> VERY ODD </b> but since there are no errorlog files or at least I can't find them, it's hard to diagnose ...
<!-- Comment: Thu Jan 22 21;42;57 comment by...Rainer -->
More of that quiet shift type reading  <a href='http://www-cdf.fnal.gov/internal/silicon/silicon_commission/daq/sidaq_sop.html#sidaq_failedVrbInit'> here </a> 
<!-- Date: Thu Jan 22 20:58:01.806 2004 -->
<pre>L2 Decision Timeout ERROR! (error with pop-up window this time) 
-> HRR -> fine</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310' target=_top>178310</a>) -->
<!-- Comment: Thu Jan 22 21;23;18 comment by...Irina/Jan -->
<pre>HRR means auto-HRR. After restarting RC , auto-HRR works. </pre>




<!-- Comment: Thu Jan 22 22;02;00 comment by...Frank Chlebana -->
<pre>Looks like something on dap58 ran amok.

Was connecting to the rtserver causing us to go over the license
limit - leading to crates losing connection.

Had to reboot dap58 - after that no longer saw 

  b0dap58.fnal.gov:38712  FIN_WAIT1   

messages on dap33

</pre>
<!-- Comment: Thu Jan 22 22;15;21 comment by...Jeff, Irina, Jan, Anadi -->
To flesh out the situation with a more new-ace-friendly description:<br>
When the rtserver connection license limit is hit, the various processes which communicate through it are suddenly blind/deaf, and practically every monitoring GUI goes all red, e.g. VxWorks Mon, DAQ Mon, the L3 state.  One should immediately log in to b0dap33, as per the instructions from Frank pasted between the monitors for b0dap59 at the DAQ station, and look at the results of <br>
`netstat | grep FIN`.<br>
This reveals connections which are persistently hogging rtserver licenses.  It this case, as Frank writes, something went crazy on b0dap58 (the rightmost station in the DAQ area) and it had to be rebooted.  As soon as that was done, all these FIN_WAIT1 connections went away, and the monitoring processes displayed green again.  We also restarted the rtserver.x process on b0dap33, following the rtserver help page instructions, although this was probably unnecessary.<br>
Once b0dap58 came back up again and the VxWorks Mon was restarted, we see that everything is fine again.<br>
The run continued undisturbed throughout this procedure.
<!-- Date: Thu Jan 22 21:32:16.787 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1717&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1717' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1718&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1718' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1719&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1719' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1720&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1720' width='300'></a></center>
<!-- Author: anadi & jeff -->
<!-- Comment: Thu Jan 22 21;32;51 comment by...Anadi & Jeff -->
Hourly Plots, 20:00 - 21:00 






<!-- Date: Thu Jan 22 22:04:08.246 2004 -->
<pre>While we delt with the rtserver/dap58 problem the run was  
still going. Looks like communication is restored.</pre>
<!-- Author: Frank Chlebana -->


<!-- Date: Thu Jan 22 22:10:48.664 2004 -->
<pre>From the runControl log file, it looks like the RunControlStatus 
publish thread crashed 12:56 today with an ArrayIndexOutOf 
BoundsException somewhere down in a low-level smart sockets 
routine.    So, ErrorLogger probably did not know that RunControl 
was ACTIVE at the time of the L2 timeouts, and hence did not 
issue an auto-HRR. 
 
Anyhow, I've put in a protection against this in the devel 
version of fer, which will be current at some point soon... 
</pre>
<!-- Author: W.Badgett -->




<!-- Date: Thu Jan 22 22:16:30.096 2004 -->
<pre>SpyAlpha has been running more than 6 hours  
without a problem, so I'd suggest enabling the  
SpyAlpha button in ProcMon.   (also see note early in shift) 
</pre>
<!-- Author: W.Badgett -->


<!-- Date: Thu Jan 22 22:26:47.997 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1721&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1721' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1722&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1722' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1723&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1723' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1724&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1724' width='300'></a></center>
<!-- Author: Anadi & Jeff  -->
<!-- Comment: Thu Jan 22 22;27;36 comment by...Anadi & Jeff  -->
Hourly plots: 21:00 - 22:00.
Drop in the abort gap losses


<!-- Date: Thu Jan 22 22:34:01.088 2004 -->
<pre>Online DB cdfonprd was red on DAQ Monitor. Jeff  stopped it by using ResMGr Proxy viewer (I was looking if he was doing right) and it restarted by itself (I was looking on it too). It's green now. We did it without paging DAQ expert!:) 
</pre>
<!-- Author: Irina & Jeff :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310' target=_top>178310</a>) -->
<!-- Comment: Thu Jan 22 22;37;59 comment by...Jeff -->
but it was Irina's dazzling display of e-log search wizardry that discovered the <a href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=copyentry&page=738&time=18:58:34.233">entry</a> which contains the information about how to do it :)


<!-- Comment: Thu Jan 22 23;12;01 comment by...JJ -->
<pre>You become dazzling ACEs and then you leave... 
sniff... gonna miss you guys!</pre>
<!-- Date: Thu Jan 22 22:43:00.951 2004 -->
<table border=0><td><tr><td bgcolor=#e0e030>Dan Cyr, BMU and TDC expert called back. He is in the area again,  
and we shall page him from now onwards, instead calling on his cell phone. </td></tr></td></table>
<!-- Author: cigdem  -->



<!-- Date: Thu Jan 22 23:14:33.136 2004 -->
<pre>communication test between SRC and RTServer: 
HRR request by Rainer approved by the ace crew & SciCo 
 
-> test successful</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310' target=_top>178310</a>) -->
<!-- Comment: Thu Jan 22 23;22;27 comment by...Rainer -->
we are now monitoring the # of L1A seen by the SRC and by the L1 scalers. so far all L1As accounted for! <p>
NB: unstable RTserver will screw this experiment up ....

<!-- Date: Thu Jan 22 23:15:19.632 2004 -->
<pre>busy timeout for crate B0SVX06</pre>
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310' target=_top>178310</a>) -->



<!-- Date: Thu Jan 22 23:26:31.624 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1725&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1725' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1726&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1726' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1727&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1727' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1728&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1728' width='300'></a></center>
<!-- Author: Anadi & Jeff -->
<!-- Comment: Thu Jan 22 23;27;15 comment by...Anadi & Jeff  -->
Hourly Plots: 22:00 - 23:00


<!-- Date: Thu Jan 22 23:46:14.795 2004 -->
b0fcal00 shepherded since vxmon indicated an error 
<!-- Author: Irina & Jan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310' target=_top>178310</a>) -->

<!-- Date: Thu Jan 22 23:55:52.655 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178305>178305</A> 
x2B881</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,387,419]</TD>                                       
<TD ALIGN=CENTER>14:13:18</TD>                                          
<TD ALIGN=CENTER>18:51:59</TD>                                          
<TD ALIGN=CENTER>04:11:29</TD>                                          
<TD ALIGN=RIGHT>    210,204,247</TD>                                    
<TD ALIGN=RIGHT>     3,243,198</TD>                                     
<TD ALIGN=RIGHT>     749,903</TD>                                       
<TD ALIGN=RIGHT>  284.169</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178307>178307</A> 
x2B883</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,387,419]</TD>                                       
<TD ALIGN=CENTER>18:57:02</TD>                                          
<TD ALIGN=CENTER>20:32:18</TD>                                          
<TD ALIGN=CENTER>01:27:31</TD>                                          
<TD ALIGN=RIGHT>     58,711,769</TD>                                    
<TD ALIGN=RIGHT>       925,486</TD>                                     
<TD ALIGN=RIGHT>     221,574</TD>                                       
<TD ALIGN=RIGHT>   85.209</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178310>178310</A> 
x2B886</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,387,419]</TD>                                       
<TD ALIGN=CENTER>20:46:53</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>03:04:57</TD>                                          
<TD ALIGN=RIGHT>    116,029,091</TD>                                    
<TD ALIGN=RIGHT>     1,810,215</TD>                                     
<TD ALIGN=RIGHT>     438,152</TD>                                       
<TD ALIGN=RIGHT>  161.310</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:02</TD>                                                       
<TD ALIGN=CENTER>08:43:59</TD>                                          
<TD ALIGN=RIGHT>     384,945,107</TD>                                   
<TD ALIGN=RIGHT>      5,978,899</TD>                                    
<TD ALIGN=RIGHT>     1,409,629</TD>                                     
<TD ALIGN=RIGHT>  530.688</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Jan 22 23:56:56.768 2004 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Summary of the evening shift 22 Jan 2004 
======================================== 
 
 
16:00 Start shift, taking data.  
 
Store 3185  
TeVPR 710  
TeVPB  63  
B0LUM 18.00 E30 
 
AbortGap Losses: 9320Hz 
 
 
17:04 High dead time alarm. 
       
18:20 Jim Cyro tech, found rack 1RR 21I single fan failure 
      bad electrical connection 
      upper fan bag  
      nothing obvious  
      Will investigate problem when we don't have store in. 
      Please pass this information to the next shifts.  
 
18:38 Silicon tripped. Can't turn on ladder..Handshake problem.  
 
19:06 trying to get hold of DAQ Expert-Frank, because HRR is not working. 
      Frank said, we do have to restart the error logger if we have a chance.  
      I am worried that silicon is unprotected, since I don't know if the  
      Aces will get informed when Silicon is in a high occupancy state.  
      Or if the only way we can tell is the SVXMON red ladder status. 
 
19:45 Donatella called. She restarted the errorlogger. It is not needed to  
      stop the run to do it:  Click on the Clear button at the top of  
      the EH GUI.  
 
20:04 Donatelle called. the reason why the Auto-HRR was not issued is that  
      Error Handler expects an RXPT message (in this case L2 Decision Timeout) 
      to issue the Auto-HRR command. This RXPT message is missing here and in run 178306 as well, therefore no Auto-HRR was issued.  
      Did something change ii the Return Cross Point? 
      I contacted Frank and he tells me that according to his knowledge, return cross point did not change. To be sure, he will write Bill. Meanwhile, if we do have to stop the run and shall restart the RC.  
 
20:47 We had to abort run, halt transition got stuck for several minutes. We than restarted the RC as Frank told us to do.  
 
20:58 We do have auto HRR working again.  
 
21:40 We have problems with RTserver. Instructions did not work.  
      Called DAQ expert. 
  
22:41 Dan Cyr, BMU and TDC expert called back. He is in the area again, and we shall page him from now onwards, instead calling on his cell phone.  
 
Delivered luminosity: 459.30 nb-1 
Acquired luminosity: 406.00 nb-1 
Efficiency: 88.40 
 
Plan is to end the store at 7am. Please inform the Cyro Techinician when store ends, such that he can check the fan failure 
of rack IRR 21I.</pre></td></tr></td></table>
<!-- Author: cigdem -->


<!-- Date: Thu Jan 22 23:58:44.270 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1729&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1729' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1730&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1730' width='300'></a></center>
<!-- Author: Anadi & Jeff  -->
<!-- Comment: Thu Jan 22 23;59;31 comment by...Anadi & Jeff -->
Integrated radiation and shift luminosity