<!-- Title: Eve shift. Mon Jun 26, 2006 -->
<!-- SciCo: Frank Wuerthwein (fkw) -->
<!-- DAQAce: Yanjun Tu -->
<!-- MonAce: Enrique Palencia -->
<!-- CO: Arkadios Manousakis-Katsikakis -->
<!-- OpManager: Bob Wagner -->
<!-- Notes: -->
<pre>lumi 66E30 
 
expect smooth running throughout shift. 
</pre>

<!-- Date: Mon Jun 26 16:07:38.310 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219095 target=_runSummary>219095</A> x357D7</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,626]</FONT></TD>
<TD ALIGN=CENTER>15:24:19</TD>
<TD ALIGN=CENTER>09:26:57</TD>
<TD ALIGN=CENTER>17:02:57</TD>
<TD ALIGN=RIGHT>   3,733,682</TD>   <TD ALIGN=RIGHT>  1,356.26</TD>
<TD ALIGN=RIGHT>  1,266.83</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219108 target=_runSummary>219108</A> x357E4</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,626]</FONT></TD>
<TD ALIGN=CENTER>12:28:42</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>03:21:19</TD>
<TD ALIGN=RIGHT>   1,226,501</TD>   <TD ALIGN=RIGHT>  1,055.83</TD>
<TD ALIGN=RIGHT>    979.09</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>16:04:23</TD>
<TD ALIGN=CENTER>20:24:16</TD>
<TD ALIGN=RIGHT>     4,960,183</TD> <TD ALIGN=RIGHT>  2,412.09</TD>
<TD ALIGN=RIGHT>  2,245.92</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->
<!-- Comment: Mon Jun 26 16;17;36 comment by...badgett -->
<table border=0><td><tr><td bgcolor=#e0e030><P>
<H3>
This was a test end-of-shift report − please ignore!
</H3>
<P>
</td></tr></td></table>
<!-- Date: Mon Jun 26 16:13:18.081 2006 -->
<pre>CMX trip (wedges 18-19). I turned them on.</pre>
<!-- Author: Kike :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219108' target=_top>219108</a>) -->
<!-- Comment: Mon Jun 26 16;15;18 comment by...Kike -->
One minute later the same wedges tripped.

<!-- Comment: Mon Jun 26 16;18;48 comment by...Yanjun -->
got timeout message, recovered

Busy Timeout: CMX_01 


<!-- Comment: Mon Jun 26 16;38;28 comment by...Kike -->
It tripped three more times. I talked to the expert and we decided to turned that wedge off.
<!-- Date: Mon Jun 26 16:17:06.250 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219095 target=_runSummary>219095</A> x357D7</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,626]</FONT></TD>
<TD ALIGN=CENTER>15:24:19</TD>
<TD ALIGN=CENTER>09:26:57</TD>
<TD ALIGN=CENTER>17:02:57</TD>
<TD ALIGN=RIGHT>   3,733,682</TD>   <TD ALIGN=RIGHT>  1,356.26</TD>
<TD ALIGN=RIGHT>  1,266.83</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219108 target=_runSummary>219108</A> x357E4</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,626]</FONT></TD>
<TD ALIGN=CENTER>12:28:42</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>03:28:12</TD>
<TD ALIGN=RIGHT>   1,282,884</TD>   <TD ALIGN=RIGHT>  1,103.94</TD>
<TD ALIGN=RIGHT>  1,011.03</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>16:16:54</TD>
<TD ALIGN=CENTER>20:31:10</TD>
<TD ALIGN=RIGHT>     5,016,566</TD> <TD ALIGN=RIGHT>  2,460.20</TD>
<TD ALIGN=RIGHT>  2,277.86</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->
<!-- Comment: Mon Jun 26 16;17;59 comment by...badgett -->
<P>
<H3>This was a test end-of-store summary − please ignore!
</H3>
<p>



<!-- Date: Mon Jun 26 16:25:26.478 2006 -->
<P> 
OK, the end-of-shift automatic summary is now working  
again &minus; but you'll have to wait until the end of this  
current shift to see it work for real. 
<P> 
The summary runs in a cron script on bzora1 and had the  
oracle software version hardcoded (!);  the old version disappeared  
recently and so the script stopped working.    But it's  
back now. 
<P>  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219108' target=_top>219108</a>) -->

<!-- Date: Mon Jun 26 16:34:36.216 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219108>219108</A>
 Terminated at 2006.06.26 16:34:28
<!-- Author: RunControl -->


<!-- Date: Mon Jun 26 16:35:12.745 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219108>219108</A>
TERMINATE: end run to fix cmx problem
<!-- Author: Yanjun -->
<!-- Comment: Mon Jun 26 16;39;09 comment by...Dale -->
I've asked that (NE) East CMX 18-19 be turned off and that a new run to be started.  If you need to mask any inhibits due so (only one that might give you trouble is the PC ones - the others will be happy that the trip is cleared and the HV is off).

<!-- Comment: Mon Jun 26 17;08;46 comment by...Dale Stentz -->
<table border=0><td><tr><td bgcolor=#e0e030><h3>Instructions for ramping up CMX durning run:</h3>
<ul>
<li>You will need to mask two inhibits (if you haven't done so already).  These are<ul>
<li>CMX HV:iFix</li>
<li>CMX HV:PC</li></ul>
It is ok to mask these inhibits as, 1) the data is bad and 2) I have the "expert authority" to do so.</li>
<li>
Ramp up the CMX HV for the suspect wedges (NE 18-19)
</li>
<ul><li>
if it trips you will still get an inhibit (CMU00 West Trip)<br>
reset the trip (turn the wedge off) and wait for another chance to try the HV, try to get back to data-taking quickly.
</li>
<li>if it makes it up and holds, we still need to make sure that it is ok for the long hall.<br>
If it holds ok for ~15-30 minutes you can think to start a new run.  This is more the sci-co/ops call and depends on what else is happening at the moment.
</li></ul>
<li>
I would try ramping up the HV every hour or if there is another source of downtime (you need to end the run for something else).
</li>
</ul></td></tr></td></table>
<!-- Comment: Mon Jun 26 17;22;19 comment by...Dale -->
Also note that with the two inhibits masked you can turn the wedge to stand-by (assuming it doesn't trip before it gets to the default STB voltages).  This may help "condition" the wedge.  Feel free to try this as well but keep the wedge off if it cannot make it to stand-by.
<!-- Date: Mon Jun 26 16:39:39.314 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
 Activated at 2006.06.26 16:39:11
<!-- Author: RunControl -->


<!-- Date: Mon Jun 26 16:39:40.989 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
ACTIVATE: RunSet: AAA_CURRENT_RH  Trigger: PHYSICS_4_00 [3,798,626] 
<!-- Author: Yanjun -->
<!-- Comment: Mon Jun 26 16;40;29 comment by...Kike -->
For this run, wedges 18-19 in cmx are off.


<!-- Comment: Mon Jun 26 16;52;02 comment by...Dale -->
Aye, note that CMX will have to be marked bad for this run - it fails the HV criteria and should show up as a 15-20 degree hole in coverage.  Also let DQMon catch the problem and see that it points you to the correct problem, etc.<br><br>
The game plan (more details to follow) is to try to turn on this wedge in such a way to minimize any lost time in data-taking.  If we get the wedge back up we can think about starting a new run.
<!-- Date: Mon Jun 26 16:43:16.835 2006 -->
got error message during partition 
 
 Attention!!!. L3_RELAY_FAILURE Error !!!.  
 Level3 internal communications failure. One or more L3 nodes is likely 
 to have died. 
   Do not panic! Level3 will try to repair itself. We recommend: 
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS    
     - exclude all affected subfarms from your run configuration 
       (find out list of bad subfarms from Error Logger:  
          if None, exclude nothing!) 
     - start another run and page L3 farm pager at closest day/evening shift 
     - if >2 subfarms affected page L3 farm IMMEDIATELY 
[MLE] 
(MLE) b0eproxy02.fnal.gov:main:Jun 26, 2006 04:36:53 PM->Error on L3 node b0eproxy02.fnal.gov (partition 0) Mon Jun 26 16:36:52 2006 l3proxy  
  in End (L3ProxyPartition.cc:2234) 
  @L3_TRANS_FAIL - transition actions failed:      b0l3s08p07(Unrecognized response type for request) b0l3s08p15(Unrecognized response type for request) 
 
(MLE) b0eproxy02.fnal.gov:main:Jun 26, 2006 04:37:22 PM->Error on L3 node b0eproxy02.fnal.gov (partition 0) Mon Jun 26 16:37:21 2006 l3proxy  
  in Reset (L3ProxyPartition.cc:718) 
  @L3_INVALID_TRANS - Reset from Halted 
 
(RC)   16:37:22 Reset -> IDLE
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 16:44:55.049 2006 -->
config b0cot10 failed, shapherd success. 
  
 Attention !!!. FERML_HEAP_CORRUPT ERROR !!!  
  
 - Heap Corrupt for crate  b0cot10  
 - Please SHEPHERD crate b0cot10.  
 -->  
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 16:47:35.387 2006 -->
<p>Just an entry to note that the iFIX Workspace on VNODE1 failed 
with an inscrutable Microsoft error message (you know, the one 
with a long string of hex characters and no usable info).  On 
clicking the OK button, the iFIX Workspace window closed.  So 
far, just opening the window again (via the Start menu) has  
worked.</p> 
 
<p>JJ also noted that there was a VOI alarm; this alarm is noting 
the the Voice Scheduler, which produces the alarms, was not running.  This is very bad.  There are instructions to restart 
the Voice Scheduler posted on the lower monitor for VNODE1.  It 
is now back up and running.</p>`
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->



<!-- Date: Mon Jun 26 16:57:26.274 2006 -->
<pre>Gor RXPT error, auto HRR is issued, recovered 
 
1 crate/s: b0pcal09(43),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-42:Jun 26, 2006 04:54:28 PM-&gt;FrontEnd Crate Error Condition from: 
PCAL_09 
 
(MLW) b0control21.fnal.gov:Thread-42:Jun 26, 2006 04:54:28 PM-&gt;Received error message: errmon Not a state HaltRecoverRun 1 crate/s: b0pcal09(43),  in error. [RXPT](MLE) b0control21.fnal.gov:Thread-42:Jun 26, 2006 04:54:28 PM-&gt;Requested Halt-Recover-Run issued (errmon) 
(MLE) b0control21.fnal.gov:Thread-42:Jun 26, 2006 04:54:28 PM-&gt;FrontEnd Error: PCAL_09 </pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 17:05:56.785 2006 -->
<pre>got warning in DQMon 
CLC info 
Slides/CLAD_Occupancy_Canvas(east2) 
asked CO to take look at that plots. 
 </pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 17:06:39.339 2006 -->
Icicle heartbeat alarm. I restarted the program.
<!-- Author: Kike -->
<!-- Comment: Mon Jun 26 17;26;50 comment by...Kike -->
It is green now.
<!-- Date: Mon Jun 26 17:18:08.918 2006 -->
<pre>got warning on DQMon 
CMX 
Slide/SlideMuon(CMX_EAST) 
ask CO to take look at it</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 17;20;43 comment by...Yanjun -->
<pre>ignore this hist, because there is one wedge off</pre>




<!-- Date: Mon Jun 26 17:28:38.882 2006 -->
<pre>inhibit CMX HV:PC  
        CMX HV:IFix 
as required by CMX expert</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 17;40;06 comment by...Kike -->
<pre>This was the result of the first try to turn wdeges 18-19 on. We masked the CMX inhibits, went to standby and then to on. It was ok for a minute or so and then trey tripped, we got a cmp inhibit. We turned the wedges off and continued taking data. Will try again in one hour.</pre>
<!-- Date: Mon Jun 26 17:30:59.219 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
fwd: <tt>The known problem of empty bins(61-63) for MPA (West OUT MP Occupancy).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 17:31:00.484 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
smx: <tt>Shower Max have dead channels due to low statistics.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 17:31:01.376 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
cal: <tt>Calorimeters have dead channels due to low statistics.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 17:31:02.858 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
monitors: <tt>Low statistics for SiliMon.<br>
Hot and cold triggers.Some plots have weird shapes. Check Available XMon Triggers of Interest Pages.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 17:31:03.665 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
cal: <tt>Calorimeters have dead channels due to low statistics.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 17:31:04.408 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
monitors: <tt>Low statistics for SiliMon.<br>
Hot and cold triggers.Some plots have weird shapes. Check Available XMon Triggers of Interest Pages.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->



<!-- Date: Mon Jun 26 17:31:10.949 2006 -->
<pre>Got High deadtime warning, it was back to normal in one second   
 
Attention!!!. FERML_HIGH_DEADTIME !!!  
 - Trigger/DAQ Deadtime is too high (at 28.662128 %).  
 - Look for source of hot triggers.  
 - Contact a trigger expert for further assistance.  
</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->



<!-- Date: Mon Jun 26 17:50:20.123 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5045&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5045' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5046&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5046' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5047&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5047' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 15.00-16.00  -->


<!-- Date: Mon Jun 26 17:58:44.819 2006 -->
<pre>Heartbeat alarm for CLC, BSC, MNP and CPR. Went downstairs and restarted the HV monitoring.</pre>
<!-- Author: Kike :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 18:05:25.032 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5048&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5048' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5049&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5049' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5050&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5050' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 16.00-17.00  -->


<!-- Date: Mon Jun 26 18:20:52.824 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5051&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5051' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5052&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5052' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5053&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5053' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 17.00-18.00  -->


<!-- Date: Mon Jun 26 18:30:19.828 2006 -->
Some silicon power supplies spontaneously switched off. Expert fixed it.
<!-- Author: Kike :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 18;32;17 comment by...Ignacio & Ulrich -->
The spontaneous switch-off is a "feature" of our CAEN power supplies. We switched on the wedges again, and everything came back fine.


<!-- Date: Mon Jun 26 18:38:28.610 2006 -->
<pre>Error Detected in ProcMon 
restart TOMCAT following instructions on ACE Page</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 18;46;16 comment by...Yanjun -->
<pre>disable it </pre>

<!-- Comment: Mon Jun 26 18;46;59 comment by...Kike -->
Bill is working on that.


<!-- Date: Mon Jun 26 18:47:50.751 2006 -->
<pre>RXPT error. resonance detected, HRR works 
 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-42:Jun 26, 2006 06:45:01 PM-&gt;FrontEnd Error: VRB_SVX_02  
(MLE) b0svx08:Messenger:Jun 26, 2006 06:45:21 PM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 26, 2006 06:45:21 PM-&gt;Silicon Resonator: Sl 13 Ch 3 -&gt; fa23: Bytes: 00780 
 
b0svx02:Messenger:Jun 26, 2006 06:45:21 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected  
 --&gt;  </pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 18:51:02.609 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><P> 
I am working on the tomcat java web server, so you may  
see problems accessing runSummary information... 
<P></td></tr></td></table>
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 19:03:25.698 2006 -->
CMU trigger inhibit
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 19;04;46 comment by...Kike -->
We tried to recover CMX wedges 18-19 but they tripped after a coupled of minutes or so...


<!-- Date: Mon Jun 26 19:10:10.742 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5054&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5054' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5055&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5055' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5056&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5056' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 18.00-19.00  -->


<!-- Date: Mon Jun 26 19:21:50.864 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5057&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5057  width='300' align='top'></a></td><td>AVDD_B0W1L2 pinkie. I unmarked it.</td></tr></table>
<!-- Author: Kike -->
<!-- Comment: Mon Jun 26 19;25;13 comment by...Kike -->
It is still ping after HRR.

<!-- Comment: Mon Jun 26 19;27;32 comment by...Ulrich -->
Came back to normal values after another minute of waiting.


<!-- Date: Mon Jun 26 19:52:01.365 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5058&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5058  width='300' align='top'></a></td><td>There are some off-diagonal entries in L1-ADMEM.</td></tr></table>
<!-- Author: Arkadios -->


<!-- Date: Mon Jun 26 20:00:16.594 2006 -->
<P> 
I am finished working on the tomcat java servlets &minus;  
please re&euml;nable the ProcMon alarm/check. 
<P>  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 20:08:11.212 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5059&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5059' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5060&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5060' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5061&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5061' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 19.00-20.00  -->

<!-- Date: Mon Jun 26 20:14:19.797 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
fwd: <tt>The known problem of empty bins(61-63) for MPA (West OUT MP Occupancy).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 20:14:20.411 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
muons: <tt>The known BMU problem.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 20:14:21.978 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
trigger: <tt>There are some off-diagonal entries in L1-ADMEM and an error rate<br>
> 0.001 in the 125MEV DCAS Occupancy(see plot).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 20:14:22.429 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
monitors: <tt>Hot and cold triggers.Check Available XMon Triggers of Interest Pages.<br>
Ignore the SiliMon output according to the known problem list.<br>
Bad chi-2 for CMX-stub.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->



<!-- Date: Mon Jun 26 20:16:28.403 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5063&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5063  width='300' align='top'></a></td><td>The error rate is > 0.001 in the 125MEV DCAS Occupancy.</td></tr></table>
<!-- Author: Arkadios -->
<!-- Comment: Mon Jun 26 22;05;38 comment by...carla -->
<pre>As I mentioned earlier, this is a low level disagreement between the expected trigger energy and the one seen by the L1 trigger, with a difference of a few counts.
Please shepard PCAL04 when it's convenient.</pre>
<!-- Date: Mon Jun 26 20:18:00.455 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5062&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5062' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5064&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5064' width='300'></a></center>
<!-- Author: Ankush -->
<!-- Comment: Mon Jun 26 20;24;53 comment by...Ankush -->
<h3>A real Silicon resonance</h3>

<p>A real resonance occured today during data taking at 6:45pm.  During the error, only one SRC FATAL ERROR box appeared. There was no L1DONE TIMEOUT, no RXPT ERROR and no 'error box explosion'.  The LA traces above were triggered by the resonance.  The left-hand trace shows the whole data and the trace on the left is a zoomed in section about the SRC error.  Note, the L1A rate did not suddenly go to a high rate as observed in other LA traces.  In fact the resonance is driven by the L2R line.</p>

<p>This is significant as it shows under a genuine resonance, the GhostBuster, the SRC, the error-handler and Run Control all behave correctly. Further details can be found in the 
<a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=307&frame=2&anchor=19:21:54'>
Silicon e-log</a>.</p>


<!-- Date: Mon Jun 26 21:11:50.314 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5065&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5065' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5066&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5066' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5067&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5067' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 20.00-21.00  -->


<!-- Date: Mon Jun 26 21:31:22.965 2006 -->
<pre>We switched on CMX wedges 18-19 ~20 min ago. They are working fine so far. We might consider end this run in order to start a new one with all CMX channels in.</pre>
<!-- Author: Kike :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 21;47;21 comment by...Kike -->
They tripped again. I turned them on.

<!-- Comment: Mon Jun 26 21;56;28 comment by...Kike -->
Another trip. We turned them off. Will try again later.



<!-- Date: Mon Jun 26 22:05:39.611 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5068&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5068' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5069&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5069' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5070&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5070' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 21.00-22.00  -->

<!-- Date: Mon Jun 26 22:12:31.889 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
fwd: <tt>The known problem of empty bins(61-63) for MPA (marked BAD according to the suggestion of expert Mary Convey from the known problem list).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 22:12:32.151 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
muons: <tt>The known BMU problem.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 22:12:34.489 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
trigger: <tt>There are some off-diagonal entries in L1-ADMEM and an error rate<br>
> 0.001 in the 125MEV DCAS Occupancy(see the plots from the previous<br>
checklist).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 22:12:35.522 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
monitors: <tt>Hot and cold triggers.Check Available XMon Triggers of Interest Pages.<br>
Ignore the SiliMon output according to the known problem list.<br>
Bad chi-2 for CMX-stub.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->



<!-- Date: Mon Jun 26 22:45:51.892 2006 -->
I will try to power-up CMX 18-19 NE manually on the first floor.  I let the shift crew know about it.
<!-- Author: John Strologas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->


<!-- Date: Mon Jun 26 23:15:48.949 2006 -->
<pre>Some silicon power supplies spontaneously switched off. I turned bach on the affected wedges.</pre>
<!-- Author: Kike :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->
<!-- Comment: Mon Jun 26 23;23;39 comment by...Ulrich -->
We are still getting automated pages on any iFix alarm, so I was aware of the problem, but did not have to intervene. Good job! It is a little weird that this spontaneous switch-off happened to the same crate as a couple of hours ago, but maybe it is just coincidence.
<!-- Date: Mon Jun 26 23:16:27.588 2006 -->
<pre>Trigger Ihibit, Recovered 
 
Trigger Inhibit is set from the components: 
 ISL HV:iFix  		[129] Intermediate Silicon High Voltage from iFix 
 L00:iFix  		[150] Silicon Layer 00 from iFix 
Trigger Inhibit is set from the components: 
 L00:iFix  		[150] Silicon Layer 00 from iFix 
</pre>
<!-- Author: Yanjun :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->



<!-- Date: Mon Jun 26 23:31:10.404 2006 -->
<pre>We had the same proble with ifix as the one described by Stev at 16.47. I had to restarted it. Now it is ok. 
</pre>
<!-- Author: Kike -->


<!-- Date: Mon Jun 26 23:38:49.750 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5071&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5071' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5072&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5072' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-5073&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-5073' width='300'></a></center>
<!-- Author: Kike&#39;s hourly plots: 22.00-23.00  -->


<!-- Date: Mon Jun 26 23:41:23.868 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5074&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5074  width='300' align='top'></a></td><td>Several consecutive Red ladders.</td></tr></table>
<!-- Author: Arkadios -->
<!-- Comment: Tue Jun 27 00;00;16 comment by...Ulrich -->
I have instructed the shift crew to wait for an update and page me again if the cells are still red. Looking at the most recent consumer slides from 23:49, the red ladders seem to have disappeared. Although the red cells cover more of L00 than the wedges which switched off themselves shortly before the above plot was produced, the problem may be related.
<!-- Date: Mon Jun 26 23:41:45.554 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5075&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5075  width='300' align='top'></a></td><td>AVDD_B1W4L0 pinki. I Unmarked it and is green now.</td></tr></table>
<!-- Author: Kike -->
<!-- Comment: Tue Jun 27 00;04;31 comment by...Ulrich -->
This pinkie happened during the spontaneous switch-off and recovered, most probably during the HRR.
<!-- Date: Mon Jun 26 23:46:48.882 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-5076&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-5076  width='300' align='top'></a></td><td>AVDD_B3_W2L0 pinkie. I unmarked and it is green now.</td></tr></table>
<!-- Author: Kike -->
<!-- Comment: Tue Jun 27 00;05;07 comment by...Ulrich -->
This pinkie happened during the spontaneous switch-off and recovered, most probably during the HRR.
<!-- Date: Mon Jun 26 23:54:08.898 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
fwd: <tt>The known problem of empty bins(61-63) for MPA(see the Known Problem List).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 23:54:09.875 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
si: <tt> Several consecutive Red ladders in L00.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 23:54:10.268 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
muons: <tt> The known BMU problem.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 23:54:11.160 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
trigger: <tt>There are some off-diagonal entries in L1-ADMEM and an error rate<br>
> 0.001 in the 125MEV DCAS Occupancy(see the plots from the previous<br>
checklist).</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 23:54:12.314 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121>219121</A>
monitors: <tt>Hot and cold triggers.Check Available XMon Triggers of Interest Pages.<br>
Ignore the SiliMon output according to the known problem list.<br>
Bad chi-2 for CMX-stub.</tt>
<!-- Author: Consumer Operator( Manousakis-Katsikakis Arkadios) -->


<!-- Date: Mon Jun 26 23:55:10.371 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219108 target=_runSummary>219108</A> x357E4</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,626]</FONT></TD>
<TD ALIGN=CENTER>12:28:42</TD>
<TD ALIGN=CENTER>16:34:28</TD>
<TD ALIGN=CENTER>03:30:23</TD>
<TD ALIGN=RIGHT>   1,287,205</TD>   <TD ALIGN=RIGHT>  1,176.31</TD>
<TD ALIGN=RIGHT>  1,016.91</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219121 target=_runSummary>219121</A> x357F1</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,626]</FONT></TD>
<TD ALIGN=CENTER>16:39:11</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>06:30:20</TD>
<TD ALIGN=RIGHT>   1,884,228</TD>   <TD ALIGN=RIGHT>  1,243.43</TD>
<TD ALIGN=RIGHT>  1,107.20</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:00</TD>
<TD ALIGN=CENTER>10:00:43</TD>
<TD ALIGN=RIGHT>     3,171,433</TD> <TD ALIGN=RIGHT>  2,419.74</TD>
<TD ALIGN=RIGHT>  2,124.11</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Tue Jun 27 00:01:10.540 2006 -->
<font color=red>Shift Summary: </font><pre>expect smooth running. 
 
~16:30 paged cmx because 4 trips in 20minutes. 
see elog for details. 
 
18:30 part of silicon detector power 
turned itself off spontaneously. 
Apparently this is business as usual. 
 
~20:00 pcal04 having some issues. 
We saw them as inconsistencies in DCAS vs Addmem plot. 
see elog for details. 
Carla said we should shepherd the b0pcal04 crate 
next time something else breaks, and we loose time anyway. 
 
~20:30 called cmx expert again because the prescription he suggested 
isn't helping. Are told to keep on trying once an hour. 
 
22:40 Strologas came in to work on the cmx wedge that doesn't hold power. 
 
23:30 paged svx expert because of red L00 in consumer. 
resolved itself by itself. Maybe was related to the power going off randomly. 
 
</pre> 
<!-- Author: fkw -->


<!-- Date: Tue Jun 27 00:04:06.368 2006 -->
At the end of the evening shift after an update of the plots, L00 plot is green again. Thus L00 looks OK again.
<!-- Author: aurore :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219121' target=_top>219121</a>) -->