<!-- Title: Eve shift. Mon Jul 30, 2007 -->
<!-- SciCo: Rob (Sasha) Harr -->
<!-- DAQAce: Sasha Paramonov -->
<!-- MonAce: &nbsp; -->
<!-- CO: Tara (Sahsa) Spears -->
<!-- OpManager: Sasha Pronko -->
<!-- Notes: -->
<pre>Returning to store 5588, the store we started on last night.  Lumi at 43e30, stash of 250e9 and stack of 20mA.  We'll nurse this store as long as it stays.</pre>


<!-- Date: Mon Jul 30 16:16:05.814 2007 -->
<pre>FrontEnd Crate Error Condition from: VRB_SVX_02 
(SRC Fatality Error for b0svx02: Sl 6 Resonance Detected ) 
was followed by a few L2 Timeouts. 
Was fixed by a few HRRs. 
</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 16:51:52.666 2007 -->
ICICLE is not working: Allocation of shared memory failed.<br> 
IT cano not connect to the RT server and the network drive is "busy". We are trying to fix it.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->
<!-- Comment: Mon Jul 30 16;55;09 comment by...Rob Harr -->
Called Steve Hahn and he'll take a look.


<!-- Comment: Mon Jul 30 17;09;34 comment by...Sasha -->
Steve has fixed it.
<!-- Comment: Mon Jul 30 17;23;45 comment by...Steve Hahn -->
<p>There are two problems: one involving disk usage of ICICLE
debug logs, which Margherita has agreed to cleam up, and the
second being the usual "disconnected network disk" problem with
the S: disk, which in turn means the RTserver config file 
cannot be found.  Fixing the latter required a reboot, which
I did.  ICICLE is now running OK.</p>
<!-- Date: Mon Jul 30 17:00:15.797 2007 -->
eveb2b went rd while in ACTIVE. HRRed.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->



<!-- Date: Mon Jul 30 17:19:15.639 2007 -->
<br>Scaler Mon Error. 
<br>- Trigger/DAQ Deadtime is too high (at 22.797394 %).
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->



<!-- Date: Mon Jul 30 17:26:28.567 2007 -->
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected  followed by a L2 Decision Timeout.<br> I had to HRR twice to fix the issue.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 17:26:56.571 2007 -->
Done Timeout detected in crates: COT_07<br> 
It was fixed by an auto HRR.  
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->
<!-- Comment: Mon Jul 30 20;14;24 comment by...Sasha -->
The same thing just happened with COT_06.
<!-- Comment: Mon Jul 30 20;18;21 comment by...Sasha -->
Again. COT_07
<!-- Comment: Mon Jul 30 23;30;05 comment by...Sasha -->
Done Timeout detected in crates: COT_06. I just HRRed it.

<!-- Date: Mon Jul 30 17:30:04.499 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-6215&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-6215  width='300' align='top'></a></td><td>Possible hot trigger channel in TrigMon 125 MeV DCAS occupancy plot (rate of 0.14 at eta 0 (west) phi 4.</td></tr></table>
<!-- Author: Tara -->
<!-- Comment: Mon Jul 30 17;39;11 comment by...Tara -->
SciCo paged expert - this is a known problem and doesn't affect data quality.
<!-- Date: Mon Jul 30 17:37:38.913 2007 -->
PSM alarm from cate 1RR27D_2 (1st floor). The voltages are unphysicsl.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->
<!-- Comment: Mon Jul 30 17;38;17 comment by...Sasha -->
It did not cause a trigger inhibit..



<!-- Date: Mon Jul 30 17:40:34.755 2007 -->
<p>After some discussion with Jonathan, I have masked all the 
SMD crates in the SW and SE arches in addition to the four 
bad PEI crates in the CASWT and CASEB.  Thus, PSM is now yellow 
indicating there are masked power supplies.  Any new problems 
that are not in these masked crates will turn PSM red, and  
should be considered serious.</p> 
 
<p>The other method I could have used was to disable dataline 
#6 which is the entire south side.  This would speed up the 
readout, since VOLTMAN would not be making repeated attempts at 
reading out all the uncommunicative crates, but it would mean 
the four PEIs that are OK on the south side would not be 
monitored.</p>
<!-- Author: Steve Hahn :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->

<!-- Date: Mon Jul 30 17:41:26.443 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986>245986</A>
fwd: <tt>MNP east occupancy zero, as before.</tt>
<!-- Author: Consumer Operator( Tara) -->



<!-- Date: Mon Jul 30 17:58:22.151 2007 -->
Cryo wants to power down the solenoid because dripping water filled the bucket and may present a safety hazard. We will do that between stores. More details in e-log from cryo and e-mail to cdf-ops-managers@fnal.gov with an official request would still be appreciated. 
 
<!-- Author: Sasha Pronko (OpsMan)  -->


<!-- Date: Mon Jul 30 18:24:54.779 2007 -->
Busy Timeout. It was fixed by an HRR
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 18:55:50.342 2007 -->
<pre>Done Timeout detected in crates: CMX_00 
then  
FrontEnd Crate Error Condition from: VRB_SVX_02 
then 
SRC Fatality Error for b0svx02: Sl 6 Resonance Detected  
 
I HRRed it. 
 
=&gt; L2 Decision Timeout. 
 
I HRRed it again and the problem was fixed.</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 19:07:26.868 2007 -->
RXPT error for b0SVX02 followed by 2 L2 Decision Timeouts. I had to HRR multiple times. It looks like we get L2 timeouts after every RXPT error.
<!-- Author: Sash :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->
<!-- Comment: Mon Jul 30 19;20;23 comment by...Miguel M. -->
It seems that four resonance errors have been produced in the past three hours. You should consider to page silicon.
<!-- Date: Mon Jul 30 19:08:24.951 2007 -->
Trigger inhibit from 1RR21E_2. LV are unphysicsl. This is the 1st floor PSU.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->



<!-- Date: Mon Jul 30 19:24:46.210 2007 -->
"Done Timeout detected in crates: COT_04" was fixed by an HRR.  
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 19:46:27.564 2007 -->
TM2 Events Pending Timeout Error!. It was fixed by an HRR.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 19:52:22.931 2007 -->
<pre>AcNet data not updated since 2007-07-30 19:39:41.121 
AcNet data is critical to data taking! 
Restart AcnetMonitor or contact expert immediately! 
 
AcNEt looks ok.</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->
<!-- Comment: Mon Jul 30 19;54;26 comment by...Sasha -->
TeVMon is not updating

<!-- Comment: Mon Jul 30 20;02;08 comment by...Miguel M. -->
They had the same problem in the <a href=
'/cgi/elog/elog.pl?nb=2007&action=view&page=631&frame=2&anchor=06:43:15.759'>
owl shift</a>. Have a look.
<!-- Comment: Mon Jul 30 20;13;00 comment by...Sasha -->
Jose restarted AcNet monitor. TeVMon is working.
<!-- Date: Mon Jul 30 19:55:47.557 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986>245986</A>
fwd: <tt>MNP east occupancy zero, as before.</tt>
<!-- Author: Consumer Operator( Tara) -->



<!-- Date: Mon Jul 30 19:59:09.538 2007 -->
Stopping and restarting YMON, as requested.
<!-- Author: Tara :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->






<!-- Date: Mon Jul 30 20:39:19.675 2007 -->
PSM alarm. Crate 1RR21H_1. 
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->
<!-- Comment: Mon Jul 30 20;42;05 comment by...Sasha -->
Why dowe get so many problems withthe 1st floor power supplies? 

<!-- Comment: Mon Jul 30 20;52;52 comment by...Sasha -->
I went downstairs. The power supply is fine ( the LEDs are green). I will ignor these alarms from now on. It funny that the monotoring system fails much more often then the power supplies.


<!-- Comment: Mon Jul 30 22;18;56 comment by...Phil -->
well, it's a bit worse than usual. Channeling Larry on the 2nd sentence, not unexpected.
<!-- Comment: Mon Jul 30 23;24;47 comment by...Sasha -->
one more.
<!-- Date: Mon Jul 30 21:02:55.019 2007 -->
<pre>iFix on SOLENOID@ (ICI)  has an error message: 
 
7/30/2007 19:31:14.6 !SystemAlert! The alarm queue 'SUMMARY LIST' has overflowed. At least one alarm may have been  lost.</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 21:27:51.889 2007 -->
"RXPT Error for crate/s: b0svx02" was followed by L2TO. I had to HRR twice.  
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->

<!-- Date: Mon Jul 30 21:45:24.089 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986>245986</A>
fwd: <tt>MNP East occupancy zero</tt>
<!-- Author: Consumer Operator( Tara) -->




<!-- Date: Mon Jul 30 22:38:03.489 2007 -->
<pre>Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
Stuck Cellid S/B3/W9/L3/C0-5 .  
 
Fixed by an auto HRR.</pre>
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 23:12:00.070 2007 -->
<p>The run coordinator has posted his plan for the end of this store:  he will make the call in the morning sometime between 8 and 10am (wants 430e10 in the recycler). 
He would like to try a high intensity shot before the shutdown comes, and trying now leaves another chance if this one doesn't work. 
</p> 
<p>Phil called to ask whether the plan included ramping the solenoid to empty the overflowing bucket of LCW. 
It does, and I conferred with cryo on checking in the morning that there will be sufficient time before shot setup (at least 90 min.) and ask if they can think about putting in a hose or something so that next time the bucket can be drained without having to ramp the solenoid. 
They will do both.</p> 
<p>I called Phil back, and he agreed that with this plan there should be time in the morning for him to check on solenoid ramping, or if the store is lost during the night, there would be time before the next store during recovery.</p> 
  
<!-- Author: Rob Harr -->


<!-- Date: Mon Jul 30 23:13:36.158 2007 -->
<p>Alison Lister called in. 
She asked to be called when the store is terminated; she wants to run some tests with L2. 
I've added a note to the white board for the following shifts.</p>
<!-- Author: Rob Harr -->



<!-- Date: Mon Jul 30 23:28:38.307 2007 -->
<br>Done Timeout detected in crates: IMU_00 
<br>Fixed by an HRR.
<!-- Author: Sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986' target=_top>245986</a>) -->


<!-- Date: Mon Jul 30 23:34:14.887 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245986>245986</A>
fwd: <tt>MNP East occupancy zero.</tt>
<!-- Author: Consumer Operator( Tara) -->


<!-- Date: Mon Jul 30 23:55:06.063 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=245986 target=_runSummary>245986</A> x3C0E2</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_02 [11,935,700]</FONT></TD>
<TD ALIGN=CENTER>09:18:13</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>13:01:38</TD>
<TD ALIGN=RIGHT>   6,906,774</TD>   <TD ALIGN=RIGHT>  2,277.96</TD>
<TD ALIGN=RIGHT>  2,023.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>13:01:38</TD>
<TD ALIGN=RIGHT>     6,906,774</TD> <TD ALIGN=RIGHT>  2,277.96</TD>
<TD ALIGN=RIGHT>  2,023.00</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->
<!-- Comment: Tue Jul 31 06;11;02 comment by...Sunil Somalwar (day SciCo) -->
End of shift report for this shift appears at the end of previous (day) shift for some reason