<!-- Title: Owl shift. Wed Dec 19, 2007 -->
<!-- SciCo: Wolfgang Wagner -->
<!-- DAQAce: Sam Hewamanage -->
<!-- MonAce: &nbsp; -->
<!-- CO: Guiseppe Latino -->
<!-- OpManager: Max Goncharov -->
<!-- Notes: -->
<pre>Store #5809 is still spinning. We take data, Run 255224. According to run coordinator e-log "The plan is to keep the store 5809 in as long as the experiments feel that it is useful." The work on the Pelletron will continue until 15:00 hrs today. New beams are expected at 16:30 hrs.</pre>


<!-- Date: Wed Dec 19 00:14:19.969 2007 -->
L2 decision TO.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->
<!-- Comment: Wed Dec 19 00;21;35 comment by...sam -->
again


<!-- Comment: Wed Dec 19 01;09;02 comment by...sam -->
again
<!-- Comment: Wed Dec 19 01;40;37 comment by...sam -->
again*2
<!-- Date: Wed Dec 19 00:24:48.857 2007 -->
momentary SI. nothing obvious! VNODE2 has a 'Information' pop up 'Error Number -xxxxxxxxx. A network transaction is in progress.' I acknowledged and 'Skipped All'. also P-bar AgGaps has shot to ~18kHz once. So it is most probably due to this. The nominal value for the P-bar Ag Loss is 12kHz. 
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->
<!-- Comment: Wed Dec 19 00;25;11 comment by...sam -->
I mean momentary Si alarm...


<!-- Date: Wed Dec 19 00:46:01.491 2007 -->
Forgot to add the lumi upfront at the beginning: it's now 
L = 52.6E30 cm-2s-1
<!-- Author: Wolfgang :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->

<!-- Date: Wed Dec 19 00:51:19.153 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224>255224</A>
central: <tt>TOF marked bad: usual 9 dead channels problem...</tt>
<!-- Author: Consumer Operator( Giuseppe Latino) -->



<!-- Date: Wed Dec 19 01:00:36.941 2007 -->
ProcMon warning. It seems like ENCP was stuck. First check failed. checked again after several minutes. check passed.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->
<!-- Comment: Wed Dec 19 01;14;32 comment by...sam -->
again.
<!-- Date: Wed Dec 19 01:06:33.322 2007 -->
I think I left my laptop power adaptor in the control. Could you please put it to a side (or just leave it there), and I will get it tomorrow morning. Thanks!
<!-- Author: Si :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->
<!-- Comment: Wed Dec 19 01;06;59 comment by...Si -->
in the control room...


<!-- Comment: Wed Dec 19 01;14;18 comment by...sam -->
yep. I put it in the drawer next to printer.




<!-- Date: Wed Dec 19 02:12:11.109 2007 -->
We are currently running at 96.3% efficiency. Very quiet, hope it remains like this.
<!-- Author: Wolfgang :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->


<!-- Date: Wed Dec 19 03:28:59.473 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7425&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7425  width='300' align='top'></a></td><td>there was a PAGC spike above >20kHz in the fast time plotter. but i don't see it in the history!</td></tr></table>
<!-- Author: sa -->


<!-- Date: Wed Dec 19 03:32:18.930 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7427&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7427  width='300' align='top'></a></td><td>here is the history plot. i made sure the time period is correct.</td></tr></table>
<!-- Author: sam -->


<!-- Date: Wed Dec 19 03:45:18.662 2007 -->
ev2b stuck. 
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->
<!-- Comment: Wed Dec 19 03;49;12 comment by...sam -->
RC was stuck in HALT mode. Abort was the only actuve button. manually overide the state manage to end the run and restarting.
<!-- Comment: Wed Dec 19 04;04;38 comment by...sam -->
i checked the EVB proxy and SCPUs. evme05 was out of sync. somehow in the rush i forgot to clean up EVB after ending the run as the instructions says. i checked the SCPUs. all are up. still warns about possible out of sync in evme05. i am not seeing any erros so far.

<!-- Comment: Wed Dec 19 04;05;54 comment by...sam -->
so i'll keep running till something happens again.
<!-- Comment: Wed Dec 19 04;22;45 comment by...sam -->
i started from START.
<!-- Date: Wed Dec 19 03:45:51.134 2007 -->
We have an event builder problem. Crate evb2b got stuck in the recover transition. Sam aborted the run for that reason.
<!-- Author: Wolfgang :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224' target=_top>255224</a>) -->

<!-- Date: Wed Dec 19 03:46:35.588 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224>255224</A>
 Terminated at 2007.12.19 03:46:08
<!-- Author: RunControl -->


<!-- Date: Wed Dec 19 03:46:36.512 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224>255224</A>
TERMINATE: evb2b stuck.
<!-- Author: Sam -->



<!-- Date: Wed Dec 19 03:49:43.881 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232>255232</A>
 Activated at 2007.12.19 03:49:21
<!-- Author: RunControl -->


<!-- Date: Wed Dec 19 03:49:44.798 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232>255232</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Sam -->



<!-- Date: Wed Dec 19 03:50:01.433 2007 -->
Done Timeout detected in crates: 
SVT_PULSAR_09 
-- hrr'ed
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->


<!-- Date: Wed Dec 19 03:52:22.830 2007 -->
Done TO. for crate/s:  
    b0pcal04  -- auto hrr did not help. shepherding.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->
<!-- Comment: Wed Dec 19 03;53;26 comment by...sam -->
said it was successfull. but still yellow. doing it again.

<!-- Date: Wed Dec 19 03:57:15.980 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255224>255224</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
TOF 
</PRE>is BAD because of 9 dead channels which need to be fixed in an access.
<!-- Author: cdfscico -->



<!-- Date: Wed Dec 19 04:03:42.977 2007 -->
Brief report being 1/2 way into the shift: <BR> 
<TABLE bgcolor=#cdc9a5> 
<tr><th colspan=2>Totals</th></tr> 
 
<tr><td>Begin shift:</td><td>2007.12.19 at 00:00:38</td> 
<tr><td>End report period:</td><td>04:00:38</td> 
<tr><td>Delivered luminosity:</td><td> 706.9 nb<sup>-1</sup></td></tr> 
<tr><td>Acquired luminosity:</td><td> 644.0 nb<sup>-1</sup></td></tr> 
<tr><td>Efficiency:</td><td> 91.1</td></tr> 
 
</TABLE>
<!-- Author: Wolfgang -->
<!-- Comment: Wed Dec 19 04;05;37 comment by...Wolfgang -->
Current luminosity is L = 45.22	E30 cm-2 s-1.



<!-- Date: Wed Dec 19 04:10:57.254 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7428&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7428  width='300' align='top'></a></td><td>ok, when i look at the ace history plot, it shows the spike. as i guessed it was flying wire. so i should always look at ACE.... plots. what is the diff between this and Recall->Loss/Abgaps?</td></tr></table>
<!-- Author: sam -->
<!-- Comment: Wed Dec 19 04;29;04 comment by...sam -->
hmm i was looking at the wrong qty! so still don't see this >20kHz spike?


<!-- Date: Wed Dec 19 05:26:55.891 2007 -->
Weird XMON warning canvases still observed. 
 
As already reported by CO of previous shift, XMON shows several canvases which are in a directory called Warning and which are all red. They have messages like: "ERROR found by SL1D Monitor: Unprescaled L2 trigger 129 L2_TWO_EM16 is firing at a rate of 0.0 hz. You should double check all of the L2 trigger rates!". 
I checked the trigger rates in XMonSummary web page and they look ok. For instance, for the above example:  
129  L2_TWO_EM16 -1    4835    1.10    1.09    25.61    24.78 26.41±5.38  -0.15 => the trigger rate is ~ 1.1Hz and the xsec is well within expectations. 
We tried to restart XMon (this solved the problem during last shift) but the red canvases reappeared again.  
It looks there is some software problem connected with XMon which should be investigated.    
<!-- Author: Giuseppe :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->


<!-- Date: Wed Dec 19 06:05:14.054 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7429&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7429  width='300' align='top'></a></td><td>1.2.2: L1_TFRD_Unprescaled_Triggers shows an unexpected error...</td></tr></table>
<!-- Author: Giuseppe -->


<!-- Date: Wed Dec 19 06:18:34.982 2007 -->
Done TO. for crate/s:  
    b0pcal10 -- shepherded to give life.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->


<!-- Date: Wed Dec 19 06:19:35.709 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7432&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7433  width='300' align='top'></a></td><td>Warning Canvas of TrigMon concerning TFRD_Monitor</td></tr></table>
<!-- Author: Wolfgang -->
<!-- Comment: Wed Dec 19 06;20;19 comment by...Wolfgang -->
Image is too big to be displayed. Click on the image to view it.

<!-- Comment: Wed Dec 19 06;22;47 comment by...Wolfgang -->
The instructions on the canvas say, if there are only a few errors we should just report the problem in the elog. The experts should comment. I'll mention it in the 08:00 am meeting.

<!-- Date: Wed Dec 19 06:23:30.858 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232>255232</A>
central: <tt>TOF marked bad because of known problem of 9 dead channels.</tt>
<!-- Author: Consumer Operator(Giuseppe Latino) -->


<!-- Date: Wed Dec 19 06:23:31.213 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232>255232</A>
cal: <tt>A dead PMT is observed in west PHA. Plot put on e-log. Run is still marked good. E-mail sent to experts.</tt>
<!-- Author: Consumer Operator(Giuseppe Latino) -->


<!-- Date: Wed Dec 19 06:23:32.208 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232>255232</A>
trigger: <tt>1.2.2: L1_TFRD_Unprescaled_Triggers shows an unexpected error. As required: plot put on e-log and experts paged. L1 marked "Other" while waiting for expert response.</tt>
<!-- Author: Consumer Operator(Giuseppe Latino) -->



<!-- Date: Wed Dec 19 06:26:05.311 2007 -->
<table><tr><td><a href=/elog/notebooks/2007/Wed_Dec_19_2007_Owl.062555.ps align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7435  align='top'></a></td><td>A dead PMT is observed in west PHA...</td></tr></table>
<!-- Author: Giuseppe -->
<!-- Comment: Wed Dec 19 07;17;23 comment by...Larry N -->
should be on the known list
<!-- Comment: Wed Dec 19 07;26;34 comment by...Giuseppe -->
The same dead PMT was also observed by eve CO on Dec. 17th shift. From comments by experts reported on e-log for that shift, it depends on geometrical segmentation of that tower and sometimes appear with low statistics. This dead channel typically go away with more statistic. It indeed is gone now ! 
This PHA behavior at low statistics should already have been put on  
CO known problems list.... I'll do that....
<!-- Comment: Wed Dec 19 07;27;24 comment by...Giuseppe -->
... Sorry, Larry wrote his comment when I was writing mine..
<!-- Date: Wed Dec 19 06:50:02.124 2007 -->
We got a trigger inhibit from aPSM rack: RR18G_2. Cryo called. Dervin was in the control room and will look at it.
<!-- Author: Wolfgang :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->
<!-- Comment: Wed Dec 19 06;52;20 comment by...Wolfgang -->
Dervin will have to replace the fan in the crate. It's a hadron TDC crate.
<!-- Comment: Wed Dec 19 07;19;09 comment by...Wolfgang -->
Dervin replaced the fan in RR18G_2. Trigger inhibit has cleared.
<!-- Date: Wed Dec 19 06:50:48.780 2007 -->
htdc00 died. shepherdin it. and cooling alarm from PSM, rack 1RR18G_2 fans failed.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->
<!-- Comment: Wed Dec 19 06;53;08 comment by...sam -->
We'll wait till this is fixed. htdc00 is in this rack.


<!-- Date: Wed Dec 19 06:55:09.939 2007 -->
 ProcMon says RTSERVER is not running. check says the same.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->
<!-- Comment: Wed Dec 19 06;57;21 comment by...sam -->
this is due to hdtc00 that is in trouble.


<!-- Comment: Wed Dec 19 07;00;06 comment by...sam -->
check summary<br>
<pre>

2007.12.19 06:57:49 Status: CHECKING
2007.12.19 06:58:57 Status: CHECK FAILED 
Wed Dec 19 06:57:54 CST 2007   procAction  rtserver   check   b0server04
Checking rtserver
Found 1 copy of the rtserver watcher 'sscheck' script running (1 expected)
Found 1 copy of the rtserver (rtserver.x) running (1 expected)
Found 200 rtserver connections (near >235 will degrade performance)
Found 0 rtserver FIN_WAIT connections (>0 may degrade performance)
Found 0 rtserver FIN_WAIT connections from www-cdfonline (TOMCAT)
Found 1 potentially stuck ports 
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode     
tcp        0  12568 b0server04.fnal.gov:5101    b0htdc00.fnal.gov:1920      ESTABLISHED cdfdaq     346869201  
RTserver is not running correctly



</pre>
<!-- Comment: Wed Dec 19 07;08;47 comment by...sam -->
its green again. i think it dropped the connection to htdc.
<!-- Date: Wed Dec 19 06:59:06.783 2007 -->
arghhhh.. we were getting close to 94%.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->




<!-- Date: Wed Dec 19 07:10:02.688 2007 -->
Consumer monitor says its not geeting HB from SVXMon. Since we are in halt???
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->

<!-- Date: Wed Dec 19 07:16:03.761 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232>255232</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
TOF 
</PRE> because of dead channels
<!-- Author: cdfscico -->
<!-- Comment: Wed Dec 19 07;19;50 comment by...Wolfgang -->
Sorry, I was too fast. That's our current run. I reverted the marking in the data base.
<!-- Date: Wed Dec 19 07:18:45.078 2007 -->
Thanx to Dervin, we up. Shepherding the htdc.
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->
<!-- Comment: Wed Dec 19 07;20;01 comment by...Sam -->
what a bummer. we are back in business.




<!-- Date: Wed Dec 19 07:40:41.876 2007 -->
TFRD errors in TrigMon (#slide 13, reported earlier) are now at 7 counts. This was 5 before.
<!-- Author: Wolfgang :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255232' target=_top>255232</a>) -->


<!-- Date: Wed Dec 19 07:52:32.962 2007 -->
<font color=red>Shift Summary: </font><pre>00:00  Store 5809 is still in from previous owl shift. 
       Take data, Run 255224. 
 
00:24  We got a spurious silicon sono alarm which went away immediately. 
       We speculate that this is related to a spike in the antiproton 
       abort gap halo.  
 
03:45  Event builder problem: evb2b got stuck in recovery state of an HRR. 
       We had to end the run. Start run # 255232.  
 
05:26  CO Guiseppe reports inconsistency with XMon: HistoDisplay shows warning 
       messages, while rates on Summary page look okay. During the day that 
       should be further investigated.  
 
06:00  Guiseppe reports an error in TrigMon: L1_TFRD 
       Since there were only a few errors we contiued running. 
 
06:50  Fan in PSM crate fails and is replaced by Dervin. 
07:19  Trigger inhibit is cleared. Back to data taking. 
 
07:45  Luminosity now at 39.25E30 cm-2 s-1 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   255224, 255232
Delivered Luminosity   1288.1 nb-1  
Acquired Luminosity    1125.7 nb-1  
Efficiency             87.4%
   
</td></tr></table></pre>
<!-- Author: Wolfgang -->

<!-- Date: Wed Dec 19 07:55:11.828 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255224 target=_runSummary>255224</A> x3E4F8</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>16:15:27</TD>
<TD ALIGN=CENTER>03:46:08</TD>
<TD ALIGN=CENTER>10:37:17</TD>
<TD ALIGN=RIGHT>   3,242,499</TD>   <TD ALIGN=RIGHT>  2,462.88</TD>
<TD ALIGN=RIGHT>  2,269.04</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255232 target=_runSummary>255232</A> x3E500</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>03:49:21</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>03:20:57</TD>
<TD ALIGN=RIGHT>     855,970</TD>   <TD ALIGN=RIGHT>    619.14</TD>
<TD ALIGN=RIGHT>    509.45</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>13:58:14</TD>
<TD ALIGN=RIGHT>     4,098,469</TD> <TD ALIGN=RIGHT>  3,082.01</TD>
<TD ALIGN=RIGHT>  2,778.49</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->