<!-- Title: Owl shift. Sun Aug 24, 2008 -->
<!-- SciCo: Rob Harr -->
<!-- DAQAce: YuChul Chang / David Mietlicki -->
<!-- MonAce: &nbsp; -->
<!-- CO: Michele Giunta -->
<!-- OpManager: JJ -->
<!-- Notes: -->
<pre>Trouble with the remote CO video connection from Pisa.  Consulting with JJ.  Inheriting store 6378, run 266511, L = 118E30.</pre>


<!-- Date: Sun Aug 24 00:15:13.981 2008 -->
When try to connect to CDF via Polycom I get "Your call cannot be completed because the call was routed through an intermediate network that does not service the far site. Contact your network administrator...". 
I restarted my system and the PVX program with no success. 
Going throu checklist for run 266511.  
<!-- Author: Mikele :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 00:15:30.624 2008 -->
L2 Decision TO, fixed by auto HRR
<!-- Author: David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->
<!-- Comment: Sun Aug 24 00;23;10 comment by...Yuchul -->
again twice
<!-- Comment: Sun Aug 24 00;30;51 comment by...David -->
again
<!-- Comment: Sun Aug 24 01;08;00 comment by...David -->
again
<!-- Comment: Sun Aug 24 01;17;11 comment by...David -->
again
<!-- Comment: Sun Aug 24 01;29;29 comment by...Yuchul -->
again
<!-- Comment: Sun Aug 24 01;31;04 comment by...David -->
again
<!-- Comment: Sun Aug 24 01;36;22 comment by...David -->
again
<!-- Comment: Sun Aug 24 01;54;54 comment by...Yuchul -->
again
<!-- Comment: Sun Aug 24 01;55;52 comment by...Yuchul -->
again
<!-- Comment: Sun Aug 24 02;09;30 comment by...David -->
again
<!-- Comment: Sun Aug 24 02;10;53 comment by...David -->
again
<!-- Comment: Sun Aug 24 02;17;21 comment by...David -->
again
<!-- Comment: Sun Aug 24 02;18;03 comment by...David -->
again
<!-- Comment: Sun Aug 24 02;45;16 comment by...David -->
again
<!-- Comment: Sun Aug 24 02;53;56 comment by...David -->
again
<!-- Comment: Sun Aug 24 03;01;27 comment by...David -->
again
<!-- Comment: Sun Aug 24 03;36;22 comment by...David -->
again
<!-- Comment: Sun Aug 24 03;51;33 comment by...David -->
again
<!-- Comment: Sun Aug 24 04;25;11 comment by...David -->
again
<!-- Comment: Sun Aug 24 04;28;43 comment by...Yuchul -->
again
<!-- Comment: Sun Aug 24 04;34;42 comment by...David -->
again
<!-- Comment: Sun Aug 24 04;56;05 comment by...David -->
again
<!-- Comment: Sun Aug 24 05;17;04 comment by...David -->
again
<!-- Comment: Sun Aug 24 05;27;17 comment by...David -->
again
<!-- Comment: Sun Aug 24 05;48;54 comment by...David -->
again
<!-- Comment: Sun Aug 24 06;04;30 comment by...David -->
again
<!-- Comment: Sun Aug 24 06;13;33 comment by...David -->
again
<!-- Comment: Sun Aug 24 06;18;18 comment by...David -->
again
<!-- Comment: Sun Aug 24 06;21;10 comment by...David/Yuchul -->
again
<!-- Comment: Sun Aug 24 06;39;41 comment by...Yuchul/David -->
again
<!-- Comment: Sun Aug 24 07;25;34 comment by...David/Yuchul -->
Again
<!-- Date: Sun Aug 24 00:16:44.428 2008 -->
I assume all HV are ON. If not, please comment this entry.
<!-- Author: Mikele :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->
<!-- Comment: Sun Aug 24 00;18;20 comment by...Yuchul -->
All HV is ON now



<!-- Date: Sun Aug 24 00:28:19.746 2008 -->
CMP HV trip.  Checked detail summary, trip in CMP Bottom A4 and Bottom D4.  Turned HV back on.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 00:28:54.830 2008 -->
Plug HV 07 trip.  Recovered by itself as we checked on it.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->
<!-- Comment: Sun Aug 24 00;49;16 comment by...David/Yuchul -->
As we exited plug HV monitor, LED 12 turned back on
<!-- Comment: Sun Aug 24 00;49;54 comment by...David/Yuchul -->
Sorry, I meant LED 07
<!-- Date: Sun Aug 24 00:30:36.581 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266509>266509</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=266509 target=_runSummary>266509</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->




<!-- Date: Sun Aug 24 00:33:02.655 2008 -->
I can ping the Control Room polycom from home. SciCo Rob Harr 
can connect to Control Room polycom from Pump Room polycom.<br> 
I suggested that Pisa:<br> 
<ul> 
<li>Try to connect to Pump Room (see IP address at 
   https://www-cdfonline.fnal.gov/internal/ops/cohelp/co_remote_communications.html) 
<li>Try to ping Control Room Polycom from computer running 
    PVX software in Pisa. 
</ul> 
Above will give us more data.<br><br> 
Another approach would be to have both units try to 
connect to a meeting. Try using call 85cdfco (8523326). 
Both unit would have to dial this meeting.<br><br> 
Finally I note that Remote Co can also<br> 
    *  1) Stop (kill) and restart the PVX application <br> 
 
    * 2) Reboot the machine hosting the video-conferencing tools and restart PVX <br> 
(I think #1 has been tried.)
<!-- Author: jj -->
<!-- Comment: Sun Aug 24 00;34;49 comment by...jj -->
It should be noted that Remote Co can proceed without Video
Connection but must use telephone to discuss all problems.

<!-- Date: Sun Aug 24 00:35:45.138 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
cal: <tt>(dead3) WHAPMT Canvas: eta=-11, phi wedge 6 new HOT PMT. Red is 0.24 wrt blu 0.04, which is more than 10% higher. I guess this meets criteri to be defined as BAD. I did not find this on the Known problem list.</tt>
<!-- Author: Consumer Operator( Mikele ) -->
<!-- Comment: Sun Aug 24 01;27;36 comment by...Rob -->
emailed cdf_cal_op per instructions.
<!-- Date: Sun Aug 24 00:35:46.174 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
general: <tt>Exept for the WHA PMT, it looks good. Rob, please cross-check if criterion to mark it bad is met..</tt>
<!-- Author: Consumer Operator( Mikele ) -->



<!-- Date: Sun Aug 24 00:46:03.023 2008 -->
Plug HV 12 trip and PEM alarm in iFix, we tried to on LED 12 and restart plug HV program, And we had stuck b0pcal05 in recover transition, shepherded it. taking data again.
<!-- Author: Yuchul/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 00:48:06.346 2008 -->
ENCP TO TAPE is red in ProcMon
<!-- Author: Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->
<!-- Comment: Sun Aug 24 01;32;36 comment by...Yuchul -->
Now, it back green
<!-- Date: Sun Aug 24 00:50:17.418 2008 -->
Regarding the videoconference problem: 
My PC was restarted, and the PVX polycom program too. Problem persists. 
We tried to connect to a common number, 8523326, video is OK but we  have no audio.
<!-- Author: Mikele -->
<!-- Comment: Sun Aug 24 00;51;49 comment by...Rob -->
We hear you fine.  Check your speaker volume and we'll check the microphone here.

<!-- Comment: Sun Aug 24 00;54;30 comment by...rob -->
We'll try disconnecting and reconnecting... Please stay connected.


<!-- Comment: Sun Aug 24 01;28;04 comment by...Rob -->
Direct connection is working again.
<!-- Date: Sun Aug 24 00:55:11.421 2008 -->
You hear me?? Good. From this side there's no problem on speakers, it rather looks a network problem because there are noises coming and going, like the bandwith was very tight.
<!-- Author: Mikele -->


<!-- Date: Sun Aug 24 01:04:45.867 2008 -->
ISL05 trip, followed by auto-Hockerization of crate 12.  Silicon auto recovery was successful, now taking data.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->




<!-- Date: Sun Aug 24 01:22:47.373 2008 -->
Done TO b0pcal04, had to shepherd it.
<!-- Author: Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 01:26:08.043 2008 -->
Hourly plots not updated.  Restarted Sequencer
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->











<!-- Date: Sun Aug 24 02:15:04.687 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
cal: <tt>Same as previous checklist. Expert already mailed by Rob.</tt>
<!-- Author: Consumer Operator( Mikele ) -->





<!-- Date: Sun Aug 24 02:22:30.218 2008 -->
PTM came back green. It was red from previous shift. PW01 77.7 and PE10 80.0 now.
<!-- Author: Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->






<!-- Date: Sun Aug 24 03:37:42.567 2008 -->
<pre> 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid S/B3/W7/L2/C5-9 .  
 AUTO HRR will be issued   
</pre> 
Fixed by auto HRR
<!-- Author: David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->



<!-- Date: Sun Aug 24 04:15:55.727 2008 -->
<pre> 
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
Stuck Cellid S/B2/W7/L1/C3-5 .  
AUTO HRR will be issued   
</pre> 
Auto HRR isseud, fellow by L2TO, manunally HRR fixed it.
<!-- Author: Yuchul/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->

<!-- Date: Sun Aug 24 04:22:45.638 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
cal: <tt>Still that hot PMT Phi Wedge5, Eta West -11 </tt>
<!-- Author: Consumer Operator( Mikele ) -->


<!-- Date: Sun Aug 24 04:22:46.720 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
general: <tt>Check OK</tt>
<!-- Author: Consumer Operator( Mikele ) -->






<!-- Date: Sun Aug 24 04:37:06.169 2008 -->
Done TO b0pcal05, auto HRR issued.  b0pcal05 stuck during halt transition, had to be shepherded.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->



<!-- Date: Sun Aug 24 05:03:50.682 2008 -->
<pre>  
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid S/B2/W4/L4/C0-6 .  
 AUTO HRR will be issued   
</pre> 
 
L2 Decision TO after auto HRR, fixed by an additional HRR.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->




<!-- Date: Sun Aug 24 05:43:43.162 2008 -->
<pre> 
*** Node b0pcal07 in Error *** 
Last data more than 120s ago!  Node is not responsive 
</pre> 
 
Fixed with two HRRs
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->



<!-- Date: Sun Aug 24 06:01:04.260 2008 -->
<pre> 
  RXPT Error for crate/s:  
    b0svx02    
</pre> 
followed by 
<pre> 
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run      
 - record any relevant information in e-log      
 - RECOVER - RUN      
</pre> 
 
Fixed with two HRRs
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->



<!-- Date: Sun Aug 24 06:10:45.155 2008 -->
<pre> 
*** Node b0pcal07 in Error *** 
Last data more than 120s ago!  Node is not responsive 
</pre> 
 
Shepherded b0pcal07.  Followed by L2 Decision TO, fixed by HRR.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->




<!-- Date: Sun Aug 24 06:26:52.226 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
cal: <tt>As before, new hot PMT WHA at eta=-11</tt>
<!-- Author: Consumer Operator( Mikele ) -->


<!-- Date: Sun Aug 24 06:26:53.590 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
general: <tt>Checklist done.</tt>
<!-- Author: Consumer Operator( Mikele ) -->


<!-- Date: Sun Aug 24 06:26:54.908 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511>266511</A>
trigger: <tt>1.5.1.3 plot concerning XFT T2FT has cells/sucells red. The color scale corresponds to 10-6 wich is less than the 3% mentioned in the criterion. Please SciCo give your OK about that or call expert. </tt>
<!-- Author: Consumer Operator( Mikele ) -->



<!-- Date: Sun Aug 24 06:37:57.804 2008 -->
ENCP TO TAPE is red in ProcMon
<!-- Author: Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 06:39:15.198 2008 -->
ENCP TO TAPE red again in ProcMon
<!-- Author: David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->



<!-- Date: Sun Aug 24 06:59:52.581 2008 -->
Trigger inhibits from ISL HV and L00.  auto-Hockerization of silicon crate 12.  Silicon auto recovery successful.
<!-- Author: David/Yuchul :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 07:10:44.173 2008 -->
SlowControl ICICLE is red, restated icicle program.
<!-- Author: Yuchul/David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->



<!-- Date: Sun Aug 24 07:38:44.154 2008 -->
 <pre>  
  RXPT Error for crate/s:  
    b0svx02    
</pre> 
followed by 
<pre> 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run      
 - record any relevant information in e-log      
 - RECOVER - RUN      
</pre> 
Required two HRRs, now ok.
<!-- Author: David :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=266511' target=_top>266511</a>) -->


<!-- Date: Sun Aug 24 07:54:15.607 2008 -->
<font color=red>Shift Summary: </font><pre>After we got the remote CO connected, no real problems.  Still many DTO's.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   266511
Delivered Luminosity   2567.0  
Acquired Luminosity    2316.3  
Efficiency             90.2
   
</td></tr></table></pre>
<!-- Author: Rob Harr -->

<!-- Date: Sun Aug 24 07:55:20.610 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=266511 target=_runSummary>266511</A> x4110F</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [9,1119,725]</FONT></TD>
<TD ALIGN=CENTER>22:53:41</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>08:08:16</TD>
<TD ALIGN=RIGHT>   3,215,565</TD>   <TD ALIGN=RIGHT>  3,083.82</TD>
<TD ALIGN=RIGHT>  2,764.74</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:02</TD>
<TD ALIGN=CENTER>08:08:16</TD>
<TD ALIGN=RIGHT>     3,215,565</TD> <TD ALIGN=RIGHT>  3,083.82</TD>
<TD ALIGN=RIGHT>  2,764.74</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->