<!-- Title: Owl shift. Fri Jun 18, 2004 -->
<!-- SciCo: Steve Kuhlmann -->
<!-- DAQAce: Masato Aoki -->
<!-- MonAce: Else Lytken -->
<!-- CO: Seog Oh -->
<!-- OpManager: Mary Convery -->
<!-- Notes: -->
<pre>Waiting for shot setup.</pre>

<!-- Date: Fri Jun 18 00:21:33.108 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 00;22;47 comment by...Masato -->
auto HRR fixed it.

<!-- Date: Fri Jun 18 00:26:09.057 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_07). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 00;27;10 comment by...Masato -->
auto HRR worked.

<!-- Comment: Fri Jun 18 00;28;22 comment by...Masato -->
It happened again. Auto HRR worked.

<!-- Date: Fri Jun 18 00:38:01.080 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_11). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 00;38;20 comment by...Masato -->
fixed by auto HRR.

<!-- Comment: Fri Jun 18 01;07;13 comment by...Masato -->
happened again. auto HRR fixed it.
<!-- Comment: Fri Jun 18 06;01;51 comment by...Masato -->
happened again.now ok.
<!-- Date: Fri Jun 18 00:56:02.462 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate COT_PULSER_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 00;56;25 comment by...Masato -->
fixed by auto HRR.

<!-- Comment: Fri Jun 18 00;58;40 comment by...Masato -->
happened again. auto HRR worked.


<!-- Comment: Fri Jun 18 01;18;42 comment by...Masato -->
It happened again. fixed by auto HRR.


<!-- Comment: Fri Jun 18 02;03;15 comment by...Masato -->
happened again. now fine.
<!-- Comment: Fri Jun 18 02;21;58 comment by...Masato -->
again. now fine
<!-- Comment: Fri Jun 18 02;48;07 comment by...Masato -->
happened again.
<!-- Comment: Fri Jun 18 02;58;35 comment by...Masato -->
happened again.
<!-- Comment: Fri Jun 18 03;19;06 comment by...Masato -->
it happened again.now fine.
<!-- Comment: Fri Jun 18 05;04;07 comment by...Masato -->
again.now ok.
<!-- Comment: Fri Jun 18 05;43;50 comment by...Masato -->
again.
<!-- Date: Fri Jun 18 01:56:22.520 2004 -->
<pre>Lost connection to the TOF HV PC. The computer was not reponding so we rebooted it according to the instructions. It is now up and running again.</pre>
<!-- Author: Else -->

<!-- Date: Fri Jun 18 02:01:18.729 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 02;02;20 comment by...Masato -->
auto HRR worked.




<!-- Date: Fri Jun 18 02:35:15.138 2004 -->
<pre>Getting ready for shot setup: the wireless access power cords in 1RR04F and 1RR34F have been unplugged.</pre>
<!-- Author: Else -->

<!-- Date: Fri Jun 18 02:40:53.905 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 02;41;27 comment by...Masato -->
fixed by auto HRR


<!-- Date: Fri Jun 18 02:41:34.337 2004 -->
<pre>When the protons started loading, SVX AVDD B1 W8 L4 turned pink. As with B0 W0 L2, unmarking only helps for a short while, then it goes back to pink. 
For B1W8L4: nominal: 1074, current: 1105</pre>
<!-- Author: Else -->
<!-- Comment: Fri Jun 18 03;55;22 comment by...Else -->
<pre>B1W8L4 is now back to nominal.</pre>
<!-- Date: Fri Jun 18 02:52:19.307 2004 -->
<pre>Another presumably false heart beat alarm from Icicle. It happened shotly after the "done with current cycle .. .  Oracle flag is 1 " lines, between the process transition lines RECOVERED and ACTIVE.</pre>
<!-- Author: Else -->
<!-- Comment: Fri Jun 18 04;17;56 comment by...Else -->
<pre>Two times Icicle heart beats. It seemed to happen when Icicle had starting running again after hanging at the same line for a while. (not in the same place as the earlier entry, maybe it is just random). </pre>
<!-- Comment: Fri Jun 18 05;47;47 comment by...Else -->
<pre>Two more Icicle heart beat alarms. This time between the same lines as in the first entry.</pre>
<!-- Date: Fri Jun 18 03:12:16.311 2004 -->
<pre>Two alarms in a row form BSC. Everything was marked green when I opened the detector page.</pre>
<!-- Author: Else -->
<!-- Comment: Fri Jun 18 07;00;19 comment by...Else -->
<pre>This just happened again at the same time as we had big spikes in the losses.</pre>
<!-- Date: Fri Jun 18 03:37:33.910 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb16 (forwarded by FER crate COT_18). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 03;41;34 comment by...Masato -->
When we got this error,done timeout detected in crate COT_18.Auto HRR fixed it.

<!-- Comment: Fri Jun 18 04;47;10 comment by...Masato -->
We had the same error. Auto HRR worked.
<!-- Date: Fri Jun 18 03:43:12.131 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11. 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 03;43;29 comment by...Masato -->
now fine.



<!-- Date: Fri Jun 18 04:07:21.070 2004 -->
<pre>Call MCR.  Shot setup in about 45 minutes.</pre>
<!-- Author: sk -->

<!-- Date: Fri Jun 18 04:09:00.146 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 04;09;28 comment by...Masato -->
auto HRR fixed it.

<!-- Comment: Fri Jun 18 05;22;16 comment by...Masato -->
happened again.
<!-- Comment: Fri Jun 18 05;37;53 comment by...Masato -->
again.
<!-- Comment: Fri Jun 18 05;45;24 comment by...Masato -->
again.
<!-- Date: Fri Jun 18 04:13:12.437 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE: 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb11 (forwarded by FER crate CCAL_07). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed)  
 in 30 secs. 
<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 04;13;49 comment by...Masato -->
fixed by auto HRR.




<!-- Comment: Fri Jun 18 05;12;24 comment by...Masato -->
We had the same error.Now ok.
<!-- Date: Fri Jun 18 04:47:55.380 2004 -->
<pre>MCR calls.   Shot setup in 20 minutes.</pre>
<!-- Author: sk -->


<!-- Date: Fri Jun 18 04:55:29.501 2004 -->
<pre>From MCR log: 
 
04:41:20-   Tev did one more study store with bunches 24 and 25  
to look at peculiar abort loss patterns at B0. Things look  
good, and the Tev is progressing through a dry squeeze  
prior to shot setup. - Mike V.</pre>
<!-- Author: sk -->




<!-- Date: Fri Jun 18 05:17:40.339 2004 -->
<pre>Shot setup has begun.</pre>
<!-- Author: sk -->
<!-- Comment: Fri Jun 18 05;24;17 comment by...rainer -->
could you please put in some summary ACNET shift plots for the beam studies during your shift ? please also check the BLM FIFOs whether any abort during the proton only studies has been messy.  The BLM part is described in <a href='http://www-cdfonline.fnal.gov/internal/ops/svxii/runii/quench.html'> these </a> instructions (you don't need to perform the full checkout, only see whether the BLMs latched in the last 12 hours). thanks.
<p>
ps.: your EVB does not sound healthy. I would be surprised if you had a smooth startup with these recurring SCPU errors I see in the elog.

<!-- Date: Fri Jun 18 05:23:56.459 2004 -->
<pre>ProcMon Error due to TOMCAT. 
We followed the instructions to restart the process.</pre>
<!-- Author: Masato :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187' target=_top>184187</a>) -->




<!-- Date: Fri Jun 18 05:39:59.083 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-9006&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-9006' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-9007&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-9007' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-9008&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-9008' width='300'></a></center>
<!-- Author: Else -->
<!-- Comment: Fri Jun 18 05;41;01 comment by...Else -->
<pre>Rainer, is this what you wanted to see?</pre>




<!-- Comment: Fri Jun 18 07;36;08 comment by...rainer -->
more or less, thanks. the right plot, which timestamp is the right plot ?
<!-- Date: Fri Jun 18 05:52:57.944 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
ACTIVE:   
  Done TO. for crate/s: 
    b0cot18   

<!-- Author: ErrorHandler -->
<!-- Comment: Fri Jun 18 05;53;37 comment by...Masato -->
Auto HRR worked.Now fine.


<!-- Comment: Fri Jun 18 06;08;25 comment by...Masato -->
Happened again.


<!-- Date: Fri Jun 18 06:22:41.642 2004 -->
<pre>Final protons are being loaded.</pre>
<!-- Author: sk -->

<!-- Date: Fri Jun 18 06:23:04.533 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184187">184187</A>
TERMINATE: End L2_TORTURE, switch to shot setup
<!-- Author: Masato (x2080) -->



<!-- Date: Fri Jun 18 06:25:18.091 2004 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2004.06.18</td><td>06:21:47</td><td>W Inner BLM</td><td>0.53</td><td>RADS</td></tr> 
<tr><td>2004.06.18</td><td>06:21:47</td><td>W Outer BLM</td><td>0.07</td><td>RADS</td></tr> 
<tr><td>2004.06.18</td><td>06:21:47</td><td>E Inner BLM</td><td>0.50</td><td>RADS</td></tr> 
<tr><td>2004.06.18</td><td>06:21:47</td><td>E Outer BLM</td><td>1.22</td><td>RADS</td></tr> 
</TABLE> 
<strong>Integrated dosage</strong>
<!-- Author: Else -->


<!-- Date: Fri Jun 18 06:37:28.986 2004 -->
<pre>We cannot switch to AAA_SHOTSETUP. 
When we push partition button, L3 turned red. And got following error message, 
 
(MLE) b0l3pcom2.fnal.gov:main:6:27:33 AM-&gt;Error on L3 node b0l3pcom2.fnal.gov (partition 0) Fri Jun 18 06:27:32 2004 l3proxy  
  in Define (L3ProxyFarmHandler.cc:656) 
  @L3_INVALID_SUBFARM - Subfarm no 15, is marked offline and can not be used. 
All known bad subfarms: 15   
 
paging L3 
 
</pre>
<!-- Author: Masato -->
<!-- Comment: Fri Jun 18 07;31;01 comment by...Guillelmo -->
Actually the problem was in subfarm 16<p>
If one converter node is down clean up L3 when a L3 expert is not around is worse because it will try to go there, and (of course) it will fail. Aces should try to run without that subfarm, and page L3 right away.
<!-- Date: Fri Jun 18 06:56:57.047 2004 -->
<pre>pbars in</pre>
<!-- Author: sk -->



<!-- Date: Fri Jun 18 07:13:53.196 2004 -->
<pre>Scraping complete,  store 3582</pre>
<!-- Author: sk -->
<!-- Comment: Fri Jun 18 07;17;34 comment by...sk -->
<pre>L=80.65,  Tevpr=8971,  Tevpb=1134</pre>


<!-- Date: Fri Jun 18 07:20:33.627 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-9009&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-9009  width='300' align='top'></a></td><td> After scraping</td></tr></table>
<!-- Author: Else -->


<!-- Date: Fri Jun 18 07:22:32.830 2004 -->
<pre>Losses still high,  L3 broken,  experts here. </pre>
<!-- Author: sk -->




<!-- Date: Fri Jun 18 07:42:31.733 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-9010&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-9010  width='300' align='top'></a></td><td> After 2nd scraping</td></tr></table>
<!-- Author: Else -->


<!-- Date: Fri Jun 18 07:47:46.731 2004 -->
<pre>Frank did some work on vrb.  L3 may work.  Losses 
went back up...</pre>
<!-- Author: sk -->

<!-- Date: Fri Jun 18 07:52:01.043 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184188">184188</A>
TERMINATE:             
<!-- Author: Masato (x2080) -->


<!-- Date: Fri Jun 18 07:55:50.241 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:05</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Fri Jun 18 07:55:52.427 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184189">184189</A>
TERMINATE: end    
<!-- Author: Masato (x2080) -->



<!-- Date: Fri Jun 18 07:56:39.833 2004 -->
<font color=red>Shift Summary: </font><pre>Shot setup starts at 5:20. 
Scraping complete at 7:14,  store 3582 
L=80.65,  Tevpr=8971,  Tevpb=1134 
Losses too high. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   
Delivered Luminosity   177  
Acquired Luminosity    0  
Efficiency             0
   
</td></tr></table></pre>
<!-- Author: sk -->


<!-- Date: Fri Jun 18 07:59:38.579 2004 -->
<pre>There have been many SCPU-P0-E-TracerEventId errors while running  
overnight. 
 
The errors appear to be associated with  
 
b0eb11 
slot 10 
channel 8  
 
Ran some standalone tests to check the connection and did not 
see any errors. I was not able to run for an extended time. 
 
 
 
26'36" b0l3pcom1.fnal.gov:main:12:26:36 AM-&gt;Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 667706, crate 8, channel 8 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 0, should be 10. 
Tracer word = 0x00000008 (I think). 
Channel data = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000001 0x00040000 ... 
... 0x00000000 0x00000000 0x00000038 0x00000000 0x0a800028 0x55555555 0x00000008 0x80000000 --&gt;   
 
The longword before the 0x55555555 should be 0xaaaaaaaa 
 
SCPU-P0-E-BadVrbByteCount: Event 667707 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 10. 
Count = 1048575. --&gt;   
 
  
 
</pre>
<!-- Author: Frank Chlebana -->

<!-- Date: Fri Jun 18 08:01:21.021 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=184190">184190</A>
ACTIVATE: PHYSICS_2_05[4,489,465] - Going into AAA_SHOTSETUP in preparation for taking data.
<!-- Author: Mark (x2080) -->