<!-- Title: Owl shift. Fri Oct 13, 2000 -->
<!-- Leader: Rick Tesar-X -->
<!-- Ace: Peter Wittich -->
<!-- Co1: Antoni Munar -->
<!-- Co2: Jonatan Piedra -->
<!-- Co3: Nigel Lockyer (SOS1) -->
<!-- Co4: Dave Toback (SOS2) -->
<!-- Co5: &nbsp; -->
<!-- Notes: -->
<h1>Continuing run plan from evening shift:</h1> 
<li>MINIMUM BIAS (bit 21) 
<li>2_JET_4_TEST (bit 15) 
<li>CMU_LOW_PT_DPHI_120_TEST (bit41) 
<li>COMMISSIONING_A 
<li>COMMISSIONING_B (this one was done at the evening shift.) 
<h3> Each of these runs should be with </h3> 
<li>Early Myron Mode 
<li>L3 Reconstruction (L3 TAG=46) 
<li>100K events  


<!-- Date: Fri Oct 13 00:38:00 2000 -->
<pre>Running COMMISSIONING_A, taken over from evening shift. 
 
We have a few reformatter errors during the run.</pre>
<!-- Author: Peter Wittich :: (run 101267) -->
<!-- Comment: Fri Oct 13 00;58;09 comment by...Peter W. -->
<p>Get a BUSY condition in XPT from crate VRB_SVX_02 (RXPT 
timeout), after 56601 events.  Try HRR, get 11 more events and 
the same error re-occurs.  End the run.

<p>EVB proxy is dead, according to the ACE control panel (message is, "Not running  ZMconverter, zwgc"). Run "Clean up EVB".  Start new run after this.
<!-- Comment: Fri Oct 13 01;41;31 comment by...dave -->
<pre>After cleaning up evb, we start run 101268 and get RXPT timeout
after 71 events.  The error is again svx02, with following:

   SCPU-P0-E-BuffeOverflow: Data from VRB in slot 13 would not
      fit in remaining buffer space.  Buffer size 51200, event
      size 27824
   1 crate/s: b0svx02(112), busy [RXPT]

After HRR, get another 11 events (same as before), then get RXPT
timeout again.  Going to try one more time from scratch:

   --> new run control
   --> reboot tsi00, svx02
   --> clean evb, L3

After this, a new run was started but we now get a Done timeout
from CMU_00, after 795 events.  HRR works this time (run 101269).
</pre>
<!-- Date: Fri Oct 13 00:50:24 2000 -->
I recommend rebooting all SVT crates at soonest opportunity. 
I found a Makefile bug that caused inconsistently-compiled 
code to be loaded into crate CPUs.  Now fixed. 
<p> 
If anything weird happens, feel free to call 773-263-5249.
<!-- Author: Bill Ashmanskas -->
<!-- Comment: Fri Oct 13 01;00;20 comment by...Peter Wittich -->
We're in between runs, so we take this opportunity to reboot b0svt01, b0svt02, b0svt06.



<!-- Date: Fri Oct 13 01:59:57 2000 -->
<pre>After 28272 events, run 101269 finally got RXPT timeout for 
crate svx02.  Prior to this, had gotten several reformatter 
errors linked to this crate.</pre>
<!-- Author: dave :: (run 101269) -->
<!-- Comment: Fri Oct 13 02;20;05 comment by...dave -->
<pre>At this point, we are removing SVX so that we can finish the
different trigger table studies, after which we will try to
reintegrate.  We are also attempting to add back in PCAL04.
Our new run is 101270:

   Trigtype: COMMISSIONING_A[1,46]
   SvxSet: (none)
   TS flags: Myron(L1Early)
   Output: HEVB
   Subfarms: 8
   Crates:
      CCAL (all), WCAL (all), PCAL (all)
      CLC
      CMP, IMU00-01, CMU00, MUTRIG
      XFT_FINDER00-05, XFT_XTRP_00

   Rates:  bit 15 =  5.6
           bit 23 = 19.3
           evt rate = 29
</pre>


<!-- Date: Fri Oct 13 03:23:07 2000 -->
<pre>Ended run 101270 after 101635 events, now moving onto 
CMU_LOW_PT_DPHI_120_TEST[1,46] trigger table.  Removing 
CLC for Alexei. 
 
   Run 101271:  bit 41 = 14.7 Hz 
                ev rate = 29 ev/sec</pre>
<!-- Author: dave :: (run 101271) -->
<!-- Comment: Fri Oct 13 03;34;52 comment by...dave -->
<pre>Noticed ERROR LEDS were on in CLK00 D0 PCC module (SYNC missing,
SYNC timing, RF missing).  Hopefully have just not been cleared
from a previous problem.  Will clear at end of this run.</pre>


<!-- Comment: Fri Oct 13 03;51;12 comment by...dave -->
<pre>For this run, get RXPT timeout after 29068 events.  Get:

   Runtime error #: see wrong event number (5) in L2 message

in many crates, and:

   Runtime error #: No valid header for TDC in slot #

for:

   cmu00 slot 7,10,13,17
   cmp00 slot 6,15

Ended this run and reset module in 2RR221 (as stated above).
Started another run (101272).
</pre>
<!-- Comment: Fri Oct 13 07;06;29 comment by...patrick -->
<pre>The message:
Runtime error #: see wrong event number (5) in L2 message

in many crates

general means the clock has glitched.</pre>
<!-- Comment: Fri Oct 13 07;56;49 comment by...SCN -->
Clock glitch can also cause reformatter errors and 
vrb byte count errors in the silicon (see above)...
<!-- Date: Fri Oct 13 03:35:07 2000 -->
<pre>We observe our first beam of the shift in the Tevatron.   
Ch 13 message indicates Tev studies and PBAR stacking.</pre>
<!-- Author: R.J. Tesarek :: (run 101271) -->



<!-- Date: Fri Oct 13 04:27:12 2000 -->
<pre><br><b>CLC crate:</b> 
<br> 
Alexei Safonov and Ivan Furic take the CLC crate.  TOF PMT 
signals are removed from the crate.  Alexei re-establishes 
the default CLC configuration and recalibrates. 
<br></pre>
<!-- Author: R.J. Tesarek -->


<!-- Date: Fri Oct 13 04:48:35 2000 -->
<pre>Run 101272 ended with 100464 events.  This run is the two muon 
trigger (CMU_LOW_PT_DPHI_120_TEST, bit 41). 
 
Trigger rate for bit 41 was 16.5 Hz (event rate was 29 Hz).</pre>
<!-- Author: Peter Wittich :: (run 101272) -->


<!-- Date: Fri Oct 13 04:52:27 2000 -->
<pre><br><b>BMU Trip:</b> 
<br>Outgoing SOS discovers BMU tripped off.  Turn off and ramp 
up to standby. 
<br></pre>
<!-- Author: R.J. Tesarek :: (run 101275) -->


<!-- Date: Fri Oct 13 04:56:14 2000 -->
<pre>Starting run 101275: 
 
   Trigtype: 2_JET_4_TEST[1,46] 
   SvxSet: (none) 
   TS flags: Myron(L1Early) 
   Output: HEVB 
   Subfarms: 8 
   Crates: 
      CCAL (all), WCAL (all), PCAL (all) 
      CLC 
      CMP, IMU00-01, CMU00, MUTRIG 
      XFT_FINDER00-05, XFT_XTRP_00 
 
   Rates:  bit 15 =  6.0 
           evt rate = 20 
</pre>
<!-- Author: dave :: (run 101275) -->
<!-- Comment: Fri Oct 13 05;47;06 comment by...Dave -->
Run 101275 gets RXPT timeout after 50569 events from pcal04,
HRR gives 15 more events, then same timeout.  pcal04 crate
gave:
<pre>
   Data size = 4460
   TRACER_enableDone: Front End Fatal Error (-1): Tracer done 
      not set
   Runtime Error 0, event 50569: Tracer FIFO more than half full
   Got error message through error_q
</pre>



<!-- Comment: Fri Oct 13 05;49;20 comment by...Peter Wittich -->
This is the same error we saw  yesterday, which went away when we got rid of the PESD banks in this crate. <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=162&anchor=06:50:56 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a>

<!-- Comment: Fri Oct 13 05;57;37 comment by...dave -->
<pre>trying again after pcal04 reboot, run 101277</pre>


<!-- Comment: Fri Oct 13 06;05;39 comment by...dave -->
<pre>The previous run 101276 was an attempt at a coldstart only,
after ending run 101275.  It failed (timeout on pcal04) after
about 70 events.</pre>
<!-- Date: Fri Oct 13 06:01:21 2000 -->
<br><b>HVMON:</b> 
<br> 
Randomly looking at monitors, look at the CMU page and learn 
that HVMON is not running for CMU.  Message displayed when 
bringing bringing up the CMU display is: 
<pre> 
  HVMON++ DOES NOT SEEM TO BE RUNNING, CHECK THE LOCAL PC. 
</pre> 
The HV PC on the first floor appears to be functioning correctly 
(currents updating).  Try rebooting the  
There are no clear instructions on how to reboot the machine so 
we try logging stopping all the processes and logging out.  Upon 
logging back in, *NO* HV status.  When bringing up the CMU page, 
we get the message: 
<pre> 
    HVMON++ Intentionally down for now, check back later. 
</pre> 
In going through the web pages documenting the MCS(slow controls) 
machine, we find no documentiation on how to reboot the machine 
or how to recover the updating HV status page.  
<br>
<!-- Author: R.J. Tesarek :: (run 101275) -->



<!-- Date: Fri Oct 13 06:47:25 2000 -->
<pre>Run 101277 diead after 46203 events with the same PCAL04 error. 
As we now have roughly 100K events from the two last runs, 
we will move on to the next trigger table.</pre>
<!-- Author: Peter Wittich :: (run 101277) -->


<!-- Date: Fri Oct 13 06:50:23 2000 -->
<pre>During a slow point in the shift, I stole a script which lists the disk  
space allocated by the top 10 files/subdirectories in some parent directory. 
When I run this scrip on the /cdf/people1 and /cdf/people2 directories, 
I get a list of the top 10 disk users and the amount of space they are using. 
Below are my results for the two main CDF directories.  If you find your 
name on the list, please make an attempt to reduce your disk usage.   
 
Top 10 /cdf/people1 Users on Fri Oct 13 02:34:34 CDT 2000: 
       Usage is in kbytes 
 
5418028 cdfdaq 
1733452 chchung 
1580864 kovalev 
1430048 csl 
1356112 chlebana 
1264296 cdf_cal 
1253432 bishai 
898904  cdf_clc 
818428  litvinse 
800688  mkruse 
 
Top 10 /cdf/people2 Users on Fri Oct 13 01:14:14 CDT 2000: 
       Usage is in kbytes 
 
5213992	safonov 
2569128	slaughte 
2117388	stadie 
1797684	pamaral 
1433816	pauletta 
1327804	vejcik 
1251564	smwang 
1150784	nlai 
1045028	nachtman 
1030608	phgris 
</pre>
<!-- Author: R.J. Tesarek -->


<!-- Date: Fri Oct 13 06:54:00 2000 -->
<pre>Bob Kephart calls- they are going to power the solenoid.</pre>
<!-- Author: Peter Wittich -->



<!-- Date: Fri Oct 13 07:12:12 2000 -->
<br><b>HVMON:  The saga continues...</b> 
<br>Mark Knapp comes in and helps explain the issues with HVMON. 
The shocked shift crew learns that the only times that HVMON 
updates the control room screen updates is on HVMON startup 
and shutdown of the process on the PC located on the first floor. 
<br> 
<br>We also learn that the current message, 
<pre> 
HVMON++ Intentionally down for now, back up later. 
</pre> 
means that someone shut down the program.  Since no one notified 
the shift crew, we can only assume that this was done remotely. 
<br> 
<br><i>Editorial comment: 
<br>It strikes me as a bad idea to control HV and other critical 
systems remotely.  This fosters miscommunication with the shift 
crew which could easily result in the loss of data or worse, a 
potential safety hazzard. 
<br></i>  
<!-- Author: R.J. Tesarek -->


<!-- Date: Fri Oct 13 07:12:38 2000 -->
<pre>Run 101279 started, which continues from 101277 after integrating 
SVX: 
 
   Trigtype: 2_JET_4_TEST[1,46] 
   SvxSet: FIBTEST 
   TS flags: Myron(L1Early) 
   Output: HEVB 
   Subfarms: 8 
   Crates: 
      CCAL (all), WCAL (all), PCAL (all) 
      CLC 
      CMP, IMU00-01, CMU00, MUTRIG 
      XFT_FINDER00-05, XFT_XTRP_00 
      SVX 
 
   Rates:  bit 15 =  5.8 
           evt rate = 21 
</pre>
<!-- Author: DAVE :: (run 101279) -->



<!-- Date: Fri Oct 13 08:01:29 2000 -->
<font color=red>Shift Summary: </font><pre> 
Owl Shift Summary 
~~~~~~~~~~~~~~~~~ 
Plan: 
  Take the following data sets: 
   o MINIMUM BIAS              100k 
   o 2_JET_4_TEST              100k 
   o CMU_LOW_PT_DPHI_120_TEST  100k  
   o COMMISSIONING_A           100k 
   o COMMISSIONING_B           100k (done by previous shift) 
 
Execution: 
  All runs taken except for MIN BIAS due to the lack of beam  
most of the shift.  Runs taken are listed in the table below. 
A few problems occured during the shift and are noted above. 
</pre> 
 <h2>
Runs during the OWL shift:</h2>

<center><table BORDER COLS=4 WIDTH="100%" NOSAVE >
<tr ALIGN=CENTER NOSAVE>
<td ALIGN=CENTER NOSAVE><b>Run Number</b></td>

<td><b>Trigger Table</b></td>

<td><b>Number&nbsp; of Events</b></td>

<td><b>Event Rate (Hz)</b></td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101267</td>

<td NOSAVE>COMMISSIONING_A</td>

<td>56K</td>

<td>7.7, 29.1, 18.2
<br>(Bits 15, 21, 23)</td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101269</td>

<td>COMMISSIONING_A</td>

<td>28K</td>

<td>5.7,0.0,18</td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101270</td>

<td>COMMISSIONING_A</td>

<td>100K</td>

<td>5.5,0.0,19.3</td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101271</td>

<td>CMU_LOW_PT_DPHI
<br>_120_TEST</td>

<td>29K</td>

<td>14.7 (bit 41)</td>
</tr>
<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101272</td>

<td>CMU_LOW_PT_DPHI
<br>_120_TEST</td>

<td>100K</td>

<td>16.2 (bit 41)</td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101275</td>

<td>2_JET_4_TEST</td>

<td>50K</td>

<td>6.0&nbsp; (bit 15)</td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101277</td>

<td>2_JET_4_TEST</td>

<td>46K</td>

<td>6.5 (bit 15)</td>
</tr>

<tr ALIGN=CENTER NOSAVE>
<td NOSAVE>101279</td>

<td>2_JET_4_TEST</td>

<td></td>

<td>5.6 (bit 15)</td>
</tr>
</table></center>

<p>Note:&nbsp; After run 101269, we removed the SVX and SVT from the run
configuration as it was causing timeouts - see the shift report <a href="http://www-b0.fnal.g
ov:8000/cgi-bin/elog.pl?nb=2000&action=view&page=165&anchor=01:59:57">entry</a>
for more details.&nbsp; We included the SVX again starting with run 101279.


<!-- Author: R.J. Tesarek -->