<!-- Title: Owl shift. Sat Apr 28, 2001 -->
<!-- SciCo: Dan Amidei/Alan Sill -->
<!-- DAQAce: Francesco Palmonari -->
<!-- MonAce: Nancy Lai -->
<!-- CO: M.Hennecke, Javier Fernandez -->
<!-- OpManager: Kevin Burkett -->
<!-- Notes: -->
<pre>In store, looking at tracks! 
Plan is to keep store till 8:30.  Run this all night, 
if and when it drops, do calibrations.</pre>


<!-- Date: Sat Apr 28 00:54:54 2001 -->
Have our 500K early Myron events -- ending run.  Next one 
will be normal mode.
<!-- Author: Alan Sill -->


<!-- Date: Sat Apr 28 01:13:41 2001 -->
<pre>PHA alarm - Crate 11.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?113485' target=_top>113485</a>) -->
<!-- Comment: Sat Apr 28 03;06;58 comment by...Nancy -->
<pre>Again.</pre>
<!-- Comment: Sat Apr 28 03;28;16 comment by...Nancy -->
<pre>PHA alarm in Crate 8 this time.</pre>
<!-- Comment: Sat Apr 28 04;48;43 comment by...Nancy -->
<pre>PHA alarm in Crate 11.</pre>
<!-- Comment: Sat Apr 28 05;54;49 comment by...Nancy -->
<pre>Again in Crate 11.</pre>
<!-- Date: Sat Apr 28 01:15:18 2001 -->
<pre>[>> ended run 113484 (after 526579 events) 	] 
[   -> made record in the Good Run List		] 
 
>> started run 113485 : 
 
-> run config: 36X36_DEFAULT 
 
-> trig table: 36X36_1[10,111] 
 
-> MyronMode: FALSE	L1Early: FALSE  
 
->	SILICON:	5 wedges 
	COT:		SL4 below Vop 
	CAL:		OK 
	MUON:		OK</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113485' target=_top>113485</a>) -->
<!-- Comment: Sat Apr 28 02;52;59 comment by...f. -->
<pre>NB: run activate time is 00:59:43.0</pre>
<!-- Date: Sat Apr 28 01:34:45 2001 -->
<pre>Various ACnet stuff is not working: 
 
- can't save plot from D44 (steps to save it seem to work, 
  but plot does not show up on adcon.fnal.gov/userb/pic.capture.cdf 
  page even if we are very careful to specify that location) 
 
- Accelerator display of beam intensities on TV monitors is 
  not working -- displays "waiting for supercycle reset" 
  message
</pre>
<!-- Author: AFS -->
<!-- Comment: Sat Apr 28 02;41;29 comment by...AFS -->
TV display information came back about 20 minutes later.
<!-- Comment: Sat Apr 28 04;21;09 comment by...AFS -->
TV display shows luminosity values about twice as high as
the ones we are reporting, which come vrom B0ILUM...
<!-- Date: Sat Apr 28 01:36:30 2001 -->
Luminosity now 2.5E29 (B0ILUM on ACNet page C65)
<!-- Author: AFS -->


<!-- Date: Sat Apr 28 01:48:00 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-805&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-805  width='300' align='top'></td><td>.</a></td></tr></table>
<!-- Author: francesco -->
<!-- Comment: Sat Apr 28 01;51;32 comment by...f. -->
<pre>Some broken sockets connections errors
in the L3 converters nodes.
(... never seen these errors messages)</pre>


<!-- Date: Sat Apr 28 02:01:26 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-806&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-806  width='300' align='top'></td><td>Standalone silicon tracks finding the beam</a></td></tr></table>
<!-- Author: Aaron -->


<!-- Date: Sat Apr 28 02:09:32 2001 -->
<pre>CPR east tripped.  Put on standby and then turned voltage on  
again.</pre>
<!-- Author: Nancy -->


<!-- Date: Sat Apr 28 02:31:27 2001 -->
<pre>Lost connection to S4 3 times already this shift.  Connection 
to S1 lost 5 minutes ago.  </pre>
<!-- Author: Nancy -->
<!-- Comment: Sat Apr 28 02;58;25 comment by...Nancy -->
<pre>S4 connection lost again.</pre>
<!-- Comment: Sat Apr 28 02;59;46 comment by...Nancy -->
These connection loss warnings are all on the computer cdf_solenoid2.
<!-- Comment: Sat Apr 28 03;02;59 comment by...AFS -->
(This is as reported on the IFIX computer on the northwest rack, 2RR06B, which I think is note solenoid2?  The other IFIX computers
in the control room do not seem to be having trouble communicating with the server S1 - S4 nodes - odd.)

<!-- Comment: Sat Apr 28 07;43;46 comment by...Nancy -->
<pre>Lost connection to S4 on solenoid2 computer roughly once an 
hour or so.</pre>
<!-- Date: Sat Apr 28 02:42:36 2001 -->
<pre>COT tripped in SL 4.  Aseet notified.</pre>
<!-- Author: Nancy -->
<!-- Comment: Sat Apr 28 02;47;53 comment by...Nancy -->
<pre>Turning COT back on.  Make sure that SL 4 goes to 98.5% and 
not 100% of HV setting for COT.</pre>


<!-- Comment: Sat Apr 28 02;56;49 comment by...AFS -->
Trip was on SL4 wire S5.  Watched to verify that it returned
to 98.5% upon turning it back on, which it did.






<!-- Date: Sat Apr 28 03:14:01 2001 -->
<pre>>> one CPU in processor node b0l3087 die</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113485' target=_top>113485</a>) -->


<!-- Date: Sat Apr 28 03:18:08 2001 -->
</pre>Found some useful information about the HV monitor warning  
messages and other FAQ's of monitoring aces at: </pre>
 
<a href=http://www-cdf.fnal.gov/internal/people/links/HyunsooKim/muon/hvmon_recovery.html>http://www-cdf.fnal.gov/internal/people/links/HyunsooKim/muon/hvmon_recovery.html</a> 
 
<pre>It's also good for a few late-night laughs as well....</pre>
<!-- Author: Nancy -->



<!-- Date: Sat Apr 28 03:43:01 2001 -->
<center><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-807&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-807' width='300'></a><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-808&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-808' width='300'></a><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-809&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-809' width='300'></a><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-810&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-810' width='300'></a></center>
<!-- Author: francesco -->
<!-- Comment: Sat Apr 28 03;44;18 comment by...francesco -->
<pre>&gt;&gt; trig. rates for run 113485</pre>

<!-- Comment: Sat Apr 28 03;59;18 comment by...Alan Sill -->
Luminosity was about 2E29 for this entry.


<!-- Date: Sat Apr 28 04:15:35 2001 -->
<pre>&gt;&gt; L3: a converter node and all the column of processors 
   turned to glod: they are no more monitored   </pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113485' target=_top>113485</a>) -->




<!-- Date: Sat Apr 28 05:09:48 2001 -->
<pre>&gt;&gt; run 113485 end after 1119653 events for BUSY TIMEOUT	 
 
-&gt; try to HRR 3 times but didn't help</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113485' target=_top>113485</a>) -->


<!-- Date: Sat Apr 28 05:24:56 2001 -->
<pre>&gt;&gt; start a new run 113486</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113486' target=_top>113486</a>) -->


<!-- Date: Sat Apr 28 05:45:33 2001 -->
<pre>&gt;&gt; cannot realy be sure of what causes the previous  
   busy timeout in run 113485 
 
   Try to report error that cause the timeout 
 
54'12" triggered readout sent events mismatch[EOR]b0dau32.fnal.gov:csl_mon_send:2:47:43 AM-&gt; 
CSL received a "BAD EVENT" in partition 4.[MLW]1 crate/s: b0svx02(112),  busy.[RXPT] 
(MLE) b0tsi02:Messenger:4:54:28 AM-&gt;BUSY timeout 
(MLE) b0tsi02:Messenger:4:54:32 AM-&gt;BUSY timeout 
(MLE) b0tsi02:Messenger:4:54:36 AM-&gt;BUSY timeout 
(MLE) b0tsi02:Messenger:4:54:40 AM-&gt;BUSY timeout 
(MLE) b0tsi02:Messenger:4:54:44 AM-&gt;BUSY timeout 
(MLE) b0tsi02:Messenger:4:54:48 AM-&gt;BUSY timeout 
... 
... 
... 
 
(much more in logfile of the run...)</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?ex 113485' target=_top>ex 113485</a>) -->


<!-- Date: Sat Apr 28 05:46:43 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-811&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-811  width='300' align='top'></td><td>fp</a></td></tr></table>
<!-- Author: fp -->
<!-- Comment: Sat Apr 28 05;48;02 comment by...fra. -->
<pre>Problems with SCPU ? here the rates we got while trying to HRR</pre>


<!-- Comment: Sat Apr 28 06;31;55 comment by...AFS -->
<pre>Wonder whether these are the symptoms of ATM switch problems
we have been asked to look for...</pre>
<!-- Comment: Sat Apr 28 17;08;26 comment by...Steve Tether -->
No, this wasn't the switch.
<!-- Date: Sat Apr 28 05:51:26 2001 -->
<pre>&gt;&gt; run 113486: one CPU of a processor node in the farm  
   is dead (in b0l3047)</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113486' target=_top>113486</a>) -->



<!-- Date: Sat Apr 28 06:12:08 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-812&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-812  width='300' align='top'></td><td>cc</a></td></tr></table>
<!-- Author: cc -->
<!-- Comment: Sat Apr 28 06;29;43 comment by...francesco -->
<pre>** sorry for the question:
   run 113484 and 113485 are linked toghether in the
   CSL logging plot is this expected ? (run 113485 
   started from IDLE)
   [run 113485 starts at 00:59:43.0 --&gt; finish at 05:11:46
    and recorded ~1.1 10^6 events not ~1.6 10^6 ]

</pre>
<!-- Date: Sat Apr 28 06:12:55 2001 -->
<pre>COT tripped: SL 7, wire S6.  Aseet notified and says to turn  
voltage back on.</pre>
<!-- Author: Nancy -->
<!-- Comment: Sat Apr 28 06;24;03 comment by...Nancy -->
<pre>COT tripped again immediately.  Same SL and wire.  Bob Wagner
paged at Aseet's suggestion.  Bob says to reset it again.  We
do so, but it trips again.  There is nothing unusual that we 
can see from the COT HV display for SL 7, wire S6.  We will 
wait for Bob, who will be here shortly.  </pre>


<!-- Comment: Sat Apr 28 06;54;32 comment by...Nancy -->
<pre>Bob is here now to take care of the COT.</pre>
<!-- Date: Sat Apr 28 06:29:22 2001 -->
<pre>Proton intensity is still at about 1800E9 on the TV display, 
but antiprtons are down to about 8E9 now.  ==&gt; most of the 
drop in luminosity may be due to antiproton lifetime...</pre>
<!-- Author: Alan Sill -->





<!-- Date: Sat Apr 28 07:03:09 2001 -->
<pre>Bob Wagner is here working on SL7</pre>
<!-- Author: Dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?113486' target=_top>113486</a>) -->


<!-- Date: Sat Apr 28 07:19:35 2001 -->
Bob W. says he has slightly raised a current limit in SL7; it is 
now ramped back up.  We are continuing the run still in progress.
<!-- Author: Alan Sill :: (run <a href='/cgi-bin/runSummaryRun.cgi?113486' target=_top>113486</a>) -->


<!-- Date: Sat Apr 28 07:22:21 2001 -->
<pre>&gt;&gt; L3: 8 converters are red (presence of an error) 
 
-&gt; checking their status shows some inputs failures 
   (~ 1/15000)</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113486' target=_top>113486</a>) -->


<!-- Date: Sat Apr 28 07:28:16 2001 -->
<pre>&gt;&gt; end run 113486 after 556212 events 
</pre>
<!-- Author: francesco :: (run <a href='/cgi-bin/runSummaryRun.cgi?113486' target=_top>113486</a>) -->
<!-- Comment: Sat Apr 28 07;31;04 comment by...AFS -->
<pre>Ended run now that COT looks like SL7 will stay up.
Will start a new one.</pre>
<!-- Date: Sat Apr 28 07:29:30 2001 -->
<pre>We cannot start the Luminosity Monitor. The error given by the script was: 
 
 
b0dap51:31:~&gt;./startMon.sh LumMon 4 
  
This script should be executed by a Consumer Operator 
  
./startMon.sh: [-e: command not found 
starting LumMon partition 4 on b0dap04 
  
to view the results of consumer monitors, 
go to a terminal where you like to display, 
open a window, then type: 
  
source ~cdfsoft/cdf2.cshrc 
setup cdfsoftb0 
HistoDisplayMain                         
b0dap51:32:~&gt;/usr/X11R6/bin/xterm^Q: Command not found.       
                   </pre>
<!-- Author: Javier -->




<!-- Date: Sat Apr 28 07:44:30 2001 -->
<pre>&gt;&gt; Busy timeout while starting the run 
 
148 /cdf/code-common/cdfonline/errmon/log/daq % more errorfile113487.log 
Error Handler Log File 
All timestamps are referred to CT 
CT: 4/28/2001, 7:34'33" 
Partition:4 Run number: 113487 (0x1bb4f) 
(RC) Config -&gt; READY 
(RC) Activate -&gt; ACTIVE 
(MLE) b0xtrp00:Messenger:7:38:32 AM-&gt;Event 36: Bunchcounters in slot 6 and slot 15 disagree(at least) 
(MLE) b0xtrp00:Messenger:7:38:32 AM-&gt;Runtime Error 1, Event 36: Bunch counter mismatch, mism 
atch count = 1 
38'10" 1 crate/s: b0svx02(112),  busy.[RXPT] 
(MLE) b0tsi02:Messenger:7:38:27 AM-&gt;BUSY timeout 
(MLE) b0tsi02:Messenger:7:38:31 AM-&gt;BUSY timeout 
 
</pre>
<!-- Author: f :: (run <a href='/cgi-bin/runSummaryRun.cgi?113487' target=_top>113487</a>) -->
<!-- Comment: Sat Apr 28 07;56;09 comment by...fra. -->
<pre>&gt;&gt; (Suggested also by the SAFE Check Scram Net Ring)
    -&gt; b0eb13 with suspended proc. 
-&gt; decided to clean up of the EVB  </pre>


<!-- Date: Sat Apr 28 07:58:50 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-813&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-813  width='300' align='top'></td><td>luminosity...</a></td></tr></table>
<!-- Author: Alan Sill -->


<!-- Date: Sat Apr 28 08:01:07 2001 -->
<font color=red>Shift Summary: </font><pre>Shot 49 still in progress 
 
We inherited Run 113484 from swing. 
36x36_1[10,111] Early Myron 
End with 526K events ca 1am. GOOD RUN 
 
Run 113485 
36x36_1[10,111] NO Myron 
L ~ 0.2E30, %live ~ 50% 
End with ~1.1M events GOOD RUN 
 
Run 113486 36x36_1[10,111] 
COT SL 7 off for part of this 
 
Run 113487 36x36_1[10,111] 
running as we leave, COT all OK  
 
Problems:  
 
1. Sporadic PHA trips in crate 11 and 8. COT trips in SL4,7 
2. In run 113485, some L3 converters went red with 
   socket problems, but kept running 
   Eventually L3 crashes on some kind of BUSY TIMEOUT 
3. Connection loss warnings on the computer cdf_solenoid2. 
   Cryo guys came to look, will put  
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   113485 - 113487
Delivered Luminosity   6  pb-1
Acquired Luminosity    3.8  pb-1
Efficiency             65%
   
</td></tr></table></pre>
<!-- Author: Dan Amidei, Alan Sill -->


<!-- Date: Sat Apr 28 08:22:52 2001 -->
Busy condition from VRB_SVX_02 after 84K events.  Will try HRR.  See errors in merlin window about bad Tracer id and data truncated from one of the CTO crates.<font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Peter -->
<!-- Comment: Sat Apr 28 08;49;34 comment by...patrick -->
<pre>This was the b0l1cl03 crate:


(MLE) b0l3gate1.fnal.gov:main:8:20:15 AM-&gt;Host b0eb12.fnal.gov, task tRec_0
SCPU-P4-E-TracerEventId: Event 84300, crate 131, channel 3 has either bad Tracer ID or bad markers around Tracer word.
 Tracer word event ID is 7, should be 12.
Tracer word = 0x02f7fc83 (I think).
Channel data = 0x00000347 0x540c0464 0x58464644 0x00060004 0x00000000 0x00000193 0x00000003 0x00000008 ...
... 0x00000000 0x00000000 0x02fff700 0x00ffe3d8 0x007ff0f0 0x00a7fc1c 0x02f7fc83 0x00fbf6f8
(MLE) b0l3gate1.fnal.gov:main:8:20:15 AM-&gt;Host b0eb12.fnal.gov, task tRec_0
SCPU-P4-E-VrbHeader: Dump of header words for event 84300 from VRB in slot 10:
0x00000000 0x00004ca0 0x000b004b 0x05540d1c 0x08200bd4 0x08200d1c 0x08200000 0x08b00000
                                                   ---
CT: 4/28/2001, 8:20'5"

This number should have been 0d1c, a false end of data
was detected by the VRB.

</pre>
<!-- Date: Sat Apr 28 08:38:36 2001 -->
MCR called, they are dumping the beam.  Petra turns off the silicon systems, bob turns off the COT, I turned off the TOF and Alexei turned off the muon systems.  We call back and tell them we've ready. 
 
<p>Run 113488 ended before all this, with 125k events.<font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Peter -->