<!-- Title: Eve shift. Wed Apr 11, 2001 -->
<!-- SciCo: Christoph Paus -->
<!-- DAQAce: Dan Cyr -->
<!-- MonAce: David Goldstein -->
<!-- CO: Barry Wicklund -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre> 
a) do calibrations 
 
b) do Rick run 
 
c) work with level2 people to integrate the level2 trigger 
 
d) perform a rate test of the DAQ, the Frank run 
 
e) gather the sytem and run it as much as possible, maybe include 
   silicon?!</pre>


<!-- Date: Wed Apr 11 16:24:17 2001 -->
<pre>The B0IMU01 crate was being continuously reset, 
by the remote reset line. This made it impossible 
to login to it, get a response via the console,
or do any data taking with this crate.

I have disconnected the reset line from this
crate for now.
 
</pre>
<!-- Author: W. Stuermer -->


<!-- Date: Wed Apr 11 16:24:22 2001 -->
<pre>have had a couple isl cooling audible alarms.  system is being 
tested/worked on for a while, so we are just going to silence the  
alarms until further notice. 
</pre>
<!-- Author: david -->


<!-- Date: Wed Apr 11 16:32:53 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>Status update on 'new run control' and calibrations: 
<P> 
- Muon calibrations work since last night 
<BR> 
- CLC QIE calibration works since 5 minutes 
<P> 
This means that the only type of calibrations under ace control 
that requires the 'old run control' is LED/XEF/FLASHPED 
</td></tr></td></table>
<!-- Author: Arnd -->


<!-- Date: Wed Apr 11 16:33:44 2001 -->
<pre>
Pcal02 is reachable from minicom, however "i" doesnt work, and we can't reboot it. We saw this in pcal00 as well, and sysreset it. Resetting pcal02 now. 
 
Seeing Heap errors in pcal01 (and others while monica is using).  
 
b0pcal01-> Monitor: interval 6000, state ACTIVE for partition 2  
Monitor: Evts sent/readout/triggered 105954/105954/105955 
Monitor: Error state:    0, VRB_Busy:    1, TV count: 0 
Monitor: readout/sent rate    195.9 Hz /    195.9 Hz, cur evt is 800 bytes 
0xe85488 (Messenger): Heap Error: 'Internal Inconsistency' at   c78828 (count = 9910) 
Monitor: interval 6000, state ACTIVE for partition 2  
Monitor: Evts sent/readout/triggered 107291/107291/107291 
Monitor: Error state:    0, VRB_Busy:    1, TV count: 0 
 
-------------------------------
</pre>
<!-- Author: dan + robin -->
<!-- Comment: Wed Apr 11 18;09;46 comment by...Robin Erbacher -->
It turns out the whole west side had gone down at 15:15 (wall tripped so the clocks for the plug go down).  On the heap useage panel it said that pcal02 had a corrupt heap like the rest of them but also said SUSP! in red with a time of 15:15.  Perhaps both pcal00 and pcal02 originally hung at 15:15 when the wall went down.  In any case it took a "reset_crate" to get them back.
Voltages look ok in voltman.
<!-- Date: Wed Apr 11 16:38:17 2001 -->
<pre>NEW RC : Apparently switching partitions in new RC caused a situation where crates were booked in partition 0, and we switched to partition 6. Then the crates were "booked" according to resource manager. We couldnt release them, had to kill RC and after a few minutes the partition (and crates) were released 
</pre>
<!-- Author: dan -->


<!-- Date: Wed Apr 11 16:39:02 2001 -->
The AC power on the west wall appeared to go down about 15:15.  The COT low voltage power was still off at 16:30.  I reset the power and it is now OK.
<!-- Author: Morris Binkley -->


<!-- Date: Wed Apr 11 17:08:35 2001 -->
<pre>Rebooted the "Corrupt heap!" cal crates (ccal00-05 and pcal 01,03-05) 
note minicom from pcal01 is "dead" 
</pre>
<!-- Author: dan -->


<!-- Date: Wed Apr 11 17:16:03 2001 -->
cmx01 and cot09 - failed to reset cards in slots. cmx i dropped, cot09 i reset and brought back into run.  
 
NOTE : BEAM data was accidentally set in this run  - pilot error.  
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111786' target=_top>111786</a>) -->
<!-- Comment: Wed Apr 11 17;17;12 comment by...dan -->
<pre>ended run at 28k events to switch to cosmic triggers AND data type to COSMIC!!!
</pre>


<!-- Date: Wed Apr 11 17:19:42 2001 -->
Doug, Francesco and I were doing some ISL tests 
which involved trying to power an ISL ladder with the coolant 
flow turned off. We forgot that the Quadlog safety interlocks  
just became operational, and trying to do this test caused the 
AC power to the silicon racks to be dropped. After talking to Rich Schmitt, we got the AC power back up (using the secret toggle switch!). We'll have to rethink doing this test without causing a rack power trip!
<!-- Author: Steven Blusk -->


<!-- Date: Wed Apr 11 17:32:28 2001 -->
<pre>I'm done with the imu and cot crates I had. TDC test should work with the new RC now.</pre>
<!-- Author: Arnd -->


<!-- Date: Wed Apr 11 17:51:09 2001 -->
<pre>BSC want to test their system and therefore would need quiet time 
without beam in the machine. I called the MCR and they told me 
that they are going to put beam into the machine very soon. In 
case they have quiet time they will call back. 
 
BSC expert wants to be called if this is before midnight 
 
 pager (630) 218-4400 
</pre>
<!-- Author: Christoph Paus -->


<!-- Date: Wed Apr 11 17:55:38 2001 -->
<pre>A "stable" run is going. Missing most of the muons (EJ is working on muon timing). I'm going to past plots of scalermon. 
</pre>
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111791' target=_top>111791</a>) -->
<!-- Comment: Wed Apr 11 18;00;40 comment by...dan -->
and cosmic data type is checked!

<!-- Date: Wed Apr 11 17:58:18 2001 -->
<center><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-614&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-614' width='300'></a><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-615&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-615' width='300'></a></center>
<!-- Author: dan -->
<!-- Comment: Wed Apr 11 17;59;12 comment by...dan -->
<pre>Scalermon plots for cosmic / highrate test.</pre>




<!-- Date: Wed Apr 11 18:37:08 2001 -->
<pre>Beam was injected in to Tev, about the same time our run halted with busy timeout. It appears to be the same VRB for the plug (eb16) 
--------------------- 
MLE) b0cot11:Messenger:6:30:17 PM->Runtime Error 4, Event 1135596: data truncated 
(MLE) b0l3gate1.fnal.gov:main:6:31:42 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 1164259, crate 0, channel 0 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 8, should be 3. 
Tracer word = 0x78781400 (I think). 
Channel data = 0x000000bb 0x78030029 0x50484144 0x000c0007 0x00000000 0x00000049 0x00000001 0x0dd60030 ... 
... 0x00000002 0x00000000 0x00000000 0xaa000000 0x55aaaaaa 0x29555555 0x78781400 0x80000000 
(MLE) b0l3gate1.fnal.gov:main:6:31:42 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 1164259, crate 0, channel 1 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 2, should be 3. 
Tracer word = 0x72721400 (I think). 
Channel data = 0x00000101 0x7203002a 0x50484144 0x000c0008 0x00000000 0x00000049 0x00000001 0x08d60030 ... 
... 0x00000003 0x00000000 0x00000000 0xaa000000 0x55aaaaaa 0x2a555555 0x72721400 0x80000000 
(MLE) b0l3gate1.fnal.gov:main:6:31:42 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 1164259, crate 0, channel 2 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 6, should be 3. 
Tracer word = 0x46461400 (I think). 
Channel data = 0x000000d8 0x4603002b 0x50484144 0x000c0009 0x00000000 0x00000049 0x00000001 0x09d60030 ... 
... 0x00000003 0x00000000 0x00000000 0xaa000000 0x55aaaaaa 0x2b555555 0x46461400 0x80000000 
(MLE) b0l3gate1.fnal.gov:main:6:31:42 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 1164259, crate 0, channel 3 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 14, should be 3. 
</pre>
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111791' target=_top>111791</a>) -->
<!-- Comment: Wed Apr 11 18;39;04 comment by...dan -->
<pre>HRR wasnt successful.</pre>
<!-- Date: Wed Apr 11 18:38:18 2001 -->
<pre>Tried to start with warm-start. Failed with same VRB errors. 
</pre>
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111797' target=_top>111797</a>) -->
<!-- Comment: Wed Apr 11 21;58;18 comment by...patrick -->
<pre>This was the VRB that was swapped today.
It looks like maybe it died...
Though it did work for a while.</pre>
<!-- Comment: Wed Apr 11 22;32;44 comment by...patrick -->
<pre>Note that this VRB only affects pcal06-11.
It would be worth running with pcal00-05</pre>
<!-- Date: Wed Apr 11 18:54:36 2001 -->
<pre>MCR called to announce that they are going to do an aperature 
scan (called 3 bumps) in the A4 area. 
 
We put up beamloss monitors and SVX rad monitors on ACNET. 
</pre>
<!-- Author: Christoph Paus :: (run <a href='/cgi-bin/runSummaryRun.cgi?111707' target=_top>111707</a>) -->


<!-- Date: Wed Apr 11 19:17:13 2001 -->
Power cycling the eb16 crate didnt help the problem. Removed all the plug crates for now. Expect the trigger experts to claim large portions of system soon. Up and running @ 350hz again....
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111799' target=_top>111799</a>) -->
<!-- Comment: Wed Apr 11 22;33;47 comment by...patrick -->
<pre>The VRB problem should only affect b0pcal06-11.
It would be worth trying to run with b0pcal00-05.</pre>
<!-- Date: Wed Apr 11 19:32:42 2001 -->
<pre>This run had a done timeout after 500k events. HRR successful - here is log. 
 
(MLE) b0ccal03:Messenger:7:30:49 PM->Event 506318: Bunchcounters in slot 17 and slot 18 disagree (at least) 
(MLE) b0ccal03:Messenger:7:30:49 PM->Runtime Error 1, Event 506318: Bunch counter mismatch, mismatch count = 1 
30'58" 1 crate/s: b0ccal03(3),  Done TO.[RXPT] 
(MLE) b0tsi02:Messenger:7:31:24 PM->DONE timeout 
</pre>
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111799' target=_top>111799</a>) -->
<!-- Comment: Wed Apr 11 20;35;10 comment by...dan -->
<pre>Ended run to give CAL to monica 1.28M events with L2 processors!!! Not one state 14 error. . Integrating Si with the calibration trigger. 

</pre>


<!-- Date: Wed Apr 11 21:09:45 2001 -->
<pre>TOF TAC calibration timing-in, using b0wcal02 and DDG module. 
Taken  runs with several CalibInterval Settings 1, 2, 3 & 4 
 
Run numbers 111800,111801,111802,111804</pre>
<!-- Author: Gervasio and Ivan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111800-111804' target=_top>111800-111804</a>) -->


<!-- Date: Wed Apr 11 21:09:52 2001 -->
<pre>New RC : has failed "partition" transition - several crates are "yellow" in replies/acks window, and after 4-5 min, i hit abt+rst and try again. Fail again, restart RC, works. </pre>
<!-- Author: dan -->



<!-- Date: Wed Apr 11 22:02:57 2001 -->
<pre>COT calibration : 
number of missing channels = 192 
number of good channels = 29597 
number of drifted good channels = 96 
ave drift = -0.201505 
no drift detected 
numBad channels = 451                                                                                 </pre>
<!-- Author: dan :: (run <a href='/cgi-bin/runSummaryRun.cgi?111812' target=_top>111812</a>) -->




<!-- Date: Wed Apr 11 23:26:56 2001 -->
<pre>Calibrations in last minutes of shift : 
 
111812 : COT (new RC) 
111816 : QIE (new RC)    
</pre>
<!-- Author: dan -->


<!-- Date: Wed Apr 11 23:34:07 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>THIS IS INTOLERABLY SLOW : 
setub0dap59:21:~>setup fer v1_03 
rc 
CDFVME_COMMON: Last updated by nahn on Thu Mar 23 13:03:15 CST 2000 
b0dap59:22:~>rc 
Run Control log file: /cdf/code-common/cdfonline/fer/log/rc_b0dap59_2001-04-11_23:28:49.log 
2001.04.11 23:28:49 Attempting connection to <tcp:b0dau30:5101> RTserver. 
2001.04.11 23:28:50 Connected to <tcp:b0dau30:5101> RTserver. 
2001.04.11 23:28:51 Fetching from HdwDb... 50846 ms 2001-04-11 23:27:13.0 
2001.04.11 23:29:42 Fetching from RunDb... 137780 ms 
2001.04.11 23:32:01 Connected RunControl to RTServer 
2001.04.11 23:33:15 Selected Partition: 1 
2001.04.11 23:33:15 ListenToEndOfRun: </partition-1/endRun/*> 
--------------------------- 
 
184s to start RC is REALLY BAD!!!!</pre></td></tr></td></table>
<!-- Author: dan -->


<!-- Date: Wed Apr 11 23:57:28 2001 -->
<font color=red>Shift Summary: </font><pre>Shift Summary 
============= 
 
 a) main time was given to Monica and Co for Level 2 development we 
    ran with level2 (no silicon) for roughly 2 hours without problems 
  
 b) plug PCAL10 fails after a million events and cannot be integrated 
    even after power cycling -- needs to be fixed!! 
 
 c) we started with the calibrations and were very much slowed down by 
    the speed of the computers .... network problems, other jobs ?? 
 
 d) did not have time to do the Rick Run (leave it for the owl) 
 
 e) ran with high rate (350Hz/60kB) and could not find a culprit 
    for being regularly late on the done 
 
 f) we had a short silicon integration run which worked only for part of the  
    system 
 
 g) the HV alarms are back at a regular rate 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   111772-111818
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             100 %
   
</td></tr></table></pre>
<!-- Author: Christoph Paus -->


<!-- Date: Thu Apr 12 00:00:57 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>The L2/track prototype board (Jane Nachtman's board, which 
has faithfully provided SVTD readout since September) 
is now in b0svt06 slot 13, no longer b0l2de01.  When we 
next read out SVT in DAQ tests (hopefully soon) we will 
use an SVT crate for the readout, not an L2 crate.  This 
allows L2 experts to use b0l2de01 for its intended purpose, 
i.e. L2 testing. 
</td></tr></td></table><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Ashmanskas -->


<!-- Date: Thu Apr 12 00:06:50 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-616&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-616  width='300' align='top'></td><td>THIS is the status of our RC. </a></td></tr></table><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: dan -->
<!-- Comment: Thu Apr 12 00;20;58 comment by...disgruntled ace -->
it was about 3 minutes till we could pick a partition.
<!-- Date: Thu Apr 12 00:20:12 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>BTW : Partitioning for TOF QIE took EIGHT minutes. 
</td></tr></td></table><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: dan -->