<!-- Title: Eve shift. Fri Jun  8, 2001 -->
<!-- SciCo: Alan Sill -->
<!-- DAQAce: Michael Riveline -->
<!-- MonAce: Koji Ikado -->
<!-- CO: M.Dell'Orso/S.Tourneur -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Accelerator recovering from quench.  L2 studies are done. 
Silicon will integrate soon.  RC crashed at beginning of shift.</pre>


<!-- Date: Fri Jun 8 16:17:30 2001 -->
Silicon integrating.  Larry Nodulman will take opportunity to download new code into CCAL04 crate for testing.
<!-- Author: Alan Sill -->


<!-- Date: Fri Jun 8 16:20:15 2001 -->
Bad temperature probe on 1RR18C (IMU01) being replaced -- 
 will pull this out of the partition.
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 16:20:34 2001 -->
<pre>Pull IMU01 out of the partition for some tests.</pre>
<!-- Author: michael -->


<!-- Date: Fri Jun 8 16:24:03 2001 -->
<pre>I have removed older L3 exe tags to avoid confusion in the  
control room.  Aces should be testing out the new 3.17.0 
exes.  Default L3tag should be 131 - which has all the  
reconstruction code and filters needed for commissioning  
trigger tables.  If crashes are occurring with the 3.17.0  
version than you can fall back to tagset 125 which is indentical  
to 131 but made in 3.16.0.  There is a new Null exe which  
tests the new calibration loading at Level-3, tagset 133.</pre>
<!-- Author: Kirsten -->


<!-- Date: Fri Jun 8 16:34:54 2001 -->
BMU HV off while they are investigating noise problems.  Remember to page them at 218-9361 if beam reappears.
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 16:41:12 2001 -->
<pre>Start partitionning for new run after Larry loaded some code on 
a cal crate (CCAL04). 
Take a run for Steve Nahn. 
Get a lot of L3 errors: 
 
(MLE) b0l3gate2.fnal.gov:main:4:38:53 PM->Error on L3 node b0l3c14 (partition 0) Fri Jun 8 16:38:51 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:39:03 PM->Error on L3 node b0l3c03 (partition 0) Fri Jun 8 16:39:01 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:39:14 PM->Error on L3 node b0l3c15 (partition 0) Fri Jun 8 16:39:12 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:39:36 PM->Error on L3 node b0l3c13 (partition 0) Fri Jun 8 16:39:33 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:39:42 PM->Error on L3 node b0l3c11 (partition 0) Fri Jun 8 16:39:40 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad)             </pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117200' target=_top>117200</a>) -->


<!-- Date: Fri Jun 8 16:44:50 2001 -->
MCR log shows the quench that ended last shift's store was at B0 low beta quads.  They were doing alpha scans at the time.
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 16:55:09 2001 -->
<pre>Change run configuration -> use COSMICS[3,131]. 
Remove crate FIB_ISL03 and start new run. 
 
Get a lot of errors from L3: 
 
 
 
 
 
(MLE) b0tsi02:Messenger:4:51:03 PM->BUSY timeout 
(MLE) b0l3gate2.fnal.gov:main:4:51:07 PM->Error on L3 node b0l3c02 (partition 0) Fri Jun 8 16:51:05 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:51:08 PM->Error on L3 node b0l3c01 (partition 0) Fri Jun 8 16:51:05 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:51:08 PM->Error on L3 node b0l3c03 (partition 0) Fri Jun 8 16:51:06 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:51:08 PM->Error on L3 node b0l3c06 (partition 0) Fri Jun 8 16:51:06 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad) 
  
(MLE) b0l3gate2.fnal.gov:main:4:51:08 PM->Error on L3 node b0l3c05 (partition 0) Fri Jun 8 16:51:06 2001 l3_node 
  in chain_loop (l3_chain.c:599) 
  @L3_INPUT_FAILURE:  chain 0 code Success (0 good, 1 bad)                                    
 
 
 
 
and from the event builder, producing a busy time out: 
 
 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb21 for event 60. 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb25 for event 60. 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb20 for event 60. 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb14 for event 60. 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb13 for event 60. 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb23 for event 60. 
(MLE) b0l3gate1.fnal.gov:main:4:50:58 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-DuplicateAck: Duplicate Load Event ack from SCPU b0eb22 for event 60.                 
 
 
</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117201' target=_top>117201</a>) -->


<!-- Date: Fri Jun 8 17:14:32 2001 -->
MCR will try for another store.  Expect injection to begin if all goes well around 7 pm.  Asked them to warn us before putting beam back in, which they will do.
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 17:34:42 2001 -->
<pre>Checked the noise on BMU. Even with HV off, the 
noise is still present. Also, found 15 us noise 
which is characteristics of CAEN power supply noise. 
 
BMU channel 30 on CAEN supply 1 trips while ramping 
up (at around 4.7kV). The HV setting for the channel 
and its partner (channel 26) have been set to 100 V 
so that it won't trip till the next access.</pre>
<!-- Author: hyunsoo & woo-hyun -->
<!-- Comment: Fri Jun  8 17;54;20 comment by...hyunsoo -->
<pre>BMU West Cathode 24 (CAEN Crate 2 channel 2) tripped 
3 times under no beam. Setting on HV to 100 V for 
both cathode and its corresponding anode (channel 6).</pre>
<!-- Date: Fri Jun 8 17:45:07 2001 -->
<pre>Looks like old ATM problem. We thought we  
solved it a while ago by changing ATM fabrique. 
When we add Silicon, we increase the number  
of fragments. It makes ATM connection unstable.  
Report will be send to Steve Tether.    
 
I cleaned up EVB and L3. Start run again with 4 subfarms. 
It works but reformatter rejects 100% of events.  
Looks like reformatter does not understand SVX format.  
 
Trying to run with all subfarms.  
No input errors on converters.  
We processing events VERY slowly  
32 events for 10 minutes with a help of HRR.  
Scanner Manager does not receive triggers.  
 
Clean up EVB again. </pre>
<!-- Author: Boris/Michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117202' target=_top>117202</a>) -->
<!-- Comment: Sat Jun  9 01;06;11 comment by...Steve Tether -->
There were no complaints about bad ATM packets in the /var/log/messages file of any converter node, so this is not a repeat of the earlier ATM hardware problem. It looks like the SCPUs were screwed up and told the Scanner Manager that they sent their fragments but really did not.
<!-- Date: Fri Jun 8 18:02:23 2001 -->
<pre>Restart new run with tag 131 and only 4 L3 sub-farms. 
RC crashes again (out of memory) in cold start.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117203' target=_top>117203</a>) -->


<!-- Date: Fri Jun 8 18:22:15 2001 -->
<pre>Restart RC and load 131 again. crate b0imu00 fails twice on cold start. Try to reboot the crate and restart the run. 
Crate still fails -> remove it and page the expert. 
Run started -> run slightly faster than previous ones.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117209' target=_top>117209</a>) -->


<!-- Date: Fri Jun 8 18:29:25 2001 -->
<pre>Analysis of reformatter error shows that the CCAL crates are 
the culprits (SteveN please relax it is not the silicon). 
 
Might bee that downloading new code into CCAL04 causes the 
problem since the error appeared in CCAL05. Reboot the crate. 
</pre>
<!-- Author: Christoph Paus -->


<!-- Date: Fri Jun 8 18:37:36 2001 -->
<pre>Reboot ccal04 so that we load back the old code. 
Start new run with tagset 133 (requested by Farrukh). Still pretty slow but no RAWREF error any more. </pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117210' target=_top>117210</a>) -->


<!-- Date: Fri Jun 8 18:46:01 2001 -->
<pre>Previous run was started with version 4. Change it to 3 and restart the run. Still tagset 133. 
Rate a little bit higher but still not high enough.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117211' target=_top>117211</a>) -->


<!-- Date: Fri Jun 8 18:58:03 2001 -->
MUON scintillator times out 3 times. 
 
(MLE) b0musc00:Messenger:6:53:37 PM->VISIONread/write failed in slot 4 
53'41" 1 crate/s: b0musc00(85),  Done TO.[RXPT]                      
 
 
Can recover with HRR
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117211' target=_top>117211</a>) -->


<!-- Date: Fri Jun 8 19:00:33 2001 -->
<pre>Farrukh calls again. Wants a run with tagset 134. Anyway, muon scintillator is timing out again. End the run</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117211' target=_top>117211</a>) -->


<!-- Date: Fri Jun 8 19:08:28 2001 -->
PTM (Plug Temperature Control System) Looks like 
a second circuit breaker tripped yesterday (probably).   
The temperatures are not controlled on PE02, PE03,  
PE05, and PE06 now.  I disabled the alarms for  
those boxes until the circuit breakers can be 
reset on access (and apparently the toroids must be 
moved to allow us to open the breaker box).  Cause  
of the trips is not known yet.  System was running  
for 9 months with no trips, now 2 in a week.  
<!-- Author: David Dagenhart -->


<!-- Date: Fri Jun 8 19:12:14 2001 -->
<pre>Reset crate 19 in IMU00 and remove slot 4 from MUON scintillator. 
Restart the run with tagset 134 (as requested by Farrukh) but failed in cold start: 
 
 
(MLE) b0l3gate2.fnal.gov:main:7:09:25 PM->Error on L3 node b0l3gate2 (partition 0) Fri Jun 8 
19:09:24 2001 l3proxy 
  in Config (L3ProxyPartition.cc:1140) 
  @L3_PACKAGE_FAIL - failed to download TCL package           
</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117212' target=_top>117212</a>) -->


<!-- Date: Fri Jun 8 19:25:29 2001 -->
We have tested the new Level3 scheme of picking up calibrations 
from Valid-Sets (at run-start time as usual), this scheme works. 
 
The Null executable built from 3.17.0 is  
l3 tagset=133 
 
The DEFAULT Exe (which runs reconstruction-> COT tracking 
Muons CdfEmObject etc) and writes a single stream is 
l3 tagset=134. (This does the same thing as tagset 131  
except 131 uses the old calibration scheme) 
  
<!-- Author: Farrukh Azfar :: (run <a href='/cgi-bin/runSummaryRun.cgi?117213' target=_top>117213</a>) -->


<!-- Date: Fri Jun 8 19:26:58 2001 -->
I forgot to enter this amidst the chaos earlier. 
<p> 
About 1600, I changed the clock from 36x36 mode to 1x8 mode.  I'd forgotten to do this before, mea culpa.  It confused the L2 people no end. 
<p> 
We will keep the clock in this mode until the first 36x36 store.
<!-- Author: Steve Hahn -->


<!-- Date: Fri Jun 8 20:07:56 2001 -->
<pre>PSM sometimes alarms. 
Expert playing with it. 
</pre>
<!-- Author: Koji -->


<!-- Date: Fri Jun 8 20:12:05 2001 -->
<pre>Finished updating PSM monitoring program. Digital words from Pei PSM are readout together with their complement word for data consistency cross-checking, which should improve the readout reliability. Also the way of treating readout errors with "analog" voltages is changed, which should slightly speed up the program. Due to this activity few heart beat alarms were generated. </pre>
<!-- Author: Marcin -->


<!-- Date: Fri Jun 8 20:20:28 2001 -->
MCR will start shot setup within next 15 minutes... turned COT, CES, CPR, and CCR to standby and informed silicon people.
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 20:34:32 2001 -->
<pre>Set COT, CMU, CMP, CMX, BMU, CES, CPR and CCR HVs standby.</pre>
<!-- Author: Koji -->


<!-- Date: Fri Jun 8 20:45:57 2001 -->
Silicon powering down.  Beam is now arriving...
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 21:04:12 2001 -->
<pre>David Saltzberg asks for L1CAL, L1Global, XFT, L2Decision, CCAL02... 
 
So we start with calibrations... 
 
 
Well, there is beam, so here go the calibrations... 
 
We can still take a muon calibration. It hangs in partition 0 but works on partition 1. Times out at the end of the calibration: 
crates b0imu00(80), b0imu01(81),b0cmx01(87), Done Time out. 
 
</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117215' target=_top>117215</a>) -->


<!-- Date: Fri Jun 8 21:10:40 2001 -->
We successfully ran w/ 6 wedges of silicon integrated for 
a couple hours.  TeV in shot set-up right now and plan to 
spend the first 4 hours of the store performing various studies. 
So, we're taking ourselves out of the run and turning the silicon 
off for now.  Once TeV is done playing, Rob Napora can bring the 
designated wedges (he knows which they are) to STDBY, then into 
the run.  If all is well, we'll bring them to full volts. 
 
p.s. Rob, please read your email.
<!-- Author: Doug G -->


<!-- Date: Fri Jun 8 21:12:32 2001 -->
<pre>Try another muon calibration but same timeout.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117217' target=_top>117217</a>) -->


<!-- Date: Fri Jun 8 21:19:03 2001 -->
<pre>CMP box in Global Alarms sometimes becomes yellow 
even in standby state.</pre>
<!-- Author: Koji -->


<!-- Date: Fri Jun 8 22:29:58 2001 -->
<pre>David gave us back the CAL and L1 and 2, so we start again a cosmic run [3,134].</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117223' target=_top>117223</a>) -->


<!-- Date: Fri Jun 8 22:48:04 2001 -->
<pre>Beam quench happened. So there won't be beam for a while. 
Let's do some calibrations: 
 
ACE_QIE_NOBEAM -&gt; calibrate CAL and CLC; hangs on partition 0!!! Does not hang in partition 0 but crashes in the setup. No error log file is written. 
 
OK, the SEB was not starting so we restarted it by hand. 
 
Still crashes. So change configuration to ACE_QIE. This time, it works on partition 0.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117227' target=_top>117227</a>) -->


<!-- Date: Fri Jun 8 23:10:55 2001 -->
<pre>BSC alarm. 
This alarm disapeared in 5 sec.</pre>
<!-- Author: Koji -->


<!-- Date: Fri Jun 8 23:16:34 2001 -->
<pre>BSC alarm again which disapeared soon. 
</pre>
<!-- Author: Koji -->


<!-- Date: Fri Jun 8 23:18:34 2001 -->
<pre>BSC alarm. It's gone.</pre>
<!-- Author: Koji -->


<!-- Date: Fri Jun 8 23:18:47 2001 -->
<pre>TOF calibration: 117229. 
 
ShwrMax calibration: 117230 -&gt; calibration hangs.  
Some ccal are not responding: ccal02, ccal05, ccal07, ccal11, ccal13. 
 
Try to log in using minicom (in b0dap10): does not work-&gt; prompt does not appear (it does in all the other). ping the crate-&gt; no response. Page the expert. 
 
Some ccal are not responding: ccal02, ccal05... ooops should not have done that. 
 
Expert calls. Well, will reset crates anyway after all -&gt; 
 ccal07, ccal11, ccal13. 
 
Alll the crates reappear as green. unfortunately, there is beam in the machine so shower max calibration cannot be done. 
</pre>
<!-- Author: michael -->


<!-- Date: Fri Jun 8 23:31:19 2001 -->
Store is in, but accelerator says they do not plan to do scans 
yet; instead will wait until day shift and in the meantime want ro give the store to us.  We see high antiproton losses (>4 KHz) and have asked them to do what they can to cleant this up in order to make the store usable to us.  <p> 
Meanwhile, the DAQ is not happy -- ace is trying to clean up EVB etc. to get it back.
<!-- Author: AFS -->


<!-- Date: Fri Jun 8 23:40:12 2001 -->
<pre>Try a new cosmic[3,134]. 
Event builder failure -&gt;b0eb17 could not connect. 
 
Start luminosity (1x8)-&gt; reset run to 36x36[20,134] as asked and physics run. 
 
Still failure of b0eb17, although run passes partition transition. But will probably not pass cold start.  
So reset run and clean up EVB -&gt; complains server not found. 
Finishes cleaning up okay. 
Start partition again -&gt; no more complains from EVB.  
 
Start data taking 
</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117231' target=_top>117231</a>) -->
<!-- Comment: Fri Jun  8 23;48;33 comment by...AFS -->
Voltages are still at standby for sensitive detectors.  CLC shows
collisions are basically centered in z, so have asked MCR to try to improve loss rate so that we can run.  They also see poor Pbar lifetime and so we agree they should try to clean this up.  Note separators are ON during this store so far (different from previous ones where they were off).
<!-- Date: Fri Jun 8 23:46:41 2001 -->
<pre>2 L3 nodes die: b0l3068 and b0l3017.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?117231' target=_top>117231</a>) -->



<!-- Date: Fri Jun 8 23:55:18 2001 -->
<center><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-1083&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-1083' width='300'></a><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-1084&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-1084' width='300'></a></center>
<!-- Author: Mauro & Stephane -->
<!-- Comment: Fri Jun  8 23;58;10 comment by...Mauro -->
<pre>Thanks God it's Friday !
(and we are getting collisions on the right spot)</pre>


<!-- Date: Fri Jun 8 23:58:15 2001 -->
<font color=red>Shift Summary: </font><pre>Spent most of the shift waiting for the store.  David Saltzberg 
did some L2 studies and we also ran silicon in integration during 
other runs.   
 
CLC calibration should NOT be done until cleared by expert. 
Did some calibrations (see above); others remain to be done. 
Encountered problems with SMX crates and reset them on instructions  
from expert.  See board for calibrations that need to be done. 
 
Store went in near end of shift.  We have it through the night 
and are trying to get MCR to clean it up so we can use it.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   117200 - 117231
Delivered Luminosity   .1 nb-1  pb-1
Acquired Luminosity    0  pb-1
Efficiency             0
   
</td></tr></table></pre>
<!-- Author: Alan Sill -->


<!-- Date: Sat Jun 9 00:01:05 2001 -->
<pre>SVX radiation monitoring summary : 
 
W Inner BLM : .3716307   (Rads) 
W Outer BLM : .0471198   (Rads) 
E Inner BLM : .181965    (Rads) 
E Outer BLM : .7015953   (Rads)</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Koji -->