<!-- Title: Owl shift. Sat Aug  4, 2001 -->
<!-- SciCo: S. Errede/A. Clark -->
<!-- DAQAce: Anyes Taffard/ Valdimi -->
<!-- MonAce: Victoria Martin/Boris -->
<!-- CO: Jaison Lee/ William We -->
<!-- OpManager: Kevin Burkett -->
<!-- Notes: -->
<pre> 
  Subject to no problems: 
   
  a) Aces have just started physics_0_00 run. All in except 
     for SVT because the L2 cable pulled out 
 
  b) We should run this trigger all night, +  
     - approx 20 mins muon_test 
     - approx 20 mins test_two_tracks</pre>


<!-- Date: Sat Aug 4 00:35:06 2001 -->
Kaori was here, Event Builder now operational. Problem  
was due to silicon data
<!-- Author: allan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122673' target=_top>122673</a>) -->
<!-- Comment: Sat Aug  4 17;53;46 comment by...Kaori -->
<pre>Andrey had fixed Event Display code, which was having a
problem reading silicon data.  We built a new 'mon' on
the online machine.  See CO e-log for "how to run this Evd"
(until we make a new cdfsoftb0 release with this fix).</pre>
<!-- Date: Sat Aug 4 00:41:37 2001 -->
<pre>Reminder from Vaia: When we do LED calibrations tomorrow, please reboot the central calorimeter crates before the calibration.  
We want to make sure that new front end code takes effect.</pre>
<!-- Author: allan/steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122673' target=_top>122673</a>) -->


<!-- Date: Sat Aug 4 00:47:53 2001 -->
<ul> 
<h3> 
<br> Since the begining of the shift and also previous shift we have been getting rather very ofter SVT_07 BTO with can take few HRR to get sorted. <br> This is due to some "junk" event that there are receiving from XTRP and that trigger CDF error. <br> They do not want to mask out this problem since this will be equivalent as having SVT out of the configuration. 
<br> After talking with SVT expert, the conclusion was that if this keeps occuring to often we can remove SVT from the run. 
<br> <br> After insisting with HRR, we are now running for few mins without getting any errors... Let's hope it stays that way<br>  
</h3> 
</ul>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122673' target=_top>122673</a>) -->
<!-- Comment: Sat Aug  4 00;55;46 comment by...Anyes -->
<br> After 63k we started getting SVT error {XTRP] so we are taking out SVT. <br>


<!-- Date: Sat Aug 4 00:58:12 2001 -->
<pre>Kaori Maeshima has (temporarily) turned off the TRKMON module in ObjectMon - because it was crashing ObjectMon. Thus, people should ignore the associated histograms normally output from TRKMON - they will now not be filled. When TRKMON is fixed, it will be re-enabled.</pre>
<!-- Author: allan/steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122673' target=_top>122673</a>) -->


<!-- Date: Sat Aug 4 00:58:46 2001 -->
Vladimir is taking over daq ACE position 
Boris is taking over the monitoring ACE position.
<!-- Author: new ACEs -->


<!-- Date: Sat Aug 4 00:59:16 2001 -->
<pre>W03 and E07 have higher occupancy than the rest. 
Input to the TDC shows train of pulses coming and 
going.</pre>
<!-- Author: hyunsoo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122673' target=_top>122673</a>) -->


<!-- Date: Sat Aug 4 01:07:38 2001 -->
<br> fib02 Bdidn't cold start -> abort -> coldstart  
<br> Then Fib00 b0fib00 B0w0 -> coldstart, but then RC got stuck doing nothing so reset... <br>
<!-- Author: Aces -->
<!-- Comment: Sat Aug  4 01;31;40 comment by...allan/steve -->
<pre>This is now run 122676 as of 1:12 am</pre>
<!-- Date: Sat Aug 4 01:23:30 2001 -->
Event display occassionally shows error messages of the following type when the COT display is alive: 
SVXchipID_xxxxxx is invalid on output 
(xxxxxx=large numbers in a large loop) 
<!-- Author: William Wester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122676' target=_top>122676</a>) -->
<!-- Comment: Sat Aug  4 01;32;07 comment by...W. Wester -->
"SVX chip Id -xxxxxx is invalid on output" errors appear during other displays than just COT


<!-- Comment: Sat Aug  4 06;34;01 comment by...W Wester -->
See the note on the terminal (left most) for more information and apparently a way to supress the messages.
<!-- Date: Sat Aug 4 02:17:42 2001 -->
BTO in VRB/EVB   crate VRB_SVX_02  -> HRR
<!-- Author: Vladimir  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122676' target=_top>122676</a>) -->


<!-- Date: Sat Aug 4 02:47:35 2001 -->
<pre>02:30 
 
 CMP trip.  
 Pause run. Put whole of CMP to standby. 
 Call MCR 
 Major losses of both LOSTP and LOSTPB 
 They are investigating. 
 
 Losses now under control. 
 
 Wait for Flying Wire. Then wait for some additional  
 beam losses. 
 
 02:45 .... continue run </pre>
<!-- Author: allan :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122676' target=_top>122676</a>) -->
<!-- Comment: Sat Aug  4 07;04;56 comment by...Victoria -->
See Boris' nice losses plot below.
<!-- Date: Sat Aug 4 03:10:57 2001 -->
<pre>Found that TOF HV was off and the HV software hanged up at  
around 1:10. 
After looking into what happened and reseting the software, 
put on the HV again at 2:25.</pre>
<!-- Author: Koji Sato -->


<!-- Date: Sat Aug 4 03:19:10 2001 -->
<br> 
While trying to restart run, some fib crates  were not coldstarting (fib00, fib04). <br>fib00 got fixed at next coldstart, but then fib04 start failling for no apparent reason: 
<br> Processing Config Message FailedSent ack to Run Control: 0 <br> READY ERROR 
<br> Error Initializing HDI Slot 12 HDI 4: SVX B4W0 
<br> Statechange failed                                 
 
We finally rebooted it. 
<br>
<!-- Author: Aces -->
<!-- Comment: Sat Aug  4 03;23;04 comment by...allan -->
error repeats. Page silicon


<!-- Comment: Sat Aug  4 03;31;42 comment by...Anyes -->
<br>
After reboot the crate (fib04 b4w0) failed again coldstart 3 times. Removing crate from configuration file and paging SVX.  
<br>
<!-- Comment: Sat Aug  4 04;06;09 comment by...allan -->
Attempted to call silicon pagers (primary, Jean, primary DAQ, Lester). No luck. To sort out this morning. At present removed. 
<!-- Date: Sat Aug 4 03:30:56 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1756&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1756  width='300' align='top'></td><td>Losses between 2 am and 3 am. </a></td></tr></table>
<!-- Author: Boris -->


<!-- Date: Sat Aug 4 03:31:19 2001 -->
TOF calibration request. 
If possible at the end of store, and if there is quiet time, 
TOF requests time (~ 1 hour) to make a full TOF calibration,  
in particular pulse01
<!-- Author: allan/steve -->



<!-- Date: Sat Aug 4 03:38:41 2001 -->
CMP trip
<!-- Author: allan/steve -->


<!-- Date: Sat Aug 4 03:47:48 2001 -->
<br> 
Got new run starting but getting a series of BTO which do not get <br> fixed by HRR. There is eb10 errors. End run to clean EVB and L3 (just in case) while we are waiting for flying wire... <br>  
<!-- Author: Anyes -->


<!-- Date: Sat Aug 4 03:56:30 2001 -->
Start run 122686
<!-- Author: allan/steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122686' target=_top>122686</a>) -->



<!-- Date: Sat Aug 4 04:17:48 2001 -->
</br>  Spike in proton losses -> CMP side 1 (top), part 1  tripped 
</br>
<!-- Author: Vladimir  -->


<!-- Date: Sat Aug 4 04:32:34 2001 -->
<br> 
<li> VME Bus error from eb19 crate. 
<li> Stop taking event, xft02 + l2dec00. Got stuck in HRR, then in abort.  
<li> Reset + partition , but crates are getting reset signal but not partition. So restart RC. 
<li> remove 8 L3 subfarms to get started faster. We will hopefully be able to check if the rates get changed. 
<li> And in fact.... we are running ...well RC didn't save the configuration file before, <br> so SVT crates sniked in the configuration and we have to take them out. 
<li> Got started for good... may be. 
<br>
<!-- Author: Aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122688' target=_top>122688</a>) -->


<!-- Date: Sat Aug 4 04:35:18 2001 -->
Start run 122688, after DAQ problems.
<!-- Author: allan/steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122688' target=_top>122688</a>) -->


<!-- Date: Sat Aug 4 04:49:13 2001 -->
<pre>While restarting the runs we were trying to use 8 and 12  
Level3 subfarms. Rate remains the same within the limits  
of fluctuation.  
 
When we used 8 subfarms we  saw bright yellow boxes  
here and there on Level3 Dsplay. It caused Level3 Display  
window to be redrawn every time which does not look nice.  
At least we noticed that feature of monitoring.  
 
 
 
 
 
</pre>
<!-- Author: Boris -->


<!-- Date: Sat Aug 4 04:55:29 2001 -->
<pre>One of the nodes does not start run sometimes .  
I've seen 023 yesterday and 013 and 014 today.  
 
see /log/l3_node/l3_node.out_200108040444  on 014  
 
I remember 002 in the same state last week.  
All those nodes are the old ones. 
 
</pre>
<!-- Author: Boris.  -->


<!-- Date: Sat Aug 4 04:59:27 2001 -->
<pre>Done timeout from b0l2de00  
 
-> internal buffer too  small 
 
HRR does not help.  
 
Ended run, rebooted the crate.  
Sent partition message to the crate.  
Resumed run from Idle. </pre>
<!-- Author: Boris/Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 06:26:24 2001 -->
<pre>CEM and CHA have yellow HV bars.  It looks like they are over-voltage.  There is no IFIX panels for these detectors, so I can't check this directly.</pre>
<!-- Author: Victoria -->
<!-- Comment: Sat Aug  4 07;00;08 comment by...Victoria -->
<pre>Steve went down the check the power supplys and apparently they are not complaning about the HV.
I've paged the calorimeter pager, but no reply as yet.</pre>
<!-- Comment: Sat Aug  4 07;03;20 comment by...Victoria -->
Steve called Fotis Ptohos and he's coming in...
<!-- Date: Sat Aug 4 06:35:17 2001 -->
<pre>CMP Top HV trips.  Seems to be correlelated with a little spike in the proton losses.  HV reset ok.</pre>
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 06:39:49 2001 -->
<pre>DTO imu00 -> HRR -> ok .</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 06:41:22 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>All crew thank our PschyCo for bringing breakfast!!!</td></tr></td></table>
<!-- Author: Shift crew :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 06:42:27 2001 -->
<pre>TOF alarm, but the TOF stays green.  
No trigger inhibit, so I don't think it's a HV alarm. 
So other signs of any problem.</pre>
<!-- Author: Victoria -->
<!-- Comment: Sat Aug  4 08;19;55 comment by...rolf -->
TOF does not (yet) generate Trigger Inhibits
<!-- Date: Sat Aug 4 06:54:33 2001 -->
 
 Call from George Bush on the Red Phone.  
 No words from him, but believe it has to 
 do with his Health Plan.
<!-- Author: Allan Clark :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->





<!-- Date: Sat Aug 4 07:18:16 2001 -->
Visually inspected Currents and Voltages on CEM & CHA HV power supplies. All 4 CEM HV power supplies show HV readings on analog meter consistent w/ settings, and currents consistent w/ each other. Same is true for CHA power supplies. No overvoltage indicators are lit up on these power supplies. Suspect CEM/CHA 
~ 20% overvoltage warning alarm is not real. We also looked 
at CEM/CHA YMON plots, found nothing overtly wrong since this 
warning started.
<!-- Author: steve e. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 07:19:32 2001 -->
<br>  
I have notice that this evening we have been getting much far less SVX02 BTO (1 or 2 in 600k so far). <br> Can this be related to the fact that the SVT crate are out of the configuration? <br>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 07:24:03 2001 -->
Fotis calls back - he says he won't be in, but Nikolay will be in at ~ 8:15 instead.
<!-- Author: steve e. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 07:37:23 2001 -->
<pre>Since run 122687 we have been using trigger table  
Physics_0_01 [10,167]. This table has corrupted prescale values 
for some of the L3 trigger paths. 
 
Unless this was intentional should switch to Physics_0_00 [2,161]. 
</pre>
<!-- Author: Frank Chlebana -->


<!-- Date: Sat Aug 4 07:43:09 2001 -->
Frank Cheblana calls - points out that we have Physics_0_01 
trigger table for run 122690, not Physics_0_00. We end this run. 
Near flying wire time. We will do Muon_Test [4,165]. Run for 
~ 20 mins. Then day shift should do Test_Two_Track[10,165], 
then carry on with Physics_0_00[2,161]. We called MCR earlier, 
the store will continue thru morning.
<!-- Author: Steve E. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=122690' target=_top>122690</a>) -->


<!-- Date: Sat Aug 4 07:58:32 2001 -->
<pre>Busy timeout after 49. HRR. Busy timeout after 59 events.  
SM statistics: Got 12 more triggers than loaded events.  
suspended processes on eb14.  
 
Cleanup evb. Remove all SVX from the run.  
Silicon expert plays with voltage. Ready to rum Muon  
trigger table.</pre>
<!-- Author: Boris/Anyes -->


<!-- Date: Sat Aug 4 07:59:06 2001 -->
<font color=red>Shift Summary: </font><pre> 
 Shift summary. 
 
 a) XTRP timing problem persisting. Presumably this affects 
    all triggers involving tracks ... not just SVT. Important  
    to fix. 
 
 b) Calibration request from TOF at end of store      
    If quiet time, request ~ 1 hour for TOF calibration, 
    in particular pulse01. 
 
 c) Had various DAQ/Detector problems much of the night. 
    We got what we thought was a good slug of data taking 
    on run 122690, but then discovered that we picked up  
    the wrong trigger table because of DAQ problems, with 
    the previous configuration not being savable. 
 
 d) We had problems with an SVX crate, which was removed from 
    the DAQ. 
 
 e) We had overvoltage warning from CEM/CHA HV. There is no 		    information as to the nature of the problem. We visually          inspected these power supplies and found no obvious problems.     We called the experts. They will investigate on day shift         this morning. 
 
 f) We will start Muon_Test[4,165] which day shift should end,        then do Test_Two_Track[10,165].</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   122673-122691
Delivered Luminosity   48 nb-1  
Acquired Luminosity    26 nb-1  
Efficiency             54%
   
</td></tr></table></pre>
<!-- Author: allan/steve -->
<!-- Comment: Sat Aug  4 12;07;28 comment by...Kevin Pitts -->
<pre>To clarify, the problem with XTRP-&gt;L2/SVT interface affects 
none of the L1 triggers.   We are not yet cutting at L2, so
the problem has no effect on the performance of any of the 
existing trigger paths.  It of course does affect the
commissioning of the SVT and L2 track-based triggers, in 
addition to giving Bill A. headaches.  We are working on it.</pre>
<!-- Comment: Sat Aug  4 17;02;11 comment by...whoever -->
<pre>Actually, the rates and deadtime scalers display in the control room havent been updated for 7 hours, when I got to the control room this morning. 

</pre>
<!-- Date: Sat Aug 4 08:01:14 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1758&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1758  width='300' align='top'></td><td>Luminosities and </a></td></tr></table>
<!-- Author: Vladimir -->


<!-- Date: Sat Aug 4 08:03:01 2001 -->
<center><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=-1756&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2001&action=view&page=-1756' width='300'></a></center>
<!-- Author: Vladimir  -->
<!-- Comment: Sat Aug  4 08;03;39 comment by...Vladimir -->
Losses
<!-- Date: Sat Aug 4 08:03:07 2001 -->
<br> We seem to still have pb with save and save as from RC which caused tonight to get the wrong trigger table in the<br> configuration because the correct configuration was not saved when I thought I just saved it before exitting RC. <br> 
<!-- Author: Anyes -->



<!-- Date: Sat Aug 4 08:05:37 2001 -->
SVRad are all zero.
<!-- Author: vladimir  -->



<!-- Date: Sat Aug 4 08:31:55 2001 -->
We took MUON_TEST[4,165] for 20 min. 
And we start TEST_TWO_TRACK[10,165]<font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Y.Liu/M.Ahn -->
<!-- Comment: Sat Aug  4 08;35;58 comment by...Y.Liu/M.Ahn -->
Run Number 122693