<!-- Title: Day shift. Tue Jun 12, 2001 -->
<!-- SciCo: Tom Phillips -->
<!-- DAQAce: Anyes Taffard -->
<!-- MonAce: Victoria Martin -->
<!-- CO: Arie Bodek -->
<!-- OpManager: Doug -->
<!-- Notes: -->
<pre></pre>


<!-- Date: Tue Jun 12 08:26:45 2001 -->
Dervin reports 1st floor racks are back on.
<!-- Author: Tom -->


<!-- Date: Tue Jun 12 08:34:31 2001 -->
Doug turned 1st floor tripped breakers back on (tripped by cryo.  Details?)
<!-- Author: Tom -->


<!-- Date: Tue Jun 12 08:42:03 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>All the CAEN racks were off, now they are back on (see above). 
The COT, Muon, CCAL and COT HV systems should be checked.</td></tr></td></table>
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 08:43:21 2001 -->
<pre>Could not end cosmic Run 117496, probably due to the racks tripped.  
 
</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117496' target=_top>117496</a>) -->


<!-- Date: Tue Jun 12 08:50:50 2001 -->
<pre>Cannot start the L3 Ace Control panel from b0dap58. 
Get following error: 
/usr/krb5/bin/kinit -k -t /var/adm/krb5/cdfdaq.keytab cdfdaq/cdf/`hostname`  
kinit: No such file or directory while getting initial credentials 
I guess the machine need to be reconfigure (bodap58 is node 
which was down for a week)</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 08:58:45 2001 -->
<pre>cdfsoftb0 current version is 3.17.0a now.</pre>
<!-- Author: kaori -->


<!-- Date: Tue Jun 12 09:02:00 2001 -->
Summary of events  
 
Earlier this morning, Petra hit the silicon racks abort crash button which turned off power to all 
racks.    In so doing, this created (caused?) a severe error in that particular quadlog module.   
There are two easy ways to determine this.  One can look in the APACS page and look for severe 
errors.   QL puts out -0.78 volts when it means zero (so that you know it is working).   In t his 
case, I saw 0.0 which means that QL was latched in a bad state.  
 
 
To fix this problem, one needs to go to 4-mation and remove the latch.  This will clear this 
particular severe error.  
 
 
We were overly exhuberent and tried to reset this by reseating the card.   This is a no-no for a 
quadlog safety system.   The system is configured now to drop AC power, flammable gas etc if it sees 
an abnormal condition (reseating a card, forcing a variable in 4-mation etc).   Thats what happened. 
  We tripped off all colision hall power, tripped off flammable gas, tripped off solenoid, etc.  
 
 
Everything is now back up and running -- except solenoid which is now being pumped down.  
<!-- Author: RR -->


<!-- Date: Tue Jun 12 09:18:39 2001 -->
Starting new run with everything to get system back up. 
COT and TOF in STBY (waiting for solenoid to get back on) 
CES,CPR,CCR in STBY.
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 09:25:15 2001 -->
<pre>HV status:  Hynsoo came in and started the CMU, CMP, CMX, BMU HV.  The CMX SE wedges 16&17 tripped, even in standby, so we switched them to off. 
The Plug systems expert called and I switched on the HV following his instructions. 
The CCal expert came in and put the CES, CPR and CCR systems to standby. 
Experts are working with the COT and TOF.  Their HV cannot be switched on until the Solenoid is on.</pre>
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 09:31:44 2001 -->
The PSM reports an error in the arches in crate 21E and the plug systems report a missing heartbeat.  These errors clear themselves.  I'm trying to contact Marcin re the PSM.
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 09:38:44 2001 -->
<pre>Partition:0 Run number: 117497 (0x1caf9) 
(MLE) b0xft02:Messenger:9:18:20 AM->VISIONread/write failed in slot 19 
(MLE) b0xft02:Messenger:9:18:20 AM->Wrong card type in slot 19 
(MLE) b0xft02:Messenger:9:18:20 AM->Setup failed with error code 3 
(MLE) b0xft02:Messenger:9:18:21 AM->Statechange failed 
(RC)   9:17:2 Config -> ERROR 
(MLE) b0cot13:Messenger:9:17:11 AM->Resetting card in slot 15 failed 
(MLE) b0cot13:Messenger:9:17:11 AM->Statechange failed   
 
Reset cot13 and xft02</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117497' target=_top>117497</a>) -->


<!-- Date: Tue Jun 12 09:46:57 2001 -->
CEM turns yellow (warnings are present) in IFIX.  The expert is coming.
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 09:48:35 2001 -->
<pre>TRied again after reset to start a run. Got exactly same errors 
from xft02 and cot13 
Tried reboot xft02 via vxlogin gui... the procedure cratch the gui! 
Paged xft experts.</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 10:02:24 2001 -->
<pre>Release svt00 & 01 to experts.</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 10:20:51 2001 -->
After the morning outage SW CEM gamma gives 30v more, waiting 
till 12am. 
<!-- Author: N. Luzhetskiy -->


<!-- Date: Tue Jun 12 10:26:59 2001 -->
<pre>SVT expert is looking into SVT02 pb. Remove it from run for the time being. 
Starting new run. COT 13 is still giving  
(MLE) b0cot13:Messenger:10:17:25 AM->Resetting card in slot 15 failed 
(MLE) b0cot13:Messenger:10:17:25 AM->Statechange failed   
Rebooting crate via Minicom.</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 10:28:16 2001 -->
b0xft02 slot 19 (finder SL24, #06) was dead, 
it has been changed to new board (#31) 
 
Now it works OK.
<!-- Author: Jong-young Chung -->


<!-- Date: Tue Jun 12 10:35:12 2001 -->
Alarms from SVX, ISL, L00.  I think they were related to IFIX not seeing the heartbeat.  They have cleared themselves.
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 10:36:05 2001 -->
<pre>Pb with COT13 crate: 
Minicom reboot with crtl x didn't help 
Use Minicom to do a bus reset. The offending card came back on.</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 10:42:19 2001 -->
<pre>COT13 and SVT02 now work 
 
New run get error from SMXR: 
 b0ccal01:Messenger:10:40:00 AM->Runtime Error 4, Event 38: SMXR in slot 3 is not responding, header 0x0x3bfe350  
Remove card in slot 3</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117502' target=_top>117502</a>) -->


<!-- Date: Tue Jun 12 10:52:38 2001 -->
<pre>imu01 busy rxpt -> HRR worked</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117503' target=_top>117503</a>) -->


<!-- Date: Tue Jun 12 10:56:41 2001 -->
PSM error with SMDCE_21.  The errors on this crate have been masked.  The voltage on channel 2 of this crate is 6V cf 7.5V. 
However the potential difference between channels 7 and 2 is still nearly 15V so this is okay.  We should continue to monitor this crate
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 11:09:53 2001 -->
<br><b>ACE_CALORIMETER_MINIMAL Configuration:</b> 
<br>Changed the above configuration to use tagset 1,133.  I'm still confused about why tagset 130 no longer works. 
<br>
<!-- Author: R.J. Tesarek -->


<!-- Date: Tue Jun 12 11:10:59 2001 -->
<pre>I recompiled level3 code abd cleaned up level3. There was a change made by Jeff to the code. The change basically allows for level3 to process events having sizes greater than 1 MB   </pre>
<!-- Author: Arkadiy :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117503' target=_top>117503</a>) -->


<!-- Date: Tue Jun 12 11:22:26 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>All Level3 tagsets BELOW 133 have been disabled.  
Tagsets 133 (NULL) and 134 (TestL3_Filters single stream: with full reconstruction) use the scheme for the new calibrations. 
As a precautionary measure we had tagsets 125  and 131 (both  
TestL3Filters single stream with reconstruction) still available 
for use. 
 
Thus the tagsets available for use are: 
133 which is Null. 
134 which does all reconstruction and has single stream output. 
 
cheers</td></tr></td></table>
<!-- Author: Farrukh -->


<!-- Date: Tue Jun 12 11:29:55 2001 -->
<pre>We did some Silicon stuff this morning but now it looks like 
the BD guys want to see why the Pbar losses were so high, so 
we turned off the Silicon.  Call us when A/Plosses < 5 kHz and  
going to stay that way...8611</pre>
<!-- Author: SCN -->


<!-- Date: Tue Jun 12 11:31:30 2001 -->
<pre>COT west side racks have lost power. 
We think the west wall is off. 
Someone is coming to reset the power. 
 
A consquence of this is that we cannot take data, we cannot communicate to some of the cot crates.</pre>
<!-- Author: Victoria -->
<!-- Comment: Tue Jun 12 11;40;07 comment by...Tom -->
Drip sensor on CONWB caused trip.  It was reading 35-40%, but reset when page reopened.  This drip sensor has caused trouble in the past.
<!-- Comment: Tue Jun 12 11;49;25 comment by...Tom -->
Rob reports this sensor was measuring 100 ohms, while the trip point is 20 ohms, so it is probably a piece of dust.  Reading is now back to normal.
<!-- Date: Tue Jun 12 11:34:32 2001 -->
CMX, CMP and BMU HV trip.  Almost all of the wedges/stacks trip.  There have just put protons in the machine.  I set the CMP, CMX and BMU to standby.  The CMX 16&17 SE wedges are still off.
<!-- Author: Victoria -->



<!-- Date: Tue Jun 12 11:40:50 2001 -->
<pre>Released the COT crates to the COT people. 
Released the CCAL crates and the L1cal and L2cal crates to Peter W for tests.</pre>
<!-- Author: Victoria -->
<!-- Comment: Tue Jun 12 11;43;40 comment by...ambrose -->
<pre>Reset the COT low voltage.  Booked all crates for a quick
calibration scan, and all TDC's appear to have survived the
power outage.</pre>

<!-- Comment: Tue Jun 12 11;45;28 comment by...Victoria -->
Peter has also taken the trigger scaler crate and the L1global and L2decision crates.


<!-- Date: Tue Jun 12 11:46:01 2001 -->
New features in run control.  Restart rc.
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 11:46:45 2001 -->
<pre>Run Control "setup fer" now points to devel in order  
to test out some new bug fixes. 
</pre>
<!-- Author: W.Badgett -->



<!-- Date: Tue Jun 12 11:51:35 2001 -->
<pre>All configurations with the Fred crate have been updated  
to include the Fred card itself.   They had been mistakenly  
excluded yesterday while adding a new PreFred card. 
 
This was the so-called "argsueisis" problem. 
</pre>
<!-- Author: W.Badgett -->


<!-- Date: Tue Jun 12 11:54:27 2001 -->
Dave Ambrose checked COT TDCs, and it appears that none were lost in the power trip.
<!-- Author: Tom -->


<!-- Date: Tue Jun 12 11:54:33 2001 -->
Test run using calib continuous and auto L2 accept. 
No problems with the COT crates.
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117505' target=_top>117505</a>) -->


<!-- Date: Tue Jun 12 12:04:57 2001 -->
Using all the CCAL crates, we get mysterious errors in CCAL00 and CCAL05, as was seen yesterday.  SMXR in CCAL01 slot 3 is now giving an error also (not responding).
<!-- Author: Peter -->


<!-- Date: Tue Jun 12 12:10:40 2001 -->
Busy timeout in b0imu01.  HRR doesn't work.  End the run.  Took 116764 events.
<!-- Author: Victoria :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117505' target=_top>117505</a>) -->


<!-- Date: Tue Jun 12 12:19:51 2001 -->
Try to start a new run. 
 
Get the following error in cold start. 
<pre>Partition:0 Run number: 117506 (0x1cb02) 
16'15" triggered readout sent events mismatch[EOR]b0dau32.fnal.gov:csl_mon_send:12:16:14 PM->Logger partition 0 cannot log this event. Run 117505 not accepted. Last run = 117505.</pre>CSL error meaning no events will be written to disk.  It suggests we should end the run.  We cannot Reset, so we Abort. 
              
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117506' target=_top>117506</a>) -->


<!-- Date: Tue Jun 12 12:42:28 2001 -->
<pre>Run hangs at partition.
imu01 not responding anymore (cannot map crate with minicom) 
reseting crate</pre>
<!-- Author: Anyes -->
<!-- Comment: Tue Jun 12 12;47;00 comment by...Anyes -->
<pre>Minicom doesn't seem to be able to reset crate:
Cannot register service: RPC: Timeout
rdbtask: couldn'tregister RDB service. Trying to find the 
expert in the building since the expert with the pager
is not the crate expert</pre>


<!-- Date: Tue Jun 12 12:58:26 2001 -->
CEM SW gamma -> output is reduced by 34v; 
<br> 
In gamma program: set = 1270v -> changed to 1245v;
<!-- Author: N. luzhetskiy -->


<!-- Date: Tue Jun 12 13:13:00 2001 -->
Audio SVX, ISL and L00 cooling alarms.  But there is no record of them in IFIX and they are green.
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 13:13:22 2001 -->
<pre>IMU01 crate cannot be reseted via software, i.e. need  
do manual reset.</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 13:20:20 2001 -->
<pre>Release CLC00 crate for study. 
Started new run with what's left. Change trigger table to [2,133] 
-> null flag</pre>
<!-- Author: anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117508' target=_top>117508</a>) -->


<!-- Date: Tue Jun 12 13:24:17 2001 -->
Hynsoo says that the CMU should be on standby when the other muon systems are on standby, so I put the CMU on standby.
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 13:25:38 2001 -->
<pre>Rearranging fibers to VRB for CCAL, ECAL, HTDC.  Can't use Hardware Event Builder for those crates until further notice. Estimate 2 hours.</pre>
<!-- Author: Jonathan Lewis -->


<!-- Date: Tue Jun 12 13:26:36 2001 -->
In ACNET both the proton and antiproton losses are reported to be exaclty zero.  There are protons in the machine.  Is the wrong information being reported??
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 13:31:22 2001 -->
<pre>Release Hadron timing, and wcal crates for study. 
</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 14:12:53 2001 -->
<pre>While restarting a new run, L3 fails partition (run 117508 I think). 
Check using ACE L3 gui, which gives OK's back 
Clean up EVB and L3 farms... worked 
</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117508' target=_top>117508</a>) -->


<!-- Date: Tue Jun 12 14:17:41 2001 -->
<pre>Included CCAL00 &01 for check (Greg request) 
CCAL01 gives a done time out 
 
Also get error from CSL: 
  RunNumber             117511
  PartitionId           0
  nEventsInput          90
  nEventsOutput         90
  nEventsError          0
  FlushedOrRejected     0
  LoadingErrors         0
  SendingErrors         0
(EOR) CSL: L3 TimeOut waiting EOR; 0 events received, 0 sent to loggers, 0 logged.
(EOR) CSL: stream 0, 0 events received, 0 events logged.
(EOR) CSL: stream 1, 0 events received, 0 events logged.
(EOR) CSL: stream 2, 0 events received, 0 events logged.
(EOR) CSL: stream 3, 0 events received, 0 events logged.
(EOR) CSL: stream 4, 0 events received, 0 events logged.
(EOR) CSL: stream 5, 0 events received, 0 events logged.
(EOR) CSL: stream 6, 0 events received, 0 events logged.
(EOR) CSL: stream 7, 0 events received, 0 events logged.
(EOR) CSL: stream 8, 0 events received, 0 events logged.
(EOR) CSL: stream 9, 0 events received, 0 events logged.
(EOR) CSL: errors, 243 bad events.
(EOR) End of Run System Summary                               
</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117510' target=_top>117510</a>) -->


<!-- Date: Tue Jun 12 14:26:09 2001 -->
<pre>Run 117512 remove CCAL00&01 included CCAL05. 
Works. Stop run to give all muon system to Dan for test</pre>
<!-- Author: Anyes :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117512' target=_top>117512</a>) -->


<!-- Date: Tue Jun 12 14:28:10 2001 -->
<pre>Removed CCAL05, start run, L3 proxy crashed. Restart it.</pre>
<!-- Author: Anyes -->


<!-- Date: Tue Jun 12 14:32:33 2001 -->
<pre>Trying to run tdctest on the muons:  
 
b0imu01 gives this error during setup transition :  
data access 
Exception current instruction address: 0x00e10110 
Machine Status Register: 0x0000b130 
Data Access Register: 0x7c6802da 
Condition Register: 0x24000088 
Data storage interrupt Register: 0x0000b130 
Task: 0xe85178 "Messenger" 
  
the messenger task gets suspended - and the crate needs to be rebooted after this. 
 
Messenger  FER_messenge   e85178 200 SUSPEND      e10110   e84dd8      46     0 
</pre>
<!-- Author: dan -->
<!-- Comment: Tue Jun 12 14;50;55 comment by...Arnd -->
<pre>Bug in the code, should be fixed now. Please reboot crates where
TDC test crashed.
</pre>
<!-- Date: Tue Jun 12 14:38:57 2001 -->
Alarm from the PSM.  The 2RR30C_1 crate on the second floor has two wrong voltages.  Channel 0 is at 0V not +5V.  Channel 1 is at .84V not +12V.  I've paged Fred Lewis about this.
<!-- Author: Victoria -->
<!-- Comment: Tue Jun 12 14;42;58 comment by...Victoria -->
The error disappered by itself.  Channel 0 is now at +5.07V and channel 1 is now at 11.94V.


<!-- Date: Tue Jun 12 14:43:30 2001 -->
We are upping the CLC pressure to 8 PSIG (from 3.5 PSIG). 
This is to test the response at this pressure with large 
stats. We'll keep it like this for the next store at least. 
Bill Noe will change the alarm lmits from {2.5,4.5} to 
{7.5,8.5}
<!-- Author: Jaco -->



<!-- Date: Tue Jun 12 14:54:54 2001 -->
<pre>Having trouble with running TDCtest on HADRON_TIMING - perhaps the work occuring on 1st floor is related. Fails setup transition. 
</pre>
<!-- Author: dan -->


<!-- Date: Tue Jun 12 15:00:26 2001 -->
<pre>Gave CCAL, L1, L2 back to shift crew.</pre>
<!-- Author: Stephen Miller -->


<!-- Date: Tue Jun 12 15:16:20 2001 -->
<pre>Finished runnning TDCtest on muons (except htdc) after Arnd's fixes.  
 
</pre>
<!-- Author: dan -->


<!-- Date: Tue Jun 12 15:18:51 2001 -->
<pre>VRB fibers rearranged for WCAL crates at patch panel.  Ace should test.  Hadron TDC cables moved from TSI chan 88 to 153.  Bill Badgett is updating hardware database as I type.  Should do reset, TSI, and VRB communication tests for b0htdc00.  Need all tests for b0fcal00.</pre>
<!-- Author: Jonathan Lewis -->


<!-- Date: Tue Jun 12 15:27:20 2001 -->
New run to test new L3 tagset 135, 36X36_1_Coupled (this name will be changed).  It test the L3 express stream. 
 
We not using SVT, WCAL, Muons, other people are working with these.
<!-- Author: Victoria, Anyes and Farrkur :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117517' target=_top>117517</a>) -->
<!-- Comment: Tue Jun 12 15;50;27 comment by...Victoria -->
Timeout from CCAL_01 slot3.  HRR fixed it.
<!-- Date: Tue Jun 12 15:41:35 2001 -->
<pre>End of run SVX rad numbers: 
 
W Inner BLM   .0008 
W Outer BLM   0 
E Inner BLM   .0077 
W Outer BLM   .264 </pre>
<!-- Author: Victoria -->


<!-- Date: Tue Jun 12 15:44:51 2001 -->
<pre>L3 express stream test seems to work. 
Data get written to streams 1, 6, 8, 9 with total rate 
.29Mb/s. 
The files do not yet get written to fcdfsgi2</pre>
<!-- Author: aces :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117517' target=_top>117517</a>) -->
<!-- Comment: Tue Jun 12 16;29;06 comment by...Anyes -->
<pre>I check again if the file were on fcdfsgi2 and I found two in data08:
_ATESTR.0000/ar01cb0d.0001test
_ITESTR.0000/ir01cb0d.0001test</pre>
<!-- Date: Tue Jun 12 16:01:47 2001 -->
<pre>
Calorimetery QIE calibration at the request of Karen. 
The look-up tables got corrupted today (because of the power cut?) so we need to derive new clibration constants. 
 
Software event builder dosen't initalise on partition 0 -> reset -> select partition 1. 
 
We try to load the new calibration constants (as Karen told us to)  we get the error: "***Error: CalibLoad not allowed without LoadQIEFRAM***" </pre>
<!-- Author: Victoria & Anyes -->
<!-- Comment: Tue Jun 12 18;29;00 comment by...W.Badgett -->
<pre>The expert should have told you to click the 
"LoadQIEFRAM" check box in the Expert Run Settings.
</pre>
<!-- Date: Tue Jun 12 16:06:29 2001 -->
<font color=red>Shift Summary: </font><pre>Started with recover from power trip. 
Power on west endwall then tripped with drip sensor. 
Priority to L2 work. 
New L3 exe (coupled). 
Accelerator still trying to get a pbar only store.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   117496-117518
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             
   
</td></tr></table></pre>
<!-- Author: Tom -->