<!-- Title: Owl shift. Wed Apr 11, 2001 -->
<!-- SciCo: Jimmy Proudfoot -->
<!-- DAQAce: Vladimir Nagaslaev -->
<!-- MonAce: Ben Kilminster -->
<!-- CO: Joao Guimaraes da Costa -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre>Last muon trigger rate test run in progress.</pre>


<!-- Date: Wed Apr 11 00:17:55 2001 -->
<pre>MCR - ready to inject beam into Tevatron in about 10 mins. No problems here - we've just completed the muon test run. 
</pre>
<!-- Author: Jimmy Proudfoot -->


<!-- Date: Wed Apr 11 00:21:36 2001 -->
<pre>Included CMU E16 and W20 to the muon matchbox  
with threshold set at 30 (all the rest are set  
at 20, however). The trigger rates (single muon 
stub triggers) look ok.</pre>
<!-- Author: hyunsoo :: (run <a href='/cgi-bin/runSummaryRun.cgi?111704' target=_top>111704</a>) -->


<!-- Date: Wed Apr 11 00:33:35 2001 -->
<pre>Plug High voltage alarm summary for the night: 
</pre>
<!-- Author: Ben -->
<!-- Comment: Wed Apr 11 00;34;06 comment by...Ben -->
PSH crate 11 tweets at 12:15 am.
<!-- Comment: Wed Apr 11 01;33;28 comment by...Ben -->
PHA crate 11 tweets at 1:30 am.
<!-- Comment: Wed Apr 11 01;48;39 comment by...Ben -->
PEM HV crate 11 tweets at 1:45 am.
<!-- Comment: Wed Apr 11 02;04;13 comment by...Ben -->
PHA and PSH alarms for crate 11 go off at 2:00 am.
<!-- Comment: Wed Apr 11 02;45;56 comment by...Ben -->
PHA crate 2 2:45 am.
<!-- Comment: Wed Apr 11 03;29;46 comment by...Ben -->
PHA crate 11 3:30 am
<!-- Comment: Wed Apr 11 03;34;38 comment by...Ben -->
<pre>PSH crate 11  3:34 am.</pre>
<!-- Date: Wed Apr 11 00:45:31 2001 -->
<pre>WATER alarm from CHILLER TE66. 
Temperature went over high limit by 0.1 deg F. 
It quickly fell back into tolerance.  Cryo Room called 
and said that they have been observing this chiller. 
</pre>
<!-- Author: Ben -->
<!-- Comment: Wed Apr 11 01;43;01 comment by...Ben -->
Alarm repeats and clearts at 1:42 am.
<!-- Comment: Wed Apr 11 01;51;45 comment by...Ben -->
And again at 1:51 am.
<!-- Date: Wed Apr 11 00:57:40 2001 -->
Starting Rick's run. L3 fails partition transition. Error state 
in b0l3007. Run configuration has subfarms 0 and 4. Changed 0 to 
1. Now running. Trigger rate 190 Hz. 

<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111705' target=_top>111705</a>) -->
<!-- Comment: Wed Apr 11 01;06;10 comment by...Vladimir -->
<pre>The error message was:
MLE) b0l3gate2.fnal.gov:main:12:36:21 AM->Error on L3 node b0l3gate2 (partition 0) Wed Apr 11 00:36:20 2001 l3proxy 
  in Initialize (L3ProxyPartition.cc:788)
  @L3_TRANS_FAIL - transition actions failed:      b0l3007(Target failed)

Looking back to the note from April 6 18:03:54. There was an
error in this node b0l3007, but the blamed was PCAL10. Could
it be just the problem of the processor node?
</pre>


<!-- Date: Wed Apr 11 01:08:54 2001 -->
Rick's run stopped with 169K events.
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111705' target=_top>111705</a>) -->


<!-- Date: Wed Apr 11 01:11:46 2001 -->
<pre>Scanning over SLow Controls IFIX for red things: 
 
RACKS ->  
   2nd floor cooling -> 
      2RR 31C: AC power ON alarm, double fan failure 
      32C    : "" 
      33C    : "" 
      35C    : AC POWER ON, DRIP Detector, Power permit, water VLV open, double fan failure 
 
  SVX Set Detector Rack : AC Power ON, Fan failure, power permit. 
 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Wed Apr 11 01:28:29 2001 -->
PCAL is released to Alexei. 
Running the SM QIE calibrations with new RC. CCAL only. <br>
Successful. Calibrations reached the database. <br> 
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111706' target=_top>111706</a>) -->






<!-- Date: Wed Apr 11 02:01:43 2001 -->
<pre>Silicon Cooling Alarm,  ISL. 
CHSUP - Chiller outlet temp dropped to 4.5 deg F. (lo limit is 5) 
Silenced by techs in cryo area. 
Seems to be related to water chiller alarms reported above. 
Cryo techs looking into it.</pre>
<!-- Author: Ben -->
<!-- Comment: Wed Apr 11 05;44;16 comment by...Ben -->
<pre>Same alarms come back at 5:40 am.</pre>
<!-- Date: Wed Apr 11 02:08:23 2001 -->
<pre>There are uncoalesced 150 GeV protons in Tevatron. 
CLC tight min bias triggers are seen to go up to 66 Hz at  
one point.</pre>
<!-- Author: Ben -->


<!-- Date: Wed Apr 11 02:20:18 2001 -->
<pre>Running integration from ACE_APR_9_NEWRC_NO_SI, 
trigger DRY_RUN_TABLE[3,83]. No PCAL, no silicon, CCAL04. 
First and last subfarms excluded. 
L2 processor on.
Very long ColdStart transition.  
RC reports missing cards in HTDC slot 4- TDC_TIMING_00 
			    CMX  slot 10 TDC_CMX_16 
                                 slot 11 TDC_CMX_17 
				 slot 15 TDC_CMX_20 
Trigger rate 107 Hz.
Event size 66K.
</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111707' target=_top>111707</a>) -->
<!-- Comment: Wed Apr 11 02;28;38 comment by...Vladimir -->
Stopped with 100K events.




<!-- Date: Wed Apr 11 03:02:49 2001 -->
<pre>Starting another run. No way to integrate Silicon: it is off. 
  Cards in cmx01 and htdc00 are added by the standard trick 
of removing/adding the crates in the configuration, but 
it turned out that there are physically no cards in slots 10,  
11 and 15 in cmx01. Removed them back. 
 
  Often partition transition hangs at one of the crates 
(b0l1gl00, tsi00, tsi02, for instance) Those crates looks 
fine. After Abort/Reset transition either stops at another 
crate or goes on and finishes ok. 
Running</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111709' target=_top>111709</a>) -->



<!-- Date: Wed Apr 11 03:34:06 2001 -->
<pre>IFIX Heartbeats: 
COT heartbeat is pink but sometimes flashes green. 
PSM heartbeat pink. 
GAM heartbeat purple.</pre>
<!-- Author: Ben -->



<!-- Date: Wed Apr 11 03:42:46 2001 -->
PHA crate 11 3:40 am.
<!-- Author: Ben -->


<!-- Date: Wed Apr 11 03:43:55 2001 -->
PHA crate 11 3:40 am.
<!-- Author: Ben -->


<!-- Date: Wed Apr 11 04:21:58 2001 -->
<pre>I've tested my program today. I used one source : West 3, 
channels 7 & 8 (hadron). Some part of the program didn't  
work so I plan to continue my debugging tomorrow.</pre>
<!-- Author: Alexei Sedov -->


<!-- Date: Wed Apr 11 04:30:39 2001 -->
<pre>NEW RC 
 
I can't get settings for configuration message (PCAL_SOURCE_00) 
in a reasonable amount of time. It takes more than 10 minutes to 
get everything  in the "CalorCalibSet: PCAL_SOURCE_00" so it's 
impossible to work with it.</pre>
<!-- Author: Alexei Sedov -->


<!-- Date: Wed Apr 11 04:41:18 2001 -->
<pre>Put PCAL back into RC. Took the short run with it. 
Trigger rate jumped from 100 to 170Hz.   30K events. 
</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111714' target=_top>111714</a>) -->


<!-- Date: Wed Apr 11 05:05:07 2001 -->
<pre>Running SM QIE calibrations in West Plug only. 
Partition hung- tried to switch to partition 1 (from 0). 
Partition passed quickly. 
  Run ended successfully, no error messages. Tables 
appeared in the database about 5 minuts later.</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111717' target=_top>111717</a>) -->


<!-- Date: Wed Apr 11 05:23:46 2001 -->
<pre>Tried partition 0 again- transitions pass fine. 
Now SM QIE calibrations for the East Plug ShowerMax. 
Errors, DB tables not created. 
 
MLE) b0dap62.fnal.gov:sevb:5:22:29 AM->Reformatter error: 
Returning code: 2; position: 5 
Line: 345; file: src/library/rr_rawevt.c 
Description: @L3_REFORMAT_ERROR 
 
RAWREF Error - MINI-INSTA - with code 2 -- START 
 Line -  345 in: src/library/rr_rawevt.c 
 Buffer position: 5 
 Error count: 1 of 1 
 Reject rate: -99.000 percent 
 
 Instance number of mini fragments out of 
 allowed range. It as to be positive/non zero. 
 Ntotal      Ninstance 
  0x00000008  0x0000000b 
 
 Error location code: 0x00000000 
 
 
 Crate Id?: 0x00000000 
 
 Error during scanning MINI structure! 
 Last Unit prob. worked on (all counts from 0): 
  SCPU: 0 
  VRB : 0 
  LINK: 0 
  MINI: 0 
 
 Buffer (0 until posi+2=7) 
 0x43534550  0x0b000800  0x00000000  0x01880200  0x03000000  0x0000200a  
 0x06e5493e  0x355e1a40  
 
</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111719, 111725' target=_top>111719, 111725</a>) -->


<!-- Date: Wed Apr 11 05:24:04 2001 -->
Tested new calibration/LUT code for central crates in ccal04. Bugs==true & I will need to try again after I stare at the code long enogh. I reloaded the default code and re-ran calibration program to correctly load the LUTs in the 3 SMXRs in this crate.
<!-- Author: Jimmy Proudfoot -->


<!-- Date: Wed Apr 11 05:38:37 2001 -->
<pre>Run Control slowed down on b0dap59 so much (updating windows), that I had to restart it from scratch. No other processing 
were running on the dap59 console, and the processor CPU load  
was low.</pre>
<!-- Author: Vladimir -->



<!-- Date: Wed Apr 11 05:53:26 2001 -->
<pre>Running SMX calibrations in East plug, excluding one by one 
crates, aiming to find a culprit. 
 remove pcal07    errors 
	pcal07 back, pcal08 off	    errors   run 111727 
	08 back		10  off     errors	 111728 
	10 back		11  off	    errors	 111729 
apparently, this tackle doesn't help. 
</pre>
<!-- Author: Vladimir -->


<!-- Date: Wed Apr 11 05:59:09 2001 -->
<pre>Si cooling alarm - this time overtemperature. Not clear what is going on downstairs with chillers & so page SVX Cooling expert for advice.</pre>
<!-- Author: Jimmy Proudfoot -->


<!-- Date: Wed Apr 11 06:14:45 2001 -->
<pre>  Running integrated system again to make sure it works 
before this morning access. Resources: 
  CCAL, PCAL, WCAL, COT, CMU, CMX, IMU, MUSC, HTDC, CLC, XFT, 
L1CL, L1GL, L2CL, L2DE_00, VRB, L1, L2, L3, SCALERS. 
L3 display's dead. 
  Stopped with 50k events.
</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111731' target=_top>111731</a>) -->


<!-- Date: Wed Apr 11 06:25:21 2001 -->
<pre>Restart after L3 cleanup. 
This time crates PCAL07, 08, 10, 11 released for playing in 
a separate partition.</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111732' target=_top>111732</a>) -->


<!-- Date: Wed Apr 11 06:33:56 2001 -->
<pre>MCR: they are bringing the TeV down to prepare for access.</pre>
<!-- Author: Jimmy Proudfoot -->


<!-- Date: Wed Apr 11 06:42:08 2001 -->
<pre>111732 stopped with 55k events. COT and XFT dropped on 
request. PCAL 7,8,10,11 are back. 
</pre>
<!-- Author: Vladimir :: (run <a href='/cgi-bin/runSummaryRun.cgi?111735' target=_top>111735</a>) -->


<!-- Date: Wed Apr 11 06:48:23 2001 -->
<pre>ISL Chiller temperature alarm goes off for 
an over temperature. (rises from 5 deg F to 30 deg F) 
Alarm silenced by techs.  Followed SVX monitoring guide and 
paged Steve Blusk. 
 
We went through displays and decided that things were safe  
at the moment since several other temperatures were okay  
and the chiller temp had since returned to within tolerance. 
 
Noticed problems with IFIX.  Some links were coming up pointing 
to the wrong pages.  Steve recommends I restart IFIX on  
svxiicon.  Tried to do this, but was unable to restart due 
to some other application still using IFIX. 
 
Only other applications running were PSMON (which was looping 
meaninglessly) and a Mircea program.  Restarted PSMON, IFIX still 
wouldn't start. 
 
Paged Steve again and called Mark Knapp.  Steve will be here 
in half an hour to take a look.</pre>
<!-- Author: Ben -->
<!-- Comment: Wed Apr 11 07;05;02 comment by...Ben -->
<pre>Mark Knapp helps me restart IFIX by logging out of svx pc and
logging in.  Restarted PSMON process, and Mircea process. 

Went upstairs, and IFIX upstairs is not displaying 
proper information for SVX.  Perhaps not all processes started
correctly.  Steve should be here soon to look at this.
</pre>
<!-- Date: Wed Apr 11 06:58:27 2001 -->
<pre>Expansion tank level exceeded maximum very briefly and went into alarm. No big deal according to the cryo/gas tech.</pre>
<!-- Author: Jimmy Proudfppt -->



<!-- Date: Wed Apr 11 07:06:54 2001 -->
<pre>WATER alarm as chiller temp TE66 goes up over 45 deg F. 
Looks like Cryo guys raise the HILIMIT to 46.0 deg F to  
quiet alarm down. 
 
This appears correlated to ISL over temp mentioned above.</pre>
<!-- Author: Ben -->


<!-- Date: Wed Apr 11 07:23:58 2001 -->
<pre>Kevin Lannon: Work to begin on CMP HV - ignore it for the timebeing (HV may go up and down)</pre>
<!-- Author: Jimmy Proudfoot -->


<!-- Date: Wed Apr 11 07:28:22 2001 -->
<pre>Jim Ellermeier - they will be working on ISL and SVX cooling systems. It will last about 2 hrs. Ignore alarms.</pre>
<!-- Author: Jimmy Proudfoot -->


<!-- Date: Wed Apr 11 07:54:23 2001 -->
<pre>Steve is in to listen to the host of WATER and SILICON  
Cooling alarms going off from CHILLER over-temps and  
TE-CHSUP Chiller outer temp fluctuations. 
 
There is a problem with chiller but most alarms are from  
the shutdown of silicon cooling because of the access. 
 
Rich Stanek calls in and will take a look today (in  
addition to Steve )</pre>
<!-- Author: Ben -->


<!-- Date: Wed Apr 11 07:59:30 2001 -->
<font color=red>Shift Summary: </font><pre>Completed the final in a set of muon trigger rate test runs 
 
Completed Rick's Run (ACE Calorimeter Minimal) 
 
Alexei did more work on plug source (modest step forward) 
 
Ran integration test with most systems except Si 
 
Ran SMX calibrations -> datbase for central and West plug crates 
 
Tested modified SMX calibration code - not successfull 
 
Had several SVX cooling alarms - paged several experts. Techs are working on this system. 
 
FIX Heartbeats:  
        COT heartbeat is pink but sometimes flashes green.  
        PSM heartbeat pink.  
        GAM heartbeat purple. 
 
A few RC features: 
        Often partition transition hangs at one of the crates  
        (b0l1gl00, tsi00, tsi02, for instance) 
         
        It takes more than 10 minutes to get everything  in the  
        "CalorCalibSet: PCAL_SOURCE_00" (Alexei) 
 
        Run Control slowed down on b0dap59 so much (updating 	windows),  
        that AceI had to restart it from scratch 
 
Controlled chaos started at 7am.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   111704-111735
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             100
   
</td></tr></table></pre>
<!-- Author: Jimmy Proudfoot -->