<!-- Title: Owl shift. Wed Jul 25, 2001 -->
<!-- SciCo: Christoph Paus -->
<!-- DAQAce: Heather Ray -->
<!-- MonAce: Rob Napora -->
<!-- CO: ---- -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre>Shift Plan 
========== 
 
Finish Level2 tests. 
 
Integrate Silicon. 
 
Finish up the left over calibrations. 
 
Test new two track trigger table (magnet on, COT HV on) 
</pre>


<!-- Date: Wed Jul 25 00:19:36 2001 -->
Masked off slot 4 in CCAL08 with cardEditor.  I had thought this SMX card was swapped out earlier this week?
<!-- Author: hray -->
<!-- Comment: Wed Jul 25 01;42;35 comment by...hray -->
<table border=0><td><tr><td bgcolor=#e0e030>note - the persons using the system before I came on shift said that there were problems with this slot and they removed the entire crate from running.  I did a search of the log book from 7/14/01 for "ccal08".  There is mention of Mike and Karen not finding anything wrong with this slot and asking the shift crew (me, during day last week) to include it in the run and call them if any problems occur.  At the next end of run I will unmask the slot from cardEditor and try running to reproduce problems which occured earlier</td></tr></td></table>
<!-- Comment: Wed Jul 25 09;09;15 comment by...R.J. Tesarek -->
<table border=0><td><tr><td bgcolor=#e0e030>Heather is absolutely correct in removing only the slot.  I made
this comment to the previous shift crew, but was ignored and we
lost data from this crate in all the previous runs.  This situation is unacceptable.  When in doubt, page the expert and
get their opinion *BEFORE* removing anything.</td></tr></td></table>
<!-- Date: Wed Jul 25 00:26:36 2001 -->
<pre>I was called in because Plug source runs failed in Setup with 
messages like 
 
Partition:3 Run number: 121502 (0x1da9e) 
(MLE) b0dap55.fnal.gov:AWT-EventQueue-0:10:44:34 PM->*** Error fetching L3CalibTag info for run 121502 
(RC)   22:44:37 Setup -> ERROR 
(MLE) b0l3gate2.fnal.gov:main:10:44:37 PM->Error on L3 node b0l3gate2 (partition 3) Tue Jul 24 22:44:36 2001 l3proxy 
  in Setup (L3ProxyPartition.cc:958) 
  @L3_INVALID_TAGS - invalid CALIB tag 
 
As a workaround, I told Alex to use the DAQ state manager instead 
of the 'Generic Calibration' - something is broken in the 
calibration state manager. A fix will have to wait until 
tomorrow. 
</pre>
<!-- Author: Arnd -->


<!-- Date: Wed Jul 25 00:28:22 2001 -->
Finished L2 reces studies.  Details in <a href= 
'/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=13&frame=2&anchor=00:19:42'>L2 e-log</a>.
<!-- Author: Saltzberg and Worcester -->


<!-- Date: Wed Jul 25 01:34:16 2001 -->
We got the system back from L2/Silicon.  Added CCAL08 (with slot 4 masked out) and SVT07 (slot 11 added) to run.  Per Arnds request we try to change partitions to 1.  We are unable to start a run.  Verified with Ilya that EVB doesn't see any triggers.  When the plug sourcing is finished for the night and we have the entire L3 system in one partition we will try to clean up EVB and attempt runs in partition 1 again.  (runs 121529,121531,121533)
<!-- Author: hray -->
<!-- Comment: Wed Jul 25 06;04;31 comment by...Napora -->
<p>Silicon was integrated starting with run #121525.
<!-- Date: Wed Jul 25 01:35:39 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>subfarm b0l3079 will remain "pink" until Ilya fixes it.  It should not cause any problems with running.</td></tr></td></table>
<!-- Author: hray -->



<!-- Date: Wed Jul 25 02:23:33 2001 -->
<p>CMX West 21 tripped. Reset.
<!-- Author: Napora -->
<!-- Comment: Wed Jul 25 02;49;01 comment by...Napora -->
<p>And again.  Wait 20 minutes and reset.
<!-- Comment: Wed Jul 25 03;09;45 comment by...Napora -->
<table border=0><td><tr><td bgcolor=#e0e030><p>It tripped again.  I am going to leave it on standby for the night and e-mail Hyunsoo about it.</td></tr></td></table>
<!-- Date: Wed Jul 25 02:25:25 2001 -->
<ul> 
<li>Run 121537 get several BTO's (it claims from VRB_SVX_02).  one DTO from this crate, as well.  End run to remove Silicon to see if BTO's go away (ie is it really Silicon's fault?) 
<li>Keep getting BTO's.  End run (121540), coordinate cleanup of EVB and L3 with plug sourcing people 
<li>add back in Silicon.  add in WCAL (not in last runs - no mention in log book as to why they were removed and not included again).  add slot 4 in ccal08 back in.  try again 
</ul>
<!-- Author: hray -->


<!-- Date: Wed Jul 25 02:27:46 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>Notice with Cosmics_All[2,165] there is a total DT of 99.9%, all of it coming from Readout.  L1A,L2A are around 80 Hz, but FredL1A is 1.7 MHz!  Is this table correct?  Is a prescale not set/threshold table value wrong?  </td></tr></td></table>
<!-- Author: hray -->



<!-- Date: Wed Jul 25 02:53:50 2001 -->
<p>|| <font color='red'>COT Cooling alarm on PDT-SEP PC Storage Diff. Pressure</font>.  It drops below the low limit (<1 PSID) and then comes back up for a short while and drops again.  It has done this three times in the last couple of minutes.
<!-- Author: Napora -->
<!-- Comment: Wed Jul 25 03;06;38 comment by...Napora -->
<p>Spoke with the Cryo-Techs about this.  The flow in the SUVA tank causes Pump 7 to temporarily turn off which causes the differential pressure to drop.  When the pump turns back on the pressure comes back up.  It went throught this iteration about 5-6 times, but they assure me it is (reasonably) normal operation.  Things have been stable again for the last few minutes.
<!-- Comment: Wed Jul 25 04;20;44 comment by...Napora -->
Ditto.
<!-- Comment: Wed Jul 25 07;43;36 comment by...Napora -->
<p>...and we are having one more iteration of these cooling trips to celebrate the end of this shift!
<!-- Date: Wed Jul 25 02:54:01 2001 -->
<ul> 
<li>Run 121544 get DTO from CCAL08 - SMXR in slot 4 not responding: header x3bfe46a, so this is the same error as before - it should be x38, not x3b.   
<li>end run and removed this card via cardEditor 
<li>Run 121546 get L3 errors from the start and are unable to run (see below) 
</ul>  
<!-- Author: hray -->
<!-- Comment: Wed Jul 25 13;09;03 comment by...Boris.  -->
<pre>RC sent us 0 CALIB tag number. 
The problem is fixed (?) by Bill Badget </pre>
<!-- Date: Wed Jul 25 02:55:03 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1678&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1678  width='300' align='top'></td><td>This is the L3 error that sent RC into an error state during coldstart for run 121546</a></td></tr></table>
<!-- Author: hray -->


<!-- Date: Wed Jul 25 03:00:26 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1679&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1679  width='300' align='top'></td><td>Run 121548.  Got BTO at very start from vrb_svx_02.  HRR and  we see this error - also one I haven't seen before.  BTO's comming  from all parts of detector...</a></td></tr></table>
<!-- Author: hray -->




<!-- Date: Wed Jul 25 03:21:19 2001 -->
<pre>Sourcing Plug from 22:00 until 03:15.  First two hours spent getting hardware event builder to work with source calibration running.  Remainder of time sourced 10 tubes (per quadrant) in 10 separate runs:  all of sector 0  PEM and PHA, half of sector 1, PEM only. (plus corresponding sectors in other quadrants).  Numerous problems with run control.  Even when no problems, setup of each run takes as long as poking one tube. 
The run numbers are 121521, 524, 526, 532, 534, 536, 538, 539, 541, and 547.   </pre>
<!-- Author: A. Sedov, V. Barnes  -->


<!-- Date: Wed Jul 25 04:15:01 2001 -->
<p>CMX West 6 tripped.  Reset.
<!-- Author: Napora -->
<!-- Comment: Wed Jul 25 04;22;43 comment by...Napora -->
<p>Again.  Wait 5 minutes and reset.  It trips again--won't even go to standby.  Turn it off and send a message to Hyunsoo.  Humidity in the collision hall is around 35% at this time.


<!-- Date: Wed Jul 25 04:38:22 2001 -->
<ul> 
Did several calibrations.  As requested by Vaia today in ace meeting I checked the Consumer gui under Proxy on RC gui...I watched the consumers and noted when they started/stopped : 
<li>CAL QIE run 121551.  get two failure errors - from wcal05 and wcal07.  Vaia said this would be a "normal" amount, and to proceed with calib_load (after verifying this calibration made it to the DB and examining the log file for errors).  I did calib_load for the entire calorimeter (called Vaia to verify).  she said this will ensure all parts have the same tag.  vxlogin of one each of pcal,ccal,wcal crates shows correct run number loaded 
<li>CAL QIE run 121552.  get three failures - wcal04, ccal12, ccal05.  done after calib_load to check all went well.  Consumer = QIE.  this stayed on until I switched from Qie to Generic calibration, three calibrations later...(cot) 
<li>SMX QIE run 121553.  okay.  Consumer = CES CALIB.  stayed on until switched from Qie to Generic calibration 
<li>TOF run 121554.  showed a lot of failures in all wcal crates.   
Consumer = TOF QIE.  stayed on until switched from Qie to  Generic calibration 
<li>COT run 121555.  okay.  Consumer = COT CTT.  all consumers cleared, and this consumer stops when end of calibration reached 
<li>MUON run 121556.  okay.  Consumer = NONE SHOWN.  This calibration did NOT make it to DB. 
<li>CLC QIE run 121557.  one failure in ADMEM 16, CLC00.  Consumer = QIE.  consumer stops when end of calibration reached 
<li>CAL LED run 121558.  okay.  Consumer = LED_0.  consumer stops when end of calibration reached 
<li>XENON FPED run 121559.  okay.  Consumer = XEF_0.  consumer stops when end of calibration reached 
<li>XENON ALL run 121560 fails setup - all ccal crates show no SEVB error 
<li>XENON ALL run 121561.  okay.  Consumer = XEF_0. 
<li>conclusion: do calibrations in such an order as to avoid conflicts between consumers that take awhile to stop.  All calibrations are then written to db.
</ul>
<!-- Author: hray -->


<!-- Date: Wed Jul 25 04:58:04 2001 -->
Run 121562 attempted Rick Run.  several BTO at beginning of run.  End and clean up L3 and EVB.  Run 121563 Rick Run successful and ended with 36K.  one minor(?) problem with L3 = all subfarms of CN13,14,15,16 turned green and stayed that way the entire run.  The rest of L3 flashed green/dk blue and behaved.  
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121563' target=_top>121563</a>) -->
<!-- Comment: Wed Jul 25 08;48;42 comment by...Boris.  -->
<pre>b0dau32d is down again. Page CSL.</pre>
<!-- Date: Wed Jul 25 05:22:39 2001 -->
<p>The solenoid has been ramping up.  The COT has been set to standby.
<!-- Author: Napora -->


<!-- Date: Wed Jul 25 05:46:17 2001 -->
<p>Solenoid is up.  COT and TOF turned on.
<!-- Author: Napora -->



<!-- Date: Wed Jul 25 06:06:37 2001 -->
<ul> 
<li>run 121567 everything times out waiting for r/o list 
<li>Run 121568 with Test_Two_Track[10,165], Solenoid on, COT and TOF HV on.  L1A/L2A = 90 Hz.  Fred L1A = 1.7 MHz. Rates: L1_MB_XING_PS1M = 1.7 Hz.  L1_TWO_TAU0_PT2_OPP, L1_CEM0_PT4_PS1, L1_TWO_CEM0_PT2_OPP, L1_CMU0_PT4_PS1 all = 1.7 MHz.   
<li>the email received about this said two things: people would like 5 - 10K events to check this table, and it should be run for 30 - 60 mins.  Running for 5 mins got us 35K events.  Are the rates okay for this table? 
</ul>
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121568' target=_top>121568</a>) -->


<!-- Date: Wed Jul 25 06:08:21 2001 -->
During Run 121568 see the last 4 CN of L3 turning green again, and not doing anything during the run.  Also get a lot of DTO and BTO from all crates, most notably SVX/FIB.  DTO end run at 70K.  Will clean up EVB and L3 again to 1. see  if this helps DTO (I realize DTO are normally FE crates fault, but we're seeing most of the DTO message without any message telling us which crate it is coming from, along with SCPU VRB error messages on RC error screen), 2. see if this helps CN green problem
<!-- Author: hray -->
<!-- Comment: Wed Jul 25 08;01;23 comment by...SCN -->
<a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=55&frame=2&anchor=08:00:23'>
Comment</a> on TOs from Si
<!-- Date: Wed Jul 25 06:17:59 2001 -->
Getting a lot of "Failed to book any crates" errors in RC.  Have to either exit partition and reselect, or exit RC, cleanup, and restart.
<!-- Author: hray -->


<!-- Date: Wed Jul 25 06:27:19 2001 -->
<p>Solenoid ramping back down.  COT put on standby, TOF turned off.
<!-- Author: Napora -->


<!-- Date: Wed Jul 25 06:57:07 2001 -->
Run 121569 ErrorLogger fails CS.  Reset and partition.  Run 121570 get several BTO and DTO, both from VRB_SVX_02.  Have to hit the Silicon crash button at 68K.  During this run the last 4 CN on L3 were again green the entire time.  Apparently cleanup of L3 and EVB didn't help.  Waiting until Boris comes in for 8 am meeting, to ask him to reboot b0l3u04 (Christoph's idea)
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121570' target=_top>121570</a>) -->
<!-- Comment: Wed Jul 25 07;23;57 comment by...hray -->
of course, we removed all Silicon from the runs after the crash button was hit...
<!-- Date: Wed Jul 25 07:17:17 2001 -->
<p>||<font color='red'>SVX Cooling Alarm</font>.  Pretty much the exact same thing as occured <a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=513&frame=2&anchor=15:04:22'>yesterday</a>.  A L00 gas trip caused flow to be lost on the east side.  This resulted in sono-alarms and brought down the silicon racks (normal?) which in turn gave us foghorn alarms.  Cryo-techs were able to get things working again in a few minutes.  Main silicon pager was notified.
<!-- Author: Napora -->
<!-- Comment: Wed Jul 25 07;37;34 comment by...Napora -->
<p>Of course when the racks went down the GUI promptly hung (it has difficulty communicating with crates that aren't turned on.)  When the racks came back on after the mandatory 10 minute waiting period we got the expected 'tweety bird' alarms from the lack of communication. I had a difficult time getting the GUI back up and eventually resorted to reseting <i>svxinter</i> by hand, but it is up now.
<p>When I first restarted the GUI I got another tweety-alarm on the SVX::<i>SVX_PS_TRIP  SVX Power Supply Trip</i>.  I do not think this is real since no power supplies had been turned on since the rack regained power.  I attributed it to GUI wierdness and reset.
<!-- Comment: Wed Jul 25 07;47;29 comment by...Napora -->
<p>Sono-alarms were unsilenced shortly after the problem was fixed.
<!-- Date: Wed Jul 25 07:23:20 2001 -->
Run 121571 get several BTO from CMX_01 and from "unknown" crates (no messages anywhere).  L3 SF13 pink.  End run and try coldstart.  Run 121572 again several BTO from CMX_01.  end and manually reboot TSI00.  -> From our L3 info: BTO are from EVB (HRR or cleanup fixes), SM (need to reboot eb10), TS(reboot tsi00), or downstream of L3/CSL.  So far this shift we've tried HRR and cleaning up EVB/L3.  Check of SM/EVB/L3 doesn't show anything hung on eb10.  Now trying reboot of TSI00......
<!-- Author: hray -->



<!-- Date: Wed Jul 25 07:35:05 2001 -->
Run 121573 started with Cosmics_All[2,165] after reboot of TSI00 - so far at 80K with no BTO.  Looks like that did the trick
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121573' target=_top>121573</a>) -->





<!-- Date: Wed Jul 25 07:54:50 2001 -->
<font color=red>Shift Summary: </font><pre>Shift Summary 
============= 
 
No beam in the machine, search and secure training in progress. 
 
Level2 tests finished, with not too conclusive results. More tests needed. 
 
Source runs did 10/52 tubes. Procedure is really painful and slow. 
 
CCAL08 crate had problems again. Experts should check this out. 
 
Silicon integration went smoothly. 
 
Did all calibrations. All fine except for the known problem with muon 
calibrations related to consumer not starting up. 
 
Tested trigger table TEST_TWO_TRACK v10. Strange results which need to 
be looked at by experts. 
 
Silicon cooling alarm went off! 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   121529 - 121573
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             100%
   
</td></tr></table></pre>
<!-- Author: Christoph Paus -->



<!-- Date: Wed Jul 25 08:38:09 2001 -->
<br><b>b0pcal01, crate hang:</b> 
<br>Wandered into the control room to great fanfare, 
<!-- Author: R.J. Tesarek -->