<!-- Title: Eve shift. Sat Oct  6, 2007 -->
<!-- SciCo: Frank Chlebana -->
<!-- DAQAce: Glenn Strycker -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>One hour into the shift...</pre>


<!-- Date: Sat Oct 6 16:34:45.861 2007 -->
<pre>Beginning of Shift 
Short Status Summary:  
-------------------------  
 
- currently not running anything 
  - L3 guys are working, so they are using the event builder and other critical systems 
  - Paolo and Fotis were working on WHA, so those crates were removed for a while. 
  - Cosmics was run in previous shifts with "ignoring reces" set, without SMX_15 in CCAL05 
    (that is, SMX readout card 15 (slot 3) from ccal05), and without the whole SVT_08 crate. 
 
- cooling rack problem from Sunday seems fixed 
 
- all crates are green on the VxWork monitor (but some other bad crates listed below) 
 
- HV status:  
  - ON  (+HV):   CHA,WHA,CLC 
  - ON  (no HV): MNP,BSC 
  - NOT (+HV):   GAM  
  - STD:         PEM/PHA/PSH  
  - OFF:         everything else  
  - WARN:        CEM (yellow) -- see note above 
 
Global Alarms status: 
  - Red:    PSM,PTM  (see "known problems" below) 
  - Yellow: CEM      (see "known problems" below) 
  - Green:  everything else 
  - all heartbeat are ok (green), although sporatic COT heartbeat problems 
  - no "pink" or "grey" heartbeats 
 
- Process System Alarms:  
  - status seems to be consistant with known problems 
  - yellow: GAS, COT, PSM (new) 
  - green: CRYO, HVAC, WATER, SILICON, RACKS, SOLENOID 
 
- Racks not in use 
  - CCAL05 slot 3 (SMX READOUT 15) 
  - SVT_08 
  - WCAL_04 should be checked (previously had problems) 
  - all other racks in the L2_TORTURE [19,952,639] mode are presumably working 
  - all other racks in COSMICS mode [15,954,639] are presumably working 
 
- Known issues  
  - TOFQIE calib is not working properly 
    - We've had a request to troubleshoot TOFTAC and narrow down which crate(s) 
      and which ADMEM(s) are bad. 
  - in CEMLED, CALXEF, and TOFTAC calibs, wedges 9-11West are still empty 
  - Since my last shift (Sunday), CLC now has HV turned on (apparently Thursday). 
  - PTM is red on the Global alarms due to high temperatures on PW12 and PW11. 
    PW11 is sporatic, so whenever it becomes ok and then goes red again, a nasty 
    warning alarm sounds. 
  - PSM is red --> 
     -First floor: 1RR18I_2 and 1RR18F_1 are red 
     -End Plug:    SMD01E and SMD12E are red 
     -Arches:      18,19,20,21West and 04,05East are red 
     -End Wall:    CMP_T and CONET_4 are red 
  - CEM is periodically turning "yellow" (warning) on HV and Global alarms, but this is 
    known.  When I clicked the details button, many channels are flashing red-and-white, 
    so it appears to be a communications error. 
  - COT heartbeat was reported to have periodic problems. 
  - The Tevatron turned on at the beginning of my shift, so Channel 13, ACNET, and the 
    Tevatron crates all came back up.  Whenever the TEV crates restart, many sound alarms 
    go off (SVX dose, etc), so we silenced them.  We couldn't figure out how to turn the 
    silence off, though, so we need to follow up at the next Ace meeting 
  - Cosmics was run in previous shifts with "ignoring reces" set, without SMX_15 in CCAL05 
    (that is, SMX readout card 15 (slot 3) from ccal05), and without the whole SVT_08 crate. 
  - Possible noise reported in COT on Friday evening. 
 
- Shift plan  
  - COSMICS run to check L3 work 
  - run L2_TORTURE with all possible crates (see which ones are really out) 
  - run all calibrations 
  - Troubleshoot TOFTAC calibration 
  - end shift with a nice, quiet COSMICS run 
</pre>
<!-- Author: Glenn Strycker (Ace) -->
<!-- Comment: Sat Oct  6 16;40;39 comment by...Ulrich -->
Please check my comment in the e-log for the previous shift concerning the Silicon radiation alarms.


<!-- Comment: Sat Oct  6 16;48;12 comment by...Glenn Strycker -->
Thanks for your help!  The "silence" LED is still lit for the Integrated alarm, though, so I'll check the help pages you listed and perhaps call the RadCo pager if we can't figure out how to turn it off.
<!-- Comment: Sat Oct  6 17;03;39 comment by...Ulrich -->
The light with go off after you press the Alarm Acknowledge button. I would say it is fine to press it, but you can double-check with Steve Hahn if you feel unsure.
<!-- Comment: Sat Oct  6 18;29;48 comment by...Glenn Strycker -->
There is no "alarm acknowledge" button for this crate, only on the crate with the SVX Integrated Alarm.  This one is a "Diamond Integrated Alarm".  Willis pointed me to <a href="http://www-cdfonline.fnal.gov/ops/mcs/radmon/diamonds/">this</a> page that describes the updates to the Diamond monitor.  Evidently we need to make a different post and the experts turn this alarm off themselves.  I will make a post below.
<!-- Comment: Sat Oct  6 18;37;08 comment by...Glenn Strycker -->
Sorry about the confusion, Ulrich.  Thanks for your help!  Ricardo called and explained the alarm and the silence mode feature to us, so I think we got it now.  See the post below for more details.
<!-- Date: Sat Oct 6 16:45:35.050 2007 -->
<pre>Dell called. 
Power is back on and any new alarms from the gas shed should be 
taken seriously</pre>
<!-- Author: Chlebana -->



<!-- Date: Sat Oct 6 17:07:52.696 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250997>250997</A>
 Activated at 2007.10.06 17:07:08
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 17:07:53.407 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250997>250997</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 17:07:54.806 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250997>250997</A>
 Terminated at 2007.10.06 17:07:31
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 17:07:55.372 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250997>250997</A>
TERMINATE: It worked.
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 17:14:34.759 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250998>250998</A>
 Activated at 2007.10.06 17:14:25
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 17:15:31.902 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250998>250998</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 17:19:41.223 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250998>250998</A>
 Terminated at 2007.10.06 17:18:14
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 17:44:14.860 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250999>250999</A>
 Activated at 2007.10.06 17:44:04
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 17:49:41.758 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250999>250999</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 17:51:51.784 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=250999>250999</A>
 Terminated at 2007.10.06 17:50:56
<!-- Author: RunControl -->



<!-- Date: Sat Oct 6 18:02:12.304 2007 -->
<pre>Got the 6pm emergency pager test... 
Called operator to inform them.</pre>
<!-- Author: Chlebana -->
<!-- Comment: Sun Oct  7 10;31;13 comment by...dee  -->
you don't have to call them
<!-- Date: Sat Oct 6 18:16:43.797 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251000>251000</A>
 Activated at 2007.10.06 18:16:13
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 18:17:28.647 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251000>251000</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 18:23:43.504 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251000>251000</A>
 Terminated at 2007.10.06 18:22:10
<!-- Author: RunControl -->




<!-- Date: Sat Oct 6 18:35:12.000 2007 -->
Diamond Integrated alarm triggered 
<br><br> 
This alarm came on earlier when the TEVATRON power came back up (see previous posts at the end of the morning shift).  We silenced this alarm as instructed on the new Diamond Radiation Monitor page.  <br><br> 
We were worried about how to take the alarm out of silence mode.  Ricardo called and said not to worry -- this alarm will be reset automatically when the next store begins.
<!-- Author: Glenn Strycker -->



<!-- Date: Sat Oct 6 18:53:52.203 2007 -->
Level3 Work: 
 
We first recovered the monitoring system, from last night's tests. Then we tested the error handling system by forcing a relay failure. First attempt failed. We reverted to an older version of oracle which fixed the relay failure issue.  
 
Finally we tested the L3 filter executable delivery system from b0spool to our farm. The first step of this process takes forever (~10min) to get the package (80MB) from b0spool to the EVB gateway node. Currently, we don't understand why it is doing it.  
 
We have converted all subfarms except subfarm 1-4. Run control will have 4 subfarms to run with until the total conversion to SL4 which will probably happen some time next week.
<!-- Author: Si and Georgios -->

<!-- Date: Sat Oct 6 19:03:59.593 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251002>251002</A>
 Activated at 2007.10.06 19:03:52
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 19:05:21.026 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251002>251002</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] -- Ok, now were are finally attempting a "normal" cosmics run to check that we, in fact, can.  The L3 guys are supposedly done for the day, and only 4 nodes are available on the L3 farm.  The others have been converted to SL4 and will be unavailable until all of the problems are fixed.
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 19:06:23.787 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251002>251002</A>
 Terminated at 2007.10.06 19:05:43
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 19:06:24.175 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251002>251002</A>
TERMINATE: It worked ok.  I'm going to run an L2 Torture while the L3 guys are still here.
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 19:09:47.541 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251003>251003</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,952,639] 
<!-- Author: Glenn Strycker -->
<!-- Comment: Sat Oct  6 19;10;38 comment by...G.S. -->
Seems to have started ok and is now working.  Si and Georgios went home for the day.


<!-- Date: Sat Oct 6 19:35:23.535 2007 -->
We looked at the deadtime and prescales on the trigger rate window.  Frank adjusted the prescale on L2_PS151_L1_MB_XING_PS49 from 151 to 47.  We noticed the deadtime jump from 4.9% to about 9% -- it later came down to around 6%.  Then he changed the prescale to 37 and the L2 rate went to 942 Hz (7% deadtime), which is up from the L2 rate I've seen previously. 
<br><br> 
Meanwhile, we saw on the Return/Done Crosspoint window that the last crate to be done is always Cot09.  We looked on the consumer monitor, and the DAQMon Cot plots showed a slow crate for COT_09.  Frank pointed out that there is noise on the Event Display in a dark region.  It looks like the noise is causing Cot09 to run slow (or perhaps a problem with the crate is causing the noise).  Someone during the day shift should contact the COT guys and figure out what's going on.
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 19:37:36.355 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-6522&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-6522  width='300' align='top'></a></td><td>Here's the COT plot from the consumer...</td></tr></table>
<!-- Author: Glenn Strycker -->
<!-- Comment: Sat Oct  6 19;41;15 comment by...G.S. -->
This noise seems to be in the same spot as noticed in previous shifts.
<!-- Date: Sat Oct 6 19:39:22.408 2007 -->
See Eiko's elog post on 
<a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=776&frame=2&anchor=10:30:24.101'>Thu Oct 4 10:30:24</a> for more details.<br><br> 
In the meantime, we'll try to run L2_TORTURE without COT09 and see how this affects the rates. 
  
<!-- Author: Glenn Strycker :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251003' target=_top>251003</a>) -->

<!-- Date: Sat Oct 6 19:40:44.674 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251003>251003</A>
TERMINATE: We're quitting in order to remove the COT09 crate from the partition.
<!-- Author: Glenn Strycker -->



<!-- Date: Sat Oct 6 19:45:08.869 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251004>251004</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,952,639] 
<!-- Author: Glenn Strycker -->
<!-- Comment: Sat Oct  6 19;46;35 comment by...Glenn Strycker -->
<pre>Run is without COT_09.  We see
L1A = 40 kHz
L2A = 250 Hz
Deadtime = 3.95% (down from 5,9,7% seen earlier)</pre>

<!-- Comment: Sat Oct  6 19;47;53 comment by...G.S. -->
<pre>
I changed the prescale from 151 (last entry above) down to 47.  We now see:

L1A = 40 kHz
L2A = 748 Hz
Deadtime = 6.34%
</pre>

<!-- Comment: Sat Oct  6 19;51;22 comment by...G.S. -->
One of the L3 farms is not working properly -- all channels on the L3 monitor are green in one column.  Frank says that this means it is running slow.  This isn't too unexpected since we're only running with 4 crates, but it is surprising that the effect is asymmetric -- all other nodes are running fine.

<!-- Comment: Sat Oct  6 19;53;15 comment by...G.S. -->
<pre>
Now we're at a prescale of 27

L1A = 40 kHz
L2A = 800 Hz
Deadtime = 67% !

There are now 3 of the 4 L3 sub-farm stations showing slow (green).
</pre>

<!-- Date: Sat Oct 6 19:59:07.935 2007 -->
Ok, enough tests for now.  Our conclusion is that the COT experts need to look at the noise issue and also investigate the COT09 crate, which appears to be running slower than the others.  Now I will quit the L2_TORTURE and begin the calibrations.
<!-- Author: Glenn Strycker -->

<!-- Date: Sat Oct 6 20:05:24.553 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251005>251005</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 20:07:09.543 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251005>251005</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;15;26 comment by...Glenn Strycker -->
BAD -- CHA 11W (ch 0-4) are still damaged, see previous elog entries.  This is a "known" problem to the Aces, but I'm not sure if an expert has looked at it yet.
<br><br>
Many CHA and CEM channels have a mean of 0 and/or a sigma of 0 or 1.  The CEM is in yellow on the global alarms window, so there may be several problems with that.
<!-- Date: Sat Oct 6 20:09:17.158 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251006>251006</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Glenn Strycker -->
<!-- Comment: Sat Oct  6 20;36;57 comment by...G.S. -->
results are on the web, but nothing showed up on the elog.  I'm going to re-run this calibration.
<!-- Date: Sat Oct 6 20:11:57.352 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251008>251008</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Glenn Strycker -->
<!-- Comment: Sat Oct  6 20;40;34 comment by...G.S. -->
Results are on the web, but never showed up on the elog.  I'm going to re-run and see if they'll show up now.
<!-- Date: Sat Oct 6 20:11:58.035 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251007>251007</A>
CALIBRATION: <U><B>bscqie</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 20;12;37 comment by...G.S. -->
I didn't hit reset before I loaded the parameters, so I had to abort and try again below.  It's ok.

<!-- Date: Sat Oct 6 20:19:06.627 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251009>251009</A>
ACTIVATE: RunSet: BOTHPLUGS_NORMAL 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 20:28:15.362 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251010>251010</A>
ACTIVATE: RunSet: BOTHPLUGS_MM_C 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 20:30:25.136 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251010>251010</A>
CALIBRATION: <U><B>pluglaser</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#pluglaser target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;17;09 comment by...Glenn Strycker -->
Good (?) -- Now the last plot has a mean of 540, which is BELOW the interval 550-580.
<!-- Date: Sat Oct 6 20:32:17.279 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251011>251011</A>
ACTIVATE: RunSet: BOTHPLUGS_MM 
<!-- Author: Glenn Strycker -->



<!-- Date: Sat Oct 6 20:38:32.258 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251012>251012</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 20:39:20.874 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251012>251012</A>
CALIBRATION: <U><B>clcqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#clcqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;22;21 comment by...Glenn Strycker -->
Good
<!-- Date: Sat Oct 6 20:41:14.899 2007 -->
The trigmon consumer keeps dying -- I've had to restart it 3 times or so.
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 20:41:47.852 2007 -->
RTServer just died -- I'm restarting it.
<!-- Author: Glenn Strycker -->
<!-- Comment: Sat Oct  6 20;58;51 comment by...Glenn Strycker -->
That didn't work.  Frank noticed that the problem is a stuck port on b0control20 -- the L3/EventBuilder Ace-Monitor computer.  We don't want to kill any processes, but it seems the Event Builder is stuck, so we'll hit the "Clean-up" button for the L3 farm and the Event Builder.
<!-- Comment: Sat Oct  6 21;08;28 comment by...Glenn Strycker -->
That didn't work.  We're going to run without it for now -- calibrations should go ok.  If we still have problems we'll reset the computer at the actual terminal.  No plan to page the DAQ expert for this, since it's only a stuck port and shouldn't affect our operations.
<!-- Comment: Sat Oct  6 21;44;37 comment by...Conor -->
The monitors may run on b0control20, but the actual EVB and L3 processes that connect to the RTserver run on other nodes in the EVB/L3 network
<!-- Date: Sat Oct 6 20:54:15.300 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251013>251013</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 20:55:20.424 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251013>251013</A>
CALIBRATION: <U><B>bscqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#bscqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;23;54 comment by...Glenn Strycker -->
Good, although BSC 1-east looks a little lighter-blue/green than the reference plot.  Keep an eye on this one.
<!-- Date: Sat Oct 6 21:10:34.978 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251015>251015</A>
 Activated at 2007.10.06 21:10:22
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 21:12:20.991 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251015>251015</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,957,639] 
<!-- Author: Glenn Strycker -->




<!-- Date: Sat Oct 6 21:18:26.731 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251015>251015</A>
 Terminated at 2007.10.06 21:18:11
<!-- Author: RunControl -->
<!-- Comment: Sat Oct  6 21;20;54 comment by...Glenn Strycker -->
L1A = 64.6 Hz, which is between 40 and 100.


<!-- Date: Sat Oct 6 21:23:09.476 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251016>251016</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:23:54.196 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251016>251016</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;27;30 comment by...Glenn Strycker -->
Good!
<!-- Date: Sat Oct 6 21:26:00.485 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251017>251017</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:27:24.493 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251017>251017</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;28;39 comment by...Glenn Strycker -->
Bad -- Wedges 9-11 west are still white

<!-- Date: Sat Oct 6 21:29:35.602 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251018>251018</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:32:30.344 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251019>251019</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:36:18.720 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251019>251019</A>
CALIBRATION: <U><B>cot</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cot target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;38;40 comment by...Glenn Strycker -->
BAD -- Constants have been updated 8 times out of the last 10.  Otherwise good.
<!-- Date: Sat Oct 6 21:37:57.829 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251020>251020</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:37:59.190 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251018>251018</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;45;11 comment by...Glenn Strycker -->
Good.
<!-- Date: Sat Oct 6 21:38:40.471 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251020>251020</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;46;07 comment by...Glenn Strycker -->
Bad -- same problem as the CEM
<!-- Date: Sat Oct 6 21:41:31.357 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251021>251021</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:44:36.378 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251022>251022</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Glenn Strycker -->





<!-- Date: Sat Oct 6 21:52:29.082 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251022>251022</A>
TERMINATE: This run took 6 minutes, which is still far short of the 10 minute estimate
<!-- Author: Glenn Strycker -->


<!-- Date: Sat Oct 6 21:52:30.808 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251022>251022</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sat Oct  6 21;57;17 comment by...Glenn Strycker -->
BAD -- same bad channels as usual.  It's the end of the shift, so I didn't have time to do any troubleshooting.

<!-- Date: Sat Oct 6 21:59:29.223 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251023>251023</A>
 Activated at 2007.10.06 21:58:23
<!-- Author: RunControl -->


<!-- Date: Sat Oct 6 21:59:30.355 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251023>251023</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] 
<!-- Author: Glenn Strycker -->



<!-- Date: Sat Oct 6 22:00:39.276 2007 -->
<font color=red>Shift Summary: </font><pre>L3 work during the start of the shift - some progress made. 
We now have the L3 display back 
 
CSL work 
 
Noticed that COT09 had long readout times. Also saw the same region in the COT with noise hits. 
 
Ran through calibrations 
 
Left with a cosmic run going... 
</pre> 
<!-- Author: Chlebana -->


<!-- Date: Sat Oct 6 22:02:11.385 2007 -->
<pre>End of shift summary 
---------------------- 
 
  - Even though previous shifts removed SMX_15 (slot 3) in CCAL05, removed the SVT_08 crate, and ran with "ignore reces", I was able to run both COSMICS and L2_TORTURE with all of the normal settings. 
  - COT noise (see Friday's post) is still present.  COT_09 crate seems to be having problems (it's running slower than the other COT crates) 
  - We found out that the Diamond Radiation monitor has a new webpage and that we don't have to worry about resetting the "Integrated" alarm on that crate. 
  - Most L3 farms are down while we wait for an SL4 operating system upgrade.  For now we have 4 sub-farms that are remaining in SL3 so we can do work.  Be prepared for further work after the weekend on the L3 systems. 
  - RTServer acted up a little -- there is evidently a stuck port on b0control20, the Ace computer for the L3 farm and DAQ Monitor.  It may be a problem with EVB since that is a large process running on this machine, but we don't know.  If it is still on red tomorrow morning, you (the Ace or SciCo) may want to page the DAQ expert. 
  - Trigger consumer monitor (TrigMon) is unstable and has been restarted a lot this shift. 
 
  - We're ending today with a COSMIC run. 
 
  - Notre Dame is ahead of UCLA 20-6 at the end of the third quarter.  This might be the win that ends our longest-ever losing streak.  GO IRISH!!!</pre>
<!-- Author: Glenn Strycker -->
<!-- Comment: Mon Oct  8 18;33;16 comment by...A Bruin -->
Boo.
<!-- Date: Sat Oct 6 22:04:48.086 2007 -->
We turned off the CLC HV for the night -- Morning Ace, please turn it back on.
<!-- Author: Glenn Strycker -->

<!-- Date: Sat Oct 6 23:55:34.389 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Sun Oct 7 07:55:26.029 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Oct 7 08:11:56.728 2007 -->
I found a cosmics run running ok. The only problems are from RTServer which has a bad connection from b0control20 and has been complaining about this all night. I'm investigating.
<!-- Author: Francesco :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251023' target=_top>251023</a>) -->