<!-- Title: Eve shift. Sun Oct  7, 2007 -->
<!-- SciCo: Frank Chlebana -->
<!-- DAQAce: Glenn Strycker -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>One hour into shift...</pre>


<!-- Date: Sun Oct 7 16:06:57.751 2007 -->
<pre>Beginning of Shift Short Status Summary (BoS4) 
------------------------------------------------ 
 
- currently running COSMICS 
  - L1A = 28.2 Hz 
  - L2A = 28.2 Hz 
  - Deadtime = 0.0% 
 
- all crates are green on the VxWorks monitor 
 
- HV status: 
  - ON  (+HV):   CHA,WHA 
  - ON  (no HV): none 
  - NOT (+HV):   GAM 
  - STD:         PEM/PHA/PSH 
  - WARN:        CEM (known) 
  - GRAY:        CMU,CMP,CMX,BMU,CES,CPR,CLC,MNP,BSC (known, see previous shift) 
  - OFF:         everything else 
 
Global Alarms status: 
  - Red:    PSM,PTM  (known) 
  - Yellow: CEM      (known) 
  - Grey:   CLC,CPR,BSC,MNP,CES,CMU,CMP,CMX,BMU 
  - Green:  everything else 
  - all green systems have heartbeats 
  - no "pink" or "grey" heartbeats, except for gray locked-out systems 
 
- Process System Alarms: 
  - status seems to be consistant with known problems 
  - yellow: GAS, COT, PSM 
  - green:  CRYO, HVAC, WATER, SILICON, RACKS, SOLENOID 
 
- Racks not in use 
  - all racks in the L2_TORTURE [19,952,639] mode are presumably working 
  - all racks in COSMICS mode [15,954,639] are presumably working 
  - we'll try to track down the bad slots/racks for TOFTAC calibration 
 
- Known issues   
  - TOFQIE calib is not working properly  
  - in CEMLED, CALXEF, and TOFTAC calibs, wedges 9-11West are still empty  
  - Noise in COT yesterday, COT_09 crate is still slow today 
 
- Shift plan   
  - COSMICS run 
  - run L2_TORTURE with all possible crates 
  - run all calibrations 
  - Troubleshoot TOFTAC calibration 
  - end shift with a nice, quiet COSMICS run</pre>
<!-- Author: Glenn Strycker (Ace) -->

<!-- Date: Sun Oct 7 16:33:41.587 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251046>251046</A>
 Terminated at 2007.10.07 16:32:57
<!-- Author: RunControl -->


<!-- Date: Sun Oct 7 16:33:42.810 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251046>251046</A>
TERMINATE: Let's end COSMICS and do an L2_TORTURE run.
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 16:36:18.369 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251047>251047</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,952,639] 
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 17:17:47.365 2007 -->
<pre>I'm noticing some noise on the event display, except this time it isn't in the central tracking, but the muon tracking (at about the 3 o'clock prosition).</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 17:24:33.439 2007 -->
<center><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6523&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6523' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6524&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6524' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6525&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6525' width='300'></a><a href='/cgi/elog/elog.pl?nb=2007&action=view&page=-6526&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2007&action=view&page=-6526' width='300'></a></center>
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 17;27;34 comment by...Glenn Strycker -->
<pre>Plot 1: COT_07 2d Occupancy
Plot 2: COT_08 "
Plot 3: COT_09 "
Plot 4: COT_09 1d Occupancy

I took snapshots of these plots from the consumers to show where the problem is on crate 9 and how if differs from good crates.  It looks like slots 4,5,6 are bad.</pre>


<!-- Date: Sun Oct 7 17:34:51.992 2007 -->
<pre>HVAC audible alarm -- Process Systems alarm button was green when I looked at it, though, and nothing appears bad on the details.</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 18:10:27.944 2007 -->
<pre>I'm going to end the L2_TORTURE run now and take calibrations.  First, however, a few statistics for the run: 
 
L1A: 39.7 kHz 
L2A: 248.5 Hz 
Deadtime: 4.95%</pre>
<!-- Author: Glenn Strycker -->

<!-- Date: Sun Oct 7 18:16:41.405 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251048>251048</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 18:18:12.064 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251048>251048</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;01;14 comment by...Glenn Strycker -->
BAD -- CEM 11W are bad as noted in previous shift
<!-- Date: Sun Oct 7 18:21:18.249 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251049>251049</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 18:21:19.835 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251049>251049</A>
CALIBRATION: <U><B>clcqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#clcqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;01;54 comment by...Glenn Strycker -->
Good.
<!-- Date: Sun Oct 7 18:28:55.373 2007 -->
During the "Partition" step for setting up the BSC QIE calibration, the crate b0clc00 hung.  I rebooted and reshepherded the crate, and then everything worked ok.
<!-- Author: Glenn Strycker -->

<!-- Date: Sun Oct 7 18:30:11.985 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251050>251050</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 18:31:05.123 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251050>251050</A>
CALIBRATION: <U><B>bscqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#bscqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;03;00 comment by...Glenn Strycker -->
OK -- BSC 1-E is a little lighter blue than usual
<!-- Date: Sun Oct 7 18:32:23.273 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251051>251051</A>
ACTIVATE: RunSet: BOTHPLUGS_NORMAL 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 18:45:16.470 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251052>251052</A>
ACTIVATE: RunSet: BOTHPLUGS_MM_C 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 18:48:10.293 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251052>251052</A>
CALIBRATION: <U><B>pluglaser</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#pluglaser target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;03;36 comment by...Glenn Strycker -->
Good.
<!-- Date: Sun Oct 7 18:56:20.305 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251053>251053</A>
ACTIVATE: RunSet: BOTHPLUGS_MM 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:00:42.052 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251054>251054</A>
 Activated at 2007.10.07 19:00:14
<!-- Author: RunControl -->


<!-- Date: Sun Oct 7 19:00:43.891 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251054>251054</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,957,639] 
<!-- Author: Glenn Strycker -->







<!-- Date: Sun Oct 7 19:07:38.502 2007 -->
TrigMon died again... I restarted in on the concon.
<!-- Author: Glenn Strycker -->

<!-- Date: Sun Oct 7 19:08:11.059 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251054>251054</A>
 Terminated at 2007.10.07 19:08:04
<!-- Author: RunControl -->
<!-- Comment: Sun Oct  7 19;10;21 comment by...Glenn Strycker -->
L1A rate = 64.8 Hz -> within the 40-100Hz allowed range

<!-- Date: Sun Oct 7 19:12:28.582 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251055>251055</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:14:20.367 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251055>251055</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;42;17 comment by...Glenn Strycker -->
Good.
<!-- Date: Sun Oct 7 19:14:21.778 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251056>251056</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:18:31.993 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251057>251057</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:18:32.158 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251057>251057</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;42;42 comment by...Glenn Strycker -->
BAD -- same missing spots as usual.
<!-- Date: Sun Oct 7 19:18:33.421 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251056>251056</A>
CALIBRATION: <U><B>smx</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#smx target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;43;21 comment by...Glenn Strycker -->
Good.
<!-- Date: Sun Oct 7 19:20:44.232 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251058>251058</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:23:04.980 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251059>251059</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:23:05.210 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251058>251058</A>
CALIBRATION: <U><B>cot</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cot target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;45;01 comment by...Glenn Strycker -->
Good (?) -- see previous shifts for an explanation of my '?'.  The calibration is ok, but the line "Out of the last 10 calibrations, the constants were updated 9 times" means we should technically mark it as bad.  Instead we mark it as good and we notified Eiko about it.  I assume this will be addressed Monday.
<!-- Date: Sun Oct 7 19:25:17.945 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251059>251059</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;45;34 comment by...Glenn Strycker -->
BAD -- same missing channels as usual.
<!-- Date: Sun Oct 7 19:27:22.011 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251060>251060</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:29:48.351 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251061>251061</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 19;39;21 comment by...Glenn Strycker -->
We used all of the crates for this run (that is, the default settings)
<!-- Date: Sun Oct 7 19:38:24.032 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251061>251061</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;39;11 comment by...Glenn Strycker -->
We used all of the crates for this run (that is, the default settings)


<!-- Comment: Sun Oct  7 19;41;50 comment by...Glenn Strycker -->
BAD -- same missing channels as usual.  The next few "calibration" runs will actually be test runs, so I won't evaluate them as "good" or "bad".  This calibration result here is the one I will be posting on the white board.







<!-- Date: Sun Oct 7 19:46:43.695 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-6527&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-6527  width='300' align='top'></a></td><td>Here is the calibration plot for TOF TAC with ALL crates (well, the default ones, anyway) included.  I will try to post plots for other configurations.</td></tr></table>
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 19;53;38 comment by...Glenn Strycker -->
<pre>NEVER MIND...
Now that I re-read the original TOFTAC troubleshooting challenge (and Francesco's work from two days ago), I see that the problem isn't with the plots.  I may post a plot for TOFTAC with wcal04 removed just for kicks, but don't expect 9 plots below!</pre>
<!-- Date: Sun Oct 7 19:50:26.978 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251062>251062</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- This one is with crate wcal00 removed from the partition
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 19:58:19.294 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251062>251062</A>
TERMINATE: Calibration finished sucessfully... hopefully results will post to the elog soon.
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 19:58:20.737 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251062>251062</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 19;59;41 comment by...Glenn Strycker -->
It looks exactly like you would expect... just like the plot I posted above except with a white block for wcal00.  I think I may skip to removing wcal04 now to see if I see the same error(s) as Francesco

<!-- Date: Sun Oct 7 20:03:54.321 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251063>251063</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL04 removed, all others in
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 20:08:28.832 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251063>251063</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Sun Oct  7 20;17;39 comment by...Glenn Strycker -->
Looks ok -- I finally see the "floating point error".  See my comments below.
<!-- Date: Sun Oct 7 20:08:28.030 2007 -->
<pre>I gave the walkie talkie that was found outside during the  
last shift to the security gaurd who passed by. 
 
He thinks it probably belongs to the Grounds Dept. 
</pre>
<!-- Author: Chlebana -->



<!-- Date: Sun Oct 7 20:24:33.292 2007 -->
<pre>
I was not seeing any errors floating point errors in any of my TOFTAC calibrations.  I read and re-read Francesco's posts and couldn't understand what he was seeing, but I was able to follow his methods and find the log files. 
 
So far I've run 3 trials... 
 
251061 -- all crates included 
251062 -- wcal00 removed 
251063 -- wcal04 removed 
 
I did not have ANY problems with the actual runs freezing, not posting log files, or not posting plots to DBANA or the elog.  No, everything ran fine, and I wasn't sure what errors everyone is talking about. 
 
Then I finally tried the following: 
cat runTacCalib_251063* | grep floating 
 
If you want a COUNT of the errors, use: 
cat runTacCalib_251063* | grep floating | wc -l 
 
I tried substituting various run numbers in, but only 251063 (wcal04 removed) gave any such errors.  I was surprised (as was Francesco) that the errors occur when I REMOVE a crate, not ADD.  The original email made it sound as if this calibration would die if certain slots/crates were INCLUDED.  Anyway, I'm still a little confused, but now I'll try running with various other crates removed and make a post showing which runs give errors.  Then I will track down that rascally ADMEM slot.</pre>
<!-- Author: Glenn Strycker -->

<!-- Date: Sun Oct 7 20:31:52.708 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251064>251064</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL01 removed
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 20:43:05.643 2007 -->
<pre>About noise in COT crate b0cot09 slot 4-6: 
 The TDC expert Eric James said he might have a clue about what's going on with these three slots. He will find a chance to work on them. The TDC in three slots had shifted 13-16 ns in the average start time for part of the channels several times. See  
http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry 
&page=699&time=10:51:10.383 
The funniness went away after we reseated these TDCs.</pre>
<!-- Author: Shin-Shan Yu (Eiko) -->

<!-- Date: Sun Oct 7 20:44:19.433 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251064>251064</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Sun Oct 7 20:44:20.928 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251065>251065</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL01 removed, 2nd attempt
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 20:44:20.785 2007 -->
No results came up for the last run, so I'm redoing it.  I noticed, however, that I had previously forgotten to click "ok" for the "terminate" window for that run AND the previous run.  Perhaps having open windows in RunControl leads to these errors in elog posts.  Anyway, we're going to re-run just to see if we can get this to work.
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 20;46;03 comment by...G.S. -->
The log file for 251064 JUST APPEARED in the directory, so maybe things are just going slow.
<!-- Date: Sun Oct 7 20:45:05.345 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251065>251065</A>
TERMINATE: This one seemed to work ok.
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 21;00;13 comment by...G.S. -->
That's true, it ended ok, but results were never posted.  Results from 251064 (which had the same settings) finally did post results, so perhaps the log file will emerge for this run later.  If not, we can just use the results for 251064 or else do this run again later.
<!-- Date: Sun Oct 7 20:48:48.052 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251066>251066</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL02 removed
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 20:54:50.814 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251066>251066</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Sun Oct 7 20:55:03.518 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251066>251066</A>
TERMINATE: ended ok.
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 20:58:25.434 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251067>251067</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL03 removed
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 21:04:26.221 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251067>251067</A>
TERMINATE: ran and ended ok
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:04:27.066 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251067>251067</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Sun Oct 7 21:07:20.767 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251068>251068</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL05 removed
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:12:03.732 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251068>251068</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Sun Oct 7 21:12:04.271 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251068>251068</A>
TERMINATE: run ran ok, ended ok
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:18:18.393 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251069>251069</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL06 removed
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:21:12.764 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251069>251069</A>
TERMINATE: ended ok
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 21;22;20 comment by...Glenn Strycker -->
Actually, no, it didn't.  We're stuck on DONE and it won't go to IDLE.  SEVB appears stuck, so I'll reset and reshepherd.

<!-- Comment: Sun Oct  7 21;22;56 comment by...G.S. -->
I'm not allowed to reshepherd this crate... I'll try ABORTing or ending RunControl or something.

<!-- Comment: Sun Oct  7 21;24;33 comment by...G.S. -->
I hit reset and that worked... sort of.  I tried re-partitioning and that got stuck, again on the SEVB crate.  I may close RunControl and restart it, and/or look up info on sevb.

<!-- Comment: Sun Oct  7 21;36;21 comment by...G.S. -->
I closed RunControl, restarted, and then I cleaned up the L3 farm and EVB using the Ace Controls on b0control20.  THAT finally worked.
<!-- Comment: Sun Oct  7 21;42;58 comment by...Conor -->
The software  event builder (SEVB) has nothing to do with the hardware event builder. The ACE help page has a link to SEVB help, under the DAQ section.
<!-- Date: Sun Oct 7 21:32:33.017 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251070>251070</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL06 removed, 2nd try
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 21:36:42.196 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251070>251070</A>
TERMINATE: run ended ok!
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:36:43.388 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251070>251070</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->


<!-- Date: Sun Oct 7 21:38:49.185 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251071>251071</A>
ACTIVATE: RunSet: TOF_TAC_CALIB -- WCAL07 removed
<!-- Author: Glenn Strycker -->



<!-- Date: Sun Oct 7 21:45:21.599 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251071>251071</A>
TERMINATE: ended well.
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:45:27.989 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251071>251071</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->



<!-- Date: Sun Oct 7 21:47:31.348 2007 -->
<pre>"TOFTAC w/ Crate Removal" Results... 
 
251061 -- all crates included -- 0 floating point errors 
251062 -- WCAL00 removed -- 0 errors 
251063 -- WCAL04 removed -- 1 
251064 -- WCAL01 removed -- 0 
251065 -- WCAL01 removed -- did not post log file 
251066 -- WCAL02 removed -- 0 
251067 -- WCAL03 removed -- 0 
251068 -- WCAL05 removed -- 0 
251069 -- WCAL06 removed -- did not post log file 
251070 -- WCAL06 removed -- 0 
251071 -- WCAL07 removed -- 0 
 
Verdict: only WCAL04 is bad and causing floating-point errors.  Next task: find the faulty ADMEMs within this crate.  I suggest that the next ace moves on to analyzing these individual slots -- for each run, include all crates, but hit the "EDIT" button when 04 is highlighted instead of "REMOVE", and then remove a slot.</pre>
<!-- Author: Glenn Strycker -->

<!-- Date: Sun Oct 7 21:53:19.668 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251072>251072</A>
 Activated at 2007.10.07 21:52:34
<!-- Author: RunControl -->


<!-- Date: Sun Oct 7 21:53:20.628 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251072>251072</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,954,639] -- This should be the last run of the day, hopefully.  We may try to tweak the trigger prescale levels before we leave, if there's time.
<!-- Author: Glenn Strycker -->
<!-- Comment: Sun Oct  7 21;56;05 comment by...Glenn Strycker -->
Never mind, tweaking triggers is more for L2_TORTURE mode.  We'll leave this COSMICS run as it is.


<!-- Date: Sun Oct 7 21:58:49.222 2007 -->
<pre>End-of-Shift Summary (EosS) 
----------------------------- 
 
TrigMon died a few more times today -- We should email the experts and look into this.  I emailed Charles already. 
 
Laser Calibration better than it has been, but the mean is right on the edge of the allowed 550nm-580nm window. 
 
The BSC QIE calibration looks pretty good, but the BCS 1-E plot is slightly more light-blue than usual, and lighter than the comparison reference plots. 
 
We still see slow slots in COT09.  Eiko emailed me and other aces back, saying that experts will look at this soon. 
 
We're ending the day with a nice, quiet COSMICS run -- Enjoy. 
 
 
--------------------------- 
TOFTAC UPDATE 
--------------------------- 
 
Note to the next Ace: I didn't have time to figure out all the bad crates, but I do have a good method down (see above epost). 
 
If you have time, here is a sequence you can use: 
1) run with some slot removed from WCAL04 
2) after it is finished, run the following on a terminal: 
      cd /cdf/onln/data/cdfcalib/log 
      ls -la *Tac*2007-10-08* (or whatever day it is) 
      cat runTacCalib_[runnumber]* | grep floating | wc -l 
3) note whether the answer is 0 or >0 and post results</pre>
<!-- Author: Glenn Strycker -->


<!-- Date: Sun Oct 7 21:59:55.483 2007 -->
<pre>Ending the shift with a cosmic run running...</pre>
<!-- Author: Chlebana -->

<!-- Date: Sun Oct 7 23:55:52.002 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Mon Oct 8 07:55:46.954 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Mon Oct 8 08:01:06.201 2007 -->
solenoid2 had a crash on tcptask.exe (I guess part of windows TCP/IP stack). Unluckly the timestap is in hex and counting I don't know what starting from I don't know when, but the value is 0x43743b60 for the first error in the detailed data. I suppose this correspond to the first ICICLE error reported by ProcMon which was at 5.40am. I'll acknowledge the error and eventually restart solenoid2.
<!-- Author: Francesco :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=251072' target=_top>251072</a>) -->
<!-- Comment: Mon Oct  8 08;04;37 comment by...Francesco -->
I have to correct myself, ICICLE is not updating since about 2.43am and the first ProcMon error about this is at about 3.19am. I've checked internet (with internet explorer) on solenoid2 and it is working. Maybe the tcptask.exe crash is unrelated.

<!-- Comment: Mon Oct  8 08;08;41 comment by...Francesco -->
I've restarted ICICLE (ctrl-c and then doubleclick on the icon) but it say the following: "Error in init EDA, maybe IFIX is not running". I guess I'll have to ask an expert about this, iFix seems to be running.