<!-- Title: Owl shift. Thu Jan 10, 2008 -->
<!-- SciCo: Gerry Bauer -->
<!-- DAQAce: Eunsin Lee -->
<!-- MonAce: &nbsp; -->
<!-- CO: Jason Galyardt -->
<!-- OpManager: Farrukh &nbsp; -->
<!-- Notes: -->
<pre>Run 255955 end JUST before  start of shift 
and store 5835 was terminated. 
 
Solenoid will be cycled, and D0 will go into access 
</pre>

<!-- Date: Thu Jan 10 00:05:56.359 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255956>255956</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:06:10.442 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255956>255956</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:06:11.993 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255901>255901</A>
CALIBRATION: <U><B>cot</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->


<!-- Date: Thu Jan 10 00:06:12.772 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255956>255956</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:06:13.977 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255901>255901</A>
CALIBRATION: <U><B>cot</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->


<!-- Date: Thu Jan 10 00:08:16.358 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255956>255956</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 00;28;17 comment by...Jason -->
OK
<!-- Date: Thu Jan 10 00:08:17.863 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255957>255957</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:09:41.590 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255957>255957</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:11:15.299 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255958>255958</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:12:12.474 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255958>255958</A>
CALIBRATION: <U><B>bscqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#bscqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 00;29;01 comment by...Jason -->
OK
<!-- Date: Thu Jan 10 00:12:39.619 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255958>255958</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:14:49.133 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255959>255959</A>
ACTIVATE: RunSet: BOTHPLUGS_NORMAL 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:21:03.822 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255959>255959</A>
TERMINATE: done

<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:22:55.490 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255960>255960</A>
ACTIVATE: RunSet: BOTHPLUGS_MM_C 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:24:38.333 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255960>255960</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:25:05.155 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255960>255960</A>
CALIBRATION: <U><B>pluglaser</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#pluglaser target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 00;31;52 comment by...Jason -->
OK
<!-- Date: Thu Jan 10 00:27:58.306 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255961>255961</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:27:59.901 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255961>255961</A>
ACTIVATE: RunSet: BOTHPLUGS_MM 
<!-- Author: Eunsin -->





<!-- Date: Thu Jan 10 00:33:17.815 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255962>255962</A>
ACTIVATE: RunSet: CALIB_CLC_QIE
The previous clc calibration result didn't come out. redoing..
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:34:05.733 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255962>255962</A>
CALIBRATION: <U><B>clcqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#clcqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 00;36;28 comment by...Jason -->
OK
<!-- Date: Thu Jan 10 00:35:35.303 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255962>255962</A>
TERMINATE: done
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 00:38:36.678 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255963>255963</A>
 Activated at 2008.01.10 00:37:53
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 00:38:37.986 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255963>255963</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,1023,706] 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:39:41.545 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255963>255963</A>
 Terminated at 2008.01.10 00:39:05
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 00:39:42.320 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255963>255963</A>
TERMINATE: consumers switching to cosmic..redo
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:49:35.941 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255964>255964</A>
 Activated at 2008.01.10 00:48:48
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 00:49:36.115 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255964>255964</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,1023,706] 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:51:41.544 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255964>255964</A>
 Terminated at 2008.01.10 00:50:17
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 00:51:42.424 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255964>255964</A>
TERMINATE: still complaining about the mode matching...Jason(CO) will restart concon in cosmic mode.
Will do the calibration later..
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:53:48.445 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255965>255965</A>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:53:49.035 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255965>255965</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 00:53:50.789 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255965>255965</A>
CALIBRATION: <U><B>tofqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#tofqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 00;55;19 comment by...Jason -->
OK

<!-- Date: Thu Jan 10 00:57:27.639 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255966>255966</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:00:33.689 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255966>255966</A>
TERMINATE: b0pcal01 not responsive..will do again
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 01:02:31.687 2008 -->
bopcal01 shepherding failed: 
 
Connecting to project <cdf_daq> on <tcp:b0server04> RTserver. 
 
Using tcp protocol. 
 
RTserver denied access. 
 
Message: Client name </_b0pcal01_28731400> is not unique.WARNING: closed connInternal validation failed, file: ectionipcmsg.c , line 1314Message from RTlm: 1Checking in <RTclient> license. 
 
27.0.0.1:1043 on request 
 
FER_smartInitMS: Could not connect to RTserver!  
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255966' target=_top>255966</a>) -->
<!-- Comment: Thu Jan 10 01;04;06 comment by...Eunsin -->
The second time shepherding works!

<!-- Date: Thu Jan 10 01:06:14.743 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255967>255967</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 01:06:19.758 2008 -->
Having trouble getting the consumers and the Consumer Controller (concon) to play nice; stopped all monitors and displays, then exited concon.  As per the concon instructions, I did 
<pre> 
 cd /cdf/onln/home/cdfdaq/consumer_scripts 
 concon & 
</pre> 
Tried several times, but to no avail. finally did 
<pre> 
 cd /cdf/onln/home/cdfdaq/consumer_scripts 
 ./concon & 
</pre> 
And this worked.  Not sure why, as <pre>which concon</pre> brings up the directory above.  The Consumer Controller and all monitors are now running again (in the right mode: Cosmics). 
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255966' target=_top>255966</a>) -->
<!-- Comment: Thu Jan 10 01;12;27 comment by...Jason -->
To elaborate on the conditions which lead me to stop the monitors and exit concon to begin with:  
<ul>
<li>the mode (Physics/Cosmics) refused to change properly (says it's in Cosmics, but not broadcasting this to RunControl)
<li> Monitors failed to stop and restart properly with "End Store" button
<li> Displays failed to connect to their sockets
<li> Multiple monitors stopped individually ended badly (some sort of memory corruption error)
</ul>
<!-- Comment: Thu Jan 10 01;33;30 comment by...Jason -->
Almost all Consumer Displays have trouble connecting to their assigned sockets; especially SvxMon (but perhaps this is because we're in Cosmics mode).  Opening the sockets manually in each CDF Consumer Display Main Frame works for unknown reasons, whereas the Consumer Controller interface fails.
<!-- Date: Thu Jan 10 01:06:50.991 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255967>255967</A>
TERMINATE: done

<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:11:00.530 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255968>255968</A>
CALIBRATION: <U><B>cemled</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#cemled target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 01;14;54 comment by...Jason -->
BAD due to <a href="http://www-cdfonline.fnal.gov/cgi/internal/elog/fixlist.pl?nb=fixlist&action=copyentry&page=1&time=08:31:20.654"> known calibration problem</a>
<!-- Date: Thu Jan 10 01:11:01.987 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255966>255966</A>
CALIBRATION: <U><B>smx</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->



<!-- Date: Thu Jan 10 01:13:09.372 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255969>255969</A>
 Activated at 2008.01.10 01:13:00
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 01:14:54.244 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255969>255969</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,1023,706] 
<!-- Author: Eunsin -->




<!-- Date: Thu Jan 10 01:22:08.300 2008 -->
KC calls: the solenoid is all set!!
<!-- Author: bauer -->

<!-- Date: Thu Jan 10 01:24:16.479 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255969>255969</A>
 Terminated at 2008.01.10 01:24:08
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 01:25:41.763 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255968>255968</A>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:25:42.579 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255968>255968</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:25:43.857 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255969>255969</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:27:15.003 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255970>255970</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:27:38.687 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255970>255970</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:30:39.121 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255971>255971</A>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:32:41.258 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255970>255970</A>
CALIBRATION: <U><B>cot</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->


<!-- Date: Thu Jan 10 01:32:42.765 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255971>255971</A>
CALIBRATION: <U><B>calxef</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calxef target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 01;34;50 comment by...Jason -->
BAD due to a <a href="http://www-cdfonline.fnal.gov/cgi/internal/elog/fixlist.pl?nb=fixlist&action=copyentry&page=1&time=08:31:20.654">known calibration problem</a>.
<!-- Date: Thu Jan 10 01:34:47.825 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255971>255971</A>
TERMINATE: done
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 01:43:20.681 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255972>255972</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:43:21.964 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255972>255972</A>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:46:29.430 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255973>255973</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:50:42.063 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255973>255973</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:51:32.525 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255973>255973</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jan 10 01;57;41 comment by...Jason -->
OK, <b>but</b> there's a problem with the calibrations results web page for toftac: the run number is correct on the web page, but the run number in the image is <b>255898</b>.  Opening the image with gv yields the correct set of plots for this run.
<!-- Date: Thu Jan 10 01:54:24.031 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255974>255974</A>
TERMINATE: done

<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 01:54:25.365 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255974>255974</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Eunsin -->




<!-- Date: Thu Jan 10 02:07:22.109 2008 -->
SMX Calibration results message did not make it to the elog, but the results are in the Calibrations results web page.  Status: <em>OK</em>
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255967' target=_top>255967</a>) -->
<!-- Comment: Thu Jan 10 02;08;19 comment by...Eunsin -->
Thanks, Jason.

<!-- Date: Thu Jan 10 02:09:24.075 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255975>255975</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,1021,705] 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:13:38.312 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255975>255975</A>
TERMINATE: to rerun the cot calibration
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:17:11.961 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255976>255976</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:17:12.596 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255976>255976</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:21:39.665 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255977>255977</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,1021,705] 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:42:15.824 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255977>255977</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:44:14.441 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255978>255978</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 02:44:15.723 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255978>255978</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 03:13:15.044 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255979>255979</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,1021,705] 
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 03:36:19.793 2008 -->
<table border=0><td><tr><td bgcolor=#e0e030>COT calibration has failed since Jan 8. 
I emailed Eiko about this. 
Not sure if this is critical to data taking...</td></tr></td></table>
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255979' target=_top>255979</a>) -->


<!-- Date: Thu Jan 10 03:50:16.509 2008 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Dear shift crew: 
  Could you describe what happened when the COT calibration  
failed? Could you please also check if the calibration proxy is  
at the "green" state when COT calibration is taken? Thanks!! 
Please make sure the calibration proxy is green during  
calibration, and you dont see any errors reported in the error  
log window. 
It's crucial to have COT calibrations updated every time before  
the stores. 
  
I checked the run control error log files and didnt find any  
failure due to done timeouts of COT crates. The calibration just  
didnt seem to go through the complete procedure as usual.  
The calibrations have failed since Jan 8 owl shift with the same  
error reported:  
COTCHANT0 : COT off in tcl or container empty 
 
The tcl file and COT calibration executable, scripts have not  
been changed at all. I will also double check with Bill Badgett  
too. I am at CERN now so cant be reached by phone. I will  
check elogs. 
 
</pre></td></tr></td></table>
<!-- Author: Shin-Shan Yu -->
<!-- Comment: Thu Jan 10 04;04;57 comment by...Eunsin -->
<pre>I redid the COT calibraion.
I don't see any symptom you referred to while runing COT calibration.(the calibration proxy is green, no errors in the error log window, no failure due to done timeouts of COT crates)
The only thing weird is that Calibrating state last only few seconds and goes to done state. It that what its supposed to be?</pre>
<!-- Comment: Thu Jan 10 04;31;18 comment by...Shin-Shan Yu -->
<pre>Thanks for checking!!! 

Usually the calibration process takes ~ 15-30 seconds. If the time 
is less than that, it's not going through calibration procedure 
completely and probably have nothing written to the database. 
I also didnt see any calibration log files produced at 
/cdf/onln/data/cdfcalib/log. 

I will bug Bill Badgett and Willis Sakumoto.</pre>
<!-- Comment: Thu Jan 10 04;40;20 comment by...bauer -->
<pre>There WERE a couple of log files produced...not for every attempt, for two calibration attempts there were:
/cdf/onln/data/cdfcalib/log/runCotCtt_0_2008-01-10_01:26:15.log
/cdf/onln/data/cdfcalib/log/runCotCtt_0_2008-01-10_01:51:03.log
</pre>
<!-- Date: Thu Jan 10 03:58:37.254 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255979>255979</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 04:00:31.434 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255980>255980</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 04:00:59.861 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255980>255980</A>
TERMINATE: done
<!-- Author: Eunsin -->




<!-- Date: Thu Jan 10 04:05:42.342 2008 -->
Notice in MCR log: 
  03:36:11-   D0 is returning their keys 
 
...so maybe something could yet happen on our shift?...
<!-- Author: bauer -->

<!-- Date: Thu Jan 10 04:08:07.095 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255981>255981</A>
 Activated at 2008.01.10 04:08:00
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 04:09:50.701 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255981>255981</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,1019,706] 
<!-- Author: Eunsin -->





<!-- Date: Thu Jan 10 04:40:57.842 2008 -->
<pre>Dear shift crew, 
 
   another way to check COT calibration, could you please  
 restart run control and take another COT calibration?  
  
 A copy from an old email of Bill B: 
 
The calibration consumer is not restarted if it is  
already running. 
 
So, if the first calibration CONFIGs, but fails  
to ACTIVATE, then the calibration consumer will  
be started, but not receive any events. Then, on the second  
calibration, the CONFIG will not restart the calibration  
consumer, but it will still be there waiting for events after the  
next ACTIVATE. 
</pre>
<!-- Author: Shin-Shan Yu -->
<!-- Comment: Thu Jan 10 05;05;52 comment by...Jason -->
Restarting RunControl and re-running the COT calibration (run 255982) did not seem to work.  I regenerated the COT portion of the calibration web page, but it's still got the failure message associated with the run taken at 1:51 am.
<!-- Comment: Thu Jan 10 07;24;59 comment by...Shin-Shan Yu -->
Yes, for some reason, the log files from the latter runs are 
missing.
<!-- Date: Thu Jan 10 04:48:38.085 2008 -->
MCR calls for shot set-up!
<!-- Author: bauer -->

<!-- Date: Thu Jan 10 04:51:17.511 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255981>255981</A>
 Terminated at 2008.01.10 04:50:50
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 04:51:31.044 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255981>255981</A>
TERMINATE: done
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 04:55:28.238 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255982>255982</A>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 04:55:29.961 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255982>255982</A>
TERMINATE: done after restarting RC
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 05:03:50.277 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255983>255983</A>
 Activated at 2008.01.10 05:03:40
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 05:03:51.255 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255983>255983</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,1019,706] 
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 05:34:33.747 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255983>255983</A>
 Terminated at 2008.01.10 05:34:28
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 05:35:03.913 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255983>255983</A>
TERMINATE: final protons being injected
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 05:38:20.107 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255984>255984</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 05:41:00.906 2008 -->
MCR log claims: 
05:33:35-   Loading final protons  
<!-- Author: bauer -->


<!-- Date: Thu Jan 10 05:51:35.649 2008 -->
Done TO. for crate/s:  
    b0cmu01  auto hrr issued
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255984' target=_top>255984</a>) -->


<!-- Date: Thu Jan 10 06:01:09.807 2008 -->
<pre>MCR log: 
05:59:18-   Preparing to load PBars</pre>
<!-- Author: bauer -->


<!-- Date: Thu Jan 10 06:05:53.636 2008 -->
CPR, CLC hb alarm in HV summary. 
I've checked CLC1 machine, the program crashed. so I restarted the program.
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255984' target=_top>255984</a>) -->


<!-- Date: Thu Jan 10 06:09:43.866 2008 -->
PBars appearing in machine now....
<!-- Author: bauer -->


<!-- Date: Thu Jan 10 06:38:33.570 2008 -->
MCR Log: 
06:37:26-   2034e09 PBars loaded.  
<!-- Author: bauer -->

<!-- Date: Thu Jan 10 06:41:30.668 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255984>255984</A>
TERMINATE: scraping..
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 06:56:13.467 2008 -->
MCR calls: scraping complete 
...Si calls, they are happy with the rates 
...inform MCR we ready....
<!-- Author: bauer :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985' target=_top>255985</a>) -->
<!-- Comment: Thu Jan 10 06;57;04 comment by...bauer -->
Store:	5837


<!-- Date: Thu Jan 10 06:59:35.890 2008 -->
C:LOSTP	 10104.690 	   <--increased slightly to warning <br>
C:LOSTPB	3504.688 	
<!-- Author: bauer -->


<!-- Date: Thu Jan 10 07:01:44.494 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-313&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-313  width='300' align='top'></a></td><td>  </td></tr></table>
<!-- Author: Eunsin -->

<!-- Date: Thu Jan 10 07:05:15.314 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985>255985</A>
 Activated at 2008.01.10 07:04:48
<!-- Author: RunControl -->


<!-- Date: Thu Jan 10 07:05:16.585 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985>255985</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Eunsin -->



<!-- Date: Thu Jan 10 07:05:52.823 2008 -->
Done TO. for crate/s:  
    b0clc00   auto hrr issued.
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985' target=_top>255985</a>) -->


<!-- Date: Thu Jan 10 07:07:46.656 2008 -->
C:B0ILUM	178.844E30
<!-- Author: bauer :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985' target=_top>255985</a>) -->


<!-- Date: Thu Jan 10 07:12:18.243 2008 -->
<center><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-315&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-315' width='300'></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-316&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-316' width='300'></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-317&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-317' width='300'></a></center>
<!-- Author: Eunsin -->


<!-- Date: Thu Jan 10 07:15:02.410 2008 -->
Done TO. for crate/s:  
    b0clc00   auto hrr issued
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985' target=_top>255985</a>) -->
<!-- Comment: Thu Jan 10 07;18;13 comment by...Eunsin -->
again

<!-- Comment: Thu Jan 10 07;22;52 comment by...Eunsin -->
happend again, active turned red.
Shpherding clc00 successful.


<!-- Date: Thu Jan 10 07:33:22.685 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255755>255755</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=255755 target=_runSummary>255755</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->



<!-- Date: Thu Jan 10 07:42:01.437 2008 -->
L2 Decision Timeout ERROR !!. AUTO HRR issued.
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985' target=_top>255985</a>) -->
<!-- Comment: Thu Jan 10 08;04;21 comment by...Eunsin -->
again
<!-- Date: Thu Jan 10 07:42:44.197 2008 -->
BMU HV East Trip (SE 144). Power cycled.
<!-- Author: Eunsin :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255985' target=_top>255985</a>) -->

<!-- Date: Thu Jan 10 07:55:11.628 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255985 target=_runSummary>255985</A> x3E7F1</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>07:04:48</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:41:26</TD>
<TD ALIGN=RIGHT>     430,524</TD>   <TD ALIGN=RIGHT>    494.61</TD>
<TD ALIGN=RIGHT>    415.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:00</TD>
<TD ALIGN=CENTER>00:41:26</TD>
<TD ALIGN=RIGHT>       430,524</TD> <TD ALIGN=RIGHT>    494.61</TD>
<TD ALIGN=RIGHT>    415.00</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Jan 10 07:58:29.885 2008 -->
<font color=red>Shift Summary: </font><pre>Run 255955 end JUST before  start of shift  
and store 5835 was terminated.  
 
Solenoid will be cycled, and D0 will go into access  
 
COT Calib fails....looking back, its been failing for last few days, ACE emails problem to Eiko 
 
01:22:08  KC calls: the solenoid is all set 
 
06:37:26- 2034e09 PBars loaded. 
 
06:56:13  MCR calls: scraping complete 
 
C:LOSTP 10104.690  
C:LOSTP 3504.688 
 
07:05:15  Run 255985  Activated 
 
07:07:46  C:B0ILUM 178.844E30  
 
07:42:44  BMU HV East Trip (SE 144).  
 
 
 
 
From Last shift: 
    DO NOT  power cycle b0wcal06 in between stores -- let Phil do it 
(only affects monitoring, so don't screw with it)  --(shower max prob?) 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   255985
Delivered Luminosity   603.5 nb-1  
Acquired Luminosity    427.3 nb-1  
Efficiency             71.6
   
</td></tr></table></pre>
<!-- Author: bauer -->