<!-- Title: Day shift. Sun Dec 14, 2003 -->
<!-- SciCo: Rick Field -->
<!-- DAQAce: Jeffrey Miles -->
<!-- MonAce: Irina Shreyber -->
<!-- CO: Giorgio Bellettini -->
<!-- OpManager: Mary Convery -->
<!-- Notes: -->
<pre>Running COSMIC_NOTRACKS for COT gas flow tests.</pre>


<!-- Date: Sun Dec 14 08:25:35.869 2003 -->
<pre>We paged iFIX because we are having problems resetting the heartbeat alarm.</pre>
<!-- Author: Rick -->
<!-- Comment: Sun Dec 14 08;26;42 comment by...Rick -->
iFIX called and Irina is talking with him.

<!-- Comment: Sun Dec 14 09;00;00 comment by...Irina -->
<pre>Restarted IFix on vnode1. Problem's returned back- HBM alarm is present and all hearbeat boxes are grey.
 JC's suggested to 
1. Restart iFix on vnode2
2. Restart iFix on vnode1
If it doesn't help -3.reboot vnode1
Hope, it will help</pre>

<!-- Comment: Sun Dec 14 09;34;01 comment by...Irina -->
<pre>Hmmm...
I've restarted iFix on vnode2 and vnode2.
I did it 20!!!minutes ago on vnode2 and aroun 15 minutes on vnode 1, but we still don't have HV bars on HV Summary Pages and almost all buttons are grey on Global Alarms (exept of SVX,ISL,L00,GAM,ICI,INH,VOI- they are green (but their hearbeats are grey), some "D" buttons are green, HBM is red). Why is it taking so much time?!?! I'll wait 10 minutes and then ,if situation doesn't change, reboot vnode1.</pre>
<!-- Comment: Sun Dec 14 10;01;24 comment by...Irina -->
<pre> I've rebooted vnode1 7 minutes ago.
No HV bars yet, almost all is grey on Global Alarms but  (I really don't like it!!!) - HBM is red. Seems we still have the problem.</pre>
<!-- Comment: Sun Dec 14 10;49;57 comment by...Irina -->
<pre>After recovery HV bars've returned back and everything is green on Global Alarm, just HBM is red and hearbeats are grey.
(we have returned back to the morning state).
  JC will stop by the control room this afternoon. </pre>
<!-- Date: Sun Dec 14 09:05:32.876 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176381">176381</A>
TERMINATE: In retrospect, it seems that the desired run config was to include the COT crates (to facilitate COT gas flow tests), but use the NOTRACKS trigger table to exclude these trigger bits
<!-- Author: Jeff x2080 -->



<!-- Date: Sun Dec 14 09:07:11.963 2003 -->
In the past hour or so there was no signal in COT1, COT2, COTAveStart displays of YMon.
<!-- Author: Giorgiob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176381' target=_top>176381</a>) -->

<!-- Date: Sun Dec 14 09:10:57.198 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176382">176382</A>
 Activated at 2003.12.14 09:09:38
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 09:12:28.480 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176382">176382</A>
ACTIVATE: Added the COT crates;  this is the full cosmics configuration, with the NOTRACKS cosmics table [6,353,403], and all files to tape and look area
<!-- Author: Jeff x2080 -->



<!-- Date: Sun Dec 14 09:19:12.779 2003 -->
Called MCR about search and secure this evening.  Crew chief said a time wouldn't be determined until after the alignment crew is done. 
<P> 
All detector steel is closed.  Shielding blocks will be restacked after the alignment work. 
<P> 
Possible first beam on owl shift.
<!-- Author: convery -->




<!-- Date: Sun Dec 14 10:03:26.307 2003 -->
At 10 am we observe that the consumer status  as displaied in the Consumer Server List is incorrect: 
1) it says "updating" for all but one entry but all rates are constant 
2) it shows an old run number
<!-- Author: giorgiob :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176382' target=_top>176382</a>) -->


<!-- Date: Sun Dec 14 10:03:45.379 2003 -->
<pre>I noticed that the "Consumer Status"  webpage at 
http://www-cdfonline.fnal.gov/consumer/consumer_status.html 
is "goofy"... wrong date... wrong run, etc.</pre>
<!-- Author: Rick :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176382' target=_top>176382</a>) -->
<!-- Comment: Sun Dec 14 10;23;21 comment by...Rick -->
Kaori called and she and Jeffrey fixed the problem.  We are going to restart all the consermers at the end of this run.
<!-- Comment: Sun Dec 14 10;23;30 comment by...kaori -->
Asked daqace person (jeff) to restart state manager.  The consumer
status page is updated ok, now.
<!-- Date: Sun Dec 14 10:21:51.239 2003 -->
<pre>I paged iFIX again for Irina.</pre>
<!-- Author: Rick -->




<!-- Date: Sun Dec 14 10:25:30.289 2003 -->
At some point, the calorimeter ADMEMs (in ccal, pcal, and wcal) will need a new set of FRAM calibrations.  I have talked with Rick and Jeff, who would like to continue taking data for COT.  I will call back at 11 to guide the ACE through the (expert) procedure.<p> 
Besides getting the calorimeter ready after whatever work went on the last two weeks, recently the dead PHA 05W channel was restored.  (This channel had zero pedestals.  Howard and Willis diagnosed a faulty connector, and Mike L. fixed it.)  This channel has an unknown set of calibrations, which should be remedied.
<!-- Author: m mattson -->



<!-- Date: Sun Dec 14 11:07:52.151 2003 -->
I noticed that the /data1 disk is agin 96% full. 
 I asked the Ace to close/restart EH GUi so I could remove a big errlog log file. 
 After deleting this file,the disk is still 96% full. So something else is building up...
<!-- Author: Donatella T. -->

<!-- Date: Sun Dec 14 11:13:31.603 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176382">176382</A>
 Terminated at 2003.12.14 11:12:19
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 11:14:14.074 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176382">176382</A>
TERMINATE: need to take new calorimetry constants and load them
<!-- Author: Jeff x2080 -->



<!-- Date: Sun Dec 14 11:16:46.065 2003 -->
<pre>We are stopping the run to determine "calorimeter constants".  It should take less than 30 min and then we will go back to COSMIC_NOTRACKS.  We are also restarting all the consumers now.</pre>
<!-- Author: Rick -->

<!-- Date: Sun Dec 14 11:18:57.286 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176383">176383</A>
ACTIVATE: CAL QUI
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 11:18:58.125 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176383">176383</A>
TERMINATE: CAL QIE
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 11:25:12.674 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176384">176384</A>
ACTIVATE: CAL QIE (again, because I didn't click calib_load)
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 11:33:37.118 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176384">176384</A>
TERMINATE: CAL QIE: constants loaded successfully
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 11:51:00.705 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176385">176385</A>
 Activated at 2003.12.14 11:49:31
<!-- Author: RunControl -->



<!-- Date: Sun Dec 14 11:53:18.686 2003 -->
<pre>We are doing a "spike run" to verify the "calorimeter constants". </pre>
<!-- Author: Rick -->

<!-- Date: Sun Dec 14 11:55:14.288 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176385">176385</A>
ACTIVATE: PMT spike run; Mark Mattson advises that this is a way to identify whether any bad constants were just loaded
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 12:01:09.700 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176385">176385</A>
 Terminated at 2003.12.14 11:57:38
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 12:03:16.240 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176385">176385</A>
TERMINATE: took 30K events;  avg L1 accept rate this run: 63.0 Hz  (within the 50-100 Hz acceptability range). This is with absolute quiet time, of course.
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 12:27:11.431 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176386">176386</A>
TERMINATE: backing out of cosmics start-up;  been asked to do take cal constants again
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 12:27:12.234 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176387">176387</A>
ACTIVATE: CAL QIE
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 12:48:37.763 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176387">176387</A>
TERMINATE: <pre>CAL QIE: calib_load'ing is complete, but again with the messages that are the reason for our redoing it:
(b0wcal05) QIE calib: slot 21, qie 10:Warn 17: Mult. FRAM erasures, val 02.0000, cap 0, rng0.
(b0wcal03) QIE calib: slot 21, qie 03:Warn 17: Mult. FRAM erasures, val 02.0000, cap 0, rng0.
(b0wcal04) QIE calib: slot 21, qie 10:Warn 17: Mult. FRAM erasures, val 03.0000, cap 0, rng0.
(b0pcal11) QIE calib: slot 20, qie 15:Warn 17: Mult. FRAM erasures, val 02.0000, cap 0, rng0.
(b0pcal11) QIE calib: slot 20, qie 17:Warn 17: Mult. FRAM erasures, val 02.0000, cap 0, rng0.
(b0pcal02) QIE calib: slot 21, qie 15:Warn 17: Mult. FRAM erasures, val 02.0000, cap 0, rng0.
(MLE) b0pcal00:Messenger:12:33:06 PM->ADMEM=17 QIE=15 Fail 15: Wrong vals in FRAM  (65535.0) (0) (18)
(b0pcal00) QIE calib: slot 17, qie 15:Fail 15: Wrong vals in FRAM , val 65535.0000, cap 0, rng18.
(RC)   12:38:21 Calib_Load -> LOADED
</pre>

<!-- Author: Jeff x2080 -->
<!-- Comment: Sun Dec 14 12;59;42 comment by...m mattson -->
<br>The multiple FRAM erasures are not a problem.  On the other hand, pcal00 slot 17 ch 15 does look bad.  Last time, the "wrong vals in FRAM" was for range 3, so I thought it was a glitch that might be fixed by a second calibration.  This time, it complains about range 18.  The problem is that there are only 8 ranges (from 0-7).  There is always a chance that this is a monitoring problem, but I suspect the worse.<p>
I will make a repair submission to the ops manager, and call into the control room to see what they think.  This would require opening the west plug steel, to replace the CAFE card.<p>
This channel is PHA, phi 0W, eta 17.  The CO should check any YMon plots during future cosmic runs, to see what strange behavior comes out of this channel.
<!-- Date: Sun Dec 14 12:51:38.559 2003 -->
<pre>There seems to be a problem in loading the "calorimeter constants".  We are getting errors.  I will wait for a short time to see if Mark Mattson calles back... he was also going to check the log files.  </pre>
<!-- Author: Rick -->
<!-- Comment: Sun Dec 14 13;14;14 comment by...m mattson -->
The DAQ Ace can use pcal00 again.  I've finished doing diagnostics.
<!-- Comment: Sun Dec 14 13;23;07 comment by...m mattson -->
<br>The CAFE in pcal00 slot 17 ch 15 is clearly losing its mind.  When running qietest, I did a map of the crate (checking the id numbers of the ADMEMs and CAFE cards).  That channel reported 65534, which is a nonsense number (should be 4 digits only).<p>
On the other hand, pedestals (in both pass-through and FRAM-lookup) are normal, as was the QIE calibration (slopes).  I looked at parts of the FRAM table, which appear okay.  There is a chance that this channel will behave normal enough (during datataking) that it will not disrupt the DAQ.<p>
I am submitting a repair request, and have alerted the shift crew that if the channel acts up, it may be necessary to have the L1 trigger pager mask the channel.  I'm just not certain when or if the CAFE will flake out completely.
<!-- Date: Sun Dec 14 13:10:44.075 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176388">176388</A>
 Activated at 2003.12.14 13:09:17
<!-- Author: RunControl -->



<!-- Date: Sun Dec 14 13:12:08.821 2003 -->
<pre>I finally got a hold of Hyunsoo Kim.  He is coming down to set-up the "telescope" and he should be finished before "search & secure".</pre>
<!-- Author: Rick -->

<!-- Date: Sun Dec 14 13:13:36.318 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176388">176388</A>
ACTIVATE: we return to cosmics; this is with all of the standard cosmics crates except for pcal00, which I've released to Mark so he can debug the origin of the FRAM failure messages during calib_load'ing. Using COSMICS_NOTRACKS [6,353,403] table, all files to tape and look area
<!-- Author: Jeff x2080 -->
<!-- Comment: Sun Dec 14 13;16;03 comment by...m mattson -->
<br>You can put pcal00 back in the run, when ready.  If there are any trigger problems from this channel, it would show up in HAD phi 0, eta 3 (trigger coordinates).

<!-- Comment: Sun Dec 14 13;22;17 comment by...Jeff -->
Also, I have switched Level3's l3proxy process to the one that runs on b0l3pcom1 so that we can try to understand its (occassional) failure modes.  The state/transition area of the L3 Display is always red when we test this, because the monitoring does not yet function there.


<!-- Date: Sun Dec 14 13:24:24.254 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176388">176388</A>
 Terminated at 2003.12.14 13:22:01
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 13:24:25.513 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176388">176388</A>
TERMINATE: will include pcal00 now, per instructions
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 13:29:16.708 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176389">176389</A>
 Activated at 2003.12.14 13:27:27
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 13:29:17.763 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176389">176389</A>
ACTIVATE: same setup as the previous run, except that I've included pcal00
<!-- Author: Jeff x2080 -->



<!-- Date: Sun Dec 14 13:29:46.483 2003 -->
There were problems with TOF channel 308 last night ( trips ). 
I told an ACE to turn the channel off. 
 
I just came in and turned that channel back on, ACEs should 
keep an eye on it and if it trips again, go downstairs and click 'reset trip and leave off' button for that channel in SMACs. 
Then please make an entry in TOF e-log. 
 
  
<!-- Author: Denys (TOF pager) -->
<!-- Comment: Sun Dec 14 13;35;50 comment by...Denys -->
See also <a href=
'/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=14&frame=2&anchor=13:34:49'>this TOF e-log entry</a>
<!-- Date: Sun Dec 14 13:31:15.258 2003 -->
<pre>ToF expert cam by to check channel #308.  He reset it and it seems okay now.  If it trips again we should turn it off and wait for the high voltage expert on Monday.  If any other channel trips we should page him again.</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 14 13:41:49.859 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176389">176389</A>
 Terminated at 2003.12.14 13:40:40
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 13:43:59.724 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176389">176389</A>
TERMINATE: the run failed to make the activation transition.  no error windows came up, although b0svt05, b0tsi0*, l3, and evb were the non-green clients in the Replies and Ack. window
<!-- Author: Jeff x2080 -->
<!-- Comment: Sun Dec 14 13;44;43 comment by...Jeff -->
and svt05 refuses to abort now

<!-- Comment: Sun Dec 14 13;49;05 comment by...Jeff -->
svt05 won't partition either, when everyone else would.  I'm issuing a vxlogin reboot to it

<!-- Date: Sun Dec 14 14:01:37.111 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176390">176390</A>
 Activated at 2003.12.14 13:59:43
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 14:06:02.776 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176390">176390</A>
ACTIVATE: rebooting svt05 seemed to have worked. (in retrospect, a shepard would have produced this result more directly). back to cosmics, with pcal00 now
<!-- Author: Jeff x2080 -->



<!-- Date: Sun Dec 14 14:18:33.149 2003 -->
<pre>Hyunsoo Kim arrived and he is down working on the "telescope".</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 14 14:20:05.675 2003 -->
<pre>We are running COSMIC_NOTRACKS and watching closely the calarimeter rates.</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 14 15:21:22.326 2003 -->
<pre>Kim has finished his task.  Rob has finished moving the steel back.  Mary called MCR and told them we are ready for "search & secure"... but MCR did not give us a time yet.</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 14 15:32:05.933 2003 -->
<pre>JC's called. He'll be here approximately in an hour  
to investigate iFix problem.</pre>
<!-- Author: Irina -->


<!-- Date: Sun Dec 14 15:34:48.132 2003 -->
<pre>We notified "Silicon" that we might get beam during owl shift.</pre>
<!-- Author: Rick -->
<!-- Comment: Sun Dec 14 15;37;43 comment by...Jason Nielsen -->
The silicon will be OFF for these first studies/stores.


<!-- Date: Sun Dec 14 15:38:00.777 2003 -->
Paged silicon to let them know there may be beam on owl shift.  He recommended putting the HV in standby now (since we aren't including the silicon crates in the cosmic run anyway).  He will check with the SPL whether the HV should be at standby or off when we anticipate beam.
<!-- Author: convery -->


<!-- Date: Sun Dec 14 15:49:06.184 2003 -->
<pre>Looks like they will "search & secure" at 5:15pm today.</pre>
<!-- Author: Rick -->

<!-- Date: Sun Dec 14 15:56:10.838 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Dec 14 15:57:56.624 2003 -->
<font color=red>Shift Summary: </font><pre>** COSMIC_NOTRACKS ** 
Run cosmics for COT gas flow tests. 
 
** Calorimeter Constants Determined ** 
We determined the "calorimeter constants" and 
did a "Rick's Run" which looked okay.  We 
had to do it twice since Mark found "loading 
errors" on our first try.  There were errors 
on the second try as well.  We will monitor 
it closely and if the rates are bad we will 
page L1_CAL. 
 
** ToF Channel #308 ** 
The ToF expert came by to check channel #308.   
He  reset it and it seems okay now.  If it trips  
again we should turn it off and wait for the high  
voltage expert on Monday.  If any other channel  
trips we should page him again. 
 
** Search & Secure ** 
Hyunsoo Kim set-up the "telescope".  Rob finished 
moving the steel back.  Mary called MCR and told 
them we are ready.  Looks like 5:15pm for  
"search & secure".</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   176381-176390
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             NA
   
</td></tr></table></pre>
<!-- Author: Rick Field -->

<!-- Date: Sun Dec 14 16:01:51.998 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176390">176390</A>
 Terminated at 2003.12.14 15:59:57
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 16:01:52.664 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176390">176390</A>
TERMINATE: need to switch l3proxy back to the (stable) version on b0l3pcom2
<!-- Author: Jeff x2080 -->


<!-- Date: Sun Dec 14 16:13:42.847 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176391">176391</A>
 Activated at 2003.12.14 16:12:38
<!-- Author: RunControl -->


<!-- Date: Sun Dec 14 16:13:58.933 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=176391">176391</A>
ACTIVATE: starting a COSMICS NO TRACKS
<!-- Author: Jeff x2080 -->