<!-- Title: Eve shift. Thu Dec 15, 2005 -->
<!-- SciCo: Beate Heinemann -->
<!-- DAQAce: Sergo Jindariani -->
<!-- MonAce: Conor Henderson, Dean Hidas -->
<!-- CO: Gianluca Introzzi -->
<!-- OpManager: Bob Wagner -->
<!-- Notes: -->
<pre>Nothing to do...no beam expected druing this shift. Will run cosmics, L2 torture and calibrations.</pre>


<!-- Date: Thu Dec 15 16:32:05.644 2005 -->
MCR called to let us know that there will be no beam before 08:00 tomorrow morning and to ask if we needed to access the Collision Hall between now and then.  I noted that we had no pressing need for an access and ascertained again that they would like to keep us out of the hall unless there's an urgent need to access.
<!-- Author: Bob Wagner (ANL) -->

<!-- Date: Thu Dec 15 16:43:56.293 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208954">208954</A>
 Terminated at 2005.12.15 16:43:42
<!-- Author: RunControl -->



<!-- Date: Thu Dec 15 16:43:56.030 2005 -->
<pre>We have finished our XTC2 tests for today with the COT crates.</pre>
<!-- Author: Greg -->

<!-- Date: Thu Dec 15 16:44:34.950 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208954">208954</A>
TERMINATE: will try L2 torture
<!-- Author: Sergo x2080 -->



<!-- Date: Thu Dec 15 16:53:06.034 2005 -->
With permission of Markus, I changed the network connection 
between the L3 MAIN switch in 3RR21J and the SCPU in 3RR08x. 
This was part of a cabling cleanup associated with installing 
new L3 racks. I had problems with getting a network link but 
eventually succeeded after replacing a crossover cable. Sometime 
during this process, DAQ Ace started a new run and had problems 
with L3 partition. Markus is coming to take a look.
<!-- Author: jj -->


<!-- Date: Thu Dec 15 17:01:11.740 2005 -->
TOF tripped and recovered immediately.
<!-- Author: Sergo -->

<!-- Date: Thu Dec 15 17:04:55.942 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208960">208960</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,729,599] 
<!-- Author: Sergo x2080 -->



<!-- Date: Thu Dec 15 17:10:06.010 2005 -->
<pre>L3 didn't partition while trying to start new run. I did L3 farm check and there was a whole bunch of nodes that it wasn't able to ssh into. We called Markus, we stopped by and fixed it. Nodes were hung up b/c of cable switching that JJ was doing.</pre>
<!-- Author: Sergo -->


<!-- Date: Thu Dec 15 17:11:28.887 2005 -->
<pre>Running L2torture, COT crates appear to be fine. Get occasional L2 decision timeouts ( ~every few minutes). I'll switch to cosmics if they become too frequent.</pre>
<!-- Author: Sergo -->


<!-- Date: Thu Dec 15 17:17:58.219 2005 -->
<pre>Bob Wagner ( FNAL ) asked me to keep COT on ( while there's no beam ) to see if it trips. If it does and it's before midnight - he wants shift crew to call him.  </pre>
<!-- Author: Sergo -->
<!-- Comment: Thu Dec 15 17;48;20 comment by...Sergo -->
<pre>COT S6S8 tripped 10 minutes after I turned COT on. I called Bob and he's coming over to check on it. In a meanwhile I put COT to standby.</pre>
<!-- Date: Thu Dec 15 17:19:28.911 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208960">208960</A>
TERMINATE: too many L2 decision TO's. Switching to cosmics
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 17:21:39.540 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208961">208961</A>
 Activated at 2005.12.15 17:20:30
<!-- Author: RunControl -->


<!-- Date: Thu Dec 15 17:21:40.084 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208961">208961</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,598] 
<!-- Author: Sergo x2080 -->



<!-- Date: Thu Dec 15 17:23:25.806 2005 -->
Done TO ccal00.
<!-- Author: Sergo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208961' target=_top>208961</a>) -->


<!-- Date: Thu Dec 15 18:16:01.744 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208961">208961</A>
 Terminated at 2005.12.15 18:15:13
<!-- Author: RunControl -->


<!-- Date: Thu Dec 15 18:16:02.818 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208961">208961</A>
TERMINATE: Will try cosmics nosilicon tape to cot data to tape 
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 18:31:01.903 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208968">208968</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,725,597] Run to test COT spike trigger 
for Bob Wagner
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 18:32:30.846 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208968">208968</A>
TERMINATE: Need to write to disk
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 18:36:41.559 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208969">208969</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,725,597] 
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 18:52:14.030 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208969">208969</A>
TERMINATE: errlog was stuck in Halt transition
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 18:55:11.997 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970">208970</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,725,597] 
<!-- Author: Sergo x2080 -->



<!-- Date: Thu Dec 15 18:59:05.253 2005 -->
<pre>COT Spike Run: 
 
Bob Wagner (Fnal) requested a special run to test COT Spike Trigger. I didn't know which run configuration has to be used in order for this trigger to be included. Luckily Bill Badgett was around and helped us: 
 
 One should use AAA_Shotsetup configuration. In addition in order to write it into look area in CSL menu one should select "All Files". 
 
 
  While running I got Busy TO and then errlog got stuck in transition to Halt. Bringging it to Halt manually didn't help. So I had to reset and start a new run.</pre>
<!-- Author: Sergo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->


<!-- Date: Thu Dec 15 19:17:33.156 2005 -->
We had another problem with errlog.  It hung when issuing an HRR.  did a few manual overrides to get everything but errlog to the halted state.  closing the error display and bringing it back through the transitions seems to have it working again.  We noticed that this is v3_02_a which seems to be a recent update. (within the last two days)
<!-- Author: Dean :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->
<!-- Comment: Thu Dec 15 19;27;15 comment by...Dean -->
this was after a busy TO
<!-- Date: Thu Dec 15 19:18:23.566 2005 -->
TOF HV alarm.  cleared in two seconds.
<!-- Author: Dean :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->



<!-- Date: Thu Dec 15 19:32:40.260 2005 -->
<pre>DAQ Aces reported that ErrorHandler was not responding  
to the HALT transition;  I noticed a Java stack trace  
in the latest errorLogger session log file -- perhaps  
Donatella can have a look in the morning;   I put the  
errmon version pointer back to v3_02.</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->
<!-- Comment: Thu Dec 15 19;38;39 comment by...Donatella T. -->
thanks... I was just looking at it. And fixed the java problem.
I've reinstated  v3_02_a. Please restart Eh to use this latest version. I'll be checking the e-log so I can fix other problems if any.
Thanks



<!-- Comment: Thu Dec 15 19;56;13 comment by...Conor -->
I have restarted EH - it is now v3_02_a. Everything looks fine so far.
<!-- Comment: Thu Dec 15 19;59;57 comment by...Conor -->
have got 2 busy TOs, accompanied by the new-look crate-less window
<!-- Date: Thu Dec 15 19:54:47.889 2005 -->
heartbeat alarm for CLC, VSC, RPS, MNP, and CPR.  Went downstairs and restarted the HV monitoring.  All are back in touch now
<!-- Author: Dean :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->




<!-- Date: Thu Dec 15 20:00:36.594 2005 -->
Got ProcMOn error for RESMGR PROXY. The check now shows it is okay.
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->


<!-- Date: Thu Dec 15 20:11:12.923 2005 -->
Busy TO and/or TM2 TO
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->
<!-- Comment: Thu Dec 15 20;11;28 comment by...Conor -->
another


<!-- Comment: Thu Dec 15 20;24;49 comment by...Conor -->
Another. An auto-HRR is implied in these entries...
<!-- Comment: Thu Dec 15 20;29;10 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 20;34;51 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 20;36;07 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 20;47;15 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 21;16;22 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 21;23;31 comment by...Conor -->
two more
<!-- Comment: Thu Dec 15 21;30;49 comment by...Conor -->
three more
<!-- Comment: Thu Dec 15 21;46;47 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 21;50;51 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 21;58;06 comment by...Conor -->
another
<!-- Comment: Thu Dec 15 22;14;10 comment by...Conor -->
If only there were a way of writing tally marks in the elog for these kind of errors. It would save a lot of space...
<!-- Comment: Thu Dec 15 22;21;11 comment by...greg feild -->
<pre>The BUSY timeouts seem to be due to xft.
I'm guessing that readout lists are off
for special xft studies.  If so, you may 
stop tallying, I'd say.</pre>
<!-- Comment: Thu Dec 15 22;34;27 comment by...Conor -->
I see an entry in the previous shift's elog at 2pm asking the shift crew to reboot b0xft00-05. But I find no entry saying that this was in fact done. Perhaps this reboot might help with the busy TOs we are seeing? I could check if the reboot had been done if I knew the VxWorks equivalent for 'uptime' ...
<!-- Comment: Thu Dec 15 22;48;20 comment by...Conor -->
<pre>Aha, I found it in our super-helpful VxWorks page linked from the DAQ-ACE-help page:

b0xft00-&gt; printBootTime
System was booted at WED DEC 14 19:24:30 2005

So it looks like these crates have not been rebooted since Jay Dittman made the request in the elog at 2pm today. 
I will reboot them at 11pm, after the current COT spike run stops.</pre>
<!-- Date: Thu Dec 15 20:21:51.583 2005 -->
<pre>&lt;quote&gt; 
17'30" b0server03.fnal.gov:ConsumerErrorRe:Dec 15, 2005 8:17:30 PM-&gt;DAQErrMon:DataMissingError: Error 2640 Event 11681 Run 208970 Bank COTD_StorableBank missing from event record, nErrors 2543  --&gt;   
 Additional Information:  
 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!!  
  
 DAQErrMon:DataMissingError:  
 Error 2640 Event 11681 Run 208970 Bank COTD_StorableBank missing from event record, nErrors 2543  
 
 - We suggest issuing an HRR.  
  
 - if the problem persists for more than a few minutes,  
   starting a new Run may be necessary.  
 
&lt;endquote&gt; 
 
So, I did as it suggested and HRRed. I had never seen this error before the last few days. Can any elog-reader out there explain if it is a serious problem? I will continue to follow the instructions from the error display. 
 
</pre>
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->
<!-- Comment: Thu Dec 15 21;20;18 comment by...Dean -->
<pre>
again with:
  Error: 5200
  Event: 22367
  nErrors 5103

I did the suggested HRR.</pre>






<!-- Comment: Thu Dec 15 21;59;06 comment by...greg feild -->
<pre>Since you asked, here is what I know:

DAQERRMON has been with us for a while but it has
never actively participated in a run.  Someone
(Bill?) must think it is ready for prime time.  Is it?
Do you see the missing COTD banks in the L3 or Error log
files?

PS - It is a CONSUMER ... runs on one of the middle 
CO terminals.

</pre>

<!-- Comment: Thu Dec 15 22;05;08 comment by...jj -->
If you go to Consumer Operator Help page, there is link
to DAQERRMON with basic details on what this consumer does.





<!-- Comment: Thu Dec 15 23;36;41 comment by...Conor -->
Thanks for the information. Bill mentioned that these errors most often result from the consumers being in the wrong mode (physics vs. cosmics) And true enough, the comsumers were still in cosmics mode while for the COT spike run we ran in AAA_shotsetup, which should require physics mode for consumers.
<!-- Comment: Fri Dec 16 15;20;03 comment by...badgett -->
DAQErrMon has been participating in the run since 
many months, Greg!
<!-- Date: Thu Dec 15 22:57:43.136 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970">208970</A>
TERMINATE: end of COT spike run
<!-- Author: Sergo x2080 -->



<!-- Date: Thu Dec 15 22:57:44.734 2005 -->
End of COT spike run. The COT did not trip again; nor did we see any counts for the trigger L1_COT_SPIKE. 
<!-- Author: Conor :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208970' target=_top>208970</a>) -->
<!-- Comment: Thu Dec 15 23;48;12 comment by...Conor and Dean -->
The COT has been turned off again 
<!-- Date: Thu Dec 15 23:11:25.052 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208971">208971</A>
 Activated at 2005.12.15 23:10:50
<!-- Author: RunControl -->


<!-- Date: Thu Dec 15 23:11:26.383 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208971">208971</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,598] 
<!-- Author: Sergo x2080 -->



<!-- Date: Thu Dec 15 23:18:34.165 2005 -->
I have rebooted boxft00-05 using the LocalClientController from RunControl. We are now running in cosmics mode.
<!-- Author: Conor -->
<!-- Comment: Thu Dec 15 23;19;05 comment by...Conor -->
This reboot was in response to Jay Dittman's request at 2pm today


<!-- Comment: Thu Dec 15 23;33;27 comment by...Conor -->
I had a bit of a scare while doing this. In order to try to find out if the xft crates had already been rebooted, I vxlogin-ed to b0xft00. While trying to type 'printBootTime' I made a typo and then could not backspace. Instinctively I hit Ctrl-C, without thinking that this was not Linux. Under VxWorks, Ctrl-C, I have since discovered, aborts and restarts the shell. This was very dumb of me. RunControl was in Active state during our COT spike run at the time. b0xft00 showed up as unresponsive and failed shperherding. I ended the run, brought all clients to start state (using manual override) and then used the LocalClientController to reboot and shepherd the crates b0xft00-05. Shepherding was successful for all of them, and now we are taking cosmics with no problems. I am relieved that there appear to be no lasting ill-effects resulting from my dumbness.
<!-- Date: Thu Dec 15 23:26:34.574 2005 -->
TOF PS alarm.  gone in 2 seconds.
<!-- Author: Dean -->
<!-- Comment: Thu Dec 15 23;37;02 comment by...Dean -->
and another


<!-- Date: Thu Dec 15 23:39:16.585 2005 -->
Incidentally, earlier we got an 'RXPT Loop error' and I see that with this version of the ErrorHandler, this error comes with an audible alert. This is very helpful since this particular error disables auto-HRRs - thanks Donatella!
<!-- Author: Conor -->


<!-- Date: Thu Dec 15 23:53:14.246 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208971">208971</A>
 Terminated at 2005.12.15 23:53:09
<!-- Author: RunControl -->



<!-- Date: Thu Dec 15 23:53:28.327 2005 -->
<font color=red>Shift Summary: </font><pre>- no beam until tomorrow morning. Tevatron are stacking pbars though. 
 
- ran COT spike run for Bob Wagner but did not get any event and also no trip.  
 
- else had some confusion due to errors from DAQErrMon which got probably resolved since the consumers were in COSMICS mode</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   -
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             0
   
</td></tr></table></pre>
<!-- Author: Beate Heinemann -->

<!-- Date: Thu Dec 15 23:55:38.492 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=208971">208971</A>
TERMINATE: end to include level3 node 070
<!-- Author: Sergo x2080 -->


<!-- Date: Thu Dec 15 23:55:40.023 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>Physics able</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L1Accepts</TH>
<TH>L2Accepts</TH>
<TH>L3Accepts</TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->