<!-- Title: Owl shift. Wed Jun 19, 2002 -->
<!-- SciCo: Simona Rolli -->
<!-- DAQAce: Ke Li -->
<!-- MonAce: Alon Attal -->
<!-- CO: Nicola Turini -->
<!-- OpManager: Arnd Meyer -->
<!-- Notes: -->
<pre>Taking over from the previous shift: preparation for access is 
in progress. 
We paged: 
Gino and Lester 
Rob Roser 
M. Lindgren, K. Byrum 
 
MCR so far has given us an hour of access</pre>


<!-- Date: Wed Jun 19 00:52:33 2002 -->
Waiting for access crew to show up ( Rob Roser)
<!-- Author: Simona -->
<!-- Comment: Wed Jun 19 02;27;20 comment by...Simona -->
There was some miscommunication.
The previous SciCo told us that Rob was on his way here while
he was actually waiting for a second confirmation call.
<!-- Date: Wed Jun 19 00:58:47 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146783">146783</A>
TERMINATE: Cal Qie calib completed.
<!-- Author: Alon -->


<!-- Date: Wed Jun 19 01:03:09 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146784">146784</A>
ACTIVATE: CLC QIE
<!-- Author: ACE 2080 -->


<!-- Date: Wed Jun 19 01:07:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146785">146785</A>
TERMINATE: END TOF QIE
<!-- Author: ACE 2080 -->


<!-- Date: Wed Jun 19 01:11:44 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146786">146786</A>
ACTIVATE: BSC MINiplug Comb Calibration
<!-- Author: ACE 2080 -->


<!-- Date: Wed Jun 19 01:17:58 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146787">146787</A>
ACTIVATE: CAL LED Calibration 
<!-- Author: ACE 2080 -->



<!-- Date: Wed Jun 19 01:18:58 2002 -->
Access starting: 
Mike and Karen, Lester and Alon
<!-- Author: Simona -->


<!-- Date: Wed Jun 19 01:29:34 2002 -->
Want to turn on the Xenon High Voltage for XEF calibration.  
When open Xenonflasher, see a bar on the computer saying  
it is intentionally down and check back later.   
<!-- Author: Ke -->

<!-- Date: Wed Jun 19 01:33:44 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146789">146789</A>
ACTIVATE: COT calibration
<!-- Author: Jim Kraus 2080 -->


<!-- Date: Wed Jun 19 01:34:17 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146790">146790</A>
ACTIVATE: TOF TAC
<!-- Author: ACE 2080 -->


<!-- Date: Wed Jun 19 01:41:45 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146791">146791</A>
ACTIVATE: plug laser 
<!-- Author: ace 2080 -->


<!-- Date: Wed Jun 19 02:00:02 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146792">146792</A>
 Activated at 2002.06.19 01:58:19
<!-- Author: RunControl -->


<!-- Date: Wed Jun 19 02:09:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146792">146792</A>
 Terminated at 2002.06.19 02:00:30
<!-- Author: RunControl -->


<!-- Date: Wed Jun 19 02:09:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146792">146792</A>
TERMINATE: We got B0ccal05 done time out. HRR doesn't solve the problem. End the run
<!-- Author: Jim Kraus 2080 -->


<!-- Date: Wed Jun 19 02:11:09 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146793">146793</A>
ACTIVATE: XEF 
<!-- Author: Jim Kraus 2080 -->
<!-- Comment: Wed Jun 19 02;17;14 comment by...ace -->
<pre>When we want to turn off the XENON voltage, the 
15, 16, 17 SW are red. So we turn them off and back on 
and do the XEF calibration again. </pre>
<!-- Date: Wed Jun 19 02:17:13 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146794">146794</A>
ACTIVATE: Xenon Flashers calibration
<!-- Author: ACE 2080 -->



<!-- Date: Wed Jun 19 02:25:32 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146795">146795</A>
 Activated at 2002.06.19 02:24:31
<!-- Author: RunControl -->


<!-- Date: Wed Jun 19 02:27:20 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146795">146795</A>
ACTIVATE: We were going to do the rick's run, but we have to stop since 
Karem want to do smx calibration.
<!-- Author: Jim Kraus 2080 -->


<!-- Date: Wed Jun 19 02:27:20 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146795">146795</A>
 Terminated at 2002.06.19 02:26:18
<!-- Author: RunControl -->



<!-- Date: Wed Jun 19 02:29:25 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146796">146796</A>
ACTIVATE: SMX calibration

<!-- Author: ACE 2080 -->


<!-- Date: Wed Jun 19 02:55:33 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146799">146799</A>
ACTIVATE: SM QIE calibration
<!-- Author: ACE 2080 -->



<!-- Date: Wed Jun 19 03:18:06 2002 -->
<pre>SMX Access report. 
 
Determined Calibration failure in Ccal05, slot 4 (West Phi=16) 
to be a bad SMXR channel.  Swapped this board. 
 
We also swapped a SMD board in ccal11, slot 5 (East Phi=11) due 
to error bits in header word and channels failing in calibrations. 
 
Investigated cable disconnect identified  
from YMON plots on East Phi=14 (Wire channels in SMX2, ch 17-32).  
We found all cables connected. 
 
We ran stand alone  code to check swapped boards and these 
looked ok.  Couldn't confirm with DBANA due to 
problems getting calibrations into database. 
 
</pre>
<!-- Author: byrum,lindgren -->


<!-- Date: Wed Jun 19 03:24:30 2002 -->
Moved some shorting jumpers for the ISL during the controlled access.  <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=113&frame=2&anchor=03:21:08'> 
details here</a>.<p>
<!-- Author: Napora -->


<!-- Date: Wed Jun 19 03:37:46 2002 -->
access completed at around 3.00am
<!-- Author: Simona -->


<!-- Date: Wed Jun 19 03:43:50 2002 -->
<pre>The result of all calibrations tonight didn't go to DBANA, and  
we checked the latest calibration in database which was  
18.06.2002 16:02:00    146,779 PHAPED, not any of ours. 
We checked the consumer log, the final part of the log file 
was like:  
  
%ERLOG-e XRCPControlFileError:  
         rcp-E-XRCPControlFileError [#1] 
  Error while reading control file for FileSystemDB with name: 
        level3 
The control file is named: 
        /cdf/code/cdfoffline/dist/rcpdb/level3/control.dat 
The error occured while reading RCPNames and RCPIDs, on entry number: 9992 
(remember, we start counting from 0) 
  Wed Jun 19 02:15:20 2002 
  -- ZMthrow was issued at line 1215 
  of file "/home/cdfsoft/dist/packages/rcp/V00-00-14/src/FileSystemDB.cpp" 
  ... Exception thrown! 
          
          19-Jun-2002 02:15:20 run = 0 event = 0 
 
We tried several times, all the symptoms were same.  
We don't know what's going wrong. In the morning, we should  
get help from calibration expert</pre>
<!-- Author: Ke/Rocio -->
<!-- Comment: Wed Jun 19 04;02;28 comment by...Simona -->
From the offline cdf_software_help we got the same kind of errors, it looks like the 
~cdfsoft/dist/rcpdb/level3/control.dat file was corrupted.

http://listserv.fnal.gov/scripts/wa.exe?A2=ind0206&L=cdf_software_help&F=&S=&P=16320
<!-- Comment: Wed Jun 19 04;04;37 comment by...Simona -->
Also, all the monitors are crashing, possibly due to the same
problem....
<!-- Date: Wed Jun 19 03:47:45 2002 -->
In the stage of partition in RC, we got stuck with errlog and l3. And because of some yellow nodes in L3 display,  
we clean up L3 farm and L3 monitor and restart L3 proxy,  
this didn't help. So we close the partition and it works. 
<!-- Author: Ke/Rocio/Alon -->

<!-- Date: Wed Jun 19 03:49:50 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146801">146801</A>
 Activated at 2002.06.19 03:47:19
<!-- Author: RunControl -->


<!-- Date: Wed Jun 19 03:49:50 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146801">146801</A>
 Terminated at 2002.06.19 03:48:13
<!-- Author: RunControl -->




<!-- Date: Wed Jun 19 04:06:48 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146803">146803</A>
 Activated at 2002.06.19 04:05:06
<!-- Author: RunControl -->


<!-- Date: Wed Jun 19 04:11:43 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146803">146803</A>
ACTIVATE: Cosmics with default COSMICS_NOTRACKS, there is one node from the farms: 203 ended
<!-- Author: ACE 2080 -->



<!-- Date: Wed Jun 19 04:11:44 2002 -->
Monitor crashes: 
Xmon chash 
ConsumerSummaryDB::initialize(): INFO: Jun 19 03:49:31 2002: 
                   Opening connection to database 'cdfonprd.fnal.gov'. 
DisplayServer::pollConsumer(): INFO: XMon sent message: END CONNECTION 
                             ==> Going to terminate connection to XMon running on 
                             b0dap70.fnal.gov 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
runXMon: Consumer ended with value  11. 
Stage0 
%ERLOG?? Stage0 Summary: ::Evaluation:  3 ::evtProcessed:  1761::Comment:  
         Fit results NOT valid: DB Not Filled  Stage0Module Stage0Module()  
         19-Jun-2002 03:47:30 run = 146801 event = 0 
DisplayServer: Got an unexpected string from the consumer. Expected: 'CONSUMER SEND'. Received: '---'. 
DisplayServer::loop(): ERROR: poll consumer returned with: -5 
DisplayServer::loop(): Protocol between consumer and display server 
was not obeyed! Display Sever will exit! 
runStage0: Consumer ended with value  0. 
Press C-d to close window. 
 
Ymon, LumMon and TrigMon all crashed 
Display also crashed  
<!-- Author: N.T. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146801' target=_top>146801</a>) -->
<!-- Comment: Wed Jun 19 05;55;40 comment by...Simona -->
Same as for DBana.
The previous SciCo told us that Farruck called reporting 
problems with RCP database and that D. Waters should be working now on it from London.
<!-- Date: Wed Jun 19 04:26:26 2002 -->
MCR called and they are injecting a proton only beam at low beta for studies
<!-- Author: Simona -->


<!-- Date: Wed Jun 19 05:03:41 2002 -->
Called MCR and asked if they have an estimate for the next shot setup: they told us not before 7.30-8.00am
<!-- Author: Simona -->


<!-- Date: Wed Jun 19 05:03:54 2002 -->
<pre>there are two nodes (202 and 167) with one of analysis chains dead 
 
(MLE) b0l3gate2:main:4:38:12 AM-&gt;Error on L3 node b0l3202 (partition 0) Wed Jun 19 04:38:11 2002 l3_node  
  in exe_exchange (l3_ana_exe.c:559) 
  @L3_EXE_TIMEOUT:  Filter 1 timeout 
 
(MLE) b0l3gate2:main:4:55:12 AM-&gt;Error on L3 node b0l3167 (partition 0) Wed Jun 19 04:55:11 2002 l3_node  
  in exe_exchange (l3_ana_exe.c:559) 
  @L3_EXE_TIMEOUT:  Filter 1 timeout 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146803' target=_top>146803</a>) -->
<!-- Comment: Wed Jun 19 07;56;40 comment by...ace -->
<pre>We get 183/198 nodes dead. And node 203 turns pink. 
(MLE) b0l3gate2:main:7:42:41 AM-&gt;Error on L3 node b0l3183 (partition 0) Wed Jun 19 07:42:40 2002 l3_node 
  in exe_exchange (l3_ana_exe.c:559)
  @L3_EXE_TIMEOUT:  Filter 0 timeout

(MLE) b0l3gate2:main:7:48:41 AM-&gt;Error on L3 node b0l3198 (partition 0) Wed Jun 19 07:48:40 2002 l3_node 
  in exe_exchange (l3_ana_exe.c:559)
  @L3_EXE_TIMEOUT:  Filter 0 timeout
</pre>
<!-- Date: Wed Jun 19 06:27:39 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-3802&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-3802  width='300' align='top'></td><td>Only interesting plot during proton only studies.  Most losses negligible.</a></td></tr></table>
<!-- Author: Alon -->


<!-- Date: Wed Jun 19 07:37:00 2002 -->
Starting shot setup
<!-- Author: Simona -->


<!-- Date: Wed Jun 19 07:53:53 2002 -->
<font color=red>Shift Summary: </font><pre>We had an access that ended aroun 3.00am 
Silicon people ( Gino, Lester and Bob Napora) did some work 
Karen, Michael and Rob did also work. 
 
Tev studies with protono only followed. 
 
We ran with cosmic_notrack. 
 
Shot setup is under way now, it started at aroun 7.30am 
 
 
The RCPdatabase problem seem to affect the monitors ( crashing all time) and the fact that we were not able to log any of our  
calibration into DBana. 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             
   
</td></tr></table></pre>
<!-- Author: Simona -->

<!-- Date: Wed Jun 19 07:55:58 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Wed Jun 19 08:24:02 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146803">146803</A>
 Terminated at 2002.06.19 08:22:54
<!-- Author: RunControl -->


<!-- Date: Wed Jun 19 08:24:16 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=146803">146803</A>
TERMINATE: Stop cosmic run for Jimmy P. 
<!-- Author: ACE 2080 -->