<!-- Title: Owl shift. Sat Nov 15, 2003 -->
<!-- SciCo: Beate Heinemann -->
<!-- DAQAce: Massimo Casarsa -->
<!-- MonAce: Eli Ben-Haim -->
<!-- CO: Subir Sarkar -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>running Silicon calibrations from Rainer's wishlist. Will run cosmics afterwards. L3 experts working on fixing crashes at end of run.</pre>

<!-- Date: Sat Nov 15 00:06:06.149 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174097">174097</a>
ACTIVATE: SVX bias verification run with extra0=4359 and bias in STDBY 
<!-- Author: Massimo x2080 -->



<!-- Date: Sat Nov 15 00:20:12.797 2003 -->
There was an ISL alarm. ISL B5 W0 all ladders tripped. Immediately after we tried to recover and put the wedge on standby it tripped again. 
The run is in halt. 
We paged silicon.  
<!-- Author: Eli :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174097' target=_top>174097</a>) -->


<!-- Date: Sat Nov 15 00:25:47.927 2003 -->
The Level3 network services seem to crash on end-transition after nearly every run, for all converter nodes included in that run. Issuing a 'network restart' command to the affected machines followed by a L3 farm cleanup will solve the problem, though it only breaks again after the next run. Boris suspects an ATM crosstalk problem, although this is beyond my experience to handle. If the failure persists after the current Si calib run, I will switch L3 back to the older kernel subfarms (1-8).
<!-- Author: Jeff Miles/Boris Iyutin -->
<!-- Comment: Sat Nov 15 00;35;19 comment by...Jeff -->
For the record, the console messages from misbehaving converter nodes are of the form:<br>
-------<br>
nicstar0: Opening vpi.vci *.0 RCVBUF quota = 2097152<br>
eth1: Too much work during an interrupt, csr5 = 0xfc6***<br>
-------<br>
As these are apparently ATM-related, I will restart all upgraded converters and associated switches before defaulting to the old farms.


<!-- Date: Sat Nov 15 00:39:31.096 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12037&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12037  width='300' align='top'></td><td>.</a></td></tr></table>
<!-- Author: Eli -->
<!-- Comment: Sat Nov 15 00;57;26 comment by...Eli -->
History plot for the ISL 2VDOIM B5 W0
<!-- Date: Sat Nov 15 00:41:36.829 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174097">174097</a>
TERMINATE: Run aborted, as requested by L3 expert, who is trying to fix the level-3.
<!-- Author: Massimo x2080 -->



<!-- Date: Sat Nov 15 00:54:03.874 2003 -->
Rainer asked us to wait for instructions from him before we continue to run.
<!-- Author: Eli -->


<!-- Date: Sat Nov 15 01:36:19.635 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174099">174099</a>
 Activated at 2003.11.15 01:35:17
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 01:36:20.935 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174099">174099</a>
ACTIVATE: L3 test
<!-- Author: Jeff/Massimo x2080 -->


<!-- Date: Sat Nov 15 01:36:39.869 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174099">174099</a>
 Terminated at 2003.11.15 01:36:29
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 01:37:29.109 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174099">174099</a>
TERMINATE: watching for L3 converter failure now
<!-- Author: Jeff/Massimo x2080 -->


<!-- Date: Sat Nov 15 04:15:01.574 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174100">174100</a>
 Activated at 2003.11.15 04:14:38
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 04:15:03.959 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174100">174100</a>
ACTIVATE: L3 test
<!-- Author: Jeff/Massimo x2080 -->


<!-- Date: Sat Nov 15 04:28:22.121 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174100">174100</a>
 Terminated at 2003.11.15 04:17:45
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 04:28:23.826 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174101">174101</a>
 Activated at 2003.11.15 04:27:37
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 04:30:33.205 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174101">174101</a>
 Terminated at 2003.11.15 04:28:24
<!-- Author: RunControl -->



<!-- Date: Sat Nov 15 04:46:22.087 2003 -->
Further Level3 upgarde work will continue tomorrow.
<!-- Author: Nuno/Jeff -->

<!-- Date: Sat Nov 15 04:56:54.923 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174103">174103</a>
ACTIVATE: SVX bias verification run with extra0=4359, bias in standby, Solenoid off.
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 05:11:06.596 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174103">174103</a>
TERMINATE: SVX biasverification run
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 05:43:09.550 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174104">174104</a>
TERMINATE: Run aborted because L3 proxy process crashed
<!-- Author: Massimo x2080 -->
<!-- Comment: Sat Nov 15 05;59;19 comment by...Jeff Miles -->
Level3 proxy process died during the config transition, as signaled by the total red-ness of the State/Transition area in the Level3 Display.  A simple Clean-up L3 Farm command solves the problem, although the fact that it happened at all is a little worrisome.
<!-- Date: Sat Nov 15 05:53:04.506 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174105">174105</a>
ACTIVATE: SVX bias verification run (extra0=4359, HV ON, Solenoid off)
<!-- Author: Massimo x2080 -->



<!-- Date: Sat Nov 15 06:20:58.288 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174105">174105</a>
TERMINATE: Ended SVX bias verification run
<!-- Author: Massimo x2080 -->
<!-- Comment: Sat Nov 15 06;27;49 comment by...Massimo/Eli -->
During the end transition we had a L3 internal communication failure. We aborted the run and reset RC.

<!-- Date: Sat Nov 15 06:48:33.703 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174106">174106</a>
TERMINATE: L3 proxy crashes again during configuration.
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 06:48:34.536 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174107">174107</a>
TERMINATE: Another L3 proxy crash
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 06:48:35.603 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174108">174108</a>
ACTIVATE: SVX pedestal D mode run (PDS ON, Solenoid off).
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 07:08:22.185 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174108">174108</a>
TERMINATE: Ended SVX pedestal run (DPS ON, Solenoid off)
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 07:21:10.553 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174109">174109</a>
ACTIVATE: SVX pedestal D mode run (DPS OFF, Solenoid off)
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 07:36:08.200 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174109">174109</a>
TERMINATE: Ended silicon pedestal Dmode run with DPS OFF and Solenoid off.
<!-- Author: Massimo x2080 -->


<!-- Date: Sat Nov 15 07:55:34.175 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->

<!-- Date: Sat Nov 15 07:58:04.190 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174110">174110</A>
 Activated at 2003.11.15 07:57:33
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 07:58:33.884 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174110">174110</A>
ACTIVATE: L3 test with as many crates as possible except for Si
<!-- Author: Jeff x2080 -->


<!-- Date: Sat Nov 15 08:00:26.251 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174110">174110</A>
 Terminated at 2003.11.15 07:59:10
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 08:00:28.114 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174110">174110</A>
TERMINATE: L3 test, watching for failure on end transition
<!-- Author: Jeff x2080 -->


<!-- Date: Sat Nov 15 08:02:33.095 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174111">174111</A>
 Activated at 2003.11.15 08:02:18
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 08:02:57.078 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174111">174111</A>
ACTIVATE: L3 test; same config as previous
<!-- Author: Jeff x2080 -->


<!-- Date: Sat Nov 15 08:05:42.613 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174111">174111</A>
 Terminated at 2003.11.15 08:05:25
<!-- Author: RunControl -->


<!-- Date: Sat Nov 15 08:06:42.173 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174111">174111</A>
TERMINATE: end L3 verification; system seems stable as long as Silicon crates are excluded
<!-- Author: Jeff x2080 -->



<!-- Date: Sat Nov 15 08:08:07.023 2003 -->
<font color=red>Shift Summary: </font><pre>- started shift with taking Silicon calibration runs 
- trip of Silcon ladder ISL B5 W0 in StandBy was worked on by expert  
for several hours. The ladder is okay now. 
- L3 proxy crashed al the time. Experts have been working on it all night 
and are still working on it 
- took all Silicon calibrations again after change of settings by Rainer</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   0
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             0
   
</td></tr></table></pre>
<!-- Author: beate -->