<!-- Title: Owl shift. Tue Jul 24, 2001 -->
<!-- SciCo: Christoph Paus -->
<!-- DAQAce: Heather Ray -->
<!-- MonAce: Rob Napora -->
<!-- CO: ---- &nbsp; -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre>

Shift Plan
==========

Integrate Silicon, continue TOF installation, Aseet does some
COT HV tests.

Finish with the calibrations which started already.

Take a Rick Run as soon as TOF are done.

</pre>

<!-- Date: Tue Jul 24 01:11:30 2001 -->
<pre>Testing of the new QiePed calibration consumer. 
 
Several problems encountered.  
 
New consumer(devel): 
 
- The consumer never ends and it does not write calibration 
  constans to the db. 
 
- A calibration consumer started in a previous run still  
  can process the following qie calibration runs, since it  
  remains alive. Producing extra confusion and missing  
  consumer log files. 
 
Old consumer (3.14.0) 
 
- We can not download the FRAM, event if the calibrations  
are in the database and the consumer stopped. I have the following error message: "Calibload not allowed without 
loadQIEFRAM" 
 
 
	</pre>
<!-- Author: Ivan -->


<!-- Date: Tue Jul 24 01:38:34 2001 -->
<pre>I made ups product of the latest version of Event Builder proxy and made it current version. I also modified evbproxy account accordingly so , the change will take effect as soon as the aces clean up EVB  </pre>
<!-- Author: Arkadiy -->


<!-- Date: Tue Jul 24 01:49:34 2001 -->
<ul> 
<li>Inherit Run 121363 from previous shift with Cosmics_All[2,165].  No Si or WCAL in.  
<li>Ended run with 437K due to many BTO from  COT_Pulser00.  this happened a few minutes after Rob did a reset_crate on the Silicon.  Not sure if there was a connection or not... 
<li>Clean up EVB and try another run 
<li>Run 121373 get DTO from CMX01 immediately.  Gets stuck in Halt of HRR.  end run.  Crate pings okay, can't VxLogin.  issue reset_crate 
<li>Run 121378 - get lots of SCPU Duplicate Load errors.  RC increments events, L3 display is not doing anything.  HRR doesn't fix.   
<li>end run and L3 display begins flashing - for several minutes it appears to run on it's own accord.  abort does not fix  
<li>Kill L3 gui and restart - it appears to be okay now 
<li>Run 121381 started - get the same problem with L3 display not doing anything and RC gui incrementing events 
<li>So far there have been no OOM errors from RC 
<li>end run and clean up EVB and L3 
<li>On ColdStart a bunch of crates give waiting for r/o list error - now see java OOM error 
<li>Kill RC and restart....  
</ul>
<!-- Author: hray -->


<!-- Date: Tue Jul 24 02:57:18 2001 -->
Installed TOF transistion boards in b0wcal05 and did some repairs  
in other crates. See <a href= 
'/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=1&frame=2&anchor=02:40:03'> 
TOF logbook</a>. 
 
Need wcal03 and wcal07 for another 40 min 
for cafe-delay scans.
<!-- Author: Matthew,Gerry,Rolf -->


<!-- Date: Tue Jul 24 03:14:35 2001 -->
<pre>Among all the problems with Level3 Display we observe  
one of the nodes going to end state during ColdStart  
transition. That means that problem is not solved from  
Level3 side. </pre>
<!-- Author: Boris.  -->


<!-- Date: Tue Jul 24 03:18:36 2001 -->
<p>CMP Top Ch.3 tripped.  Reset.
<!-- Author: Napora -->


<!-- Date: Tue Jul 24 03:48:06 2001 -->
<pre>L3Display stop showing any data flow and catch up RC 
very fast on End transition. Looked like messages were  
stored somewhere and sended to L3Display in very fast burst.  
Shift crew tried to cleanup Level3 and restart L3Display.   
It did not help.  
 
When I came we found a lot of pending messages in different  
processes connected to RT server from b0l3pcom1 b0l3pcom2  
b0dap57. First we killed those processes and tried to start  
run again. It did not help. We restarted RT server and Relay 
service on Level3. Everything went ok but on end run it took  
L3Display ~2 min to catch up RC.   
 
Level3 control code was changed today at ~17:30. We switched  
back to older version of Level3 control code and wait for  
results.  
 
 
 
</pre>
<!-- Author: Boris.  -->


<!-- Date: Tue Jul 24 04:22:27 2001 -->
<ul> 
so, to recap all things tried to fix L3 by both expert and shift crew/sci-co 
<li>at first sign of problem (~1:30) did HRR 
<li>cleaned up L3, EVB, closed RC and killall -9 java 
<li>killed L3 gui and restarted 
<li>on b0dau30 did:/usr/etc/netstat | grep 5101 | sort -u -k5 
and see l3pcom1 shows bytes send'ed but not yet read by client >48K.  the fix for this from Ace page says to log into the PC and kill processes with large RecvQ values.  Christoph Paus has account on this machine so he tries this.  It does not work.  This is the point when we paged L3 (~2:15)
<li>Tried stopping/starting L3 and EVB proxies 
<li>Tried closing all java on b0dap57, kilall -9 java 
<li>Tried logging out of b0dap57  
<li>Tried killing and restarting RTserver, and then RTserver and RTLM 
<li>Tried changing to older version of L3 code 
<li>call both Jim Patrick and Ilya (~4:00)
</ul>
<!-- Author: hray -->


<!-- Date: Tue Jul 24 04:46:28 2001 -->
Boris is still on the phone with Ilya.  Jim called back.  He checked the RTserver logs - so far tonight it has always been under the max number of licenses, it has been logging every 10 minutes, and logging monitor info ~every minute.  The web page:/daqmon/servlet/DaqMonServlet3 
receives the same information as the L3 display.  This page showed L3 has been sending out appropriately, so his conclusion is that this problem has nothing to do with the RTserver.  It is a L3 monitoring issue.
<!-- Author: hray -->


<!-- Date: Tue Jul 24 05:08:02 2001 -->
<pre>Finally swithed to one of the private versions  
of L3Display. It works.  
 
L3Display code was not changed since 13 of July.  
We do not have a clear understanding of the failure.  
Decided to leave it for the morning. </pre>
<!-- Author: Boris.  -->


<!-- Date: Tue Jul 24 05:33:27 2001 -->
<ul> 
Took some calibrations while Boris was chugging along with L3 stuff 
<li>CAL QIE - run 121401 fails 
<li>CAL LED - run 121404 passes 
<li>TOF - run 121403 fails 
<li>XENON FLASHPED - run 121405 passes 
<li>XENON ALL - run 121406 passes 
all calibrations above made it to the DB.  See below for some error messages from QIE and TOF
</ul>  
<!-- Author: hray -->


<!-- Date: Tue Jul 24 05:35:10 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1675&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1675  width='300' align='top'></td><td>CAL QIE RC error window, part 1, from failed calibration</a></td></tr></table>
<!-- Author: hray -->


<!-- Date: Tue Jul 24 05:35:52 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1676&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1676  width='300' align='top'></td><td>CAL QIE RC errors - part 2</a></td></tr></table>
<!-- Author: hray -->


<!-- Date: Tue Jul 24 05:37:00 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1677&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1677  width='300' align='top'></td><td>TOF RC errors from failed calibration</a></td></tr></table>
<!-- Author: hray -->


<!-- Date: Tue Jul 24 05:43:29 2001 -->
Run 121407 Rick Run (PMT spike) with 61K events) accomplished okay
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121407' target=_top>121407</a>) -->


<!-- Date: Tue Jul 24 06:07:49 2001 -->
Run 121408 ended at 58k due to OOM errors.  Run with entire system (silicon integrated) and Cosmics_All[2,165] 
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121408' target=_top>121408</a>) -->


<!-- Date: Tue Jul 24 06:13:31 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>the VxWorks window from DAQMON keeps killing itself!  This has been going on for the past hour or so.  Restarting DAQMON does not help.  PLEASE FIX THIS</td></tr></td></table>
<!-- Author: hray -->
<!-- Comment: Tue Jul 24 06;18;49 comment by...hray -->
<table border=0><td><tr><td bgcolor=#e0e030>L3 display started to do the same thing - it kills itself!  </td></tr></td></table>


<!-- Date: Tue Jul 24 07:11:49 2001 -->
Run 121409 ended at 213K due to  OOM  
<!-- Author: hra :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121409' target=_top>121409</a>) -->


<!-- Date: Tue Jul 24 07:37:54 2001 -->
Run 121410 HRR at 106 K because RC gui and L3 show no events incrementing.  After HRR RC gui increments; L3 display does not.  Try end and Coldstart.  
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121410' target=_top>121410</a>) -->


<!-- Date: Tue Jul 24 07:46:40 2001 -->
Problems with the displays on the b0dap57 computer all night.  At sci-co's urging I reboot the computer (instead of logging out, as was done earlier this morning).  Aside from the above mentioned problems with L3 display not working properly there were other problems of first VxWorks display killing itself and not staying open, and then of this same thing happening with L3 display.  Killing DAQMON/killing all java/restarting did not help.  
<!-- Author: hray -->


<!-- Date: Tue Jul 24 07:53:08 2001 -->
<font color=red>Shift Summary: </font><pre>Shift Summary 
============= 
 
No beam in the machine, access in progress. 
 
TOF Qie tests not very successful, see details above. 
 
Level3 display does not work properly. Data taking is going on without 
problems?! Experts will resume work today. 
 
TOF electronics installation made great progress (3/4 done). Errors in 
the calibrations should be looked at by experts. 
 
Finished of the calibration (Rick run included). 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   121363-121412
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             100%
   
</td></tr></table></pre>
<!-- Author: Christoph Paus -->


<!-- Date: Tue Jul 24 08:01:28 2001 -->
Trying to start a new run - FIB00 then FIB02 fail.  Rob et al say this is due to flakey-ness, not a real problem.  So far rebooting the computer helped and at least the L3 and VxWorks guis are staying up on the dap57 screen (for now - keep your fingers crossed!).  Second L3 display running on one of the CO computers, too.  Passing the torch on to the day shift
<!-- Author: hray -->


<!-- Date: Tue Jul 24 08:12:54 2001 -->
<pre>PEM alarm on Crate 1.</pre>
<!-- Author: Andras -->
<!-- Comment: Tue Jul 24 08;13;31 comment by...Andras -->
<pre>Cleared itself within 2 minutes.</pre>