<!-- Title: Day shift. Fri May 25, 2001 -->
<!-- SciCo: Tom LeCompte -->
<!-- DAQAce: Heather Ray -->
<!-- MonAce: Bill Orejudos -->
<!-- CO: none -->
<!-- OpManager: Kevin Burkett -->
<!-- Notes: -->
<pre></pre>


<!-- Date: Fri May 25 08:30:40 2001 -->
Nodes 001 & 006 are dead again.  
L3proxy can not start monitoring. 
<!-- Author: Boris.  -->


<!-- Date: Fri May 25 08:47:35 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>There was some confusion in the owl shift about why the 
data for several runs did not appear in the look area on 
fcdfsgi2. They did not appear there because they were taken 
with older Level-3 tag sets which put all events into stream 
A. Recently a change was made so that stream A events marked 
as 'cosmic ray' do not go to the look area. Unfortunately 
not all old Level-3 tagsets were removed. The tagsets, 
120 for example, will be removed to avoid this mistake  
from happening again. 
 
ACES should now use the most recent Level-3 tagsets: 125 or 126 
(or later) 
when they want to do a monica test or jimmy test. This will 
put all events into stream I. Some of these files will go 
to the look area. 
</pre></td></tr></td></table>
<!-- Author: Tony Vaiciulis -->


<!-- Date: Fri May 25 09:09:27 2001 -->
<pre>Just a note to mention that we got a call from the collision  
hall around 8:15 mentioning that one of the MX241 oxygen  
monitors was found on the ground and was indicating low 
levels of oxygen. Turned out that its batteries were low. 
Unit is now back in the control room, being charged.</pre>
<!-- Author: Bill Orejudos -->


<!-- Date: Fri May 25 09:10:56 2001 -->
<pre>Took a bunch of "RAW" cot calibrations to test  
changes I made to the front end code.</pre>
<!-- Author: Bill Orejudos :: (run <a href='/cgi-bin/runSummaryRun.cgi?115878-115881, 115883-115887' target=_top>115878-115881, 115883-115887</a>) -->


<!-- Date: Fri May 25 09:15:38 2001 -->
<pre>Boris is here to try to fix L3. We have taken a couple 
of runs so far and are still trying to get things going... 
 
Ok, run 115890 running smoothly (only 1 subfarm). The 
Scanner Manager was having problems and we had to clean up EVB. 
Note that the original problem Boris fixed was to  
reboot a couple L3 processor nodes. We have also been 
told not to use the first (0) or last (15) subfarm.</pre>
<!-- Author: Bill Orejudos :: (run <a href='/cgi-bin/runSummaryRun.cgi?115890' target=_top>115890</a>) -->


<!-- Date: Fri May 25 09:16:32 2001 -->
<pre>I restarted 001 and 006 and asked aces not to use  
subfarm 00 (c01) untill we understand the problem.  
 
l3proxy run into problems trying to connect RelayServer 
on a dead node.  
</pre>
<!-- Author: Boris.  -->


<!-- Date: Fri May 25 09:47:27 2001 -->
This is the Data Handling Test.  Kirsten T. and Tony V. are the on-site experts handling this.  Run was done using subfarms 1 through 14.  We included all crates except for the silicon, XFT, and IMU_00 - 01.  
<!-- Author: Rob :: (run <a href='/cgi-bin/runSummaryRun.cgi?115892' target=_top>115892</a>) -->


<!-- Date: Fri May 25 09:47:45 2001 -->
CCAL01 pins fixed
<!-- Author: Fred Lewis -->


<!-- Date: Fri May 25 10:28:08 2001 -->
<pre>Data Handing test ended after 500K events (took about 50  
minutes). During the test Morris was changing COT DAC  
voltages, but we saw very little change in the rates.  
 
Tape got mounted, data is being written to tape 
CA3340. Should take 6 hrs for write to tape to be 
complete.</pre>
<!-- Author: heather :: (run <a href='/cgi-bin/runSummaryRun.cgi?115892' target=_top>115892</a>) -->


<!-- Date: Fri May 25 10:39:36 2001 -->
<pre>The Alcohol bubbler temperature for CMP has  
been raised from -5C to 0C as of 7pm yesterday 
in an attempt to suppress HV trips on  
stack 42 North wall.</pre>
<!-- Author: hyunsoo & tony L. -->


<!-- Date: Fri May 25 11:21:41 2001 -->
<pre>Taking runs for Steve K. so he can test a firmware 
upgrade. Using INTEGRATION_SUMET_2(2,126). Our  
first attempt (115893) ended with: 
the "Alpha in slot 3 is no longer processing at spot
16777216". We 
ended run, did a warmstart, and tried again (115894). Better 
results - we did get a done timeout (internal buffer size 
too small) from b0l2de00 after ~30k events, but HRR 
worked and we are running again.</pre>
<!-- Author: Bill Orejudos :: (run <a href='/cgi-bin/runSummaryRun.cgi?115893-115894' target=_top>115893-115894</a>) -->


<!-- Date: Fri May 25 11:23:12 2001 -->
<pre>Run 115894 ended with the "Alpha in slot 3 is no longer  
processing at spot 0" after 64k events.</pre>
<!-- Author: bill :: (run <a href='/cgi-bin/runSummaryRun.cgi?115894' target=_top>115894</a>) -->


<!-- Date: Fri May 25 11:30:21 2001 -->
New firmware for L1 interface and Isolist were inserted 
to possibly help MB bus arbitration.  Scope pictures 
showed the logic did change as expected, but  
new features appeared...  Scope pictures are being 
sent to experts and other L2 experts should run  
their tests again,  there is a chance it helped.
<!-- Author: Steve Kuhlmann -->


<!-- Date: Fri May 25 11:47:44 2001 -->
<pre>Added more debug printouts to Run Control for help in solving  
the Java OutOfMemory problem. 
 
Please restart Run Control. 
 
</pre>
<!-- Author: W.Badgett -->


<!-- Date: Fri May 25 11:49:24 2001 -->
<pre>Still getting the alpha errors. Here is the output of 
the alpha_check: 
 
Error: Alpha is no longer processing at spot 0:   
 TSI handshake bits are 4fff0000 
 TSI startload bits are 1010194 
 TSI MODDONE bits are 4007ffef 
 Number of words for TL2D, buffer 0: 2 
 Number of words for TL2D, buffer 1: 0 
 Number of words for TL2D, buffer 2: 3 
 Number of words for TL2D, buffer 3: 0 
 Address for DMA buffer 0 is  518da0 
 Address for DMA buffer 1 is  510100 
 Address for DMA buffer 2 is  500670 
 Address for DMA buffer 3 is  500ff0 
 Address for DMA buffer 4 is  5046a0 
 Address for DMA buffer 5 is  508d80 
 Address for DMA buffer 6 is  100000 
 Address for DMA buffer 7 is  100000 
 Address for DMA buffer 8 is  100000 
 Address for DMA buffer 9 is  100000 
 Executable name is   
 
 Note that while we were running, we were getting 
errors in minicom like: 
         
Error: Number of Iso MBus words 60 
Error: Number of Iso MBus words 44 
Error: Number of Iso MBus words 43 
Error: Number of Iso MBus words 32 
Error: Number of Iso MBus words 34 
Error: Number of Iso MBus words 46 
Error: Number of Iso MBus words 46 
Error: Number of Iso MBus words 37 
Error: Number of Iso MBus words 40 
Error: Number of Iso MBus words 64 
Error: Number of Iso MBus words 47 
Error: Number of Iso MBus words 81 
Error: Number of Iso MBus words 39 
Error: Number of Iso MBus words 79 
Error: Number of Iso MBus words 42 
Error: Number of Iso MBus words 34 
Error: Number of Iso MBus words 33 
Error: Number of Iso MBus words 53 
Error: Number of Iso MBus words 37 
Error: Number of Iso MBus words 95 
Error: Number of Iso MBus words 45 
Error: Number of Iso MBus words 31 
Error: Number of Iso MBus words 95 
Error: Number of Iso MBus words 32 
Error: Number of Iso MBus words 45 
Error: Number of Iso MBus words 37 
Error: Number of Iso MBus words 40 
Error: Number of Iso MBus words 48 
Error: Number of Iso MBus words 46 
Error: Number of Iso MBus words 130 
Error: Number of Iso MBus words 31 
Error: Number of Iso MBus words 32 
 
For this run the ISO board was plugged in, but the BOSS jumper was in place on the backplane behind the ISO board.  If you don't want a board to send data, you need to physically disconnect the cables from the frontpanel, or unplug the board from the crate.  Having the board plugged in with a jumper in place may still allow data to be sent out from that board. </pre>
<!-- Author: hray :: (run <a href='/cgi-bin/runSummaryRun.cgi?115895' target=_top>115895</a>) -->


<!-- Date: Fri May 25 12:17:58 2001 -->
<pre>The Global Alarms system chirped and the PHA button turned red.  We investigated and found that Power Crate 08 had caused the error, but it came back to life while we were looking at it. Tom tells me that this is a "feature" of the system that has been known for a while.</pre>
<!-- Author: Rob -->
<!-- Comment: Fri May 25 12;24;21 comment by...TJL -->
<pre>I was under the impression that the PHA bird problem of last month went away.  But that's exactly what this looked like.</pre>


<!-- Date: Fri May 25 13:27:02 2001 -->
<pre>TrigMon wouldn't come up properly.  We paged an expert and were told that the tcl file is old and out of date.  He will come by shortly to fix it.</pre>
<!-- Author: Napora -->


<!-- Date: Fri May 25 14:01:25 2001 -->
<b>The Level 2 Tests</b><br> 
<br> 
<ul> 
  <li>First, the acknowledge window has disappeared without a trace. 
  <li><b>Run 115897:</b>crashed with the error: Alpha in slot 3 is no longer processing at spot 16777216.  We reset the crate and tried to HRR with no luck.  Run aborted. 
  <li><b>Run 115898:</b>Ran again...didn't crash, but it is not taking any data. 
  <li>Jane unplugged the cables from her board (in desperation, not because we really thought that was the problem.) 
  <li>Ended RC and killed all java.  Restarted RC and got our Ack window back. 
  <li><b>Run 115899:</b>Heather called Steve Miller who had us log into the crate and watch.  When we coldstarted we got the error: FISSION: S_errno_EIO (0x5): VME BERR received! We activated anyways and got events.  Paused the run at 115500 events to change out the <b>Clique Board</b>.  Resumed without errors, but without getting any events either--Alpha hung again.  Ended run. 
  <li>Steve Hahn took CCAL01-02 crates. 
  <li><b>Run 115901:</b>Included Clist, Isolist, and L1.  Alpha loaded ok, running ok, currently at 54506 events. 
</ul>
<!-- Author: Napora :: (run <a href='/cgi-bin/runSummaryRun.cgi?115897-115901' target=_top>115897-115901</a>) -->
<!-- Comment: Fri May 25 14;11;37 comment by...Napora -->
<pre>Run 115901 ended at 62097 events.</pre>


<!-- Date: Fri May 25 14:17:23 2001 -->
<pre>Switched to SEVB and removed the sub-farms so that Boris could use them.  Restarted under the (othwerwise) same run conditions.</pre>
<!-- Author: Napora :: (run <a href='/cgi-bin/runSummaryRun.cgi?115902' target=_top>115902</a>) -->
<!-- Comment: Fri May 25 14;34;21 comment by...Napora -->
<pre>Run ended with 1242 events.</pre>


<!-- Date: Fri May 25 14:36:30 2001 -->
<pre>Released all the WCAL crates for Nicolai and restarted under the same conditions.</pre>
<!-- Author: Napora :: (run <a href='/cgi-bin/runSummaryRun.cgi?115903' target=_top>115903</a>) -->


<!-- Date: Fri May 25 14:44:36 2001 -->
<pre>Tape CA3340 dismounted, copy job over.  Notifying Eric James and Carla G.-P.</pre>
<!-- Author: Napora -->


<!-- Date: Fri May 25 15:07:45 2001 -->
Turn off WHA HV to work on the fibers. <br> 
Central hadron laser will be on only for WHA wedges.   
<!-- Author: fotis -->


<!-- Date: Fri May 25 15:10:18 2001 -->
Made WHA source run 115904 (0x1c4c0)
<!-- Author: N.Luzhetskiy :: (run <a href='/cgi-bin/runSummaryRun.cgi?115904' target=_top>115904</a>) -->


<!-- Date: Fri May 25 15:10:34 2001 -->
<pre>The acknowledgment window has disappeared from RC again.  This new "feature" is particularly annoying and needs fixing.</pre>
<!-- Author: Napora -->


<!-- Date: Fri May 25 15:17:21 2001 -->
<pre>We did a QIE CLC Calibration run.  It ran fine and showed up in DBAna.  The experts looked at the plots. The troubles they had doing this on last night's owl shift are a mystery.</pre>
<!-- Author: Napora -->


<!-- Date: Fri May 25 15:35:49 2001 -->
<pre>Carla called, tells us she was able to run a job off 
the tape that was written to earlier (CA3340).</pre>
<!-- Author: Bill  -->


<!-- Date: Fri May 25 15:49:50 2001 -->
At 15:45 an ethernet switch crashed.  This wiped out (among other things) our access to the E-log.  Jim Patrick reset it.
<!-- Author: TJL -->


<!-- Date: Fri May 25 15:55:38 2001 -->
<pre>TS #002 has returned from prep with the new VME chip 
in place and I put it  back into partittion 2.   
Please exercise this partition! 
 
</pre>
<!-- Author: Greg F. -->


<!-- Date: Fri May 25 16:00:59 2001 -->
<font color=red>Shift Summary: </font><pre>Bill Orejudos' shifts have come to an end....</pre> 
<!-- Author: TJL -->


<!-- Date: Fri May 25 16:04:18 2001 -->
<pre>We took level3 for 2 hours. 
  
I tested some database connection fixes.  
Jeff worked on a new code distribution routines.    
</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Boris/Jeff -->