<!-- Title: Owl shift. Fri Sep 22, 2000 -->
<!-- Leader: Bob Wagner (ANL) -->
<!-- Ace: Ben Kilminster -->
<!-- Co1: Yuji Takeuchi -->
<!-- Co2: Ivan Furic -->
<!-- Co3: &nbsp; -->
<!-- Co4: &nbsp; -->
<!-- Co5: &nbsp; -->
<!-- Notes: -->
<pre>Our instructions for the shift listed on the white board are to 
take cosmic ray trigger data.  There is a carry over problem with 
the Consumer Server Logger from the evening shift.  Ben is 
dealing with the problem now.  He notes that some code was 
recently committed and appears to have introduced a bug into the 
CSL.</pre>


<!-- Date: Fri Sep 22 01:02:30 2000 -->
<pre>CSL bugs appear to be introduced from using new disk with  
different file protections. Problem is fixed and a slightly different CSL startup script is being used.  
 
CSL corrected itself without needing to be restarted, but  
restarted it anyway just to be sure.</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 01:20:21 2000 -->
<pre>Did QIE Calibration. Looked at b0scope01. COnfirmed that there  
was no beam around. So we did CLC QIE Calibration. 
 
Errors:  
 
Fail 25: Self calib fail on three crates: 
	b0ccal14 
	b0wcal06 
	b0pcal02 
 
 
</pre>
<!-- Author: Ben -->
<!-- Comment: Fri Sep 22 02;18;22 comment by...Ben -->
Run 98955  (QIE)

<!-- Comment: Fri Sep 22 02;19;17 comment by...Ben -->
WE didn't actually load the calibration onto the CLC.
To do this we would have had to kill the process on the crate,
which we didn't do.
<!-- Date: Fri Sep 22 01:20:38 2000 -->
<pre>Did LED CAlibration.</pre>
<!-- Author: Ben -->
<!-- Comment: Fri Sep 22 02;20;44 comment by...Ben -->
Run 98956 (LED)


<!-- Date: Fri Sep 22 02:21:28 2000 -->
<pre>Did XEF Calibration: RUN 98957</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 02:25:07 2000 -->
<pre>Did COT Calibration. 
One problem was that the Run configuration COTTEST_BB didn't have  
Run Number checked off. 
We checked this off, and save it to configuration COTTEST_BB. 
 
Run NUMBER 98958 (COT).</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 02:52:30 2000 -->
<pre>Run 98959: 
Busy Condition in XPT from some PCAL crates. 
 
We catch a VME  error on event builder: 
 
ALIGNMENT OR VME BUS ERROR FOR EVENT 16556 on b0eb16, slot 12. 
This corresponds to one of the folling sources: 
PCAL 6->11. 
 
Of these, 7,8,10,11 all have shower max banks. 
 
Perhaps it is a good idea to reboot these crates to verify  
Arnd's changes have been loaded. 
 
Reboot PCAL_6->11</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 03:08:23 2000 -->
<pre>Some Run control state manager errors that keep popping up are:  
 
java sql exception: Must be logged onto server. 
Error entering crateset RUN_98960_CLC link to BankSet to 
RUN_98960_CLC_00_FDA_1 in rundb. 
 
This error appears for a list of most, if not all, crates. 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 03:10:12 2000 -->
<pre>Run 98960:  Run dies after 31,738 events after a RXPT Timeout, 
	BUSY Condition.  
	Event builder message logger identifies  
	eb16, slot 12 as the culprit again. 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 03:11:44 2000 -->
<pre>Bunch counter errors are coming out from:  
 
b0l2cl01, slot 4 disagrees with everybody. 
 
b0clc00 Nuch counter errors with slot 3. 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 03:38:09 2000 -->
<pre>Run Control was in some werid state where nothing would 
respond to the partition command, It ended up being an error: 
transition failure for everything except the CSL and the trigger  
supervisors. 
RC eventually froze. Restarted Run Control.</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 04:23:28 2000 -->
<pre>Run 98962: 
Rate is only 1 Hz. 
We check with level 3. All event builder nodes are on line 
and have no memory problems. Not Level 3 or CSL either. 
 
Is XTRP a problem ? We vaguely remember XTRP being mentioned as 
being slow, so we remove it.</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 04:31:00 2000 -->
<pre>Run 98963: 1 Event. 
XTP Errors: Done Timeout from CES_TEST_00. 
 
Only thing weird is that b0tsi03 is showing up red on Vxworks 
display. We are not actually booking tsi 3, but maybe it is the problem.   Before rebooting, we see by logging in that it says: 
Internal Inconsistency Error. 
 
Reboot b0tsi03. 
 
 </pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 04:37:29 2000 -->
<pre>Run 98964: 1 event. 
Same Error: XPT CES_TEST_00 DOne TImeout 
 
For the fun of it,  
remove all XFT, 
Since we recently added some in, and maybe it is the problem. 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 04:38:08 2000 -->
<pre>Run 98965:  5695 Events. 
RXPT Timeout. 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 04:46:19 2000 -->
<pre>Run 98966: 13,965 Events. 
Event rate: 40 Hz. (three level 3 sub-farms running) 
CSL rate ~ 3.5 MB/s. 
 
Premature Death: 
 
VRB ReadFailed: b0eb14, Error reading VRB in slot 10 for event 13936. 
VME_Bus_ERROR: ALignment or VME bus error. 
 
Attempt  
HRR:  
b0tsi00 failed to return and is red on receive ack display. 
 
Results in RXPT Timeout. 
 
ABT+RST 
   </pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 04:55:05 2000 -->
<pre>Run 98967:  Fails on ACTIVATE Transition. 
EVB failed to respond, and has apparently died. 
b0eb19: RIng broken (no carrier). 
 
Ivan goes to push the reset button on b0eb19. 
WE hit the new ->Cleanup EVB>- button o the ACE display to  
reboot all the event builders to get them in sync again. 
 
try, TRy again ...</pre>
<!-- Author: Ben -->
<!-- Comment: Fri Sep 22 04;59;54 comment by...Ben -->
WE should have reset eb 19 after the previous run.

<!-- Date: Fri Sep 22 04:59:18 2000 -->
<pre>The Tevatron has been ramping up to 980 GeV.  Also injecting some 
beam into the Tevatron.</pre>
<!-- Author: Bob Wagner (ANL) -->



<!-- Date: Fri Sep 22 05:37:24 2000 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2000&action=view&page=-167&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2000&action=view&page=-167  width='250' align='top'></td><td>In Run:98966 CEMD West Phi Tower #0 went to very high occupancy.  However, it disappeared in the following run.</a></td></tr></table>
<!-- Author: Yuji -->


<!-- Date: Fri Sep 22 05:40:44 2000 -->
<pre>Parallelized coldstart included in default l3 run control.</pre>
<!-- Author: Ivan -->


<!-- Date: Fri Sep 22 05:51:14 2000 -->
<pre>Run 98968: 13,965 events. 
 
NOTE: These reformatter errors are seen:  (not fatal) 
1) SCPU 3, Link 3, b0eb13, slot 10, GVRB 6, mask 462, TracerVRBconnection 6 --> COT_6 
1) SCPU 3, Link 4, b0eb13, slot 10, GVRB 6, mask 462, TracerVRBconnection 7 --> COT_7 
1) SCPU 2, Link 2, b0eb14, slot 10, GVRB 7, mask 910, TracerVRBconnection 3 --> COT_13 
1) SCPU 2, Link 0, b0eb14, slot 10, GVRB 7, mask 910, TracerVRBconnection 1 --> COT_11 
1) SCPU 2, Link 1, b0eb14, slot 10, GVRB 7, mask 910, TracerVRBconnection 2 --> COT_12 
1) SCPU 1, Link 1, b0eb11, slot 10, GVRB 8, mask 255, TracerVRBconnection 1 --> IMU_1 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 05:54:28 2000 -->
<pre>Run 98968: Events 13965: 
 
Death: RXPT Timeout: Busy condition in XPT from crates: 
			COT_E_18 
 
On Event builder message window, we see from b0eb14:  
Error reading VRB in slot 10 for event 19121. 
VRB_BUs_error: ALignment or VME bus error. 
Run continues for 2800 events, so this may not have been fatal. 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 06:01:20 2000 -->
<pre>RC crashes in next run attempt: all windows are unresponsive. 
Slows down all operations on PC, so we do:  
killall -9 java. 
 
Restart RC. Do rejoin orphaned partition. WORKS !!! 
WE get back control of run 98968. 
 
Unfortunately, new error handler doesn't seem to have joined okay. 
It is unresponsive on the Partition transition. 
We kill all the error handler processes running, and start  
a new one. This works okay. 
 
TRy new run. EVB doesn't acknowledge. 
b0eb19: has ring broken again. Ivan checks and this makes  
sense since eb19 is bi-passed.  
 
 
 
</pre>
<!-- Author: Ben -->
<!-- Comment: Fri Sep 22 06;12;25 comment by...Ben -->
We clean up the EVB. And try again ...


<!-- Date: Fri Sep 22 06:20:29 2000 -->
<pre>Run 98970:  
Events: 4028 
Death: eb13 VRBRead failed in slot 10 for event 3999. 
BTW, eb13 is handling COT. </pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 06:59:55 2000 -->
<pre>Run: 98971 
Events: 14,298 events. 
TRy changing COT control voltage setting from 350 mV to 500 mV 
to see if this discriminates better against noise. 
 
Death:  eb13 goes off-line scramnet: 
eb13 gives error:  VME bus error. 
 
WE clean up EVBs. 
 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 07:25:32 2000 -->
<pre>Run 98972: 
Events: 9,179 
Death:  eb13 went offline-s due to VME bus error. 
 
Since eb13 serves COT, we decide to remove all COT.  
Because we are getting tired of writing eb13 in this log. 
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 07:43:17 2000 -->
<pre>Run 98973: 
NO COT 
Rate jumps to 180 events/ sec. 
CSL logs 17 MB/s right before it dies. 
 
Death:  RXPT Timeout due to busy condition from PCAL crates. 
Check CSL display and notice that receiver queue is empty.  
All events messages are in the distributor queue.  
 
After 10 minutes, CSL feature automatically cleans up distributor  
queue.  
 
So CSL didn't have to be rebooted. 
We lower the number of level 3 sub-farms to see if the high rate: 
17 MB/s was the problem. 
  
 
  
</pre><font color=red><cite>(Entry started on Fri Sep 22 07:43:15 2000)</cite></font>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 07:45:30 2000 -->
<pre>Run 98974: 
4 level 3 sub-farms. 
2450 Events 
Death: RXPT Error: PCAL crates. 
CSL dies again with 47 messages in distributor queue, 2 in  
conssend, and 1 in the distributor process. 
 
Restart CSL.</pre><font color=red><cite>(Entry started on Fri Sep 22 07:45:28 2000)</cite></font>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 08:01:57 2000 -->
<pre>Run 98975: 
DIes from RXPT error:  
This time it is a DONE Timeout from CES_TEST_00, and CCAL_WEST04. 
 
We aren't including CES_TEST_00. So we don't know why we keep 
getting errors related to this. Jim says crate is physically  
located at Argon, so we shouldn't be using it.  However, it is  
not explicitly included in the configuration file. 
 
 
(NO CSL errors) 
 
Try Halt-Recover-Run. 
Result: Transition failed for /daq/tsi/ts. 
 
b0tsi00 is found to be bad. It is red on all displays that care. 
Upon closer inspection, it has 2 susplended processes:  
tpn_0. 
 
tpn_0 is the process that runs the partition.  
 
As Ivan and the Highlander say, <<There can be only one>>. 
 
We rebooted the trigger supervisor.  
Sorry Jay and Ted.  
 
 
 </pre><font color=red><cite>(Entry started on Fri Sep 22 08:01:55 2000)</cite></font>
<!-- Author: Ben -->
<!-- Comment: Fri Sep 22 08;03;20 comment by...Ben -->
Try again: 

As Ivan and the Highlander say:  There can be only one.


<!-- Date: Fri Sep 22 08:04:44 2000 -->
<font color=red>Shift Summary: </font>Not really long runs during the night but identified many error 
conditions. 
<!-- Author: Bob Wagner -->


<!-- Date: Fri Sep 22 08:37:40 2000 -->
<pre>Run 98976: 
Events : 42,000 events.  
 
We started this just before we left. 
 
Rate: 80 Hz  (4 level 3 sub-farms)  
CSL Rate:  7 MB/sec 
 
 
</pre><font color=red><cite>(Entry started on Fri Sep 22 08:37:38 2000)</cite></font>
<!-- Author: Ben -->


<!-- Date: Fri Sep 22 08:38:13 2000 -->
<pre>Run 98976: 
 
Still going strong at 118,000 events ...</pre><font color=red><cite>(Entry started on Fri Sep 22 08:38:12 2000)</cite></font>
<!-- Author: Ben -->