<!-- Title: Owl shift. Mon Oct 16, 2000 -->
<!-- Leader: Phillip Koehn -->
<!-- Ace: Eiko Yu/Peter Wittich -->
<!-- Co1: Mako Shimojima -->
<!-- Co2: Ashkan Alavi -->
<!-- Co3: David Saltzberg(SOS) -->
<!-- Co4: &nbsp; -->
<!-- Co5: &nbsp; -->
<!-- Notes: -->
<pre>Shift Goals: 
	* Take cosmics with 2_jet_2 trigger 
	* Take calibrations 
	-> If we get quiet time before access, take CLC LED. 
 
</pre>


<!-- Date: Mon Oct 16 00:26:08 2000 -->
<pre><!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
b0cot02 gave a done timeout. Reset the crate and took another run again. 
<br>This time VRB_SVX_02 gave a busy timeout. 
<br>The error message in the green screen showed 
<p>---- 16 Oct 2000 00:23:30 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-BufferOverflow: Data for event 1725 from VRB in slot 12 would 
not fit in remaining buffer space. 
<br> Total buffer size = 51200 bytes, remaining buffer space = 18320 
bytes. 
<p>----- 16 Oct 2000 00:23:30 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbHeader: Dump of header words for event 1725 from VRB in 
slot 12: 
<br>0x6080000d 0x04104100 0x80080180 0x00000000 0x00000000 0x00000060 0x1ffc0000 
0x1ffc1ffc 
<p>----- 16 Oct 2000 00:23:31 CDT ----- b0eb19.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbReadFailed: Error reading VRB in slot 12 for event 1727. 
<br>The VRB had no event. 
<p>----- 16 Oct 2000 00:23:32 CDT ----- b0eb19.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbReadFailed: Error reading VRB in slot 13 for event 1727. 
<br>The VRB had no event. 
<br>  
</body> 
</html></pre>
<!-- Author: Eiko :: (run 101521) -->




<!-- Date: Mon Oct 16 00:40:25 2000 -->
<pre>b0ccal06 hangs during coldstart, Peter reset the crate. 
Try another run. 
 
Saw many crates red in the vxworks window. 
b0ccal03, b0ccal11 heap error... 
Reset all the red crates.  
</pre>
<!-- Author: Eiko :: (run 101522) -->
<!-- Comment: Mon Oct 16 00;54;07 comment by...Peter Wittich -->
All of the following crates appear to have internal heap inconsistencies all of a sudden.
<pre>
ccal:
=======
02,08,09,11,12,13,15
pcal:
=======
03,07,08,09,10
wcal:
=======
00,03,04,07
clc
=======
00 (as if there were other crates)
</pre>
The error message says, 'Hear error : 'Internal INconsistency' at xxxxx (count = 224444)'.  The hex address appears to be in the address space of the 'Messenger' process, according to the 'SP' column which one gets when one types "i" at the vxworks prompt from one of the affected crates.</p>

<p>Note that the CMX tripped twice at this time, too.  Coincidence or conspiracy?</p>

<p>Strangely enough, b0vrb03 also is read in the vxmonitor window.  Heap is not corrupt on that machine.  Let's ignore that for now.



<!-- Date: Mon Oct 16 01:02:41 2000 -->
<pre>Restart a run. vrb_svx_02 gave a busy timeout. 
Cleanup L3 farm and evb at ACE control winodw.</pre>
<!-- Author: Eiko :: (run 101523) -->


<!-- Date: Mon Oct 16 01:05:51 2000 -->
<pre>CMX keeps tripping. 
 
There is no CMX pager listed so we called the CMU pager. 
 
Instructed to put CMX on standby. 
 
Will try to bring it up periodically. 
 
We will continue to look for CMX pager number-- we are told 
there is one. 
 
</pre>
<!-- Author: Saltzberg  (SOS) :: (run 101533) -->


<!-- Date: Mon Oct 16 01:30:13 2000 -->
 <!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
  
<p>vrb_svx_02 gave a busy timeout again! 
<p>The green error message screen says: 
<p>----- 16 Oct 2000 01:27:11 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-BufferOverflow: Data for event 4779 from VRB in slot 12 would 
not fit in remaining buffer space. 
<br> Total buffer size = 51200 bytes, remaining buffer space = 10184 
bytes. 
<p>----- 16 Oct 2000 01:27:11 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbHeader: Dump of header words for event 4779 from VRB in 
slot 12: 
<br>0x8020000b 0x04104100 0x80080180 0x00000000 0x00000000 0x00001ffc 0x1ffc0000 
0x1ffc1ffc 
<p>----- 16 Oct 2000 01:27:11 CDT ----- b0eb19.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-BufferOverflow: Data for event 4779 from VRB in slot 13 would 
not fit in remaining buffer space. 
<br> Total buffer size = 51200 bytes, remaining buffer space = 18376 
bytes. 
<p>----- 16 Oct 2000 01:27:11 CDT ----- b0eb19.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbHeader: Dump of header words for event 4779 from VRB in 
slot 13: 
<br>0xa020000b 0x04104100 0x82080180 0x00000000 0x00000000 0x00001ffc 0x1ffc1ffc 
0x1ffc1ffc 
<p>----- 16 Oct 2000 01:29:11 CDT ----- b0eb10.fnal.gov ----- tsmDisp 
<br>SM-P0-W-Loitering: Event 4780 has had a load pending for a long time. 
Pending SCPU mask = 0x00000204. 
<br>  
</body> 
</html>  
<!-- Author: Eiko :: (run 101524) -->
<!-- Comment: Mon Oct 16 01;34;36 comment by...Eiko -->
Tried HRR, still got busy timeout!



<!-- Comment: Mon Oct 16 01;49;11 comment by...Eiko -->
run ended after 4786 events.
<!-- Date: Mon Oct 16 01:43:53 2000 -->
<pre>Actually we figured out CMX was supposed to be on standby. 
</pre>
<!-- Author: saltzberg (SOS) -->



<!-- Date: Mon Oct 16 02:06:28 2000 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.14-1.3.0f2 i686) [Netscape]"> 
</head> 
<body> 
run 101525 ended at event=3897. 
<br>Lots of crates gave done timeout. 
<p>Here is the list 
<br>ccal00, 02,05,06,07,09,12,13,14 
<br>wcal00,01,02,03,04,05,06,07 
<br>pcal00,01,02,03,04,05,06,07,08,09 
<br>cot01,02,03,06,09,10,11,12,13,16,18,19 
<br>imu00,01 
<br>cmu00,01 
<br>cmp00 
<br>xft_xtrp_00 
<br>xft00,02,03,04,05 
<br>clc00 
<br>svt_tracking_02 
<br>svt_fitter_06 
<br>l1cal00,01,03,04,05 
<br>l1global00,scalers_00, 
<br>l2cal00,01,02,03,04 
</body> 
</html> 
  
<!-- Author: Eiko :: (run 101525) -->


<!-- Date: Mon Oct 16 02:09:06 2000 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
Related to the last comment: 
<br>The green error message window says 
<br>---- 16 Oct 2000 02:00:00 CDT ----- b0eb12.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-TracerEventId: Event 3897, crate 170, channel 4 has Tracer 
event ID of 5 instead of 9. 
<br>Tracer word = 0x5555aaaa (I think). 
<br>Channel data = 0x14141414 0x14148282 0x14141414 0x82828282 0x82828282 
0x82828282 0x82828282 0x14148282 ... 
<br>... 0x4410f339 0x021c009c 0x00000000 0x00000000 0xaaaa0000 0x5555aaaa 
0x00855555 0x0000e809 
<p>----- 16 Oct 2000 02:00:01 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbReadFailed: Error reading VRB in slot 10 for event 3897. 
<br>The VRB had no event. 
<p>----- 16 Oct 2000 02:00:01 CDT ----- b0eb19.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbReadFailed: Error reading VRB in slot 12 for event 3897. 
<br>The VRB had no event. 
<br>  
</body> 
</html>
<!-- Author: Eiko :: (run 101525) -->


<!-- Date: Mon Oct 16 02:23:29 2000 -->
<pre>Reset master clock module, as all the error lights were on.</pre>
<!-- Author: Peter Wittich -->


<!-- Date: Mon Oct 16 02:25:28 2000 -->
<pre>Take a QIE calibration run with all the calorimeter crates. 
 
b0pcal00 slot14, QIE07 gave no fitting error. </pre>
<!-- Author: Eiko :: (run 101527) -->


<!-- Date: Mon Oct 16 02:49:30 2000 -->
<pre>Peter W. notices that there are errors with masterclock module, and 
resets it.</pre>
<!-- Author: Phillip Koehn -->


<!-- Date: Mon Oct 16 02:54:21 2000 -->
<pre>Bill Noe calls, from home, to report an auto page of flammable gas alarm. 
 
We have no alarms in Control room. Dave checks IFIX pages. 
Observes Rack Differential Pressure alarms in storage shed. Expert paged, Pat Lukens returns call. Pat says they will deal 
with it in the morning. </pre>
<!-- Author: P. Koehn -->


<!-- Date: Mon Oct 16 03:12:17 2000 -->
<pre>Masterclock module has errors. I reset module.</pre>
<!-- Author: Phillip Koehn -->


<!-- Date: Mon Oct 16 03:13:28 2000 -->
<pre>Bill Noe's pager keeps going off for this alarm. 
He can't turn off his pager or he would miss a new alarm. 
He is coming in to investigate.</pre>
<!-- Author: saltzberg (sos) -->


<!-- Date: Mon Oct 16 03:24:00 2000 -->
<pre>LED calibration: run number = 101528 
XEF calibration: run number = 101529 
COT TDC calibration: run number= 101530 
Muon calibration without b0cmx01: run number= 101531 
COs are looking at the data.</pre>
<!-- Author: Eiko -->


<!-- Date: Mon Oct 16 03:48:25 2000 -->
<pre>Masterclock has errors. Reset.</pre>
<!-- Author: P.Koehn :: (run 101533) -->


<!-- Date: Mon Oct 16 03:50:24 2000 -->
SVX power trip. L0, 2E, number 38.  Page SVX pager.  Upon instruction, we turn it back on.
<!-- Author: Peter Wittich -->


<!-- Date: Mon Oct 16 04:32:18 2000 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
Run 101534: 
<br>vrb_svx_02 gave a busy timeout at event 5350. 
<p>The green error message window says 
<p>----- 16 Oct 2000 04:12:42 CDT ----- b0eb14.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-BadChannelCounts: Event 5340 has the sum of channel byte 
counts exceeding the total byte count, VRB slot 10. 
<p>----- 16 Oct 2000 04:12:42 CDT ----- b0eb14.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbHeader: Dump of header words for event 5340 from VRB in 
slot 10: 
<br>0x00000000 0x00000c88 0x01980200 0x00dc0000 0x01040108 0x01f800d4 0x00000055 
0x103c003b 
<p>----- 16 Oct 2000 04:12:42 CDT ----- b0eb10.fnal.gov ----- tsmDisp 
<br>SM-P0-I-Suspending: Due to errors, data-taking in this partition is 
suspended until the next Recover transition. 
<p>Tried HRR, at event=6824, busy timeout again!! 
<p>----- 16 Oct 2000 04:18:23 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-BufferOverflow: Data for event 6814 from VRB in slot 12 would 
not fit in remaining buffer space. 
<br> Total buffer size = 51200 bytes, remaining buffer space = 10184 
bytes. 
<p>----- 16 Oct 2000 04:18:23 CDT ----- b0eb20.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-VrbHeader: Dump of header words for event 6814 from VRB in 
slot 12: 
<br>0x40c0000e 0x04104100 0x80000100 0x00000000 0x00000000 0x00001ffc 0x1ffc0000 
0x00340062 
<p>----- 16 Oct 2000 04:18:23 CDT ----- b0eb19.fnal.gov ----- tRec_0 
<br>SCPU-P0-E-BufferOverflow: Data for event 6814 from VRB in slot 13 would 
not fit in remaining buffer space. 
<br> Total buffer size = 51200 bytes, remaining buffer space = 24976 
bytes. 
<p>Another busy timeout after HRR. 
<br> Ended the run at event=6835 
<br>  
<br>  
<br>  
<br>  
</body> 
</html>
<!-- Author: Eiko :: (run 101534) -->


<!-- Date: Mon Oct 16 04:33:20 2000 -->
<p>Mon Oct 16 04:30:20  GAS... "Rack Differential Pressure" went into alarm. Auto Dialer called me at 02:13AM. I called x-2080 to determine what the alarm was. PTL was called and he said it could wait until later but the auto dialer had other ideas. It paged Dean Beckner and called me 2 more times. I came in to find Dean looking into the problem. We found the rack ventilation intake screen plugged with dirt. After cleaning the screen the alarm reset ok. Will deal with the auto dialer later today. - Bill Noe  
 
</p>
<!-- Author: (Copied from Cryo & Gas Log by Bill Noe) -->


<!-- Date: Mon Oct 16 05:03:33 2000 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
Tried to resolve the busy timeout again! 
<br>1. cleanup EVB and L3 proxy. 
<br>2. run the script at b0l3pcom1:  bin/setVrbDelays.exp 1000 1 , 
so that increase the time of EVB 
<br>    waits with 1s. 
<p>Run 101535 had done timeouts from many crates due to the mismatch of 
master clock with 
<br>beam clock. 
<table border=0><td><tr><td bgcolor=#e0e030> Reset the crate b0clk00 by accident. </td></tr></td></table>
</body> 
</html>
<!-- Author: Eiko :: (run 101535) -->


<!-- Date: Mon Oct 16 05:13:40 2000 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
Run 101536 gave a evb transition error at the activate state. 
<br>The green message screen shows 
<p>****************************************************************************** 
<br>SCPU-P0-E-CantResetVrb: Reset of VRB in slot 12 failed. 
<br>VRB_NOT: The module at the given VME address has the wrong module ID. 
<p>----- 16 Oct 2000 05:08:19 CDT ----- b0eb10.fnal.gov ----- tsmHot 
<br>SM-P0-E-ScpuUnresponsive: One or more SCPUs failed to respond to Define 
Partition. Check SCRAMNet. 
<br>******************************************************************************* 
<br>  
</body> 
</html>
<!-- Author: Eiko :: (run 101536) -->
<!-- Comment: Mon Oct 16 05;26;02 comment by...Eiko -->
Cleanup evb and L3 proxy.
Failed SCRAMNet check at b0eb16. Reset the crate b0eb16.
Now check result shows o.k.

 


<!-- Date: Mon Oct 16 05:35:31 2000 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.16-3 i686) [Netscape]"> 
</head> 
<body> 
Run 101537: 
<br>busy timeout from vrb_svx_02 again! at event=410. This time HRR doesn't 
help much! 
</body> 
</html>
<!-- Author: Eiko :: (run 101537) -->


<!-- Date: Mon Oct 16 05:46:21 2000 -->
<pre>Masterclock errors, reset module.</pre>
<!-- Author: P.Koehn :: (run 101537) -->


<!-- Date: Mon Oct 16 06:23:40 2000 -->
<pre>Jay Dittmann requests all XFT Finder Crates,  
and all available COT crates. We give them to him 
as we are preparing to do CLC calibrations during 
the quiet time before the access.</pre>
<!-- Author: Phillip Koehn -->
<!-- Comment: Mon Oct 16 06;32;35 comment by...P. Koehn -->
<pre>Jay will be performing Finder board input capture tests.</pre>


<!-- Date: Mon Oct 16 06:33:27 2000 -->
<pre>I am starting to use the XFT Finder crates and the COT 
crates for tests and to prepare for this morning's access.</pre>
<!-- Author: Jay Dittmann -->


<!-- Date: Mon Oct 16 06:35:52 2000 -->
<pre>Cleanup EVB and L3 Proxy. b0eb23~25 had to be reset by pressing  
the bottom on the CPU.  
 Started a run.  
 Run number= 101538 
 Ended the run at event=6228 to do CLC calibration</pre>
<!-- Author: Eiko :: (run 101538) -->


<!-- Date: Mon Oct 16 06:46:20 2000 -->
<p>Running CLC LED calibration.  The file name is ar018ca4.0001.cal.  this is our second attempt - in the first one, it appeared that there was no ack from the CLC crate at some point, but I am not sure at which point, as there is too much text being written to the screen to be understood. 
<p> 
It would be useful if  
<li>the documentation for this calibration was linked to the ACE calibration page 
<li>the documentation mentioned how long this calibration takes, so we can tell if the run is progressing normally. 
<p> 
Run appears to end successfully. 
</p>
<!-- Author: Peter Wittich :: (run 101540) -->
<!-- Comment: Mon Oct 16 06;56;24 comment by...Peter Wittich -->
<p>The run does not appear to be in either the online production or development data base as of now, though I am not sure I am checking it correctly.</p>

<!-- Comment: Mon Oct 16 07;01;40 comment by...Peter Wittich -->
<p>Mea culpa, it is in the online development data base.</p>


<!-- Date: Mon Oct 16 07:06:15 2000 -->
<pre>Dee arrives at 7am to prepare for the access.</pre>
<!-- Author: Phillip Koehn -->


<!-- Date: Mon Oct 16 07:20:07 2000 -->
<pre>The result of COT calibration run 101530 was compared to run 
1015489 (the previous calibration run). The only significant  
difference is the offset rms in crate 3. The data for slots 1-6 seems to be missing.  
Are there a set of reference plots somewhere? We couldn't find any in the CDF folders.</pre>
<!-- Author: Ashkan -->


<!-- Date: Mon Oct 16 07:54:07 2000 -->
<pre>* Reset masterclock module. Error lights on - clock glitches. >4 occurances. 
 
* Calibrations taken: 
  - QIE 
  - CAL Led  
  - CAL XENON  
  - COT  
  - MUON (CMX01 not included) 
 
* SVX power trip: L0, 2E, number 38. Page SVX pager.  
  Upon instruction, we turn it back on 
   
* Jay Dittmann requests all XFT FINDER CRATES and all available COT CRATES 
  for input capture studies. 
   
* Dee gets in at 7am to get access rolling. 
 
* Taking cosmic ray data with: 
  - 2_jet_2_TEST[1,47] trigger @ 2hz, early myron mode 
  - ces/cpr off, cot standby, Muon system standby 
  - all healthy crates used, including COT 
  - run summary(101519-101540): 
Run     Trigger        		  L1_A    Comment 
	 
101519  2_jet_2_test[1,47] @2hz        1744     	  
101520      "           "         2436     	  
101521      "           "          548     	  
101522      "           "            0   Heap Error in Cal crates 	  
101523      "           "          276	 vrb_svx_02 busy timeout 
101524      "           "         4812	  
101525      "           "         4003 
101526	    ???????? 
101527	 QIE       CALIBRATIONS 
101528	 CAL LED   CALIBRATIONS 
101529	 CAL XENON CALIBRATIONS 
101530	 COT TDC   CALIBRATIONS 
101531	 MUON      CALIBRATIONS 
101532  2_jet_2_test[1,47]          0    	  
101533       "                      0  
101534       "                    6840  
101535       "                    4765 	vrb done timeouts 
101536       "                      0   evb transition error at the activate state 
101537       "                     424  
101538       "                    6228  ended run to do CLC calibrations 
101539	1st attempt at CLC LED    CALIBRATIONS 
101540	CLC LED    CALIBRATIONS 
101541       "                    in progress  NO XFT LINKER CRATES, NO COT CRATES</pre>
<!-- Author: Phillip Koehn :: (run 101541) -->
<!-- Comment: Mon Oct 16 07;54;44 comment by...P. Koehn -->
This is the end of shift summary.


<!-- Comment: Tue Oct 17 00;49;37 comment by...P. Koehn -->
Run 101541 is 2_jet_2 trigger 
<!-- Date: Mon Oct 16 07:58:59 2000 -->
<font color=red>Shift Summary: </font><pre>See entry above.</pre> 
<!-- Author: P.Koehn -->