<!-- Title: Day shift. Sat Oct 21, 2000 -->
<!-- Leader: Mel Shochet -->
<!-- Ace: Dan Cyr -->
<!-- Co1: Carla Pilcher -->
<!-- Co2: Rob Snihur -->
<!-- Co3: Stan Thompson (SOS) -->
<!-- Co4: Slawek Tkaczyk (SOS) -->
<!-- Co5: &nbsp; -->
<!-- Notes: -->
<pre>This page intentionally left blank as the powers that decide such important matters are eating donuts....

Young-Kee's donuts were indeed tasty!  The plan for the 
shift is to wait and hope!!</pre>


<!-- Date: Sat Oct 21 08:41:48 2000 -->
Inherited a cfg from eiko.  
 
I set+verified cot voltages (249 SL1-7 /349 SL8) 
 
Was running very smoothly - till i was checking how many L3 farms. Run Control has just died. Just in time, Jim walks in, performs the "Rejoin orphaned partition" and it works! Ended run gracefully at 127625 events, comm_0_bias(2,58) to investigate the rate mismatch when not running enough subfarms. Reformatter errors were seen, i will parse the log to find out the (multiple) guilty crates.  
<!-- Author: dan :: (run 102096) -->
<!-- Comment: Sat Oct 21 08;53;01 comment by...dan & jim -->
<pre>Only one guilty crate - FIB_SVX_04 (card FIB_19) this corresponds to VRB_30. 
Jim Patrick says this may not be an SVX problem : here is a snippet from the log 

Last Unit prob. worked on (all counts from 0):
  SCPU: 1
  VRB : 0
  LINK: 0
  MINI: 0

 Event Buffer dump (7759 until posi+2=7781)
 0x00000000  0xaaaaaaaa  0x55555555  0x98001262  0x00000080 0x00000000

 0x280c0000  0x415e3800  0xf2c20110  0x00431414  0x37643804  0x03500000

 0x0200e809  0x00000000  0x00401004  0x00012200  0x80000403  0x00001805

 0x00000000  0x00000000  0x2a20e3c4  0x001e83fc  0xa11e8203     
              
--------------------------------------
the first 6 words from this event buffer dump should be the last 6 of a non-svx VRB. 
Perhaps this is carryover from SCPU 0.</pre>
<!-- Comment: Mon Oct 23 09;36;06 comment by...SCN -->
Looks like a "c4" byte added to first channel header from
SVX Vrb:
0x2a20e3c4  0x001e83fc -byte Swap-> c4e3202a fc831e00
Remove erroneous c4 from 1st word-> e3202afc 831e0003

e320 is  e->SVX  3-> Bulkhead 3  2-> Phi 2 0-> Layer 0
Clock Counter (aka BC) 2a, Back End State fc, first
Chip ID 0x83, which is exactly (to the bit) what one should
expect.

Typically this happens when some ladder (not necessarily this
one) screws up the readout -- probably a vestige of the
remnant OBDV clock problem, now at a much lower rate.
<!-- Comment: Mon Oct 23 09;36;27 comment by...SCN -->
<pre>Looks like a "c4" byte added to first channel header from
SVX Vrb:
0x2a20e3c4  0x001e83fc -byte Swap-> c4e3202a fc831e00
Remove erroneous c4 from 1st word-> e3202afc 831e0003

e320 is  e->SVX  3-> Bulkhead 3  2-> Phi 2 0-> Layer 0
Clock Counter (aka BC) 2a, Back End State fc, first
Chip ID 0x83, which is exactly (to the bit) what one should
expect.

Typically this happens when some ladder (not necessarily this
one) screws up the readout -- probably a vestige of the
remnant OBDV clock problem, now at a much lower rate.</pre>
<!-- Date: Sat Oct 21 08:44:48 2000 -->
<pre>Jim Patrick is working on the COT readout.</pre>
<!-- Author: M. Shochet -->



<!-- Date: Sat Oct 21 09:17:07 2000 -->
<pre>Carla notices that the CMP mapping in the event display is still 
wrong when using cdfsoftb0, but correct when using development. 
We will use the development version if we don't have any problems 
when we start a run after Jim is finished.</pre>
<!-- Author: M. Shochet -->


<!-- Date: Sat Oct 21 09:45:43 2000 -->
<pre>CMP tripped.  Dan is investigating.</pre>
<!-- Author: M. Shochet -->
<!-- Comment: Sat Oct 21 09;53;31 comment by...dan -->
the 2 top center channels of CMP tripped (no beam) I successfully brought them up to operatingV. 


<!-- Date: Sat Oct 21 10:24:21 2000 -->
<pre>Look at COT busy assertion with limited Level 3 subfarms: 
First switched back to 1 subfarm and saw error after 12000 
events. 
Reprogrammed firmware in COT VRBs to older version 0A12 
before some changes in the busy logic in later versions. 
System ran for 25K events with no errors. 
Then unfortunately attempted to make all VRBs have same 
firmware. There was a problem reprogramming the one for 
the plug calorimeter east. It gave bus errors partway through 
programming the second flash bank. Although it passed 
the standalone test, there were some busy timeouts 
though not obviously attributable to this VRB. 
Anyway, a spare VRB was installed and programed and the 
system has now run for 25K events with one subfarm, 
and is still running.</pre>
<!-- Author: patrick :: (run 102101) -->
<!-- Comment: Sat Oct 21 11;04;52 comment by...dan -->
ended run successfully @ 103,000 !!! starting a run adding silicon back in. still one L3 subfarm. next run will test all L3 subfarms as Ivan says they are good to go


<!-- Date: Sat Oct 21 11:40:08 2000 -->
<pre>One L3 subfarm test 30k events. I presume reformatter errors from VRB_30 are present, as they are in the next run!</pre>
<!-- Author: dan :: (run 102102) -->


<!-- Date: Sat Oct 21 11:43:27 2000 -->
<pre>Mike Church called and reported the following: 
 
	The TeV RF anode power supply failed yesterday and was  
	repaired.  It later failed again.  The engineer is 
	now working on it.  Mike's plan is to start the next 
	shot setup Monday morning, with luminosity delivered 
	to us by the evening.  Until then (after the supply 
	is repaired), various TeV studies will be carried out. 
	During the day today, they will work with uncoalesced 
	beam.  Tonight they plan to run with 36 coalesced, high 
	intensity bunches.  This assumes, of course, that the 
	engineer is successful in repairing the power supply.</pre>
<!-- Author: M. Shochet -->


<!-- Date: Sat Oct 21 11:43:40 2000 -->
This run had a done timeout at 72k events in cot01 (slot12 no valid header word) - HRR worked! However the CO's started seeing some errors, check CO log for more info. Unknown to me, svt wasnt in the run config - next run i will add it. 
<!-- Author: dan :: (run 102103) -->


<!-- Date: Sat Oct 21 11:46:16 2000 -->
<table border=0><td><tr><td bgcolor=#e0e030>I think it would be VERY useful to have the "Replies & Ack" display window display the buttons in alphabetical order. This may seem like a minor(bells&whistle) thing but I think it is very important. Many mistakes could be avoided if this feature were implemented immediately. </td></tr></td></table>
<!-- Author: dan -->


<!-- Date: Sat Oct 21 11:59:35 2000 -->
<pre>Took 125k events with ALL L3 subfarms (30 L3 processor nodes!)  
 
Trigger comm_0_bias (2,58) 
L1 rate : 114hz 
readout : 95-100hz (throttled by the 10ms cot delay i think) 
 
Lots of reformatter errors in SCPU 1, VRB 0 LINK 0 (VRB_30)  
This the prev. mentioned FIB_SVX_04 (FIB_19).  
I wont continue to report these, as it's Jim Patrick's opinion that SVX_04 isnt at fault. 
</pre>
<!-- Author: dan :: (run 102103) -->
<!-- Comment: Sat Oct 21 12;48;19 comment by...patrick -->
<pre>As in some runs many events are being rejected due to reformatter
errors, at some point these should be looked at more.
Crate Id?: 0x00000000

 Error during scanning MINI structure!
 Last Unit prob. worked on (all counts from 0):
  SCPU: 1
  VRB : 0
  LINK: 0
  MINI: 0

 Event Buffer dump (7759 until posi+2=7781)
 0x00000000  0xaaaaaaaa  0x55555555  0x980023ee  0x00000080  0x00000000
 0x50080000  0x613c0100  0xe3090010  0x01431414  0x4be30e01  0x00a00000
 0x03007806  0x00000000  0x00401004  0xe0002600  0xf8003200  0x00001c04
 0x00000000  0x00000000  0x2b20e3c4  0x000683fc  0x2b068203


 NOTE: Event is rejected!

As noted above, the reformatter says it is working on the
SVX data, however, the first 6 words of this block
look like what the last 6 words should be from the previous
VRB, which reads the trigger. The 0/a/5 pattern is expected.
The "98" in the next word is the crate number, 0x98 corresponds
to b0l2de01, the last crate read in the trigger.
The next two words are end of data words.
Will send the message to Christoph and see if he has any insight.</pre>
<!-- Comment: Mon Oct 23 09;42;45 comment by...SCN -->
Bad Layer 0 explanation <a
href=/cgi/elog/elog.pl?nb=2000&action=view&page=190&anchor=08:41:48
target=_top>here</a>, though looks like not such a low rate...
<!-- Date: Sat Oct 21 12:14:50 2000 -->
Bookmarks changed and reloaded one too many times on b0dap16. ;-) Well actually killall java failed to kill some java processes. The machine became unbearably slow. I logged out, and back in. I got a gray stippled X screen for 2 min. Rebooted computer.  
<!-- Author: dan -->


<!-- Date: Sat Oct 21 12:31:16 2000 -->
<pre>CMP tripped again.  It was the same two channels.  Dan reset 
it.  If it trips again, the two channels will be put on
standby.</pre>
<!-- Author: M. Shochet -->
<!-- Comment: Sat Oct 21 12;39;11 comment by...M. Shochet -->
And yet again .... It is going to standby.


<!-- Date: Sat Oct 21 12:43:40 2000 -->
<pre>CMP and CMX tripped.  There is beam again in the machine! 
The muon systems are all being taken down to standby.</pre>
<!-- Author: M. Shochet -->
<!-- Comment: Sat Oct 21 14;51;16 comment by...M. Shochet -->
Steve Hahn sees on ACNET that the Tevatron has not been ramped
today.  The beam that has been in the machine has been at 150
GeV.  So perhaps they are still working on the RF system.
<!-- Date: Sat Oct 21 12:48:12 2000 -->
<pre>We started a run using Commissining_Photon_Plug_2.  Since 
the accelerator was off, we got 2 events in a half-hour. 
We will try this table again when there is beam in the machine.</pre>
<!-- Author: M. Shochet :: (run 102105) -->



<!-- Date: Sat Oct 21 13:24:02 2000 -->
<pre>With some beam in the machine, we tried again to run the plug 
photon trigger.  The rate was still too small.  We will go 
back to the zero_bias trigger to exercise the DAQ.</pre>
<!-- Author: M. Shochet :: (run 102107) -->


<!-- Date: Sat Oct 21 13:47:25 2000 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>We ran the event display on fcdfsgi2 on the great Plug_Photon 
run containing two real triggers (8 events, since Myron Mode 
was used).  Only 5 of the events were on the file (also seen 
in Carla's program): 
 
		event number		Myron Mode number 
		____________		_________________ 
 
		     1				0 
		     3				2 
		     4				3 
		     6				1 
		     8				3 
 
Why aren't events 2, 5, and 7 on the output file?</pre></td></tr></td></table>
<!-- Author: M. Shochet :: (run 102105) -->
<!-- Comment: Sat Oct 21 13;54;14 comment by...M. Shochet -->
PS, In the one correct-bunch event (event 6), the trigger did
do what it was supposed to do.
<!-- Comment: Sat Oct 21 13;57;31 comment by...M. Shochet -->
The two triggers came in approximately 15 minutes apart.  So all
of the missing events cannot be due to L3 processors that
weren't yet finished processing and for some reason weren't 
read out.
<!-- Comment: Sat Oct 21 16;52;45 comment by...patrick -->
There were reformatter (data corruption) errors of
the type mentioned in two of the above entries
for 3 events. Currently these events are always
rejected by Level 3.
<!-- Date: Sat Oct 21 13:47:33 2000 -->
<pre>Going back to comm_0_bias (2,58) :  
ccal04 gave a vision r/w error on slot20.  
i reset ccal04 and tried to recover run by taking ccal04 thru 
partition+config individually.  
Failed. and the attempt annoyed evb + l3. had to close partition + reset.  </pre>
<!-- Author: dan :: (run 102108) -->



<!-- Date: Sat Oct 21 13:56:59 2000 -->
<pre>Started running again.  
All "healthy" crates (part 0 reports 127 crates fwiw) the usual 113hzL1 + 92hz readout rate.  
~16% deadtime (all of which is readout) 
 
stopping run soon to calibrate silicon....</pre>
<!-- Author: dan :: (run 102109) -->
<!-- Comment: Sat Oct 21 14;11;42 comment by...dan -->
ended gracefully at 100k events

<!-- Date: Sat Oct 21 13:57:24 2000 -->
<pre>Even tough SVT is in the run, we get many TrigMon errros 
of the form: 
out-of-bounds exception occurred while accessing Bank SVTD, 1 
StorableBank::get_I4_element(): out-of-bounds exception 
offset = 0, index = 15, data_size = 15 
Finally, TrigMon crashed.</pre>
<!-- Author: Rob Snihur :: (run 102109) -->
<!-- Comment: Sat Oct 21 14;25;47 comment by...dan -->
i wonder if this might be related to the 800+mb that trigmon was consuming yesterday on b0dap19


<!-- Comment: Sat Oct 21 14;32;53 comment by...Rob Snihur -->
Jane Nachtman is not sure about this possible reason,
but both she and Matt say it's NOT related to the SVTD bank.
The SVTD bank msg is reporting exactly what TrigMon was designed for.  If TrigMon crashes again, the COs should grab any relevent
msgs and store them in the E-Log or email to Jane & Matt.


<!-- Date: Sat Oct 21 14:35:40 2000 -->
I am using b0l2de00 in partition 4 for some Level 2 tests.
<!-- Author: Jane -->



<!-- Date: Sat Oct 21 15:37:11 2000 -->
SVX Calibration : 102110 + 102111 using run config SVX_CALIB  
this uses the calib continuous and hevb. Note these files will be in sequential root as they use the hardware event builder.
<!-- Author: dan :: (run 102110) -->


<!-- Date: Sat Oct 21 15:43:30 2000 -->
<pre>Started a run with the l3 exe tag 64. Had a done timeout at 79k events.  
Experts will comment on the filter and the power trip which may have caused the done timeout.  
We had the usual reformatter errors, then almost all crates started giving errors such as : 
(MLE) b0imu00:Messenger:3:34:36 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0cot04:Messenger:3:34:30 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0xft02:Messenger:3:38:31 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0xft00:Messenger:3:34:31 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0cmx00:Messenger:3:34:35 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0xtrp00:Messenger:3:34:29 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0l1cl01:Messenger:3:34:30 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0cot00:Messenger:3:34:29 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0cmu00:Messenger:3:34:31 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0l1cl02:Messenger:3:34:27 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0pcal11:Messenger:3:34:32 PM->Runtime Error 1: See wrong event number (0) in L2 message 
(MLE) b0l1gl00:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0l1cl03:Messenger:3:34:28 PM->Bunchcounters in slot 4 and slot 8 disagree (at least) 
(MLE) b0cmu01:Messenger:3:34:31 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0xft04:Messenger:3:34:31 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0cmx01:Messenger:3:34:29 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0musc00:Messenger:3:34:29 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0cot15:Messenger:3:34:30 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0xft03:Messenger:3:34:32 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0tsi03:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0l1cl05:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0l1cl00:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0pcal08:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (0) in L2 message 
(MLE) b0ccal04:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (0) in L2 message 
(MLE) b0mutr00:Messenger:3:34:31 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0xft01:Messenger:3:34:30 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0cmp00:Messenger:3:34:30 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0imu01:Messenger:3:34:26 PM->Runtime Error 1: See wrong event number (1) in L2 message 
(MLE) b0l1cl04:Messenger:3:34:31 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0xft05:Messenger:3:34:28 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0cot05:Messenger:3:34:30 PM->Runtime Error 1: See wrong event number (9) in L2 message 
(MLE) b0svt06:Messenger:3:34:29 PM->Runtime Error 1: See wrong event number (0) in L2 message 
(MLE) b0cot11:Messenger:3:34:29 PM->Runtime Error 1: See wrong event number (0) in L2 message                     
 
-------------------------------- 
then more errors like : 
(MLE) b0cmp00:Messenger:3:34:30 PM->Runtime Error 5: No valid header for TDC in slot 15 
(MLE) b0l1cl03:Messenger:3:34:29 PM->Bunchcounters in slot 4 and slot 5 disagree (at least) 
(MLE) b0xft02:Messenger:3:38:31 PM->Runtime Error 2: Bunch counter mismatch in event 79506, mismatch count = 4 
(MLE) b0l1cl01:Messenger:3:34:30 PM->Bunchcounters in slot 4 and slot 11 disagree (at least) 
(MLE) b0l1cl02:Messenger:3:34:27 PM->Bunchcounters in slot 4 and slot 8 disagree (at least) 
(MLE) b0xft03:Messenger:3:34:32 PM->Runtime Error 2: Bunch counter mismatch in event 79506, mismatch count = 4 
******************************************** 
 
</pre>
<!-- Author: dan :: (run 102114) -->
<!-- Comment: Sat Oct 21 16;54;46 comment by...patrick -->
This is the classic symptom of a Master Clock glitch.
Next week we should have some better monitoring of the
clock.
<!-- Date: Sat Oct 21 15:50:14 2000 -->
<pre>CMX east 18 19 tripped FROM standby. resetting to standby. 
</pre>
<!-- Author: dan -->


<!-- Date: Sat Oct 21 15:59:00 2000 -->
<font color=red>Shift Summary: </font><pre>This shift crew sadly understands that it will not see any 
beam.</pre> 
<!-- Author: M. Shochet -->


<!-- Date: Sat Oct 21 15:59:51 2000 -->
<pre>Tested new Level-3 executable with COT global tracking.  Seems 
to work fine but HV not on for COT so no CdfTracks.  Tagset 64 
should become the default tagset to be used in Run Control.</pre>
<!-- Author: Kirsten :: (run 102114) -->


<!-- Date: Sat Oct 21 16:02:17 2000 -->
<font color=red>Shift Summary: </font><pre>This shift crew sadly understands it will not see any collisions!</pre> 
<!-- Author: M. Shochet -->


<!-- Date: Sat Oct 21 16:09:28 2000 -->
<pre>Level 3 may have died horribly. Trying to perform L3 cleanup - doesnt look promising... Alexi is taking over and may have to page L3 expert. 
</pre> <font color=red><cite>(Entry started on Sat Oct 21 16:09:26 2000)</cite></font>
<!-- Author: dan -->