<!-- Title: 14:33:37  Mon Jan  6 2003 -->
<!-- Crewchief: Yale University -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New year, new page!</pre>


<!-- Date: Mon Jan 6 15:13:08 2003 -->
<pre>Waiting for resources so we start simply. 
----------------------------------------- 
1) P6 with v61, calib continous, interval 0 = 60, 
   Auto L2 Accept. NO Readout.  All Silicon VRB crates. 
   No L1 DONE timeouts. 
 
2) Steve adds simulated Silicon Readout.   
   Emulated data Event size = 15 kBytes. 
   No L1 DONE timeouts. 
 
</pre>
<!-- Author: Greg, Steve, Michael -->


<!-- Date: Mon Jan 6 15:44:49 2003 -->
<pre>We make one change: auto L2A --&gt; auto L2R. 
Still no problems with L1-Done. 
 
Next we include the Trigger: run number 156159. 
Table = cosmic test L2 inclusive 3 190 318. 
 
We wanted to use hevb but can not if fibs are not 
included and have Steve create the data. 
We tried without hevb and got an SRC error. 
 
We decide to switch in svxfib02 and include hevb. 
Run 156162. 
 
</pre>
<!-- Author: Michael, Greg, Steve -->
<!-- Comment: Mon Jan  6 15;47;53 comment by...Michael -->
<pre>The SRC error reported above was "L2A without available
scan buffer".  We did not try to reproduce this error.
It was horrible and opened a million orange windows.</pre>


<!-- Date: Mon Jan 6 16:22:32 2003 -->
<pre>Run 156167.  Still cosmic table with hevb and one silicon 
ladder.  Changed prescales to lower L1A input rate to  
about 9kHz.  No troubles with L1-Done.  Run ends before 
we can check for ROL on scope due bad vrb (eb12-slot 10). 
 
</pre>
<!-- Author: Michael, Greg and Steve -->


<!-- Date: Mon Jan 6 16:28:08 2003 -->
<pre>Actually we did not have to end the run, just reset vrb. 
 
We check ROL on tracer upstairs.  Greg does a special 
alpha loading (and he will have to explain).</pre>
<!-- Author: Michael -->
<!-- Comment: Mon Jan  6 16;44;51 comment by...Michael, Greg and Steve -->
<pre>We verified that ROL are changing; run 152167.

We are using reload_alpha(exe=l2runexe-2). This means ROL= 0 or 7
alternating.  The pattern observed is not stable as we expect.

~9 million L1A with no L1-Done trouble.  We drop silicon to
see if there is any change in the ROL pattern on the tracer.</pre>

<!-- Comment: Mon Jan  6 16;45;50 comment by...michael -->
<pre>the run number is 156167.</pre>


<!-- Comment: Mon Jan  6 19;05;46 comment by...Michael -->
<pre>Peter did check and there are no ROL problems offline
(expectation vs. reality).</pre>
<!-- Date: Mon Jan 6 17:04:57 2003 -->
<pre>156173 was a run without silicon; we decide for now 
that the ROL looks fine; sent email to Peter in hopes 
that he can run his offline job. 
 
We'll try again now with silicon and then we will go 
and eat and let others work for a while and resume this 
evening.</pre>
<!-- Author: Michael, Greg and Steve -->


<!-- Date: Mon Jan 6 17:17:55 2003 -->
<pre> 
Run 156177 is with silicon, hevb and cosmic table we 
raise rates to 11.5kHz L1A and 200Hz L2A and have no 
troubles with L1-Done. 
 
Note triggers here are basically given by `randomized' 
BC's (prescales not divisible by 2 or 3). 
</pre>
<!-- Author: Mi chael, Greg, Steve -->



<!-- Date: Mon Jan 6 19:19:22 2003 -->
<pre>156189, same as before with svxfib02 and same cosmic 
table.  Still no L1-Done. 
 
Steve points out that I reseated TS for P5 and P6 by 
virtue of checking the PROM1 version labels. 
 
Could this have made the problem go away. 
 
We plan to move now to partition 1 while we wait 
for beam in the machine (studies) with which we 
could go back and try P6 again with the same physics 
trigger table that Peter, Greg and Steve used when 
they were able to reproducibly get L1-Done timeouts. 
</pre>
<!-- Author: Michael -->


<!-- Date: Mon Jan 6 19:47:55 2003 -->
<pre>Before moving to P1 we use P6 with l2 torture 4 209 318 
(not ROL): run number 156191.  Again no L1-done troubles. 
 
We cranked the rate up to 10kHz for L1 and 300Hz for L2. 
 
Now we switch to P1. 
 
</pre>
<!-- Author: Michael, Greg and Steve -->


<!-- Date: Mon Jan 6 20:15:49 2003 -->
<pre>Run 156192; P1 with L2 torture 4 209 318. 
 
We do see L1-Done timeouts. No ROL involved and 
old (v58 prom1) firmware.  11kHz L1A and 300Hz L2A. 
 
We get L1-Done timeout about once every 10 minutes. 
 
</pre>
<!-- Author: Steve, Greg and Michael -->
<!-- Comment: Mon Jan  6 20;24;53 comment by...Greg, Steve, Michael -->
<pre>we got two L1-dones and gave up

we couldn't wait any longer

we drop the ladder so that we increase the L1A rate</pre>


<!-- Date: Mon Jan 6 20:41:05 2003 -->
<pre>Removed ladder. Turned OFF ladder. 
Removed Silicon Emergency brake in scaler crate. 
-------------------------------------------------- 
Running L2Torture in P1 we had many L1DONE timeouts  
in run 156193.  Typical rates: L1A=30 kHz, L2A=300 Hz. 
------------------------------------------------------- 
Going back to P6 to see if it happens there. 
</pre>
<!-- Author: Greg, Steve, Michael -->


<!-- Date: Mon Jan 6 21:04:40 2003 -->
Tested Partition 0 at high L1A rate (~30 kHz mostly) and high  
L2A (~300 Hz) and found <B>No L1 Done Timeouts</B> Run was 
run <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=156198'>156198</a>.  Now marching through the partitions. 
<h3>Partition 0 is fine</h3>
<!-- Author: SCN -->


<!-- Date: Mon Jan 6 21:09:36 2003 -->
Partition 1 gets L1 Done timeout almost immediately after setting 
rate up...Prescales are <ul> 
<li> L1_MB_XING_PS101 = 53 
<li> L2_PS10K_L1_MB_XING_PS101 = 101 
</ul> 
and trigger table is L2_TORTURE [4,209,318].  On to P2...
<!-- Author: SCN -->


<!-- Date: Mon Jan 6 21:14:16 2003 -->
P2 suffers same problem, 2 L1 DTOs within the the first 100k events running at 30 kHz L1A.
<!-- Author: SCN -->


<!-- Date: Mon Jan 6 21:29:35 2003 -->
<pre>Partition 3 checks out fine!</pre>
<!-- Author: Steve, Michael, Greg -->


<!-- Date: Mon Jan 6 21:40:45 2003 -->
<pre>Partition 4 is fine! 
</pre>
<!-- Author: Greg, Steve, Michael -->


<!-- Date: Mon Jan 6 21:57:20 2003 -->
<pre>Partition is 5 is fine! 
----------------------- 
Partition 6 gave L1 DONE timeouts right away.</pre>
<!-- Author: Michael, Steve, Greg -->


<!-- Date: Mon Jan 6 22:11:52 2003 -->
<pre>Partition 7 tests perfectly fine. 
</pre>
<!-- Author: steve et al. -->


<!-- Date: Mon Jan 6 22:13:37 2003 -->
<pre>We have removed PROM1 v61 from P6 and replaced it 
with v58.  We are going to restest P6 now.</pre>
<!-- Author: Michael, Steve, Greg -->
<!-- Comment: Mon Jan  6 22;17;23 comment by...Greg et al -->
<pre>P6 still give L1 DONE timeouts as expected.</pre>


<!-- Date: Mon Jan 6 22:22:01 2003 -->
<pre>We are now swapping the cards for P6 and P7 to make 
sure the problem is with the card and not the slot. 
TS #0 now serves P7.  TS #8 is now in the P6 slot. 
</pre>
<!-- Author: Greg, Steve, Michael -->


<!-- Date: Mon Jan 6 22:32:24 2003 -->
<pre>Testing P6 now with the card that used to be in P7. 
This card works perfectly indicating the problem is not  
with the slot. 
Going to test the bad card which is now in P7.</pre>
<!-- Author: greg and the rest -->
<!-- Comment: Mon Jan  6 22;39;14 comment by...GSM -->
<pre>The bad board is still bad in the P7 slot.
</pre>


<!-- Date: Mon Jan 6 22:41:58 2003 -->
<pre>We are now removing the HEVB (and all readout) to see if 
the problem, now in P7, persists.  This is to learn how 
much of the DAQ we need to in a teststand to debug the  
bad TS cards.</pre>
<!-- Author: Greg -->
<!-- Comment: Mon Jan  6 22;43;46 comment by...msg -->
<pre>We had an L1 DONE timeout immediately.
</pre>


<!-- Date: Mon Jan 6 22:51:22 2003 -->
<pre>We have put prom1, v61 (ROL firmware) into a good TS (#8)  
and placed it in Partition 7.  
----------------------------------------------------------- 
The bad TS (#0) is now back in P6 but with the older non-ROL 
firmware (v58). 
</pre>
<!-- Author: Michael et al. -->
<!-- Comment: Mon Jan  6 22;54;17 comment by...SCN -->
<pre>Current TS map (after Swap and Reswap)
--------------------------------------
Slot  2: 0024 001 TRACER__V2             
Slot  3: 0005 040 TS V2.0
Slot  5: 0013 040 TS V2.0
Slot  7: 0014 040 TS V2.0
Slot  9: 0007 040 TS V2.0
Slot 11: 0003 040 TS V2.0`
Slot 13: 0006 040 TS V2.0
Slot 15: 0000 040 TS V2.0
Slot 17: 0008 040 TS V2.0
Slot 19: 0002 042 GL2 V1 @
-----------------------------
#0013, #0014, and #0000 are the bad ones vis a vis L1 Done timeouts and high rates</pre>


<!-- Date: Mon Jan 6 22:54:23 2003 -->
<pre>Running with the new P7. No HEVB, L2 TORTURE, No ROL in Trigger. 
---------------------------------------------------------------- 
As expected no L1 DONE problems or the like. 
</pre>
<!-- Author: msg -->


<!-- Date: Mon Jan 6 23:01:35 2003 -->
<pre>Replacing the cards in P1 and P2 with spares Michael 
brought from Yale. Now P2 has TS #10 and P1 has TS #12.</pre>
<!-- Author: msg -->


<!-- Date: Mon Jan 6 23:05:53 2003 -->
<pre>Removing TS #0 from P6 to take to Yale for testing. 
Replacing it with TS #13 which was seen to be bad in  
slot 5 aka P1.  I.e. P6 is still bad! 
</pre>
<!-- Author: msg -->


<!-- Date: Mon Jan 6 23:08:12 2003 -->
<pre>Tested the newly arrived spare in P1 and had L1 DONE 
timeouts immediately. 
</pre>
<!-- Author: msg -->


<!-- Date: Mon Jan 6 23:13:26 2003 -->
<pre>Now testing the new TS in P2 ... this one works!</pre>
<!-- Author: msg -->


<!-- Date: Mon Jan 6 23:20:51 2003 -->
<pre>Summary up to now: 
Testing the eight resident TS cards and two spares we find that  
four of these cards do not handle L1 DONE properly at high  
but reasonable DAQ rates.  Replaced one bad card with a good 
spare and taking one bad card back to Yale for study. 
------------------------------------------------------------ 
! There are still two bad TS cards for L1 DONE: P1 and P6 ! 
------------------------------------------------------------</pre>
<!-- Author: Feild, Schmidt, and Nahn, Inc. -->


<!-- Date: Mon Jan 6 23:26:59 2003 -->
Now running with Zombie Silicon ladder (dead, but runnable) and 
HEVB still with Torture Trigger table.  However, now we have a TS  
(in P7) with new ROL firmware.  It is working well,  
so we will next move to 
a trigger table which utilizes ROLs.
<!-- Author: Greg (the critic) Michael (the impatient) and Steve (the sickly) -->
<!-- Comment: Mon Jan  6 23;30;19 comment by...Rand McNally -->
Current TS Map
<pre>
-------------------------
Slot  3: 0005 040 TS V2.0
Slot  5: 0012 040 TS V2.0
Slot  7: 0010 040 TS V2.0@
Slot  9: 0007 040 TS V2.0
Slot 11: 0003 040 TS V2.0`
Slot 13: 0006 040 TS V2.0
Slot 15: 0013 040 TS V2.0
Slot 17: 0008 040 TS V2.0
-------------------------</pre>
Slot 5 (#12) and Slot 15 (#13) known not to like L1 DONE too much


<!-- Date: Mon Jan 6 23:30:38 2003 -->
<pre>Same as above but now we are switching to a trigger table 
with readout lists: COSMICS_TEST_L2INCLUSIVE[3,190,318]. 
</pre>
<!-- Author: greg, steve, michael -->
<!-- Comment: Mon Jan  6 23;49;26 comment by...Steve Nahn, Michael Schmidt, Greg Feild -->
<pre>Run number = 156216. P7. This run has it all!
1) ROL firmware
2) ROL trigger table
3) HEVB
4) Zombie ladder (lives!)
5) Trigger rate bordering on 12 kHz.
-------------------------------------------------------
There are no problems with ROL or L1 DONE or anything!
</pre>


<!-- Date: Tue Jan 7 00:01:46 2003 -->
<pre>Run number 156217 - Same as above but we loaded the 
3 ROL executable at coldStart.  DId it take ...?</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Jan  7 07;34;57 comment by...Michael -->
<pre>The tracer scope shot of two ROL lines sure looks
like `it' took.</pre>


<!-- Comment: Tue Jan  7 11;08;20 comment by...Michael -->
<pre>Peter checked and these runs look good offline for ROL !</pre>
<!-- Date: Tue Jan 7 07:58:39 2003 -->
<pre>Before leaving for the night we went back to Partition 6 
and tried to replicate the L1-done timeout problem using 
the L2 torture table specified above.  We succeeded. 
 
Hopefully this morning we will be able to replicate this 
result and pursue in more detail under what conditions 
we can provoke L1-Done timeouts.</pre>
<!-- Author: Michael -->


<!-- Date: Tue Jan 7 10:18:48 2003 -->
<pre>Working with partition 1, VRBs, L2 torture table and no hevb. 
 
Get L1-Done timeouts at 30k L1A, 300 L2A 
 
Drop L2A to 3Hz and it gets boring. 
 
Drop L1A to 7k and bring L2A back to 300Hz; still boring 
 
Return L1A to 30kHz and L2A to 300 Hz; L1-Done immediately. 
 
Swapping boards: 
 
</pre>
<!-- Author: Michael -->


<!-- Date: Tue Jan 7 10:22:35 2003 -->
<pre>Swapped TS id=0 with TS id=12 in partition 1 
 
Will bring TS id=12 to Yale since it fails to handle 
L1-Done properly</pre>
<!-- Author: Michael -->


<!-- Date: Tue Jan 7 10:29:29 2003 -->
<pre>Working with partition 1 (run 156242) as above, after swapping 
TS modules.  Find that we get L1-Done timeout as soon as 
we cross the 25kHz L1A level (with about 1% L2A). 
 
</pre>
<!-- Author: Michael -->


<!-- Date: Tue Jan 7 10:44:49 2003 -->
<pre>Moved to partition 6 and see the same result. 
 
Crossing the 25kHz L1A mark causes L1-Done timeout 
(1% L2A) 
 
Also tried higher (40kHz) rates; the problem does not 
go away. 
 
Andrew was on the line and found it interesting that 
the L2 machine is in state 18 before trapping on L1-Done 
timeout.  This means it has a L2R to operate on and 
is `waiting' for L1-Done. 
 
All last evening and this morning for any L1-Done we 
see the same L2 FSM pattern --&gt; 18 --&gt; 14. 
 
</pre>
<!-- Author: Michael -->


<!-- Date: Tue Jan 7 10:46:18 2003 -->
<pre>Andrew said it would be nice if there was a way to get 
L2P to always reject. 
 
For now I save the run configuration as mps_l1done in 
daq-test</pre>
<!-- Author: Michael -->



<!-- Date: Tue Jan 7 11:28:54 2003 -->
<pre>We tried forcing L2P to only give L2R (secret download; 
Peter Wittich will provide the recipe). 
 
We had to pause the run to read the scalers.  L1A rate 
of 30 kHz caused no L1-Done problems (partition 6). 
 
Conclusion a mix of L2A and L2R is needed. 
 
Prior to this I was running for a half an hour or so with 
20kHz L1A and 200 Hz L2A and no problems (again partition 6). 
 
</pre>
<!-- Author: Michael, Peter, Steve -->


<!-- Date: Wed Jan 15 14:48:27 2003 -->
<pre>Testing theories about ROL and massive Done Timeout. 
 
We ran with partition 7 and L1,L2 trigger crates, 
cosmic_test_l2inclusive 3,222,318. 
 
See no evidence for ROL being sent at TRACER. 
 
Run special script for L2 to get ROL change every L2 
event. 
 
See DONE timeout from all involved crates.  Symptom 
is verified to be (DAQmon RXPT and cdfvme RXPT panel) 
that crates claim they did not see the L2A message! 
 
We can force the timeout to happen more often if we 
increase the L2A rate. 
 
We switch the v61 prom from partition 7 to 5 and see 
the timeout happen again. 
 
Next we rebooted some of the readout crates affected 
and verify that the ROL is in fact actually enabled 
when we coldstart/activate the trigger table mentioned 
above. 
 
We again the see the massive timeout. 
 
CSL is back up so we try to run with HEVB.</pre>
<!-- Author: Michael -->


<!-- Date: Wed Jan 15 17:26:16 2003 -->
<pre>We have now taken a some runs with each partition 5, 6 and 
7 and HEVB and cosmic_l2inclusive.  In each case we get 
massive done timeout (crates not seeing L2A). 
 
</pre>
<!-- Author: Michael -->
<!-- Comment: Wed Jan 15 17;30;24 comment by...Michael -->
<pre>This is all with V61 prom1 in TS P5,6,7</pre>
<!-- Date: Wed Jan 15 17:29:54 2003 -->
<pre>156631,633,634,639  are the run numbers with ROL 
changing and v61 prom1.</pre>
<!-- Author: Michael -->



<!-- Date: Wed Jan 15 17:33:33 2003 -->
<pre>We also tried a L2 torture table with no 
ROL changes and partition 7 (V61 prom1) 
 
No problems with done timeouts. 
 
</pre>
<!-- Author: Michael -->


<!-- Date: Wed Jan 15 20:01:37 2003 -->
<pre>We return to partition 6 and the cosmic l2 inclusive 
table. 
 
Peter ran his ROL job on the recent runs and all showed 
the symptom of having NO ROL mismatches until the event 
after the massive DTO (done not de-asserted). 
 
We make one more run 156655 to verify the same problems 
before switching to TS prom1 V62 (this is only meant to 
fix L1-Done problems which we are not testing now). 
 
</pre>
<!-- Author: Michael -->
<!-- Comment: Wed Jan 15 21;16;55 comment by...Michael -->
<pre>We tried a run and did indeed get the massive done timeouts.

This was run 156658.</pre>
<!-- Date: Wed Jan 15 21:14:03 2003 -->
<pre>We switch the GL2 firmware: prom GL23_1q  --&gt; GL23_1t 
 
This version includes the capability of simulating 
L1-Done, emulating L2P with prescalable L2A and 
changing ROL, and emulating FRED with prescaled BC 
L1A.</pre>
<!-- Author: Michael -->
<!-- Comment: Wed Jan 15 21;44;38 comment by...Michael -->
<pre>This is the same version (not fully GUI supported, etc.)
that is in place in the teststand at Yale.</pre>


<!-- Date: Thu Jan 16 00:41:32 2003 -->
<pre>Well after much fun and joy we got the new GL2 panel 
up and running (don't forget to gmake on b0dap30, but 
first setup fer and then override it for java). 
 
Then we re-discover the fact that the L2A prescale 
emulator does not read back the upper bits. 
 
But we use it any way, get more TSI deadtime than 
expected and have to remove the L2P crates from the 
run so they don't complain. 
 
We try without HEVB and do finally get a massive 
DTO (156678) and now try with the GL2 emulator.</pre>
<!-- Author: Michael -->


<!-- Date: Thu Jan 16 01:06:55 2003 -->
<pre>Unfortunately we could not find settings using 
the GL2 L2 emulation that would cause the massive 
Done timeout.  We tried to vary the L1 and L2 accept 
rates and the L2 latency. 
 
</pre>
<!-- Author: Michael -->


<!-- Date: Thu Jan 16 13:01:08 2003 -->
<pre>Update of TS board status at B0: 
 
 
partition	Slot	ID	prom1	vme chipset a 
 
0		3	5	58	yes 
1		5	0	58	yes 
2		7	10	58	no (?) 
3		9	7	58	yes 
4		11	3	58	yes 
5		13	6	61	yes 
6		15	13	62	no (?) 
7		17	8	63	yes 
 
spare			12		yes 
spare			14		yes 
 
 
re: vme chip set a 
 
no(?) means we have no record of vme chip set change 
but will have to inspect board 
 
yes means we have record of the vme chip set change 
 
prom1	L1	L2	RO 
61	27	29	12 
62	27	30	12 
63	28	30	13</pre>
<!-- Author: Michael -->
<!-- Comment: Thu Jan 16 13;07;38 comment by...Michael -->
<pre>We will update partition 5 and 6 with prom1 v63,
since v61 is known to be bad for L1-Done and v62
is bad for ROL.  v63 should be fine for L1-Done
and needs testing by silicon.  v62 is not good for
ROL (after any HALT; we are trying mightly to test
this theory now).</pre>


<!-- Comment: Thu Jan 16 14;45;54 comment by...Michael Schmidt -->
<pre>Partition 6 or 7 as currently configured with prom1
v62 and v63 should both be good for silicon tests
of L1-done.

</pre>
<!-- Date: Thu Jan 16 14:06:03 2003 -->
<pre>My bad, but at least I figured it out before I left. 
 
The new GL2 firmware needed to be backed out (along 
with supporting software).  The L2 emulation was not 
being turned off so it was interferring with being 
able to control L2A rates with the L2P engaged. 
 
Everything is back to normal now.</pre>
<!-- Author: Michael -->


<!-- Date: Thu Jan 16 14:39:52 2003 -->
<pre>Forgot to make this entry earlier: 
 
We are trying to run with partition 7 and v63 of prom1. 
 
This version has new firmware for L1 and RO FSM in 
TS.  Before this version there was a FIFO for ROL that 
was NOT being reset on HRR.  This could lead to ROL 
mismatch after an HRR if the case that the FIFO was 
not already empty. 
 
The expectation is that now neither a massive DTO or 
any HRR should mess up the ROL. 
 
Then we `just' have to focus on why we get massive 
DTO when ROL are enabled.</pre>
<!-- Author: Michael -->



<!-- Date: Thu Jan 16 15:30:19 2003 -->
<pre>Yet another update: 
 
partition       Slot    ID      prom1   vme chipset a  
 
0               3       5       58      yes  
1               5       6       58      yes  
2               7       10      58      no (I looked)  
3               9       7       58      yes  
4               11      3       58      yes  
5               13      0       63      yes  
6               15      13      63      yes (I looked)  
7               17      8       63      yes  
 
spare                   12              yes  
spare                   14              yes  
 
as is clear we have updated the TS prom1 firmware to 
version 63 in partitions 5,6, & 7.  Version 61 and 62 
were superceded as per discussion with Andrew.</pre>
<!-- Author: Michael -->


<!-- Date: Fri Jan 17 13:47:07 2003 -->
<pre>Took two ROL runs in Partition 6 with PROM1_63. 
Runs 156781, 156782.  Had massive DTOs in both. 
Sent to Peter W for ttheofflien anaysis.</pre>
<!-- Author: Greg  -->


<!-- Date: Fri Jan 17 13:55:45 2003 -->
<pre>Steve takes a run with P1 thinking it was one of the 
partitions known to be flaky for Level 1 DONE.   
This was two establish a baseline for our L1 DONE study. 
Little did he know that this card was moved to P5 and  
had a firmware upgrade! Run 156793.  No L1 DONE TO. 
 
</pre>
<!-- Author: Greg, Steve -->
<!-- Comment: Fri Jan 17 14;36;21 comment by...Michael -->
<pre>Sorry, I thought I did communicate this to Steve,
but in any case I can see the wisdom of his test
(getting back to the bad square 0 so to speak).</pre>
<!-- Date: Fri Jan 17 14:01:10 2003 -->
Hacked TS P4 and P5 so as to get a "bad for L1 Done Board (ID 0)" 
with old firmware (v58) to reproduce problem (using P5).  Then we 
will switch to P6 and see if L1 Done problem reoccurs
<!-- Author: SCN, RGF -->
<!-- Comment: Fri Jan 17 14;05;56 comment by...SCN -->
L1 Done timeouts occur basically immediately (using MPS_L1DONE
configuration, adjusting prescales as before).


<!-- Comment: Fri Jan 17 14;38;11 comment by...Michael -->
<pre>Does this mean that the firmware was switched in partition
5 to be prom1 v58?   Presumably nothing had to be done
to partition 4 which was left loaded with this prom.</pre>
<!-- Comment: Fri Jan 17 16;15;36 comment by...Greg -->
<pre>It was switched  to 58 for the baseline test and then 
put back to 63.  We borrowed the v58 prom from P4 and 
then put it back.

</pre>
<!-- Date: Fri Jan 17 14:11:07 2003 -->
Now running on partition 6, which is a "Bad L1 Done board" but 
has new firmware (v63 I believe).  Run number is 156799.  So far, no trouble after 5 million L1A.  Prescales et al as before...
<!-- Author: SCN -->
<!-- Comment: Fri Jan 17 14;20;36 comment by...SCN, RGF -->
No trouble after 24 million L1As.  Passing token of command
to Greg with request that he use all VRB crates and either P5 or P6 to do whatever tests he wishes...

<!-- Comment: Fri Jan 17 14;29;38 comment by...Greg -->
<pre>End run after 40 M L1As to switch to a readout list table.</pre>



<!-- Comment: Fri Jan 17 14;39;43 comment by...Michael -->
<pre>My entry at 15:30 yesterday in this e-log gives the
partition, TS ID and prom1 version number as we left
this system yesterday</pre>


<!-- Date: Fri Jan 17 14:40:34 2003 -->
<pre>Still in P6.  Switching to the ROL table COSMIC_TEST_L2INCLUSIVE. 
Silicon VRBs still in.  Run number = 156803. 
</pre>
<!-- Author: Greg, Steve -->
<!-- Comment: Fri Jan 17 14;45;56 comment by...Greg -->
<pre>Massive DTO after 30313 events.  Seemed to occur after
I changed prescales.  Specultations:  Rate dependent?
PAUSE dependent?

</pre>

<!-- Comment: Fri Jan 17 14;47;59 comment by...Greg -->
<pre>Another MDTO after 30899 events.
L1A = 11 KHz, L2 = ?? due to deadtime</pre>

<!-- Comment: Fri Jan 17 14;48;37 comment by...Greg -->
<pre>Another MDTO ....
</pre>

<!-- Comment: Fri Jan 17 14;49;28 comment by...Greg -->
<pre>I'm going to say L2A = 100 Hz.</pre>

<!-- Comment: Fri Jan 17 14;52;28 comment by...Greg -->
<pre>MDTO after 52024 events</pre>


<!-- Comment: Fri Jan 17 16;19;33 comment by...Michael -->
<pre>From studying the Run Summary entries it does not
look like the problem is caused by pausing the
run.  Perhaps you have decided the same thing.

</pre>


<!-- Date: Fri Jan 24 12:12:25 2003 -->
<pre>I measured the offset between the falling edge of L1A and Bunch0 
at 4 central calorimeter crates.  wcal02 and ccal13 were 4.0ns 
and ccal05 and wcal 06 were 3.2 ns.  I will check against the 
numbers to see where that puts the data arrival.</pre>
<!-- Author: Andrew -->


<!-- Date: Fri Jan 24 12:14:47 2003 -->
<pre>Yesterday I looked at the massive done timeout problem.  I found 
that the timeout would occur after what looked like a collision 
between a L2R, ROL change, and L2A messages.  This means the 
problem is in the scheduler prioritization.</pre>
<!-- Author: Andrew -->


<!-- Date: Mon Jan 27 16:37:17 2003 -->
<pre>Updated all partitions except P0 to have PROM1_63. 
 
</pre>
<!-- Author: Greg  -->


<!-- Date: Fri Jan 31 18:56:50 2003 -->
<pre>Placed PROM2_30 into P7.  Started an ROL run at high rate 
----------------------------------------------------------- 
Fri Jan 31 17:17:29 Run 157802 ACTIVATE: rol stress test 
----------------------------------------------------------- 
40 minutes later, no problems.  Two earlier runs today 
ended  with problems in the alpha processor.  We had thought 
they could possibly be correlated with the new firmware 
although Stephen could not really see the mechanism.  Aslo, 
there were other DAQ issues (rtserver, DTOs, etc.) 
-------------------------------------------------------------- 
Run 157802 finally ends with error handler crash. All files  
were sent to the look area.  Will ask Peter to run his job. 
 
</pre>
<!-- Author: Greg -->


<!-- Date: Tue Feb 4 10:29:14 2003 -->
<pre>Yesterday we put prom2_30 into all partitions except for 
P0. 
 
</pre>
<!-- Author: Greg -->


<!-- Date: Thu Feb 6 09:32:45 2003 -->
<pre>Partition 3 with the `new' proms not working well 
with the Silicon giving many L1 DONE timeouts. 
I swapped out TS #7 from this slot and replaced it  
with spare TS #14.</pre>
<!-- Author: Greg -->


<!-- Date: Thu Feb 6 10:18:58 2003 -->
<pre>Placed the prom BG_GSYNC in the Bunch Gates scaler card. 
 
</pre>
<!-- Author: Greg  -->
<!-- Comment: Thu Feb  6 15;28;14 comment by...Michael Schmidt -->
<pre>Seems to be working fine as of 158214 (that is no
sync errors are being reported on either board and
no check sum errors for L1A or Livetime per bunch).</pre>


<!-- Date: Mon Feb 10 15:59:27 2003 -->
<pre>Due to small Livetime accounting issues I swapped 
TS #10 out of P2 and replaced it with TS #12. 
 
 </pre>
<!-- Author: Greg Feild -->


<!-- Date: Mon Feb 17 04:24:16 2003 -->
<pre>Just some bookkeeping entries: 
 
Greg puts a note in main e-log on Wed Feb 12 at 16:52 
indicating that P0 has old firmware re-installed. 
 
FRED PROM update happened on Feb. 2 and stayed in until 
just before run 158876. 
 
Moving to P0 from P1 (new firmware) and reverting the 
FRED PROM makes no difference in the rash of mismatches 
being seen in prescale accounting (low level) since 
run 158731. 
 
</pre>
<!-- Author: Michael Schmidt -->
<!-- Comment: Wed Feb 19 13;33;28 comment by...Greg Feild -->
<pre>Today, I upgraded P0 to have the latest TS firmware
and I put back the latest FRED prom compiled  with 
XILINX 1.3

Note:  The flurry of prescale accounting errors was
traced to a final reading of the scaler data before
the run was completely ended.  This has been fixed.

</pre>