<!-- Title: 10:45:05  Tue Apr 16 2002 -->
<!-- Crewchief: L2 et al -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Making bottom L2 crate work</pre>


<!-- Date: Tue Apr 16 10:49:05 2002 -->
<pre>See previous page for earlier work. 
 
Crate configuration 
 
SLot  board  delay 
L1      10    0 
Track    9     1us 
Clist   14    400ns 
Svt     12    2us 
Iso     20    0 
 
Using latest TTL firmware in all boards.</pre>
<!-- Author: Greg, Natalia, Stephen, Jane -->


<!-- Date: Tue Apr 16 11:31:27 2002 -->
<pre>Types of errors 
 
1. NO board sends anything. On the scope we see that boss and 
ddone are always held low. This was typical for Tracklist,  
Svtlist collisions in the past. Triggering on ddone* being low 
for > than 600ns, we see that Tracklist gets the bus just 
after Clist is done sending (4 Mbus words since a cable is 
pulled.).  Boss and ddone are then held low.  
  When all boards are sending, Svtlist sends about 500ns after 
Tracklist is done, so it isn't clear if the collision is 
with CList or Svtlist. 
  If ddone is low when Tracklist gets the bus, then the Tracklist 
state machine will be hung. On the scope we saw ddone going 
back high -BUT the high level for ddone maxes out at 2V. Unlike 
the 3V for the other signals. 
</pre>
<!-- Author: G,J,N,S -->
<!-- Comment: Tue Apr 16 12;08;52 comment by...greg -->
<pre>Found a blown fuse on the MBUS termination card.
Stephen replaces it.</pre>
<!-- Date: Tue Apr 16 12:04:43 2002 -->
<pre>Masa arrives!  We go to lunch.</pre>
<!-- Author: G,J,N,S,M -->



<!-- Date: Tue Apr 16 14:40:42 2002 -->
<pre>Added new firmware to Clist (v9.1.1) (old firmware is 9-1). 
This firmware waits 150ns after the last dstrobe before 
releasing boss. This gives more time for ddone* to get pulled 
high.  
  HOwever, we still see the same errors with the tracklist board 
not sending (shows up quickly at high rate). Triggering on 
mod_done for the tracklist board being low for more than 
10us, we see that tracklist gets the bus right after clist but 
then doesn't assert dstrobe. It holds the bus for about 150ns 
and then gives it up. Following that the bc counters between 
tracklist and svtlist disagree and are out of sync by n events. 
  We have also seen events where svtlist didn't send any MBUS 
words. These are classic tracklist board sync errors.</pre>
<!-- Author: J, S, N, G -->


<!-- Date: Tue Apr 16 14:55:34 2002 -->
<pre>Run number 143341, CLIST not in and the XTRPList board still did send anything on every event. 
Stephen is removing delays from both SVTlist and Tracklist, 
so there will be no delays in the arbitration.  Clist still out.</pre>
<!-- Author: G,S,J,N -->
<!-- Comment: Tue Apr 16 14;57;14 comment by...natalia -->
<pre>The run number is a palindrome!!</pre>

<!-- Comment: Tue Apr 16 15;05;25 comment by...natalia -->
<pre>In my previous note, I meant "did NOT" send of course...</pre>


<!-- Date: Tue Apr 16 15:07:45 2002 -->
<pre>Run 143342, with the conditions described in the previous 
entry -- it works!  So Stephen is putting CLIST back in.</pre>
<!-- Author: G, S, J, N  -->


<!-- Date: Tue Apr 16 15:12:41 2002 -->
<pre>Run 143343 with CLIST back in. 
Using software evb, changed L2 prescales to get the higher rate of L1 (L1 rate is now ~17kHz).  It just works!</pre>
<!-- Author: G, S, J, N -->


<!-- Date: Tue Apr 16 15:15:29 2002 -->
<pre>Ended run 143343 after 409 L2 accepts with no errors. 
At the moment we have no calorimeter. 
Starting a new run using hardware event builder.</pre>
<!-- Author: G, S, J, N -->


<!-- Date: Tue Apr 16 15:21:35 2002 -->
<pre>Run 143347 -- ending the run after 475 L2 accepts with no errors. </pre>
<!-- Author: G, S, J, N -->


<!-- Date: Tue Apr 16 15:55:08 2002 -->
<pre>Got calorimeter back except for CCAL00.
 Plugged DCAS cables back in.
143352 with L2_TORTURE, fake tracks and real clusters.  No errors in approx 100k L1A. 
143353 same as previous.  </pre>
<!-- Author: G, S, J, M1, M2, N, -->


<!-- Date: Tue Apr 16 15:57:11 2002 -->
<pre>Run 143353 ending after 179,970 L2 accepts with no errors. 
Masa is here, he will put the RECES board to the bottom crate.</pre>
<!-- Author: G, S, J, M1, M2, N -->


<!-- Date: Tue Apr 16 16:09:36 2002 -->
<pre>Run 143357 - L2 Torture, no delays in track cards, funky 
new CLIST firmware, RECES in bottom crate but not readout by 
L2. (still using L2_TORTURE_v-4_nomon_noreces exe). 
Also, DCAS cables are plugged back in</pre>
<!-- Author: S,G,J,M,M,N -->


<!-- Date: Tue Apr 16 16:14:11 2002 -->
<pre>SciCo visits and comments on the hand stenciled UCLA front panels 
</pre>
<!-- Author: Greg -->


<!-- Date: Tue Apr 16 16:19:38 2002 -->
<pre>Run # 145359 but now running L2_TORTURE_v-4_nomon_fourreces 
to readout all 4 Reces on every L2A. 
 
</pre>
<!-- Author: S,G,J -->
<!-- Comment: Tue Apr 16 16;25;16 comment by...J and G -->
<pre>Errors!
=======
Error: Clist data has buffer number 0 expected 1 for counter = 1882176         
Number of Clist MBUS words is 3                                   Clist data 0 is 8f 8017c801                                       Clist data 1 is 8f 17c801                                         Clist data 2 is 8f 57c801                                         Clist data 3 is 8f 8057c801                                      Clist data 4 is 8f 8197c801                                       Clist data 5 is 8f 197c801                                                     </pre>

<!-- Comment: Tue Apr 16 16;32;52 comment by...Greg and Jane -->
<pre>More about this run
====================
deadtime=2%, 60% of that is from L2, 
the remainder is readout.
FL1A = 25 kHz
 L1A = 24 KHz
 L2A = 157 Hz
15 Million L1As and 109,000 L2As and counting</pre>




<!-- Date: Tue Apr 16 16:44:03 2002 -->
<pre>More Errors 
============ 
 Error: Clist data has buffer number 0 expected 1 for counter = 19181855         
Number of Clist MBUS words is 2   
Clist data 0 is 18 8017a001  
Clist data 1 is 18 17a001 
Clist data 2 is 20023 197a002 
Clist data 3 is 20023 8197a002              </pre>
<!-- Author: Greg -->


<!-- Date: Tue Apr 16 16:47:16 2002 -->
<pre>Run 143359 ends.  DONE timeout from XTRP.  The XTRP 
problem also seems to have caused  many track related  
sync erros in L2 crate.  HRR doesn't help so we abort 
</pre>
<!-- Author: Greg -->


<!-- Date: Tue Apr 16 16:52:56 2002 -->
<pre>New run. 143368</pre>
<!-- Author: Jane and Greg -->
<!-- Comment: Tue Apr 16 16;55;51 comment by...greg -->
<pre>CLIST ERRORS
=============
 Error: Clist data has buffer number 0 expected 1 for counter = 1008423         
Number of Clist MBUS words is 3                                   Clist data 0 is 61 8017c401                                       Clist data 1 is 61 17c401                                         Clist data 2 is 61 57c401                                         Clist data 3 is 61 8057c401                                       Clist data 4 is 61 8197c401                                       Clist data 5 is 61 197c401                                                     </pre>

<!-- Comment: Tue Apr 16 16;59;11 comment by...greg -->
<pre>These may be the famous "psuedo-errors".
Jane sez they are in TrigMon too.</pre>


<!-- Date: Tue Apr 16 17:01:14 2002 -->
<pre>All this near perfection is boring so I am going to  
try and double the L1 rate at 90,000 L2As</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 16 17;04;31 comment by...Greg -->
<pre>L1 rate is now 40 KHz - still no errors! (but the 
deadtime is HUGE)</pre>


<!-- Date: Tue Apr 16 17:13:32 2002 -->
<pre>Broke last run trrying to adjust L2 prescales on the  
fly -- Special FER/RC interaction in bottom crate 
may need some further study. Starting new run 143374</pre>
<!-- Author: Greg, Stephen, Mas -->
<!-- Comment: Tue Apr 16 17;18;16 comment by...everybody -->
<pre>The last run showed some low level errors with 
RECES data so MAsa jiggled some fibers.  We 
will see if that helped!\</pre>

<!-- Comment: Tue Apr 16 17;21;52 comment by...Greg -->
<pre>DTO from XTRP - no adverse effects in L2!
HRR and continue
</pre>


<!-- Date: Tue Apr 16 17:53:03 2002 -->
<pre>Masa determines the RECES errors may be due to  
the missing ccal00 crate.  We add this crate and 
atart run 143385</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 16 17;57;55 comment by...Greg -->
<pre>The  fake pseudo errors return:
================================
Error: Clist data has buffer number 1 expected 0 for counter = 1543764        
Number of Clist MBUS words is 3                                   Clist data 0 is cc 8007c401                                       Clist data 1 is cc 7c401                                          Clist data 2 is cc 47c401                                         Clist data 3 is cc 8047c401                                       Clist data 4 is cc 8387c401                                       Clist data 5 is cc 387c401                                                     </pre>

<!-- Comment: Tue Apr 16 18;20;59 comment by...Stephen and Greg -->
<pre>After 270,000 event we change the L1 BC prescale to 
51  jacking the L1 rate to 38 KHz -- no errors.</pre>

<!-- Comment: Tue Apr 16 18;28;54 comment by...Tom, Stephen, Greg -->
<pre>Ending this run with 54 Million L1As and no L2 
errors to do more ambitious tests.</pre>


<!-- Date: Tue Apr 16 18:32:28 2002 -->
<pre>New run with physics_test_1_01[16,15,-] 
Keeping fake track and SVT data. 
This table has new code for HRR recovery 
after L2 timeouts. 
</pre>
<!-- Author: G,T,S -->
<!-- Comment: Tue Apr 16 18;40;35 comment by...Greg -->
<pre>Success - killed run by trying to change prescales
via the GUI. This feature is not yet 
implemented in the bottom crate; a lesson we
supposedly learned earlier today
Starting a new run.
</pre>


<!-- Date: Tue Apr 16 21:29:11 2002 -->
<pre>Run # 143391 ran for over 200ML1A, 900K L2A. 
Using PHYSICS_TEST_1_01[15,16,282] and bottom TTL crate. 
This crate has all the cards in including Reces and the nodelay 
firmware for track boards. 
 We only had 1 error where the L1 board sent 0 MBus words. 
This was using the default error code which caused a L2 
Decision Timeout. Auto HRR was in use, so the run recovered fine 
(since we were at dinner). 
 Because of the fake tracks in the system, we had a 100KHz rate 
from the L1_7trk trigger which was prescaled by 8 to  
get the total rate down to 23KHz Fred L1A rate. The L2_AUTO 
triggers fed by the L1 track triggers were prescaled by 100. 
The L2A rate was 99Hz. The deadtime was 10% with most of that 
from L2. The L2 latency was 44.7us.  
  TrigMon didn't show any errors except for a few sticky bits 
in Reces, plus some low level Reces errors. Perhaps Masa can 
comment on those errors.</pre>
<!-- Author: Stephen, Tom, Greg -->


<!-- Date: Tue Apr 16 22:51:22 2002 -->
<pre>Loaded some unpack on demand code of Tom's to see if the 
L2 latency would go down. We then had some bc errors 
between the tracklist boards. 
 Tried the default PHYSICS_TEST exe and still had the bc errors. 
Tried to hook up the scope to look at the errors and then 
started to run L2_TORTURE table to run the debugflag. 
 The run stoppped on the first event. Running alpha_check 
gave VISION errors. Turned out that Reces SN05 blew a fuse 
so the alpha crashed when trying to readout that board. 
Replaced the board with SN07.  
Powercycled the crate when replacing Reces and then couldn't 
coldstart because of errors reading the tracer. The solution 
was to unplug and plug CList back in with the power on. 
Then it worked fine. L2_TORTURE ran briefly without errors. 
Ended run to try PHYSICS_TEST table.</pre>
<!-- Author: Stephen, Tom -->
<!-- Comment: Tue Apr 16 23;59;08 comment by...Stephen -->
<pre>Note: It looks like the unpack on demand code was the cause
of the bc errors. We could get the crate working again with
the original exe. Tom will look at the code tomorrow.</pre>


<!-- Date: Wed Apr 17 10:23:44 2002 -->
<pre>Note: 
  If you powercycle the crate, make sure to hit reset on the 
Clist board before continuing. Otherwise you will get VME 
errors (there will be problems reading the tracer idprom). 
 If hitting reset doesn't work, then remove and insert Clist with 
the power on.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Apr 17 11:19:42 2002 -->
<pre>Test of Tracklist firmware 
 Keeping the crate in the standard configuration 
Board Slot Delay 
L1     10   0 
Track   9   (change from 0 to 1us) 
Svt    12    0 
Clist  14    400ns 
Iso    20     0 
Reces  15,16,17,18 
 
Running L2_torture_v-4_nomon_fourreces exe 
 
With the 1us delay in Tracklist we get errors where the board 
won't send for awhile and then start sending but with bc 
sync errors. This happens both with and without Clist in the 
crate. 
 Putting back the no delay firmware, we don't see these errors. 
We did get one error where Tracklist grabbed the bus and  
didn't let go.</pre>
<!-- Author: Stephen,Jane, Greg -->


<!-- Date: Wed Apr 17 14:55:26 2002 -->
<pre>Run 143346 - No delays in track boards. HEVB.  High L1A 
rate due to Carla's pulled cables. L1 and L2 rates  
varied as we played with the prescales.  No  
errors seen in TrigMon --- it is not clear if the  
alpha debug code was running.  Run ended when  we  
tried to make scalerGUI prescale changing work in  
the bottom crate.  Bill B went back to the drawing 
board. 
 
</pre>
<!-- Author: Greg -->
<!-- Comment: Wed Apr 17 15;05;00 comment by...Greg, Stephen, Jane -->
<pre>Run 143347 - 36209 L2 events at 24 kHz L1A rate 
and no MBUS errors.  Bill's scaler test ends run.
</pre>


<!-- Date: Wed Apr 17 22:10:39 2002 -->
<pre>Note: Bill Badgett fixed the Trigger Rates Gui so that L2 
prescales can be changed from the GUI for running with the  
bottom crate.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Apr 17 22:34:59 2002 -->
<pre>With the 1us delay in Tracklist board, we get frequent  
errors where the Tracklist board doesn't send any data. 
From the scope we see that Tracklist asserts boss for about 100 
ns but never sends dstrobe.  
 Hooking the LA up to the board, we can see that the firmware 
falls out of the state machine. (In at least some cases it 
gets the bus and sends the data a few us later).</pre>
<!-- Author: Jane, Matt, Stephen -->
<!-- Comment: Wed Apr 17 23;55;02 comment by...Stephen -->
<pre>Jane makes new firmware for the Tracklist board (version 4/17/02)
This has a 1us delay for Tracklist, 2us for Clist but with
the fix to the state machine.
Putting this design in Tracklist causes the errors to go away.
</pre>


<!-- Comment: Fri Apr 19 08;54;17 comment by...S -->
<pre>That should say 2us for Svtlist</pre>
<!-- Date: Thu Apr 18 00:34:38 2002 -->
<pre>RUn# 143487 using L2_torture_v-4_nomon_fourreces. 
THis has Jane's new firmware in Tracklist. 
Ran for 100M L1A with 0 MBus errors. 
(There was one L2 state 14 error, but for no apparent reason) 
Ran at 20KHz with a few percent deadtime from L2. 
 
We will leave the bottom crate as the default for shift crew 
cosmic running tonight.</pre>
<!-- Author: Stephen -->


<!-- Date: Thu Apr 18 11:39:24 2002 -->
<pre>We put in the new firmware into SVTlist. This has the state 
machine fix and gives svtlist a 2us delay. We unplugged 
several DCAS cables to give several clusters in eac event. 
On the scope we see that Tracklist immediately follows Clist 
with Svtlist sending right after that. 
 We ran both L2_TORTURE and PHYSICS_TEST_1_01[15,16..] 
without any errors detected in the alpha.  
For Run 143509 we did see an error in Clist in TrigMon. For 2 
events there were 0 clusters in TL2D with 3 and 5 expected. 
There were no errors with the number of Mbus words or any 
of the other boards.</pre>
<!-- Author: Stephen,Greg, Matt, Jane -->
<!-- Comment: Thu Apr 18 11;41;51 comment by...Stephen -->
<pre>That should be run# 143506</pre>


<!-- Date: Thu Apr 18 11:52:34 2002 -->
<pre>Jane made new firmware that had no delays for the Tracklist 
boards. Started run 143509 with this firmware.  
Looking on the scope, the boards are clearly arbitrating 
with Tracklist sometimes sending first and sometimes last.</pre>
<!-- Author: Stephen -->



<!-- Date: Fri Apr 19 21:10:15 2002 -->
I am finally following up on the work I was doing on 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=39&frame=2&anchor=20:47:11'> 
March 19</a>  
to try to explain the 2 or 3 ladders that dominate SVT/svtsim 
mismatches.  I am looking at 'SVT_CALIBRATION' table data runs 
142000 and 142009.  To set the scale of the problem, in run 
142009, out of 131K events, there were 126 tracks having mismatched data between SVT output and svtsim output.  93/126 
come from SB4W3L0, 14/126 from SB2W0L4, and 9/126 from SB0W7L3. 
The rest are more scattered. 
<p> 
For SB2W0L4 and SB0W7L3, I can look at the hits actually written 
by the Hit Finders, in the SVDD bank.  SB2W0L4 appears to be a 
problem in bit 0 of the G-link data between the splitters and 
the VRB, in which 0's are often incorrectly turned into 1's.  In 
svtsim (and by examing the SIXD bank directly) I see many  
examples of wrong chip IDs (e.g. 8a becomes 8b), BC mismatches 
(e.g. 06 becomes 07), and wrong strip numbers (e.g. the sequence 
72,73 appears as 73,73).  The Hit Finder emulator reports error 
bits in these events, but the real Hit Finder does not.  Also, 
the real Hit Finder finds clusters with centroids where they 
would be if the correct data had been sent.  I conclude that 
the data reaching SVT are correct, but the data read into SIXD 
are not, i.e. the problem must be on the VRB side of the fiber 
split.  The errors are always on bit 0. 
<p> 
Continued in next entry ... 
<p>
<!-- Author: Bill A -->


<!-- Date: Sun Apr 21 14:42:27 2002 -->
<p> 
The alpha.log date stampp at the beginning of every run for the bottom crate is off by one hour - it never got adjusted for daylight savings time. So if you're trying to decipher times in the ~cdfdaq/alpha.log file, keep this in mind. 
</p>
<!-- Author: Peter Wittich -->
<!-- Comment: Mon Apr 22 14;19;52 comment by...Peter -->
<pre>Actually, the top crate is off by one hour, and the bottom crate
is off by some other weird number. Not sure what it is....</pre>
<!-- Date: Sun Apr 21 16:19:47 2002 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-645&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-645' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-646&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-646' width='300'></a></center>
<!-- Author: Masa -->



<!-- Date: Mon Apr 22 17:00:27 2002 -->
<pre>Replacing LOCOS #3 with LOCOS #9 in b0l2cl04 
as a possible solution to the new clustering  
problem (per Monica's request). 
</pre>
<!-- Author: Masa, (Greg) -->
<!-- Comment: Mon Apr 22 18;15;38 comment by...Greg -->
<pre>This did not fix the  clustering problem
</pre>


<!-- Date: Mon Apr 22 18:18:05 2002 -->
Generated a new L2 tagset w/o reces readout based on the current defaults for PHYSICS_1_01 v4. The tagset number is 31. it also has the changed SVTmon code, which now sends and error if either XTRP or SVT send zero words. 
 
<pre> 
 Tags  Alphatag         L2runTag        l2codeTag       N  Table           Ver 
 31 v2_1b_SvtErr        v2_3a_noreces   v2_0_repoMar26  1  PHYSICS_1_01     4    
</pre>
<!-- Author: Peter wittich. -->
<!-- Comment: Mon Apr 22 18;46;50 comment by...Peter -->
<p>Made a new one with l1Mon error printout; tagset 32, alpha tag v2_1d_l1err. Note that this does not have Masa's fixes to the l2code package.</p>
<!-- Comment: Mon Apr 22 19;00;20 comment by...Peter, burin&#39; through the tagsets -->
<p>there was a misunderstanding between masa and me - it turns out that this version of the l2code package had a mismatch to the l2run package, and hence reces was being read out. now fixed with yet another tagset:
</p>

<pre>
33 v2_1d_l1err         v2_3a_noreces   v2_1_nElecFix   1  PHYSICS_1_01     4
</pre>
<!-- Comment: Mon Apr 22 19;20;39 comment by...peter wittich -->
<p>Same as above, but <b>with</b> reces readout.
<pre>
34 v2_1d_l1err         v2_3_hrrFix     v2_1_nElecFix   1  PHYSICS_1_01     4 
</pre>
<!-- Date: Mon Apr 22 18:22:25 2002 -->
<pre> I tried to understand the 5-10% Reces error problem. 
I changed the prescale for some L1 triggers 
 
 (1) Two track trigger (most of the L1 triggers) 
 (2) All the L1 trigger to cause Reces read for 
     trigger decision. 
 
 They didn't cause significant changes in the error rate. 
I also checked the scope to make sure there are always 
twelve MB read in total for L2A events (before and after L2A). 
 
 SO I think it can be real Reces bit errors just coincidence 
with COT to be in. I want to go back to top crate and 
check what happens.</pre>
<!-- Author: masa -->
<!-- Comment: Mon Apr 22 20;40;48 comment by...Greg -->
<pre>RECES worked properly in the top crate.
</pre>
<!-- Date: Mon Apr 22 19:51:02 2002 -->
<pre>At least one problem solved tonight. 
It seems the XTRP-TL2D number of tracks 
mismatch is due to a bad XTRP card  
(single pin problem). 
Jim K is going to try to reseat the card  
and he also gave Matt a tip on how to  
check for this before making the data  
comparison between the two crates. 
</pre>
<!-- Author: Greg -->
<!-- Comment: Mon Apr 22 20;16;00 comment by...Matt -->
Jim was successful.  The XTRP-TL2D Ntracks mismatches are gone as of run 142712 both in Jim's code and mine.  TrigMon will be updated with Jim's suggestion to only count tracks in XTRD if the pT is less than 96 when comparing to TL2D, to remove single bit errors.



<!-- Date: Mon Apr 22 20:42:32 2002 -->
<pre>We swapped LOCOS to CLIST fibers between crates 
b0l2cl03 and 04.  The clustering problem did NOT 
follow the fiber.  We believe the problem lies in  
the  DCAS crate itself. 
</pre>
<!-- Author: Greg -->
<!-- Comment: Mon Apr 22 20;47;04 comment by...Greg -->
<pre>We also tried reset_crate b0l2cl04 without 
success.
</pre>


<!-- Date: Mon Apr 22 21:38:04 2002 -->
<pre>Summary of this evening's studies. 
================================== 
1) CLIST clustering errors seen in  
both top and bottom crates.  We find that the  
LOCOS to CLIST fiber is not the cause of cause of the error. 
2) RECES inefficinecy not seen in top  
crate.  Must be due to readout in the bottom crate ...? 
3) XTRP-TRACKList track number mismatch solved.  Traced 
   to flaky XTRP card which was reseated. 
4) Improved error checking code in L2 tagset 34 causes 
   CPU to crash so we fall back to tagset 22. 
5) The problem where the L1 card would not send on buffers 
   2 and 3 reappeared forcing us to leave the shift crew in 
   the top crate for this evening.  Will return to study this        tomorrow. </pre>
<!-- Author: Greg, Peter, Masa, Matt -->


<!-- Date: Tue Apr 23 10:42:35 2002 -->
<pre>Tev Studies.  Beginning L2 tests.  Start with  
bottom crate and all Interface boards engaged. 
 
</pre>
<!-- Author: Greg and Masa -->


<!-- Date: Tue Apr 23 10:56:53 2002 -->
<pre>We begin with L2 torture. SEVB. Run number 143737. 
Alpha claims L1 not sending on buffers 2 and 3. 
Going to try and see why on the backplane.</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 23 10;59;03 comment by...Greg -->
<pre>As I prepare to go to the back of the crate 
I see that now L1 is also not sending
for buffers 0 and 1.  This decreases my 
confidence that the problem is actually on the 
L1 interface card.

</pre>

<!-- Comment: Tue Apr 23 11;06;50 comment by...Greg -->
<pre>Went back and jiggled L1 cables and neighboring jumpers.
When I return, I see the following.
==========================================================
L1 data is 40830000 1034000 30000
 .
 .
L1 data is ffffffff ffffffff ffffffff 
 .
 .
L1 data is fdefffff ffffffff ffffffff
L1 data is 2 0 0                                                  L1 data is 2 0 0</pre>

<!-- Comment: Tue Apr 23 11;07;54 comment by...Greg -->
<pre>Also see "Alpha is no longer processing at spot 64" 
in Error Handler.
</pre>


<!-- Date: Tue Apr 23 11:13:35 2002 -->
<pre>End run.  Reset alpha.  New run number is 143738.</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 23 11;18;09 comment by...Greg -->
<pre>It occurs to me that we did not reset CLIST after
powering down the crate to move RECES.  I end the
run to go and reseat CLIST!
</pre>


<!-- Date: Tue Apr 23 11:19:57 2002 -->
<pre>CLIST reseated.  Run 143739</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 23 11;26;29 comment by...Greg -->
<pre>Alpha still claims L1 not sending for buffers 0,1, 2, and 3.
I end run and change the "little black jumper" currently 
in slot 9.

</pre>


<!-- Date: Tue Apr 23 11:27:17 2002 -->
<pre>New run. 143740</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 23 11;36;37 comment by...Greg -->
<pre>Still errors from L1 and other cards.
Here are some of the claims:
======================================
Error: Number of Iso MBus words 4096
Error: Number of L1 words sent is 1106 for event 331580 
buffer 0 buffswitch  - it then proceeds to print them all
out!
==========================================================
Peter arrives to help.  He is hacking some code to print out 
number of words sent from each board.

</pre>

<!-- Comment: Tue Apr 23 11;50;31 comment by...Stephen -->
<pre>Whenever the number of MBus words is reported as being so large
it is probably due to a failure in reading the register that
says how many words were sent. If another board was sending
data while the register was being read, then this could give
this problem.</pre>


<!-- Date: Tue Apr 23 11:51:11 2002 -->
<pre>New run.  Setting SVT mode to "blank". 
Run number 143741.  Peter loading some secrect  
code ny hand. 
</pre>
<!-- Author: Greg -->
<!-- Comment: Tue Apr 23 12;05;40 comment by...Greg -->
<pre>After intermittently not sending any data for 
many events, the L1 board sends 498 words!
==============================================
L1Mon: saw 498 L1 DMA transfers, expect 1 (buffer number 0)                     L1Mon: Dumping data for &gt; 1 word.                                               Word       upper 32 bits  lower 32 bits                                            0: 0x00000000        0x00000008                                                 1: 0x00000000        0x00000000                                                 2: 0x00000000        0x00000002                                                 3: 0x00000000        0x00000000                                                 4: 0x00030000        0x00030000                                                 5: 0x00030000        0x00030000                                                 6: 0x00030000        0x40030001                                                 7: 0x00030000        0x01030080                                                 8: 0x00000000        0x00031000                                                 9: 0x00020000        0x00830000                                             </pre>

<!-- Comment: Tue Apr 23 12;19;17 comment by...Greg -->
<pre>Had another occurance wher L1 sent 498 events.
Ending run
</pre>


<!-- Comment: Tue Apr 23 12;20;58 comment by...Stephen -->
<pre>If the error is caused by a board sending at the wrong time,
it should be possible to trigger on startload being high
and dstrobe being low.</pre>
<!-- Comment: Tue Apr 23 12;28;06 comment by...Stephen -->
<pre>To trigger on events where L1 doesn't actually send anything,
setup the scope to trigger whenever startload and mod_done(7)
are both low for time&gt;500ns. Also look at bossreq and boss.

Alternatively, if the debugflag is set in the alpha code, one
can trigger on mod_done(0) going low. This will check errors
for the previously sent data.</pre>
<!-- Date: Tue Apr 23 12:20:14 2002 -->
<pre>New run.  Extra1 =0 
</pre>
<!-- Author: Greg, Wittich -->
<!-- Comment: Tue Apr 23 12;21;02 comment by...Greg -->
<pre>Run number 143743</pre>

<!-- Comment: Tue Apr 23 12;27;06 comment by...Greg -->
<pre>Same errors.  Going to hook up the scope.
</pre>



<!-- Date: Tue Apr 23 17:03:45 2002 -->
<pre>Run 143762 -  Bottom crate. 
(Finally) saw on the scope that L1 is not 
sending data sometimes.  Ending run to put big  
delays in TRACKboards to see what happens.</pre>
<!-- Author: Greg  -->
<!-- Comment: Tue Apr 23 17;04;32 comment by...Greg -->
<pre>and Peter, Peter, Masa, Matt</pre>


<!-- Date: Tue Apr 23 17:05:32 2002 -->
<pre>Run 143767 - 24 and 34 us delays in TRACK boards. 
Fake RECES data. 
</pre>
<!-- Author: GPPMM -->
<!-- Comment: Tue Apr 23 17;09;24 comment by...Greg -->
<pre>Correction.  Masa is running standalone test with 
fake data.  When he is done we will run with the 
delays.  There is no run 152767</pre>


<!-- Date: Tue Apr 23 17:37:30 2002 -->
<pre>Now run with track delays - Run 143770.   
Still Level 1 sometimes does not send. 
It does not even send out BOSSREQ. 
I am finally convinced something is wrong with 
it!  Taking it out for studies ... 
Masa is going to run standalone RECES test and then we 
are going to go back to the top crate and try to  
fix the Clustering problem following Monica's  
suggestions. 
</pre>
<!-- Author: Greg, Masa, Matt -->
<!-- Comment: Tue Apr 23 17;43;34 comment by...masa -->
<pre> I test the Reces boad by runing the diagnostic code just
read/write the Reces boad using alpha. It works no problem 
as fas as I can check.</pre>


<!-- Date: Tue Apr 23 17:50:25 2002 -->
<pre>Back in top crate.  First we will see if we can run  
and reproduce the Cluster errors.  Then we will try  
Monica's two suggestions. 
==================================================== 
- used a different cable. There should be a spare cable  
in the L2CAL spare 
cable box in the shelves behind the ACE terminals.  
Note that you will have 
to change the connections to both LOCOS in b0l2cl04 and  
b0l2cl05 since the 
cable is doubled-up on the CLIQUE end. 
 
- change CLIQUE with spare. You will have to make sure that  
the PROM1 prom 
is moved frm the old to the new board and that the JP1, JP2  
and JP3 
jumpers are connected as in the old board (which I think are  
all not connected). 
</pre>
<!-- Author: Greg and Masa -->


<!-- Date: Tue Apr 23 18:59:24 2002 -->
<pre>One success today!  Run 143783 
====================================================== 
We swapped to the spare LOCOS to CLIQUE cable and the 
cluster errors went away.  We swapped back to the  
original cable and still saw no errors.  We conclude 
the cable was poorly seated or something. 
</pre>
<!-- Author: Masa and Greg -->
<!-- Comment: Wed Apr 24 08;02;23 comment by...Monica -->
<pre>You are my heroes!</pre>


<!-- Date: Thu Apr 25 10:21:09 2002 -->
<pre>Trying to fix the L1 interface board. 
Starting with L2 tortur and SEVB. 
All interface boards in run. 
</pre>
<!-- Author: Greg, Karen, Monica -->


<!-- Date: Thu Apr 25 12:11:23 2002 -->
<pre>Greg and Karen asked me to modify the spare Magic termination board with testpoint so that it can be used on the bottom crate. 
I removed terminations for BOSS* and BOSSREQ* and I shorted together BOSSGRIN and BOSSGROUT. Now the board is labelled and should be ready to go in slot 21 of top crate.  
Also, I visually inspected the reser logic TTL CLIST. Found nothing wrong still I disconnected one unused old input to FPGA. 
After I powered up bottom crate, CLIST now seems to be responding to VME without having to do hot insertion.</pre>
<!-- Author: Monica -->
<!-- Comment: Thu Apr 25 14;23;37 comment by...Monica -->
<pre>We find sometimes you still have to do a hot insert.
</pre>


<!-- Date: Thu Apr 25 14:26:01 2002 -->
<pre>Found that chip U5 on the TTL L1 interface board was dead. 
This chip brings in CDF_L1A and the two buffer address  
bit from the J2  backplane.  We replaced the chip and  
are trying to make a run now, 
</pre>
<!-- Author: Greg, Peter, Monica, Karen -->
<!-- Comment: Thu Apr 25 15;02;06 comment by...Greg -->
<pre>Run 143860 - Everybody have a look!</pre>


<!-- Date: Thu Apr 25 15:48:40 2002 -->
<pre>L2 Plan 
 
1. Run using bottom crate (with delays) as the default unless  
there are lots of problems running the crate. Problems could 
be frequent Decision timeouts or a high level of errors in the 
Reces boards. 
 
2. If bottom crate works fine over the weekend, then starting on 
Tuesday we would remove at least some of the delays and see how 
the system works. 
 
3. The shift crew is using L2 tagset 22. If that keeps crashing, 
we will need to go back to tagset 20.</pre>
<!-- Author: Stephen -->
<!-- Comment: Thu Apr 25 15;52;17 comment by...S -->
<pre>This was the plan discussed at today's L2 meeting.</pre>