<!-- Title: 19:42:15  Tue Jul 17 2001 -->
<!-- Crewchief: L2 and SVT folks -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Time for a new page</pre>


<!-- Date: Tue Jul 17 19:43:04 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-48&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-48' width='300'></a></center>
<!-- Author: Matt and David -->
<!-- Comment: Tue Jul 17 19;51;53 comment by...Matt and David -->
We captured the problem of the BOSS hang on the scope by
triggering on START_LOAD and BOSS both being high.   One
can see the Fred board grabbing BOSS but does not issue
any DSTROBE.  However, it looks like DSTROBE changes level a little, possibly corresponding to a buffer being enabled.  If you look at a good event, you see the same level shift before DSTROBE goes low.  
<p>
We know this is the L1 board since that is the one which always
goes first and we also see the level shift in DSTROBE at exactly
the same time as it usually comes.  Also, hitting reset on the
other boards does not release BOSS.
<p>
There is something special about these events.  It always comes
immediately after another L2 cycle.   Ie, START_LOAD is only 
high for 2.4 usec, and for "good" events it is high for about
100 usec between cycles. 
<p>
While this is being fixed, can someone make a torture test that
does not need Fred bits?  Another question is why does the L2 processor not time out under these conditions? (It keeps asserting START_LOAD over and over even though BOSS is asserted.)

<!-- Comment: Tue Jul 17 20;31;05 comment by...Stephen -->
<pre>We should also look at the L1A line (from the J2 backplane)
on the scope - as we might see one coming close in time to
when this hangs.

The reason alpha doesn't hang is because it doesn't pay any 
attention to boss. When we get our error code working, then
it will cause a CDF error when the boards don't send any data
for an event.

I'll think about some way to implement L2 tagging mode such
that it can prescale during tagging mode. This will be 
independent of the L1 bits.

</pre>


<!-- Date: Wed Jul 18 03:32:19 2001 -->
New "smart" HF coldstart, using FIB channel enable data, 
seems to work.  One exception is that hf_w10_b0b1 is in 
run mode, but it seems to see DAV always asserted; need 
to track this down.   
<p> 
I masked off SVT_ERROR for all HF 
error bits, all XTFA error bits, all Merger error bits 
except lost sync (I hope; I set the mask to 0x100), and 
all TF error bits.  I don't know why XTFA was asserting 
error sometimes; it didn't happen until the run had been 
going for a while.  The only TF to assert error was the 
one in wedge 10, which perhaps was pulling error because 
it saw lost sync on input?  The XTRP emulator seems to 
assert SVT_ERROR during each L1A, which I think started 
out as a trick for me to make the LED blink--oops. 
<p> 
By midway through run 120709, I finally had all of the 
error LEDs off, except for the XTRP emulator in crate 7. 
Run 120710 should be good data, but it ended after about 
12K events (with no hardware sync error detected) when 
b0svt07 spontaneously rebooted itself?!  Run 120711  
should be good, and has been running for about 10 minutes 
now, with no error LEDs. 
<p> 
Because the "smart" cold start code currently affects only  
the HFs, not the Mergers, I needed to turn off the Merger 
inputs corresponding to the disabled HFs, to avoid generating 
sync errors.  Unfortunately, at least mrg_w04 and mrg_w09  
still have old firmware, which means I had to pull the boards 
and change the DIP switches. 
<p> 
I notice that someone wrote "hw=bruins" under SVT on the 
whiteboard.  Because of this, the first couple of runs I 
took had no SVX data flowing through SVT, because the 
normal SVT configuration (default.hwset) was not being 
loaded.  I intended SvtMode to be set to a non-null value 
only while some expert is sitting at the controls, not 
while the system is left to normal use.  Otherwise, the 
Ace's saved runsets get polluted with special-case SVT 
configurations, and SVT experts lose the ability to choose what 
is loaded into SVT by default. 
<p>  
<!-- Author: Bill -->
<!-- Comment: Wed Jul 18 04;14;04 comment by...Bill -->
I also noticed (as other people have reported) that the Mergers seem to include error bits
seen on the input cable in their decision to pull SVT_ERROR.
This is convenient for me, but I gather that it may be
different from the behavior of other SVT boards.  (I wouldn't
know, since Hit Finders don't have to make this distinction!)



<!-- Comment: Wed Jul 18 17;13;20 comment by...Bill -->
I should have also noted that I turned off the XTRP track
list in the SVT readout, so that the presence of single-word
packets (other than EE) does not interfere with L2.

<!-- Date: Wed Jul 18 12:17:15 2001 -->
Alpha was crashing.  For some reason SPYMON was running in  
crate 7 which we know causes L2 to crash.  This must have been done recently since it was working fine this morning.   We killed it with "td". 
<!-- Author: David and Peter Wit. -->
<!-- Comment: Thu Jul 19 12;00;31 comment by...Bill -->
As noted below, a Hit Finder firmware change has been made
that eliminates the interference between this HF test mode
and the spymon task.  If this interference is seen again,
then there is a bug somewhere, and it should be reported
to the SVT pager.  (The symptom was that the spymon task
would asynchronously rewind the Hit Finder that was driving
test data out to L2, which would cause some events to
contain tracks with Nwords different from 7, which would
cause the L2/track board to get into a state that crashes
the L2 alpha.  The easiest fix was to make the HF immune
to the spymon task's activity.)

<!-- Date: Wed Jul 18 16:26:29 2001 -->
<pre>We looked at the L1 interface failure problem. 
 We couldn't reproduce it but do have a good idea of what the 
failure is. We notice a noise glitch on the ddone* line 
right after boss was asserted by the L1 board. According 
to Bill Haberichter, this noise could cause a reset of the 
dstrobe* latch causing dstrobe* to not be asserted. In that 
case, the L1 board would hold boss high and not send any data. 
He is making new firmware where if the board doesn't send any 
data 1us after getting boss, then it will reset itself and  
give up the bus without sending any data. This firmware should 
arrive on Thurs. 
 
One question is whether all of the additional boards plugged 
in caused this problem. We saw another feature where there 
would be a short ddone* strobe after both the L1 and Clist  
boards had sent their data. This was definitely a driven signal 
and may be caused by the Reces boards. We will verify this  
later but for now it doesn't seem to be causing a problem.</pre>
<!-- Author: Stephen, Greg, David, Matt -->


<!-- Date: Wed Jul 18 17:12:06 2001 -->
Firmware for HF in b0svt07 slot 6 has been updated to ignore 
VME spy buffer reset when in "ospy test" mode, so that spymon 
will no longer interfere with using this SVT as a source of 
test data.  The change is in, but has not yet been tested. 
<p> 
svtvme_fer.c has been updated so that 'default.hwset' is  
replaced by 'nosvx.hwset' when SVX is not in the current 
partition and no other 'hw=' value has been provided in 
SvtMode.  The change will take effect next time crates are 
rebooted.  It has not yet been tested. 
<p>  
<!-- Author: Bill -->
<!-- Comment: Thu Jul 19 11;55;22 comment by...Bill -->
These changes seem to have been a success.  The second
definitely works; the first seems to have worked, but I
never had a chance to test it explicitly by seeing that
the board ignores the VME command while sending test
data.

<!-- Date: Wed Jul 18 18:35:07 2001 -->
<pre>We modified the L2 code to readout the Reces board during 
running. The executable is L2_TORTURE_v-1_reces_B0L2PR00. 
We would get DONE TIMEOUTS from the L2 crate somewhat regularly. 
When running with L2 cutting, this was especially true. 
 
We would see start_load go low for 200ms with data being 
sent by the interface boards but reces not being read.  
That would be followed by several regular start_load events 
where reces was read. 
 
 
We were able to take some runs with L2 accepting all of the  
time and only reading reces once there was a L2A decision. 
The data read matched quite well was was in smxr, but needs 
to be looked at in more detail. 
 
In summary it was a good start for reading out Reces, but 
see below for the bad Reces news.</pre>
<!-- Author: Stephen, Masa, Karen -->


<!-- Date: Wed Jul 18 18:37:30 2001 -->
<pre>We looked at the noise on DDONE* with Reces in and pulled out. 
The noise on DDONE* is due to having the Reces boards in. 
As discussed above, this is the cause of the L1 board holding 
boss high.</pre>
<!-- Author: Stephen, Greg -->


<!-- Date: Wed Jul 18 18:39:07 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-50&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-50  width='300' align='top'></td><td>Backplane signals with Reces</a></td></tr></table>
<!-- Author: Stephen -->


<!-- Date: Wed Jul 18 18:39:46 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-51&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-51  width='300' align='top'></td><td>Backplane signals without Reces</a></td></tr></table>
<!-- Author: Stephen -->


<!-- Date: Wed Jul 18 18:42:57 2001 -->
<pre>I have taken all svt crates for testing</pre>
<!-- Author: Alexei@Franco -->


<!-- Date: Wed Jul 18 18:47:00 2001 -->
SvtMode = 'hw=bruins' will no longer work.  See recent shift 
logbook entry.  SVT will now automatically send test data if 
SVX is missing and process SVX data if SVX is present.
<!-- Author: Bill -->


<!-- Date: Wed Jul 18 22:31:46 2001 -->
I noticed that SPYMON is running in SVT crate 7.  It does not seem to have any adverse effect on Level-2 now.
<!-- Author: David -->
<!-- Comment: Thu Jul 19 12;02;36 comment by...Bill -->
That's good news.  Thanks for noting that!
<!-- Date: Wed Jul 18 23:13:36 2001 -->
<pre>Runs taken tonight for L2 torture tests.  All runs were taken 
with L2 cutting, trigger table L2_TORTURE[1,150], SVT sending 
tracks from their tests, no COT high voltage.  The rates given 
in the table are L2A, in the runs where we were successfully 
cutting the L1A rate was approximately 2 kHz. 
 
run     Nevts   analyse?  comments 
----------------------------------------------------------------- 
120793  225k    NO        Cutting at L2.  Early in the run the 
        123 Hz            alpha reported L1interface sending 0 mb 
                          words in 3/4 of the events, and filling 
                          the TL2D bank with junk from L1.  This 
                          could NOT be seen on the backplane: L1 
                          took BOSS and sent data every event. 
                          The alpha also began accepting all the 
                          bad events.  Checking the L1 cables did 
                          not help, so we power cycled the crate. 
 
120798  59k     YES       L1 and XTRPlist were good.  Clist was 
        121 Hz            reporting num of clusters mismatches 
                          almost every event.  SVTlist unknown  
                          while TrigMon code is updated. 
 
120802  16k     NO        We mysteriously were accepting every 
        132 Hz            L1A.  Note that in this run Clist had 
                          no errors. 
 
120803  359k    YES       Cutting again.  Clist now has Nclust 
        119 Hz            errors.  L1, XTRP, and SVT (with new 
                          code) check out okay.  6.8 MILLION L1 
                          ACCEPTS in 50 minutes! 
 
120806  92k     MAYBE     In both this run and 120805, L2 was  
        125 Hz            accepting every L1A on the L2_JET_5 
                          trigger.  Deadtime was at 95%.  HRR 
                          cleared this problem.  Note that when 
                          we were essentially tagging, Clist was 
                          fine, after the HRR it began to have 
                          errors again.  Also note: solenoid and 
                          COT were on in this run. 
 
General comments: when we have high L1A rate, Clist has many 
errors.  XTRPlist still sees bc errors at a low rate that can 
be explained by the known XTRD bc problem.  New code in SVTlist 
solves the problem that TrigMon had with even numbers of tracks 
from SVTlist, but we now see other errors, such as multiple 32  
bit track words from SVTlist in 1 mb transaction that have the 
end event bit set. 
 
Leaving the system to the ace and silicon people with all 4  
boards in and instructions to run in cutting mode.</pre>
<!-- Author: Greg, David, Matt -->


<!-- Date: Thu Jul 19 01:29:07 2001 -->
<pre>I have finished the testing of svt crates.</pre>
<!-- Author: Alexei -->


<!-- Date: Thu Jul 19 10:39:02 2001 -->
<pre>Comments from 7/17, 7/18 L2 meeting 
 
1. Need to develop L2 error handling code so that Alpha can 
pull cdf_error when interface boards stop sending any data. 
 
2. There was a problem with the svtmode being changed in run 
control by non svt experts when in fact some other things  
needed to be changed in the svt for things to work properly. 
The svtmode should only be changed by an svt expert!! 
(Note that since the meeting, Bill has configured the svt 
to operate properly either with or without svx.) 
 
3. Run control should send down additional information to the  
Alpha depending on what interface boards are in the crate.  
That way the alpha will know to ignore the mod_done lines for 
those slots. That will also make it easier for the system to 
identify which board/system is no longer sending data. 
 
4. All 48 Reces cables have been tested and are okay. The 2  
problem cables were a bad smxr board and a mislabeled cable. 
 
5. (As noted in this week's log entries) The board that held 
boss high was identified as the L1 board, which was responding 
to spurious ddone* signals which in fact were caused by the 
Reces boards being plugged in. New firmware is being made 
for the L1 board. 
 
 
 </pre>
<!-- Author: Stephen -->





<!-- Date: Thu Jul 19 16:44:50 2001 -->
<pre>Looked at DDONE* with ReCes some more.   Basically, i wanted 
to see if the spurious glitches in DDONE* were due to 
one particular board or to one board address.  I 
plugged in each board separatly and  
all 4 ReCes boards generated the spurious DDONE* 
signal.  WIth only one or two boards in, I only 
saw one spurious DDONE* spike (after the first DSTROBE). 
 
 
SO it looks like the generation of the spurious DDONE 
is not  because of one board or one board address.   
On Monday, John Dawson and Bill Habericter  will be here.</pre>
<!-- Author: byrum,saltzberg -->


<!-- Date: Thu Jul 19 18:48:22 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-52&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-52  width='300' align='top'></td><td>Clever SVT people.</a></td></tr></table>
<!-- Author: Matt -->


<!-- Date: Thu Jul 19 22:43:12 2001 -->
<pre>Looked into cluster merging problem. I can reproduce it quite easily when all DCAS are in by simulating two clusters in the same crate (as the no. of towers of the second cluster increases, I can see that 3 instead of 6 clusters are received by CLIST). 
 
When I take some DCAS out to attach Logic Analizer to LOCOS, the problem almost disappear. Still I triggered on two instances of merged clusters, both for pass 0 and pass 2 (while pass 1 was OK). It looks like the two seeds are found as separate but after the first DCAS with seed is selected, only one propagation takes place and the second seed is lost by the time the second SEED_REQ is sent....looks similar to cluster hanging problem! Will look more in details in Michigan. 
</pre>
<!-- Author: monica -->


<!-- Date: Thu Jul 19 23:00:07 2001 -->
<pre>Run L2_TORTURE table with new CLIST (#2), XTRP + SVT interface and L1 Interface. L1A~3KHz, L2A~110Hz. 
&lt;LI&gt;  
Run 120880 has Alpha in tagging mode, run 120881 has Alpha in cutting mode. 
&lt;LI&gt; 
Found a stuck bit (bit 19) in the the first L1 mask word in the alpha (i.e. the copy of the first FRED L1 mask word). 
&lt;LI&gt; 
CLIST looks OK in the  run with the Alpha tagging but it looses sync in a similar way to the one observed for run 120803 of last night after 11614 L2A's when the Alpha is cutting. 
&lt;LI&gt; 
SVT, XTRP and L1 interface (except for the stuck bit) look OK.</pre>
<!-- Author: monica and matt -->


<!-- Date: Thu Jul 19 23:27:42 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-53&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-53  width='300' align='top'></td><td>Plot of stuck bit from run 120880.</a></td></tr></table>
<!-- Author: Matt -->


<!-- Date: Fri Jul 20 11:23:39 2001 -->
<pre>Current L1 interface board status. 
 
SN_02 is the serial number of the board 
known to work and curently working in b0l2de00. 
It has firmware versions "Yellow D" and "Green O". 
 
SN_01 is the spare.  How well it works in b0l2de00 
will be determind today and then given to Peter for 
noise tests in b0l2de01.  It has firmware versions 
"Yellow D" and "Green O". 
 
I believe the two cards have had the same number of  
modifications to beat down the noise problems but I am 
not sure.  Will try to re-create the modification  
history of the two cards (with Bill H., Stephen, and  
Peter Wit.) and post in the log book later. 
 
There is new firmware ("Green P")to make the L1 interface  
board less susceptible to noise on the DDONE line but I  
wait until there have been more RECES refinements  
before putting this in the boards.</pre>
<!-- Author: Greg -->


<!-- Date: Fri Jul 20 15:18:05 2001 -->
<p>Found that the L1 interface board was plugged into the same slot as one of the tracklist boards.  Moved it to slot 12.</p> 
 
<p>Managed to get the second L1 interface board (SN01) working in the top crate. It appears to be working reliably from the point of view of looking at the MB backplane. Currently running for ~10K events without any hangs (no other boards are sending data).</p>
<!-- Author: Peter Wittich -->


<!-- Date: Fri Jul 20 16:16:20 2001 -->
<p>Moved L1 interface board SN01 (spare) to the lower L2 decision crate, b0l2de01. Current run configuration is </p> 
 
<ul> 
<li>central calorimeter w/o ccal00</li> 
<li>L1 cal crates</li> 
<li>L2 global</li> 
<li>L2 decision 01</li> 
<li>FRED and L2PROCESSORS selected</li> 
</ul> 
<p>This worked briefly at some level - I saw that the processor loaded the executable, and was able to get events going (judging from the RC counter.) However, I never saw the `select' light on the L1 interface board blink. </p> 
<p>In subsequent runs, we never managed to load the L2 executable: we're always getting BOOTP errors: 
 
<pre> 
Booting file is l2exe/Cosmics_All_v-2_B0L2PR00  
Digital Semiconductor DC21143 (0): Initializing 
        Hardware address = 00-c0-f0-4b-c4-6d 
        Checking ports, 10BaseT, 10Base2 (BNC), MII 10/100Base-Tx 
Attempting BOOTP... 
Bootp Error, retrying (Press Control-C to abort). 
......No response 
 
BootP failed 
 
 File  failed to load 
Booting file is L2autoconfig </pre> 
 
<p>Not sure how to solve this, nor what it meant that we were getting data but no SELECTs on L1interface board.  Of course, it doesn't work like this: we get "alpha no longer processing" errors immediately.</p>  
<!-- Author: Peter Wittich -->
<!-- Comment: Fri Jul 20 16;19;21 comment by...Peter Wittich -->
By the way, the crate configuration at this point was:
<ol>
<li>Alpha in slot 3 (we did enable this one in the hardware db)</li>
<li>L1 board in slot 7 (all jumpers in place)</li>
<li>RECES in slot 18 (not all jumpers in place between it and slot 3)</li>
</ol>
I guess it is possible that the jumpers in the slots between the alpha and the L1interface board aren't all making a good connection. I should've checked the START_LOAD on the backplane when I had a chance.....


<!-- Comment: Fri Jul 20 17;16;48 comment by...Greg -->
<pre>I don't think the SELlight ever workedd on that one...
</pre>
<!-- Date: Fri Jul 20 17:04:44 2001 -->
Tried Tracklist prototype - saw same large noise level 
 
<a href="/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-21&button=yes">as we saw before.</a> 
 
Abandon using this board for now for noise studies, though it does have a lot of noise :)
<!-- Author: David S & Peter W -->



<!-- Date: Fri Jul 20 19:12:29 2001 -->
Took many runs (120003-120011).  The only interesting one is 120005, in which L2 was cutting and we changed the prescales by hand to run at higher rates (~3.5 kHz) than we have done previously.  At this point the alpha hung.  For all subsequent runs the alpha stopped processing early in the run.  We sent the results from minicom and alpha_check to Stephen.
<!-- Author: David and co. -->


<!-- Date: Fri Jul 20 19:23:14 2001 -->
In the process of trying to debug this problem we killed 
SPYMON in SVT07.   It was not causing the problem, so that wasn't it.   I don't know how to restart it, sorry.
<!-- Author: David -->
<!-- Comment: Sat Jul 21 18;07;09 comment by...Luciano -->
<pre>You should have called an SVT expert in. If none is around
please call the SVT pager.</pre>


<!-- Comment: Mon Jul 23 09;56;32 comment by...Ashmanskas -->
<table border=0><td><tr><td bgcolor=#e0e030>This was completely inappropriate.  Shall we begin
routinely unplugging L2 boards in random, superstistious
attempts to debug SVT problems?
</td></tr></td></table>
<!-- Comment: Mon Jul 23 11;37;07 comment by...David -->
Sorry, in the past (a few days earlier) this was an allowed operation.  I must have missed the new instructions.  
Since it had previously been the cause of my losing a few days of my life that I don't get back, it didn't seem so superstitious.
<!-- Comment: Mon Jul 23 13;40;42 comment by...Bill -->
Just sorry, without mitigating, would be adequate.  
You were in fact at the L2 meeting at which Luciano
explicitly asked that any change to the SVT crates or configuration, including tasks running on crates,
should be OKed first by calling the SVT pager.  (This is 
the same meeting that generated Stephen's minutes,
above.)  If we all are careful to communicate with each other, (we're getting better at it) 
months of all of our lives can still be saved by keeping integration going smoothly.  (We can defer discussions of
rationalism to a different thread.)

<!-- Date: Sun Jul 22 10:57:16 2001 -->
<pre>Put Isolist in slot 17 of de00 and shift crew ran  
cosmics_2_1_15 table.  All previous versions of firmware 
had the incorrect assumption about Clique Done, mentioned 
above.  So in many ways this version is back to square one.  
This one certainly worked better,  but the board appeared 
to get out of sync in the first multi-cluster event,  
although this may be a coincidence.  In terms of messing 
up the other boards,  we saw no evidence of holding boss 
high,  but in events without a cluster it appears that  
startload was very long,  indicating a possible problem  
with the moddone lines.   
 
Dawson/Haberichter are coming to fnal tomorrow to work  
on L1,Reces,Iso.   </pre>
<!-- Author: Steve Kuhlmann -->
<!-- Comment: Sun Jul 22 11;12;18 comment by...SK -->
<pre>Should have mentioned, I compared the Trigmon # diagnostic
words plot before and after Isolist was added,  and it
looked identical except for the new data in word #5 (Iso) 
that had a reasonable distribution.  Therefore other than
slowing things down due to Startload timing out there
was no gross evidence of a problem caused by adding 
Isolist.  In addition,  the scope signals also looked
reasonable.  It might be useful for other experts to 
look at run 121211 which is in the Look area to see 
if there are problems.</pre>


<!-- Date: Sun Jul 22 17:20:04 2001 -->
Continuing L2 torture tests. 
<p> 
Since Steve K. had success running with the alpha we tried again. 
This time we WERE able to keep the alpha going.  We ran with 
L2_TORTURE at 2kHz with the four boards.   They appear to be running as usual according to TrigMon. 
<p> 
We then wanted to increase the rate so we switch to L2_RATETEST. 
We also set the prescales to get high rate...this time 17kHz 
of L1A.  L1 interface was fine.  SVTlist went out of sync once but recovered with HRR.  Clist still needs the right code in TrigMon.  XTRPlist went out of sync nearly immediately.   We 
could see on the backplane occasions where XTRPlist did not 
send a word (by triggering the scope on ST_LOAD=hi and MODDONE=lo from XTRPlist). 
<p> 
Note for Stephen.  There may be something not right about the 
L2_TORTURE executable.   For this executable only we find 
that we always get 3 tracks in the XTRP section but the 
data are mostly zero.  This does not happen for COSMICS_ALL 
or L2_RATETEST and has been pretty repeatable. 
<p> 
Meanwhile good runs to look at at 17kHz L1A are: 121234, 121235, 121236.  
<!-- Author: Matt and David -->


<!-- Date: Sun Jul 22 18:34:08 2001 -->
For future reference here are the prescales we used for 
the L2_RATETEST table to make it run at 20kHz: 
<p> 
<TABLE border> 
<TR><TH> Bit </TH><TH>PRESCALE</TH></TR> 
<TR><TD>1</TD><TD>1,000,000</TD></TR> 
<TR><TD>4</TD><TD>1000</TD></TR> 
<TR><TD>6</TD><TD>1000</TD></TR> 
<TR><TD>3</TD><TD>5</TD></TR> 
</TABLE>
<!-- Author: David -->


<!-- Date: Sun Jul 22 19:07:26 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-54&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-54' width='300'></a></center>
<!-- Author: David -->
<!-- Comment: Sun Jul 22 19;18;27 comment by...David -->
Matt and I found (see above) that there were occasional events
where the XTRPlist did not send a word to the alpha and then became out of sync.   We set up the scopes so that we could look at DS from the XTRP and L1A from the tracer when this happened.   One can see that there is a L1A without a corresponding DS from the XTRP. This is presumably why we are going out of sync.
<p>
(Note that the DS signal looks a little odd above, but that
is just because I needed to get as long of a time slice as possible and the scopes only have 32K points.  On a finer scale they look okay and this is just due to undersampling.   Because DS is so narrow this is the maximum time slice we can grab without losing some DS's altogether.  BTW, the noise on the L1A is a feature of the tracer we've seen for a long time and is probably not relevant--although it makes the MagicBus backplane look good by comparison.)
<p>
Anyway it is interesting that this happens exactly before a pause
in the L1A's.   I don't know what causes these pauses.    Note
that Matt checked that that XTRD does not go out of sync with respect to the rest of CDF so the XTRP is sending its VME information out, just not the DS with an EE word up to us.
<p>
There really isn't anything I can think of to protect ourselves
other than having the alpha pull CDF_ERROR.   Note also that this was with 17kHz L1A's...we never saw this at lower rate.   We'll ask the XTRP people to look into this.
<p>
It would be interesting to see if the SVT people also see this in
their spy buffers at the same time.  This could be the source
of sync errors for them.



<!-- Comment: Mon Jul 23 10;00;12 comment by...Ashmanskas -->
<table border=0><td><tr><td bgcolor=#e0e030>Indeed it would, as this would cause a synch problem for us,
and this is exactly the reason we keep
svtspymon running in crate 7, which, we remind you, is
never, under any circumstances,
 to be interfered with in any way without
first calling the SVT pager, no matter what time of day 
or night.
</td></tr></td></table>


<!-- Comment: Mon Jul 23 10;28;36 comment by...Bill -->
Since a missing EE word from XTRP would cause SVT, L2/SVTlist,
and L2/XTRPlist to get out of synch, it certainly deserves a
CDF_ERROR.  I think we may be able to arrange to have SVT pull
CDF_ERROR when this happens, i.e. if there is a L1A with no
corresponding EE from XTRP after some programmable length of
time has elapsed.  What length of time would be appropriate?
100 usec?


<!-- Date: Mon Jul 23 10:10:17 2001 -->
<pre>I needed a 5A fuse, so I borrowed one from the L2/track board 
in the active L2 crate. 
</pre>
<!-- Author: Bill -->
<!-- Comment: Mon Jul 23 10;10;37 comment by...Bill -->
<pre>Just kidding.
</pre>




<!-- Date: Mon Jul 23 13:38:15 2001 -->
<pre>L2 debugging of Spikes on DDone generated by ReCes. 
 
We ran the Level2 torture test and observed the spikes on 
the DDone line with ReCes in.  The DDone signal is  
generated by an FPGA at the top of the ReCes board and 
is driven out onto the magic bus by a drive at the bottom 
of the ReCes board.  A trace equal to the length of the 
board separates the FPGA and the driver.  we observed 
that the DDone spikes were being produced by noise at the 
input to the driver.  We added a RC filter to the input 
of the driver and this killed the noise spikes on DDone. 
 
We then ran standalone code to verify that ReCes could 
still produce DDone.    
 
Bill will now modify all 6 ReCes boards by adding a filter 
at the input of the DDone driver.   We also  discovered 
by accident that the ReCes board does not have the BossIn 
and Bossout pins jumpered together so this fix will also 
be done.</pre>
<!-- Author: byrum,takano,dawson,habericter -->



<!-- Date: Mon Jul 23 13:53:56 2001 -->
<pre>Whoops - I mispelled masa's last name on the last entry. 
It should be Tanaka instead of (Takano)</pre>
<!-- Author: byrum -->


<!-- Date: Mon Jul 23 17:14:15 2001 -->
<pre>L2 Meeting Notes 
 
1. Test runs taken at higher rate (2Khz and 17 Khz) showed 
some problems. The Tracklist board would fail to send an MBUS 
word and get out of sync. This is due to a failure on the part 
of the XTRP to send data to that board.  The Clist board 
gets out of sync after about 10K events. In this case it will 
still send some data, but it is partially incorrect. This will 
be studied this week. 
 
2. The DDONE* glitch due to Reces has been diagnosed and  
a fix prescribed. (See above entry). Modified boards will be 
back for testing in a few days. 
 
3. A new  version of the Alpha will be arriving and will 
hopefully be tested at Fermilab the end of this week. 
 
4. Plan for beam is to start out in tagging mode but then do some 
tests with L2 cutting. Initially simply prescaling the L1  
triggers and then cutting on clusters if the Clist sync error 
is solved.</pre>
<!-- Author: Stephen -->


<!-- Date: Tue Jul 24 13:26:37 2001 -->
Working on hardware enforcement of XTRP BC number, i.e. to 
pull CDF_ERROR when XTRP forgets one EE or sends wrong BC 
in EE, since this is currently a problem for L2 (and probably 
for us).  Also doing some general clean-up. 
<p> 
Removed TF 3 from slot 11 of b0svt07.  It was there for a 
special test for Jane Nachtman, a few months ago, which 
needed data to be sent at low speed from SVT to a prototype 
L2/track board.  The 4MHz oscillator has been removed from 
that TF.  As I was unable to locate a 30MHz oscillator, I 
used a 28MHz oscillator and left this TF in the test stand, 
so that we can make sure it is happy.  (We should also find 
a 30MHz oscillator for it.) 
<p> 
Took Merger 25 from test stand, loaded Standard_2001  
firmware, placed in slot 11 of b0svt07.  b0svt07/10 is 
now "mrg_xtrpcheckbc" and b0svt07/11 now takes over role  
of "mrg_xtrpemu" previously played by b0svt07/10. 
(More explained in next entry.)  
<!-- Author: Bill, Alex -->
<!-- Comment: Tue Jul 24 17;21;09 comment by...Bill -->
<p>
I also reprogrammed XTRPemu so that it would no longer
assert SVT_ERROR on every L1A. (Oops.)
<p>
I made coldstart for "mrg_xtrpcheckbc" notice whether R_C
sends "noxtrp=1", in which case it disables input A, since
no data would be expected from XTRP.
<p>

<!-- Date: Tue Jul 24 14:50:34 2001 -->
Loaded Standard_2001 firmware into Merger boards 25, 10, 17, 20, and 18, with some supervision from Stefano.  
<!-- Author: Bill -->


<!-- Date: Tue Jul 24 15:54:17 2001 -->
<pre>Steve K. pulls out the track list board for his iso-tests. 
The board is not getting anything good from XTRP. 
</pre>
<!-- Author: Greg -->


<!-- Date: Tue Jul 24 16:52:32 2001 -->
<pre>Sunday with everything working well startload was 3us or so, 
today before touching the system it was always 20us or 
longer.  We thought this may be due to no tracks in  
xtrp,  so after consulting with Matt we unplugged them. 
This didn't seem to help.  We plugged in Isolist anyway,  
and things seemed to run well with it,  but startload was 
still 20us.   
 
A different logic change not related to moddone was to hopefully 
improve the out-of-sync problem.  The data taken today 
was no better than Sunday's.   
 
It would be good while I'm gone for 9 days to test the  
moddone out of Isolist and if it works run with it as  
much as possible to find conflicts. 
 
 
Also,  4 modified RECES boards are here ready to be tested.</pre>
<!-- Author: Steve K -->
<!-- Comment: Tue Jul 24 16;53;59 comment by...sk -->
<pre>As David was returning tonight we left the track-related
boards unplugged.</pre>


<!-- Date: Tue Jul 24 17:12:40 2001 -->
<pre>I am taking SVT crates 00-05 for testing a new version of AM diagnostics. Will probably be done in ~ 1 hour</pre>
<!-- Author: Luciano -->
<!-- Comment: Tue Jul 24 17;35;43 comment by...Luciano -->
<pre>I'm done now.</pre>


<!-- Date: Tue Jul 24 20:15:17 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>Bunch counter number sent by SVT was off by one 
when SVT was running stand-alone (i.e. no SVX). Now it agrees. SVT has been set up to pull 
CDF_ERROR if SVT's own BC disagrees with BC sent by XTRP, to catch possible low-rate 
problems seen by L2 XTRPlist board. Thus, <b><u>ERRORDISABLE is no longer 1 for the 
TRACER in b0svt07</b></u>. The intent is that only this synchronization problem can currently 
cause SVT to pull CDF_ERROR.</td></tr></td></table>
<!-- Author: Bill -->


<!-- Date: Tue Jul 24 21:30:29 2001 -->
L2 rate test was a very useful test of our lost-sync detection. 
When initially the XTRPemu's BC number was off by one, the 
SVT Merger pulled CDF_ERROR immediately after each H/R/R and 
at the start of the run.  Once that was fixed, we ran for 
10M+ events at ~10kHz L1A rate and detected no mismatches 
between the XTRP cable and the XTRPemu. 
<p> 
So it's looking more as if our sync problem is in the silicon 
handling, not the COT track handling.  This is consistent with 
Roberto's observation of the sync problem before we included 
XTRP data into SVT. 
<p>
<!-- Author: Bill -->


<!-- Date: Tue Jul 24 23:58:33 2001 -->
XTRPlist studies 
<p> 
SVT, L2, and XTRP guys converged to track down the missing DS from XTRP that supposedly caused the XTRPlist board to go out of sync.  See David's <a href= 
'/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=13&frame=2&anchor=19:07:26'>previous entry</a>.  It turns out that we were actually undersampling with the scope in the screenshot that looks like we were missing a DS, and upon more detailed checking we were not missing any DSes.  SVT confirmed with their new CDF_ERROR code that they didn't see any events where XTRP did not send them a DS. 
<p> 
The conclusion is that we screwed up and the problem is probably somewhere in the XTRPlist board.  We began looking at backplane signals and our FPGA logic, but at the high rate (L1A = 16 kHz) it was very hard to trigger on the error condition in time to see anything on the scope.  Further studies will be needed. 
<p> 
We have 2 requests.  We really need the error bit set by the alpha if a board sends no mb words, to definitively trigger on the sync error condition.  We also would like a L2 exe with a faster timeout than 60 usec.
<!-- Author: Matt and David -->
<!-- Comment: Wed Jul 25 00;23;49 comment by...David and Matt -->
<pre>We mean timeout less than 1 msec.</pre>
<!-- Date: Wed Jul 25 00:19:42 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-55&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-55  width='300' align='top'></td><td>DDONE signals with 4 reces boards in L2 decision.</a></td></tr></table>
<!-- Author: Matt and David -->
<!-- Comment: Wed Jul 25 00;26;29 comment by...D a v i d and ttaM -->
These are 4 of the reces boards with the mods on the input to the DDONE driver on each reces board, that was supposed to remove a spike on DDONE that was hanging the other boards.  See Karen et al's <a href=
'/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=13&frame=2&anchor=13:38:15'>previous entry</a> for details.  In the short test we were able to take due to systemwide problems (21 events), the fix appears to have worked.


<!-- Date: Wed Jul 25 23:22:08 2001 -->
<pre>Notes from 7/25 L2 meeting 
 
1. New Alpha board has been tested and works quite well. 
(Although there is some flakiness possibly associated with 
a bga socket). The alpha will be tested at Fermilab on Thursday. 
 
2. Reces boards were tested with the ddone* noise fix, and the 
fix is successful. Now need to test Reces with test runs. 
 
3. Clist out of sync error occurs when only 1 of 2 MBus words 
are sent. This is still being studied. 
 
4. The XTRPlist board "goes crazy" at high rate (17Khz) and gets 
out of sync. To trigger on the error, will need Alpha code to 
compare the XTRP and SVT BC values and drive some line to  
trigger the scope/LA.  Note that the SVT board doesn't show any 
problem. 
 
5. Thursday's work will include testing the new Alpha at B0, 
doing some Reces test runs and continuing with the XTRP and Clist 
sync errors. 
                Stephen</pre>
<!-- Author: Stephen -->


<!-- Date: Thu Jul 26 05:27:57 2001 -->
<pre>UPDATE ON CLIST TESTS of 7/25/01  
 
- found possible problem with noisy DSTROBE* and/or DDONE* when     running at highest possible rate with one cable out (i.e. 3       simulated clusters, one per pass). High rates (~650Hz) are        reached by not requiring any SEVB or HEVB and using               L2_RATE_TEST trigger table (i.e. 17KHz of ZERO_BIAS events). 
 
  After fixing the FPGA (downloading FIFO_ADDER_3_13 via xcehcker) we can run for over 2M events without 
  any error detected on the Logic Analizer. 
 
- when running with real cosmics at high rates (L2_RATE_TEST or     L2_TORTURE) with Alpha in cutting mode, loss of sync error with 
  partial FIFO update shows up again (see runs 121687/89/90).  
  Also noticed in 2 occasion DDONE* from CLIST disappearing         altogether. Tried to run with CLIST#1 (121694) and observed same behaviour. Decided to leave CLIST#1 here (with old FPGA design) and take CLIST#2 back to Michigan to discuss situation with Myron and John. 
 
- pushed XTRP and SVT interface boards into L2 crate as found earlier this morning.</pre>
<!-- Author: monica -->


<!-- Date: Thu Jul 26 19:34:19 2001 -->
Stephen has made a new executable:  ~jsm/alpha/alpha_debug_linux 
<p> 
This will allow a bit to be set on the backplane (alpha's moddone) under various 
conditions:  BC mismatch between interface boards and/or 
zero MBus words having been sent by the board of your choice. 
There is also an option to print the error to the minicom. 
<p> 
The executable gives instructions. 
<!-- Author: for Stephen -->


<!-- Date: Thu Jul 26 21:46:01 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-57&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-57' width='300'></a></center>
<!-- Author: Matt and David -->
<!-- Comment: Thu Jul 26 21;55;33 comment by...David and Matt -->
Continuing XTRPlist studies:
<p>
OK, within minutes of Stephen's new output bit being implemented
we could reproduce the error.  Seen here is the XTRPlist taking BOSS twice (count 'em!).  Then we would be one event ahead of the rest of CDF as we suspected.  Note that the time between the two is exactly our start-load delay of 1.1 usec.
<p>
We tried to capture MODDONE on the same trace, but then we had
other problems and it was time to give the system away.
<p>
This debug feature is really cool. 


<!-- Date: Fri Jul 27 00:59:55 2001 -->
<pre>We ran the L2 torture test trying to Read out the ReCes boards. 
We had a special version of the L2 torture exec which read out  
all 4 reces boards. We first tested new front end initialization 
code and then verified the standalone code was writing correct  
trigger thesholds.   
 
 
We next ran the L2 torture test to strobe data. We observed some 
phi's that made sense, while others which were inconsistant. 
i.e. Here is an example for one of the ReCes boards 0X400 which 
was reading out the west detector phi0-11. (buffer 0)  
 
 
Phi Expected  Read  
     bits     bits 
 0 aaaaaaaa aaaaaaaa  
 1 55555555 55555555  
 2 aaaaaaaa aaaaaaaa  
 3 55555555 55555555  
 4 aaaaaaaa ffffffff  
 5 55555555 ffffffff  
 6 aaaaaaaa ffffffff  
 7 55555555 ffffffff  
 8 aaaaaaaa aaaaaaaa  
 9 55555555 55555555 
10 aaaaaaaa aaaaaaaa  
11 55555555 55555555  
 
Each board behaved differently and there was no correlation with 
magic bus bits, buffers, etc..  
 
We changed the l2 exec to read out only 1 reces board and 
the pattern looked exactly the same. In fact, the pattern looked the same after unplugging several fibers so we concluded we were 
never strobing data into the boards during the torture test, although we must have occassionally strobed in some data since 
some patterns were correct.  
 
Using our standalone program, we have successfully stobed 
alternating paatterns into the Reces boards so we compared the 
timing between stand alone and the l2 torture. The first plot is 
the timing during the L2 torture test.  The second plot is 
the timing using stand alone  code.  There is a difference 
in the DDONE and data relative to Dstrobe.       
  
During the course of this, we took several longish runs 
which someone else may find useful too.   
 
Run 121753 (this ended in a funny state) 
Run 121758</pre>
<!-- Author: byrum,tanaka -->
<!-- Comment: Fri Jul 27 13;26;56 comment by...Matt -->
Both runs 121753 and 121758 have L1A rate = L2A rate = 2-3Hz, with no L2 rejects.  I think that somehow these runs were with the alpaha tagging, and not cutting.
<!-- Date: Fri Jul 27 01:01:10 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-59&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-59  width='300' align='top'></td><td>Using L2 torture exec.</a></td></tr></table>
<!-- Author: byrum,tanaka -->


<!-- Date: Fri Jul 27 01:01:39 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-60&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-60  width='300' align='top'></td><td>Using stand alone code</a></td></tr></table>
<!-- Author: byrum,tanaka -->



<!-- Date: Fri Jul 27 14:51:38 2001 -->
<pre> We continued yesterday's Reces test. 
There are two stages of problems we need to understand: 
(i)  strobe XCES bit from SMXR to Reces 
(ii) read Reces register with L2 processor 
Today's test is mostly concentrated to establish (i). 
 
(1) We at first set all the trigger threshold to 
 0xffff, thus all SMXRs are expected to send zero data 
 to Reces. Then we took a l2-ratetest run with software 
 event builder, and check the Reces register by using the 
 stand alone code while pausing the run. This test looks 
 ok, all the data in the Reces is set to be zero except the 
 know problems.  
 
(2) We changed the trigger threshold for the SMXR 
 west phi = 0 and 1, and repeated the same test. But the 
 Reces registers were still all zero. Note that the stand  
 alone code which strobes XCES bit from SMXR to Reces works  
 with no error for both the west phi =0 and 1.  
 
(3) We change all the 48 trigger thresholds, and test 
 again but none of the Reces register were changed. 
 
(4) We try the same test with hardware event builder 
 but no difference as before. 
  
(5) We suspected Reces not to be in the data mode 
 but in the diagnostic mode while data taking, so put  
 Reces to data mode by using stand alone code while 
 pausing a run, then resume the run. It didn't work. 
 
(6) We tried to scope the Reces readout timing, but found 
   no signal that alpha reading Reces. 
 
(7) In summary we need to make sure 
     we are using the correct run configulation 
     Reces is set to be data mode 
     data strobe and readout timing is correct. 
 
 
 
    
</pre>
<!-- Author: masa, karen -->


<!-- Date: Fri Jul 27 17:17:14 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-62&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-62' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-61&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-61' width='300'></a></center>
<!-- Author: Matt and David -->
<!-- Comment: Fri Jul 27 18;28;08 comment by...David and Matt -->
<p>
XTRPlist sync studies continued:
<p>
We captured the double-event sending again today.  This time
we used the scope behind the rack (left plot: Ch1=ST_LOAD, Ch2=ERROR, Ch3=DSTROBE, Ch4=MOD_DONE, slot 8) to trigger the scope in front of the rack (right plot).  Note that our firmware is designed to hold the S4_OUT signal high once we have asserted DONE_OUT (aka MOD_DONE on the backplane) to prevent exactly this multiple sending per event condition.  Also note that we don't see this condition in myron mode.


<!-- Date: Fri Jul 27 18:39:56 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-63&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-63' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-64&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-64' width='300'></a></center>
<!-- Author: David and Matt -->
<!-- Comment: Fri Jul 27 18;52;16 comment by...Matt and David -->
<p>
More XTRPlist results:
<p>
We were also able to see events (left plot) where the XTRPlist board went out of sync by triggering on the bc error condition from the alpha (green), and by looking at START_LOAD (yellow), MOD_DONE from slot 8 (pink), and BOSS (purple) we could see that XTRPlist clearly took BOSS a second time in 1 START_LOAD (just above the purple "boss" label).  This is equivalent to our previous entry.
<p>
However, the screen capture from the front scope (right plot) triggering on the SAME event from the back scope clearly shows that START_LOAD, BOSS, and MOD_DONE in our firmware look perfectly normal: no multiple BOSS assertions in one ST_LOAD.  Note also that here we hold S4_OUT high as expected for the entire suspect event.  So in this case the backplane and the XTRPlist firmware seem to disagree.
<p>
We were repeatably able to observe both this case and the previous entry's condition in which we could also see the multiple BOSS assertions in our firmware (even going back and forth between them with several HRRs in one run).

<!-- Comment: Fri Jul 27 20;25;57 comment by...Stephen -->
<pre>Could it be that the SVT board was the one which had the BC error
and/or sent the two events worth of data?
</pre>


<!-- Date: Sat Jul 28 23:16:26 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-65&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-65' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-66&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-66' width='300'></a></center>
<!-- Author: David and Matt -->
<!-- Comment: Sat Jul 28 23;25;44 comment by...Matt and David -->
As expected, the store ended quickly due to a quench so the shift crew called us in for more track-list studies.    We actually see the problem from both SVTlist and XTRPlist approximately equally.
(See above posting.)
<p>
We looked in more detail and with more time resolution at the state machine.   We see that S4_OUT is up for exactly one clock cycle (50 nsec) and then is turned off.  According to the FPGA, S4_OUT should keep itself high until START_LOAD is released so this may be due to noise on START_LOAD, in fact we do see about 500mV noise (according to the testpoint) at the critical time. 
<p>
We looked at S4_IN (right plot) and even with the START_LOAD noise, we would have expected S4_IN to go back high as least at some time during the next 50 nsec due to the S4_OUT feedback even if there were some START_LOAD noise but it never does.   This is puzzling.


<!-- Date: Mon Jul 30 21:42:03 2001 -->
<p> 
Tracklist studies 
<p> 
Trying to verify and remove the (possible) glitch on START_LOAD causing the tracklist boards to send 2 events of data in 1 START_LOAD.  Tried swapping XTRPlist and SVTlist into slots 7 and 8, respectively.  It appears that the board in slot 7 tends to favor going out of sync sooner than the board in slot 8.  Moved SVTlist from slot 8 to slot 11.  It was inconclusive whether this had any effect. 
<p> 
Put new firmware (7jul01) onto XTRPlist board, which should protect against the glitches in START_LOAD.  Due to problems with softevb and run control in general, it was inconclusive whether this firmware had any effect.  We could not get the system to run faster than 1 kHz L1A rate in a configuration that previously ran at 15-17 kHz.  We saw a strange state where all L2 boards stopped sending along the backplane even thought they were getting data, but BOSS was LOW!  All jumpers were double-checked and in place.
<!-- Author: David, Matt -->
<!-- Comment: Tue Jul 31 08;33;19 comment by...Stephen -->
<pre>In this case check BOSSREQ, to make sure a deadlock condition
hasn't occurred.</pre>
<!-- Comment: Tue Jul 31 15;48;59 comment by...David and Matt -->
We saw a similar condition today.  BOSSREQ was low as it should be.  We contacted Stephen.  It turns out that alpha_check says it was waiting for L1A.  Looking
at the TS for our partion we see lights are all orange which means it was a DAQ problem since L2 already sent its decsion.  Steve Nahn (who was doing silicon daq studies with the other half of the HEVB) and Greg cleaned up L3 and it worked again.   Anyway this TS LED knowlege is generally good to know.
<!-- Date: Tue Jul 31 01:22:46 2001 -->
<pre>We ran the L2 torture test executable with L1, L2, all cal 
crates, XFT,SVT, trigger scalers, hardware event builder.   We 
had to perform HRR continously (after about each 10 evts) due 
to L2 crate timeout. This was related to the Vision read failed 
 in slot 3 with b0l2de00: done time out errors.  
   
We were only reading out one ReCes board, 0x400.  
 
Overall, the data appears to be strobed from the SMXR into 
the reces correctly.   We don't know if the reason everything 
looks correct tonight is related to the slow event rate.  
Several days ago when we performed a very similar test, not all 
fibers were getting correct data (we were reading out  4 boards 
at that time and the event rate was faster than today). 
 
Our results: 
We cleared all the ReCes registers and set 
the SMXR thresholds to read alternately aaaaaaaa and 55555555 
for alternating phis.   We started RC, strobed data 
through the SMXRs, paused the run and used stand alone 
to read the ReCes registers.   After 10 events, we changed the 
SMXR thresholds so each phi would read the alternate pattern, 
resumed RC, paused after a few more events and reread the  
ReCes registers.    The data in all 4 cards looked  
correct (except for 2 swapped fiber bundles which we later 
verifed were swapped using standalone code). 
 
We also checked the TL2D bank and found this to be partly 
correct.   We looked at software, but couldn't find any bugs. 
 
Finally, we managed to get some scope pictures of the timing 
of the data coming into ReCes. 
See below.</pre>
<!-- Author: karen, masa -->


<!-- Date: Tue Jul 31 01:34:29 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-68&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-68' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-67&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-67' width='300'></a></center>
<!-- Author: karen,masa -->



<!-- Date: Tue Jul 31 11:40:20 2001 -->
<pre>Please start a new page of this log as soon as possible. 
It takes forever to read if you have a slow connection 
as I have now :( 
 
Thanks 
</pre>
<!-- Author: Luciano -->