<!-- Title: 23:21:22  Tue Aug 20 2002 -->
<!-- Crewchief: L2 crew -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New page.</pre>


<!-- Date: Tue Aug 20 23:24:30 2002 -->
<pre>In starting a L2 torture run, I had the L1A rate limited at  
about 700Hz with high deadtime from L2. THis was due to the alpha 
timing out (after 1.2ms) waiting for mod_done from isolist. 
 
 I paused the run and did a hot reseat of the iso-clique board  
in b0l2cl00. That solved the problem.</pre>
<!-- Author: Stephen -->


<!-- Date: Tue Aug 20 23:56:28 2002 -->
<pre>I replaced SVTlist #6 with Svtlist #8. The firmware has the 
same version. We will see if the L2 SVT errors of too many 
SVT tracks goes away when beam comes back.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Aug 21 00:35:57 2002 -->
Did some SVT timing studies during DAQ idle time.  Replaying 
some old hits from run 149484 (which has occupancy that should 
be typical of what comes out of the L1 track trigger), I see  
average SVT processing time is about 9.3usec above average 
wedge 3 merger done time.  (Wedge 3 chosen arbitrarily.) 
Wedge 3 AMS done minus wedge 3 Merger done averages 2.1usec. 
Wedge 3 HB done minus wedge 3 AMS done averages 0.3usec. 
Wedge 3 TF done minus wedge 3 HB done averages 0.8usec. 
<p> 
So it looks as if pattern recognition and fitting for a typical 
(i.e. usually empty) wedge is done about 3.2usec after last hit 
arrives. 
<p> 
For a data sample that is biased to have at least one 
track per wedge per event, wedge 3 AMS done minus wedge 3 Merger done averages 2.5usec, wedge 3 HB done minus wedge 3 AMS done averages 1.5usec, and wedge 3 TF done minus wedge 3 HB done averages 2.0usec.  This adds up to 6.0usec.  So one might deduce that the wedges that contain the "two tracks" take 2.8usec longer than the "empty" wedges. 
<p> 
So we can guess that the last TF is typically done 6.0usec after 
its corresponding Merger is done. 
<p> 
Because the Mergers run in 
"deterministic" mode, and the mean number of SVX hits per SVT 
wedge is about 150 (200 in wedges biased to have tracks in them), 
the Mergers should be done about (2/3)*150*0.033=3.3usec (4.4usec for busy wedges) after the typical time Hit Finders are done.  (Is the "deterministic" effect really this large?  We should check more carefully; I may have overestimated.) 
<p> 
We measured <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=48&frame=2&anchor=00:16:44'> 
earlier</a>  
that a typical Hit Finder is done 16.5usec after L1A.  About 
9usec of this is waiting for first SVX data.  So a typical Merger should be done about 19.8usec after L1A, and one in a busy 
wedge should be about 21usec after L1A, with current cluster 
occupancies. 
<p> 
This says the last TF is done about 27usec after L1A; then add 
about 2usec for the merging and the current GhostBuster latency, 
and you get 29usec, compared with 27usec measured in recent 
runs.  Maybe I was pessimistic somewhere. 
<p> 
In test run 150119 (SVX readout threshold=9), SVT processing time averaged 24.78usec, compared with 25.44usec in normal 150117.  In run 150118 (threshold=11), processing time was 23.48usec.  So we 
gain maybe 2usec with thresholds.  (Gino said the SvxMon occupancy went from about 15% in the normal run to about 3% in the two test runs with increased thresholds.  3% is the nominal occupancy, I think.) 
<p> 
In these runs, the number of SVT clusters per wedge was about 100, so deterministic merging might buy us (2/3)*100*0.033=2.2usec. 
<p> 
Removing some unnecessary GB latency should buy us 1.0-1.5usec (say 1.2).   
<p> 
So maybe there is 5.5usec to gain in all of the above. 
We may also gain a few usec by going back to 7-bit digitization 
in SVXII.  This would bring us down to 18 or 19 usec. 
<p> 
We should figure out what is the most optimistic possible value 
for time SVX data arrive at Hit Finders and when Hit Finders are 
done.  I think the old estimates were 3 usec to get the data, 2 usec readout time, 1 usec added HF latency, total 6 usec, compared with the 16 or so we see now. 
<p> 
It's after midnight, and it shows.  Sorry for poor S/N. 
<p>  
<!-- Author: Bill -->


<!-- Date: Thu Aug 22 23:42:21 2002 -->
Am investigating why SVT crates crash when AMB pattern write 
fails.  Tried cold start several times, OK.  Unseated one HF, 
then tried cold start several times.  It fails, but does not 
crash, and abort can be followed by another cold start.  
Unseated one AMB, then tried cold start several times.  It 
also fails gracefully.  It seems that any time I need to 
call svtvme_getRAM() for the AMB patterns, I get a suspended 
messenger task ...  
<!-- Author: Bill A -->
<!-- Comment: Fri Aug 23 00;23;02 comment by...Bill -->
I didn't figure it out before time ran out.  But the crash 
still happens even when I disable the getRAM, writeMem(),
and checkMem() calls, my three prime suspects.


<!-- Date: Tue Aug 27 11:36:39 2002 -->
<pre>We successfully tested a L2exe sending readoutlist bits to the 
TS. Look in the TS log book for details.</pre>
<!-- Author: Stephen, Greg -->


<!-- Date: Wed Aug 28 21:29:08 2002 -->
<pre>Description of L2 tests on Tuesday and Wednesday. 
 
Summary: PIO fpga (D0 version), pipeline fpga and code optimizations 
were all tested successfully. The L2 TORTURE trigger table 
ran with a L1A rate of 143KHz with 1% deadtime with all the  
optimizations. VME BLT still doesn't work with the new firmware.  
 
PIO: Test of new firmware from Rick Kwarciany. This is used for  
reading Reces. This firmware allows Reces to be read at any time 
and was successfully tested with the alpha arbitrating with 
other interface boards for the MBus.  The biggest change is 
that the PCI transaction must be a 64 bit burst transaction which 
means new software is required when this prom is used.  
Another advantage is that it only takes 13us to read all of the 
Reces boards on a L2A (versus 48us now). 
 Caveat: In tests where Reces was read on every event, at high 
rate, the PIO would hang the PCI bus and MBus. That is 
because the Reces board would not respond with DDONE* to finish  
the MBUS transfer. This happened for one specific Reces address,  
but not for another, so it isn't clear if it is actually a Reces  
problem or a PIO problem. 
In the test it was always the same address being read. For the 
problem address, the error would happen every few minutes and 
would require a reboot of the alpha to fix it. The old fpga 
design has a timeout if no device responds on the MBUS. Perhaps 
this feature can be added to the new design.  
 
Pipeline fpga: Currently the control of MagicBus loading is 
all done in software. The new fpga allows the MagicBus loading 
to be done in firmware. This speeds up the time for turning 
the control signals on and off, as well as making loading 
more independent of algorithm processing. It also saves several  
us for the L2-TS handshake. This new firmware  
worked very well. 
 
DMA config: Some optimized software was tested which writes 
the starting address to the DMA buffers using one PCI burst  
transaction instead of 6 separate PCI writes. This saves 1us 
of overhead time every event. 
 
Full optimization test: In addition to the above, I made a  
stripped down exe which didn't read the DMA buffers, didn't do 
the L1 scalers, and didn't calculate timing numbers. The 
tracklist, Clist boards only sent the EoE word as no detector 
elements were booked. The only trigger was the BC trigger. With 
a prescale of 12, the L1A rate was 143KHz and the deadtime was 
1%. The L2A rate was about 5Hz. The loading time on the MBUs was 
about 4us. The time between data being loaded through the 
decision being sent to the TS was 6us. Both loading and  
processing were happening at the same time. The breakdown of the 
6us is:  
Configuring alpha for next loading 1.1us.  
Unpack, Algorithms (autoaccept triggers), error checking: 2.6us 
L2-TS handshake 2.3us. 
Conclusion: The alpha configuration time is now at our initial 
target of 1us. The L2-TS handshake can be improved with  
firmware changes in the TS and the alpha.    
 
VME Block transfers with the new firmware was tested and also 
gave problems on the PCI bus. The options are 
1. Use readoutlists to only readout the following on some events 
L1 scalers- 64 words 
Reces - 48 words 
Diagnostic - 50 words 
L2 trigger word - Nwords = Ntriggers 
L2 scalers - Ntriggers*4 
Total is 2,248 bytes per event 
2. Option 2 is to try and slightly slow down the BLTransfers 
using software. 
 
Near term Plan: Put the code for the new fpga's and  
optimizations into an official tagset and physics table exe. 
Test this without beam and then schedule a beam test. 
Not all the optimizations will be done at once, so several 
tests will be required.</pre>
<!-- Author: Stephen -->


<!-- Date: Fri Aug 30 18:01:23 2002 -->
<p> 
We are attempting to use the SVDD bank to measure the 
timing of the hits coming out of Merger mrg_w03, using 
a spare Merger board in b0svt07/21 as a fanout and then 
a "utility" channel of a GB in b0svt07/12 to read this 
into a bank.  This GB has previously been used to 
measure the timing and output data of a couple of 
Hit Finders (including one in w03).  This should fill 
in a missing piece of the timing study begun about a 
week ago. 
<p>  
<!-- Author: Alex and Bill -->


<!-- Date: Sat Aug 31 02:08:34 2002 -->
<pre>&lt;p&gt; 
Apparently the timing tool we inserted this afternoon on W03 
is causing L2 errors. Symptom is L2 complaining about 0 words 
coming off the SVT: 
 
SvtMon: Error: no data from XTRP(1 words) or SVT(0) 
 
The error was showing up every 2 minutes at the beginning of  
the run, every 5-10 later on. It disappeared after bypassing 
the timing measurement kludge. 
The bypass went in sometimes before event 571342/run 150779.  
 
Just as a cross check I'm going to put it in again and see 
if the L2 error shows up again! 
Halted at event 881242 and put the timing stuff in again...  
the error shows up at event 900978. 
 
So it's definitly us... sorry but I'll leave the thing out  
for tonight.... The data we have should be already enough  
for our timing measurement. 
 
I couldn't put together one single reason why this happens... 
maybe it's too early in the morning to have my brain  
"functioning properly" 
&lt;p&gt;</pre>
<!-- Author: Alex -->


<!-- Date: Sat Aug 31 03:43:26 2002 -->
<pre>W0B1L2 disappeared so we recovered W0B1 by switching 
from X0123X5 to X01X235.... This required two steps 
because in the first one I wrote X012X35.... 
</pre>
<!-- Author: Alex -->


<!-- Date: Sat Aug 31 10:50:56 2002 -->
<p> 
I don't think we recovered anything, because we were still 
using the same W01 pattern file.  Also, since Gino thought 
that the ladder he had removed would soon be back, it may 
not make sense for us to change our configuration so 
quickly.  People would like us to try to be stable.  Finally, 
I suspect (say 80% confidence) that our timing cable had  
nothing to do with the timeouts, that the problem was that 
the FIB containing the troublesome ladder was not sending 
data, and that the two fixes were implemented at about the 
same time.  Maybe we can glean more information from the 
spy buffer archives. 
<p> 
It would be nice to have a unique minicom error message from 
L2 indicating 0 words sent by SVT.  It would then be nice for 
SVT to attempt to diagnose this, or at least track down which 
inputs are responsible.  For example, one could see which 
Merger inputs in SVT are still waiting for input. 
<p>  
<!-- Author: Bill -->
<!-- Comment: Sat Aug 31 12;50;53 comment by...Bill -->
OK, it's a good thing I left myself 20% instead of just 5%.
If I carefully re-read Alex's e-log entry, it's clear that the
problem was our cable.  Inconceivable!
<p>
Sorry, Alex.


<!-- Comment: Sat Aug 31 12;51;51 comment by...Bill -->
To be clear, that's Bill saying: "Sorry, Alex."



<!-- Date: Sat Aug 31 19:29:48 2002 -->
<pre>&lt;p&gt; 
Still investigating why the timing bypass gives problems... 
 
The "bypass" is a merger with one single input enabled and 
output #1 connected to the "timing" GB, with the hold line 
sense disabled. 
output #2 is the return line for the bypass and has the 
hold line enabled. 
 
The error was detected when bypassing the line (W03, MRG-&gt;HB). 
A possible explanation could have been that the HB didn't like 
the idea of receiving EE from the AMS while the merger was 
still sending out (from the HB point of view) data... 
 
At the end of the last store I used 3 min of beam time to  
realize that the same problem showed up applying the tool  
to (W03, MRG-&gt;AMS). So the HB confusion is ruled out 
as an explanation of the problem. 
 
Taking advantage of the interstore time, I took SVT in a 
separate partition and exercised it with hw=fakehits 
 
I tried everythig I could think of: 
- unplugging the HB output 
- unplugging in turn each of the HB inputs 
- unplugging the GB input 
 
each time the behaviour was as expected and the 
spy dumps were normal. 
 
Could that still be an hardware failure of the merger 
we use for bypassing? or maybe a failure of the bypass 
wires? 
 
We need additional end-of-the-store beam time to cross 
check this using another pair of cables and a new merger. 
 
Waiting for that time to come I replaced the merger we were 
using (merger #2) with the one in the test-stand (merger #06). 
&lt;p&gt;</pre>
<!-- Author: Alex -->
<!-- Comment: Mon Sep  2 15;13;14 comment by...Stefano -->
<pre>It was not a merger you started with. It was an XTFC.
Some programs (like the test stand random test) read the
prom and crash if they do not find the board they expect.
</pre>
<!-- Comment: Mon Sep  2 17;29;20 comment by...Bill & Stefano -->
<pre>We tested XTFC 2 in the test stand with the random test and it works. We found that switches are configured to have Hold enabled on output 1 and disabled on output 2. From Bill's memory
this is the opposite of what it should be. This may possibly explain the timeouts.</pre>
<!-- Comment: Mon Sep  2 18;40;11 comment by...Alex -->
<p>Im surprised: I don't remember moving the switches on XTFC2
but i remember:
<ul>
<li> checking that:
 <ul>
     <li> unplugging the MRG->GB line had no effect on the                  dataflow with no beam data
     <li> unplugging the MRG->b0svt01 crate line stopped the
           upstream dataflow
  </ul>
(thats what I meant in the original logbook entry saying that
 the behaviour was as expected)
<li> checking that the switches on XTFC 2 were as expected
<br>
  (hold sense disabled on input 1, enabled on input 2)
<li>setting the same config on merger 06
</ul>
<p>

<!-- Date: Mon Sep 2 09:07:30 2002 -->
<pre>&lt;p&gt;We didn't get our beam time yet... I showed up this morning 
at the 8am meeting to ask again for it. Maybe at the end of next 
store? 
In the mean time I ran also a new pair of cables between 2RR35H 
 (top) [this is where the timing bypass merger sits] and 2RR35I 
[the rack where W03 sits]. Hopefully I'll be ready in 5 minutes 
when they give us beam... 
&lt;br&gt; 
This is the plan of the tests I want to make: 
&lt;ul&gt; 
&lt;li&gt; reproduce the error with old merger,cables and bypass position 
&lt;li&gt; swap everything (cables, merger) and see if error shows up again 
&lt;li&gt; try another wedge 
&lt;li&gt; try to bypass a less "sensitive" line (that is, one where 
 relative timing with another data pipe is not an issue), 
 downstream from the HB or elsewhere 
&lt;li&gt; if all this fails try also disconnecting the GB input line 
&lt;li&gt; try to investigate any pathology with the merger being  
used in the configuration in which it's running for this bypass 
exercise (1 input enabled [input A], 2 outputs used one of which 
[out 1] has the hold sense disabled): 
&lt;ul&gt; 
&lt;li&gt; swap the output lines so that hold is sensed on out 1 but not in out 2 
&lt;li&gt; use a different input line as the only input 
&lt;li&gt; enable hold sense also for the output connected to the GB 
(dangerous?) 
&lt;/ul&gt; 
&lt;li&gt; still having the problem? Then maybe we should make our 
timing measurement using a digital scope and forget about the 
GB+MRG stuff at least until after the SVT workshop 
&lt;/ul&gt; 
 
&lt;p&gt;.</pre>
<!-- Author: Alex -->



<!-- Date: Mon Sep 2 17:18:06 2002 -->
<p> 
We wanted to get the Spinella test stand program running 
again, after a few days' hiatus.  When we first started up, 
we found strange road output from AMS.  We traced it to a 
bad AMplug. 
<p> 
AMB SN04 was in the test stand (slot 9) when we arrived, and still is. 
<p> 
We temporarily borrowed AMB SN20 from the spares 
cabinet.  This is the board that Alex swapped out a few 
weeks ago because it didn't pass coldstart.  We confirm 
that it doesn't pass coldstart.  It is back in the spares 
cabinet, and still needs diagnosis. 
<p> 
AMB SN04 was producing spurious/duplicate roads from plug #3, 
which was AMplug SN223.  We swapped AMplug SN223 with AMplug 
SN210 (which was plug #3 on AMboard SN20). 
<p> 
Now the Spinella random test is running happily on the test 
stand crate.  Random test has a ~3E-5 failure rate, which seems 
always to be due to Track Fitter output.  It seems that the 
Track Fitter sometimes produces a second identical copy of track when the simulation does not expect it to do so. 
<p> 
NB also Stefano's modification to teststand boot script has been 
undone, i.e. once again the random test starts up automatically 
when the crate is booted. 
<p> 
  
<!-- Author: Stefano and Bill -->



<!-- Date: Mon Sep 2 18:06:51 2002 -->
I modified the svtvme software so that it treats XTF-C as 
it if it were a Merger board. This will be fine until (if ever) 
we modify XTF-C firmware, which now is identical to merger one. 
<p> 
This now allows the test stand random test to work even if 
one XFT-C board is used in place of a Merger. 
(See comment to <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=49&frame=2&anchor=19:29:48'> 
Saturday's Alex entry </a> 
<!-- Author: Stefano -->


<!-- Date: Mon Sep 2 18:07:53 2002 -->
<p> 
I just overwrote teststand.mapset with the current mapset from 
the database (gshift_mixed_20020831150236), and rebooted the 
test stand.  It is still happy. 
<p>
<!-- Author: Bill -->



<!-- Date: Mon Sep 2 19:58:50 2002 -->
<p> 
Another episode of the timing measurement saga.... 
<br> 
This time I tried to raise the rate at which fake hits 
flow through svt (hw=fakehits).<br> 
I tried to bypass the hit lines for either the HB and 
the AMS, using: 
<ul> 
<li> the old cables and merger (xtfc 02) without touching the switches (xtfc02 now sits in slot #20). 
<li> the merger taken from the test stend, seated in slot #21 
</ul> 
In all cases the cables are the same as yesterday, even if 
I have a new pair there ready for the new beam test. 
<br> 
I don't see abnormal behaviours at all. I tried to change 
by factors 10 the rate (interval0 = 10 thru 100) without 
noticing any desynch problem in the HB. 
<br> 
Spy dumps are always in synch. 
I'm more and more puzzled. 
<p>
<!-- Author: Alex -->
<!-- Comment: Mon Sep  2 19;59;44 comment by...Alex -->
<pre>I leave all the hardware in place so that I can try several different cables/boards configurations with beam data, whenever
it comes.</pre>


<!-- Date: Mon Sep 2 21:58:39 2002 -->
<p> 
I noticed that SPYMON was not running in b0svt06 (and thus 
there were no beam fits). I started this task.  I also 
noticed (unrelated) that tf_w00 and mrg_final were not 
respecting the HOLD lines on their output cables, so I 
changed this in the database.  It will take effect next 
run. 
<p>
<!-- Author: Bill A -->


<!-- Date: Tue Sep 3 09:00:22 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-736&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-736  width='300' align='top'></td><td>Starting from store 1723 Isolist has high rate  (~50%) eta-phi errors. This error wasn't in the previous store 1726.</a></td></tr></table>
<!-- Author: Masa -->


<!-- Date: Tue Sep 3 09:08:36 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-737&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-737  width='300' align='top'></td><td>The other TrigMon plot shows the error rate is <1%. I suspects it's TrigMon problem. </a></td></tr></table>
<!-- Author: Masa -->
<!-- Comment: Mon Sep  9 17;02;03 comment by...Nate Goldschmidt -->
<pre>I believe I've found the bug in the ISOlist_Monitor code. Fix is forthcoming!</pre>
<!-- Date: Tue Sep 3 11:06:59 2002 -->
I updated the mapset in the database to go from 8 to 4 SVX 
hits per superstrip, so that the superstrip numbers for 
bulkhead 5 no longer collide with the layer bits in the HLM 
address space.  I updated the hwset in the database to enable 
CDF_ERROR for PE,FO for AMS and PE,FO,LS for HB.  If this goes 
well, we can get even more agressive with the error bits.  
<!-- Author: Bill A -->


<!-- Date: Wed Sep 4 10:16:34 2002 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-738&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-738' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-739&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-739' width='300'></a></center>
<!-- Author: Masa  -->
<!-- Comment: Wed Sep  4 11;01;55 comment by...Masa -->
<pre>

 
    Left plot is the processing time for sum of all 
   the Electron triggers in the current PHYSICS_1_02_v3 
   trigger table before (top) and after (bottom) the
   code optimization. The processing time is reduced
   by ~x2. The data is Stream J, and I prerequisite 
   at least on of the the L1_CEM4_PT4 or L1_TWO_CEM2_PT2_OPPQ 
   to be on. Also the processing time doesn't include the
   unpacking time (~5-10 us)

    Except the L2_CEM16_PT8 which is the only the 
   electron trigger which isn't require Reces, all 
   the electron code now uses exactly the same lookup table 
   for XFT track extrapolation. The extrapolation information 
   is cached under the L2trigobjects/TrackData, so the 
   extrapolation is done only once per track through 
   the event.

    I haven't yet implemented for caching the matching
   information. It turns out not that easy since we need
   to handle the case if one XFT track associate to
   two or more clusters. 

    Right plots shows the processing time after the 
   optimization as a function of N_XFT (top) and 
   N_cluster (bottom). The tail in the processing time
   comes from the events with large number of tracks
   or clusters. I still have some ideas for further
   optimization.

    The other thing is I noticed the processing time
   for the photon triggers under L1_EM8 is larger than
   the electron triggers,  since  more than
   10 L2 photon triggers are fed to the L1_EM8.
   I think we can reduce the processing time by 
   caching the information.



  

 </pre>


<!-- Date: Wed Sep 4 23:00:45 2002 -->
<p> 
I did an analysis of the spy buffer dumps from run 150946. 
I see that in the dumps from 23:04, 23:05, 23:06, 23:09, and 
23:10, the Track Fitter in wedge 1 was several events behind 
every other board in SVT whose spy buffers were dumped.  In 
3 of these 5 sets of dumps, the b0svt06 dump is in time with 
the wedge crate dumps, and I can see that the final Merger is 
stuck at the same point as the wedge 1 Track Fitter.  Here is 
an example, in which I am simply dumping out the bunch IDs 
from each spy buffer, right-aligned, because different numbers 
of events are available at the tail of different spy buffers. 
<p> 
<pre> 
 
b0svt00/11/HB_HIT_SPY     .. .. .. 21 76 56 06 47 4a 
b0svt00/11/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt00/11/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt00/12/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt00/20/HB_HIT_SPY     .. .. .. .. 76 56 06 47 4a 
b0svt00/20/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt00/20/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt00/21/TF_OSPY        .. .. .. .. .. 03 18 21 76 
b0svt01/11/HB_HIT_SPY     .. .. .. .. 76 56 06 47 4a 
b0svt01/11/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt01/11/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt01/12/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt01/20/HB_HIT_SPY     .. .. .. .. .. 56 06 47 4a 
b0svt01/20/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt01/20/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt01/21/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt02/11/HB_HIT_SPY     .. .. .. 21 76 56 06 47 4a 
b0svt02/11/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt02/11/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt02/12/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt02/20/HB_HIT_SPY     .. .. .. 21 76 56 06 47 4a 
b0svt02/20/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt02/20/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt02/21/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt03/11/HB_HIT_SPY     .. .. .. 21 76 56 06 47 4a 
b0svt03/11/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt03/11/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt03/12/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt03/20/HB_HIT_SPY     .. .. .. 21 76 56 06 47 4a 
b0svt03/20/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt03/20/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt03/21/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt04/11/HB_HIT_SPY     .. .. 18 21 76 56 06 47 4a 
b0svt04/11/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt04/11/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt04/12/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt04/20/HB_HIT_SPY     .. .. .. .. 76 56 06 47 4a 
b0svt04/20/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt04/20/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt04/21/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt05/11/HB_HIT_SPY     .. .. .. .. 76 56 06 47 4a 
b0svt05/11/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt05/11/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt05/12/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt05/20/HB_HIT_SPY     .. .. .. .. 76 56 06 47 4a 
b0svt05/20/HB_ROAD_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt05/20/HB_OUT_SPY     .. 03 18 21 76 56 06 47 4a 
b0svt05/21/TF_OSPY        .. 03 18 21 76 56 06 47 4a 
b0svt06/12/MRG_A_SPY      .. .. .. .. .. 03 18 21 76 
b0svt06/20/MRG_OUT_SPY    .. .. .. .. .. 03 18 21 76 
b0svt07/10/MRG_A_SPY      .. 03 18 21 76 56 06 47 4a 
b0svt07/10/MRG_B_SPY      .. 03 18 21 76 56 06 47 4a 
b0svt07/10/MRG_OUT_SPY    .. 03 18 21 76 56 06 47 4a 
b0svt07/17/XTFA_TRK_SPY   .. 03 18 21 76 56 06 47 4a 
 
</pre> 
<p> 
It is clear that the reason that SVT was sometimes not 
sending data to L2 was an internal SVT problem, and was 
somewhere between the output stage of the Wedge 1 Hit 
Buffer and the last processing stage of the Wedge 1 
Track Fitter. 
<p> 
This condition could have been immediately made known to 
the operator if we had a program that polled the status 
of each Merger in b0svt06 on the Halt transition (or 
whenever the trigger rate dropped to zero).  This condition 
could be detected as the very unusual state that data are 
waiting in the input FIFOs of 11 of the 12 wedges' Merger 
inputs in the final fan-in crate.  This handles the case of 
one wedge missing an event.  If one wedge sent an extra event, 
there would be a BC mismatch, and we could detect that in the 
hardware.  The next case is all wedges failing to send data. 
This could be detected by polling the Merger that lives inside 
each wedge crate any time the trigger rate goes to zero.  Then 
one could detect one Hit Finder failing to send, all Hit Finders 
failing to send, or XTRP failing to send.  It seems very unlikely 
that all wedges would simultaneously fail to send data for any 
reason other than lack of input data; a hardware problem is much 
more likely to be localized. 
<p> 
We could also count both L1A from backplane and EE from SVT cable 
in the GhostBuster board in b0svt06, in counters reset on CDF_RECOVER.  If these counter mismatch when polled after the 
trigger rate drops to zero, it is a sign of trouble.  In the 
same GB, we could implmement a O(500usec) timeout from L1A to 
end-of-event, on which we could report an error condition. 
<p> 
We should also change our "nosvx" configuration so that it  
actually flows some kind of data (even if only end-event 
words) through every SVT board, instead of bypassing most of 
the SVT hardware.  That would probably have allowed this problem 
to be detected before the silicon was integrated. 
<p> 
It is still unclear whether the problem is TF, HB, or cable 
in between. 
<p>  
<!-- Author: Bill Ashmanskas -->


<!-- Date: Thu Sep 5 10:59:01 2002 -->
 
 
<p>The silicon/L2 run numbers from hell are: <p>  
<pre>  
147267  
144562  
146652  
150948 
</pre> <p> 
<BR>  
<!-- Author: Bill -->


<!-- Date: Thu Sep 5 12:21:34 2002 -->
<p> 
<b>Some people expressed doubt that the wedge 1 problem was 
present in the last couple of runs before we replaced SVT  
data flow with empty events.  I clearly see the same pattern 
in spy dumps from run 150947 at 23:28, 23:29, 23:30, 23:31, 
23:32, and 23:40.  I didn't check 23:33-39.  I have every 
reason to believe this problem will still be present whenever 
the silicon comes back. 
</b> 
<p>  
<!-- Author: Bill A -->
<!-- Comment: Thu Sep  5 16;30;04 comment by...David -->
<pre>Would it be helpful to know if the same or related problem was 
present at:
  Sept 2: 21:51
  Aug 31: 03:14
  Aug 31: 02:26
?</pre>


<!-- Comment: Thu Sep  5 20;44;43 comment by...Bill -->
I'm not sure that it was helpful, but it seemed that it
couldn't hurt, so I took a look.  I didn't see anything
fishy.  Unfortunately, the spy dump only catches the
problem if either the trigger is paused for a time 
comparable to the 1-minute polling interval or the
spy buffers are frozen by the boards' detecting an 
error condition and using the SVT_ERROR backplane line.
To say for sure that the problem was not there would
require us to know that the spy dump had happened while
the trigger was paused for the error in question.
(Maybe a 'halt' transition is always an interesting
time for a spy buffer dump?)

<!-- Comment: Thu Sep  5 22;53;13 comment by...David -->
<pre>I see.   The "halt" idea is excellent.
(SVT's spy buffers are great.)</pre>
<!-- Comment: Wed Sep 11 07;25;47 comment by...Bill -->
Marco says this should be easy, as spymon is already listening
to Run_Control messages, including e.g. the Halt message.

<!-- Date: Thu Sep 5 17:17:24 2002 -->
<pre>Between 15:10 and 17:13 today i've been trying to catch a 
 de-synch like the one Bill saw flowing fake data through SVT. 
 Couldn't cath one. I've ran the data throught at different L1A 
 rates but spy buffers are always consistent. 
For future records, the spy buffer dumps coming from these tests  
include 2 time intervals where I on purpose generated de-synch 
in svt (first case was using non aligned input data files, 
second case was unplugging W1 MRG-&gt;HB hit line).</pre>
<!-- Author: Alex -->
<!-- Comment: Thu Sep  5 19;56;28 comment by...Alex -->
<pre>Note that the data used in this test was the default 
fakehits.hwset one, which at this time is Bill's selection
of events with at least one track.</pre>
<!-- Date: Thu Sep 5 19:55:30 2002 -->
<p>Changed the default nosvx entry so that we're sending 
through SVT events taken from real data <b>rather than</b> 
empty events (like the infamous ones that caused the 
SVX accident two days ago). 
This way we can also use the spydump data from runs with 
no silicon integrated to look for symptoms similar 
to the one we had right before the SVX accident.<p>
<!-- Author: Alex -->







<!-- Date: Wed Sep 11 17:29:33 2002 -->
<pre>Firmware tests today. 
--------------------- 
Put Stephen's new firmware in the alpha  
and ran with the code he tested several  
weeks ago -  
 
1) Hit reset on CLISt board and crashed the run. 
   HRR recovers. 
 
2) Run 151351 sent to TrigMon.  Data looks OK. 
 
3) Tried new software to go with the firmware. 
   Runs would crash after a couple hundred events. 
   Stephen is investigating. 
 
</pre>
<!-- Author: Greg, Tom, Stephen -->


<!-- Date: Thu Sep 12 20:59:35 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-740&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-741  width='300' align='top'></td><td>logic analyzer trace of SVT board-to-board timing</a></td></tr></table>
<!-- Author: Franco & Bill -->


<!-- Date: Fri Sep 13 20:35:16 2002 -->
<p> 
hmmm, we see that there is definitely a data flow problem between 
HB0 and TF0.  we illustrate with '~ashmansk/vst/eeflow.py' 
python script, first showing wedge 1 (ok) and then wedge 0 (bad). 
 
<pre> 
> python eeflow.py 1 
    AMS_HIT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
    AMS_OUT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
     HB_HIT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
    HB_ROAD_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
     HB_OUT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
        TF_ISPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
        TF_OSPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
      MRG_B_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
> python eeflow.py 0 
    AMS_HIT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
    AMS_OUT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
     HB_HIT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
    HB_ROAD_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
     HB_OUT_SPY(P): 600000 600001 600002 600003 600004 600005 600006 600007 
        TF_ISPY(F): 6ff0f0 6ff0f1 6ff0f2 6ff0f3 6ff0f4 6ff0f5 6ff0f6 6ff0f7 
        TF_OSPY(F): 6ff0f0 6ff0f1 6ff0f2 6ff0f3 6ff0f4 6ff0f5 6ff0f6 6ff0f7 
      MRG_A_SPY(F): 6ff0f0 6ff0f1 6ff0f2 6ff0f3 6ff0f4 6ff0f5 6ff0f6 6ff0f7 
</pre> 
<p>  
<!-- Author: Bill & Taka -->
<!-- Comment: Fri Sep 13 20;42;04 comment by...Taka & Bill -->
<pre>

We touch the cable--no change.
We reseat the cable--no change.
We swap the cable--no change.
We run the cold start for all boards--no change.
Cycle power, run cold start--it's OK now.
We don't know whether the HB or the TF was at fault.
Let's try to make it happen again.

</pre>


<!-- Comment: Sat Sep 14 00;32;55 comment by...Bill & Taka -->
We ran until final protons were loaded, and error didn't
happen again.  It seems likely that the confusion we were
seeing after the "two minute" track fitter swap would have
gone away if we had cycled the power on the crate.  These
symptoms look not completely unlike other recent cases in
which SVT failed to send data to L2 for internal reasons
(as opposed to e.g. SVX reasons).  We went to the test
stand and checked out both an old board and the new board.
Both agree at the same level with the simulation, once the
difference in phi computation is accounted for (i.e. the 
sign bit problem in phi).  It seems that some overflow 
status bits are not always in agreement between the simulation
and the Track Fitter, but this is pretty minor stuff.
I think if we swap the same TF again tomorrow, it will work.
This time, let's do it between stores and have plenty of time
to replay old hits and look at the monitoring distributions.
<p>

<!-- Comment: Sat Sep 14 10;13;36 comment by...Bill & Taka -->
Please see also
<a href=
'/cgi/elog/elog.pl?nb=2002&action=view&page=768&frame=2&anchor=18:13:01'>this entry</a> and
<a href=
'/cgi/elog/elog.pl?nb=2002&action=view&page=768&frame=2&anchor=20:10:02'>this one</a> from shift e-log.



<!-- Date: Sat Sep 14 18:55:45 2002 -->
<p> 
We did the TF swap for real today.  We replayed in a  
separate partition using hw=fakehits, and we saw what 
looked like reasonable distributions (e.g. of phi) in 
svtspymon.  Now we're having the shift crew run with 
hw=fakehits during shot setup, so that Level 2 can 
check that we always send something.  (Gotta implement 
that SVT timeout mechanism so that this becomes  
unnecessary....) 
<p>
<!-- Author: Bill & Taka -->


<!-- Date: Mon Sep 16 21:46:35 2002 -->
<p> 
To make sure the recent Track Fitter update was a success, as 
well as to confirm the hypothesis that going back to 4 hits 
per superstrip in the SVX layers would eliminate SVT/svtsim 
mismatches seen in much of August data (which can be emulated 
after a minor patch, still forthcoming), I looked at SVT vs 
svtsim for 204K events in the look area for run 151573. 
<p> 
I find 120 mismatches (0.06% rate).  All but 2 of them are from 
these 4 ladders: SB2W0L3 (0.6% rate), SB2W1L1 (0.9% rate), 
SB4W3L0 (0.7% rate), SB0W7L3 (0.5% rate). 
<p> 
Eliminating these 4 wedges from the comparison leaves us with 1E-5 rate of tracks 
having svt/svtsim mismatch. 
<p> 
This is after having informed the hardware database of the TF 
phi=0 bug fix.  So both the fixed and the pre-fixed firmware are 
correctly emulated, and the status is known from the database. 
<p>  
<!-- Author: Bill A -->
<!-- Comment: Mon Sep 16 21;47;37 comment by...Bill -->
Sorry, that was 204K tracks, not 204K events.


<!-- Date: Tue Sep 17 11:42:02 2002 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-742&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-742' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-743&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-743' width='300'></a></center>
<!-- Author: Matt -->
<!-- Comment: Tue Sep 17 11;43;04 comment by...Matt -->
Both Clist and ISOlist have Nclusters mismatches between simulation and data in the last run 151641 from the last store 1758.


<!-- Date: Wed Sep 18 11:01:08 2002 -->
<pre>There is no board in Slot 16! 
-------------------------------Read L2 prescales 
In Tagging Mode 
Mod done mask read from crate controller: 0x000008c1 
TIMEOUT: Mod done not set for slot 16 (0x0006ffff, cnt 1) 
MonitorBase::doErrorAction: Tried to call Global Error, but none defined? 
TIMEOUT: Mod done not set for slot 16 (0x0006ffff, cnt 2) 
MonitorBase::doErrorAction: Tried to call Global Error, but none defined? 
TIMEOUT: Mod done not set for slot 16 (0x0006ffff, cnt 3) 
MonitorBase::doErrorAction: Monitor Unknown saw 3 sequential errors 
FaultManager: Fatal error, sending L2 timeout directly 
Stop or Recover state 
Stop or Recover state 
</pre>
<!-- Author: Greg -->
<!-- Comment: Sun Sep 22 21;05;59 comment by...Peter W -->
<pre>This was a bug in the printout, it should have been slot 19.</pre>
<!-- Date: Fri Sep 20 19:23:07 2002 -->
<p> 
<pre> 
svt spy buffer dumps from 2002-09-20 17:33 (L2 buffer + BC number) 
b0svt00/11/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt00/11/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt00/11/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt00/12/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt00/20/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt00/20/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt00/20/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt00/21/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt01/11/HB_HIT_SPY     ... ... ... ... ... ... ... 147 056 176 012 003 
b0svt01/11/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt01/11/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt01/12/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt01/20/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt01/20/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt01/20/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt01/21/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt02/11/HB_HIT_SPY     ... ... ... ... ... 01b 00f 147 056 176 012 003 
b0svt02/11/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt02/11/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt02/12/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt02/20/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt02/20/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt02/20/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt02/21/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt03/11/HB_HIT_SPY     ... ... ... ... 156 01b 00f 147 056 176 012 003 
b0svt03/11/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt03/11/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt03/12/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt03/20/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt03/20/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt03/20/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt03/21/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt04/11/HB_HIT_SPY     ... ... ... ... 156 01b 00f 147 056 176 012 003 
b0svt04/11/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt04/11/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt04/12/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt04/20/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt04/20/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt04/20/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt04/21/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt05/11/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt05/11/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt05/11/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt05/12/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt05/20/HB_HIT_SPY     ... ... ... ... ... ... 00f 147 056 176 012 003 
b0svt05/20/HB_ROAD_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt05/20/HB_OUT_SPY     021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt05/21/TF_OSPY        021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt06/12/MRG_A_SPY      021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt06/20/MRG_OUT_SPY    021 06a 073 00f 156 01b 00f 147 056 176 012 003 
b0svt07/10/MRG_A_SPY      156 01b 00f 147 056 176 012 003 100 206 01b 350 
b0svt07/10/MRG_B_SPY      156 01b 00f 147 056 176 012 003 100 206 01b 350 
b0svt07/10/MRG_OUT_SPY    156 01b 00f 147 056 176 012 003 100 206 01b 350 
b0svt07/17/XTFA_TRK_SPY   156 01b 00f 147 056 176 012 003 100 206 01b 350 
 
Everybody tells a consistent story except for b0svt07, which fans out  
the XFT data.  It looks as if SVT processed correctly events up through 
056, 176, 012, 003, but then never sent any output for 100, 206, 01b, 350. 
So we sent data on buffer 0, and there were subsequent L1A's on buffers 
1 and 2 (so 0 was still waiting for L2+readout), then buffer 0, then 
buffer 3, then everybody was full.  This is all consistent with SVT's 
having stopped sending data after the 003 event.  Since this happens in 
every SVT wedge at the same time, the cause must be either all of SVX 
failing to send data or SVT's XTFA board failing to send data.  We would 
need some information from the b0svt00-05 Merger boards to distinguish 
these two cases.  But since there were Silicon L1 Done Timeouts around 
the same time, the former hypothesis seems more likely: no data from 
silicon. 
</pre> 
<p>
<!-- Author: Bill A -->


<!-- Date: Sat Sep 21 13:00:00 2002 -->
<p> 
Went through list of dead/inefficient SVT wedges with Lester, 
since we were both sitting in the control room.  SVT reports 
<p> 
dead: B0W6, B1W3, B1W7, B2W5, B2WA, B4W6, B4W8<br> 
ineff: B0W0, B0WB, B2W7, B3W9, B5W0<br> 
<p> 
Lester says: 
<p> 
B0W6: 123 OK<br> 
B1W3: 1234 possibly OK<br> 
B1W7: hosed<br> 
B2W5: hosed<br> 
B2WA: hosed<br> 
B4W6: 014 OK<br> 
B4W8: 034 OK<br> 
<p> 
B0W0: L2 dead, L4 hosed for the past 4 days (we use 0134)<br> 
B2W7: 023 OK<br> 
B3W9: 0134 should be OK (we're using 0123)<br> 
B5W0: 1234 may be OK<br> 
<p>  
<!-- Author: Bill A -->
<!-- Comment: Sat Sep 21 13;10;02 comment by...Bill -->
and B0WB is also hosed



<!-- Date: Sat Sep 21 15:27:19 2002 -->
<p> 
SVT data are read out through VRB in b0eb11 slot 12.  (See  
<a href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/Crates"> 
table</a>.)  Slot 10 reads out the muon TDCs.  One of these 
two VRBs has been causing a lot of busy timeouts lately; it 
is not clear whether it is SVT or muon.  To help narrow this 
down, I just shortened the SVDD readout from the ghostbuster 
in b0svt07.  It was putting two cables' data into the event 
record.  Now it just puts their word count and timing into 
the event record.  This is controlled by  
spy_fanout_gbReadoutMask, which was '55' and is now '05'. 
See svtvme/src/svtvme_fer.c for details. 
<p>
<!-- Author: Bill A -->


<!-- Date: Sat Sep 21 15:54:45 2002 -->
<pre>b0svt00/7/MRG_A_SPY     ... ... ... ... 038 179 
b0svt00/7/MRG_B_SPY     ... ... ... ... ... 038 
b0svt00/7/MRG_C_SPY     ... ... ... ... ... 038 
b0svt00/7/MRG_D_SPY     ... ... ... ... ... 038 
b0svt00/16/MRG_A_SPY    ... ... ... ... 038 179 
b0svt00/16/MRG_B_SPY    ... ... ... ... ... 038 
b0svt00/16/MRG_C_SPY    ... ... ... ... ... 038 
b0svt00/16/MRG_D_SPY    ... ... ... ... ... 038 
b0svt01/7/MRG_A_SPY     ... ... ... ... 038 179 
b0svt01/7/MRG_B_SPY     ... ... ... ... ... 038 
b0svt01/7/MRG_C_SPY     ... ... ... ... ... 038 
b0svt01/7/MRG_D_SPY     ... ... ... ... ... 038 
b0svt01/16/MRG_A_SPY    ... ... ... ... 038 179 
b0svt01/16/MRG_B_SPY    ... ... ... ... ... 038 
b0svt01/16/MRG_C_SPY    ... ... ... ... ... 038 
b0svt01/16/MRG_D_SPY    ... ... ... ... ... 038 
b0svt02/7/MRG_A_SPY     ... ... ... ... 038 179 
b0svt02/7/MRG_B_SPY     ... ... ... ... ... 038 
b0svt02/7/MRG_C_SPY     ... ... ... ... ... 038 
b0svt02/7/MRG_D_SPY     ... ... ... ... ... 038 
b0svt02/16/MRG_A_SPY    ... ... ... ... 038 179 
b0svt02/16/MRG_B_SPY    ... ... ... ... ... 038 
b0svt02/16/MRG_C_SPY    ... ... ... ... ... 038 
b0svt02/16/MRG_D_SPY    ... ... ... ... ... 038 
b0svt03/7/MRG_A_SPY     ... ... ... ... 038 179 
b0svt03/7/MRG_B_SPY     ... ... ... ... ... 038 
b0svt03/7/MRG_C_SPY     ... ... ... ... ... 038 
b0svt03/7/MRG_D_SPY     ... ... ... ... ... 038 
b0svt03/16/MRG_A_SPY    ... ... ... ... 038 179 
b0svt03/16/MRG_B_SPY    ... ... ... ... ... 038 
b0svt03/16/MRG_C_SPY    ... ... ... ... ... 038 
b0svt03/16/MRG_D_SPY    ... ... ... ... ... 038 
b0svt04/7/MRG_A_SPY     ... ... ... ... 038 179 
b0svt04/7/MRG_B_SPY     ... ... ... ... ... 038 
b0svt04/7/MRG_C_SPY     ... ... ... ... ... 038 
b0svt04/7/MRG_D_SPY     ... ... ... ... ... 038 
b0svt04/16/MRG_A_SPY    ... ... ... ... 038 179 
b0svt04/16/MRG_B_SPY    ... ... ... ... ... 038 
b0svt04/16/MRG_C_SPY    ... ... ... ... ... 038 
b0svt04/16/MRG_D_SPY    ... ... ... ... ... 038 
b0svt05/7/MRG_A_SPY     ... ... ... ... 038 179 
b0svt05/7/MRG_B_SPY     ... ... ... ... ... 038 
b0svt05/7/MRG_C_SPY     ... ... ... ... ... 038 
b0svt05/7/MRG_D_SPY     ... ... ... ... ... 038 
b0svt05/16/MRG_A_SPY    ... ... ... ... 038 179 
b0svt05/16/MRG_B_SPY    ... ... ... ... ... 038 
b0svt05/16/MRG_C_SPY    ... ... ... ... ... 038 
b0svt05/16/MRG_D_SPY    ... ... ... ... ... 038 
b0svt07/10/MRG_A_SPY    ... 038 179 238 370 085 
b0svt07/10/MRG_B_SPY    ... 038 179 238 370 085 
b0svt07/10/MRG_OUT_SPY  ... 038 179 238 370 085</pre>
<!-- Author: Bill A -->
<!-- Comment: Sat Sep 21 15;56;37 comment by...Bill -->
<pre>This time SVX is clearly implicated, thanks to Marco's
improvement (see shift e-log 9/20 20:33) to the spy dump information.
</pre>


<!-- Date: Sun Sep 22 11:01:49 2002 -->
<pre>b0svt00/16/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt00/16/MRG_B_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt00/16/MRG_C_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt00/16/MRG_D_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt00/7/MRG_A_SPY       315 270 10c 26d 170 321 06a 244 144 
b0svt00/7/MRG_B_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt00/7/MRG_C_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt00/7/MRG_D_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt01/16/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt01/16/MRG_B_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt01/16/MRG_C_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt01/16/MRG_D_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt01/7/MRG_A_SPY       315 270 10c 26d 170 321 06a 244 144 
b0svt01/7/MRG_B_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt01/7/MRG_C_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt01/7/MRG_D_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt02/16/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt02/16/MRG_B_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt02/16/MRG_C_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt02/16/MRG_D_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt02/7/MRG_A_SPY       315 270 10c 26d 170 321 06a 244 144 
b0svt02/7/MRG_B_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt02/7/MRG_C_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt02/7/MRG_D_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt03/16/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt03/16/MRG_B_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt03/16/MRG_C_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt03/16/MRG_D_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt03/7/MRG_A_SPY       315 270 10c 26d 170 321 06a 244 144 
b0svt03/7/MRG_B_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt03/7/MRG_C_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt03/7/MRG_D_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt04/16/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt04/16/MRG_B_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt04/16/MRG_C_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt04/16/MRG_D_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt04/7/MRG_A_SPY       315 270 10c 26d 170 321 06a 244 144 
b0svt04/7/MRG_B_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt04/7/MRG_C_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt04/7/MRG_D_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt05/16/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt05/16/MRG_B_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt05/16/MRG_C_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt05/16/MRG_D_SPY      015 070 00c 06d 070 021 06a 044 044 
b0svt05/7/MRG_A_SPY       315 270 10c 26d 170 321 06a 244 144 
b0svt05/7/MRG_B_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt05/7/MRG_C_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt05/7/MRG_D_SPY       015 070 00c 06d 070 021 06a 044 044 
b0svt06/12/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt06/20/MRG_OUT_SPY    315 270 10c 26d 170 321 06a 244 144 
b0svt07/10/MRG_A_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt07/10/MRG_B_SPY      315 270 10c 26d 170 321 06a 244 144 
b0svt07/10/MRG_OUT_SPY    315 270 10c 26d 170 321 06a 244 144</pre>
<!-- Author: Bill -->
<!-- Comment: Sun Sep 22 11;02;56 comment by...Bill  -->
<a href=
'/cgi/elog/elog.pl?nb=2002&action=view&page=793&frame=2&anchor=09:24:03'>This time</a> there is no evidence of a problem from
either SVX or SVT.



<!-- Date: Mon Sep 23 09:52:15 2002 -->
<pre>
 
<b>This one is just a test of the whodunnit script.</b>
 
Merger b0svt07/10: EEreg=0011, FIFOMT=1111, InpEna=1100, Ena&!MT=0000
Merger b0svt07/11: EEreg=0111, FIFOMT=1001, InpEna=1000, Ena&!MT=0000
Merger b0svt00/07: EEreg=1000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt00/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/16: EEreg=1110, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/07: EEreg=1000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 000000000000000000
 
</pre>
<!-- Author: SVT whodunnit script -->


<!-- Date: Mon Sep 23 10:37:57 2002 -->
<pre>
 
Merger b0svt07/10: EEreg=0011, FIFOMT=1111, InpEna=1100, Ena&!MT=0000
Merger b0svt07/11: EEreg=0111, FIFOMT=1001, InpEna=1000, Ena&!MT=0000
Merger b0svt00/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt00/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt01/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt01/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt03/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt03/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt04/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt04/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt05/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt05/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 001111111111110000
 
Executive summary: silicon dunnit
 
</pre>
<!-- Author: SVT whodunnit script -->


<!-- Date: Tue Sep 24 08:58:32 2002 -->
<pre>
 
Merger b0svt07/10: EEreg=0011, FIFOMT=1111, InpEna=1100, Ena&!MT=0000
Merger b0svt07/11: EEreg=0111, FIFOMT=1001, InpEna=1000, Ena&!MT=0000
Merger b0svt00/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt00/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt01/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt01/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt03/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt03/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt04/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt04/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt05/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt05/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 001111111111110000
 
Executive summary: silicon dunnit
 
</pre>
<!-- Author: SVT whodunnit script -->


<!-- Date: Tue Sep 24 08:58:32 2002 -->
<pre>
 
Merger b0svt07/10: EEreg=0011, FIFOMT=1111, InpEna=1100, Ena&!MT=0000
Merger b0svt07/11: EEreg=0111, FIFOMT=1001, InpEna=1000, Ena&!MT=0000
Merger b0svt00/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt00/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt01/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt01/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt03/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt03/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt04/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt04/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt05/07: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt05/16: EEreg=1000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 001111111111110000
 
Executive summary: silicon dunnit
 
</pre>
<!-- Author: SVT whodunnit script -->


<!-- Date: Tue Sep 24 15:15:54 2002 -->
<pre>
 
Merger b0svt07/10: EEreg=0011, FIFOMT=1111, InpEna=1100, Ena&!MT=0000
Merger b0svt07/11: EEreg=0111, FIFOMT=1001, InpEna=1000, Ena&!MT=0000
Merger b0svt00/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt00/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/07: EEreg=1000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 000000000000000000
 
</pre>
<!-- Author: SVT whodunnit script -->
<!-- Comment: Tue Sep 24 15;16;42 comment by...wja -->
Oops, was done after recover.



<!-- Date: Tue Sep 24 15:40:10 2002 -->
<pre>I dont know if the previous entries from Bill refer to this: 
 
b0svt02_20020923131913.dat 
b0svt02/7/MRG_A_SPY       04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/11/HB_HIT_SPY     04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/11/HB_ROAD_SPY    04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/11/HB_OUT_SPY     04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/12/TF_OSPY        04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 088 
b0svt02/16/MRG_A_SPY      04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/20/HB_HIT_SPY     04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/20/HB_ROAD_SPY    04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/20/HB_OUT_SPY     04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
b0svt02/21/TF_OSPY        04d 07c 000 003 07f 056 053 14a 053 021 056 01e 053 050 056 06a 076 00c 085 015 
 
This I think explains at least in part what happens in run 151969: there is one single case where one of the SVT track fitters goes out of sync. 
</pre>
<!-- Author: Alex -->

<!-- Date: Tue Sep 24 21:20:00 2002 -->
<pre>
 
Merger b0svt07/10: EEreg=0011, FIFOMT=1111, InpEna=1100, Ena&!MT=0000
Merger b0svt07/11: EEreg=0111, FIFOMT=1001, InpEna=1000, Ena&!MT=0000
Merger b0svt00/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt00/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/07: EEreg=0000, FIFOMT=0111, InpEna=1111, Ena&!MT=1000
Merger b0svt02/16: EEreg=1000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/07: EEreg=1000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 000000100000000000
 
</pre>
<!-- Author: SVT whodunnit script -->
<!-- Comment: Tue Sep 24 23;35;56 comment by...Confused in Los Angeles -->
Bill,   Can you tell us how to read these?   

<!-- Comment: Wed Sep 25 07;15;23 comment by...wja -->
You have to know the topology of SVT.  XTRP comes into crate 7,
is fanned out to the first input of each of the 12 mergers in
crates 0-6.  tracks are collected by the 3 mergers in crate 6
slots 8,16,18, then those 3 outputs are collected by slot 20.
channels have an enable mask.  each channel has a fifo, which
has an empty flag.  if an enabled channel has a non-empty fifo
(when data taking is paused), there are data stuck there,
perhaps waiting for another input on the same merger to see
data.

<!-- Comment: Wed Sep 25 07;21;56 comment by...wja -->
This is another case (like the one Alex pointed out above)
where something odd is happening in the left half of crate
2 (i.e. wedge 4).  To understand it will require the full
spy buffer dump.  I can't tell whether the XTFA in crate 7
sent an extra word to wedge 4 or something bad happened 
within wedge 4 itself.  But it looks fishy, assuming that
the run was really stopped at the time.


<!-- Date: Wed Sep 25 12:54:40 2002 -->
<pre>L2 tests of new firmware. 
Using JTSI1-5 for TSI and KIO32 for PIO. 
Running L2_123_L2_TORTURE_v-4_B0L2PR00_fastloop_statemachine_unpackall 
This is code for the new pipeline fpga.  
The code worked fine out of the box. Must be something to 
writing code as a state machine. 
Typical latency is about 20us. 
There is beam in the machine so trigger rates can get up to  
100Khz, so I prescaled the 0.5 Gev L1 trigger to get 40KHz. 
 
SCN setup the LA to look at the time between the L1A and L2R  
for a given buffer. The minimum seen was 17us. Looking at 
time between succesive L2R, the minimum is 11.5us with an 
average of 24us. This is using the unpackall code.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 12:56:24 2002 -->
<pre>For RUn 152005, we added the pitwib. SVT is sending just the EOE 
word. At 100KHz most of the deadtime is L1DOne deadtime. 
Dialing down the rate to 40KHz, the deadtime was reasonably low. 
There was no redout for this run, but the L2A rate was 300-400Hz. 
</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 14:29:52 2002 -->
<pre>Run  152007. 
 Added sevb to the run. L1A rate is about 20-40KHz, L2A rate 
of 20Hz. (these are the livetime rates). 
 Trigmon hists look good for L2. 
No errors from alpha. 
 
I can force L2 errors by pushing reset on the alpha. The code 
prints out the error, but doesn't force a L2 decision timeout. 
However, since it gets stuck in a loop, the TS gives a decision 
timeout after 6sec.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 15:04:14 2002 -->
<pre>Run 152019: 
  Using L3 and pitwib. No problems in running. 
L1A rate is 25KHz and L2 rate is 150Hz.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 16:29:57 2002 -->
<pre>Run 152027 -another run with pitwib 
 
 This run ended because the alpha hung. Looking on backplane 
dstrobe was held low. This might be the error due to reading 
Reces with Reces not responding.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 16:32:34 2002 -->
<pre>Run 152028: 
 
  Test of isolist only sending pass1 clusters. TrigMon 
still has no simulated clusters, but no obvious problems from 
the hardware. 
Going back to default of isolist sending all 3 clusters.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 16:41:48 2002 -->
<pre>Run 152029 
  Isolist back to normal and looks okay. 
We are also now running with a new isoclique card which works well. 
 
At this point we drop the pitwib from the run.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 19:24:35 2002 -->
<pre>Done with L2 studies.  
Summary the new firmware works well. Now we need a beam test. 
 
Proms are 
JTSI1-5 replaces TSI7-5 chip 
KIO32  replaces  pione4-11 
 
Old proms are in the alpha. 
New proms are in L2 cabinet.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed Sep 25 21:01:59 2002 -->
See 
<a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=803&frame=2&anchor=21:01:04'>main shift log book</a> about GhostBuster testing.  
<!-- Author: Taka & Bill -->


<!-- Date: Fri Sep 27 16:05:54 2002 -->
<pre>Last store was the first with the new "production" 
IsoCliques,  this was the first time in history we had  
zero energy errors with beam for about 30K Trigmon  
events.  We still had eta errors 
which are of much less importance,  we will 
reevaluate these after we switch to pass=1 running.   
I would like to run for a week  
with no changes to see if energy errors return, and 
if the problem of Isolist not sending data returns. </pre>
<!-- Author: Steve Kuhlmann -->


<!-- Date: Sat Sep 28 14:56:42 2002 -->
On Friday afternoon, according to the main shift log, there was 
a beam test of the new L2 firmware that was tested without beam on Wednseday.  I can't find anything in either log about whether the beamtest was successful, failed, or something in between.
<!-- Author: David -->