<!-- Title: 03:01:13  Tue Nov 25 2003 -->
<!-- Crewchief: SVT people -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New page</pre>


<!-- Date: Tue Nov 25 03:11:34 2003 -->
We had large number of "L2 decision timeout" errors in the first collision with SVX after shutdown. <br> 
L2 people is sure that L2 itself is good enough. <br> 
I tried fakehits mode, L2 torture, cosmic ray run (w/ SVX), but 
SVT is fine for these runs. <br> 
Reiner, Sal et al for SVX group is also sure that SVX (in particular, SRC system) is OK. <br> 
We are totally confused that we have such errors. <p> 
 
One possibility of having this problem is final GB readout configuration changing. But, I tried both old and new configurations and had same symptom. <br> 
This looks like not a source of the problem. 
(In that sense, SVT does not change any configuraiton.) 
 
Anyway, in next collision, Alex and I follow the data using 
spydumps.  
   
   
<!-- Author: Taka -->

<!-- Date: Tue Nov 25 19:07:14 2003 -->
<pre>
 
Merger b0svt07/10: EEreg=0001, FIFOMT=1111, InpEna=0001, Ena&!MT=0000
Merger b0svt07/11: EEreg=0001, FIFOMT=1111, InpEna=1110, Ena&!MT=0000
Merger b0svt00/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt00/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt01/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt02/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt03/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt04/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/07: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt05/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/08: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/16: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/18: EEreg=0000, FIFOMT=1111, InpEna=1111, Ena&!MT=0000
Merger b0svt06/20: EEreg=0001, FIFOMT=1110, InpEna=1110, Ena&!MT=0000
 
Ena&!MT summary: 000000000000000000
 
</pre>
<!-- Author: svt whodunnit script -->



<!-- Date: Wed Nov 26 01:55:59 2003 -->
We (Alex and I) tried several tests to terminate error from SVT.  (L2 decision timeout) <br> But, most crucial thing is  following: 
After final MRG (slot#20) in b0svt06, L1 accept and SVT data 
are both existed just before level2 decision timeout error event. 
We checked this using Osciloscope. But, we don't have any SVT data before the L2 slot#12 (tracklist). <br> 
This means that GhostBuster board in slot#6 is guilty!!!!!!!!!!!! <br> 
We have a few candidates for this error. <br> 
(0) Merging functionality into one board is wrong: but this   board is already 
   used from Sep-2003 before shutdown. So, this is not a source. 
   <br> 
(1) SVTD and SVDD readout shouldn't be used in slot#6 chip#1. 
   but, if we changed readout chip from slot#6 chip#1 to slot#13 
   chip#0, we still had a same error (L2 decision timeout). 
   So, this should also be excluded. <br> 
(2) GhostBuster slot#6 has something broken. Personally, I think  
   this is most likely since this only explains that above two  
   items.I need to make spare GhostBuster board to confirm this. 
   (If I can use teststand PC) <br> 
 
  I could think item (1) also has small possibility since we have  
 several types of L2 decision timeout. <p> 
 
  Anyway, Operation manager thinks that current error rate is 
 acceptable. (L2 decision timeout / 5min. @ ~5.0x10^30). 
 Now physics run is running. 
 We don't know whether this acceptable situation continues or  
 not at high rate run. 
   
  
 
    
   I don't know this is possible, but even if we use   
     
<!-- Author: Alex and Taka -->


<!-- Date: Wed Nov 26 02:46:45 2003 -->
svtsim and board output is different at 8% level (reported from 
daily autoSVTmon: usually 0.2% before shutdown). I think we may have problems in physics run. 
<!-- Author: Taka -->


<!-- Date: Wed Nov 26 02:48:01 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030>Beam position y moved to 5.7mm!!. 
We need to create new pattern to avoid decrease the efficiency.</td></tr></td></table>
<!-- Author: Taka -->


<!-- Date: Wed Nov 26 12:47:15 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Let's try to summarize all we know about this "L2 Timeout" problem.  
 
When the problem occurs: 
^^^^^^^^^^^^^^^^^^^^^^^ 
- The SVT chain does not appear to be clogged (all "hold" lines 
  are down, no DS is active, no error LED is set) 
 
- the "whodunnit" output looks ok 
 
- One of the last resorts was the good old oscilloscope. 
  Triggering on L1A we looked at the DS of the last SVT 
  merger (which is before the GB) and we saw the DS firing 
  at least once: the EE word at least is going out 
 
- The spy dumps are not completely clear on what's happening. 
  It would be useful to be able to sample all the spy buffers 
  of one wedge and the b0svt06 spy buffers for one of these 
  events to make sure that the pipeline really works all the 
  way through without any problem. Marco is working on 
  helping us on this one, but the time lag is making things 
  a little difficult! 
 
- L2 is not anymore positive that they are not getting data 
  from us. This conclusion was based on inconclusive evidence. 
  We revised that yesterday night and found out that in 
  fact it is likely that the DS line is being moved. 
 
- we have an unusual number of mismatches between SVT 
  and svtsim in last night's data (8%). Part of these 
  are probably caused by us fiddling with the hardware, 
  but I don't think it's likely for 8% of the time... 
  moreover, after we left (~2AM) there is at least one 
  run which has been taken without any hardware change. 
  AFAIK we see the svt/svtsim discrepancy is  
  there as well! 
 
- it is my understanding that this problem has been showing 
  up occasionaly in the past, leaving all the SVT experts 
  puzzled (yes, this includes Bill A.). It always faded away  
  before further investigation was possible. The assumption 
  has always been that SVX was not sending data through 
  in these cases, but the fact that we do send out an EE 
  word would lead us to argue the contrary now! 
 
Our conclusions so far: 
^^^^^^^^^^^^^^^^^^^^^^ 
 
- SVX/XFT seem to be sending data (otherwise why would we see 
  the EE in the final merger) 
 
- L2 is evidently not receiving data 
 
- Remaining possibilities: 
  - the GB misbehaves in particular timing situations 
    (the problem so far occured only with beam data!!!) 
 
  - the cable between us and L2 has transmission problems 
 
  - the tracklist board gets confused 
 
    - note that last night the L2 folks swapped the 
      tracklist board with a backup, but the error 
      was still present, so if this is a problem 
      between us and the tracklist it must be some 
      kind of protocol issue showing up only in 
      "extreme" situations 
 
 
 
What we can do next: 
^^^^^^^^^^^^^^^^^^^ 
 
- Get a bunch of good probes for the oscilloscope! 
 
- We would like to inspect one complete event in the spy 
  buffers (all of them!!!) for these cases where we have the    
  timeout. We need help from Marco with the spydump guts, and 
  also from the L2 folks to figure out what's the bunch  
  crossing id for the events that cause the mess. 
 
- Put one additional merger at the end of the SVT chain 
  so that we can sample the GB output independently from 
  the GB itself. b0svt07/11 seems an obvious candidate. We 
  tried to use it yesterday but got continuous timeouts. 
  - figure out if merger was configured correctly 
  - understand if L2 needs the hold sense enabled or disabled 
    on the merger's output 
  Once this is solved, we need also Marco to implement the 
  readout of this merger's spy buffer following the protocol 
  used for the rest of the SB, in the same spirit as the 
  previous point. 
 
- swap the GB being used right now with a new one to make 
  sure that we're not dealing with "just" a faulty piece 
  of hardware. 
 
- bypass the GB. We tried this yesterday, taking the L2 input 
  cable and attaching it to directly to the final merger's output 
  but in this condition we kept getting very frequent L2 buffer 
  ID mismatches! Anybody has ideas on this one??? 
 
- Have the SVT "sniffer" board working (we tried to use it  
  yesterday, but still with not much success... data wasn't 
  able to flow through the card). We could use some help 
  from somebody who used this before (franco?). 
  Once this is working we can go around on cables and intercept 
  the EE words when things go belly up. This would be 
  a cross check that the content of the spy dumps is 
  consistent with the "last event", and that we do get 
  data from SVX. 
 
- Look at the tracklist board's input with a logic analyzer, 
  triggering on the infamous "last L1A before timeout". 
 
</pre></td></tr></td></table>
<!-- Author: Alex & Taka -->


<!-- Date: Wed Nov 26 20:23:13 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-164&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-164  width='300' align='top'></td><td>All svtsim/SVTD difference is come from  wedge#10. I will investigate this one more.</a></td></tr></table>
<!-- Author: Taka -->
<!-- Comment: Wed Nov 26 20;28;07 comment by...Taka -->
<pre>Wedge#10 corresponds to b0svt05. (As you know).</pre>
<!-- Date: Wed Nov 26 20:26:45 2003 -->
Many people complain that we don't have beam position for L2 and 
SVDD. I found that beam position text file "lastbeam.txt" was never updated. What's happen?? 
 
 
<!-- Author: Taka -->
<!-- Comment: Wed Nov 26 20;28;50 comment by...Taka -->
<pre>typo!! not for "L2" but "L3"</pre>


<!-- Date: Wed Nov 26 23:23:08 2003 -->
 
 
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12563&button=yes' target='xxx' 
onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12563' width='300'></a><a 
href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12564&button=yes' target='xxx' 
onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12564' 
width='300'></a></center> <BR> 
<cite>-- Wed Nov 26 23:16:49 comment by...Taka --  </cite>Left: Impact parameter distribution from Online 
monitor (run#175078), right: impact parameter vs phi from offline data (run#175075). Online board has 
good beam information (and impact parameter), but offline has odd distribution?? Couldbe a bug, but very 
very strange!!. <BR> 
<cite>-- Wed Nov 26 23:18:31 comment by...Taka --  </cite>Information used here here are come from 
SVTSPYMON (left), SVTD (right).   
<!-- Author: Taka -->
<!-- Comment: Thu Nov 27 13;01;18 comment by...rescigno -->
<table border=0><td><tr><td bgcolor=#e0e030>I believe this could only be an artifact of what you are plotting here. SVTD data seems to make sense. It is the SVDD info that you may use for the second plot that is corrupted and hence make you see the sinus wave. Are you plotting the simulated track impact parameter? This may explain your findings!
  </td></tr></td></table>


<!-- Comment: Thu Nov 27 14;52;29 comment by...Taka -->
<table border=0><td><tr><td bgcolor=#e0e030>This is actually both svtsim and SVTD at same time.
svtsim/SVTD has quite similar tendency except for wedge#10.

</td></tr></td></table>
<!-- Date: Thu Nov 27 13:53:40 2003 -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> 
<html> 
<head> 
   <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> 
   <meta name="GENERATOR" content="Mozilla/4.77 [en] (X11; U; Linux 2.4.2-2 i686) [Netscape]"> 
</head> 
<body> 
Investigating the SVDD problem.... 
<p>Here is a typical hex dump of the current SVDD&nbsp;bank. (run 175066): 
<p>AC++Dump cannot print() the SVDD,&nbsp;complaining about pointers corrupted 
, but just dump it.<tt></tt> 
<p><tt>Printing contents of SVDD, 1</tt> 
<br><tt>**********************************************************</tt> 
<br><tt>Dumping Bank with Bank Name&nbsp;&nbsp;&nbsp; = SVDD</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Bank Number&nbsp; = 1</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Bank Version = 0</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Bank Type&nbsp;&nbsp;&nbsp; = 67I4</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Body Size&nbsp;&nbsp;&nbsp; = 68</tt> 
<br><tt>----------------------------------------------------------</tt> 
<br><tt>&nbsp;</tt> 
<br><tt>Bank Type: as unsigned int (4-byte words)</tt> 
<br><tt>&nbsp;&nbsp;&nbsp; Index&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3</tt> 
<br><tt>---------&nbsp; -------- -------- -------- --------</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0: 00000003</tt> 
<br><tt>&nbsp;</tt> 
<br><tt>Bank Data: as unsigned int (4-byte words)</tt> 
<br><tt>&nbsp;&nbsp;&nbsp; Index&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3</tt> 
<br><tt>---------&nbsp; -------- -------- -------- --------</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0: 00000008 0000000a 
0000000c 0000000e</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4: 00000010 00000012 
00000014 00000041</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8: 00000043 00000043 
00000000 00000002</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 12: 00000000 00000002 00000000 
00000002</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 16: 00000000 00000002 00000000 
00000002</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20: 00000005 <font color="#33FF33">00000007</font> 
<font color="#3366FF">0000000f</font> <font color="#FF0000">00000007</font></tt> 
<br><tt><font color="#FF0000">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 24: 
00000013 00000007</font> 0000002d <font color="#33FF33">000a00ff</font></tt> 
<br><tt><font color="#33FF33">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 28: 
f74a15cc f7ae15fe f81c1630 f880166c</font></tt> 
<br><tt><font color="#33FF33">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 32: 
f8ee169e f97016d0 02108421</font> <font color="#3366FF">000b00ff</font></tt> 
<br><tt><font color="#3366FF">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 36: 
f84e164e 02b50155 00000000</font> <font color="#FFFF00">000e00ff</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 40: 
00060001 03740358 00000016 00100947</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 44: 
00045417 0006060b 000ec2d3 000b6264</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 48: 
00001000 00201052 00100986 00004e05</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 52: 
00080448 000952a5 00099a4f 00000800</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 56: 
00200257 001009c9 0000e217 00080ddf</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 60: 
000002f5 0008de0d 00000669 0020085b</font></tt> 
<br><tt><font color="#FFFF00">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 64: 
00641050</font> 00000000 00000002</tt><tt></tt> 
<p><tt>Color code:</tt><tt></tt> 
<p><tt><font color="#33FF33">green:</font> pointers and data to the 6 2d 
beam fit (one for barrel)</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; This info looks OK!</tt><tt></tt> 
<p><tt><font color="#3333FF">blue:</font>&nbsp; pointers and data to the 
3D beam info (this is the crucial info)</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; This looks OK too...</tt><tt></tt> 
<p><tt><font color="#FF0000">red:</font>&nbsp;&nbsp; pointers to 3 additional 
card in the block that should contain the</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; data from the three possible 
readout channels of the readout GB.</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <font color="#FF0000">These 
pointers are not healthy to me. In fact only the second one</font></tt> 
<br><tt><font color="#FF0000">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (corresponding 
to&nbsp; the only active channel) is correct. The others</font></tt> 
<br><tt><font color="#FF0000">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; are 
probably just confusing the bank unpacking code.</font></tt><tt></tt> 
<p><tt><font color="#FFFF00">yellow:</font>data from channel 2 of the GB&nbsp;readout, 
currently this data is a replica</tt> 
<br><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; of SVTD (not much of an information 
so).</tt><tt></tt> 
<p><tt>Conclusions:</tt><tt></tt> 
<p><tt>The beam data info itself is healty and present in the SVDD. However 
we should discard all this data from the analysis since level3 won't be 
able to cut on ip.</tt><tt></tt> 
<p><tt>I believe we can solve this problem dropping the yellow info from 
SVDD,</tt> 
<br><tt>changing appropriately the hwset to instruc the readout code not 
to</tt> 
<br><tt>append any data after the beam info, practically disabling the 
SVDD&nbsp;for any other purpose than holding the beaminfo data. It is not 
clear to me whether the</tt> 
<br><tt>present readout code will allow to write anyhow the SVTD&nbsp;bank. 
In this case we may also have to switch back to the old configuration where 
SVT&nbsp;readout is done by the old GB&nbsp;in slot 13, until some likely 
trivial code patch is applied.</tt><tt></tt> 
<p><tt>If my diagnosis is correct we should write asap the appropriate 
piece of</tt> 
<br><tt>code to deal with the situation where we want to append additional 
cards in</tt> 
<br><tt>the same block of the SVDD while preserving the possibility of&nbsp; 
skipping</tt> 
<br><tt>any readout channel of the GB in whichever position it appears 
to be in the cards numbering. It's my understanding that present code is 
only able</tt> 
<br><tt>to skip channels from GB&nbsp;only if those are the first in the 
sequence...</tt> 
<br><tt></tt>&nbsp; 
<br><tt></tt>&nbsp; 
<br><tt></tt>&nbsp; 
<br><tt></tt>&nbsp; 
<br><tt></tt>&nbsp; 
<br><tt></tt>&nbsp; 
</body> 
</html>
<!-- Author: Marco  -->



<!-- Date: Fri Nov 28 07:03:30 2003 -->
<pre>Beam was lost. So I ran diagnostics on AM Boards in wege 10. 
ssh -l cdf_svt b0dap72 
cd tools/am_test 
./am_test_vme 
and 
./am_test_data_loop 
(we have two test programs for AM boards). 
Board #0 (slot 9) passes all tests (100 iterations) 
Board #1 (slot 10 #25) oddily passes the data loop test (100 
iterations) but it always fails the 1st iteration of am_test_vme 
with a road read error from chip 0 of plug 10: 
 
Comparing... 
1600 25/03/11/28/06/58/36 Error: Plug 10: 576 chip 0 road 47 layer 5 w X4B6 r X6B6 
1700 25/03/11/28/06/58/36 Plug 10:576 has 1 errors: 
1710 25/03/11/28/06/58/36    Chip 0 has 1 errors: 
 
 
---------Final test summary--------- 
0860 25/03/11/28/06/58/36 Plug 00:314 all tests OK 
0860 25/03/11/28/06/58/36 Plug 01:587 all tests OK 
0860 25/03/11/28/06/58/36 Plug 02:315 all tests OK 
0860 25/03/11/28/06/58/36 Plug 03:586 all tests OK 
0860 25/03/11/28/06/58/36 Plug 04:313 all tests OK 
0860 25/03/11/28/06/58/36 Plug 05:584 all tests OK 
0860 25/03/11/28/06/58/36 Plug 06:855 all tests OK 
0860 25/03/11/28/06/58/36 Plug 07:733 all tests OK 
0860 25/03/11/28/06/58/36 Plug 08:856 all tests OK 
0860 25/03/11/28/06/58/36 Plug 09:386 all tests OK 
1872 25/03/11/28/06/58/36 Plug 10:576 Chip 0 has 1 errors: 
0860 25/03/11/28/06/58/36 Plug 11:385 all tests OK 
0860 25/03/11/28/06/58/36 Plug 12:387 all tests OK 
0860 25/03/11/28/06/58/36 Plug 13:382 all tests OK 
0860 25/03/11/28/06/58/36 Plug 14:249 all tests OK 
0860 25/03/11/28/06/58/36 Plug 15:384 all tests OK 
1900 25/03/11/28/06/58/36 ******* TEST FAILED ******* at cycle 1 
 
I repeated the am_test_vme on board#25 several times, always 
this one error. 
It is strange that data_loop is OK. But... it is worth 
trying to swap the AM board in slot 10 with a spare one 
and see if the mismatches from svtmon go away. 
 
</pre>
<!-- Author: Stefano -->
<!-- Comment: Fri Nov 28 07;09;43 comment by...Stefano -->
<table border=0><td><tr><td bgcolor=#e0e030>Everybody that wants to use am_test_data_loop, <b>BEWARE</b>,
in order to make it work after the first few iterations, I
had to disable the merger hold on output 2 (hit to HitBuffer).
I did this from merger_menu by writing 1f in the switch register (swr 1f). THis will be reset by next run control config transition.
Sometimes I will change the test program to do this automatically.</td></tr></td></table>

<!-- Comment: Fri Nov 28 07;28;07 comment by...Stefano -->
I modified am_test_vme in my private area to keep going after
errors. All tests after AM read/write are passed, so it appears
that the error from reading back the patterns does not affect
the AM workings. Out of 50 iterations of the test I found only
15 to have a single error in radback, always
from Plug 10, chip 0, road 47, layer 5. Always a single bit error.

<p>
<b>AT THIS POINT I THINK IT IS VERY UNLIKELY THAT THIS PROBLEM AMY CASUE THE MISMATCHES IN SVT VS. SIMULATION</B> ALL IN ALL
<br><b>SWAPPING BOARD 10 IN B0SVT05 IS NOT NEEDED</B>


<!-- Date: Mon Dec 1 04:37:43 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-165&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-165  width='300' align='top'></td><td>Wedge#10 problem. Road Id is something wrong.  (But, Stefano said AM is fine via test program)</a></td></tr></table>
<!-- Author: Taka Maruyama -->


<!-- Date: Mon Dec 1 04:42:03 2003 -->
I was paged at 2:00a.m. since wedge#11 has quite small occupancy. 
(See main e-log: <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=1002&frame=2&anchor=02:05:57.425'> 
here </a> )  <p> 
 
This is due to Track Fitter Problem which might be caused in 
yesterday's power outage. See below figures.
<!-- Author: Taka -->


<!-- Date: Mon Dec 1 04:47:53 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-167&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-167  width='300' align='top'></td><td>Figures.</a></td></tr></table>
<!-- Author: Taka Maruyama -->
<!-- Comment: Mon Dec  1 04;51;27 comment by...Taka Maruyama -->
<pre>SVT was paged at 2:00 am since we do have small occupancy in wedge#11. (from Rainer, Thanks). 
As shown in this figure, this is due to Track Fitter 
board broken (most likely due to yesterday power outage.) 

Left: SVT road ID just before Track Fitter, 
Right:SVT road ID after Track Fitter. 
Top: wedge#0 (typical good one), 
Bottom: wedge#11 (problematic one)
 
Track Fitter (serial#12:slot21) was swapped to serial#10 in test stand. After swapping, it seems that wedge#11 is fine. </pre>


<!-- Date: Mon Dec 1 04:53:15 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-168&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-168  width='300' align='top'></td><td>Road ID before/after TF after board swapping.</a></td></tr></table>
<!-- Author: Taka -->


<!-- Date: Mon Dec 1 04:54:09 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-169&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-169  width='300' align='top'></td><td>SVT occupancy plot after fixing.</a></td></tr></table>
<!-- Author: Taka -->


<!-- Date: Mon Dec 1 05:01:37 2003 -->
At the same time of TF swapping, I also changed AM boards in 
b0svt05, slot#18 and #19:  serial#27->04 (slot18) and serial#24->07 (slot19). This is simply my mistakes since 
histograms were unavailable at my paged time. <p> 
 
Anyway, both spare boards are quite fine right now. 
Serial#27 and serial#24 AM boards are now in teststand.  
   
<!-- Author: Taka Maruyama -->


<!-- Date: Mon Dec 1 08:52:01 2003 -->
Still trying to find what is wrong wtih Wedge 10. 
<p> 
I ran hit buffer random test on the HB of wedge 10. It gives no error. 50K iteration of the test OK so far. If the HB has a problem, it usually shows in <1000 iterations. 
<p> 
So the empty road IDs are not due to HB either. 
<p> 
At this point it is either AMS, or an AM problem that the 
AM diagnostics can not reproduce.
<!-- Author: Stefano -->


<!-- Date: Tue Dec 2 13:51:01 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-173&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-173  width='300' align='top'></a></td><td>Wedge#10 has 30-40% svtsim/SVTD mismatches, recently. 
This is hardwired thing. (day by day, we have gradually worse  
situation it looks like. 
But, Stefano's test shows AMS(B) and HB is not to blame.) 
What should we do??</td></tr></table>
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Dec  2 16;51;15 comment by...Stefano -->
Here is what we should do: run svtsim_diag (addpkg svtsim, gmake svtsim.nobin, gmake svtsim.tbin) and look for events that fail. look at kind of failure, is there a pattern ? dump merger output for some
of those events (i.e. AMS input) and play it through the boards
with Bill's python scripts (e.g.b0dap30:~belforte/vme/python/mrg2tf.py). I can help, but I can not do it all. Anna Maria volunteered to help with second part, someone a bit familiar with svtsim (or willing to become so) may do the first.


<!-- Date: Tue Dec 2 18:50:49 2003 -->
<pre>We set up the logic analyzer to intercept the first EE word 
after L1 Accept and decode the event tag. We did this both 
at the end of the final GB (b0svt06/13) and at the output 
of a TF (b0svt03/21). In both cases we are not able to 
match the event tag we see with the LA with what we 
see on the spy dumps. Maybe something is resetting 
the spy buffers between the last L1A and the time the 
spy buffers are dumped to file? We need feedback from 
Marco on this...</pre>
<!-- Author: Alex Jahred, Taka -->
<!-- Comment: Wed Dec  3 04;50;26 comment by...Stefano -->
Spy buffers are reset every 60 seconds by svtspymon.
I sugegst you disable SVTSPYMON, then do your test, and
use manual freeze and dump of spy buffers when you want to
compare. Instructions on stopping SVTSPYMON are
<a href="http://www-cdfonline.fnal.gov/internal/ops/svt/spymon/instructions_expert.html">here</a>. For freeze/unfreeze/dump you can use
the beautifull graphic interface that Subir wrote, it will
deliver new people from having to learn all those ugly tricks
and scripts that SVT experts have used so far
<pre>
setup svtvme -d
svtgui &
</pre>
<!-- Comment: Wed Dec  3 12;13;31 comment by...Marco -->
Beware that with the fake RT licence we are working now when you stop SPYMON process on the crate you then have to necessarily reboot the crate because there is a memory corruption. You cannot restart it with the svtStart command any more. Also I would not trust the readout code in this instance. "normal" vme and of course SVT are perfectly fine though. 
<!-- Date: Tue Dec 2 19:20:08 2003 -->
<pre>Swapped TF serial #10 out for # 13 in b0svt05, slot#21. Placed #10 in test stand. Un-Ki claims that fake hit output looks fine. 
 
Fake hit output here looks fine, too.</pre>
<!-- Author: Jahred, Taka -->
<!-- Comment: Wed Dec  3 00;21;50 comment by...Taka -->
<pre>In teststand, we have two TFs, one is in slot#12 (occupancy small one) and the other is slot#21 (coldstart might be problematic). </pre>

<!-- Comment: Wed Dec  3 04;44;54 comment by...Stefano -->
I would not worry too much about the problem at cold start in
TF #10. It is not clear that a board swawp was neede in the first place. In several other occasions it was enough to reseat the TF after the crate power cycle. All we know is that sometimes the TF goes into an ill-degined illegal state at power up and does not
respond to some vme comamnds or drives some vme line when it shouldn't. As far as we know it could be a subtle bug in some state machine that only appears in particular power up situations. Maybe in crate b0svt03 the +3 and _5 supplies come up with a particularly unfortunate ramp and have a few % chance to set the TF in the illegal state. But as long as this is fixed by reseating the board, just let it be, we only need to do it a few times a year.


<!-- Comment: Wed Dec  3 08;34;42 comment by...Taka -->
<pre>I agree, Stefano.
I tested with different hwsets (100 times) and power cycle turning on/off (20 times).
This is only for conservative way.

We can swap again to make sure this thing.</pre>

<!-- Comment: Wed Dec  3 08;35;14 comment by...Taka -->
<pre>Tests has no problems.</pre>



<!-- Date: Wed Dec 3 15:23:35 2003 -->
<pre>&lt;pre&gt;Un-Ki should 
report his work but didn't. I guess he re-swapped SVT Track Fitter board at slot21 in b0svt05 from 
serial#13 to no serial number prototype one, but, I don't know what is the reason and where is another 
Track Fitter spare board.(And what validation was performed.) Anyway, he and I tried coldstart test and 
fakehits mode test and they are fine. If you have problems in SVT (with SVX run), please page SVT or call 
Taka.&lt;/pre&gt;  
 
</pre>
<!-- Author: Taka -->
<!-- Comment: Fri Dec  5 17;29;41 comment by...Un-ki -->
Just to be clear. I DID report what work I did in shift-Elog 
right after I finished my TF's work. And I was going to put a detailed report in SVT-elog later. But because of my online account problem, I was not able to access E-log outside FNAL.
My account is just cleared-up. Now I am IN SVT-Elog. </pre>

Anyway, here is a brief summary. </pre>

This Tuesday, I delivered the TF #13 to FNAL after I replaced 10 old EPC1 chips with new one (which contain version 6 TF's firmware).
But the 2nd spare TF (#12) was not happy with new EPC1 chips. I made one set, still this board was not happy. I couldn't try to make another set, because this problem maybe due to this board problem, and I don't have enough EPC1 chips to play same exercise </pre>. So the simple way was to take all EPC1 chips from TF #13 and put them into TF #12. This time, TF #12 was happy in b0svt05 cate. So I decided to leave this TF #12 with all EPC1 chips from TF #13, after few others tests on TF#12, and then took the TF #13 to Chicago. TF#13 has many special hooks for debugging purpose. </pre>
Today, I was able to figure out which EPC1 chips (out of twenty) were bad. Now the TF #12 with fixed EPC1 chips are happy. High-statistics data-flow test (againt SVTSIM prediction) is under way. </pre>
<!-- Date: Fri Dec 5 13:53:33 2003 -->
<pre>New patterns and fit constants loaded. 
Beam position used: 
x=-0.1950, y=0.5900   
Slopes: m_x= 0.715 urad, m_y= 0.396 urad</pre>
<!-- Author: Roberto -->


<!-- Date: Fri Dec 5 16:52:22 2003 -->
<pre>We tested wedge 10 with script mrg2tf.py (which sends 
data from the merger to TF and checks all spy buffer 
data with simulation). 
 
We used few events which showed a discrepancy between  
SVT and svtsim in W10, from run 175283. 
 
We found that at the AMS output few roads were missing 
compared to the simulation. 
 
Missing roads were from board 0 and plugs 0 or 3. 
We have swapped the AM board 0. Now  AM # 20 is in 
b0svt05 and AM # 15 is in the cabinet. 
 
We run again the mrg2tf script, now everything seems ok.</pre>
<!-- Author: Anna Maria and Roberto -->
<!-- Comment: Sat Dec  6 14;53;21 comment by...stefano -->
<pre>are the missing roads the same discrepancyt that was showing in the data ? i.e. is the problem that showed during data taking reproducible ? Note that missing roads can not explain the plots from SVTSPYMON showing roads with no associated hits.</pre>
<!-- Comment: Wed Dec 10 21;59;31 comment by...Taka Maruyama -->
<pre>As far as I know, Anna Maria and Roberto reproduced the
problem with very small statistics. (2 events?)
And, after swapping, there is no error.</pre>
<!-- Date: Fri Dec 5 17:01:55 2003 -->
<pre>Jahred and I have training to put the constant (mapset and 
hwset) to database. So, we have two more mapset after Roberto 
putting, but constant itself is completely same as latest greatest one. 
 
If you have problems, please call Taka or page SVT.</pre>
<!-- Author: Jahred and Taka -->
<!-- Comment: Sat Dec  6 12;35;24 comment by...Roberto -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>The mapset was modified and corrupted during this operation.
I have restored the right version.</pre></td></tr></td></table>
<!-- Comment: Sat Dec  6 22;27;42 comment by...Jahred and Taka -->
<pre>This is obviously our fault.
We didn't check this mapset completely, so that we had mistakes.
For a while, Jahred and Taka won't install mapset.</pre>
<!-- Comment: Sun Dec  7 05;42;57 comment by...Un-ki -->
<pre>Is this mapset for the shifted beam-position? then don't we expect different constant setting?</pre>
<!-- Date: Fri Dec 5 21:49:49 2003 -->
<pre>Can anyone reboot or recycle B0 SVT test-stand crate? It doesn't respond. Thank.</pre>
<!-- Author: UN-ki -->


<!-- Date: Sat Dec 6 02:06:29 2003 -->
<pre> I try to modify the svtvme_fer.c to get final GB configuration with Marco's suggestions. 
Marco thinks that changing the code to get correct card number of SVDD is enough at present.  
And I tried this with fakehits run using software event builder. (we can use offline data) 
 
 Unfortunately, this test was failed with unknown problem. 
Symptom is that "timing data (SVDD cable data)" was omitted  
somehow. And beam position data, so called 3D beam information 
is not correctly filled up. (Using offline analysis, this was  
confirmed). 
 
 We need to investigate this more. 
 
 </pre>
<!-- Author: Marco and Taka -->







<!-- Date: Thu Dec 11 11:33:43 2003 -->
<pre>I finished SVT test (beam poition readout test with latest b0svt06 configuration) I checked this bank readout 
using offline and currently looks this test was succeeded. Marco will check the data of this test, and if it is 
OK, I will install latest hardware configuration.  
 
lookarea data: /cdf/data08/look/itestr/ir02afd5.0001test  (run#176085) is test data. 
 
We have 2D and 3D beam position and timing in SVDD. 
 
</pre>
<!-- Author: Marco and Taka -->


<!-- Date: Thu Dec 11 11:37:39 2003 -->
 
 
<pre>PROCMON complains about SVTSPYMON. I tried to re-start the this process, but somehow disk 
quota does not allow to restart the process. error in b0dap75 is: >cdf_svt@b0dap75a/~/svtspy#45> 
spy_manager >Starting spy_manager: 
>/cdf/code-common/cdfonline/svtmon/devel/bin/spy_manager_Linux.exe >&! SPY_MANA\ 
GER_2003-12-11_11:12:25.log & >[1] 32365 >cdf_svt@b0dap75a/~/svtspy#46> 
SPY_MANAGER_2003-12-11_11:12:25.log: Disk quota \ exceeded. </pre>   
<!-- Author: Taka -->
<!-- Comment: Thu Dec 11 11;39;10 comment by...Taka -->
<pre>So, I disabled PROCMON process right now. </pre>


<!-- Comment: Fri Dec 12 01;33;57 comment by...Marco, Stefano and Taka -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Taka re-started the "SVTSPYMON" process again, so that you can enable for ProcMon for "SVTSPYMON" again. <br>
We erased unnesessary files for spydumps, this is the source of problem. </pre></td></tr></td></table> 
<!-- Date: Thu Dec 11 15:47:38 2003 -->
<pre>Followed the instructions of Un-Ki and was able to upload the latest control/fitter firmware to the track fitter in test stand slot #12 (which had supposedly sometimes had problems on cold start). 
 
Tried to put into wedge 00 and run the variant of mrg2tf.py (w00.py) that Anna Maria was using to send svtsim input from the merger thru the TF and compare output. We used the latest mapset and hw file - but see mismatch between AMS_OUT_SPY for hardware and svtsim (and subsequent mismatch in later boards after that). Not sure what this is/means... (And thus - the TF still has not been tested) 
 
For example, 
 
event 1: nroads=3 
MRG_D_SPY: good (len=150) 
MRG_OUT_SPY: good (len=150) 
AMS_HIT_SPY: good (len=150) 
AMS_OUT_SPY(svt): 340165 3407f6 340b99 340e9a 341e8b 34351f 344432 3449a8 347140 602a8b 
AMS_OUT_SPY(sim): 340834 340b0f 34114b 60208b 
HB_HIT_SPY: good (len=150) 
HB_ROAD_SPY(svt): 340165 3407f6 340b99 340e9a 341e8b 34351f 344432 3449a8 347140 602a8b 
HB_ROAD_SPY(sim): 340834 340b0f 34114b 60208b 
.......</pre>
<!-- Author: jahred, taka -->
<!-- Comment: Sat Dec 13 15;54;54 comment by...stefano -->
I suspect boards have not been initialised with same patterns
as simulation. Otherwise one should think that AM Boards in wedge 0 are broken.. but data from that wedge so far was good.
<!-- Date: Fri Dec 12 19:24:10 2003 -->
<pre>All label numbers on the front side of the TFs are done to be consistent with serial numbers (from ID_PROM register). 
I plan to glue the ID_PROM dip switch setting later.</pre>
<!-- Author: Un-ki -->