<!-- Title: 13:17:22  Wed Jun 13 2007 -->
<!-- Crewchief: Usual Suspects -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<h2>TDC Problems and Repairs for Early 2007</h2>


<!-- Date: Wed Jun 13 13:26:41 2007 -->
<b>Spares Inventory</b> 
<ul> 
<li>LVDS Rev C, OK for XTC: 3 
<li>LVDS Rev C, bad for XTC: 9 
<li>LVDS Rev C, single channel problems: 2 
<li>LVDS Rev F, OK for XTC: 1 
<li>ECL Rev D: 0 
<li>ECL Rev F: 5 
</ul> 
In principal we can generate an ECL/D and LVDS/F spare by rearranging cards.  
<!-- Author: jdl -->


<!-- Date: Wed Jun 27 09:42:35 2007 -->
<pre>Nils took the flaky TDCs #259 (ECL), #313 (LVDS), #827 (LVDS) equipped them with mezzanine boards and sent them to Baylor for their test stand.  Baylor is sending back 3 boards that we can test out for possible spares.</pre>
<!-- Author: jdl -->


<!-- Date: Wed Jun 27 09:47:21 2007 -->
In the past 10 days or so, we have had two power trips on the west side that have led to TDC problems: 
<ul> 
<li>cot00, slot 3 
<li>cot09, slots 2 & 15 
<li>cot06, slot 10 
</ul> 
The board in cot06 is recovered by power-cycling the crate.  The other three must be reseated to be recovered.  Jianrong told me that she had tried swapping at least one of the boards in cot09 when the problem occured previously, but the problem persisted.  She suspects the crate.  That seems strange, but we live in strange times.
<!-- Author: jdl -->


<!-- Date: Wed Jun 27 09:54:11 2007 -->
The TDC in cot01, slot 6 (S/N 546) has been giving T0 problems.  See <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2007&action=view&page=531&frame=2&anchor=18:02:39.303'> 
evening shift yesterday</a> for details.  Access has been requested.
<!-- Author: jdl -->
<!-- Comment: Wed Jun 27 22;30;25 comment by...jdl -->
<pre>I got the wrong serial number.  It was #710.  It was replaced today by #670 which has mezzanine card #59. </pre>


<!-- Date: Wed Jun 27 22:44:58 2007 -->
Eric and I took another look at the card (S/N 603) that we pulled a few months ago from the COT that had a large t0 offset.  While this card shows a large offset in the calibration, when we pulsed it from the backplane, we saw the signal at the TDC chip input at the same time referenced to CDF clock as on a normal board. 
<p> 
Today, we compared to boards 164 and 719.  We applied an LVDS input pulse at the front panel and looked at the output of the DS9032 receiver chips.  We saw that the signal appears 9ns faster on the bad board.  We saw a similar offset on board 710 that was pulled today. 
<p> 
The normal propogation delay on a reciever seems to be of order 30ns. The strangest thing is that if there were a difference in receiver chip batches or a natural variation among components, we would expect channel-to-channel variations.  However, the uniformity of calibrations among good boards is quite high, while the bad boards are uniformly off by about 8ns on all channels.  Moreover, boards have gone from the normal state to the bad t0 state as was the case on board 710.  The only thing we could think of that was global to the board that might affect the receiver is the power voltage.  We checked both boards and got 4.90V, compared to 4.99 on power lug of the crate.   
  
<!-- Author: jdl -->


<!-- Date: Fri Aug 10 12:22:47 2007 -->
<pre>I have been testing some of the XFT TDC boards for the last few days using Dr. Krumnack's program and the Registry Test 
 
Here are the results of the registry test: 
 
Board 684: PASSED 
Board 342: PASSED 
Board 171: PASSED 
Board 586: PASSED 
Board 733: PASSED 
Board 719: PASSED 
 
*Note: Board 719 was labeled "NO XFT" but it seems to be able to complete the tests even though it fails so should it be "BAD XFT" instead? 
 
 
Board 307:  
 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONread: tried to read register 0x02041c68 on slot 11; read ffffffff and returned 5 
Testing register address (VME): 0x02041c68. w: 01 r: ff x: fe FAILED 
 
 
Register test failed. 304 discrepancies found. 
value = 304 = 0x130 
tstxft2-&gt; on slot 11; read ffffffff and returned 5 
w: 08 r: ff x: f7 FAILED 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONwrite: tried to write 0x04000000 to register 0x02041c68 on slot 11; returned 5 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONread: tried to read register 0x02041c68 on slot 11; read ffffffff and returned 5 
Testing register address (VME): 0x02041c68. w: 04 r: ff x: fb FAILED 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONwrite: tried to write 0x02000000 to register 0x02041c68 on slot 11; returned 5 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONread: tried to read register 0x02041c68 on slot 11; read ffffffff and returned 5 
Testing register address (VME): 0x02041c68. w: 02 r: ff x: fd FAILED 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONwrite: tried to write 0x01000000 to register 0x02041c68 on slot 11; returned 5 
FISION: S_errno_EIO (0x5): VME BERR received! 
Error during VISIONread: tried to read register 0x02041c68 on slot 11; read ffffffff and returned 5 
Testing register address (VME): 0x02041c68. w: 01 r: ff x: fe FAILED 
undefined symbol: slot 
 
 
Earlier I could not get Dr. Krumnack's software to recognize this board.</pre>
<!-- Author: Scott Ruhnau -->


<!-- Date: Fri Aug 10 12:35:05 2007 -->
<pre>I have been testing some of the TDC Boards for the last few days using Dr. Krumnack's program and the Registry Test. 
 
 
Here are the results from Dr. Krumack's Program 
 
 
*These were obtained with the new version of the program* 
Board 342 
XTC 133 
 
Test 0 
 
Found 0 errors. 
value = 16 = 0x10 
 
Test 1 
 
48/100: 48/100:1ffffe 
49/104: 49/104:1ffffe 
50/108: 50/108:1ffffe 
51/10c: 51/10c:1ffffe 
60/101: 60/101:1ffffe 
61/105: 61/105:1ffffe 
62/109: 62/109:1ffffe 
63/10d: 63/10d:1ffffe 
72/102: 72/102:1ffffe 
73/106: 73/106:1ffffe 
74/10a: 74/10a:1ffffe 
75/10e: 75/10e:1ffffe 
84/103: 84/103:1ffffe 
85/107: 85/107:1ffffe 
86/10b: 86/10b:1ffffe 
87/10f: 87/10f:1ffffe 
Found 16 errors. 
value = 17 = 0x11 
 
Test 2 
 
49/104: 49/104:1ffffe 
50/108: 50/108:1ffffe 
51/10c: 51/10c:1ffffe 
60/101: 60/101:1ffffe 
61/105: 61/105:1ffffe 
62/109: 62/109:1ffffe 
63/10d: 63/10d:1ffffe 
72/102: 72/102:1ffffe 
73/106: 73/106:1ffffe 
74/10a: 74/10a:1ffffe 
75/10e: 75/10e:1ffffe 
84/103: 84/103:1ffffe 
85/107: 85/107:1ffffe 
86/10b: 86/10b:1ffffe 
87/10f: 87/10f:1ffffe 
Found 15 errors. 
value = 17 = 0x11 
 
Test 3 
 
Found 0 errors. 
value = 16 = 0x10 
 
 
In test 1 the errors come grouped in sets of 4 consectutuive erros, it seems like the timing is off on this board. 
 
 
Board 530 
XTC 133 
Unable to get software to recognize board 
 
 
Board 530 
XTC 150 
Unable to get software to recognize board 
 
 
Board 719 
XTC 150 
 
Test 0 
Found 0 errors. 
value = 16 = 0x10 
 
Test 1 
0/000: 0/000:1ffffe 
1/004: 1/004:1ffffe 
2/008: 2/008:1ffffe 
3/00c: 3/00c:1ffffe 
12/001: 12/001:1ffffe 
13/005: 13/005:1ffffe 
14/009: 14/009:1ffffe 
15/00d: 15/00d:1ffffe 
24/002: 24/002:1ffffe 
25/006: 25/006:1ffffe 
26/00a: 26/00a:1ffffe 
27/00e: 27/00e:1ffffe 
36/003: 36/003:1ffffe 
37/007: 37/007:1ffffe 
38/00b: 38/00b:1ffffe 
39/00f: 39/00f:1ffffe 
Found 16 errors. 
value = 17 = 0x11 
 
Test 2  
48/100: 
49/104: 
50/108: 
51/10c: 
52/110: 
53/114: 
54/118: 
55/11c: 
56/120: 
57/124: 
58/128: 
59/12c: 
60/101: 
61/105: 
62/109: 
63/10d: 
64/111: 
65/115: 
66/119: 
67/11d: 
68/121: 
69/125: 
70/129: 
71/12d: 
72/102: 
73/106: 
74/10a: 
75/10e: 
76/112: 
77/116: 
78/11a: 
79/11e: 
80/122: 
81/126: 
82/12a: 
83/12e: 
84/103: 
85/107: 
86/10b: 
87/10f: 
88/113: 
89/117: 
90/11b: 
91/11f: 
92/123: 
93/127: 
94/12b: 
95/12f: 
Found 48 errors. 
value = 17 = 0x11 
 
Test 3 
 
Found 0 errors. 
value = 16 = 0x10 
 
 
I believe these errors are with the timimg. 
 
In test 1 all the errors are in the first half of the channels while in test 2 the whole 2nd half has errors. I don't know why the errors are blank but I would say that since the errors only apear in either prompt or delayed test and are affecting a variety of locations on the board that the error is in the timing. 
 
 
 
*These results were obtained with the old vesion of the program 
 
The complete results from the tests are here: 
https://hep.baylor.edu/elog/ruhnau/22 
https://hep.baylor.edu/elog/ruhnau/21 
 
I have provided some of the data and my analysis here: 
 
 
Board 586 
The board consistently showed these errors 
 
2/08: 
6/18: 
10/28: 
 
Since the second didgit is the same and the first digits are 0,1,2 this probably means there is an error in one output cable for all three time slices.  So an output cable is most likely damaged 
 
 
Boards 171 
This board ocassionally shows this error  
26/0a: 
 
Which suggests a problem in one of the inputs. 
 
 
Board 733 is showing errors on each channel and a variety of errors in those channels. There is something wrong with this board but I am not sure what it is. 
 
 
Board 684 is showing these errors occasionally and showing no errors when paired with certain XTCs 
26/0a: 
72/02: 72/02:2afffe 
They both occur in the first time slice but it different channels meaning there may be a slight error on the timing of the first time slice or something else all together</pre>
<!-- Author: Scott Ruhnau -->
<!-- Comment: Thu Aug 23 12;28;20 comment by...jdl -->
<pre>Here is Nils's summary of the above:

171: broken for XTC input line 26
307: broken for mezzanine
342: timing problems
530: broken for mezzanine
586: broken for XTC output line 8
684: broken for XTC input line 26
719: broken VME interface line
733: other kind of error
</pre>

<!-- Comment: Thu Aug 23 12;30;39 comment by...jdl -->
Since channel 26 is used for HTDC in BMU triggers and I don't know what output line 8 is for muon mezz cards, my conclusion is that we can't use any of these TDCs in HTDC slots that have trigger functions.


<!-- Date: Wed Sep 19 15:54:26 2007 -->
<pre>work on etdc00 crate  
 
board 329 has channel 10 dead -&gt; gave to Lou for repairs 
 
board 586 gives done timeouts in both slots 12 and 4  
(after some period of running) -&gt; marked as flaky</pre>
<!-- Author: Eric James -->


<!-- Date: Wed Sep 19 20:00:43 2007 -->
On 9/12, Bill fixed a problem with the etdc00 crate.  We were getting errors associated with the last TDC that is used to read loss monitor counters that is not part of the EMTD bank and had been dropping the EM timing crate from running for a week or so.  Bill's e-mail said, "The boardtype problem is now fixed in the fer version v5_55.  Now the bank list will be sorted in the same order as the global list of TDCs, in case there are multiple banks."
<!-- Author: jdl -->


<!-- Date: Wed Sep 19 20:38:44 2007 -->
Today, Eric and I continued investigation of the t0 offset issue.  We connected an LVDS pulser to the first 12 channels of the TDCs.  We looked consistently at the same channel on many boards.  I think it is #2.  In any event, it is pins 6&7 of the LVDS receiver at A27.  We also looked at the associated mux input.  The times are defined as follows: 
<ul> 
<li>t0 = Input pulse from NIM-TTL level shifter + 8ns (before LVDS fanout), 50% point of leading edge. 
<li>t1 = Input to receiver using differential probe.  Use point where crosses 0.8V from ground level.  (Messy signal) 
<li>t2 = Input to MUX (pin 6) using normal probe. 
</ul> 
Because there is circuitry in the differential probe, they delays are not absolute, but what we are looking for is board-to-board differences.  The results were pretty surprising:<p> 
<table border=0 cellpadding=0 cellspacing=0 width=292 style='border-collapse: 
 collapse;table-layout:fixed;width:219pt'> 
 <col width=64 span=3 style='width:48pt'> 
 <col width=100 style='mso-width-source:userset;mso-width-alt:3657;width:75pt'> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 width=64 style='height:15.0pt;width:48pt'>Board</td> 
  <td width=64 style='width:48pt'>t1-t0</td> 
  <td width=64 style='width:48pt'>t2-t0</td> 
  <td width=100 style='width:75pt'>Comment</td> 
 </tr> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 align=left style='height:15.0pt'>603</td> 
  <td align=left>18.8</td> 
  <td align=left>18.8</td> 
  <td>Offset board</td> 
 </tr> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 align=left style='height:15.0pt'>733</td> 
  <td align=left>18.8</td> 
  <td align=left>18.8</td> 
  <td>Reference</td> 
 </tr> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 align=left style='height:15.0pt'>653</td> 
  <td align=left>19.8</td> 
  <td align=left>19.8</td> 
  <td>Reference</td> 
 </tr> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 align=left style='height:15.0pt'>710</td> 
  <td align=left>18.1</td> 
  <td align=left>19.5</td> 
  <td>Offset board</td> 
 </tr> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 align=left style='height:15.0pt'>690</td> 
  <td align=left>17.8</td> 
  <td align=left>19.5</td> 
  <td>cot07/16</td> 
 </tr> 
 <tr height=20 style='height:15.0pt'> 
  <td height=20 align=left style='height:15.0pt'>504</td> 
  <td align=left>18.1</td> 
  <td align=left>18.8</td> 
  <td>cot07/14</td> 
 </tr> 
 <![if supportMisalignedColumns]> 
 <tr height=0 style='display:none'> 
  <td width=64 style='width:48pt'></td> 
  <td width=64 style='width:48pt'></td> 
  <td width=64 style='width:48pt'></td> 
  <td width=100 style='width:75pt'></td> 
 </tr> 
 <![endif]> 
</table> 
<p> 
Because the offset boards all look the same with the test pulse and the time offset boards have different calibrations from Michigan, the sensible conclusion is that there is a difference in the analog input.  We thought we had confirmed that when we first looked at this problem on a scope several months ago.  As of last week, I could not reproduce any significant timing delay.   
<p> 
The boards from cot07 are a big clue.  They have an offset calibration from Michigan.  They had been giving "normal" timing in COT calibrations (i.e. where the ASD is pulsed).  That would mean that the input continues to be offset and the application of the Michigan calibration makes them look the same as other boards.  We pulled them this afternoon.  They looked normal on the bench, and we put the back.  Eiko now reports that the calibration has shifted by 6ns.  Since these boards have the offset Michigan calibration, that means that by hot-seating the boards, we removed the offset. 
<p> 
This is significant.  We don't know the Michigan calibration procedure.  I don't recall if we power-cycled the cards when we did the bench study last time.  Since last week, I've been hot-seating the cards every time.  So, my latest hypothesis is that if we hot-seat a card with a timing offset, it will go away.  What I don't know is whether a time-offset board will shift back if we cycle the power on the crate.  We need to try that next. 
<p> 
In other news, Eric tried moving the board that gave Done Timeouts in slot 12 of the ETDC crate to slot 4 to test my hypothesis about mitigating backplane loading effects by putting the board closer to the Tracer.  It still didn't work. 
<!-- Author: jdl -->


<!-- Date: Thu Sep 20 10:31:26 2007 -->
<pre>TDC SN329 pulled yesterday from system, reporting CH10 dead. Symptom was reproducible in B0 Teststand. This morning I found open net (C_EMPTY_10) TDC10 pin 37 - &gt; N11, pin 12. Net was repaired and card retested OK at B0 Teststand. 
 
Return TDC SN329 to service</pre>
<!-- Author: Lou DalMonte -->


<!-- Date: Thu Sep 20 16:43:43 2007 -->
<pre>More TDC's returned to spares. 
 
TDC SN562 
This TDC never had  failures in B0 Teststand. 
Both on Aug 29 and Sept 20 it was tested w/o failures. 
Mods were  completed Sept 20. Board retested, Passing  test. 
TDC SN562 returned to spares. 
 
TDC SN369 
08/23 Ch19 Fails. Reproduced from previous testing. 
08/27 I found net (DATAOUT3) not connected to data bus. 
Repair with wire. 
This is an ECL Board and does not need mods. 
Passes tests at  B0 after repair. 
TDC SN369 returned to spares. 
 
TDC SN307 
09/11 Fails test. Ch8 -&gt; Ch12 Bad, also Ch54 Bad. 
Found open net (TDC12 pin 27 to TDC13 pin27). Repair with wire. 
Found 2nd open net (D24 pin85 to C12 pin 14). Repair with wire. 
Board has mods. 
09/13 In retesting it was discovered that Ch 72 -&gt; Ch were failing. Back to bench for diagnosis. 
Found 3rd open net between TDC75 pin41 to TDC76 pin 41. 
Net was repaired by flowing solder of vias and pads of  net. 
09/19 Retest at B0 TestStand. Board  passing, repeated testing. 
TDC SN307 returned to spares. 
</pre>
<!-- Author: Lou DalMonte -->


<!-- Date: Tue Sep 25 14:23:52 2007 -->
<pre>More TDC repairs and Return to spares.  
 
TDC SN198  
09/06 Failing Ch14 at Teststand. Reproducible. 
09/11 Thourough testing of nets yields nothing. Replace TDC14. 
09/12 Passes test at Teststand. Reproducible. 
Send to WH14 for mods. 
09/25 Mods are done. Passes test at Teststand. Reproducible. 
TDC SN198 returned to spares. 
</pre>
<!-- Author: Lou DalMonte -->


<!-- Date: Tue Sep 25 17:01:15 2007 -->
<pre>Four TDC's were graded for XTC: 
730 -&gt; Grade F 
198 -&gt; Grade B 
562 -&gt; Grade C 
703 -&gt; Grade D</pre>
<!-- Author: Lou DalMonte -->


<!-- Date: Fri Sep 28 11:45:09 2007 -->
<pre>More TDC's moved to spares and some XFT Grading. 
 
TDC SN324 
History of repairs  follow. 
Was failing ch72-&gt; ch77. 
Due to broken net (L2_BUF0_9). 
Repair with wire. 
Also was failing CH25 
Due to broken net (TDC_ACK25). 
Repair with wire. 
Passing test, reproducible. 
Mods were done at WH14. 
09/28 Pass final testing. Reproducible. 
TDC SN324  moved to spares. 
 
3 TDC's were graded for XTC:  
324 -&gt; Grade F  
537 -&gt; Grade C  
503 -&gt; Grade B</pre>
<!-- Author: Lou DalMonte -->


<!-- Date: Fri Oct 12 11:25:50 2007 -->
<pre>Compared 1.5 and 3.3V in cot10/17 that has had 1/2 board with offset calibration to normal card.  Both cards had correct values.  When loaded 1.5V sagged to 1.37V.  Compared top and bottom of slot 17.  Was all the same.  Need to take another calibration to see if problem is still there.</pre>
<!-- Author: jdl -->