<!-- Title: 15:56:22  Sun May  6 2001 -->
<!-- Crewchief: L2 group -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>L2 commissioning during shutdown. 
</pre>


<!-- Date: Mon May 7 00:14:56 2001 -->
b0svt03 failed cold start Friday night; I got e-mail from Ace. 
David Saltzberg and Stephen Miller found that the TF in slot 
21 was complaining that writeMap(TF_SHIFTS) failed.  I suspect 
that the failure is not the shift values themselves but the 
extra word I added to the "shift" pseudo-RAM to make it start 
on an even word boundary.  Anyway, there seems to be some  
state that the TF can get into that the coldstart can't get it 
out of; need to figure it out and add the appropriate VME  
commands.  Power-cycling the board got it unstuck.  (Yuck.) 
<p>  
<!-- Author: Bill -->


<!-- Date: Mon May 7 14:22:58 2001 -->
<pre>Plugged in 24 DCAS boards; the following crates now have a full compliment of DCAS: 2RR28C, 2RR27C & 2RR25C. 
 
</pre>
<!-- Author: Nathan Goldschmidt -->


<!-- Date: Mon May 7 17:21:35 2001 -->
<pre>We're using svt 2,3,4,6,7 and b0l2de01 for Level 2 tests.</pre>
<!-- Author: Matt & Jane -->
<!-- Comment: Mon May  7 20;09;56 comment by...Jane and Matt and Stephen -->
 We are finished with the crates for now.  We successfully tested
 the new firmware with standalone tests.  When we used the SVT
 crates we saw that SVT was sending a lot of data, which is
 interesting but made the Level 2 situation harder to figure out,
 and may have caused the problems seen over the weekend.  After
 talking to Bill Ashmanskas, who advised as to how to change the
 svt data being sent, we were able to send just one word per
 event.  But then other problems came up, so we weren't able to
 finish the test.  


<!-- Date: Tue May 8 10:57:23 2001 -->
<pre>I'm taking svt crates 2,3,4,6,7 and b0l2de01 for Level 2 tests.</pre>
<!-- Author: Jane -->
<!-- Comment: Tue May  8 18;40;25 comment by...Jane and Stephen and Matt -->
I am finished with the SVT crates and b0l2de01.  We successfully
took data using the alphas with the L1 interface board and two
track interface boards receiving data from the SVT and sending
it to two memory locations on the alpha.  
<!-- Date: Tue May 8 12:44:44 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Removed CLIQUE from 2RR28C. 
 
ACE: Don't add L2 calorimeter crates to the 
run until 5/9/01 afternoon.  </pre></td></tr></td></table>
<!-- Author: Nathan Goldschmidt -->



<!-- Date: Wed May 9 14:43:19 2001 -->
<pre>L2CAL crates are back in business; found and changed 2 bad boards in b0l2cl02 and b0l2cl04 which were preventing COLDSTART for those crates. 
Run 114304 taken with ACE_CALORIMETER_MINIMAL confirms that boards are still reading out the correct energies in the corresponding tower, i.e. the changes to the hardware database also work! 
There is still a problem in crate b0l2cl02 (2RR26C) with the daisy-chain energies passed to LOCOS...maybe a backplane open? 
Still investigating. 
</pre>
<!-- Author: monica -->
<!-- Comment: Wed May  9 17;03;29 comment by...monica -->
Yep, problem was an open in the backplane between DCAS in slot 9 and slot 10 for daisy-chained HAD energy, bit(8).

<!-- Comment: Wed May  9 17;04;10 comment by...monica -->
Jumper between those 2 pins in the back of the crate solves the problem.

<!-- Date: Wed May 9 20:07:43 2001 -->

<b>HB/TF movements</b> <BR>
I completed the HB/TF rearrangement. Now all TF's are in the "tracking"
crates, in slots 12/21, all HB's are in slots 11/20. Crate b0svt00 is
completed again with all board in and all cabled. Only the TF outputs
from crate b0svt00 (wedge 0 and 1) are now not connected, because the
cables previously used for HB->TF are not long enough for the new
TF->final merger path. Sometime soon myself or someone of good will
should spend 	15 min trying to find a solution with existing cables, or
clearly defining which new lengths are needed.
<p>
About mergers, note that at present there is no "free" merger, so no
merger is in the test stand. We have 4 additional mergers at FNAL,
actually, they are:
<pre>
1 programmed as XTFA replacemente
2 used to test new firmware, work in progress, not fully functional yet
3 used by Jane in b0l2de01
4 used in b0svt07 to trasparently switch between XTRP and XTRP-emu
</pre>
Of course mergers 3 and 4 can be used as spares, and 1 or 2 can be
repogrammed if/when needed.
<!-- Author: Stefano -->



<!-- Date: Wed May 9 22:09:24 2001 -->
<pre>Run 0x1bed2 was run with INTEGRATION SUMET_2 v-2, but I had 
compiled the latest L2 code for that table. No L2 triggers  
are run, but the svt and track data is handled differently. 
We had seen the Cluster data being overwritten in TL2D by the  
svt data. With this latest code, that problem has gone away. 
 
We took out the svt board for run 0x1bed3 to look at the Cluster 
data and look for errors without having to worry about the 
SVT board.</pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 10 00:37:00 2001 -->
<pre>WE added Isolist into the mix and saw a failure where 1 
of the interface boards would hold boss high and no more 
data would be sent by any board.  
 
Looking at the backplane signals on the scope, the SVT interface  
board appeared to be the culprit, although the WEAREBOSS LED  
wasn't lit on that board. On the scope, it looked like the SVT 
board sent its data but then didn't give up boss. 
For that event, CList had sent it's data before the SVT board, 
whereas usually the SVT board would send its data before Clist. 
(That is an observation, not an explanation of the problem). 
 
If we took out either Clist or the SVT board or Isolist, 
then this problem no longer occurs. 
 
We then took run 0x1bed9(?) without the SVT board in order 
to look at the Isolist and Clist data. It appears that for 
some events, neither board sent any data.</pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 10 00:44:53 2001 -->
<pre>Run 114397 (ox1bedd - all these runs have the word "bed" in them. 
It's not helping to keep me awake). 
 
This run has the Reces boards plugged in, to see if they disrupt 
the other boards. They are not being readout by the alpha. 
We'll look at the data offline to see if there was any affect.</pre>
<!-- Author: Stephen -->
<!-- Comment: Thu May 10 00;53;00 comment by...S -->
Monica saw that Clist didn't send data for some events
with the Reces boards in, so we took them out.

The theory is that Clist doesn't like to have any board
in a higher slot in the crate. (Although the Reces boards
are slaves so shouldn't affect the arbitration lines)



<!-- Comment: Thu May 10 01;00;51 comment by...Stephen, Monica -->
Doing a bankdump, we see that after the first few events,
Clist stopped sending any MBus words (it should always send
at least 1) and that every few events the SVT board didn't 
send any (although it should also send at least 1).



<!-- Date: Thu May 10 01:07:49 2001 -->
<pre>We have frequently had "Alpha no longer processing errors" 
in some runs about every 5K events. This seemed more likely to happen when Isolist was in (but also Reces!). Taking those 
out the Alpha is less likely to hang. 
 
Anyways, I modified the code to put in more checkpoints, 
so when it fails, the spot in the code it failed at will  
be easier to find. Of course now it isn't failing.</pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 10 01:21:14 2001 -->
<pre>Looking at the data in run 114398 (0x1bede), we see 
that Clist always sends MBUS words.  
 
Monica is continuing to track down some single bit errors that 
are showing up in the EM energy word. 
</pre>
<!-- Author: Stephen, Monica -->


<!-- Date: Thu May 10 01:46:49 2001 -->
<pre>Made a new L2_RATE_TEST exe and tried to run it. 
Get about 19KHz L1 rate reduced to 84Hz L2A rate. 
With about 4% deadtime 
 
The alpha then crashed after a few 10K's of events (rejected).</pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 10 02:15:40 2001 -->
<pre>This is where L2CAL system stands after putting the crates back together with changed P3 backplane and new 960A insatalled: 
 
1) clustering still runs very reliably 
 
2) cluster info sent to CLIST has still (eta,phi) problem  
 
3) found open in backplane for b0l2cl02 for daisy-chained EM 
  energy bit 8, between slot 9 and 10. Solved with a jumer. Need    more reliable solution. 
 
4) energy of clusters sent to CLIST has new single bits errors      (EM bit 0 and bit 3) confined to seeds in phi=0 (any eta). 
  This problem shows up only for cluster in pass1 and pass 2,  
   i.e. the energy for pass 0 cluster in same event is OK! 
   Sound like a timing clustering issue; after all clusters in 
  phi=0 are always processed first. 
 
5) CLIST works reliably with SVX interface in slot 13, on top of     L1 inteface in slot 10. CLIST stops sending data when either     ISOLIST (slot 16) or RECESS boards (slot 17 and higher) are in. 
 
6) slot 15 of b0l2cl05 consistently fails at warmstart bit never    at coldstart. 
 
Problem 4), 5) and 6) need further investigation.</pre>
<!-- Author: monica -->


<!-- Date: Thu May 10 18:24:56 2001 -->
<pre>Investigating the 3 board limit problem in the alpha crate. 
Plugged in 
 L1 - slot 10 
 SVT slot 13 
 Clist slot 14 
 Isolist slot 16 
We observe the SVT board will sometimes always assert boss. 
In this state it will sometimes send data or sometimes stop 
sending any data. In either case the other boards won't send 
any data. Sometimes it will stop asserting boss for a short time 
and the other boards will start sending data again during 
that time. 
 
We moved the SVT board to slot 8 to attach the LA.</pre>
<!-- Author: Stephen, Jane, Peter -->


<!-- Date: Thu May 10 19:22:06 2001 -->
<pre>3 board limit cont... 
We trigger on boss being high and startload* going high to 
trigger the LA. Jane observed that the SVT board was in 
state 3 sending dstrobe, but never went to state 3 wait because 
there was a ddone* before the dstrobe*. It left state 3 when 
it sent dstrobe*, but since it didn't see a ddone* it fell 
into no state so ended up in state 0. Normally it goes through 
state 4 to clear the local request line.  
We are uncertain as to how ddone* went low before dstrobe*. 
Perhaps another board sent a dstrobe* when it shouldn't have. 
 
We then pulled CList out and run with just isolist, svt and L1 
interface. In this case, we see that isolist doesn't turn off 
boss after sending it's data, however, it does deassert boss 
once startload* is deasserted. That means when startload* is 
asserted again, the interface boards can keep sending data. 
Note that boss is not kept asserted on every event, but on many 
events. </pre>
<!-- Author: Stephen, Jane -->


<!-- Date: Thu May 10 20:10:27 2001 -->
<pre>We plug the Clist board back in and look at dstrobe on 
the backplane along with MBAD(1:0). We definitely see 
dstrobe being asserted by some board while SVT has the bus. 
Looking at the MBAD lines we conclude that it is Isolist 
that is sending the dstrobe while SVT has boss. 
This means ddone* is asserted before SVT expects it to be, 
which means the SVT state machine ends up in a wrong state 
with boss always asserted. 
 
It seems that with Clist plugged in before Isolist, causes 
Isolist to get the bus and assert dstrobe while SVT thinks 
it has the bus.  
We'll try another test with Isolist plugged into a lower 
slot than Clist. 
</pre>
<!-- Author: Stephen, Jane -->


<!-- Date: Fri May 11 00:46:00 2001 -->
<pre>Continue to study interface board problems. 
 
Several observations. 
1. With all 4 boards in, get the "Alpha no longer processing 
error" much more frequently. Than with fewer boards. 
I will study this later. 
 
2. With SVT  in slot 8, L1 interface in slot 10, Isolist in  
slot 16, running COSMICS V-1 trigger table. I see on backplane 
that the Isolist will often hold boss high until startload 
is deasserted. (This was also noted above) However, 
another strange behavior is that other times the Isolist board 
will assert the MBUS address and drive boss high and dstrobe high 
for a short time (about 100 ns) shortly before startload is 
asserted. In this case it doesn't send data once startload is 
asserted and it doesn't hold boss high for that time. 
Since it drives dstrobe high instead of asserting it low, 
there is no ddone from the alpha.</pre>
<!-- Author: Stephen -->


<!-- Date: Fri May 11 01:29:28 2001 -->
<pre>I put Clist back in slot 14, but put Isolist in slot 12. 
SVt is in slot 8 and L1 in slot 10. 
In this mode, I see behavior similar to when Clist is not in. 
The SVT board doesn't get confused and hang, but Isolist 
sometimes will keep holding boss high for the duration of time 
that startload is asserted. In that case the L1 and SVT boards 
are each sending 1 MBus word, then Isolist sends 1 word but 
then holds boss high so Clist doesn't send anything. 
Some of the time, Isolist asserts address and boss before 
startload (but it doesn't assert dstrobe* low but instead 
drives it high). In that case, L1, SVT and Clist each send  
one word and Isolist sends none. 
 
</pre>
<!-- Author: Stephen -->


<!-- Date: Fri May 11 01:37:05 2001 -->
<pre>The configuration for the above tests was based 
on the ACE_MAY10 settings but with some crates deleted. 
 
Crucial pieces are ccal crates, L1 global, L1CAL, L2CAL, L2de00, 
and b0svt06 and b0svt07 crates. The b0svt06 (SVT_FITTER_06) crate 
should have L2SVT_01 removed from slot13.  
Also, the run parameter SVTMode should be hw=l2test. 
Used COSMICS_v-1 physics table. 
Set TS mode to Standard and L2Mode to L2Processors. 
I also did these tests without readout since I was just 
looking at backplane signals. 
</pre>
<!-- Author: Stephen -->


<!-- Date: Fri May 11 02:49:50 2001 -->
<pre>I ran the L2_RATE_TEST exe and with tagging off, would get 
"Alpha no longer processing" errors frequently. I removed 
the SVT board and these errors disappeared. There must be 
something wrong with the code in how it gets the svt data 
or perhaps in when/how the data is sent to the alpha. 
 
Without SVT and with cutting at L2, the L2_RATE TEST had a 
L1A rate of about 18KHz with a 15% deadtime mostly due to L2. 
Note that there is no hevb or sevb for these tests. 
Looking with the scope, it looks like there is about 17us 
between start_loads (seen in Myron mode which has quick L1A). 
 
 
</pre>
<!-- Author: Stephen -->


<!-- Date: Fri May 11 02:51:42 2001 -->
<pre>With the Clist and L1 interface boards plugged in, I looked 
at the MBUS signals while running L2_RATE_TEST in myron mode. 
The L1 interface board would only send its data on the first 
startload, and not the other 3. The Clist sent dstrobe for all 
4 buffers. This matches what was seen in looking at myron mode 
data from the 36x36.</pre>
<!-- Author: Stephen -->


<!-- Date: Fri May 11 18:57:54 2001 -->
I take b0svt05 for a bit, to test some code changes.  
<!-- Author: Bill -->
<!-- Comment: Fri May 11 20;52;42 comment by...Bill -->
Done.  But note that all .mapset files will need to have
AMS USEQ CRC updated.




<!-- Date: Sat May 12 10:14:30 2001 -->
<pre>We looked at the problem with Isolist and Clist some more. 
The fundamental problem seems to be that boards will pass 
on bossgrout even when they aren't supposed to. The signal 
passed on is a small one (more of a blip) and is not as clean 
as the signal passed on when bossgrout is supposed to be passed. 
The first fix to try is better termination of the boss and 
bossreq lines. We'll try to get that done next week.</pre>
<!-- Author: Stephen and Steve -->


<!-- Date: Tue May 15 12:26:06 2001 -->
<pre>Plan to use all Track Fitters to update the firmwares 
from 2:00 pm to 5:00 pm. I may reset the power of each  
SVT crates.</pre>
<!-- Author: Un-ki -->
<!-- Comment: Tue May 15 18;37;47 comment by...Un-ki -->
<pre>Uploading the update firmware are done successfully.
Few comments: TF #4 fails in the 1st try. But it works 
in the 2nd try. And I noticed that cables from the TF to Merger
were swapped between TF #6(wedge) and #7, no effect because
they went to same merger. But I switch them back to original
positions.</pre>
<!-- Comment: Wed May 16 10;30;31 comment by...Un-ki -->
<pre>The dip switches for the TF's ID  are now properly setup
to assign TF's ID (from 0 to 11).</pre>
<!-- Date: Tue May 15 16:44:52 2001 -->
<pre>The Alpha processor which had been in the top crate, board C-2 
is completely dead at this point. It is likely another bad 
via which needs to be fixed. For now, don't try to use that  
board.</pre>
<!-- Author: Stephen -->




<!-- Date: Thu May 17 17:54:43 2001 -->
All 42 non-spare Hit Finders now have up-to-date firmware. 
Spares are not yet done.  In case of emergency in the next 
few weeks, use L00 Hit Finders as spares.  I hope next month 
to have firmware rewrite code integrated into standard SVT 
software tools. 
  
<!-- Author: Bill -->


<!-- Date: Fri May 18 17:30:26 2001 -->
<pre>I did some tests with several boards in the crate today, and I'll  
try to summarize some of the things that I saw.  
The boards used were: 
 
slot 9: a spare Track Interface board with no data source  
slot 10: L1 board 
slot 11: SVT board sending to MBAD=1 
slot 12: isolist board 
slot 13: SVT board sending to MBAD=2 
slot 14: CLIST board 
 
For a short while, four out of the five boards that could send 
data, were sending data. Sometimes only three boards would send  
data.  The run stopped due to other problems (we were using the  
shift integration run) and when we started again it seemed that  
only two boards were sending data because Isolist was holding  
boss high.  After removing the isolist board it seemed that both  
SVT boards were sending at the same time.  I saw three  
boss_req's,  and two of them seemed to come from L1 and CLIST ( 
first and third), but on the middle one it seemed that the  
address lines were not very clean, as if they were being driven  
by the two SVT boards writing to MBAD 1 and 2.   To check this, I  
used the logic analyzer but I had to move the boards farther apart to do it.    
 
This configuration had these boards: 
slot 9 :SVT sending to MBAD 1 
slot 10: L1 
slot 14: CLIST 
slot 16: SVT sending to MBAD 2 
 
On the logic analyzer I saw that sometimes both SVT boards  
asserted boss at the same time.  I also saw that sometimes the  
board in slot 9 did not give up the bus, but since I couldn't  
correlate it with what the other boards were doing (except the  
other SVT board) it is hard to say what was going on. 
On the scope, I checked 
signals on the backplane, and mostly saw four boards sending  
data.  Occasionally I saw that the SVT board in slot 9 would  
send data first, then the L1 board (asserting Local_req with a  
pulse+blip) then CLIST (where Local_req was not really asserted,  
it was smaller than the blip), and then the SVT board in slot 16. 
  </pre>
<!-- Author: Jane -->


<!-- Date: Mon May 21 14:35:06 2001 -->
<pre>The shift crew gave up b0l2de00 and the svt crates so that I can do level 2 tests.</pre>
<!-- Author: Jane -->


<!-- Date: Mon May 21 18:48:08 2001 -->
We did some work on the SVT-L2 interface.  We tried to reproduce a problem that Stephen reported, where with the SVT board in the crate, where the "Alpha no longer processing in spot 6" message shows up.  A possible theory is that the SVT board  
sends data when the alpha is not asserting Start_load.  We ran for a total of about 150K events using various trigger sets and 
speeds, where the SVT board in slot 9 sent data to MBAD=1 and the  
SVT board in slot 16 sent data to MBAD=2.  The L1 board was plugged in but not receiving data.  We never saw the alpha hang  
with the error, nor did we see the SVT board sending data at 
inappropriate times.  We did see other problems with the SVT boards.  A couple of times they stopped sending data, and we weren't able to determine why.  Resetting the boards fixed the problem, HRR would also do it.  Another problem is that at higher rates, we saw errors in the svt data in the TL2D bank.  We will investigate these problems further as soon as possible.
<!-- Author: Jane Nachtman and Matt Worcester -->


<!-- Date: Mon May 21 19:11:34 2001 -->
<pre>We did some tests with the XTRP - Level 2 interface.  Due to problems with the XFT configuration, no tracks could be sent from XTRP to the XTRP/L2 board, just End-of-Event word.  We were able to run using Run Control, and stopped after 5K events to check the data. The data was being sent to MBAD=2, so it shows up in the TL2D bank in the svt block.  The L2 buffer numbers agree for the entire run. To fully test the firmware in the XTRP/L2 board, multiple-track events must be sent, and this will have to be done as soon as possible.</pre>
<!-- Author: Jane and Matt and Nathan -->


<!-- Date: Tue May 22 12:28:47 2001 -->
I am using b0svt00. Stay out.
<!-- Author: Stefano -->
<!-- Comment: Tue May 22 15;12;16 comment by...Stefano -->
I am finished.


<!-- Date: Tue May 22 18:36:21 2001 -->
Work doen today on SVT cold start 
<ul> 
<li> moved TF for wedge 0/1 in HDWDB to b0svt00 
<li> checked that R_C cold start works on all SVT crates 
<li> fixed AMS microcode CRC in default.mapset : now file is the same as actual microcode on boards 
</ul>
<!-- Author: Stefano -->


<!-- Date: Wed May 23 02:11:11 2001 -->
<pre>Summary of L2 MBus arbitration studies 
 
We concentrated on the L1 interface board and see that 
shortly after the board has asserted BOSS, it passes on 
BOSSGROUT. This would allow another interface board to take 
the bus and send data at the same time, although with only 
Clist sending data, there isn't an actual conflict. 
 
We also observe a second peak on BOSSREQ that is likely a  
reflection of some kind. Removing the termination board 
at one end doesn't have any effect on this signal.  
I then modified the termination on both boards to have 
50 Ohm termination with about 83 ohms to VCC and 125 Ohms to ground. 
 
If I look at the arbitration signals when the alpha is using 
the bus, in a standalone test, I don't see the same reflection 
on BOSSREQ. In fact there is a small downgoing pulse. Also, 
the BOSSGROUT signal is not passed on. Of course the Alpha is 
in slot 3 with the termination board right behind it. 
 
The CList board (in slot 14) doesn't have the reflection 
on BOSSREQ. The Isolist and SVT boards were plugged in but 
were not sending data.</pre>
<!-- Author: Stephen -->


<!-- Date: Wed May 23 10:08:49 2001 -->
<pre>Plan to use B0SVT03 for debugging using randomtest</pre>
<!-- Author: Un-ki -->


<!-- Date: Wed May 23 10:51:24 2001 -->
<pre>I am going to use svt06, svt07, l2de01.</pre>
<!-- Author: Jane -->
<!-- Comment: Wed May 23 19;41;09 comment by...Jane -->
<pre>I'm finished with these crates.</pre>
<!-- Date: Wed May 23 12:10:19 2001 -->
<pre>
SVT crates were not possible to coldstart since yesterday evening due to
missing default.mapset in svt_config area. 
I recovered a 'default.mapset~' from the same area and copied to
marco.mapset and then linked this to a default.mapset.
Plese check the content of this file since I do not know if it
is completely correct. It claims to be a 0123X_wide_fakeXFT_20010426 mapset.
<pre>
<!-- Author: Marco -->
<!-- Comment: Thu May 24 19;08;50 comment by...Stefano -->
The missing file was a fult of mine in having used ci command to save default.mapset modification in RCS sub-directory of svt_config, but I did not realise I had to do ci -l to avoid the file being removed (sort of chec-in + chek-out). The file that Marco put ther was anyhow OK.
<!-- Date: Wed May 23 18:28:17 2001 -->
<pre>Useful information for multiple board tests in the L2 crate is 
now available in TrigMon.  We ran tests with L1, svt, and clist 
interface boards looking at the number of magic bus words sent by 
each board in TrigMon.  To access this information, in a new 
terminal, start TrigMon by typing: 
    &gt; source ~cdfsoft/cdf2.cshrc 
    &gt; setup cdfsoft2 development 
    &gt; runTrigMon [partitionID] 
where partitionID is your partition number.  At the beginning 
of the job you will see the socket server and port information 
for the Display.  Remember it.  Then to see the relevant plots, 
launch the HistoDisplay GUI from another terminal by setting up 
cdfsoft2 development and typing: 
    &gt; HistoDisplayMain & 
In the GUI, click on Automatic Update, enter the socket server 
and port information in the appropriate window and click OPEN. 
You should see a list of folders appear in the large window. 
For L2, you want to double click on the L2TriggerMonitor folder. 
You should see a list of histograms; to view one, double click 
on it.  To see the number of magic bus words sent by each board, 
see the TL2D Diagnostic Word plot.  If you have any questions, 
page TrigMon at 266-2583.</pre>
<!-- Author: Jane & Matt -->
<!-- Comment: Thu May 24 09;18;37 comment by...Stephen -->
<pre>The TL2D Diagnostic Word plot is a very cool plot that Matt made.
The x axis is the interface board number and the y axis is the
number of words sent. The order of boards on the x-axis
is L1 interface, Track, SVT, Clist, Isolist, Muon.
The L1 interface board should always send 1 word. The Clist 
board should always send at least 1 word.
This plot can directly show how well the interface boards
are working with each other.</pre>
<!-- Date: Thu May 24 01:41:39 2001 -->
<pre>Let crate b0svt06 and b0svt05 run all night in partion 6 for SVTmonitoring test</pre>
<!-- Author: Marco Rescigno -->



<!-- Date: Thu May 24 09:44:01 2001 -->
<pre>Summary of L2 arbitration studies: 
 
The problem of L2 interface boards sending data at the same 
time is the result of spurious arbitration signals being  
generated at the wrong time. Specifically, BOSSREQ is sent 
twice and BOSSGROUT is passed on at the start and end of 
a transaction by the board sending data. In this case BOSSGROUT 
should not be sent at all. 
This problem only shows up with the L1 interface and Isolist  
boards. We focused on the L1 interface board, and observed that 
when the TTL drivers which drive the MagicBus data 
are being turned on and off,  ground and VCC on the interface 
board have 0.5 volt fluctuations. This causes the PECL  
arbitration drivers on the board to send out the  spurious  
BOSSGROUT and BOSSREQ signals, as well as put some spikes on 
the BOSS line. We compared ground on the MBus and Crate with 
ground on the interface board and saw a 0.5 V difference in 
their value when the interface board TTL drivers were turning 
on and off.  
 
The Isolist and L1 interface board are based on the same basic 
design. These boards don't have a P1 connector and also were not 
connecting to a row of ground pins on P0. This is in contrast 
to the other boards in the crate which have these connections. 
Additional grounding was added to the L1 interface board but 
this did not have an effect on the problem. 
 
The arbitration circuitry is such that the extra BOSSREQ,  
and BOSSGROUT signals will not have an effect while BOSS is  
asserted. Thus if BOSS is asserted the whole time the TTL drivers 
are being turned on and off, then this problem may be solved. 
New firmware will be made for the L1 and Isolist boards so that 
is the case.  </pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 24 10:00:25 2001 -->
<pre>Additional Observations: 
 
The Isolist board would occasionally continue to hold BOSS high 
after it had sent all of its data. BOSS would stay high until 
START_LOAD* was deasserted. This problem had been observed before 
and new firmware will be made to fix this. 
 
With L1, SVT, Isolist and Clist in slots 9,11,12, and 14,  
it was observed that Clist would stop sending data. It would 
at least send data on the first event but would stop after that. 
If the SVT cable was unplugged, so that it wouldn't send data, 
then Clist would continue to send data. If Isolist and L1  
interface were removed, then SVT and Clist would both send data. 
(Note: With just those two, we did see an occasional error 
were both words sent would end up in the Clist address) 
With L1 interface in slot 10, SVT in slot 7 and Clist in slot 14, 
then all the boards would continue to send data. However, looking 
at the Trigmon plot of MBUS words per interface board, it 
looked like sometimes the MBUS word for 1 board would end 
up in the address for another board. 
If just Clist and L1 were sending data, then this was never a 
problem. 
 
These tests should be repeated once the new firmware is in 
for the L1 and Isolist boards. </pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 24 10:02:44 2001 -->
<pre>The Alpha with the broken via was brought back to Michigan 
as it failed again late in the day on Wed. 
 
I also brought a Reces card to Michigan for tests with 
the Alpha board. 
</pre>
<!-- Author: Stephen -->



<!-- Date: Fri May 25 00:55:40 2001 -->
<pre>We looked at the problem of CLIST and SVT track interface sending 
 data at the same time.  We triggered one scope in front of the 
 rack on LOCALBOSS from either board and used this condition to 
trigger a scope in the back of the rack via some RG-58. 
 
(Hint of the day:  We used AUX OUT from this scope to trigger 
 another scope that was looking at signals on the backplane.  
 Note that you have to go into the calib menu on  the scope to 
change aux out from a calibration sq. wave to "trig out".   The 
 trigger corresponds to the positive leading edge.) 
 
 
 When the SVT board received BOSSGRIN and didn't become boss it 
gave (correctly) a 4.2 V pulse on  BOSSGROUT.   However, when 
the SVT board did keep BOSS, it still created a small glitch 
(&lt;~500 psec) on BOSSGROUT (about 5-10ns later).   Most of the 
 time, this glitch is about 3.6 V which did not cause the CLIST 
 board to think it was getting BOSSGRIN.   HOwever, when we 
triggered the front scope on the pathological case, ie, both 
boards asserted LOCALBOSS, then we saw the glitch was about 
 150--200 mV higher (about 3.7 or 3.8 V) which was enough to cause 
CLIST to think it was boss too.    By the way, there are also  
other glitches on BOSSGROUT which were not associated with 
BOSSGRIN. 
</pre>
<!-- Author: David, Monica, Matt -->


<!-- Date: Fri May 25 01:29:36 2001 -->
<pre>Worked on fixing the clustering. We found that: 
1) LOCOS in crate 00 (SNO 6) had an history of not clearing 
 itself properly at the beginnig of the run.  
 This would cause the clustering hangs observed yesterday which 
 were solved by power cycling the crate.  
 Problem went away when LOCOS was changed with SNO 8. 
2) when DCAS_transtion boards are not installed, they cause the 
 shoulder tower usage bits to randomly float HIGH thus creating 
 spurious cluster propagations. 
3) EM(3) problem in phi 0 is still there even when changing the 
 RIPPLE_START to CRSUM_DONE delay from 100 ns (now the default 
 for all crates) to 150 ns and even 200 nsec. 
4) new CLIST (#2) seems to have some energy and NTOW bits stuck 
 high. Need more standalone testing. 
5) new CLIQUE(#3) works reliably in creating the right clusters. 
 Not clear if it causes more instances of bad CLIST behaviour.     Need more investigation. Will leave it in the crate for the 
 time being.</pre>
<!-- Author: monica and heather -->


<!-- Date: Fri May 25 14:53:18 2001 -->
<pre>Continue to use b0svt03 for randomtest unless someone want to use it.</pre>
<!-- Author: Un-ki -->


<!-- Date: Fri May 25 17:31:13 2001 -->
<pre>We are going to use svt06, svt07, l2de00, and some of the ccal crates to do SVT/CLIST/L2 tests.</pre>
<!-- Author: Jane and Matt -->


<!-- Date: Fri May 25 18:13:07 2001 -->
<pre>I had to switch the minicom cable for the alpha in b0l2de00 since 
the shift crew accidently killed it badly and it seems to be  
hung.  The new minicom is b0l2pr02.</pre>
<!-- Author: Jane -->


<!-- Date: Fri May 25 20:23:08 2001 -->
Results from further clist/svt track interface board testing.  We added a mod to svt track interface board #7 to try and remove the glitch in the BOSS arbitration that was seen <a href= 
'/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=9&frame=2&anchor=00:55:40'> yesterday</a>.  This glitch may have been the cause of the problem where clist and svt track interface were sending data at the same time.  We encountered reformatter errors in run control that prevented looking at the data in TrigMon.  However, we were able to run without sending data through the CSL.  We took several thousand events without triggering on the condition that the 2 interface boards are holding local BOSS at the same time.  Yesterday this condition would come much more frequently.  We didn't check the signals on the backplane to see how much the glitch has been reduced, if at all.
<!-- Author: Jane, Matt -->


<!-- Date: Sat May 26 16:55:46 2001 -->
<pre>We're working on the SVT/CLIST collision problem with Level 2,  
Level 1, SVT, and calorimter crates.  We are testing yesterday's 
modification to the bus arbitration circuit on the SVT board.  
Also, we had all of the calorimeter crates so real clustering 
should have been occurring.  The trigger was integration_sumet2. 
 
At first we had problems getting started, and when logged in to  
the crate cpu we saw fision errors from the tracer.  Power  
cycling the crate helped.   
 
We had CLIST in slot 14, L1 in slot 10, and SVT in slot 7.  We  
were not able to get a run going.  On the backplane it seemed  
like the L1 board was colliding with SVT.  We removed the L1  
board and then CLIST and SVT worked well together. We still saw 
the condition where SVT and CLIST sent data at the same time, 
although it seemed to be at a lower level than previously.  The  
glitch on BOSSGROUT that David, Matt, and Monica observed  
the other night seemed lower, although still there at some level.  
It also  
seemed that the BOSSGRIN signal (looking at the backplane at the 
SVT board's slot) was sort of sick-looking.  The 
run number is 115975 (0x1c507) and we took 310,000 events with  
no done timeouts. 
 
Next we tried switching the order of the boards, with CLIST in  
slot 14 and SVT in slot 17.  In this configuration we had a hard 
time keeping a run going, and we took Run 115981 (0x1c50d) with 
only 1800 events. 
 
Next, we put the SVT board in slot 7, CLIST in slot 14, and  
between them in slot 10 we put an XTRP board that was not  
receiving data, and this made the bossgrin and bossgrout signals 
look better and added delay to the bossgrout signal.  We never 
saw the condition on the scope where CLIST and SVT sent data at 
the same time, although in TrigMon it appears that at a very low 
level something is still going wrong since SVT and CLIST  
sometimes send the wrong number of magic bus words.  This run  
was 115982 (0x1c50e) and had 165K events.</pre>
<!-- Author: Jane, David, Heather  -->


<!-- Date: Sat May 26 16:57:18 2001 -->
I bent a ground pin on P3 in slot 10 of the b0l2de00. 
<BR> 
<BR> 
Sorry.
<!-- Author: David -->


<!-- Date: Sun May 27 21:13:33 2001 -->
<pre>Today I worked on the SVT/L1 collision problems.  After spending 
most of the day fighting with run control, level 3, rt server,  
etc, I finally tried to look at signals on the backplane. 
 
The SVT board was in slot 7, XTRP (with no input) in slot 10, and 
L1 in slot 11.  I could only get the boards to send events for a 
few events, and then they would stop although the run would 
continue.  I observed two types of events. 
 In the first type, L1 gets the bus first, and 
the BOSSGROUT line is very noisy.  The BOSSREQ signal from SVT  
looks very tiny, and in the noise of BOSSGROUT as L1 is releasing 
the bus the SVT board gets the bus, asserts BOSS, and sends data. 
In the second case, it seems as though both boards have the bus 
at the same time, as I saw the MBAD1 line driven neither low  
nor high.  Then BOSS is released, but then asserted again  
(there was another BOSSREQ sent). 
Sometimes in this case BOSS is held forever, stopping any  
further data transfer.  I saw that SVT did not have WEAREBOSS  
(Local boss) true in this case, so I think it is the L1 board  
that holds the BOSS. 
 
Next I tried switching the boards' positions in the crate.  The  
L1 board is in slot 8, SVT in slot 10, XTRP (no data input) in  
slot 11.  This worked much better; signals on the backplane 
looked healthier and Run 116068 went for 60K events 
before the alpha crashed for unknown reasons.  Looking at the  
data with TrigMon shows that the L1 and SVT boards each sent 
one word per event, for every event.  At this point I put CLIST 
in but immediately started having problems with run control again,  
along with b0l1gl00 and the ts. I gave the crates back to 
the shift crew, as I have to go now.</pre>
<!-- Author: Jane -->
<!-- Comment: Sun May 27 21;30;57 comment by...Jane -->
<pre>I'm leaving the L1, CLIST, and SVT boards in the crate, although 
I may take the L1 out if the shift crew can't get things to work.
(They are having trouble -- runs only go for a few thousand
events before the alpha crashes.)</pre>

<!-- Comment: Sun May 27 21;45;43 comment by...Jane -->
<pre>Since the longest run the shift crew could get was only 13K 
events, I took the L1 board out.  But it is interesting that
in those 13K events, TrigMon saw that each board sent one word
per event, so it seemed that there were none of the SVT-CLIST
collisions that we saw previously...</pre>


<!-- Date: Mon May 28 20:16:15 2001 -->
<pre>Today we tried to get L1, SVT, and CLIST working together.  We  
did not succeed.  It seems as though the SVT board can't work  
with the new L1 firmware -- possibly it is taking BOSS in the  
noisy part of the L1 BOSS signal that was introduced with the  
new firmware.  Note that the only way that SVT and L1 would 
work together yesterday is if the L1 board was in a higher  
priority slot.   
 
We also saw lots of alpha errors that we didn't understand, and 
sometimes the alpha could not recover it's executable except by 
taking all of the crates through cold start.  Sometimes simply 
taking b0l2de00 through the states would recover it.  Twice we 
were in a situation where we could not recover any way (rebooting 
the crate, cold start) except by power-cycling. It was not always 
possible to tell if the interface boards were causing these  
problems or something else. 
 
For a while our tests were being messed up because the SVT was 
not sending data in the correct format.  It is hard to say  
when this behavior started, but it added much confusion to the 
entire situation.  We reset the crates to clear it, but don't  
understand why it came up.</pre>
<!-- Author: Jane, Heather, Matt, David -->


<!-- Date: Mon May 28 21:00:14 2001 -->
<pre>I made an updated version of the INTEGRATION_SUMET_2 v-2  
l2 executable. The code was made more robust so that the 
alpha should be less likely to crash. 
 
Please test this code and make note of any errors, 
especially any new ones.</pre>
<!-- Author: Stephen  -->
<!-- Comment: Tue May 29 22;48;21 comment by...Stephen -->
<pre>This new code also checks the number of MBUS words sent,
and if they are greater than some amount, it sets them
to a maximum value and prints out an error message.
This error message is the one that was mentioned in the 
next entry. Apparently this checking still hasn't made the
code robust against crashing.</pre>
<!-- Date: Tue May 29 19:49:05 2001 -->
<pre>Summary of today's Level 2 work.  We started with a working  
configuration of SVT and CLIST, plus an extra board between them. 
We changed the SVT configuration so that three tracks per event 
were being sent, and everything worked fine.  Then we decided to 
try the L1 board once again, and everything broke so we removed 
it.  Things continued in a downward spiral after that, even  
though the L1 board was not in the crate.  We saw errors in the 
minicom window that said that the CLIST board was sending 5K+  
clusters, and it was set to 30.  This would happen at the same 
time as the alpha crashed.  At first this was infrequent, but  
got worse until finally we couldn't get the run to go at all.  
We didn't see the clustering error message every time. We 
tried coldstart, rebooting the crate, even power cycling, none 
of which seemed to help.  Then, after a couple of cycles of this, 
everything started working again and we ran for 385K events with 
SVT + CLIST and no errors reported anywhere.  This was run 
116182.   
 
Next we tried new firmware from Argonne for the L1 board.  It  
was supposed to do the bus arbitration but not enable the data 
lines, which should make the bus arbitration  
less noisy and possibly help us with debugging.  We had the L1, 
SVT, and CLIST boards in the crate. It didn't work, no boards 
sent data but there were no errors from the alpha.  
 
We went back to Friday's L1 firmware which did not work with SVT.  
To speed things up we weren't using the event builder, so we  
couldn't see any data, but now the three boards seemed to work 
together.  We ran for 90K events this way, and then stopped to 
try to put the event builder back in.  But then we couldn't run  
at all, we got VISION errors immediately from b0l2de00 and  
couldn't recover.  We removed L1 board, but it still took a  
couple of iterations to recover -- we saw more of the messages 
about many clusters from CLIST.  Eventually power-cycling worked 
but we saw that CLIST was no longer sending data. 
 
We moved on to XTRP tests, as we wanted to test the handling of 
the data by the newest L2 exec.  We were only able to get things 
to work if the XTRP board was alone, but that could be because 
the XTRP board doesn't have the same magic bus mod that the SVT 
board has to coexist with the CLIST board.  Or it could be the  
same sort of random problems that seem to be happening a lot 
lately.  We took a run with XTRP sending 0-9 tracks per event,  
and we need to analyze the data.  The run number is 116198, and 
has 39K events.</pre>
<!-- Author: Jane N, Matt W, Heather R, Nathan E -->
<!-- Comment: Tue May 29 20;02;01 comment by...Jane -->
<pre>I forgot to put Steve Kuhlmann's name on the list of people 
above...</pre>

<!-- Comment: Tue May 29 21;32;23 comment by...SK -->
<pre>Somehow it seemed better that way...</pre>



<!-- Date: Wed May 30 22:08:41 2001 -->
<pre>We did some work with the Clist, SVT and L1 interface boards. 
 
Originally, we had SVT in slot 11, an SVT board with no cable 
in slot 12 and Clist in slot 14. Clist would send bossreq but 
not get bossgrin. Somehow the board in slot 12 (which wasn't  
being used to send data) was holding bossreq high so that 
Clist was never seeing the upgoing strobe on bossgrin.  
This Track/SVT board is #6.  Note that PECL only drives signals 
high and needs a pulldown resistor on the output for a signal 
to go low. I tried to check this on the board but didn't find 
anything obviously wrong. Will need to check voltage levels 
on each chip with a scope.  
 
Jane created a new prom that has the SVT board wait about 1us 
after start_load* is asserted in order to start sending data. 
This is to avoid having an arbitration problem with the L1 
interface board which always sends its data with 100-200ns after 
start_load* is asserted. SVT was in slot 7 and Clist was in 
slot 14. In looking on the scope, we saw that the SVT board  
would occasionally send only 1 word or sometimes 6 words. 
It should always be sending 2 MBUS words (for 3 tracks) with 
the current hw=l2test svt configuration.  
  We added the L1 interface board to slot 9 and issued an HRR. 
All 3 boards were reliably sending data. L1 would send 1 word, 
immediately followed by Clist sending 1 or 2 words, and then 
SVT always sending 2 words about 1 us later. This was run 
#116321</pre>
<!-- Author: Stephen Miller -->


<!-- Date: Thu May 31 00:05:11 2001 -->
<pre>New code tool of the day. 
 
Made ~jsm/alpha/clistconfig_linux  which can be used to  
configure Clist and Clique if not all 6 L2cal crates are used 
in a run. The program prompts for a 6 bit bitmask. Set a bit 
to zero if that crate is not used.  
E.g. enter 1 to only use the b0l2cl00 crate.  Enter 2f to  
exclude crate b0l2cl04.</pre>
<!-- Author: Stephen and Monica -->


<!-- Date: Thu May 31 18:37:47 2001 -->
<pre>Running with new SVT interface board firmware to delay 
sending data, we take a run with L1 interface, Clist, SVT boards. 
We notice that occasionally the SVT board will continue to send 
large amounts of data, for a long time, even after start_load 
is deasserted. This causes the alpha program to crash. 
This may be a problem with the SVT crates sending too much data 
or in the wrong format. 
   
For now, the SVT board is plugged in, but the cable going to the 
front has been unplugged to allow for stable operation for the 
shift crew.</pre>
<!-- Author: Stephen -->


<!-- Date: Thu May 31 18:39:42 2001 -->
<pre>I am taking the Alpha in the bottom crate back to Michigan. 
We will put the arbitration fix on that board, and bring it 
back next week. 
 
           </pre>
<!-- Author: Stephen -->


<!-- Date: Fri Jun 1 13:02:07 2001 -->
We swapped in SVTD readout board 6 into the b0svt06 crate so that we can have ECO #3 made to the board (#4) that was previously in 
there.   We checked that the swapped in board works in run 116486. 
  
<!-- Author: Matt, David -->
<!-- Comment: Fri Jun  1 14;39;08 comment by...David, Matt -->
Run ended after 65k events with:  "Alpha in slot 3 is no longer processing at spot 61" errors.  Took rc through coldstart and are running again: 116502 (1c716).  116502 ended after 12k events.


<!-- Date: Sat Jun 2 21:28:25 2001 -->
<pre> 
Currently, the SVT cable cannot be plugged in, in order to have 
stable operation of L2. (See a few entries above which  
describes how the SVT board won't stop sending data, and that 
will cause the alpha to crash.) Currently, just L1 and Clist 
boards should be sending data to L2. 
 
If you do tests with other interface boards, either remove 
them or unplug their input cables before handing the crate 
back to the shift crew.</pre>
<!-- Author: Stephen -->
<!-- Comment: Mon Jun  4 10;55;32 comment by...Matt, David -->
We would prefer if you kept it plugged in during the day runs and called us if there is this problem.   If this is the problem we think it is, then the SVT people have wanted to see why this is happening.  So call us and we'll determine if we should page the SVT people.

<!-- Comment: Mon Jun  4 11;18;57 comment by...Stephen -->
<pre>Since this error condition is reliably reproduced, I would 
suggest setting a time with the SVT people and shift crew to 
reproduce the error. Otherwise the it will disrupt normal
running by the shift crew. </pre>


<!-- Date: Mon Jun 4 19:59:36 2001 -->
Today we investigated the sync problem we have been seeing 
in the SVT data in TL2D vs. SVTD.   We had seen that the 
data would start out of phase (as measured by the bunch ID) but stay locked with that constant offset stably. 
 
<p> 
We triggered the 'scope off of the strobes being sent 
by the SVT to the interface board.   We found that lots of data 
were sent by the SVT during cold start and also that event fragments could come in on the first event. 
 
<p> 
We called Stefano in to investigate the SVT behavior. He found that because the SVT test-data is sent asynchoronously, the first event can begin with a fragment of a test event.  (BTW, it turns out that the behavior on cold start is irrelevant since all runs start with an HRR after "activate" which Jane says resets the FIFOs on the SVT interface board upon CDF-HALT.)  The event fragment causes the end-event bit to come at the wrong time in the protocol and it would take several events (up to 6) before synchronizing.  Depending on the phase of the SVT data w.r.t. the svt interface board's registers, a very large single "event" could be formed from the point of view of the L2 crate.   This would appear as if the SVT boards were spewing lots of data-- which is what has been blamed for the L2 processor hangs. 
 
<p> 
Stefano wrote new programs  that will start and stop test-data that we can issue from a unix prompt after a HRR so that the SVT will always send the test data with the proper phase. 
 
<p> 
We have this working now, but unfortunately we have magic bus problems so we are going to look tomorrow.
<!-- Author: Matt, David -->