<!-- Title: 09:18:56  Wed Sep  1 1999 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Christoph Paus -->
<!-- Op3: Ilya Kravchenko -->
<!-- Op4: Sasha Rakitine -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Run II hardware event builder


<!-- Date: Wed Sep 1 09:22:24 1999 -->
We've had an overnight run with 10 SCPUs using the new ATM driver on the converter node. We still get packets with bad CRCs, which are now thrown away properly. This is the only way to test this feature of the driver since we can't make bad packets to order (not without risking permanent damage to an optical fiber).<p>
<!-- Author: Steve -->


<!-- Date: Thu Sep 2 18:00:42 1999 -->
I've updated scramnet_reset.* in b0l3pcom1:/home/vxworks/cdfatm/all. The scramnet_reset routine now recognizes whether a new or an old SCRAMNet is present, and takes the approriate action. The new SCRAMnet cards don't have the VME address of the RAM set from switches anymore; now that's done by software. The node number is also settable by software. Also, a little more information is printed. All SCRAMNet modules are still set to be compatible with the old SCRAMNet network protocol. When we have new SCRAMNets everywhere we can go to PLATINUM. 
<p> 
Here are examples of reset printout for new and old SCRAMNets: 
<p> 
<pre> 
-> scramnet_reset              
SCRAMNet CSR base address = 0xefff6000 
New SCRAMNet with 512K detected. 
SCRAMNet node number is 26 (set by software) 
Contents of CSR 1 = 0x8000 
value = 0 = 0x0 
<p> 
-> scramnet_reset              
SCRAMNet CSR base address = 0xefff6000 
Old SCRAMNet detected. 
SCRAMNet node number is 11 (set by switches) 
Contents of CSR 1 = 0x3000 
value = 0 = 0x0 
 </pre> 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Sep 3 16:58:13 1999 -->
Ilya and I have installed SCPUs b0eb25,26, and 27 along with new SCRAMNet modules in the top SVX crates in relay racks 1RR21G, 1RR21H, and 1RR21I. The SCRAMNet modules checked out when hooked up in their own mini-ring and are now connected to cables going up to third floor. One of those cables is bad, though, so we may have to commandeer two of the 20 new ATM fibers when they are laid. JJ has an adapter that will let us do this. He'll also try to get DataComm in to attach new connectors to the bad cable, since most problems are due to bad connectors. Due to the bad cable, the three SCPUs on the first floor are not yet inserted into the existing SCRAMNet ring.<p> 
 
The ethernet connections need work, too. All the CPUs can boot, but we get a lot of messages about dribbling bits and CRC errors. Presently the first-floor SCPUs are connected to a thin-wire ethernet that goes up to our private net on the third floor. JJ suggests that we recable this using CAT5 cable going to the nearby patch panel and connect the panel to one of our ethernet switches directly. 
<p> 
The ATM cards all respond to the "zot" command and recognize the optical carrier when connected to themselves.<p>
<!-- Author: Steve -->


<!-- Date: Wed Sep 8 14:14:43 1999 -->
I've moved all the ATM fibers over to the new fabric and network 
modules. The relative placement of the fibers is the same, i.e., fibers 
connected to ports 1A1, 1A2, etc. are now connected to 2A1, 2A2, etc. 
Since I don't have an IP number for the new fabric yet, I used the 
serial port to type in only the minimal routing table needed for SCPUs 
to send to converter nodes 2, 3, 4, or 5. I checked it manually by 
sending single-cell messages from each SCPU to each converter, then 
examining the switch's cell statistics (how many cells went from X to 
Y). It works. Bear in mind that for now only SCPUs sending to converters 
will work, so any data sent between SCPUs, between converters, or from 
converters to SCPUs will just be thrown away inside the switch. Once the 
IP address is assigned I have an expect script that will install the 
full routing where anybody can talk to anybody. 
<p> 
VPI numbers are unchanged. 
  
<!-- Author: Steve -->


<!-- Date: Wed Sep 8 16:25:24 1999 -->
We've had time to do a small test on the new fabric. The results are 
very entertaining. 
<p> 
This time we had just one sender, repeatedly sending a packet 261 cells 
long to (VPI, VCI) = (11, 3), (11, 2), (11, 1), (11, 0), (11, 3), etc. 
<p> 
We had a trace placed inside the ATM driver used on the receiving node, 
so we could see directly which VPI and VCI an AAL5 packet was received 
on. The reception pattern was (1, 2), (1, 1), (1, 0), (1, 2), etc. The 
change of VPI from 11 to 1 is specified in our routing table, no problem 
there. As you can see, we never got packets in which VCI = 3. I checked 
the cell counts for the input and output ports, and sure enough exactly 
3/4 of the cells received were transmitted (the rejection counts were 
zero, though). The cell count for the virtual path between the two ports 
was also about 3/4 of what it should have been (it doesn't show as many 
significant figures so I can't say exactly). 
<p> 
We tried using more VCIs in cyclical order from 5 down to 0, and all 
packets come through except the ones with VCI 3. This is pretty damn 
strange.  
<p> 
Ilya is running the wedge test now, and the event builder is happy as long as we use no more than three Builders on the converter, because that means that only VCIs 0, 1, and 2 are used. 
<p> 
We never use VPI 0, but maybe are there other reserved VPI/VCI combinations we 
need to avoid on Fore equipment? I've asked Fore for help. This looks like some sort of 
configuration mistake rather than a true hardware problem. 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Sep 10 13:46:24 1999 -->
By international convention, VCIs 0-31 are reserved for various tasks such as signalling and flow control. It doesn't matter what the VPI is. Fore doesn't guarantee that the switch will properly handle permanent virtual circuits using these VCIs, no matter how many features we turn off. As it happens, VCI 3 is reserved for "Segment OAM F4 Flow" cells, and it appears that the new fabric is doing the "right" thing by throwing them away since it isn't doing any flow control (there are other flow control cells allowing the sending node to negotiate with the receiving node). 
<p> 
We therefore need to use VCIs greater than 31. Right now the Fore LE155 ATM card is set up to use 8 bits of VPI and four bits of VCI to make the index into its reception virtual circuit table, held in SRAM on the card. 12 bits means 4096 entries in that table, which is the maximum the card allows. This means that we can't just use more VCI bits. The next step down uses two VPI bits and 10 VCI bits, and we can also use 0 VPI and 12 VCI bits. Either choice means abandoning our current scheme of using the VPI to identify sender and receiver. 
<p> 
The LE card uses only 12 of the 24 bits available to find a connection table entry, but reports the full VPI and VCI back to the driver. The driver uses the full VPI/VCI to match incoming packets with read requests, and only allows you to request VCIs in the range determined by the hardware, 0-15. I don't see why the driver can't be modified to allow only requests for VCIs 32-47 instead. Then a transmission would work as follows: 
<ol> 
<li>The converter software opens VPI 1, VCI 32 for reading. 
<li>The driver activates the hardware table entry for VPI 1, VCI 32 & 0xf = 0 for incoming data, but uses 1:32 in the software table. 
<li>The SCPU transmits on 1:32. 
<li>The cells for 1:32 go through the switch and arrive at the LE card. 
<li>The card looks up the table entry for (1 & 0xff):(32 & 0xf) = 1:0, finds it activated. 
<li> The buffer fills and the card reports back a buffer for 1:32. 
<li> The driver matches the buffer to the read request. 
</ol> 
<p> 
In the meantime, we can avoid VCI 3 by using only three builders on the converter. There's no guarantee, though, that some data won't occasionally be mishandled. 
<p>
<!-- Author: Steve -->
<!-- Comment: Tue Sep 14 10;47;40 comment by...Steve -->
It turns out that the driver ignores cells with too many significant bits in the VPI or VCI; it doesn't mask out bits it doesn't need to use. In hindsight this makes sense, who would want two data streams to be silently merged? I found this out when testing a modified driver which required all VCIs to be greater than 31 but masked out all but the four lower bits when doing table lookups. The hardware refused to respond to VCIs outside the range 0-15.
<!-- Date: Fri Sep 10 13:53:21 1999 -->
Ilya ran an event builder test using the new fabric last night. Here's a summary: 
<pre> 
I have tried our old test with 10SCPU->1CV->12PROC with  
current setup which is the new ATM fabric but the VCI numbers 
are not corrected yet. The test is done with 3 builders. 
<p> 
  There are still bad events. The first bad event appeared 
after ~580000 events (approx 2 hours). 
<p> 
  I have stopped the test and also saved the trace in case 
anyone is interested, trace stopped 2 events after the 1st bad one: 
b0l3pcom1:/home/ikrav/ 
          trace-090999.dat 
          ps-090999.dat 
<p> 
Ilya 
----------------------------------------------------------------- 
From Steve: 
<p> 
The converter node was b0l3c02. 
All fibers were connected to the new switching fabric. 
The trace isn't much use, it just shows one good event. 
We had a power failure last night, so all switch statistics were lost. 
The message log shows that: 
  The test started at 17:54. 
  One packet with a CRC error arrived at 20:19. 
  Ten packets with bad CRCs arrived at 21:35. We can't tell if they're all in the same event. 
  The test ended at 22:16 with no further CRC errors. 
</pre> 
<p> 
I've moved the trace files and an extract of /var/log/messages to b0l3pcom1:~tether/atm-tests/09091999. 
<p>
<!-- Author: Steve -->
<!-- Comment: Fri Sep 10 16;40;34 comment by...Steve -->
One more thing: the nicstar driver was version 1_4.
<!-- Date: Fri Sep 10 16:39:01 1999 -->
We ran 17 100-ft ATM cables between first and third floors today (with Dervan's help, of course :-)). They now come out at the switch where it is now, but we have more than 20 ft of slack, so no problem reaching the future ATM rack. 
<p> 
Both switch fabrics are on the private network now. The old one is b0atm1 (192.168.5.2) and the new one is b0atm2 (192.168.5.3). The hosts table on b0l3pcom1 now has entries for them both. 
<p> 
When you telnet to a fabric you have to enter a user name AND a password. See me if you need to know them.<p> 
 
On Monday, some people from DataComm will come and try to repair the bad 1st-3rd floor SCRAMNet cable by putting on new connectors.<p>
<!-- Author: Steve -->



<!-- Date: Fri Sep 10 16:51:03 1999 -->
Ilya will rerun last night's test. I've slightly modified the v1_4 nicstar driver on b0l3c02 so that it will print the VPI and VCI of any bad packet in /var/log/messages.<p>
<!-- Author: Steve -->



<!-- Date: Tue Sep 14 11:13:28 1999 -->
Since we can't avoid the reserved range 0-31 using only four bits of VCI, we'll have to set the hardware to use more VCI bits but fewer VPI bits. The total number of bits used must be less than or equal to 12 since the ForeRunner LE ATM card supports at most 4096 simultaneous connections. <p> 
 
The next step down from eight VPI bits is two bits. That's not enough to let us continue to use the VPI as a node identifier. We might as well give up using the VPI and set the hardware for zero VPI bits and 12 VCI bits. I'll have to study the ASX-1000 manual to see what kind of routing would have to be programmed to 
<ul> 
<li>Let the sending node identify the receiving node, 
<li>Let the receiving node identify the sending node, 
<li>Select the right Builder on the receiving node, 
<li>Avoid VCIs 0-31. 
</ul> 
<p>
<!-- Author: Steve -->
<!-- Comment: Tue Sep 14 11;17;32 comment by...Steve -->
 <a href=/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=3&anchor=13:46:24 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a> Previous entry on reserved VCI problem


<!-- Date: Wed Sep 15 17:54:28 1999 -->
Two more tests are summarized in b0l3pcom1:~tether/atm-tests. In directory 09121999 we see a test run with the standard timout settings (10/3 ms between tests, 15 tests, total 50 ms). About 130 events out of about 5 million had tardy fragments, but according to the hardware counters all fragments made it to the converter node except 20 which had CRC errors. The test ran roughly 17.5 hours, making the total event rate.<p> 
 
In directory 09131999 is the summary of a test made with the timeout test interval set to 300/3 ms and the number of tests set to 100, for a total timeout of 10 sec. This time we got no complaints about tardy fragments, except in the very first event where we lost one fragment due to a CRC error. The price we paid for this massive expansion of the timeout was a low event rate: about 700,000 events in 14 hours, or about 14 Hz. Good enough for cosmics but not true DAQ. Of course, we can reducing the time between tests while increasing the number of tests in proportion. I'm not sure that the Linux scheduler will reliably run the testing thread more than once every 10 ms, so that should be our lower limit on the testing interval. Also, a 12 KB fragment takes 12 ms to send at 1 MB/sec, and we don't know how much has been transmitted when the SCPU tells the SM that fragment has been sent. That's because the write call to the ATM driver returns when the fragment has been DMA'd into the ATM card's memory, not when the actual transmission is complete.<p>
<!-- Author: Steve -->


<!-- Date: Wed Sep 15 18:31:51 1999 -->
ATM switch changes made today: 
<ul> 
<li>The old SCP running ForeThought software version 3.4 has been replaced by a new one running version 6.0, so both switch boards are running the latest software. We can upgrade the old SCP to version 4.1 or perhaps 5.3 later and keep it as a spare. 
<li>All ten SCPUs currently in use have been attached to the first board, b0atm1, while the five current converters have been moved to b0atm2.  
<li>The routing tables in both boards have been updated to allow SCPUs to send to converters. As before, the VPI is used to identify sender and receiver, and VCIs 0-15 are still in use. There will be a test event builder run tonight. 
</ul> 
<p> 
I've changed the host tables for converter 2 and for VxWorks to reflect new VPI assignments. If the overnight test works I'll update the other converters. 
<p>In the table below, the nodes whose names are in red haven't been connected to the switch yet. 
<center> 
<table BORDER COLS=3 WIDTH="50%" > 
<tr> 
<td></td> 
 
<td> 
<center>New Port Assignments</center> 
</td> 
 
<td>&nbsp;</td> 
</tr> 
 
<tr> 
<td> 
<center>Port<hr></center> 
</td> 
 
<td> 
<center>b0atm1<hr></center> 
</td> 
 
<td> 
<center>b0atm2<hr></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>a1</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0eb11</font></center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c01</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>a2</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0eb12</font></center> 
</td> 
 
<td> 
<center>b0l3c02</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>a3</center> 
</td> 
 
<td> 
<center>b0eb13</center> 
</td> 
 
<td> 
<center>b0l3c03</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>a4</center> 
</td> 
 
<td> 
<center>b0eb14</center> 
</td> 
 
<td> 
<center>b0l3c04</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>b1</center> 
</td> 
 
<td> 
<center>b0eb15</center> 
</td> 
 
<td> 
<center>b0l3c05</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>b2</center> 
</td> 
 
<td> 
<center>b0eb16</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c06</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>b3</center> 
</td> 
 
<td> 
<center>b0eb17</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c07</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>b4</center> 
</td> 
 
<td> 
<center>b0eb18</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c08</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>c1</center> 
</td> 
 
<td> 
<center>b0eb19</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c09</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>c2</center> 
</td> 
 
<td> 
<center>b0eb20</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c10</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>c3</center> 
</td> 
 
<td> 
<center>b0eb21</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c11</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>c4</center> 
</td> 
 
<td> 
<center>b0eb22</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c12</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>d1</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0eb23</font></center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c13</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>d2</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0eb24</font></center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c14</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>d3</center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0eb25</font></center> 
</td> 
 
<td> 
<center><font color="#CC0000">b0l3c15</font></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>d4</center> 
</td> 
 
<td> 
<center>b0eb02</center> 
</td> 
 
<td>&nbsp;</td> 
</tr> 
</table> 
</center> 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Sep 17 13:35:38 1999 -->
We had a big cabling party yesterday afternoon, so now converters 6-11 are on the private network. Of these only 6 has two network connections, the others have only one for now. There were a number of minor mistakes I made while configuring these nodes which have now been fixed: 
<ol> 
<li>The /etc/hosts table didn't have the unlettered aliases for the new nodes, e.g., b0l3c06 for b0l3c06a.fnal.gov, etc. This caused some control software to break. 
<li>The host name in /etc/sysconfig/network didn't exactly match the first name for the host given in /etc/hosts. This caused the hostname -s command to fail with host lookup failure. Apparently hostname doesn't deal too well with aliases. 
<li>The unconnected ethernet ports had ONBOOT=yes in their /etc/sysconfig/network-scripts/ifcfg-eth? files. This caused trouble the first time the network was used after a reboot. The switch and the node negotiated full-duplex mode on the connected ports but then the node hung for a very long time. During this time the network was unreachable and the mounting of remote file systems failed. You have to be logged in to the console to see this happen. This has been corrected on the old converter nodes as well, getting rid of all those "No link beat" messages. 
</ol><p> 
The new converter nodes are not directly connected to the old ethernet switch. They are connected to a new one in the new network rack, which is connected by a crossover cable to the old switch. We originally made two connections between the switches this way, but this caused all nodes on the private network to become unreachable. We don't know why.<p> 
 
An attempt to fix the bad 1st-3rd floor SCRAMNet cable by replacing a bad connector didn't work, so we have commandeered a duplex ATM cable. We used an SC-ST adapter plus a short length of ST-ST cable to attach it to a SCRAMNet node on the first floor. We now have four working SCRAMNet nodes on the first floor that are part of the event builder ring, replacing for now the 2nd floor SCRAMNet that was being experimented with by Steve Vejcik.<p>
<!-- Author: Steve -->


<!-- Date: Sun Sep 19 10:37:55 1999 -->
More mistakes: 
<ul> 
<li>I didn't install scramnet software properly (or at all?) on the new converter nodes. Jeff and Ilya fixed this. 
<li>I left an experimental version of the nicstar driver running on b0l3c05. This is now fixed. There's a chance that the bad version got onto c06-11, so I reinstalled the correct version on them, which will run the next time the nodes are rebooted. 
<li>I forgot to install the 2.0.25 kernel sources on c06-c11. I'll fix this on Monday. 
</ul><p>
<!-- Author: Steve -->


<!-- Date: Sun Sep 19 11:25:41 1999 -->
I've renamed .cshrc for root on b0l3c02 to ikrav.cshrc copied root.cshrc to .cshrc so that remote commands will work. The ikrav version was intended to to let Ilya try running the converter node code as a privileged user. It doesn't work properly now anyway; it always drops you into RedHat setup.<p>
<!-- Author: Steve -->


<!-- Date: Mon Sep 20 13:10:55 1999 -->
I've looked at the /var/log/messages file for last weekend's tests. Besides a few complaints about CRC errors, there are messages about 
packets being received on inactive virtual connections, that is, 
connections which are not open for reading. The CRC errors happen very 
near the beginnings of runs, within 3 events. The inactive VC messages 
occur near the ends of runs, probably because the converter node 
software is shut down before the SCPUs have quite finished sending. 
 
<pre> 
Sep 19 14:04:34 b0l3c02a kernel: nicstar0: SDU received on inactive VC 1.0. 
 
Sep 19 14:04:34 b0l3c02a kernel: nicstar0: SDU received on inactive VC 1.0. 
 
Sep 19 14:09:41 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 2. 
 
Sep 19 14:09:41 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 2, VCI 
= 0. 
 
Sep 19 14:10:19 b0l3c02a kernel: nicstar0: SDU received on inactive VC 3.2. 
 
Sep 19 14:10:19 b0l3c02a kernel: nicstar0: SDU received on inactive VC 3.2. 
 
Sep 19 14:12:13 b0l3c02a kernel: nicstar0: SDU received on inactive VC 3.1. 
 
Sep 19 14:14:29 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 2, VCI 
= 2. 
 
Sep 19 14:14:29 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 1. 
 
Sep 19 14:14:29 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 3, VCI 
= 0. 
 
Sep 19 14:15:38 b0l3c02a kernel: nicstar0: SDU received on inactive VC 3.2. 
 
Sep 19 14:18:52 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 1. 
 
Sep 19 14:18:52 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 3, VCI 
= 1. 
 
Sep 19 14:18:52 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 2, VCI 
= 1. 
 
Sep 19 14:38:18 b0l3c02a kernel: nicstar0: SDU received on inactive VC 4.2. 
 
Sep 19 14:43:39 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 1. 
 
Sep 19 14:43:39 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 5, VCI 
= 1. 
 
Sep 19 14:43:39 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 4, VCI 
= 1. 
 
Sep 19 14:43:39 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 3, VCI 
= 1. 
 
Sep 19 14:43:39 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 2, VCI 
= 1. 
 
Sep 19 19:37:22 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 14, VCI 
= 2. 
 
Sep 19 19:52:27 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 2, VCI 
= 2. 
 
Sep 19 20:16:13 b0l3c02a kernel: nicstar0: SDU received on inactive VC 2.1. 
 
Sep 19 20:18:32 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 0. 
 
Sep 19 20:24:41 b0l3c02a kernel: nicstar0: SDU received on inactive VC 1.1. 
 
 
-------- I couldn't find log files for the messages below ---------- 
 
Sep 19 20:27:36 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 0. 
 
Sep 19 20:38:33 b0l3c02a kernel: nicstar0: SDU received on inactive VC 1.1. 
 
Sep 19 20:42:18 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI 
= 0. 
 
Sep 19 20:42:18 b0l3c02a kernel: nicstar0: SDU CRC is bad. VPI = 2, VCI 
= 0. 
</pre><p>
<!-- Author: Steve -->


<!-- Date: Mon Sep 20 16:17:05 1999 -->
<p>Today we clean up converter nodes.<p> 
c07 and c10 have some sort of hardware problem involving ATM. The card is recognized properly on boot-up, but both of these PCs lock up when the first data arrives over the link. This looks like bad ATM cards.<p> 
 
The cause of c03 and c04 not responding to ATM over the weekend is a shortcoming in the driver. It doesn't handle the so-called PHY (physical, i.e. optical transceiver) interface of the board very well. In particular, it doesn't do the right thing when carrier is lost and then restored, so the card may hang whenever the ATM fiber is disconnected, however briefly. The only sure way to cure the hang is to shutdown -h then turn off the power for about 30 seconds. I see from  messages on the linux-atm mailing list that someone has written better PHY code for this driver. One more thing to investigate when we get around to updating the driver. 
 
<p>c08, c09, and c11 just needed to be rebooted to run the correct version of the driver.<p> 
 
Jeff has put c01 (the new PC) on the ethernet and SCRAMNet.<p> 
 
For now, Ilya will do his tests with up to nine converters; c01-06, c08, c09, and c11. After that we'll mess around trying to pinpoint the problem with c07 and c10.<p>
<!-- Author: Steve -->


<!-- Date: Tue Sep 21 09:47:25 1999 -->
Yesterday, when Ilya first tried to run using two converters, he found that the first connected to VCIs 0-2, but that the second used VCIs 3-5. Since the switch will not pass cells with VCI 3, the second converter timed out on 1/3 of its events.<p> 
 
This pattern of VCI use harks back to the old days when we used BSD sockets for event transfer. The Builders acted as clients and the SCPUs as servers. To prevent Builders contending for a single service port, each SCPU provided each possible Builder with its own port to connect to. Builders are assigned global ID numbers by the Scanner Manager, so the port number for each Builder was some base port number + the global ID number. When we switched from using IP number and port to using VPI and VCI, the pattern of VCI use was modeled after the pattern of port use.<p> 
 
We can't do this in a large system now because the PC ATM card is set to accept only VCIs in the range 0-15. Also, because the switch may mishandle some VCIs in the range 0-31, we must for now restrict ourselves to VCIs 0-2. File socketry.c in the Ilya's copy of the SCPU code and boss.c in the Jeff's copy of the L3 receiver code have been modified so that the VCI argument given to new_seqio_atm is reduced modulo 3. <p> 
 
Of course, this only works when there are exactly three Builders per converter, which is what we'll use for this week's tests. Clearly, VCIs need only be assigned locally per converter node rather than globally. That would allow 16 Builders per converter, more than enough.<p>
<!-- Author: Steve -->


<!-- Date: Wed Sep 22 12:11:56 1999 -->
Yesterday afternoon Ilya and I discovered the once in a while a SCRAMNet message would be garbled. The more converter nodes we tried to use, the more frequent the garbling would become. With four converter nodes we would see the problem about once an hour or so, but with five we would see half a dozen occurrences in half an hour. With more than five things were hopeless.<p> 
  
The lower eight bits of the second 32-bit word in a BUILDER_READY message would be changed from 0x31 to 0x10. This part of the message is the command type, so the messaage was changed from BUILDER_READY to a value not assigned to any command. Because of this the Scanner Manager would not know that the Builder involved was free to receive a new event, effectively taking the Builder out of circulation for the rest of the run. So as the run continued, the number of Builders receiving events decreased steadily. All converter nodes were equally affected.<p> 
 
Once, when b0l3c05 had had all three of its Builders removed, we stopped the run, located the last bad command in the SM SCRAMNet, and dumped the same region on all other nodes. The command was written properly at c05 and b0eb16 but became corrupted at c06, remaining that way for all nodes downstream of it. After we removed c06 from the SCRAMNet ring we were able to run 1 SPU to five converter nodes for several hours with no complaints. <p> 
 
During our investigations we replace the old-model SCRAMNet module with a new one. It didn't solve the problem, but there's no reason now to change it back.<p> 
 
We also found that the old b0l3c01 had not been removed from the SCRAMNet ring and was still turned on. Since converter node SCRAMNet module IDs are assigned based on the PC host name, this SCRAMNet module had the same ID as the one in the new b0l3c01. We have turned off the old converter node. We may as well remove its SCRAMNet card. Perhaps this machine would be useful as a test stand.<p> 
  
<!-- Author: Steve -->


<!-- Date: Wed Sep 22 21:28:03 1999 -->
This evening Ilya and I replaced the old-model SCRAMNet cards for b0eb21 and b0eb16 with new cards. We also tried to make SCRAMNet work for b0l3c06, but it was futile. We tried three different PCI SCRAMNet cards with that PC (one of them in two different slots). One refused to work at all and two others (including the original card we had in there) garbled messages when we had a lot of SCRAMNet traffic. Either the PCI bus is bad on b0l3c06 or there is some sore of conflict with one of the other PCI cards in the machine. We've taken c06 out of the SCRAMNet ring.<p> 
 
We've run 1 SCPU -> 5 converters and 14 SCPUs -> 5 converters each for half an hour without problems.<p> 
 
Starting a run is still a little chancy, though. Sometimes a converter node will get a few events and then stop, or we will see lots of messages about CRC errors and running out of large ATM buffers in /var/log/messages. Rebooting the affected converter node got rid of both types of problem.<p>
<!-- Author: Steve -->