<!-- Title: 09:37:12  Wed May  5 1999 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Christoph Paus -->
<!-- Op3: Ilya Kravchenko -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Log book for the Run II event builder.</pre>


<!-- Date: Wed May 5 09:38:03 1999 -->
<pre>Yesterday afternoon I took b0l3c02, 03, and 04 out of the rack in 
order to test and swap their ATM cards. I found that the card for  
03 is dead; it doesn't work in any machine. Data going into the  
card never makes it into main memory, although the receive LED  
does light up. Data sent for transmission is accepted by the card  
but never goes out onto the fiber, although the optical carrier is  
present. The cards in 01, 02, 04, and 05 check out perfectly using  
both a direct optical loop-back and a loop going through the ATM  
switch. I don't know why b0l3c04 didn't work in Ilya's tests. A  
card transplanted from another converter node still worked  
properly in b0l3c03. For the night's tests I put 02 and 04 back in  
the rack.</pre><p> 
 
<pre>This morning I will take the ATM card from b0l3c01 and put it into  
03, which will make all operating converter nodes the same make  
and model of machine. I'll check the Fore manual on CD-ROM to see  
if the problem we see in the bad card is one that is already  
known. If not, I'll see about warranty, etc.</pre><p> 
 
<pre>To assist stand-alone testing, I've copied the ATM software in  
~djholm/atm to /root/atm on all converter nodes.</pre><p> 
 
<pre>I noticed one small problem while testing the ATM interfaces. If  
you run the br/bw pair of test programs, which sends ATM data as  
quickly as possible without attempting to synchronize, the ATM  
driver will sometimes complain that it has run out of reception  
buffers (either large or small). This complaint appears in /var/ 
log/messages. After it occurs, the count of packets transmitted  
may start to get ahead of the count of packets received (in /proc/ 
atm/devices), although the count of packets dropped remains zero.  
Occasionally, the sending program bw will hang before it sends all  
the packets it was supposed to.</pre><p> 
 
<pre>I ran all my tests with transmission and reception on the same  
machine. 
</pre>
<!-- Author: Steve Tether -->


<!-- Date: Wed May 5 11:11:15 1999 -->
The ATM card that was in b0l3c01 is now in 03 and seems to work fine. There was one small surprise: this card has a circuit board very different from (and smaller than) the others. It only has one LED instead of two. Must be an early version. 
  
<!-- Author: Steve Tether -->


<!-- Date: Fri May 7 09:33:08 1999 -->
The ForeRunner LE card that failed is still under warranty, so we can send it in and get it repeaired. This kind of thing requires filling out a requisition (even though there is no charge) and a material move form (so that Shipping and Receiving knows it went out and who to give it to when it comes back). Jim Patrick is doing this paperwork. The following shipping instructions came from Fore, which is handling the repeair directly. 
<p> 
Dear Stephen, 
<p> 
 
Thank you for being a valued FORE customer, we always appreciate your 
business. 
 
<p> 
Please note that your RMA number is 
<bold><underline>80054222</underline></bold> for the return of your 
<bold><underline>4402552</underline></bold>.  Please place this number on 
the outside of the box for our Receiving department to accept the 
shipment and proceed with the repair.  Your defective equipment will be 
repaired and returned to you per your agreement, which is 
<bold><bigger>(10)</bigger></bold> day return to factory. 
<p> 
 
The address for the product return is: 
<br> 
FORE Systems, Inc. 
<br> 
Attn: <bold><underline>80054222 
<br> 
</underline></bold>174 Thorn Hill Road 
<br> 
Warrendale, PA  15086 
<br> 
(724) 742-6655 
<p> 
 
For any further technical questions, please contact our Technical 
Assistance Center at 800-671-FORE (3673) or +1-724-742-6999 and refer to 
your case # <bold><underline>CC136721</underline></bold>. 
<p> 
 
Best Regards, 
 
<p> 
 
<italic><bigger> 
 
<bigger>Debrah Coryea            
<br> 
</bigger></bigger></italic><bold>RMA 
DEPARTMENT              
<br> 
<bigger>FORE 
Systems Inc. 
<br> 
</bigger><smaller>1000 FORE Drive 
<br> 
Warrendale, PA 15086 
<br> 
</smaller><bigger><bigger> 
 
</bigger></bigger></bold>Direct:   724.742.6651 
>br> 
Fax:            724.742.7900 
<br> 
Email:          dcoryea@Fore.com                 
<br> 
RMA Group:      724.742.6655 
<br> 
URL:            <underline>http://www.fore.com/ 
<br> 
</underline><italic>FORE 
NETWORKS OF STEEL 
<br> 
</italic><bigger><bigger><bigger> 
 
</bigger></bigger><italic>   
 
</italic></bigger>  
<!-- Author: Steve Tether -->


<!-- Date: Sat May 15 11:26:24 1999 -->
I've installed the two new 2603s in the crate that used to have b0eb03 and 04. The node names are b0eb23 and 24 and have been entered in the .rhosts file for vxworks on b0l3pcom1. Only 23 is hooked up to the network at the moment (though there is a free tee nearby). I'll be using b0eb23 for ILU tests. 
<p> 
These are the last of the new 2603s. We still have five out for upgrade to Universe II, and Dave Berg has one 2603 and one 2604 that need to be upgraded. We'll get them from Dave when the 2600 BSP is stable again. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Sat May 15 12:04:00 1999 -->
Since Friday morning I've been running tests of a simple ILU server running under VxWorks. The server is generated from the following IDL spec: 
<p> 
<pre> 
module Test { 
  struct Stats { 
    long id; 
    sequence&lt;long&gt; grunch; 
  }; 
 
  interface Joe { 
    void getStatus(out Stats info); 
  }; 
}; 
</pre><p> 
The implementation returns 1234 in info.id and the squares of the integers 0-10 in info.grunch. The server is the primitive kind which prints an IOR on the console, which you must manually copy to the client. Here's a summary of the results 
<ul> 
<li>The ILU kernel + C interface library take up 496K. 
<li>Starting a server task with <code>sp()</code> takes another 41K This doesn't include the size of the server code itself, but it does include the stack and task overhead. 
<li> The server can perform 960 calls/second. The client was written in C and ran on b0l3pcom1, a 400 MHz Pentium II. The server ran on a 200 MHz MVME2603 (PPC 603e). 
<li>Neither client nor server has a memory leak when the user-written code is correct. 
<li>The VxWorks server can be started, stopped, and restarted by using <code>ILU_C_StoppableRun()</code> and <code>ILU_C_StopRun()</code> instead of just <code>ILU_C_Run()</code>. You won't find this in the ILU manual. 
</ul> 
I started a long test at 12:52 yesterday afternoon which is still running (more than 68 million calls). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Jun 14 15:31:17 1999 -->
<pre>Installed b0eb25 (refurbished MVME2603).  Had to re-install 
boot image.  Instructions on doing so (courtesy Dave Berg): 
 
1.  J10 set to ROM bank B. 
2.  Connect to serial port (talk through minicom). 
3.  In PPCbug (which we'll keep in bank B as we program bank A), 
 
  PPC1-Bug>niot 
  ...set:  Client IP Address = 192.168.1.25 
           Server IP Address = 192.168.3.1 
           Subnet IP Address Mask = 255.255.0.0 
           Gateway IP Address = 0.0.0.0 (leave zeroed) 
           Boot File Name = (leave blank) 
           Argument File Name = (leave blank) 
  PPC1-Bug>bf 200000 24ffff 0		(force known values) 
  PPC1-Bug>niop 
  ...set:  File Name = /tftpboot/mv2603_5_3b.bin 
           (copied from /home/vxworks/config/mv2603/boot.bin) 
           Memory Address = 00200100 
  PPC1-Bug>pflash 200000 24ffff ff000000 
  Program FLASH Memory?  y 
 
4.  This programs the FLASH memory.  We power off b0eb25, 
    change J10 back to ROM bank A (two pins closer to front 
    panel), and reboot.  It comes back with the VxWorks Boot 
    prompt. 
5.  Enter appropriate values for the new node. 
6.  Enter b0eb25 into /home/vxworks/.rhosts.  Also make 
    sure it's listed in /etc/hosts 
7.  Boot VxWorks with '@' 
</pre>
<!-- Author: Jeff -->


<!-- Date: Tue Jun 15 10:44:58 1999 -->
<p> 
The i4515 driver package has been updated in the online 
cvs repository, and it has been rebuilt on gobi.  The 
main changes from the previous repository version include 
a lot of debugging info and the change to correct the 
"stuck buffer" problem.  The perf methods, formerly 
loaded in the atmtest.a module, have been incorporated 
into perf.c, so no separate loading is necessary; the 
startup script (b0l3pcom1:~vxworks/cdfatm/all/startup.vxsh) 
has been correspondingly modified, and atm.a has been 
copied from gobi to that same directory. 
</p>
<!-- Author: Jeff -->


<!-- Date: Wed Jun 30 16:07:13 1999 -->
I have loaded a new version of the NICStAR ATM driver into the kernel on b0l3c02 for testing. The installation isn't permanent yet; rebooting the system will reload the old version. 
 
<p>The new version waits for entries in the Receive Status Queue that are known to be in transit from ATM card to PC memory.The old version returned after taking entries that had already arrived, which sometimes left a late-arriving entry to be processed next time the end-of-packet interrupt occurred, i.e.,  at the arrivial of data from the next event.
<!-- Author: Stephen Tether -->


<!-- Date: Fri Jul 2 13:16:26 1999 -->
ATM reception does not seem to work for b0l3c02 although 
it does work for b0l3c03. We will use c03 for ATM driver 
tests for now.  
<!-- Author: Steve -->


<!-- Date: Fri Jul 2 15:10:04 1999 -->
I've replaced the SCRAMNet module for b0eb21 because it couldn't drive the relay in the optical bypass, although nothing else seems wrong with it. This is the second time we've seen this problem, and both times the affected module was in the lower VIPA crate on the third floor. Coincidence? (The last time was April 27 of this year).
<!-- Author: Steve -->


<!-- Date: Fri Jul 2 15:22:08 1999 -->
The lights are out on the ATM card in b0l3c02. Looks like another ATM card has died.<p> 
 
You can't change the current nicstar driver by using rmmod followed by insmod for the new version. You have to load the driver you want once, at boot time. Otherwise, ATM sends and receives will simply not work.<p> 
 
I'll be using b0l3c03 to test new nicstar drivers and I've "permanently" installed the experimental driver there. Other users should use one of the other converter nodes.<p> 
 
The new version of the nicstar driver that uses a busy-wait on the "valid" bit of RSQ entries doesn't work. For small jobs, a few senders, things are OK. When we tried to use ten SCPUs at once, reproducing the conditions which cause the tardy fragments to appear, the PC hangs and must be reset. I suspect that the busy-wait becomes an endless loop somehow.<p>
<!-- Author: Steve -->


<!-- Date: Mon Jul 12 11:07:23 1999 -->
Converter nodes 2 and 3 have new optional kernels, thanks to Ron and Don. The new kernel is called convTrace by LILO and has traces in the kernel IRQ handler. convTrace is the default now on b0l3c03, but the old "converter" kernel is still the default on c02 (you have to ask for convTrace at the LILO prompt to get the new kernel).  
<!-- Author: Steve -->


<!-- Date: Wed Jul 14 15:03:56 1999 -->
Yesterday Don and Ron made a kernel zImage-cliRon which is running now on b0l3c03. It can trace nearly every change of the CPU interrupt enable bit. Using that, I found that printk was running for long times, e.g. 80 ms, with interrupts disabled. The reason was that we had turned on console echoing to COM1. At 9600 baud, that means printk took 1 ms per character printed, and it disabled interrupts for the entire line. Naturally, the scheduler could not run during this time, causing delays in servicing  interrupts, running bottom halves and servicing the kernel timer list. At my request Don and Ron have taken taken out from zImage-cliRon all echoing to the serial port. This kernel will become the standard one for converter nodes.<p> 
 
Ilya has run the event builder in the configuration 10 SCPUs -> 
b0l3c03 -> 4 processors (no rejection) -> 1 output node for more than an hour, using a version of the driver I made that is called every 5 ms or so by the Linux timer bottom half (using the kernel timer list with an interval of 5 ticks). There were no tardy fragments reported. The timeout was set to 10 tries at intervals of 5 ms.<p> 
<!-- Author: Steve -->


<!-- Date: Fri Jul 16 17:21:36 1999 -->
I've reconfigured converter nodes b0l3c02-05. c01 will take a little 
more work because it has different hardware.<p> 
 
/etc/sysconfig/nicstar tells rc.local which version of the driver to 
use. Version 1_0 is Ron's version and 1_2 is the new one. Also, 
POLLING_INTERVAL is the interval in jiffies (clock ticks) between polls 
of the ATM card. On the converter nodes a jiffie is 1 ms.<p> 
 
The source code in /usr/src/linux/drivers/atm for nicstar is no longer 
up to date. The real source code is in the CDF online repository as cdfevb/linux/nicstar. 
The tag for the new version is v1_2.<p> 
 
You can see which nicstar driver is running via lsmod or cat /proc/modules.<p> 
 
There is only one kernel, named zImage, in /boot now. /etc/lilo.conf is 
set up to run it (of course) and calls it "converter". This kernel 
doesn't echo console messages to the serial port and adds tracing for 
each possible change of the CPU interrupt enable flag.<p> 
 
The source code for the new kernel is in /usr/src/linux-2.0.25.tar.gz 
and untarred in /usr/src/linux.<p> 
  
<!-- Author: Steve -->


<!-- Date: Fri Jul 23 13:55:06 1999 -->
A problem with SCRAMNet turned out to be  in the connection going from 
b0eb22(xmit) to b0eb13(receive). The CARRIER DETECT light was not on for 
13's module. I switched the two connectors for this cable at its 
entrance to 13's bypass module, and voila. I think that we have a 
combination of a half-bad cable with a half-bad transciever on one of 
the modules. Anyway, it should be OK for the weekend.<p> 
  
<!-- Author: Steve -->


<!-- Date: Mon Jul 26 17:28:16 1999 -->
We still have a problem with the event builder that showed up last Friday. For some reason the Scanner Manager writes a flood of messages to its SCRAMNet, causing its queue of outgoing messages to overflow, which forces the module offline. Thereafter every attempt to read from the SM SCRAMNet RAM results in a bus error (and data is all one-bits) until the module is reset. Garbled data in the SCRAMNet network might cause the SM to become confused about how many messages it has received.<p> 
 
Today I wrote a small program to write a pattern of a rotating bit to all 128K of SCRAMNet memory on one of the 2603's. The pattern was duplicated correctly on all other SCRAMNet modules on line, including those in converter nodes. Next I'll try writing into each module in turn to see if the originating module makes a difference.<p> 
 
The power supply for the crate containing b0eb18 and 19 has blown out, so I took their SCRAMNet module out of the loop.<p>
<!-- Author: Steve -->
<!-- Comment: Tue Aug  3 12;59;22 comment by...Steve -->
Power was off for a few hours on Monday morning, so everything had been power-cycled between the time the problem was discovered and the time I made my tests. I guess that cleared the problem.
<!-- Date: Fri Jul 30 13:37:52 1999 -->
I finally got around to making b0l3c01 reboot. It turns out that its root partition is /dev/hda7 and not /dev/hda6, so I had the LILO configuration set incorrectly. All the older machines (b0l3pcom1, b0l2002, b0l3003, b0l3c01) have the root partition on /dev/hda7 and all the others have it on /dev/hda6.<p> 
 
I have also changed the root password on the older machines to match the one on all the others.<p>
<!-- Author: Steve -->


<!-- Date: Fri Jul 30 14:14:41 1999 -->
This morning Alex, Ilya, and I tested Alex's Java UCI. There were a few interesting glitches, but we were able to define partitions and start/stop runs.<p> 
 
Glitch #1: The Scanner Manager expects each UCI message to be sent  in a single packet, but each Java write of an int to the socket output stream was sent by itself. The SM should be smarter about message reception, but we worked around the problem by wrapping the output stream in a BufferedOutputStream with a buffer size larger than any possible UCI message. After the last int of the message is written, we flush the buffer.<p> 
 
Glitch #2: When the SM receives a UCI End Run message, it complains about not being able to communicate with the Trigger Supervisor for the partition. During Run 1b the SM had a direct link with the TS for each partition, and was responsible for telling them to stop triggering when End Run was received. Now the TS's will be controlled by the Trigger Manager, which is told to stop a TS by Run Control. Why haven't we seen this (non-fatal) complaint before?<p> 
 
Glitch #3: When the SM receives End Run, it sends a DELETE_PART SCRAMNet message the the SCPUs, which obediently erase all record of the partition. However, several events for the partition may still be awaiting sending, and when the SM tells the SCPUs to send them they complain that the partition no longer exists. So the events are lost. Evidently the steps required to stop a run correctly are: 
<ol> 
<li>Run Control disables the trigger for the partition. 
<li>Run Control tells the EVB proxy to do an End transition. 
<li>The UCI sends QUERY commands to the SM to find out how many events are still queued in the SCPUs and waits until this number has been stable for a few seconds. 
<li>If there are still events waiting in the SCPUs, the UCI should send Flush DAQ for the partition in order to release the stuck buffers. 
<li>The UCI sends End Run to the SM. 
<li>The EVB proxy goes to the Ended state and informs Run Control. 
</ol> 
<p> 
 
The new UCI code will go into the online CVS repository under cdfevb/proxy/juci. There are three packages 
<ul> 
<li>juci.action, which communicates with the SM. 
<li>juci.gui, a simple GUI. 
<li>juci.shell, a simple command parser which can be driven with an Expect script. 
</ul> 
Right now the shell only parses, but we'll be adding action calls soon. 
<p>  
<!-- Author: Steve -->