<!-- Title: 14:24:44  Mon Nov 15 1999 -->
<!-- Crewchief: Christoph Paus -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Steve Tether -->
<!-- Op3: Ilya Kravchenko -->
<!-- Op4: Andreas Korn -->
<!-- Op5: Michael Mulhearn -->
<!-- Op6: Ivan Furic -->
<!-- Notes: -->
<pre>  The previous log is getting long, started the new one. 
</pre>


<!-- Date: Mon Nov 15 14:29:10 1999 -->
<pre>  Last days I had a problem running a simple 1-1 system. 
The sympthoms: 
  - Zephyr message "SM partitions internal error 
 in push_builder 1" appears in large numbers. 
  - Events go through, only the event rate is unstable 
  - In scramnet memory there is no apparent error, only 
 there are more messages than expected for 1-1 system 
The cause: 
  One of the converter nodes had l3_converter running 
and forgotten with the same base key. 
Solution: 
  Make sure that the whole system is clean. 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Sun Nov 21 22:46:55 1999 -->
<pre> I have been having problems with b0l3c03 node lately. 
When used as L3 converter, it behaved strangely. Some runs 
no events were received, some runs events were observed, 
but only small number and unstable. Once I saw exactly 
19 good - 1 bad (input failure) pattern. 
 
  It appears that the ATM card is having problems. General 
check of scramnet is ok and there are no error messages 
anywhere (except for "input failure" in /log/l3_converter). 
  A test with atm_write from SCPU to b0l3c03 always works 
well. The self test  when atm input and output on the PC 
are connected gives unusual results: ./awrite to ./br 
test works well (single message is sent), but the  
test ./bw to ./br does not work at all (multiple blocks 
of garbage are sent). In this latter test on the concole 
(and /var/log/messages) I see the following: 
Nov 21 23:00:55 b0l3c03a kernel: nicstar0: SDU CRC is bad. VPI = 1, VCI = 1. 
Then, I noticed the same message in the messages file 
during my previous tests of EVB/L3 of today. 
 
  So I did in-out procedure for PCI cards, and bw->br results 
remained as above. Finally, I decided to quit and reassembled 
everything. Miraculously, at this moment it is  working in  
1->1 EVB/L3 test. But I suspect that we will see this problem 
again in the future. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Tue Nov 23 12:59:06 1999 -->
Today b0l3pcom1's IP number has changed to 131.225.236.8, gateway 131.225.236.200. I've made these changes on pcom1 itself and I've updated the /etc/hosts file on converters 1-16, processors 1-30, and output nodes 1-4.  
<!-- Author: Steve -->


<!-- Date: Wed Nov 24 12:02:35 1999 -->
We ran out of space on b0l3pcom1:/usr, so I've moved the products directory to /cdf. /usr/products is now a symbolic link to the new location. This shouldn't cause any problems, but people should log out and back in to make sure that all space for deleted files is reclaimed.<p>
<!-- Author: Steve -->


<!-- Date: Wed Nov 24 13:52:13 1999 -->
Now the current version of cdfevb_sm on b0l3pcom1 is v3_6_patched, which is a branched version of cdfevb_sm that has MAX_FRC set properly in evbLimits.h. This means that the slot 15 fix is complete. The current versions of SCPU (v3_5) and VRB (v3_0) have been rebuilt using the new SM. 
 
v3_6 of SM is now the "old" version, obtainable by using setup -o or explicitly specifying that version.<p>
<!-- Author: Steve -->


<!-- Date: Wed Nov 24 13:58:13 1999 -->
<pre>Ilya reported a network problem when rebooting 
all 15 SCPU's. In the log file inetd stopped rsh 
with the message: 
  inetd[728]: shell/tcp server failing (looping), service           terminated 
For diagnostics I used netstat -c and also started inetd with 
the debug option. It seems during reboot a number of reconnections are made so that rsh exceeds the default limit of 40 attempts. The problem was fixed by editing /etc/inetd.conf 
and setting the limit to 100: 
  shell   stream  tcp     nowait.100      root    /usr/sbin/tcpd    in.rshd</pre>
<!-- Author: Andreas Korn -->
<!-- Comment: Mon Nov 29 12;31;45 comment by...Steve -->
The limit Andreas refers to is on the number of instances of the server that can be created in 60 seconds. The default is 40 under Linux inetd.


<!-- Date: Thu Dec 9 11:39:04 1999 -->
<pre>ATM card in b0l3c03 appears to be not   completely good. 
Used as a converter node sometimes it wors normally 
but sometimes no events go through. To test ATM I  
connected input to output with a fiber and  
started br and awrite. This worked. Then I tried  
bw to br and there was no responce (same procedure 
works on all other converters). I have replaced the 
card, now br to br test work. The bad card I have put 
back into cabinet, marked.</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Sun Dec 12 23:07:14 1999 -->
<pre>  As I mentioned at the EVB/L3 meeting last Thursd., I have  
been having problems with very long time that Init procedure  
takes (i.e., "init" button of Jeff's kludge). Often for a big 
system when "partition is full" and "Begin" button is pressed, 
the system breaks.  
 
  So I see 2 problems starting the big system: 
    1) takes too long to initialize. >10 min - too much. 
    2) "SM lost connection to SCPUs" problem. 
In this e-mail I am addressing mostly the first one. 
(actually, there is #3, with running complicated 
system with fake trigger manager at high rate). 
 
  At first I measured time that "init" stage takes. 
Here I can see that at some point there is a sudden increase 
in time required. Here are my data on this time interval in seconds 
for different configurations.  
 
               N SCPUs   1      3     7     8    9    10   11    15 
 N converters        
      3                 10     11    --    11   12    14   --    16       
      5                 20     19    --    24   39    34   --    75 
      7                 28     26    --    64   75   135   --   165      
      9                 34     34    --   205  204   233   --   270 
     11                 43     42    --   332  362   346  394   XXX 
     13                 51     51    --   ---  ---   ---  ---   --- 
     15                 58     58    59   535  535   ---  595   XXX 
 
XXX - could not go past "start", tried 3 times. Either scanner manager 
hangs or "lost connection to b0ebxx, still trying". "--" means 
that I did not take that point. 
 
So, we see the problem first at the point 8SCPUs->8CV (it takes 
138 seconds, although this point is not in the table) and then 
it appears to move to  ->7CV for 10 and larger number of SCPUs. 
 
  Typically, for complicated system init goes quickly through 
the first several nodes but then it starts to take 45-60 seconds 
per node. Judging from the messages in Zephyr window. 
 
  Does anyone have any idea about this problem?  
I did try Jeff's suggestion to increase listen(5..) to listen(15...), 
did not have much effect as he predicted. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Thu Dec 16 10:26:43 1999 -->
Ilya and I found three problems yesterday while examining the startup process for the event builder with 15 SCPUs and at least ten converter nodes. We were using simulated events (no VRB access) and writing to trash on the converters. 
<ol> 
<li> If there is some other activity such as VRB tests in an SCPU crate, it interferes with the SCPU's reading of SCRAMNet. Apparently an occasional VME bus error occurs, causing a value of 0xffffffff to returned from a SCRAMNet read operation. By far the most common type of SCRAMNet read is the testing of the first word in a message slot, the tag, for a change indicating the presence of a new message. The problem therefore tends to show up as an erroneous decision that a new message has arrived. When the SCPU goes to read the message later, the bus error doesn't occur and the true value of the tag word is seen. The command picked up is garbage, of course, and the the SCPU rejects it, but the damage has been done, since the loop checking for new messages has advanced to the next message slot. When a real message arrives in the previous slot it won't be detected. The problem was most frequent in crates 1RR21C-B and 1RR21F-B, but did not occure when these creates were observed to be idle when we started the event builder. 
<li>The Scanner Manager's code in registry.c, which handles the registration of Builders during grand init, seems to be woefully inefficient. The network connections from the Bosses are accepted promptly and a task is spawned for each Boss (one per converter). Each registration involves messages to and from every SCPU, so to prevent confusion each registration task must take a semaphore which it only releases when all message exchanges are done. The first few registrations go quickly, but later ones slow down enormously for no apparent reason. We will have to carefully examine registry.c to understand what is going on here. 
<li>Sometime late in the registration process, the TCP connections between the Scanner Manager and five of so SCPUs just disappear. Sometimes this happens too late to affect the last registrations, but often the SM will end up trying to read from a connection that no longer exists and complain rightly that contact with the SCPU has been lost. These network connections are established at event builder startup and are supposed to be kept until an event builder restart. We have not seen any of these connections enter a FIN_WAIT or CLOSE_WAIT states using inetstatShow. The connections affected seem to be among the third or so of the SCPUs as they are listed in the SM config file. 
</ol> 
The first problem can be gotten partially around by treating 0xffffffff as an invalid tag instead of the start of a new message, but there is no guarantee the a non-tag word of a SCRAMNet message won't eventually be misread.<p>
<!-- Author: Steve Tether -->


<!-- Date: Thu Dec 16 20:03:05 1999 -->
<p>Today I instrumented the Scanner Manager code in registry.c in 
order to see where the time was being spent during Builder 
registration, which happens after you click the Init button on the 
control kludge. Each time a converter node sends a registration 
request to the SM, each SCPU listed in the config file is sent a 
message telling it to expect connections from converter nodes over 
ATM. The converter node is sent a message telling it to make the 
connections. Each SCPU and the converter node sends back to the SM a 
message saying which Builders managed to connect. Since we use 
permanent virtual circuits for ATM, the connection process is 
virtually instantaneous, so the replies ought to come back very 
rapidly. [Note 1] 
 
<p>What Ilya and saw at first was very different. The replies from the 
SCPUs came most of the time in two bursts with roughly equal numbers 
of replies in each and a noticable delay between bursts. Furthermore, 
each time a new converter registration handshake was performed, the 
delay between bursts increased. It started at about a second or so and 
went up to more than a minute. Thus most of the time during 
registration was spent waiting for replies from SCPUs. 
 
<p>The situation changed dramatically if we put a modest delay before 
each transmission of the "expect connections" message to an 
SCPU. Using half a second, the entire registration sequence for 15 
SCPUs and 10 converters takes about 90 seconds instead of more than 
ten minutes. As a bonus, the TCP connections between SCPUs and SM 
established at startup remain intact. 
 
<p>Although all the converter nodes connect to the Scanner Manager on 
the same port at about the same time during registration, it doesn't 
seem to cause a problem. The accept() call returns promptly every 
time. Each connection gets its own task, but the internal registration 
database in the SM is protected with a semaphore which is not released 
until the registration process is complete for that converter 
node. Therefore only one registration task is really ready to run at 
any given time. 
 
<p>At present we don't know what causes the delay in SCPU replies to 
the SM when all are sent messages as quickly as possible. It could be 
a delay in the transmission of the original messages to the SCPUs, a 
delay inside the SCPU, coming back, or a delay internal to VxWorks at 
the SM. It seems more likely that there would be problems with fifteen 
nodes trying to send to one than the other way around. 
 
<p> 
 
<h3>Notes</h3> 
<ol> 
<li>In Run 1b we used UltraNet, so the usual TCP connection handshake took place between L3 and SCPUs. 
</ol><p>  
<!-- Author: Steve Tether -->
<!-- Comment: Mon Dec 20 15;43;40 comment by...Steve -->
Further notes on problems with a large event builder will be in the EVB E-log from now on.
<!-- Date: Sun Dec 19 22:16:16 1999 -->
I've turned off all L3 PCs, the ATM switch,mand the ethernet switches in anticipation of tomorrow's power outage. The Scanner Manager and SCPU VME crates were already off. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Dec 20 15:41:15 1999 -->
The L3 farm should be left running over the holidays for the mock data challenge. I've made two small changes on b0l3pcom1 at Kevin McFarland's request. 
<ol> 
<li> User "mdc" is now defined in /etc/passwd.nis. It belongs to group "cdf". 
<li> Root version 2.22/06 compiled with kcc 3.3 is available in /usr/products. To get it do "setup root v2_22_06" or "setup -q KCC_3_3 root v2_22_06". Just "setup root" still gets you the old version. 
</ol> 
<p>I've also made yppasswd active by default for init modes 3 and 5. I had to change /etc/rc.d/init.d/yppasswd in order to get the daemon to use /etc/passwd.nis instead of /etc/passwd.<p>
<!-- Author: Steve Tether -->



<!-- Date: Thu Jan 6 11:26:01 2000 -->
<p>Kevin sent us mail regarding the fact that b0l3014 was 
inaccessible.  When I went to take a look, the hard disk 
indicator was on.  I reset the machine by pressing the 
front button.</p> 
 
<p>The machine still didn't come back, so I connected it to 
a monitor and keyboard.  The bootup procedure was stuck in 
fsck, which was refusing to auto-correct /dev/hda8.  It had 
to be done manually.  This was done:  fsck found too many bad 
blocks in inode 124934, so the inode itself had to be cleared. 
This inode was reported to belong to the file 
/mdc/run90077_exe2/core (why /mdc and not /cdf/mdc, I don't 
know---there is no /mdc directory, but /cdf/mdc/run90077_exe2 
does exist).  At this point I did not have fsck clear the file 
itself.  Unfortunately, when the computer was rebooted, fsck 
auto-cleared it.</p> 
 
<p>Two things to watch for next time this happens:</p> 
<ul> 
<li>Connect the monitor and keyboard <i>before</i> rebooting. 
    The computer itself might still be up, in which case a 
    more graceful termination might be dreamed up (unlikely in 
    the case of a real core dump, but worth a try). 
<li>Check the size of the file being dumped before restarting 
    the computer.  Better yet, if it's not excessively large, 
    copy it somewhere for later debugging (and it might be 
    useful for filter debugging). 
</ul>
<!-- Author: Jeff -->


<!-- Date: Fri Jan 7 19:44:57 2000 -->
<pre>  Today we have been sending DEM data from SVX crates (2 
VRBs) through EVB/L3/CS to a calibration Consumer of  
Rick St Dennis that also connects to OracleDb. 
 
  There were several problems, I list them here mostly for my reference: 
 
1. l3_wedge_exe dies, dumps core file. I had the wrong command_input.tcl file. 
 
2. Reformatter complains to SVX data in the following way: 
   == ERROR rr_refevt_bank_add_mini ==  
 
  bank name    : SIXD 
  bank instance: 61 fragment instance: 61 
  mini fragment appeared with ambiguous instance 
  mini fragment not added to bank 
  >>EXIT  rr_refevt_bank_add_mini 
 
 The reason is that the FIB ID is the same for the two  
VRBs that I read. To correct this ask SVX people to change 
their setup. 
 
3.  This problem is more difficult. The reformatter error: 
 
     == ERROR rr_refevt_bank_set_vrbword ==  
 
    VRB word is not consistent with the existing one. 
    Name     : SIXD 
    Type(hex): 1 
    >>EXIT  rr_refevt_bank_set_vrbword 
 
 In the event itself, there are unusual things: vrb status 
words are not zero (indicates errors), event number is 
different for different VRBs (should be the same).  
Also, a Glink error was found, not clear if it is something 
to worry about. Steve Nahn wiggled cables and the problem 
mostly went away. 
   Right now it appears in the first several events of the run. 
The first event is ok, but from approx. second to eight this reformatter 
message comes and also the VRB size read out is smaller than  
expected and is changing. After the 10th event it stabilizes. 
   In previously described tests EVB/L3 was started after SVX (automatic mode). 
In several tests when EVB/L3 is started first and SVX second 
I observed no errors. 
 
4. There were errors in the data content today that 
disappeared by the evening. See more in SVX-II e-log entry  
by Rick St Dennis of today. 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Tue Jan 11 23:37:18 2000 -->
<pre>Performance tests done tonight. For the first test we go up to 
15SCPU->16 converters. No processors yet. 
 
------------------ 
   N SCPU -> 16 converters -> trash. size=16k (really 16632B),  
n=188, ClkRate=1000 
 
NCPU  Rate, Hz       Rate, Hz/CV  total MB/s    MB/s/CV      Ev. size    mis? 
  1   570.3+-0.1       35.6          9 MB/s      0.6           16kB       no 
  3   500.7+-0.2       31.3         24 MB/s      1.5           49kB       no 
  5   499.7+-0.1       31.2         40 MB/s      2.5           81kB       no 
  7   499.8+-0.1       31.2         56 MB/s      3.5          114kB       no 
  9   499.5+-0.2       31.2         72 MB/s      4.5          147kB       no 
 11   499.7+-0.2       31.2         87 MB/s      5.4          179kB       no 
 15   499.2+-0.2       31.2        119 MB/s      7.4          244kB      yes 
 
(mis? means Tag mismatch error in converter log files) 
 
------------------- 
  N SCPU -> 1 converter -> trash. size=16k, n=188, ClkRate=1000 
 
NCPU  Rate, Hz       Rate, Hz/CV  total MB/s    MB/s/CV      Ev. size, kB 
  1   65.8+-0.01         -          1.0           -             16             
  5   65.8+-0.02         -          3.1           -             81 
 11   65.7+-0.04         -         11.3           -            179 
 15   65.6+-0.07         -         15.6           -            244 
 
------------------- 
  1 SCPU -> N converters -> trash. size=16k, n=188, ClkRate=1000 
 NCV  Rate, Hz       Rate, Hz/CV  total MB/s    MB/s/CV      Ev. size  
  1   65.8+-0.01        65.8        1.0          1.0           16 
  2  131.5+-0.05        65.8        2.1          1.0           32 
  4  262.9+-0.08        65.7        4.1          1.0           65 
  6  391.4+-0.03        65.2        6.1          1.0           97 
  8  511.2+-0.11        63.9        8.0          1.0          130 
 10  545.2+-0.08        54.5        8.5          0.9          162         
 12  570.3+-0.08        47.5        9.1          0.8          194 
 14  570.6+-0.10        40.8        9.1          0.7          227 
 16  570.3+-0.1         35.6        9.1          0.6          244 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon Jan 17 10:59:10 2000 -->
<pre>  Some work that has been done lately: 
 
We have swapped two PCs in the farm because of PCI bus problems: 
   - b0l3c06 contains: hard drive of c06 but the body of b0l3030 
   - b0l3030 has hard drive of 030 but body of b0l3c06 
  This is done because the processor nodes do not use PCI bus  
  cards. 
 
A bad bypass switch has been replaced, it was formerly attached 
to b0l3c15. The switch has been marked and put in the cabinet. 
 
  Level 3 PC activity of last Friday: 
 
     - I have changed the interrupts BIOS settings on all the  
         converter nodes, disabling the 24-IRQ option. 
 
     - In the process, I have noticed that b0l3c08 shows only 
         one 500MHz Pentium III processor during start time. 
         Also, hard drive LED is red while for the others it is  
         yellow. Is this computer of different model? 
 
     - After the change so far I have not noticed any differences 
         running big systems. I.e., it is still 570Hz  
         for 1SCPU->16 converters 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon Jan 17 11:44:32 2000 -->
<pre>   This weekend I excercised EVB/L3, this time including processor 
nodes. I hoped to get some results for the chep paper. In the 
end I did not put in anything except for the plot shown by Steve 
at the last meeting, up to 8 converters. 
 
  I have used up to 8 subfarms, each with 4 processor nodes, 
and 4 output nodes. The rejection factor was 90%, fragment size 16k 
and Nscpu=15. The rate goes linearly at first, but then saturates 
at 260 Hz. Christoph suggests that this limitation comes from 
the level 3 network architecture. 
 
  Also, I did a longer run. It included 6 subfarms with 3-4 processor  
nodes each and the output nodes. This test lasted for about 8.5 hours 
with all subfarms running. At some point, my workstation ws rebooted 
which caused everything to die. 
 
  I observed unusual behavior of some processor nodes, possibly 
I will have problem with them later. For example, b0l3021 I could not 
use. It looks normal when I check, but l3_farm_user can not connect to  
it. In addition, I get this: "connection reset by peer" - from time to  
time. I could not use b0l3001 as a processor as well, but did not try 
to understand why. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Thu Jan 20 16:38:09 2000 -->
<pre>Studying rate limitations of the L3-EVB system. 
 
These tests did not change the fact of saturation at 570Hz 
 for 1->16 system: 
   Modified SCPU's N of ATM buffers and buffer size 
   Modified primary and aux. system clock rates on SCPU 
             (weak effect, 5-10 Hz) 
   Modified R between 188 and 222 (1-2 MB/s) 
   Modified TM frequency 
   Run TM on different SCPU  
 
This test makes a difference: 
   Change fragment size 
 
Other: 
 
- Test 1scpu-1converter system at high data troughput. Checking,  
for example for optical fiber limitations. 
First, 16kB fragment size. 
  N    Exp. throughput    Real throughput     Event rate 
 188     1.0 MB/s           1.0MB/s            65.8+-0.0Hz 
 215     1.7 MB/s           1.7MB/s           109.0+-0.1Hz 
 225     2.2 MB/s           2.3MB/s           143.5+-0.2Hz 
 235     3.3 MB/s           3.2MB/s           199.4+-0.0Hz     
 240     4.6 MB/s           3.5MB/s           219.5+-1.6Hz 
 243     5.6 MB/s           3.6MB/s           229.4+-0.7Hz 
 246     7.4 MB/s           3.9MB/s           247.9+-2.6Hz 
 248     9.5 MB/s           4.1MB/s           257.6+-1.4Hz 
 249    11.1 MB/s           4.1MB/s           261.1+-1.2Hz 
 
Second, 24kB fragment size 
 188     1.0                1.0                44.1+-0.1                    
 225     2.2                2.3                96.2+-0.2 
 240     4.6                3.5               149.6+-0.2 
 246     7.4                3.5               149.6+-0.1              
 249    11.1                3.5               149.6+-0.2 
 
Third, 24kB fragment size with Aux. clock of SCPU at 1000 Hz 
 249    11.1                5.8               245.8+-0.1 
 
 
Random comment: 
  - Not everything is ok with booting. Sometimes "Failed to hang..." 
  message does not appear although SCPU seems to be ok.</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Thu Jan 20 18:51:53 2000 -->
<pre>Ilya found out a few days ago that b0l3c08 was using 
only one of its 2 pentium processors. I rectified this 
as follows: 
 
BIOS server tab has a field where one can select "retest 
 processor" on next boot. I selected this, rebooted, both  
pentiums were recognized. I then verified that retest is  
again deselected in BIOS (as was the default), rebooted again, 
both CPUs still recognized. 
 
We need to check all processor nodes to make sure that 
they are using both CPUs. Ilya did so for the converter nodes 
already. 
 
</pre>
<!-- Author: fkw -->


<!-- Date: Fri Jan 21 23:29:43 2000 -->
<pre>The following e-mail was sent around today by myself: 
 
  We had some progress in understanding the rate limitations of 
the large EVB/level 3 system. The major problem was in event 
simulation when filling the event buffer with dummy data. 
Christoph has modified the reformatter (in a temporary way, the 
real change will have to be made and submitted).  
   The change is the following. Before, the event fragment was  
created with normal structure and filled with longs "12345678". 
Now, the content of the memory buffer is modified only  
to create appropriate structure while the data section contains 
unmodified junk that was there before - just like Jeff does in  
his Fake Vrb. 
   Now the rate observed for 1scpu -> 16 converters is approx. 1kHz 
as expected for 16kB fragments and 1MB/s/converter output of SCPUs. 
The degradation of the rate with the increased number of SCPUs is 
not significant up to about 8 SCPUs. 
   However, I had trouble starting bigger system. Typically, only  
some subfarms start and also some number of tardy fragments is 
observed. This needs more testing. Also, SCPUs hang more often 
and require hardware reset. 
   I am going to put a set of performance tests done yesturday and 
today to the Level 3 e-log. 
 
  Note, that the following changes to the official libraries are pending: 
    1) SM: Steve's fixes to various problems. 
       Trigger Manager: my modification that can run at high rate 
    2) VRB: vrb_simu is changed so that simulated events handle 
           slot numbers 6-20 properly. 
    3) RAWREF: the fast simulation for SCPUs 
Out of these, items 1) and 2) currently exist only in my directories. 
 
   Also, we lost 1 SCPU apparently because one adapter card is not  
working anymore. I am in the process of getting a replacement from 
Fred Lewis. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Fri Jan 21 23:33:17 2000 -->
<pre>These are some performance tests done yesturday and today. 
The problems are described in several previous log entries. 
 
----> 
---->   Studying 1->1 system, sending 16kB fragments 
----> 
  When running with the VrbFake, all of the following make a  
difference: 
   o l3_wait on the converter node 
   o sysAuxClkRateSet on SCPU (default limits to 300 Hz) 
   o sysClkRateSet on SCPU (minor influence) 
   o sysClkRateSet on SM 
Maximum observed rate is a little over 400 Hz. 
 
  When running with VrbSimu, the above makes little (5-10Hz) 
difference while rate saturates at about 260Hz. 
 
----> 
---->   Conditions: size 16kB, 1MB/s/scpu, using Jeff's FakeVrb. 
----> 
  1. 1scpu->Nconverters 
   Nconv    Rate, Hz 
     1       66.7+-0.1 
     2      132.6+-0.1 
     4      265.8+-0.1 
     6      398.8+-0.1 
     8      532.0+-0.3 
    10      665.4+-0.2 
    11      731.7+-0.1 
    12      798.0+-0.5 
    13      864.0+-0.7 
    14      931.1+-0.1 
    15      996.3+-0.2 
    16     1022.9+-0.8 
 
  2. 16scpu -> N converters 
   Nconv     Rate, Hz    
    16  
 
  3. Nscpu->16conv 
   Nscpu     Rate, Hz 
     1      1022.9+-0.8 
     3       995.9+-0.3 
     7      1006.2+-0.3 
     9      1015.3+-0.2 
    13       957.0+-3.3 
 
Comment: sometimes difficult to sturt, SCPU(s) hangs. Could 
   not start >10scpu->16conv. 
 
-----> 
----->   Now conditions are as above, but using VrbSimu 
----->  fixed by Christoph. 
-----> 
 
  1. 1scpu->N converters 
    N conv        Rate, Hz 
      1             65.9+-0.1 
      2            131.3+-0.3 
      4            263.1+-0.1 
      6            394.6+-0.1 
      8            526.1+-0.1  
     10            657.3+-0.1 
     12            788.3+-0.2 
     14            918.0+-0.1 
     16           1010.6+-0.2 
 
  2. Nscpu->16conv 
   N scpu         Rate, Hz 
      1           1010.6+-0.2 
      3           1002.9+-0.3 
      5           1002.7+-0.2 
      7            989.8+-1.9 
      9         could not start all subfarms    
Comment: 
   - have been having problems starting the large system of  
order 10 or so SCPUs to 16 converters at 1kHz rate. More tests 
are needed. 
 
  3. A test 1 scpu -> 1 converter -> 1 processor -> 1 output node 
was ok. There was only 1 complaint from the reformatter in the  
beginning. This test was necessary to check if the fixed/fast 
reformatter is ok. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Wed Feb 16 22:39:35 2000 -->
<pre>Here is e-mail from Tony: 
 
I've moved the four b0dau31 ethernet cables from switch 5  
to switch 2 and switch 3 to get connections directly from the  
Level-3 PCs to the CSL. For documentation purposes, I describe 
below exactly what I did. 
 
name convention:  b0dau31a = NET-CS-01 
                  b0dau31b = NET-CS-02 
                  b0dau31c = NET-CS-03 
                  b0dau31d = NET-CS-04 
 
original connections: 
NET-CS-01 : SW 5 PT 11  (switch 5, port 11) 
NET-CS-02 : SW 5 PT 12 
NET-CS-03 : SW 5 PT 23 
NET-CS-04 : SW 5 PT 24 
 
I changed to this: 
NET-CS-01 : SW 2 PT  8 
NET-CS-02 : SW 2 PT 20 
NET-CS-03 : SW 3 PT  8 
NET-CS-04 : SW 3 PT 20 
 
 
Tony 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Fri Mar 10 01:03:14 2000 -->
<pre>A test node with FRHL 611 is available as b0l3040. 
Installed with cdfonline workgroup. 
Changes needed after install: 
/etc/fstab ,/etc/hosts/, /etc/ntp.conf /etc/ntp ,/etc/yp.conf /etc/sysconfig/network, /etc/sysconfig/static-routes /etc/sysconfig/network-scripts/ifcfg-eth0, /etc/rc.d/init.d/xntpd 
Modified UPS: /local/ups/etc/upsdb_list added /usr/products/upsdb 
              ups undeclare shrc 
</pre>
<!-- Author: Andreas -->


<!-- Date: Fri Mar 17 17:15:55 2000 -->
<b>Turned off telnet, ftp, and gopher inet services.</b> 
All user access should be conducted through ssh and 
related utilities.  
<!-- Author: Jeff -->


<!-- Date: Fri Mar 24 17:19:41 2000 -->
<pre>Ilya and I made some changes to port assignments last Tuesday 
to get farm back after power outage. 
 
switch 1 not working presently. The only ports on that switch that we really need are: 
 
sw1p2, p14, p13   (net-atm, net-sm, net-gate01) 
 
we switched those to: 
 
sw5p8, p20, p19 
 
Currently unavailable are thus converters1-4, proc.2-8 and 40 , 
output node 1. 
 
Recall that proc40 is the one that has redhat 6.1 . 
 
This was the simplest re-arrangement. Alternative options, like 
replacing switch with hub were not attractive because hup has 12ports 
while 16ports on switch are presently used. 
</pre>
<!-- Author: Frank -->


<!-- Date: Tue Mar 28 14:32:54 2000 -->
<pre>I installed 30Gb disks into b0l3c12,c14,c15 last night. 
One disk is still unavailable as c13 has problems. 
The are mounted to /consumer12 /consumer14 /consumer15 
on the respective machines. 
As the 2.0.25 kernel did not proper recognize the disks, 
I had to do the following: 
change CHS in the bios to the REAL value (59598,16,63)[the fake 8Gb value reported is a fix for large disks].  
add hdc=59598,16,63 to lilo.conf. Run lilo. 
Run fdisk and mke2fs from 2.0.37 bootdisk to format and partition disk proper.</pre>
<!-- Author: Andreas -->


<!-- Date: Tue Mar 28 20:48:39 2000 -->
<pre>Installed ups v4_5_1 and upd v4_5_2. I used modified bootstrap scripts. The standard version is UPD_VER=v4_4_3 UPS_VER=v4_4a. 
I had to edit the localboot.sh and the two! (one in current one in /var/tmp/bootstrap) cfg.version files.  
After installation the /local/ups/etc/upsdb_list had to be updated. Entries added:  
/usr/products/upsdb 
/cdf/code-common/products/upsii_database 
/cdf/code-common/cdfonline/upsii_database 
 
 </pre>
<!-- Author: Andreas -->


<!-- Date: Wed May 31 23:02:07 2000 -->
<pre>I made a second gateway node b0l3pcom2/b0l3gate2. 
I needed to compile a new tulip driver  
in order to get the Linksys card to work. 
Files that needed to be changed: /etc/conf.module and /etc/sysconfig/network-scripts/ifcfg-ethx 
On b0l3pcom1 I used: 
 upd install -z /local/ups/db/ oracle_client 
to install oracle.</pre>
<!-- Author: Andreas -->


<!-- Date: Sun Jun 18 16:21:29 2000 -->
<p>I've added the following lines to /local/ups/etc/upsdb_list on b0l3pcom1: 
<pre> 
/cdf/code-Linux-2.0.34/products/upsii_database 
/cdf/code-Linux-2.0.34/freeware/upsii_database 
/cdf/code-Linux-2.0.34/cdfoffline/upsdb 
</pre> 
They come just after the local UPS databases. I've also changed /etc/fstab so that /cdf/code-Linux-2.0.34 is mounted from b0dap30. 
<p> 
I did this to make it more likely that setups of CDF software will work. For example "setup oracle_jdbc" failed because the version of oracle_jdbc available from /cdf/code-common was not declared to be on any chain, including "current". This in turn caused "setup fer devel" to fail.<p>
<!-- Author: Steve Tether -->
<!-- Comment: Fri Jun 23 17;17;27 comment by...Andreas -->
I was trying to make the same change for b0l3gate2
and 2.2.14smp Kernel ....
<!-- Date: Thu Jun 22 17:28:24 2000 -->
<p>Due to problems this morning with b0dap30, b0l3gate1 is mounting the Linux versions of CDF products from b0dau30:/linux_backup (altered /etc/fstab). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jun 23 16:57:25 2000 -->
<pre>Two problems on gate1: 
I moved files from /home/save to free space. 
I also restarted portmap,ypserv,ypbind as 
ypserv used lots of processing time ... 
 
</pre>
<!-- Author: Andreas -->



<!-- Date: Tue Jul 18 13:56:55 2000 -->
<p>Setting up oracle_tnsnames on gate2 resulted in a complaint from UPS that PROD_DIR was undefined (the table file used $PROD_DIR). The problem seems to have been caused by attempting to redefine the "current" chain of the product to be v12 in the local UPS database while it is v6 in /cdf/code-Linux-2.0.34/products/upsii_database. I don't know why this didn't work. Anyway, I got rid of the local database entry for oracle_tnsnames and changed the "setupRequired" in the table file for oracle_client v8_0_5_1a to ask for v12 of oracle_tnsnames rather than the current version. <p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Jul 26 09:40:46 2000 -->
<p>At the request of OOC I've installed ORBacus v3_3_2 on b0l3pcom1 for testing with the event builder proxy. This is the latest of the release 3 ORBacuses. I've only installed the Java support for now because the C++ support requires gcc 2.95 and we only have 2.91. After setting up ooc v3_3_2 $OOC_DIR/lib will be on LD_LIBRARY_PATH and $OOC_DIR/bin will be on PATH. Put $OOC_DIR/lib/OB.jar on CLASSPATH.<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jul 28 16:04:40 2000 -->
<pre>Test message from Dave Waters. We might use this log book  
for filter debugging info as well.</pre>
<!-- Author: David Waters -->