<!-- Title: 10:03:07  Tue Jun 22 1999 -->
<!-- Crewchief: Makoto Shimojima -->
<!-- Op1: Kevin McFarland -->
<!-- Op2: Tony Vaiciulis -->
<!-- Op3: Ben Kilminster -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<center><H1>Consumer-Server/Logger Activities</H1></center>


<!-- Date: Fri Mar 3 19:56:39 2000 -->
(Entered by Tony on March 1; moved here by mako on March 3) 
<br><br> 
Here are the results of CSL rate testing on the morning of Wednesday, March 1. The main point of today's test was to check the effect of having the logger send the messages directly back to the receiver queue instead of to the distributor queue. It seems like the effect is small, though the 8 sender rate did increase a little, on average, from about 18 MB/s to about 19 MB/s when the distributor was effectively turned off.  
<br>   
<center><table BORDER COLS=4 WIDTH="100%" NOSAVE >  
<tr>  
<td>  
<center> # of senders</center>  
</td>  
<td>  
<center> nominal, no logging</center>  
</td>  
<td>  
<center>nominal, logging</center>  
</td>  
<td>  
<center> no distributor, logging</center>  
</td>  
</tr>  
<tr>  
<td>  
<center>1</center>  
</td>  
<td>  
<center>11</center>  
</td>  
<td>  
<center>11</center>  
</td>  
<td>  
<center>11</center>  
</td>  
</tr>  
<tr>  
<td>  
<center>2</center>  
</td>  
<td>  
<center>22</center>  
</td>  
<td>  
<center>21</center>  
</td>  
<td>  
<center>21</center>  
</td>  
</tr>  
<tr>  
<td>  
<center>3</center>  
</td>  
<td>  
<center>29</center>  
</td>  
<td>  
<center>21</center>  
</td>  
<td>  
<center>21</center>  
</td>  
</tr>  
<tr>  
<td>  
<center>4</center>  
</td>  
<td>  
<center>37</center>  
</td>  
<td>  
<center>21</center>  
</td>  
<td>  
<center>21</center>  
</td>  
</tr>  
<tr>  
<td>  
<center>8</center>  
</td>  
<td>  
<center>43</center>  
</td>  
<td>  
<center>18</center>  
</td>  
<td>  
<center>19</center>  
</td>  
</tr>  
</table></center>  
<p>I consider the uncertainty on the 4 and 8 sender points to be about +/- 1 MB/s.
<!-- Author: Tony -->
<!-- Comment: Fri Mar  3 20;01;02 comment by...mako -->
I believe the logger was writing TRYBOS files for these tests.
<!-- Date: Fri Mar 3 19:58:48 2000 -->
<pre>Status of b0dau32:  
  
01-MAR-2000:  
  - Mounted b0dau32 at the bottom of the rack  
    and b0dau31 on top of it  
  
02-MAR-2000:  
  - Power up b0dau32 for the first time  
  - Configure network ports ef0-ef8  
  - Install license and driver for Emulex LP8000-N1 FC HBA  
  
03-MAR-2000:  
  - Configure DNS and NIS  
  - Mount /cdf/* disks  
  - Disable ef1-ef8 for the moment 
 
</pre>
<!-- Author: mako -->



<!-- Date: Tue Mar 7 10:57:39 2000 -->
<pre>March 5-6 CSL rate testing results 
 
default settings: 
 - XXX data format 
 - all events logged to one stream 
 - 250 kB event size 
 - 50 buffers 
 - events from logger queue go to distributor queue 
 - Unison disk 
 
All units are MB/s. 
First column is number of senders. 
Each column is a different test. 
Non-default settings are noted below. 
 
                no logging   150      log queue 
   def.  trybos  trybos     buffers   -> rec queue 
--------------------------------------------------- 
1  11.3   11.3    11.3       11.3      11.3 
2  20.4   20.2    22.4       21.1      20.1 
3  19.1   19.5    29.1       20.0      20.8 
4  19.2   19.5    37.5       18.4      20.1 
8  16.7   16.7    43.6       16.2      17.3 
 
notes:  
 -- the rate decreases when the number of senders gets large 
 -- the difference between trybos and XXX in this test 
    was about negligible 
 -- the 'no logging' rates agree very well with the 
    rates measured last summer 
 -- changing from 50 buffers to 150 buffers does not 
    have a large effect (for the default settings) 
 -- removing the distributor queue from the loop 
    slightly increases the rate 
 
 
test rate dependence on event size: 
 
                                def. 
    1kB  10 kB  50 kB  100 kB  250 kB  1 MB 
---------------------------------------------- 
1  0.36   3.0    11.0   11.3    11.3   11.3 
2  0.37   3.0    10.3   15.9    20.4   22.3 
3  0.36   3.1    10.2   15.4    19.1   22.3 
4  0.37   3.1     9.9   13.0    19.2   21.7 
8  0.37   3.1     9.8   13.1    16.7   20.7 
 
notes: 
 -- for small event sizes, the rate curve is flat 
 -- rates decrease quickly as event size decreases 
 -- 100 kB events have significantly lower rate 
    than 250 kB events 
 
 
test rate dependence on event size again, but with events 
moved directly from logger queue to receiver 
queue: 
 
                                def. 
    1kB  10 kB  50 kB  100 kB  250 kB 
---------------------------------------------- 
1  0.60   4.5    11.3   11.3    11.3    
2  0.51   3.7    12.3   16.5    20.1    
3  0.51   3.8    11.3   16.8    20.8    
4  0.53   3.7    11.7   14.5    20.1    
8  0.56   3.8    11.1   13.1    17.3    
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Mar 7 11:00:50 2000 -->
<pre> 
March 7 CSL rate testing results 
-------------------------------- 
defaults: 
 - XXX format 
 - all events logged to one stream 
 - 250 kB event size 
 - 50 buffers 
 - Unison disk 
 
All units are MB/s. 
First column is number of senders. 
Each column is a different test. 
Non-default settings are noted below. 
------------------------------------------------ 
   def.   no monitoring    Ciprico disk (72 GB) 
------------------------------------------------ 
1  11.3      11.3             11.3  
2  19.5      19.5             21.7   
3  19.2      19.7             28.6 
4  19.1      18.6             26.6   
8  16.6      17.1             24.6 
------------------------------------------------ 
Turning the monitoring off (in csl_mon.h file) 
makes little difference. The rates are much 
higher when writing to the Ciprico disk instead 
of the Unison disk. 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Mar 9 18:59:54 2000 -->
<pre>b0dau32 status 2 (work on the new CIPRICO disks) 
 
08-MAR-2000 Unpacked the new CIPRICO disks and 
            installed them in the rack 
 
09-MAR-2000 Installed software (Storage Manager)</pre>
<!-- Author: mako -->


<!-- Date: Mon Mar 13 09:36:24 2000 -->
<pre>March 11 CSL rate testing 
------------------------- 
default settings: 
 xxx, logging to one stream, 
 250kB size, 50 buffers, 
 log to Ciprico, no consumer 
---------- 
default: 
1  11.3 MB/s 
2  21.8 
3  29.4 
4  29.2 
8  26.5 
12 26.4 
Comments 
 -- The rates are higher than they were on March 7 
---------- 
one consumer requesting any event (one at a time) 
SA_consumer program used. Consumer running 
on b0dau30. 
1  11.2 
2  18.3 
3  18.3 
4  18.1 
8  18.1 
12 18.1 
Some comments 
 -- almost all buffers were in conssend message 
    queue. bufman was working very hard to remove  
    messages from conssend and put them onto 
    receiver queue. A change can be made to the 
    distributor code to keep only, say, 5 messages 
    for each consumer on the conssend queue. 
 -- This consumer requested one event at a 
    time. This can be changed to N events 
    at a time, reducing the number of handshaking 
    messages. 
 -- This consumer was in a very tight loop, 
    requesting an event as soon as it received 
    the last event. A real consumer would 
    have some processing time before it 
    received the next event. The effect of this 
    will be tested. 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Mar 14 09:26:27 2000 -->
<pre>March 14 CSL rate testing 
------------------------- 
default settings: 
 format XXX output, logging to one stream, 
 250kB size, 50 buffers, 
 log to Ciprico, no consumer 
default: 
1 
2 
3 
4 29.0 MB/s 
8 26.5 
----------------------------- 
with one consumer on b0dau30: 
1 
2 
3 
4 19.4 
8 19.4 
note: almost all messages are in conssend queue, 
rate to consumer was about 5 MB/s according to 
csl monitor. The sharp drop in throughput 
when a consumer is attached was seen in  
previous tests. 
----------------------------- 
Now change bufman parameter from 10 ms to 1 ms 
(i.e. bufman now takes a message from conssend 
queue and puts it onto receiver queue every millisecond): 
1 
2 
3 
4 24.6 
8 26.6 
note: now most messages are in logger queue, rate 
has increased significantly, rate to consumer seems 
to be about the same. 
----------------------------- 
Note: I was pressed for time today so each rate value 
above represents a single measurement rather than 
the more usual average of a few measurements. 
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Mar 15 09:32:25 2000 -->
<pre>March 15 CSL rate testing 
------------------------- 
default settings: 
 format XXX output, logging to one stream, 
 250kB size, 50 buffers, 
 log to Ciprico, no consumer 
 bufman parameter = 0.1ms 
---------------------- 
default: 
1 
2 
3 29.8 MB/s 
4 28.5  
8 24.7 
---------------------- 
One consumer running on b0dau30, requesting 
events one at a time, no sleep time: 
1 11.2 
2 20.5 
3 27.9 
4 27.1 
8 26.7  
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Mar 16 09:18:01 2000 -->
<pre>March 16 CSL rate testing 
------------------------- 
default settings: 
 format XXX output, logging to one stream, 
 250kB size, 50 buffers, 
 log to Ciprico (72 GB), no consumer 
 bufman parameter = 0.1 ms 
---------------------- 
default: 
1 
2 
3 28.7 
4 27.2 
8 23.1 
---------------------- 
Write to two streams (mask = 3). 
Each event goes to two streams. 
1 11.3 
2 18.2 
3 18.3 
4 16.4 
8 15.7 
---------------------- 
Write to two streams (mask = 3). 
Each event goes to two streams. 
Use Unison disk instead of Ciprico. 
1 10.8 
2 10.4 
3 10.5 
4 9.5 
8 9.4</pre>
<!-- Author: Tony -->


<!-- Date: Thu Mar 16 19:28:11 2000 -->
<pre>b0dau32 status: 
 
12-MAR-2000: Created /logger/disk1 and /logger/disk2. Strangely 
             the two disks are not quite of equal size. 
 
14-MAR-2000: Attempted to install ups and products by copying 
             files from b0dau30; ups (at least) did not quite 
             work until Jim Patrick created a symlink on 
             16-MAR-2000... 
 
16-MAR-2000: Finally, managed to install MIPSpro 7.3 compilers; 
             Installed ssh; 
             Increased msgtql system parameter to 1000.</pre>
<!-- Author: mako -->


<!-- Date: Mon Mar 20 08:58:16 2000 -->
<pre>March 19 CSL rate testing 
------------------------- 
default settings: 
 format XXX output, logging to one stream, 
 250 kB event size, 50 buffers, 
 log to Ciprico (72 GB), no consumer 
 bufman parameter = 0.1 ms 
 1.5 MB buffer size 
----------------------------------------------- 
vary the buffer size (default = 1.5 MB): 
---------- 
   256 kB  0.5 MB  1.5 MB    3 MB   5 MB 
   ------  ------  ------    ----   ---- 
1   11.3   11.3    11.3      11.1   10.4 
2   22.2   22.2    21.7      20.5   17.5 
3   29.8   28.5    28.0      24.8   17.8 
4   34.7   29.6    26.3      23.0   17.1 
8   29.1   25.8    25.9      21.6   16.1 
Although the event size is constant at 250 kB, the 
rate decreases significantly as buffer size increases. 
Is this due to the time it takes to attach to the shared 
memory segment? 
----------------------------------------------- 
Look at very large event sizes to make sure 
code doesn't crash. Reduce number of 
buffers to 10 due to limited memory on b0dau31. 
---------- 
# buffers:    10       10        10 
buffer size:  1.5      15 MB     15 MB 
event size:   250 kB   250 kB    15000 kB 
             -------  -------   --------- 
1             11.3     7.3       11.2 
2             21.6     7.8       22.0 
3             26.5     7.9       28.5 
4             24.9     8.1       39.9 
8             24.5     8.2       42.2 
----------------------------------------------- 
Vary number of buffers: 
---------- 
      10     20     50 
1    11.3   11.3   11.3 
2    21.6   21.7   21.7 
3    26.5   28.0   28.0 
4    24.9   25.8   26.3 
8    24.5   25.9   25.9 
</pre>
<!-- Author: Tony -->
<!-- Comment: Mon Mar 20 12;07;45 comment by...mako -->
> Is this due to the time it takes to attach to the shared
> memory segment?

Time to enable CSL_ONE_BIG_SHM in csl_logger_message.c?


<!-- Date: Mon Mar 20 18:31:04 2000 -->
<pre>b0dau32 status: 
 
20-MAR-2000: 
  - Updated tcsh to v6.09 fnal version 
  - Fixed emacs 
  - Installed less</pre>
<!-- Author: mako -->


<!-- Date: Sun Mar 26 17:55:03 2000 -->
<pre>March 26 CSL rate testing 
------------------------- 
default settings: 
 format XXX output, logging to one stream, 
 250 kB event size, 50 buffers, 
 log to Ciprico (72 GB), no consumer 
 bufman parameter = 0.1 ms 
 a separate shared memory segment for each event buffer 
---------------------- 
  1.5 MB buffer size: 
1 11.3 
2 21.9 
3 29.8 
4 29.3 
8 27.2 
---------------------- 
  3 MB buffer size: 
4 22.6 
8 20.8 
---------------------- 
one big shared memory: 
   5 MB    3 MB    1.5 MB  0.5 MB  256 kB  <-- buf. size 
1  11.3    11.3    11.3    11.3    11.3 
2  22.0    22.2    21.8    21.8    22.4 
3  30.6    28.8    28.6    28.8    29.7 
4  40.5    39.8    39.5    38.1    37.6 
8  33.2    36.5    33.9    31.0    34.0 
 
comments: 
In tests of March 19 rate was seen to decrease 
significantly as buffer size increased. Using 
one big shared memory in today's tests, instead 
of a separate shared memory for each buffer, 
I see that rates don't depend strongly on buffer 
size and are much higher for the 4 and 8 sender  
tests compared to the 'one shm segment for each 
buffer' case. 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Apr 6 16:44:50 2000 -->
<pre>(Message sent to Jim and Marjorie last night...) 
 
Earlier this evening, Stephan and I sat together and tried to test 
the connectivity between the B0 and FCC Loggers using "real" hardware 
(b0dau32 and new Ciprico FibreSTORE RAID disk at B0 and fcdfsgi1 at 
FCC connected via single-mode fibre). There was no difficulty re- 
establishing the connection, though it was a bit difficult to mount 
the file system on fcdfsgi1 in read-only mode by hand when b0dau32 
is logging at full rate. (I have a little C program that does this 
and it seems to work just fine, so after I figure out how to update 
/etc/mtab, we can use this tool.) 
 
Here is how the system is set up: 
                                              fcdfsgi1 
                  b0dau31  b0dau32                : 
                     |        |                   : single-mode 
                     |        |                   : 
                [ FibreChannel Hub ]    [ FibreChannel Hub ]  
                        |                       / 
                        |                      / 
                [ Ciprico FibreSTORE RAID disk array ] 
                [   mounted read/write on b0dau32    ] 
                [   mounted read-only on fcdfsgi1    ] 
                [            and later on b0dau31    ] 
 
We first tested how fast the current version of the csl_logger could 
write using the new hardware: 
 
        1. CS/L software running on b0dau32 
        2. Sending events from b0dau32 using loopback device 
        3. No consumers 
        4. Single stream logging 
 
Results: csl_logger in the repository           32 MB/sec 
           + CSL_ONE_BIG_SHM optimisation       44 MB/sec 
 
Initial rate test with fcdfsgi1 reading a file on the same disk as 
the B0 Logger writes to it was however rather disappointing: there 
was no noticeable difference in the logging rate (30 MB/sec or so 
without CSL_ONE_BIG_SHM), but the reader's rate was limited to 2-3 
MB/sec. 
 
After dinner, I spent some time reading the documentation that 
came with the RAID disk and found a way to enable read prefetch, 
so I came back and repeated the test using b0dau31, where I have 
super-user privileges. (Stephan: the trick was to select LUN #0 
on the RAID controller, and various configuration and tuning 
parameters become accessible. They are not available for LUN #1!) 
 
Results: WRITER: csl_logger + CSL_ONE_BIG_SHM   42 MB/sec 
         READER: "cat" on fcdfsgi1              22 MB/sec 
 
There are some minor problems we need to sort out but as far as 
connectivity and data transfer rate between the two Loggers is 
concerned, it looks very promising. 
 
</pre>
<!-- Author: mako -->
<!-- Comment: Tue Apr 11 12;28;13 comment by...mako -->
Errata: '"cat" on fcdfsgi1' -> '"cat" on b0dau31.'


<!-- Date: Fri Apr 14 17:53:38 2000 -->
<pre>b0dau32 update... 
- ef0  : netmask 255.255.255.0; adapted network.local script 
- ef5-8: registered & installed cables; ports enabled</pre>
<!-- Author: mako -->


<!-- Date: Thu Apr 20 15:58:09 2000 -->
<pre>April 19 CSL rate testing 
------------------------- 
Today rate tests were done using the new CSL machine  
(b0dau32) and connecting several consumers. 
------- 
SUMMARY TABLE (average values) (MB/s) 
------- 
     rec    rec+log    rec+log+con 
     ---    -------    ----------- 
1   11.3     11.3        11.3 
2   22.1     22.0        20.8 
3   29.5     29.7        30.7 
4   43.9     36.3        35.8 
8   45.2     33.0        33.0 
12  45.2     33.7        37.7 
------------------------------------- 
notes 
----- 
250 k event size, 3 MB buffer size, 
one big shmem, format XXX, one stream, 
log to /logger/disk1 on b0dau32, 
csl running on b0dau32, 
bufman turned on (the current default mode) 
--------- 
Each row corresponds to a different number of senders,  
from 1 to 12. The rate is the rate at which senders were  
able to push out data, which is effectively the rate at  
which events were logged to disk because every event sent  
is logged to disk. The uncertainty on the 4/8/12 sender  
points for the rec+log and rec+log+con tests is 1-2 MB/s.  
Uncertainty on other points is smaller. 
-------- 
rec = CSL received events only, no logging was done because  
      stream mask was zero 
-------- 
rec+log = stream mask was 1 so events were logged to the new  
      Ciprico disk (/logger/disk1) 
-------- 
rec+log+con = 5 consumers were connected to CSL, running on 
      b0dap14-b0dap18, each consumer received about 1.2 MB/s, 
      each requested a different level3 trigger bit, 
      total rate to consumers was about 6 MB/s regardless of  
      number of senders, each sender scanned through level3byte0  
      setting a different (single) bit for each event. 
-------- 
my comments: 
 -- the performance is healthily above the design goal of 20 MB/s  
    to disk plus at least 5 MB/s to consumers 
 -- rate differences between rec+log and rec+log+con cases are  
    negligibly small, except maybe 12 sender case 
 -- every node other than the 12 I had reserved was running some B 
    analysis program, though in principle any network traffic due  
    to this shouldn't affect the CSL rate tests (?) 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Aug 14 13:44:32 2000 -->
<pre>CSL does not respond to RC or rc_sim messages. 
I finally try to shut it down but the csl processes do not die. 
kill -9 does not work. After talking with Jim P., I reboot b0dau31 which clears all of these processes. 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Aug 14 13:54:37 2000 -->
<pre>Greg cannot connect to CSL using software event builder. 
Using Level3Sender program I can send events to CSL. 
Also, other people are using sevb to send events to CSL. 
So the problem is specific to Greg. We see that his 
version of the sevb is using base key 4200 instead of 4000 
(16384 decimal). When this is changed, he can connect to CSL. 
</pre>
<!-- Author: Tony -->
<!-- Comment: Mon Aug 14 15;00;09 comment by...mako -->
Last Friday, we tried to run CS/L under csl_dev account and
asked Nancy to change the CS/L base key. She did but apparently
her change was lost or so we thought at that time because it
was already reset to 16384 when we tried to undo her change
after we fixed the problem with the CS/L running under csl
account...


<!-- Date: Tue Aug 15 11:16:30 2000 -->
<pre>NOTE: b0dau30 was rebooted, and therefore down for 10-15 minutes. 
The official cdf_daq csl, as well as a private CSL, both running on b0dau31, managed to function during this down-time, and continued to work after b0dau30 was restarted. 
 
What is CSL Dependence on b0dau30? 
1) RTserver runs on b0dau30, so csl_mon_send and csl_driver both rely on b0dau30 to communicate with smartsockets.  These processes were able to  continually (every 10 seconds) wait for RTserver to begin running and then contiue as normal. 
 
2) Although all log files are kept on disks not mounted on b0dau31, the actual CSL executables are on a b0dau30 home area disk. Apparently, the executables were loaded into memory on b0dau31, and the reboot of b0dau30 did not affect them. 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Tue Aug 15 18:09:56 2000 -->
<pre>Called to control room by Greg Feild. 
CSL is not writing events to disk. Logfiles 
show that logger is no longer accepting events 
from the current run number. Apparently an EOR 
was received and the logger timeout expired. 
The timer should have been reset since the CSL 
was still receiving good events. It's not understood 
why the timer expired. 
The EOR was received because one of the processes 
on a particular Level-3 farm PC crashed. Removing 
this PC from the farm and running again was successful. 
</pre>
<!-- Author: Tony, Mako -->


<!-- Date: Wed Aug 16 14:08:50 2000 -->
<pre>Shutdown CSL processes at about 8:00 this morning to update the 
code. When restarting csl, notice that I can't list  
/logger/disk1 or /logger/control disks on b0dau31. 
There is an i/o error. 
I do a umount -k /logger/control but then cannot remount it: 
  root@b0dau31 6# mount /logger/control 
  mount: /dev/dsk/dks8d1s11 on /logger/control: No such device 
  mount: giving up on: /logger/control 
Talk with Jim Patrick. Try to reset scsi bus: 
  scsiha -r /hw/scsi_ctlr/8/bus 
but this gives message 'failed to reset' after one minute. 
Console on third floor of B0 periodically gives message 
like "em0: link failed..." 
Finally reboot b0dau31 which seems to 'fix' problem. 
Disks (/logger/disk1, /logger/control) are mounted again. 
CSL processes finally restarted at 8:47. 
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Aug 16 14:33:09 2000 -->
<pre> 
April CSL notes 
--------------- 
thu, apr 20 
Date: Thu, 20 Apr 2000 22:23:42 -0500 
From: Jim Patrick <patrick@fnal.gov> 
Subject: ethernet througput 
To: vaiciuli@fnal.gov, mako@fnal.gov 
Cc: bourov@fnal.gov 
 
In case Sergei didn't get back to you late in the day: 
 
It appears that throughput from 
  SGI <-> SGI is fast 
  Linux <-> Linux is fast 
  Linux -> SGI is moderately fast (not as fast as above) 
  SGI -> Linux is slow (< 1 MB/sec) 
 
Also, there appear to be issues with b0dap21-28 
which use different ethernet cards than b0dap11-20. 
 
There is a simplified version of ttcp (stripped 
down so as to be easily ported to vxworks) that 
is obtainable by  
setup ttcp 
on either SGI or Linux. 
To run it, just type ttcp, 
answer r for receive, t for transmit 
and if transmitting, give the IP address (not node name) 
and the number of buffers to send (~1000 is a good number). 
This does not appear to interoperate with the SGI ttcp. 
It has a fixed packet size of 8192 bytes. 
--------------------------------------------- 
thu, apr 20 
 -- measure rate sending data from b0dau32 to b0dau30: 
    b0dau30> /usr/etc/ttcp -r > /dev/null 
    b0dau32> cat A20000420101544.RAW | /usr/etc/ttcp -f M -t -v b0dau30 
    output:  
     ttcp-t: 1073987120 bytes in 91.43 real seconds = 11.20 MB/sec +++ 
     ttcp-t: 1073987120 bytes in 35.64 CPU seconds = 28.73 MB/cpu sec 
     ttcp-t: 174087 I/O calls, msec/call = 0.54, calls/sec = 1904.08 
     ttcp-t: 0.6user 34.9sys 1:31real 38% 32maxrss 0+0pf 188352+414csw 
--------------------------------------------- 
fri, apr 21 
Date: Fri, 21 Apr 2000 12:08:32 -0500 
From: Jim Patrick <patrick@fnal.gov> 
To: bourov@fnal.gov, mako@fnal.gov, vaiciuli@fnal.gov 
Subject: more on ethernet througput 
 
It looks like the transmit throughput from SGI to the PCs is 
slow for PCs running RedHat 6.1.1. However, we still have a few 
running RedHat 5.2, and the throughput to those is fast. 
For the PCs, both where the throughput is fast and slow, 
there does not seem to be a large dependence on ethernet 
card type (we have 3 types) and even tried two drivers for 
one of the cards. 
In the offline production farms they have an O2000 as an 
I/O node to the farm of PCs, you might contact them and see 
if they see anything like this as well. 
 
                        Jim P. 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Aug 17 09:25:50 2000 -->
<pre>Restart CSL after updating all code. 
Latest changes include sending messages to 
error logger. 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Aug 17 16:54:59 2000 -->
<pre>CSL rate tests of Tuesday August 15: 
1) Using Level3Sender on b0dap17 to send events to CSL 
running on b0dau32, I get throughput of only 2.5 MB/s. 
2) Run Level3Sender on b0dau32 and CSL on b0dau32. Sending 
rate is about 10-12 MB/s. Run 5 consumers (b0dap14,b0dap15, 
b0dap15,b0dap17,b0dap18) and get a total rate to consumers 
of about 7.5 MB/s (about 1.5 MB/s to each consumer). 
8 consumer test (14,14,15,15,17,17,18,18): total rate to 
consumers = 9 MB/s. Note: receiver queue was always empty. 
Each consumer was requesting every event so distributor had 
to make 7 copies of each event. 
3) max rate to a single consumer, running on a PC in control 
room, seems to be about 2.5 MB/s. 
 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Fri Aug 18 14:42:42 2000 -->
<pre>Rc was unable to connect to CSL. We believe the current 
CSL smartsockets code did not appropriately handle the 
change of rtserver from b0dau30 to b0dau31. 
 
CSL display wasn't updating but this was fixed by  
killing and then starting csl_mon_send.  
As for the problem of communicating with RC, we needed 
to restart the driver. Currently, that means we have 
to kill all csl processes. 
 
</pre>
<!-- Author: Mako and tony -->


<!-- Date: Mon Aug 21 10:34:04 2000 -->
<pre>Try ttcp test sending from b0dau32, receiving on b0dap15 
to test how fast data can be sent: get only 0.8 MB/s 
This is strange since I get about 2.5 
MB/s when sending from CSL on b0dau32 to a consumer on b0dap15. 
These numbers are not consistent. 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Aug 21 11:04:57 2000 -->
<pre> 
June CSL notes 
-------------- 
 
Mon, June 5 
 -- talk to Stephan about his copy job: 
    any files in /b0dau31/disk1/csl/xx on b0dau32  
    (i.e. the 250 GB Unison disk) 
    of the type A*phys or a*phys or A*cosm or a*cosm are 
    copied every few minutes to fcdfsgi2:/cdf/data08 area. 
    The files in b0 are then deleted. 
----------------------------------------------------- 
tue, June 6 
Date: Tue, 6 Jun 2000 08:33:06 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: chlebana@fnal.gov 
Subject: recognizing trybos vs XXX file 
Frank, 
 
First the summary: 
 
second word 
byte pattern   data format 
------------   ----------- 
0101           TRYBOS, big endian 
0202           TRYBOS, little endian 
0505           XXX,    big endian 
0606           XXX,    little endian 
 
EXAMPLES 
-------- 
 
Here's a hex dump on b0dau31 of the first few lines of  
a CSL output file in TRYBOS format: 
 
od -X B20000605193552.test  
 
0000000 ff7f0000 04020200 00000000 04000000 
0000020 94130000 54433144 01000000 00000000 
0000040 7f020000 03000000 0c000000 0e000000 
 
The 0202 byte pattern in the second word tells me this file 
is TRYBOS,little endian format. You don't need to know about 
the endianness, but it does tell me that the events were 
assembled on a PC rather than an IRIX machine (e.g. if sevb 
was used, it was run on b0dap30 and not b0dau30). 
 
Here's a hex dump on b0dau31 of the first few lines of 
a CSL output file in XXX (what Rob K. calls flatroot) format: 
 
od -X ar01fb43.0193dc2i 
 
0000000 8c000000 04060600 00000000 04000000 
0000020 40000220 00010001 00000000 03000000 
0000040 00000005 00000004 4000006e ffffffff 
0000060 4c524944 5f53746f 7261626c 6542616e 
 
The 0606 byte pattern in the second word tells me this file  
is XXX, little endian. 
 
Note: all data files going through the hardware event builder 
and Level-3 executable are assembled on a PC (by reformatter) 
and written out in flatroot format (by Level-3 output module), 
so the byte pattern is 0606. 
 
Tony 
-------------------------------------------------- 
mdc2a rate testing on June 22 
------------------------------ 
 
software: Level-3 control code, null Level-3 Filter  
  executable, current CSL code 
hardware: two Level-3 converter nodes, 16 processor nodes, two  
  output nodes, CSL on b0dau32, 400 GB RAID 
 
starting time        = 10:17 am 
starting run         = 90022 
starting run section = 0x21020 = 135200 decimal 
stop time            = 11:20 am 
last run section     = 0x2105e = 135262 
 
write 98 files to disk <--> about 90 GB data 
stream A approximately 50% size of stream B output 
  (B passes all events) 
 
note: 
flat2seq conversion rate << csl rate to disk! 
 
After end of run, continue converting flat root files to  
sequential root. 
 
top 
PGRP USERNAME PRI  SIZE   RES STATE    TIME WCPU% CPU% COMMAND 
75530  75212 csl   20  154M  152M sleep   16:02  9.8 34.00 csl_log 
75520  75225 csl   20  154M  152M sleep   16:23  9.3 31.73 csl_rec 
75527  75225 csl   20  154M  152M run/0   16:02  9.0 28.97 csl_rec 
76792  73507 csl   20   14M 8960K run/3    0:00  5.6 16.33 Edm_Fla 
 
summary of 1 hour test today: 
The level-3 farm can output a maximum of 16 MB/s in the current 
hardware test configuration. 
All file information seems to have been put into DFC correctly 
during the run. The flat2seq conversion worked properly but 
at a rate much less than the CSL output rate (not a stable 
situation). Files were not moved to tape while the CSL 
wrote to disk because the rate would have been too low. 
Instead, the movement of files to tape began after the CSL 
stopped writing to disk. 
 
---------------------------------------------------------------- 
studies of how fast the Edm_FlatRoot2SeqRoot utility runs: 
 
disks used in study = b0dau32:/logger/disk2   (400 GB Ciprico) 
                      b0dau32:/logger/control ( 72 GB Ciprico) 
 
--- 
TEST 1 (same configuration as the mdc2 rate test done today) 
measure conversion rate while two Level-3 output nodes send to CSL, 
CSL writes stream A at 6 MB/s, stream B at 16 MB/s 
 -- csl writes to /logger/disk2 
 -- converter reads from /logger/disk2 
 -- converter writes to  /logger/disk2 
time to convert 1 GB = 3 minutes, 10 seconds  --> 6.2 MB/s 
--- 
TEST 2 
measure conversion rate with no events being sent to CSL: 
 -- converter reads from /logger/disk2 
 -- converter writes to  /logger/disk2 
time to convert 1 GB = 55 seconds --> 18 MB/s 
--- 
TEST 3 
measure conversion rate while two Level-3 output nodes send to CSL, 
CSL writes stream A at 6 MB/s, stream B at about 15 MB/s 
 -- csl writes to /logger/control 
 -- converter reads from /logger/control 
 -- converter writes to  /logger/control 
time to convert 687 GB = 80 seconds --> 8.6 MB/s 
--- 
TEST 4 
measure conversion rate while two Level-3 output nodes send to CSL, 
CSL writes stream A at 6 MB/s, stream B at about 15 MB/s 
 -- csl writes to /logger/control 
 -- converter reads from /logger/control 
 -- converter writes to  /logger/disk2 
time to convert 738 GB = 75 seconds --> 9.8 MB/s 
The question is what is really limiting the conversion rate? 
 
Tony 
------------------------------------------------- 
Date: Thu, 29 Jun 2000 22:04:37 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: Makoto Shimojima <mako@fnal.gov>, Kevin McFarland <ksmcf@pas.rochester.edu>, 
     Kirsten Tollefson <kirsten@b0urpc03.fnal.gov>, murat@murat.fnal.gov, 
     ratnikov@fnal.gov, lammel@fnal.gov, Ben Kilminster <bjk@fnal.gov>, 
     Jeff Tseng <jtseng@fnal.gov> 
Subject: mdc2 rate test of June 29 
 
 
Some notes on the rate test from CSL point of view (used latest 
version of all CSL code). 
 
mdc2 rate test, june 29 
----------------------- 
 
start time           = 9:15 am 
starting run         = 15000 hex = 86016 
starting run section = 15000 hex = 86016 
using 1 output node  = b0l3u03 
rate to CSL          = about 8 MB/s 
rate to disk: stream A = 3.5 MB/s, stream B = 8 MB/s 
total CSL rate to disk = about 11.5 MB/s 
 
9:25 about 25 GB left on /logger/control 
 
10:23am  44 GB on disk (converted) in 68 minutes 
  --> about 11 MB/s average rate to disk 
 
10:25am  about 23.5 Gb left on /logger/control disk. 
 --> converter is keeping up with CSL output 
 
9:15 start writing to disk1 
~11:15 switch to disk 2 
13:50 switch to disk 1 (had to wait about 25 minutes for lock) 
15:10 switch to disk 2 
--- 
notice one of the disks in the 72 GB Ciprico died.  
  continue rate test. 
15:15 average rate to DISK is ~11.7 MB/s according 
      to csl monitor. 
 
check at 3:50 pm: 6 hours 35 minutes, ~260 GB --> ~11.2 MB/s 
 
looks like about 1 run section per minute. 
 
18:54 test still going. 1.9 million events to disk. 
19:45 stopped the Jeff's thing. 2,072,146 events written, 
      average rate 11.87713 
      trying to stop the CS/L 
 
CSL received shutdown messages from Level-3 at 19:41. 
last stream a file = ar015271.0003dc2i 
last stream b file = br015272.0002dc2i 
last run section = 15273 hex = 86643, but DFC entry looks strange 
 for this run section: 
   run sec  run#   low evt   high evt 
   86641    86016  15000005  15023986 
   86642    86016  15024009  15047991 
   86643    86016         0  15055285    why zero here? 
CSL wrote 155 stream A files 
(from ar015000.0005dc2i to ar015271.0003dc2i) 
and 314 stream B files 
(from br015000.0002dc2i to br015272.0002dc2i). 
 
At approximately 950 MB per file, CSL wrote a total of (155+314)*0.950 
= ~445 GB in a time of 19:41 - 9:15 = 10 hours and 26 minutes 
--> about 12 MB/s 
 
Handshaking between FCC and B0 when switching disks works. 
Rate was intentionally limited by using only 1 Level-3 output node. 
 
Tony 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Aug 21 11:32:16 2000 -->
<pre>July CSL notes 
--------------- 
mdc2 rate test July 13, 2000 
---------------------------- 
 
start time: 09:25 
start run:  15000 hex 
start RS:   15000 hex 
two L3 output nodes used: b0l3u03, b0l3u04 
rate from each output node to CSL = ~5.2 MB/s 
total rate to CSL      = ~10.5 MB/s 
stream A rate to disk  =   2.7 MB/s 
stream B rate to disk  =  10.5 MB/s 
total CSL rate to disk =  13.4 MB/s 
 
CSL seems to be working properly. Output file conversion 
is working fast enough. DFC script is entering information into  
database. 
 
I see a few (3) messages in logger log file like this: 
[58472983] ERROR--> hostname: b0dau32.fnal.gov 
         prg name: csl_logger (in C) 
         function: csl_logger_writetostream() 
         datetime: Thu Jul 13 09:31:25 2000 
         src file: /usr/people/csl/cs-logger/src/programs/csl_logger.c 
         src line: 1884 
         src info: File already closed for this RS 
 
possibly indicating the CSL received an event much later than  
other events of that run section. Mako, what is the timeout period  
now for this? or what is the exact condition that would cause this? 
 
 
~10am: notice that input rate to CSL has dropped 
to about 3 MB/s. 
 
10:52  
stream A rate to disk  = 0.8 MB/s 
stream B rate to disk  = 2.2 MB/s 
total CSL rate to disk = 3.0 MB/s 
 
Rate from Level-3 farm to CSL has decreased from 10.5 to 3 MB/s. 
Why? [many packets in Level-3 were wandering around unable to find 
the target address]  
Rate test is stopped. 
 
Tony 
--------------------------------------------------- 
mdc2 rate test, july 18 [P. Murat] 
---------------------------------- 
Ivan is not around, so ask Ilya to start L3 farm. Later Ivan confirmed  
that Ilya did everything right. 
 
l3root -b started on b0l3gate2 
 
start time           = 10:45 am 
starting run         = 15000 hex = 86016 
starting run section = 15000 hex = 86016 
using 1 output node  = b0l3u03 
rate to CSL          = about 8 MB/s 
rate to disk: stream A = 3.5 MB/s, stream B = 8 MB/s 
total CSL rate to disk = about 11.5 MB/s 
mistakes: 
        forgotten to start DFC_script 
        then started it w/o redirecting its output to the file 
        after figuring it out, killed DFC_script and restarted it again 
 
12:00pm: rate to disk is sustained (about 11.5 MB/sec) 
12:05    52 files written to /logger/disk1 
         filesets are being created 
 
... lunch break, nobody in the control room, everything works 
 
12:58pm:switch to disk2 has happened already 
 
        the rate to disk is 12.6 MB/sec 
        the input rate is 8 MB/sec 
        stager is setting the read locks in /logger/disk1/stager  
        directory: 7 read locks were set at 12:43-12:44pm.  
        Are we using 7 drives?  -no, it is 7 filesets which have  
        been formed 
 
1:05pm: rate to disk is 10.9 MB/sec 
 
1:35pm: rate to disk ~10.3MB/sec, 409,503 events total logged  
  (average rate 11.6) 
 
1:50pm: rate to disk ~9.9MB/sec (stream A = 1.7 MB/sec, stream B =  
  8.2MB/sec) 
 
2:05pm: read locks on DISK1 have been deleted which means that the 
corresponding filesets have been written to tape 
  12 filesets have been created in the file catalog ->  
  writing 2 filesets / tape  
        takes about 80 min 
 
2:05pm : first request from the Production Farm 
 
2:10pm: Ivan is checking if L3 executables are alive - not all of them are. 
        using 16 2-processor nodes 
 
        B0L3010: one process got stuck after 54:00 minutes (5434 events) 
        - it didn't crash, but the number of processed events is not  
          being incremented 
          time consumed by the running exec - 120:00 minutes  
          (16000 events) 
                 size of the stuck executable - 59,420 KBytes 
                 of the running one           - 54,572 KBytes 
        B0L3012: both executables died,  
                 one of them at 2:09pm, the core file is available,  
                 13209 events  
                 processed another one died w/o core file left  
        B0L3023: both executables died, 
                 one process at 1:12pm, core file is available 
                 another one - at 1:18pm, core file is available 
 
        Ivan also setup 16 windows to monitor L3 filters running on  
        the L3 farm 
 
2:20pm: rate to disk is 9.0 MB/sec (1.0 + 8.0) 
 
2:45pm: rate to disk 11.3 MB/sec (3.3 + 8.0) 
 
2:47pm: second switch , now writing to DISK1 again 
        8 write locks created on DISK2 
        3 files left in /logger/disk2/csl/mdc2 
 
3:00pm: somehow the 1st conversion to ROOT reported 15 GB total writen  
to DISK1 (?) 
 
4:05pm: rate to disk: 10.6 MB/sec(2.3 + 8.3) 
        all remaining L3 execs are alive 
 
4:08: Eric noticed first entries in the DFC produced by the Production Farm 
       fcdfsgi1:/cdf/data01/ 
 
        take one fileset /cdf/data01/_ADC2LA.0000 and see how much data are 
going there: 
 
       10,453,264  5.22 GB 
                               after 2 minutes 
      11,241,232: 5.62 GB 
       ------------------- 
 
        400 GBytes /120 sec = 3GBytes/sec 
 
4:45pm: rate to disk 10.6 MB/sec (2.3 + 8.3) 
 
        fcdfsgi1:/cdf/user/stager/bin/stage_output.csh - the script doing 
output 
 
4:52pm: rate 12.3 MB/sec (4.0+8.3) 
 
5:20pm: rate to disk ~12.3 MBytes/sec 
        CA3026: the 1st fileset created for the output of the Production Farm  
 
  /cdf/data01/CA3026.0: 
  total 10113704 
  drwxr-xr-x    2 stager   cdf              113 Jul 18 17:18 . 
  drwxr-xr-x   27 stager   cdf             4096 Jul 18 17:18 .. 
  -rw-r--r--    1 stager   cdf       1130245803 Jul 18 15:57 aa015000.0006dc2l 
  -rw-r--r--    1 stager   cdf       1031166005 Jul 18 15:42 aa015010.0005dc2l 
  -rw-r--r--    1 stager   cdf       1999988939 Jul 18 16:52 aa015015.000adc2l 
<----- ? 
  -rw-r--r--    1 stager   cdf       1016789349 Jul 18 16:51 aa01501f.0005dc2l 
 
 
5:30pm: noticed that more execs died at L3 farm 
        B0L3010: both executables died, 
                 one process   at 4:54pm, core file is available 
(/log/l3_node_exe_1) 
                 another one - at 5:00pm, core file is available 
(/log/l3_node_exe_0) 
 
        B0L3011: one executable got stuck (is not using CPU any longer)  
                 after 11718 events and 89:50 min (size 57,388 KBytes) 
                 the 2nd one consumed 232:56 min and its size is 64,640 KBytes 
 
5:45pm: 
        for comparison: on B0l3014 both filters are running and they consumed  
        25:53 min each per about 40000 events; the same for B0L3013 
        B0L3009: both filters are running, they consumed about 220:00 min each 
 
        total number of events written: stream A:   345,000 
                                        stream B: 1,023,386 
 
5:55pm: rate to disk 10.0 MBytes/sec(1.7+8.3) 
 
 
6:15pm: rate to disk is 10.0 MBytes/sec(1.7+8.3) 
 
        B0L3021: one executable got stuck (is not using CPU any longer) after 
                 consuming 132:26 min and processing 35235 events  
                 (its size is 51112 KBytes) 
                 the still running one is 60692 KBytes big  
 
6:20pm: noticed that the Production Farm didn't put into catalog any new file 
after 4:47pm. 
 
6:40pm: notified (by email) Miro. 
        rate to disk is 9.0 MBytes/sec (0.7+8.3) 
        events on disk: 1,462,000 total 
 
6:45pm: 
        B0L3011: both executables died 
                 /log/l3_node_exe_0: at 18:24 (core file available) 
                 didn't find the core for the 2nd one 
        B0L309:  both executables died, no core files found 
 
6:50pm: rate to disk 9.0 MBytes/sec(0.7+8.3) 
7:00pm: noticed that consumer server stopped writing the events to disk  
while L3 farm is 
continuing to send them. No diagnostics generated.  
It looks like it happened at least at 6:40pm - the rate and the  
total number of events were the same.  
  terminating the run. 
 
7:10pm: end of the test. 
----------------------------------- 
excerpt from July 19 Harris/Mcfarland mail 
--------- 
   Yesterday the CDF offline system sustained a rate of 12 MB/s for over 
7 hours, successfully completing the first full rate test of Mock Data 
Challenge 2. This rate is sufficient for the commissioning run.  The next 
step in Mock Data Challenge 2, the generation of large data samples for 
the physics groups, will begin shortly. 
  The test started with passing Monte Carlo events through the Level 3 
farm. The events were written to disk at B0 by the consumer server logger, 
then read by the data logger in the Feynman Computing Center and written 
to the tape robot.  The data were then staged from tape to disk, 
reconstructed by the production farm, and written back to the tape robot. 
The calibration database was accessed, the trigger database was used, and 
the data file catalog kept track of the files.  About 300 GB were 
successfully written to the robot. 
   
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Aug 23 11:08:41 2000 -->
<pre>how fast can data be sent from b0dau32 to a PC: 
----------------------------------------------- 
I see that S0_SNDBUF and SO_RCVBUF are 60Kb when csl_receiver 
code creates a socket. I don't know if it is the same for  
the ConsumerInputModule though. 
to get IPERF gui: > setup iperf 
                  > jperf 
Using iperf GUI I send from b0dau32 to b0dap16 and get 
0.8 MB/s using TCP window size of 9Kb or 2Kb. 
I get 11.3 MB/s using TCP window size of 60Kb. 
Throughput is very dependent on window size. 
---- 
documentation on iperf is at 
http://dast.nlanr.net/Projects/Iperf/ 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Aug 28 11:56:39 2000 -->
<pre>Date: Fri, 25 Aug 2000 19:22:26 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: Makoto Shimojima <mako@fnal.gov> 
Cc: Kevin Mcfarland <ksmcf@fnal.gov>, Benjamin Kilminster <bjk@fnal.gov>, 
     Kirsten Tollefson <kirsten@fnal.gov>, Ilya Kravchenko <ikrav@fnal.gov>, 
     Frank Wuerthwein <fkw@fnal.gov>, Jim Patrick <patrick@fnal.gov>, 
     Marjorie Shapiro <mdshapiro@lbl.gov>, Robert Harris <rharris@fnal.gov>, 
     Pasha Murat <murat@fnal.gov>, Ivan Furic <ikfuric@fnal.gov>, 
     Hiroyuki Matsunaga <matunaga@fnal.gov>, ratnikov@fnal.gov, 
     Christoph Paus <paus@fnal.gov>, wicklund@cdfsga.fnal.gov 
Subject: Level-3--CSL testing today 
 
Hi, 
 
In mid-afternoon the event builder/Level-3 became available for 
the front-end/event builder/Level-3/CSL/tape test today. 
A short run showed a problem with the section number in the 
LRIH bank. The reformatter was modified and then the section 
number was as expected (starting at one and incrementing about 
every minute). Then the CSL processes on b0dau31 were shut 
down and restarted on b0dau32, the Run II machine, using 
the code which writes sequential root files directly. 
 
Run 96643 was taken using data from the front end 
crates passing through the hardware event builder and Level-3 
filter executable and then to the CSL and disk. The input 
rate to the CSL was only about 2 MB/sec so this run was stopped 
and additional processors were added in the Level-3 farm, 
making a total of twenty processor nodes and three output nodes. 
Run 96645 went smoothly at about 17 MB/sec input to the CSL 
until the rate decreased sharply to less than 1 MB/sec, about 
ten minutes after the run began. 
No experts were available for debugging. The run was stopped. 
About 5 Gb were written to disk in B0. These events were not 
sent over the fiberlink to FCC. 
 
The test should be repeated when hardware evb/level-3 experts are 
available. 
 
Tony 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Aug 29 17:13:48 2000 -->
<pre> 
Restart CSL. This does not include updated code.  
The receiver display was not showing anything event though 
csl_mon_send was restarted and the shared_memory_dump 
showed two receivers connected. For some reason most buffers 
were in the distributor queue, probably limiting the rate. 
Couldn't understand why looking at log files, etc.</pre>
<!-- Author: Tony -->


<!-- Date: Tue Aug 29 22:42:32 2000 -->
<pre>~9:30 pm: control room called 
No sevb or hevb could connect to CSL. Find that CSL already  
had 16 receiver connections. 48 messages in distributor 
queue. 4 hevb clients are in active state. The other 12 
slots are taken up by sevb and hevb in connected state. 
 
what happened: for some reason messages piled up in distributor 
queue so there were no free buffers in receiver queue. 
I see in bufman log file many messages like the following: 
 
 Number of messages in receiver queue: 0 
 Returned message to original queue: not of type L3_CS_RUN_EVENT. 
 
Then when sevb tried to connect, the connection was accepted 
and the hadshaking was done but then the receiver child did 
a blocking wait for a message from the receiver queue and was stuck. 
So more connections attempts were made until all 16 slots were 
filled. 
 
The main problem is that the distributor queue is being filled 
with messages that are not normal events OR bufman is not properly 
recognizing that they are normal events and it is incorrectly 
putting them back onto the distributor queue because they could 
be saved BOR events... 
 
But maybe we also want to change the receiver child so it does a 
nonblocking wait (with timeout) for a message from receiver queue 
in such a way that the receiver child doesn't hang. 
 
Restarted CSL after diagnosing the problem for awhile and then  
everything was fine. 
 
Tony 
 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Aug 30 14:23:31 2000 -->
<pre>CSL topped accepting connections. Same problem as 
last night. many messages in distributor queue. 
receiver already has 16 connections. 
Bufman says it is returning messages to original queue. 
Put a line of debug code in bufman to say what 
the message type is and restart CSL. 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Aug 31 11:57:51 2000 -->
<pre>CSL monitoring is not working properly. Restart of 
csl_mon_send doesn't help. The receiver display shows 
no connections although there are connections. 
dump_shared_memory shows correctly that there are connections. 
Need to debug this 'offline'.</pre>
<!-- Author: Tony -->


<!-- Date: Fri Sep 1 16:54:57 2000 -->
<pre>The Logger was consuming 100% (per process) of CPUs 
and no events were logged, holding the DAQ operation. 
</pre>
<!-- Author: mako -->
<!-- Comment: Fri Sep  1 16;57;48 comment by...mako -->
Entry in the Shift E-log <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=70&anchor=12:25:57 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a>


<!-- Date: Sat Sep 2 00:44:00 2000 -->
<pre>Root files seem to have 2 GB file size limitation.</pre>
<!-- Author: mako -->
<!-- Comment: Sat Sep  2 00;44;29 comment by...mako -->
See this Shift E-log entry <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=70&anchor=19:33:54 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a>


<!-- Date: Sat Sep 2 10:47:51 2000 -->
<pre>The CS/L was restarted this morning. See <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=71&anchor=10:44:46 target=_top>Shit E-log entry</a> 
for details.</pre>
<!-- Author: mako -->
<!-- Comment: Sat Sep  2 11;11;01 comment by...mako -->
The 17.5 GB log file was moved to /logger/disk1/csl/error/.


<!-- Date: Sun Sep 3 10:03:08 2000 -->
<pre><pre>The CS/L was found to be in a strange state this morning: 
- no messages in the receiver queue; 
- 4 in the distributor queue; 
- 3 taken out by receiver children; 
- the rest in the conssend queue; 
- no consumers (according to bstate monitor); but 
- 3 conssend processes on the system (according to ps). 
 
What happened to the bufman??? 
 
The CS/L was restarted...</pre></pre>
<!-- Author: mako -->


<!-- Date: Sun Sep 3 14:04:21 2000 -->
<pre>The CS/L was restarted again. We need to understand 
and fix <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=72&anchor=14:01:57 target=_top>this distributor queue problem</a> ASAP. </pre>
<!-- Author: mako -->


<!-- Date: Sun Sep 3 16:50:51 2000 -->
<pre> <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=72&anchor=16:49:55 target=_top>Restarted</a>.</pre>
<!-- Author: mako -->


<!-- Date: Mon Sep 4 15:59:04 2000 -->
<pre>The csl_mon_send process was <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=73&anchor=15:51:31 target=_top>restarted</a></pre>
<!-- Author: mako -->


<!-- Date: Wed Sep 6 01:40:04 2000 -->
<pre>b0dau32 was upgraded from IRIX 6.5.7f to 6.5.9f to make 
use of better CXFS support in the new release. Two new 
logger disks (CXFS) are added to the system as well: 
 
  /logger/disk3	- 237 GB  Unison Info  moved from b0dau31 
  /logger/disk4	- 203 GB  DigiData     moved from FCC 
 
b0dau31 still has its own /logger/control disk but no real 
logger disk; /logger/disk1/* is now a symlink to the control 
disk /logger/control/* so the CS/L should still be able to 
run, at least for a short period of time.</pre>
<!-- Author: mako -->
<!-- Comment: Thu Sep  7 01;51;12 comment by...mako -->

setup_csl.csh script was updated a while ago to detect
available logger disks automatically.


<!-- Date: Mon Sep 11 11:59:17 2000 -->
<pre>b0dau31:/logger/control disk (72 GB Ciprico) is unreadable 
    umount hangs. The disk is power cycled, and comes up 
    without any errors on LCD panel. But disk still is not 
    accessible. Reboot b0dau31. 
    See messages in /var/adm/SYSLOG: 
  Sep 10 19:00:30 6A:b0dau31 unix: dksc8d1s11:  
    <6>SCSI driver error: device does not respond to selection 
  Sep 10 19:00:30 1A:b0dau31 unix: ALERT:  
    I/O Error Detected.  Shutting down filesystem:/logger/control 
  Sep 10 19:00:30 1A:b0dau31 unix: ALERT:  
    I/O error in filesystem ("/logger/control")  
    meta-data dev 0x1b6 block 0x21eba25 ("xlog_sync") 
  Sep 10 19:00:03 5A:b0dau31 unix: NOTICE:  
    em00: invalid license key for option emfc 
  Sep 11 11:24:25 5A:b0dau31 unix: NOTICE:  
    em0: expired license key for option emfc 
</pre>
<!-- Author: Tony -->
<!-- Comment: Wed Oct  4 20;58;16 comment by...mako -->
 <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=133&anchor=09:37:54 target=_top>Fixed by Hiro</a>
<!-- Date: Wed Sep 13 23:21:25 2000 -->
<pre>control called. can't send data to CSL. 
receiver queue empty. logger queue empty. dist que has 11 
messages, conssend has 36, two buffers used by receivers, 
1 buffer used by distributor. bufman was repeatedly 
grabbing bor messages from conssend and returning them. 
Use dump_msg_que and see that conssend is filld with BORs 
for partition 0. 
 
Restart CSL so data can be taken again.</pre>
<!-- Author: Tony -->


<!-- Date: Wed Sep 27 22:46:19 2000 -->
<pre>Event Display still making multiple connections to consumer 
server when click on next event.  Also saw some one connect 
from fcdfsgi2 tonight and run the Event Display.</pre>
<!-- Author: Kirsten Tollefson -->


<!-- Date: Fri Sep 29 17:28:03 2000 -->
<pre> 
b0dau32 was rebooted. I accidentally started up a CSL 
on b0dau32 for testing purposes without reducing the 
number of buffers or the size of each buffer (default 150 
x 3 MB each). I believe this started to cause strange problems. 
The control room called claiming there was a problem with the 
CSL. The driver was not running. I shut down the CSL but 
still ~450 MB memory was being used. Several conssend processes 
could not be killed with kill -9. b0dau32 was rebooted. 
after about 1 minute the /logger/disk? areas were visible 
from b0dau32 but not from fcdfsgi1. Stephan was notified and 
he determined fcdfsgi1 needed a reboot which he will do. 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Oct 2 04:34:38 2000 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cs-logger&action=view&page=-1&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cs-logger&action=view&page=-1  width='250' align='top'></td><td>An example of my new Logger Display screenshot</a></td></tr></table>
<!-- Author: mako -->


<!-- Date: Wed Oct 4 20:53:32 2000 -->
<pre> <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=138&anchor=06:21:14 target=_top>Another problem?</a></pre>
<!-- Author: mako -->



<!-- Date: Fri Oct 6 03:40:37 2000 -->
<pre>CSL <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=143&anchor=00:02:03 target=_top>crashed</a>? Tony had to <a 
href=/cgi/elog/elog.pl?nb=2000&action=view&page=144&anchor=02:39:03 
target=_top>reboot</a> b0dau32.</pre>
<!-- Author: mako -->


<!-- Date: Thu Oct 12 08:55:31 2000 -->
<pre>Date: Thu, 12 Oct 2000 01:15:36 -0500 
From: Petar Maksimovic <petar@fnal.gov> 
To: cdfdh_grp@cdfaix.fnal.gov 
Cc: csl@b0dau30.fnal.gov, cdf_silicon_monitor@fnal.gov, patrick@fnal.gov 
Subject: what happened to ar018a1c.0001phys??? 
 
 
Dear Data Handlers, 
 
I have a look-area mystery:  on Monday night, I scanned by hand 
(using vdf) about 650 events from the file ar018a1c.0001phys in  
/cdf/data08/_APHYSR.0000, including some good SVX track candidates 
from collisions with Barrel 4.  A few of these events are near 
and dear to our hearts, but (alas!) it didn't occur to me that I 
won't be able to look at them at a later date. 
 
However, the same file now contains only 61 events!  It is much smaller 
in size, and has a *later* modification date: 
 
fcdfsgi2/cdf/data08/_APHYSR.0000] ls -l ar018a1c.* 
-rw-r--r--    1 stager   cdf          2622501 Oct 10 17:26 ar018a1c.0001phys 
-rw-r--r--    1 stager   cdf       1039864861 Oct  9 15:33 ar018a1c.0009phys 
-rw-r--r--    1 stager   cdf       1039991701 Oct  9 15:40 ar018a1c.0011phys 
-rw-r--r--    1 stager   cdf       1039900369 Oct  9 15:42 ar018a1c.0019phys 
-rw-r--r--    1 stager   cdf       1040107644 Oct  9 15:49 ar018a1c.0021phys 
-rw-r--r--    1 stager   cdf       1040075908 Oct  9 15:51 ar018a1c.0029phys 
-rw-r--r--    1 stager   cdf       1040164228 Oct  9 15:52 ar018a1c.0031phys 
-rw-r--r--    1 stager   cdf       1040272872 Oct  9 16:03 ar018a1c.0039phys 
-rw-r--r--    1 stager   cdf       1040773628 Oct  9 16:07 ar018a1c.0041phys 
-rw-r--r--    1 stager   cdf       1040924796 Oct  9 16:05 ar018a1c.0049phys 
-rw-r--r--    1 stager   cdf        244310625 Oct  9 16:05 ar018a1c.0051phys 
 
So I'm curious as to what happened to ar018a1c.0001phys (including the 
first nice SVX tracks)?  Shall we (the users of this area) expect that 
the files will be disappearing at random? (... Which will have 
ramifications on what we would like to copy elsewhere before you guys 
delete it.)   
 
Also, is there any way I can retrieve this file in it entirety (I'm 
particularly keen on regaining access to the events 622 and 623...) 
 
Thanks a lot, 
 
         Petar 
</pre>
<!-- Author: mako -->


<!-- Date: Sun Oct 15 11:58:36 2000 -->
<pre>A <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=172&anchor=10:33:28 target=_top>problem</a> with the driver. It seems it is receiving a 
signal that we do not handle properly... Not sure who 
is sending it, though.</pre>
<!-- Author: mako -->


<!-- Date: Mon Oct 16 08:27:26 2000 -->
<pre>Last night, around 23:00, Ciprico's second RAID controller 
failed again... To be cold-reset sometime soon.</pre>
<!-- Author: mako -->
<!-- Comment: Tue Oct 17 00;42;05 comment by...mako -->
<a href=/cgi/elog/elog.pl?nb=2000&action=view&page=176&anchor=00:01:12 target=_top>done</a>


<!-- Date: Tue Oct 17 02:32:28 2000 -->
<pre><a href=/cgi/elog/elog.pl?nb=2000&action=view&page=177&anchor=02:30:53 target=_top>/logger/disk4</a> back online.</pre>
<!-- Author: mako -->
<!-- Comment: Tue Oct 17 02;39;24 comment by...mako -->
Somehow the original entry disappeared from the
Shift E-log page, so I have reentered it <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=177&anchor=02:37:41 target=_top>here</a>.


<!-- Date: Tue Oct 24 10:37:24 2000 -->
<pre>

Last night there was a database <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=198&anchor=03:04:06 target=_top>problem</a> and run had
to be started without a valid run number. I am happy
to see that the Logger handled this "properly" and
let events through to disk (on B0 at least).

How the DH group should deal with these files is
another matter. It should not be too difficult to
come up with a special sender which reads events
in from these files, modifies the LRIH bank, and
sends them to the CS/L, I believe. Sounds very
much like what we used for MDC test...

</pre>
<!-- Author: mako -->



<!-- Date: Wed Oct 25 17:46:40 2000 -->
<pre>There was a test today of switching the database instance used 
by the resource manager (from production to integration) which 
had the effect of reusing some run nubmers. 
For example: 
logger/disk1/to_be_deleted/ar018f5d.0069phys etc. 
and 
/logger/disk1/csl/error/ar018f5d.0001cali 
 
Other run numbers were also used twice apparently. Some examples: 
 
 /logger/disk1/to_be_deleted/ar018f5b.0039phys (102235) 
 /logger/disk1/csl/error/ar018f5b.0001cali 
 
 /logger/disk1/csl/error/ar018f58.0001phys  (102232) 
 /logger/disk1/csl/error/ar018f58.0001cali 
 
Rick StDenis says the data taken while the integration instance  
was used (i.e. the calibration runs) should never be used. 
 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Oct 25 17:47:44 2000 -->
<pre>This afternoon a QIE calorimeter calibration was done but no information 
was entered into the database. I was there while they repeated the 
calibration and I saw that the CSL did not receive the calibration 
event, so of course the consumer never received it. The software evb 
rejected the event with a reformatter code error which was not  
understood.  
 
In any case, this was not a CSL problem. 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Oct 26 19:44:13 2000 -->
<pre>Restart CSL to inculded updated code which has 
run summary information sent as merlin messages. 
</pre>
<!-- Author: Tony Ben -->


<!-- Date: Tue Oct 31 15:17:57 2000 -->
<pre>CSL restarted just now and also yesterday because there were 
too many connections. This is due to a problem with the 
run summary code included in the last code update. 
</pre>
<!-- Author: Tony V -->


<!-- Date: Tue Oct 31 15:21:14 2000 -->
<pre>Date: Tue, 31 Oct 2000 15:15:21 -0600 (CST) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: cdf_csl@fnal.gov 
Subject: b0dap11 'calibration CSL' 
 
Hi, 
 
I've started a 'calibration CSL' on b0dap11. In order to create 
this special CSL which blocks instead of throwing events away 
I did the following: 
 
-- turn on CSL_NO_BUFMAN flag in csl_driver.c  (bufman not started) 
-- turn on DONT_LOG_EVENT flag in csl_logger.c (events not put on disk) 
-- turn on CALIB_CSL flag in csl_distributor.c (don't throw events away) 
-- turn on CALIB_CSL flag in csl_conssend.c (increae max msgs per 
consumer) 
-- decrease # buffers from 150 to 8 in csl_logger_message.h (don't use 
   up too much PC memory) 
-- don't define ONE_BIG_SHM in csl_logger_message.c (due to Linux limits) 
-- keep max event size 3 MB in l3_cs_interface.h (may increase to 15 MB in 
future) 
 
I sent 100 events and the consumer received all 100 events. The sender 
rate was limited by the rate at which the consumer took events. 
 
There has been no official calibration with this 'calibration csl' 
yet because there are collisions now and no calibrations have been 
done recently. 
 
The code for the 'calibration csl' is in  
/cdf/people1/csl/cs-logger_calib 
on the online cluster. 
 
Tony 
 
 
</pre>
<!-- Author: Tony  -->


<!-- Date: Wed Nov 1 15:04:42 2000 -->
<pre>Restart CSL to include code which fixes the 'too many 
connections to CSL' problem seen recently. 
</pre>
<!-- Author: Tony  -->


<!-- Date: Mon Nov 20 13:46:35 2000 -->
<pre>Power up CSL disks and b0dau32 after the scheduled power outage finishes.  
Only /logger/disk1 had membership delivered. df -k shows only 
disk1. 
 
/logger/control can't be mounted. 
Stephan manually sets Ciprico 72 GB (/logger/control) disk 
to have AL_SELID = 1 (instead of 2). 
 
Use scsiha -p 3 to probe for all devices on scsi controller 3. 
Use ioconfig -f /hw to assign logical controller numbers to 
all devices.  
Use hinv to see status of system hardware (disks). 
Use /usr/cluster/bin/cmgr -p to reenable disk4 and disk5 which 
Stephan disabled last Sunday.  
 
From hinv -c disk I see 
Integral SCSI controller 1: Version QL1040B (rev. 2), single ended 
Integral SCSI controller 0: Version QL1040B (rev. 2), single ended 
  Disk drive: unit 1 on SCSI controller 0 
  Disk drive: unit 2 on SCSI controller 0 
Integral SCSI controller 3: Version Fibre Channel AIC-1160, revision 2 
  Disk drive: unit 1 on SCSI controller 3 
  Disk drive: unit 2 on SCSI controller 3 (XVM Cluster Disk) 
  Disk drive: unit 3 on SCSI controller 3 (XVM Cluster Disk) 
  Disk drive: unit 11 on SCSI controller 3 (XVM Cluster Disk) 
  Disk drive: unit 12 on SCSI controller 3 (XVM Cluster Disk) 
Integral SCSI controller 4: Version Fibre Channel AIC-1160, revision 2 
  Disk drive: unit 2, lun 1 on SCSI controller -1 
 
The 72 GB Ciprico, /logger/control, is on controller 3, unit 1. 
So I change the /etc/fstab entry for /logger/control to 
/dev/dsk/dks3d1s12  /logger/control  xfs  rw,raw=/dev/rdsk/dks3d1s12  0 0 
 
/logger/disk2 fails to be reenabled. Start up CSL without it. 
</pre>
<!-- Author: Tony  -->


<!-- Date: Mon Nov 20 16:57:50 2000 -->
<pre>Reboot b0dau32 to try to get /logger/disk2 recognized 
(at Stephan's suggestion) but still have the following cmgr 
error: 
 
 Could not enable filesystem (/dev/cxvm/dks3d0l1s0) on (/logger/disk2).  
 Device (/dev/cxvm/dks3d0l1s0) is not a valid block device. 
 Failed to modify cluster 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Dec 5 10:34:00 2000 -->
<pre>All 16 receiver connections are used. No more data can 
be sent to CSL. All messages are in logger queue. 
No error messages in logger log file. Restart CSL.  
 
Also, I notice that b0dau32:/logger/disk2 doesn't appear 
with a 'df'. 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Dec 5 11:21:37 2000 -->
<pre>Mon Dec 4 13:08:51   
 
Went to follow up on a problem that CSL was not responding.  
Found out that NO CSL processes were running on b0dau32.  
driver log file ended after TIMEOUT forever print statement  
indicating it was in a smartsockets receive message loop.  
 
Looks like the problem of CSL death is still there.  
 
 - Ben  
</pre>
<!-- Author: Ben -->
<!-- Comment: Wed Dec  6 15;57;06 comment by...tony -->
<pre>This particular instance of processes dying was apparently due to
a reboot of b0dau32.
</pre>


<!-- Date: Thu Dec 7 14:34:43 2000 -->
<pre>problem #1 
I worked with Luciano (sevb) and Alexei Safonov (CLC consumer) 
to see why CLC cailbrations sometimes have problems. We 
determine that when Alexei makes the rate very high, 
the CSL buffers are often all in the conssend queue and 
eventually the software evb dies. The sevb does a blocking 
send to the CSL, but for some reason the sevb process just dies 
at a high rate, after sending a few hundred events. The CLC 
calibration is a total of 5000 events separated into 10 sub-runs 
of 500 events each with a pause between each sub-run. 
Each sub-run takes about 5 seconds and sends 500 events to 
the CSL. Alexei would like the total calibration to take no more 
than about four minutes. We turned off the filling of debug 
histograms in the CLC consumer and the print statements for 
every event to try to make it faster. 
 
In the testing, we used the calibration CSL on b0dap11 which 
uses only 8 buffers currently. The idea is that the calibration 
CSL will need to be able to handle events as large as 15 MB 
(e.g. showermax calibration) so we are restricted to only a 
handful of buffers when using a PC with 256 MB memory. 
 
When using the CSL on b0dau32, Alexei reports no problem 
with his CLC calibration. The CSL on b0dau32 does not cause 
the software evb to block thanks to bufman, but not all events 
sent to the consumer. 
 
Luciano plans to modify the software evb to send events to 
the CSL using a timeout. If there is a timeout, that event 
will be dropped and an error/warning merlin message will be  
sent. 
 
problem #2 
If the software evb is told by run control to start a new run 
before it has received an end or abort for the last run, it 
will happily make a new connection to the CSL without closing 
the previous connection. This problem has happened while 
doing CLC calibrations. Luciano will modify the software evb 
to check that the previous connection is closed before 
opening a new connection to the CSL. 
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Dec 20 15:13:06 2000 -->
<pre>Control room paged. CSL does not respond to run control commands. 
I see that driver process had died. ~140 messages in rec queue, 
3 in logger, 2 in dist., and 2 in conssend queue. The driver 
log file has the following message: 
 
The CSL believes logger 0 is in state: 0 
about to external_message_send a CSL_LOGGER_OKAY 
 entered external_message_send 
 about to set partition in external_message_send 
 about to MLinit in external_message_send 
 about to MLencodedCreate 
 about to MLencodedSendEncoded 
 
*** FATAL ERROR *** 
T_MALLOC <((msg->variable_data_buf).data)>: 1024 bytes:  
  out of heap: file ipcmsg.c, line 85 
Wall Clock Time: Wed Dec 20 08:01:50 2000 
RTworks Time: 0.000000 
Last RTworks Error Code: 6 
Last C Error Code: 11 
Last OS Error Code: 11 
Last Socket Error Code: 11 
To create a core dump, please generate a QUIT signal 
(usually by typing CTRL-\) now. 
You have four seconds to do this. 
 
Ben sent mail to Luciano about this error, which occurs in 
MLencodedSendEncoded. 
 
I started up the 24 hour sender and it was able to send events 
to the CSL (e.g. only driver process had died). 
The driver did not leave a core file when it died. 
The logger log file had a message I haven't seen before: 
 
********************************************************* 
[20] Child 3012631 terminated (status = 0) 
 
---------------------------------------------------------------- 
ERROR--> hostname: b0dau32.fnal.gov 
         prg name: $Id: csl_logger.c,v 1.110.2.14 2000/11/01 
                   12:50:11 bjk Exp $ (in C++) 
         function: csl_logger_new_partition() 
         datetime: Wed Dec 20 08:01:49 2000 
         src file: /usr/people/csl/cs-logger-kcc/src/programs/ 
                   csl_logger.c 
         src line: 1233 
         src info: [20] Error forking partition 00 
         sys perr: Resource temporarily unavailable 
 
I used csl_cleanup to kill the CSL processes but one receiver 
process would not die. I rebooted b0dau32. Before the reboot, 
I saw the following message scroll by continuously on the third 
floor bodau32 console: 
 
dksc3d11 vol: SCSI driver error: device does not respond to   
 selection 
dksc3d12 vol: SCSI driver error: device does not respond to  
 selection 
 
This message was also filling up the file 
b0dau32:/var/adm/SYSLOG 
I guess this is because Stephan had turned off these two disks. 
 
Reboot went smoothly and CSL was restarted. 
We have seen the driver die before. This time we have some debug 
info which may be helpful in understanding what is causing the 
death. 
</pre>
<!-- Author: Tony -->


<!-- Date: Wed Jan 3 18:31:16 2001 -->
<pre>Restart CSL to use up-to-date code which includes a fix 
in driver for a memory leak. 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Jan 8 15:07:51 2001 -->
<pre>Restart CSL to clean up partition zero after the problem of 
Jan 5 causing wrong data representation errors. 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Jan 15 14:33:52 2001 -->
<pre>This is a TEST entry to check if the URL of this entry 
will be automatically mailed to the CSL group. 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Jan 15 14:51:14 2001 -->
<pre>Another test of email functionality. 
</pre>
<!-- Author: Tony -->


<!-- Date: Sun Jan 28 16:40:40 2001 -->
I noticed that between 3:00 and 5:00 on Sunday morning  
fsr was running on b0dau32. There is a root cronjob that calls  
fsr every Sunday morning at 3:00 in order to "improve the organization  
of mounted filesystems". 
 
b0dau32> grep -i fsr /var/spool/cron/crontabs/* 
root:0  3       *       *       0       if test -x /usr/etc/fsr;  
        then (cd /usr/tmp; /usr/etc/fsr) fi 
 
fsr had the following effects on the CSL monitoring plots: 
 
--Total CPU usage was at about 90% during this time, compared to 
a few percent before and after. Almost all of this CPU was 
in the 'idle waiting for I/O' state. 
 
--The load doubled from 0.8 before 3:00 to 1.6 between 
3:00 and 5:00. 
 
--The free memory decreased from 300 MB before 3:00 to 
about 210 MB between 3:00 and 5:00. After 5:00 the free memory 
on b0dau32 went up to 350 GB, 50 GB higher than before fsr 
ran. 
 
--/logger/disk3 and /logger/control showed ~5 GB increase in used 
disk space for a short period of time between 3:00 and 5:00 
 
 
b0dau32> cat /var/adm/SYSLOG 
 
Jan 28 01:01:05 6F:b0dau32 syslogd: restart 
Jan 28 03:00:05 6B:b0dau32 fsr[1150480]: no rw efs file systems in mtab 
Jan 28 03:00:06 6B:b0dau32 fsr[1058218]: fsr_xfs -m /etc/mtab -t 7200  
  -f /var/tmp/.fsrlast_xfs ... 
Jan 28 03:00:06 6B:b0dau32 fsr[1149197]: /logger/disk3  
  startino=654311972 
Jan 28 03:09:19 6B:b0dau32 fsr[1151169]: / startino=0 
Jan 28 03:09:29 6B:b0dau32 fsr[1150021]: /logger/control startino=0 
Jan 28 03:35:00 6B:b0dau32 fsr[1128143]: /logger/disk1 startino=0 
Jan 28 04:08:19 6B:b0dau32 fsr[1154689]: /logger/disk3 startino=0 
Jan 28 05:02:54 6B:b0dau32 fsr[1154689]: fsr_xfs startpass 5,  
  endpass 6, time 7368 seconds 
 
I document some of the effects of fsr here so we know what happens 
when fsr runs while there is no real CSL activity and we are 
not surprised by it in the future.  
<!-- Author: Tony -->


<!-- Date: Tue Jan 30 14:00:04 2001 -->
Restart CSL using Ben's new monitoring code for history plots.  
<!-- Author: Tony -->


<!-- Date: Mon Feb 12 13:27:56 2001 -->
<pre>Move calibration CSL from b0dap11 to b0dap60. b0dap60 
is a monitorless PC on the third floor of B0 meant for 
CSL use. 256 kB memory, dual processor, 700 MHz clock speed. 
A copy of the latest CSL code was put into 
b0dap60:/home/csl/cs-logger-calib 
The log files also get written to that area so if b0dau30 
(where all home areas are) is down the calibration CSL should 
still be able to run. The calibration CSL has 8 buffers each 
of size 15 MB. 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Mon Feb 12 14:28:53 2001 -->
<pre> 
Set up a Level3Sender process on b0dap60 to send events 24 
hours a day to CSL in partition 14. Put the executable, log 
file, and input data file on b0dap60:/home/csl so a b0dau30 
reboot won't affect this process. 
The 24 hour sender is controlled by commands issued by user csl: 
senderd start 14 
senderd stop 14 
senderd check 14 
</pre>
<!-- Author: Tony -->

<!-- Date: Tue Feb 13 18:49:49 2001 -->
<pre>

Yesterday, I upgraded b0dau31 from IRIX 6.5.7m to 6.5.9f and
added the CXFS software. The upgrade went quite smoothly until
at the very end when flashing PROMs took forever (>1 hour), so
I killed the process (which may not have been the right thing
to do, I agree) and discovered that one of the IO boards had
its PROM header damaged. This is not a fatal error, as the
system can boot fine, but I still do not feel very comfortable
leaving the system like this.

I called SGI to have it fixed today, and that was attempted this
morning; two XIO boards were replaced but the flash command still
fails; we will try to replace the midplane for the GIGAchannel
module tomorrow morning. Hopefully that is all we need...

					mako

PS: I learned 6.5.11 was already out and there were a couple of
    patches for CXFS, so we might consider upgrading to the
    latest version with all the relevant patches before the run
    really starts.

</pre>
<!-- Author: mako -->


<!-- Date: Wed Feb 14 16:30:34 2001 -->
<pre>

The flash problem went away after replacing the midplane
for the GIGAchannel module this afternoon.

				mako

PS: We borrowed IRIX 6.5.10 and MIPSpro compiler 7.3.1.2m CDs
    from Greg Siekas <siekas@sgi.com>. Serguei has them now.
 
</pre>
<!-- Author: mako -->



<!-- Date: Tue Feb 20 15:12:51 2001 -->
<pre>Subject: b0dap60 crash 
 
Noticed that the 24 hour sender process was not working. 
Could not connect to b0dap60. Needed to reboot b0dap60. 
No indication in /var/log/messages about why it crashed, 
but it seems that it crashed at about 15:20 on Friday, 
Feb 16. Jeff (new sys admin) put b0dap60 on the list of 
high priority machines so if it crashes again he will 
be mailed within five minutes.  
Possible that someone hit the power switch on Friday accidentally? 
 
</pre>
<!-- Author: Tony  -->


<!-- Date: Wed Feb 21 18:02:14 2001 -->
<pre>Alexei says there is a problem with calibration CSL on b0dap60. 
I see that there are no msgs in rec or log or cons queue. 
6 are in dist que. Here is dump of dist queue: 
type   buffer type       length strms  format part    run  evtnum section 
--------------------------------------------------------------------------- 
1     L3_CS_Run_Event     652     1       0    0      0       1       1 
1     L3_CS_Run_Event     652     1       0    0      0       2       1 
1     L3_CS_Run_Event     652     1       0    0      0       3       1 
1     L3_CS_Run_Event     652     1       0    0      0       4       1 
1     L3_CS_Run_Event     652     1       0    0      0       5       1 
1     L3_CS_Run_Event     652     1       0    0      0       6       1 
 
Restart calib csl so Alexei can work. 
Don't know why it got into this state. (bufman is off, so it can't help.)  
The format and run number are zero for some reason. 
Calib csl was started with 8 buffers of 15 MB each. 
 
 
 
 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Mar 1 15:29:38 2001 -->
<pre>Subject: b0dap60 crash 
 
Last night at about 18:30 I could no longer ping b0dap60. 
I was copying a large file from b0dap60:/home to b0dau30:/data1 
when b0dap60 crashed. I hit the reset button on b0dap60 to 
reboot it. Jeff S. made a modification so log messages go 
to a virtual terminal so we can see what hapened before 
it died. This crash, similar to the one 9 days ago, left no 
traces in the log files to explain why it crashed. 
Restart 24 hour sender and calib csl. 
</pre>
<!-- Author: Tony  -->


<!-- Date: Mon Mar 5 20:45:27 2001 -->
<pre>restart CSL using binaries resulting from Hiro's recent changes 
to the makefile. Also CSL now accepts up to 10 streams instead 
of only 8. 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Mar 6 14:06:04 2001 -->
<pre>There was a problem such that RC was unable to end runs because 
the RC Error Handler was choking on the CSL's EOR messages. 
 
The problem was figured out with the help of Donatella Torretta, 
who is taking over the Error Handler program from Marco.  
The CSL had updated the number of data streams from 8 to 10.  
The Error handler, however, had the number of streams hard coded  
as 8, and was therefore choking on the extra information.  
 
Donatella is going to read the number of streams from our  
java class: mon_struct_j.class in the future. 
 
</pre>
<!-- Author: Ben -->


<!-- Date: Tue Mar 6 18:01:28 2001 -->
<pre>Restart calibration csl on b0dap60 using all the latest 
code from the head of the online repository (i.e. 10 streams, 
new makefile). 
</pre>
<!-- Author: tony -->


<!-- Date: Wed Mar 7 18:26:04 2001 -->
<pre>The CSL is now running with the new root version (2_26_00g). 
I reposited all code changes necessary to run with the new root. 
Tests were done today using new root L3 exes. 
I used Edm_ObjectLister and AC++Dump on the CSL output files 
to verify that the data is readable. Also I ran an event display 
(development version) and YMon consumers and they worked fine. 
Kirsten has instructed the ACEs to use Level-3 exes that were 
built with the new root version. 
 
I updated the partition 14 24 hour sender so it sends 
events in the new root format. 
</pre>
<!-- Author: tony -->


<!-- Date: Thu Mar 8 14:17:14 2001 -->
<pre>Hi Stephan, 
 
Today we wrote a total of about 18.5 GB in stream A contained in exactly 
21 files. Most files are on disk1, the last run is on disk2. 
The information for these 21 files is in the Data File Catalog. 
Please collect these stream A files onto a tape. 
 
The input rate to the CSL was about 16 MB/sec with event size ~53kb 
and event builder rate about 310 Hz. Four Level-3 output nodes were 
used, two subfarms per node, six PCs per subfarm. 
 
Tony 
</pre>
<!-- Author: tony -->


<!-- Date: Tue Mar 13 15:34:59 2001 -->
<pre><a href='/cgi/elog/elog.pl?nb=2001&action=view&page=114&frame=2&anchor=13:20:41'> 
upgraded b0dau32 OS </a> </pre>
<!-- Author: tony and Hiro -->


<!-- Date: Tue Mar 13 15:38:48 2001 -->
<pre>Two patches from SGI, 4060 and 4147, are applied to b0dau31. 
Also kerberos is installed on the machine. 
We had to manually configure /logger/control disk. 
</pre>
<!-- Author: Tony and Hiro -->


<!-- Date: Tue Mar 13 22:35:32 2001 -->
<pre>Restart CSL using new code to fix a bug which prevented 
long runs to be taken when using multiple output streams. 
Now more files can be opened simultaneously. 
</pre>
<!-- Author: tony -->


<!-- Date: Thu Mar 15 04:30:40 2001 -->
<pre>FROM: CDF e-log 
 
 Thu Mar 15 04:06:13  
 
Run ended due to BUSY CONDITION.  
CSL sent warning BAD EVENT received.  
Inspection of CSL message queue shows that CSL has   
no buffers available in the receiver que for events.  
 
Apparently, the CSL got stuck while trying to handle a bad event.  
 
CSL restarted  as logger process was in no-man's land. 
 
 - Ben :: (run 109304)  
</pre>
<!-- Author: Ben -->


<!-- Date: Fri Mar 16 08:57:36 2001 -->
<pre><a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=121&frame=2&anchor=21:02:37'> 
Restart CSL last night because of bad event problem. </a></pre>
<!-- Author: Tony V. -->


<!-- Date: Tue Mar 20 16:36:39 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cs-logger&action=view&page=-2&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cs-logger&action=view&page=-2  width='300' align='top'></td><td>CSL restarted due to receiver connections full. Uncertain how this state happened.</a></td></tr></table>
<!-- Author: Ben -->


<!-- Date: Thu Mar 22 11:25:07 2001 -->
<pre><a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=141&frame=2&anchor=11:21:33'> 
OS upgrade on b0dau31 </a>  
</pre>
<!-- Author: Hiro M -->


<!-- Date: Mon Apr 9 13:44:56 2001 -->
<pre>Date: Mon, 09 Apr 2001 13:43:19 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: cdf_csl@fnal.gov 
Cc: Stephan Lammel <lammel@fnal.gov> 
Subject: running fsr on b0dau32 
 
Hi, 
 
I modified b0dau32:/var/spool/cron/crontabs/root, the root cron table 
file, so fsr runs at 10:00 on Sundays instead of at 3:00. I also 
made a change so fsr ignores the following disks: 
 
/logger/disk1 
/logger/disk2 
/logger/disk3 
/logger/disk4 
/logger/disk5 
/logger/control 
 
I think this means it will act only on the root filesystem, which doesn't 
take long. If we see performance problems in the future with 
the RAID systems maybe we can run fsr manually on them, when we're not 
taking data, to improve the file layout. 
</pre>
<!-- Author: Tony  -->


<!-- Date: Thu Apr 12 11:28:35 2001 -->
<pre>Date: Thu, 12 Apr 2001 11:25:53 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: cdf_csl@fnal.gov 
Subject: CSL restarted 
 
 
Hi, 
 
Jim Patrick et al. were working on some problems with B0 machines 
and needed to reboot b0dap30 and b0dau30. Then they rebooted all other 
PCs. I remounted on b0dau32 the nfs exported disks 
from b0dau30. I restarted the CSL on b0dau32 and the calibration CSL and 
the 24 hour sender on b0dap60. We cannont use ssh to access b0dap60 
anymore. A kerberized telnet, for example, is OK.  
 
One consequence of all this is that the CSL is now running all the latest 
code, which I think includes some monitoring fixes. It had not been 
restarted since April 2. 
 
Tony 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Apr 17 16:50:08 2001 -->
<pre>Restart CSL as user ace on b0dau32. Now shift crew can 
use cslcom script to start/stop/check the CSL. 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue Apr 17 18:00:38 2001 -->
<pre>Date: Tue, 17 Apr 2001 17:54:40 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: cdf_csl@fnal.gov 
Subject: CSL restarted under new user name 
 
 
Hi, 
 
I restarted the CSL as user 'ace' on b0dau32. 
You can use the cslcom script to start/stop/check  
the CSL now. 
 
example: 
 
ssh -l vaiciuli b0dau32 
su - ace 
cslcom stop 
  sends stop message to CSL 
cslcom start 
  starts all CSL processes 
cslcom check 
  checks that CSL processes are running and that  
  shared memory and message queues exist 
The shiftcrew can execute these commands also: 
ssh -l cdface b0dau32 
su - ace 
cslcom stop 
etc. 
 
I created the 'ace' account on b0dau32. Only CSL  
experts and user cdface can become user ace. 
 
The CSL now writes data files to  
b0dau32:/logger/disk?/ace/?? instead of  
b0dau32:/logger/disk?/csl/??.  
I told Stephan about this change 
and he will soon make the necessary changes on his side. 
The DataFileCatalog script and CSL monitoring code  
needed small modifications.  
 
The binaries which are now running are in the area 
b0dau32:/logger/control/ace/software/bin 
They were copied from  
b0dau32:/usr/people/csl/cs-logger-official 
by sourcing the script  
b0dau32:/logger/control/ace/software/copy_official 
 
The csh/csl_logger_config file was changed to reflect  
the 'csl' to 'ace' switch. The csh/setup_csl.csh file  
was changed so user ace correspondsto the same base_key  
we have been using for user csl. User csl now  
has a different base_key. 
 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Apr 19 10:04:13 2001 -->
<pre>   Wed Apr 18 21:33:49  
torture test comments:  
Don't understand why stream A switched from disk 4 to disk1.  
Restart CSL inbetween runs. Streams c,d,e,f,g,h,i  
switched from disk1 to disk2 earlier than I expected.  
[was due to DH process ignoring write locks] 
Realize that disk3 is not big enough to contain 7*40GB  
which means that disk3 is probably now effectively dead.  
These are some things that must be thought about. 
</pre>
<!-- Author: tony -->


<!-- Date: Thu Apr 19 10:04:55 2001 -->
<pre>executive summary of the april 18 torture test from CSL point of view: 
 
-- ran for about 24 hours. 
--Unfortunately disk write locks were ignored during first half 
  of test. Corrected at 2:30 pm. 
--stream A switched from disk4 to disk1, not understood why. 
--When switching to disk 3, rate decreased to about 15 mB/sec. 
  rate had been close to 20 MB/sec. Disk3, unison, is known 
  to be slower than others, but this seems too low.  
--During first half of test, wrote at about 19 MB/sec average rate. 
--Had one long run, 7 hours! 
--Disk 3 became effectively 'dead' in the evening when it filled 
  up (85%) but there still wasn't enough data in any one stream 
  to trigger a tape job (total disk size = ~250 GB and we were 
  writing 7 streams to it) This would not be a problem for a 
  4 stream torture test, for example. 
-- we were aproaching a dead lock condition in which we would 
   have had to stop taking data, but we managed to make it until 
   midnight before that happened. 
-- I want to rethink how long we keep a file open before we 
   decide that no more events could possibly belong to that 
   run section. 
 
Note: 2 new RAIDs are expected in about 4 weeks. They each have 
700GB available space. Maybe we can remove (or relegate to "overflow" 
atatus) disk1 and disk2, which 
require locking, as well as disk 3, which is slower than the others 
when the new disks arrive. With disk 4 and disk 5 (digidata) and 
the two new disks we will have about 2100 GB, quite a bit more than 
the 1500 we have now spread over 5 RAIDs. 
</pre>
<!-- Author: tony -->


<!-- Date: Sat Apr 21 20:13:52 2001 -->
<pre>CAlibration CSL isn't working. All messages are in logger 
queue. My standalone tests also fail. Debug. Modify code so 
dbInstance and dataType parameter are ignored by calibration 
CSL, which does not need them. 
</pre>
<!-- Author: Tony  -->


<!-- Date: Thu May 10 10:12:21 2001 -->
<pre>Reboot b0dau32 (today) and b0dau31 (yesterday) 
with new IP address, moving to upper part of subnet 236 
for security reasons. FNAL DataCom group will set firewall  
from outside attacks for upper part of the subnet. 
The databaes machines had the same change done. 
</pre>
<!-- Author: Tony -->


<!-- Date: Tue May 15 14:15:01 2001 -->
<pre>Date: Tue, 15 May 2001 14:06:34 -0500 (CDT) 
From: Tony Vaiciulis <vaiciuli@b0urpc07.fnal.gov> 
To: cdf_csl@fnal.gov 
Subject: b0dau32 
 
Hi, 
 
b0dau32 (and '31) were rebooted today after the network 
configuration  
was changed. Machines which are not in the protected 
subnet can no longer communicate with b0dau32 or b0dau31. 
A few exceptions include fcdfsgi2, fcdfora1 (offline database) 
and fcdfora2 (backup for offline database). 
 
The csl on b0dau32, the calibration CSL on b0dap60, the 24 hour 
sender on b0dap60, and the DFC script on b0dau32 have all been 
restarted and seem to be working.vaiciuli.  
Just after the reboot of b0dau32, I could not log in as user 
vaiciuli. Sergei saw the problem and created the file 
/var/yp/binding/b0daq/ypservers 
which contains the string "b0dau30". This fixed the problem 
of b0dau32 not recognizing the ypserver. 
 
Tony 
 
 
</pre>
<!-- Author: Tony -->


<!-- Date: Thu Oct 25 08:44:52 2001 -->
<pre>A few IRIX patches were applied to b0dau31 and b0dau32. 
 
b0dau31 : 4236, 4320, 4345 
b0dau32 : 4236, 4345 
 
Perhaps, we need one more reboot ? 
</pre>
<!-- Author: Hiro Matsunaga -->


<!-- Date: Tue Jan 8 19:06:55 2002 -->
<pre>Here is a summary of the steps that were taken to install 
the two new Digidata RAIDs. 
 
Mount both RAIDs in a rack on third floor and connect 
cables according to operations manual provided by Digidata. 
Remove disk1/2 (Ciprico) from CXFS cluster and from  
b0dau32 side hub to make room for two more connections 
on the hub for the two new RAIDs. 
 
We purchased two complete systems from Digidata Corporation. 
This includes the FibreSabre 2132 RAID controller, the  
enclosure, and the disks. Each RAID was already configured 
to consist of a single tier of 5 disks (4 data + 1 parity) 
plus one spare disk. The process of creating Logical UNit 
zero (LUN 0) and initializing array parity were done. 
The FibreSabre 2100/2300 Controller family Operations 
Manual describes what we would have had to do if we had 
put the system together ourselves instead of buying 
a complete system. 
 
hinv shows the two new disks at unit/drive = 0 and 1 on 
SCSI controller 5: 
 
b0dau32> hinv -c disk 
Integral SCSI controller 5: Version Fibre Channel QL2200 
  Disk drive: unit 0 on SCSI controller 5 (XVM Cluster Disk)    
  Disk drive: unit 1 on SCSI controller 5 (XVM Cluster Disk) 
  Disk drive: unit 3 on SCSI controller 5 (XVM Cluster Disk) 
  Disk drive: unit 5 on SCSI controller 5 
  Disk drive: unit 11 on SCSI controller 5 (XVM Cluster Disk) 
  Disk drive: unit 12 on SCSI controller 5 (XVM Cluster Disk) 
  Disk drive: unit 13 on SCSI controller 5 (XVM Cluster Disk) 
  Disk drive: unit 14 on SCSI controller 5 (XVM Cluster Disk) 
 
where the mapping to the names we use is 
unit  name 
0     disk8 
1     disk9 
3     disk3 
11    disk4 
12    disk5 
13    disk6 
14    disk7 
 
Initialize each new RAID by running 'fx -x' on them. 
This created the default partitions 0, 1, 8, and 10. The 
root (0) and swap (1) partitions were removed using 
repartition/resize in fx. 
 
Show parameters of one of the new RAIDs. 
fx> label/show/all 
----- current drive parameters----- 
Error correction enabled          Enable data transfer on error 
Don't report recovered errors     Do delay for error recovery 
Don't transfer bad blocks         Error retry attempts           1 
Do auto bad block reallocation (read) 
Do auto bad block reallocation (write) 
Drive readahead  enabled          Drive buffered writes  enabled 
Drive disable prefetch   65535    Drive minimum prefetch         0 
Drive maximum prefetch       0    Drive prefetch ceiling         0 
Number of cache segments     1 
Read buffer ratio        0/256    Write buffer ratio         0/256 
Command Tag Queueing disabled 
 
----- current drive geometry----- 
      Tracks/zone = 4358       Sect/track = 1489 
    Alt sect/zone =    0       Interleave =    1        Cylinders = 190516 
 Alt track/volume =    0    Cylinder skew =   84            Heads =    5 
   Alt track/zone =   16       Track skew =   90   Data bytes/sec =  512 
  Rotational rate = 7177 
----- partitions----- 
part  type        blocks            Megabytes   (base+size) 
  8: volhdr        0 + 3072           0 + 1     
 10: volume        0 + 1418395648       0 + 692576 
----- bootinfo----- 
 root partition = 0     swap partition = 1    bootfile = /unix 
----- directory entries----- 
 0: sgilabel   block    2 size     512  1: backvh     block    3 size     512 
----- sgi-info----- 
   serial =                              name = RAID    Controller      2.02 
 
 
Label each disk in xvm. 
 xvm:cluster> label -name logger9 dks5d1vol 
Now disk dks5d1 shows up as a physical volume in xvm. 
 
Create one slice covering the whole usable space of the XVM 
 physical volume logger9: 
 xvm:cluster> slice -all logger9   
This has the effect of creating a volume with name vol/logger9s0 
which I change to vol/logger9: 
xvm:cluster> change name logger9 vol/logger9s0 
 
Check topology of the new RAID: 
xvm:cluster> show -verbose -topology vol/logger9 
vol/logger9                       0 online 
    subvol/logger9/data      1418391552 online 
        slice/logger9s0          1418391552 online 
 
Note that subvolume logger9/data was automatically created. 
 
Create a filesystem on logger9: 
b0dau32 30# mkfs /dev/cxvm/logger9 
 
Add both logger8 and logger9 to CXFS cluster using mount 
points /cdf/b0/disk8 and /cdf/b0/disk9 and enable them. 
(use command /usr/cluster/bin/cmgr -p) 
 
After an fcdfsgi1 reboot, it can see the two new RAIDs. 
</pre>
<!-- Author: Tony -->

<!-- Date: Mon Jan 21 09:47:29 2002 -->
<pre>
my attempt to make an entry from email
</pre>
<!-- Author: vaiciuli -->



<!-- Date: Mon Feb 4 16:51:59 2002 -->
<pre>Here is a description of what was done on b0dau32 in order 
to upgrade the bandwidth to fcdfsgi2.  
------------------------------------------------ 
The goal is to add another FastEthernet line to b0dau32 and use 
it to copy files to fcdfsgi2. The data sent to consumers 
will continue to go through the primary interface. In this 
way the file copy to fcdfsgi2 will use a different path 
(cable) than the data sent to consumers. 
 
Most testing was done on b0dau31 but here I just describe 
the final system on b0dau32.  
 
Useful commands: 
 
netstat -r 
ping -f 
osview 
ifconfig -a 
route get fcdfsgi2 
 
An empty FastEthernet port on b0dau32 was chosen: interface ef2. 
 
interface  description 
---------  ----------- 
1 ef0      primary interface 
 
2 ef1      CXFS private line 
3 ef2      previously unused, now used to copy files to fcdfsgi2 
4 ef3      unused 
5 ef4      unused 
 
6 ef5      Level-3 
7 ef6      Level-3 
8 ef7      Level-3 
9 ef8      Level-3 
 
ef1, ef2, ef3, ef4 are on one quad FastEthernet card 
ef5, ef6, ef7, ef8 are on another quad FastEthernet card 
 
A cable for a FastEthernet connection was put into port ef2 on b0dau32. 
This cable makes a connection from b0dau32 to the online Cisco switch. 
A new IP address was requested, b0dau32a.fnal.gov (131.225.236.183). 
b0dau32 has IP address 131.225.236.182. 
File modifications on b0dau32: 
 
------------------------ 
/etc/hosts 
remove  
 192.168.30.2  b0dau32-ef2     # (unused) 
and add 
 131.225.236.183         b0dau32a.fnal.gov b0dau32a 
 
------------------------ 
/etc/config/netif.options 
Change address of interface ef2  
from 
 if3addr=b0dau32-ef2 
to 
 if3addr=b0dau32a 
 
------------------------ 
/etc/config/ifconfig-3.options 
This file was created. It did not exist before. 
It contains the following line: 
 netmask 255.255.255.128 broadcast 131.225.236.255 
 
------------------------ 
/etc/config/static-route.options.1 
Create this file. It is a copy of static-route.options file. 
 
------------------------ 
/etc/config/static-route.options.3 
Create this file. It contains the following line: 
 $ROUTE $QUIET add -host fcdfsgi2 131.225.236.200 
 
The route command in this file will be executed just after 
interface ef2 is configured (see modification to /etc/init.d/network 
below). 
 
------------------------ 
/etc/init.d/network 
First attempt to use the new interface, ef2, failed because 
home area disks from b0dau30 were not mounted. This is because 
the default route on b0dau32 used interface ef2, instead of 
the primary interface ef0. The "/etc/init.d/network" script was 
changed to fix this. See the following description. 
 
Need to execute static-route.options.1 file to add default 
routes after ef0 has been configured. In this way, the default 
routes will use ef0 as the interface. 
 
Add the lines  
 # Add static routes before any routing daemons get theirs installed 
 if test -s $CONFIG/static-route.options.1; then 
     . $CONFIG/static-route.options.1 
  fi 
 
Need to execute static-route.options.3 file to add fcdfsgi2 
route after ef2 has been configured. In this way, the route to 
fcdfsgi2 will use ef2 as the interface. The lines below guarantee 
that the file static-route.options.$num will be executed just 
AFTER the $num interface is configured.  
 
Also add the lines 
  # 
  # Added for multi-home issue 
  # 
                      if test -s $CONFIG/static-route.options.$num; then 
                          . $CONFIG/static-route.options.$num 
                      fi 
 
Original network script is at /etc/init.d/network_2002_0130 
 
--------------------- 
 
After making changes to the files above, b0dau32 was  
rebooted. The rate limiting program, "slowpipe", was replaced 
in the lookarea copy script with kerberized rcp. Several days 
after this I see an average rate of 6.7 MB/sec for copying large  
files to fcdfsgi2 (averaged over 230 files). This is much 
faster than the limit of ~1.2 MB/sec we were using before 
we added the new cable.  
 
</pre>
<!-- Author: Tony Vaiciulis -->


<!-- Date: Tue Dec 24 19:13:05 2002 -->
<pre>About CS/L problem at the day shift: 
 
When I was paged at Dec 24, aces cannot use "cslcom" command  
to control CS/L on machine(b0dau32). 
Because we cannot see the disk(b0dau32:/logger/control) and  
there were "cslcom" in this disk. 
 
I and joao(DAQ ACE) tried to reboot b0dau32. 
But unfortunately I didn't know root password.  
So we contacted Serguei and he let us know how to reboot without root password. 
We tried to do reboot more than 2 times but it didn't work. 
 
After then I checked fiber and fiber-adapter. 
I swapped fiber and fiber-adapter separately. 
But it also wasn't cause. 
 
And then I and Rob tried to check fiber-channel itseft. 
We just did power cycle it. It worked! 
I started all csl process again. 
 
These same serious CS/L problem happened at Nov 11. 
 
- hirokazu</pre>
<!-- Author: Hirokazu Kobayashi -->


<!-- Date: Tue Jan 14 18:40:39 2003 -->
<pre>b0dau31 was upgraded from 6.5.11f to 6.5.18f. 
It seems to be working good, but I will check more 
before upgrading b0dau32.</pre>
<!-- Author: Hiro Matsunaga -->
<!-- Comment: Wed Feb 26 19;15;14 comment by...mako -->
<table border=0><td><tr><td bgcolor=#e0e030>
<pre>This is year 2003. Perhaps we should open a new file
before we lose track of what year each entry is in...</pre></td></tr></td></table>
<!-- Date: Sun Feb 2 03:06:52 2003 -->
Evening shift today noted an <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=96&frame=2&anchor=17:39:51'> 
error </a> during the lookarea check which appears spurious.
<!-- Author: Kevin McFarland -->


<!-- Date: Wed Feb 26 17:51:06 2003 -->
<pre>New hardware installed by Greg Siekas of SGI: 
 
b0dau32: 
  2 internal SCSI disks, 73 GB each; 
  1 Gigabit Ethernet NIC (copper); and 
  4 dual-port Fibre Channel interface cards with JNI chips. 
 
b0dau31: 
  1 Gigabit Ethernet NIC (copper). 
 
Before putting all 4 FC cards (and we have additional spare), 
we installed one card and tested if it worked at all --- it 
uses JNI chips, which according to Greg are compatible with 
the old Adaptec chip, the one in the old single-port FC card 
from SGI, which caused compatibility problems before --- using 
one of the DigiData disks and in single user mode. It looked 
OK; the disk was recognised properly. 
 
We also replaced the bad fan in the bottom DigiData disk, 
disk4. 
 
These cards are not yet configured.</pre>
<!-- Author: mako -->
<!-- Comment: Wed Feb 26 17;53;51 comment by...mako -->
<pre>Serial numbers:

System SN K0025084 b0dau32
PR-S73-SU SN 3EK2DG37  (73GB disk)
PR-S73-SU SN 3EK2PDXT  (73GB disk)
XT-GIGENET-C-SU SN MPZ504  (Copper GigE card)
XT-FC-2P-M-SU SN MRR278  (2-port XIO FC card)
XT-FC-2P-M-SU SN MRR297  (2-port XIO FC card)
XT-FC-2P-M-SU SN MRR267  (2-port XIO FC card)
XT-FC-2P-M-SU SN MRR298  (2-port XIO FC card)
XT-FC-2P-M-SU SN MRR291  (2-port XIO FC card spare, not installed)

System SN 6900D1E81 b0dau31
XT-GIGENET-C-SU SN MPZ527  (Copper GigE card)</pre>



<!-- Date: Wed Apr 16 00:42:28 2003 -->
Incurring periodic low level CSL deadtime at CSL output rates 
of 18MB/s.  (Note, about 40% of output is in a single stream, 
stream H.)<br> 
 
See <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=316&frame=2&anchor=00:28:03.560'> 
note</a> in shift e-log. <br> 
 
Is it possible to add some monitoring code so that when the CSL is blocking because all the buffers are in the logger that we 
can write periodic snapshots of the buffer distribution as a  
function of logger process number?  That might help to understand 
the role that the uneven distribution of streams causes.
<!-- Author: Kevin -->
<!-- Comment: Fri Apr 18 12;04;21 comment by...mako -->
<pre>PIDs of the loggers can be obtained by simply
running my little "bstate" utility program,
but since we only have one logger process per
RC partition writing events for all streams,
this is not very informative.</pre>
<!-- Date: Thu Apr 17 01:36:04 2003 -->
More comments on the start of run CSL blocking problem in the shift e-log <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=319&frame=2&anchor=01:02:06.755'> 
here</a>.<p> 
Also note crash in CSL monitoring <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=319&frame=2&anchor=00:10:19.488'> 
here </a>. 
<!-- Author: Kevin -->


<!-- Date: Thu Apr 17 20:25:15 2003 -->
CSL code modification: I switched stream e files from disk9 to disk7 to make a more uniform distribution of rate across the CSL disks. I also turned off the copying of 
stream h files to the lookarea to check how this affects the higher than usual deadtimes seen when CSL copies files to the lookarea. Investigation of this morning's long run (xxx633) indicate the 
Level3 output rate decreased precisely when the CSL was copying a file to the look area on fcdfsgi2.
<!-- Author: Tony Vaiciulis -->
<!-- Comment: Fri Apr 18 12;06;11 comment by...mako -->
<pre>I wonder why we are still copying files from b0dau32
at all??? I would once again propose we run the look
area copy jobs on fcdfsgi1, not on b0dau32.

When the copy jobs run, which is typically at the
beginning of a store, the CS/L system needs every
CPU cycle and I/O bandwidth to cope with the high
rate. Meanwhile fcdfsgi1 mostly waits for a file
set worth of files to accumulate on the disk.

Distributing disk access pattern uniformly within
the CS/L is good but we should also try to balance
the I/O rates between the two logger machines.

Remember the original design was that b0dau32 was
to be a write-only machine and fcdfsgi1 a read-only
machine. Mixing the two modes should be avoided.</pre>


<!-- Date: Wed Apr 23 13:51:05 2003 -->
<pre>We changed the hardware configuration of  
the Consumer-Server/Logger system. Now all  
but one of the logger disks are connected  
directly to the new Fibre Channel interface  
cards on b0dau32. The one that is left in  
the hub, which is our overflow disk, has a  
special cable, which is too short to reach  
b0dau32. This should not cause any problems,  
however.</pre>
<!-- Author: mako -->
<!-- Comment: Wed Apr 23 13;53;00 comment by...mako -->
<pre>Disk6 remains unreachable from fcdfsgi1.
From clconf_info, we see disk6 has only
server (b0logger) connected to it and no
clients. Other disks have fcclogger
connected as a client.</pre>

<!-- Comment: Wed Apr 23 15;22;04 comment by...mako -->
<pre>I have run a simple dd test

  % time dd if=/dev/zero of=/cdf/b0/diskN bs=250k count=7500

(where N = 3..9)
on all the logger disks and found
that disk6 was particularly slow:

  disk3  79.545 sec     23 MB/sec
  disk4  34.875 sec     53 MB/sec
  disk5  33.828 sec     54 MB/sec
  disk6 167.635 sec     11 MB/sec
  disk7  30.929 sec     59 MB/sec
  disk8  34.101 sec     54 MB/sec
  disk9  29.342 sec     62 MB/sec

So there really is some hardware
problem on our side as well.

A more thoroughful test with the
DAQ system will be performed later
today when it becomes available.
</pre>

<!-- Date: Wed Apr 23 22:53:19 2003 -->
<pre>

Here is a summary of what Hiro and I did on disk6
this evening.

1. As I wrote in my previous mail, I ran a simple
   dd test to write a file on each of the disks on
   b0dau32 and found disk6 to be way too slow. 10
   MB/sec vs. 60 MB/sec on other disks.

   We already knew disk6 had a problem on the FCC
   side --- that fcdfsgi1 cannot read files from
   disk6 --- so I thought it had a problem on our
   side as well.

2. We replaced the controller with a spare and retried
   the dd test. It was as fast as other disks. The disk
   however was not accessible from fcdfsgi1, so we put
   the original controller back in. The dd test still
   showed it was as good, so we concluded there was
   nothing wrong with the controller after all. Maybe
   the cable was a bit loose?

3. Next we swapped the two FC cables going into disk6,
   labelled 506 and 516, so 506 is now plugged into
   channel 1 of the disk and 516 into channel 0. If
   channel 1 were broken, the disk should now become
   inaccessible on our side. This did not happen. It
   was still the FCC side that had the trouble. So
   the controller looked fine.

4. We then tried to plug the cable 516 into another
   port in the hub. We tried one with the unconnected
   cable as well as the one on the rear without the
   GBIC module. Whichever port we used, it stayed
   unavailable on fcdfsgi1. This meant the hub was
   OK, too.

5. Finally we replaced the cable 516 with a new one
   and nothing changed at all, so the cable was not
   a problem, either.

So, we ran out of ideas... Since we can see other
disks on fcdfsgi1, the fibre that goes to the FCC
and beyond should be OK. Yet, CXFS refuses to mount
disk6 on fcdfsgi1, now generating lots of error
messages continuously there, which makes it difficult
to keep it mounted on b0dau32, too.

It turned out that unmounting the disk altogether
made Hiro's diskinfo process unhappy, so I just
mounted disk6 locally on b0dau32 using the normal
mount command. I also tried this on fcdfsgi1 but
got an error "/dev/cxvm/logger6: Invalid argument",
so there is something wrong with it somewhere.

I have noticed the xvm command returns exactly
the same output on both b0dau32 and fcdfsgi1
including disk6. You can see disk6 on fcdfsgi1
using xvm. So, perhaps the Fibre Channel link
is not that bad.

Here comes the interesting bit:

The strange thing is that when I read some bytes
from the device:

  dd if=/dev/cxvm/logger6  of=X bs=1k count=1

I get different checksum values while they agree
on both machines if I read from other CXFS disks.
What is even more bizarre is that I get different
value on fcdfsgi1 each time I read the same block
from disk6.

What could this possible be???

</pre>
<!-- Author: mako -->


<!-- Date: Thu Apr 24 10:12:12 2003 -->
<pre>
I ran a short CS/L torture test, 161977(?),
with L2&L3 autoaccept, and saw the messages
filling up the logger queue. The event size
was something like 113 KB --- I forgot to
check it for this run, but that was the size
in the previous runs we tried. The overall
throughput was still 23 MB/sec. This is a
single stream run, so it must be the limit
from the ROOT I/O overhead (+ our bookkeeping
overhead to some extent). Since ROOT I/O does
some internal buffering, I do not expect
much improvement in throughput even when
the event is back to the nominal value of
250 KB.

Having a Logger process per stream running
in parallel might help in Run IIa to write
perhaps at 40 MB/sec, but we probably will
not be able to do much better than that.
</pre>
<!-- Author: mako -->


<!-- Date: Thu Apr 24 19:09:34 2003 -->
<pre>

I have discovered that disk6 does not show up
in hinv output on fcdfsgi1, so we are seeing
a ghost in the xvm subsystem and not a real
physical device. Stephan suggests shutting
down everything and reboot, but Tony says he
and Glenn have tried this before.

I will be out of my day shift as soon as the
next SciCo is comfortable alone, and that
would hopefully be right after the 8:00 am
meeting, so I suppose we can do some more
tests on the disk before I leave for Japan
in the early morning on Saturday.

We are expecting beam tomorrow so I am not
sure how much time we will have, though.
</pre>
<!-- Author: mako -->



<!-- Date: Fri Apr 25 10:22:35 2003 -->
<pre>(This did not get through, so I am cut- 
and-pasting from my E-mail copy.) 
 
This morning, I asked Glenn to shutdown 
fcdfsgi1 and I shutdown b0dau32 after I 
saw fcdfsgi1 go down. (I saw a message on 
the console reporting "I/O error" while 
unmounting disk6, which was locally 
mounted on b0dau32.) 
 
(I also realised after I hit the return 
key that I forgot to stop the CS/L 
processes beforehand, so that is why you 
did not receive the CSL STOP mail before 
CSL START.) 
 
I powercycled the disk (only disk6) 
and brought b0dau32 back up. The disk 
came up fine. 
 
I then asked Glenn to boot fcdfsgi1, 
too, and saw all 7 disks show up in 
the hinv output. The cxfs_mount command 
worked just fine from fcdfsgi1, and we 
saw the disk mounted on both machines. 
 
The first 1KB of the logical disk now 
returns the same checksum value on both 
machines, so I believe the problem is 
gone, but we should keep it running for 
a while before we put it back in the 
logger spool. Tony please do this when 
you feel comfortable with the disk. 
</pre>
<!-- Author: mako -->


<!-- Date: Sun May 11 09:18:10 2003 -->
We start and stop DAQMon for a <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=392&frame=2&anchor=09:15:36.552'>bandwidth test</a>.  Tony's conjecture that the Consumer Bandwidth is sink rather than source limited in the absence of DAQMon is verified. 
 
Total rate drops to ~4.5MB/s without DAQMon.
<!-- Author: Kevin, Kaori -->


<!-- Date: Sun May 11 09:32:51 2003 -->
<pre>/disk4 and /disk6 are somewhat full (40% and 60% respectively) with files that aren't being logged to disk.  (Presumably no log runs and error events?)</pre>
<!-- Author: Kevin -->


<!-- Date: Tue May 27 10:08:05 2003 -->
<pre>May 27, 2003 
after power outage /cdf/b0/disk8 comes up with the following message: 
 
"configuration error 
 check disk1A 
 restart to continue" 
 
Restart command on VFD fails. Powering down and then up again gives 
same error. 
Connect the disk to the VT220 terminal using b0dau31 cable plus a spare 
cable. 
Call Digidata at 410-730-6880. 
Use CLI commands to cripple disk 1A and reconstruct onto disk 6A: 
 
RAID&gt; ds 
disk status: A-?23456 
(i.e. disk 1A is bad) 
 
RAID&gt; dc autocripple=on 
RAID&gt; par save   [save the changed autocripple parameter] 
RAID&gt; mode offline 
RAID&gt; restart 
This now shows disk status A-c23456 
and automatically begins reconstructing the contents of disk1A onto 
the hot spare, disk 6A. 
 
We will send disk 1A to seagate for repair/replacement. 
 
 
</pre>
<!-- Author: Tony and Jedong -->


<!-- Date: Thu Jun 5 08:45:40 2003 -->
<pre>Modifications needed to make a point-to-point connection 
between b0dau31 and b0csl01 
--------------------------------------------------------- 
 
Need to choose a subnet on the private network for the b0dau31 
to b0csl01 connection. New CSL PCs had GigE interfaces set  
to be on subnet zero, so subnet zero was chosen (it is  
not used yet for anything else). 
 
Connect crossover cable (borrowed from Sergei) directly from 
GigE port on b0dau31 (eg0) to b0csl01-eth1. 
 
b0dau31:/etc/hosts 
Add following lines 
192.168.0.170    b0dau31-eg0 
192.168.0.171    b0csl01-eth0 
192.168.0.172    b0csl01-eth1 
 
b0csl01:/etc/hosts 
Add following lines 
192.168.0.170    b0dau31-eg0 
192.168.0.171    b0csl01-eth0 
192.168.0.172    b0csl01-eth1 
 
b0dau31:/etc/config/netif.options 
Add these lines 
if7name=eg0 
if7addr=b0dau31-eg0 
 
Create file 
b0dau31:/etc/config/ifconfig-7.options 
netmask 0xffffff00 broadcast 192.168.0.255 
 
Reboot b0dau31. 
 
On b0csl01, 
route add -net 192.168.0.0 network 255.255.255.0 eth1 
 
Some useful commands: 
/usr/etc/ifconfig on b0dau31 
/sbin/ifconfig on b0csl01 
/usr/etc/netstat -r [-i] [-in] 
ping -f b0dau31-eg0 
osview            (to see traffic on b0dau32 interfaces) 
watch -d ifconfig (to see traffic on b0csl01 interfaces) 
route get b0csl01-eth0 
traceroute b0csl01-eth0 
 
Now a ping works in both directions between b0dau31-eg0 
and b0csl01-eth1 (though b0csl01-eth0 doesn't seem to work properly). Telnet currently works only in the b0csl01-&gt;b0dau31 
direction. 
 
</pre>
<!-- Author: Tony -->