<!-- Title: 17:17:43  Sat Jan 20 2001 -->
<!-- Crewchief: Christoph Paus -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Steve Tether -->
<!-- Op3: Ilya Kravchenko -->
<!-- Op4: Andreas Korn -->
<!-- Op5: Ivan K Furic -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
We should keep pages short ....<p> 
<b>Please catch up with new entries 
in previous Log! </b>


<!-- Date: Thu Jan 25 15:57:56 2001 -->
<pre> 
  I have done some investigation on the rate of reformatter  
error messages appearing outside of level 3 farm. The goal 
was to determine if all messages from every processor nodes 
make it out and if prescaling is correct. 
 
  In my test, I configured FE so that every event would 
cause reformatter error. The rate in all test was close to 400Hz. 
 
  The expected behavior is to see errors on the first 10 events 
from each reformatter and then one report per 1000 errors. 
 
  The observed behavior:  
    - reformatter generates error in expected way, they all appear 
  in the log files on individual nodes 
    - some fraction of the messages is lost on the way to the outside 
   world 
 
 Jeff suggested and I verified that messages are lost in level3. 
The cause is most likely to be in the mechanism that we use now, 
that is using orbacus(CORBA) standard utilities that do not guarantee  
the delivery. Thus it is possible to overflow buffers at really high 
rate of errors. 
  The errors are collected on the gateway by l3_mon_client (in C) and  
converted into Merlin using MConverter (in java). Both of these 
programs seem to function a bit too slow to handle all incoming  
messages. l3_mon_client by itself shows less messages at the output 
comparatively to what was generated on the farm. At the output of  
l3_mon_client I observe slightly more messages than at the output  
of l3_mon_client+MConverter. 
 
   To quantify the effect I have done the following tests: 
 
1. With 4 subfarms. Expect to see 160 reformatter messages  
in the first second of the run (each instance will report about  
the first 10 events). I am looking at the output of l3_mon_client 
that has artificial delay inserted per message (this is directly  
on the gateway): 
    default version           41 message out 
    delay with usleep(0)      17 out 
    delay with usleep         10 out    (here the rate did not change 
        for 1mks to 10ms                  when the delay was increased) 
     
2. All default software, run with 14 subfarms. Rate 390Hz. The number 
of reformatters on the farm is 56. I am looking at the error messages that 
actually reach Error Handler.  The expected number in the right-hand column 
is 56. 
   Error     Number of times this count 
   count      is found in Error Handler 
      1               12 
      2                4 
      3                0 
      4                0 
      5                8 
      6                0 
      7                0 
      8                0 
      9                4 
     10                3 
   1000                5 
   2000                4 
   3000                5  
   4000                4  
   5000                5 
   6000                3 
Note that the first 10 error counts come all in one burst, so you see 
zeros there in some lines. 
 
  Conclusions and suggested plan: 
  ------------------------------- 
    At reasonably low rate of errors (say, <10 Hz), all are expected 
to make it to the EH. 
    At high rate, such as in above test, the fraction that does go 
through might be sufficient, especially since every message contain 
indication on the integrated error level for that reformatter. 
    One can also see the rate of errors in L3 Display. 
 
   To increase fraction of messages going through during bursts 
is possible, but would take moderate amount of work. 
   To achieve guaranteed delivery (my guess) would require major 
monitoring changes. 
  
   The proposal is to leave the mechanism as it is, at least for 
now, being aware of its effects. Because the information that does  
go out appears to be  sufficient. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon Jan 29 17:17:30 2001 -->
On b0l3gate2 the internal interface eth0 had timeouts. 
A network restart made it go away. 
All nodes were rebooted earlier, made necessary by NFS 
timeouts due to harddrive and switch work.
<!-- Author: Andreas -->


<!-- Date: Wed Jan 31 13:46:44 2001 -->
After an unanounced power outage, the farm was brought up. 
b0l3gate2 did not show any BIOS bootup. 
Reset lines seemed to work. The Motherboard beep codes 
worked too. So I concluded the Processor as likely culprit. 
Removing the secondary Processor brought the machine 
back to life. Carefully reseating the processor solved it finally.
<!-- Author: Andreas -->


<!-- Date: Thu Feb 1 21:17:13 2001 -->
The following will go into some sort of documentation 
web page, some day. 
 
When switching L3 farm on after power-down, one has to  
do actions in certain sequence to avoid problems and  
wasted time: 
   1) plug in all Ethernet switches (they have no on/off button) 
   2) turn on b0l3pcom1 (requires [1] or will complain about  
                   network). Check if finished with concole. 
   3) turn on b0l3pcom2 (requires [1] and [2], mounts disk from  
                   the first gateway). Check if finished with concole. 
   4) turn on Scanner Manager crate (requires [1] and [2]). Check 
          if finished looking at SM and scramnet lights. 
   5) turn on ATM switch (requires [1] and [2]). Check if finished 
          on ATM text display. 
   6) push "Reset" on all 15 SCPU crates downstares for they have  
      probably lost network or do software reboot if possible 
      (requires [1], [2] and [5]. Make sure that ATM has finished  
       booting!). Check if finished looking at lights on SCPUs and  
       scramnet. 
   7) Turn on converter nodes (requires [1], [2] and [5]. Make sure 
       that ATM is done, as well.) 
   8) Turn on processor and output nodes (requires [1] and [2]).  
<!-- Author: Ilya -->


<!-- Date: Thu Feb 1 22:20:15 2001 -->
Testing log book.  
Plain mode seems to be broken.
<!-- Author: Ilya -->
<!-- Comment: Thu Feb  1 22;20;43 comment by...Ilya -->
<pre>Testing comments.
Plain mode works?</pre>

<!-- Comment: Thu Feb  1 22;28;29 comment by...Ilya -->
<pre>This is a summary of the problems encountered during today's shift,
for which I was asked for asistance. Just an example of a an 
average shift.

 - reformatter reports error from non-existing VRB! All are confused.
     Guess that this is because of XFT crates. Guess was correct.

 - Level3 fails partition transition. This is because Run Control
     sees Abort+Reset acknowledgement while it expects to get
     Partition ackn. from L3.

 - Level3 reports reformatter error: crate ID found twice (crateID
     is actually zero). Question to me: problems with large
     fragments code? (fragments are actually small).
     Is it worth checking for non-zero crateID in the event builder? Today, I blamed this on FE.
 
 - Event Builder was not responding transition for a long time.
     It appears that SM has rebooted itself. Steve was on the third
     floor and says that nobody touched it. So:
        - cause? Scanner Manager code died in a catastrophic way? 
        - apparently, reconnecting sockets are not in this
            version of proxy/sm. I will need to check, where they are.
        - on DAQMON EventBuilder bar turned red during that. Does this
            mean that Heartbeat message was not sent until I restarted
            EvbProxy? (It was not received for >330s!). Why did it
            stop?

 - Event Builder failed on Cold Start transition complaining that
      Level3 did not register in time. My explanation was that 
      this is because of the way how RC handles our acknowledgements
      mistaking ack. for Abort+Reset for ack. for Partition.

 - Event Builder failed on End. "Lost connection to SCPU". Possibly
      because of the problem related to HRR.

 - (after I left) the aces noticed that monitoring is not coming
      out for the run in progress, reported in shift log file.
      I did not get any pages.

Summary:
   1) Two transition failures because of the way RC handles acknowledgements
   2) One Event Builder crash(SM): code crash or hardware glitch
   3) Two runs with reformatter errors that confused aces.
   4) HRR: needs to be studied, L3 monitoring: unknown.

</pre>


<!-- Date: Fri Feb 2 14:36:07 2001 -->
<p> 
Ivan, Philip and I installed 12 new nodes yesterday. 
Processors b0l3031-b0l3042.<br> 
Typical benchmark: 294/447/20 (streams/tiny/disk)<br>  
Here is the procedure:<br></p> 
<pre> 
-record Serial Nr.  
-boot up separate 
-hit <return> to run benchmark 
-check and record benchmark 
-get label 
-insert floppy (need a few) 
-cd /root 
-mount /mnt/floppy 
-run /mnt/floppy/initnode  
-enter and record Node number 
-affix label 
-umount /mnt/floppy 
-/sbin/halt and power down 
-move to final location 
-plug in eth, reboot 
-check with mobile terminal 
-remove configuration from kudzu  
 (you will be prompted) 
-final test: ping node 
<\pre>
<!-- Author: Andreas -->


<!-- Date: Sat Feb 3 16:18:27 2001 -->
</pre><pre> 
I changed the BIOS settings on all 12 machines. 
HD LED's work now. B0l3032 had the HD LED and RESET 
connectors switched, thats corrected as well. 
</pre>
<!-- Author: Andreas -->
<!-- Comment: Sat Feb  3 16;21;50 comment by...Andreas -->
<pre>The propper BIOS location to enable is:
/advanced/pci/scsi/rom scan</pre>


<!-- Date: Thu Feb 8 16:32:32 2001 -->
<pre><pre>All 40 PC's got installed today. 
Thanks to Installers: Frank, Ivan, Konstantin and Sasha and me. 
No major defects found. Up to B0L3080 should be available. 
Only problem was a velcro shortage. 
</pre></pre>
<!-- Author: Andreas -->


<!-- Date: Sun Feb 11 21:04:55 2001 -->
<p>The disk setup for b0l3pcom1 has 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=12&target=_top&anchor=21:01:08'> 
changed.</a> <p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Feb 12 23:59:46 2001 -->
<pre>Additional 48 PC's got installed today.  
Thanks to Installers: Ivan, Konstantin, Sasha, Phillip and me.  
One node (Ser. Nr. 02015155) with slow disk (14Mb/s) found.  
Up to B0L3127 should be available.  
Velcro shortage was resolved by Jeff.  
Two glitches in /etc/hosts (b0l3101-b0l3109 were mapped to b0l3001-b0l3009) and /etc/rc.d/rc.local (missing disk tuning) 
were recognized and fixed. Some reboots were neccessary due to 
stale NFS handles. 
Special Thanks to Mark Knapp and Sergei Bourov for lending equipment.</pre>
<!-- Author: Andreas -->


<!-- Date: Tue Feb 13 00:00:36 2001 -->
<pre>b0l3c04 had a null passord -> fixed!</pre>
<!-- Author: Andreas -->


<!-- Date: Mon Feb 19 19:01:53 2001 -->
<pre>Steve and I improved security on gate1 and gate2. 
b0l3gate1 was completely reinstalled. 
Both machines are kerberized now. You'll need to 
call Yolanda Valedez to get a principal. 
Access to gate1 and gate2 is as b0l3pcom1  
and b0l3pcom2 only! 
Kerberized ssh and sshd are installed. 
The vxworks booting moved to new node b0l3backup.</pre>
<!-- Author: Andreas -->


<!-- Date: Wed Feb 21 20:34:44 2001 -->
<pre> we have succeeded running L3 with 15 subfarms having 8 processor  
nodes each. The transitions take long, but not long enough for RC to time out: 
  Partition: 195s 
  ColdStart: 320s 
  WarmStart: 232s 
  End:        35s 
 
(Since I have measured this as well, here are the numbers for 15 subfarms with  
2 
processors each: 
   Partition: 145s 
   ColdStart: 147s 
   End:        13s 
) 
 
On the negative side, it possible to do only 2 good runs before we are forced to 
reboot EVB. These are memory problems, I will send more details to Steve. 
 
We will start an overnight run and see how much of L3 survives until the  
morning. 
</pre>
<!-- Author: Ilya -->


<!-- Date: Tue Mar 13 12:09:38 2001 -->
<p> 
According to the  <a href="http://linuxperf.nl.linux.org/general/common.html">common problems</a> web page, the error "Unable to load interpreter" that we've been getting on the converter nodes recently is almost certainly due to exceeding the limit on open files, which by default is 1024. On a system with 128 MB of RAM the recommended value is 8192. There is also a limit on the size of the inode cache, which is recommended to be 3-4 times the size of the open file limit.<p> 
The easiest way to set these values for our machines is to put the following lines in rc.local:<p> 
<code> 
echo "8192" &nbsp;> /proc/sys/kernel/file-max<br> 
echo "24576" > /proc/sys/kernel/inode-max 
</code><p> 
The inode cache never shrinks but inode-max limits its size, in this case to about 6 MB. 
<p>  
<!-- Author: Steve Tether -->
<!-- Comment: Tue Mar 13 12;14;24 comment by...ST -->
By the way, the interpreter being referred to is the the one for ELF files, in other words shared libraries. Later versions of the kernel give the name of the interpreter, /lib/ld.so.1.


<!-- Date: Sun Mar 25 00:41:45 2001 -->
<pre>We replaced ATM card on converter c02 earlier in the afternoon  
because we were losing ATM connection on it several times 
in the last few days  </pre>
<!-- Author: Arkadiy/Ilya -->


<!-- Date: Sat Apr 21 13:24:08 2001 -->
<pre>Hello, 
 
Here is a new Level-3 tagset based on 3.15.0pre2.  
It includes Ces and Cpr reconstruction and some new filters  
for the first time.  
Due to the ongoing problems with the track reconstruction,  
there is no tracking in this tagset. 
 
Regards, David. 
 
 
 
 EXE_TAG  = 152 exists : Checksum = 1748031203 ; Filesize = 33300461  
 TCL_TAG  = 202 exists : Checksum = 1982384392 ; Filesize = 4193  
 CALIB_TAG = 90 exists : Checksum = 4039960152 ; Filesize = 352081  
 Inserting new tag-set : L3TAGSET        = 107  
                         (EXE,TCL,CALIB) = (152, 202, 90)  
                         DAQVALID        = 2  
                         DESCRIPTION     = Exercise Ces and Cpr 
                                           reconstruction and filters.  
                                           No tracking.  
</pre>
<!-- Author: Dave Waters -->


<!-- Date: Mon Jun 25 20:45:22 2001 -->
<pre>  Today we had several hours of access, so some work was done on L3. 
 
 1. Boris tried to recover converter c02, no luck 
      - BIOS setting about IRQs is set correctly "disabled" (checked with c05) 
      - pulled out all cards. Put in ATM only, checked connection with atm_write, 
        added scramnet card, checked atm connection, added network card, checked 
        atm connection. In all checks HelloWorld worked. 
      - tried to run: no events, converter complains about late fragments. Also: 
           b0l3c02 1% more /proc/atm/devices  
           Itf Type    ESI/"MAC"addr AAL(TX,err,RX,err,drop) ... 
             0 nicstar 001c40000f48  0 ( 0 0 0 0 0 )  5 ( 0 0 2 0 4 ) 
        I.e., 4 packets are indicated as dropped. 
 
  2. Rebooted converter c05 back to experimental kernel "localports" 
 
  3. EventFunnels are now started on converters 
 
  4. l3_mon_reporter now has rendomized seed for random() that is based on the node 
       name as opposed to timer (that caused problems before). The changed reporter 
       is now in use, but have not been committed to cvs. 
 
  5. Boris has fixed a bug in the error message reported to RC when offline subfarm 
        is included in the run. Committed to cvs. 
 
  6. We are back to smartsockets 5_5 on gateway2. Everything has been recompiled. 
 
  7. Output of all event funnels is sent to log files on all nodes. The change is 
       committed to cvs. 
 
  8. On the gateway, "l3_mon_client | MConverter" are now started so that  
       l3_mon_client output is sent to both MConverter and the log file. 
       This is to help us trace lost messages. The change to  
l3_start_mon_gateway 
       is committed to cvs. 
 
  9. Everything is recompiled on both gateways and distributed on the farm as v2_3_5, 
      which is made current in .env file. 
</pre>
<!-- Author: Ilya -->


<!-- Date: Wed Jun 27 17:23:43 2001 -->
<pre> What we have accomplished today: 
 
  1. Steve+Boris have added new realm FNAL.GOV to both gateways 
 
  2. Arkadiy has moved memory from b0l3016-030 to b0l3001-015. 
     Currently processors 016-030 are powered off. 
 
  3. Arkadiy tested his l3_mon_client with main loop and declared 
     it working. The change is committed to cvs. 
 
  4. Arkadiy testes Sasha's ace and proxy new versions. We did not 
     have time today to make ups products, so we are running from  
     Sasha's home for now. Arkadiy plans to install in ups  
     today/tomorrow. 
 
  5. Boris has installed new "localports" kernel on all converter, 
     this is what is running now. 
 
  6. I have installed 2 changes from Jeff: 
        1) EventFunnels print some infinitesimal debug messages 
             to log files 
        2) Now converter nodes print the circular buffers of l3recv 
             to log files every 5 min. 
 
  7. I have distributed over the farm and made current level3 as v2_3_6  
       and relay as v1_2 (the new relay has only a small change to the 
       EventFunnel). level3 and relay under l3proxy account are now 
       up to date 
 
  8. Recovering converter c02: 
       - Boris has replaced scramnet card. The result was the same, 
           Hello World works, but in real run all events time out. 
       - nothing else happened because hevb/level3 was strongly  
           requested by many people 
 
  Other notes: 
 
  The work is finished for today, a shot is being set up now. Tomorrow 
there will be machine studies, so there might be a chance to get 
the system for a few hours. 
 
  Subfarm status: 
    c01: good converter 8 good processors 
    c02: bad converter  8 good PR 
    c03: good converter 0 good PR 
    c04: good converter 4 good PR 
    c05-c16: nominal 
Aces will not use subfarms 1 and 2. Those are marked offline in  
database. 
 
  Work that was planned but not done: 
    - converter c02 problem 
    - tuning of monitoring from Arkadiy (by now his changes are  
            ready, he says) 
    - changes to config classes (there were conflicts with Jeff) 
 
  People say in the control room that it is very desirable to  
get HEVB back approx. 2 hours before access ends, in order to  
test the detector readout. This might mean, e.g., that if  
access starts at 7am, we should start at that time, too. Although 
today Arkadiy started pretty early, at 8:30. 
</pre>
<!-- Author: Ilya -->


<!-- Date: Thu Jun 28 17:41:05 2001 -->
<pre>I installed the memory in the nodes 016 -030 
All of them will now have one chip of 256 Mb. 
To make those nodes available will have to mark them online and redistribute level3 code. We will do that as soon as we get 2 hour level3 time   </pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Jun 29 00:36:42 2001 -->
<pre>Ok, one thing at a time.  
 
&lt;hr&gt; 
&lt;font color=navy&gt;&lt;h2&gt;Exploring Eventserv Crashes&lt;/h2&gt;&lt;/font&gt; 
&lt;hr&gt; 
&lt;ul&gt; 
&lt;strong&gt; Featured monitoring past/present problems &lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt;Messages from processor nodes (subfarms) get lost on  
transitions.  
&lt;li&gt; Eventserv dies on a gateway or on a subfarm. 
 
&lt;strong&gt;I read some documentation and figured out that &lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt; Eventserv is a kind of smart mediator between "Suppliers" 
 and "Consumers". It can handle several connections simultaneosly 
 from each side. 
&lt;li&gt; Suppliers/Consumers could be Push/Pull 
Push Supplier - active 
Pull Supplier - passive 
Push Consumer - passive 
Pull Consumer - active 
Im our system we use Push Supplier - Push Consumer model.  
&lt;li&gt; Eventserv properties depends on the implementstion  
and have to be tested before use.  
&lt;br&gt;&lt;br&gt; 
&lt;strong&gt;Lost Messages Tests&lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt; Eventserv has buffer, which size could be defined by config file.  
&lt;li&gt; If buffer is full we start loosing messages.  
&lt;li&gt; Switching to a completely passive Supplier/Consumer namely 
 Pull Supplier -  Push consumer did not change anyhting. 
 Eventserv can easily overfill itself.  
&lt;li&gt; Size of the buffer does not strongly affect memory using.  
Change buffer size from 10 to 200 increased mem usage on pcom2 
 from 1.5% to 2.2% (buffer is empty) and 3% (buffer is full)  
for 1K long messages.  
&lt;br&gt;&lt;br&gt; 
&lt;strong&gt;Crash Test&lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt;Eventserv can crash with different messages.  
&lt;ul&gt; 
&lt;li&gt;&lt;i&gt;Segmentation fault&lt;/i&gt; 
&lt;li&gt;&lt;i&gt;Non-compliant application error detected: 
Object was already deleted (object reference count was already zero) 
eventserv: ../../include/OB/Basic.h:643: void OBSimpleRefCount::_OB_decRef():  
Assertion `(int)(ref_ &gt; 0)' failed. 
Abort&lt;/i&gt; 
&lt;li&gt; No messages at all at eventserv. On Supplier:  
&lt;i&gt;system exception `COMM_FAILURE' 
Completed: maybe 
Minor code: 17 (not a GIOP message)&lt;/i&gt; 
&lt;/ul&gt; 
In all cases messages stop going through. 
&lt;li&gt;I wrote "Crush Stand" which allowes to change size of 
sended messages, sending frequency (sleeping time) number 
of senders etc.  
&lt;li&gt;Arkadiy did a thorough test changing the size of the message  
in "Crash Stand" and taking 10 runs for each one. 
&lt;li&gt; Results: 
&lt;ul&gt; 
&lt;li&gt; Eventserv is more stable on the system which is less busy.  
 (b0l3127 compare to pcom2) 
&lt;li&gt; Throughput rate grows with the size of event and saturates 
near 4Mb/sec. (The test was run within one computer, no network 
 connections were used.) 
&lt;li&gt; 0.5 and 1K events crashed eventserver within 50,000 messages  
with ~100% probability on pcom2 and 30% probability on 127   
&lt;li&gt; 5K and 10 K events did not crash eventserv at all on 127, 
 probably because of the smaller rate of messages. (see below)  
&lt;/ul&gt;  
&lt;li&gt; Next time I put a sleep time 10 ms. It could not be made  
less because 10 ms is a quantum of time which could be granted  
to process in Linux system. Eventserv survived 2 times after  
100,000  events on pcom2. 
  
&lt;strong&gt; Farm  Testing&lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt; After Jeff added info from monitoring to logfiles we could  
figure out that eventserv on a gateway died when the birst of  
messages comes from converter nodes 8messages*16converters. 
Together with "sleep" test it indicates that crash happens when  
messages come too frequently. 
 
&lt;strong&gt;Changes &lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt; I increased the eventserv buffer size to 50 messages.  
It solved the problem with lost messages inside the farm.  
&lt;li&gt; We put event Funnels on converters. &lt;i&gt;It supposed&lt;/i&gt; to 
 shield eventserv on a gateway from the bursts of messages. 
&lt;li&gt; I add a random sleep (10ms - 150 ms) in l3_mon _reporter 
before sending event. &lt;i&gt;It supposed &lt;/i&gt;to spread messages over 
 safe period of time and shield subfarm eventserv.  
&lt;li&gt; &lt;i&gt;"It supposed"&lt;/i&gt; in last two buttons means that it does 
 not work the way it should cause we still see eventserv crashes.    
 
&lt;strong&gt;More things to try&lt;/strong&gt; 
&lt;br&gt; 
&lt;li&gt; Move eventserv on a less busy node.  
&lt;li&gt; Split message channel to two (processors and converters).  
Probably do it together with previous measure.  
&lt;li&gt; try to play with random sleep for event funnel. 
&lt;li&gt; Figure out the size of the event which comes from event 
 funnel, which is probably big (100K) and make a new eventserv 
crash test with big messages. 
&lt;li&gt; If big messages can crash eventserv we could try to make  
event funnel smarter and try to either filter messages (unlikely)  
or store messages and spread message flow over longer time.  
&lt;li&gt; remove some &lt;strong&gt;asserts&lt;/strong&gt; from eventserv code  
and hope that it will help. 
</pre>
<!-- Author: Boris -->


<!-- Date: Fri Jun 29 00:56:48 2001 -->
 
 
<hr>  
<font color=navy><h2>Exploring Eventserv Crashes</h2></font>  
<hr>  
<ul>  
<strong> Featured monitoring past/present problems </strong> <br>  
<br>  
<li>Messages from processor nodes (subfarms) get lost on  <br> 
transitions.   
<li> Eventserv dies on a gateway or on a subfarm. <br><br>  
 
<strong>I read some documentation and figured out that </strong>  
<br> <br>  
<li> Eventserv is a kind of smart mediator between "Suppliers" <br> 
 and "Consumers". It can handle several connections simultaneosly<br>  
 from each side.  
<li> Suppliers/Consumers could be Push/Pull <br><br>  
Push Supplier - active <br> 
Pull Supplier - passive <br> 
Push Consumer - passive <br> 
Pull Consumer - active <br><br> 
In our system we use Push Supplier - Push Consumer model.  <br> 
<li> Eventserv properties depends on the implementstion  <br> 
and have to be tested before use.   
<br><br>  
<strong>Lost Messages Tests</strong> <br> 
<br>  
<li> Eventserv has buffer, which size could be defined by config<br> file.   
<li> If buffer is full we start loosing messages.   
<li> Switching to a completely passive Supplier/Consumer namely <br> 
 Pull Supplier -  Push consumer did not change anyhting. <br> 
 Eventserv can easily overfill itself.  <br> 
<li> Size of the buffer does not strongly affect memory using.  <br> 
Change buffer size from 10 to 200 increased mem usage on pcom2 <br> 
 from 1.5% to 2.2% (buffer is empty) and 3% (buffer is full)  <br> 
for 1K long messages.   
<br><br>  
<strong>Crash Test</strong> <br> 
<br>  
<li>Eventserv can crash with different messages.   
<ul>  
<li><i>Segmentation fault</i>  
<li><i>Non-compliant application error detected: <br> 
Object was already deleted (object reference count was already zero) <br> 
eventserv: ../../include/OB/Basic.h:643: void<br> OBSimpleRefCount::_OB_decRef():  <br> 
Assertion `(int)(ref_ > 0)' failed. <br> 
Abort</i>  
<li> No messages at all at eventserv. On Supplier:   
<i>system exception `COMM_FAILURE' <br> 
Completed: maybe <br> 
Minor code: 17 (not a GIOP message)</i>  
</ul>  
In all cases messages stop going through. <br> 
<li>I wrote "Crush Stand" which allowes to change size of <br> 
sended messages, sending frequency (sleeping time) number <br> 
of senders etc.   
<li>Arkadiy did a thorough test changing the size of the message <br>  
in "Crash Stand" and taking 10 runs for each one. <br> 
<li> Results:  
<ul>  
<li> Eventserv is more stable on the system which is less busy. <br>  
 (b0l3127 compare to pcom2)  
<li> Throughput rate grows with the size of event and saturates <br> 
near 4Mb/sec. (The test was run within one computer, no network <br> 
 connections were used.)  
<li> 0.5 and 1K events crashed eventserver within 50,000 messages<br>   
with ~100% probability on pcom2 and 30% probability on 127    
<li> 5K and 10 K events did not crash eventserv at all on 127, <br> 
 probably because of the smaller rate of messages. (see below)   
</ul>   
<li> Next time I put a sleep time 10 ms. It could not be made  <br> 
less because 10 ms is a quantum of time which could be granted  <br> 
to process in Linux system. Eventserv survived 2 times after  <br> 
100,000  events on pcom2. <br>  
<br>  
<strong> Farm  Testing</strong> <br>  
<br>  
<li> After Jeff added info from monitoring to logfiles we could <br>  
figure out that eventserv on a gateway died when the birst of  <br> 
messages comes from converter nodes 8messages*16converters. <br> 
Together with "sleep" test it indicates that crash happens when <br>  
messages come too frequently. <br>  
<br>  
<strong>Changes </strong> <br>  
<br>  
<li> I increased the eventserv buffer size to 50 messages.  <br> 
It solved the problem with lost messages inside the farm.   
<li> We put event Funnels on converters. <i>It supposed</i> to <br> 
 shield eventserv on a gateway from the bursts of messages.  
<li> I add a random sleep (10ms - 150 ms) in l3_mon _reporter <br> 
before sending event. <i>It supposed </i>to spread messages over <br> 
 safe period of time and shield subfarm eventserv.   
<li> <i>"It supposed"</i> in last two buttons means that it does <br> 
 not work the way it should cause we still see eventserv crashes.     
<br> <br>  
<strong>More things to try</strong> <br>  
<br>  
<li> Move eventserv on a less busy node.   
<li> Split message channel to two (processors and converters).  <br> 
Probably do it together with previous measure.   
<li> try to play with random sleep for event funnel.  
<li> Figure out the size of the event which comes from event <br> 
 funnel, which is probably big (100K) and make a new eventserv <br> 
crash test with big messages.  
<li> If big messages can crash eventserv we could try to make  <br> 
event funnel smarter and try to either filter messages (unlikely)<br>   
or store messages and spread message flow over longer time.   
<li> remove some <strong>asserts</strong> from eventserv code  <br> 
and hope that it will help. 
<!-- Author: Boris.  -->


<!-- Date: Fri Jun 29 13:00:12 2001 -->
<a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=438&frame=2&anchor=12:56:51'> 
user link text </a> 
<!-- Author: Arkadiy -->


<!-- Date: Sat Jun 30 12:12:49 2001 -->
<hr>   
<font color=navy><h2>Access report</h2></font>   
<hr>   
<ul> 
<br> 
<strong> Gateways</strong><br><br> 
I installed 256M of new memory into gateways. Old 128M memory chips  <br> 
from pcom1 and pcom2 are of different kind and could not be put together. <br> 
They are now in my office together with 3 more 256M spares. 
<br> 
<br> 
<strong> C02</strong><br> 
<br> 
I tested different combinations of ATM & ethernet cards on c02 and c03.<br> 
I found two non working ATM cards. I left them near Steve's test stand <br> 
on the third floor with appropriate description.  
<br><br> 
After all:   
<br><br> 
  <li> c02 has ATM and ethernet  cards from c03.  
  <li> c03 has ethernet card from c02 and a new ATM card.  
 
<br><br> 
Steve suggested to keep an eye on c02.   
<br> 
 
<br> 
<strong>xntpd</strong><br><br> 
I tried to reproduce xntpd problem by rebooting pcom1 as well as   
random nodes.<br> 
 The problem is not reproduced.  Time is still synchronized  
on all the nodes.<br>  
<br> 
After we restart xntpd, time usually stay synchronized for a period  <br> 
about 2 weeks. <br> 
<br> 
 
<strong>Changes to Level3 Software</strong> 
<br><br> 
I tested my changes. <br><br> 
   <li>creating configuration file for cleanup script by using   
     online flags from database.  
   <li>creating relay  map so that relay and "clean up" cover   
     the same nodes.   
   <li> moved Print() and PrintCleanupConfig() methods  from  
     L3ProxyConfigHWDB to the base class L3ProxyConfig. <br><br> 
Changes work ok in general. I did not commit them because of some  
numerical (?) bug <br>which I had troubles to localize.   
<br> 
<br> 
</ul>  
<!-- Author: Boris.  -->