<!-- Title: 14:49:28  Wed Apr 14 1999 -->
<!-- Crewchief: Ilya Kravchenko -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Steve Tether -->
<!-- Op3: Christoph Paus -->
<!-- Op4: Andreas Korn -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
During installation and comissioning phase the level-3 activities will concentrate on standalone and integration tests



<!-- Date: Wed Apr 14 15:27:16 1999 -->
<pre>======================================================================== 
 
          HISTORY OF EVENT-BUILDER AND LEVEL-3 WEDGE TEST 
                     BEFORE THE E-LOGBOOK 
 
======================================================================== 
 
23 MARCH 1999 
============= 
 
 First time real data were sent from a real detector "the Wedge" to 
 the data acquisition system. The Event-Builder and the level-3 trigger 
 were therefore used. 
 
 Standard setup: Lower Crate on 3rd floor 
                 VRB slot 9 
                 SCPU: b0eb21 
                 SM: b0eb20 and 
                 CV: b0l3c04 (write to disk immediately) 
 
 Event rate was 55 events per 30000 ticks reading four ADMEM boards. 
 Event size was around 1400 words: 
 
                 28000 bytes/sec  = 5*4*1400 bytes/sec 
 
 Started a long time test around 8 o'clock in the evening. 
 
24 MARCH 1999 
============= 
 
 This test finished itself at 02:26 in the morning. (123k events) 
 
 Symptoms: 
 
 GUIreadout was giving VRB busy (restart did not help) this means 
 nobody picked up the events in the VRB? 
 
 login to SM on b0eb10 did not count this means the Scanner Manager 
 crashed. 
 
 UCI and SM communications seemed broken .. Steve says this is normal 
 after startup. 
 
 level3 (converter) was sitting and waiting for events this means it 
 looked Okay. 
 
 Started a long time test around 8 o'clock in the evening. 
 
25 MARCH 1999 
============= 
 
 This test finished itself at 02:49 in the morning (120k events) 
 
 It was not possible to recover 
 
 Symptoms: 
 
 GUIreadout was giving VRB busy (restart did not help) this means 
 nobody picked up the events in the VRB? 
 
 login to SM on b0eb10: SM did count this time 
 
 event rate jumped suddenly from 55 events/30k ticks to more than 2000 
 events/30k ticks. Asked Jim and Jeff for help with this. They did not 
 do anything to the system 
 
 level3 (converter) was sitting and waiting for events this means it 
 looked Okay. 
 
 Started a long time test around 8 o'clock in the evening. 
 
26 MARCH 1999 
============= 
 
 Started immediately a new test. Test crashed with the same symptoms as 
 the one over night. The event count suddenly jumped from 55 to more 
 than 2000 event/30k ticks and it died afterwards. In total this test 
 wrote 800k events.  Suspiciously the event size was very small (40 
 words) with the high rate. So something screwed up in the hardware. 
 Looked at the events with Frank: They do not contain any structure of 
 the Data. 
 
 It turns out that at the moment when the event size jumped Ed 
 McKingey turned off the power of the crate. This might well be the 
 reason why things failed. I unfortunately do not remember whether I 
 checked the two rlogin processes whether they were still alive. 
 
 
 Tried to start a new test to run the Wedge over the VRB and the 
 Event-Builder into the Level-3. Problem appeared in the startup of the 
 cosmic_readout program. 
  
 Symptom: 
 
 when "take data" button was pressed not the usual print out appeared 
 but there came some error messages or there was not just any strange 
 message but the good ones of telling the control sequence did not pop 
 up. This happened also after very many tries. I also tried the secret 
 vmeIntService on the wegtst01 vxworks node. It did not help! 
 
 Each try the Event-Builder got screwed up. The SCPU b0eb21 seems to 
 be is trying to read data from the VRB, and it gets stuck. Actually 
 this means that one cannot log into the node which means press the 
 reset button. 
 
 When looking at the SCPU and the VRB the following LED are on: 
 
 VRB:  +5V   green  on 
       CPU   yellow on 
       VME   yellow on 
       SRC   yellow off 
              
 SCPU: CHS   yellow off 
       BFL   orange off 
       CPU   green  on 
       PCI   green  on 
       FUS   green  on 
       SYS   green  on 
 
 this problem might be the same as the one we are having when the SCPU 
 screws up some random time. 
 
27 MARCH 1999 
============= 
 
 Could not make the readout work! 
 
28 MARCH 1999 
============= 
 
 Could not make the readout work! 
 
29 MARCH 1999 
============= 
 
 Could not make the readout work! 
 
30 MARCH 1999 
============= 
 
 Could not make the readout work! 
 VME Crates were replaced by new ones. 
 
31 MARCH 1999 
============= 
 
 Readout now works! 
 .. Not clear why it was not working yesterday. 
 
 Run some Scanner CPU standalone tests. I turned out that the 
 reformatter code as implemented on the Linux system was not very 
 efficient and due to large objects passed to the stack overflowed the 
 stack memory. These problems were solved by implementing the 
 reformatter somewhat more efficiently. To trace down these problems 
 some VxWorks commands have been extremely useful to put into the C 
 code: 
  
     printf("\ncheckStack this TAG\n"); 
     checkStack(taskIdSelf()); 
     fflush(stdout); 
  
 On the other hand also in VxWorks there are some nice commands as 
  
     d & input,16,4    this means  16 elements at the size of 4 bytes 
     d & input,100,4              100                         4 
     d &output,100,4              100                         4 
  
 Here I should mention that without Jeff i would have been completely 
 lost. 
 
 Ran the system locally which is Frank's cosmic readout and my 
 reformatter on the SCPU b0eb21. More then 670,000 events were 
 succesfully read and reformatted. 
 
 Stopped the test to run a Level-3 test. 
 
 Could not start up the Event-Builder properly since Jeff and myself 
 had removed the Zephyr and other stuff from the SCPU boot script .... 
 
 Start stand alone at 22:36. 
 
 > Emulation Mask = 0 
 > I do not touch the data 
 >  
 > 			     
 >     0-0                   
 > 	L                    
 >     \_/     promised!     
 >  
 > Buffer address = 0xc57b80. 
 
01 APRIL 1999 
============= 
 
 When I came in the morning the debug output was giving me  
 
   ERROR: event scan did not work: -1 
 
 And I could not determine when the thing broke. In total 22,959,712 
 events were transferred which points to the fact that it broke rather 
 quickly. The hexdump shows that the event is 40 bytes long which 
 means the error is the same as we have seen before when the event 
 size/rate jumped. 
  
 I could not restart the two processes (readout and event-builder) in 
 parallel. The reason was that I have to disable the busy in the VRB 
 first. When it is not disabled the readout will hang. The tricky part 
 here is that the readout crashes always at first and only works at 
 the second start. Unfortunately, when it crashes it also enables the 
 busy and one has to reset the VRB before starting all over again. 
 Simple if one knows but tricky if one does not. 
 
 Started new test at 14:00 with Frank. Standalone b0eb21 from 
 wegtst01. Ran for 204,000 events and then somebody rebooted the wedge 
 crate. Too bad. 
 
 Started a level-3 run test on the wedge: 
 
    b0l3c04 -> b0l3009, b0l3010 -> b0l3022 
 
 Event rate is 582(3) in 30000 ticks which is about 58 events/sec. 
 
</pre>
<!-- Author: Christoph Paus -->


<!-- Date: Thu Apr 15 11:25:59 1999 -->
<pre>Frank Chlebana and Steve Hahn replaced the clock signal. 
 
In all tests before April 14 the real clock (external to the crate) was used. On April 14 in the afternoon the clock signal was switched to the TestClock (in the crate). 
</pre>
<!-- Author: Christoph Paus -->


<!-- Date: Thu Apr 15 20:12:32 1999 -->
<pre>Wedge test - the system wedge->EVB->LVL3 was run for 
about 10 hours with no error messages in LVL3 log files. 
Stopped today in the morning because of other work  
done around the wedge. 3.7e6 events are taken. 
Configuration: 
   Wedge - 4 ADMEM boards, wegtst01 CPU, using testclock 
   EVB   - b0eb10 as SM, b0eb21 as VSPU 
   LVL3  - b0l3c002 converter, b0l3005,006 processors, 
           b0l3021 output node 
 </pre>
<!-- Author: Ilya Kravcheno -->
<!-- Comment: Sat Apr 17 22;44;37 comment by...Ilya Kravchenko + Christoph Paus -->
Unfortunately it turned out that the level-3 executable was old
and therefore not giving error messages in the l3_event_input.
This test should be ignored.


<!-- Date: Mon Apr 19 10:47:15 1999 -->
<pre>Long test was run on the system 
 wedge-> Event Builder -> Level 3 
over the weekend (Fri 22:00 - Sun 8:30). Sunday morning  
level 3 started to give error messages  
 
>> == ERROR l3_rawevt_vrb_scan ==  
>>upper 32 bits of 64 bit event count word should be ZERO 
>>exit without completion. 
 
which continued until Mon. morning when I exited everything. 
 
  From the wedge ADMEMs 17,18 and 19, real CLOCK 
(see wedge log of Friday evening) were used. 
L3 and EVB configuration is as in the previous test. 
 
  This test ran for ~32-34 hours without errors. Which 
rather good. 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Fri Apr 23 16:34:42 1999 -->
Ilya
<!-- Author: {Quick_Entry} -->


<!-- Date: Fri Apr 30 20:42:27 1999 -->
<pre>NOTE: Startup procedures - when LEvel 3 system is brough up 
after shutdown one needs to restart zephyr service  
manually.  
   To do it  
     - login to b0l3pcom1 to any account 
     - first run this:  ~jtseng/gnu/sbin/zephyrd 
     - then type:       ~jtseng/gnu/sbin/zhm b0l3pcom1 
 
Zephyr service is needed for testbed monitoring,  
among other things. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon May 3 12:00:36 1999 -->
<pre>On Fri Apr 30 message from Jeff: 
-------------- 
Ilya and I looked with some detail into what was failing and so forth, and we 
found at least what was wrong with b0eb18 and b0eb19---it turns out that 
their fibers had been swapped, so the switch was routing their cells 
to the wrong destinations. 
-------------- 
   The symptoms of this problem: when running a test 
of EVB-LVL3 with fake data, SM and the converter node 
time out: too low event rate at SM and "error -18" at Conv. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon May 3 12:02:25 1999 -->
<pre>   Summary of my weekend tests of Level 3 and event builder: 
 
The full configuration is  
  SM b0eb10 
  SCPUs b0eb13 -> up to b0eb22 
  SubFarm1   c01,  002,003,005,006,  021 
  SubFarm2   c02,  007,008,009,010,  022 
using fake data from SCPUs 
 
The tests: 
 
  Configuration         Subfarm 1             Subfarm 2          Comment 
diff from "full"    time (good/bad ev)     time (good/bad ev) 
 
  SCPUs 13,14       1h24min (?/?)died      51min (?/?)died      b0l3003 died 
 
  SCPUs 13-16       23min (167k/135)died   23min (112k/64)died   
   
  SCPUs 13-18       19min (89k/76)died     5h22min (1.7M/10)     
   
  SCPUs 13-20       5h8min (0.9M/2339)died 9h3min (1.6M/3k)     b0l3003 died 
 
SCPU 13-22, no 003  7h0min (192k/648)died  10h44min (2.4M/5k)   b0l3002 died 
 
 
The distribution of tardy fragments among SCPUs is close to uniform. 
 
Note that a level3 processor node crashed 3 times.  
 
At some point in these tests converter node just stops receiving data 
(no increas in .log file). At some point in testbed window the 
following error starts to appear: 
 
>DAQ INFO message at 00:57:36 on Mon May  3 1999 from b0eb10.fnal.gov 
>Process tsmDisp 
>SM Partitions: internal error. 
>in push_builder 1. 
 
Also, when I tried to run the full system with 10 SCPUs on Friday 2 times 
it did not last for 15min whilelast night it went for much longer. 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Wed May 5 09:26:43 1999 -->
<pre>Yesterday afternoon I took b0l3c02, 03, and 04 out of the rack in 
order to test and swap their ATM cards. I found that the card for  
03 is dead; it doesn't work in any machine. Data going into the  
card never makes it into main memory, although the receive LED  
does light up. Data sent for transmission is accepted by the card  
but never goes out onto the fiber, although the optical carrier is  
present. The cards in 01, 02, 04, and 05 check out perfectly using  
both a direct optical loop-back and a loop going through the ATM  
switch. I don't know why b0l3c04 didn't work in Ilya's tests. A  
card transplanted from another converter node still worked  
properly in b0l3c03. For the night's tests I put 02 and 04 back in  
the rack. 
 
This morning I will take the ATM card from b0l3c01 and put it into  
03, which will make all operating converter nodes the same make  
and model of machine. I'll check the Fore manual on CD-ROM to see  
if the problem we see in the bad card is one that is already  
known. If not, I'll see about warranty, etc. 
 
To assist stand-alone testing, I've copied the ATM software in  
~djholm/atm to /root/atm on all converter nodes. 
 
I noticed one small problem while testing the ATM interfaces. If  
you run the br/bw pair of test programs, which sends ATM data as  
quickly as possible without attempting to synchronize, the ATM  
driver will sometimes complain that it has run out of reception  
buffers (either large or small). This complaint appears in /var/ 
log/messages. After it occurs, the count of packets transmitted  
may start to get ahead of the count of packets received (in /proc/ 
atm/devices), although the count of packets dropped remains zero.  
Occasionally, the sending program bw will hang before it sends all  
the packets it was supposed to. 
 
I ran all my tests with transmission and reception on the same  
machine. 
</pre>
<!-- Author: Steve tether -->


<!-- Date: Sat Jun 5 23:29:33 1999 -->
<HTML>  
<BODY> 
<H2> 
    This note contains summary of the tests with the 
Event Builder and Level 3 done in April and May 1999.  
</H2> <BR> 
In all of these tests events are generated on the SCPUs 
using Christoph's code. The output of SCPUs is throttled 
to be equal to some predefined value. 
 
  In addition to testing overall performance of the system, 
many of the tests are directed to understanding why 
and at what conditions we experience variation in event 
rate and observe late fragments. 
 
<hr> 
In this test we study the <B> fraction of bad events 
as a function of timeout </B>. Configuration 
5 SCPU -> 1 CV -> 4 PR -> 1 OUT, throughput 1MB/s/SCPU, 
fragment size 12k. 
 
<table border=1>  
<tr><th> l3_tries <th> 10 <th>50<th> 100<th> 200<th> 300 
                      <th> 500<th> 1000 <th>1500 <th>3000 
<tr><th> good ev <td> 105295 <td>  101043<td>  99633  
     <td> 100587 <td> 104319<td>  103004 
     <td>  152721 <td>  133080<td>  108015 
<tr><th> bad ev <td> 94 <td>  74<td>  75  
     <td> 47 <td> 26<td>  29 
     <td>  31 <td>  19<td>  12 
</table> 
<P> 
<LI> see results in this  
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/late-timeout.ps">  
                        Figure </a> 
<LI> timeout is calculated as l3_tries*4/3ms 
<LI> in all measurements only 1 late fragments per bad 
event is observed 
<LI> in the next tests (<B>A-D</B>) timeout is set to 2000*4/3ms 
to reduce late fragments as much as possible 
 
<hr> 
 
 <B> Test A </B> Configuration:  EVB -> 1 CV -> 4 PR -> 1 OUT, 
fragment size 12k, n=188, timeout = 2000*4/3 ms 
<table border=1>  
<tr><th> N SCPU <th> 2 <th> 4 <th> 6 <th> 8 <th> 10 
<tr><th> rate, ev/sec <td> 87.5 0.2 <td> 86.0 0.5 
          <td> 78 2.5 <td> 35 2<td> 40 2 
<tr><th> good/bad ev <td> 31950/0 <td> 197798/1 <td> 129673/2 
          <td> 222387/331 <td> 61121/97  
</table> 
<P> 
<B> Observations: </B>  
<UL> 
<LI> see this  
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/testA.ps">  
                        Figure </a> 
<LI> in configuration EVB->1 CV -> TRASH event rate is approx. 87.5 ev/s 
for any number of SCPU, no bad fragments are observed. 
<LI> problems with rate are accompanied by bad events 
<LI> sometimes l3_converter is using close to 100% of CPU on b0l3c02. 
<LI> the event rate drops so significantly here and in some 
of the following tests (e.g., <B> C </B>) because timeout 
is so large 
</UL> 
<hr> 
 
 <B> Test B </B> Configuration:  EVB -> 1 CV -> 4 PR -> 1 OUT, 
fragment size 12k, n=32 == 0.3MB/s, timeout = 2000*4/3 ms 
<table border=1>  
<tr><th> N SCPU <th> 2 <th> 4 <th> 6 <th> 8 <th> 10 
<tr><th> rate, ev/sec <td> 26.7 0.1 <td>   26.7 0.1 
          <td> 26.7 0.01 <td> 26.7 0.1<td> 26.7 0.01 
<tr><th> good/bad ev <td> 8276/0 <td> 21824/0 <td> 9669/0 <td>  
                     10505/0 <td> 11501/0  
</table> 
<P> 
<UL> 
<LI> see this 
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/testB.ps">  
                        Figure </a> 
<LI> no problems observed at this low event rate and throughput 
</UL> 
<hr> 
 <B> Test C </B> Configuration:  10 SCPU -> 1 CV -> 4 PR -> 1 OUT, 
fragment size 12k, timeout = 2000*4/3 ms 
<table border=1>  
<tr><th> throughput, MB/sec/SCPU<td> 0.3 <td> 0.5 <td> 0.6 <td> 0.7 
      <td> 0.8 <td> 0.9 <td> 1.0 <td> 1.2 
<tr><th> expected rate, ev/sec <td> 26.7 <td> 44.3 <td> 53.4 <td> 62.3 
    <td> 71.2 <td> 79.5 <td> 87.5 <td> 104.5 
<tr><th> rate, ev/sec <td> 26.7&plusmn0.01 <td> 44.3&plusmn0.01 
    <td> 53.4&plusmn0.1 <td> 52.4&plusmn13.6 <td> 44.0&plusmn17.7 
    <td> 35.2&plusmn17.9 <td> 40.1&plusmn18.8 <td> 37.1&plusmn18.9 
<tr><th> good/bad ev. <td> 11501/0 <td> 16727/0 <td> 13310/0  
    <td> 40118/16 <td> 59404/61 <td> 56452/109 <td> 61121/97 
    <td> 70042/112 
</table> 
<P> 
<UL> 
<LI> see this 
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/testC.ps">  
                        Figure </a> 
<LI> expected is defined as the rate observedfor 1->1->trash system 
<LI> ths system breakes down at approximately 5 MB/s 
</UL> 
<hr> 
 <B> Test D </B> Configuration:  10 SCPU -> 1 CV -> 4 PR -> 1 OUT, 
fragment size 24k, timeout = 2000*4/3 ms 
<table border=1>  
<tr><th> N SCPU <th> 2 <th> 4 <th> 6 <th> 8 <th> 10 
<tr><th> rate, ev/sec <td> 44.8 0.1 <td> 44.9 0.1 
       <td> 44.5 0.4 <td> 44.7 0.1 <td> 37.0 0.01  
<tr><th> good/bad ev <td> 19196/0 <td> 9120/0 <td> 32802/0 <td> 22823/0 
       <td> 1721906/858 
</table> 
<P> 
<UL> 
<LI> see this 
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/testD.ps">  
                        Figure </a> 
<LI> the test with 10 SCPU ran for 12hrs without errors untill 
   was stopped 
<LI> tests <B> A-D </B> show obvious problems. At this point 
we tried to reproduce tests that were done earlier and gave 
good results. 
</UL> 
 
<hr> 
 <B> Reproducing an old test. </B> Awhile ago (Dec.1998?) a test was done 
by Jeff in the following configuration: 
14 SCPU -> 1 CV -> 3 PR -> TRASH with fragment size 16k 
and n=188 == 1MB/s. l3_converter timeout is 20*4/2 ms. 
Observed event is between 50 and 60 ev/sec, late fragments 
are not checked. 
<P>  
In the first attempt EVB -> 1 CV -> 3 PR -> 1 OUT 
fragment size is 12k,  timeout is the same. 
The expected event rate is 87.7 ev/sec. The measurements: 
 
<table border=1>  
<TR><Th>N SCPU        <Th>2      <Th>6      <Th>8    <Th>10 
<TR><TD>Rate, ev/sec  <TD>87     <TD>86     <TD>50   <TD> 59 
</table> 
<hr> 
 
Then, in the same configuration with the fragments size 16k 
the rate is 50 ev/sec <BR> 
The difference between this test and the previous is the presence 
of the output node. <B> By removing the output node in this test 
the older results were reproduced.</B> <BR> 
 Therefore, later we study the output node effects. 
 
<hr>  
 But first several tests done for DAQ meeting will be 
described. 
<B>Test G </B> Configuration:  10 SCPU -> 1 CV -> 3 PR -> 1 OUT, 
fragment size 16k, timeout = 20*4/3 ms. This configuration 
is chosen to resemble closely Jeff's early test. 
<table border=1>  
<tr><th> throughput, MB/sec/SCPU<td> 0.3 <td> 0.5 <td> 0.6 <td> 0.7 
      <td> 0.8 <td> 0.9 <td> 1.0  
<tr><th> rate, ev/sec <td> 20.09&plusmn0.03 <td> 33.34&plusmn0.05 
        <td> 40.18&plusmn0.04 <td> 46.87&plusmn0.04  
        <td> 50.00&plusmn0.06 <td> 49.95&plusmn0.32 <td> 50.0&plusmn0.1 
<tr><th> good/bad ev <td> 2998/2 <td> 15222/0 <td> 11064/0 <td> 23840/0 
        <td> 16989/18 <td> 56495/91 <td> 13935/65 
</table> 
<hr> 
<B> Test H.</B> The plot from this test was shown at DAQ meeting 
on May 19, 1999. Configuration:  N SCPU -> 1 CV -> 3 PR -> 1 OUT, 
fragment size 16k, timeout = 20*4/3 ms.  
<table border=1>  
<tr><th> N SCPU <th> rate, ev/sec <th> good/bad ev <th> bad ev fraction  
                                           <th> observed throughput, MB/sec 
<tr><th> 1  <td> 65.99&plusmn0.04 <td> 32735/0 <td> 0 <td> 1.031&plusmn0.001 
<tr><th> 2  <td> 65.99&plusmn0.03 <td> 14918/0 <td> 0 <td> 2.06&plusmn0.001 
<tr><th> 3  <td> 65.72&plusmn0.97 <td> 26600/0 <td> 0 <td> 3.08&plusmn0.01 
<tr><th> 4  <td> 59.98&plusmn0.05 <td> 19396/3 <td>  (1.5&plusmn0.9)e-4 <td>  
			3.75&plusmn0.001 
<tr><th> 5  <td> 59.98&plusmn0.09 <td> 35830/16 <td> (3.3&plusmn1.1)e-4 <td>  
			4.69&plusmn0.007 
<tr><th> 6  <td> 49.98&plusmn0.06 <td> 21303/22 <td> (1.0&plusmn0.2)e-3 <td>  
			4.68&plusmn0.006 
<tr><th> 7  <td> 49.89&plusmn0.59 <td> 18596/46 <td> (2.5&plusmn0.4)e-3 <td>  
			5.46&plusmn0.01 
<tr><th> 8  <td> 50.06&plusmn0.06 <td> 23572/38 <td> (1.6&plusmn0.3)e-3 <td>  
			6.26&plusmn0.001 
<tr><th> 9  <td> 49.36&plusmn3.29 <td> 20409/15 <td> (7.3&plusmn1.9)e-4 <td>  
			6.94&plusmn0.46 
<tr><th> 10 <td> 50.00&plusmn0.10 <td> 13935/65 <td> (4.7&plusmn0.6)e-3 <td>  
			7.81&plusmn0.02 
</table> 
 
<LI> for this test expected rate is 66 ev/sec 
<LI> see figures  for this test in the next section  
 
<hr> 
<B> May 19, 1999 DAQ meeting presentation </B> 
 Trancparencies can be found on the Level 3 web page. 
<DT> 1. Basic event builder test with configuration 
   EVB -> 1 CV -> TRASH, fragment size 12k, timeout 20*4/3ms, 
   throughput 1MB/s/SCPU 
<LI> see results in this 
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/trash.ps">  
                        Figure </a> 
<LI> throughput as expected 
<LI> subfarms factorize 
 
<DT> 2. Integrated EVB/Level3 test with configuration 
   EVB -> 1 CV -> 3 PR -> 1 OUT, fragment size 16k, throughput 1MB/s/SCPU, 
   timeout 20*4/3ms 
<LI> see results in this 
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/problems_my.ps">  
                        Figure </a> 
<LI> problems with event rate and late fragments are observed 
 
<DT> 3. Observations: 
<LI> non-linear behavior correlated to late fragments 
<LI> larger timeout - fewer late fragments 
<LI> larger fragments - fewer late fragments 
<LI> smaller throttled rate - less late fragments 
 
 
<hr> 
 
  <B> Influence of the output node </B> 
The several tests are done with various subfarm configurations. 
The expected event rate is 87.7 ev/sec as follows from the 
fragment size 12k and n=188. <BR> 
<DT>1. 10 SCPU -> CV -> 4 PR -> OUT(b03l3021) 
<DD>    rate = 59.9 0.7 
<DD> good/bad=25300/57 
<P> 
<DT>2. 10 SCPU -> CV -> 2 SF, where SF -> 2 PR -> OUT 
<DD>    rate = 83.4 9.5  
<DD>    good/bad = 32379/1 
<P> 
<DT>3. Using 2 ethernet ports on CV and only 1 on OUT (there is only 
       one on b0l3021) 
<PRE> 
              -> 2 PR     
 10SCPU -> CF          -> 1 OUT 
              -> 2 PR 
</PRE> 
<DD>     rate = 59.0 0.9 
<DD>     good/bad = 20365/62 
<P> 
<DT>4. Using 2 ethernet ports on both CV (b0l3c02) and OUT (b0l3c03) 
<PRE> 
              -> 2 PR ->    
 10SCPU -> CF            1 OUT 
              -> 2 PR -> 
</PRE> 
<DD>     rate = 83 ev/sec 
<DD>     good/bad = 50000:1 
<P> 
<DT>5. Using 2 ethernet ports on CV and  1 on OUT (this time it is b0l3c03 
      that has 2 ports). <I> Note: Christoph suspects that even 
   though I specify port b0l3c03a in software, both are actually used.</I> 
<PRE> 
              -> 2 PR  
 10SCPU -> CF         -> 1 OUT 
              -> 2 PR  
</PRE> 
 
<B> Conclusions: </B> 
  It appears that the output node has a strong influence 
on the farm performance. Both event rate and the number  
of bad events are affected. Backward TCP/IP pressure? 
Not clear. The network connection is supposed to handle 
up to 10 MB/s while we see effects already for 5-8 MB/sec <BR> 
  So in the next tests we play with rejection on the processors nodes. 
 
<hr> 
<B> Rejection rate effects. </B> This test is performed with 
variable rejection fraction, events are rejected on the  
processor nodes. Configuration 10 SCPU -> 1 CV -> 4 PR -> 1 OUT, 
fragment size 12k, throughput 1MB/s/SCPU, timeout 20*4/3 
 
 
<table border=1>  
<tr><th> Rejection <th> 0% <th> 5% <th> 10% <th> 15% <th> 20%  
            <th> 33% <th> 50% <th> 80% 
<tr><th> rate, ev/sec <td> 59.9&plusmn0.7 <td> 62.9&plusmn3.2 
   <td> 65.5&plusmn4.7 <td> 62.2&plusmn4.1 <td> 71&plusmn8 
   <td> 81&plusmn12 <td> 84&plusmn9 <td> 85.2&plusmn7.2  
<tr><th> good/bad ev <td> 25300/57 <td> 101631/169 <td> 39187/85 
          <td> 37997/92 <td> 42848/65 <td> 54992/7  
          <td> 58900/1 <td> 97045/1 
</table> 
<P> 
<LI> see results in  
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/late-rejection.ps">  
                        Figure 1 </a> and 
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/rate-rejection.ps">  
                        Figure 2</a> 
 
<LI> event with high rejection late fragments occur, although 
 only rarely 
<LI> behavior changes between 20 and 30% rejection 
 
<hr> 
  This test is run with 80% rejection done on the output node. 
Two configurations are tested, with one subfarm and with two subfarms: <BR> 
10 SCPU -> N*( 1 CV -> 4 PR -> 1 OUT), where N=1,2. The results  
are shown in  
    <a href="http://mit2.fnal.gov/~ikrav/work/figures/subfarms.ps">  
                        Figure </a> 
 and in the table below: 
<table border=1>  
<tr><th>  <th> N SCPU <th> 2 <th> 4 <th> 6 <th> 8 <th> 10 
<tr><th> 1 subfarm <th> rate, ev/sec <td> 87.5 0.9 <td> 87.3 1.3 
          <td> 86.7 3.6 <td> 85.3 5.7 <td> 85.2 7.2  
<tr><th>           <th> good/bad <TD> 48722/5 <td> 86873/3 <td> 44742/0  
                        <td> 89712/1       <td>  97045/1    
<tr><th> 2 subfarms <th> rate, ev/sec <td> 174.6 1.7 <td> 174.4 2.2 
          <td> 173.1 5.2 <td> 170.1 12.2 <td> 141.3 6.8 
<tr><th>           <th> good/bad <td> 55433/3 <td> 53414/7 <td> 60453/0 
                        <td> 64910/0 <td>    
</table> 
<B>Conclusions:</B> In spite of the rejection at high throughput 
the event rate drops, even though not many late fragments observed. 
 
<hr> 
<B>   Assorted tests: </B> 
 
In these tests we make sure that all nodes are sitting  on the same network  
switch. Also, in the following  tests we use b0l3c02 as a converter   
and b0l3c04 as an output so both have 2 eithernet ports.  
  These tests were done at "quiet" times when there was no activity on the   
network, not even logging in and out.  
<P> 
<DT>1. 10 SCPU-> 1 CV -> 4 PR -> 1 OUT, n=188 (1MB/s/SCPU), <BR> 
   fragment size = 12kB, rejection fraction 90%, timeout 15*4/3=20ms  
<DD>        rate 85.5 7.7 ev/s <BR> 
<DD>        good/bad events (fragments) = 156405/1(1) 
<P> 
<DT>2. 10 SCPU-> 1 CV -> 4 PR -> trash, otherwise the same as 1. 
<DD>        rate 85.6 6.6 ev/s 
<DD>        good/bad = 179032/0 
<P> 
<DT>3. 8 SCPU-> 1 CV -> 4 PR -> 1 OUT, otherwise same as 1. 
<DD>        rate 85.7 6.0 ev/s 
<DD>        good/bad = 185594/10(71) 
<P> 
<DT>4. 10 SCPU-> 1 CV -> 8 PR -> 1 OUT, otherwise same as 1. 
<DD>         rate = 85.2 7.9 ev/s 
<DD>         good/bad = 155257/3(17) 
<P> 
<DT>5. timeout 750*4/3=1sec, otherwise the same as 1 
<DD>         rate = 84.8 7.7 ev/s 
<DD>         good/bad = 161152/1(1) 
<P> 
<B>  Some conclusions:</B> With converter node used as an output 
node, fragment size of 12k and 90% rejection the problem 
with the late fragment is at the level of ~1/50000. 
  Large timeout does not lead to the drop in the 
event rate as observed in tests with larger number  
of late fragments. 
 
<hr> 
 
<B> <a href="http://mit2.fnal.gov/~ikrav/work/figures/doe.ps">  
                        Plot, </a> 
 shown by Christoph at DOE review </B> 
<BR> 
 The test configuration: 
  10 SCPU -> 1 CV -> 4 PROC -> 1 OUT (w 2 ports), 1MB/sec/SCPU, 
  fragment size 16k, timeout 750*4/3 = 1sec, rejection 90% 
The measurements for the plot: 
 
<table border=1>  
<tr><th> NSCPU               <th> 2 <th> 4 <th> 6 <th> 8 <th> 10   
<tr><td> rate, ev/s          <td> 66.0 0.18 <td> 65.8 1.3 
          <td> 65.7 1.7 <td> 64.6 5.7 <td> 64.4 5.1  
<tr><td> good/bad events     <td> 245131/0 <td> 107339/0  
          <td> 104981/0 <td> 103784/0 <td> 102750/0 
<tr><td> good/bad fragments  <td> 490262/0  <td> 429336/0 
          <td> 629886/0 <td> 830280/0 <td> 1027500/0  
</table> 
<HR> 
<B> Conclusions: </B>  
At present the problem with the late fragments is not solved. 
We, and especially Steve and Jeff, are looking into it in more and 
more details. 
</BODY>  
</HTML>  
  
<!-- Author: Ilya Kravchenko -->


<!-- Date: Thu Jul 22 15:25:38 1999 -->
<H1>Results from the wedge test crate: "final" reformatter test</H1> 
 
<P> 
990720 - 990722 -- running the full wedge test 
</P> 
 
<P> 
 
Here is what we want to find: it is an event read from the wedge by 
the SCPU stand alone program. 
 
</P> 
 
<PRE> 
 
Command sequence used (VRB might need hardware reset): 
====================================================== 
 
 > cd "~paus/vrb" 
 > ld < libvrb.o 
 > reset_vrb,9 
 > reformat_vrb,9,0x200,0x000 
 
Event on SCPU in RAW format: 
============================ 
 
 nwords:    347, offset:      0 
 
  0-  7:      56c      568        2        3        4        5        6        0  
  8- 15:      550    90009        0        0        0        0      52c      14b  
 16- 23: 50454d44   180001        0      141        1       32    10030    20031  
 24- 31:    30036    40034    50032    60032    70034    80034    90032    a0035  
 32- 39:    b0033    c0032    d0031    e0034    f0033   100033   110033   120030  
 40- 47:   130032   400018   41001d   420016   430018   440019   45001d   460018  
 48- 55:   47001a   480017   49001c   4a001b   4b001a   4c0016   4d001a   4e0019  
 56- 63:   4f001a   500018   51001c   520016   530019   800039   810037   820037  
 64- 71:   830036   840039   850035   860038   870034   880033   890037   8a0037  
 72- 79:   8b0038   8c003c   8d0036   8e0037   8f0037   900031   910034   920031  
 80- 87:   930034   c0003d   c10040   c2003d   c30042   c4003e   c50045   c6003a  
 88- 95:   c70040   c80041   c90040   ca003b   cb003e   cc003d   cd003f   ce003b  
 96-103:   cf0041   d0003e   d1003e   d2003f   d3003a  1400033  1410036  1420034  
104-111:  1430032  1440032  1450033  1460034  1470035  1480032  1490034  14a0034  
112-119:  14b0033  14c0031  14d0035  14e0031  14f0032  1500032  1510032  1520033  
120-127:  1530032  1800050  181004f  1820054  1830053  1840053  185004b  1860053  
128-135:  1870050  1880052  189004e  18a004c  18b004e  18c0053  18d004c  18e0050  
136-143:  18f0051  1900052  1910050  1920051  193004e  1c00034  1c10034  1c20033  
144-151:  1c30036  1c40032  1c50032  1c60031  1c70036  1c80034  1c90035  1ca0032  
152-159:  1cb002e  1cc0030  1cd0030  1ce0030  1cf002f  1d00031  1d10033  1d20033  
160-167:  1d30032  200002f  201002e  202002e  203002a  204002f  2050029  206002b  
168-175:  207002d  2080029  209002c  20a002b  20b002c  20c002a  20d002b  20e0028  
176-183:  20f0029  210002d  211002b  2120029  213002b  280002e  2810031  2820034  
184-191:  2830032  2840031  2850030  286002f  2870030  2880031  2890032  28a002f  
192-199:  28b0032  28c0031  28d0033  28e0032  28f0030  2900034  2910032  2920031  
200-207:  2930030  2c00025  2c10026  2c20020  2c3001f  2c40024  2c50027  2c6001f  
208-215:  2c70023  2c8001d  2c90024  2ca0026  2cb0022  2cc001d  2cd0023  2ce001f  
216-223:  2cf001f  2d0001e  2d1001f  2d2001e  2d30021  3000037  3010033  3020036  
224-231:  3030032  3040035  3050034  3060037  3070032  3080035  3090034  30a0036  
232-239:  30b0037  30c0037  30d0038  30e003a  30f0038  3100033  3110032  3120031  
240-247:  3130032  3400035  3410039  3420035  343003b  344003c  345003c  3460038  
248-255:  3470039  348003f  3490038  34a0038  34b0039  34c0039  34d0038  34e0035  
256-263:  34f0039  350003b  3510039  3520040  353003a  3c00032  3c10034  3c20036  
264-271:  3c30033  3c40033  3c50034  3c60030  3c70034  3c80035  3c90032  3ca0031  
272-279:  3cb0033  3cc0033  3cd0034  3ce0032  3cf0031  3d00032  3d10032  3d20031  
280-287:  3d30032  400003f  401003a  4020040  403003b  4040040  405003f  4060042  
288-295:  407003e  4080042  409003d  40a003d  40b003c  40c0040  40d003d  40e003f  
296-303:  40f003d  410003e  411003c  412003b  413003e  4400028  441002d  442002a  
304-311:  443002d  444002b  445002e  446002b  447002c  448002b  449002f  44a0029  
312-319:  44b002b  44c002a  44d002c  44e0028  44f002a  4500031  4510033  4520031  
320-327:  4530031  4800027  4810027  482002a  4830025  4840024  4850023  4860028  
328-335:  4870027  4880022  489002b  48a0029  48b0022  48c0028  48d0027  48e0028  
336-343:  48f0025  4900029  4910026  4920027  4930025        0 aaaaaaaa 55555555  
344-351:   170102 80000000        0  
 
</PRE> 
 
<PRE> 
 
Event read from wedge through EVB and Level3 
============================================ 
 
 RAW format, print on processor node 
 
 nwords:    352, offset:      0 
 
  0-  7: 446b945f      614 1f56be00 33fafb00 6c050000 68050000  1000000 21000000  
  8- 15:   746e43     4201 5500d900        0 50050000        0        0        0  
 16- 23:        0        0 2c050000 4b010000 444d4550  1001800        0 41010000  
 24- 31:  1000000 35000000 34000100 34000200 31000300 33000400 30000500 32000600  
 32- 39: 35000700 33000800 34000900 34000a00 32000b00 33000c00 33000d00 32000e00  
 40- 47: 38000f00 32001000 30001100 30001200 32001300 3f004000 3c004100 43004200  
 48- 55: 3f004300 3e004400 3c004500 42004600 3f004700 3f004800 3e004900 3c004a00  
 56- 63: 3f004b00 41004c00 3d004d00 40004e00 41004f00 3a005000 3e005100 3e005200  
 64- 71: 41005300 30008000 32008100 33008200 30008300 2b008400 33008500 30008600  
 72- 79: 32008700 34008800 2e008900 30008a00 2f008b00 2f008c00 2f008d00 31008e00  
 80- 87: 31008f00 32009000 33009100 30009200 31009300 2900c000 3000c100 2b00c200  
 88- 95: 2a00c300 2c00c400 2600c500 2900c600 2900c700 2b00c800 2c00c900 2a00ca00  
 96-103: 2a00cb00 2900cc00 2c00cd00 2900ce00 2800cf00 2c00d000 2a00d100 2b00d200  
104-111: 2900d300 30004001 30004101 32004201 32004301 36004401 36004501 33004601  
112-119: 32004701 32004801 31004901 34004a01 32004b01 31004c01 31004d01 32004e01  
120-127: 31004f01 30005001 33005101 34005201 31005301 18008001 1b008101 12008201  
128-135: 17008301 17008401 1a008501 1a008601 18008701 15008801 1c008901 19008a01  
136-143: 16008b01 14008c01 17008d01 15008e01 14008f01 1a009001 18009101 14009201  
144-151: 17009301 3800c001 3600c101 3600c201 3400c301 3600c401 3200c501 3400c601  
152-159: 3000c701 3300c801 3500c901 3700ca01 3900cb01 3700cc01 3900cd01 3300ce01  
160-167: 3900cf01 3100d001 3500d101 3300d201 3000d301 3e000002 40000102 3e000202  
168-175: 42000302 3f000402 43000502 3d000602 3f000702 40000802 43000902 3d000a02  
176-183: 3f000b02 43000c02 3d000d02 3f000e02 3e000f02 3a001002 3c001102 40001202  
184-191: 3c001302 36008002 36008102 31008202 33008302 35008402 32008502 30008602  
192-199: 37008702 31008802 35008902 34008a02 32008b02 31008c02 33008d02 33008e02  
200-207: 33008f02 33009002 31009102 32009202 33009302 5100c002 4d00c102 5300c202  
208-215: 4f00c302 5100c402 4d00c502 5300c602 4f00c702 5200c802 4c00c902 5100ca02  
216-223: 4f00cb02 5500cc02 4d00cd02 5200ce02 4f00cf02 4f00d002 4c00d102 5300d202  
224-231: 4b00d302 30000003 34000103 34000203 31000303 31000403 33000503 30000603  
232-239: 33000703 32000803 34000903 2f000a03 2f000b03 2e000c03 31000d03 2b000e03  
240-247: 2f000f03 32001003 33001103 32001203 32001303 29004003 2f004103 30004203  
248-255: 2d004303 2d004403 2b004503 2a004603 2f004703 2e004803 2d004903 2a004a03  
256-263: 2b004b03 29004c03 2b004d03 25004e03 2a004f03 2d005003 2b005103 2c005203  
264-271: 2b005303 3100c003 3000c103 3100c203 3300c303 3000c403 3300c503 3100c603  
272-279: 3300c703 3200c803 3200c903 3200ca03 3200cb03 3200cc03 3400cd03 3000ce03  
280-287: 3100cf03 3300d003 3200d103 3100d203 3200d303 1f000004 25000104 1b000204  
288-295: 1e000304 22000404 24000504 20000604 20000704 20000804 24000904 21000a04  
296-303: 22000b04 1b000c04 24000d04 20000e04 1e000f04 21001004 20001104 18001204  
304-311: 20001304 33004004 31004104 35004204 32004304 33004404 2f004504 34004604  
312-319: 33004704 32004804 33004904 37004a04 37004b04 35004c04 37004d04 38004e04  
320-327: 37004f04 33005004 32005104 33005204 32005304 34008004 37008104 33008204  
328-335: 37008304 37008404 3a008504 36008604 35008704 38008804 36008904 34008a04  
336-343: 34008b04 38008c04 35008d04 34008e04 35008f04 37009004 36009104 3d009204  
344-351: 35009304        0 aaaaaaaa 55555555  2011700       80        0        0  
 
</PRE> 
 
<PRE> 
 
Event read from wedge through EVB and Level3 
============================================ 
 
 REF format, print on processor node 
 
 nwords:    389, offset:      0 
 
  0-  7: 446b945f      57c 1f56be00 33fafb00        0        0        0        0  
  8- 15:        0        0        0        0        0        0        0        0  
 16- 23:        0        0        0        0        0        0        0        0  
 24- 31:        0        0        0        0        0        0        0        0  
 32- 39:        0        0        0        0      161 4849524c        1        0  
 40- 47:        a        1 ffffffff        0        1        2        3        4  
 48- 55:        5        6        7 444d4550        1        0      144        1  
 56- 63:   1a0018  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  
 64- 71:  29a029a  29a029a  29a029a  29a029a  29a029a       34    10032    20031  
 72- 79:    30035    40033    50032    60033    70030    80031    90030    a0032  
 80- 87:    b0031    c0031    d0033    e0030    f0031   100033   110032   120032  
 88- 95:   130033   400021   410025   420019   43001f   44001d   450022   460019  
 96-103:   47001e   48001c   49001f   4a0022   4b001f   4c0019   4d0020   4e001a  
104-111:   4f0019   50001a   51001e   520019   53001d   800032   810034   820035  
112-119:   830032   840031   850032   860033   870032   880031   890030   8a0034  
120-127:   8b0034   8c0033   8d0033   8e0034   8f0035   900032   910031   920031  
128-135:   930031   c00037   c10036   c20034   c30038   c40032   c50039   c60031  
136-143:   c70039   c80033   c90036   ca0035   cb0035   cc0036   cd0035   ce0033  
144-151:   cf0035   d00037   d10039   d20039   d30035  1400032  1410035  1420036  
152-159:  1430032  1440034  1450034  1460033  1470033  1480032  1490031  14a002f  
160-167:  14b0032  14c0033  14d0033  14e0035  14f0035  1500030  1510032  1520031  
168-175:  1530030  1800044  181003c  1820043  1830043  1840040  185003f  1860042  
176-183:  1870043  1880043  189003f  18a003f  18b0043  18c0041  18d0041  18e0044  
184-191:  18f0042  1900042  1910042  1920041  193003f  1c0002d  1c1002d  1c20030  
192-199:  1c30032  1c4002d  1c50031  1c6002f  1c7002f  1c8002e  1c90033  1ca002e  
200-207:  1cb002c  1cc0031  1cd002e  1ce002b  1cf002d  1d00031  1d10031  1d20033  
208-215:  1d30034  2000029  201002b  2020029  2030028  2040029  2050024  2060027  
216-223:  207002b  208002a  209002a  20a0026  20b0026  20c0026  20d002a  20e0025  
224-231:  20f0025  2100028  2110029  2120024  2130027  2800033  281002d  2820033  
232-239:  2830034  2840030  2850033  2860032  2870033  2880036  2890036  28a0035  
240-247:  28b0034  28c0030  28d0035  28e0030  28f0033  2900030  2910034  2920031  
248-255:  2930032  2c00016  2c1001a  2c20011  2c30015  2c40017  2c5001b  2c60012  
256-263:  2c70017  2c80015  2c90016  2ca0018  2cb0018  2cc0017  2cd001a  2ce0014  
264-271:  2cf0016  2d00015  2d1001c  2d20017  2d3001a  3000036  3010036  3020034  
272-279:  3030032  3040036  3050032  3060034  3070031  3080031  3090032  30a0035  
280-287:  30b0036  30c0036  30d0036  30e0036  30f0037  3100031  3110034  3120032  
288-295:  3130032  340003a  341003b  342003b  343003f  344003e  3450042  346003c  
296-303:  347003b  348003c  349003b  34a003b  34b003d  34c003e  34d003a  34e003a  
304-311:  34f003d  350003d  351003c  352003f  353003a  3c00033  3c10036  3c20034  
312-319:  3c30033  3c40035  3c50035  3c60035  3c70036  3c80033  3c90031  3ca0031  
320-327:  3cb0034  3cc0031  3cd0033  3ce0036  3cf0037  3d00034  3d10035  3d20033  
328-335:  3d30032  4000052  4010051  4020057  4030054  4040053  4050051  4060059  
336-343:  4070054  4080056  4090053  40a004e  40b0050  40c0053  40d0051  40e0054  
344-351:  40f0053  4100053  411004e  4120054  413004e  4400031  4410033  4420034  
352-359:  4430036  4440034  4450033  4460035  4470037  4480035  4490035  44a002f  
360-367:  44b002c  44c0032  44d0031  44e002d  44f0030  4500033  4510033  4520031  
368-375:  4530032  480002f  4810031  4820031  483002c  484002f  485002c  486002f  
376-383:  4870032  488002e  489002d  48a002c  48b002c  48c002d  48d0029  48e002c  
384-391:  48f002a  490002f  491002c  492002c  493002d 
 
</PRE> 
 
<P> 
Bottom line is that the reformatted events printed on the processor 
node now look exactly the same as the event printed on the SCPU. 
Exactly means concerning the structure, the event contents is 
obviously different. 
</P> 
 
 
<P> 
 
One more remark about the YBOS NAME in the header of the event 
Although represented as an integer as: 
<PRE> 
 
 444d4550 
(D M E P ) 
 
</PRE> 
 
It should come out as PEMD when read as a character array since these 
are little endian numbers. 
</P>  
<!-- Author: Christoph Paus -->



<!-- Date: Fri Aug 6 19:59:15 1999 -->
<H3> Tests of the Executable/Control Interface</H3> 
 
Executable link info:<p> 
<ul> 
<li>level3/control tagged version 
RunII_v1_0 </li> 
<li>development version of offline Aug 6, 1999 </li> 
<li>Level3Mods package has uncommitted changes for debugging </li> 
</ul> 
 
Version of level3/control is current ups release<p> 
 
<h4>Observed problems:</h4> 
<ul> 
<li> l3_dispatcher on processor node hangs and then crashes upon 
        receiving the L3_BEGIN_RUN event sent by the executable.</b> <p> 
Unfortunately, no debugging information is present <p> 
<i> Work around: </i> run for now in trash output mode on processor <p></li> 
<li> 
active_run -> abort transition is not successful <p> 
l3_box_manager reports:<p> 
<pre><small><tt> L3_Box_Manager: Received msg attributes 
type   = L3_Run_Halt 
length = 20 bytes 
origin = L3_RC_Server 
destin = L3_Box_Manager 
time   = Fri Aug  6 19:37:15 1999 
 
 
-------------------------------------------------------------------------------- 
ERROR--> hostname: b0l3pcom1.fnal.gov 
         prg name: l3_box_manager_atm_shared 
         function: l3_msgque_receive_acknowledge_from_any() 
         datetime: Fri Aug  6 19:37:15 1999 
         src file: /home/paus/level3/control/src/library/l3_msgque.c 
         src line: 1377 
         src info: Acknowledgement received is not of expected type 
         sys perr: Success 
l3_msgque_receive_acknowledge_from_any(WARNING)---------- 
Msg: L3_Acknowledge, received by L3_Box_Manager from L3_Node_Exe 
Msg: goal 32 bytes 
Ack: L3_Run_End, SUCCESS 
</tt></small></pre> <p> 
<i> Conclusion: </i> appears to be the fault of the executable interface in <tt>APPLevel3InputModule.cc</tt>.  Will investigate 
<p> </li> 
<li> Event format as returned by reformatter is corrupt in several places <p> 
<ul> 
<li> Total word count is inclusive instead of exclusive. <p> 
     <i> Solution: </i> test with a kludge in place in the 
          executable to correct this word count <p></li> 
<li> PEMD bank length is incorrect by 10 words.<p> 
     <i> Solution: </i> Fix reformatter <p></li> 
<li> Illustration from the executable debug output: <p> 
    <pre><small><tt> 
nextEvent:KLUDGE... set reformatted event one word smaller as a test 
TRY_Ybos_Simple_Record::iter_first(key): Attempt to advance past end of record -- corrupt record suspected 
next Event: WIN address = 0x40565010, logical len = 351, physical len = 1048592, event offset = 32 
********************************************************* 
nextEvent: WIN buffer 
--------------------------------------------------------- 
nwords:    512, offset:      0 
--------------------------------------------------------- 
 
  0-  7:        0        0        0        0        0        0        0        0  
  8- 15:        0        0        0        0        0        0        0        0  
 16- 23:        0        0        0        0        0        0        0        0  
 24- 31:        0        0        0        0        0        0        0        0  
 32- 39:      15f 4449524c        1        0        9        1 ffffffff        0  
 40- 47:        1        2        3        4        5        6 444d4550        1  
 48- 55:        0      144        1   1a0018  29a029a  29a029a  29a029a  29a029a  
 56- 63:  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  
 64- 71:     80a7    180e0    280d0    380c4    480d2    580b3    680e4    780b6  
 72- 79:    880eb    980bf    a808e    b808a    c80d9    d80a9    e80c0    f80c7  
 80- 87:   1080a3   1180db   12809b   1380cb   4080bb   4180d1   4280c1   4380c9  
. 
. 
. 
368-375:  4848058  48581af  486804f  4878074  4888096  489806c  48a806b  48b8069  
376-383:  48c80a7  48d8043  48e806e  48f806a  4908096  4918072  49280bc  493804b  
384-391:        0        0        0        0        0        0        0        0  
********************************************************* 
Bank List:LRID #1 (9 size) 
PEMD #1 (324 size) 
k<80><8A>^D #76251241 (76382275 size) 
TRY_Ybos_Simple_Record::iter_any_next(): Attempt to advance past end of record -- corrupt record suspected 
</tt></small></pre> 
</ul> 
</li> 
<li> Kludging fixes by hand in the InputModule results in 
           <i> successful interpretation of the event </i> <p> 
<pre><small><tt> 
Bank List: 
LRIH #1 (9 size) 
PEMD #1 (334 size) 
</tt></small></pre> 
</ul> 
 
<h4> The Level-3 executable has, for the first time since the upgrades to the control code, successfully read an event.  It 
has also, for the first time, successfully read an event produced by the reformatter (modulo a few fixes)</h4> 
<small> (yea!) </small>
<!-- Author: Kevin McFarland -->


<!-- Date: Sat Aug 7 18:33:41 1999 -->
<h3> Continuing debugging the offline exec I/O interface...</h3> 
 
<ul> 
<li> Dispatcher crashes fixed 
<ul> 
<li> Problem was that <tt>_win</tt> buffer length in I/O modules was not being updated when banks were added to the event.  An update just before the send to the dispatcher is now forced 
</li> 
<li> The level3 exec can now run almost indefinitely (limited due to disk space limitations exhausted by the error messages produced from the continuing problems of the reformatter in writing valid <tt>PEMD</tt> banks) 
</li> 
</ul> 
</li> 
<li> Failure of HALT->ABORT transition fixed 
<ul> 
<li> <tt>APPLevel3OutputModule</tt> was erroneously sending an acknowledge of an <tt>L3_RUN_END</tt> message every time it sent the EoR event to the dispatcher.  The appropriate message is now sent. 
</li> 
</ul> 
</li> 
<li> Output of <tt>TL3B</tt> and <tt>TL3D</tt> banks added 
<ul> 
<li>  Banks are successfully added to the event and sent to the dispatcher</li> 
<li> Continuing problems with <tt> PEMD</tt> bank and with word count in reformatter mean that bank appears in the wrong place and that bank cannot be accessed <p> 
<tt><small><pre> 
fillBegin: WIN address = 0x40565010, logical len = 362, physical len = 1048592, event  
offset = 32 
********************************************************* 
fillBegin: WIN buffer 
--------------------------------------------------------- 
nwords:    512, offset:      0 
--------------------------------------------------------- 
 
  0-  7:        0        0        0        0        0        0        0        0  
  8- 15:        0        0        0        0        0        0        0        0  
 16- 23:        0        0        0        0        0        0        0        0  
 24- 31:        0        0        0        0        0        0        0        0  
 32- 39:      16a 4849524c        1        0        a        1 ffffffff        0  
 40- 47:        1        2        3        4        5        6        7 444d4550  
 48- 55:        1        0      144        1   1a0018  29a029a  29a029a  29a029a  
 56- 63:  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  29a029a  
 64- 71:  29a029a     80a7    180e0    280d0    380c4    480d2    580b3    680e4  
 72- 79:    780b6    880eb    980bf    a808e    b808a    c80d9    d80a9    e80c0  
 80- 87:    f80c7   1080a3   1180db   12809b   1380cb   4080bb   4180d1   4280c1  
 88- 95:   4380c9   44807c   4580b2   468074   478097   488087   498092   4a8097  
 96-103:   4b80ac   4c8084   4d80bf   4e80b8   4f80c0   5080cc   5180bc   5280b0  
. 
. 
. 
368-375:  483808c  4848058  48581af  486804f  4878074  4888096  489806c  48a806b  
376-383:  48b8069  48c80a7  48d8043  48e806e  48f806a  4908096  4918072  49280bc  
384-391:  493804b        0 44334c54        1        0        5        1        0  
                            D 3 L T 
392-399:        0        0        0        0        0        0        0        0  
Bank List:LRIH #1 (10 size) 
PEMD #1 (324 size) 
k<80><8A>^D #76251241 (76382275 size) 
TRY_Ybos_Simple_Record::iter_any_next(): Attempt to advance past end of record -- corrupt record suspected 
</pre></small></tt> 
 
</li> 
</ul> 
 
</li> 
</ul> 
  
<!-- Author: Kevin McFarland -->


<!-- Date: Thu Aug 12 14:46:02 1999 -->
<pre>Starting the new series of tests. The major change -  
new atm driver. 
 
    Jul 12 - had problems with network. b0l3pcom1 
was rebooted. In subsequent tests this day reformatter 
crashes until 5:30pm, after which tests where fine. 
 
  A. Throughput tests: 
System configuration is N_SCPU->1CV->4PROC->1OUT. 
n=188 (1MB/s/SCPU), fragment size 12k, 90% rejection  
at the processor nodes. 
  N_SCPU     Rate, ev/s        N events in run 
    2        87.3+-0.001         40048 
    4        85.7+-0.5           85436 
    6        86.3+-0.5           80206 
    8        85.5+-0.6          100438 
   10        81.9+-1.2          100475 
 
 - No tardy fragments in these tests 
 - Slight degradation is seen at high throughput 
 
 B. Test with 4 subfarms.  
Configuration: 10SCPU->4x(1CV->4PROC->1OUT) 
event rate is 83.0+-0.5 ev/s/CV 
  - Two tardy fragments are observed on b0l3c02.  
  - The system is left running overnight  
by the morning 1 subfarm was still working.  
  - All subfarms processed a few of 10^6 events. 
</pre>
<!-- Author: Ilya Kravchenko -->