<!-- Title: 08:51:54  Wed Apr 11 2001 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Ilya Kravchenko -->
<!-- Op3: Sasha Rakitin -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Run II hardware event builder</pre>


<!-- Date: Wed Apr 11 08:57:13 2001 -->
<p>Several times over the last month we've had large rates of CRC errors reported by the ATM drivers on converter nodes that lasted for many minutes and were reported by many converter nodes. At 18:49 last Monday (the ninth) the ATM switch was cold-booted: switched off for 30 seconds then turned back on. So far we have not seen any CRC errors after that.  
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Apr 11 10:43:36 2001 -->
<p>I've installed cdfevb_scpu v3_24 and cdfevb_vrb v3_9 as the current UPS versions on pcom1, have installed them in vxboot/all on b0l3backup, and changed vxboot's .cshrc to load them. These are the versions we've been using for some time, loading them from ~vxboot/tether/ as scpu_14Mar2001.o and vrb_15Mar2001.o. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Apr 11 15:41:25 2001 -->
<p>We've had zwgc die on us, without any error message, when the 
user that started the program logged out afterward. Sasha looked in in the source code and found that when started zwgc tests whether /dev/tty is connected to a terminal (has a valid process-group ID). If it was, and later on /dev/tty becomes disconnected, zwgc exits. Evidently the authors never thought the program would be used to feed a pipe. 
<p>The solution is to test whether the standard output, not /dev/tty, is a TTY or not. This is accomplished by replacing 
the code of check_tty() in zwgc/mux.c by 
<pre> 
    return isatty(1); 
</pre> 
"1" being the file descriptor for the standard output. 
<p>I've installed this change in the UPS zephyr v2_0_4 on pcom1. I've also imported this version into the online CVS repository as cdfevb/zephyr tag v2_0_4. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Thu Apr 12 14:09:04 2001 -->
<p>Yesterday I rebuilt the UPS ilu 2_0a14 so that its home directory is /usr/products/ilu/v2_0a14 instead of /home/tether/ilu, so I could delete the symbolic link in my home directory that pointed back to the actual product directory. After I did that the SCPU rebuild still worked so c-stubber et al. were being found. 
<p>ILU's home directory is hardwired into the code by the building process so you can never just copy it. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Sun Jun 10 03:52:18 2001 -->
<p>Occasionally we've been seeing the dispatcher task tsmDisp get suspended during a Recover. The immediate cause was an incorrect partition number being passed to SM_Disp_Suspend(); the given partition didn't exist and so garbage was fetched for the address of the partition-object.<p> 
The fault was in two places. In cdfevb/sm/inc/evbCtrl.h the DP_PNUM field overlapped the ERR_TYPE field, so that when the SCPU turned the original SCRAMNet command into an error reply the contents of DP_NUM were destroyed. A few commands, Recover, Define Partition, and Delete Partition don't store the partition number in the tag word (word 0) but in the DP_PNUM field. Therefore when an SCPU error occurred during the processing of one of these commands the Scanner Manager extracted a bogus partition number from the error reply. I moved ERR_TYPE and ERR_NAME from word 1 to word 2 so that DP_PNUM is preserved. 
<p>Also, the ERR_CODE_NO_STATS macro (in cdfevb/scpu/inc/scpu.h) that inserts the error information into the original SCRAMNet was careless about clearing fields other than CMD, ERR_NAME, and ERR_TYPE. Now it alters only those three fields. 
<p>
<!-- Author: Steve -->
<!-- Comment: Sun Jun 10 03;54;43 comment by...ST -->
<pre>The new SM and SCPU programs are in b0l3backup:~vxboot/tether as sm_10Jun2001.o and scpu_10Jun2001.o to make them available for testing.</pre>


<!-- Date: Mon Jun 18 11:23:57 2001 -->
<p>Of course now that the SM exercises its event-flushing code we see bugs in that code. Flushed events were not being removed properly from the linked list of pending events. Fixed it. 
<p>The latest SM and SCPU code is available via UPS on pcom1 ad cdfevb_sm v3_22 and cdfevb_scpu v3_26. They are also in the official "all" directory on b0backup:~vxboot. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jul 6 02:28:04 2001 -->
<p>The dumps of suspended tasks now made as part of EVB cleanup have over the last turned up a couple of bugs to be investigated. These have resulted in two SCPU crashes and one SM crash so far. 
<p>In the Scanner Manager there appears to be an invalid subscript for ssi.network_address[] on line 264 of registry.c. 
<p>In the SCPU ATM driver the semaphore ID l->tx_dma is invalid on line 759 of i4515.c. 
<p>
<!-- Author: Steve -->
<!-- Comment: Fri Jul  6 03;29;59 comment by...Steve -->
In two of the crashes the Ace had used HRR several times in a row while trying to cure a persistent problem. In another there is no mention of an EVB cleanup but other e-log entries tell of problems which would normally provoke an HRR or two. Single HRR's work quite well so I suspect that the problem occurs when two HRR's come close together, or perhaps an HRR is not handled properly when a previous one has already flushed internal queues. The code where the crashes occur may not be buggy itself; instead I think that an HRR problem corrupted some data structures used by unrelated code, causing crashes later on. 

<!-- Comment: Tue Jul 10 09;25;42 comment by...Steve -->
We've had a few more of the ATM-thread suspensions. It has happened on both silicon and non-Si SCPUs. It has happened with beam and without beam. The only common factor I can make out is that the shift crew were struggling with some DAQ difficulty for an extended period, i.e., ten minutes or more.


<!-- Date: Wed Jul 18 16:15:40 2001 -->
<p>Last week the SVX crew found that the event builder was dropping about 4-5% of their calibration events, taken with Si readout in the ALL mode (no zero suppression). For an example see the shift E-log for July 12 17:00-18:00. The total event size was about 1.5 MB with at most 120 KB per SCPU per event (the current 
limit is about 230 KB). 
<p> 
It doesn't appear to be due to bad ATM cards or fibers, since a check of the SONET port statistics via "stat port sonet" on b0atm1 shows all SCPUs with about the same number of uncorrectable ATM cell-header errors (most likely due to the reboots done by EVB cleanups). 
<p>I then checked the rate division setting in the SCPU startup script. Surprise, there was none! Neither the script nor the SCPU code called i4515_rate_division(), so we've been living dangerously for a long time by sending data at top speed on every virtual connection. I'll put this back into the startup script. Also, if the SCPU code calls i4515_reset() it must also call i4515_rate_division() since the reset clobbers the rate register. 
<p>While I'm at it I'll define some new global variables in the script which can be read by the SCPU code if needed: 
<ul> 
<li>scpuNumAtmBuffers 
<li>scpuAtmBufferSize 
<li>scpuAtmTransmitMode 
<li>scpuAtmRateDivision 
</ul>  
<!-- Author: Steve Tether -->


<!-- Date: Wed Jul 18 16:31:38 2001 -->
<p>In order to avoid possible overloading the input ATM link of an converter node, each virtual connection from SCPU to converter must have its bandwidth limited to (max rate)/(N) where "max rate" is the maximim link data rate, nominally 155 Mbps, and N is the number of SCPUs. From the formula in section 4.5 of the SARA-S manual we get the following table for R, the value to give as the argument to i4515_reset(): 
<center><table BORDER COLS=2 WIDTH="30%" > 
<tr> 
<td> 
<center><b><font color="#FF0000">N</font></b></center> 
</td> 
 
<td> 
<center><b><font color="#FF0000">R</font></b></center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>1</center> 
</td> 
 
<td> 
<center>252</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>2</center> 
</td> 
 
<td> 
<center>249</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>3</center> 
</td> 
 
<td> 
<center>245</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>4</center> 
</td> 
 
<td> 
<center>242</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>5</center> 
</td> 
 
<td> 
<center>238</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>6</center> 
</td> 
 
<td> 
<center>235</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>7</center> 
</td> 
 
<td> 
<center>232</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>8</center> 
</td> 
 
<td> 
<center>228</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>9</center> 
</td> 
 
<td> 
<center>225</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>10</center> 
</td> 
 
<td> 
<center>221</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>11</center> 
</td> 
 
<td> 
<center>218</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>12</center> 
</td> 
 
<td> 
<center>214</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>13</center> 
</td> 
 
<td> 
<center>211</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>14</center> 
</td> 
 
<td> 
<center>208</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>15</center> 
</td> 
 
<td> 
<center>204</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>16</center> 
</td> 
 
<td> 
<center>201</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>17</center> 
</td> 
 
<td> 
<center>197</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>18</center> 
</td> 
 
<td> 
<center>194</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>19</center> 
</td> 
 
<td> 
<center>191</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>20</center> 
</td> 
 
<td> 
<center>187</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>21</center> 
</td> 
 
<td> 
<center>184</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>22</center> 
</td> 
 
<td> 
<center>180</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>23</center> 
</td> 
 
<td> 
<center>177</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>24</center> 
</td> 
 
<td> 
<center>173</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>25</center> 
</td> 
 
<td> 
<center>170</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>26</center> 
</td> 
 
<td> 
<center>167</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>27</center> 
</td> 
 
<td> 
<center>163</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>28</center> 
</td> 
 
<td> 
<center>160</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>29</center> 
</td> 
 
<td> 
<center>156</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>30</center> 
</td> 
 
<td> 
<center>153</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>31</center> 
</td> 
 
<td> 
<center>149</center> 
</td> 
</tr> 
 
<tr> 
<td> 
<center>32</center> 
</td> 
 
<td> 
<center>146</center> 
</td> 
</tr> 
</table></center> 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Sep 17 16;12;42 comment by...ST -->
Correction: the R value is passed to i4515_rate_division().
<!-- Date: Wed Aug 22 08:56:33 2001 -->
There is new experimental SCPU code in directory ~vxboot/tether on b0l3backup: vrb_21Aug2001.o and scpu_21Aug2001.o. This is another attempt to fix the problem causing suspension of task i4515int.  
<!-- Author: Steve Tether -->
<!-- Comment: Thu Aug 23 10;49;42 comment by...ST -->
We ran a <a href=
'/cgi/elog/elog.pl?nb=2001&action=view&page=603&anchor=09:44:47' target=_top>test</a> of the new SCPU and VRB code today, looks OK. We also tried a Scanner Manager code change which prevents <a href=
'/cgi/elog/elog.pl?nb=2001&action=view&page=599&anchor=05:12:51' target=_top>two EventLoaded messages being sent to the TM for the same event</a> (tether/sm_23Aug2001.o). The EVB is loading the new code from my directory on b0l3backup while I prepare the official releases.


<!-- Date: Thu Sep 6 13:17:17 2001 -->
In order to track down the cause of the suspended ATM task problem I've set up an SCPU test stand on the third floor. The SCPU code is lightly modified so that it can be driven by high-priority tasks rather than SCRAMNet and LAN. An ATM card is installed so that we can test the driver. 
 
<p>For five straight days it has been running Load-Send pairs in an endless loop with no memory leaks and no task crashes. Event size was fixed at 300,000 bytes and the SCPU thinks it is sending to the full complement of 16 L3 subfarms. 
 
<p>I've started a second test where the fake event size has a truncated gaussian distribution of mean 150,000 bytes, sigma of 150,000, maximum of 300,000 and minimum of 0. After a day or so of that I'll try interspersing Recover commands among the Loads and Sends. Repeated Begin Run-End Run pairs are also worth trying. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Sep 7 14:04:50 2001 -->
After 24 hours the variable-sized event test was still running. There was no change in the amount of free storage, and yesterday I performed a five-minute run with malloc tracing which showed no memory leaks. 
 
<p>One peculiar thing did happen near the end of the test. The "i" command which I ran from the shell crashed in the middle of its printout. After I restarted the shell with control-C I found that a number of key tasks were suspended. However, these could not have been suspended before the "i" because the test was still running then. In fact, I was able to restart the test simply by using taskResume() on those key tasks. The problem was apparently caused by a floating-point error in the shell task, here is the "ti" output: 
<pre> 
ti tShell 
 
  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY 
---------- ------------ -------- --- ---------- -------- -------- ------- ----- 
tShell     shell          bd6350   1 READY        163ef4   bd5fd8  3d0001     0 
 
stack: base 0xbd6350  end 0xbd1530  size 19056  high 4320   margin 14736 
 
options: 0xe 
VX_UNBREAKABLE      VX_DEALLOC_STACK    VX_FP_TASK 
 
Floating point underflow 
Floating point inexact 
Floating point invalid operation for interger convert 
Fp Control and Status Register: 0xea024170 
value = 0 = 0x0 
</pre> 
<p>I've now started a test in which four Recover operations in a row are done every 1000 events. The debugRecoverer() function waits until no sends have been done for five consecutive seconds before starting the Recover operations (this is similar to what the Scanner Manager does when a Recover transition is sent). A five-minute run of a similar test with malloc tracing and Recovers every 10 events showed no memory leaks. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Sep 10 16:34:45 2001 -->
I left the Recover SCPU test running all weekend and still haven't been able to get the ATM task suspended. Next test: repeated SetPartition-Begin Run-End Run-Delete Partition sequences with 10000 events per run. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Thu Sep 13 11:15:39 2001 -->
SCPU 3.27 and VRB 3.11 are installed in ~vxboot/all on b0l3backup. They are also 
available on pcom1 via UPS as the development versions (-d). They must 
be used together. 
<p> 
Changes: 
<ul> 
<li>Suppression of console printout during SetPartition transition. 
<li>Use of fragbuff object to manage fragment buffers. 
<li>The last 128 bytes of a fragment buffer will never be used (guards 
    against DMAs from VRBs possibly giving more data than asked for). 
<li>Checking the consistency of the total and per-channel byte counts in SVX 
    VRB data headers. 
</ul> 
 
These new versions have been in use for real data taking since 
<a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=664&anchor=22:53:13' target=_top> 
last night</a>.  
<!-- Author: Steve Tether -->


<!-- Date: Fri Sep 14 11:48:42 2001 -->
Yesterday I was finally able to reproduce the suspended ATM task problem using the SCPU in my test system. Task t4515int, which services interrupts from the ATM card, gets suspended at exactly the same place as it does for the EVB SCPUs. The immediate cause is a data access exception. The problem doesn't appear until you close, then re-open the partition. It doesn't always appear right away, it seems to depend on how busy the SCPU was at the end of the last run before the partition was closed; if it had been idle for a while then all goes well. 
<p>Closing the partition is done by calling the functions delete_builders() and simple_delete_partition(). The first function shuts down tSend tasks and closes the ATM connections to the L3 farm. It may be that the connection closing is being done too soon, recycling data structures that the ATM driver still needs to use (because of a backlog of work stored in the ATM card memory). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Sep 14 14:55:49 2001 -->
When I keep ATM connections from being closed and just use the same ones every time the partition is re-created, the problem doesn't appear. This version of the code has been through several cycles of partition destruction so far, each with ten runs of 30 seconds duration. Let's see if it can run overnight. 
<p>There is also a memory leak in ATM connection closing; two semaphores created when a connection is opened are not destroyed when the connection is closed. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Sep 17 09:57:43 2001 -->
The test-stand SCPU code has run overnight with the normal closing of ATM virtual connections restored and partition closing/opening taking place about every 40 seconds. The ATM driver's transmission mode was changed from 4 to 5. Setting the LSB in the mode forces the driver to wait for DMA completion on each packet before going to the next one. 
<p>The driver should theoretically be able to operate in a "fast mode" in which DMA requests are allowed to queue up at the ATM card. The driver uses a ring FIFO to keep track of which virtual connection objects have pending DMAs. The ATM card generates an interrupt each time a DMA is completed, at which time the driver takes the front entry off the ring FIFO. 
<p>This mode of operation does not work quite right. The DMAs are done but somehow the driver loses track of some DMA-completion interrupts, which causes old pointers to connection objects to build up in the ring FIFO. When ATM connections are closed, their corresponding objects are freed, which makes invalid all the pointers to them that are still (mistakenly) sitting in the ring FIFO. The next time the driver does see a DMA-completion interrupt it will therefore take an invalid pointer from the ring FIFO and try to use it, which causes a data access error. 
<p>The solution is to use the wait-for-DMA mode, which does not allow any connection-object pointers to stay long in the ring FIFO. Therefore, if the SCPU is idle when ATM connections are closed (a condition which the Scanner Manager enforces) there will be no old pointers in the FIFO to cause trouble.  
<p>Even if the "fast" mode were working properly it shouldn't be used in the SCPUs right now, because there is no notification from the ATM driver to the rest of the SCPU code of DMA completion. That means that the SCPU can't be certain when it is safe to re-use a fragment buffer. In practice DMA is very fast so it's unlikely to be a problem, but the safeguard should be in place nonetheless. 
<p>On Friday afternoon I changed the startup script for the SCPUs to set ATM transmission mode 5, instead of 4. A couple of hours later the deadtime of the DAQ system suddenly jumped to over 90% with the source being the EVB. Thinking that the mode change had something to do with it, I told Boris to change it back. The problem seemed to go away, but came back with a vengeance late Sunday evening; no run would last more than about an hour before the EVB deadtime suddenly jumped. Therefore I conclude that this deadtime problem, which we've been seeing once a twice a shift for a month or so, is independent of the ATM task-suspension problem. 
I don't know why the deadtime problem has suddenly become so severe, nor why the test-stand SCPU can make runs lasting for days using very large simulated events. It would appear that the problem lies not in the SCPU code by itself but in the interaction of the EVB with the rest of the DAQ system. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Sep 17 10;06;42 comment by...ST -->
The EVB is now using version 3.10 of the VRB code and version 3.26 of the SCPU code. These versions predate all the changes I made attempting to solve the suspended task problem. In fact, the reversion was done last night as part of the attempts to cure the deadtime problem. It didn't have any effect. The only thing that <em>seemed</em> to have any efect was rebooting all the converter nodes. They were able to take 750,000 events in a run after that, stopping the run for other reasons. Still we can't be sure that it's not a coincidence until we can run for several hours at normal rates.




<!-- Date: Wed Sep 19 16:55:38 2001 -->
Ilya, Boris, and I discovered a number of fascinating things about the so-called "12 Hz" problem on Monday and Tuesday: 
<ol> 
<li>SCPUs can be sick but still do their job well enough for the shift crew not to notice that something's wrong. The symptoms of illness are unresponsiveness to rlogin (and to the LAN in general) and complaints about failed mallocs if you try to issue any shell command using the serial port. The SCPUs still respond to SCRAMNet commands, and memory becomes available when the partition is closed. Only sometimes will the 
SCPU start issuing an a console error message (also about memory) for every event, which then causes the deadtime to skyrocket. 
 
<li>The problem can occur even when the luminosity (and therefore the event rate) is low. For instance, Tuesday morning b0eb13 and 14 had the memory problems but were cranking out events at a rate which the Ace and SciCo told me was normal for the luminosity and trigger. 
 
<li>Certain mode settings for the ATM driver will make the NOTICABLE symptom of high deadtime appear much sooner. We don't know if it has 
any effect on the sneakier version of the problem. The trouble is that the mode setting that causes the quick appearance of symptoms is the one we need to use to fix the suspended ATM task problem (see previous EVB E-log entries). 
 
<li>The SCPU tasks (threads) that send data to the ATM card can overflow their stack allocations in certain circumstances. Their stack allocations are small because there are lots of them: 48 when using the full L3 farm (one per reception buffer of which there are three per converter node). Any attempt by one of these "tSend" tasks to issue a Zephyr message (NOT a console message) causes the stack overflow. Such an error would need to happen only once to screw up the SCPU until the next reboot. Other SCPU tasks may be experiencing a similar problem. 
 
<li>During the last access, another quadrant of the COT was activated which increases the amount of data handled by b0eb13 and 14 by 30%-40%. 
That in itself should not be a problem, but I'll bet this also makes more likely the occurrence of huge events caused by occasional noise spikes. 
 
<li>Monday night the test-stand SCPU started a series of runs in which some of the faked fragments were allowed to be larger than the fragment buffer size. It eventually crashed with memory allocation problems, though not exactly the same ones seen in the DAQ (there are some code changes for stand-alone running). Also, buffer overflow Zephyr messages failed to appear. After one run the checkStack command showed that almost all tSend tasks had overflowed their stacks. It turns out that the function that formats Zephyr messages uses a 1K buffer on the stack, so calling the function is guaranteed to make the stack grow by a bit more than 1K. That was enough. After I changed the function to use a statically allocated buffer protected by a semaphore the stack overflows went away and the buffer overflow messages appeared when appropriate. 
The tSend tasks do not send buffer overflow messages, so the tRec 
task may also have had a problem. 
</ol> 
 
We have some spare memory space in the SCPUs so I'll try increasing the stack space a little for the tSend and tRec tasks. I'll install the change to the Zephyr-message formatting routine. Also, I can remove the only malloc() call that is done once per event. This malloc() is not essential to SCPU operation but is only for debugging. Removing it should stop the flood of console messages in case the SCPU memory still gets quietly screwed up, and that will prevent the large deadtimes. 
<p>  
<!-- Author: Steve Tether -->


<!-- Date: Wed Sep 19 17:00:19 2001 -->
An overnight test of the revised SCPU 3.27 code on the test stand did not crash or screw up memory. About 10% of the simulated fragments had sizes greater than the fragment buffer size of about 330K and these were handled properly. 
<p> 
This afternoon we took a cosmic run with the SVX in the normal readout mode and COT DAC thresholds set low to increase the data size for b0eb13 and 14. We were able to get the COT fragment sizes up to about 120K; past that we started getting BUSY timeouts from all the COT crates. We ran at the maximum rate the CSL would take, about 120 Hz. 
<p> 
After that we took an extended SVX calibration run. This puts the SVX in read-all mode, i.e., it turns off zero suppression. Once again the fragment sizes rose to about 120K, this time for b0eb17 et al., and we drove the CSL as hard as we could (about 30 Hz). 
<p> 
All the above tests were run with the ATM driver in the correct operating mode. There were no signs of memory corruption problems in any SCPU. We've left the new code set as the default for the next shot.<p>  
<!-- Author: Steve -->


<!-- Date: Wed Nov 21 18:03:47 2001 -->
<p>Last Sunday we noticed that boot failures of the EVB VxWorks nodes had become much more common. During an EVB cleanup fully half of them would get stuck at the loading of the kernel image and eventually time out. Sunday was the first EVB cleanup done since the last of the L3 farm PCs were connected to our private network. Coincidence? You decide. 
 
<p>We have isolated the VxWorks nodes and their boot host, now named b0l3boot-vx, on a new private network 192.168.20.0 controlled by the uppermost switch in the ethernet switch rack. We activated the third ethernet port eth2 on the first gateway machine, assigned it the IP address 192.168.20.128 (b0l3gate1-vx), and connected it to the "new" switch. We plan to do the same for the second gateway in the near future. 
 
<p>That left the converter nodes with no direct connection to the Scanner Manager, but they need to talk to it during the Partition transition. We've therefore enabled IP forwarding in the Linux kernel in gate1 and have routed traffic for 192.168.20.0 to eth2. A corresponding route back to the converter node network 192.168.21.0 through gate1 has been defined for the Scanner Manager. 
 
<p>All VxWorks boot parameters have been changed to make b0l3boot-vx (192.168.20.130) the host. 
 
<p>Files changed on gate1: 
<ul> 
<li> Added /etc/sysconfig/network-scripts/ifcfg-eth2. 
<li> /etc/hosts: 
<ul> 
<li> b0l3gate1-vx 192.168.20.128 
<li> b0l3gate2-vx 192.168.20.129 
<li> bol3boot-vx 192.168.20.130 
</ul> 
<li> /etc/rc.d/rc.local: Routing added for the new private net (will later put this in /etc/sysconfig/static-routes). 
<li> /etc/conf.modules: alias eth2 tulip 
<li> /usr/src/2.0.38+atm...: In "make menuconfig" we turned on the networking options Network firewalls, Network aliasing, IP:forwarding/gatewaying, IP:firewalling, and IP:firewall packet logging. The rebuilt zImage is in /boot/vmlinuz-ipfwd. IP forwarding must be configured in the kernel so that /proc/net/ip_forward is created. 
<li> /etc/lilo.conf: Added a new entry for vmlinuz-ipfwd and made it the default. Ran lilo. 
<li>/etc/sysconfig/network: FORWARD_IPV4=true to make sure that IP forwarding is enabled at boot time by a write to /proc/net/ip_forward. 
</ul>For now gate1 will relay packets from anyone for network 20, but we can use ipfwadm later to restrict the eligible hosts and ports involved. 
 
<p>On the converter nodes /etc/syconfig/static-routes now defines  the routing from converter to Scanner Manager. 
 
<p>On b0l3boot-vx ~vxboot/all/startSm.vxsh defines the routing back from Scanner Manager to converter nodes (net 21). 
 
<p>All this works now and eliminates the VxWorks booting problem with the L3 farm at its full size. 
 
<p>While we were working on this we found that the entries in /etc/sysconfig/static-routes were out of date on most converter nodes: most of the processor nodes were not listed which forced the converters to use the default eth0 for their packets all the time. Boris will fix this. 
<p>
<!-- Author: Steve Tether -->