<!-- Title: 11:36:26  Tue Sep  5 2000 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Christoph Paus -->
<!-- Op3: Frank Wuerthwein -->
<!-- Op4: Ilya Kravchenko -->
<!-- Op5: Sasha Rakitine -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Run II hardware event builder


<!-- Date: Tue Sep 5 16:04:14 2000 -->
<pre>  Here are several e-mail messages that I have sent around  
lately concerning Event Builder operations. 
 
----------------------------------------------------- 
 
  Lately, we have been trying to understand causes of abnormal 
run terminations with hardware event builder, in particular, 
those with "Error reading VRB" from the event builder. 
 
  As Jim suggested, in several runs that stopped like that 
I looked into VRB FIFO using cdfvme_vme program. 
  - in one case, the "bad" VRB has no data at all 
  - in several other cases, the first 64 bits were zero, 
       followed by a few normal events 
 
How VRB readout is normally done: Steve Tether's code 
    attempts to read VRB 10 times with 1ms unterval. It 
    gives up if no data were found after 10ms and reports 
    the error. 
 
Today and yesterday evening, as Greg suggests, verbosity 
  levels on COT crates were turned to the level when statistics 
  is printed out every 10000 events on COT CPUs. This caused, 
  I believe, delays in data arriving to VRBs relative to trigger 
  message from TS/TM to Evb, so that Evb was complaining. 
  The runs were also stopping with software event builder. 
As an experiment, I have increased number of tries to read out 
VRB from 10 to 10000 for the Scanner CPUs (i.e., wait for 
data for 10 seconds while testing VRB). With this change, 
event builder was able sometimes to go through 10000/20000/... 
boundaries with trigger rate dropping to 0 for several seconds.  
10,000 tries is not going to be a normal mode for event builder,  
we are reverting back to 10 tries. 
 
  Last day's problems seem to have been caused by verbosity 
and debug printout on FE crates, and we observed triggers 
coming before data. This problem is cured by resetting  
the verbosity, but anyone can turn it on again. Also, 
we saw "Error loading VRB"  problem associated with b0htdc..  
crate before it was fixed on the FE side, and randomly with  
other crates. 
 
  At present, I do not see indications that Event Builder 
or VRB readout or VRB do wrong things. We do seem to see 
that triggers arrive before data appear at the VRBs because 
of different features of the Front End crates. It would be 
good to make trigger synchronization more robust or/and 
have experts available. So far I typically ask Jim(VRB) 
and Arnd/Greg(FE, ..XPT), please let me know if there are 
other people who are knowledgeable on this. Normally, 
it is difficult for me, Steve Tether or Ivan to say why data 
do not arrive. 
 
   Right now, running with "good" crates, we appear to be okay, 
we are using COTs, large section of Central Calorimeter and 
real trigger , rate is about 90Hz for about an hour. 
 
------------------------------------------------------------ 
 
 
   There are several issues that operations people might want to know. 
 
1. Halt Recover Run transitions fails from the TS side when  
Event Builder is included in run. This is possibly Trigger Manager 
problem, or something else. Greg wrote about it in the shift e-log. 
Will the leaders ask appropriate experts to look into it? 
 
2. I have observed one clear case today when a CSL problem 
(Mako had to restart afterwards) propagated all the way back 
and resulted in BUSY timeout from several FE crates. That  
might not be easy for an Ace to diagnose. Except that the next 
run will fail from the start. 
 
3. Occasionally we keep getting event fragments that are too large  
at the Event Builder side. More examples are recorded 
by Nancy in today's shift e-log. Once VRB data were as large  
as 71kB. This looks more like data corruption problem than 
real large-size events (i.e., a stuck bit/flipped in the data size 
in VRB header). Nobody is working on this problem now (not clear 
to me who should and how to understand it). 
 
-------------------------------------------------------- 
 
  By the end of this weekend, there are three problems 
appearing in runs with the event builder and level 3, I  
mentioned all of them earlier, this is to summarize: 
 
  1. "Error reading VRB" from Event Builder that suspends the run. 
 As you saw from Jim's mail, it is possible for trigger to 
 arrive before the data and it is not going to change, so we 
 will have to play with the time of VRB readout. Unless Steve 
 has something else in mind. 
 
  2. VME bus errors.  We have observed that in high-rate (~250-300Hz) 
 VRBs get out of synchronization (reformatter complains about VRB 
 word not consistent between SCPUs), after 5-20 thousand of events. 
 This is accompanied by bus error message in the corresponding VRB 
 crate. I saw this in a crate with b0eb14 that at that time had  
 only SCPU, scramnet module, and a single VRB. 
    Jim says that he will help us to update VRB firmware to see if 
 it makes any difference. 
 
   3. Tag mismatch error reported by converter nodes at high rates. 
--------------------------------------------------------------- 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Thu Sep 7 16:07:21 2000 -->
<p>It's now possible to have SCPU Zephyr messages relayed as Merlin encoded messages. Use cdfevb_scpu v3_17 and cdfevb_proxy v1_6. 
 
<p>This new SCPU code can also 
<ul> 
<li>Adjust the rate at which a VRB is polled for an event after a trigger is received. Global variable scpuVrbWaitMs is the total number of millisec to wait for data (default 10). scpuVrbRetryTicks is the number of system clock ticks to wait in between polls (default 1). The system clock rate is now set to 4 KHz. b0l3pcom1:~evbproxy/bin/setVrbDelays.exp is an Expect script which sets these variables in all SCPUs. 
<li> Dump the first 8 32-bit words of each loaded VRB's data whenever the SCPU buffer is overflowed. The dump is in hex. 
</ul> 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Sep 8 14:50:01 2000 -->
<p>cdfevb_sm v3_16 is now available. This version checks for gaps in event IDs from the Trigger Manager. It also waits only 5 seconds instead of 31 before trying to recontact SCPUs after they have been sent INITIALIZE (which happens after the EVB proxy sends new configuration info to the Scanner Manager). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Sep 13 10:35:22 2000 -->
<p>Fixed a bug in SCPU 3.17 that caused it to crash when the SCPU had no VRBs to read for an event. It was using a pointer to the last-used VRB object but that pointer was never being set since no VRB was used. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Sep 13 10:51:55 2000 -->
<p>These are now the current (default) releases of event builder-related software:<p> 
<table BORDER COLS=2 WIDTH="50%" NOSAVE > 
<tr NOSAVE> 
<td NOSAVE>cdfevb_sm</td> 
 
<td>v3_16</td> 
</tr> 
 
<tr> 
<td>cdfevb_scpu</td> 
 
<td>v3_17</td> 
</tr> 
 
<tr> 
<td>cdfevb_proxy</td> 
 
<td>v1_6</td> 
</tr> 
 
<tr> 
<td>cdfevb_ace</td> 
 
<td>v2_7</td> 
</tr> 
</table> 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Sep 13 16:50:07 2000 -->
<p>The experimental cdfevb_proxy v1_7 is now available for testing. It sends its own messages as encoded Merlin messages. Once this works the entire event builder will be using only encoded Merlin messages. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Thu Sep 14 13:07:02 2000 -->
<p>The experimental cdfevb_ace v2_8 is now available. It adds the button to reboot the SM and SCPUs then restart the EVB proxy. It also makes a click on the main window's close box act exactly the same as a click on the Exit button in the same window. 
<p> 
rebootVx.exp will have to be copied from $CDFEVB_ACE_DIR to the bin directory of user evbproxy before you try this release. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Sep 15 12:16:11 2000 -->
<p>I have changed /etc/inetd.conf and restarted inetd on gate1. I increased the limit of spawns-per-minute for service "shell" from 200 to 500. Looking inside /var/log/messages I saw that one VxWorks CPU makes about 17 rsh connections while rebooting, so booting all SCPUs + SM would cause 272 connections to be made and broken all pretty much within one minute. That exceeded the old limit of 200, which would cause inetd to ignore all rsh requests for ten minutes (as designed). A limit of 500 ought to let us boot all our VxWorks machines at the same time. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Sep 15 13:05:10 2000 -->
<p>Experimental cdfevb_scpu v3_18 is now available. Changes: 
<p> 
	init.c contains a string giving the SCPU code release number, e.g., 
	"3.18". scpuRelease() prints the string. 
<p> 
	transmit.c now defines a histogram showing how many events had a 
	given number of VRB read retries. 0 retries means the data was 
	already present. Events having VME errors don't get 
	counted. scpuVrbPrintHist() prints the histogram in tab-delimited 
	form. scpuVrbClearHist() resets all bins to zero. Also added the 
	global variable scpuVrbBeforeMs which contains the number of 
	millisec to wait in each event before trying to read the first 
	VRB. 
<p> 
	Added the Expect script setVrbDelays.exp to src/transmit. 
<p> 
	Reduced the event buffer size to 50K (51200) to match what's used 
	on the ATM card. This way a too-large event will always provoke 
	a dump of VRB header words. 
 
<p>  
<!-- Author: Steve Tether -->


<!-- Date: Mon Sep 18 17:26:46 2000 -->
<p>A new experimental trio of event builder products is available for testing on gate1: 
<ul> 
<li>cdfevb_sm v3_17 
<li>cdfevb_proxy v1_8 
<li>cdfevb_ace v2_9 
</ul> 
These three releases have to be used together. In addition, you'll have to install $CDFEVB_PROXY_DIR/EvbProxy_db in the Merlin central message database (then do it again after re-setting up the old proxy after tests are finished).<p> 
The proxy now remembers the run number and the Scanner Manager statistics collection and display has been given a facelift. The proxy sends a Merlin run summary message at the end of a run. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Sep 20 12:10:42 2000 -->
<pre> 
    As you know, we have been working on bus errors in  
VRB crates.  
 
   Current theory on these bus errors (i.e., my understanding 
  of Jim's explanations): 
 
Facts: 
  - bus errors occur while accessing the VRB at the correct 
  address (as seen by Jim with VME bus tool) 
 
  - after such event (and EVB halts at that moment), a partial 
  event is found in the VRB (as seen by Jim with cdfvme_vme tool) 
 
  - FE people tell us that an event is written into the tracer->VRB 
   piece by piece (bank by bank). The link control words are written 
   last. All the writing happens after Done is returned to TS. 
 
 
Observations: 
  - the problem is mostly happening in COT VRB crates and sometimes 
  in the past in muon VRB crate. 
 
  - the problem seems to be related to large events, slow-coming 
  events, FE-truncated events. 
 
  - VRB test utility does see bus errors occasionally,  
  in which case Jim just retries several times and it  
   appears to work well. 
 
What happens: 
  - possibly when Evb receives a trigger and goes to get that event, 
   not full event is in the VRB, so when we try to read it, 
   we get this bus error. Ideally, VRB is supposed be protected 
   from this happening. 
 
What can we do: 
  - reduce the severity of the problem by writing data faster 
   from the FE side (Arnd made changes in this direction, 
   removed some delays, etc. Still, we saw the problem after that.) 
    
  - run with meaningful (non-zero) thresholds on COT TDCs  
   again to reduce data size. 
 
  - protect from bus errors in the Event Builder. For example, 
   to retry reading several times after a bus error. (Everyone 
   except for me appears to think that this is easy). 
 
  - study problem more by trying to reproduce real data  
   evnironment/timing with a simple test program similar to  
   VRB test. Of course, this would take some work. Depending 
   on Jim's schedule and priorities... 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Tue Sep 26 15:44:34 2000 -->
<p>Experimental cdfevb_proxy v1_9 is available for testing. This version allows a Reset transition from any state and tries to give a run-summary message no matter how the run is stopped (End, Abort, Reset). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Sat Sep 30 15:13:39 2000 -->
<p>Experimental cdfevb_scpu v3_19 and cdfevb_vrb v3_6 are now availble via 
UPS setup on b0l3pcom1. See the ChangeLog files. A new Merlin message is 
defined so the file $CDFEVB_SCPU_DIR/src/msg/Scpu_db has to be 
"concatenated" before testing (and the old one restored afterward). The 
Ace control panel and EVB proxy must both be restarted after setup so 
that ZMConverter can have the new message definitions on its class path. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Oct 2 14:14:13 2000 -->
<p> 
I've changed cdfevb_scpu v3_19 so that it also tries to check the event ID in Tracer words against the TS event ID (DAQ VRBs only). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Thu Oct 5 21:54:03 2000 -->
<p>Experimental cdfevb_scpu v3_20 and cdfevb_vrb v3_7 are now available for testing on pcom1. More new Merlin messages. Fixed an erroneous calculation of the space available for VRB data in an event buffer (could have caused writing past the end of the buffer). VRB reading for an event continues after errors, though the event will still be thrown away. If a bus error occurs but no data was ever transferred, the SCPU acts as if an empty FIFO were found: it waits a bit and tries again. There is a good chance the next event will be found intact. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Oct 6 13:00:20 2000 -->
<p> 
cdfevb_scpu v3_20 and cdfevb_vrb v3_7 are now the defaults. 
<p>  
<!-- Author: Stephen Tether -->


<!-- Date: Sun Oct 8 12:43:52 2000 -->
<p> 
Lately we've been noticing complaints from the SCPUs about their not being able to find an event for sending to L3 (NoSuchEvent). Afterward the SM complains that the event in question has had a load pending for a long time, which indicates that the SCPU tried to send when it should have been loading. (Why then do we not always see a WrongWay message from the SM, telling us that the SCPU was trying to do the wrong thing?) It's possible that update messages are being lost resulting in the SCPU reading stale data from SCRAMNet memory. 
<p> 
After checking the error monitor logs I find that NoSuchEvent errors from 
SCPUs started appearing on September 21 in run 98898. Somehow I didn't 
notice them; guess I was distracted. 
 
I've made a small change to SCPU 3.20. It's not in CVS, just in the 
product directory. In src/control/control.c:readSMforever() I've moved 
the following three lines 
<pre> 
    semGive(sScramPoll); 
    msleep(2); 
    semTake(sScramPoll, WAIT_FOREVER); 
 
</pre>from before the call to pollSm() to between the calls to pollSM() and 
readSM(). This leaves the polling rate unchanged but puts the 2 msec 
delay between the detection of a new SCRAMNet message from the SM and 
the reading of that message. If, as I suspect, SCRAMNet update messages 
are getting lost or corrupted resulting in retransmissions by the 
originating nodes, this delay will give the retransmissions time in 
which to succeed and drastically reduce the frequency run disruption. 
<p> 
Of course we'll have to inspect cables, check SCRAMNet error histograms, 
yada yada yada. I will also work on making the SM send Stop Flow to the 
TM when NoSuchEvent and related errors occur.<p>  
<!-- Author: Steve Tether -->


<!-- Date: Wed Oct 18 16:26:51 2000 -->
<p>A new release, v3_8, of cdfevb_vrb has been created and is now the default for CDF running. This version reads the slow FIFO on the VRB because reading the fast FIFO resulted in corrupt data. The problem was most frequent by far in COT VRB crates but occurred once in a while in others. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Wed Oct 18 16;28;34 comment by...Steve -->
 <a href=/cgi/elog/elog.pl?nb=2000&action=view&page=181&anchor=14:41:18 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a> Shift E-log entry


<!-- Date: Wed Oct 25 14:50:38 2000 -->
<p>Since today was an access day I took the opportunity to track down the cause of the garbled and lost SCRAMNet messages. Eventually I noticed that the SCRAMNet modules on b0l3c06 and b0l3c09 had the Carrier Detect and Protocol Violation error bits on. They stayed on even after several resets of the modules. 
<p> 
Activating the bypass for b0l3c06 caused the problem to move down to b0l3c07, next in line after c06. Bypassing b0l3c05, which transmits to c06, just made things worse. Removing c05 completely from the loop made c06 happy. Something is probably wrong with the bypass module for c05. 
<p> 
Bypassing or unlooping c08 didn't make c09 happy, though. Here the problem seems to be in the SCRAMNet card itself (for c09). 
<p>I've unlooped both c05 and c09 so that data taking can continue while these nodes are being worked on. After doing that I was able to run a SCRAMNet test with SM + SCPUs for 20 minutes without seeing any errors. 
<p> 
During all this I've occasionally seen the Carrier Detect and Protocol Violation bits set for c12, but not always, and for now at least it doesn't seem to be a problem.  
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Oct 25 15:15:43 2000 -->
I've finally managed to find something wrong with the MVME2603 that used to be installed as b0eb15 in the event builder; I've kept it in my test crate.  I've had it shipped to the Motorola Computer Group for repairs (in warranty, Dispatch picked it up today). Here's an excerpt from the problem report I sent to Motorola: 
<pre> 
Hardware description 
-------------------- 
MVME2603-2121 serial no. FP41335 
 
Description of problem 
---------------------- 
DMA transfers from VME to local RAM corrupt the data. Every eighth byte has 
bit 3 forced to 1. The DMA engine was set to perform D32 non-BLT transfers. 
The problem persists when our PMC card was removed, 
when another VME memory module was used, and when the MVME was moved to  
another crate. There seems to be no problem with DMA writes to VME but we 
have not made exhaustive tests. Non-DMA transfers also seem to be 
OK but again we haven't made a lot of tests. 
 
Example of problem (OS is VxWorks 5.3.1) 
---------------------------------------- 
-> vmeMemoryCpuAddr=0xd3000000 
new symbol "vmeMemoryCpuAddr" added to symbol table. 
vmeMemoryCpuAddr = 0xf535e0: value = -754974720 = 0xd3000000 = __tx_dma_unwait + 0xd20abf58 
 
-> vmeMemoryVmeAddr=0x03000000 
new symbol "vmeMemoryVmeAddr" added to symbol table. 
vmeMemoryVmeAddr = 0xf535a0: value = 50331648 = 0x3000000 
 
-> testBuffer=malloc(1024) 
new symbol "testBuffer" added to symbol table. 
testBuffer = 0xf53560: value = 16759184 = 0xffb990 
 
-> nnn=1024 
new symbol "nnn" added to symbol table. 
nnn = 0xf53528: value = 1024 = 0x400 
 
-> bzero(testBuffer,nnn) 
value = 16760208 = 0xffbd90 
 
-> d testBuffer 
00ffb990:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffb9a0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffb9b0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffb9c0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffb9d0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffb9e0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffb9f0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba00:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba10:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba20:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba30:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba40:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba50:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba60:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba70:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
00ffba80:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
value = 21 = 0x15 
 
-> bzero(vmeMemoryCpuAddr,nnn) 
value = -754973696 = 0xd3000400 = __tx_dma_unwait + 0xd20ac358 
 
-> d vmeMemoryCpuAddr 
d3000000:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000010:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000020:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000030:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000040:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000050:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000060:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000070:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000080:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d3000090:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d30000a0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d30000b0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d30000c0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d30000d0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d30000e0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
d30000f0:  0000 0000 0000 0000 0000 0000 0000 0000   *................* 
value = 21 = 0x15 
 
-> vmeDma(testBuffer,vmeMemoryVmeAddr, &nnn, 0, 0, 0) 
value = 1 = 0x1 
 
-> d testBuffer 
00ffb990:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffb9a0:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffb9b0:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffb9c0:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffb9d0:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffb9e0:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffb9f0:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba00:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba10:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba20:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba30:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba40:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba50:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba60:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba70:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
00ffba80:  0000 0000 0000 0008 0000 0000 0000 0008   *................* 
value = 21 = 0x15 
</pre>
<!-- Author: Steve Tether -->


<!-- Date: Fri Oct 27 14:08:53 2000 -->
<p>I've started an overnight test with the SCRAMnet modules of b0l3c09 and c05 connected to themselves with 
<ul> 
<li>Bypasses disabled, 
<li>Write-own-slot mode, 
<li>Host-to-memory write disabled. 
</ul> 
With these settings the SCRAMNet memory is not updated at the time of writing but only when the module receives back the update message it generated for the write. I used ~tether/rwTest to reproduce the SCRAMNet usage of an SM and an SCPU: the "SM" writes a message about every 2 ms, the "SCPU" polls for then, reads them, and echos them in another region of SCRAMNet memory. The reading code histograms the bits in the error register CSR1 and prints the histogram every 20 sec or so. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Sat Oct 28 10:52:32 2000 -->
After running for more than 20 hours of the test described above, c05 had no errors. c09 had: 
<ul> 
<li>1 instance of Transmit Retry (CSR1 bit 9, garbled packet) 
<li>2 of Bad Message (bit 7) 
<li>2 of Protocol Violation (bit 5) 
</ul> 
So these errors still occur even when the module is connected only to the bypass switch and a six-inch fiber running from the bypass output to the input. 
<p> 
Now I'll start a test in which the bypass switches on both nodes are activated, eliminating the short pieces of external fiber from the loops. 
<p>
<!-- Author: Steve tether -->


<!-- Date: Sun Oct 29 16:43:43 2000 -->
<p>Still no errors found for c05. For c09 there were thousands of counts for Carrier Detect, Transmit Retry, and Timeout errors. There were millions of counts for Bad Message and Protocol Violation. Here is the final histogram for the bits in CSR1: 
<pre> 
Bit     Count 
 0          0 
 1    7943012 
 2          0 
 3          0 
 4          0 
 5   16447857 
 6       1327 
 7    8933362 
 8          0 
 9    1642617 
10      21548 
11       1147 
12        303 
13 1117108424 
14          0 
15          0 
</pre> 
<p>I'll now start tests for c09 in which the SCRAMNet module is connected directly to itself using a simple fiber instead of the bypass switch. Further tests on c05 will have to wait until another SCRANet module is available, since I want to test it in a loop with more than one module generating messages and it wouldn't be fair to test it together with the module in c09. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Oct 30 16:17:38 2000 -->
<p>The latest SCRAMNet test on b0l3c09 has run for about 24 hours with no errors. It looks as if something is fishy in its bypass switch. I'll now start a test in which c05 and c09 are hooked together without bypasses and in which both write to SCRAMNet at about the same rate (about 500 Hz). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Nov 1 11:23:07 2000 -->
<p>That last test turned up lots of errors on c09; a few on c05 as well could have been due to failures of c09. I've started a test in which c05 and a module in the VME test crate are hooked together. The VME SCRAMNet is known to be OK. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Wed Nov  1 11;31;47 comment by...Steve -->
Niehter module is using a bypass; connections are direct.


<!-- Date: Thu Nov 2 12:59:27 2000 -->
<p>No errors on c05 or the VME SCRAMNet for 24 hours. Now I'll do the same test but with c05 connected through its bypass.<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Nov 3 09:45:20 2000 -->
<p>c05's SCRAMNet passes with flying colors. Now I'm testing c09's module directly connected to the VME module.<P>
<!-- Author: Steve Tether -->


<!-- Date: Fri Nov 3 11:35:41 2000 -->
<p>It didn't take long for c09's SCRAMNet to show errors. Already I see 645 Protocol Violations (error bit 5), 42 Carrier Detect Failures (bit 6), 457 Bad Messages (bit 7), and 50 Transmit Retries (garbled packets, bit 9). When I stopped the test the counts for Protocol Violation and Bad Message were still incrementing by two to four counts every ten seconds. Each module was sending one message per millisecond so the error rate was 2 to 4 per 10,000 messages. 
<p> 
It's clear now that c09's SCRAMNet card is bad and should be replaced. Probably nothing is wrong with the corresponding bypass. I can't find anything at all wrong with c05's SCRAMNet equipment so c05 should be put back in the ring. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Nov 6 09:55:09 2000 -->
<pre>  New versions from Steve: 
 
cdfevb_sm    3.19 
cdfevb_ace   2.11 
 
These contain changes that make decoding of reformatter errors 
easier. I have made these to be the versions used by SM  
startup script and by evbproxy account. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon Nov 6 10:00:36 2000 -->
<pre>  Converter node c05 and c09 are both working. Converter c09 
had scramnet card replaced. </pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Fri Nov 17 14:38:54 2000 -->
<p>The MVME board I sent out on Oct. 25 came back around Nov. 3rd. I tested it and found that the problem hadn't been fixed. Apparently some technician ran a standard set of diagnostics, didn't find anything wrong, and they shipped the board back to us without trying to talk to us first. I complained to Julian De La Cruze, got a new RMA number, and had the packaged board picked up by Dispatch today. This time Motorola is going to have someone more knowledgable look at the sick board.<p>
<!-- Author: Steve Tether -->
<!-- Comment: Fri Nov 17 14;39;55 comment by...Steve -->
 <a href=/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=11&anchor=15:15:43 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a>Link to problem description.


<!-- Date: Fri Nov 17 14:49:05 2000 -->
<p>All the SCPU crates have had the 2300 boards placed in slot 1, the remote resets installed, and the 712M transition modules mounted using the new large front panels. By the way, the 712Ms are upside down on purpose; that way they don't stick out beyond the width of their slots. 
 
<p>All the SCPUs now boot properly and the Ace's Check SCRAMNet command returns OK for them all. On the first booting attempt b0eb12's Board Fault light didn't go out but the problem went away after I power-cycled the crate again. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Nov 27 14:32:17 2000 -->
<p>b0eb11 has been acting flakey again, so I checked the 5V supply fuse on the adapter and it was dark brown in the 
middle. I showed this to Fred Lewis who agreed that this was definitely 
NOT normal. Since this fuse keeps blowing Fred will replace the adapter, 
probably sometime this afternoon. I tested the CPU in the 6U crate 
upstairs and it worked fine. I think the old adapter is bad but we can't 
rule out a CPU drawing more power than the specs say or some sort of 
crate problem (although other cards in the crate don't seem to be affected). 
<p>  
<!-- Author: Steve Tether -->
<!-- Comment: Mon Nov 27 17;09;11 comment by...Steve -->
b0eb11 is back up with a new adapter.


<!-- Date: Mon Nov 27 19:09:14 2000 -->
<p>Ilya reported many complaints from the Scanner Manager when he tried to run an EVB test with simulated data late last week. Typical complaints were "unexpected error" and "SCPU loaded when it should have sent", etc. These are all complaints about SCRAMNet messages so it looked like a SCRAMNet problem. 
 
<p>I checked the error registers of all converter and SCPU SCRAMNet modules and found a persistent Carrier Detect error for b0eb17. In fact looking at the unit I could see the carrier detect light blink off every 5-10 seconds. The problem was that the front-panel transmission connectors for b0eb15's module were loose and one had nearly worked itself out. Acting on a hunch, I checked the connectors for all the other SCPUs and found one or more loose connectors on five more SCPU SCRAMNet modules. The people who moved the CPUs around a while back must know how to stick but not how to twist. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Nov 29 15:17:16 2000 -->
<pre>  Progress with the simulated data tests: 
 
  Now after hardware is back in order the tests work well. In  
the first series of tests I have not observed any EVB glitches 
at all. Here is a list of tests that I have done until now: 
 
Simulated data, 16kB fragment size, converters write to trash 
 
  Nscpu->Ncv    Duaration    Rate   Total Throughput   Total events 
     1->1         1h20min    65Hz       1MB/s             312k 
     4->15        5h30min   239Hz      15MB/s            4700k 
     8->15          15min   239Hz      30MB/s             200k 
     8->15        9h        239Hz      30MB/s            7700k 
 
Some observations/details: 
  - in the long 9h run I do not know if there were any complaints 
from the Event Builder because in the morning the system was taken 
over by other people. 
 
  - the difference between the 3d and 4th lines is in the rate division 
constant for SCPUs. It looks like we have reached the limit of the 
rate here. The limited factors are not understood. 
 
  Plan: do several long overnight runs watching for 1) scramnet glitches 
and 2)SCPU memory problems. Then perhaps look into the rate limiting 
factors. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon Dec 4 11:21:56 2000 -->
<pre>  More tests with simulated data have been done over the last 
days. The general conclusion is that the tests show no sign of  
any problems. 
   Here is a table of several biggest tests: 
 
Nscpu->Ncv   Time   Rate    size/spcu   Events  Throughput (total) 
  7->16       1h    239Hz     30kB       850k      49MB/s 
  7->16      6h15m  460Hz     12kB        10M      38MB/s 
  7->16      18h5m  342Hz     48kB        23M     112MB/s 
 
Observations/comments: 
  - no EVB failures of any kind observed 
 
  - in the results of previous days the rate was limited at 239Hz 
     regardless of data size. This is because of the fake TM max rate 
     at SysClkRate=60Hz. When I removed this limitation, max observed 
     rate went up to 460-470Hz. I have not been able to find the next 
     bottleneck so far. 
 
  - in the third test the data are sent at 16MB/SCPU, i.e. fiber-limited, 
    so we observe 340 Hz instead of 460 max rate 
 
  - I have also done order of 10 tests with varying data sizes and 
      rates of about 15 min duration while studying the rate (200-470Hz) 
      without seeing any problems. 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Sun Dec 17 18:27:31 2000 -->
<p>Today I tried to install one of the new IDE controllers and big disk on b0l3pcom1. I wasn't able to do it; too many things went wrong: 
<ul> 
<li>One of the new IDE cables was faulty, so the new drive didn't seem to be recognized. Finally the cable came apart, leaving me to pry the connector out of the IDE card. 
<li>Linux didn't like the partition table I'd set up on b0l3pcom4. I had to zero the first part of the new disk (on /dev/hde) using dd. 
<li>fdisk seemed to be able to make partitions on the new disk, but after I added the last partition and tried to write the table to disk, fdisk complained that writing to /dev/hde failed. After that I had to use dd again and start from scratch. I was able to define partitions 1 and 5-9 and write the table to disk, but saving the partition table would always fail after I added partition 10. Also, fdisk didn't seem to be able to recognize the right number of cylinders for the new disk, so I had to use expert mode and guess the correct number. Even when I used a number I knew grossly underestimated the disk's capacity I still had problems writing the partition table (I thought perhaps the last partition was running off the end of the disk, but apparently that's not what's wrong). By the way, during booting the kernel also doesn't get a correct number of cylinders. 
<li>I reconnected the ethernet cables, thinking that I would try copying a later version of fdisk to pcom1. During rebooting the 4-port ethernet card went dead (lights out). This happens if the new IDE card is installed and a drive is connected to it. It might be an IRQ conflict since both PCI cards are assigned to IRQ 11. Perhaps the one of the drivers doesn't want to share interrupts. Eventually I was able to log in and try using ifup to bring up eth0 and eth1, but all I got is a message saying "resource temorarily unavailable". I'm guessing that the resource is an IRQ. 
</ul> 
I've put the machine back in its original configuration. To make this work we may need to upgrade Linux, get a different ethernet card, or both. 
<p> 
FYI, the parameters to put in the "append" clause for LILO are 
<pre>     ide2=0x10a8,0x10a0 ide3=0x10b0,0x10a4 
</pre> 
The numbers are the first two I/O port addresses assigned by the BIOS, which you can see by cat-ing /proc/pci. IDE2 is function 0 and IDE3 is function 1. Look for "I/O at ...". This controller is vendor ID 1103, device ID 4. 
<p>  
<!-- Author: Steve Tether -->
<!-- Comment: Sun Dec 17 18;36;29 comment by...Steve -->
One more thing: The BIOS on the motherboard didn't recognize the BIOS on the new IDE card, so the BIOS setup dialog for the card never came up during booting, making it impossible to set device characteristics or choose a boot device on the card. I don't know what we do about this, a motherboard BIOS upgrade? The BIOS on b0pcmit04 gave us the dialog after printing a message about looking for Plug-and-Play devices. Perhaps the simplest thing would be to get new machines for the gateway nodes.



<!-- Date: Tue Dec 19 11:06:53 2000 -->
<p>The 5V fuse has blown again on b0eb11's 6U->9U adapter. The last time this happened (Nov 27) we replaced the adapter. It seems that we can expect about three weeks of operation per fuse. Either the CPU is drawing just a smidgen more current over the fuse's 5A rating or something is wrong with the crate. Fred Lewis replaced the fuse and after Christmas vacation will see about buying slightly heftier fuses. The next step up we have in stock is 10A. 
<p>
<!-- Author: Steve Tether -->