<!-- Title: 13:47:02  Fri Oct  1 1999 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Christoph Paus -->
<!-- Op3: Ilya Kravchenko -->
<!-- Op4: Sasha Rakitine -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Run II hardware event builder


<!-- Date: Tue Oct 12 11:29:50 1999 -->
<h3>First floor SCPU crates</h3> 
15 SCPUs have been installed on the first floor. Only the ones in the four SVX crates that we've been using have working ethernet (the top crates in 1RR21 F through I). 
 
<p>The bottom crates in 1RR21 F-I, which are also SVX crates, have no +12V power, which Pete Wilson told me will not be rectified until the production power supplies are installed. 
 
<p>All DAQ crates, 1RR21 C-E, have no +12V coming to the ethernet port in the 712M transition modules mounted in the backs of the crates. This is because the 6U-9U adapter cards with the DC-DC converters, as they are now,  don't make any connections from the CPU to the user-defined pins in rows A and C of the P2 connector. There is a place on each card for a set of jumper posts for each row, but no posts have yet been installed. Pete Wilson knows of this omission (in fact discovered it) and he will talk to the techs about putting in the jumper posts. Then we will have to put in the jumpers.  Just to be sure we should install all of them. 
 
<p>The ISL crate in 1RR18I also has the problem of no jumpers on the adapter card, so no ethernet there either. 
 
<p>Two crates need adapters for SCRAMNet modules. All the others have been installed. 
 
<h3>Third floor power</h3> 
The power in the two banks of  outlets under the floor near our shelves was turned on yesterday afternoon. 
 
<h3>SCRAMNet</h3> 
The ring is complete (bypasses are installed even where modules are not). 
 
<h3>ATM</h3> 
All SCPUs and converter nodes have been hooked up. Sasha found a problem with one ATM cable. The outer covering of the cable from the b0l3c14 doesn't reach all the way to the connector at the switch end. He wrapped it up with an electrical tape on the chance that it may still work. 
 
<h3>Local area network</h3> 
Our private ethernet has been extended to the first floor, and each SCPU now has its own 100baseT connection instead of having to share a thin-wire daisy-chain.  
 
<p>b0l3pcom1 has been brought back up to the third floor and installed next to the old Sun workstation. Those SCPUs with working ethernet can boot from it, and converter nodes can mount the NFS disk with home directories an can run ypbind successfully. 
 
<p>Most processor nodes and all output nodes remain to be connected.  
<!-- Author: Steve -->
<!-- Comment: Tue Oct 12 11;32;08 comment by...Steve -->
This entry was made on Oct 12, not the first. The E-log apparently assumes that the first entry in a file is on the same date that the file was created.


<!-- Date: Tue Oct 12 13:58:40 1999 -->
<p>The 11 SCPUs without working ethernet have been removed by Ilya and me. The CPU modules are in our cabinet on the third floor. Their adapters have gone back to the high rise to have the jumper posts added starting this afternoon. We expect them back in a couple of days. 
 
<p>The final power supplies for the first-floor SCPU crates are due to be installed "soon". The actual installation schedule is complicated by the presence of FIB modules in the lower SVX crates. They need -5V, which will not be available once the final power supplies are installed. Pete Wilson has said that if installation is delayed by more than a few days, he'll have the 12V supplies in the SVX racks hooked up to both crates in each rack. We can get away with this because only the 2600 CPU needs 12V in each crate, and the supply is rated at 24 amps. 
 
<p>One of the new SCRAMNet modules appears to have developed a defect. If you change the contents of its memory, it broadcasts the change message repeatedly instead of just once. You can see the Message Waiting light stay on continuously (this means that there is an outgoing message queued for sending). Also, the other SCRAMNet modules pick up the messages, so any changes you make to the affected locations in their memories get overridden almost at once. We'll look into this further. 
<p>
<!-- Author: Steve -->


<!-- Date: Tue Oct 12 15:32:47 1999 -->
The SCRAMNet module that we thought was bad works just fine in a smaller ring with only four nodes. In the big ring it looks as if the module was timing out waiting for its change message to come back to it. When this happens the module re-transmits the message. There are a couple of ways in which a large ring could be responsible for this kind of failure: 
<ol> 
<li>Too many consecutive bypasses are in use. Each bypass attenuates the signal by about 2 dB they just pass light through without using an amplifier. The SCRAMNet Media User's Guide says not to have  more than  four or so bypasses active in a row. With so many SCPUs removed to have their adapters modified we easily exceed that limit. 
<li>The time it takes a message to go all the way around the ring is too long. Each node and each 50 m of cable delays the message by about 240 ns. Register CSR5 can be used to set how long a module will wait before retransmitting, in units of 240 ns. 
</ol> 
Booting more converter nodes upstairs and cutting the bypasses for the DAQ SCPUs out of the ring leaves use with at most five consecutive bypasses engaged. This seems to work for now. I will alter the SCRAMNet reset procedure on converter nodes and SCPUs to set a decent timeout value for the full-sized ring. I can't see what's being used now because the timeout setting is write-only. 
 
<p>FYI, the above info indicates the messages travel at about 0.7c in the fiber-optic cable. Also, the figure for node delay is for new SCRAMNet modules. The old ones required something like 800 ns. 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Oct 15 18:39:06 1999 -->
I've updated the SCRAMNet software on converter nodes 1-15. /usr/local/bin/scram_reset now calculates the SCRAMNet ID number from the converter node's name, 128 + nn for b0l3cnn. The timeout value is now set to 50. 
 
<p>I've changed the scramnet_reset routine for VxWorks to also set timeout to 50. 
 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Oct 15 20:17:42 1999 -->
We got back six 6U->9U adapter cards with jumpers installed for the DAQ SCPU crates. Ilya re-installed the SCPUs (b0eb11-16). All but b0eb15 have working ethernet now. In the one bad crate, +12V is still not coming out to pin C7 of the P2 connector in back of slot 1. Could be the adapter or CPU is badly seated, could be a missing jumper or a bad solder. We can't power down the crate now to check because someone is using it (Petar, I think). 
<p>
<!-- Author: Steve -->


<!-- Date: Mon Oct 18 15:25:25 1999 -->
From Ilya: 
 
<p><pre> 
1. Scramnet ring. Now everything is connected to the ring, 
   all the bypass switches are in but not all CPUs are connected 
   to bypasses. Here is the list of nodes and their status: 
     b0l3c01, c03-c05, c08, c09, c11-c14 - ok 
     b0l3c02, c07, c10 - problems.  
                Hang when anything is sent through ATM 
     b0l3c06 - did not bother to  check, this is where PCI  
                problems were observed 
     b0l3c15 - a different problem. Scramnet is not working. Lights 
        "insert" and "carrier detect" are on, but when it is in, 
         the system acts as if the ring is broken. I checked all  
         the fibers around, they seem ok. Finally, I pulled out 
         the thick white cable of bypass for this node so it is 
         not in scramnet ring. 
     b0l3c16 - it seems that all the hardware is in place, but scramnet 
         software is not set up and host tables, ATM not as well. 
 
     b0eb10  - scanner manager is ok 
 
     b0eb12-17, 19, 21, 23 are ok 
     b0eb11, 18, 20, 22, 24, 25 are still missing adapters and right  
            power supplies. 
<p> 
   I have tested all the above sending simulated data in 1 SCPU -> 1 converter 
    configuration.  
<p> 
 
2. In my last email I mentioned that the event rate in my latest tests 
    is lower than expected. Eventially, I tried to use Christoph's 
    sm.a and scpu.o, and the rate was back to 86 Hz while with mine 
    I get 75. Will look at it tomorrow. (My only consious change to 
    the code was MAX_FRC = 24) 
<p> 
3. Not everything is set up for b0l3pcom1. When I log in I get 
this message: "Unable to initialize the UPSII environment". I 
can not do "setup xxx". Also, I was not able to recompile SM and SCPU 
code: "/bin/sh: makedepend: command not found". I guess, I will have to  
bug Andreas. 
</pre> 
<p> 
 
Ilya has also noticed that power cycling one of the first floor VIPA crates may cause the SCPU to lose its boot parameters. Petar and Steve Nahn have noticed the same problem with the SVX CPUs.
<!-- Author: Steve -->


<!-- Date: Mon Oct 18 15:28:34 1999 -->
We are to have 16 sub-farms after all, so I will install the needed software on b0l3c16 today. 
 
<p>Also in progress: Andreas and Jeff are changing host names and IP assignments in order to separate b0l3serv/b0l3pcom1 from b0l3001. 
 
<p>
<!-- Author: Steve -->


<!-- Date: Mon Oct 18 16:47:17 1999 -->
I've just set up b0l3c16 as a proper converter node. While at it I noticed that c02-05 and c12-15 were running version 1.3 of the nicstar driver instead of 1.4. No harm done, the only difference is the addition of CRC error reporting. All converters are running 1.4 now. 
<p>
<!-- Author: Steve -->


<!-- Date: Tue Oct 19 18:31:36 1999 -->
Fred Lewis replaced the faulty adapter for b0eb11, and now I can log in to that SCPU. 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Oct 22 14:29:36 1999 -->
<h3>Code changes to SM, SCPU, and L3 receiver. Coming soon!</h3> 
 
<h4>Communication with Trigger Manager</h4> 
It looks like we'll be trying this next Thursday. 
 
<ol> 
<li><b>(Done)</b> Add TM messaging to the SM. This step also adds new header 
files describing system limits, SCRAMNet messages, SCRAMNet memory 
layout, etc., to the SM inc/ directory. Also in this directory are new 
versions of sm_scpu.h and l3recv.h which are redefined in terms of the 
new header files. 
 
<li>Change the makefiles and rebuild scripts for the the SCPU and L3 
receiver code so that, at a minimum, they look to the SM inc/ directory 
for l3recv.h and sm_scpu.h. Recompile both packages. 
 
<li>Have the SCPU code copy the last four 32-bit words from each 
SCRAMNet LOAD_EVENT message to the place reserved for them in the 
appropriate fragment header. 
</ol> 
 
<h4>Communication with Run Control</h4> 
<ol> 
<li>Alter the UCI BEGIN_RUN message format accepted by the SM so that 
VRB channel masks are included as well as VRB index numbers. 
 
<li>Alter the SCRAMNet DEFINE_PART (begin run) message format used by SM 
and SCPU so that VRB channel masks are sent as well as the bitmask 
showing which VRB slots to use. 
</ol> 
 
<h4>Bug fixes</h4> 
<ol> 
<li>Let the SPU read VRBs whose slot number is greater than fifteen. The 
changes needed have been found, we just have to try them out. 
 
<li>Remove the buffer-deleting watchdog from the SCPU. 
 
<li>When the SCPU receives a FLUSH_DAQ SCRAMNet message from the SM, it 
should only delete the events belonging to the specified partition. Now 
it deletes the events from all partitions. 
</ol> 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Oct 22 16:05:01 1999 -->
Nine more 6U-9U adapters for 2600 CPUs are going to have the notches put in on Monday morning. Installing these will free two old adapters which we can then use for SCRAMNet. Then all SCPUs and SCRAMNet modules on the first floors will have an adapter of some kind. We can then take our time replacing old adapters. 
<p>The notches are required in order to leave clearance for a couple of small mezzanine boards attached to the motherboard. One could force them in but this puts a lot of stress on the mezzanine connectors. 
<p>
<!-- Author: Steve -->


<!-- Date: Thu Oct 28 12:30:35 1999 -->
Yesterday Jeff and I put into the online repository versions of the Scanner Manager, SCPU, and L3 receiver code that use the new, longer SCRAMNet messages. Packages affected: sm, scpu, l3recv, seqio. The SM understands Trigger Manager commands, as it has done for some time. The SCPU code now saves the last four 32-bit words of the LOAD_EVENT message from the SM in the fragment header. A trial run using the SM, one SCPU, and one converter ran correctly using a simulated TM running as a task in the SM. Although the code has been committed we haven't tagged it yet, there may still be bugs to fix. 
<p> 
We had the first attempt today at making the real Trigger Manager talk to the Scanner Manager via SCRAMNet, with one SCPU and one converter node. The SCPU was set to generate fake events. Due to a blunder on our part we loaded the wrong seqio code for the SM and SCPU, so only the first message between any two nodes was picked up properly. The first event message from the TM was picked up, the SM told the SCPU to send it, and the SCPU sent it. We'll try again tomorrow, when we'll have use of the event builder again. 
<p> 
We did see a real bug, though. At first, the TM was sending trigger messages for partition 8, while the event builder was set up for partition 1 only. This caused the SM dispatcher, task smDisp, to crash in the function SM_Disp_Trigger. Apparently the immediate cause of the crash was an attempt to jump to address 0. We'll have to look over the code to make sure that messages for undefined partitions are rejected properly. 
<p>
<!-- Author: Steve -->
<!-- Comment: Thu Oct 28 13;26;39 comment by...Steve -->
SM_Disp_Trigger() didn't check for a NULL partition object in the "partlist" array, which is how undefined partitions are marked. I've put in the check.

<!-- Comment: Thu Oct 28 13;30;12 comment by...Steve -->
By the way, the TM software numbers partitions from 0 to 7, whereas we use 1-8. We both agree on the 0-7 numbering in the TM<->SM messages, so no harm done. You just have to be careful when you ask a TM person for triggers in a certain partition. The control kludge only works with partition no. 1 (our numbering).


<!-- Date: Thu Oct 28 16:39:09 1999 -->
We tried the TM-to-SM stuff again this afternoon, this time loading the right libraries into the event builder VME CPUs. Worked like a charm on multiple events, which got all the way to the converter node. The only remaining problem is on the TM side: the replies from the SM are not being detected, although we can see them in the right places in the TM SCRAMNet. 
<p>
<!-- Author: Steve -->


<!-- Date: Fri Oct 29 11:08:00 1999 -->
The event builder and L3 farm are back up after this morning's blackout. Running are 
<ul> 
<li>SCRAMNet and most Scanner CPUs (with power in their crates). 
<li>Converter nodes 1-15. 16 appears to be off the SCRAMNet. 
<li>Processor nodes 1-30. 31 has apparently never been set up. 
<li>Output nodes 1 and 2. 3 and 4 still think that they are converter nodes 3 and 4. 
<li>ATM switch. 
<li>Scanner Manager. 
</ul>
<!-- Author: Steve -->


<!-- Date: Fri Oct 29 13:04:54 1999 -->
Zhaoou found the problem in the TM code, so now we have run normal triggers via the Trigger Manager with no problem. We also crashed the SCPU in the middle of a run and saw the Scanner Manager send a Stop Flow message to the TM.<p>
<!-- Author: Steve -->