<!-- Title: 15:51:37  Fri Jan 12 2001 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Ilya Kravchenko -->
<!-- Op3: Christoph Paus -->
<!-- Op4: Frank Wuerthwein -->
<!-- Op5: Sasha Rakitin -->
<!-- Op6: Ivan Furic -->
<!-- Notes: -->
<pre>Run II hardware event builder 
</pre>


<!-- Date: Fri Jan 12 15:57:35 2001 -->
<p>I've just finished the paperwork for PCI SCRAMNet 1096/7319 and have called dispatch to pick it up. Now I'll have to do it all again for PCI SCRAMNet 2206/11112, which keeps the PC it's in from booting (even the BIOS setup screen doesn't appear). We tried it in several PCs of different makes, they all behave in the same way. This card was in converter c09; luckily we had just received two new spare boards so we were able to put c09 back in service right away. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Jan 15 10:42:20 2001 -->
<p>Last weekend I was able to upgrade the BIOS in b0l3pcom1 from version P11 to version P14. This BIOS seems to do a much better job of auto-detecting the operating parameters of IDE disks. The only thing I had to do in the Main BIOS menu was to turn on 32-bit IDE transfers; other settings such as PIO mode were already correct. Under the Advanced BIOS menu I changed the Use MultiProcessor Specification back to the value of 1.4; it was changed to 1.1 because the last step in upgrading the BIOS is use F9 to cause the new BIOS to change all settings to the default. See http://support.intel.com/support/motherboards/server/N440BX. 
 
<p>I did have to make one slight change to the upgrade procedure. BIOS P14 requires the the so-called Board Management Controller chip be running version 57 of its firmware. After unpacking the BMC upgrade materials onto a 1.44 MB floppy you're supposed to make the floppy bootable using the SYS command in DOS, in order to use the CONFIG.SYS and AUTOEXEC.BAT automatically when you boot from the floppy. Alas, there was no room for the invisible MSDOS.SYS and IO.SYS files that the SYS command tries to write. I had to create a blank bootable floppy, then copy CONFIG.SYS and AUTOEXEC.BAT onto it. After getting to the upgrade menu when booting from this floppy I inserted the upgrade floppy, which now needs no more than COMMAND.COM. 
 
<p>Once the BIOS was upgraded Linux had to be made to recognize really large IDE disks. The IDE code for 2.0.25 is hopelessly outdated; I had to upgrade the Linux kernel to 2.0.38 in order to apply a patch for IDE disks larger than about 34 GB. This isn't a Fermilab kernel but comes straight from ftp.kernel.org. The only problem there was that the tarball was compressed with bzip2 which is not installed on pcom1; I had to expand it on pcom2. The large disk patch comes from ftp://ftp.kernel.org/pub/linux/kernel/people/aeb/2.0.38-largediskpatch. 
 
<p>The networking for this kernel is configured with Tulip chip support built in (not a module) and all other PCI and ISA ethernet card support disabled. eth0 and eth1 become the rightmost and second-rightmost ports on the Adaptec 4-port card, as they should. This Tulip driver seems to function well except for one funny glitch at boot-up: a message appears for each port saying that the transmitter has stopped. After that it pauses for about a minute, then resumes after sayting that full duplex operation has been negotiated. There don't seem to be any problems after that. Perhaps a later version of the Tulip driver can be installed at some other time. I tried the very latest one from www.scyld.com/network/tulip.html but I wan't able to get it to work properly. That driver is supposed to work for series 2.0 and 2.2 kernels but something appears to be wrong with the 2.0 compatibility macros. It might be worth while to get a later tulip.c directly from an later 2.0 kernel from ftp.kernel.org. 
 
<p>Since Tulip support is built into the new kernel, I've removed the loading of the 2.0.35-2 Tulip module that was done in /etc/rc.d/rc.sysinit. Also, I've had to alias the module names net-pf-4 and net-pf-5 to "off" in /etc/conf.modules in order to suppress messages at boot time that modprobe cannot find these modules. These are the kernel's names for IPX and Appletalk protocol modules, which the kernel tries to load even though support for these protocols was turned off in the kernel configuration. 
 
<p>The compressed image for the new kernel is /boot/zImage-2.0.38. In /etc/lilo.conf this kernel is given the name "bigdisk" and is now the default. 
 
<p>The new kernel has not had ATM patches applied so at least for now converter node software must be compiled on a converter node. 
 
<p>Next weekend I'll do the BMC+BIOS upgrade for b0l3pcom2. It's running a 2.2.14 kernel, which according to the Large Disk HOWTO has support for IDE disks larger than 34 GB already in place. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Jan 15 11;43;42 comment by...ST -->
<pre>I forgot to say that the new disk is attached as /dev/hdb. fdisk and cfdisk recognize the partition table I set up on b0pcmit04 and the kernel was able to create a filesystem on the one partition I tried that on. The new disk is empty at present.</pre>
<!-- Date: Mon Jan 15 10:50:06 2001 -->
<p>b0l3030 is now out of commission, since I've removed its hard disk to take it to Eternal Graphics for copying to the new processor nodes. Last weekend I added the bencmarking code to /root/benchmarks and the sensor-reading code to /cdf/sdr (Sensor Data record). That way I'll be able to get our performance figures for CPU and disk as well as seeing if the SDR code will run unchanged on a Lancewood motherboard (of course, even if the interface is the same the sensor ID numbers may be different for the new chassis). 
<p>
<!-- Author: Steve Tether -->



<!-- Date: Mon Jan 15 12:17:35 2001 -->
<p>SCPU release 3.22 is now available on b0l3pcom1. This version has some extra checking of the marker words around the Tracer word for non-SVX VRB data. It also has simplified code for large fragments (not yet activated) which expects the ATM driver to cut up large transmissions. 
<p>  
<!-- Author: Steve Tether -->
<!-- Comment: Tue Jan 16 11;21;22 comment by...ST -->
Since the value of the EOF word in non-Si data is unreliable in certain circumstances I've remove the check for it from SCPU 3.22.



<!-- Date: Sat Jan 20 16:57:36 2001 -->
<p>I had to make some changes to system files on pcom1 for security reasons. See the link below. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Sat Jan 20 16;59;15 comment by...ST -->
 <a href=/cgi/elog/det-system-elog.pl?nb=l3&action=view&page=4&anchor=16:50:31 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a>


<!-- Date: Sat Jan 20 17:18:14 2001 -->
<p>cdfevb_l3recv v3_7 for large fragments is now UPS declared on pcom1. v3_6, which didn't work, is deleted. The problem with low reception rates was caused by the thread that was waiting for event data to arrive; it didn't release the lock it had on the Builder object before going to sleep. The reception threads need to acquire that lock before setting the data-arrival flag, so there was deadlock unless the reception thread got the lock first. The deadlock was eventually cleared by the event timeout but this enormously slowed the reception rate. 
<p>
<!-- Author: Steve Tether -->

<!-- Date: Mon Jan 22 11:32:14 2001 -->
<p>This is a test of making entries via e-mail.<p>
<!-- Author: Steve Tether -->


<!-- Date: Tue Jan 23 12:22:27 2001 -->
<p>Experimental cdfevb_proxy v1_11 is now available via UPS on pcom1 as
the new development version (sertup -d ...). 
Changes:
<ul>
<li>New EvbHeartbeat message with state and event-backlog information.
<li>Corrected transition table entries for Reset.
<li>The DaqMsg rundb.HardEvbSummary is sent at the end of a run.
For this to work $RUNDB_DIR/classes must be on CLASSPATH.
</ul><p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jan 26 12:24:49 2001 -->
<p>Release 2.12 of the Ace control panel is now available on pcom1 as
cdfevb_ace v2_12 (development chain). This version adds a check by
pinging of the CPUs in the ATM switch. Any operations started from the
main panel are done in threads other than the GUI thread, so the GUI
will update even if the program is busy.<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jan 26 18:38:59 2001 -->
<p>The following event builder packages are for debugging large fragments:
<ul>
<li>
cdfevb_i4515 v3_4 - the VxWorks ATM driver,</li>

<li>
cdfevb_scpu v3_23,</li>

<li>
cdfevb_l3recv v3_7,</li>

<li>
cdfevb_sm v3_20.</li>
</ul>
For large fragments the fragment header has changed, see the typedef fragmentHeader
in the SM include file fragment.h. The reformatter will have to skip over
the new header. The new header has a count of the number of packets used
for the fragment which is set properly by the SCPU (it calls an ATM driver
routine to do the calculation). Data should be extracted from the header
by casting a pointer to the beginning of the fragment to type fragmentHeader*
then using the header fields by name. That way a change in the header means
you only have to recompile not change source code.<p>
<!-- Author: Steve Tether -->



<!-- Date: Wed Jan 31 16:09:06 2001 -->
<p>Fixed a problem with fragment header alignment in SM 3.20 then recompiled SCPU 3.23. Reintroduced MAX_FRAGMENT_PACKET_SIZE in SM 3.20.<p>
<!-- Author: Steve Tether -->

<!-- Date: Wed Jan 31 17:49:10 2001 -->
<p>I won't be recreating user testbed since a user with a null password
is a security risk. To get a Zephyr window while logged on to gate1, run
the script /usr/local/bin/greenscreen.<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Feb  5 12;48;52 comment by...patrick -->
<pre>Doesn't seem to exist:

[evbproxy@b0l3gate1 ~]$ cd /usr/local
[evbproxy@b0l3gate1 local]$ find . -name "*green*" -print
[evbproxy@b0l3gate1 local]$</pre>


<!-- Date: Mon Feb 5 16:04:03 2001 -->
<p>Over the weekend I switched pcom1 back to booting from its old 6 GB disk. Forgot to copy the greenscreen stuff to the old /usr; it's there now. 
<p>pcom1 now gets / and /usr from the old disk /dev/hda. /tmp, /home, and /cdf come from the new disk /dev/hdc. I've removed from the machine the previous /dev/hdc which had /home. 
<p>I was going to repartiiton /dev/hda to make a new / (with /usr) and a large /tmp, but during one reboot I started getting hundreds of e2fsck errors for the /cdf on /dev/hdc. I thought that this was a delayed reaction to some problem related to the large size of the new disk or to the power outages we had late last week. I was well into "correcting" the errors one-by-one when the penny dropped: I had thoughtlessly typed "linux s" at the LILO prompt instead of "bigdisk s". That booted a really old kernel we had that of course hadn't been patched to handle large IDE disks. When I booted the bigdisk kernel I found that the primary superblock for the partition was corrupt. Luckily I was able to run e2fsck with the -b option to use a duplicate superblock. Saved! It seems no data was lost. I've removed from /etc/lilo.conf all mention of kernels other than the correct one, now labelled "linux" instead of "bigdisk". 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Feb 5 16:12:19 2001 -->
<p>Converter nodes c11 and c12 are now out of the main SCRAMNet ring and form a 2-node ring of their own. I'll use this to test a PCI SCRAMNet that has come back from the "repair shop". 
<p>The MVME2603 that had the DMA problem has come back cured at last. I had to send it in to Motorola twice because the first time their standard test procedure didn't detect the fault. Finally an engineer at Motorola had to borrow our VxWorks kernel and DMA software. Replacing the Universe II chip made the problem go away. Last time I checked, Motorola programmers were going over the standard test code to see why it missed this problem. Heh, heh, heh. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Feb  5 16;18;44 comment by...ST -->
Last e-log entry for the <a href=
'/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=11&frame=2&anchor=14:38:54'>
MVME2603 DMA problem. </a>

<!-- Comment: Mon Feb  5 16;24;19 comment by...ST -->
Last e-log entry for the <a href=
'/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=11&frame=2&anchor=11:35:41'>PCI SCRAMNet problem.</a> 



<!-- Date: Mon Feb 5 18:07:08 2001 -->
<p>I've just tested the "repaired" SCRAMNet board and it is if anything worse than it was 
before. I put it in c12 which was in a 2-node ring with c11, and I didn't even 
have to try sending any messages before things went wrong. It was 
continually dropping the carrier and I could even see by eye the carrier 
detect light blinking from time to time. I tried replacing this board with 
the one we had in c12 before: no problems. That makes it unlikely that the cabling or bypasses are at fault. I've sent e-mail to Kellie Kobes at Systran. 
<p>  
<!-- Author: Steve Tether -->


<!-- Date: Tue Feb 6 18:07:59 2001 -->
<pre>We had several problems with the SCPUs today: 
 
 1. Jim Patrick has updated flash rams for b0vrbXX cpus, 
 in doing that the processor number assigned to those 
 became 0. The default processor numbers are 
      1 - monitoring cpu (b0vrbXX, b0svxXX, those are 
          in the first slots) 
      2 - our SCPUs. Currently in slot 3 everywhere. 
 When the processor number is wrong, the memory mapping 
 changes. The effect of this as far as we can tell was that 
  one could not write to SCRAMNet memory on the affected 
 crates. 
 
  2. We have observed some SCPUs going into eternal reboot 
  loop. First, the affected SCPU was b0eb18, then b0eb17 
  while b0eb18 became normal. 
    The reboot would happen when atm code was loaded and 
  started. We would get queue-overflow-kernel-panic message 
  and it would keep rebooting. It would boot with atm stuff 
  commented out. Actions taken: 
     - powercycle crate: did not help 
     - replace ATM card: did not help 
     - reseat adapter in crate: did not help 
     - take b0eb17 to the test (6U) crate upstares: booted ok 
     - reseat b0eb17 back into its adapter/crate: all works 
  Steve says: a) possibly bad contact, b)looks like atm card, 
      when started, would give interrupts at too high rate  
      for the system to handle (12V power problem?) 
   Now everything works, but we do not exactly understand what  
      was wrong. 
 
</pre>
<!-- Author: Ilya -->


<!-- Date: Sun Feb 11 21:01:08 2001 -->
<p>I've repartitoned the old 6 GB disk on b0l3pcom1. It now holds only a root partition, a large /tmp, and swap space. It's the primary IDE master (/dev/hda) and the default boot disk. 
<p>The new, 46 GB disk has the /home and /cdf partitions, plus duplicates of the system disk partitions. It's the secondary IDE master (/dev/hdc).  
<p>To make the system boot from the big disk, bring the system down and shut it off, then take the IDE cable from the small disk (lower bay) and plug it into the big disk (upper bay). This will make the big disk /dev/hda. Reboot. 
<p>Both disks are set up to be bootable only as /dev/hda. LILO will only boot properly from the first hard disk the BIOS sees, and this BIOS will always see /dev/hda first if any such disk is connected . You can see from the BIOS boot page the ordering assigned to the disks. There is a Boot menu in the BIOS setup which sets which master boot record to use but which doesn't cause the BIOS to change the ordering of the disks as LILO wants it to change.<p>
<!-- Author: Steve Tether -->


<!-- Date: Thu Feb 22 15:37:22 2001 -->
<p>We definitely have an SCPU memory leak associated with the Partition transition (Grand Init). When an SCPU gets down to about 1 MB of free memory it starts to complain about not being able to connect to converter nodes. The leak seems to get larger per partition transition when the number of converter nodes increases. Today Ilya and I did a "memShow" and "checkStack" between various transitions over and over, saving the results in a file. This will at least tell us how big the leak can get. 
<p>One thing we already found out is that the stack allocation for each "tSend" process is grossly oversized at 64K. "checkStack" shows that even after several Activate transitions they never used more than 1K of stack space each. If we reduce the allocation per task to 2K  it should leave a good margin of safety and save up to 2.9 MB, assuming three Builders per converter, 16 converters, and one tSend task per Builder. We can reduce the stack allocation right away, which will give us some breathing room while we look for the cause of the leak. 
<p> 
<!-- Author: Steve Tether -->
<!-- Comment: Thu Feb 22 19;35;25 comment by...ST -->
Each pair of transitions Reset+Partition leaks about 9,600 bytes for each Builder used in L3. We currently run with three Builders per converter node. No other transitions show a memory leak.


<!-- Date: Mon Feb 26 17:23:00 2001 -->
<pre>  We seem to have understood the problem with dying zwgc 
that prevents Run Control from seeing error messages from  
the Event Builder. 
 
  When we restart EVB proxy, we start zwgc piped to  
ZMConverter, and zwgc creates temporary file /tmp/wg.877 
that contains process ID of the zwgc program. 
  If someone else opens the green zephyr window after that, 
the new instance of zwgc rewrites the /tmp/wg.877 file  
and puts there the process ID of the new zwgc. When this 
green window is closed, the file wg.877 is deleted. At the 
same time zwgc originally started along with EvbProxy is still 
running. At some point it realizes that the file is gone and 
crashes. 
 
  We will have to think how to go around this problem.</pre>
<!-- Author: Ilya -->


<!-- Date: Tue Feb 27 21:37:14 2001 -->
<p>We've squashed some bugs over the last 4-5 days: 
<ol> 
<li>The memory leak in the SCPUs. By tracing all the malloc() and free() calls we found <em>two</em> leaks. One was the needless reallocation of the ScpuEventList object every time a sub-farm was registered, instead of just once. That lost more than 26K each time. The other leak was eight bytes for every Zephyr message sent (a message descriptor was not freed). Updated cdfevb_scpu v3_23 and cdfevb_vxZephyr v3_2. 
<li>The problem with zwgc's files in /tmp. Now the green-screen launcher and the proxyops.py script set the environment variable WGFILE to a more suitable temp-file name before running zwgc. The name is "/tmp/zwgc." followed by a process ID. Updated /usr/local/greenscreen and cdfevb_proxy v1_11. 
<li>A deadlock in the VxWorks ATM driver. Yesterday it was found that when using a fragment size of about 1K or less in excess of the ATM packet buffer size caused the sending tasks in the SCPU to hang in the driver. The driver's sending function went to sleep to wait for some hardware resources but didn't release the global hardware locking semaphore beforehand, thus keeping the interrupt service task from freeing those same resources. Updated cdfevb_i4515 v3_4. 
</ol><p>  
<!-- Author: Steve Tether -->


<!-- Date: Wed Feb 28 20:25:02 2001 -->
<P>Testing the making of log entries from FNAL machines not on the B0 sub-nets (e.g., TACACS modem connection). If you see this then it worked. 
<P>
<!-- Author: Steve Tether -->


<!-- Date: Sun Mar 4 13:59:06 2001 -->
<pre>I have rebooted b0l3pcom1 because there seemed to be problems  
with the internal network. When SCPUs were rebooted all at once, 
most failed to boot. Rebooting b0l3pcom1 solved this problem.</pre>
<!-- Author: Ilya -->


<!-- Date: Fri Mar 9 18:16:51 2001 -->
<p>Some Zephyr cleanup today. First, the executables for zephyrd and zhm started by rc.local on b0l3pcom1 were not the official ones from /usr/products/zephyr/v2_0_4/sbin. One has to use our official Zephyr binaries because the location of configuration files is hardwired into the code at build time. Second, the server list in $ZEPHYR_DIR/etc/zephyr contained the names mit2 and b0pcmit03 as well as b0l3gate1. Every 15 minutes zephyrd would try to contact the other servers to check if any messages were to be forwarded either way. As b0pcmit03 is no longer running a zephyrd the message "Connection refused from 192.168.24.11" would appear in /var/log/messages. I've removed the other server names from the list since there's no good reason to have EVB messages forwarded, one can always log into pcom1. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Mar 12 14:37:11 2001 -->
<p> 
Ivan, Arkadii, and I were able to reproduce the SCPU booting problem by telling all the SCPUs to reboot then quickly disconnecting b0l3pcom1 from the private network. In theory this should have had no effect; instead most SCPUs hung and those that continued to boot went through the process <em>very</em> slowly. So in some way the boot host b0l3backup was still relying on pcom1. Ivan suggested NIS (ypbind) as a possible culprit. We therefore killed ypbind on b0l3backup (and unmounted the NFS filesystems, too). This seemed to clear up the problem; after rebooting all SCPUs by hand we were able to reboot them all normally using the same Expect script that the Ace control panel uses.<p> 
Now b0l3backup is permanently configured not to start ypbind or do NFS mounts. This means that the only recognized users are root and vxboot. The only remaining use of pcom1 as a server is via xntpd.<p> 
We don't know what configuration file was being requested from pcom1. Our /etc/nsswitch.conf puts "files" first for hosts and group. 
<p>
<!-- Author: Steve Tether -->