<!-- Title: 02:30:19  Sat Feb  1 2003 -->
<!-- Crewchief: Level3 team -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<h2><font color=orange size=+5><center><b>February</b> 2003</center></font></h2>


<!-- Date: Sat Feb 1 03:07:27 2003 -->
<br> 
<i>I have now completed the upgrade of the recovery tools for the farm. Below it is added the instructions for the procedure and  
explanation of tools, for <u>reference</u>.</i>  - Nuno 
<p> 
<h4>	 INSTRUCTIONS FOR RECOVERY</h4> 
    The procedure for restoring a filesystem on a node 
  involves the following steps.  
 
    <br><li> Go to remote recovery directory<br>  
       eg b0l3pcom2> cd /home/recovery/remote 
    <br><li> Specify filesystem information<br> 
       check/edit file recover.config-remote 
    <br><li> Check for existence of corresponding tarball<br> 
       ls tarballs; otherwise create one 
     <br><li>For Output and Converter nodes only: 
       re-plug Ethernet cable(A) on MB Ethernet port 
    <br><li> Check/edit recovery script if needed<br> 
       fixnode.all.remote script   
     <br><li>Execute recovery script on broken node<br> 
       insert (remote) floppy on the drive of broken  
       node and hit reset; 
       this will execute locally the remote script 
       named fixnode.all.remote on remote directory 
    <br><li> Remove floppy when done 
       after ~4min; or check with tail -f stat.log;       
       the procedure will wait until the floppy has been  
       removed from the node's drive; 
       also, for Output or Converter node, the Ethernet  
       cable should be re-plug into its original port 
    <br><li>Check that node is up<br> 
       ping and ssh; 
       for new nodes (212-276), a network problem is  
       corrected by executing the script network.fixit 
       stored on both /root/recovery/ and /cdf/recovery 
    <br><li> Distribute relay/l3 code<br> 
       this step is necessary only when the /cdf filesystem  
       has been restored 
       (see recovery web-page for how to do this) 
<p> 
 
 <h4>FILES AND LOCATION</h4> 
     The remote recovery directory is /home/recovery/remote/.<br> 
     A description of the main files follows.<br> 
 
     <li>  "fixnode.all.remote", 
        this is the actual file which is executed by the floppy 
 
      <br><li> "recover.config-remote", 
 	this file contains a map which specifies which filesystem  
        is to be recovered, and further needed information; 
        see its header for editing instructions; 
        it is read by the main script, and needs to be edited 
        before executing it. 
  
       <br><li>"ipmaclistL3-remote"  
        this contains a list of names, IP and MAC addresses of 
        each single node; 
        this list ought to always be up-to-date (for updating 
        instructions see below); 
        it is read by the main script. 
 
       <br><li>"sbin/sulogin" 
        this is a statically compiled version of a login binary; 
        by default it is not used (the system configuration of the 
        image has been modified so to avoid logging in); 
        it should be called however when a problem during the  
        execution of the main script takes place; 
        it can also be called by choice, by adding the line 
          /mnt/extdisk/remote/bin/sulogin 
        to the main script where convenient (what follows in the  
        script will not be executed); 
        the result of its call is the interruption of the procedure 
        by exiting to a user shell prompt on the local machine  
        (this has then the function of an uncustomized floppy) 
 
<br><li>       "stat.log" 
        this is the general recovery log-file; 
        it is automatically edited each time the procedure is used, 
        with information about the node and filesystems recovered. 
 
<br><li>       "tarballs/" 
	this directory should contain (at the time of execution) 
        the appropriate tarball with the filesystem to be installed; 
        correct description of these files should appear on the 
        file "recover.config-remote"; 
        some tarballs may be available on  
           b0l3gate2> /root/recovery/tarballs/ 
        in this case, the needed tarball still have to be temporarily  
        copied to the remote/tarballs directory. 
 
<br><li>        "store/" 
        this directory stores the working version of all files; 
        prior to modify any of the files, one ought to be sure 
        that a backup version is saved. 
<p> 
<i>Furher information available from the <a href="http://www-cdfonline.fnal.gov/ops/evbl3shift/pager/recovery/">recovery</a> / <a href="http://www-cdfonline.fnal.gov/ops/evbl3shift/evbl3pager.html">expert's</a> web-pages.</i>
<!-- Author: Nuno Leonardo -->


<!-- Date: Sun Feb 9 10:15:28 2003 -->
Script used by Ace Control Panel actions (<i>l3_all_nodes_exe</i>) 
continues giving problems. 
<li>it hangs in arbitrary subfarms, without timing out 
<li>it seems not to execute in all nodes (in <i>l3_node_list</i>) 
<br>  
<!-- Author: Nuno -->


<!-- Date: Sun Feb 9 10:21:19 2003 -->
Output node b0l3u02 hung this morning.  
In var/log/messages one reads: 
<font size=-1> 
<pre> 
Feb  9 04:02:04 b0l3u02 syslogd 1.3-3: restart. 
Feb  9 04:22:11 b0l3u02 kernel: hda: lost interrupt  
Feb  9 04:22:21 b0l3u02 kernel: hda: irq timeout: status=0x80 { Busy }  
Feb  9 04:22:22 b0l3u02 kernel: ide0: reset: success  
Feb  9 09:04:25 b0l3u02 syslogd 1.3-3: restart. 
Feb  9 09:04:25 b0l3u02 syslog: syslogd startup succeeded 
</pre> 
</font>
<!-- Author: Nuno -->


<!-- Date: Wed Feb 12 11:23:43 2003 -->
<p>I've built a new kernel for b0l3pcom1 and have reconfigured LILO to use it the next time the machine is rebooted. The new kernel is /boot/vmlinux-nocookie and is just like the old kernel vmlinux-ipfwd except that the networking option "SYN cookies" has been turned off. This option is supposed to protect against so-called SYN flooding attacks in which an attacker begins opening lots of connections at once by sending SYN packets but never completing the opening process. The trouble is that we get many false positives from the L3 farm resulting in lots of bogus warning messages in the system log 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Thu Feb 13 12;28;25 comment by...Steve -->
By the way, the false alarms were caused by times when many L3 processor nodes were making nearly simultaneous requests of the ypserv daemon running on pcom1. This sometimes results in a large backlog of packets in between some random node's SYN packet starting a TCP opening handshake and its SYNACK packet completing the handshake. A large number of packets backlogged with no SYNACK yet seen to match a given SYN is interpreted as a possible SYN flooding attack if SYN cookies are enabled in the kernel.</i>


<!-- Date: Thu Feb 13 12:29:10 2003 -->
Are italics off yet?
<!-- Author: Steve -->
<!-- Comment: Thu Feb 13 18;54;29 comment by...Nuno -->
</i></i></i>Are <i>italics</i> still around...?



<!-- Date: Thu Feb 20 11:54:54 2003 -->
<a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=152&frame=2&anchor=10:54:49'> 
Comment </a>  by Ilya concerning errors in the executables at the beginning of the run.
<!-- Author: Guillelmo -->


<!-- Date: Tue Mar 4 17:00:45 2003 -->
<pre>HD_REGULAR_FCHECK:  
Rebooting of the full farm (Processor, Converter, Output nodes).  
In the following 31 nodes the root filesystem had been repaired. 
b0l3005,b0l3006,b0l3009,b0l3011,b0l3015,b0l3020,b0l3021, 
b0l3029,b0l3073,b0l3122,b0l3127,b0l3130,b0l3135,b0l3138, 
b0l3139,b0l3144,b0l3146,b0l3149,b0l3150,b0l3153,b0l3162, 
b0l3163,b0l3169,b0l3173,b0l3175,b0l3176,b0l3187,b0l3190, 
b0l3192,b0l3200,b0l3203 
 
b0l3256 didn't boot up. It will be sent to repair.</pre>
<!-- Author: Alberto,Nuno,Guillelmo,Arkadiy -->


<!-- Date: Tue Mar 11 20:43:21 2003 -->
<pre>Today I updated tcsh in the two gateways: 
b0l3pcom1 now has 6.11.00 (prev. 6.07) 
b0l3pcom2 now has 6.12.00 (prev. 6.10) 
The source code and the configuration files produces, as well as 
various logs, are in b0l3gate1(2):/tmp/tcsh/</pre>
<!-- Author: Alberto -->


<!-- Date: Tue Apr 8 18:32:34 2003 -->
<pre>Backup drive replaced on pcom2.  
The hard drive was taken from b0l3219. 
/home/ and /cdf/ synchronized with pcom1 by running by hand the script copyuser.sh in /root/synchronize/. 
It took a couple of hours to copy /home/ and it's still working on /cdf/ (~1 hour total time needed for this partition, hdc10). 
</pre>
<!-- Author: Alberto/Guillelmo/Nuno -->


<!-- Date: Wed Apr 9 16:17:35 2003 -->
<pre>I upgraded rpm and glibc on c02 and processor nodes 019->034 
(subfarm 02). 
rpm-3.0.1 -> rpm-3.0.5-9 
glibc-2.0.2 (I guess) -> glibc-2.1.3-22</pre>
<!-- Author: Alberto -->


<!-- Date: Wed Apr 9 17:59:37 2003 -->
I tried to upgrade rpm and glibc on c02 too, but it didn't work. I needed to install popt-1.4.1 and bzip2-0.9.5c; bzip needs glib, and glib can't be installed because it conflics with util-linux-2.7-15. I tried forcing the installation (the file which causes the conflict is /sbin/sln) and after that I couldn't log in to b0l3c02 from the gateways (while it was still possible to log in from other nodes). So, I moved sln to /sbin/sln.bak and copied /sbin/sln from c03. After rebooting c02 everything was fine. I gave up the idea of running fermitest on converters, I installed by hand nettest and I'm running only that test.
<!-- Author: Alberto -->


<!-- Date: Fri Apr 11 16:39:55 2003 -->
<pre>As required, I restarted the cron jobs responsible for the backup of /cdf/ and /home/ partitions on the backup disk in pcom2.</pre>
<!-- Author: Alberto -->


<!-- Date: Sun Apr 13 06:52:50 2003 -->
DAQ ACEs can't find the error message from Level-3 corresponding to a reported filter SEVERE error.  See  
<a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=307&frame=2&anchor=06:31:19.942'> this link</a>.<p> 
Is this a gap in training/documentation, or is the error really not passed on by the control code?
<!-- Author: Kevin -->
<!-- Comment: Fri Apr 18 13;26;59 comment by...Nuno -->
<i>Below follows a related comment sent by Ilya to the Aces' list:</i>
<br>
  Looking back trough e-logs we see that last Sunday there was some 
confusion about severe filter errors. The question was how to find
out what processor nodes these errors happen on. 
<br>Once a severe filter error occurs, the corresponding error message
is sent to level3 display. These messages are counted by the counters
on the bottom-right. Also, the name of the processor node (3-digit
number of the corresponding box on the GUI) turns red. In order to
see the text of the error messages one has to click on that box
and a new window will appear with a text field.
  <br>Note that the processor node names turn red not only because of
the filter errors, but also because of reformatter errors and filter
crashes, in all cases there is a corresponding text message.
   <br>
   This information can be found in Nuno's tutorial talk for the Aces
given last week. For those who would like to review it, it is available
on our main page
<a href=http://www-cdfonline.fnal.gov/ops/evbl3shift/evbl3shift.html>http://www-cdfonline.fnal.gov/evbl3shift/evbl3shift.html</a>.
Ilya
<!-- Date: Thu Apr 17 23:58:22 2003 -->
<pre>HD_REGULAR_FCHECK: 
Farm maintenace rebooting; 29 nodes failed fsck. 
002, 010, 012, 013, 014, 017, 122, 124, 125, 130,  
132, 137, 138, 139, 144, 146, 149, 150, 151, 159,  
163, 169, 170, 1178, 180, 194, 199 
Restored hda6 in all, and hda5 in: 125, 139, 176 
 
I have produced 6 floppies after fixing the problem 
associated with simultaneous recovery instances. 
 
This took place on Tuesday, April 15th.</pre>
<!-- Author: Nuno -->


<!-- Date: Fri Apr 18 00:08:56 2003 -->
<pre>CONVERTERs' recovery. 
 
Note the following when recovering filesystems on a Converter 
node. Convertes use an older kernel version, which does not  
support the default options of the newer versions (in particular  
the one used in the recovery procedure). As a result you need to 
set the proper option when creating filesystems on these nodes. 
Namely, and as noted on the main recovery script, the following  
options to mke2fs should be set: 
/usr/bin/mke2fs  -O none /dev/$fsname 
(this will disable the sparse_super and filetype features). 
 
We figured this out recently with Steve when attempting to 
recover c02. 
</pre>
<!-- Author: Nuno -->


<!-- Date: Fri Apr 18 00:14:45 2003 -->
<pre>Problems and fixes to be noted: 
 
-c02 
  problem with atm connection when taking data; 
  swapped ATM cards with c01. 
 
-223 
  problem with Ethernet card 
  after recovering filesystems, noticed network was unreachable; 
  activated second Ethernet port  
  it boots up properly now, but un-desirable in principle 
  (farm non-uniformity)  
  </pre>
<!-- Author: Nuno, Guillelmo -->



<!-- Date: Wed Apr 30 15:51:48 2003 -->
<pre>Updated all the rootfs_###.tar.gz files in tarballs.reservoir, adding the correct /etc/hosts file. Also updated rootfs_242.tar, in /home/recovery/remote/tarballs/</pre>
<!-- Author: Alberto -->


<!-- Date: Wed Apr 30 15:57:31 2003 -->
<pre>Node 011 still giving hard disk problems. After rebooting, it was not possible to log in. Errors in hda1 and hda5. 
I restored, using Nuno's remote floppy procedure, all the partitions, then distributed filter and relay software. Now it is possible to log in. I'll check if the usual hda irq errors appear again. If they do, it might be needed to ask for the help of HelpDesk again. 
In Control Room they didn't notice I was tarring/untarring partitions (recovering one node only).</pre>
<!-- Author: Alberto -->


<!-- Date: Tue May 6 19:33:16 2003 -->
<pre>HD_REGULAR_FCHECK: 
Farm rebooted. Nodes which failed: 
5,17,21,29,126,127,130(hda5,hda1),134,139(hda5),144,
146(hda1,hda5),148,150,166,169,176,185,197,198,203. 
Also 6 new nodes were recovered, though they might just 
have been (really) slow to come up after rebooting: 
217,220,227,236,241,253. 
 
NOTE: one floppy doesn't set the correct permissions as for 
/cdf/log and /cdf/level3. </pre>
<!-- Author: Guillelmo/Alberto -->
<!-- Comment: Wed May  7 07;46;41 comment by...Guillelmo -->
<pre>The problem actually is in the tar file, not in the floppy.</pre>


<!-- Date: Mon May 12 17:11:59 2003 -->
<pre>I have finished preparing a few level3 nodes  
which have been repaired recently. 
These include: 
- 011 
  this new had its hard drive replaced 
  i repartioned the disk and installed 
   the proper filesystems 
- 195 
  this had the power supply replaced 
- 251 
  updated sshd_config 
  it showed problems during booting complaining 
  that hda1 is mounted, and therefore aborts... 
  i've re-installed the root fs and specified  
  boot device in bios; problems do not seem to 
  appear anymore after several consecutive rebootings 
- 223 
  this node is also up, although it had shown 
  ethernet problems in one port sometime ago 
All these nodes are ready to be included online  
(first for some testing runs). 
Nuno</pre>
<!-- Author: Nuno -->


<!-- Date: Fri May 16 19:23:36 2003 -->
<pre>Node 251 behaved misteriously in the last days. Rebooting the node different error messages appeared; when we tried to run l3_node it the node hung. The last error I received was caused by a read() failure of gpm. I checked if gpm (mouse stuff) was consistently installed in all the nodes and I found that the farm is (was) quite inhomogeneous. In ~l3proxy/gpmlog.log the output of /sbin/chkconfig --list | grep gpm is listed, in ~l3proxy/gpmlocatelog.log I put the output of "locate gpm". In the new nodes, a few of them had gpm service running and a few more files than the other nodes. I removed the files (except /etc/rc.d/init.d/gpm, in some of them: I kept it just for taking a look at what it does exactly). As for the old nodes, gpm service is on, level 2 and 4 (on the new nodes with the service on, levels 2 to 5 were on; maybe the problem is with level 3 and 5 on, I don't know what they refer to).</pre>
<!-- Author: Alberto -->


<!-- Date: Mon May 19 21:16:14 2003 -->
<pre>During coldstart (before L2 torture), phase 3 (distributing packages), the L3 display turned red and got stuck for some 20 minutes. I killed by hand the RelayService and cleaned up the L3 farm. Then, we could partition and coldstart again. The log file is b0l3pcom2:$L3_LOGDIR/l3proxy/l3proxy.out_20035160834.  
There's a break due to a segmentation violation right after l3_checksum checked all the total+l3 tarballs. 
 
By the way, during the L3 clean-up nodes 43 and 73 couldn't be reached and timed out. I could log in in bothb of them and we had no problems when we tried to coldstart.</pre>
<!-- Author: Alberto -->


<!-- Date: Wed May 21 15:55:38 2003 -->
<p>I've installed in ~evbproxy/test/ace a new checkScram.exp script that checks converter nodes, SCPUs, the Scanner Manager and the Trigger Manager. It displays descriptions of which error bits are on, etc. It's also installed in the current Ace control panel running out of Alberto's home directory. The renamed old version is available in the same places. 
<p> 
We tried the script and it worked several times. To check the converters it relies on user evbproxy being able to ssh to a converter node as user l3proxy using RSA authentication. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed May 28 19:16:29 2003 -->
<pre>ACE_COSMICS_ALL 
Select single, random crate 
Calib Continuous 
Auto L2 Accept  
Uncheck the expert's UseScaler option,  
if it doesn't work and no one around  
can propose a better solution. </pre>
<!-- Author: Nuno / Alberto -->


<!-- Date: Wed May 28 19:21:17 2003 -->
<pre>Marked b0l3119 offline. 
This node was taken by Steve for  
Gateway related work.  
</pre>
<!-- Author: Nuno/Alberto -->


<!-- Date: Wed May 28 19:39:46 2003 -->
<pre>Included 13 nodes in the farm. 
Recovered partitions on the following: 
017, 021, 138, 179, 156, 153, 173, 195, 217, 219 
003 was returned from repair this afternoon 
223, 251 run network fixing problem (network.fixit) 
Two nodes are not included at the moment: 
238 awaiting return from repair (for many weeks) 
248 blank screen, may require repair 
Fixed directory permissions on tarballs.</pre>
<!-- Author: Nuno/Alberto -->


<!-- Date: Sat May 31 02:28:11 2003 -->
<pre>L3 farm up after power shutdown. I needed to recover nodes 
13,18,120,122,133,156,195. hda6 (/ partition) failed disk check. 
I kept the converter node c12 off because it emits a weird sound. 
After startup, also node 9 sounded strange, maybe the fan was not working well. It is fine now. 
Node 238 finally came back. I'm about to repartition the disks and test the node.  
I also removed c12 from the HDdatabase for the time being.</pre>
<!-- Author: Alberto -->
<!-- Comment: Sat May 31 04;41;24 comment by...Alberto -->
<pre>Node 238 recovered. Disk partitioned, stuff copied....
By the way, cdf_273.tar is a 1.6GB tarball. It contain executables and lots of control_2_X_X_debug directories, which are useless. I put the node online</pre>

<!-- Comment: Sat May 31 06;47;06 comment by...Alberto -->
<pre>Node 248 has been picked up.</pre>


<!-- Date: Sun Jun 1 13:02:27 2003 -->
<pre>Node 245 was unpingable. Monitoring of farm 15 was lost. Actually the node was up and running, with a 'nfs b0l3saerv unreacheable' message at the login prompt. I replugged the ethernet cable and  
everything was fine. The ace was taking a L2 torture using the first four subfarms only. When I took another look at L3 display after replugging the cables, the node turned light blue... </pre>
<!-- Author: Alberto -->


<!-- Date: Tue Jun 3 12:16:06 2003 -->
<pre>I just put online nodes50 and 248. 246 is up and apparently fine after a night spent cooling. 251 gives an error message, when logging as l3proxy, which is surely related to the problems Nuno and I had copying the tarball usr_273.tar  
It apparently cannot find "setup" and ups commands, which are likely to be just missing. I'll try again to restore /usr partition.</pre>
<!-- Author: Alberto -->


<!-- Date: Tue Jun 3 12:53:35 2003 -->
<pre>248: it was impossible to start the filter executables. 
total+l3.tar was corrupted, ~11Mb instead of 47Mb. I removed it and copied the tarball from another node.</pre>
<!-- Author: Alberto -->


<!-- Date: Mon Jun 16 16:48:28 2003 -->
<p>Last Friday (June 13) Guillelmo and I tried to install a former processor node running Fermi Linux 7.3.1 as the new gateway #1. After many hours work we put the old gateway back in. Here's a summary of what happened: 
<ul> 
<li>The Generic Desktop installation of FL 7.3.1 doesn't include ypserv or expect. Installed them by hand from the CD-ROMs. 
<li>Had to add an entry for rpc.statd to /etc/hosts.allow. 
<li>The ntpd program now requires a user named "ntp" (ID 38) and 
group "ntp" (also 38). 
<li>Just after ntpd was started on the new gateway, starting xntpd on a converter node failed. A few minutes later it worked. 
<li>Changing from one NFS server to another requires that NFS disks on the clients be first unmounted then remounted. Otherwise the client's mounts appear to work but you get lots of "statfs error=5" messages if you try something even as simple as "df". 
<li>The EvbProxy requires the presence of the directory /cdf/log/cdfevb owned and writable by user evbproxy. 
<li>There must be a symbolic link /log -> /cdf/log. 
<li>UPS setup of jdk v1_4_0 failed because we didn't have a symbolic link /cdf/code-Linux -> /cdf/code-Linux-2.0.34. 
<li>Zephyr needed to be rebuilt because zwgc program would not run due to a missing shared library. 
<li>The current class files in cdfevb_proxy won't run under jdk v1_4_0 and the old jdk v1_1_7 won't run under 7.3.1. When we tried to recompile we got lots of error messages from javac. 
</ul> 
So much for trying to switch in a new gateway quickly! Now it's a longer-term project involving code changes to the EVB proxy. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Tue Jun 17 10:29:17 2003 -->
<pre>Status of the level3 nodes: 
 
119 - taken out forever  
195 - chronic, taken out 
223 - chronic, taken out 
226 - was off last week with blank monitor ,  
went up fine this morning after cooling down,  
and i included it   
238 - chronic, taken out  
246 - the same as 226 
248 - the same as 226 
251 - chronic , taken out    
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Jun 20 18:14:33 2003 -->
<p>The new gateway b0l3pcom3 is kerberized and connected to the KVM switch with the other two. After sourcing /local/ups/etc/setups.cshrc one can setup jdk v1_4_0 and then compile and run a Hello World program. That means we can start to work seeing why the EVB proxy won't compile. 
<p> 
I had to modify /etc/exports on pcom1 to export to pcom3 over the public net as pcom3 has no connection to the private networks (we'd have to run a long cable). 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jun 20 21:21:59 2003 -->
<i>L3 farm maintenance update performed this afternoon:</i> 
<p> 
Recovered filesystems: 223 (all), 248 (all), 251 (usr); 
<br>new cdf partition tarball, w/ maneageable size, has been made. 
<p> 
Request for complete replacement sent for nodes 238, 223. 
<p> 
Individual status for nodes currently offline (all atipa's): 
<br> 
<li>238: replacement requested 
<li>223: replacement requested; appears fine 
<li>251: appears okay 
<li>226: appears okay (had blank screen) 
<li>246: blank screen 
<li>248: gets shortly stuck 
<p> 
Notes: 
<li>all mentioned six nodes are for now being left offline 
<li>nodes with seemingly heating problems have now been  
    reseated on nearby shelves 
<li>monitor cable needs urgent replacement 
<p>
<!-- Author: Nuno -->


<!-- Date: Wed Jun 25 00:10:49 2003 -->
<pre>Opened nodes 226,246,251 to check for broken fans. The interior 
of the three nodes looks fine. Additionally, nodes 226 and 251 were up and running, while 246 was stuck (blank screen; we rebooted it without any improvement; we turned off the node waiting for it to cool). 
</pre>
<!-- Author: Alberto/Jeff -->
<!-- Comment: Wed Jun 25 00;35;48 comment by...Alberto/Jeff -->
We are in the middle of a shot setup, final protons will be injected soon. We didn't have a chance to test any node. We tried to put 36,178,195,226,251 online, but the clean up script got stuck before starting l3proxy (the error message is
[1] +Exit 1 /home/ceballos/checkout/level3/control/bin/Linux+2.2.14-1.3.0f2smp/l3proxy -b......)
We guess the symbolic link to 2.2.14-1.3.0f2smp is missing, but we couldn't find it, wasting the few minutes we could run.



<!-- Date: Wed Jun 25 13:21:03 2003 -->
We are making a <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=527&frame=2&anchor=13:14:11.313'> 
test run </a> 
using nodes 36,195,226 and 251. 223 and 238 will never been used (people don't feel comfortable using them) and we'll try to replace them entirely. Node 178 died after a shout from the inside (and a nasty smell coming out from its back side...). We placed a repair request for the node and put it in the small table ion front of the rack of new nodes. 
246: we turned it on after a 12-hours sleep but it is still too warm. The screen turned blank during disk check, or soon after.  
We plan to let it rest for 24 hour and try to turn it on tomorrow morning.
<!-- Author: Alberto/Jeff -->


<!-- Date: Fri Jun 27 13:31:17 2003 -->
<pre>Farm summary: 
36: BIST error during boot up. If it concerns anyone, we'll place a repair request for the node 
192,193: to be put onlina as soon as possible 
178: being repaired 
226,246: being tested for overheating (now dead) 
248,251: being tested with l3_node 
238: bad errors on the screen (unable to handle interrupts....)-  we rebooted the pc and it looks fine, there's something wrong inside, didn't we ecide to try to have the node replaced? 
 
</pre>
<!-- Author: Alberto/Jeffrey -->
<!-- Comment: Fri Jun 27 14;05;57 comment by...Alberto -->
<pre>195: being tested with l3_node; it will be probably put online with 192 and 193 (and 36, if anyone has complaints)</pre>


<!-- Date: Mon Jun 30 19:11:12 2003 -->
<pre>node 178 is back. They changed the power supply and "levelled down"  
the hard drive, whatever that means. I restored the three partitions (the disk was empty) and checked that it works. I'm running l3_node standalone. 
Today I also sent a couple of repair requests: 246 (overheating) and 36 (processor 2 fails some self-tests). 
226 died this night but someone booted it up in the afternoon. If it dies again this week I'd place a repair request (identical to the one for 246). 
251 has been showing new error messages for the whole afternoon. I restored the root partition yesterday and it had problems: it booted up as localhost and when I entered the command "hostname b0l3251" it started showing "Unable to handle paging request; kernel panic" errors.</pre>
<!-- Author: Alberto -->


<!-- Date: Wed Jul 2 14:48:28 2003 -->
<p>I've replaced checkScram.exp in version 2.20 of cdfevb_ace. Both the online CVS repository and the /cdf/prodcts copy now have the updated script. It's essentially the same as the one we've been using in b0l3pcom1:~evbproxy/test/ace/; I made only a few minor changes to comments. 
<p>Now any checkout of release v2_20 will contain the new script; I didn't think it worth while to make a new release just for this. 
<p>The new script checks all nodes in the SCRAMNet ring: SCPUs, converter nodes, the Scanner Manager and the TriggerManager. It prints the names of any error bits that it finds are on. Comments in the script provide an explanation of what the bits mean. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Jul 2 18:02:10 2003 -->
<p>I noticed that the syslog daemon was not running on b0l3pcom1. Apparently it was not started when I did "telinit 3" from single-user mode a few days ago (after fixing a Zephyr problem). 
<p>After I restarted the logger I saw a lot of messages of messages about various servers being out of memory.  Pcom1 was rebooted at my request. 
<p>Before the reboot I noticed that the xntpd time server was not running on either pcom1 or pcom2. The ntpdate program in the startup script complained about not finding suitable servers to sync to at startup. The list of servers to try is in /etc/ntp/step-tickers which I set to contain "vlan302.r-cdf-online-fnal.gov", the translation of 131.225.236.200 which is the router for the 236 subnet. Since for security reasons pcom1 may not use DNS I had to put an entry for the router in /etc/hosts. After these changes I got the time servers running again. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Jul 2 19:24:33 2003 -->
<p>I've created the following new product versions in CVS and registered them with UPS: cdfevb_proxy v1_16, cdfevb_sm v3_24a, and cdfevb_scpu v3_28a. Don't make these versions current; they are intended for the replacement for pcom1. 
<p>The SM and SCPU versions are in branch "old_evb" bases on versions v2_24 and v3_28 respectively. This branch will be used for bug fixes and minor improvments while major changes will be made on the main branch. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Sep 15 18;12;49 comment by...Steve -->
By the way the reason for the new versions is to fix the problem we had building these packages using JDK 1.4.0 and later. The compiler no longer allows importing from the unnamed package.

<!-- Date: Fri Jul 11 17:47:33 2003 -->
<p>On the first try Orbacus for Java 3.3.2 didn't build when using the java compiler in jdk v1_4_0. The problem was that the Java libraries now come with the packages org.omg.* in $JDK_DIR/lib/server.jar. There have been some changes to the CORBA binding for Java since Orbacus 3.3.2 which make them incompatible with some Orbacus source files. 
<p>The solution is to make javac and java use the org.omg.* class files that come with Orbacus. Go to the Make.rules file in the Orbacus config/ directory and add the following to JAVACFLAGS: <b>-bootclasspath     $(JDK_DIR)/lib/core.jar:$(JDK_DIR)/lib/graphics.jar</b>    . The CORBA classes will be found using the CLASSPATH set in the Orbacus make files. 
<p>After I built Orbacus like this I tried the Hello demo and it worked. 
<p>
<!-- Author: Steve Tether -->