<!-- Title: 18:33:19  Mon Jan 14 2002 -->
<!-- Crewchief: Level3 people -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<h2><font color=navy><center> January 2002</center></font></h2>


<!-- Date: Mon Jan 14 18:49:58 2002 -->
<pre> 
I made two resque tarballs on b0l3242.  
 
/mnt/recovery/hda1.tar 
/mnt/recovery/hda5.tar 
 
Together with tarred hda6: 
/mnt/recovery/rootfs.tar 
 
they make a complete recovery  
set for Level3 node. the recovery  
procedure is analogous to one for hda6.</pre> 
<a href = "http://mit.fnal.gov/~iyutin/L3EVB/notes/disk_fix.html">Procedure </a> 
<pre> 
hda1 tarball does not contain Level3 Filter  
executables and has relay v1_3 and level3  
control code v2_5_0. To make node fully working  
the distribution of new versions of relay and  
level3 control code is required.  
</pre>
<!-- Author: Boris.  -->


<!-- Date: Thu Jan 17 16:02:55 2002 -->
<p>Scanner Manager changes I've incorporated into v3_24: 
<ol> 
<li>Doubled command buffer size in hotline.c in order to allow 
Begin Run commands with more than 61 VRB entries. 
<li>Corrected values of MAX_FRAGMENT_PACKET_SIZE and MAX_FRAGMENTS_PER_EVENT in evbLimits.h. 
</ol> 
Both of these adjustments were made to reflect the size of SVX D-mode calibrations (our biggest events so far). The UPS instance 
of cdfevb_sm v3_24 is rebuilt with the changes on pcom1 and will propagate to pcom2 tonight. I've also replaced all/sm_v3_24.o on b0l3boot-vx. 
<p>I've installed nicstar v1_6 (ATM driver) on all converter nodes. It will be loaded used after the next reboot of the converters (and it's already running on c01). This version allocates the maximum of 512 reception buffers, again due to the size of SVX calibration events. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Thu Jan 17 16;15;33 comment by...Steve -->
If for some reason you need to revert to the old ATM driver, go to the file /etc/sysconfig/nicstar on each converter node and change "v1_6" to "v1_5". The reboot the converter.


<!-- Date: Thu Jan 24 23:33:58 2002 -->
<pre>HD_REGULAR_FCHECK: 
   While rebooting nodes b0l3124-b0l3241 the following nodes 
showed file system damage and have been repared: 
 
 133, 153, 173, 212, 238 
</pre>
<!-- Author: Ilya/Nuno/Mark -->


<!-- Date: Fri Jan 25 19:18:41 2002 -->
<pre>Submit log files management into CVS 
 
$L3_SRCDIR/remote/L3ProxyPartition.cc - Call FetchLogFiles method 
 
$L3_SRCDIR/remote/L3CoreManager.cc - Added FetchLogFiles method 
$L3_INCDIR/remote/L3CoreManager.h - Added FetchLogFiles method 
 
$L3_CSHDIR/l3_core_copier - redirected copying to 
                           /cdf/level3/filter/core  
                           instead of /cdf/level3/filter/log 
 
$L3_CSHDIR/l3_log_manager - new!!! 
 
$L3_CSHDIR/l3_all_node_exe - added new debug mode 
 
Number of computers (3) is now hardcoded into L3ProxyPartition.cc.  
 
logfiles (6) are fetched from computers correspondent to  
3 last entries in the run list forwarded to the constructor 
 of L3CoreManager. Due to the historical reasons the list is generated so that  computer IDs are either odd or even  
( 066, 064, 062 for exampile). 
  
</pre>
<!-- Author: Boris.  -->


<!-- Date: Wed Jan 30 15:11:55 2002 -->
<pre> HD_REGULAR_FCHECK: 
    Today we rebooted nodes 124-241  
       excluding 125, 135, 136, 138, 139, 176, 197 
 There are 10 nodes that are found bad. The bad node list: 
     124, 149, 151, 175, 187, 207, 208, 222, 233, 240 
  All nodes have been fixed. It took 2 hours 10min with 
three floppies.                                    
</pre>
<!-- Author: Ilya/Nuno -->


<!-- Date: Fri Feb 1 08:53:27 2002 -->
<p> 
Roughly half of the Level 3 farm will be regularly rebooted and forced a file system check. For those nodes that fail, we have put together a time effective recovery procedure.  
<p> 
The failure <a target=_top href="http://mit.fnal.gov/~leonardo/l3evb/recovery/nodelist.html">statistics</a> of nodes in the farm will be kept up to date, and detailed description of the <a target="new" href="http://mit.fnal.gov/~leonardo/l3evb/recovery/">recovery</a> procedure employed is presented as well. 
  
<!-- Author: Nuno Leonardo -->


<!-- Date: Fri Feb 1 19:07:33 2002 -->
<p>I've updated rsync on pcom1 and 2 in order to fix a security bug. See the RedHat <a href="https://www.redhat.com/support/errata/RHSA-2002-018.html">security announcement</a>. Upgrading on pcom2 was easy as RedHat provided 
an RPM for RH 6 systems, rsync-2.4.6-10.i386.rpm, now stored in /usr/src/redhat/RPMS/i386 on pcom2. For pcom1 I had to go to rsync.samba.org to get a tarball, in this case for version 2.4.8, which is the lowest "old-version" they list that has the security bug fixed. The tarball is in /usr/src on pcom1. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Fri Feb  1 19;14;49 comment by...Steve -->
The RPM was rsync-2.4.6-1.6.i386.rpm.



<!-- Date: Sat Feb 2 22:17:16 2002 -->
<pre>    Friday level3: 
 1. b0l3001 marked offline, it hung second time this week 
 
 2. b0l3044 taken offline. It can not boot anymore, appears 
      to have troubles with its hard drive</pre>
<!-- Author: Ilya -->


<!-- Date: Tue Feb 5 13:44:56 2002 -->
<pre>HD_REGULAR_FCHECK:  
We have performed another (2nd) regular rebooting of the L3 farm. 
Rebooted nodes 124-241   
       excluding 125, 135, 136, 138, 139, 176, 197  
The number of bad nodes this time was 2, namely 
       b0l3192, b0l3198 
These 2 nodes have been recovered. </pre>
<!-- Author: Nuno -->


<!-- Date: Wed Feb 6 12:50:09 2002 -->
<p>The new version of rsync (see a couple of entries back) hangs in the select() system call on pcom1. Small transfers of a few dozen files worked but not the transfer of the thousands of files  in the /home partition. I had to go back to version 2.3.3 in order to get something that works. Using <b>strace -p</b> I found a slight difference between versions in the way the rsync process sending from pcom1 used select: 
<ul> 
<li>2.4.8 - select(2, NULL, [1], NULL, ...)</li> 
<li>2.3.3 - select(2, [0], [1], NULL, ...)</li> 
</ul> 
Since rsync is started from ksh via inetd, fd 0, 1, and 2 are all 
aliases for the socket leading to b0l3pcom2. Perhaps there is some reason why one must wait for input as well as output. Anyway I'll leave 2.3.3 installed on both gateways. I don't think the security bug is all that important given that both L3 gateways are on the private DAQ subnet and given that the kerberized rsh encrypts all packets. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Thu Mar  7 16;18;57 comment by...ST -->
I had to revert eventually to rsync 2.3.1. It turns out that 2.3.3 doesn't update the copy properly when the new version of a file is shorter than the old. It gives an error message which seems to be non-fatal and is all technical spew anyhow.
<!-- Date: Fri Feb 8 18:04:10 2002 -->
<pre>  After Nuno had fixed b0l3044 with Tom's disk, it still  
showed "Processor 2 failed BIST" on startup. Following  
Steve's advice I have set processor recheck flag in BIOS 
and rebooted the node. Now the message is gone. I have 
integrated it back into the farm. 
</pre>
<!-- Author: Ilya -->


<!-- Date: Tue Feb 12 11:04:40 2002 -->
<pre>HD_REGULAR_FCHECK:   
 
L3 farm weekly rebooting.  
Two consecutive rebootings of subfarms 8-16:  
 processor nodes 124-241  
 w/ excluding 125, 135, 136, 138, 139, 176, 197 
 
During 1st rebooting, the following 6 nodes failed 
128 132 151 171 219 227  
These were successfully recovered. 
 
After the 2nd rebooting (again forcing fsck) failed node 
173 
It failed as well the standard recovery procedure.  
It turns out it fails checking hda5.</pre>
<!-- Author: Nuno -->


<!-- Date: Thu Feb 14 21:50:27 2002 -->
<pre>  I have recompiled L3 with the latest cdfevb_l3recv v3_8 
and distributed level3 under the version v2_5_8 
This new l3recv is supposed to cure the problem with 
converter nodes crashing (l3_node) during shot setup 
due to the huge events (occasionally) and not enough 
ATM reception buffers.</pre>
<!-- Author: Ilya -->


<!-- Date: Tue Feb 19 14:26:06 2002 -->
<pre>HD_REGULAR_FCHECK: 
Two consecutive rebootings of subfarms 8-16:   
 processor nodes 124-241   
 w/ excluding 125, 135, 136, 138, 139, 176, 197  
 
During 1st rebooting, the following 6 nodes failed  
127 146 150 169 223 231 
Node 146 needed to be recovered filesystem /dev/hda5/ as well 
The 6 nodes were recovered. 
 
During 2nd rebooting, the following 6 nodes failed  
151 166 173 181 200 203 
These 6 nodes were also recovered. 
 
Additionally: 
- Collected Mac addresses of single ethernet port in output nodes 
  (this is not used usually; it is needed for recovery procedure 
  to work on outputs as well) 
- Included output node u03 and respective subfarms online in 
  database (we had fixed u03 yesterday, after day long struggle) 
 
</pre>
<!-- Author: Nuno -->


<!-- Date: Tue Feb 26 19:48:11 2002 -->
<pre>HD_REGULAR_FCHECK: 
5 nodes failed root filesystem check: 
130, 149, 171, 173, 192 
I am leaving max mount count set to 1 in processors 124-241. 
This will impose a filesystem check everytime one of these  
nodes needs to be rebooted; it also means there is some 
probability that it will need to be recovered (so please be 
aware of the procedure).  
</pre>
<!-- Author: Nuno -->


<!-- Date: Tue Mar 5 11:34:52 2002 -->
<pre>HD_REGULAR_FCHECK: 
Seven nodes failed fsck  
b0l3128 b0l3151 b0l3173 b0l3209 b0l3216 b0l3230 b0l3236  
and were recovered (hda5 was restored in 173). 
Filesystem check was performed, avoiding tune2fs, as max mount count was set to 1 since last week. Recovery script leaves this set to 1.</pre>
<!-- Author: Nuno -->
<!-- Comment: Thu Mar  7 16;21;51 comment by...Steve Tether -->
It looks as if b0l3173 has a broken serial port ttyS0 (COM1). mingetty complains that it isn't a tty. Commented out the line in /etc/inittab that starts mingetty on ttyS0.


<!-- Date: Wed Mar 13 10:20:09 2002 -->
<p>A little after 2 AM on the morning of the 12th we saw all but two of the converter nodes crash simultaneously. The immediate cause was an attempt to divide by zero in the kernel during the execution of the gettimeofday() system call. It looks like the "jiffies" counter that is incremented on each clock tick go set to zero somehow. My guess is that a broadcast NTP protocol packet got garbled and set the local time to some ridiculous value. Here's a kernel dump that was made to the system log of c01, stripped of timestamps: 
<p><pre> 
divide error: 0000 
CPU:    0 
EIP:    0010:[<00110a47>] 
EFLAGS: 00010002 
eax: 0042387b   ebx: 00836ba7   ecx: 00000000   edx: 00079bf2 
esi: 7a2f6d12   edi: 00000000   ebp: bfffeb3c   esp: 0418df88 
ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018 
Process RemoteServer (pid: 4856, process nr: 33, stackpage=0418d000) 
Stack: 0418dfb0 00000256 080944f8 00110b46 bfffeb34 00000000 00118120 0418dfb0  
       03b5a018 bfffeb70 3c8db916 00046a70 0010a92d bfffeb34 00000000 404b2a68  
       bfffeb70 080944f8 bfffeb3c ffffffda ffff002b 0010002b 0000002b 0000002b  
Call Trace: [<080944f8>] [<00110b46>] [<00118120>] [<0010a92d>] [<080944f8>]  
Code: f7 f1 ba e8 03 00 00 89 c1 31 c0 f7 f1 a3 54 21 1b 00 89 c3  
</pre><p> 
Running that through the ksymoops program in /usr/src/2.0.38+atm/scripts yielded: 
<p><pre> 
[root@b0l3c01 /root]# /usr/src/2.0.38+atm/scripts/ksymoops /usr/src/2.0.38+atm/System.map < ~/oops.1 
Using `/usr/src/2.0.38+atm/System.map' to map addresses to symbols. 
 
>>EIP: 110a47 < do_fast_gettimeoffset+3b/78> 
Trace: 80944f8 
Trace: 110b46 < do_gettimeofday+6/3c> 
Trace: 118120 < sys_gettimeofday+1c/70> 
Trace: 10a92d < ret_from_sys_call+4d/88> 
Trace: 80944f8 
 
Code: 110a47 < do_fast_gettimeoffset+3b/78>  
Code: 110a47 < do_fast_gettimeoffset+3b/78>  f7 f1                       divl   %ecx,%eax 
Code: 110a49 < do_fast_gettimeoffset+3d/78>  ba e8 03 00 00              movl   $0x3e8,%edx 
Code: 110a4e < do_fast_gettimeoffset+42/78>  89 c1                       movl   %eax,%ecx 
Code: 110a50 < do_fast_gettimeoffset+44/78>  31 c0                       xorl   %eax,%eax 
Code: 110a58 < do_fast_gettimeoffset+4c/78>  f7 f1                       divl   %ecx,%eax 
Code: 110a5a < do_fast_gettimeoffset+4e/78>  a3 54 21 1b 00              movl   %eax,0x1b2154 
Code: 110a5f < do_fast_gettimeoffset+53/78>  89 c3                       movl   %eax,%ebx 
</pre> 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Tue Mar 19 11:33:32 2002 -->
<pre>HD_REGULAR_FCHECK: 
Consecutive rebootings of subfarms 8-16:    
processor nodes 124-241    
w/ exclusion of 125, 128, 135, 136, 138, 139, 149, 176, 197   
 
First rebooting: 
Twelve nodes failed fsck after two week period since last farm reboot  
b0l3127 b0l3146 b0l3150 b0l3151 b0l3159 b0l3163  
b0l3171 b0l3192 b0l3196 b0l3205 b0l3212 b0l3223 
and were recovered (hda5 was restored in 159) 
 
Second rebooting: 
Three nodes failed fsck 
b0l3126 b0l3144 b0l3220 
 
Third rebooting: 
Two nodes failed fsck 
b0l3169 b0l3215 
 
Fourth rebooting: 
One node failed fsck 
b0l3159 
(hda1 was restored) 
 
Fifth rebooting: 
One node failed fsck 
b0l3231 
 
Sixth rebooting: 
No nodes failed 
 
Seventh rebooting: 
One node failed 
b0l3134 
(hda1 only was restored) 
 
Eight rebooting: 
No node failed 
 
Nineth rebooting: 
No node failed 
 
Tenth rebooting: 
No node failed 
 
 
</pre>
<!-- Author: Nuno -->


<!-- Date: Tue Apr 2 11:07:07 2002 -->
<pre>HD_REGULAR_FCHECK:  
2 weeks after last rebooting 
Nine nodes failed fsck 
130, 139, 150, 151,171,203,209,223,230</pre>
<!-- Author: Nuno -->


<!-- Date: Mon Apr 15 15:44:58 2002 -->
<p>Converter node 1 has been running nicstar v1_7 for a while. That version was a general cleanup with tests added to prevent writing past the end of the array used to hold buffer addresses and data lengths during packet reception. It sometimes seems to mis-handle the data, so I'm withdrawing it and replacing it with v1_6b1 which is v1_6 with only the overrun test. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Tue Apr 16 12:36:08 2002 -->
<pre>HD_REGULAR_FCHECK: 
L3 farm regular rebooting. 
Nine nodes failed fsck: 
124 130 144 146 169 175 192 223 233 
hda1 (/cdf) needed to be restored in 130, current level3 and relay code were delivered to the node.</pre>
<!-- Author: Nuno -->
<!-- Comment: Tue Apr 16 13;14;47 comment by...Nuno -->
<pre>b0l3149 also failed fsck on hda6, but failed to read floppy properly, maybe problems with the driver. It is left online but was not recovered (boot ignoring fsck errors).
</pre>


<!-- Date: Wed Apr 24 17:30:26 2002 -->
<pre>Processor node 044 crashed this morning, with hda errors.  
hda1 has been recovered.</pre>
<!-- Author: Nuno -->


<!-- Date: Tue Apr 30 19:50:00 2002 -->
<pre>HD_REGULAR_FCHECK: 
L3 farm regular rebooting. 
19 nodes failed fsck: 
 
127 146 149 150 151 153 163 171 173 190   
198 205 208 209 221 223 228 231 239  
 
17 were recovered (rootfs repaired), and two marked offline: 
- 149 (floppy driver problem) 
- 151 (recall: this node was running since April 10 with a new 
  ide card, which unfortunately didn't change the node's bad 
  behaviour; the node got stuck while performing fsck on reboot.) 
</pre>
<!-- Author: Nuno -->


<!-- Date: Tue Apr 30 21:01:22 2002 -->
<ull> 
<li>folowing MT's advice, try to make sure we can reproduce this missing energy bit problem. 
 
<li>Run 144123 pull cable in dcas slot 4 bottom cable crate b0l2cl02.  this should be phi 8 eta 22,23.  If the problem is in dcas/locos/backplane with dropped bit I should see x7de in the cdfvme locos and dcas pages. 
 
<li>When I pull the cable I see x7fe (correct value for no  dropped bits) in locos, and in all  dcas on the lhs of locos, meaning no bits are being dropped across the backplane or in the daisychain. 
 
<li>not sure what this means - haven't been keeping abreast of the problem.  possible flakeyness in the cable between dcas and dirac?? 
</ul>
<!-- Author: hray -->
<!-- Comment: Wed May  1 12;50;06 comment by...Ilya -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>  THIS ENTRY WAS PUT HERE BY MISTAKE, PLEASE IGNORE.</pre></td></tr></td></table>


<!-- Date: Sat May 11 09:04:56 2002 -->
<pre>When trying to reset with minicom one of the scpus it 
hanged for a while in: 
 
 
# Load and start the SCPU code 
ld &lt; ${SCPU} 
value = 15330640 = 0xe9ed50 = n_proc + 0x18 
 
 
 
Also, in b0l3boot-vx date is not synchronized (w/ pcom1 eg).</pre>
<!-- Author: Nuno -->


<!-- Date: Thu May 16 09:51:42 2002 -->
<p>It appears that converter node 1 ran into network difficulties early this morning and died with divide by zero in the gettimeofday() system call. The other converter nodes don't appear to be affected. 
<p> 
<pre>May 16 03:11:22 b0l3c01 ypbind[586]: b0l3gate1.fnal.gov: RPC: Remote system error   
May 16 03:11:22 b0l3c01 ypbind[586]: clnt_create for server b0l3gate1.fnal.gov failed  
May 16 03:11:22 b0l3c01 ypbind[586]: clnt_create for server b0l3gate1.fnal.gov failed  
May 16 03:11:23 b0l3c01 ypbind[586]: b0l3gate1.fnal.gov: RPC: Remote system error   
May 16 03:11:23 b0l3c01 ypbind[586]: b0l3gate1.fnal.gov: RPC: Remote system error   
May 16 03:11:23 b0l3c01 ypbind[586]: clnt_create for server b0l3gate1.fnal.gov failed  
May 16 03:11:26 b0l3c01 kernel: divide error: 0000 
May 16 03:11:26 b0l3c01 kernel: CPU:    0 
May 16 03:11:26 b0l3c01 kernel: EIP:    0010:[do_fast_gettimeoffset+71/120] 
May 16 03:11:26 b0l3c01 kernel: EFLAGS: 00010002 
May 16 03:11:26 b0l3c01 kernel: eax: 0044dae5   ebx: 00836ba7   ecx: 00000000   edx: 00079bf2 
May 16 03:11:26 b0l3c01 kernel: esi: 796fc016   edi: 00000000   ebp: bfffeb3c   esp: 018ccf88 
May 16 03:11:26 b0l3c01 kernel: ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018 
May 16 03:11:26 b0l3c01 kernel: Process RemoteServer (pid: 22078, process nr: 33, stackpage=018cc000) 
May 16 03:11:26 b0l3c01 kernel: Stack: 018ccfb0 00000256 08094cf0 00110b46 bfffeb34 00000000 00118120 018ccfb0  
May 16 03:11:26 b0l3c01 kernel:        07f83018 bfffeb70 3ce369ae 0009ccba 0010a92d bfffeb34 00000000 404b2a68  
May 16 03:11:26 b0l3c01 kernel:        bfffeb70 08094cf0 bfffeb3c ffffffda 4026002b 0807002b 0806002b 0000002b  
May 16 03:11:26 b0l3c01 kernel: Call Trace: [<08094cf0>] [do_gettimeofday+34/68] [sys_gettimeofday+44/112] [system_call+85/124] [<08094cf0>] [<0807002b>] [<0806002b>]  
May 16 03:11:26 b0l3c01 kernel: Code: f7 f1 ba e8 03 00 00 89 c1 31 c0 f7 f1 a3 5c 21 1b 00 89 c3  
May 16 03:11:26 b0l3c01 kernel: divide error: 0000 
May 16 03:11:26 b0l3c01 kernel: CPU:    0 
May 16 03:11:26 b0l3c01 kernel: EIP:    0010:[do_fast_gettimeoffset+71/120] 
May 16 03:11:26 b0l3c01 kernel: EFLAGS: 00010002 
May 16 03:11:26 b0l3c01 kernel: eax: 0081bb53   ebx: 00836ba7   ecx: 00000008   edx: 00079bf2 
May 16 03:11:26 b0l3c01 kernel: esi: 79aca084   edi: 00000000   ebp: 079b6f38   esp: 079b6f04 
May 16 03:11:26 b0l3c01 kernel: ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018 
May 16 03:11:26 b0l3c01 kernel: Process xntpd (pid: 595, process nr: 21, stackpage=079b6000) 
May 16 03:11:26 b0l3c01 kernel: Stack: 079b6f5c 00000046 00008001 00110b46 00000000 00000000 001185eb 079b6f5c  
May 16 03:11:26 b0l3c01 kernel:        079bb414 00000000 ffffff30 bffffd18 00008001 00008001 00000000 00000000  
May 16 03:11:26 b0l3c01 kernel:        00fa0000 00fa0000 00000040 00000002 00000001 00c80000 3ce369ae 0009ebfa  
</pre> 
<p>  
<!-- Author: Steve Tether -->


<!-- Date: Tue May 21 21:39:26 2002 -->
<pre>HD_REGULAR_FCHECK: 
16 processor nodes failed fsck while rebooting 
 130 144 146 150 163 169 171 192  
 200 209 223 224 230 231 236 240 
hda6 was repaired in all of them, and  
hda5 in 146, hda1 in 230</pre>
<!-- Author: Nuno/Arkadiy -->


<!-- Date: Sun May 26 15:31:10 2002 -->
Following a site-wide <a  href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=436&frame=2&anchor=09:33:55'>power outage</a> on May 26, 05:39, the consequences for EVB/L3 were as follows. 
<p>Ethernet swich 04, serving subfarm 3 (converter node b0l3c04), failed to come up successfully; it hangs performing its own power tests. The subfarm was marked offline, and a replacement for the switch is found necessary. 
<p>HD_REGULAR_FCHECK: 
<br>It was not a <i>regular</i> maintenance rebooting, but we register here as usual the filesystem repairs that were needed: 
<li>hda6: 124 127 170 194 203 207 
<li>hda5: 133 
<br>Two floppies were also found damaged when attempting to recover 194. 
<br>This information has also been used to update the <a href="http://mit.fnal.gov/~leonardo/l3evb/recovery/nodelist.html">statistics</a> page for the L3 farm. 
 
<p> Processor nodes 108 and 128 were not yet recovered: 
<li>108 seems to present problems with the disk itself rather than its partitions. When attempting to use fdisk, it shows several error messages. This way the floppy (which is used to recover filesystems) cannot be used (as hda cannot be properly accessed). 
<li>128 showed another strange behaviour; this is one of the nodes used for ide cards testing (Mark's); other such nodes (125, 135, 138, 151, 176, 197) also failed; however we didn't attempt to recover them, as they are not to be used online, and so not to interfere with possible current tests. 
 
<p>Node 041 showed twice problems when booting, hanging even before Lilo started. The front power button would have no effect at this point; it did come up after power cycling (main power supply button, on the back of the box), still displaying however some error messages. 
 
<p>Several other random nodes which were down after the outage did come up successfully after pwer cycling.  
 
<p>Initial problems with ypbind also disappeared when rebooting the full farm in the correct order (gate1, gate2, vxboot, converters, processors, outputs, respectively).  
 
<p>No problems with EVB were noticed, and EVB cleanup was performed successfully. 
 
  
<!-- Author: Nuno/Ilya -->


<!-- Date: Fri Jun 14 15:22:48 2002 -->
<p>I've just installed scpu_v3_29.o and sm_v3_25.o on b0l3boot-vx and have made the corresponding UPS product builds on pcom1. These are the official versions of the recently tested code to prevent incomplete events from being sent to L3. I haven't yet changed the .cshrc file for vxboot on b0l3boot-vx. The UPS products are now the current ones for cdfevb_sm and cdfevb_scpu. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Jun 14 16:14:42 2002 -->
<p>I've just installed nicstar driver 1.6b1 on all converter nodes except b0l3c04 which is currently unreachable. The change will take effect the next time the converters are rebooted. This retracts the massive changes in release 1.7 which seemed to cause runs to hang and adds only a check against index out of range in the buffer of iovecs. Until further notice all production versions of the nicstar ATM driver will come from the branch labelled v1_6b and not from the main branch. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Jun 26 15:32:06 2002 -->
<p> 
I've updated b0l3boot-vx:~vxboot/all/scpu_v3_29.o with the changes I made for Steve Nahn (dumping of SVX VRB header for more kinds of errors). When convenient the .cshrc should be made to point to sm_v3_25.o and scpu_v3_29.o. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Tue Jul 16 16:49:47 2002 -->
<p>I've just installed nicstar 1.6b1 on converter node c04. It will be loaded after the next reboot. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Tue Jul 16 18;40;02 comment by...Nuno -->
<pre>c04 has been rebooted, and now all converter nodes are
using the newer version  nicstar1_6b1.</pre>


<!-- Date: Tue Jul 16 18:45:12 2002 -->
<pre>HD_REGULAR_FCHECK: 
hda6 was repaired on 11 nodes during this maintenance reboot: 
138 144 150 151 154 163 171 176 205 220 223  
 
Additionally, an extra partition (hda3; leftover from ide tests) 
was removed from nodes 125,135,136,138,139,176 (to avoid problems 
when running lilo).</pre>
<!-- Author: Nuno -->


<!-- Date: Tue Jul 30 18:23:33 2002 -->
<pre>HD_REGULAR_FCHECK:  
Seventeen nodes had been repaired (hda6) 
125 127 130 144 149 150 151 156 171 176 180 192 202 207 223 230 239  
Additionaly: hda5 on 125, hda1 on 127. 
Relay and L3 code were distributed to 127.</pre>
<!-- Author: Nuno -->


<!-- Date: Thu Aug 15 11:55:01 2002 -->
<pre>farm reboot 
 
22 nodes needed to be taken care of  
124 146 150 151 152 159 161 163 169 171  
180 200 203 208 209 223 228 230 231 233 
236 240  
 
i recovered 20 nodes and marked offline  
nodes 146 and 230  
because i could not recover them neigher with floppy nor manually 
they both have a persistent error when checking hda1 </pre>
<!-- Author: Arkadiy -->


<!-- Date: Thu Aug 29 11:25:28 2002 -->
<p>Yesterday /dev/hdc on b0l3pcom1 failed and we were obliged to use the backup copies of /home and /cdf that were being kept on gate2. We ran into a few problems not covered in the instructions for the changeover. 
<ul> 
<li>The L3 farm could not import /home and /usr/products from gate1 any more because gate1 itself was importing them from pcom2. We had to change everbody's /etc/hosts so that b0l3serv is an alias for gate2. 
<li>The master NIS maps are still on gate1 so /etc/yp.conf on each node now has to refer to gate1 and not b0l3serv. 
<li>/etc/hosts.allow on the gateways only allows networks 192.168.0.0 to import; use b0l3gate1 instead of b0l3pcom1, etc. 
<li>We had to create directory cdfevb on the duplicate /cdf because the EVB proxy creates log files there. 
<li>Simply remounting the imported disks on each L3 node didn't always work; we had to reboot about ten machines or at least use the stop/start script for network disks in /etc/rc.d/init.d (nfsfs on RH5, netfs on RH6). 
</ul> 
Converter node c10 has not been updated yet because it is off-line awaiting a new power supply. The changes to yp.conf still need to be made on processor nodes but is not urgent because they don't import /usr/products or use non-local user names for data taking. 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Mon Sep 9 17:24:25 2002 -->
<p>It turns out that pcom1 is running a version of glibc that is older than 2.1.3 and so is vulnerable to DNS-based attacks even for looking up ordinary host names as well as for the less common lookup of network names. Therefore I've made <b>/etc/hosts</b> contain all the IP numbers for online group machines that need to communicate with pcom1. Tomorrow during EVB/L3 downtime the <b>/etc/nsswitch.conf</b> file of pcom1 should be edited to remove "dns" from the "hosts" line. This will force it to rely solely on <b>/etc/hosts</b> and so protect it fron DNS attacks. 
<p>  
<!-- Author: Steve Tether -->


<!-- Date: Tue Sep 10 17:27:04 2002 -->
<pre>HD_REGULAR_FCHECK: 
Rebooting of the full farm (Processor, Converter, Output nodes). 
In the following 30 nodes the root filesystem hads been repaired. 
b0l3073, b0l3125, b0l3127, b0l3130, b0l3135, b0l3139, b0l3144, 
b0l3146, b0l3149, b0l3151, b0l3159, b0l3163, b0l3164, b0l3169,  
b0l3170, b0l3171, b0l3173, b0l3175, b0l3176, b0l3188, b0l3192,  
b0l3197, b0l3198, b0l3205, b0l3209, b0l3212, b0l3221, b0l3223, 
b0l3224, b0l3239  
Additionally, hda5 to be repaired on 125, 139,146,159;  
135 was fixed also running lilo from floppy.  
</pre>
<!-- Author: Nuno -->


<!-- Date: Wed Sep 11 08:21:48 2002 -->
<pre>test</pre>
<!-- Author: Todd -->


<!-- Date: Wed Sep 25 18:16:12 2002 -->
<pre>All L3 farm has been rebooted (Processor, Converter, Output 
nodes) this morning . 
 
In the following 26 nodes the root filesystem has been repaired: 
b0l3075,b0l3124,b0l3126,b0l3134,b0l3135,b0l3139,b0l3144, 
b0l3150,b0l3151,b0l3163,b0l3168,b0l3169,b0l3171,b0l3176, 
b0l3187,b0l3194,b0l3199,b0l3204,b0l3209,b0l3211,b0l3219, 
b0l3222,b0l3223,b0l3227,b0l3230,b0l3231 
 
b0l3159 was recovered "by hand". 
 
Output node 3 is still down, maybe it has corrupted the hard drive. 
 
The script "allProc_w_excl.exp" to include all nodes in it has been changed (the old one is called "allProc_w_excl.exp.old")</pre>
<!-- Author: Guillelmo -->


<!-- Date: Wed Oct 30 09:07:27 2002 -->
<pre>Most Converters - all except c05, c08, c10 - died, w/ RIP status. 
hda6 check failed on c03, c04, c06, c09, c11; these have been 
temporarily fixed with manual fsck - a full (floppy) recovery of 
these filesystems should be done. 
 
This appears to have been caused by Kernel division by 0, eg 
 
  Oct 30 06:08:59 b0l3c16 kernel: divide error: 0000 
 
which occurred on all converter nodes except on: 
 c05, c08, c10, c13, c14, c15 
on 
Oct 30 6:09-6:12</pre>
<!-- Author: Nuno/Guillelmo -->
<!-- Comment: Wed Oct 30 10;39;43 comment by...Nuno/Guillelmo -->
<pre>ATM driver fail to connect with a single converter reboot.</pre>

<!-- Comment: Wed Oct 30 11;54;24 comment by...Nuno/Guillelmo/Ilya -->
<pre>A second reboot was not enough; a third did it.

Note: 
One can check in /var/log/messages of each converter
if ATM driver has not been correctly loaded; when there has 
been a problem the following kernel "page" message appears:

Oct 30 10:02:35 b0l3c01 kernel: Couldn't get a free page.....

just preceding usual messages like:

Oct 30 10:02:35 b0l3c01 kernel: nicstar: 1 devices found.
Oct 30 10:02:36 b0l3c01 kernel: scramnet: PCI memory-space address = 0xfa000000
</pre>


<!-- Date: Tue Nov 5 18:09:04 2002 -->
<pre>Rebooted farm today. 35 nodes failed.  
29 nodes: hda6 recovered with a floppy.    
3  nodes: hda1 or hda5 recovered with manual fsck 
3  nodes: hda1 or hda5 recovered with a floppy.  
 
Put new labes on 001-030 nodes.  
Put the labels for 212-242 in the box above the  
output nodes.  
 
 
b0l3226 and b0l3230 had a weird problem with hostname.  
They were thinking that they are still b0l3016 and b0l3020.  
This is the leftover from our IP exchange and I 
already fixed it once. Will take a close look on this  
problem later.  
</pre>
<!-- Author: Boris  -->


<!-- Date: Tue Nov 19 22:14:09 2002 -->
<pre>Power cut in the 3rd floor of B0 this afternoon. Everything was down. Then, we used the standard procedure to reboot the farm (available on the web). 
The main problems came from the proccesor nodes. 
b0l3... 11,13,20,21,27,29,131,135,139,150,151,175,181,191,196,209 failed, and they were recovered by the standard floppy. 
 
b0l3... 134,170,176,185 were recovered by the controlled floppy (in 134,170,185 hda1 and hda5 failed!, in 176 just hda5) 
 
b0l3172 was recovered by the 'remote' floppy. I needed to put the IP address in ifcfg-eth0 by hand. 
 
In addition, I marked b0l3075 online.</pre>
<!-- Author: Guillelmo -->


<!-- Date: Wed Jan 22 16:03:55 2003 -->
<pre>I shutdown the system late last night. 
Today a few nodes failed with usual  
filesystem failures, and were repaired: 
(HD_REGULAR_FCHECK:)  
135, 139, 149,176, 183. 
One of the new nodes (256) is failling 
with hardware problems; it was marked  
offline. 
Ping script has been corrected.</pre>
<!-- Author: Nuno -->