10/11/98 - nis on b0dau30
  Do man yp, man nsd. yp and many other naming services have been
  merged into the "nsd" daemon (network directory service?).
  To activate nds and yp:
    chkconfig nds on
    chkconfig yp on
    The above will cause it to start at boot time.
    To start while the system is running, just type nsd.
    To restart after changing configuration file, killall -HUP nsd (or close)
    in file /var/yp/ypdomain put the name b0daq
    in /etc/netgroup on b0sgh add b0dau30
    in hosts file on b0sgh, b0sgz add b0dau30 (for nfs exports)
    in hosts file on b0dau30 add b0sgh and b0sgz (same reason)

10/20/98
  Install apache server
  Straightforward except problem with cgi - had to have trailing /
	in ScriptAlias line
  Redirect line in srm.conf to redirect
	Redirect / http://b0dau30.fnal.gov:8000/
  To make new log files:
	mv access_log access_log.old
	kill -1 `cat httpd.pid` (SIGHUP signal)
  Still to do - understand .htaccess files
  robots.txt:
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /tmp/
  
10/23/98
  To get the ethernet address on an SGI, do netstat -ai

10/28/98
  Try to make b0dau30 a yp server
  On b0sgh, /var/yp, edit ypservers
	on /var/yp/ypmake, doesn't reread this
  edit /etc/netgroup
  edit /etc/hosts on all machines that nfs mount
  chkconfig ypserv on on b0dau30

10/29/98
  Installing b0dau31. Problem with yp
    Turns out network.local missing from /etc/init.d
    Also /etc/passwd needs +::0:0::: at end
    Also make sure /etc/sys_id is fully qualified name

11/02/98
--------
    Access to b0sgh had disappeared ~2300 last Friday (10/30).
    Tried reboot, no help. Power cycle no help. Mess around
    switching ethernet cables, AUI/thinwire interface, no help.
    Finally, with different ethernet line (old b0sgz, direct
    to dempr), and power cycle, comes back.

    Further working to make b0dau30 the yp master:

    After doing ypinit -s b0sgh -d b0daq
     then the yp updates start working

    Presumably the next step is to copy over all the files,
    then set ypmaster on b0dau30, off b0sgh, and reboot.

    Enable proxy server in apache:
	AddModule in src/Configuration
	ProxyRequests on in httpd.conf
	ProxyVia on (don't understand)
		Actually, need this off to bypass later blocking
		Makes requests appear to come from b0dau30 if off
	Don't enable cache for now
	Add to access.conf:
<Directory proxy:*>
order deny,allow
deny from all
allow from fnal.gov
</Directory>

11/03/98
--------
    Get Samba working. Pretty much follow instructions
    Did test of ROBIN server, seemed to work as long as usual
	no // in ROBIN_REPOSITORY

    Try to get systools working
    Find /usr/local/systools/library/functions has function/people
      directory must not have world read access
      have to copy funame from UPS to /usr/local/bin
      then can't find getlock. Copy from b0sgh, won't execute

11/09/98
--------
    In /etc/aliases, modified so root mail goes to patrick

11/09/98
--------
    Major hacks to get cvsweb.cgi working again
    Put in links from /usr/bin -> /usr/local/bin -> /cdf/code/products/cvs/v1_9/bin
    Still fails, complains can't find /usr/local/products/cvs/v1_9/bin/co
    Can't figure out, so link /usr/local/products/cvs /cdf/code/cvs
    Finally now works.
    Note need code-IRIX-5.3 disk mounted on b0dau30

11/12/98
--------
    Power outage OK. Fail again to convert yp master

    Move cvs repository to b0dau30:
	Cross mount code directories writable on b0dau30<->b0sgh
	Tar files from h to dau30
	Make link /cdf/code-IRIX-5.3/cvs -> /cdf/code-IRIX-6.5/cvs
	To support access through b0dau30, add b0dau30 location
	  of cvs bin file to PATH in ~cvs/.cvshrc

11/19/98
--------
    Finally install ssh. Download and build version 2, which takes
    a LONG time. Turns out to apparently be incompatible with version 1.
    So get version 1, build, install.
    Really is as simple as untar, ./configure, gmake,
    then as root gmake install.
    Modify /etc/sshd_config to set RhostsAuthentication yes
    so a password will not be requested, and vxworks nodes can boot.
    Copy /etc/init.d/sshd from fndaub, add link to rc2.d
    Turns out to really be remarkably simple. It does make the
    assumption that the first time someone connects, it is the true
    host and not a fake.

11/24
    To get offline to build on b0dau30, copy libXpm.a and libXt.a
    from b0sgz. Not clear if it will really do what is desired though.

11/26
    Install new 18GB disks.
    Repartition with fx command
    Make file of form
    dks0d5s5 4096:17000000 xfs
    dks0d5s6 followspart5:remainder xfs
    then do fx -x -s filename "dksc(0,5)" for disk 5 on controller 0

11/27
    Attempt to make mirror set:
    xlv_make
    vol people1
    data
    plex
    ve dks0d5s5
    plex
    ve dks14d5s5
    end
    exit
    complains on exit that license not installed.
    would then do:
    mkfs_xfs /dev/rxlv/people1
    then mount /dev/xlv/people1, raw=/dev/rxlv/people1
    To delete object, xlv_mgr then delete object people1

    To make non-mirror set, do
    plex
    ve dks0d5s5 dks14d5s5
    end
    exit

11/30/98
--------
    Make logical volumes out of half of each new disk for
    people1, people2, code-IRIX-6.5, code-common.
    Transfer code-IRIX-6.5, code-common, people1 to new disks
    Adjust fstab, exports everywhere.
    Note now need to include all of b0daq in /etc/hosts on dau30

    Also install new PCI-SCSI adapter, and tape drive from b0sgh.
    Comes up as /dev/rmt/tps16d5...

12/01/98
--------
    Transfer people2 to b0dau30.
    Finally make b0dau30 the yp master.
    Get passwd, group, netgroup, aliases from b0sgh.
    Leave other files the same.
    Do ypinit -m -d b0daq -l sahara,b0dau31
      Asks to delete old (slave) database.
    Other things bind OK except IRIX 5.
    On b0sgh, sahara, do ypsetme b0dau30 in /etc/config/ypbind.options
    Have to add b0dau30 to /etc/hosts on these for that to work.

12/07/98
--------
    Physically move /cdf/commercial and /cdf/code-IRIX-5.3 disks from
    b0sgh to b0dau30. Put in 5 1/4" carrier on top, installation is
    straightforward. Note SCSI address on the disk is used.
    Also discover that VxWorks68k is really on system disk.

12/08/98
--------
    Trouble with cvs2html again, cron job not running. Run on SGI,
    rediscover the reason it doesn't work is it needs a "find -maxdepth".
    Also, perl needs to be set up.

12/10/98
--------
    systools on b0dau30: adduser calls /usr/local/bin/obtain
    This tries to fetch from giduid.fnal.gov, but that doesn't work.
    Get version of obtain from b0sgh, which gets it from fnalv.
    Seems to work?

    ypinit on b0dau30 does not help but makes things worse...

12/11/98
--------
    nfs authentication wasn't working.
    Reason turned out to be in nsswitch.conf, netgroup had been
    changed to files rather than nis to see if it would cure
    the yppush problem.

12/15/98
--------
    people backup overflows tape. Switch to compressed mode
    on tape drives - c at end of device name

12/22/98
---------
    Last night in middle of xemacs install code-IRIX64-6.2 disk on floor
    starts giving "I/O error" on any access.
    Doesn't come back this morning, so restore from backup made
    11/19/98 to /cdf/code-common/code-IRIX64-6.2.
    Then have to make link to here from /cdf/code-IRIX64-6.2
    all machines, particularly O2s

01/11/99
--------
    Add swap space to b0sgz:
    see man swap
    swap -l to list space usage
    /usr/sbin/mkfile 128m /cdf/code-common-old/swap
    /sbin/swap -a /cdf/code-common-old/swap
    edit fstab - add line:
    /cdf/code-common-old/swap swap swap pri=2 0 0

01/22/99
--------
    Have been getting memory errors on b0dau30:

Jan 18 11:34:04 4A:b0dau30 unix: WARNING: Correctable memory ECC error (intr)
Jan 18 11:34:04 4A:b0dau30 unix: WARNING:  Mem: 0x4232a4a0, Syn: 0x43 (data_2)
Jan 18 11:34:04 4A:b0dau30 unix: WARNING:  Loc: /hw/module/1/slot/MotherBoard/node/memory/dimm_bank/2 (SLOT5 line 10)
Jan 18 11:34:04 4A:b0dau30 unix: WARNING:  Single-bit error has been repaired.
Jan 18 11:34:04 4A:b0dau30 unix: WARNING: Excessive single-bit ECC corrections in pfn 67786

    Turns out to be Kingston memory. SGI person interchanges slot 3 and 5,
    errors go away. But should run memory diagnostic.

03/22/99
--------
Problems changing group on gobi. Find it is bound to sahara.
Turns out possible nodes are listed in a file pointed to
in /etc/init.d/network. Found and changed to b0dau30 and rebooted.

04/09/99
--------
Find in apache server to enable htaccess files, must have
AllowOverride Limit
Also to use password authentication, must have AllowOverride AuthConfig
Example syntax for use is in a .htaccess file in cgi-bin

04/13/99
--------
Memory errors last couple of days:
Apr 13 09:26:27 4A:b0dau30 unix: WARNING:  Single-bit error has been repaired.
Apr 13 09:26:28 4A:b0dau30 unix: WARNING: Correctable memory ECC error (polled)
Apr 13 09:26:28 4A:b0dau30 unix: WARNING:  Mem: 0x100217fb8, Syn: 0xe (data_55)
Apr 13 09:26:28 4A:b0dau30 unix: WARNING:  Loc: /hw/module/2/slot/MotherBoard/node/memory/dimm_bank/0 (SLOT2 line 63)

It is SGI memory. Reseat DIMM, no errors so far after a couple of hours.

04/20/99
--------
Needed to change ARG_MAX in order for YBOS include files to work:

sysconf | grep ARG_MAX (default = 20480 = insufficient)
systune ncargs 40960 to change
no reboot needed (from Chris Green)

05/13/99
--------
After installing new cisco switch on 3rd floor, still see that
throughput is more like slow ethernet.
Note that b0dap07 however gives good throughput (5-7MB/sec on ftp).
Note that b0dap07 uses de4x5 as the ethernet driver, while
b0dap00 uses tulip.
To change the ethernet driver that is used on linux,
- log on as root
- select the "kernel daemon configuration" button on the control panel
- click on "tulip" to select it
- then click "remove"
- click on "add"
- module type is "eth" - click OK
- which module should be "de4x5" - click OK
- on the next window, there are no arguments, so just click OK
- click "restart kerneld"
- However it seems necessary to reboot the system for things to work

05/18/99
--------
Subject: 
        bootp/dhcp on 6.5 machines
   Date: 
        Tue, 18 May 1999 10:51:38 -0500 (CDT)
   From: 
        Mark Schmitz <schmitz@cdfserv0.fnal.gov>
     To: 
        patrick@cdfserv0.fnal.gov, geri@b0cc06.fnal.gov

I noticed the following messages in the SYSLOG of the machines
I upgraded last month. Bootp/DHCP was starting every few minutes. 

SGI bootp/dhcp Server V3.0.3 starting at Tue May 18 08:34:04 1999

I turned bootp/dhcp off by commenting out the following lines in
/etc/inetd.conf and reruning inetd.

# bootp dgram   udp     wait    root    /usr/etc/bootp          bootp
# bootp  dgram   udp     wait    root    /usr/etc/dhcp_bootp dhcp_bootp -o

/etc/killall -HUP inetd

Mark


06/01/99
--------
Install fortran license transfered from b0sgh on b0dau30

06/01/99
--------
Find that xemacs doesn't work on some linux systems. Apparantly linked
to libjpeg.so.6. So need to make link:
ln -s libjpeg.so.62 libjpeg.so.6

06/07/99
--------
Upgrade to IRIX 6.5.3. Found:
Very old CD drives don't appear to work.
Do network install, need bootp enabled on server to boot mini-root.
Also, rsh access for root or guest.
Also, tftp?

Later, deleted old versions of ups causing problems.
/usr/local/etc/setups.cache.IRIX+6 contains ups bootstrap
directory. Was pointing to an old (now deleted) version.
Doesn't use setups.cache.IRIX+6.5 for some reason.
Because ups declared with -f IRIX and not -f IRIX+6.5?

06/09/99
--------
Turned dks14d4s5 into /data1

06/20/99 (approx)
-----------------
Tricks needed to install on Micron quad Netframe:

- need to pass memory size to floppy boot kernel:
  Hit <F-something>
  linux mem=32M (or vmlinux?)

- aic7xxx=noprobe also - during installation procedure
  when asked for scsi options

07/07/99
--------
/disk3, with oracle database files, starts giving repetitive I/O errors

07/08/99
--------
Power cycle system, /disk3 comes back.
Do fx read diagnostics, no problems shown.
Also do xfs_check, no problems shown.
Oracle database restored.

Later, system appears to hang.
Turns out root disk partition is full.
Kill a process consuming large amounts of CPU, doesn't help.
(Probably this was syslogd?)
Do /etc/init.d/network stop then start, clears things up

07/09/99
--------
In afternoon complaints from OSU and Michigan about access.
Ping for example cdfpc7.mps.ohio-state.edu, get "Network not accessible"
However, OSU is able to boot vxworks nodes using gobi
after fixing ~vxworks/.cshrc to deal with unset REMOTEHOST on Sun machines.
Also notice can't get out to many other places as well.

07/12/99
--------
Reboot machine. Networking now looks OK. Perhaps problem was not
running network.local after network start?

Install 36GB disk and attempt to partition.
fx core dumps.

09/03/99
--------
Complete move of code products.
Bring down to single user mode. Some netscape process fails to stop.
Makes 1.6 GB core file which later causes problems.
Turns out ns_fasttrack is off, but nss_fasttrack was on.
So do chkconfing nss_fasttrack off so this shouldn't get started again.

09/21/99
--------
b0dau30 crash! kernel panic. first crash.
Note that systune ncargs ~60000 had been done less than a day earlier.

09/27/99
--------
When system is moved for power outage, fails to reboot.
Kernel panic - "Write access error" after local disk mounts
Try disk in b0dau31, does same thing.
Try dua31 disk on dua30, boots OK.
Bring to single user mode, note that netstat or even systune cause crash.
./network start seems to complain about hosts file, so strip that
down but no improvement.
Finally, note that previous kernel was stored in /var/adm/crash/unix.0
on the crash from 9/21. Note although the sizes are identical, the
kernels are different (from cmp). Try copying this one back into /unix,
then the system boots.
Not clear why it rebooted OK after crash. Note that old kernels do not
appear to be saved when the kernel is modified.
Thought that oracle work was before 9/21.

10/06/99
--------
Install new vxworks v5_3d.
upd install vxworks v5_3d
upd install -q ppc vxworks v5_3d
upd install -q 68k vxworks v5_3d
upd install -q mv2301 vxworks v5_3d
upd install -q mv162 vxworks v5_3d (not done yet)
To kill bad prototype messages in vxworks header compile,
go to vxworks_generic/v5_3c/ups, modify setup.csh to
remove -strict-prototypes from the compiler options

10/17/99
--------
Finally get Apache working with servlets.

Unpack ApacheJServ-1.1b1.tar.gz and apache_1_3_9.tar.gz

- in Apache top directory:
    ./configure --prefix=/cdf/code/freeware/apache/v1_3_9

- in JServ top directory:
    ./configure --prefix=/cdf/code/freeware/apache/v1_3_9 \
   		--with-apache-src=/cdf/code/freeware/apache/v1_3_9 \
		--with-jdk-home=/cdf/code/freeware/jdk/v1_1_6
		--with-JSDK=/cdf/code-common/freeware/jsdk/v2_0/lib/jsdk.jar
    (options can be put in configure-options)

    gmake install

- back to Apache directory:
    ./configure --prefix=/cdf/code/freeware/apache/v1_3_9 \
		--activate-module=src/modules/jserv/libjserv.a
    gmake (install)

- Copy configuration files (httpd, access, srm) from current
  Modify port number, ServerRoot directory, add at end:

  Include /cdf/code/freeware/apache/v1_3_9/etc/jserv.conf

- Modify apache_dir/etc/zone.properties to move default .class location from
  /cdf/code/freeware/apache/v1_3_9/servlets to somewhere else

10/20/99
--------
  Get DLT drive fixed finally. RVC said not really broken, another
  tape should have worked. In general, putting a good tape into a
  drive with the "broken leader" will not break the good tape.
  To recreate the drive:
    scsiha -p /hw/scsi_ctlr/16/bus (no messages are printed,
		however new entries appear in /hw/scsi_ctlr/16/target)
    ioconfig -f /hw/scsi_ctlr/16
    ls /dev/rmt then shows the newly attached drive, that was missing before

11/13/99
--------
  System crashes ~0130, fails to reboot:

Nov 13 13:42:30 6D:b0dau30 sn0log: The following are messages stored in the flashlog from a previous system boot.
Nov 13 13:42:30 6D:b0dau30 sn0log: Flashlog for /hw/module/2/slot/MotherBoard/node/hub/mon
Nov 13 13:42:30 5D:b0dau30 sn0log: B Fatal: Serial #: i^M!!
Nov 13 13:42:30 5D:b0dau30 sn0log: B Fatal:  HARDWARE ERROR STATE:
Nov 13 13:42:30 5D:b0dau30 sn0log: B Fatal: WARNING:
Nov 13 13:42:30 5D:b0dau30 sn0log: B Fatal: /hw/module/2/slot/MotherBoard/node/cpu/b:
Nov 13 13:42:30 5D:b0dau30 sn0log: B Fatal: Exception during show hardware state
Nov 13 13:42:30 6D:b0dau30 sn0log: End of flashlog for /hw/module/2/slot/MotherBoard/node/hub/mon
Nov 13 13:42:30 6D:b0dau30 sn0log: End of flashlog messages.

  Reboots OK after power cycle at 13:42

11/19/99
--------
 Note in evening, can't telnet to b0cc06. Have to give IP number

11/20/99
--------
 No backup mail messages. Can't telnet more places. Look in SYSLOG.
 See:
Nov 19 20:16:18 6D:b0dau30 sshd[4733571]: log: Closing connection to 131.225.233
.231
Nov 19 20:24:37 3D:b0dau30 nsd[283]: lookup (dns): failed malloc
Nov 19 20:25:02 6A:b0dau30 unix: NFS3 lookup failed for server localhost (nsd):
Timed out [filter /usr/sbin/klogpp failed: too slow]
Nov 19 20:25:15 6A:b0dau30 unix: NFS3 lookup failed for server localhost (nsd):
Timed out [filter /usr/sbin/klogpp failed: too slow]
Nov 19 20:25:15 6A:b0dau30 unix: NFS3 getattr failed for server localhost (nsd):
 Timed out [filter /usr/sbin/klogpp failed: too slow]
Nov 19 20:25:12 6D:b0dau30 ftpd[5157740]: connection from 195.76.117.5
Nov 19 20:25:44 6A:b0dau30 unix: NFS3 lookup failed for server localhost (nsd):
Timed out [filter /usr/sbin/klogpp failed: too slow]

 Note nothing in /ns
 Try nsadmin restart, get

nsadmin> restart
Undefined subroutine &main::EBADFD called at /usr/sbin/nsadmin line 395.
b0dau30.fnal.gov# telnet fnalp

Reboot, now OK

12/01/99
--------
b0dau30 has problems.
Initial symptom is smartsockets functions very slowly.
Restart rtserver 
  (as root, /etc/init.d/smartsockets start)
However, when this is done it is immediately restarted by all the clients.
After several hours, the problem does not subside.
Do:
  source /usr/local/etc/setups.csh
  setup smartsockets
  rtserver -stop_all 
to kill clients, and prevent auto restart.
Restart rtserver again. Still bad. Get messages like:
 WARNING: Could only recv -1/8 first bytes from connection server:tcp:b0dau30.fnal.gov:5101:589.
Reboot system.
Still get above messages, and smartsockets clients cannot connect.
Do /usr/etc/netstat -a
See many many (>100) sockets connected to b0dap09 on port 5101
Nothing obviously offensive running on b0dap09.
Don't know what port 5101 is.
Run tcpdump on b0dau30 - ~patrick/tcpdump-3.4/tcpdump
 /cdf/people2/patrick/tcpdump-3.4/tcpdump -c 100 host b0dap09
See packets like:
15:55:56.651433 b0dap09.fnal.gov.29795 > b0dau30.fnal.gov.5101: S 998298352:998298352(0) win 512 <mss 1460>
15:55:56.651670 b0dau30.fnal.gov.5101 > b0dap09.fnal.gov.29795: S 228357793:228357793(0) ack 998298353 win 48180 <mss 1460> (DF)
15:55:56.651798 b0dap09.fnal.gov.29795 > b0dau30.fnal.gov.5101: . ack 1 win 30660 (DF)
15:55:58.671668 b0dap09.fnal.gov.29796 > b0dau30.fnal.gov.5101: S 2952953508:2952953508(0) win 512 <mss 1460>

The sequence information in the packets is absurd. Apparently this
causes problems for the SGI tcp stack.

Reboot b0dap09, smartsockets immediately starts working fine.

12/09/99
--------
Enable bootp for Level 2 processors:

edit /etc/inetd.conf to enable bootp
/etc/killall -HUP inetd

edit /etc/bootptab to add line for l2 processor

(If not found here, will look in /etc/ethers)

01/07/00
--------
Ethernet connection to outside world lost.
CD networking group determines fiber uplink
from small Cisco switch bad. Put in different
switch

01/12/00
--------
When shutting down Oracle database to increase
number of allowed processes parameter, system
runs out of swap space. This also prevents
new processes from being created.
Oracle shutdown process thrashes for a long time.
Nelly gets advice from Oracle on shutting it down.

Add swap space (using unused partition on Flashdisk):
swap -a /dev/dsk/dks17d10s7

Add to fstab:
/dev/dsk/dks17d10s7 swap swap pri=0

Smartsockets still flaky, so reboot system.

02/10/00
--------
Note on boot the Micron box gives Level 2 cache
failures on processors 1 and 2.
Mike M. comes out to look.
Calls Micron, they suggest voltage regulator
problems. Turns out there are 6 regulators for
4 processors. Move around suspect regulators
and cause processors 3 and 4 to fail as expected.
On Friday, replace regulator and all is well.

02/14/00
--------
reports of b0dau31 being slow previous day. Mako attempted
reboot. Didn't come up.
Tony comes by and says Level 3 network seems flaky.
Unplug 4 Level 3 network ethernet cables.
System them comes up fine.

Also note Micron appears to have same problem. Unplug
Level 3 network cable from there, it then also appears OK.

02/17/00
--------
Finally install cvs version 1.10. In addition to just upd install,
need to:

  Make current in current.chain
  Make sure version file points to old table file that sets CVSROOT
  Make link from "current" to v1_10
  Make sure cvsh is still accessible as cvs login shell

02/24/00
--------
Change various things in offices to subnet 234.
Apparently the gateway is supposed to be 131.225.235.200, not 234.
So, in Linux:

  in /etc/sysconfig/network:
    GATEWAY=131.225.235.200

  in /etc/sysconfig/network-scripts/ifconfig-0
    NETWORK=131.225.235.0
    BROADCAST=131.225.235.255
    NETMASK=255.255.255.0  (255.255.0.0 is rejected by route add command)

On Sun:

  in /etc/inet/netmasks
    change line to
    131.225.235.0 255.255.252.0
    seems to automatically convert the netmask to ffff0000
    Actually fffffc00 would be sufficient, but couldn't get this to happen

On SGI:

  in /etc/config/ifconfig-1.options
    netmask 255.255.252.0 broadcast 131.225.235.255
    This does cause the netmask to get set to fffffc00

02/28/00
--------
  36GB disk on Micron:
    Previously found SCSI ID jumpers were set. Once they were
    removed, the drive showed up on the BIOS SCSI scan. Also,
    Linux fdisk would print the partition table. Interestingly
    it can now recognize SGI format drives, though can't do
    anything with them. However, attempts to rewrite the
    partition table resulted in SCSI errors showing up in
    the /var/log/messages file, and no change to the table.
    Finally found old note from Stephan suggesting to insert
    the "Disable sync negotiation" jumper. After doing this,
    the drive is now writeable with partition magic.
    First try using this hung at 59% of the bad block check.
    Disabled this, second try hung at 60% of extend inode table.
    Give up to try later under Linux.

    Finally installed 6.1. Problems of last 2 weeks were in
    passing the "linux mem=128M" to the install kernel caused
    it to use the GUI installer without FNAL modifications.
    Turns out passing the memory size is no longer needed.
    System then installs fine selecting the cdf online workgroup.

    Then, use fdisk to just add a partition covering the entire
    disk. Creates fine. Then make file system with mkfs -t ext2

02/29/00
--------
    To install zapf-dingbats fonts on redhat 6:

    Get urw-fonts.tar.gz from gimp.org
    Untar in /usr/X11R6/lib/X11/fonts
    chown root:root URW - WON'T WORK UNLESS THIS IS DONE!
    In /etc/X11/fs/config, add a line pointing to the URW
    fonts right after the unscaled lines. This is because
    RedHat 6 uses a font server (xfs) to render fonts, rather
    that the X server.

    See www.gimp.org/fonts.html

03/08/00
--------
    Connected online switch to new rsa router, blocked all
    traffic except that destined for subnet 236.
    Created problems with slow controls PCs which
    were connected to the switch but had 234/5 addresses.
    Put in new satellite switch connected to building
    ethernet, plugged all those PCs into that.

    Still have problem with workstations in pit.
    Eventually resolved by converting thinwire
    to RJ45 and plugging into switch port in
    trigger room. Problems with ports being
    automatically disabled eventually resolved.
    MCS PC in pit re-addressed to 236.12

03/16/00
--------
    Converted MVME ethernet ports back to autonegotiate.
    In the course of testing, note that throughput to
    some PCs (b0dap06, b0dap09) is slow, but fast to
    b0dap20. Turns out PC-PC throughput is even slow.
    Note that these have the Digital ethernet cards,
    and the Linux installation selected the tulip
    driver rather than the de4x5 driver.
    Modify /etc/conf.modules to replace tulip with
    de4x5 and reboot. Things work much better.

03/17/00
--------
    It was suggested that problems with overflowing
    PATH length could be alleviated by a newer version
    of tcsh. Installed and declared current latest
    version from kits (6.09). This puts version in
    /usr/local/bin/tcsh. Move SGI version in /bin
    out of the way and do
    ln -s /usr/local/bin/tcsh /bin/tcsh
    Thus the default is the new version.

03/20/00
--------
    Do the same things as above on b0dap30.
    This must be done on the other machines as well:
    Do:
     ups installasroot tcsh v6_09"
     mv /bin/tcsh /bin/tcsh.rpm   (to move it out of the way)
     ln -s /usr/local/bin/tcsh /bin/tcsh

    Skipped b0dap15, dap19

04/07/00
--------
    To make emacs work on b0dau00-3 it appears necessary to do:

    mkdir /usr/local/products
    mkdir /usr/local/products/emacs
    ln -s /cdf/code/products/emacs/v19_34b /usr/local/products/emacs/v19_34

04/18/00
--------
    Install new 36 GB disks on b0dau30, targeted for more people space.
    Try and install disks on b0dap30. 36GB disk doesn't really fit.
    9 GB disk causes drive letters to change, so remove for now.

04/20/00
--------
    Subnet 206 completely removed with 1 minute notice!#%@#$^@$#%.
    Within an hour, b0dau30 networking gets in odd state.
    rlogin anywhere says host not found, but nslookup finds it.
    Also NIS clients can't bind to b0dau30, only b0dau32.
    Reboot.
    Following reboot, a few PCs can't nfs mount Linux code from b0dap30.
    Reboot b0dap30.

05/05/00
--------
    flpr was not in /usr/local/bin on b0dau0x.
    Log on to those machines as root
    source /usr/local/bin/setups.sh
    ups declare -c -f IRIX+6 flpr v1_12
    this does the copy with suid bit set.
    Can't update ups database in /cdf/code but that is OK

05/09/00
--------
    Moved people1 to new disk.

    System crashed at 15:40. Oracle database corrupted.

    START|957912242|Tue May  9 17:44:02 2000
SUMMARY|Begin
    PANIC STRING: PANIC: CPU 2: TLBMISS: KERNEL FAULT
SUMMARY|End

05/10/00
--------
    Look at fru.x files in /var/crash.
    All three say software problems at 90% c.l.

    Make "cold" backup of database:
    cd /dbs1
    find . -print | cpio -pdum /gobi_backup/dbs1
    same for dbs2, dbs3

    Then tar to tape:
    cd /gobi_backup
    tar cKf /dev/rmt/tps16d5nrnsc .
xfs_db -r -c frag /dev/dsk/dks1d2s7

05/15/00
--------
    b0dap30 was hung when came in. Rebooted OK.

05/20/00
--------
    Did some things to b0dap30 in working on smartsockets problem.
    Installed strace 4.2 from linux-rep RedHat 6.2 distribution
    Stll cannot strace threaded programs.
    Also installed source rpm for this. Goes to /usr/src/redhat
    Made netstat suid so everyone can see what process is bad.

05/22/00
--------
    Informed that b0dau32 cannot access cdfsga. Found in
    /etc/init.d/network.local that it was still configuring
    routes assuming 109 was on the same wire.
    Changed this. Tried to implement without reboot.
    Flush routes, then execute network.local
    However no path to LOCALHOST even.
    Do network stop; network start, still doesn't work.
    So reboot.
    After reboot, b0dau31 crashes, and b0dau30 cannot
    access outside nodes.
    Note that network.local on b0dau30 also still bad,
    so have no idea why it never had trouble accessing cdfsga.
    Have to reboot b0dau30.
    Forget that smartsockets doesn't autostart still.....

05/23/00
--------
    Removed offline upsdb from /usr/local/etc/upsdb_list
    as per request of Art Kreymer.
    Now ~cdfsoft/cdf2.cshrc will prepend the offline
    upsdb to $PRODUCTS.

06/01/00
--------
    On b0dap30, set up cron job in /root to do code disk backup.
    Modified /etc/aliases to send root mail to patrick, run newaliases

06/9/00
-------
    After adding accounts chech and eremina, nis passwd file
    appears corrupt. Noone can log in, or further log into
    another system. Remove accounts for now. Never on Friday...

06/13/00
--------
    Copy current system to old system disk (with zero'd passwd file).
    Then copy last nights /system_backup to spare disk in cabinet.
    Could not copy current system disk, presumably because it
    would have involved copying a 9GB file system to a 4 GB filesystem.
    Remove spare disk and put back in cabinet.

    Try moving chech/eremina elsewhere in passwd file.
    Seems to work at first, but stops before long.
    Note problem seems associated with chech account.

06/14/00
--------
    Finally find that changing chech uid from 9634 to 9635 allows it
    to work. Very strange......

06/16/00
--------
    /var/ns/domains/b0daq/ypservers has relevant list,
    NOT /var/yp/ypservers

    Again get errors adding account peitz:
    b0dau30.fnal.gov# yppush[1715418]: From b0dau31 about passwd.byuid: Failed - no order number in map; use ypxfr -f

06/22/00
--------
    Problems with b0dap30:

    0830 Complaints from control room that PCs are slow to log in. 

Note that b0dap30 is hung, can't log in from console.
Hit reset. On boot, fschk fails on /dev/sdb1 (code disk):

/dev/sdb1 Inode 4482482 has illegal block(s)
/dev/sdb1 UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY
          (i.e. without -a or -p options)

Continue boot to single user mode.
Edit /etc/fstab to remove mount of sdb1

Boot to multiuser mode.

Each evening, the code disk is copied to b0dau30:/linux_backup
So modify fstab on each PC to pick up code-Linux-2.0.34 from there:

b0dau30:/linux_backup    /cdf/code-Linux-2.0.34 nfs soft,rsize=8192,wsize=8192 0 0

(comment out the line that mounts it from b0dap30)

On each PC:
  edit /etc/fstab as above
  umount /cdf/code-Linux-2.0.34
  mount -a

  If the umount fails because the device is busy, it is necessary
  to reboot the PC

Also, on b0dap30 add the mount of /cdf/code-Linux-2.0.34 from b0dau30
to /etc/fstab and do mount -a

The attempt to repair the original filesystem:

[root@b0dap30 /root]# /sbin/e2fsck -r /dev/sdb1
e2fsck 1.17, 26-Oct-1999 for EXT2 FS 0.5b, 95/08/09
/dev/sdb1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 4482482 has illegal block(s).  Clear<y>? yes

Illegal block #12 (3252553934) in inode 4482482.  CLEARED.
Illegal block #13 (1857029597) in inode 4482482.  CLEARED.
Illegal block #14 (3403155662) in inode 4482482.  CLEARED.
Illegal block #15 (98419166) in inode 4482482.  CLEARED.
Illegal block #16 (147746488) in inode 4482482.  CLEARED.
Illegal block #17 (3252603864) in inode 4482482.  CLEARED.
Illegal block #18 (3680509405) in inode 4482482.  CLEARED.
Illegal block #19 (98371955) in inode 4482482.  CLEARED.
Illegal block #20 (147746496) in inode 4482482.  CLEARED.
Illegal block #21 (98421720) in inode 4482482.  CLEARED.
Illegal block #22 (147746504) in inode 4482482.  CLEARED.
Too many illegal blocks in inode 4482482.
Clear inode<y>? yes
Inode 4482485 has illegal block(s).  Clear<y>? yes

Illegal block #14 (9437252) in inode 4482485.  CLEARED.
Illegal block #80 (17104932) in inode 4482485.  CLEARED.
Illegal block #83 (17105056) in inode 4482485.  CLEARED.
Illegal block #89 (17104964) in inode 4482485.  CLEARED.
Illegal block #92 (17104964) in inode 4482485.  CLEARED.
Illegal block #95 (17104964) in inode 4482485.  CLEARED.
Illegal block #101 (17301540) in inode 4482485.  CLEARED.
Illegal block #104 (17301664) in inode 4482485.  CLEARED.
Illegal block #107 (17301664) in inode 4482485.  CLEARED.
Illegal block #110 (17301572) in inode 4482485.  CLEARED.
Illegal block #113 (17301572) in inode 4482485.  CLEARED.
Too many illegal blocks in inode 4482485.
Clear inode<y>? yes

Restarting e2fsck from the beginning...
/dev/sdb1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'put__71IoPackage_f953eaa1.int.c' in /cdfoffline/dist/releases/development/tmp/Linux2-KCC_3_3/CalibDBTables/libCalibDBTables_oci-static/ti_files (4480678) has deleted/unused inode 4482482.  Clear<y>? 

Entry 'keyExists__71IoP_5a62c458.int.c' in /cdfoffline/dist/releases/development/tmp/Linux2-KCC_3_3/CalibDBTables/libCalibDBTables_oci-static/ti_files (4480678) has deleted/unused inode 4482485.  Clear<y>? yes

ss 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  +3385 +3386 -3470 -3471 -5440355 -5440356 -8985786 -8985787 -8985788 -8985789 -8985790 -8985791 -8985792 -8985793 -8985794 -8985795 +8985823 +8985824 +8985825 +8985947 +8985948 +8985949 +8985950 +8985951 +8985952 +8985953 -8987191 -8987192 -8987193 -8987194 -8987195 -8987196 -8987197 -8987198 -8987199 +8987200 -8987234 -8987235 -8987236 -8987237 -8987238 -8987239 -8987240 -8987241 +8987243 +8987244 +8987245 +8987246 +8987247 +8987248 +8987249 -8987302 -8987303 -8987304 -8987305 -8987306 -8987307 -8987308 -8987435 -8987436 -8987437 -8987438 -8987439 -8987440 -8987441 -8987442 -8987443 -8987444 -8987445 -8987446 -8987447 -8987448 +8987489 +8987490 +8987491 +8987492 +8987493 +8987914 +8987915 +8987916 -8988201 -8988202 -8988203 -8988204 -8988205 -8988206 -8988207 -8988208 -8988209 +8988229

Fix<y>? yes

Free blocks count wrong for group #166 (28540, counted=28542).
Fix<y>? yes

Free blocks count wrong for group #274 (24626, counted=24656).
Fix<y>? yes

Free blocks count wrong (6276415, counted=6276447).
Fix<y>? yes

Inode bitmap differences:  -4482482 -4482485
Fix<y>? yes

Free inodes count wrong for group #274 (14308, counted=14310).
Fix<y>? yes

Free inodes count wrong (4296545, counted=4296547).
Fix<y>? yes


/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 216605/4513152 files (11.3% non-contiguous), 2744733/9021180 blocks


It looks like the files that are corrupt are temporary files
generated during C++ compilation. Most likely the  filesystem
is OK, and sometime soon the above procedure will be reversed.

06/22/00
--------
  Computer security comes by and sez FBI sez that root password
  on sahara may be compromised.
  Removed from net.
  However system then thrashes too much to examine.
  Start changing root passwords on other machines.
  Shut down unused systems at end of day.

06/23/00
--------
  Change root passwords on all machines to new paradigm
  CSL passwords are slightly different
  SGI workstations only have 8 characters
  Modify /etc/inetd.conf to disable telnet/rlogin/ftp

  Looking at sahara:
    Note bncIRIX program running as user guest
    Find ~guest/.stuff/bncIRIX program dated June 22
    bnc is an IRC proxy server program
    modification time on passwd file 00:50 June 22
    guest login from ip216.memphis14.tn.pub-ip.psi.net on June 19
                from fnsf223 from 12:51-4 June 22
                from fn835y from 0048:53 June 22
    no anomalous root logins
    CD says ftp from sahara to 195.89.4.20 (shmi.globalcom.net)
                from 12:53-12:55 June 22

  Also, go through and restore use of original code-Linux-2.0.34 disk

07/03/00
--------
  Add swap space to b0dap30:

  dd if=/dev/zero of=/cdf/code-Linux-2.0.34/swapfile bs=1024 count=524288
  chmod 600 /cdf/code/swapfile
  /sbin/mkswap /cdf/code-Linux-2.0.34/swapfile
  /sbin/swapon /cdf/code-Linux-2.0.34/swapfile
  
  should add to /etc/fstab
  /cdf/code-Linux-2.0.34/swapfile none swap defaults

07/06/00
--------
  Note people1 backup tape is filling up.
  For obsolete users, tar/gzip their directories.
  Move o and > back to people2 disk

07/14/00
--------
  b0dap30 hangs, apparently due to problems with swapper.
  Note that memory had been saturated lately.

07/26/00
--------
   nsd dies on b0dau30, leaving 220+MB core file in /var/ns.
    root partition fills. Restart nsd by just "nsd" command.

07/28/00
--------
   Install tcsh v6_09 on b0dau0x machines following similar
   instructions to above for Linux machines.
   Found that password for b0dau00 was still the old format,
   so changed it to the new format.

08/01/00
--------
   Add router module to switch

   Reboot b0dau30.

   Note after reboot, accounts reichold, mjones, yun, and rekovic
   seem to add OK. They all gave problems before the reboot.

08/07/00
--------
   Serial numbers for b0dau30/31:
    690D1E81 system #S04449 property #'s 086756, 086757
    690D21A1 system #S04452 property #'s 086632, 086754,086755

08/09/00
--------
   System Crash:

(analysis.5)
=======================
ICRASH CORE FILE REPORT
=======================

SYSTEM:
    system name:    IRIX64
    release:        6.5
    node name:      b0dau30
    version:        01221553
    machine name:   IP27

GENERATED ON:
    Wed Aug  9 18:07:46 2000

TIME OF CRASH:
    965862069 Wed Aug  9 18:01:09 2000

PANIC STRING:
    PANIC: CPU 3: TLBMISS: KERNEL FAULT

...

FRU ANALYSIS BEGIN
    <6>++
    <6>++
    <6>++                       FRU Analysis Summary
    <6>++
    <6>++       Software                                        : 90%
    <6>++
    <6>++FRU ANALYSIS END

    <0>PANIC: CPU 3: TLBMISS: KERNEL FAULT
    PC: 0xc00000000014f290 ep: 0xa8000001005f1c28
    EXC code:8, `Read TLB Miss '
    Bad addr: 0xc0000fc000000000, cause: 0x8<CE=0
    <6>

08/15/00
--------
    System hangs. From messages in log file, it looks like nsd crashed.
Aug 15 10:17:21 6A:b0dau30 unix: NFS3 lookup failed for server localhost (nsd):
Timed out
Aug 15 10:17:27 3D:b0dau30 nsd[200]: NIS server for domain b0daq not responding.
Aug 15 10:17:41 6A:b0dau30 unix: NFS3 lookup failed for server localhost (nsd):
Timed out
Aug 15 10:17:51 6A:b0dau30 unix: NFS3 getattr failed for server localhost (nsd):
 Timed out

08/16/00
--------
Moved code-IRIX-6.5 area and rebooted.
Note have to re-nfs mount everywhere

08/18/00
--------
Had moved code-common-backup to dks17d10s7
Did successful rsync during the day.
However, the evening rsync indicated a corrupt file system.
Can't unmount it.
Make code-common-backup2 on dks17d10s13 and try

08/22/00
--------
b0dap30 hangs. 
Responds to ping, but no response at console or to . 
Reboot by hardware reset button. 
On reboot, remote PCs cannot mount /cdf/code-Linux-2.0.34 disk. 
On b0dap30, note nfs mount daemon not running 
  (ps -ef | grep mountd shows nothing) 
So do /etc/rc.d/init.d/nfs stop 
      /etc/rc.d/init.d/nfs start 
This seems to correct the problem. 
Don't understand why the system hung, or why nfs crashed after reboot.

08/24/00
--------
Prior to 0700, the root partition on b0dau30 filled. 
The system was rebooted shortly after 0800. 
It is unknown what additional files were added.

 - patrick 

08/24/00
--------
b0dap30 hung again, exactly as yesterday 
at about this time. It was reset, and again 
the nfs mount daemon was not running when 
the reboot finished.

08/24/00
--------
It is likely that the b0dap30 hangs were due to TrigSim++. 
This program has a memory leak, shortly after the reboot 
it was observed to be consuming 1GB of virtual memory 
and was still increasing when it was killed.


08/29/00
--------
Mako finds the following at ~2300 last evening:

About half an hour ago, b0dau30 suddenly hung. I went over to B0 and
saw the following message on the console, which happened to be showing
b0dau30's output:

  WARNING: /hw/module/2/slot/MotherBoard/node/cpu/a: Exception during show hardware state

The console itself was frozen, so I HRRed the node. It looks OK now,
though disks feels somewhat slow on NFS clients.

                                        mako


In the morning, the RAID array is not accessible.
Power cycle machine, it returns.

08/29/00
--------
b0dap30 hangs. Reboot. Again nfs exports do not come up.
Note finally that nfs is not in the rc3/rc5 areas.
Install the links
ln -s /etc/rc.d/init/d/nfs /etc/rc.d/rc3.d/S70nfs
ln -s /etc/rc.d/init/d/nfs /etc/rc.d/rc5.d/S70nfs

08/29/00
--------
root partition on b0dau30 nearly full.
nsd crashes.
Restart nsd by "nsd" command
Find cause of excessive disk use in root partition
was "orphaned" partial /cdf/people2_backup.
This apparently was not fully deleted when an
rsync was started without this mounted.
So:
umount /cdf/people2_backup
cd /cdf/people2_backup
rm -rf *
mount -a
and it is much better.

08/29/00
--------
To install new netscape:
rpm -e netscape-communicator-4.73-1
rpm -e netscape-common-4.73-1
rpm -i ftp://linux-rep.fnal.gov/pub/mirrors/redhat/updates/6.0/i386/netscape-common-4.75-0.6.2.i386.rpm
rpm -i ftp://linux-rep.fnal.gov/pub/mirrors/redhat/updates/6.0/i386/netscape-communicator-4.75-0.6.2.i386.rpm

09/09/00
--------
System appears to hang at 08:59:

Sep  9 08:58:32 6E:b0dau30 rshd[8153965]: vxworksb@b0clc01.fnal.gov as vxworksb:
 cmd='/bin/cat < /cdf/people1/cdf_clc/online/clcdaq/DPMdata.txt'
Sep  9 08:58:34 6E:b0dau30 rshd[7922409]: vxworksb@b0clc01.fnal.gov as vxworksb:
 cmd='/bin/cat > /cdf/people1/cdf_clc/online/clcdaq/DPMdata.txt'
Feb 28 11:43:14 6F:b0dau30 syslogd: restart
Feb 28 11:43:14 2A:b0dau30 unix: : Ending XFS recovery for filesystem: /hot_back
up (/hw/module/2/slot/MotherBoard/node/xtalk/8/pci/0/scsi_ctlr/0/target/1/lun/0/
disk/partition/0/block)
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending clean XFS mount for filesystem:
/backup
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /cd
f/code-common-backup2 (/hw/module/1/slot/MotherBoard/node/xtalk/8/pci/6/scsi_ctl
r/0/target/10/lun/0/disk/partition/13/block)
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /li
nux_backup (/hw/module/1/slot/MotherBoard/node/xtalk/8/pci/6/scsi_ctlr/0/target/
10/lun/0/disk/partition/6/block)
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /go
bi_backup (/hw/module/1/slot/MotherBoard/node/xtalk/8/pci/6/scsi_ctlr/0/target/1
0/lun/0/disk/partition/3/block)
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /cd
f/people1_backup (/hw/major/192/minor/7)
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /cd
f/people2_backup (/hw/major/192/minor/8)
Feb 28 11:43:14 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /da
ta1 (/hw/major/192/minor/9)
Feb 28 11:43:21 3D:b0dau30 nsd[200]: WARNING: Future date on /etc/passwd: Fri Se
p  8 08:29:05 2000
Feb 28 11:43:22 3D:b0dau30 nsd[200]: WARNING: Future date on /etc/hosts: Fri Sep
  8 12:54:58 2000
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.8.12
0: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.8.12
0: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.8.12
0: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.8.12
0: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.8.12
0: No route to host
Feb 28 11:43:22 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.17.1
50: No route to host
Feb 28 11:43:23 3D:b0dau30 nsd[200]: dns_query: failed final sendto 131.225.8.12
0: No route to host
Feb 28 11:43:29 6D:b0dau30 sn0log: The following are messages stored in the flas
hlog from a previous system boot.
@

Feb 28 11:43:29 6D:b0dau30 sn0log: Flashlog for /hw/module/1/slot/MotherBoard/no
de/hub/mon
Feb 28 11:43:29 5D:b0dau30 sn0log: B Fatal: Serial #: i^M!!
Feb 28 11:43:29 5D:b0dau30 sn0log: B Fatal: B Fatal: B Fatal: B Fatal: (
Feb 28 11:43:29 5D:b0dau30 sn0log: B Fatal:
Feb 28 11:43:29 6D:b0dau30 sn0log: End of flashlog for /hw/module/1/slot/MotherB
oard/node/hub/mon
Feb 28 11:43:29 6D:b0dau30 sn0log: End of flashlog messages.
Feb 28 11:43:31 3D:b0dau30 nsd[200]: WARNING: Future date on /etc/group: Wed Sep
  6 12:31:18 2000
Feb 28 11:43:31 5B:b0dau30 sendmail: Freezing sendmail.cf
Feb 28 11:43:34 6C:b0dau30 sendmail[632]: starting daemon (980427.SGI.8.8.8): SM
TP+queueing@00:15:00
Feb 28 11:43:34 6E:b0dau30 rshd[637]: vxworksb@b0clc01.fnal.gov as vxworksb: cmd
='/bin/cat > /cdf/people1/cdf_clc/online/clcdaq/DPMdata.txt'
Feb 28 11:43:37 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 1 second
Feb 28 11:43:37 6B:b0dau30 syslog: FCAGENT exiting
Feb 28 11:43:38 7B:b0dau30 syslog: {start,stop}midi entered
Feb 28 11:43:38 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 2 seconds
Feb 28 11:43:38 4D:b0dau30 rtmond[967]: /tmp/.rtmond_pid: Purging old lock file;
 pid was 1518
Feb 28 11:43:39 3B:b0dau30 syslog: sdpd: fam error (Operation not permitted)
Feb 28 11:43:40 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 3 seconds
Feb 28 11:43:43 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 4 seconds
Feb 28 11:43:47 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 5 seconds
Feb 28 11:43:50 5E:b0dau30 su[1045]: succeeded: console changing from root to or
acle
Feb 28 11:43:52 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 6 seconds
Feb 28 11:43:58 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 7 seconds
Feb 28 11:44:00 5E:b0dau30 su[1061]: succeeded: console changing from root to or
acle
Feb 28 11:44:01 5E:b0dau30 su[1066]: succeeded: console changing from root to or
acle
Feb 28 11:44:05 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 8 seconds
Feb 28 11:44:13 1D:b0dau30 mediad: Can't connect to fam: No such file or directo
ry.  Sleeping 9 seconds
Feb 28 11:44:18 5E:b0dau30 su[1131]: succeeded: console changing from root to hy
pertxt
Feb 28 11:44:19 5E:b0dau30 su[1137]: succeeded: console changing from root to pi
ccoli
Feb 28 11:44:20 5E:b0dau30 su[1209]: succeeded: console changing from root to cd
fdaq
Feb 28 11:44:20 5E:b0dau30 su[1259]: succeeded: console changing from root to cd
fdaq
Feb 28 11:44:21 5E:b0dau30 su[1354]: succeeded: console changing from root to cd
fdaq
Feb 28 11:44:22 6D:b0dau30 sshd[1443]: log: Server listening on port 22.
Feb 28 11:44:22 6D:b0dau30 sshd[1443]: log: Generating 768 bit RSA key.

09/13/00
--------
b0dap30 crashes and reboots. Reason unknown.
Satellite PCs don't immediately remount disks?
Redo exportfs -a on b0dap30, after this everything
came back. But it is unknown whether they had
come back anyway.

09/15/00
--------
b0dap30 hangs, reboot via reset button.
Again, satellite PCs don't immediately
remount disks. Happened immediately
upon doing exportfs -a

b0dap30 hangs again around 8 p.m.
people in area eventually reset

09/18/00
--------
b0dap30 hangs around 1 p.m.
cpumon dump shows physical memory mostly
used up, very little free. But had been
almost used up earlier.
Note that swap space is only 117MB.
Line to add /cdf/code/swapfile is commented
out, from when disk did not fschk a while ago.
Uncomment line, then do
swapon -a
and this adds another 500MB of swap space.

09/18/00
--------
root partition fills up. Frantically
free space, keeps refilling.
Upon doing compare with hotspace,
it magically drops by 15%...
Don't have to reboot.
????????????????????????????????

b0dap30 hangs again at ~16:00

Install 2.2.16-3 kernel manually
as per Connie Sieh's instructions:

rpm install kernel related files
mkinitrd
edit lilo.conf

09/19/00
--------
Enable routing function on 6509 switch.
Also add 48 new ports

Also did a disk shuffle to get more space
for the system disk:

Current system disk (4GB) removed
Previous 9GB backup system disk made the current
  system disk
/dbs1 disk moved from dks0d6 to dks14d1,
  divided into to partitions (/hot_backup and /stable_backup)
  to as to be the new backup system disk

Copied contents of the 5 1/4" drive (/cdf/commercial)
to /backup/commercial and made the symbolic link
ln -s /backup/commercial /cdf/commercial

Removed this disk and replaced it with a CDROM drive


09/20/00
--------
b0dau30 S04449 serial # 690d1e81
b0dau31 S04452 serial # 690d21a1

09/29/00
--------
Major disk shuffle on b0dau30, create /cdf/people3

Add disk to b0dap30 for level 3 code

10/02/00
--------
b0dap30 hangs cold, doesn't respond to ping.
Reset from front panel. First time in 10 days...

10/03/00
--------
Tried to mount the new level 3 disk on the gateway node.
Gives rpc portmap timeout.
Same problem observed on b0dap23, so not l3 specific.
Try /etc/rc.d/init.d/nfs restart plus other flailing around,
it doesn't work.
Finally stop then start portmapper, then nfs restart works.
Then can mount disk everywhere.

10/05/00
--------
Interpolating from 10/10, b0dap30 must have been
rebooted today?

10/10/00
--------
b0dap30 hangs, had been up 5 days, 16:31
Responded to ping.
Repeated messages on console:

nfsd Security: /// bad export

tail of cpumon program:

  3:11pm  up 5 days, 16:31, 36 users,  load average: 0.19, 0.55, 0.66
227 processes: 226 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  2.1% user,  3.2% system,  0.0% nice, 94.5% idle
Mem:  1036276K av, 955172K used,  81104K free,  84444K shrd,  20976K buff
Swap: 642032K av,  18440K used, 623592K free                450800K cached
  3:11pm  up 5 days, 16:31, 36 users,  load average: 0.25, 0.56, 0.67
234 processes: 233 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  6.9% user,  4.6% system,  0.0% nice, 88.4% idle
Mem:  1036276K av, 957820K used,  78456K free,  88300K shrd,  20980K buff
Swap: 642032K av,  18440K used, 623592K free                451952K cached
36K used,  74840K free,  88300K shrd,  20980K buff
Swap: 642032K av,  18440K used, 623592K free                455636K cached
  3:11pm  up 5 days, 16:31, 36 users,  load average: 0.42, 0.58, 0.67
234 processes: 231 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 16.0% user, 18.2% system,  0.0% nice, 65.6% idle
Mem:  1036276K av, 964064K used,  72212K free,  88316K shrd,  20980K buff
Swap: 642032K av,  18440K used, 623592K free                458144K cached

System was reset at 15:25, so the cpumon program was hung before then.

DISCOVER THAT THE DEFAULT KERNEL WAS STILL THE 2.2.14 KERNEL.
B0DAP30 had run for a while with no crashes, following the
original upgrade (09/18 to 10/02) however on 9/29
was rebooted back to the 2.2.14 kernel when a disk was added.
Change default in /etc/lilo.conf then run /sbin/lilo

10/11/00
--------
Rebooted b0dap30 to 2.2.16-3 kernel.
Also added 2 more 36GB disks.
Recent summary of b0dap30 reboots from remaining messages files:
Sept 13 - 14:31
Sept 15 - 11:41 and again @20:43
Sept 18 - 13:23 and again @16:19
At this point boot to kernel 2.2.16-3
Sept 29 - 08:01 - intentional shutdown to install disks
Rebooted back to 2.2.14 kernel
Oct 02 - 12:51
Oct 04 - 22:45
Oct 10 - 15:33
Oct 11 - Intentional reboot to revert to 2.2.16 kernel
Oct 02


11/06/00
--------
Locations of oracle java files:
<b0dau30-1-> cd /cdf/code-common
<b0dau30-2-> find . -name "classes111.zip" -print
./products/jdbc_oracle/v8_1_6/lib/classes111.zip
./products/oracle_jdbc/v8_0_4/classes111.zip
./cdfonline/dbbroker/v2_0/classes/classes111.zip
./cdfonline/dbbroker/v2_2/classes/classes111.zip
./cdfonline/dbbroker/debug/classes/classes111.zip
./cdfonline/resman/v0_0/lib/classes111.zip
./cdfonline/resman/v0_1/lib/classes111.zip
./cdfonline/resman/v1_0/lib/classes111.zip
./cdfonline/resman/v1_1/lib/classes111.zip
./code-IRIX64-6.2/oracle/client/v8_0_4_1/jdbc/lib/classes111.zip

<b0dau30-3-> cd /cdf/code
<b0dau30-4-> find . -name "classes111.zip" -print
./products/oracle/v8_0_5/jdbc/lib/classes111.zip
./products/oracle/v8_0_5a/jdbc/lib/classes111.zip
./products/oracle/v8_0_5b/jdbc/lib/classes111.zip
./products/oracle_jdbc/v8_0_4/classes111.zip
./freeware/jdbc_oracle/v7_3_4/lib/classes111.zip
./cdfoffline/products/oracle_client/v8_0_5b/IRIX+6/jdbc/lib/classes111.zip


<b0dau30-5-> cd /cdf/people1
<b0dau30-6-> find . -name "classes111.zip" -print
./fkw/dbanalyze/classes111.zip
./rvidal/brokerv2_2/classes/classes111.zip
./rvidal/classes111.zip
./iyutin/java/classes111.zip
./johnbrit/dbAccess/classes111.zip


<b0dau30-7-> cd /cdf/people2
<b0dau30-8-> find . -name "classes111.zip" -print
./yuyi/cvs/resman/v0_0/lib/classes111.zip
cannot stat ./svxii/SVX-voltages.html
No such file or directory
./patrick/resmanx/lib/classes111.zip
./patrick/resman/lib/classes111.zip
./rakitin/java/classes111.zip
./wicklund/public_html/classes111.zip

11/07/00
--------
In past weeks have sometimes seen on the b0dau30 console:

increase svc_maxdupreqs from 1501
presumably has to do with nfs.

11/29/00
--------
b0dau30 hangs dead. Console switch was elsewhere. 
Message in log file:
Nov 28 04:34:43 4A:b0dau30 unix: WARNING: Correctable memory ECC error (polled)
Nov 28 04:34:43 4A:b0dau30 unix: WARNING:  Mem: 0x121048ee0, Syn: 0x4 (check_2)
Nov 28 04:34:43 4A:b0dau30 unix: WARNING:  Loc: /hw/module/2/slot/MotherBoard/node/memory/dimm_bank/1 (SLOT3 line 2)
Nov 28 04:34:43 4A:b0dau30 unix: WARNING:  Single-bit error has been repaired.
Nov 28 04:34:43 4A:b0dau30 unix: WARNING: Correctable memory ECC error (intr)
Nov 28 04:34:43 4A:b0dau30 unix: WARNING:  Mem: 0x121048ee0, Syn: 0x4 (check_2)
Nov 28 04:34:43 4A:b0dau30 unix: WARNING:  Loc: /hw/module/2/slot/MotherBoard/node/memory/dimm_bank/1 (SLOT3 line 2)
Nov 28 04:34:43 4A:b0dau30 unix: WARNING:  Single-bit error has been repaired.

Reset and reboots OK. Don't initially see messages,
presumably have to wait for more memory usage...