From: kls@argus.SLAC.Stanford.EDU (Karl L. Swartz)
Newsgroups: comp.sys.sun.admin,comp.unix.wizards
Subject: process(es) stuck in 'D' wait (Sun OS 4.1)
Summary: Help!
Date: 9 Oct 92 03:18:57 GMT
Organization: Stanford Linear Accelerator Center

About a month ago one of our fileservers started running itself out
of various system table slots or nfsds roughly several times per week.
There does not seem to be any frequency pattern -- things are fine for
some time, maybe hours, maybe days, then problems start cropping up.

Background info:  The machine is a Sun 4/390 (big fileserver) with 4
1.2 GB IPI disks, two that came with it and two more identical drives
all on one controller.  It's a bit over two years old and has been
running Sun OS 4.1 since installed.  The kernel has a few patches --
not many, as I'd just as soon upgrade the sucker to 4.1.2 -- and was
last rebuilt in March when I added an FDDI interface.  It's a pretty
lean 'n mean kernel with support for a CD ROM and tapes on SCSI, the
FDDI, a pair of additional Ethernet interfaces, and one controller's
worth of IPI drives.

The problem manifests itself as one or more processes getting stuck
in a 'D' state, a non-interruptible short-term wait, often for disk.
The most likely victims are nfsd and the nightly 'findtrash' script
which gets rid of old cores and the like.  For example, the current
occurance shows up as follows:

$ ps axgu | grep ' D'
root         0  0.0  0.0    0    0 ?  D    Oct  7  1:06 swapper
root         2  0.0  0.0    0    0 ?  D    Oct  7  0:00 pagedaemon
root       179  0.0  0.1   24   16 ?  D    Oct  7  4:58 update
root     23123  0.0  0.0  136    0 ?  D    03:15   1:54 find /u1 /u2 /u3 ...
kls      11595  0.0  0.8   40  264 pa S    20:04   0:00 grep  D

The find (pid 23123) has been stuck there since early this morning.
Turning to pstat, I found more info (in digested form below) which
tells me more about *what* is going on, but not *why*.

$ pstat -fiu 23123     # with extraneous info filtered out and notes added
337/976 open files
f8120d7c vnode   R         1    0  ff3386a8  512
f8121700 vnode   R         1    0  ff226378  512
f812178c vnode   R         1    0  ff2e0388  512
f81217c4 vnode   R         1    0  ff2b3408  1536
f8121ab8 vnode   R         1    0  ff27a388  512

  ILOC   IFLAG  IDEVICE   INO   MODE NLK  UID  SIZE/DEV VFLAG CNT SHC EXC TYPE
ff27a380    R    22, 23 18432  40755  13 1038       512         1   0   0 VDIR
    /dev/id002h inode  18432: /u3/<18432>
ff2b3400    R    22, 23  9216  40755  18 1038      1536         1   0   0 VDIR
    /dev/id002h inode   9216: /u3/<9216>
ff2e0380    R    22, 23  4608  40755  25    0       512         4   0   0 VDIR
    /dev/id002h inode   4608: /u3/<4608>
ff226370    R    22, 23     2  40755   9    0       512 R       3   0   0 VDIR
    /dev/id002h inode      2: /u3
ff3386a0   LR    22, 23189136  40755   4 1038       512         3   0   0 VDIR
    /dev/id002h inode 189136: /u3/<189136>

pid =	23123
cdir =	ff3386a8	/u3/<189136>
fd[3] =	f8121700	/u3		/u3
fd[4] =	f812178c	/u3/<4608>	/u3/el
fd[6] =	f81217c4	/u3/<9216>	/u3/el/jjh
fd[7] =	f8121ab8	/u3/<18432>	/u3/el/jjh/viewlogic
fd[8] =	f8120d7c	/u3/<189136>

I can't identify the last file as ls or find will hang upon trying to
access the file, though after a reboot there will be no problem.  Next
time, find or nfsd may hang on another directory, perhaps on another
drive.  There's no pattern.

The key seems to be the 'L' flag in the inode table entry for the
deepest directory (note that this is also find's cwd) which indicates
the inode is locked.  But why, and why isn't it being unlocked?

Any suggestions on where I might look next would be *greatly*
appreciated!

-- 
Karl Swartz		|INet	kls@unixhub.slac.stanford.edu
SLAC Computing Services	|	   or kls@ditka.chicago.com
1-415/926-3630		|UUCP	uunet!lll-winken!unixhub!kls  -or-  ditka!kls
(SLAC and the US Dept. of Energy don't necessarily agree with my opinions.)