From: jlw@improb.cls.com (Jeff Wannamaker)
Newsgroups: comp.unix.aix
Subject: Re: 530 Performance problems
Date: Wed, 15 Apr 92 23:45:00 CDT
Organization: Improbability Drive

toma@swsrv1.cirr.com (Tom Armistead) writes:

> Is there any software out there for real-time monitoring of processes
> under AIX (something like u386mon)?
> 
> We have a 530 with 128meg ram, ~5gig of disk and about ?30? users on the
> system at any given time.  The system is connect to an IBM mainframe via
> and SNA link and a couple of other 6000's via TCP/IP.  There are also
> 5 or so X-Terminal connected to the network.
> 
> We seem to have always had performance problems on this machine (and never
> been able to nail down what the real problem is).  There are times when
> a user re-draws a screen (using vi) and the screen update will take 30
> seconds.  There are times when one can cd to another directory and have
> to wait 30+ seconds to get the prompt back.  Is this typical?  The SE's
> from IBM have always told us that there wasn't a problem.
> 
> I've always had the feeling the the SE's had absolutely NO idea what they
> were talking about.  I remember how Impressed one 'IBM Performance
> Specialist' was when I showed him that he could edit his command line
> using vi commands???????????????????????????????????????????????????????
> 
> We've had all sorts of people look at the system and no one has been able
> to help in any way.  When we have performance problems like this and stop
> to look at what's wrong, the system speeds back up.
> 
> Any help or suggestions would be greatly appreciated!!!
> 

I hear what you are saying, but there are some nice tools that can help you
monitor what your system is doing.  For example, paging space is critical.
When page space is low, memory is chewed up and thrashing occurs.  When
out of page space, your processes begin to rapidly die.  You can monitor
page space by typing "lsps -a"  whenever system performance is pitiful.
If all page spaces are approaching their capacity, you need to expand or
add more page spaces.  With the gigs of disk storage you have, you should
be able to add more.

As for your 30 users at one time, you didn't mention the types of processes
they are running.  Are they running Xwindows or straight ASCII?  Xwindows
is a memory hog and if you are running AIX 3.2, you will find X grows each
time you pull up a new display or dialog box.  (I can prove this -  IBM is
investigating the case we presented).   If your users are running Xde to
debug, you may want to convince them to use dbx to conserve.  Our shop
uses dbx (ascii debugger) because of this reason.

To monitor memory utilization by individual process, you can use the
"ps vg" command.  It will show lots of great information and you can
monitor a process by entering the following at the command line (Korn
shell):

    while true; do
        ps vg processid     (you enter the process id)
        sleep 5             (adjust delay if necessary, 5 secs in example)
    done

You will find this is pretty good for singling out processes that are
memory gobblers.  And to monitor overall virtual storage (page faults,
cpu loading, and other good stuff), use the "vmstat interval"  command
where "interval" is the refresh rate in seconds.  With this you can see
what may be causing the bottlenecks.

Another tool for monitoring CPU utilization and I/O stats, use the
"iostat interval" command (interval is in seconds).  This is good if
you suspect your system is bogged down because of CPU intensive processes
or if the processes do intensive disk I/O.

If you want VERY fine detail on a particular process, I suggest delving
into the AIX trace and trcrpt function.  This will record all kernel
events for a specified interval of time.  This will produce more detail
than you can imagine (including down to hardware latencies), so you will
have to do some planning on how you want to trigger this function on and
off during the desired window while your process is executing.  You can
find out why your programs are so dang slow.

Since you are not really on a networked system, "netstat" won't be of
much interest to you.  It's good for monitoring packets in/out of your
system, errors/retransmits, and is good for checking network loading.

All of the above have a number of switches, so you can do a lot of tweaking.
I would recommend using InfoExplorer or look at the man pages for these
commands.  Without more detail, I cannot elaborate more than these few
examples.  I wish I had some experience with u386mon, so I can't comment
with regard to similar facilities in AIX.

Jeff Wannamaker
Manager, Development & System Support
Stagg Systems Inc.
Houston, TX
713 496-3470

---
Jeff Wannamaker, jlw@improb.cls.com or ...!uunet!cls!improb!jlw