From: rmilner@zia.aoc.nrao.edu (Ruth Milner)
Subject: Re: understanding the numbers
Reply-To: rmilner@zia.aoc.nrao.edu (Ruth Milner)
Organization: National Radio Astronomy Observatory, Socorro NM
Date: Fri, 16 Oct 92 23:50:58 GMT

In article <1992Oct16.134212.13136@organpipe.uug.arizona.edu> olevin@random.hwr.arizona.edu writes:

>From lsps -a
>
>Page Space  Phal Vol   Volme Group    Size   %Used  Active  Auto  Type
>paging06    hdisk2       rootvg        40MB      46     yes   yes    lv
>paging05    hdisk2       rootvg        80MB      25     yes   yes    lv
>paging03    hdisk2       rootvg        40MB      50     yes   yes    lv
>paging04    hdisk2       rootvg        40MB      49     yes   yes    lv
>paging02    hdisk2       rootvg        40MB      50     yes   yes    lv
>paging01    hdisk0       rootvg        40MB      50     yes   yes    lv
>paging00    hdisk0       rootvg        40MB      49     yes   yes    lv
>hd61        hdisk1       rootvg        40MB      50     yes   yes    lv
>hd6         hdisk0       rootvg        40MB      66     yes   yes    lv
>
>From ps aux
>
>olevn 7840 1.7 87.0 164220 113744      - S    06:35:23 24:14 particle.x 
>
>You can see that the program is using only 1.7% of the CPU.
>There is some explenation for that? Why is the program only
>using so samll amount of CPU?.....

It probably has something to do with the fact that your program is very large,
using 87% of your 128MB of memory and a sizeable fraction of the total page 
space. This is then made much worse by the fact that half of these page spaces 
are in separate areas on the same disk, and 3 more are all on another disk. 
Your program is memory bound, and you are getting poor paging performance 
because the page requests are badly fragmented, causing lots of conflicting
seeks all over the disk. The fact that you saw iowait times of 90+% is proof
of this.

As to *why* you are doing so much paging, there are some big reasons on the 
programming side, too. A couple of suggestions: first, if this program is 
just being tested, reduce the size of your arrays to make the executable 
smaller. Make sure it works before you increase them again. Second, and much
more important, make sure you are accessing the arrays in column-major order, 
"First Index Fastest", which may not be intuitive but is nevertheless how 
Fortran arranges its data. For example, write your loops like

    DO 20 K=1,N                              DO 20 I=1,N 
      DO 20 J=1,N                not           DO 20 J=1,N 
        DO 20 I=1,N                              DO 20 K=1,N 
20        X(I,J,K)=Y(I,J,K)              20        X(I,J,K)=Y(I,J,K)

Be careful to preserve the logic of your program when you do this. You can't
always simply reverse the indices; you may have to think about it. I have seen 
programs go from taking hours and doing MB of paging, to running in a short 
time and paging only rarely, just by making this correction everywhere.

Spreading paging space across several disks is a good idea, but all the
page space on each disk should be consolidated on a single contiguous area.
As it stands, most of your requests for data in that space are causing lots
of head movement and being slowed down by seek/latency time. Contiguity of
logical volumes on a disk can be checked with "lspv -M hdisk2" (for example).

See if the system's administrator can consolidate the paging space better, so 
it winds up being reasonably contiguous, with most of it on the faster/least 
busy disks. But this may not be at all easy to do at this stage.
-- 
Ruth Milner                          NRAO/VLA                  Socorro NM
Computing Division Head      rmilner@zia.aoc.nrao.edu
Newsgroups: comp.unix.aix