From: steves@ssec.wisc.edu (Steve Shirshac)
Subject: Problem with AIX3.2 paging subsystem?
Summary: pager from hell
Organization: UW-Madison Space Science and Engineering Center
Date: Thu, 21 Jan 1993 23:05:35 GMT

I think I may have found a problem in the AIX 3.2 paging subsystem.  If
you have any experience with developing device drivers for AIX, I would
appreciate any insights you might have into this problem or how to
isolate it further.  A rather detailed description follows.

background/demographics:

        We have an IBM RISC/6000 model 520h with 32 Mb of physical memory,
        AIX 3.2, and 64 Mb of page/swap space configured.

        We have developed an adapter card which now sits on the MicroChannel
        bus and for which I have written a device driver.
        Our adapter reads a high speed stream of data from an external
        device and blocks it up to perform slave DMA transfers.

problem description:

        My device driver for this adapter seems to perform correctly in
        all major aspects but one.  Once my device driver is configured
        into the AIX kernel, I run a root privileged process to open the
        device, read data from the device into a very large buffer allo-
        cated from shared memory, and close the device once I've read
        enough data.  Fine so far - all works OK.  I can execute this root
        process exactly four times and then the machine runs out of swap
        space.  AIX then starts killing processes to start a new one and
        also complains that there's not enough...            swap space.

        It appears that there is either a problem with (1) the AIX kernel's
	paging subsystem or (2) my device driver.

        The Swapper apparently is having a problem trying to reuse the
        space used to back the memory touched by the root processes which
        have run and completed.  It still thinks that it needs to keep track
        of those used memory pages and thus runs out of space when it hits
        the 64MB limit for which it is configured.

        On the other hand, and most likely, the problem could be the way
        in which my device driver handles the memory that it touches.

        The short version is:
           The root process (call it "fred") allocates a 16Mb hunk of
           shared memory. "fred" then call a kernel extension to pin this
           shared memory hunk. "fred" then opens my device driver, tells
           it the starting location of the shared memory and a block size,
           and lets the device driver fill the shared memory with data.
           Once the shared memory is filled and "fred" is happy with it,
           "fred" closes the device, unpins (via kernel extension) the
           shared memory hunk, and releases the shared memory and term-
           inates.  At this point, the system seems to think that the shared
           memory (16MB) is indeed unpinned and free but the swapper seems
           to think that it still needs to be backed.


        The long version follows:

           privileged user process ("fred") allocates a large shared mem
           segment (16Mb), "fred" pins shared mem seg, "fred" opens our
           device and passes a pointer to the shared mem seg to my device
           driver.

           device driver receives pointer to shared mem segment start.

           Device driver then attaches cross memory descriptors to each of
           1024 16384-byte blocks ("arbitrarily" chosen buffer size) blocks
           which comprise the 16Mb shared mem seg.

           (this is done so that hardware can perform DMA transfers to the
           individual 1024 blocks within the shared mem seg. and so that we
           can still allow viewing access to individual blocks that have
           been filled with data while the remainder of the blocks are
           being filled with data from the adapter.)
           DMA transfers occur (from our hardware to shared mem seg blocks)
           cyclically through the 1024 blocks in shared mem seg. (i.e.,
           DMA transfers occur sequentially, starting with block 0 in the
           shared mem segment and increasing through block 1023.  Once block
           1023 is filled  via DMA transfer of 16384 bytes, we start anew with
           block 0.)
           Upon close of device, DMA transfers are halted and attached cross
           memory descriptors are released.

           privileged user process then unpins the shared mem seg and
           deallocates it.


           upon privileged user process completion:

              "svmon" indicates that the pages used to back the shared mem
                      segment have been unpinned.

              "ipcs" indicates that the shared memory segments which were
                     being used are no longer there.

              "lsps -a" indicates that swap space usage has increased by 16Mb.

              low-level kernel debugger indicates that there are 0 pages of
                     memory that have a xmemcnt > 0.

If the problem is not in the OS, then what could I be doing wrong in
the device driver?  I've exhausted the 'Writing Device Drivers for
AIX3.2' documentation, and found nothing there to help.

Steve Shirshac (steves@ssecmail.ssec.wisc.edu)