gpu: ion: Use list_for_each_safe on error path

On an error, the list of pages is walked and destroyed. Since the list
is being destroyed, use list_for_each_safe to avoid list corruption.

Change-Id: Ia397a11d2d890b3458745820c79df2adaa088ade
CRs-Fixed: 565183
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Also shrink memory cached in the deferred free list

When the system is low on memory, we want to shrink any cached
system memory ion is holding.  Previously we were shrinking memory
in the page pools, but not in the deferred free list.  This patch
makes it possible to shrink both.  It also moves the shrinker
code into the heaps so they can correctly manage any caches they
might contain.

Change-Id: I177f587f999a5220eddbf7af94745aae736cac75
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Git-commit: 072beca55150c0c0b652bbb24267da271e0bdf19
Git-repo: https://android.googlesource.com/kernel/common
[mitchelh@codeaurora.org: minor context fixups due to MSM changes]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Add buffer flag to skip page pooling on free

Currently, when we free a buffer it might actually just go back into a
heap-specific page pool rather than going back to the system. This poses
a problem because sometimes (like when we're running a shrinker in low
memory conditions) we need to force the memory associated with the
buffer to truly be relinquished to the system rather than just going
back into a page pool.

There isn't an obvious use case for this flag by Ion clients. The main
use case right now is to provide a mechanism for the deferred free code
to force stale buffers to bypass page pooling.

Change-Id: I724f89cc037083fe8576784363caa18a34e8705a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: create separate heap and client debugfs directories

It can be slightly annoying to figure out which files under the ion
debugfs directory are heap debug files and which ones are client debug
files. Create separate subdirectories under ion to hold the different
types of debug files.

Change-Id: Ic773ab619ef94b9b4d0f3794def9d37645c7c212
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: don't use "vmalloc" for system heap readable name

The human-readable name of the system heap is currently "vmalloc", which
is misleading since we don't use vmalloc at all for that heap, we use
alloc_pages. Make the human-readable name of the heap "system".

Change-Id: Iefb92ea8adc8d76b8eccc504010aa9bc2fa5c50c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: optimize zero'ing of pages

Instead of using __GFP_ZERO, vmap high-order pages for zero'ing all at
once (or in a few large chunks).

Test results show a ~25% speedup for allocations that are fulfilled from
lowmem, and ~100% speedup for allocations that are fulfilled from
highmem.

Change-Id: I57270361ff195780370e1c6f458985bdbb641efe
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: add page pooling to cached allocations from the system heap

Add a page pool for cached allocations from the system heap.

Change-Id: I6d7688026ff4ee65efa047a164652da1427e72c4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Add support for specifying heap type in the device tree

Currently, there is no way to swap out the underlying heap type for a
given heap ID without modifying msm_ion.c and recompiling the
kernel. This is inconsistent with the rest of Ion which is customized
through the device tree. This change eliminates the need to modify and
recompile the kernel for heap type changes (although the boot image
will still need to be regenerated).

Add a device tree binding to specify the heap_type for Ion heaps.

Change-Id: I0a2b48299c829a828764d74a1d8e3c8f97de706d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: make sure all clients are exposed in debugfs

Currently, if multiple Ion clients are created with the same name, only
the first one shows up in debugfs. Rectify this by adding a
monotonically-increasing serial number to the debug names of Ion
clients.

CRs-Fixed: 566001
Change-Id: I000e45055d5029c7bccd88c36b238736929da3a5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: remove IOMMU Heap

The IOMMU heap has been merged into the System heap since there was so
much overlap between the two. Remove the IOMMU heap. ION_IOMMU_HEAP_ID
will now alias to ION_SYSTEM_HEAP_ID.

Change-Id: I1cea1d97eba1488f8d5928e6a0f31a18b0fd92e5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Support probe deferal for client creation

Ion is treated like any other device in the system. Clients
may try to initialize before Ion is setup. If the Ion driver
has not yet probed, return -EPROBE_DEFER to try again later.
Since the flow is being restructured to support probe deferal,
also include some better error handling for when Ion returns
an error while probing.

Change-Id: I7ee6dde2c837fba8bba0ba4aaddee3af37b2fec1
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: fix zero'ing of cached buffers in system heap

Currently, in the system heap we zero out buffers at free-time when they
came from uncached allocations. The reason for this is that uncached
allocations come from the page pools, which require this free-time
zero'ing. However, the assumption that a buffer came from the page pools
iff it was an uncached allocation is no longer valid since [0204b38
"gpu: ion: add page pooling to cached allocations from the system heap"]
introduced page pooling for cached allocations as well. This results in
us not zero'ing out cached buffers before putting them back in the page
pools, which is a security bug (we could potentially leak sensitive
information).

Fix this by zero'ing out all buffers that are going back into the page
pools (cached and uncached).

Change-Id: I5c51eca75c8cc3b3823df1494bb0eef295c77b86
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: kmalloc_heap: flush cpu caches at allocation time

There's currently a problem with the kmalloc heap where buffers
occasionally come through unzero'd when mmap'd to userspace, which is a
security hole. We actually are zero'ing the buffer but that data isn't
immediately visible to folks with other mappings to the memory (e.g. the
mapping set up for userspace by the mmap handler). Fix this by flushing
the cpu caches at alloc time.

Change-Id: I3895c6bc163e2d065b48cae66e99825c5c0a865b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: system_heap: invalidate cache on cached pool allocations

We need to invalidate the cache when allocating pages for the cached
page pools to avoid the possibility of a stale cached value from a
different mapping being written back after we allocate our page.

Change-Id: I9b0ebfd143850cf4a4ef309a264f947a57abff77
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: use a list instead of a tree for heap debug memory map

Currently we use an rb tree to store information about the memory map
which gets passed to the heap print_debug functions. The reason for
using a tree instead of a simple list is to maintain sortedness as we
build the memory map. However, it can be necessary to store multiple
entries for the same address in the memory map since there can be
multiple clients with handles to the same buffer. This information is
interesting and useful but we currently can't store and display it since
the rb tree requires that the key used for sorting (the physical address
in this case) be unique. Fix this by replacing the rb tree with a linked
list. In order to maintain sorted output of the print_debug functions,
sort the list by physical address after fully building the list.

This also has the positive side-effect of simplifying the code and
making future print_debug methods less error-prone.

CRs-Fixed: 571918
Change-Id: I5b129fd809fb53c66042eab10d096238a34c2b20
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: fix locking issues in debug code

There are a few places in Ion where we are iterating over volatile rb
trees without proper locking. In some places the proper locking cannot
be added since it would require us to take locks in a different order
than they are taken in other places in Ion. Fix this by re-working some
of the debug code so that we can take locks in an allowed order.

One side-effect of the re-work is that the memory maps will now show
every client that has a handle to a particular region of memory, rather
than just showing the first one that we encounter. This will allow for
more accurate accounting and will give better insight as to who is
actually using the memory.

CRs-Fixed: 571918
Change-Id: Ia43e4dbc412cd480c828173f8c20b5095d87d858
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: add locking to traversal of volatile rb tree

In ion_debug_heap_show we're iterating over an rb tree (dev->clients)
that could change while we're iterating. Fix this by taking the lock
that is used to control access to this tree.

CRs-Fixed: 571918
Change-Id: I6832e1e98e2d2a69fc653451d3752d43ec3ef269
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Add pooling of CMA allocations

CMA provides excellent memory utilization but at the cost of
being much slower than permanently reserving the memory. Most
clients of this memory end up needing to do allocations on the
fast path. These clients may be aware of allocations on the slow
path but may not be able to actually make those allocations until
the fast path. Add support for pooling CMA allocations and allowing
prefetching into the heap. This memory may be stored until either
allocated/freed or drained via a shrinker.

Change-Id: I2f15bc9c72e5f79ffa6edbeb454fd41121efd8fc
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Add ioctls for prefetching/draining pools

Multiple heaps in Ion are begining to pool large amounts of memory.
This gives a boost on average but clients may still take a hit
if the pool needs to be filled up again. Add an ioctl to give a
'hint' to add memory to the pool. Equivalently, add an ioctl to
drain some of the memory out of the pool to ennsure the memory
isn't hanging around for too long.

Change-Id: I2974322caa1520fd199d9cf2658f2fafceffb65c
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Add support for heap walking

Clients may wish to implement custom functions on a particular
heap ID. Add a generic interface to walk all the heaps available
and call a function on the particular heap ID.

Change-Id: I8b3de7a2de74cd9ea1873020ed00ee40100ce923
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>