gpu: ion: Use list_for_each_safe on error path

On an error, the list of pages is walked and destroyed. Since the list
is being destroyed, use list_for_each_safe to avoid list corruption.

Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: Also shrink memory cached in the deferred free list

When the system is low on memory, we want to shrink any cached
system memory ion is holding.  Previously we were shrinking memory
in the page pools, but not in the deferred free list.  This patch
makes it possible to shrink both.  It also moves the shrinker
code into the heaps so they can correctly manage any caches they
might contain.

Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: Add buffer flag to skip page pooling on free

Currently, when we free a buffer it might actually just go back into a
heap-specific page pool rather than going back to the system. This poses
a problem because sometimes (like when we're running a shrinker in low
memory conditions) we need to force the memory associated with the
buffer to truly be relinquished to the system rather than just going
back into a page pool.

There isn't an obvious use case for this flag by Ion clients. The main
use case right now is to provide a mechanism for the deferred free code
to force stale buffers to bypass page pooling.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: create separate heap and client debugfs directories

It can be slightly annoying to figure out which files under the ion
debugfs directory are heap debug files and which ones are client debug
files. Create separate subdirectories under ion to hold the different
types of debug files.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: don't use "vmalloc" for system heap readable name

The human-readable name of the system heap is currently "vmalloc", which
is misleading since we don't use vmalloc at all for that heap, we use
alloc_pages. Make the human-readable name of the heap "system".

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: optimize zero'ing of pages

Instead of using __GFP_ZERO, vmap high-order pages for zero'ing all at
once (or in a few large chunks).

Test results show a ~25% speedup for allocations that are fulfilled from
lowmem, and ~100% speedup for allocations that are fulfilled from
highmem.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: add page pooling to cached allocations from the system heap

Add a page pool for cached allocations from the system heap.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: make sure all clients are exposed in debugfs

Currently, if multiple Ion clients are created with the same name, only
the first one shows up in debugfs. Rectify this by adding a
monotonically-increasing serial number to the debug names of Ion
clients.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: Support probe deferal for client creation

Ion is treated like any other device in the system. Clients
may try to initialize before Ion is setup. If the Ion driver
has not yet probed, return -EPROBE_DEFER to try again later.
Since the flow is being restructured to support probe deferal,
also include some better error handling for when Ion returns
an error while probing.

Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: fix zero'ing of cached buffers in system heap

Currently, in the system heap we zero out buffers at free-time when they
came from uncached allocations. The reason for this is that uncached
allocations come from the page pools, which require this free-time
zero'ing. However, the assumption that a buffer came from the page pools
iff it was an uncached allocation is no longer valid since [0204b38
"gpu: ion: add page pooling to cached allocations from the system heap"]
introduced page pooling for cached allocations as well. This results in
us not zero'ing out cached buffers before putting them back in the page
pools, which is a security bug (we could potentially leak sensitive
information).

Fix this by zero'ing out all buffers that are going back into the page
pools (cached and uncached).

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: kmalloc_heap: flush cpu caches at allocation time

There's currently a problem with the kmalloc heap where buffers
occasionally come through unzero'd when mmap'd to userspace, which is a
security hole. We actually are zero'ing the buffer but that data isn't
immediately visible to folks with other mappings to the memory (e.g. the
mapping set up for userspace by the mmap handler). Fix this by flushing
the cpu caches at alloc time.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: system_heap: invalidate cache on cached pool allocations

We need to invalidate the cache when allocating pages for the cached
page pools to avoid the possibility of a stale cached value from a
different mapping being written back after we allocate our page.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: use a list instead of a tree for heap debug memory map

Currently we use an rb tree to store information about the memory map
which gets passed to the heap print_debug functions. The reason for
using a tree instead of a simple list is to maintain sortedness as we
build the memory map. However, it can be necessary to store multiple
entries for the same address in the memory map since there can be
multiple clients with handles to the same buffer. This information is
interesting and useful but we currently can't store and display it since
the rb tree requires that the key used for sorting (the physical address
in this case) be unique. Fix this by replacing the rb tree with a linked
list. In order to maintain sorted output of the print_debug functions,
sort the list by physical address after fully building the list.

This also has the positive side-effect of simplifying the code and
making future print_debug methods less error-prone.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: fix locking issues in debug code

There are a few places in Ion where we are iterating over volatile rb
trees without proper locking. In some places the proper locking cannot
be added since it would require us to take locks in a different order
than they are taken in other places in Ion. Fix this by re-working some
of the debug code so that we can take locks in an allowed order.

One side-effect of the re-work is that the memory maps will now show
every client that has a handle to a particular region of memory, rather
than just showing the first one that we encounter. This will allow for
more accurate accounting and will give better insight as to who is
actually using the memory.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: add locking to traversal of volatile rb tree

In ion_debug_heap_show we're iterating over an rb tree (dev->clients)
that could change while we're iterating. Fix this by taking the lock
that is used to control access to this tree.

Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: Explictly mark DMA allocations as GFP_KERNEL

For legacy reasons, the GFP allocation for dma allocations is currently
marked as 0, which corresponds to atomic allocations. These allocations
should not be marked as atomic since the atomic pool is very small.
Switch to properly using GFP_KERNEL to always allocate from the
correct pool.

Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
gpu: ion: Flush the chunk list before passing it to trustzone

Previously, the function scm_call used to take care of flushing
the caches to ensure all data was written back to memory before
going into the secure world. The behavior of scm_call is changing
such that clients are expected to flush caches themselves. Since
the address that the chunk list is written to is cached, we need
to ensure that the data is actually written back. Add the appropriate
cache operations on the chunk list.

Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>