gpu: ion: Add page pooling to the iommu heap

The system heap currently supports pooling of uncached pages
to avoid the cost of zeroing/flushing the pages every time.
Add similar functionality to the iommu heap for both cached
and uncached pages. The cached pages won't see the benefit of
delayed cache syncing but they will see a benefit for zeroing
of the buffers and reduced allocation time. Some clients may
wish to always get pages from the allocator instead of going
through the pool so add a flage to support that feature.

Change-Id: Id972d12c18a940c4eb52553a9cb7658fd1d24a26
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

Conflicts:
	drivers/gpu/ion/ion_iommu_heap.c
gpu: ion: Ensure zeroed memory gets written to RAM

When zeroing buffers, we need to make sure the zeroed data actually
gets written back to RAM. Otherwise, it may be possible for a buffer
to be reallocated with data still needing to be written back to RAM.
This may lead to corruption from userspace allocators expecting zero'd
buffers and a potential leak of kernel data to user space. Considering
the mapping is short lived and we need to write back the data, use a
writecombine mapping to write the data directly to ram and avoid the
cost of flushing the cache. If the mapping is cached, invalidate the
kernel side mappings as well.

Change-Id: I26a49908bde411469f3b8f143a1581fb4e2c3b56
CRs-Fixed: 520044
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Ajay Dudani <adudani@codeaurora.org>
gpu: ion: Set the dma address for pages from the page pool

In order to properly call dma_sync_sg_for_device, the dma address
must be set on each scatterlist. When manually creating sg lists from
individual pages, the dma address must be set manually, else the cache
flushing will not succeed. Set the dma address of the sg list for
uncached page flushing to ensure the pages get completely flushed.

Change-Id: I1b89b5e2bd5ba75e7340c347eab256acec7d9554
CRs-Fixed: 496298
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: switch to using kmalloc for the secure chunk list

When securing buffers for secure content playback, a list of 1MB
chunks is generated. The number of buffers that will ever be in this
list will never be more than 1024, which means the buffer list will
always be less than a page. Rather than use the contiguous APIs, just
use the slab allocator to get the physical address list. The slab will
be physically contiguous and appropriate to pass to TZ.

Change-Id: Ib3f7293fbe0e5db872c9c441645ca860cc9bb9ff
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
msm: remove fmem support

This functionality is no longer supported.

Change-Id: I52df01acc7deebcfd84e78c333d860e3f3c2eb32
Signed-off-by: Larry Bassel <lbassel@codeaurora.org>
ion: msm: Add null-pointer check to info struct

Add null-pointer check after allocation to avoid null-pointer
deference and subsequent crash of the kernel.

Change-Id: Idba61f5047225323d83425863222808522e89f53
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
ion: Add null-pointer check to info structure

Add null-pointer check to info structure to avoid null-pointer
dereference and subsequent kernel crash.

Change-Id: I88135c1efa0b6ddc333e16511016d3618d9d5cf3
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
gpu: ion: Secure buffers at allocation time

Currently, buffers allocated as secure must be explicitly secured
with a lock call. Considering all buffers from secured heaps are now
always secured and will never be mapped in the non-secure world,
this extra lock call is redundant. Lock the buffer at allocation
time instead of requiring clients to do so explicitly. Reference
counting will ensure correctness until clients remove the lock/unlock
calls.

Change-Id: I1593b6ec651b13b22171997606c915b8f51a05fd
Signed-off-by: Arun Menon <avmenon@codeaurora.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Don't flush secure buffers

Secure buffers have very tight restrictions about mapping into
the non secure world. The buffers are always allocated as uncached
such that there should never be a reason to have a CPU side
non-secure mapping once the buffer is allocated. Some clients may
share the same code path for secure and non-secure transactions.
Unfortunately these clients either cannot or will not distiguish
between secure and non-secure buffers, meaning they will unconditionally
try to flush the buffers. This may lead to bad things. Until clients
manage to properly distinguish between secure and non-secure buffers,
add a check in the cache flushing code to not flush if the buffer is
secure.

Change-Id: Idb0d791a8b74367e0ce0b533f980caedc23044b3
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Attempt heaps in the order given

A previous change (47b9888cc80b224b3d341227224c44b3c884d28a
"gpu: ion: Switch heap rbtree to a prio list") changed the
priority of the heaps to attempt higher numbers first. The
current setup of heaps assumes that lower numbers will be tried
first. Rather than break the userspace ABI once again to match
what was done previously, change the heap priority back.

Change-Id: Ibb7664e11c0b9037f6968e0766486936b3d42e16
CRs-Fixed: 502722
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
iommu: msm: Use ion_phys_addr_t for iova argument

IOMMU mapping function is using the wrong type for its
IO Virtual Address (IOVA) causing the returned address
to contain garbage in the upper 32 bits when LPAE is enabled.

CRs-fixed: 564365
Change-Id: Ied592fe855a145b0ff32d2a6228ecfcf7fd0a53e
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
gpu: ion: Use list_for_each_safe on error path

On an error, the list of pages is walked and destroyed. Since the list
is being destroyed, use list_for_each_safe to avoid list corruption.

Change-Id: Ia397a11d2d890b3458745820c79df2adaa088ade
CRs-Fixed: 565183
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

Conflicts:
	drivers/gpu/ion/ion_system_heap.c
gpu: ion: Also shrink memory cached in the deferred free list

When the system is low on memory, we want to shrink any cached
system memory ion is holding.  Previously we were shrinking memory
in the page pools, but not in the deferred free list.  This patch
makes it possible to shrink both.  It also moves the shrinker
code into the heaps so they can correctly manage any caches they
might contain.

Change-Id: I177f587f999a5220eddbf7af94745aae736cac75
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Git-commit: 072beca55150c0c0b652bbb24267da271e0bdf19
Git-repo: https://android.googlesource.com/kernel/common
[mitchelh@codeaurora.org: minor context fixups due to MSM changes]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Add buffer flag to skip page pooling on free

Currently, when we free a buffer it might actually just go back into a
heap-specific page pool rather than going back to the system. This poses
a problem because sometimes (like when we're running a shrinker in low
memory conditions) we need to force the memory associated with the
buffer to truly be relinquished to the system rather than just going
back into a page pool.

There isn't an obvious use case for this flag by Ion clients. The main
use case right now is to provide a mechanism for the deferred free code
to force stale buffers to bypass page pooling.

Change-Id: I724f89cc037083fe8576784363caa18a34e8705a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: create separate heap and client debugfs directories

It can be slightly annoying to figure out which files under the ion
debugfs directory are heap debug files and which ones are client debug
files. Create separate subdirectories under ion to hold the different
types of debug files.

Change-Id: Ic773ab619ef94b9b4d0f3794def9d37645c7c212
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: don't use "vmalloc" for system heap readable name

The human-readable name of the system heap is currently "vmalloc", which
is misleading since we don't use vmalloc at all for that heap, we use
alloc_pages. Make the human-readable name of the heap "system".

Change-Id: Iefb92ea8adc8d76b8eccc504010aa9bc2fa5c50c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: optimize zero'ing of pages

Instead of using __GFP_ZERO, vmap high-order pages for zero'ing all at
once (or in a few large chunks).

Test results show a ~25% speedup for allocations that are fulfilled from
lowmem, and ~100% speedup for allocations that are fulfilled from
highmem.

Change-Id: I57270361ff195780370e1c6f458985bdbb641efe
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: add page pooling to cached allocations from the system heap

Add a page pool for cached allocations from the system heap.

Change-Id: I6d7688026ff4ee65efa047a164652da1427e72c4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: make sure all clients are exposed in debugfs

Currently, if multiple Ion clients are created with the same name, only
the first one shows up in debugfs. Rectify this by adding a
monotonically-increasing serial number to the debug names of Ion
clients.

CRs-Fixed: 566001
Change-Id: I000e45055d5029c7bccd88c36b238736929da3a5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Support probe deferal for client creation

Ion is treated like any other device in the system. Clients
may try to initialize before Ion is setup. If the Ion driver
has not yet probed, return -EPROBE_DEFER to try again later.
Since the flow is being restructured to support probe deferal,
also include some better error handling for when Ion returns
an error while probing.

Change-Id: I7ee6dde2c837fba8bba0ba4aaddee3af37b2fec1
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>