ion: remove use of qcom,msm-ion-reserve

The device tree string "qcom,msm-ion-reserve" is no
longer needed since we don't support carveout heaps any more.

Change-Id: Icb24d86fff9b478504d17518fda4c2e7b04833b8
Signed-off-by: Larry Bassel <lbassel@codeaurora.org>
ion: chunk_heap: fix leak in allocated counter

buffer->size is controlled by the outer ion layer, don't modify it
inside the heap.  Instead, compute the rounded up allocated size
on demand.

Change-Id: I288ffc1221ce96cfe2591468502ac3279065bde4
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: f526a55b400d166dafc6062e5818a2410555ef39
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ion: Squashed commit of upstream updates

ion: Add missing unlock on error path

A mutex_unlock was missing for the alloc_lock on an error
path. Add it.

Change-Id: If88b72fbc2fd18504347b784d5daa2bc9170c55f
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

ion: fix crash when alloc len is -1

If userspace passes a length between -4095 and -1 to allocate it
will pass the len != 0 check, but when len is page aligned it will
be 0.  Check len after page aligning.

Drop the warning as well, userspace shouldn't be able to trigger
a warning in the kernel.

Change-Id: I96c7142637638991f3a9af9be7cfbb50f79f3803
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 49bdc418a3c1129d49fe92d89cf77111584fb51f
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: fix dma APIs

__dma_page_cpu_to_dev is a private ARM api that is not available
on 3.10 and was never available on other architectures.  We can
get the same behavior by calling dma_sync_sg_for_device with a
scatterlist containing a single page.  It's still not quite a
kosher use of the dma apis, we still conflate physical addresses
with bus addresses, but it should at least compile on all
platforms, and work on any platform that doesn't have a physical
to bus address translation.

Change-Id: I8451c2dae4bf85841015c016640684ac28430a5a
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 51e3580bf9da57da9c860330293b73b0dbb296e0
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: conflicts due to different cache APIs being
 used. Earlier we changed __dma_page_cpu_to_dev to
 arm_dma_ops.sync_single_for_device so there was a conflict at that call
 site.]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

gpu: ion: Fix performance issue in faulting code

Previously the code to fault ion buffers in one page at a time had a
performance problem caused by the requirement to traverse the sg list
looking for the right page to load in (a result of the fact that the items in
the list may not be of uniform size).  To fix the problem, for buffers
that will be faulted in, also keep a flat array of all the pages in the buffer
to use from the fault handler.  To recover some of the additional memory
footprint this creates per buffer, dirty bits used to indicate which
pages have been faulted in to the cpu are now stored in the low bit of each
page struct pointer in the page array.

Change-Id: I891b077dc0c88ed6d416b256626d8778fd67be84
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Git-commit: bbf5538ee0e4c30ba9b4556f9699ed5f31df67da
Git-repo: https://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: conflicts due to msm modifications. Changed a
 private dma API (__dma_page_cpu_to_dev) to public (ARM-specific)
 wrapper (arm_dma_ops.sync_single_for_device).]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: convert sg_dma_len(sg) to sg->length

ion is always dealing with the allocation and not the mapping,
so it should always be using sg->length and not sg->dma_length.

Change-Id: Id9b07f1196b2bafe04636fa1aa46dfc84d003cf0
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: c1d3cea1d0b985a78a87675a07c2dab865a416f1
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: check invalid values in ion_system_heap

ion_system_heap can only satisfy page alignment, and
ion_system_contig_heap can only satisify alignment to the
allocation size.  Neither can support faulting user mappings
because they use slab pages.

Change-Id: I895c2d4184c672f647f83a17aeb862985dc92f2c
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: e8f7f42cdcbe3ae4c4bf719847369ddac44219a9
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: some context due to msm fixes]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: add test device for unit tests to interact with dma_bufs

Add a /dev/ion-test device that will be created if CONFIG_ION_TEST
is set.  The device accepts a dma_buf fd and allows reading and
writing to the backing memory using DMA-like apis or kernel mapping
apis.  Can be used to test the dma_buf mapping ops, including
the ion implementations, from userspace.

Change-Id: I30703ba69cd75bdfe7767ac642e5f0cacd8d0478
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 5c31b87cef4e03eea5b417a6e3f7c9b781aacf96
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

gpu: ion: ion_chunk_heap: Zero chunk heap memory at creation time

Allocations from the ion heap need to be zeroed to protect userspace
from seeing memory belonging to other processes.  First allocations
from this heap were not zero'd allowing users to see memory from other
processes on a warm reset.

Change-Id: I524a7b79cb76c390c870fcf8b30d213185fc85a0
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Git-commit: 34e159dd120ce57b9dbfdaecb5904c2be3000a26
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: don't use phys_to_page or __phys_to_pfn

phys_to_page and __phys_to_pfn don't exist on all platforms.
Use a combination of pfn_to_page, PFN_DOWN, page_to_pfn, and
virt_to_page to get the same results.

Change-Id: I53cef26059800bc8b7fb85ae458741574c97c257
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: f26a6f2aa0beea7eeb0a2f69d44a5975ef8c887e
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: we weren't using the APIs in question in the
 same locations. Also applied the change to some other locations]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: add free list size to heap debug files

Change-Id: I3c6309afdbd661a2f870fd1ba3fea9543e229882
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: a3041cac9fc117df010e66ce40091fa2c4f646f8
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: fix printk warnings

Use %z for size_t and %pa for dma_addr_t to avoid warnings in printks.

Change-Id: I2c72874acd0b69cb35fca691928783817deb9394
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 21e08b49ac311e3d1e9c7168de1bd42868176398
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: we already had most of these right, so this
 turned out to be pretty minimal compared to the original patch.]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: Fix two small issues in system_heap allocation

In testing ion system heap allocations, I ran across two issues:

1) Not k*z*allocing the sg table. This can cause trouble if
we end up trying call sg_alloc_table() with too many entries,
then sg_alloc_table() internally fails and tries to free what it
thinks is internal table structure, which causes bad pointer
traversals.

2) The second list_for_each_entry probably should be _safe,
since I was seeing  strange lock warnings and oopses on occasion.
This seems to resolve it, but could use some extra checking.

Change-Id: I59d4c90104a8cf23dc4ae814d0b17348f1b68ac0
Signed-off-by: John Stultz <john.stultz@linaro.org>
Git-commit: 818017a0caa6b7b3bb6e96697a2c35806e08f6a4
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: we already had list_for_each_entry_safe from
the commit (fb98460facd6: "gpu: ion: Use list_for_each_safe on error
path")]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: free low memory from page pools first

When the shrinkers are called with GFP_HIGH free low memory first,
it is more important to have free than high memory.

Change-Id: I7ad8a9c133830f04d429c3d87b781b3e862ccedb
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: b928ad649cd32b4ca5ba7429bfe897433d37dfea
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: check return value from remap_pfn_range

Check the return value of remap_pfn_range and return an error if
it fails.

Change-Id: I206cf95a24607ebe1c80274e3ed15cc7c076d007
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 3b8da00c6c5e42af7b532dbf51b2d5007095d844
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: use vm_insert_pfn for faulted pages

Most ion userspace mappings are created with remap_pfn_range.  Use
vm_insert_pfn instead of vm_insert_page to make faulted cached
mappings look more like uncached mappings.

Change-Id: I9ec5cad3fef54f3b80be8b306d7ff2f1fe3f0e66
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: ed6cfafce575cc6baa099212c69b56c0132f598d
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: remove ion_heap_alloc_pages

Now that ion_vm_fault doesn't need a struct page with a nonzero
refcount, there is no need allocate heap memory for cached pages using
split_page.  Remove the ion_heap_alloc_pages and ion_heap_free_pages
helpers in favor of direct calls to alloc_pages and __free_pages,
and remove the special handling in the system heap.

Change-Id: I5966a798f48df2d56642e662a69c1495944f6509
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: d671290eae60f4a77ab2aef79cfaed008ee0716d
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: conflicts since we have page pooling for both
 cached and uncached]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: allow cached mappings of chunk and system heap buffers

Now that ion_vm_fault uses vm_insert_pfn instead of vm_insert_page
cached buffers can be supported in any heap.  Remove the checks
in the chunk and system heaps.

Change-Id: I371a44c400ed8a342c3b0eed90d0fb7060537697
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 1f67f7520303192e46e1621639097007ab076b90
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: context differences due to prior msm related
 to flushing the cache at allocation time. The prior msm fixes are now
 no longer relevant so an additional structure was removed.]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: fix sparse warnings

Fix sparse warnings in ion.

Change-Id: Icbadf2ca53bea20914f608f619568629c178eae3
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 6db8a1c0c9817d6ebd61c47110554aa34729e9a7
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: conflicts since we have page pooling for both
 cached and uncached as well as some gfp flag differences]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: add alignment check to chunk heap

Change-Id: I4be12b9545a81f9b46339a905f00e1e64896b3ed
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 1b1cbed2f4cd92b9ccfd28b985983e2d217ecd2e
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: Cleanup whitespace issues and other checkpatch problems

Just some simple cleanups to address whitespace issues and
other issues found w/ checkpatch.

Change-Id: I181444505627894b8f3bbf59192703b0f65736ee
Signed-off-by: John Stultz <john.stultz@linaro.org>
Git-commit: 0f9ae3f7fc853efb3f36e44b1c24ca15490e54f0
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: remove support for securing entire heaps

Securing an entire heap is no longer required or implemented by any
heaps. Remove the (now unused) upper-level hooks for securing heaps.

Change-Id: I624a050c30c89522e8794768681ea39ff29bc215
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: remove obsolete stub

ion_mark_dangling_buffers_locked is no longer used anywhere. The main
implementation of the function was removed but the stub wasn't. Remove
the stub.

Change-Id: I8488c5dfdd93827c31ae41be849dfe0999a965d1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: remove calls to outer cache routines

There aren't any platforms supported on 3.10 which
have outer caches.

Change-Id: Ia624de9835dd372dcd403fa321e21fc8272cb4bd
Signed-off-by: Larry Bassel <lbassel@codeaurora.org>

gpu: ion: Add trace events for secure cma

Secure CMA allocates and stores memory differently than regular
heaps. This means the regular trace events for tracking allocation
time may not tell the whole story. Add secure CMA specific trace
events for events such as prefetching and shrinking to better collect
data.

Change-Id: I58ce8eb9e94ea380bece6bd75c7ba899a3ebc781
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

ion: index client->handles rbtree by buffer

The only remaining users of the client->handles rbtree are
iterating through it like a list.  Keep the rbtree, but change
its index to be the buffer address instead of the handle address,
which makes ion_handle_lookup a fast rbtree search.

Change-Id: Ie7d974b3a5d9831c0d664de85ddae8db3c3abdf9
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 002119eeb221cf1efd7afda836da768ff1cd0ad9
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: do not use sg_dma_len() macro

Use of sg_dma_len() is not portable to 64 bit and is not used any more
by ion or iommu. Replace the usages by direct access of the length
structure member.

Change-Id: I6c18b2a77e288be312448c36f8ed3ad6db90ac0f
Signed-off-by: Larry Bassel <lbassel@codeaurora.org>

ion: convert map_kernel to return ERR_PTR

ion is going to stop accepting NULL as an error value, use ERR_PTR.

Change-Id: I030e8b72138904e38a4a5d225beaaa98427651fb
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: d37aa151f874563a35d49a0ff0a129ef332c5e9e
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: conflicts due to msm changes]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

gpu: ion: delete ion_system_mapper.c

The mapper abstraction layer was removed before the initial ion
commit, but a stray ion_system_mapper.c file was left in.  Delete
it.

Change-Id: I0001d1ea1b866ebd7ec1306e94472c67a094855f
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: 3922298f52d193d5e014bf4d28e423cf7bd0b50e
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: change ion_user_handle_t definition to int

Turn ion_user_handle_t to int. This change reflects the underlying type
returned by the ion driver.

Change-Id: I40390dae8138327769510525bf62e55877a4b37d
Signed-off-by: Rom Lemarchand <romlem@google.com>
Git-commit: dcd2da8015c7ac2da6f57fc0d8c15b80e44c4aed
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: made changes to file in different location
 since we've already done the uapi split. Also, one function name was
 different due to some necessary patch re-ordering.]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

Conflicts:
	drivers/gpu/ion/ion_cma_secure_heap.c
staging: ion: msm: Only parse available dt nodes

ION is currently ignoring the status property of device tree
heap nodes. Make ION aware of the status of the ION heap nodes making
it possible to disable certain heaps.

Change-Id: I96e6af97df6bbd1c24ace9aa476baae28409ea96
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
ion: secure_buffer: fix iteration of sg_table

We're currently re-using the `i' loop counter to iterate over the chunk
list within our iteration of the secure sg_table. This causes the
iteration of the sg_table to get all messed up and results in crashes
when we walk off the end of the list. Fix it.

Change-Id: I09c84015f8c3cbcb2cea60886241457bb829003c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ion: cma_secure_heap: validate buffer length

The secure CMA heap only supports buffers lengths that are multiples of
1MB. Currently, we simply crash if a bad length is requested. Rather
than crashing, log an error and return a failure.

Change-Id: I0387ee9f2900e44c817d3a7497e9b0528dd8c33b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ion: fix remaining instances of sg_dma_len

Recently, [ec6c8a8f443: "ion: convert sg_dma_len(sg) to sg->length"]
moved Ion away from using sg_dma_len. However, a few instances were left
over from MSM-specific code. Fix these instances so that we're using
sg->length everywhere.

Change-Id: I76b07f5589ba3867e1584b129a0ccd9f3ef00ad7
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
staging: ion: Add source of allocated page

It might be useful for higher levels in the call chain to know whether a
page came from the page pool or from the buddy allocator. If a page came
from the page pool we know that the page has been zeroed already and we
do not have to do that again.

Change-Id: I849591a56e5af8a5bc4340bff70f23e3702e62fb
CRs-fixed: 660049
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
staging: ion: Add GFP_NO_KSWAPD flag to high order allocations

When allocating from system heap we try higher order allocations first
before falling back on lower allocations. However, we do not want to
incur much overhead by doing the fallback. So don't try to run kswapd
during these allocation attempts.

CRs-fixed: 660049
Change-Id: I80895c456c80342cc3b2100470dfd9550c81bd29
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
staging: ion: Optimize zeroing when allocating from system heap

Currently when allocating from system heap each page is individually
mapped and zero'ed in the page pool code. Under high memory pressure when
we allocate a lot of small pages this becomes a bottleneck. Instead,
do the zeroing in one fell swoop at the end of allocating all the pages
but only for the pages that are not already zero'ed and sync'ed (pages
coming from the page pool).

Test results
------------
Time recorded for allocating 20 x 19MB ION buffers from System heap with
forced 4K page allocations (all pages allocated was order-0) right after
boot to Android homescreen using performance kernel configuration.

Before change (non-cached allocations):
Run 1: 768ms
Run 2: 752ms
Run 3: 759ms

Before change (cached allocations):
Run 1: 726ms
Run 2: 769ms
Run 3: 645ms

After change (cached allocations):
Run 1: 262ms
Run 2: 249ms
Run 3: 310ms

After change (non-cached allocations):
Run 1: 275ms
Run 2: 234ms
Run 3: 259ms

Time recorded for allocating 20 x 19MB ION buffers from System heap (with no
forced page size) right after boot to Android homescreen using
performance kernel configuration. Run 1 is taken right after boot while
run 2 and 3 are subsequent runs without reboot between.

Before change (non-cached allocations):
Run 1: 181ms
Run 2: 1ms
Run 3: 1ms

Before change (cached allocations):
Run 1: 220ms
Run 2: 1ms
Run 3: 2ms

After change (non-cached allocations):
Run 1: 166ms
Run 2: 1ms
Run 3: 1ms

After change (cached allocations):
Run 1: 170ms
Run 2: 1ms
Run 3: 3ms

CRs-fixed: 660049
Change-Id: I8dcfdb4adb8d5f4480768f951523fa5c64db87b1
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
(cherry picked from commit 14fe9a42e5b6d27a5dfdf4559e60956010287250)
ion: Skip zeroing on secure buffers

Secure buffers are never passed to userspace and are controled
by the secure world so there is no real need to zero. Pass the
dma attribute to skip zeroing.

Change-Id: Iad870d0d7732d3dea09443418b9294cb9e05b5e0
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
ion: shrinker free pages even for unmovable page requests

In the secure CMA heap shrinker free pooled free CMA pages
even for unmovable page requests.

Most memory reclaim is performed by kswapd which uses a gfp
mask of GFP_KERNEL. Therefore even when the system is looking
for movable pages memory reclaim will generally be done by
kswapd.

By allowing the secure CMA heap shrinker to return free pooled
CMA pages for unmovable page requests there will generally be
a headroom improvement whenever a memory intensive userpace use
case is carried out after CMA memory is freed by its client.

Change-Id: Id6edcef4952aa86499220281470189ff93f8026a
Signed-off-by: Liam Mark <lmark@codeaurora.org>
ion: msm: move custom zero'ing functions out of core Ion

MSM targets currently use some optimized zero'ing functions. However,
there are always merge conflicts when bringing in patches from upstream
so these custom functions should be moved into the MSM space and the
original zero'ing functions from upstream should be restored.

All users of the custom zero'ing functions should continue to use the
custom functions. The goal, over time, is to unify the approaches but
for now just leave them separated.

Change-Id: I4818c09569976a82b8fecc5cd81129ce65ee7db3
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ion: optimize ion_heap_buffer_zero

ion_heap_buffer_zero can spend a long time in unmap_kernel_range
if it has to broadcast a tlb flush to every cpu for every page.
Modify it to batch pages into a larger region to clear using a
single mapping.  This may cause the mapping size to change if
the buffer size is not a multiple of the mapping size, so
switch to allocating the address space for each chunk.  This
allows us to use vm_map_ram to handle the allocation and mapping
together.

The number of pages to zero using a single mapping is set to 32
to hit the fastpath in vm_map_ram.

Change-Id: I1accfe67b285cbc9e95e387bea4246864197827d
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: fc04e484c319b8df647886d4cf17f7c450453d1d
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ion: Change secure structure to be 64-bit compatible

When passing structures between the secure and non-secure world,
pointer sizes may be different. Switch to a type of a known size
to avoid issues.

Change-Id: I11a57b3a52d0d1ef9fd3dbaf22f4164734fe71ac
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

msm: secure_buffer: use explicitly-sized types for TZ communication

Some types currently in use for passing data to TZ (like `unsigned
long') have different sizes under different build configurations (32 bit
vs 64 bit, etc). TZ dictates the sizes of the structures that are passed
back-and-forth and they cannot change. Use explicitly-sized types for
any such structures.

Change-Id: Ib6a671f137a875fbabcb045f595cefcbe3616beb
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: clean up ioctls

Convert the ion ioctls to use _IOW instead of _IOWR where
appropriate, and factor out the copy_from_user and copy_to_user
based on the _IOC_DIR bits.  For the existing incorrect ioctls,
add a function to wrap _IOC_DIR to return the corrected value.

Change-Id: I3cc34c84b9c52305bdbec27a9224447b102fadcd
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: b2ee33eb98bd8240787f6b3a6acffdbab40653dc
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: merge conflict since we never renamed
 heap_mask -> heap_id_mask]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: Always flush using kernel virtual address

Cache operations via userspace address always has a risk of a
page fault. On arm64, userspace addresses are marked as write
protect and fixed up on first accesses. If a userspace address
that has not been written to is invalidated, we will take a
fault, fixup and proceed on. If the mmap semaphore is locked at
the time however we cannot fixup the fault and will abort instead.
Rather than hope the mmap semaphore is not write protected, always
invalidate via the kernel virtual addresses instead.

Change-Id: I71eb0d214cce54ebc82a213ae653f891939f273a
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

staging: ion: Avoid using rt_mutexes directly

RT_MUTEXES can be configured out of the kernel, causing compile
problems with ION.

To quote Colin:
"rt_mutexes were added with the deferred freeing feature.  Heaps need
to return zeroed memory to userspace, but zeroing the memory on every
allocation was causing performance issues.  We added a SCHED_IDLE
thread to zero memory in the background after freeing, but locking the
heap from the SCHED_IDLE thread might block a high priority allocation
thread for a long time.

The lock is only used to protect the heap's free_list and
free_list_size members, and is not held for any long or sleeping
operations.  Converting to a spinlock should prevent priority
inversion without using the rt_mutex.  I'd also rename it to free_lock
to so it doesn't get used as a general heap lock."

Thus this patch converts the rt_mutex usage to a spinlock and
renames the lock free_lock to be more clear as to its use.

I also had to change a bit of logic in ion_heap_freelist_drain()
to safely avoid list corruption.

Change-Id: I47cf3551902a01d4441eeb6ade8eeb43ab9cd90b
Acked-by: Colin Cross <ccross@android.com>
Cc: Android Kernel Team <kernel-team@android.com>
Reported-by: Jim Davis <jim.epost@gmail.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Git-commit: c6448537fe9e8b684b214e89c6c68c3b209f549e
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: move shrinker out of heaps

Every heap that uses deferred frees is going to need a shrinker
to shrink the freelist under memory pressure.  Rather than
requiring each heap to implement a shrinker, automatically
register a shrinker if the deferred free flag is set.
The system heap also needs to shrink its page pools, so add
a shrink function to the heap ops that will be called after
shrinking the freelists.

Change-Id: Icda722d683426fadb8ddd1c8e9499264ab682c57
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: f3ae9a8286d63f13715d2082965d6b8849af05a5
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: context differences due to our addition of
 cached page pools and some additional heap callbacks]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: move "freed from shrinker" flag to buffer private_flags

ION_FLAG_FREED_FROM_SHRINKER should not be used by Ion clients. It's an
internal flag. Add a field to ion_buffer for private flags and move
ION_FLAG_FREED_FROM_SHRINKER there. Also rename
ION_FLAG_FREED_FROM_SHRINKER to ION_PRIV_FLAG_SHRINKER_FREE.

Change-Id: I52926ba055fee6256314cd73951e7a6fa3faca9d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: fix overflow and list bugs in system heap

Fix a few bugs in ion_system_heap:

Initialize the list node in the info block.

Don't store size_remaining in a signed long, allocating >2GB
could overflow, resulting in a call to sg_alloc_table with
nents=0 which panics.  alloc_largest_available will never
return a block larger than size_remanining, so it can never
go negative.

Limit a single allocation to half of all memory.  Prevents a
large allocation from taking down the whole system.

Change-Id: I7fcbd7e1d5b4d482d7612d80b6c9e8e24466f1d8
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: c69de844c80884167bb42243b36498dde81eebf6
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: we already had the signed long fix]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: msm: plumb ION_HEAP_TYPE_DMA to core Ion

ION_HEAP_TYPE_DMA has made its way into core Ion since it was originally
added to the MSM Ion code base. Move some of the hooks and function
declarations out of MSM-specific code and into core Ion. There are still
some differences in the heap implementations themselves that will need
to be rectified later.

Change-Id: I44a4aa3a48645072cee0f2d0718ce3e795b27c4b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

msm: secure_buffer: warn if we get a non-1MB-aligned chunk

We only support securing memory in 1MB chunks. If we get an sg table
with any entries that are of any other size, WARN and bail.

Change-Id: Icd96984f03dafd246da51e879cd91c77ae986724
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

staging: android: Reduce allocation time from system heap

If we are under heavy memory pressure the shrinker might be running on
the system heap. This causes contention with the page pool lock when we
try to allocate from system heap. To ensure that allocations aren't
slowed down by having to wait for the page pool mutex we will just try
to get the mutex. If we are unsuccessful we just bypass the pool
altogether.

Change-Id: I3e92e3081355a8528438e41e74b716a2b49a3891
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>

ion: msm: clean up msm custom ioctls

Similar to what was done in core Ion in [$(git rev-parse HEAD)
7ba5b038cc8 "ion: clean up ioctls"], convert the ion ioctls to use _IOW
instead of _IOWR where appropriate, and factor out the copy_from_user.
For the existing incorrect ioctls, add a function to wrap _IOC_DIR to
return the corrected value.

Change-Id: If8ad728fc9d3905d30d722299c4447a00f01992e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: simplify client name uniquification

The current method of finding a unique serial number for Ion client
names is not straightforward and prone to future breakage. Rather than
parsing all client names and extracting the max out of that just store
the client `id' in a new field in `struct ion_client'.

Change-Id: Ic25008e690ae4f9e3b4c19d1e438b8aaadb36a32
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: retry during race with shrinking/allocating

There exists a race between shrinking/allocating

Thread 1:                           Thread:2
mutex_lock(&sheap->alloc_lock)
ion_secure_cma_add_to_pool()
mutex_lock(&sheap->chunk_lock)
chunk->cnt = 0
mutex_unlock(&sheap->chunk_lock)
                                    ion_secure_cma_shrinker
                                    mutex_lock(&sheap->chunk_lock)
                                    __ion_secure_cma_shrink_pool
                                    chunk->cnt == 0 for the
                                       just allocated chunk
                                    ion_secure_cma-free_chunk
ion_secure_cma_alloc_from_pool
no memory to allocate!

If this race happens, just retry adding memory to the pool instead
of BUGing out the system.

Change-Id: Ic95782db6d88a33f01dcc5248fc957ca0f378078
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
ion: Use dma_free_attrs for freeing secure memory

Secure CMA memory is allocated using DMA_ATTR_NO_KERNEL_MAPPING.
Free the memory with the same attribute.

Change-Id: I67289af68763c5b6d4b0f3d828174ad28b711642
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

staging: ion: remove struct ion_page_pool_item

The page pool uses an internal data structure, ion_page_pool_item, for
wrapping pooled pages and constructing a list. As the struct page
already provides ways for doing exactly the same thing, we do not need
to reinvent the wheel. This commit removes the data structure and slab
allocations for it.

Change-Id: I3653a71337abc26c0cd6791be7cb175055281036
Signed-off-by: Heesub Shin <heesub.shin@samsung.com>
Reviewed-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Tested-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Git-commit: 38c003b113ceab37f80d020999d3ec2cbf784da3
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

staging: ion: shrink highmem pages on kswapd

ION system heap keeps pages in its pool for better performance. When the
system is under memory pressure, slab shrinker calls the callback
registered and then the pages pooled get freed.

When the shrinker is called, it checks gfp_mask and determines whether
the pages from highmem need to be freed or the pages from lowmem.
Usually, slab shrinker is invoked on kswapd context which gfp_mask is
always GFP_KERNEL, so only lowmem pages are released on kswapd context.
This means that highmem pages in the pool are never reclaimed until
direct reclaim occurs. This can be problematic when the page pool holds
excessive amounts of highmem.

For now, the shrinker callback cannot know exactly which zone should be
targeted for reclamation, as enough information are not passed to. Thus,
it makes sense to shrink both lowmem and highmem zone on kswapd context.

Change-Id: Ibf21e36e6d370967fa9166d6e5d66aec98851478
Reported-by: Wonseo Choi <wonseo.choi@samsung.com>
Signed-off-by: Heesub Shin <heesub.shin@samsung.com>
Reviewed-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Tested-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Git-commit: 0cd2dc4db3ba23ff8ab38b8ae81b126626f795d5
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
[shashim@codeaurora.org: resolve trivial merge conflict]
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
ion: Drop cache operation after zeroing

At one time, the zeroing algorithm used a mixture of
cached and uncached mappings which necessitated an extra
cache operation afterwards. The zeroing algorithm has been
simplified so the cache operation is no longer required if
the mapping used for zeroing matches the cache status of
the buffer (cached or not). Remove the cache operation and
use appropriate page protections instead.

Change-Id: I7d3596ed503811818782e77a0f3f7f63b1456d70
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
list: introduce list_first_entry_or_null

non-rcu variant of list_first_or_null_rcu

Change-Id: Ibc8b476ef382e93c659d14a27364c896f81bd689
Signed-off-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
cma: Allow option to use strict memblock_reserved memory

Despite all the performance optimizations, some clients are
still unable to use CMA because of the allocation latency.
Rather than make those clients use a separate set of APIs,
extend the CMA code to allow clients to keep the memory out of
the buddy allocator. Since the pages never actually go to the
buddy allocator, allocation and freeing is only based on the bitmap
allocator to find the appropriate region.

Change-Id: Ia31bb1212fd7b19280361128453c8d25369ce592
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
cma: Remove potential deadlock situation

A report was given of a deadlock situation on the cma_mutex:

mutex_lock(cma_mutex)   (*2)
dma_release_from_contiguous
ion_secure_cma_free_chunk
ion_secure_cma_shrinker
shrink_slab
try_to_free_pages
migrate_pages
alloc_contig_range
mutex_lock(cma_mutex)  (*1)
dma_alloc_from_contiguous
ion_alloc

We may need to free CMA allocations while a current CMA allocation is in
progress if CMA is freed from a shrinker. cma_mutex currently protects
two things: the bitmap indicating which pages are allocated/free and
serialization of isolation/migration calls on allocation. There is
no need to take the mutex on free calls though as the pages are
freed back into the system via the regular __free_page call. Move
the free_contig_range call outside the cma_mutex to break the
chain dependency. We can safely free the pages back into the system
before changing the bitmap without any risk of races.

Change-Id: I4989eb2891e502b08db8117a51bd86652e902778
CRs-Fixed: 619644
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>