ion: shrinker free pages even for unmovable page requests

In the secure CMA heap shrinker free pooled free CMA pages
even for unmovable page requests.

Most memory reclaim is performed by kswapd which uses a gfp
mask of GFP_KERNEL. Therefore even when the system is looking
for movable pages memory reclaim will generally be done by
kswapd.

By allowing the secure CMA heap shrinker to return free pooled
CMA pages for unmovable page requests there will generally be
a headroom improvement whenever a memory intensive userpace use
case is carried out after CMA memory is freed by its client.

Change-Id: Id6edcef4952aa86499220281470189ff93f8026a
Signed-off-by: Liam Mark <lmark@codeaurora.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ion: msm: move custom zero'ing functions out of core Ion

MSM targets currently use some optimized zero'ing functions. However,
there are always merge conflicts when bringing in patches from upstream
so these custom functions should be moved into the MSM space and the
original zero'ing functions from upstream should be restored.

All users of the custom zero'ing functions should continue to use the
custom functions. The goal, over time, is to unify the approaches but
for now just leave them separated.

Change-Id: I4818c09569976a82b8fecc5cd81129ce65ee7db3
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ion: optimize ion_heap_buffer_zero

ion_heap_buffer_zero can spend a long time in unmap_kernel_range
if it has to broadcast a tlb flush to every cpu for every page.
Modify it to batch pages into a larger region to clear using a
single mapping.  This may cause the mapping size to change if
the buffer size is not a multiple of the mapping size, so
switch to allocating the address space for each chunk.  This
allows us to use vm_map_ram to handle the allocation and mapping
together.

The number of pages to zero using a single mapping is set to 32
to hit the fastpath in vm_map_ram.

Change-Id: I1accfe67b285cbc9e95e387bea4246864197827d
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: fc04e484c319b8df647886d4cf17f7c450453d1d
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ion: Change secure structure to be 64-bit compatible

When passing structures between the secure and non-secure world,
pointer sizes may be different. Switch to a type of a known size
to avoid issues.

Change-Id: I11a57b3a52d0d1ef9fd3dbaf22f4164734fe71ac
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

msm: secure_buffer: use explicitly-sized types for TZ communication

Some types currently in use for passing data to TZ (like `unsigned
long') have different sizes under different build configurations (32 bit
vs 64 bit, etc). TZ dictates the sizes of the structures that are passed
back-and-forth and they cannot change. Use explicitly-sized types for
any such structures.

Change-Id: Ib6a671f137a875fbabcb045f595cefcbe3616beb
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: clean up ioctls

Convert the ion ioctls to use _IOW instead of _IOWR where
appropriate, and factor out the copy_from_user and copy_to_user
based on the _IOC_DIR bits.  For the existing incorrect ioctls,
add a function to wrap _IOC_DIR to return the corrected value.

Change-Id: I3cc34c84b9c52305bdbec27a9224447b102fadcd
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: b2ee33eb98bd8240787f6b3a6acffdbab40653dc
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: merge conflict since we never renamed
 heap_mask -> heap_id_mask]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: Always flush using kernel virtual address

Cache operations via userspace address always has a risk of a
page fault. On arm64, userspace addresses are marked as write
protect and fixed up on first accesses. If a userspace address
that has not been written to is invalidated, we will take a
fault, fixup and proceed on. If the mmap semaphore is locked at
the time however we cannot fixup the fault and will abort instead.
Rather than hope the mmap semaphore is not write protected, always
invalidate via the kernel virtual addresses instead.

Change-Id: I71eb0d214cce54ebc82a213ae653f891939f273a
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

staging: ion: Avoid using rt_mutexes directly

RT_MUTEXES can be configured out of the kernel, causing compile
problems with ION.

To quote Colin:
"rt_mutexes were added with the deferred freeing feature.  Heaps need
to return zeroed memory to userspace, but zeroing the memory on every
allocation was causing performance issues.  We added a SCHED_IDLE
thread to zero memory in the background after freeing, but locking the
heap from the SCHED_IDLE thread might block a high priority allocation
thread for a long time.

The lock is only used to protect the heap's free_list and
free_list_size members, and is not held for any long or sleeping
operations.  Converting to a spinlock should prevent priority
inversion without using the rt_mutex.  I'd also rename it to free_lock
to so it doesn't get used as a general heap lock."

Thus this patch converts the rt_mutex usage to a spinlock and
renames the lock free_lock to be more clear as to its use.

I also had to change a bit of logic in ion_heap_freelist_drain()
to safely avoid list corruption.

Change-Id: I47cf3551902a01d4441eeb6ade8eeb43ab9cd90b
Acked-by: Colin Cross <ccross@android.com>
Cc: Android Kernel Team <kernel-team@android.com>
Reported-by: Jim Davis <jim.epost@gmail.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Git-commit: c6448537fe9e8b684b214e89c6c68c3b209f549e
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: move shrinker out of heaps

Every heap that uses deferred frees is going to need a shrinker
to shrink the freelist under memory pressure.  Rather than
requiring each heap to implement a shrinker, automatically
register a shrinker if the deferred free flag is set.
The system heap also needs to shrink its page pools, so add
a shrink function to the heap ops that will be called after
shrinking the freelists.

Change-Id: Icda722d683426fadb8ddd1c8e9499264ab682c57
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: f3ae9a8286d63f13715d2082965d6b8849af05a5
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: context differences due to our addition of
 cached page pools and some additional heap callbacks]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: move "freed from shrinker" flag to buffer private_flags

ION_FLAG_FREED_FROM_SHRINKER should not be used by Ion clients. It's an
internal flag. Add a field to ion_buffer for private flags and move
ION_FLAG_FREED_FROM_SHRINKER there. Also rename
ION_FLAG_FREED_FROM_SHRINKER to ION_PRIV_FLAG_SHRINKER_FREE.

Change-Id: I52926ba055fee6256314cd73951e7a6fa3faca9d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: fix overflow and list bugs in system heap

Fix a few bugs in ion_system_heap:

Initialize the list node in the info block.

Don't store size_remaining in a signed long, allocating >2GB
could overflow, resulting in a call to sg_alloc_table with
nents=0 which panics.  alloc_largest_available will never
return a block larger than size_remanining, so it can never
go negative.

Limit a single allocation to half of all memory.  Prevents a
large allocation from taking down the whole system.

Change-Id: I7fcbd7e1d5b4d482d7612d80b6c9e8e24466f1d8
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: c69de844c80884167bb42243b36498dde81eebf6
Git-repo: http://android.googlesource.com/kernel/common/
[mitchelh@codeaurora.org: we already had the signed long fix]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: msm: plumb ION_HEAP_TYPE_DMA to core Ion

ION_HEAP_TYPE_DMA has made its way into core Ion since it was originally
added to the MSM Ion code base. Move some of the hooks and function
declarations out of MSM-specific code and into core Ion. There are still
some differences in the heap implementations themselves that will need
to be rectified later.

Change-Id: I44a4aa3a48645072cee0f2d0718ce3e795b27c4b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

msm: secure_buffer: warn if we get a non-1MB-aligned chunk

We only support securing memory in 1MB chunks. If we get an sg table
with any entries that are of any other size, WARN and bail.

Change-Id: Icd96984f03dafd246da51e879cd91c77ae986724
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

staging: android: Reduce allocation time from system heap

If we are under heavy memory pressure the shrinker might be running on
the system heap. This causes contention with the page pool lock when we
try to allocate from system heap. To ensure that allocations aren't
slowed down by having to wait for the page pool mutex we will just try
to get the mutex. If we are unsuccessful we just bypass the pool
altogether.

Change-Id: I3e92e3081355a8528438e41e74b716a2b49a3891
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>

ion: msm: clean up msm custom ioctls

Similar to what was done in core Ion in [$(git rev-parse HEAD)
7ba5b038cc8 "ion: clean up ioctls"], convert the ion ioctls to use _IOW
instead of _IOWR where appropriate, and factor out the copy_from_user.
For the existing incorrect ioctls, add a function to wrap _IOC_DIR to
return the corrected value.

Change-Id: If8ad728fc9d3905d30d722299c4447a00f01992e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: simplify client name uniquification

The current method of finding a unique serial number for Ion client
names is not straightforward and prone to future breakage. Rather than
parsing all client names and extracting the max out of that just store
the client `id' in a new field in `struct ion_client'.

Change-Id: Ic25008e690ae4f9e3b4c19d1e438b8aaadb36a32
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>

ion: retry during race with shrinking/allocating

There exists a race between shrinking/allocating

Thread 1:                           Thread:2
mutex_lock(&sheap->alloc_lock)
ion_secure_cma_add_to_pool()
mutex_lock(&sheap->chunk_lock)
chunk->cnt = 0
mutex_unlock(&sheap->chunk_lock)
                                    ion_secure_cma_shrinker
                                    mutex_lock(&sheap->chunk_lock)
                                    __ion_secure_cma_shrink_pool
                                    chunk->cnt == 0 for the
                                       just allocated chunk
                                    ion_secure_cma-free_chunk
ion_secure_cma_alloc_from_pool
no memory to allocate!

If this race happens, just retry adding memory to the pool instead
of BUGing out the system.

Change-Id: Ic95782db6d88a33f01dcc5248fc957ca0f378078
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
msm: Update Ion and IOMMU APIs to use dma_addr_t

The current APIs for Ion and the custom IOMMU mapping assume that
many parameters are unsigned long. This type is not big enough to
hold physical addresses on LPAE systems. Update the APIs to accomodate
these types. Because we are updating the APIs, all clients need to be
updated as well, either by using the correct type or using a cast
where necessary.

Change-Id: I12e6fb6ec8a8481a7eef374cb9316e5ccbc29090
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ion: Use dma_free_attrs for freeing secure memory

Secure CMA memory is allocated using DMA_ATTR_NO_KERNEL_MAPPING.
Free the memory with the same attribute.

Change-Id: I67289af68763c5b6d4b0f3d828174ad28b711642
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

staging: ion: remove struct ion_page_pool_item

The page pool uses an internal data structure, ion_page_pool_item, for
wrapping pooled pages and constructing a list. As the struct page
already provides ways for doing exactly the same thing, we do not need
to reinvent the wheel. This commit removes the data structure and slab
allocations for it.

Change-Id: I3653a71337abc26c0cd6791be7cb175055281036
Signed-off-by: Heesub Shin <heesub.shin@samsung.com>
Reviewed-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Tested-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Git-commit: 38c003b113ceab37f80d020999d3ec2cbf784da3
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>

staging: ion: shrink highmem pages on kswapd

ION system heap keeps pages in its pool for better performance. When the
system is under memory pressure, slab shrinker calls the callback
registered and then the pages pooled get freed.

When the shrinker is called, it checks gfp_mask and determines whether
the pages from highmem need to be freed or the pages from lowmem.
Usually, slab shrinker is invoked on kswapd context which gfp_mask is
always GFP_KERNEL, so only lowmem pages are released on kswapd context.
This means that highmem pages in the pool are never reclaimed until
direct reclaim occurs. This can be problematic when the page pool holds
excessive amounts of highmem.

For now, the shrinker callback cannot know exactly which zone should be
targeted for reclamation, as enough information are not passed to. Thus,
it makes sense to shrink both lowmem and highmem zone on kswapd context.

Change-Id: Ibf21e36e6d370967fa9166d6e5d66aec98851478
Reported-by: Wonseo Choi <wonseo.choi@samsung.com>
Signed-off-by: Heesub Shin <heesub.shin@samsung.com>
Reviewed-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Tested-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Git-commit: 0cd2dc4db3ba23ff8ab38b8ae81b126626f795d5
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
[shashim@codeaurora.org: resolve trivial merge conflict]
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>

Signed-off-by: Dan Trifan <jimsth@gmail.com>