staging: ion: Add source of allocated page

It might be useful for higher levels in the call chain to know whether a
page came from the page pool or from the buddy allocator. If a page came
from the page pool we know that the page has been zeroed already and we
do not have to do that again.

Change-Id: I849591a56e5af8a5bc4340bff70f23e3702e62fb
CRs-fixed: 660049
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
staging: ion: Add GFP_NO_KSWAPD flag to high order allocations

When allocating from system heap we try higher order allocations first
before falling back on lower allocations. However, we do not want to
incur much overhead by doing the fallback. So don't try to run kswapd
during these allocation attempts.

CRs-fixed: 660049
Change-Id: I80895c456c80342cc3b2100470dfd9550c81bd29
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
staging: ion: Optimize zeroing when allocating from system heap

Currently when allocating from system heap each page is individually
mapped and zero'ed in the page pool code. Under high memory pressure when
we allocate a lot of small pages this becomes a bottleneck. Instead,
do the zeroing in one fell swoop at the end of allocating all the pages
but only for the pages that are not already zero'ed and sync'ed (pages
coming from the page pool).

Test results
------------
Time recorded for allocating 20 x 19MB ION buffers from System heap with
forced 4K page allocations (all pages allocated was order-0) right after
boot to Android homescreen using performance kernel configuration.

Before change (non-cached allocations):
Run 1: 768ms
Run 2: 752ms
Run 3: 759ms

Before change (cached allocations):
Run 1: 726ms
Run 2: 769ms
Run 3: 645ms

After change (cached allocations):
Run 1: 262ms
Run 2: 249ms
Run 3: 310ms

After change (non-cached allocations):
Run 1: 275ms
Run 2: 234ms
Run 3: 259ms

Time recorded for allocating 20 x 19MB ION buffers from System heap (with no
forced page size) right after boot to Android homescreen using
performance kernel configuration. Run 1 is taken right after boot while
run 2 and 3 are subsequent runs without reboot between.

Before change (non-cached allocations):
Run 1: 181ms
Run 2: 1ms
Run 3: 1ms

Before change (cached allocations):
Run 1: 220ms
Run 2: 1ms
Run 3: 2ms

After change (non-cached allocations):
Run 1: 166ms
Run 2: 1ms
Run 3: 1ms

After change (cached allocations):
Run 1: 170ms
Run 2: 1ms
Run 3: 3ms

CRs-fixed: 660049
Change-Id: I8dcfdb4adb8d5f4480768f951523fa5c64db87b1
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
(cherry picked from commit 14fe9a42e5b6d27a5dfdf4559e60956010287250)

Conflicts:
	drivers/gpu/ion/ion_heap.c
ion: Skip zeroing on secure buffers

Secure buffers are never passed to userspace and are controled
by the secure world so there is no real need to zero. Pass the
dma attribute to skip zeroing.

Change-Id: Iad870d0d7732d3dea09443418b9294cb9e05b5e0
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
ion: shrinker free pages even for unmovable page requests

In the secure CMA heap shrinker free pooled free CMA pages
even for unmovable page requests.

Most memory reclaim is performed by kswapd which uses a gfp
mask of GFP_KERNEL. Therefore even when the system is looking
for movable pages memory reclaim will generally be done by
kswapd.

By allowing the secure CMA heap shrinker to return free pooled
CMA pages for unmovable page requests there will generally be
a headroom improvement whenever a memory intensive userpace use
case is carried out after CMA memory is freed by its client.

Change-Id: Id6edcef4952aa86499220281470189ff93f8026a
Signed-off-by: Liam Mark <lmark@codeaurora.org>
ion: msm: move custom zero'ing functions out of core Ion

MSM targets currently use some optimized zero'ing functions. However,
there are always merge conflicts when bringing in patches from upstream
so these custom functions should be moved into the MSM space and the
original zero'ing functions from upstream should be restored.

All users of the custom zero'ing functions should continue to use the
custom functions. The goal, over time, is to unify the approaches but
for now just leave them separated.

Change-Id: I4818c09569976a82b8fecc5cd81129ce65ee7db3
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ion: optimize ion_heap_buffer_zero

ion_heap_buffer_zero can spend a long time in unmap_kernel_range
if it has to broadcast a tlb flush to every cpu for every page.
Modify it to batch pages into a larger region to clear using a
single mapping.  This may cause the mapping size to change if
the buffer size is not a multiple of the mapping size, so
switch to allocating the address space for each chunk.  This
allows us to use vm_map_ram to handle the allocation and mapping
together.

The number of pages to zero using a single mapping is set to 32
to hit the fastpath in vm_map_ram.

Change-Id: I1accfe67b285cbc9e95e387bea4246864197827d
Signed-off-by: Colin Cross <ccross@android.com>
Git-commit: fc04e484c319b8df647886d4cf17f7c450453d1d
Git-repo: http://android.googlesource.com/kernel/common/
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>