tracing/sched: add load balancer tracepoint

When doing performance analysis it can be useful to see exactly
what is going on with the load balancer - when it runs and why
exactly it may not be redistributing load.

This additional tracepoint will show the idle context of the
load balance operation (idle, not idle, newly idle), various
values from the load balancing operation, the final result,
and the new balance interval.

Change-Id: I9e5c97ae3878bea44e60d189ff3cec2275f2c75e
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
iommu: msm: Use ion_phys_addr_t for iova argument

IOMMU mapping function is using the wrong type for its
IO Virtual Address (IOVA) causing the returned address
to contain garbage in the upper 32 bits when LPAE is enabled.

CRs-fixed: 564365
Change-Id: Ied592fe855a145b0ff32d2a6228ecfcf7fd0a53e
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
gpu: ion: Use list_for_each_safe on error path

On an error, the list of pages is walked and destroyed. Since the list
is being destroyed, use list_for_each_safe to avoid list corruption.

Change-Id: Ia397a11d2d890b3458745820c79df2adaa088ade
CRs-Fixed: 565183
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Also shrink memory cached in the deferred free list

When the system is low on memory, we want to shrink any cached
system memory ion is holding.  Previously we were shrinking memory
in the page pools, but not in the deferred free list.  This patch
makes it possible to shrink both.  It also moves the shrinker
code into the heaps so they can correctly manage any caches they
might contain.

Change-Id: I177f587f999a5220eddbf7af94745aae736cac75
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Git-commit: 072beca55150c0c0b652bbb24267da271e0bdf19
Git-repo: https://android.googlesource.com/kernel/common
[mitchelh@codeaurora.org: minor context fixups due to MSM changes]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Add buffer flag to skip page pooling on free

Currently, when we free a buffer it might actually just go back into a
heap-specific page pool rather than going back to the system. This poses
a problem because sometimes (like when we're running a shrinker in low
memory conditions) we need to force the memory associated with the
buffer to truly be relinquished to the system rather than just going
back into a page pool.

There isn't an obvious use case for this flag by Ion clients. The main
use case right now is to provide a mechanism for the deferred free code
to force stale buffers to bypass page pooling.

Change-Id: I724f89cc037083fe8576784363caa18a34e8705a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: create separate heap and client debugfs directories

It can be slightly annoying to figure out which files under the ion
debugfs directory are heap debug files and which ones are client debug
files. Create separate subdirectories under ion to hold the different
types of debug files.

Change-Id: Ic773ab619ef94b9b4d0f3794def9d37645c7c212
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: don't use "vmalloc" for system heap readable name

The human-readable name of the system heap is currently "vmalloc", which
is misleading since we don't use vmalloc at all for that heap, we use
alloc_pages. Make the human-readable name of the heap "system".

Change-Id: Iefb92ea8adc8d76b8eccc504010aa9bc2fa5c50c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: optimize zero'ing of pages

Instead of using __GFP_ZERO, vmap high-order pages for zero'ing all at
once (or in a few large chunks).

Test results show a ~25% speedup for allocations that are fulfilled from
lowmem, and ~100% speedup for allocations that are fulfilled from
highmem.

Change-Id: I57270361ff195780370e1c6f458985bdbb641efe
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: add page pooling to cached allocations from the system heap

Add a page pool for cached allocations from the system heap.

Change-Id: I6d7688026ff4ee65efa047a164652da1427e72c4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: make sure all clients are exposed in debugfs

Currently, if multiple Ion clients are created with the same name, only
the first one shows up in debugfs. Rectify this by adding a
monotonically-increasing serial number to the debug names of Ion
clients.

CRs-Fixed: 566001
Change-Id: I000e45055d5029c7bccd88c36b238736929da3a5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Support probe deferal for client creation

Ion is treated like any other device in the system. Clients
may try to initialize before Ion is setup. If the Ion driver
has not yet probed, return -EPROBE_DEFER to try again later.
Since the flow is being restructured to support probe deferal,
also include some better error handling for when Ion returns
an error while probing.

Change-Id: I7ee6dde2c837fba8bba0ba4aaddee3af37b2fec1
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: fix zero'ing of cached buffers in system heap

Currently, in the system heap we zero out buffers at free-time when they
came from uncached allocations. The reason for this is that uncached
allocations come from the page pools, which require this free-time
zero'ing. However, the assumption that a buffer came from the page pools
iff it was an uncached allocation is no longer valid since [0204b38
"gpu: ion: add page pooling to cached allocations from the system heap"]
introduced page pooling for cached allocations as well. This results in
us not zero'ing out cached buffers before putting them back in the page
pools, which is a security bug (we could potentially leak sensitive
information).

Fix this by zero'ing out all buffers that are going back into the page
pools (cached and uncached).

Change-Id: I5c51eca75c8cc3b3823df1494bb0eef295c77b86
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: kmalloc_heap: flush cpu caches at allocation time

There's currently a problem with the kmalloc heap where buffers
occasionally come through unzero'd when mmap'd to userspace, which is a
security hole. We actually are zero'ing the buffer but that data isn't
immediately visible to folks with other mappings to the memory (e.g. the
mapping set up for userspace by the mmap handler). Fix this by flushing
the cpu caches at alloc time.

Change-Id: I3895c6bc163e2d065b48cae66e99825c5c0a865b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: system_heap: invalidate cache on cached pool allocations

We need to invalidate the cache when allocating pages for the cached
page pools to avoid the possibility of a stale cached value from a
different mapping being written back after we allocate our page.

Change-Id: I9b0ebfd143850cf4a4ef309a264f947a57abff77
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: use a list instead of a tree for heap debug memory map

Currently we use an rb tree to store information about the memory map
which gets passed to the heap print_debug functions. The reason for
using a tree instead of a simple list is to maintain sortedness as we
build the memory map. However, it can be necessary to store multiple
entries for the same address in the memory map since there can be
multiple clients with handles to the same buffer. This information is
interesting and useful but we currently can't store and display it since
the rb tree requires that the key used for sorting (the physical address
in this case) be unique. Fix this by replacing the rb tree with a linked
list. In order to maintain sorted output of the print_debug functions,
sort the list by physical address after fully building the list.

This also has the positive side-effect of simplifying the code and
making future print_debug methods less error-prone.

CRs-Fixed: 571918
Change-Id: I5b129fd809fb53c66042eab10d096238a34c2b20
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: fix locking issues in debug code

There are a few places in Ion where we are iterating over volatile rb
trees without proper locking. In some places the proper locking cannot
be added since it would require us to take locks in a different order
than they are taken in other places in Ion. Fix this by re-working some
of the debug code so that we can take locks in an allowed order.

One side-effect of the re-work is that the memory maps will now show
every client that has a handle to a particular region of memory, rather
than just showing the first one that we encounter. This will allow for
more accurate accounting and will give better insight as to who is
actually using the memory.

CRs-Fixed: 571918
Change-Id: Ia43e4dbc412cd480c828173f8c20b5095d87d858
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: add locking to traversal of volatile rb tree

In ion_debug_heap_show we're iterating over an rb tree (dev->clients)
that could change while we're iterating. Fix this by taking the lock
that is used to control access to this tree.

CRs-Fixed: 571918
Change-Id: I6832e1e98e2d2a69fc653451d3752d43ec3ef269
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
gpu: ion: Explictly mark DMA allocations as GFP_KERNEL

For legacy reasons, the GFP allocation for dma allocations is currently
marked as 0, which corresponds to atomic allocations. These allocations
should not be marked as atomic since the atomic pool is very small.
Switch to properly using GFP_KERNEL to always allocate from the
correct pool.

Change-Id: Id3ce68b538a3d6dbdc8857109cca9ec8706ca4ef
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
gpu: ion: Flush the chunk list before passing it to trustzone

Previously, the function scm_call used to take care of flushing
the caches to ensure all data was written back to memory before
going into the secure world. The behavior of scm_call is changing
such that clients are expected to flush caches themselves. Since
the address that the chunk list is written to is cached, we need
to ensure that the data is actually written back. Add the appropriate
cache operations on the chunk list.

Change-Id: I05e6fa868409255e66f1f37f98199c5a7e81882c
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
kernel: cgroup: push rcu read locking from css_is_ancestor() to callsite

Library functions should not grab locks when the callsites can do it,
even if the lock nests like the rcu read-side lock does.

Push the rcu_read_lock() from css_is_ancestor() to its single user,
mem_cgroup_same_or_subtree() in preparation for another user that may
already hold the rcu read-side lock.

Change-Id: I0ec7ec0059ed588d8f85bc9be8fdc42ce0ca7f5d
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Konstantin Khlebnikov <khlebnikov@openvz.org>
Acked-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Li Zefan <lizf@cn.fujitsu.com>
Cc: Li Zefan <lizf@cn.fujitsu.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Git-commit: 91c63734f6908425903aed69c04035592f18d398
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>