msm: kgsl: use kgsl_mmu_device_setstate() if the GPU is already idle
Use the default setstate function, which directly reprograms the IOMMU,
to change IOMMU pagetables or flush the TLB if the GPU is already idle.
This condition often occurs when the GPU is being powered down. In this
case it is desirable to avoid the overhead of issuing commands, waiting
for idle and firing events that results from using the GPU
command stream to reprogram the IOMMU.

Change-Id: I633002ac49c8fe58df3f1f6a1fd1ddf705fc1733
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
Signed-off-by: Ajay Dudani <adudani@codeaurora.org>
msm: Fix race condition in domain lookup
During lookup of domains we take a pointer to the root node
of the rb tree before taking the mutex to protect the rb tree.
However, the rb root node might change if another process updates
the rb tree after we take the pointer to the root node. This can
cause us to not find the entry that we are looking for.
Fix this by ensuring we have taken the mutex before getting the
root node pointer.

CRs-fixed: 493503
Change-Id: I4702740e486fda0cd92df0401d8e4706598a899b
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Signed-off-by: Sridhar Gujje <sgujje@codeaurora.org>
sched: Implement smarter wake-affine logic / sched: Micro-optimize the smart wake-affine logic
The wake-affine scheduler feature is currently always trying to pull
the wakee close to the waker. In theory this should be beneficial if
the waker's CPU caches hot data for the wakee, and it's also beneficial
in the extreme ping-pong high context switch rate case.

Testing shows it can benefit hackbench up to 15%.

However, the feature is somewhat blind, from which some workloads
such as pgbench suffer. It's also time-consuming algorithmically.

Testing shows it can damage pgbench up to 50% - far more than the
benefit it brings in the best case.

So wake-affine should be smarter and it should realize when to
stop its thankless effort at trying to find a suitable CPU to wake on.

This patch introduces 'wakee_flips', which will be increased each
time the task flips (switches) its wakee target.

So a high 'wakee_flips' value means the task has more than one
wakee, and the bigger the number, the higher the wakeup frequency.

Now when making the decision on whether to pull or not, pay attention to
the wakee with a high 'wakee_flips', pulling such a task may benefit
the wakee. Also imply that the waker will face cruel competition later,
it could be very cruel or very fast depends on the story behind
'wakee_flips', waker therefore suffers.

Furthermore, if waker also has a high 'wakee_flips', that implies that
multiple tasks rely on it, then waker's higher latency will damage all
of them, so pulling wakee seems to be a bad deal.

Thus, when 'waker->wakee_flips / wakee->wakee_flips' becomes
higher and higher, the cost of pulling seems to be worse and worse.

The patch therefore helps the wake-affine feature to stop its pulling
work when:

wakee->wakee_flips > factor &&
waker->wakee_flips > (factor * wakee->wakee_flips)
The 'factor' here is the number of CPUs in the current CPU's NUMA node,
so a bigger node will lead to more pulling since the trial becomes more
severe.

After applying the patch, pgbench shows up to 40% improvements and no regressions.

Tested with 12 cpu x86 server and tip 3.10.0-rc7.

The percentages in the final column highlight the areas with the biggest wins,
all other areas improved as well:

pgbench    basesmart

| db_size | clients |  tps  ||  tps  |
+---------+---------+-------+   +-------+
| 22 MB   |       1 | 10598 |   | 10796 |
| 22 MB   |       2 | 21257 |   | 21336 |
| 22 MB   |       4 | 41386 |   | 41622 |
| 22 MB   |       8 | 51253 |   | 57932 |
| 22 MB   |      12 | 48570 |   | 54000 |
| 22 MB   |      16 | 46748 |   | 55982 | +19.75%
| 22 MB   |      24 | 44346 |   | 55847 | +25.93%
| 22 MB   |      32 | 43460 |   | 54614 | +25.66%
| 7484 MB |       1 |  8951 |   |  9193 |
| 7484 MB |       2 | 19233 |   | 19240 |
| 7484 MB |       4 | 37239 |   | 37302 |
| 7484 MB |       8 | 46087 |   | 50018 |
| 7484 MB |      12 | 42054 |   | 48763 |
| 7484 MB |      16 | 40765 |   | 51633 | +26.66%
| 7484 MB |      24 | 37651 |   | 52377 | +39.11%
| 7484 MB |      32 | 37056 |   | 51108 | +37.92%
| 15 GB   |       1 |  8845 |   |  9104 |
| 15 GB   |       2 | 19094 |   | 19162 |
| 15 GB   |       4 | 36979 |   | 36983 |
| 15 GB   |       8 | 46087 |   | 49977 |
| 15 GB   |      12 | 41901 |   | 48591 |
| 15 GB   |      16 | 40147 |   | 50651 | +26.16%
| 15 GB   |      24 | 37250 |   | 52365 | +40.58%
| 15 GB   |      32 | 36470 |   | 50015 | +37.14%
Signed-off-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/51D50057.9000809@linux.vnet.ibm.com
[ Improved the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>

Signed-off-by: Paul Reioux <reioux@gmail.com>

Smart wake-affine is using node-size as the factor currently, but the overhead
of the mask operation is high.

Thus, this patch introduce the 'sd_llc_size' percpu variable, which will record
the highest cache-share domain size, and make it to be the new factor, in order
to reduce the overhead and make it more reasonable.

Tested-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
Tested-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Acked-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Link: http://lkml.kernel.org/r/51D5008E.6030102@linux.vnet.ibm.com
[ Tidied up the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>

Signed-off-by: Paul Reioux <reioux@gmail.com>