sched: Implement smarter wake-affine logic

The wake-affine scheduler feature is currently always trying to pull
the wakee close to the waker. In theory this should be beneficial if
the waker's CPU caches hot data for the wakee, and it's also beneficial
in the extreme ping-pong high context switch rate case.

Testing shows it can benefit hackbench up to 15%.

However, the feature is somewhat blind, from which some workloads
such as pgbench suffer. It's also time-consuming algorithmically.

Testing shows it can damage pgbench up to 50% - far more than the
benefit it brings in the best case.

So wake-affine should be smarter and it should realize when to
stop its thankless effort at trying to find a suitable CPU to wake on.

This patch introduces 'wakee_flips', which will be increased each
time the task flips (switches) its wakee target.

So a high 'wakee_flips' value means the task has more than one
wakee, and the bigger the number, the higher the wakeup frequency.

Now when making the decision on whether to pull or not, pay attention to
the wakee with a high 'wakee_flips', pulling such a task may benefit
the wakee. Also imply that the waker will face cruel competition later,
it could be very cruel or very fast depends on the story behind
'wakee_flips', waker therefore suffers.

Furthermore, if waker also has a high 'wakee_flips', that implies that
multiple tasks rely on it, then waker's higher latency will damage all
of them, so pulling wakee seems to be a bad deal.

Thus, when 'waker->wakee_flips / wakee->wakee_flips' becomes
higher and higher, the cost of pulling seems to be worse and worse.

The patch therefore helps the wake-affine feature to stop its pulling
work when:

	wakee->wakee_flips > factor &&
	waker->wakee_flips > (factor * wakee->wakee_flips)

The 'factor' here is the number of CPUs in the current CPU's NUMA node,
so a bigger node will lead to more pulling since the trial becomes more
severe.

After applying the patch, pgbench shows up to 40% improvements and no regressions.

Tested with 12 cpu x86 server and tip 3.10.0-rc7.

The percentages in the final column highlight the areas with the biggest wins,
all other areas improved as well:

	pgbench		    base	smart

	| db_size | clients |  tps  |	|  tps  |
	+---------+---------+-------+   +-------+
	| 22 MB   |       1 | 10598 |   | 10796 |
	| 22 MB   |       2 | 21257 |   | 21336 |
	| 22 MB   |       4 | 41386 |   | 41622 |
	| 22 MB   |       8 | 51253 |   | 57932 |
	| 22 MB   |      12 | 48570 |   | 54000 |
	| 22 MB   |      16 | 46748 |   | 55982 | +19.75%
	| 22 MB   |      24 | 44346 |   | 55847 | +25.93%
	| 22 MB   |      32 | 43460 |   | 54614 | +25.66%
	| 7484 MB |       1 |  8951 |   |  9193 |
	| 7484 MB |       2 | 19233 |   | 19240 |
	| 7484 MB |       4 | 37239 |   | 37302 |
	| 7484 MB |       8 | 46087 |   | 50018 |
	| 7484 MB |      12 | 42054 |   | 48763 |
	| 7484 MB |      16 | 40765 |   | 51633 | +26.66%
	| 7484 MB |      24 | 37651 |   | 52377 | +39.11%
	| 7484 MB |      32 | 37056 |   | 51108 | +37.92%
	| 15 GB   |       1 |  8845 |   |  9104 |
	| 15 GB   |       2 | 19094 |   | 19162 |
	| 15 GB   |       4 | 36979 |   | 36983 |
	| 15 GB   |       8 | 46087 |   | 49977 |
	| 15 GB   |      12 | 41901 |   | 48591 |
	| 15 GB   |      16 | 40147 |   | 50651 | +26.16%
	| 15 GB   |      24 | 37250 |   | 52365 | +40.58%
	| 15 GB   |      32 | 36470 |   | 50015 | +37.14%

Signed-off-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/51D50057.9000809@linux.vnet.ibm.com
[ Improved the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>

Conflicts:

	kernel/sched/fair.c
sched: Micro-optimize the smart wake-affine logic

Smart wake-affine is using node-size as the factor currently, but the overhead
of the mask operation is high.

Thus, this patch introduce the 'sd_llc_size' percpu variable, which will record
the highest cache-share domain size, and make it to be the new factor, in order
to reduce the overhead and make it more reasonable.

Tested-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
Tested-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Acked-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Link: http://lkml.kernel.org/r/51D5008E.6030102@linux.vnet.ibm.com
[ Tidied up the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
slub: fix a memory leak in get_partial_node()

In the case which is below,

1. acquire slab for cpu partial list
2. free object to it by remote cpu
3. page->freelist = t

then memory leak is occurred.

Change acquire_slab() not to zap freelist when it works for cpu partial list.
I think it is a sufficient solution for fixing a memory leak.

Below is output of 'slabinfo -r kmalloc-256'
when './perf stat -r 30 hackbench 50 process 4000 > /dev/null' is done.

***Vanilla***
Sizes (bytes)     Slabs              Debug                Memory
------------------------------------------------------------------------
Object :     256  Total  :     468   Sanity Checks : Off  Total: 3833856
SlabObj:     256  Full   :     111   Redzoning     : Off  Used : 2004992
SlabSiz:    8192  Partial:     302   Poisoning     : Off  Loss : 1828864
Loss   :       0  CpuSlab:      55   Tracking      : Off  Lalig:       0
Align  :       8  Objects:      32   Tracing       : Off  Lpadd:       0

***Patched***
Sizes (bytes)     Slabs              Debug                Memory
------------------------------------------------------------------------
Object :     256  Total  :     300   Sanity Checks : Off  Total: 2457600
SlabObj:     256  Full   :     204   Redzoning     : Off  Used : 2348800
SlabSiz:    8192  Partial:      33   Poisoning     : Off  Loss :  108800
Loss   :       0  CpuSlab:      63   Tracking      : Off  Lalig:       0
Align  :       8  Objects:      32   Tracing       : Off  Lpadd:       0

Total and loss number is the impact of this patch.

Cc: <stable@vger.kernel.org>
Acked-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Joonsoo Kim <js1304@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
cpu-boost: Fix cores getting stuck. Finally.
* In exclusive hotplug system, if a core is not taken down
  explicitly, it is observed that it gets stuck at input
  boost frequency.
* This issue maybe particular to only this kernel but this
  fix resolves it completely.
* Root cause seems to be removal work is lost if queued on
  sibling cores. So simply fix this by queuing on boot CPU.

* Finally, I can use min_cpus_online > 1 without any issue.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
cpufreq: refresh tunables
cpu-boost: update tunables
msm: hotplug: Add io_is_busy interface
build: Fix some warnings with GCC 4.9

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
makefile: build with -Ofast
GPU: optimize build with -Ofast
msm: hotplug: Check down lock for lowest load cpu

As reported by neobuddy89.
arm: mach-msm: Remove the unused rmt_storage code

The rmt_storage module is no longer used.
Remove the unused module and cleanup the makefile and kconfig.

CRs-Fixed: 549951
Change-Id: Ie9ebaa2209ea9fa0df05dc892504fcb882f00f29
Signed-off-by: Nikhilesh Reddy <reddyn@codeaurora.org>

Conflicts:

	arch/arm/mach-msm/Kconfig
	arch/arm/mach-msm/Makefile
cpufreq: cpu-boost: Implement additional checks

* Return early without calculations if last
  input_boost_work is pending.
* Disable input boost if input_boost_freq value is
  less than or equal to min possible frequency.
* Disable hotplug boost if plug_boost_freq value is
  less than or equal to min possible frequency.
* When we check for current frequency greater than
  boost frequency, also check if boost is removed
  properly or not. If not, do so.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
cpufreq: cpu-boost: Unify input and hotplug boost

* This doesn't make sense to have mirror functions
  operating different boost frequencies.
* If input_boost_freq > hotplug_boost_freq, hotplug
  boost will be rendered useless in most case since
  parallel input boost will be doing it work.
* So instead having separate boost functions,
  keep only single boost function "do_input_boost"
  and let hotplug callback access this boost.
* Disable hotplug boost by default and provide
  attribute for same.

* Increased migration_load_threshold to prevent
  not-so-needed and recurring migrations.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>

Conflicts:

	drivers/cpufreq/cpu-boost.c
cpufreq: cpu-boost: Do not consider keypad events for boost

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
cpufreq: cpuboost: Switch to CAF version

* This change is under review.
* CAF version uses thread structure  for thread migration.
* Remove additional checks.
* After testing, unified hotplug boost is working great,
  so enable by default.
* Load based sync is enabled by default as well.
* Add debug prints to let devs debug and check.
* CAF default for migration threshold is 15 which seems
  way small, use 30 instead.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>

Conflicts:

	drivers/cpufreq/cpu-boost.c
cpufreq: cpuboost: Sanity optimization for hotplug boost

* Don't check if hotplug_boost is enabled for every event.
  We need to check same only when CPU_ONLINE event occurs.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
debug: remove: arm unwind: remove logspam while in debug mode

authored on 3 Nov 2013 by cyanogen
mm: slub: Default slub_max_order to 0

To avoid locking and per-cpu overhead, SLUB optimisically uses
high-order allocations up to order-3 by default and falls back to
lower allocations if they fail. While care is taken that the caller
and kswapd take no unusual steps in response to this, there are
further consequences like shrinkers who have to free more objects to
release any memory. There is anecdotal evidence that significant time
is being spent looping in shrinkers with insufficient progress being
made (https://lkml.org/lkml/2011/4/28/361) and keeping kswapd awake.

SLUB is now the default allocator and some bug reports have been
pinned down to SLUB using high orders during operations like
copying large amounts of data. SLUBs use of high-orders benefits
applications that are sized to memory appropriately but this does not
necessarily apply to large file servers or desktops.  This patch
causes SLUB to use order-0 pages like SLAB does by default.
There is further evidence that this keeps kswapd's usage lower
(https://lkml.org/lkml/2011/5/10/383).

Signed-off-by: Mel Gorman <mgorman@suse.de>
Signed-off-by: flar2 <asegaert@gmail.com>
ARM: dts: msm: increase adsp heap size for msm8974

For hevc hybrid codec, adsp heap size needs to be increased to
64MB to support 1080p video.

Change-Id: I3812cc2f0204f18fbade9ae41d8e784909055569
Signed-off-by: Arun Menon <avmenon@codeaurora.org>
Signed-off-by: flar2 <asegaert@gmail.com>