msm: kgsl: Add thresholds for all available levels and cleanup
sched: Fix select_idle_sibling() bouncing cow syndrome

If the previous CPU is cache affine and idle, select it.

The current implementation simply traverses the sd_llc domain,
taking the first idle CPU encountered, which walks buddy pairs
hand in hand over the package, inflicting excruciating pain.

1 tbench pair (worst case) in a 10 core + SMT package:

  pre   15.22 MB/sec 1 procs
  post 252.01 MB/sec 1 procs

Signed-off-by: Mike Galbraith <bitbucket@online.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1359371965.5783.127.camel@marge.simpson.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched: Implement smarter wake-affine logic

The wake-affine scheduler feature is currently always trying to pull
the wakee close to the waker. In theory this should be beneficial if
the waker's CPU caches hot data for the wakee, and it's also beneficial
in the extreme ping-pong high context switch rate case.

Testing shows it can benefit hackbench up to 15%.

However, the feature is somewhat blind, from which some workloads
such as pgbench suffer. It's also time-consuming algorithmically.

Testing shows it can damage pgbench up to 50% - far more than the
benefit it brings in the best case.

So wake-affine should be smarter and it should realize when to
stop its thankless effort at trying to find a suitable CPU to wake on.

This patch introduces 'wakee_flips', which will be increased each
time the task flips (switches) its wakee target.

So a high 'wakee_flips' value means the task has more than one
wakee, and the bigger the number, the higher the wakeup frequency.

Now when making the decision on whether to pull or not, pay attention to
the wakee with a high 'wakee_flips', pulling such a task may benefit
the wakee. Also imply that the waker will face cruel competition later,
it could be very cruel or very fast depends on the story behind
'wakee_flips', waker therefore suffers.

Furthermore, if waker also has a high 'wakee_flips', that implies that
multiple tasks rely on it, then waker's higher latency will damage all
of them, so pulling wakee seems to be a bad deal.

Thus, when 'waker->wakee_flips / wakee->wakee_flips' becomes
higher and higher, the cost of pulling seems to be worse and worse.

The patch therefore helps the wake-affine feature to stop its pulling
work when:

	wakee->wakee_flips > factor &&
	waker->wakee_flips > (factor * wakee->wakee_flips)

The 'factor' here is the number of CPUs in the current CPU's NUMA node,
so a bigger node will lead to more pulling since the trial becomes more
severe.

After applying the patch, pgbench shows up to 40% improvements and no regressions.

Tested with 12 cpu x86 server and tip 3.10.0-rc7.

The percentages in the final column highlight the areas with the biggest wins,
all other areas improved as well:

	pgbench		    base	smart

	| db_size | clients |  tps  |	|  tps  |
	+---------+---------+-------+   +-------+
	| 22 MB   |       1 | 10598 |   | 10796 |
	| 22 MB   |       2 | 21257 |   | 21336 |
	| 22 MB   |       4 | 41386 |   | 41622 |
	| 22 MB   |       8 | 51253 |   | 57932 |
	| 22 MB   |      12 | 48570 |   | 54000 |
	| 22 MB   |      16 | 46748 |   | 55982 | +19.75%
	| 22 MB   |      24 | 44346 |   | 55847 | +25.93%
	| 22 MB   |      32 | 43460 |   | 54614 | +25.66%
	| 7484 MB |       1 |  8951 |   |  9193 |
	| 7484 MB |       2 | 19233 |   | 19240 |
	| 7484 MB |       4 | 37239 |   | 37302 |
	| 7484 MB |       8 | 46087 |   | 50018 |
	| 7484 MB |      12 | 42054 |   | 48763 |
	| 7484 MB |      16 | 40765 |   | 51633 | +26.66%
	| 7484 MB |      24 | 37651 |   | 52377 | +39.11%
	| 7484 MB |      32 | 37056 |   | 51108 | +37.92%
	| 15 GB   |       1 |  8845 |   |  9104 |
	| 15 GB   |       2 | 19094 |   | 19162 |
	| 15 GB   |       4 | 36979 |   | 36983 |
	| 15 GB   |       8 | 46087 |   | 49977 |
	| 15 GB   |      12 | 41901 |   | 48591 |
	| 15 GB   |      16 | 40147 |   | 50651 | +26.16%
	| 15 GB   |      24 | 37250 |   | 52365 | +40.58%
	| 15 GB   |      32 | 36470 |   | 50015 | +37.14%

Signed-off-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/51D50057.9000809@linux.vnet.ibm.com
[ Improved the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched: Micro-optimize the smart wake-affine logic

Smart wake-affine is using node-size as the factor currently, but the overhead
of the mask operation is high.

Thus, this patch introduce the 'sd_llc_size' percpu variable, which will record
the highest cache-share domain size, and make it to be the new factor, in order
to reduce the overhead and make it more reasonable.

Tested-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
Tested-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Acked-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Link: http://lkml.kernel.org/r/51D5008E.6030102@linux.vnet.ibm.com
[ Tidied up the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
cpufreq: Add cpu-boost driver

When certain bursty and important events take place, it might take a while
for the current cpufreq governor to notice the new load and react to it.
That would result in poor user experience. To alleviate this, the cpu-boost
driver boosts the frequency of a CPU for a short duration to maintain good
user experience while the governor catches up.

Specifically, this commit deals with ensuring that when "important" tasks
migrate from a fast CPU to a slow CPU, the frequency of the slow CPU is
boosted to be at least as high as the fast CPU for a short duration.

Since this driver enforces the boost by hooking into standard cpufreq
ADJUST notifiers, it has several advantages:
- More portable across kernel versions where the cpufreq internals might
  have been rewritten.
- Governor agnostic and hence works with multiple governors like
  conservative, ondemand, interactive, etc.
- Does not affect the sampling period/logic of existing governors.
- Can have the boost period adjusted independent of governor sampling
  period.

Change-Id: Ibd814a20043d0aba64ee7637a4a79b9ffa1b0991
Signed-off-by: Saravana Kannan <skannan@codeaurora.org>
cpufreq: Add a sync limit to cpu-boost

Perform frequency synchronization only when source CPU's frequency
is less than sync_threshold, else sync to the sync_threshold.

Change-Id: I544c414568d4e015b80ce5891dd215275bac95da
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
msm: kgsl: Allow GPU start to run in a high priority queue

The GPU start sequence needs to run with low latency when coming
out of slumber for the first draw command. For those cases allow the
start function to be queued into a high priority workqueue to reduce
the chance of it getting scheduled out.

Change-Id: Ic0dedbada7ef4d63b54e826862fbefdfb71a092a
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
msm: kgsl: Add an input handler to power up the GPU on a touch event

On a device where the primary input is a touchscreen nobody is
surprised that touch events are often the catalyst for drawing
frames. If the GPU is in slumber we can get a jump on the action
by watching for touch events and turning the GPU on in anticipation
of a draw call. Register for events from all input drivers
returning EV_ABS events and queue a work task to power up the GPU
if it is currently in slumber.

Change-Id: Ic0dedbadbdc96ef9ddd78252ea65f8e1cdd00110
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

Conflicts:
	drivers/gpu/msm/adreno_ringbuffer.c
defconfig: set min/max freqs
msm: hotplug: Increase down lock duration and janitize
msm: hotplug: Use high-priority workqueue
msm: hotplug: Add input boost handler

Add an input handler to hotplug additional cpus. Down lock will
take care of keeping the cpu boosted.
cpufreq: ondemand:Fix NULL check for dbs_info->cur_policy

dbs_info->cur_policy can go NULL when processing
CPUFREQ_GOV_LIMITS, NULL check is required
before processing it

Change-Id: Iba99b563e230835dfcd24573335065d68ad25821
CRs-fixed: 576272
Signed-off-by: Swetha Chikkaboraiah <schikk@codeaurora.org>
cpufreq: ondemand:kernel NULL pointer dereference at dbs_check_cpu

Function dbs_check_cpu  calls __cpufreq_driver_getavg
With  policy as IN/OUT parameter.
There is possiblity of policy becoming NULL in
 __cpufreq_driver_getavg. Same policy is getting
accessed in the for loop which leads to NULL pointer dereference.
So NULL check is required for the policy

CRs-fixed: 582925
Change-Id: Ib8d8de8d66500430a5aa4c275937640e4f934aa8
Signed-off-by: Swetha Chikkaboraiah <schikk@codeaurora.org>
cpufreq: cpu-boost: Make boost thread start/stoppable
cpufreq: Add Input Boost feature to the cpu-boost driver

On incoming input events boost the frequency of all online cpus
for at least input_boost_ms ms. This is accomplished by changing
the policy->min of all the online cpus to input_boost_freq.

Change-Id: Idb0ab75d68ae4ceff259cbbaaec1a9bb3bc871d3
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: Add a sampling_down_factor for max frequencies

Change min_sample_time to 100ms when running at max

Change-Id: Ia7e35b0625bedf20e7ef3a1f52e5828ffbfed93e
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: Allow 1 ms error in above_hispeed_delay comparisons

Allow for an error of 1 ms while taking into account
above_hispeed_delay for a frequency

Change-Id: I744e44387152e4efb5978df4f2b9533bf79d4582
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: sync freq feature for interactive governor

1) Add load info to cpufreq_interactive_cpuinfo
2) If load on any other online cpu exceeds sync_freq_load_threshold,
   do not allow the frequency to drop below sync_freq

Change-Id: I3617e10f87b85178914a18bcf04ac2a31a4f1ec1
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
(cherry picked from commit c26e25944543c61b34b1830938bb6f353a2580ac)
msm: hotplug: Add minimum interval for input boost