sched/rt: print RT tasks when RT throttling is activated

Existing debug prints do not provide any clues about which tasks
may have triggered RT throttling. Print the names and PIDs of
all tasks on the throttled rt_rq to help narrow down the source
of the problem.

Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/rt: Add Kconfig option to enable panicking for RT throttling

This may be useful for detecting and debugging RT throttling issues.

Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: fix race between try_to_wake_up() and move_task()

Until a task's state has been seen as interruptible/uninterruptible
and it is no longer on_cpu, it is possible that the task may move
to another CPU (load balancing may cause this). Here is an example
where the race condition results in incorrect operation:

- cpu 0 calls put_prev_task on task A, task A's state is TASK_RUNNING
- cpu 0 runs task B, which attempts to wake up A
- cpu 0 begins try_to_wake_up(), recording src_cpu for task A as cpu 0
- cpu 1 then pulls task A (perhaps due to idle balance)
- cpu 1 runs task A, which then sleeps, becoming INTERRUPTIBLE
- cpu 0 continues in try_to_wake_up(), thinking task A's previous
  cpu is 0, where it is actually 1
- if select_task_rq returns cpu 0, task A will be woken up on cpu 0
  without properly updating its cpu to 0 in set_task_cpu()

Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: move task load based functions

The task load based functions will need to make use of LOAD_AVG_MAX
in a subsequent patch, so move them below the definition of that
macro.

Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Add min_max_freq and rq->max_possible_freq

rq->max_possible_freq represents the maximum frequency a cpu is
capable of attaining, while rq->max_freq represents the maximum
frequency a cpu can attain at a given instant. rq->max_freq includes
constraints imposed by user or thermal driver.
rq->max_freq <= rq->max_possible_freq.

max_possible_freq is derived as max(rq->max_possible_freq) and
represents the "best" cpu that can attain best possible frequency.

min_max_freq is derived as min(rq->max_possible_freq). For homogeneous
systems, max_possible_freq and min_max_freq will be same, while they
could be different on heterogeneous systems.

[myfluxi: revert a prior commit, caf got it right now]

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: window-based load stats improvements

Following cleanups and improvements are made to window-based load
stats feature:

* Add sysctl to pick max, avg or most recent samples as task's
  demand.

* Fix overflow possibility in calculation of sum for average policy.

* Use unscaled statistics when a task is running on a CPU which is
thermally throttled.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Introduce CONFIG_SCHED_FREQ_INPUT

Introduce a compile time flag to enable scheduler guidance of
frequency selection. This flag is also used to turn on or off
window-based load stats feature.

Having a compile time flag will let some platforms avoid any
overhead that may be present with this scheduler feature.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Add scaled task load statistics

Scheduler guided frequency selection as well as task placement on
heterogeneous systems require scaled task load statistics. This patch
adds a 'runnable_avg_sum_scaled' metric per task that is a scaled
derivative of 'runnable_avg_sum'. Load is scaled in reference to
"best" cpu, i.e one with best possible max_freq

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[neo: Make use of accessor function.]
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Add CONFIG_SCHED_HMP Kconfig option

Add a compile-time flag to enable or disable scheduler features for
HMP (heterogenous multi-processor) systems. Main feature deals with
optimizing task placement for best power/performance tradeoff.

Also extend features currently dependent on CONFIG_SCHED_FREQ_INPUT to
be enabled for CONFIG_HMP as well.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Introduce efficiency, load_scale_factor and capacity

Efficiency reflects instructions per cycle capability of a cpu.

load_scale_factor reflects magnification factor that is applied for
task load when estimating bandwidth it will consume on a cpu. It
accounts for the fact that task load is scaled in reference to "best"
cpu that has best efficiency factor and also best possible max_freq.
Note that there may be no single CPU in the system that has both the
best efficiency and best possible max_freq, but that is still the
combination that all task load in the system is scaled against.

capacity reflects max_freq and efficiency metric of a cpu. It is
defined such that the "least" performing cpu (one with lowest
efficiency factor and max_freq) gets capacity of 1024. Again, there
may not be a CPU in the system that has both the lowest efficiency
and lowest max_freq. This is still the combination that is assigned
a capacity of 1024 however, other CPU capacities are relative to this.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Use rq->efficiency in scaling load stats

Extend task load scaling function to account for cpu efficiency
factor. Task load is scaled in reference to "most" efficient cpu.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Influence cpu_power based on max_freq and efficiency

Update cpu_power metric of a cpu by accounting its efficiency and
max_freq factors. cpu_power is defined such that "least" performing
cpu (one with lowest efficiency and max_freq factor) gets cpu_power of
1024. Note that no single CPU may have both the lowest efficiency and
lowest max_freq. All CPUs will still have cpu_power values relative to
this combination receiving a cpu_power of 1024 however.

cpu_power differs from capacity metric of a cpu by accounting
real-time task activity.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Basic task placement support for HMP systems

HMP systems have cpus with different power and performance
characteristics. Some cpus could offer better power at cost of lower
performance while other cpus could offer better performance at cost of
higher power. As a result, bandwidth consumed by a task to do some
"fixed" amount of work could vary across cpus.

Optimal task placement on HMP would involve placing a task on a cpu
where it can meet its performance goals at lowest power cost. Since
kernel has little to no awareness of performance goals of
applications, we guestimate whether task is meeting its performance
goals or not by looking at its cpu bandwidth consumption. High
bandwidth consumption could imply that task's performance can improve
by running on cpus with better capacity/performance-characterisitcs.

This patch makes the basic changes to support HMP. It provides a
configurable threshold and any task consuming bandwidth in excess of
threshold will be placed on a cpu with better capacity.

Change-Id: I3fd98edd430f73342fbef06411e8b2d1cf2f56fa
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Revert "sched: Remove unused parameter from find_new_ilb()"

This reverts commit 3b462fd2a747b3bca1ee607d70285fb5f90a319f.
sched: Send NOHZ kick to idle cpu in same cluster

A busy cpu will kick (via IPI) one of the idle cpus in tickless state
to run load balance and help move tasks off itself. The cpu chosen to
receive kick is simply the "first" idle cpu found in
nohz.idle_cpus_mask. This could cause unnecessary wakeups of a
cluster. A better choice would be to look for an idle cpu that is in
the same cluster as busy cpu, which should minimize cluster wakeups.

Change-Id: Ia63038d7c34b416b53c8feef3c3b31dab5200e42
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Extend active balance to accept 'push_task' argument

Active balance currently picks one task to migrate from busy cpu to
a chosen cpu (push_cpu). This patch extends active load balance to
recognize a particular task ('push_task') that needs to be migrated to
'push_cpu'. This capability will be leveraged by HMP-aware task
placement in a subsequent patch.

Change-Id: If31320111e6cc7044e617b5c3fd6d8e0c0e16952
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Handle cpu-bound tasks stuck on wrong cpu

CPU-bound tasks that don't sleep for long intervals can stay stuck on
the wrong cpu, as the selection of "ideal" cpu for tasks largely
happens during task wakeup time. This patch adds a check in the
scheduler tick for task/cpu mismatch (big task on little cpu OR
little task on big cpu) and forces migration of such tasks to their
ideal cpu (via select_best_cpu()).

Change-Id: Icac3485b6aa4b558c4ed9df23c2e81fb8f4bb9d9
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Track number of big and small tasks on a cpu

This patch adds 'nr_big_tasks' and 'nr_small_tasks' per-cpu counters
that tracks number of big and small tasks on a cpu respectively. This
will be used in load balance decisions introduced in a subsequent
patch.

Change-Id: Ia174904140f81dd6d1946286889a50be3f16ea83
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/fair: Let minimally loaded cpu balance the group

Currently we let the leftmost (or first idle) cpu ascend the
sched_domain tree and perform load-balancing. The result is that the
busiest cpu in the group might be performing this function and pull
more load to itself. The next load balance pass will then try to
equalize this again.

Change this to pick the least loaded cpu to perform higher domain
balancing.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/n/tip-v8zlrmgmkne3bkcy9dej1fvm@git.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/fair: Add some serialization to the sched_domain load-balance walk

Since the sched_domain walk is completely unserialized (!SD_SERIALIZE)
it is possible that multiple cpus in the group get elected to do the
next level. Avoid this by adding some serialization.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/n/tip-vqh9ai6s0ewmeakjz80w4qz6@git.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>