sched: Introduce spill threshold tunables to manage
 overcommitment

When the number of tasks intended for a cluster exceed the number of
mostly idle CPUs in that cluster, the scheduler currently freely uses
CPUs in other clusters if possible. While this is optimal for
performance the power trade off can be quite significant. Introduce
spill threshold tunables that govern the extent to which the scheduler
should attempt to contain tasks within a cluster.

Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: notify cpufreq on over/underprovisioned CPUs

After a migration occurs the source and destination CPUs may
not be running at frequencies which match the new task load on
those CPUs.

Previously, the scheduler was notifying cpufreq anytime a task
greater than a certain size migrates. This is suboptimal however
since this does not take into account the CPU's current
frequency and other task activity that may be present.

Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: window-stats: apply scaling to full elapsed windows

In the event that a full window (or multiple full windows) have
elapsed when updating a task's window-based stats, the runtime of
those windows needs to be scaled based on the CPU frequency. This
is currently missing, causing full windows to be accounted as having
elapsed at maximum frequency, erroneously inflating task demand.

Change-Id: I356b4279d44d4f39c8aea881c04327b70ed66183
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Use historical load for freq governor input

Historical load maintained per task can be used to influence cpu
frequency better. For example, when a heavy demand task wakes up after
prolonged sleep, we could use the historical load information to alert
cpufreq governor about the need to raise cpu frequency. This patch
changes CPU busy statistics to be aggregation of historical task
demand. Also task's historical load (as defined by
sysctl_sched_window_stats_policy) is add to cpu's busy statistics
(rq->curr_runnable_sum) whenever it executes on a cpu.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: do not balance on exec if SCHED_HMP

Rebalancing at exec time will currently undo any beneficial placement
that has been done during fork time, since select_best_cpu() will not
discount the currently running task.

For now just skip re-evaluating task placement at exec.

Change-Id: I1e5e0fcc329b7b53c338c8c73795ebd5e85a118b
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Add new trace events

Add trace events for update_task_ravg(), update_history(), and
set_task_cpu(). These tracepoints are useful for monitoring the
per-task and per-runqueue demand statistics.

Change-Id: Ibec9f945074ff31d1fc1a76ae37c40c8fea8cda9
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: update task statistics when CPU frequency changes

A CPU may have its frequency changed by a different CPU. Because of
this, it is not guaranteed that we will update task statistics at
approximately the same time that the frequency change occurs. To
guard against accruing time to a task at the wrong frequency, update
the task's window-based statistics if the CPU it is running on
changes frequency.

Change-Id: I333c3f8aa82676bd2831797b55fd7af9c4225555
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Update capacity of all online cpus when min_max_freq changes

During bootup, its possible for min_max_freq to change as frequency
information for additional clusters is processed. That would need to
trigger recalculation of capacity/load_scale_factor for all (online)
cpus, as they strongly depend on min_max_freq variable. Not doing so
would imply some cpus will have their capacity/load_scale_factor
computed wrongly.

Change-Id: Iea5a0a517a2d71be24c2c71cdd805c0733ce37f8
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Move around code

Move up chunk of code to be defined early. This helps a subsequent
patch that needs update_min_max_capacity()

Change-Id: I9403c7b4dcc74ba4ef1034327241c81df97b01ea
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: debug: Print additional information in /proc/sched_debug

Provide information in /proc/sched_debug on min_capacity, max_capacity
and whether pelt or window-based task load statistics is in use.

Change-Id: Ie4e9450652f4c83110dda75be3ead8aa5bb355c3
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: make sched_set_window() return failure when PELT is in use

Window-based load tracking is a pre-requisite for the scheduler to
feed cpu load information to the governor. When PELT is in use, return
failure when governor attempts to set window-size. This will let
governor fall back to other APIs for retrieving cpu load statistics.

Change-Id: I0e11188594c1a54b3b7ff55447d30bfed1a01115
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Account for cpu's current frequency when calculating its power cost

In estimating cost of running a task on a given cpu, cost of cpu at
its current frequency needs to over-ride cost at frequency demanded by
task, where cur_freq exceeds required frequency of task. This is
because placing a task on a cpu can only result in an increase of
cpu's frequency.

Change-Id: I021a3bbaf179bf1ec2c7f4556870256936797eb9
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Avoid active migration of small tasks

We currently check the need to migrate the currently running task in
scheduler_tick(). Skip that check for small tasks, as its not worth
the effort!

Change-Id: Ic205cc6452f42fde6be6b85c3bf06a8542a73eba
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: fair: Reset balance_interval before sending NOHZ kick

balance_interval needs to be reset for anycpu being kicked. Otherwise it
can end up ignoring the kick (i.e not doing load balance for itself).

Also bypass the check for existence of idle cpus in tickless state for
!CONFIG_SCHED_HMP to allow for more aggressive load balance.

Change-Id: I52365ee7c2997ec09bd93c4e9ae0293a954e39a8
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Move call to trace_sched_cpu_load()

select_best_cpu() invokes trace_sched_cpu_load() for all online cpus
in a loop, before it enters the loop for core selection. Moving
invocation of trace_sched_cpu_load() in inner core loop is potentially
more efficient.

Change-Id: Iae1c58b26632edf9ec5f5da905c31356eb95c925
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/fair: Introduce scheduler boost for low latency workloads

Certain low latency bursty workloads require immediate use of highest
capacity CPUs in HMP systems. Existing load tracking mechanisms may be
unable to respond to the sudden surge in the system load within the
latency requirements. Introduce the scheduler boost feature for such
workloads. While boost is in effect the scheduler bypasses regular load
based task placement and prefers highest capacity CPUs in the system
for all non-small fair sched class tasks. Provide both a kernel and
userspace API for software that may have apriori knowledge about the
system workload.

Change-Id: I783f585d1f8c97219e629d9c54f712318821922f
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Make the scheduler aware of C-state for cpus

C-state represents a power-state of a cpu. A cpu could have one or
more C-states associated with it. C-state transitions are based on
various factors (expected sleep time for example). "Deeper" C-states
implies longer wakeup latencies.

Scheduler needs to know wakeup latency associated with various C-states.
Having this information allows the scheduler to make better decisions
during task placement. For example:

- Prefer an idle cpu that is in the least shallow C-state
- Avoid waking up small tasks on a idle cpu unless it is in the least
  shallow C-state

This patch introduces APIs in the scheduler that can be used by the
architecture specific power-management driver to inform the scheduler
about C-states for cpus.

Change-Id: I39c5ae6dbace4f8bd96e88f75cd2d72620436dd1
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/fair: Introduce C-state aware task placement for small tasks

Small tasks execute for small durations. This means that the power
cost of taking CPUs out of a low power mode outweigh any performance
advantage of using an idle core or power advantage of using the most
power efficient CPU. Introduce C-state aware task placement for small
tasks. This requires a two pass approach where we first determine the
most power effecient CPU and establish a band of CPUs offering a
similar power cost for the task. The order of preference then is as
follows:

1) Any mostly idle CPU in active C-state in the same power band.
2) A CPU with the shallowest C-state in the same power band.
3) A CPU with the least load in the same power band.
4) Lowest power CPU in a higher power band.

The patch also modifies the definition of a small task. Small tasks
are now determined relative to minimum capacity CPUs in the system
and not the task CPU.

Change-Id: Ia09840a5972881cad7ba7bea8fe34c45f909725e
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Make task and CPU load calculations safe from truncation

Load calculations have been modified to accept and return 64 bit values.
Fix up all the places where we make such calculations to store the result
in 64 bit variables. This is necessary to avoid issues caused by
truncation of values.

While at it update scale_task_load() to scale_load_to_cpu(). This is
because the API is used to scale load of both individual tasks as well as
the cumulative load of CPUs. In this sense the name was a misnomer. Also
clean up power_cost() to use max_task_load().

Change-Id: I51e683e1592a5ea3c4e4b2b06d7a7339a49cce9c
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Resolve some 64 bit compilation issues

On 64 bit architectures a pointer is no longer the same size as an
int. Therefore any place that does a conversion from int to a pointer
type gives a compilation error. Resolve these by type casting to long
first which is guaranteed to be the same size as a pointer.

Change-Id: I518ac3c562bd3f85893f91ad6dbcd2f0c7bf081b
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>