sched: Make scheduler aware of cpu frequency state

Capacity of a cpu (how much performance it can deliver) is partly
determined by its frequency (P) state, both current frequency as well
as max frequency it can reach.  Knowing frequency state of cpus will
help scheduler optimize various functions such as tracking every
task's cpu demand and placing tasks on various cpus.

This patch has scheduler registering for cpufreq notifications to
become aware of cpu's frequency state. Subsequent patches will make
use of derived information for various purposes, such as task's scaled
load (cpu demand) accounting and task placement.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Revert "tracing/sched: Track per-cpu rt and non-rt cpu_load."

This reverts commit 4dc47fb113fee708933e503f96c3c6345678088d.
sched: window-based load stats for tasks

Provide a metric per task that specifies how cpu bound a task is. Task
execution is monitored over several time windows and the fraction of
the window for which task was found to be executing or wanting to run
is recorded as task's demand. Windows over which task was sleeping are
ignored. We track last 5 recent windows for every task and the maximum
demand seen in any of the previous 5 windows (where task had some
activity) drives freq demand for every task.

A per-cpu metric (rq->cumulative_runnable_avg) is also provided which
is an aggregation of cpu demand of all tasks currently enqueued on it.
rq->cumulative_runnable_avg will be useful to know if cpu frequency
will need to be changed to match task demand.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
cpu-boost: update
sched: Call the notify_on_migrate notifier chain for wakeups as well

Add a change to send notify_on_migrate hints on wakeups of
foreground tasks from scheduler if their load is above
wakeup_load_thresholds (default value is 60).
These hints can be used to choose an appropriate CPU frequency
corresponding to the load of the task being woken up.

Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Disable wakeup hints for foreground tasks by default

By default sched_wakeup_load_threshold is set to 60 and therefore
wakeup hints are sent out for those tasks whose loads are higher
that value. This might cause unnecessary wakeup boosts to happen
when load based syncing is turned ON for cpu-boost.
Disable the wake up hints by setting the sched_wakeup_load_threshold
to a value higher than 100 so that wakeup boost doesnt happen unless
it is explicitly turned ON from adb shell.

Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Revert "sched: Fix race in task_group()"

This reverts commit e194fab8d7ebe95db5609f1cb6794c2afcc3118f.
sched: Window-based load stat improvements

Some tasks can have a sporadic load pattern such that they can suddenly
start running for longer intervals of time after running for shorter
durations. To recognize such sharp increase in tasks' demands, max
between the average of 5 window load samples and the most recent sample
is chosen as the task demand.

Make the window size (sched_ravg_window) configurable at boot up
time. To prevent users from setting inappropriate values for window
size, min and max limits are defined. As 'ravg' struct tracks load for
both real-time and non real-time tasks it is moved out of sched_entity
struct.

In order to prevent changing function signatures for move_tasks() and
move_one_task() per-cpu variables are defined to track the total load
moved. In case multiple tasks are selected to migrate in one load
balance operation, loads > 100 could be sent through migration notifiers.
Prevent this scenario by setting mnd.load to 100 in such cases.

Define wrapper functions to compute cpu demands for tasks and to change
rq->cumulative_runnable_avg.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: window-stats: Fix overflow bug

Multiplication over-flow possibility exists in update_task_ravg() when
updating task's window_start. That would lead to incorrect accounting
of task load. Fix the issue by using 64-bit arithmetic.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/debug: Limit sd->*_idx range on sysctl

Various sd->*_idx's are used for refering the rq's load average table
when selecting a cpu to run.  However they can be set to any number
with sysctl knobs so that it can crash the kernel if something bad is
given. Fix it by limiting them into the actual range.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1345104204-8317-1-git-send-email-namhyung@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Rui Xiang <rui.xiang@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched/debug: Fix sd->*_idx limit range avoiding overflow

Commit ("sched/debug: Limit sd->*_idx range on
sysctl") was an incomplete bug fix.

This patch fixes sd->*_idx limit range to [0 ~ CPU_LOAD_IDX_MAX-1]
avoiding array overflow caused by setting sd->*_idx to CPU_LOAD_IDX_MAX
on sysctl.

Signed-off-by: Libin <huawei.libin@huawei.com>
Cc: <jiang.liu@huawei.com>
Cc: <guohanjun@huawei.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/51626610.2040607@huawei.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Rui Xiang <rui.xiang@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Skip load update for idle task

Load statistics for idle tasks is not useful in any manner. Skip load
update for such idle tasks.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Revert "tracing/sched: add load balancer tracepoint"

This reverts commit 82880f10eb5402a788c854603a75e9cc84f29d6b.
sched: defining relaxed version of idle_cpu

Defining relaxed version of idle_cpu, which uses
macro cpu_relaxed_read_long, that will be used to
enhance power efficiency.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Trying to reduce power usage

Using macros cpu_relaxed_read, cpu_relaxed_read_long and
cpu_read_relax in core.c to improve power efficiency.

Bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Tested-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Force sleep on consecutive sched_yields

If a task sched_yields to itself continuously, force the task to
sleep in sched_yield. This will lower the CPU load of this task
thereby lowering the cpu frequency and improving power.

Added a stat variable to track how many times we sleep due these
consecutive sched_yields. Also added sysctl knobs to control the number
of consecutive sched_yields before which the sleep kicks in and the
duration fo the sleep in us.

Bug 1424617

Signed-off-by: Sai Gurrappadi <sgurrappadi@nvidia.com>
Reviewed-by: Wen Yi <wyi@nvidia.com>
Reviewed-by: Peter Zu <pzu@nvidia.com>
Reviewed-by: Diwakar Tundlam <dtundlam@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Clean-up dual defines

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
sched: Fix select_idle_sibling() bouncing cow syndrome

If the previous CPU is cache affine and idle, select it.

The current implementation simply traverses the sd_llc domain,
taking the first idle CPU encountered, which walks buddy pairs
hand in hand over the package, inflicting excruciating pain.

1 tbench pair (worst case) in a 10 core + SMT package:

  pre   15.22 MB/sec 1 procs
  post 252.01 MB/sec 1 procs

Signed-off-by: Mike Galbraith <bitbucket@online.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1359371965.5783.127.camel@marge.simpson.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>