cpufreq: Add cpu-boost driver

When certain bursty and important events take place, it might take a while
for the current cpufreq governor to notice the new load and react to it.
That would result in poor user experience. To alleviate this, the cpu-boost
driver boosts the frequency of a CPU for a short duration to maintain good
user experience while the governor catches up.

Specifically, this commit deals with ensuring that when "important" tasks
migrate from a fast CPU to a slow CPU, the frequency of the slow CPU is
boosted to be at least as high as the fast CPU for a short duration.

Since this driver enforces the boost by hooking into standard cpufreq
ADJUST notifiers, it has several advantages:
- More portable across kernel versions where the cpufreq internals might
  have been rewritten.
- Governor agnostic and hence works with multiple governors like
  conservative, ondemand, interactive, etc.
- Does not affect the sampling period/logic of existing governors.
- Can have the boost period adjusted independent of governor sampling
  period.

Change-Id: Ibd814a20043d0aba64ee7637a4a79b9ffa1b0991
Signed-off-by: Saravana Kannan <skannan@codeaurora.org>
cpufreq: Add a sync limit to cpu-boost

Perform frequency synchronization only when source CPU's frequency
is less than sync_threshold, else sync to the sync_threshold.

Change-Id: I544c414568d4e015b80ce5891dd215275bac95da
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: Add Input Boost feature to the cpu-boost driver

On incoming input events boost the frequency of all online cpus
for at least input_boost_ms ms. This is accomplished by changing
the policy->min of all the online cpus to input_boost_freq.

Change-Id: Idb0ab75d68ae4ceff259cbbaaec1a9bb3bc871d3
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: Disable cpu-boost by default

Disable cpu-boost by default, so that it can be turned ON only by
setting the module parameter boost_ms through command prompt when
required.

Change-Id: Ia9bc280892f65ed1d2e8051d1951e51922ad13db
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: cpu-boost: Fix queue_delayed_work_on() race with hotplug

Calling queue_delayed_work_on() on a CPU that's in the process of getting
hotplugged out can result in that CPU infinitely looping in
msm_pm_wait_cpu_shutdown(). If queue_delayed_work_on() is called after the
CPU is hotplugged out, it could wake up the CPU without going through the
hotplug path and cause instability. To avoid this, make sure the CPU is and
stays online while queuing a work on it.

Change-Id: I1b4aae3db803e476b1a7676d08f495c1f38bb154
Signed-off-by: Saravana Kannan <skannan@codeaurora.org>
cpufreq: cpu-boost: Fix deadlock in wake_up of sync threads

If wake_up() is called on the current task on a CPU, the call will wait
until the current task is switched out before it wakes it up again and
returns.

The sync notifier for a CPU always runs on that CPU.

These two together can result in a deadlock if the sync notifier on CPU A
tries to wake up the sync thread of CPU A as it goes to sleep (is the
current task). A previous commit fixed this by adding a check to the sync
notifier to not wake up the sync thread of CPU A if it's the current task.

But this is still not sufficient to prevent deadlocks.

Sync thread of CPU A could be the current task on CPU B and sync thread of
CPU B could be the current task on CPU A.  At this point, if sync notifier
of CPU A and B try to wake up the sync threads of CPU A and B, it will
result in CPU A waiting for the current task in CPU B to get switched out
and CPU B waiting for the current task in CPU A to get switched out.  This
will result in a deadlock.

Prevent this scenario from happening by pinning the sync threads of each
CPU to run on that CPU. By doing this, we guarantee that sync notifiers
will only try to wake up sync threads running on that CPU. The fix added by
"cpufreq: cpu-boost: Resolve deadlock when waking up sync thread" ensures a
deadlock doesn't happen when a sync notifier tries to wake up a sync thread
running on that CPU.

Change-Id: I864e545529722a23886dd5a82f66089155d2d193
Signed-off-by: Saravana Kannan <skannan@codeaurora.org>
cpufreq: cpu-boost: Don't register for cpufreq notifiers too early

The cpufreq notifiers should be registered only after all the data
structures used in the notifier callbacks have been initialized. So, move
the cpufreq notifier registration to a later point in the init function.

Change-Id: I043ab5bc0ebb98164c40549fe151a8d801c8c186
Signed-off-by: Saravana Kannan <skannan@codeaurora.org>
cpufreq: cpu-boost: Resolve deadlock when waking up sync thread

CPU boost driver receives notification from scheduler when threads
migrate towards a cpu and in turn wakes up a sync thread associated
with that cpu to handle any required frequency transitions. The wakeup
call however can lead to a deadlock inside scheduler under some
circumstance. The deadlock is seen when sync thread is the only thread
running on a cpu and goes to sleep (say by calling wait_event() ->
schedule()). Midway through this sleep (schedule()) call, while cpu is
still running in context of sync thread, scheduler attempts a load
balance (realizing that cpu is about to become idle) which can result
in tasks being migrated towards the cpu going idle. This will cause
migration notification to be issued and in turn a wakeup on sync
thread. The wakeup call however gets stuck in below while() loop
inside scheduler:

try_to_wake_up(struct task_struct *p, ...)
{

        /*
         * If the owning (remote) cpu is still in the middle of
	 * schedule() with this task as prev, wait until its done
	 * referencing the task.
         */
	while (p->on_cpu)
		cpu_relax();

}

A possible fix could be to teach try_to_wake_up() about this
special case. Another fix, implemented in this patch and that helps
minimize scheduler changes, is to have cpu boost driver not issue a
wakeup under this special circumstance, which was found to occur very
infrequently.

Change-Id: I92bc68a22d51595a208673fe2a1eedfa97004f9e
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
cpufreq: cpu-boost: Re-issue boosts above minimum frequency

Frequency boosts where the source CPU frequency is greater than CPU's
minimum frequency should always go through regardless of the destination
CPU's current frequency. This fixes a performance issue where the governor
lowers the CPU frequency shortly after a thread is migrated to it because
the boost wasn't re-issued.

Change-Id: I449545a688d84b0a6e834f5a51dcb499caa84d29
Signed-off-by: Patrick Cain <pcain@codeaurora.org>
cpufreq: Sync on thread migration optimizations

When threads migrate between CPUs the destination CPU will temporarily
have a boosted frequency and the source CPU will have a residual higher
frequency even though the load could have just been reduced. To avoid
leaving CPUs at a high frequencies notify the source CPU after a thread
migration and make the interactive governor's timer reschedule
conditionally.

Change-Id: Id55055f2a116ca7ffd2a34dc3d0ec519c888a111
Signed-off-by: Patrick Cain <pcain@codeaurora.org>
cpufreq: cpu-boost: Use interruptible wait to not affect load average

Using the function wait_event in cpu_boost puts the
process enter to 'D' state which contribute to the
high load average. This change will put the process
boost_sync in the 'S' state (interruptible sleep)

Change-Id: Ie121adbe1fac1d2862ac5342bb97c7c926f7d7a8
CRs-Fixed: 655484
Signed-off-by: Swetha Chikkaboraiah <schikk@codeaurora.org>
Signed-off-by: Raghavendra Ambadas <rambad@codeaurora.org>
cpufreq: cpu-boost: Support separate input_boost_freq for different CPUs

Different types of CPUs could have different frequency to satisfy same
input workload. Add support for using different input_boost_freq on
different CPUs.

input_boost_freq now either takes a single number which applies to all
CPUs, or cpuid:freq pairs separated by space for different CPUs.

Change-Id: I20506a9fbdb4d532d94168bbd61744595bebc8e5
Signed-off-by: Junjie Wu <junjiew@codeaurora.org>
cpufreq: cpu-boost: Use one work to remove input boost for all CPUs

Currently each CPU queues up its own work for removing input boost.
This is not really required as boost removal for all the CPUs can
be done at the same time. So this patch uses a single work to
remove input boost for all the CPUs and updates the policy for
the online ones.

Change-Id: I37c809f2f155548b1d8c1b9aa7626c8852b3acc6
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
sched: provide per cpu-cgroup option to notify on migrations

On systems where CPUs may run asynchronously, task migrations
between CPUs running at grossly different speeds can cause
problems.

This change provides a mechanism to notify a subsystem
in the kernel if a task in a particular cgroup migrates to a
different CPU. Other subsystems (such as cpufreq) may then
register for this notifier to take appropriate action when
such a task is migrated.

The cgroup attribute to set for this behavior is
"notify_on_migrate" .

Change-Id: Ie1868249e53ef901b89c837fdc33b0ad0c0a4590
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
sched: remove migration notification from RT class

Commit 88a7e37d265 (sched: provide per cpu-cgroup option to
notify on migrations) added a notifier call when a task is moved
to a different CPU. Unfortunately the two call sites in the RT
sched class where this occurs happens with a runqueue lock held.
This can result in a deadlock if the notifier call attempts to do
something like wake up a task.

Fortunately the benefit of 88a7e37d265 comes mainly from notifying
on migration of non-RT tasks, so we can simply ignore the movements
of RT tasks.

CRs-Fixed: 491370
Change-Id: I8849d826bf1eeaf85a6f6ad872acb475247c5926
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
switch do_fsync() to fget_light()

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

Conflicts:
	fs/sync.c
soc: qcom: rq_stats: Pass correct cpu id to update_average_load function

Pass correct cpu number to update_average_load function
from cpufreq_transition_handler function. Also avoid div by zero
condition in update_average_load function.

CRs-Fixed: 708310
Change-Id: I4ec3e54b92c760ac5fa910542524fdfe919f55e9
Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org>
[zwliew: Backport to msm-3.4]
Signed-off-by: Zhao Wei Liew <zhaoweiliew@gmail.com>

Conflicts:
	arch/arm/mach-msm/msm_rq_stats.c
cpufreq: interactive: Record hispeed_validated_time in timer function

Change 0cd554ae585f5ccbb95056e049a8fec0f0742dce
(cpufreq: interactive: Exercise hispeed settings at a policy level)
introduced policy level hispeed settings. The change is correct in
general, but it sets hispeed_validate_time after actual frequency
change has completed. Waking up speedchange task and setting
frequency takes non-trival amount of time. This period is not accounted
for in above_hispeed_delay, resulting in additional delays when
ramping up frequency. Frequency switch latency varies a lot
depending on beginning and end frequency, and thus it cannot be easily
compensated by user setting above_hispeed_delay.

Record a local hispeed_validated_time in every CPU's timer function.
Cluster hispeed_validated_time is the local hispeed_validated_time of
CPUs voting for the highest frequency

Change-Id: Id8ae547fe3a70f8710f60b6e2125954111b7a2b6
Signed-off-by: Junjie Wu <junjiew@codeaurora.org>
cpufreq: interactive: Ensure hvt variable is initialized

Signed-off-by: Zhao Wei Liew <zhaoweiliew@gmail.com>
sched: Make scheduler aware of cpu frequency state

Capacity of a cpu (how much performance it can deliver) is partly
determined by its frequency (P) state, both current frequency as well
as max frequency it can reach.  Knowing frequency state of cpus will
help scheduler optimize various functions such as tracking every
task's cpu demand and placing tasks on various cpus.

This patch has scheduler registering for cpufreq notifications to
become aware of cpu's frequency state. Subsequent patches will make
use of derived information for various purposes, such as task's scaled
load (cpu demand) accounting and task placement.

Change-Id: I376dffa1e7f3f47d0496cd7e6ef8b5642ab79016
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>