Revert "msm: cpufreq: Remove cross-calling limitation"

This reverts commit be01a17afb6b6e85456217caa1ac11eb0b6674e1.

Re-introduce cross calling in cpufreq to prevent race conditions in the
hardware when changing PLLs.

A core could be in retention, while another core, could try and set the
frequency of the core in retention, causing a change in PLL and when the
core exits retention, the PLL mismatch could hang the SPM state machine.

Change-Id: I99b454a0f146652f2d8a369f79f6d35748160dcb
Signed-off-by: Praveen Chidambaram <pchidamb@codeaurora.org>
msm: cpufreq: Initialize cpufreq driver early at boot

MSM cpufreq driver registering with the CPUFreq framework during
late_initcall() may cause the device to be available a few seconds after
the clock driver is ready for changing frequency. This could be bad for
boot and/or thermal performance, since the cpu will continue to boot at
whatever frequency the bootloaders set it to.

Initialize cpufreq driver as part of device_initcalL() but only after
all the clock drivers have had a chance to initialize their devices.

Change-Id: Iaa4867108f59a368ee5b511d35ea2e8e219d39d3
Signed-off-by: Praveen Chidambaram <pchidamb@codeaurora.org>

Conflicts:
	arch/arm/mach-msm/Makefile
msm: cpufreq: Always update frequency limits when core is online

Remove condition in msm_cpufreq_init() so that when cpu is set to
online, it will call set_cpu_freq() and update the frequency limits.

CRs-Fixed: 462816
Change-Id: I40ac33c9cb4b42590a83f9d61b74f23379f5642e
Signed-off-by: Jennifer Liu <chihning@codeaurora.org>
msm: cpufreq: Configure WQ for higer priority

When the workqueue runs at a lower priority, it gets starved by higher
priority threads. Bump the WQ priority to high to ensure cpufreq work
queue gets a fair scheduling chance

Change-Id: I9e994da94a347dceb884e72ec3dd3da468a4471d
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
msm: cpufreq: Relax constraints on "msm-cpufreq" workqueue

This workqueue is not used in memory reclaim paths, so the
WQ_MEM_RECLAIM flag is not needed. Additionally, there is no
need to restrict the queue to having one in-flight work item.
Remove these constraints.

Change-Id: I9edde40917d3ec885ce061133de20680634321d0
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
msm: cpufreq: Ensure cpufreq change happens on corresponding CPU

Checking the cpus_allowed mask of the current thread before changing the
frequency doesn't guarantee that the rest of the execution will continue on
the same CPU. The only way to guarantee this is to schedule a work on the
specific CPU and also prevent hotplug  of that CPU (already done by
existing code).

Change-Id: I51a02fcc777a47d3c16f2d83c47e96f2c59f7ae6
Signed-off-by: Saravana Kannan <skannan@codeaurora.org>
cpuidle: refactor out cpuidle_enter_state

Split the code to enter a state and update the stats into a helper
function, cpuidle_enter_state, and export it.  This function will
be called by the coupled state code to handle entering the safe
state and the final coupled state.

Change-Id: I7b18848370c61408de5960e3844e8faf426fa575
Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Reviewed-by: Kevin Hilman <khilman@ti.com>
Tested-by: Kevin Hilman <khilman@ti.com>
Signed-off-by: Colin Cross <ccross@android.com>
Reviewed-by: Rafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: Len Brown <len.brown@intel.com>
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
cpuidle: fix error handling in __cpuidle_register_device

Fix the error handling in __cpuidle_register_device to include
the missing list_del.  Move it to a label, which will simplify
the error handling when coupled states are added.

Change-Id: I1e0fcbf97e9542c3f5f7634ef9bfc126d518d51e
Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Reviewed-by: Kevin Hilman <khilman@ti.com>
Tested-by: Kevin Hilman <khilman@ti.com>
Signed-off-by: Colin Cross <ccross@android.com>
Reviewed-by: Rafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: Len Brown <len.brown@intel.com>
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
cpuidle: add support for states that affect multiple cpus

On some ARM SMP SoCs (OMAP4460, Tegra 2, and probably more), the
cpus cannot be independently powered down, either due to
sequencing restrictions (on Tegra 2, cpu 0 must be the last to
power down), or due to HW bugs (on OMAP4460, a cpu powering up
will corrupt the gic state unless the other cpu runs a work
around).  Each cpu has a power state that it can enter without
coordinating with the other cpu (usually Wait For Interrupt, or
WFI), and one or more "coupled" power states that affect blocks
shared between the cpus (L2 cache, interrupt controller, and
sometimes the whole SoC).  Entering a coupled power state must
be tightly controlled on both cpus.

The easiest solution to implementing coupled cpu power states is
to hotplug all but one cpu whenever possible, usually using a
cpufreq governor that looks at cpu load to determine when to
enable the secondary cpus.  This causes problems, as hotplug is an
expensive operation, so the number of hotplug transitions must be
minimized, leading to very slow response to loads, often on the
order of seconds.

This file implements an alternative solution, where each cpu will
wait in the WFI state until all cpus are ready to enter a coupled
state, at which point the coupled state function will be called
on all cpus at approximately the same time.

Once all cpus are ready to enter idle, they are woken by an smp
cross call.  At this point, there is a chance that one of the
cpus will find work to do, and choose not to enter idle.  A
final pass is needed to guarantee that all cpus will call the
power state enter function at the same time.  During this pass,
each cpu will increment the ready counter, and continue once the
ready counter matches the number of online coupled cpus.  If any
cpu exits idle, the other cpus will decrement their counter and
retry.

To use coupled cpuidle states, a cpuidle driver must:

   Set struct cpuidle_device.coupled_cpus to the mask of all
   coupled cpus, usually the same as cpu_possible_mask if all cpus
   are part of the same cluster.  The coupled_cpus mask must be
   set in the struct cpuidle_device for each cpu.

   Set struct cpuidle_device.safe_state to a state that is not a
   coupled state.  This is usually WFI.

   Set CPUIDLE_FLAG_COUPLED in struct cpuidle_state.flags for each
   state that affects multiple cpus.

   Provide a struct cpuidle_state.enter function for each state
   that affects multiple cpus.  This function is guaranteed to be
   called on all cpus at approximately the same time.  The driver
   should ensure that the cpus all abort together if any cpu tries
   to abort once the function is called.

update1:

cpuidle: coupled: fix count of online cpus

online_count was never incremented on boot, and was also counting
cpus that were not part of the coupled set.  Fix both issues by
introducting a new function that counts online coupled cpus, and
call it from register as well as the hotplug notifier.

update2:

cpuidle: coupled: fix decrementing ready count

cpuidle_coupled_set_not_ready sometimes refuses to decrement the
ready count in order to prevent a race condition.  This makes it
unsuitable for use when finished with idle.  Add a new function
cpuidle_coupled_set_done that decrements both the ready count and
waiting count, and call it after idle is complete.

Change-Id: If304c55c17c4aadd693f843d941fb0e6c3375ac0
Cc: Amit Kucheria <amit.kucheria@linaro.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Trinabh Gupta <g.trinabh@gmail.com>
Cc: Deepthi Dharwar <deepthi@linux.vnet.ibm.com>
Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Reviewed-by: Kevin Hilman <khilman@ti.com>
Tested-by: Kevin Hilman <khilman@ti.com>
Signed-off-by: Colin Cross <ccross@android.com>
Acked-by: Rafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: Len Brown <len.brown@intel.com>
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
cpuidle: coupled: add parallel barrier function

Adds cpuidle_coupled_parallel_barrier, which can be used by coupled
cpuidle state enter functions to handle resynchronization after
determining if any cpu needs to abort.  The normal use case will
be:

static bool abort_flag;
static atomic_t abort_barrier;

int arch_cpuidle_enter(struct cpuidle_device *dev, ...)
{
	if (arch_turn_off_irq_controller()) {
	   	/* returns an error if an irq is pending and would be lost
		   if idle continued and turned off power */
		abort_flag = true;
	}

	cpuidle_coupled_parallel_barrier(dev, &abort_barrier);

	if (abort_flag) {
	   	/* One of the cpus didn't turn off it's irq controller */
	   	arch_turn_on_irq_controller();
		return -EINTR;
	}

	/* continue with idle */
	...
}

This will cause all cpus to abort idle together if one of them needs
to abort.

Change-Id: I7be85033338d14e7089a8f5a1cfb761809984a0e
Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Reviewed-by: Kevin Hilman <khilman@ti.com>
Tested-by: Kevin Hilman <khilman@ti.com>
Signed-off-by: Colin Cross <ccross@android.com>
Signed-off-by: Len Brown <len.brown@intel.com>
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
cpuidle: coupled: fix sleeping while atomic in cpu notifier

The cpu hotplug notifier gets called in both atomic and non-atomic
contexts, it is not always safe to lock a mutex.  Filter out all events
except the six necessary ones, which are all sleepable, before taking
the mutex.

Change-Id: I3f5b09334e172d1d62620a5b9929942651f0620c
Signed-off-by: Colin Cross <ccross@android.com>
Reviewed-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>