ARM: mm: implement LoUIS API for cache maintenance ops

ARM v7 architecture introduced the concept of cache levels and related
control registers. New processors like A7 and A15 embed an L2 unified cache
controller that becomes part of the cache level hierarchy. Some operations in
the kernel like cpu_suspend and __cpu_disable do not require a flush of the
entire cache hierarchy to DRAM but just the cache levels belonging to the
Level of Unification Inner Shareable (LoUIS), which in most of ARM v7 systems
correspond to L1.

The current cache flushing API used in cpu_suspend and __cpu_disable,
flush_cache_all(), ends up flushing the whole cache hierarchy since for
v7 it cleans and invalidates all cache levels up to Level of Coherency
(LoC) which cripples system performance when used in hot paths like hotplug
and cpuidle.

Therefore a new kernel cache maintenance API must be added to cope with
latest ARM system requirements.

This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.

This function cleans and invalidates all data cache levels up to the
Level of Unification Inner Shareable (LoUIS) and invalidates the instruction
cache for processors that support it (> v7).

This patch also creates an alias of the cache LoUIS function to flush_kern_all
for all processor versions prior to v7, so that the current cache flushing
behaviour is unchanged for those processors.

v7 cache maintenance code implements a cache LoUIS function that cleans and
invalidates the D-cache up to LoUIS and invalidates the I-cache, according
to the new API.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Reviewed-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Tested-by: Shawn Guo <shawn.guo@linaro.org>

Change-Id: I561c3b187b5b87b25de07ea82a4ef73a42761e27
Signed-off-by: Karthik Parsha <kparsha@codeaurora.org>
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
cpufreq: interactive: allow arbitrary speed / delay mappings

Accept a string of delays and speeds at which to apply the delay before
raising each step above hispeed. For example, "80000 1300000:200000
1500000:40000" means that the delay at or above 1GHz, until 1.3GHz is 80 msecs,
the delay until 1.5GHz is 200 msecs and the delay at or above 1.5GHz is 40
msecs when hispeed_freq is 1GHz.

[toddpoynor@google.com: add documentation]
Change-Id: Ifeebede8b1acbdd0a53e5c6916bccbf764dc854f
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Git-commit: e43dde3b6cec1e1d36a477713bcfb1dda070bef2
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: add io_is_busy interface

Previously the idle time returned from get_cpu_idle_time_us included the
iowait time. So the iowait time was always calculated as idle time.

But now the idle time returned from get_cpu_idle_time_us does not include
the iowait time anymore because of below commit which cause the iowait time
always calculated as busy time:
    6beea0c nohz: Fix update_ts_time_stat idle accounting

Add the io_is_busy interface, as does the ondemand governor, and let the user
configure the iowait time as busy or idle through the io_is_busy sysfs
interface.

By default, io_is_busy is disabled.

[toddpoynor@google.com: minor updates]
Change-Id: If7d70ff864c43bc9c8d7fd7cfc66f930d339f9b4
Signed-off-by: Lianwei Wang <lian-wei.wang@motorola.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Git-commit: fbcad7234e0aef2725d9f33ac5a0c3a9cc61a9ac
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: fix crash on error paths in get_tokenized_data

Use separate variable for error code, free proper pointer.

Change-Id: Ia83cccb195997789ac6afbf5b8761f7b278196d6
Reported-by: Arve Hjønnevåg <arve@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Git-commit: cad1684410541be5363d4af9ae8f3f35683a4dd4
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: base above_hispeed_delay on target freq, not current

Time to wait should be based on the intended target speed, not the
actual speed (which may be held high by another CPU).

Change-Id: Ifc5bb55d06adddb9a02af90af05398a78f282272
Reported-by: Arve Hjønnevåg <arve@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Git-commit: c4483b5550079c46dbc68005920a14f371d3f4eb
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: fix uninitialized spinlock

Add missing spinlock init

Backtrace:
[<c0011ce4>] (dump_backtrace+0x0/0x10c) from [<c0662a68>] (dump_stack+0x18/0x1c)
 r6:00000032 r5:c0bd09ec r4:e6848000 r3:00000000
[<c0662a50>] (dump_stack+0x0/0x1c) from [<c06670b0>] (spin_dump+0x80/0x94)
[<c0667030>] (spin_dump+0x0/0x94) from [<c06670f0>] (spin_bug+0x2c/0x30)
 r5:c08f91fc r4:c0bd09ec
[<c06670c4>] (spin_bug+0x0/0x30) from [<c0245f74>] (do_raw_spin_unlock+0x88/0xcc)
 r5:e547bac0 r4:c0bd09ec
[<c0245eec>] (do_raw_spin_unlock+0x0/0xcc) from [<c066c9cc>] (_raw_spin_unlock_irqrestore+0x14/0x40)
 r5:e547bac0 r4:60000013
[<c066c9b8>] (_raw_spin_unlock_irqrestore+0x0/0x40) from [<c044b884>] (store_above_hispeed_delay+0x6c/0x80)
 r4:c0b4cf78 r3:00000007
[<c044b818>] (store_above_hispeed_delay+0x0/0x80) from [<c0235d24>] (kobj_attr_store+0x1c/0x28)
 r7:e68ff000 r6:00000032 r5:e58137c0 r4:e61cde80
[<c0235d08>] (kobj_attr_store+0x0/0x28) from [<c0156b78>] (sysfs_write_file+0x104/0x184)
[<c0156a74>] (sysfs_write_file+0x0/0x184) from [<c0100680>] (vfs_write+0xb0/0x140)
[<c01005d0>] (vfs_write+0x0/0x140) from [<c0100900>] (sys_write+0x44/0x70)
 r8:00000000 r7:00000004 r6:00000032 r5:bee43c90 r4:e5600300
[<c01008bc>] (sys_write+0x0/0x70) from [<c000e400>] (ret_fast_syscall+0x0/0x30)
 r9:e6842000 r8:c000e584 r6:00000032 r5:bee43c90 r4:00000009

Change-Id: I80a1e0b3fecb24adba501ff44f568479deeff7fa
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Git-commit: 2e745d359ffaa9f4acc4b00950d14322ff364919
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: handle errors from cpufreq_frequency_table_target

Add checks for error return from cpufreq_frequency_table_target, and be
less noisy on the existing call with an error check.  CPU hotplug and
system shutdown may cause this call to return -EINVAL.

Bug: 8613560
Change-Id: Id78d8829920462c0db1c7e14e717d91740d6cb44
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Git-commit: 952c6d49444061849e2e254b0d701ebb1664f4b3
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: reduce chance of zero time delta on load eval

Reschedule load sampling timer after timestamp of sample start taken,
hold spinlock across entire sequence to avoid preemption.  Avoid the
WARN for zero time delta in the load sampling timer function.

Change-Id: Idc10a756f09141decb6df92669521a1ebf0dbc10
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Git-commit: 20075d8e9d42aac6e248c7cbe6a2f8a0a00d4ba4
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: avoid underflow on active time calculation

Check for idle time delta less than elapsed time delta, avoid
underflow computing active time.

Change-Id: I3e4c6ef1ad794eec49ed379c0c50fa727fd6ad28
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Git-commit: 3022f93a1aff25b3c9174ac659d7e4a143892267
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: fix race on cpufreq TRANSITION notifier

The cpufreq TRANSTION notifier callback does not check the
governor_enabled state on affected CPUS, which will case
kernel panic in update_load because the policy object maybe
NULL or invalid when governor_enabled is false.

Change-Id: Ie0f1718124f61e2f9b5da57abc6981ada5b83908
Signed-off-by: Lianwei Wang <a22439@motorola.com>
Git-commit: 44011f4524e86ca5a5e90bb941499cbe37a21987
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: resched timer if max freq raised

When the policy max freq is raised, and before the timer is
rescheduled in idle callback, the cpu freq may stuck at a
lower freq.

The target_freq shall be updated too, else on a high load
situation, the new_freq is always equal to target_freq and
which will cause freq stuck at a lower freq too.

Reschedule the timer on gov limits callback.

Change-Id: I6c187001ab43e859731429b64f75a74eebc37a24
Signed-off-by: Lianwei Wang <a22439@motorola.com>
Git-commit: 0edc2b4c9c30a695d9500d3204acdf5f3ddfa027
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: fix show_target_loads and show_above_hispeed_delay

Remove a trailing whitespace from target_loads and above_hispeed_delay. Problem
happens when user-space program tried to restore parameters that saved before
changing parameters. In this case was returned error(EINVAL).

Change-Id: I5a74e3824602cd6f2b74651adda5ec1b627e61e9
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Git-commit: e116c66cdee9080bf8ecb35c2179d79bd00c3043
Git-repo: https://android.googlesource.com/kernel/common/
Signed-off-by: Dilip Gudlur <dgudlur@codeaurora.org>
cpufreq: interactive: Add a sampling_down_factor for max frequencies

Change min_sample_time to 100ms when running at max

Change-Id: Ia7e35b0625bedf20e7ef3a1f52e5828ffbfed93e
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: Allow 1 ms error in above_hispeed_delay comparisons

Allow for an error of 1 ms while taking into account
above_hispeed_delay for a frequency

Change-Id: I744e44387152e4efb5978df4f2b9533bf79d4582
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: Reset floor_validate_time if busy at max for 100ms

When the interactive governor selects to run at max frequency it doesn't
re-schedule the timer until it hits an idle. This change checks if the CPU
has been continuously busy for last 100ms on hitting an idle start. If yes,
then floor_validate_time is reset so that the CPU stays at max frequency
for at least another 100 ms before stepping down.
This is an important feature for detecting CPU intensive workloads which
require high frequencies for achieving better performance.

Change-Id: I7d48ffbc3d50a80af9be3bf94667ee3d0120b763
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: sync freq feature for interactive governor

1) Add load info to cpufreq_interactive_cpuinfo
2) If load on any other online cpu exceeds sync_freq_load_threshold,
   do not allow the frequency to drop below sync_freq

Change-Id: I3617e10f87b85178914a18bcf04ac2a31a4f1ec1
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
(cherry picked from commit c26e25944543c61b34b1830938bb6f353a2580ac)
cpufreq: interactive: Remove trace event from idle_start handler

Removed the trace_cpufreq_interactive_idle_start.
Also fix a crash resulting from accessing NULL policy before taking
the pcpu->enable_sem lock. The policy can be NULL if the core is
hotplugged out before the enable_sem lock is taken.

Change-Id: I7e2809cc016b3b383a44cdf3c697013e2d2b5417
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
(cherry picked from commit 3b7992b90216c74a9ae9ce0df90bc446687a2313)
timekeeping: Add missing update call in timekeeping_resume()

The leap second rework unearthed another issue of inconsistent data.

On timekeeping_resume() the timekeeper data is updated, but nothing
calls timekeeping_update(), so now the update code in the timer
interrupt sees stale values.

This has been the case before those changes, but then the timer
interrupt was using stale data as well so this went unnoticed for quite
some time.

Add the missing update call, so all the data is consistent everywhere.

CRs-fixed: 536881
Change-Id: I5f7d5219749a3f6330d074ef8866533ebda3bfca
Reported-by: Andreas Schwab <schwab@linux-m68k.org>
Reported-and-tested-by: "Rafael J. Wysocki" <rjw@sisk.pl>
Reported-and-tested-by: Martin Steigerwald <Martin@lichtvoll.de>
Cc: LKML <linux-kernel@vger.kernel.org>
Cc: Linux PM list <linux-pm@vger.kernel.org>
Cc: John Stultz <johnstul@us.ibm.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
Cc: Prarit Bhargava <prarit@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: John Stultz <johnstul@us.ibm.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 3e997130bd2e8c6f5aaa49d6e3161d4d29b43ab0
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
timekeeping: utilize the suspend-nonstop clocksource to count suspended time

There are some new processors whose TSC clocksource won't stop during
suspend. Currently, after system resumes, kernel will use persistent
clock or RTC to compensate the sleep time, but with these nonstop
clocksources, we could skip the special compensation from external
sources, and just use current clocksource for time recounting.

This can solve some time drift bugs caused by some not-so-accurate or
error-prone RTC devices.

The current way to count suspended time is first try to use the persistent
clock, and then try the RTC if persistent clock can't be used. This
patch will change the trying order to:
	suspend-nonstop clocksource -> persistent clock -> RTC

When counting the sleep time with nonstop clocksource, use an accurate way
suggested by Jason Gunthorpe to cover very large delta cycles.

CRs-fixed: 536881
Change-Id: I59c21b1ce5ae81cebc8cf3e817d9b3f951302a3d
Signed-off-by: Feng Tang <feng.tang@intel.com>
[jstultz: Small optimization, avoiding re-reading the clocksource]
Signed-off-by: John Stultz <john.stultz@linaro.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: e445cf1c4257cc0238d72e4129eb4739f46fd3de)
[sboyd: Took tk and clock local variables]
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
Kconfig: Enable config to allow vmalloc savings

Select the config to reclaim the vmalloc space in lowmem
and disable the map around the hole config, where selected,
as both the configs cannot be enabled together

CRs-Fixed: 568470
Change-Id: I9dd3e0123e4fe3e33554cdbdf5a82e344c230f47
Signed-off-by: Neeti Desai <neetid@codeaurora.org>