ARM: mm: implement LoUIS API for cache maintenance ops

ARM v7 architecture introduced the concept of cache levels and related
control registers. New processors like A7 and A15 embed an L2 unified cache
controller that becomes part of the cache level hierarchy. Some operations in
the kernel like cpu_suspend and __cpu_disable do not require a flush of the
entire cache hierarchy to DRAM but just the cache levels belonging to the
Level of Unification Inner Shareable (LoUIS), which in most of ARM v7 systems
correspond to L1.

The current cache flushing API used in cpu_suspend and __cpu_disable,
flush_cache_all(), ends up flushing the whole cache hierarchy since for
v7 it cleans and invalidates all cache levels up to Level of Coherency
(LoC) which cripples system performance when used in hot paths like hotplug
and cpuidle.

Therefore a new kernel cache maintenance API must be added to cope with
latest ARM system requirements.

This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.

This function cleans and invalidates all data cache levels up to the
Level of Unification Inner Shareable (LoUIS) and invalidates the instruction
cache for processors that support it (> v7).

This patch also creates an alias of the cache LoUIS function to flush_kern_all
for all processor versions prior to v7, so that the current cache flushing
behaviour is unchanged for those processors.

v7 cache maintenance code implements a cache LoUIS function that cleans and
invalidates the D-cache up to LoUIS and invalidates the I-cache, according
to the new API.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@xxxxxx>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@xxxxxxx>

Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: mm: rename jump labels in v7_flush_dcache_all function

This patch renames jump labels in v7_flush_dcache_all in order to define
a specific flush cache levels entry point.

TODO: factor out the level flushing loop if considered worthwhile and
      define the input registers requirements.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@xxxxxxx>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: kernel: update cpu_suspend code to use cache LoUIS operations

In processors like A15/A7 L2 cache is unified and integrated within the
processor cache hierarchy, so that it is not considered an outer cache
anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
all cache levels up to Level of Coherency (LoC) that includes
the L2 unified cache.

When a single CPU is suspended (CPU idle) a complete L2 clean is not
required, so generic cpu_suspend code must clean the data cache using the
newly introduced cache LoUIS function.

The context and stack pointer (context pointer) are cleaned to main memory
using cache area functions that operate on MVA and guarantee that the data
is written back to main memory (perform cache cleaning up to the Point of
Coherency - PoC) so that the processor can fetch the context when the MMU
is off in the cpu_resume code path.

outer_cache management remains unchanged.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@xxxxxx>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@xxxxxxx>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: kernel: update __cpu_disable to use cache LoUIS maintenance API

When a CPU is hotplugged out caches that reside in its power domain
lose their contents and so must be cleaned to the next memory level.

Currently, __cpu_disable calls flush_cache_all() that for new generation
processor like A15/A7 ends up cleaning and invalidating all cache levels
up to Level of Coherency, which includes the unified L2.

This ends up being a waste of cycles since the L2 cache contents are not
lost on power down.

This patch updates __cpu_disable to use the new LoUIS API cache operations.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@xxxxxx>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@xxxxxxx>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: mm: update __v7_setup() to the new LoUIS cache maintenance API

The ARMv7 processor setup function __v7_setup() cleans and invalidates the
CPU cache before enabling MMU to start the CPU with a clean CPU local cache.

But on ARMv7 architectures like Cortex-[A15/A8], this code will end
up flushing the L2 caches(up to level of Coherency) which is undesirable
and expensive. The setup functions are used in the CPU hotplug scenario too
and hence flushing all cache levels should be avoided.

This patch replaces the cache flushing call with the newly introduced
v7 dcache LoUIS API where only cache levels up to LoUIS are cleaned and
invalidated when a processors executes __v7_setup which is the expected
behavior.

For processors like A9 and A5 where the L2 cache is an outer one the
behavior should be unchanged.

Signed-off-by: Santosh Shilimkar <santosh.shilimkar@xxxxxx>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@xxxxxxx>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: move VFP init to an earlier boot stage

In order to use the NEON unit in the kernel, we should
initialize it a bit earlier in the boot process so NEON users
that like to do a quick benchmark at load time (like the
xor_blocks or RAID-6 code) find the NEON/VFP unit already
enabled.

Replaced late_initcall() with core_initcall().

Signed-off-by: Ard Biesheuvel <ard.biesheuvel at linaro.org>
Acked-by: Nicolas Pitre <nico at linaro.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: be strict about FP exceptions in kernel mode

The support code in vfp_support_entry does not care whether the
exception that caused it to be invoked occurred in kernel mode or
in user mode. However, neither condition that could trigger this
exception (lazy restore and VFP bounce to support code) is
currently allowable in kernel mode.

In either case, print a message describing the condition before
letting the undefined instruction handler run its course and trigger
an oops.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel at linaro.org>
Acked-by: Nicolas Pitre <nico at linaro.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: add support for kernel mode NEON

In order to safely support the use of NEON instructions in
kernel mode, some precautions need to be taken:
- the userland context that may be present in the registers (even
  if the NEON/VFP is currently disabled) must be stored under the
  correct task (which may not be 'current' in the UP case),
- to avoid having to keep track of additional vfpstates for the
  kernel side, disallow the use of NEON in interrupt context
  and run with preemption disabled,
- after use, re-enable preemption and re-enable the lazy restore
  machinery by disabling the NEON/VFP unit.

This patch adds the functions kernel_neon_begin() and
kernel_neon_end() which take care of the above. It also adds
the Kconfig symbol KERNEL_MODE_NEON to enable it.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: crypto: add NEON accelerated XOR implementation

Add a source file xor-neon.c (which is really just the reference
C implementation passed through the GCC vectorizer) and hook it
up to the XOR framework.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Nicolas Pitre <nico@linaro.org>
ARM: 7835/2: fix modular build of xor_blocks() with NEON enabled

Commit 0195659 introduced a NEON accelerated version of the xor_blocks()
function, but it needs the changes in this patch to allow it to be built
as a module rather than statically into the kernel.

This patch creates a separate module xor-neon.ko which exports the NEON
inner xor_blocks() functions depended upon by the regular xor.ko if it
is built with CONFIG_KERNEL_MODE_NEON=y

Reported-by: Josh Boyer <jwboyer@fedoraproject.org>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
ARM: only allow kernel mode neon with AEABI

This prevents the linker erroring with:

arm-linux-ld: error: arch/arm/lib/xor-neon.o uses VFP instructions, whereas arch/arm/lib/built-in.o does not
arm-linux-ld: failed to merge target specific data of file arch/arm/lib/xor-neon.o

This is due to the non-neon files being marked as containing FPA data/
instructions (even though they do not) being mixed with files which
contain VFP, which is an incompatible floating point format.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
CHROMIUM: mm: Fix calculation of dirtyable memory

The system uses global_dirtyable_memory() to calculate
number of dirtyable pages/pages that can be allocated
to the page cache.  A bug causes an underflow thus making
the page count look like a big unsigned number.  This in turn
confuses the dirty writeback throttling to aggressively write
back pages as they become dirty (usually 1 page at a time).

Fix is to ensure there is no underflow while doing the math.

Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Signed-off-by: Puneet Kumar <puneetster@chromium.org>

BUG=chrome-os-partner:16011
TEST=Manual; boot kernel, powerwash, login with testaccount and
make sure no jank occurs on sync of applications

Change-Id: I614e7c3156e014f0f28a4ef9bdd8cb8a2cd07b2a
Reviewed-on: https://gerrit.chromium.org/gerrit/37612
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Olof Johansson <olofj@chromium.org>
Commit-Ready: Puneet Kumar <puneetster@chromium.org>
Reviewed-by: Puneet Kumar <puneetster@chromium.org>
Tested-by: Puneet Kumar <puneetster@chromium.org>
writeback: fix occasional slow sync(1)

In case when system contains no dirty pages, wakeup_flusher_threads() will
submit WB_SYNC_NONE writeback for 0 pages so wb_writeback() exits
immediately without doing anything, even though there are dirty inodes in
the system.  Thus sync(1) will write all the dirty inodes from a
WB_SYNC_ALL writeback pass which is slow.

Fix the problem by using get_nr_dirty_pages() in wakeup_flusher_threads()
instead of calculating number of dirty pages manually.  That function also
takes number of dirty inodes into account.

Signed-off-by: Jan Kara <jack@suse.cz>
Reported-by: Paul Taysom <taysom@chromium.org>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
cpufreq: intelliactive: initial coding and introduction!

intelliactive is based on Latest Google Interactive cpufreq governor from
Google.  The original Interactive provides smooth UI operations but suffers
slight battery drain due to its aggressive policies.  Intelliactive is an
improvement built on top of it.

It has the following enhancements added:
1. self-boost capability from input drivers (no need for PowerHAL assist)
2. two phase scheduling (idle/busy phases to prevent from jumping directly to
   max freq
3. Checks for offline cpus and short circuits some unnecessary checks to
   improve code execution paths
4. removed trace calls

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: cj360 <ayunker551@gmail.com>

Conflicts:
	drivers/cpufreq/Kconfig
	drivers/cpufreq/Makefile
	include/linux/cpufreq.h

Change-Id: I764ee783c722d477b7b62ee2109b871a8e21e883
cpufreq/cpufreq_intellidemand: intellidemand cpufreq governor (version 4.0)

Intellidemand is inspired by ondemand governor. Its main features are:

 - auto cpu hotplugging based on runnable thread average calculations
 - detection of browsing mode (long screen on activities with moderate run
   queue depth =< 45 )
 - limit max frequencies when in browsing mode to conserve battery for long
   duration tasks

Browsing State Determination:
 1- Browsing state is enabled as power-on default. Browsing state is disabled
    when run queue depth is > 45 (very high instantaneous spikes)

Browsing State Limit Max Freq Logic (sliding window algorithm):
 1- While Screen is ON > 3 min, and average load of the CPU is < 35%, then
    CPU Max frequencies will remain power-on default
 2- While Screen is On, if average CPU load > 35%, both CPUs max frequencies
    will be limited to a lower user configurable max value
 3- While Screen is On, After both CPUs' Max is limited to lowered max, when
    the average CPU load has dropped down to less than < 25%, then both CPUs
    Max freq will be restored to back to power-on default

Runnable Thread Auto-hotplug (fast hotplug / slow unplug algorithm):
 1- When average number of running thread is < threshold, 2nd cpu will be
    unplugged.
 2- When > threshold, 2nd cpu will be hotplugged.
 3- The logic applies a fast hotplug and slow unplug algorithm so 2nd core
    will stay enabled longer to help deal with performance spikes

changes from Ver 2.0 -> Ver 3.0:
  - remove GPU Busy dependency for browsing state detection (this will make
    the governor more portable to other platforms without specific GPU
    architecture knowledge)

dependencies:
  -- Qualcomm's run queue average logic (SOC independent logic)
  -- TI's runnable thread average logic (SOC independent logic)

forward ported to Linux 3.4 for use on S-800

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: cj360 <ayunker551@gmail.com>

Conflicts:
	drivers/cpufreq/Kconfig
	drivers/cpufreq/Makefile
	include/linux/cpufreq.h

Change-Id: If7f5328dc8e00b5490dd23be5fadf25d61563587
update cpufreq with @faux123's cpu turbo boost & intellidemand variables

Change-Id: Ia72dff2e267b9b283c79b1af3e0a8ee66395be24
Signed-off-by: cj360 <ayunker551@gmail.com>

update kconfig with @faux123's cpu turbo boost & intellidemand variables

Change-Id: I380953e9032bf3d78b2c58c806642f0306711195
Signed-off-by: cj360 <ayunker551@gmail.com>