arch: arm: lib: optimize memcpy for cortex-A15

The LDRD/STRD instruction is faster than LDM/STM on Cortex-A15.
Also optimized preload cache size for Cortex-A15.

Added USE_LDRDSTRD_OVER_LDMSTM to turn on LDRD/STRD optimization.
Added ARM_PLD_SIZE, default 32. Should set to 64 for Cortex-A15.

Bug 1185248

Signed-off-by: Chandler Zhang <chazhang@nvidia.com>
Reviewed-by: Krishna Reddy <vdumpa@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
arm: atomic.h: defined relaxed atomic_read

Defining relaxed version of atomic read as cpu_relaxed_read_atomic.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
arm: asm: relaxed.h: defined relaxed.h

Defined a new header file relaxed.h, which uses generic
definitions of some macros used by arm64 for improving
power efficiency.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-on: http://git-master/r/398766
(cherry picked from commit a96e59b1959f3ee216503b4f9df3cb75f7093ed6)
Reviewed-on: http://git-master/r/422211
Reviewed-by: Automatic_Commit_Validation_User
GVS: Gerrit_Virtual_Submit
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
sched: defining relaxed version of idle_cpu

Defining relaxed version of idle_cpu, which uses
macro cpu_relaxed_read_long, that will be used to
enhance power efficiency.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
asm-generic: relaxed.h: defined relaxed.h

Defined a new header file relaxed.h, which contains basic
macros which will be used for improving power efficiency for
arm64.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
GVS: Gerrit_Virtual_Submit
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
llist: defining relaxed version of llist_empty

Defining relaxed version of llist_empty as llist_empty_relaxed,
which will be used for power-optimization.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
asm-generic: processor.h: adding new header file

Defining a new header file and adding architecture
independent macros. Using these macros we are optimizing
power usage on ARM64.

bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
ARM: including asm-generic/processor.h

Here we are including asm-generic/processor.h
because it has defined some generic macros which
can be used by any architecture for optimaztion
of power usage.

bug 1440421

Change-Id: I39db944d348b3dad9235b76d8c7ea3867c0006ce
Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Reviewed-by: Peng Du <pdu@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
cpu: enhance power efficiency

Here we are trying to reduce power usage through the
use of macros cpu_relaxed_read and relaxed version of
idle_cpu().

Bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
seqlock: enhance power efficiency

Trying to improve the power efficiency in linux/seqlock.h,
using macros cpu_relaxed_read and cpu_read_relax.

Bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
sched: Trying to reduce power usage

Using macros cpu_relaxed_read, cpu_relaxed_read_long and
cpu_read_relax in core.c to improve power efficiency.

Bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Tested-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
hrtimer: enhance power efficiency

Defining relaxed version of hrtimer_callback_running(),
which will be used to improve power efficiency through
the use of macro cpu_relaxed_read_long.

Bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
smp: enhance power efficiency

Here we are trying to reduce power usage through the
use of macros cpu_relaxed_read_short and cpu_relaxed_read.

Bug 1440421

Signed-off-by: Sumit Singh <sumsingh@nvidia.com>
Reviewed-by: Sri Krishna Chowdary <schowdary@nvidia.com>
Reviewed-by: Alexander Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>

Conflicts:
	kernel/smp.c

Conflicts:
	kernel/smp.c
kernel/smp.c: use '|=' for csd_lock

csd_lock() uses assignment to data->flags rather than |=.  That is not
buggy at present because only one bit (CSD_FLAG_LOCK) is defined in
call_single_data.flags.

But it will become buggy if we later add another flag, so fix it now.

Signed-off-by: liguang <lig.fnst@cn.fujitsu.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
arch: arm: lib: optimize memcpy for cortex-A15

* Apply to memutils we ported from msm8960pro

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>

Conflicts:
	arch/arm/mach-msm/memutils/copy_from_user.S
	arch/arm/mach-msm/memutils/copy_template.S
	arch/arm/mach-msm/memutils/copy_to_user.S
	arch/arm/mach-msm/memutils/memcpy.S
ARM: cleanup undefined instruction entry code

We don't need to keep reloading the thread into into r10 - we can do
this once and keep the value cached in the register.  Also, schedule
some instructions better so that the pipeline doesn't stall after a
load in the neon code.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
ARM: mm: Remove unnecessary CMO in Cortex A15 startup

Cortex-A15 flush L2 cache after reset, there is no need to do this
in software, if L2 is already invalidated in bootloader and
cache is disabled. For secondary startup, there is no reason to
flush L2 as well.

This change assumes the setup code is always entered as the result
of CPU reset.

Signed-off-by: Bo Yan <byan@nvidia.com>
Reviewed-by: Mrutyunjay Sawant <msawant@nvidia.com>
Tested-by: Mrutyunjay Sawant <msawant@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
ARM: mm: Skip I-cache invalidate for Cortex-A15 boot

This is not required since cache is invalidated by HW in the reset
sequence. Bootloader is supposed to do the same before it hands
over control to kernel.

Signed-off-by: Bo Yan <byan@nvidia.com>
Reviewed-by: Seshendra Gadagottu <sgadagottu@nvidia.com>
Reviewed-by: Yu-Huan Hsu <yhsu@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>

Conflicts:
	arch/arm/mm/proc-v7.S

Conflicts:
	arch/arm/mm/proc-v7.S
ARM: remove unnecessary dcache_clean_area

Cortex-A9 has PIPT D-cache which do not require clean the cache
on creating page table.

Reviewed-by: Heechul Yun <hyun@nvidia.com>
Tested-by: Heechul Yun <hyun@nvidia.com>
Reviewed-by: Krishna Reddy <vdumpa@nvidia.com>
Reviewed-by: Scott Williams <scwilliams@nvidia.com>

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
ARM: LPAE: Invalidate the TLB for module addresses during translation fault

During the free_pgtables() call all user and modules/pkmap entries are
removed. If a translation fault for the modules/pkmap area occurs before
we switched away from the current pgd, do_translation_fault() would copy
the init_mm pud into the user pud.

There is a small window between pud_clear() and pmd_free_tlb() in
free_pmd_range() where the pud entry was cleared but the TLB has not
been invalidated yet and the CPU may have cached the original (valid)
pud entry in the TLB. A scenario like below would get stuck in
continuous prefetch abort:

1. Current process exiting. The modules pmd entries not populated
2. exit_mmap() -> ... -> pmd_free_tlb()
3. pud_clear() for the 1GB pud containing user stack and modules (no TLB
   invalidation yet)
4. Interrupt -> module interrupt routine
5. Level 2 (pmd) translation fault occurs when executing the module
   interrupt routine. The CPU previously cached (TLB) the old valid pud
   value for the modules area, so we don't get a level 1 translation
   fault
6. do_translation fault() copies the pud_k into the pud
7. Linux returns to the faulty instruction. Goes back to 5

At point 7, since the CPU still has the old pud value, it goes back to
point 5 and never gets out of this loop. With this patch, the stale pud
TLB entry is invalidated after point 6 and the subsequent prefetch abort
doesn't occur.

Reported-by: Tony Thompson <Anthony.Thompson@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>