ARM: bitops: prefetch the destination word for write prior to strex

The cost of changing a cacheline from shared to exclusive state can be
significant, especially when this is triggered by an exclusive store,
since it may result in having to retry the transaction.

This patch prefixes our atomic bitops implementation with prefetchw,
to try and grab the line in exclusive state from the start. The testop
macro is left alone, since the barrier semantics limit the usefulness
of prefetching data.

Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7907/1: lib: delay-loop: Add align directive to fix BogoMIPS calculation

Currently mx53 (CortexA8) running at 1GHz reports:
Calibrating delay loop... 663.55 BogoMIPS (lpj=3317760)

Tom Evans verified that alignments of 0x0 and 0x8 run the two instructions of __loop_delay in one clock cycle (1 clock/loop), while alignments of 0x4 and 0xc take 3 clocks to run the loop twice. (1.5 clock/loop)

The original object code looks like this:

00000010 <__loop_const_udelay>:
  10:	e3e01000 	mvn	r1, #0
  14:	e51f201c 	ldr	r2, [pc, #-28]	; 0 <__loop_udelay-0x8>
  18:	e5922000 	ldr	r2, [r2]
  1c:	e0800921 	add	r0, r0, r1, lsr #18
  20:	e1a00720 	lsr	r0, r0, #14
  24:	e0822b21 	add	r2, r2, r1, lsr #22
  28:	e1a02522 	lsr	r2, r2, #10
  2c:	e0000092 	mul	r0, r2, r0
  30:	e0800d21 	add	r0, r0, r1, lsr #26
  34:	e1b00320 	lsrs	r0, r0, #6
  38:	01a0f00e 	moveq	pc, lr

0000003c <__loop_delay>:
  3c:	e2500001 	subs	r0, r0, #1
  40:	8afffffe 	bhi	3c <__loop_delay>
  44:	e1a0f00e 	mov	pc, lr

After adding the 'align 3' directive to __loop_delay (align to 8 bytes):

00000010 <__loop_const_udelay>:
  10:	e3e01000 	mvn	r1, #0
  14:	e51f201c 	ldr	r2, [pc, #-28]	; 0 <__loop_udelay-0x8>
  18:	e5922000 	ldr	r2, [r2]
  1c:	e0800921 	add	r0, r0, r1, lsr #18
  20:	e1a00720 	lsr	r0, r0, #14
  24:	e0822b21 	add	r2, r2, r1, lsr #22
  28:	e1a02522 	lsr	r2, r2, #10
  2c:	e0000092 	mul	r0, r2, r0
  30:	e0800d21 	add	r0, r0, r1, lsr #26
  34:	e1b00320 	lsrs	r0, r0, #6
  38:	01a0f00e 	moveq	pc, lr
  3c:	e320f000 	nop	{0}

00000040 <__loop_delay>:
  40:	e2500001 	subs	r0, r0, #1
  44:	8afffffe 	bhi	40 <__loop_delay>
  48:	e1a0f00e 	mov	pc, lr
  4c:	e320f000 	nop	{0}

, which now reports:
Calibrating delay loop... 996.14 BogoMIPS (lpj=4980736)

Some more test results:

On mx31 (ARM1136) running at 532 MHz, before the patch:
Calibrating delay loop... 351.43 BogoMIPS (lpj=1757184)

On mx31 (ARM1136) running at 532 MHz after the patch:
Calibrating delay loop... 528.79 BogoMIPS (lpj=2643968)

Also tested on mx6 (CortexA9) and on mx27 (ARM926), which shows the same
BogoMIPS value before and after this patch.

Reported-by: Tom Evans <tom_usenet@optusnet.com.au>
Suggested-by: Tom Evans <tom_usenet@optusnet.com.au>
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7404/1: cmpxchg64: use atomic64 and local64 routines for cmpxchg64

The cmpxchg64 routines for ARMv6+ CPUs replicate inline assembly that
already exists for atomic64 operations. Furthermore, the cmpxchg64 code
uses the "memory" constraint in the clobber list rather than identifying
the region of memory that is actually modified.

This patch replaces the ARMv6+ cmpxchg64 code with macros that expand to
the atomic64_ and local64_ variants, casting the pointer parameter to
the appropriate container type.

Cc: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7720/1: ARM v6/v7 cmpxchg64 shouldn't clear upper 32 bits of the old/new value

The implementation of cmpxchg64() for the ARM v6 and v7 architecture
casts parameter 2 and 3 (the old and new 64bit values) to an unsigned
long before calling the atomic_cmpxchg64() function. This clears
the top 32 bits of the old and new values, resulting in the wrong
values being compare-exchanged. Luckily, this only appears to be used
for 64-bit sched_clock, which we don't (yet) have on ARM.

This bug was introduced by commit 3e0f5a15f500 ("ARM: 7404/1: cmpxchg64:
use atomic64 and local64 routines for cmpxchg64").

Cc: <stable@vger.kernel.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Jaccon Bastiaansen <jaccon.bastiaansen@gmail.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7852/1: cmpxchg: implement barrier-less cmpxchg64_local

Our cmpxchg64 macros are wrappers around atomic64_cmpxchg. Whilst this is
great for code re-use, there is a case for barrier-less cmpxchg where it
is known to be safe (for example cmpxchg64_local and cmpxchg-based
lockrefs).

This patch introduces a 64-bit cmpxchg implementation specifically
for the cmpxchg64_* macros, so that it can be later used by the lockref
code.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7853/1: cmpxchg: implement cmpxchg64_relaxed

This patch introduces cmpxchg64_relaxed for arm, which performs a 64-bit
cmpxchg operation without barrier semantics. cmpxchg64_local is updated
to use the new operation.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7984/1: prefetch: add prefetchw invocations for barriered atomics

After a bunch of benchmarking on the interaction between dmb and pldw,
it turns out that issuing the pldw *after* the dmb instruction can
give modest performance gains (~3% atomic_add_return improvement on a
dual A15).

This patch adds prefetchw invocations to our barriered atomic operations
including cmpxchg, test_and_xxx and futexes.

Change-Id: Id6ebfe82ff9578c27e2b745e2cce167200112606
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: franciscofranco <franciscofranco.1990@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
cpuidle: check for need_resched() after the idle entry trace event

trace_cpu_idle_rcuidle() can end up waking ksoftirqd under certain
conditions. Since there is no check for need_resched() following
this wake up, the cpu ends up going into an idle state despite there
being a runnable task on that runqueue. Furthermore, since the
resched flag is set, any sucessive waking up tasks on the idle cpu
will not result in an IPI being sent to that cpu. Consequently tasks
on that runqueue will starve until the cpu exits idle by itself. Fix
this problem by adding a need_resched() check after the idle entry
trace event.

A side effect of this change is that the idle trace entry and exit
event may be recorded for an extremely short duration even though
the cpu does not actually enter idle.

Change-Id: I370c39a1e39ecd829e44ac783b4888a00ab05eae
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: franciscofranco <franciscofranco.1990@gmail.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>
ARM: gic: Bring inline with upstream

Change-Id: Ic2cce8da16c81b9448e1f2ea8813ae05b48f1e31

ARM: gic: fix restore order of distributor registers in gic_cpu_restore()

Similar to gic_dist_restore(), the GIC config, priority and enable
registers should be restored in that order. Fix this in gic_cpu_restore()
which currently restores the enable registers before the configuration
registers.

Change-Id: Ib75640849bef81aa87a3b110fe1bd7ea4371d659
Signed-off-by: Osvaldo Banuelos <osvaldob@codeaurora.org>

ARM: GIC: fix GIC cpumask initialization

Punit Agrawal reports:
> I was trying to boot 3.8-rc5 on Realview EB 11MPCore using
> realview-smp_defconfig as a starting point but the kernel failed to
> progress past the log below (config attached).
>
> Pawel suggested I try reverting 384a290283f - "ARM: gic: use a private
> mapping for CPU target interfaces" that you've authored. With this
> commit reverted the kernel boots.
>
> I am not quite sure why the commit breaks 11MPCore but Pawel (cc'd)
> might be able to shed light on that.

Some early GIC implementations return zero for the first distributor
CPU routing register.  This means we can't rely on that telling us
which CPU interface we're connected to.  We know that these platforms
implement PPIs for IRQs 29-31 - but we shouldn't assume that these
will always be populated.

So, instead, scan for a non-zero CPU routing register in the first
32 IRQs and use that as our CPU mask.

Change-Id: I264edd7273353e300d7d1ef096709f9d87b9ec78
Reported-by: Punit Agrawal <punit.agrawal@arm.com>
Reviewed-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

irqchip: gic: Perform the gic_secondary_init() call via CPU notifier

All the calls to gic_secondary_init() pass 0 as the first argument.
Since this function is called on each CPU when starting, it can be done
in a platform-independent way via a CPU notifier registered by the GIC
code.

Change-Id: I729e09097230eb2f93fb730f1d58a279c5ef51f8
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Acked-by: Rob Herring <rob.herring@calxeda.com>
Acked-by: Simon Horman <horms+renesas@verge.net.au>
Tested-by: Simon Horman <horms+renesas@verge.net.au>
Acked-by: Srinidhi Kasagar <srinidhi.kasagar@stericsson.com>
Tested-by: Dinh Nguyen <dinguyen@altera.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
Tested-by: Marc Zyngier <marc.zyngier@arm.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Kukjin Kim <kgene.kim@samsung.com>
Cc: Sascha Hauer <kernel@pengutronix.de>
Cc: David Brown <davidb@codeaurora.org>
Cc: Bryan Huntsman <bryanh@codeaurora.org>
Cc: Tony Lindgren <tony@atomide.com>
Cc: Magnus Damm <magnus.damm@gmail.com>
Cc: Shiraz Hashim <shiraz.hashim@st.com>
Cc: Linus Walleij <linus.walleij@linaro.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Kukjin Kim <kgene.kim@samsung.com>
Cc: Barry Song <baohua.song@csr.com>
Signed-off-by: Chet Kener <Cl3Kener@gmail.com>