netfilter: x_tables: fix ordering of jumpstack allocation and table update

During kernel stability testing on an SMP ARMv7 system, Yalin Wang
reported the following panic from the netfilter code:

  1fe0: 0000001c 5e2d3b10 4007e779 4009e110 60000010 00000032 ff565656 ff545454
  [<c06c48dc>] (ipt_do_table+0x448/0x584) from [<c0655ef0>] (nf_iterate+0x48/0x7c)
  [<c0655ef0>] (nf_iterate+0x48/0x7c) from [<c0655f7c>] (nf_hook_slow+0x58/0x104)
  [<c0655f7c>] (nf_hook_slow+0x58/0x104) from [<c0683bbc>] (ip_local_deliver+0x88/0xa8)
  [<c0683bbc>] (ip_local_deliver+0x88/0xa8) from [<c0683718>] (ip_rcv_finish+0x418/0x43c)
  [<c0683718>] (ip_rcv_finish+0x418/0x43c) from [<c062b1c4>] (__netif_receive_skb+0x4cc/0x598)
  [<c062b1c4>] (__netif_receive_skb+0x4cc/0x598) from [<c062b314>] (process_backlog+0x84/0x158)
  [<c062b314>] (process_backlog+0x84/0x158) from [<c062de84>] (net_rx_action+0x70/0x1dc)
  [<c062de84>] (net_rx_action+0x70/0x1dc) from [<c0088230>] (__do_softirq+0x11c/0x27c)
  [<c0088230>] (__do_softirq+0x11c/0x27c) from [<c008857c>] (do_softirq+0x44/0x50)
  [<c008857c>] (do_softirq+0x44/0x50) from [<c0088614>] (local_bh_enable_ip+0x8c/0xd0)
  [<c0088614>] (local_bh_enable_ip+0x8c/0xd0) from [<c06b0330>] (inet_stream_connect+0x164/0x298)
  [<c06b0330>] (inet_stream_connect+0x164/0x298) from [<c061d68c>] (sys_connect+0x88/0xc8)
  [<c061d68c>] (sys_connect+0x88/0xc8) from [<c000e340>] (ret_fast_syscall+0x0/0x30)
  Code: 2a000021 e59d2028 e59de01c e59f011c (e7824103)
  ---[ end trace da227214a82491bd ]---
  Kernel panic - not syncing: Fatal exception in interrupt

This comes about because CPU1 is executing xt_replace_table in response
to a setsockopt syscall, resulting in:

	ret = xt_jumpstack_alloc(newinfo);
		--> newinfo->jumpstack = kzalloc(size, GFP_KERNEL);

	[...]

	table->private = newinfo;
	newinfo->initial_entries = private->initial_entries;

Meanwhile, CPU0 is handling the network receive path and ends up in
ipt_do_table, resulting in:

	private = table->private;

	[...]

	jumpstack  = (struct ipt_entry **)private->jumpstack[cpu];

On weakly ordered memory architectures, the writes to table->private
and newinfo->jumpstack from CPU1 can be observed out of order by CPU0.
Furthermore, on architectures which don't respect ordering of address
dependencies (i.e. Alpha), the reads from CPU0 can also be re-ordered.

This patch adds an smp_wmb() before the assignment to table->private
(which is essentially publishing newinfo) to ensure that all writes to
newinfo will be observed before plugging it into the table structure.
A dependent-read barrier is also added on the consumer sides, to ensure
the same ordering requirements are also respected there.

CRs-Fixed: 591149

Change-Id: Ica3ec69cd6e2f77945a1a7303f92887feecf42e9
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reported-by: Wang, Yalin <Yalin.Wang@sonymobile.com>
Tested-by: Wang, Yalin <Yalin.Wang@sonymobile.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Git-commit: b416c144f46af1a30ddfa4e4319a8f077381ad63
Git-repo: https://www.codeaurora.org/cgit/quic/la/kernel/msm/common/
Signed-off-by: Vijaya Kumar T M <vtmuni@codeaurora.org>
[ARM: GENERIC] *squashing updates*

ARM: user generic termios.h

ARM: 7493/1: use generic unaligned.h

      This moves ARM over to the asm-generic/unaligned.h header. This has the
      benefit of better code generated especially for ARMv7 on gcc 4.7+
      compilers.

      As Arnd Bergmann, points out: The asm-generic version uses the "struct"
      version for native-endian unaligned access and the "byteshift" version
      for the opposite endianess. The current ARM version however uses the
      "byteshift" implementation for both.

      Thanks to Nicolas Pitre for the excellent analysis:

      Test case:

      int foo (int *x) { return get_unaligned(x); }
      long long bar (long long *x) { return get_unaligned(x); }

      With the current ARM version:

      foo:
	      ldrb	r3, [r0, #2]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 2B], MEM[(const u8 *)x_1(D) + 2B]
	      ldrb	r1, [r0, #1]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 1B], MEM[(const u8 *)x_1(D) + 1B]
	      ldrb	r2, [r0, #0]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D)], MEM[(const u8 *)x_1(D)]
	      mov	r3, r3, asl #16	@ tmp154, MEM[(const u8 *)x_1(D) + 2B],
	      ldrb	r0, [r0, #3]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 3B], MEM[(const u8 *)x_1(D) + 3B]
	      orr	r3, r3, r1, asl #8	@, tmp155, tmp154, MEM[(const u8 *)x_1(D) + 1B],
	      orr	r3, r3, r2	@ tmp157, tmp155, MEM[(const u8 *)x_1(D)]
	      orr	r0, r3, r0, asl #24	@,, tmp157, MEM[(const u8 *)x_1(D) + 3B],
	      bx	lr	@

      bar:
	      stmfd	sp!, {r4, r5, r6, r7}	@,
	      mov	r2, #0	@ tmp184,
	      ldrb	r5, [r0, #6]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 6B], MEM[(const u8 *)x_1(D) + 6B]
	      ldrb	r4, [r0, #5]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 5B], MEM[(const u8 *)x_1(D) + 5B]
	      ldrb	ip, [r0, #2]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 2B], MEM[(const u8 *)x_1(D) + 2B]
	      ldrb	r1, [r0, #4]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 4B], MEM[(const u8 *)x_1(D) + 4B]
	      mov	r5, r5, asl #16	@ tmp175, MEM[(const u8 *)x_1(D) + 6B],
	      ldrb	r7, [r0, #1]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 1B], MEM[(const u8 *)x_1(D) + 1B]
	      orr	r5, r5, r4, asl #8	@, tmp176, tmp175, MEM[(const u8 *)x_1(D) + 5B],
	      ldrb	r6, [r0, #7]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 7B], MEM[(const u8 *)x_1(D) + 7B]
	      orr	r5, r5, r1	@ tmp178, tmp176, MEM[(const u8 *)x_1(D) + 4B]
	      ldrb	r4, [r0, #0]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D)], MEM[(const u8 *)x_1(D)]
	      mov	ip, ip, asl #16	@ tmp188, MEM[(const u8 *)x_1(D) + 2B],
	      ldrb	r1, [r0, #3]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 3B], MEM[(const u8 *)x_1(D) + 3B]
	      orr	ip, ip, r7, asl #8	@, tmp189, tmp188, MEM[(const u8 *)x_1(D) + 1B],
	      orr	r3, r5, r6, asl #24	@,, tmp178, MEM[(const u8 *)x_1(D) + 7B],
	      orr	ip, ip, r4	@ tmp191, tmp189, MEM[(const u8 *)x_1(D)]
	      orr	ip, ip, r1, asl #24	@, tmp194, tmp191, MEM[(const u8 *)x_1(D) + 3B],
	      mov	r1, r3	@,
	      orr	r0, r2, ip	@ tmp171, tmp184, tmp194
	      ldmfd	sp!, {r4, r5, r6, r7}
	      bx	lr

      In both cases the code is slightly suboptimal.  One may wonder why
      wasting r2 with the constant 0 in the second case for example.  And all
      the mov's could be folded in subsequent orr's, etc.

      Now with the asm-generic version:

      foo:
	      ldr	r0, [r0, #0]	@ unaligned	@,* x
	      bx	lr	@

      bar:
	      mov	r3, r0	@ x, x
	      ldr	r0, [r0, #0]	@ unaligned	@,* x
	      ldr	r1, [r3, #4]	@ unaligned	@,
	      bx	lr	@

      This is way better of course, but only because this was compiled for
      ARMv7. In this case the compiler knows that the hardware can do
      unaligned word access.  This isn't that obvious for foo(), but if we
      remove the get_unaligned() from bar as follows:

      long long bar (long long *x) {return *x; }

      then the resulting code is:

      bar:
	      ldmia	r0, {r0, r1}	@ x,,
	      bx	lr	@

      So this proves that the presumed aligned vs unaligned cases does have
      influence on the instructions the compiler may use and that the above
      unaligned code results are not just an accident.

      Still... this isn't fully conclusive without at least looking at the
      resulting assembly fron a pre ARMv6 compilation.  Let's see with an
      ARMv5 target:

      foo:
	      ldrb	r3, [r0, #0]	@ zero_extendqisi2	@ tmp139,* x
	      ldrb	r1, [r0, #1]	@ zero_extendqisi2	@ tmp140,
	      ldrb	r2, [r0, #2]	@ zero_extendqisi2	@ tmp143,
	      ldrb	r0, [r0, #3]	@ zero_extendqisi2	@ tmp146,
	      orr	r3, r3, r1, asl #8	@, tmp142, tmp139, tmp140,
	      orr	r3, r3, r2, asl #16	@, tmp145, tmp142, tmp143,
	      orr	r0, r3, r0, asl #24	@,, tmp145, tmp146,
	      bx	lr	@

      bar:
	      stmfd	sp!, {r4, r5, r6, r7}	@,
	      ldrb	r2, [r0, #0]	@ zero_extendqisi2	@ tmp139,* x
	      ldrb	r7, [r0, #1]	@ zero_extendqisi2	@ tmp140,
	      ldrb	r3, [r0, #4]	@ zero_extendqisi2	@ tmp149,
	      ldrb	r6, [r0, #5]	@ zero_extendqisi2	@ tmp150,
	      ldrb	r5, [r0, #2]	@ zero_extendqisi2	@ tmp143,
	      ldrb	r4, [r0, #6]	@ zero_extendqisi2	@ tmp153,
	      ldrb	r1, [r0, #7]	@ zero_extendqisi2	@ tmp156,
	      ldrb	ip, [r0, #3]	@ zero_extendqisi2	@ tmp146,
	      orr	r2, r2, r7, asl #8	@, tmp142, tmp139, tmp140,
	      orr	r3, r3, r6, asl #8	@, tmp152, tmp149, tmp150,
	      orr	r2, r2, r5, asl #16	@, tmp145, tmp142, tmp143,
	      orr	r3, r3, r4, asl #16	@, tmp155, tmp152, tmp153,
	      orr	r0, r2, ip, asl #24	@,, tmp145, tmp146,
	      orr	r1, r3, r1, asl #24	@,, tmp155, tmp156,
	      ldmfd	sp!, {r4, r5, r6, r7}
	      bx	lr

      Compared to the initial results, this is really nicely optimized and I
      couldn't do much better if I were to hand code it myself.

      Signed-off-by: Rob Herring <rob.herring@calxeda.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Tested-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
      Reviewed-by: Arnd Bergmann <arnd@arndb.de>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

arm: Add XN bit to pgrprot_stronglyordered

      pgprot_stronglyordered sets page attributes to the strongly
      ordered memory type. This type is often used for regions of memory which
      cannot be speculated into. Strongly ordered memory by itself prevents
      speculative data accesses but it will not prevent speculative instruction
      accesses. Since strongly ordered memory will never be used for
      instructions, set the XN bit as well to prevent speculative instruction
      fethces.

      Bug: 10750953
      Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
      Signed-off-by: Mekala Natarajan <mekalan@codeaurora.org>

ARM: 7495/1: mutex: use generic atomic_dec-based implementation for ARMv6+

      Commit a76d7bd96d65 ("ARM: 7467/1: mutex: use generic xchg-based
      implementation for ARMv6+") removed the barrier-less, ARM-specific
      mutex implementation in favour of the generic xchg-based code.

      Since then, a bug was uncovered in the xchg code when running on SMP
      platforms, due to interactions between the locking paths and the
      MUTEX_SPIN_ON_OWNER code. This was fixed in 0bce9c46bf3b ("mutex: place
      lock in contended state after fastpath_lock failure"), however, the
      atomic_dec-based mutex algorithm is now marginally more efficient for
      ARM (~0.5% improvement in hackbench scores on dual A15).

      This patch moves ARMv6+ platforms to the atomic_dec-based mutex code.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7587/1: implement optimized percpu variable access

      Use the previously unused TPIDRPRW register to store percpu offsets.
      TPIDRPRW is only accessible in PL1, so it can only be used in the kernel.

      This replaces 2 loads with a mrc instruction for each percpu variable
      access. With hackbench, the performance improvement is 1.4% on Cortex-A9
      (highbank). Taking an average of 30 runs of "hackbench -l 1000" yields:

      Before: 6.2191
      After: 6.1348

      Will Deacon reported similar delta on v6 with 11MPCore.

      The asm "memory clobber" are needed here to ensure the percpu offset
      gets reloaded. Testing by Will found that this would not happen in
      __schedule() which is a bit of a special case as preemption is disabled
      but the execution can move cores.

      Signed-off-by: Rob Herring <rob.herring@calxeda.com>
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7747/1: pcpu: ensure __my_cpu_offset cannot be re-ordered across barrier()

      __my_cpu_offset is non-volatile, since we want its value to be cached
      when we access several per-cpu variables in a row with preemption
      disabled. This means that we rely on preempt_{en,dis}able to hazard
      with the operation via the barrier() macro, so that we can't end up
      migrating CPUs without reloading the per-cpu offset.

      Unfortunately, GCC doesn't treat a "memory" clobber on a non-volatile
      asm block as a side-effect, and will happily re-order it before other
      memory clobbers (including those in prempt_disable()) and cache the
      value. This has been observed to break the cmpxchg logic in the slub
      allocator, leading to livelock in kmem_cache_alloc in mainline kernels.

      This patch adds a dummy memory input operand to __my_cpu_offset,
      forcing it to be ordered with respect to the barrier() macro.

      Cc: <stable@vger.kernel.org>
      Cc: Rob Herring <rob.herring@calxeda.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: smp: flush L1 cache in cpu_die()

      Flush the L1 cache for the CPU which is going down in cpu_die() so
      that we don't end up with all platforms doing this.  This ensures
      that any cache lines we own are pushed out before the cache becomes
      inaccessible.

      We may end up subsequently creating some dirty cache lines - for
      example, with the complete() call, but this update must become
      visible to other CPUs before __cpu_die() can proceed.  Subsequent
      accesses from the platforms cpu_die() function should _not_ matter.

      Also place a mb() after the complete() call to ensure that this is
      visible to other CPUs.

      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7590/1: /proc/interrupts: limit the display of IPIs to online CPUs only

      This is what is done for the regular interrupts in kernel/irqs/proc.c
      already, before calling arch_show_interrupts().  Not doing so for the
      IPIs causes the column headers not to match with the content whenever
      some CPUs are offline.

      Signed-off-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
ARM: 7822/1: add workaround for ambiguous C99 stdint.h types

The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
unambiguous on ARM as you would expect. For the types below, there is a
difference on ARM between GCC built for bare metal ARM, GCC built for glibc
and the kernel itself, which results in build errors if you try to build with
-ffreestanding and include 'stdint.h' (such as when you include 'arm_neon.h'
in order to use NEON intrinsics)

As the typedefs for these types in 'stdint.h' are based on builtin defines
supplied by GCC, we can tweak these to align with the kernel's idea of those
types, so 'linux/types.h' and 'stdint.h' can be safely included from the same
source file (provided that -ffreestanding is used).

                   int32_t         uint32_t               uintptr_t
bare metal GCC     long            unsigned long          unsigned int
glibc GCC          int             unsigned int           unsigned int
kernel             int             unsigned int           unsigned long

Acked by: Dave Martin <dave.martin@arm.com>

Acked-by: Nicolas Pitre <nico@linaro.org>
Acked-by: Mikael Pettersson <mikpe@it.uu.se>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
mm: per-thread vma caching

This patch is a continuation of efforts trying to optimize find_vma(),
avoiding potentially expensive rbtree walks to locate a vma upon faults.
The original approach (https://lkml.org/lkml/2013/11/1/410), where the
largest vma was also cached, ended up being too specific and random,
thus further comparison with other approaches were needed.  There are
two things to consider when dealing with this, the cache hit rate and
the latency of find_vma().  Improving the hit-rate does not necessarily
translate in finding the vma any faster, as the overhead of any fancy
caching schemes can be too high to consider.

We currently cache the last used vma for the whole address space, which
provides a nice optimization, reducing the total cycles in find_vma() by
up to 250%, for workloads with good locality.  On the other hand, this
simple scheme is pretty much useless for workloads with poor locality.
Analyzing ebizzy runs shows that, no matter how many threads are
running, the mmap_cache hit rate is less than 2%, and in many situations
below 1%.

The proposed approach is to replace this scheme with a small per-thread
cache, maximizing hit rates at a very low maintenance cost.
Invalidations are performed by simply bumping up a 32-bit sequence
number.  The only expensive operation is in the rare case of a seq
number overflow, where all caches that share the same address space are
flushed.  Upon a miss, the proposed replacement policy is based on the
page number that contains the virtual address in question.  Concretely,
the following results are seen on an 80 core, 8 socket x86-64 box:

1) System bootup: Most programs are single threaded, so the per-thread
   scheme does improve ~50% hit rate by just adding a few more slots to
   the cache.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 50.61%   | 19.90            |
| patched        | 73.45%   | 13.58            |
+----------------+----------+------------------+

2) Kernel build: This one is already pretty good with the current
   approach as we're dealing with good locality.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 75.28%   | 11.03            |
| patched        | 88.09%   | 9.31             |
+----------------+----------+------------------+

3) Oracle 11g Data Mining (4k pages): Similar to the kernel build workload.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 70.66%   | 17.14            |
| patched        | 91.15%   | 12.57            |
+----------------+----------+------------------+

4) Ebizzy: There's a fair amount of variation from run to run, but this
   approach always shows nearly perfect hit rates, while baseline is just
   about non-existent.  The amounts of cycles can fluctuate between
   anywhere from ~60 to ~116 for the baseline scheme, but this approach
   reduces it considerably.  For instance, with 80 threads:

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 1.06%    | 91.54            |
| patched        | 99.97%   | 14.18            |
+----------------+----------+------------------+

[akpm@linux-foundation.org: fix nommu build, per Davidlohr]
[akpm@linux-foundation.org: document vmacache_valid() logic]
[akpm@linux-foundation.org: attempt to untangle header files]
[akpm@linux-foundation.org: add vmacache_find() BUG_ON]
[hughd@google.com: add vmacache_valid_mm() (from Oleg)]
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: adjust and enhance comments]
Signed-off-by: Davidlohr Bueso <davidlohr@hp.com>
Reviewed-by: Rik van Riel <riel@redhat.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Michel Lespinasse <walken@google.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Tested-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[ARM: MM] implement LoUIS API for cache maintenance ops

ARM: mm: implement LoUIS API for cache maintenance ops

      ARM v7 architecture introduced the concept of cache levels and related
      control registers. New processors like A7 and A15 embed an L2 unified cache
      controller that becomes part of the cache level hierarchy. Some operations in
      the kernel like cpu_suspend and __cpu_disable do not require a flush of the
      entire cache hierarchy to DRAM but just the cache levels belonging to the
      Level of Unification Inner Shareable (LoUIS), which in most of ARM v7 systems
      correspond to L1.

      The current cache flushing API used in cpu_suspend and __cpu_disable,
      flush_cache_all(), ends up flushing the whole cache hierarchy since for
      v7 it cleans and invalidates all cache levels up to Level of Coherency
      (LoC) which cripples system performance when used in hot paths like hotplug
      and cpuidle.

      Therefore a new kernel cache maintenance API must be added to cope with
      latest ARM system requirements.

      This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.

      This function cleans and invalidates all data cache levels up to the
      Level of Unification Inner Shareable (LoUIS) and invalidates the instruction
      cache for processors that support it (> v7).

      This patch also creates an alias of the cache LoUIS function to flush_kern_all
      for all processor versions prior to v7, so that the current cache flushing
      behaviour is unchanged for those processors.

      v7 cache maintenance code implements a cache LoUIS function that cleans and
      invalidates the D-cache up to LoUIS and invalidates the I-cache, according
      to the new API.

      Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: mm: rename jump labels in v7_flush_dcache_all function

      This patch renames jump labels in v7_flush_dcache_all in order to define
      a specific flush cache levels entry point.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: kernel: update cpu_suspend code to use cache LoUIS operations

      In processors like A15/A7 L2 cache is unified and integrated within the
      processor cache hierarchy, so that it is not considered an outer cache
      anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
      all cache levels up to Level of Coherency (LoC) that includes
      the L2 unified cache.

      When a single CPU is suspended (CPU idle) a complete L2 clean is not
      required, so generic cpu_suspend code must clean the data cache using the
      newly introduced cache LoUIS function.

      The context and stack pointer (context pointer) are cleaned to main memory
      using cache area functions that operate on MVA and guarantee that the data
      is written back to main memory (perform cache cleaning up to the Point of
      Coherency - PoC) so that the processor can fetch the context when the MMU
      is off in the cpu_resume code path.

      outer_cache management remains unchanged.

      Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: kernel: update __cpu_disable to use cache LoUIS maintenance API

      When a CPU is hotplugged out caches that reside in its power domain
      lose their contents and so must be cleaned to the next memory level.

      Currently, __cpu_disable calls flush_cache_all() that for new generation
      processor like A15/A7 ends up cleaning and invalidating all cache levels
      up to Level of Coherency, which includes the unified L2.

      This ends up being a waste of cycles since the L2 cache contents are not
      lost on power down.

      This patch updates __cpu_disable to use the new LoUIS API cache operations.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>
byteorder: allow arch to opt to use GCC intrinsics for byteswapping

Since GCC 4.4, there have been __builtin_bswap32() and __builtin_bswap16()
intrinsics. A __builtin_bswap16() came a little later (4.6 for PowerPC,
48 for other platforms).

By using these instead of the inline assembler that most architectures
have in their __arch_swabXX() macros, we let the compiler see what's
actually happening. The resulting code should be at least as good, and
much *better* in the cases where it can be combined with a nearby load
or store, using a load-and-byteswap or store-and-byteswap instruction
(e.g. lwbrx/stwbrx on PowerPC, movbe on Atom).

When GCC is sufficiently recent *and* the architecture opts in to using
the intrinsics by setting CONFIG_ARCH_USE_BUILTIN_BSWAP, they will be
used in preference to the __arch_swabXX() macros. An architecture which
does not set ARCH_USE_BUILTIN_BSWAP will continue to use its own
hand-crafted macros.

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
Acked-by: H. Peter Anvin <hpa@linux.intel.com>
binfmt_elf.c: use get_random_int() to fix entropy depleting.

Usually the entropy pool depletes very fast and stays in the 100-200 range.
After this patch, the entropy stays in the 600-1000 range without the risk of
depleting in normal workloads.

Change-Id: Ice0d0f14691046249d7b4a4d31af98ab80c6306b
introduce for_each_thread() to replace the buggy while_each_thread()

while_each_thread() and next_thread() should die, almost every lockless
usage is wrong.

1. Unless g == current, the lockless while_each_thread() is not safe.

   while_each_thread(g, t) can loop forever if g exits, next_thread()
   can't reach the unhashed thread in this case. Note that this can
   happen even if g is the group leader, it can exec.

2. Even if while_each_thread() itself was correct, people often use
   it wrongly.

   It was never safe to just take rcu_read_lock() and loop unless
   you verify that pid_alive(g) == T, even the first next_thread()
   can point to the already freed/reused memory.

This patch adds signal_struct->thread_head and task->thread_node to
create the normal rcu-safe list with the stable head.  The new
for_each_thread(g, t) helper is always safe under rcu_read_lock() as
long as this task_struct can't go away.

Note: of course it is ugly to have both task_struct->thread_node and the
old task_struct->thread_group, we will kill it later, after we change
the users of while_each_thread() to use for_each_thread().

Perhaps we can kill it even before we convert all users, we can
reimplement next_thread(t) using the new thread_head/thread_node.  But
we can't do this right now because this will lead to subtle behavioural
changes.  For example, do/while_each_thread() always sees at least one
task, while for_each_thread() can do nothing if the whole thread group
has died.  Or thread_group_empty(), currently its semantics is not clear
unless thread_group_leader(p) and we need to audit the callers before we
can change it.

So this patch adds the new interface which has to coexist with the old
one for some time, hopefully the next changes will be more or less
straightforward and the old one will go away soon.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Sergey Dyasly <dserrg@gmail.com>
Tested-by: Sergey Dyasly <dserrg@gmail.com>
Reviewed-by: Sameer Nanda <snanda@chromium.org>
Acked-by: David Rientjes <rientjes@google.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mandeep Singh Baines <msb@chromium.org>
Cc: "Ma, Xindong" <xindong.ma@intel.com>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
sched: Fix hotplug vs. set_cpus_allowed_ptr()

Lai found that:

  WARNING: CPU: 1 PID: 13 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x2d/0x4b()
  ...
  migration_cpu_stop+0x1d/0x22

was caused by set_cpus_allowed_ptr() assuming that cpu_active_mask is
always a sub-set of cpu_online_mask.

This isn't true since 5fbd036b552f ("sched: Cleanup cpu_active madness").

So set active and online at the same time to avoid this particular
problem.

Fixes: 5fbd036b552f ("sched: Cleanup cpu_active madness")
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michael wang <wangyun@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Cc: Toshi Kani <toshi.kani@hp.com>
Link: http://lkml.kernel.org/r/53758B12.8060609@cn.fujitsu.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched: Improve scalability via 'CPU buddies', which withstand random perturbations

Traversing an entire package is not only expensive, it also leads to tasks
bouncing all over a partially idle and possible quite large package.  Fix
that up by assigning a 'buddy' CPU to try to motivate.  Each buddy may try
to motivate that one other CPU, if it's busy, tough, it may then try its
SMT sibling, but that's all this optimization is allowed to cost.

Sibling cache buddies are cross-wired to prevent bouncing.

4 socket 40 core + SMT Westmere box, single 30 sec tbench runs, higher is better:

 clients     1       2       4        8       16       32       64      128
 ..........................................................................
 pre        30      41     118      645     3769     6214    12233    14312
 post      299     603    1211     2418     4697     6847    11606    14557

A nice increase in performance.

Signed-off-by: Mike Galbraith <efault@gmx.de>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1339471112.7352.32.camel@marge.simpson.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched: Reorder 'struct lb_env' members to reduce its size

Members of 'struct lb_env' are not in appropriate order to reuse compiler
added padding on 64bit architectures. In this patch we reorder those struct
members and help reduce the size of the structure from 96 bytes to 80
bytes on 64 bit architectures.

Suggested-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
Signed-off-by: Prashanth Nageshappa <prashanth@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/4FE06DDE.7000403@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched: Reset loop counters if all tasks are pinned and we need to redo load balance

While load balancing, if all tasks on the source runqueue are pinned,
we retry after excluding the corresponding source cpu. However, loop counters
env.loop and env.loop_break are not reset before retrying, which can lead
to failure in moving the tasks. In this patch we reset env.loop and
env.loop_break to their inital values before we retry.

Signed-off-by: Prashanth Nageshappa <prashanth@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/4FE06EEF.2090709@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
[MSM HOTPLUG] *squashing updates*

msm: hotplug/platsmp: Fix cpu invaldation pointer arithmetic

      These functions take two void pointers indicating where to start
      and end the invalidation and cache flush. The pointer arithmetic
      is considering the pen_release pointer as an int pointer so
      adding sizeof(pen_release) is adding 16 bytes to the pointer and
      specifying that as the end. This inadvertently invalidates data
      next to the pen_release variable which could be bad. Fix the
      pointer math to only invalidate or clean the region for the
      pen_release variable.

      Change-Id: If0cf0e5de7c9f4b8c70662b29957a1f347809124
      Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>

msm: hotplug: Change the pen variable in one place

      Other WFI based hotplug implementations set the pen variable to
      -1 in C code and do the appropriate flushing operations in one
      location. They also don't try to invalidate the cache on a
      spurious wakeup and report how many spurious wakeups there were.
      Let's do the same in our code so that we can remove our
      dependence on the dmac_{inv,flush}_range() functions (which are
      no longer upstream) and easily report spurious wakeups to the
      kernel logs.

      This does delay the time we set the pen back to its default
      value, but that doesn't actually matter because generic ARM
      hotplug code already waits for a completion before continuing.

      Change-Id: Ic36fbf744fe8abeda20ad9859a207b5e37c5e63c
      Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>

msm: hotplug: remove code related to cpu_killed

      Earlier the completion event cpu_killed was used to synchronize
      notify to hot plug framework that the CPU is dead. But now it is
      replaced with another completion event cpu_died and it is placed
      in the hotplug framework itself. So there is no need for a
      platform to implement a completion event.

      But in msm platform driver the completion event related code
      was not removed with the changes in hotplug framework. This
      patch removes the unused code.

      Change-Id: I4418e5e397821e39a7b6a6efa997c4a866e9b46b
      Signed-off-by: Venkat Devarasetty <vdevaras@codeaurora.org>

msm: hotplug: Block hotplug removal of CPU0

      Hotplug removal of CPU0 is not currently supported. Attempts to
      remove it may result in undefined behaviour. Explicitly block
      any such attempts before other hotplug notifiers get the chance
      to run by setting the notifier priority to INT_MAX.

      Change-Id: I9e1f5b4a0db554295a097220417f2bdf4c690f5f
      Signed-off-by: Matt Wagantall <mattw@codeaurora.org>

msm: platsmp-8625: modify pen_release variable to volatile

      Since pen_release is a shared variable between cpus. By marking it
      as volatile, compiler won't cache that variable into registers and
      won't optimize the code which access pen_release variable.

      Change-Id: Ib5b7aa5932b35d64d05f9e4a4672df2bb9207c59
      Signed-off-by: tirupathireddy <tirupath@codeaurora.org>

msm: platsmp-8625: Add cold_boot_done as per_cpu variable

      cold_boot_done is declared as an ordinary variable, and useful for
      single secondary cpu, right now. It will not work in the scenarios
      where we have more than two cores in the system. Usage of per-cpu
      variable will make the code more generic for multi-core environment.

      Change-Id: I4b2480243b2adaadf5c5a2a3534f66faebd87553
      Signed-off-by: tirupathireddy <tirupath@codeaurora.org>

msm: pm2: Raise SPI interrupt instread of SGI to non boot CPUs

      Once APPS comes out of PC and mpdecision want to bring the non-boot
      cpus to online, first CORE0 raises the SPI interrupt(MPCORE) to bring
      the cores(1/2/3) online. While bringing the cores online, CORE0 raises
      the SPI interrupt instead of SGI. Because once APPS does a PC QGIC
      looses its state and none of the non-boot cpus are not capable of
      receiving the SGI interrupt until cores configure the QGIC PPIs&SGIs
      by themselves.

      Change-Id: Ic4aedeffbcd3fa2f0aa15e2e8ed7e25a03468f87
      Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>
mm, mempolicy: remove per-process flag

PF_MEMPOLICY is an unnecessary optimization for CONFIG_SLAB users.
There's no significant performance degradation to checking
current->mempolicy rather than current->flags & PF_MEMPOLICY in the
allocation path, especially since this is considered unlikely().

Running TCP_RR with netperf-2.4.5 through localhost on 16 cpu machine with
64GB of memory and without a mempolicy:

	threads		before		after
	16		1249409		1244487
	32		1281786		1246783
	48		1239175		1239138
	64		1244642		1241841
	80		1244346		1248918
	96		1266436		1254316
	112		1307398		1312135
	128		1327607		1326502

Per-process flags are a scarce resource so we should free them up whenever
possible and make them available.  We'll be using it shortly for memcg oom
reserves.

Signed-off-by: David Rientjes <rientjes@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Jianguo Wu <wujianguo@huawei.com>
Cc: Tim Hockin <thockin@google.com>
Cc: Christoph Lameter <cl@linux.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[CPU HotPlug] Provide lockless versions of callback registration functions

CPU hotplug: Add lockdep annotations to get/put_online_cpus()

      Add lockdep annotations for get/put_online_cpus() and
      cpu_hotplug_begin()/cpu_hotplug_end().

      Cc: Ingo Molnar <mingo@kernel.org>
      Reviewed-by: Oleg Nesterov <oleg@redhat.com>
      Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

CPU hotplug: Provide lockless versions of callback registration functions

      The following method of CPU hotplug callback registration is not safe
      due to the possibility of an ABBA deadlock involving the cpu_add_remove_lock
      and the cpu_hotplug.lock.

	      get_online_cpus();

	      for_each_online_cpu(cpu)
		      init_cpu(cpu);

	      register_cpu_notifier(&foobar_cpu_notifier);

	      put_online_cpus();

      The deadlock is shown below:

		CPU 0                                         CPU 1
		-----                                         -----

	Acquire cpu_hotplug.lock
	[via get_online_cpus()]

						    CPU online/offline operation
						    takes cpu_add_remove_lock
						    [via cpu_maps_update_begin()]

	Try to acquire
	cpu_add_remove_lock
	[via register_cpu_notifier()]

						    CPU online/offline operation
						    tries to acquire cpu_hotplug.lock
						    [via cpu_hotplug_begin()]

				  *** DEADLOCK! ***

      The problem here is that callback registration takes the locks in one order
      whereas the CPU hotplug operations take the same locks in the opposite order.
      To avoid this issue and to provide a race-free method to register CPU hotplug
      callbacks (along with initialization of already online CPUs), introduce new
      variants of the callback registration APIs that simply register the callbacks
      without holding the cpu_add_remove_lock during the registration. That way,
      we can avoid the ABBA scenario. However, we will need to hold the
      cpu_add_remove_lock throughout the entire critical section, to protect updates
      to the callback/notifier chain.

      This can be achieved by writing the callback registration code as follows:

	      cpu_maps_update_begin(); [ or cpu_notifier_register_begin(); see below ]

	      for_each_online_cpu(cpu)
		      init_cpu(cpu);

	      /* This doesn't take the cpu_add_remove_lock */
	      __register_cpu_notifier(&foobar_cpu_notifier);

	      cpu_maps_update_done();  [ or cpu_notifier_register_done(); see below ]

      Note that we can't use get_online_cpus() here instead of cpu_maps_update_begin()
      because the cpu_hotplug.lock is dropped during the invocation of CPU_POST_DEAD
      notifiers, and hence get_online_cpus() cannot provide the necessary
      synchronization to protect the callback/notifier chains against concurrent
      reads and writes. On the other hand, since the cpu_add_remove_lock protects
      the entire hotplug operation (including CPU_POST_DEAD), we can use
      cpu_maps_update_begin/done() to guarantee proper synchronization.

      Also, since cpu_maps_update_begin/done() is like a super-set of
      get/put_online_cpus(), the former naturally protects the critical sections
      from concurrent hotplug operations.

      Since the names cpu_maps_update_begin/done() don't make much sense in CPU
      hotplug callback registration scenarios, we'll introduce new APIs named
      cpu_notifier_register_begin/done() and map them to cpu_maps_update_begin/done().

      In summary, introduce the lockless variants of un/register_cpu_notifier() and
      also export the cpu_notifier_register_begin/done() APIs for use by modules.
      This way, we provide a race-free way to register hotplug callbacks as well as
      perform initialization for the CPUs that are already online.

      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Acked-by: Oleg Nesterov <oleg@redhat.com>
      Acked-by: Toshi Kani <toshi.kani@hp.com>
      Reviewed-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Doc/cpu-hotplug: Specify race-free way to register CPU hotplug callbacks

      Recommend the usage of the new CPU hotplug callback registration APIs
      (__register_cpu_notifier() etc), when subsystems need to also perform
      initialization for already online CPUs. Provide examples of correct
      and race-free ways of achieving this, and point out the kinds of code
      that are error-prone.

      Cc: Rob Landley <rob@landley.net>
      Cc: Ingo Molnar <mingo@kernel.org>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

---------------------------------------------------------------------------------------------------

CPU hotplug, perf: Fix CPU hotplug callback registration

arm, hw-breakpoint: Fix CPU hotplug callback registration

zsmalloc: Fix CPU hotplug callback registration

drivers/base/topology.c: Fix CPU hotplug callback registration

oprofile, nmi-timer: Fix CPU hotplug callback registration

profile: Fix CPU hotplug callback registration

mm, vmstat: Fix CPU hotplug callback registration

net/core/flow.c: Fix CPU hotplug callback registration

net/iucv/iucv.c: Fix CPU hotplug callback registration
[SOFTIRQ] *squashing updates*

mm: allow PF_MEMALLOC from softirq context

      This is needed to allow network softirq packet processing to make use of
      PF_MEMALLOC.

      Currently softirq context cannot use PF_MEMALLOC due to it not being
      associated with a task, and therefore not having task flags to fiddle with
      - thus the gfp to alloc flag mapping ignores the task flags when in
      interrupts (hard or soft) context.

      Allowing softirqs to make use of PF_MEMALLOC therefore requires some
      trickery.  This patch borrows the task flags from whatever process happens
      to be preempted by the softirq.  It then modifies the gfp to alloc flags
      mapping to not exclude task flags in softirq context, and modify the
      softirq code to save, clear and restore the PF_MEMALLOC flag.

      The save and clear, ensures the preempted task's PF_MEMALLOC flag doesn't
      leak into the softirq.  The restore ensures a softirq's PF_MEMALLOC flag
      cannot leak back into the preempted process.  This should be safe due to
      the following reasons

      Softirqs can run on multiple CPUs sure but the same task should not be
	      executing the same softirq code. Neither should the softirq
	      handler be preempted by any other softirq handler so the flags
	      should not leak to an unrelated softirq.

      Softirqs re-enable hardware interrupts in __do_softirq() so can be
	      preempted by hardware interrupts so PF_MEMALLOC is inherited
	      by the hard IRQ. However, this is similar to a process in
	      reclaim being preempted by a hardirq. While PF_MEMALLOC is
	      set, gfp_to_alloc_flags() distinguishes between hard and
	      soft irqs and avoids giving a hardirq the ALLOC_NO_WATERMARKS
	      flag.

      If the softirq is deferred to ksoftirq then its flags may be used
	      instead of a normal tasks but as the softirq cannot be preempted,
	      the PF_MEMALLOC flag does not leak to other code by accident.

      [davem@davemloft.net: Document why PF_MEMALLOC is safe]
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: Mel Gorman <mgorman@suse.de>
      Cc: David Miller <davem@davemloft.net>
      Cc: Neil Brown <neilb@suse.de>
      Cc: Mike Christie <michaelc@cs.wisc.edu>
      Cc: Eric B Munson <emunson@mgebm.net>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Christoph Lameter <cl@linux.com>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

softirq: reduce latencies

      In various network workloads, __do_softirq() latencies can be up
      to 20 ms if HZ=1000, and 200 ms if HZ=100.

      This is because we iterate 10 times in the softirq dispatcher,
      and some actions can consume a lot of cycles.

      This patch changes the fallback to ksoftirqd condition to :

      - A time limit of 2 ms.
      - need_resched() being set on current task

      When one of this condition is met, we wakeup ksoftirqd for further
      softirq processing if we still have pending softirqs.

      Using need_resched() as the only condition can trigger RCU stalls,
      as we can keep BH disabled for too long.

      I ran several benchmarks and got no significant difference in
      throughput, but a very significant reduction of latencies (one order
      of magnitude) :

      In following bench, 200 antagonist "netperf -t TCP_RR" are started in
      background, using all available cpus.

      Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
      IRQ (hard+soft)

      Before patch :

      # netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
      RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
      MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
      to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
      RT_LATENCY=550110.424
      MIN_LATENCY=146858
      MAX_LATENCY=997109
      P50_LATENCY=305000
      P90_LATENCY=550000
      P99_LATENCY=710000
      MEAN_LATENCY=376989.12
      STDDEV_LATENCY=184046.92

      After patch :

      # netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
      RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
      MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
      to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
      RT_LATENCY=40545.492
      MIN_LATENCY=9834
      MAX_LATENCY=78366
      P50_LATENCY=33583
      P90_LATENCY=59000
      P99_LATENCY=69000
      MEAN_LATENCY=38364.67
      STDDEV_LATENCY=12865.26

      Signed-off-by: Eric Dumazet <edumazet@google.com>
      Cc: David Miller <davem@davemloft.net>
      Cc: Tom Herbert <therbert@google.com>
      Cc: Ben Hutchings <bhutchings@solarflare.com>
      Signed-off-by: David S. Miller <davem@davemloft.net>

irq: Ensure irq_exit() code runs with interrupts disabled

      We had already a few problems with code called from irq_exit() when
      interrupted from a nesting interrupt. This can happen on architectures
      which do not define __ARCH_IRQ_EXIT_IRQS_DISABLED.

      __ARCH_IRQ_EXIT_IRQS_DISABLED should go away and we want to make it
      mandatory to call irq_exit() with interrupts disabled.

      As a temporary protection disable interrupts for those architectures
      which do not define __ARCH_IRQ_EXIT_IRQS_DISABLED and add a WARN_ONCE
      when an architecture which defines __ARCH_IRQ_EXIT_IRQS_DISABLED calls
      irq_exit() with interrupts enabled.

      Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linuxfoundation.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302202155320.22263@ionos

irq: Sanitize invoke_softirq

      With the irq protection in irq_exit, we can remove the #ifdeffery and
      the bh_disable/enable dance in invoke_softirq()

      Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linuxfoundation.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302202155320.22263@ionos

      # This is the 5th commit message:

      irq: Remove IRQ_EXIT_OFFSET workaround

      The IRQ_EXIT_OFFSET trick was used to make sure the irq
      doesn't get preempted after we substract the HARDIRQ_OFFSET
      until we are entirely done with any code in irq_exit().

      This workaround was necessary because some archs may call
      irq_exit() with irqs enabled and there is still some code
      in the end of this function that is not covered by the
      HARDIRQ_OFFSET but want to stay non-preemptible.

      Now that irq are always disabled in irq_exit(), the whole code
      is guaranteed not to be preempted. We can thus remove this hack.

      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

irq: Don't re-enable interrupts at the end of irq_exit

      Commit 74eed0163d0def3fce27228d9ccf3d36e207b286
      "irq: Ensure irq_exit() code runs with interrupts disabled"
      restore interrupts flags in the end of irq_exit() for archs
      that don't define __ARCH_IRQ_EXIT_IRQS_DISABLED.

      However always returning from irq_exit() with interrupts
      disabled should not be a problem for these archs. Prior to
      this commit this was already happening anytime we processed
      pending softirqs anyway.

      Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Fix lockup related to stop_machine being stuck in __do_softirq.

      The stop machine logic can lock up if all but one of the migration
      threads make it through the disable-irq step and the one remaining
      thread gets stuck in __do_softirq.  The reason __do_softirq can hang is
      that it has a bail-out based on jiffies timeout, but in the lockup case,
      jiffies itself is not incremented.

      To work around this, re-add the max_restart counter in __do_irq and stop
      processing irqs after 10 restarts.

      Thanks to Tejun Heo and Rusty Russell and others for helping me track
      this down.

      This was introduced in 3.9 by commit c10d73671ad3 ("softirq: reduce
      latencies").

      It may be worth looking into ath9k to see if it has issues with its irq
      handler at a later date.

      The hang stack traces look something like this:

	  ------------[ cut here ]------------
	  WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9c/0xa7()
	  Watchdog detected hard LOCKUP on cpu 2
	  Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
	  Pid: 23, comm: migration/2 Tainted: G         C   3.9.4+ #11
	  Call Trace:
	  <NMI>   warn_slowpath_common+0x85/0x9f
	    warn_slowpath_fmt+0x46/0x48
	    watchdog_overflow_callback+0x9c/0xa7
	    __perf_event_overflow+0x137/0x1cb
	    perf_event_overflow+0x14/0x16
	    intel_pmu_handle_irq+0x2dc/0x359
	    perf_event_nmi_handler+0x19/0x1b
	    nmi_handle+0x7f/0xc2
	    do_nmi+0xbc/0x304
	    end_repeat_nmi+0x1e/0x2e
	  <<EOE>>
	    cpu_stopper_thread+0xae/0x162
	    smpboot_thread_fn+0x258/0x260
	    kthread+0xc7/0xcf
	    ret_from_fork+0x7c/0xb0
	  ---[ end trace 4947dfa9b0a4cec3 ]---
	  BUG: soft lockup - CPU#1 stuck for 22s! [migration/1:17]
	  Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
	  irq event stamp: 835637905
	  hardirqs last  enabled at (835637904): __do_softirq+0x9f/0x257
	  hardirqs last disabled at (835637905): apic_timer_interrupt+0x6d/0x80
	  softirqs last  enabled at (5654720): __do_softirq+0x1ff/0x257
	  softirqs last disabled at (5654725): irq_exit+0x5f/0xbb
	  CPU 1
	  Pid: 17, comm: migration/1 Tainted: G        WC   3.9.4+ #11 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
	  RIP: tasklet_hi_action+0xf0/0xf0
	  Process migration/1
	  Call Trace:
	  <IRQ>
	    __do_softirq+0x117/0x257
	    irq_exit+0x5f/0xbb
	    smp_apic_timer_interrupt+0x8a/0x98
	    apic_timer_interrupt+0x72/0x80
	  <EOI>
	    printk+0x4d/0x4f
	    stop_machine_cpu_stop+0x22c/0x274
	    cpu_stopper_thread+0xae/0x162
	    smpboot_thread_fn+0x258/0x260
	    kthread+0xc7/0xcf
	    ret_from_fork+0x7c/0xb0

      Signed-off-by: Ben Greear <greearb@candelatech.com>
      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Pekka Riikonen <priikone@iki.fi>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: stable@kernel.org
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

irq: Force hardirq exit's softirq processing on its own stack

      The commit facd8b80c67a3cf64a467c4a2ac5fb31f2e6745b
      ("irq: Sanitize invoke_softirq") converted irq exit
      calls of do_softirq() to __do_softirq() on all architectures,
      assuming it was only used there for its irq disablement
      properties.

      But as a side effect, the softirqs processed in the end
      of the hardirq are always called on the inline current
      stack that is used by irq_exit() instead of the softirq
      stack provided by the archs that override do_softirq().

      The result is mostly safe if the architecture runs irq_exit()
      on a separate irq stack because then softirqs are processed
      on that same stack that is near empty at this stage (assuming
      hardirq aren't nesting).

      Otherwise irq_exit() runs in the task stack and so does the softirq
      too. The interrupted call stack can be randomly deep already and
      the softirq can dig through it even further. To add insult to the
      injury, this softirq can be interrupted by a new hardirq, maximizing
      the chances for a stack overrun as reported in powerpc for example:

	      do_IRQ: stack overflow: 1920
	      CPU: 0 PID: 1602 Comm: qemu-system-ppc Not tainted 3.10.4-300.1.fc19.ppc64p7 #1
	      Call Trace:
	      [c0000000050a8740] .show_stack+0x130/0x200 (unreliable)
	      [c0000000050a8810] .dump_stack+0x28/0x3c
	      [c0000000050a8880] .do_IRQ+0x2b8/0x2c0
	      [c0000000050a8930] hardware_interrupt_common+0x154/0x180
	      --- Exception: 501 at .cp_start_xmit+0x3a4/0x820 [8139cp]
		      LR = .cp_start_xmit+0x390/0x820 [8139cp]
	      [c0000000050a8d40] .dev_hard_start_xmit+0x394/0x640
	      [c0000000050a8e00] .sch_direct_xmit+0x110/0x260
	      [c0000000050a8ea0] .dev_queue_xmit+0x260/0x630
	      [c0000000050a8f40] .br_dev_queue_push_xmit+0xc4/0x130 [bridge]
	      [c0000000050a8fc0] .br_dev_xmit+0x198/0x270 [bridge]
	      [c0000000050a9070] .dev_hard_start_xmit+0x394/0x640
	      [c0000000050a9130] .dev_queue_xmit+0x428/0x630
	      [c0000000050a91d0] .ip_finish_output+0x2a4/0x550
	      [c0000000050a9290] .ip_local_out+0x50/0x70
	      [c0000000050a9310] .ip_queue_xmit+0x148/0x420
	      [c0000000050a93b0] .tcp_transmit_skb+0x4e4/0xaf0
	      [c0000000050a94a0] .__tcp_ack_snd_check+0x7c/0xf0
	      [c0000000050a9520] .tcp_rcv_established+0x1e8/0x930
	      [c0000000050a95f0] .tcp_v4_do_rcv+0x21c/0x570
	      [c0000000050a96c0] .tcp_v4_rcv+0x734/0x930
	      [c0000000050a97a0] .ip_local_deliver_finish+0x184/0x360
	      [c0000000050a9840] .ip_rcv_finish+0x148/0x400
	      [c0000000050a98d0] .__netif_receive_skb_core+0x4f8/0xb00
	      [c0000000050a99d0] .netif_receive_skb+0x44/0x110
	      [c0000000050a9a70] .br_handle_frame_finish+0x2bc/0x3f0 [bridge]
	      [c0000000050a9b20] .br_nf_pre_routing_finish+0x2ac/0x420 [bridge]
	      [c0000000050a9bd0] .br_nf_pre_routing+0x4dc/0x7d0 [bridge]
	      [c0000000050a9c70] .nf_iterate+0x114/0x130
	      [c0000000050a9d30] .nf_hook_slow+0xb4/0x1e0
	      [c0000000050a9e00] .br_handle_frame+0x290/0x330 [bridge]
	      [c0000000050a9ea0] .__netif_receive_skb_core+0x34c/0xb00
	      [c0000000050a9fa0] .netif_receive_skb+0x44/0x110
	      [c0000000050aa040] .napi_gro_receive+0xe8/0x120
	      [c0000000050aa0c0] .cp_rx_poll+0x31c/0x590 [8139cp]
	      [c0000000050aa1d0] .net_rx_action+0x1dc/0x310
	      [c0000000050aa2b0] .__do_softirq+0x158/0x330
	      [c0000000050aa3b0] .irq_exit+0xc8/0x110
	      [c0000000050aa430] .do_IRQ+0xdc/0x2c0
	      [c0000000050aa4e0] hardware_interrupt_common+0x154/0x180
	      --- Exception: 501 at .bad_range+0x1c/0x110
		      LR = .get_page_from_freelist+0x908/0xbb0
	      [c0000000050aa7d0] .list_del+0x18/0x50 (unreliable)
	      [c0000000050aa850] .get_page_from_freelist+0x908/0xbb0
	      [c0000000050aa9e0] .__alloc_pages_nodemask+0x21c/0xae0
	      [c0000000050aaba0] .alloc_pages_vma+0xd0/0x210
	      [c0000000050aac60] .handle_pte_fault+0x814/0xb70
	      [c0000000050aad50] .__get_user_pages+0x1a4/0x640
	      [c0000000050aae60] .get_user_pages_fast+0xec/0x160
	      [c0000000050aaf10] .__gfn_to_pfn_memslot+0x3b0/0x430 [kvm]
	      [c0000000050aafd0] .kvmppc_gfn_to_pfn+0x64/0x130 [kvm]
	      [c0000000050ab070] .kvmppc_mmu_map_page+0x94/0x530 [kvm]
	      [c0000000050ab190] .kvmppc_handle_pagefault+0x174/0x610 [kvm]
	      [c0000000050ab270] .kvmppc_handle_exit_pr+0x464/0x9b0 [kvm]
	      [c0000000050ab320]  kvm_start_lightweight+0x1ec/0x1fc [kvm]
	      [c0000000050ab4f0] .kvmppc_vcpu_run_pr+0x168/0x3b0 [kvm]
	      [c0000000050ab9c0] .kvmppc_vcpu_run+0xc8/0xf0 [kvm]
	      [c0000000050aba50] .kvm_arch_vcpu_ioctl_run+0x5c/0x1a0 [kvm]
	      [c0000000050abae0] .kvm_vcpu_ioctl+0x478/0x730 [kvm]
	      [c0000000050abc90] .do_vfs_ioctl+0x4ec/0x7c0
	      [c0000000050abd80] .SyS_ioctl+0xd4/0xf0
	      [c0000000050abe30] syscall_exit+0x0/0x98

      Since this is a regression, this patch proposes a minimalistic
      and low-risk solution by blindly forcing the hardirq exit processing of
      softirqs on the softirq stack. This way we should reduce significantly
      the opportunities for task stack overflow dug by softirqs.

      Longer term solutions may involve extending the hardirq stack coverage to
      irq_exit(), etc...

      Reported-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: #3.9.. <stable@vger.kernel.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Consolidate do_softirq() arch overriden implementations

      All arch overriden implementations of do_softirq() share the following
      common code: disable irqs (to avoid races with the pending check),
      check if there are softirqs pending, then execute __do_softirq() on
      a specific stack.

      Consolidate the common parts such that archs only worry about the
      stack switch.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Optimize call to softirq on hardirq exit

      Before processing softirqs on hardirq exit, we already
      do the check for pending softirqs while hardirqs are
      guaranteed to be disabled.

      So we can take a shortcut and safely jump to the arch
      specific implementation directly.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Improve a bit softirq debugging

      do_softirq() has a debug check that verifies that it is not nesting
      on softirqs processing, nor miscounting the softirq part of the preempt
      count.

      But making sure that softirqs processing don't nest is actually a more
      generic concern that applies to any caller of __do_softirq().

      Do take it one step further and generalize that debug check to
      any softirq processing.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Justify the various softirq stack choices

      For clarity, comment the various stack choices for softirqs
      processing, whether we execute them from ksoftirqd or
      local_irq_enable() calls.

      Their use on irq_exit() is already commented.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Optimize softirq stack selection in irq exit

      If irq_exit() is called on the arch's specified irq stack,
      it should be safe to run softirqs inline under that same
      irq stack as it is near empty by the time we call irq_exit().

      For example if we use the same stack for both hard and soft irqs here,
      the worst case scenario is:
      hardirq -> softirq -> hardirq. But then the softirq supersedes the
      first hardirq as the stack user since irq_exit() is called in
      a mostly empty stack. So the stack merge in this case looks acceptable.

      Stack overrun still have a chance to happen if hardirqs have more
      opportunities to nest, but then it's another problem to solve.

      So lets adapt the irq exit's softirq stack on top of a new Kconfig symbol
      that can be defined when irq_exit() runs on the irq stack. That way
      we can spare some stack switch on irq processing and all the cache
      issues that come along.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

revert "softirq: Add support for triggering softirq work on softirqs"

      This commit was incomplete in that code to remove items from the per-cpu
      lists was missing and never acquired a user in the 5 years it has been in
      the tree.  We're going to implement what it seems to try to archive in a
      simpler way, and this code is in the way of doing so.

      Signed-off-by: Christoph Hellwig <hch@lst.de>
      Cc: Jan Kara <jack@suse.cz>
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Conflicts:
	kernel/softirq.c
ARM: Optimize memset/memcpy (Credits to: Harm Hanemaaijer)

[PATCH 1/3] Rename ARM assembler push/pull macros

      The ARM assembler library functions use a macro called "push" that
      along with a macro called "pull" is used to shift bytes around in a
      word in an endian-independent way. However, the modern unified
      ARM assembler syntax also defines the instruction "push" to push
      data onto the stack, which has specific encodings in the Thumb2
      instruction set. For prevent possible conflicts going forward, and
      to allow the use of the more transparent "push" instruction along
      with the modern unified assembler syntax, this patch renames all
      occurrences of the "push" macro to "pushbits", as well as renaming
      the macro argument, when also called "push", to "pushshift".

      For consistency, the macro called "pull" with its argument name
      "pull" are also renamed to "pullbits" and "pullshift", respectively.

      Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>

[PATCH 2/3] Optimize copy_page for modern ARM platforms

      The existing implementation of copy_page for ARM appears to be
      optimized for older platforms. Benchmark testing in a sandbox
      environment shows suboptimal performance on modern platforms
      like armv6 and armv7, with speed-ups ranging from 10% (Cortex A8)
      to 80% (armv6 used in Raspberry Pi) being achievable.

      This commit optimizes copy_page and introduces the new compile-time
      constant PREFETCH_DISTANCE, defined in cache.h, which when
      multiplied by L1_CACHE_BYTES is equal to the offset used for
      prefetches performed with the PLD instruction. For platforms where
      L1_CACHE_BYTES is 32 (armv5 and armv6), copy_page processes 32 bytes
      at a time while doing one prefetch per iteration, while for armv7
      (with L1_CACHE_BYTES equal to 64), 64 bytes are processed at at time
      with one prefetch per iteration. When no preload instruction is
      available (platforms earlier than armv5), no preload instructions
      are generated and 32 bytes are processed at at time.

      To facilitate specifying instructions for architectures with no
      preload instruction, the NO_PLD macro is added to assembler.h,
      augmenting the PLD macro.

      Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>

[PATCH 3/3] Optimize ARM memset and memzero functions

      The ARM memset and memzero functions are optimized with lower overhead
      for small requests, generation of more 16-bit Thumb2 instructions when
      compiled in Thumb2 mode, and configurable destination alignment before
      the main block-copying loop.

      The new compile-time constant MEMSET_WRITE_ALIGN_BYTES is introduced
      in assembler.h, to augment the CALGN macro that previously regulated
      32-byte write alignment but was only used on the Feroceon platform.
      MEMSET_WRITE_ALIGN_BYTES can have values of 0 (no write alignment),
      8, or 32. Apart from Feroceon, memset write alignment of 32 bytes
      appears to benefit the armv6 platform, while the armv7 platform
      seems benefit from alignment to 8 bytes for memset/memzero.

      The CALGN macro is renamed to MEMSET_CALGN for memset and memzero; the
      original CALGN macro is reserved for the memcpy family of functions
      (memcpy, copy_from_user, copy_to_user) currently implemented in
      copy_template.S, and the associated compile time constant
      WRITE_ALIGN_BYTES defines the write alignment for memcpy-related
      functions that will be utilized in subsequent contributions. Because
      the current CALGN implementation in copy_template.S only implements
      32-byte write alignment and is broken on Thumb2, it is only enabled
      when WRITE_ALIGN_BYTES is equal to 32 and Thumb2 mode is not enabled.

      Finally, memset and memzero now include a directive to enable unified
      ARM assembler syntax.

      Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
ARM: bitops: prefetch the destination word for write prior to strex

The cost of changing a cacheline from shared to exclusive state can be
significant, especially when this is triggered by an exclusive store,
since it may result in having to retry the transaction.

This patch prefixes our atomic bitops implementation with prefetchw,
to try and grab the line in exclusive state from the start. The testop
macro is left alone, since the barrier semantics limit the usefulness
of prefetching data.

Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
arm: Fix compilation error with gcc 4.5.2

Fix warning with the LDM/STM instructions. The instructions dont
have a register offset so set the offset to 0.

Change-Id: I2c988373c88e280015faa43076139650747d7ff3
Acked-by: Kaushik Sikdar <ksikdar@qualcomm.com>
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
*squashing some fixes and reverts*

msm: avs: Remove vestigial software based AVS

      Older targets like 8x50 had the first version of Apdaptive Voltage
      Scaling done in software. This feature was never enabled and supported.
      Remove the code and the associated config items.

      Change-Id: If3bdab7d0bdb5d51e099f7c37f56592826edc102
      Signed-off-by: Praveen Chidambaram <pchidamb@codeaurora.org>

drivers: msm_nand: *silence GCC warnings*

      Signed-off-by: moddingg33k <moddingg33k@t-online.de>

fs: ext4: reverting huawei specific changes

      Signed-off-by: moddingg33k <moddingg33k@t-online.de>

msm: acpuclock: Add a null check for acpuclk_data

      If acpuclock fails to probe before a client calls acpuclk_get_rate,
      a null dereference will occur. Fix this by adding the appropriate
      checks.

      Change-Id: If873a64711e9cc807b1734996f7c9358bb08161a
      Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>

Revert "ARM: armv7 no need update SCTLR for alignment"

      This reverts commit 6dfaacc2dc708f11863d144ac41882af4315d4df.

PM / tracing: remove deprecated power trace API

      The text in Documentation said it would be removed in 2.6.41;
      the text in the Kconfig said removal in the 3.1 release.  Either
      way you look at it, we are well past both, so push it off a cliff.

      Note that the POWER_CSTATE and the POWER_PSTATE are part of the
      legacy tracing API.  Remove all tracepoints which use these flags.
      As can be seen from context, most already have a trace entry via
      trace_cpu_idle anyways.

      Also, the cpufreq/cpufreq.c PSTATE one is actually unpaired, as
      compared to the CSTATE ones which all have a clear start/stop.
      As part of this, the trace_power_frequency also becomes orphaned,
      so it too is deleted.

      Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
      Acked-by: Steven Rostedt <rostedt@goodmis.org>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

cpufreq: ondemand: reverted huawei changes

Conflicts:
	arch/arm/mach-msm/Kconfig
	drivers/cpufreq/cpufreq_ondemand.c