[ARM: CRYPTO]: optimized AES and SHA1 routines & NEON accelerated XOR

arm/crypto: Add optimized AES and SHA1 routines

      Add assembler versions of AES and SHA1 for ARM platforms.  This has provided
      up to a 50% improvement in IPsec/TCP throughout for tunnels using AES128/SHA1.

      Platform   CPU SPeed    Endian   Before (bps)   After (bps)   Improvement

      IXP425      533 MHz      big     11217042        15566294        ~38%
      KS8695      166 MHz     little    3828549         5795373        ~51%

      Signed-off-by: David McCullough <ucdevel@gmail.com>
      Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

ARM: 7626/1: arm/crypto: Make asm SHA-1 and AES code Thumb-2 compatible

      This patch fixes aes-armv4.S and sha1-armv4-large.S to work
      natively in Thumb.  This allows ARM/Thumb interworking workarounds
      to be removed.

      I also take the opportunity to convert some explicit assembler
      directives for exported functions to the standard
      ENTRY()/ENDPROC().

      For the code itself:

	* In sha1_block_data_order, use of TEQ with sp is deprecated in
	  ARMv7 and not supported in Thumb.  For the branches back to
	  .L_00_15 and .L_40_59, the TEQ is converted to a CMP, under the
	  assumption that clobbering the C flag here will not cause
	  incorrect behaviour.

	  For the first branch back to .L_20_39_or_60_79 the C flag is
	  important, so sp is moved temporarily into another register so
	  that TEQ can be used for the comparison.

	* In the AES code, most forms of register-indexed addressing with
	  shifts and rotates are not permitted for loads and stores in
	  Thumb, so the address calculation is done using a separate
	  instruction for the Thumb case.

      The resulting code is unlikely to be optimally scheduled, but it
      should not have a large impact given the overall size of the code.
      I haven't run any benchmarks.

      Signed-off-by: Dave Martin <dave.martin@linaro.org>
      Tested-by: David McCullough <ucdevel@gmail.com> (ARM only)
      Acked-by: David McCullough <ucdevel@gmail.com>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7723/1: crypto: sha1-armv4-large.S: fix SP handling

      Make the SHA1 asm code ABI conformant by making sure all stack
      accesses occur above the stack pointer.

      Origin:
      http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=1a9d60d2

      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Cc: stable@vger.kernel.org
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7837/3: fix Thumb-2 bug in AES assembler code

      Patch 638591c enabled building the AES assembler code in Thumb2 mode.
      However, this code used arithmetic involving PC rather than adr{l}
      instructions to generate PC-relative references to the lookup tables,
      and this needs to take into account the different PC offset when
      running in Thumb mode.

      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Cc: stable@vger.kernel.org
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

      ARM: move AES typedefs and function prototypes to separate header

      Put the struct definitions for AES keys and the asm function prototypes in a
      separate header and export the asm functions from the module.
      This allows other drivers to use them directly.

      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

      ARM: add support for bit sliced AES using NEON instructions

      Bit sliced AES gives around 45% speedup on Cortex-A15 for encryption
      and around 25% for decryption. This implementation of the AES algorithm
      does not rely on any lookup tables so it is believed to be invulnerable
      to cache timing attacks.

      This algorithm processes up to 8 blocks in parallel in constant time. This
      means that it is not usable by chaining modes that are strictly sequential
      in nature, such as CBC encryption. CBC decryption, however, can benefit from
      this implementation and runs about 25% faster. The other chaining modes
      implemented in this module, XTS and CTR, can execute fully in parallel in
      both directions.

      The core code has been adopted from the OpenSSL project (in collaboration
      with the original author, on cc). For ease of maintenance, this version is
      identical to the upstream OpenSSL code, i.e., all modifications that were
      required to make it suitable for inclusion into the kernel have been made
      upstream. The original can be found here:

	  http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6f6a6130

      Note to integrators:
      While this implementation is significantly faster than the existing table
      based ones (generic or ARM asm), especially in CTR mode, the effects on
      power efficiency are unclear as of yet. This code does fundamentally more
      work, by calculating values that the table based code obtains by a simple
      lookup; only by doing all of that work in a SIMD fashion, it manages to
      perform better.

      Cc: Andy Polyakov <appro@openssl.org>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

ARM: add .gitignore entry for aesbs-core.S

      This avoids this file being incorrectly added to git.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

CRYPTO: Fix more AES build errors

      Building a multi-arch kernel results in:

      arch/arm/crypto/built-in.o: In function `aesbs_xts_decrypt':
      sha1_glue.c:(.text+0x15c8): undefined reference to `bsaes_xts_decrypt'
      arch/arm/crypto/built-in.o: In function `aesbs_xts_encrypt':
      sha1_glue.c:(.text+0x1664): undefined reference to `bsaes_xts_encrypt'
      arch/arm/crypto/built-in.o: In function `aesbs_ctr_encrypt':
      sha1_glue.c:(.text+0x184c): undefined reference to `bsaes_ctr32_encrypt_blocks'
      arch/arm/crypto/built-in.o: In function `aesbs_cbc_decrypt':
      sha1_glue.c:(.text+0x19b4): undefined reference to `bsaes_cbc_encrypt'

      This code is already runtime-conditional on NEON being supported, so
      there's no point compiling it out depending on the minimum build
      architecture.

      Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

crypto: create generic version of ablk_helper

      Create a generic version of ablk_helper so it can be reused
      by other architectures.

      Acked-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

---------------------------------------------------------------------------------------------------

ARM: crypto: add NEON accelerated XOR implementation

      Add a source file xor-neon.c (which is really just the reference
      C implementation passed through the GCC vectorizer) and hook it
      up to the XOR framework.

      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Acked-by: Nicolas Pitre <nico@linaro.org>

ARM: 7835/2: fix modular build of xor_blocks() with NEON enabled

      Commit 0195659 introduced a NEON accelerated version of the xor_blocks()
      function, but it needs the changes in this patch to allow it to be built
      as a module rather than statically into the kernel.

      This patch creates a separate module xor-neon.ko which exports the NEON
      inner xor_blocks() functions depended upon by the regular xor.ko if it
      is built with CONFIG_KERNEL_MODE_NEON=y

      Reported-by: Josh Boyer <jwboyer@fedoraproject.org>
      Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

Optimized ARM RWSEM algorithm

      RWSEM implementation for ARM using atomic functions.
      Heavily based on arch/sh/include/asm/rwsem.h

      Signed-off-by: Ashwin Chaugule <ashwinc@codeaurora.org>
[ARM: MM] implement LoUIS API for cache maintenance ops

ARM: mm: implement LoUIS API for cache maintenance ops

      ARM v7 architecture introduced the concept of cache levels and related
      control registers. New processors like A7 and A15 embed an L2 unified cache
      controller that becomes part of the cache level hierarchy. Some operations in
      the kernel like cpu_suspend and __cpu_disable do not require a flush of the
      entire cache hierarchy to DRAM but just the cache levels belonging to the
      Level of Unification Inner Shareable (LoUIS), which in most of ARM v7 systems
      correspond to L1.

      The current cache flushing API used in cpu_suspend and __cpu_disable,
      flush_cache_all(), ends up flushing the whole cache hierarchy since for
      v7 it cleans and invalidates all cache levels up to Level of Coherency
      (LoC) which cripples system performance when used in hot paths like hotplug
      and cpuidle.

      Therefore a new kernel cache maintenance API must be added to cope with
      latest ARM system requirements.

      This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.

      This function cleans and invalidates all data cache levels up to the
      Level of Unification Inner Shareable (LoUIS) and invalidates the instruction
      cache for processors that support it (> v7).

      This patch also creates an alias of the cache LoUIS function to flush_kern_all
      for all processor versions prior to v7, so that the current cache flushing
      behaviour is unchanged for those processors.

      v7 cache maintenance code implements a cache LoUIS function that cleans and
      invalidates the D-cache up to LoUIS and invalidates the I-cache, according
      to the new API.

      Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: mm: rename jump labels in v7_flush_dcache_all function

      This patch renames jump labels in v7_flush_dcache_all in order to define
      a specific flush cache levels entry point.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: kernel: update cpu_suspend code to use cache LoUIS operations

      In processors like A15/A7 L2 cache is unified and integrated within the
      processor cache hierarchy, so that it is not considered an outer cache
      anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
      all cache levels up to Level of Coherency (LoC) that includes
      the L2 unified cache.

      When a single CPU is suspended (CPU idle) a complete L2 clean is not
      required, so generic cpu_suspend code must clean the data cache using the
      newly introduced cache LoUIS function.

      The context and stack pointer (context pointer) are cleaned to main memory
      using cache area functions that operate on MVA and guarantee that the data
      is written back to main memory (perform cache cleaning up to the Point of
      Coherency - PoC) so that the processor can fetch the context when the MMU
      is off in the cpu_resume code path.

      outer_cache management remains unchanged.

      Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: kernel: update __cpu_disable to use cache LoUIS maintenance API

      When a CPU is hotplugged out caches that reside in its power domain
      lose their contents and so must be cleaned to the next memory level.

      Currently, __cpu_disable calls flush_cache_all() that for new generation
      processor like A15/A7 ends up cleaning and invalidating all cache levels
      up to Level of Coherency, which includes the unified L2.

      This ends up being a waste of cycles since the L2 cache contents are not
      lost on power down.

      This patch updates __cpu_disable to use the new LoUIS API cache operations.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>

ARM: mm: update __v7_setup() to the new LoUIS cache maintenance API

      The ARMv7 processor setup function __v7_setup() cleans and invalidates the
      CPU cache before enabling MMU to start the CPU with a clean CPU local cache.

      But on ARMv7 architectures like Cortex-[A15/A8], this code will end
      up flushing the L2 caches(up to level of Coherency) which is undesirable
      and expensive. The setup functions are used in the CPU hotplug scenario too
      and hence flushing all cache levels should be avoided.

      This patch replaces the cache flushing call with the newly introduced
      v7 dcache LoUIS API where only cache levels up to LoUIS are cleaned and
      invalidated when a processors executes __v7_setup which is the expected
      behavior.

      For processors like A9 and A5 where the L2 cache is an outer one the
      behavior should be unchanged.

      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Tested-by: Shawn Guo <shawn.guo@linaro.org>
[ARM: GENERIC] *squashing updates*

ARM: user generic termios.h

ARM: 7493/1: use generic unaligned.h

      This moves ARM over to the asm-generic/unaligned.h header. This has the
      benefit of better code generated especially for ARMv7 on gcc 4.7+
      compilers.

      As Arnd Bergmann, points out: The asm-generic version uses the "struct"
      version for native-endian unaligned access and the "byteshift" version
      for the opposite endianess. The current ARM version however uses the
      "byteshift" implementation for both.

      Thanks to Nicolas Pitre for the excellent analysis:

      Test case:

      int foo (int *x) { return get_unaligned(x); }
      long long bar (long long *x) { return get_unaligned(x); }

      With the current ARM version:

      foo:
	      ldrb	r3, [r0, #2]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 2B], MEM[(const u8 *)x_1(D) + 2B]
	      ldrb	r1, [r0, #1]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 1B], MEM[(const u8 *)x_1(D) + 1B]
	      ldrb	r2, [r0, #0]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D)], MEM[(const u8 *)x_1(D)]
	      mov	r3, r3, asl #16	@ tmp154, MEM[(const u8 *)x_1(D) + 2B],
	      ldrb	r0, [r0, #3]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 3B], MEM[(const u8 *)x_1(D) + 3B]
	      orr	r3, r3, r1, asl #8	@, tmp155, tmp154, MEM[(const u8 *)x_1(D) + 1B],
	      orr	r3, r3, r2	@ tmp157, tmp155, MEM[(const u8 *)x_1(D)]
	      orr	r0, r3, r0, asl #24	@,, tmp157, MEM[(const u8 *)x_1(D) + 3B],
	      bx	lr	@

      bar:
	      stmfd	sp!, {r4, r5, r6, r7}	@,
	      mov	r2, #0	@ tmp184,
	      ldrb	r5, [r0, #6]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 6B], MEM[(const u8 *)x_1(D) + 6B]
	      ldrb	r4, [r0, #5]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 5B], MEM[(const u8 *)x_1(D) + 5B]
	      ldrb	ip, [r0, #2]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 2B], MEM[(const u8 *)x_1(D) + 2B]
	      ldrb	r1, [r0, #4]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 4B], MEM[(const u8 *)x_1(D) + 4B]
	      mov	r5, r5, asl #16	@ tmp175, MEM[(const u8 *)x_1(D) + 6B],
	      ldrb	r7, [r0, #1]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 1B], MEM[(const u8 *)x_1(D) + 1B]
	      orr	r5, r5, r4, asl #8	@, tmp176, tmp175, MEM[(const u8 *)x_1(D) + 5B],
	      ldrb	r6, [r0, #7]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 7B], MEM[(const u8 *)x_1(D) + 7B]
	      orr	r5, r5, r1	@ tmp178, tmp176, MEM[(const u8 *)x_1(D) + 4B]
	      ldrb	r4, [r0, #0]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D)], MEM[(const u8 *)x_1(D)]
	      mov	ip, ip, asl #16	@ tmp188, MEM[(const u8 *)x_1(D) + 2B],
	      ldrb	r1, [r0, #3]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 3B], MEM[(const u8 *)x_1(D) + 3B]
	      orr	ip, ip, r7, asl #8	@, tmp189, tmp188, MEM[(const u8 *)x_1(D) + 1B],
	      orr	r3, r5, r6, asl #24	@,, tmp178, MEM[(const u8 *)x_1(D) + 7B],
	      orr	ip, ip, r4	@ tmp191, tmp189, MEM[(const u8 *)x_1(D)]
	      orr	ip, ip, r1, asl #24	@, tmp194, tmp191, MEM[(const u8 *)x_1(D) + 3B],
	      mov	r1, r3	@,
	      orr	r0, r2, ip	@ tmp171, tmp184, tmp194
	      ldmfd	sp!, {r4, r5, r6, r7}
	      bx	lr

      In both cases the code is slightly suboptimal.  One may wonder why
      wasting r2 with the constant 0 in the second case for example.  And all
      the mov's could be folded in subsequent orr's, etc.

      Now with the asm-generic version:

      foo:
	      ldr	r0, [r0, #0]	@ unaligned	@,* x
	      bx	lr	@

      bar:
	      mov	r3, r0	@ x, x
	      ldr	r0, [r0, #0]	@ unaligned	@,* x
	      ldr	r1, [r3, #4]	@ unaligned	@,
	      bx	lr	@

      This is way better of course, but only because this was compiled for
      ARMv7. In this case the compiler knows that the hardware can do
      unaligned word access.  This isn't that obvious for foo(), but if we
      remove the get_unaligned() from bar as follows:

      long long bar (long long *x) {return *x; }

      then the resulting code is:

      bar:
	      ldmia	r0, {r0, r1}	@ x,,
	      bx	lr	@

      So this proves that the presumed aligned vs unaligned cases does have
      influence on the instructions the compiler may use and that the above
      unaligned code results are not just an accident.

      Still... this isn't fully conclusive without at least looking at the
      resulting assembly fron a pre ARMv6 compilation.  Let's see with an
      ARMv5 target:

      foo:
	      ldrb	r3, [r0, #0]	@ zero_extendqisi2	@ tmp139,* x
	      ldrb	r1, [r0, #1]	@ zero_extendqisi2	@ tmp140,
	      ldrb	r2, [r0, #2]	@ zero_extendqisi2	@ tmp143,
	      ldrb	r0, [r0, #3]	@ zero_extendqisi2	@ tmp146,
	      orr	r3, r3, r1, asl #8	@, tmp142, tmp139, tmp140,
	      orr	r3, r3, r2, asl #16	@, tmp145, tmp142, tmp143,
	      orr	r0, r3, r0, asl #24	@,, tmp145, tmp146,
	      bx	lr	@

      bar:
	      stmfd	sp!, {r4, r5, r6, r7}	@,
	      ldrb	r2, [r0, #0]	@ zero_extendqisi2	@ tmp139,* x
	      ldrb	r7, [r0, #1]	@ zero_extendqisi2	@ tmp140,
	      ldrb	r3, [r0, #4]	@ zero_extendqisi2	@ tmp149,
	      ldrb	r6, [r0, #5]	@ zero_extendqisi2	@ tmp150,
	      ldrb	r5, [r0, #2]	@ zero_extendqisi2	@ tmp143,
	      ldrb	r4, [r0, #6]	@ zero_extendqisi2	@ tmp153,
	      ldrb	r1, [r0, #7]	@ zero_extendqisi2	@ tmp156,
	      ldrb	ip, [r0, #3]	@ zero_extendqisi2	@ tmp146,
	      orr	r2, r2, r7, asl #8	@, tmp142, tmp139, tmp140,
	      orr	r3, r3, r6, asl #8	@, tmp152, tmp149, tmp150,
	      orr	r2, r2, r5, asl #16	@, tmp145, tmp142, tmp143,
	      orr	r3, r3, r4, asl #16	@, tmp155, tmp152, tmp153,
	      orr	r0, r2, ip, asl #24	@,, tmp145, tmp146,
	      orr	r1, r3, r1, asl #24	@,, tmp155, tmp156,
	      ldmfd	sp!, {r4, r5, r6, r7}
	      bx	lr

      Compared to the initial results, this is really nicely optimized and I
      couldn't do much better if I were to hand code it myself.

      Signed-off-by: Rob Herring <rob.herring@calxeda.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Tested-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
      Reviewed-by: Arnd Bergmann <arnd@arndb.de>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

arm: Add XN bit to pgrprot_stronglyordered

      pgprot_stronglyordered sets page attributes to the strongly
      ordered memory type. This type is often used for regions of memory which
      cannot be speculated into. Strongly ordered memory by itself prevents
      speculative data accesses but it will not prevent speculative instruction
      accesses. Since strongly ordered memory will never be used for
      instructions, set the XN bit as well to prevent speculative instruction
      fethces.

      Bug: 10750953
      Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
      Signed-off-by: Mekala Natarajan <mekalan@codeaurora.org>

ARM: 7495/1: mutex: use generic atomic_dec-based implementation for ARMv6+

      Commit a76d7bd96d65 ("ARM: 7467/1: mutex: use generic xchg-based
      implementation for ARMv6+") removed the barrier-less, ARM-specific
      mutex implementation in favour of the generic xchg-based code.

      Since then, a bug was uncovered in the xchg code when running on SMP
      platforms, due to interactions between the locking paths and the
      MUTEX_SPIN_ON_OWNER code. This was fixed in 0bce9c46bf3b ("mutex: place
      lock in contended state after fastpath_lock failure"), however, the
      atomic_dec-based mutex algorithm is now marginally more efficient for
      ARM (~0.5% improvement in hackbench scores on dual A15).

      This patch moves ARMv6+ platforms to the atomic_dec-based mutex code.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7587/1: implement optimized percpu variable access

      Use the previously unused TPIDRPRW register to store percpu offsets.
      TPIDRPRW is only accessible in PL1, so it can only be used in the kernel.

      This replaces 2 loads with a mrc instruction for each percpu variable
      access. With hackbench, the performance improvement is 1.4% on Cortex-A9
      (highbank). Taking an average of 30 runs of "hackbench -l 1000" yields:

      Before: 6.2191
      After: 6.1348

      Will Deacon reported similar delta on v6 with 11MPCore.

      The asm "memory clobber" are needed here to ensure the percpu offset
      gets reloaded. Testing by Will found that this would not happen in
      __schedule() which is a bit of a special case as preemption is disabled
      but the execution can move cores.

      Signed-off-by: Rob Herring <rob.herring@calxeda.com>
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7747/1: pcpu: ensure __my_cpu_offset cannot be re-ordered across barrier()

      __my_cpu_offset is non-volatile, since we want its value to be cached
      when we access several per-cpu variables in a row with preemption
      disabled. This means that we rely on preempt_{en,dis}able to hazard
      with the operation via the barrier() macro, so that we can't end up
      migrating CPUs without reloading the per-cpu offset.

      Unfortunately, GCC doesn't treat a "memory" clobber on a non-volatile
      asm block as a side-effect, and will happily re-order it before other
      memory clobbers (including those in prempt_disable()) and cache the
      value. This has been observed to break the cmpxchg logic in the slub
      allocator, leading to livelock in kmem_cache_alloc in mainline kernels.

      This patch adds a dummy memory input operand to __my_cpu_offset,
      forcing it to be ordered with respect to the barrier() macro.

      Cc: <stable@vger.kernel.org>
      Cc: Rob Herring <rob.herring@calxeda.com>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: smp: flush L1 cache in cpu_die()

      Flush the L1 cache for the CPU which is going down in cpu_die() so
      that we don't end up with all platforms doing this.  This ensures
      that any cache lines we own are pushed out before the cache becomes
      inaccessible.

      We may end up subsequently creating some dirty cache lines - for
      example, with the complete() call, but this update must become
      visible to other CPUs before __cpu_die() can proceed.  Subsequent
      accesses from the platforms cpu_die() function should _not_ matter.

      Also place a mb() after the complete() call to ensure that this is
      visible to other CPUs.

      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7590/1: /proc/interrupts: limit the display of IPIs to online CPUs only

      This is what is done for the regular interrupts in kernel/irqs/proc.c
      already, before calling arch_show_interrupts().  Not doing so for the
      IPIs causes the column headers not to match with the content whenever
      some CPUs are offline.

      Signed-off-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

vsprintf: further optimize decimal conversion

      Previous code was using optimizations which were developed to work well
      even on narrow-word CPUs (by today's standards).  But Linux runs only on
      32-bit and wider CPUs.  We can use that.

      First: using 32x32->64 multiply and trivial 32-bit shift, we can correctly
      divide by 10 much larger numbers, and thus we can print groups of 9 digits
      instead of groups of 5 digits.

      Next: there are two algorithms to print larger numbers.  One is generic:
      divide by 1000000000 and repeatedly print groups of (up to) 9 digits.
      It's conceptually simple, but requires an (unsigned long long) /
      1000000000 division.

      Second algorithm splits 64-bit unsigned long long into 16-bit chunks,
      manipulates them cleverly and generates groups of 4 decimal digits.  It so
      happens that it does NOT require long long division.

      If long is > 32 bits, division of 64-bit values is relatively easy, and we
      will use the first algorithm.  If long long is > 64 bits (strange
      architecture with VERY large long long), second algorithm can't be used,
      and we again use the first one.

      Else (if long is 32 bits and long long is 64 bits) we use second one.

      And third: there is a simple optimization which takes fast path not only
      for zero as was done before, but for all one-digit numbers.

      In all tested cases new code is faster than old one, in many cases by 30%,
      in few cases by more than 50% (for example, on x86-32, conversion of
      12345678).  Code growth is ~0 in 32-bit case and ~130 bytes in 64-bit
      case.

      This patch is based upon an original from Michal Nazarewicz.

      [akpm@linux-foundation.org: checkpatch fixes]
      Signed-off-by: Michal Nazarewicz <mina86@mina86.com>
      Signed-off-by: Denys Vlasenko <vda.linux@googlemail.com>
      Cc: Douglas W Jones <jones@cs.uiowa.edu>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

ARM: bitops: prefetch the destination word for write prior to strex

      The cost of changing a cacheline from shared to exclusive state can be
      significant, especially when this is triggered by an exclusive store,
      since it may result in having to retry the transaction.

      This patch prefixes our atomic bitops implementation with prefetchw,
      to try and grab the line in exclusive state from the start. The testop
      macro is left alone, since the barrier semantics limit the usefulness
      of prefetching data.

      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7858/1: mm: make UACCESS_WITH_MEMCPY huge page aware

      The memory pinning code in uaccess_with_memcpy.c does not check
      for HugeTLB or THP pmds, and will enter an infinite loop should
      a __copy_to_user or __clear_user occur against a huge page.

      This patch adds detection code for huge pages to pin_page_for_write.
      As this code can be executed in a fast path it refers to the actual
      pmds rather than the vma. If a HugeTLB or THP is found (they have
      the same pmd representation on ARM), the page table spinlock is
      taken to prevent modification whilst the page is pinned.

      On ARM, huge pages are only represented as pmds, thus no huge pud
      checks are performed. (For huge puds one would lock the page table
      in a similar manner as in the pmd case).

      Two helper functions are introduced; pmd_thp_or_huge will check
      whether or not a page is huge or transparent huge (which have the
      same pmd layout on ARM), and pmd_hugewillfault will detect whether
      or not a page fault will occur on write to the page.

      Running the following test (with the chunking from read_zero
      removed):
      $ dd if=/dev/zero of=/dev/null bs=10M count=1024
      Gave:  2.3 GB/s backed by normal pages,
	    2.9 GB/s backed by huge pages,
	    5.1 GB/s backed by huge pages, with page mask=HPAGE_MASK.

      After some discussion, it was decided not to adopt the HPAGE_MASK,
      as this would have a significant detrimental effect on the overall
      system latency due to page_table_lock being held for too long.
      This could be revisited if split huge page locks are adopted.

      Signed-off-by: Steve Capper <steve.capper@linaro.org>
      Reviewed-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: mm: use inner-shareable barriers for TLB and user cache operations

      System-wide barriers aren't required for situations where we only need
      to make visibility and ordering guarantees in the inner-shareable domain
      (i.e. we are not dealing with devices or potentially incoherent CPUs).

      This patch changes the v7 TLB operations, coherent_user_range and
      dcache_clean_area functions to user inner-shareable barriers. For cache
      maintenance, only the store access type is required to ensure completion.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7919/1: mm: refactor v7 cache cleaning ops to use way/index sequence

      Set-associative caches on all v7 implementations map the index bits
      to physical addresses LSBs and tag bits to MSBs. As the last level
      of cache on current and upcoming ARM systems grows in size,
      this means that under normal DRAM controller configurations, the
      current v7 cache flush routine using set/way operations triggers a
      DRAM memory controller precharge/activate for every cache line
      writeback since the cache routine cleans lines by first fixing the
      index and then looping through ways (index bits are mapped to lower
      physical addresses on all v7 cache implementations; this means that,
      with last level cache sizes in the order of MBytes, lines belonging
      to the same set but different ways map to different DRAM pages).

      Given the random content of cache tags, swapping the order between
      indexes and ways loops do not prevent DRAM pages precharge and
      activate cycles but at least, on average, improves the chances that
      either multiple lines hit the same page or multiple lines belong to
      different DRAM banks, improving throughput significantly.

      This patch swaps the inner loops in the v7 cache flushing routine
      to carry out the clean operations first on all sets belonging to
      a given way (looping through sets) and then decrementing the way.

      Benchmarks showed that by swapping the ordering in which sets and
      ways are decremented in the v7 cache flushing routine, that uses
      set/way operations, time required to flush caches is reduced
      significantly, owing to improved writebacks throughput to the DRAM
      controller.

      Benchmarks results vary and depend heavily on the last level of
      cache tag RAM content when cache is cleaned and invalidated, ranging
      from 2x throughput when all tag RAM entries contain dirty lines
      mapping to sequential pages of RAM to 1x (ie no improvement) when
      all tag RAM accesses trigger a DRAM precharge/activate cycle, as the
      current code implies on most DRAM controller configurations.

      Acked-by: Catalin Marinas <catalin.marinas@arm.com>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
      Reviewed-by: Dave Martin <Dave.Martin@arm.com>
      Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7691/1: mm: kill unused TLB_CAN_READ_FROM_L1_CACHE and use ALT_SMP instead

      Many ARMv7 cores have hardware page table walkers that can read the L1
      cache. This is discoverable from the ID_MMFR3 register, although this
      can be expensive to access from the low-level set_pte functions and is a
      pain to cache, particularly with multi-cluster systems.

      A useful observation is that the multi-processing extensions for ARMv7
      require coherent table walks, meaning that we can make use of ALT_SMP
      patching in proc-v7-* to patch away the cache flush safely for these
      cores.

      Reported-by: Albin Tonnerre <Albin.Tonnerre@arm.com>
      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7784/1: mm: ensure SMP alternates assemble to exactly 4 bytes with Thumb-2

      Commit ae8a8b9553bd ("ARM: 7691/1: mm: kill unused TLB_CAN_READ_FROM_L1_CACHE
      and use ALT_SMP instead") added early function returns for page table
      cache flushing operations on ARMv7 SMP CPUs.

      Unfortunately, when targetting Thumb-2, these `mov pc, lr' sequences
      assemble to 2 bytes which can lead to corruption of the instruction
      stream after code patching.

      This patch fixes the alternates to use wide (32-bit) instructions for
      Thumb-2, therefore ensuring that the patching code works correctly.

      Cc: <stable@vger.kernel.org>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7553/1: proc-v7: Ensure correct instruction set after cpu_reset

      Because mov pc,<Rn> never switches instruction set when executed in
      Thumb code, Thumb-2 kernels will silently execute the target code
      after cpu_reset as Thumb code, even if the passed code pointer
      denotes ARM (bit 0 clear).

      This patch uses bx instead, ensuring the correct instruction set
      for the target code.

      Thumb code in the kernel is not supported prior to ARMv7, so other
      CPUs are not affected.

      Signed-off-by: Dave Martin <dave.martin@linaro.org>
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Nicolas Pitre <nico@linaro.org>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: Use TTBR1 instead of reserved context ID

      On ARMv7 CPUs that cache first level page table entries (like the
      Cortex-A15), using a reserved ASID while changing the TTBR or flushing
      the TLB is unsafe.

      This is because the CPU may cache the first level entry as the result of
      a speculative memory access while the reserved ASID is assigned. After
      the process owning the page tables dies, the memory will be reallocated
      and may be written with junk values which can be interpreted as global,
      valid PTEs by the processor. This will result in the TLB being populated
      with bogus global entries.

      This patch avoids the use of a reserved context ID in the v7 switch_mm
      and ASID rollover code by temporarily using the swapper_pg_dir pointed
      at by TTBR1, which contains only global entries that are not tagged
      with ASIDs.

      Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
      Tested-by: Marc Zyngier <Marc.Zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      [catalin.marinas@arm.com: add LPAE support]
      Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on ASID-capable CPUs

      Since the ASIDs must be unique to an mm across all the CPUs in a system,
      the __new_context() function needs to broadcast a context reset event to
      all the CPUs during ASID allocation if a roll-over occurred. Such IPIs
      cannot be issued with interrupts disabled and ARM had to define
      __ARCH_WANT_INTERRUPTS_ON_CTXSW.

      This patch changes the check_context() function to
      check_and_switch_context() called from switch_mm(). In case of
      ASID-capable CPUs (ARMv6 onwards), if a new ASID is needed and the
      interrupts are disabled, it defers the __new_context() and
      cpu_switch_mm() calls to the post-lock switch hook where the interrupts
      are enabled. Setting the reserved TTBR0 was also moved to
      check_and_switch_context() from cpu_v7_switch_mm().

      Reviewed-by: Will Deacon <will.deacon@arm.com>
      Tested-by: Will Deacon <will.deacon@arm.com>
      Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
      Tested-by: Marc Zyngier <Marc.Zyngier@arm.com>
      Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

ARM: Remove current_mm per-cpu variable

      The current_mm variable was used to store the new mm between the
      switch_mm() and switch_to() calls where an IPI to reset the context
      could have set the wrong mm. Since the interrupts are disabled during
      context switch, there is no need for this variable, current->active_mm
      already points to the current mm when interrupts are re-enabled.

      Reviewed-by: Will Deacon <will.deacon@arm.com>
      Tested-by: Will Deacon <will.deacon@arm.com>
      Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
      Tested-by: Marc Zyngier <Marc.Zyngier@arm.com>
      Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on pre-ARMv6 CPUs

      This patch removes the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition for
      ARMv5 and earlier processors. On such processors, the context switch
      requires a full cache flush. To avoid high interrupt latencies, this
      patch defers the mm switching to the post-lock switch hook if the
      interrupts are disabled.

      Reviewed-by: Will Deacon <will.deacon@arm.com>
      Tested-by: Will Deacon <will.deacon@arm.com>
      Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
      Tested-by: Marc Zyngier <Marc.Zyngier@arm.com>
      Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

ARM: mm: introduce L_PTE_VALID for page table entries

      For long-descriptor translation table formats, the ARMv7 architecture
      defines the last two bits of the second- and third-level descriptors to
      be:

	      x0b	- Invalid
	      01b	- Block (second-level), Reserved (third-level)
	      11b	- Table (second-level), Page (third-level)

      This allows us to define L_PTE_PRESENT as (3 << 0) and use this value to
      create ptes directly. However, when determining whether a given pte
      value is present in the low-level page table accessors, we only need to
      check the least significant bit of the descriptor, allowing us to write
      faulting, present entries which are required for PROT_NONE mappings.

      This patch introduces L_PTE_VALID, which can be used to test whether a
      pte should fault, and updates the low-level page table accessors
      accordingly.

      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: mm: introduce present, faulting entries for PAGE_NONE

      PROT_NONE mappings apply the page protection attributes defined by _P000
      which translate to PAGE_NONE for ARM. These attributes specify an XN,
      RDONLY pte that is inaccessible to userspace. However, on kernels
      configured without support for domains, such a pte *is* accessible to
      the kernel and can be read via get_user, allowing tasks to read
      PROT_NONE pages via syscalls such as read/write over a pipe.

      This patch introduces a new software pte flag, L_PTE_NONE, that is set
      to identify faulting, present entries.

      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7650/1: mm: replace direct access to mm->context.id with new macro

      The mmid macro is meant to be used to get the mm->context.id data
      from the mm structure, but it seems to have been missed in a cuple
      of files.

      Acked-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
[ARM: GENERIC *part2*] *squashing updates*

ARM: mm: remove IPI broadcasting on ASID rollover

      ASIDs are allocated to MMU contexts based on a rolling counter. This
      means that after 255 allocations we must invalidate all existing ASIDs
      via an expensive IPI mechanism to synchronise all of the online CPUs and
      ensure that all tasks execute with an ASID from the new generation.

      This patch changes the rollover behaviour so that we rely instead on the
      hardware broadcasting of the TLB invalidation to avoid the IPI calls.
      This works by keeping track of the active ASID on each core, which is
      then reserved in the case of a rollover so that currently scheduled
      tasks can continue to run. For cores without hardware TLB broadcasting,
      we keep track of pending flushes in a cpumask, so cores can flush their
      local TLB before scheduling a new mm.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Tested-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: mm: avoid taking ASID spinlock on fastpath

      When scheduling a new mm, we take a spinlock so that we can:

	1. Safely allocate a new ASID, if required
	2. Update our active_asids field without worrying about parallel
	  updates to reserved_asids
	3. Ensure that we flush our local TLB, if required

      However, this has the nasty affect of serialising context-switch across
      all CPUs in the system. The usual (fast) case is where the next mm has
      a valid ASID for the current generation. In such a scenario, we can
      avoid taking the lock and instead use atomic64_xchg to update the
      active_asids variable for the current CPU. If a rollover occurs on
      another CPU (which would take the lock), when copying the active_asids
      into the reserved_asids another atomic64_xchg is used to replace each
      active_asids with 0. The fast path can then detect this case and fall
      back to spinning on the lock.

      Tested-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: mm: use bitmap operations when allocating new ASIDs

      When allocating a new ASID, we must take care not to re-assign a
      reserved ASID-value to a new mm. This requires us to check each
      candidate ASID against those currently reserved by other cores before
      assigning a new ASID to the current mm.

      This patch improves the ASID allocation algorithm by using a
      bitmap-based approach. Rather than iterating over the reserved ASID
      array for each candidate ASID, we simply find the first zero bit,
      ensuring that those indices corresponding to reserved ASIDs are set
      when flushing during a rollover event.

      Tested-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7658/1: mm: fix race updating mm->context.id on ASID rollover

      If a thread triggers an ASID rollover, other threads of the same process
      must be made to wait until the mm->context.id for the shared mm_struct
      has been updated to new generation and associated book-keeping (e.g.
      TLB invalidation) has ben performed.

      However, there is a *tiny* window where both mm->context.id and the
      relevant active_asids entry are updated to the new generation, but the
      TLB flush has not been performed, which could allow another thread to
      return to userspace with a dirty TLB, potentially leading to data
      corruption. In reality this will never occur because one CPU would need
      to perform a context-switch in the time it takes another to do a couple
      of atomic test/set operations but we should plug the race anyway.

      This patch moves the active_asids update until after the potential TLB
      flush on context-switch.

      Cc: <stable@vger.kernel.org> # 3.8
      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7659/1: mm: make mm->context.id an atomic64_t variable

      mm->context.id is updated under asid_lock when a new ASID is allocated
      to an mm_struct. However, it is also read without the lock when a task
      is being scheduled and checking whether or not the current ASID
      generation is up-to-date.

      If two threads of the same process are being scheduled in parallel and
      the bottom bits of the generation in their mm->context.id match the
      current generation (that is, the mm_struct has not been used for ~2^24
      rollovers) then the non-atomic, lockless access to mm->context.id may
      yield the incorrect ASID.

      This patch fixes this issue by making mm->context.id and atomic64_t,
      ensuring that the generation is always read consistently. For code that
      only requires access to the ASID bits (e.g. TLB flushing by mm), then
      the value is accessed directly, which GCC converts to an ldrb.

      Cc: <stable@vger.kernel.org> # 3.8
      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7661/1: mm: perform explicit branch predictor maintenance when required

      The ARM ARM requires branch predictor maintenance if, for a given ASID,
      the instructions at a specific virtual address appear to change.

      From the kernel's point of view, that means:

	      - Changing the kernel's view of memory (e.g. switching to the
		identity map)
	      - ASID rollover (since ASIDs will be re-allocated to new tasks)

      This patch adds explicit branch predictor maintenance when either of the
      two conditions above are met.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7767/1: let the ASID allocator handle suspended animation

      When a CPU is running a process, the ASID for that process is
      held in a per-CPU variable (the "active ASIDs" array). When
      the ASID allocator handles a rollover, it copies the active
      ASIDs into a "reserved ASIDs" array to ensure that a process
      currently running on another CPU will continue to run unaffected.
      The active array is zero-ed to indicate that a rollover occurred.

      Because of this mechanism, a reserved ASID is only remembered for
      a single rollover. A subsequent rollover will completely refill
      the reserved ASIDs array.

      In a severely oversubscribed environment where a CPU can be
      prevented from running for extended periods of time (think virtual
      machines), the above has a horrible side effect:

      [P{a} denotes process P running with ASID a]

	      CPU-0		CPU-1

	      A{x}				[active = <x 0>]

	      [suspended]	runs B{y}	[active = <x y>]

					      [rollover:
					      active = <0 0>
					      reserved = <x y>]

			      runs B{y}	[active = <0 y>
					      reserved = <x y>]

					      [rollover:
					      active = <0 0>
					      reserved = <0 y>]

			      runs C{x}	[active = <0 x>]

	      [resumes]

	      runs A{x}

      At that stage, both A and C have the same ASID, with deadly
      consequences.

      The fix is to preserve reserved ASIDs across rollovers if
      the CPU doesn't have an active ASID when the rollover occurs.

      Cc: <stable@vger.kernel.org> # 3.9
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Catalin Carinas <catalin.marinas@arm.com>
      Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7768/1: prevent risks of out-of-bound access in ASID allocator

      On a CPU that never ran anything, both the active and reserved ASID
      fields are set to zero. In this case the ASID_TO_IDX() macro will
      return -1, which is not a very useful value to index a bitmap.

      Instead of trying to offset the ASID so that ASID #1 is actually
      bit 0 in the asid_map bitmap, just always ignore bit 0 and start
      the search from bit 1. This makes the code a bit more readable,
      and without risk of OoB access.

      Cc: <stable@vger.kernel.org> # 3.9
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Catalin Marinas <catalin.marinas@arm.com>
      Reported-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: tlb: don't perform inner-shareable invalidation for local TLB ops

      Inner-shareable TLB invalidation is typically more expensive than local
      (non-shareable) invalidation, so performing the broadcasting for
      local_flush_tlb_* operations is a waste of cycles and needlessly
      clobbers entries in the TLBs of other CPUs.

      This patch introduces __flush_tlb_* versions for many of the TLB
      invalidation functions, which only respect inner-shareable variants of
      the invalidation instructions when presented with the TLB_V7_UIS_FULL
      flag. The local version is also inlined to prevent SMP_ON_UP kernels
      from missing flushes, where the __flush variant would be called with
      the UP flags.

      This gains us around 0.5% in hackbench scores for a dual-core A15, but I
      would expect this to improve as more cores (and clusters) are added to
      the equation.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Reported-by: Albin Tonnerre <Albin.Tonnerre@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7660/1: tlb: add branch predictor maintenance operations

      The ARM architecture requires explicit branch predictor maintenance
      when updating an instruction stream for a given virtual address. In
      reality, this isn't so much of a burden because the branch predictor
      is flushed during the cache maintenance required to make the new
      instructions visible to the I-side of the processor.

      However, there are still some cases where explicit flushing is required,
      so add a local_bp_flush_all operation to deal with this.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: nommu: add stub local_flush_bp_all() for !CONFIG_MMUU

      Since the merging of Will's tlb-ops branch, specifically 89c7e4b8bbb3
      (ARM: 7661/1: mm: perform explicit branch predictor maintenance when required),
      building SMP without CONFIG_MMU has been broken.

      The local_flush_bp_all function is only called for operations related to
      changing the kernel's view of memory and ASID rollover - both of which are
      irrelevant to an !MMU kernel.

      This patch adds a stub local_flush_bp_all() function to the other tlb
      maintenance stubs and restores the ability to build an SMP !MMU kernel.

      Signed-off-by: Jonathan Austin <jonathan.austin@arm.com>
      Acked-by: Will Deacon <will.deacon@arm.com>

ARM: mm: remove redundant dsb() prior to range TLB invalidation

      The kernel TLB range invalidation functions already contain dsb
      instructions before and after the maintenance, so there is no need to
      introduce additional barriers.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: barrier: allow options to be passed to memory barrier instructions

      On ARMv7, the memory barrier instructions take an optional `option'
      field which can be used to constrain the effects of a memory barrier
      based on shareability and access type.

      This patch allows the caller to pass these options if required, and
      updates the smp_*() barriers to request inner-shareable barriers,
      affecting only stores for the _wmb variant. wmb() is also changed to
      use the -st version of dsb.

      Reported-by: Albin Tonnerre <albin.tonnerre@arm.com>
      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: tlb: reduce scope of barrier domains for TLB invalidation

      Our TLB invalidation routines may require a barrier before the
      maintenance (in order to ensure pending page table writes are visible to
      the hardware walker) and barriers afterwards (in order to ensure
      completion of the maintenance and visibility in the instruction stream).

      Whilst this is expensive, the cost can be reduced somewhat by reducing
      the scope of the barrier instructions:

	- The barrier before only needs to apply to stores (pte writes)
	- Local ops are required only to affect the non-shareable domain
	- Global ops are required only to affect the inner-shareable domain

      This patch makes these changes for the TLB flushing code.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: spinlock: use inner-shareable dsb variant prior to sev instruction

      When unlocking a spinlock, we use the sev instruction to signal other
      CPUs waiting on the lock. Since sev is not a memory access instruction,
      we require a dsb in order to ensure that the sev is not issued ahead
      of the store placing the lock in an unlocked state.

      However, as sev is only concerned with other processors in a
      multiprocessor system, we can restrict the scope of the preceding dsb
      to the inner-shareable domain. Furthermore, we can restrict the scope to
      consider only stores, since there are no independent loads on the unlock
      path.

      A side-effect of this change is that a spin_unlock operation no longer
      forces completion of pending TLB invalidation, something which we rely
      on when unlocking runqueues to ensure that CPU migration during TLB
      maintenance routines doesn't cause us to continue before the operation
      has completed.

      This patch adds the -ishst suffix to the ARMv7 definition of dsb_sev()
      and adds an inner-shareable dsb to the context-switch path when running
      a preemptible, SMP, v7 kernel.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: l2x0: use -st dsb option for ordering writel_relaxed with unlock

      writel_relaxed and spin_unlock are both store operations, so we only
      need to enforce store ordering in the dsb.

      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: cacheflush: use -ishst dsb variant for ensuring flush completion

      flush_cache_vmap contains a dsb to ensure that any cacheflushing
      operations to flush out newly written ptes have completed.

      This patch adds the -ishst option to the dsb, since that is all that is
      required for completing cacheflushing in the inner-shareable domain.

      Signed-off-by: Will Deacon <will.deacon@arm.com>
byteorder: allow arch to opt to use GCC intrinsics for byteswapping

Since GCC 4.4, there have been __builtin_bswap32() and __builtin_bswap16()
intrinsics. A __builtin_bswap16() came a little later (4.6 for PowerPC,
48 for other platforms).

By using these instead of the inline assembler that most architectures
have in their __arch_swabXX() macros, we let the compiler see what's
actually happening. The resulting code should be at least as good, and
much *better* in the cases where it can be combined with a nearby load
or store, using a load-and-byteswap or store-and-byteswap instruction
(e.g. lwbrx/stwbrx on PowerPC, movbe on Atom).

When GCC is sufficiently recent *and* the architecture opts in to using
the intrinsics by setting CONFIG_ARCH_USE_BUILTIN_BSWAP, they will be
used in preference to the __arch_swabXX() macros. An architecture which
does not set ARCH_USE_BUILTIN_BSWAP will continue to use its own
hand-crafted macros.

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
Acked-by: H. Peter Anvin <hpa@linux.intel.com>
binfmt_elf.c: use get_random_int() to fix entropy depleting.

Usually the entropy pool depletes very fast and stays in the 100-200 range.
After this patch, the entropy stays in the 600-1000 range without the risk of
depleting in normal workloads.

Change-Id: Ice0d0f14691046249d7b4a4d31af98ab80c6306b
introduce for_each_thread() to replace the buggy while_each_thread()

while_each_thread() and next_thread() should die, almost every lockless
usage is wrong.

1. Unless g == current, the lockless while_each_thread() is not safe.

   while_each_thread(g, t) can loop forever if g exits, next_thread()
   can't reach the unhashed thread in this case. Note that this can
   happen even if g is the group leader, it can exec.

2. Even if while_each_thread() itself was correct, people often use
   it wrongly.

   It was never safe to just take rcu_read_lock() and loop unless
   you verify that pid_alive(g) == T, even the first next_thread()
   can point to the already freed/reused memory.

This patch adds signal_struct->thread_head and task->thread_node to
create the normal rcu-safe list with the stable head.  The new
for_each_thread(g, t) helper is always safe under rcu_read_lock() as
long as this task_struct can't go away.

Note: of course it is ugly to have both task_struct->thread_node and the
old task_struct->thread_group, we will kill it later, after we change
the users of while_each_thread() to use for_each_thread().

Perhaps we can kill it even before we convert all users, we can
reimplement next_thread(t) using the new thread_head/thread_node.  But
we can't do this right now because this will lead to subtle behavioural
changes.  For example, do/while_each_thread() always sees at least one
task, while for_each_thread() can do nothing if the whole thread group
has died.  Or thread_group_empty(), currently its semantics is not clear
unless thread_group_leader(p) and we need to audit the callers before we
can change it.

So this patch adds the new interface which has to coexist with the old
one for some time, hopefully the next changes will be more or less
straightforward and the old one will go away soon.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Sergey Dyasly <dserrg@gmail.com>
Tested-by: Sergey Dyasly <dserrg@gmail.com>
Reviewed-by: Sameer Nanda <snanda@chromium.org>
Acked-by: David Rientjes <rientjes@google.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mandeep Singh Baines <msb@chromium.org>
Cc: "Ma, Xindong" <xindong.ma@intel.com>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
suspend: enable freeze timeout configuration through sys

At present, the value of timeout for freezing is 20s, which is
meaningless in case that one thread is frozen with mutex locked
and another thread is trying to lock the mutex, as this time of
freezing will fail unavoidably.
And if there is no new wakeup event registered, the system will
waste at most 20s for such meaningless trying of freezing.

With this patch, the value of timeout can be configured to smaller
value, so such meaningless trying of freezing will be aborted in
earlier time, and later freezing can be also triggered in earlier
time. And more power will be saved.
In normal case on mobile phone, it costs real little time to freeze
processes. On some platform, it only costs about 20ms to freeze
user space processes and 10ms to freeze kernel freezable threads.

Signed-off-by: Liu Chuansheng <chuansheng.liu@intel.com>
Signed-off-by: Li Fei <fei.li@intel.com>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
[FREEZER]: *squashing updates by Colin Cross (05/2013)*

freezer: add unsafe versions of freezable helpers for NFS

      NFS calls the freezable helpers with locks held, which is unsafe
      and will cause lockdep warnings when 6aa9707 "lockdep: check
      that no locks held at freeze time" is reapplied (it was reverted
      in dbf520a).  NFS shouldn't be doing this, but it has
      long-running syscalls that must hold a lock but also shouldn't
      block suspend.  Until NFS freeze handling is rewritten to use a
      signal to exit out of the critical section, add new *_unsafe
      versions of the helpers that will not run the lockdep test when
      6aa9707 is reapplied, and call them from NFS.

      In practice the likley result of holding the lock while freezing
      is that a second task blocked on the lock will never freeze,
      aborting suspend, but it is possible to manufacture a case using
      the cgroup freezer, the lock, and the suspend freezer to create
      a deadlock.  Silencing the lockdep warning here will allow
      problems to be found in other drivers that may have a more
      serious deadlock risk, and prevent new problems from being added.

      Signed-off-by: Colin Cross <ccross@android.com>
      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: add unsafe versions of freezable helpers for CIFS

      CIFS calls wait_event_freezekillable_unsafe with a VFS lock held,
      which is unsafe and will cause lockdep warnings when 6aa9707
      "lockdep: check that no locks held at freeze time" is reapplied
      (it was reverted in dbf520a).  CIFS shouldn't be doing this, but
      it has long-running syscalls that must hold a lock but also
      shouldn't block suspend.  Until CIFS freeze handling is rewritten
      to use a signal to exit out of the critical section, add a new
      wait_event_freezekillable_unsafe helper that will not run the
      lockdep test when 6aa9707 is reapplied, and call it from CIFS.

      In practice the likley result of holding the lock while freezing
      is that a second task blocked on the lock will never freeze,
      aborting suspend, but it is possible to manufacture a case using
      the cgroup freezer, the lock, and the suspend freezer to create
      a deadlock.  Silencing the lockdep warning here will allow
      problems to be found in other drivers that may have a more
      serious deadlock risk, and prevent new problems from being added.

      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Reviewed-by: Jeff Layton <jlayton@redhat.com>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

lockdep: remove task argument from debug_check_no_locks_held

      The only existing caller to debug_check_no_locks_held calls it
      with 'current' as the task, and the freezer needs to call
      debug_check_no_locks_held but doesn't already have a current
      task pointer, so remove the argument.  It is already assuming
      that the current task is relevant by dumping the current stack
      trace as part of the warning.

      This was originally part of 6aa9707099c (lockdep: check that
      no locks held at freeze time) which was reverted in
      dbf520a9d7d4.

      Original-author: Mandeep Singh Baines <msb@chromium.org>
      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: shorten freezer sleep time using exponential backoff

      All tasks can easily be frozen in under 10 ms, switch to using
      an initial 1 ms sleep followed by exponential backoff until
      8 ms.  Also convert the printed time to ms instead of centiseconds.

      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: skip waking up tasks with PF_FREEZER_SKIP set

      Android goes through suspend/resume very often (every few seconds when
      on a busy wifi network with the screen off), and a significant portion
      of the energy used to go in and out of suspend is spent in the
      freezer.  If a task has called freezer_do_not_count(), don't bother
      waking it up.  If it happens to wake up later it will call
      freezer_count() and immediately enter the refrigerator.

      Combined with patches to convert freezable helpers to use
      freezer_do_not_count() and convert common sites where idle userspace
      tasks are blocked to use the freezable helpers, this reduces the
      time and energy required to suspend and resume.

      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Pavel Machek <pavel@ucw.cz>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: convert freezable helpers to freezer_do_not_count()

      Freezing tasks will wake up almost every userspace task from
      where it is blocking and force it to run until it hits a
      call to try_to_sleep(), generally on the exit path from the syscall
      it is blocking in.  On resume each task will run again, usually
      restarting the syscall and running until it hits the same
      blocking call as it was originally blocked in.

      Convert the existing wait_event_freezable* wrappers to use
      freezer_do_not_count().  Combined with a previous patch,
      these tasks will not run during suspend or resume unless they wake
      up for another reason, in which case they will run until they hit
      the try_to_freeze() in freezer_count(), and then continue processing
      the wakeup after tasks are thawed.

      This results in a small change in behavior, previously a race
      between freezing and a normal wakeup would be won by the wakeup,
      now the task will freeze and then handle the wakeup after thawing.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: convert freezable helpers to static inline where possible

      Some of the freezable helpers have to be macros because their
      condition argument needs to get evaluated every time through
      the wait loop.  Convert the others to static inline to make
      future changes easier.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: add new freezable helpers using freezer_do_not_count()

      Freezing tasks will wake up almost every userspace task from
      where it is blocking and force it to run until it hits a
      call to try_to_sleep(), generally on the exit path from the syscall
      it is blocking in.  On resume each task will run again, usually
      restarting the syscall and running until it hits the same
      blocking call as it was originally blocked in.

      To allow tasks to avoid running on every suspend/resume cycle,
      this patch adds additional freezable wrappers around blocking calls
      that call freezer_do_not_count().  Combined with the previous patch,
      these tasks will not run during suspend or resume unless they wake
      up for another reason, in which case they will run until they hit
      the try_to_freeze() in freezer_count(), and then continue processing
      the wakeup after tasks are thawed.

      Additional patches will convert the most common locations that
      userspace blocks in to use freezable helpers.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

binder: use freezable blocking calls

      Avoid waking up every thread sleeping in a binder call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

epoll: use freezable blocking call

      Avoid waking up every thread sleeping in an epoll_wait call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

select: use freezable blocking call

      Avoid waking up every thread sleeping in a select call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

futex: use freezable blocking call

      Avoid waking up every thread sleeping in a futex_wait call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Thomas Gleixner <tglx@linutronix.de>
      Acked-by: Darren Hart <dvhart@linux.intel.com>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

nanosleep: use freezable blocking call

      Avoid waking up every thread sleeping in a nanosleep call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

sigtimedwait: use freezable blocking call

      Avoid waking up every thread sleeping in a sigtimedwait call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

af_unix: use freezable blocking calls in read

      Avoid waking up every thread sleeping in read call on an AF_UNIX
      socket during suspend and resume by calling a freezable blocking
      call.  Previous patches modified the freezer to avoid sending
      wakeups to threads that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

HACK: ARM: disable sleeping while atomic warning in do_signal

      ARM disables interrupts in do_signal, which triggers a warning in
      try_to_freeze, see details at https://lkml.org/lkml/2011/8/23/221.
      To prevent the warnings, add try_to_freeze_nowarn and call it from
      do_signal.

      Change-Id: If7482de21c386adc705fa1ac4ecb8c7ece5bb356
      Signed-off-by: Colin Cross <ccross@android.com>
[CPUFREQ] *optimizations*

*Reworked MSM-CPUfreq driver and ONDEMAND governor*

Signed-off-by: moddingg33k <moddingg33k@t-online.de>
[SCHED] *squashing updates*

sched: Fix hotplug vs. set_cpus_allowed_ptr()

      Lai found that:

	WARNING: CPU: 1 PID: 13 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x2d/0x4b()
	...
	migration_cpu_stop+0x1d/0x22

      was caused by set_cpus_allowed_ptr() assuming that cpu_active_mask is
      always a sub-set of cpu_online_mask.

      This isn't true since 5fbd036b552f ("sched: Cleanup cpu_active madness").

      So set active and online at the same time to avoid this particular
      problem.

      Fixes: 5fbd036b552f ("sched: Cleanup cpu_active madness")
      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Peter Zijlstra <peterz@infradead.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Michael wang <wangyun@linux.vnet.ibm.com>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
      Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Cc: Toshi Kani <toshi.kani@hp.com>
      Link: http://lkml.kernel.org/r/53758B12.8060609@cn.fujitsu.com
      Signed-off-by: Ingo Molnar <mingo@kernel.org>

sched: Improve scalability via 'CPU buddies', which withstand random perturbations

      Traversing an entire package is not only expensive, it also leads to tasks
      bouncing all over a partially idle and possible quite large package.  Fix
      that up by assigning a 'buddy' CPU to try to motivate.  Each buddy may try
      to motivate that one other CPU, if it's busy, tough, it may then try its
      SMT sibling, but that's all this optimization is allowed to cost.

      Sibling cache buddies are cross-wired to prevent bouncing.

      4 socket 40 core + SMT Westmere box, single 30 sec tbench runs, higher is better:

      clients     1       2       4        8       16       32       64      128
      ..........................................................................
      pre        30      41     118      645     3769     6214    12233    14312
      post      299     603    1211     2418     4697     6847    11606    14557

      A nice increase in performance.

      Signed-off-by: Mike Galbraith <efault@gmx.de>
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Link: http://lkml.kernel.org/r/1339471112.7352.32.camel@marge.simpson.net
      Signed-off-by: Ingo Molnar <mingo@kernel.org>

sched: Reorder 'struct lb_env' members to reduce its size

      Members of 'struct lb_env' are not in appropriate order to reuse compiler
      added padding on 64bit architectures. In this patch we reorder those struct
      members and help reduce the size of the structure from 96 bytes to 80
      bytes on 64 bit architectures.

      Suggested-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
      Signed-off-by: Prashanth Nageshappa <prashanth@linux.vnet.ibm.com>
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Link: http://lkml.kernel.org/r/4FE06DDE.7000403@linux.vnet.ibm.com
      Signed-off-by: Ingo Molnar <mingo@kernel.org>

sched: Reset loop counters if all tasks are pinned and we need to redo load balance

      While load balancing, if all tasks on the source runqueue are pinned,
      we retry after excluding the corresponding source cpu. However, loop counters
      env.loop and env.loop_break are not reset before retrying, which can lead
      to failure in moving the tasks. In this patch we reset env.loop and
      env.loop_break to their inital values before we retry.

      Signed-off-by: Prashanth Nageshappa <prashanth@linux.vnet.ibm.com>
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Link: http://lkml.kernel.org/r/4FE06EEF.2090709@linux.vnet.ibm.com
      Signed-off-by: Ingo Molnar <mingo@kernel.org>
touchscreen: adding fuzz to improve ovation sensitivity

Adding a fuzz factor to improve X/Y touch registration.
mm, mempolicy: remove per-process flag

PF_MEMPOLICY is an unnecessary optimization for CONFIG_SLAB users.
There's no significant performance degradation to checking
current->mempolicy rather than current->flags & PF_MEMPOLICY in the
allocation path, especially since this is considered unlikely().

Running TCP_RR with netperf-2.4.5 through localhost on 16 cpu machine with
64GB of memory and without a mempolicy:

	threads		before		after
	16		1249409		1244487
	32		1281786		1246783
	48		1239175		1239138
	64		1244642		1241841
	80		1244346		1248918
	96		1266436		1254316
	112		1307398		1312135
	128		1327607		1326502

Per-process flags are a scarce resource so we should free them up whenever
possible and make them available.  We'll be using it shortly for memcg oom
reserves.

Signed-off-by: David Rientjes <rientjes@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Jianguo Wu <wujianguo@huawei.com>
Cc: Tim Hockin <thockin@google.com>
Cc: Christoph Lameter <cl@linux.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
mm: per-thread vma caching

This patch is a continuation of efforts trying to optimize find_vma(),
avoiding potentially expensive rbtree walks to locate a vma upon faults.
The original approach (https://lkml.org/lkml/2013/11/1/410), where the
largest vma was also cached, ended up being too specific and random,
thus further comparison with other approaches were needed.  There are
two things to consider when dealing with this, the cache hit rate and
the latency of find_vma().  Improving the hit-rate does not necessarily
translate in finding the vma any faster, as the overhead of any fancy
caching schemes can be too high to consider.

We currently cache the last used vma for the whole address space, which
provides a nice optimization, reducing the total cycles in find_vma() by
up to 250%, for workloads with good locality.  On the other hand, this
simple scheme is pretty much useless for workloads with poor locality.
Analyzing ebizzy runs shows that, no matter how many threads are
running, the mmap_cache hit rate is less than 2%, and in many situations
below 1%.

The proposed approach is to replace this scheme with a small per-thread
cache, maximizing hit rates at a very low maintenance cost.
Invalidations are performed by simply bumping up a 32-bit sequence
number.  The only expensive operation is in the rare case of a seq
number overflow, where all caches that share the same address space are
flushed.  Upon a miss, the proposed replacement policy is based on the
page number that contains the virtual address in question.  Concretely,
the following results are seen on an 80 core, 8 socket x86-64 box:

1) System bootup: Most programs are single threaded, so the per-thread
   scheme does improve ~50% hit rate by just adding a few more slots to
   the cache.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 50.61%   | 19.90            |
| patched        | 73.45%   | 13.58            |
+----------------+----------+------------------+

2) Kernel build: This one is already pretty good with the current
   approach as we're dealing with good locality.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 75.28%   | 11.03            |
| patched        | 88.09%   | 9.31             |
+----------------+----------+------------------+

3) Oracle 11g Data Mining (4k pages): Similar to the kernel build workload.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 70.66%   | 17.14            |
| patched        | 91.15%   | 12.57            |
+----------------+----------+------------------+

4) Ebizzy: There's a fair amount of variation from run to run, but this
   approach always shows nearly perfect hit rates, while baseline is just
   about non-existent.  The amounts of cycles can fluctuate between
   anywhere from ~60 to ~116 for the baseline scheme, but this approach
   reduces it considerably.  For instance, with 80 threads:

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 1.06%    | 91.54            |
| patched        | 99.97%   | 14.18            |
+----------------+----------+------------------+

[akpm@linux-foundation.org: fix nommu build, per Davidlohr]
[akpm@linux-foundation.org: document vmacache_valid() logic]
[akpm@linux-foundation.org: attempt to untangle header files]
[akpm@linux-foundation.org: add vmacache_find() BUG_ON]
[hughd@google.com: add vmacache_valid_mm() (from Oleg)]
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: adjust and enhance comments]
Signed-off-by: Davidlohr Bueso <davidlohr@hp.com>
Reviewed-by: Rik van Riel <riel@redhat.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Michel Lespinasse <walken@google.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Tested-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[CPU HotPlug] Provide lockless versions of callback registration functions

CPU hotplug: Add lockdep annotations to get/put_online_cpus()

      Add lockdep annotations for get/put_online_cpus() and
      cpu_hotplug_begin()/cpu_hotplug_end().

      Cc: Ingo Molnar <mingo@kernel.org>
      Reviewed-by: Oleg Nesterov <oleg@redhat.com>
      Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

CPU hotplug: Provide lockless versions of callback registration functions

      The following method of CPU hotplug callback registration is not safe
      due to the possibility of an ABBA deadlock involving the cpu_add_remove_lock
      and the cpu_hotplug.lock.

	      get_online_cpus();

	      for_each_online_cpu(cpu)
		      init_cpu(cpu);

	      register_cpu_notifier(&foobar_cpu_notifier);

	      put_online_cpus();

      The deadlock is shown below:

		CPU 0                                         CPU 1
		-----                                         -----

	Acquire cpu_hotplug.lock
	[via get_online_cpus()]

						    CPU online/offline operation
						    takes cpu_add_remove_lock
						    [via cpu_maps_update_begin()]

	Try to acquire
	cpu_add_remove_lock
	[via register_cpu_notifier()]

						    CPU online/offline operation
						    tries to acquire cpu_hotplug.lock
						    [via cpu_hotplug_begin()]

				  *** DEADLOCK! ***

      The problem here is that callback registration takes the locks in one order
      whereas the CPU hotplug operations take the same locks in the opposite order.
      To avoid this issue and to provide a race-free method to register CPU hotplug
      callbacks (along with initialization of already online CPUs), introduce new
      variants of the callback registration APIs that simply register the callbacks
      without holding the cpu_add_remove_lock during the registration. That way,
      we can avoid the ABBA scenario. However, we will need to hold the
      cpu_add_remove_lock throughout the entire critical section, to protect updates
      to the callback/notifier chain.

      This can be achieved by writing the callback registration code as follows:

	      cpu_maps_update_begin(); [ or cpu_notifier_register_begin(); see below ]

	      for_each_online_cpu(cpu)
		      init_cpu(cpu);

	      /* This doesn't take the cpu_add_remove_lock */
	      __register_cpu_notifier(&foobar_cpu_notifier);

	      cpu_maps_update_done();  [ or cpu_notifier_register_done(); see below ]

      Note that we can't use get_online_cpus() here instead of cpu_maps_update_begin()
      because the cpu_hotplug.lock is dropped during the invocation of CPU_POST_DEAD
      notifiers, and hence get_online_cpus() cannot provide the necessary
      synchronization to protect the callback/notifier chains against concurrent
      reads and writes. On the other hand, since the cpu_add_remove_lock protects
      the entire hotplug operation (including CPU_POST_DEAD), we can use
      cpu_maps_update_begin/done() to guarantee proper synchronization.

      Also, since cpu_maps_update_begin/done() is like a super-set of
      get/put_online_cpus(), the former naturally protects the critical sections
      from concurrent hotplug operations.

      Since the names cpu_maps_update_begin/done() don't make much sense in CPU
      hotplug callback registration scenarios, we'll introduce new APIs named
      cpu_notifier_register_begin/done() and map them to cpu_maps_update_begin/done().

      In summary, introduce the lockless variants of un/register_cpu_notifier() and
      also export the cpu_notifier_register_begin/done() APIs for use by modules.
      This way, we provide a race-free way to register hotplug callbacks as well as
      perform initialization for the CPUs that are already online.

      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Acked-by: Oleg Nesterov <oleg@redhat.com>
      Acked-by: Toshi Kani <toshi.kani@hp.com>
      Reviewed-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Doc/cpu-hotplug: Specify race-free way to register CPU hotplug callbacks

      Recommend the usage of the new CPU hotplug callback registration APIs
      (__register_cpu_notifier() etc), when subsystems need to also perform
      initialization for already online CPUs. Provide examples of correct
      and race-free ways of achieving this, and point out the kinds of code
      that are error-prone.

      Cc: Rob Landley <rob@landley.net>
      Cc: Ingo Molnar <mingo@kernel.org>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

---------------------------------------------------------------------------------------------------

CPU hotplug, perf: Fix CPU hotplug callback registration

arm, hw-breakpoint: Fix CPU hotplug callback registration

zsmalloc: Fix CPU hotplug callback registration

drivers/base/topology.c: Fix CPU hotplug callback registration

oprofile, nmi-timer: Fix CPU hotplug callback registration

profile: Fix CPU hotplug callback registration

mm, vmstat: Fix CPU hotplug callback registration

net/core/flow.c: Fix CPU hotplug callback registration

net/iucv/iucv.c: Fix CPU hotplug callback registration
[SOFTIRQ] *squashing updates*

mm: allow PF_MEMALLOC from softirq context

      This is needed to allow network softirq packet processing to make use of
      PF_MEMALLOC.

      Currently softirq context cannot use PF_MEMALLOC due to it not being
      associated with a task, and therefore not having task flags to fiddle with
      - thus the gfp to alloc flag mapping ignores the task flags when in
      interrupts (hard or soft) context.

      Allowing softirqs to make use of PF_MEMALLOC therefore requires some
      trickery.  This patch borrows the task flags from whatever process happens
      to be preempted by the softirq.  It then modifies the gfp to alloc flags
      mapping to not exclude task flags in softirq context, and modify the
      softirq code to save, clear and restore the PF_MEMALLOC flag.

      The save and clear, ensures the preempted task's PF_MEMALLOC flag doesn't
      leak into the softirq.  The restore ensures a softirq's PF_MEMALLOC flag
      cannot leak back into the preempted process.  This should be safe due to
      the following reasons

      Softirqs can run on multiple CPUs sure but the same task should not be
	      executing the same softirq code. Neither should the softirq
	      handler be preempted by any other softirq handler so the flags
	      should not leak to an unrelated softirq.

      Softirqs re-enable hardware interrupts in __do_softirq() so can be
	      preempted by hardware interrupts so PF_MEMALLOC is inherited
	      by the hard IRQ. However, this is similar to a process in
	      reclaim being preempted by a hardirq. While PF_MEMALLOC is
	      set, gfp_to_alloc_flags() distinguishes between hard and
	      soft irqs and avoids giving a hardirq the ALLOC_NO_WATERMARKS
	      flag.

      If the softirq is deferred to ksoftirq then its flags may be used
	      instead of a normal tasks but as the softirq cannot be preempted,
	      the PF_MEMALLOC flag does not leak to other code by accident.

      [davem@davemloft.net: Document why PF_MEMALLOC is safe]
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: Mel Gorman <mgorman@suse.de>
      Cc: David Miller <davem@davemloft.net>
      Cc: Neil Brown <neilb@suse.de>
      Cc: Mike Christie <michaelc@cs.wisc.edu>
      Cc: Eric B Munson <emunson@mgebm.net>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Christoph Lameter <cl@linux.com>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

softirq: reduce latencies

      In various network workloads, __do_softirq() latencies can be up
      to 20 ms if HZ=1000, and 200 ms if HZ=100.

      This is because we iterate 10 times in the softirq dispatcher,
      and some actions can consume a lot of cycles.

      This patch changes the fallback to ksoftirqd condition to :

      - A time limit of 2 ms.
      - need_resched() being set on current task

      When one of this condition is met, we wakeup ksoftirqd for further
      softirq processing if we still have pending softirqs.

      Using need_resched() as the only condition can trigger RCU stalls,
      as we can keep BH disabled for too long.

      I ran several benchmarks and got no significant difference in
      throughput, but a very significant reduction of latencies (one order
      of magnitude) :

      In following bench, 200 antagonist "netperf -t TCP_RR" are started in
      background, using all available cpus.

      Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
      IRQ (hard+soft)

      Before patch :

      # netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
      RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
      MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
      to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
      RT_LATENCY=550110.424
      MIN_LATENCY=146858
      MAX_LATENCY=997109
      P50_LATENCY=305000
      P90_LATENCY=550000
      P99_LATENCY=710000
      MEAN_LATENCY=376989.12
      STDDEV_LATENCY=184046.92

      After patch :

      # netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
      RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
      MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
      to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
      RT_LATENCY=40545.492
      MIN_LATENCY=9834
      MAX_LATENCY=78366
      P50_LATENCY=33583
      P90_LATENCY=59000
      P99_LATENCY=69000
      MEAN_LATENCY=38364.67
      STDDEV_LATENCY=12865.26

      Signed-off-by: Eric Dumazet <edumazet@google.com>
      Cc: David Miller <davem@davemloft.net>
      Cc: Tom Herbert <therbert@google.com>
      Cc: Ben Hutchings <bhutchings@solarflare.com>
      Signed-off-by: David S. Miller <davem@davemloft.net>

irq: Ensure irq_exit() code runs with interrupts disabled

      We had already a few problems with code called from irq_exit() when
      interrupted from a nesting interrupt. This can happen on architectures
      which do not define __ARCH_IRQ_EXIT_IRQS_DISABLED.

      __ARCH_IRQ_EXIT_IRQS_DISABLED should go away and we want to make it
      mandatory to call irq_exit() with interrupts disabled.

      As a temporary protection disable interrupts for those architectures
      which do not define __ARCH_IRQ_EXIT_IRQS_DISABLED and add a WARN_ONCE
      when an architecture which defines __ARCH_IRQ_EXIT_IRQS_DISABLED calls
      irq_exit() with interrupts enabled.

      Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linuxfoundation.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302202155320.22263@ionos

irq: Sanitize invoke_softirq

      With the irq protection in irq_exit, we can remove the #ifdeffery and
      the bh_disable/enable dance in invoke_softirq()

      Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linuxfoundation.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302202155320.22263@ionos

      # This is the 5th commit message:

      irq: Remove IRQ_EXIT_OFFSET workaround

      The IRQ_EXIT_OFFSET trick was used to make sure the irq
      doesn't get preempted after we substract the HARDIRQ_OFFSET
      until we are entirely done with any code in irq_exit().

      This workaround was necessary because some archs may call
      irq_exit() with irqs enabled and there is still some code
      in the end of this function that is not covered by the
      HARDIRQ_OFFSET but want to stay non-preemptible.

      Now that irq are always disabled in irq_exit(), the whole code
      is guaranteed not to be preempted. We can thus remove this hack.

      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

irq: Don't re-enable interrupts at the end of irq_exit

      Commit 74eed0163d0def3fce27228d9ccf3d36e207b286
      "irq: Ensure irq_exit() code runs with interrupts disabled"
      restore interrupts flags in the end of irq_exit() for archs
      that don't define __ARCH_IRQ_EXIT_IRQS_DISABLED.

      However always returning from irq_exit() with interrupts
      disabled should not be a problem for these archs. Prior to
      this commit this was already happening anytime we processed
      pending softirqs anyway.

      Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Fix lockup related to stop_machine being stuck in __do_softirq.

      The stop machine logic can lock up if all but one of the migration
      threads make it through the disable-irq step and the one remaining
      thread gets stuck in __do_softirq.  The reason __do_softirq can hang is
      that it has a bail-out based on jiffies timeout, but in the lockup case,
      jiffies itself is not incremented.

      To work around this, re-add the max_restart counter in __do_irq and stop
      processing irqs after 10 restarts.

      Thanks to Tejun Heo and Rusty Russell and others for helping me track
      this down.

      This was introduced in 3.9 by commit c10d73671ad3 ("softirq: reduce
      latencies").

      It may be worth looking into ath9k to see if it has issues with its irq
      handler at a later date.

      The hang stack traces look something like this:

	  ------------[ cut here ]------------
	  WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9c/0xa7()
	  Watchdog detected hard LOCKUP on cpu 2
	  Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
	  Pid: 23, comm: migration/2 Tainted: G         C   3.9.4+ #11
	  Call Trace:
	  <NMI>   warn_slowpath_common+0x85/0x9f
	    warn_slowpath_fmt+0x46/0x48
	    watchdog_overflow_callback+0x9c/0xa7
	    __perf_event_overflow+0x137/0x1cb
	    perf_event_overflow+0x14/0x16
	    intel_pmu_handle_irq+0x2dc/0x359
	    perf_event_nmi_handler+0x19/0x1b
	    nmi_handle+0x7f/0xc2
	    do_nmi+0xbc/0x304
	    end_repeat_nmi+0x1e/0x2e
	  <<EOE>>
	    cpu_stopper_thread+0xae/0x162
	    smpboot_thread_fn+0x258/0x260
	    kthread+0xc7/0xcf
	    ret_from_fork+0x7c/0xb0
	  ---[ end trace 4947dfa9b0a4cec3 ]---
	  BUG: soft lockup - CPU#1 stuck for 22s! [migration/1:17]
	  Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
	  irq event stamp: 835637905
	  hardirqs last  enabled at (835637904): __do_softirq+0x9f/0x257
	  hardirqs last disabled at (835637905): apic_timer_interrupt+0x6d/0x80
	  softirqs last  enabled at (5654720): __do_softirq+0x1ff/0x257
	  softirqs last disabled at (5654725): irq_exit+0x5f/0xbb
	  CPU 1
	  Pid: 17, comm: migration/1 Tainted: G        WC   3.9.4+ #11 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
	  RIP: tasklet_hi_action+0xf0/0xf0
	  Process migration/1
	  Call Trace:
	  <IRQ>
	    __do_softirq+0x117/0x257
	    irq_exit+0x5f/0xbb
	    smp_apic_timer_interrupt+0x8a/0x98
	    apic_timer_interrupt+0x72/0x80
	  <EOI>
	    printk+0x4d/0x4f
	    stop_machine_cpu_stop+0x22c/0x274
	    cpu_stopper_thread+0xae/0x162
	    smpboot_thread_fn+0x258/0x260
	    kthread+0xc7/0xcf
	    ret_from_fork+0x7c/0xb0

      Signed-off-by: Ben Greear <greearb@candelatech.com>
      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Pekka Riikonen <priikone@iki.fi>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: stable@kernel.org
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

irq: Force hardirq exit's softirq processing on its own stack

      The commit facd8b80c67a3cf64a467c4a2ac5fb31f2e6745b
      ("irq: Sanitize invoke_softirq") converted irq exit
      calls of do_softirq() to __do_softirq() on all architectures,
      assuming it was only used there for its irq disablement
      properties.

      But as a side effect, the softirqs processed in the end
      of the hardirq are always called on the inline current
      stack that is used by irq_exit() instead of the softirq
      stack provided by the archs that override do_softirq().

      The result is mostly safe if the architecture runs irq_exit()
      on a separate irq stack because then softirqs are processed
      on that same stack that is near empty at this stage (assuming
      hardirq aren't nesting).

      Otherwise irq_exit() runs in the task stack and so does the softirq
      too. The interrupted call stack can be randomly deep already and
      the softirq can dig through it even further. To add insult to the
      injury, this softirq can be interrupted by a new hardirq, maximizing
      the chances for a stack overrun as reported in powerpc for example:

	      do_IRQ: stack overflow: 1920
	      CPU: 0 PID: 1602 Comm: qemu-system-ppc Not tainted 3.10.4-300.1.fc19.ppc64p7 #1
	      Call Trace:
	      [c0000000050a8740] .show_stack+0x130/0x200 (unreliable)
	      [c0000000050a8810] .dump_stack+0x28/0x3c
	      [c0000000050a8880] .do_IRQ+0x2b8/0x2c0
	      [c0000000050a8930] hardware_interrupt_common+0x154/0x180
	      --- Exception: 501 at .cp_start_xmit+0x3a4/0x820 [8139cp]
		      LR = .cp_start_xmit+0x390/0x820 [8139cp]
	      [c0000000050a8d40] .dev_hard_start_xmit+0x394/0x640
	      [c0000000050a8e00] .sch_direct_xmit+0x110/0x260
	      [c0000000050a8ea0] .dev_queue_xmit+0x260/0x630
	      [c0000000050a8f40] .br_dev_queue_push_xmit+0xc4/0x130 [bridge]
	      [c0000000050a8fc0] .br_dev_xmit+0x198/0x270 [bridge]
	      [c0000000050a9070] .dev_hard_start_xmit+0x394/0x640
	      [c0000000050a9130] .dev_queue_xmit+0x428/0x630
	      [c0000000050a91d0] .ip_finish_output+0x2a4/0x550
	      [c0000000050a9290] .ip_local_out+0x50/0x70
	      [c0000000050a9310] .ip_queue_xmit+0x148/0x420
	      [c0000000050a93b0] .tcp_transmit_skb+0x4e4/0xaf0
	      [c0000000050a94a0] .__tcp_ack_snd_check+0x7c/0xf0
	      [c0000000050a9520] .tcp_rcv_established+0x1e8/0x930
	      [c0000000050a95f0] .tcp_v4_do_rcv+0x21c/0x570
	      [c0000000050a96c0] .tcp_v4_rcv+0x734/0x930
	      [c0000000050a97a0] .ip_local_deliver_finish+0x184/0x360
	      [c0000000050a9840] .ip_rcv_finish+0x148/0x400
	      [c0000000050a98d0] .__netif_receive_skb_core+0x4f8/0xb00
	      [c0000000050a99d0] .netif_receive_skb+0x44/0x110
	      [c0000000050a9a70] .br_handle_frame_finish+0x2bc/0x3f0 [bridge]
	      [c0000000050a9b20] .br_nf_pre_routing_finish+0x2ac/0x420 [bridge]
	      [c0000000050a9bd0] .br_nf_pre_routing+0x4dc/0x7d0 [bridge]
	      [c0000000050a9c70] .nf_iterate+0x114/0x130
	      [c0000000050a9d30] .nf_hook_slow+0xb4/0x1e0
	      [c0000000050a9e00] .br_handle_frame+0x290/0x330 [bridge]
	      [c0000000050a9ea0] .__netif_receive_skb_core+0x34c/0xb00
	      [c0000000050a9fa0] .netif_receive_skb+0x44/0x110
	      [c0000000050aa040] .napi_gro_receive+0xe8/0x120
	      [c0000000050aa0c0] .cp_rx_poll+0x31c/0x590 [8139cp]
	      [c0000000050aa1d0] .net_rx_action+0x1dc/0x310
	      [c0000000050aa2b0] .__do_softirq+0x158/0x330
	      [c0000000050aa3b0] .irq_exit+0xc8/0x110
	      [c0000000050aa430] .do_IRQ+0xdc/0x2c0
	      [c0000000050aa4e0] hardware_interrupt_common+0x154/0x180
	      --- Exception: 501 at .bad_range+0x1c/0x110
		      LR = .get_page_from_freelist+0x908/0xbb0
	      [c0000000050aa7d0] .list_del+0x18/0x50 (unreliable)
	      [c0000000050aa850] .get_page_from_freelist+0x908/0xbb0
	      [c0000000050aa9e0] .__alloc_pages_nodemask+0x21c/0xae0
	      [c0000000050aaba0] .alloc_pages_vma+0xd0/0x210
	      [c0000000050aac60] .handle_pte_fault+0x814/0xb70
	      [c0000000050aad50] .__get_user_pages+0x1a4/0x640
	      [c0000000050aae60] .get_user_pages_fast+0xec/0x160
	      [c0000000050aaf10] .__gfn_to_pfn_memslot+0x3b0/0x430 [kvm]
	      [c0000000050aafd0] .kvmppc_gfn_to_pfn+0x64/0x130 [kvm]
	      [c0000000050ab070] .kvmppc_mmu_map_page+0x94/0x530 [kvm]
	      [c0000000050ab190] .kvmppc_handle_pagefault+0x174/0x610 [kvm]
	      [c0000000050ab270] .kvmppc_handle_exit_pr+0x464/0x9b0 [kvm]
	      [c0000000050ab320]  kvm_start_lightweight+0x1ec/0x1fc [kvm]
	      [c0000000050ab4f0] .kvmppc_vcpu_run_pr+0x168/0x3b0 [kvm]
	      [c0000000050ab9c0] .kvmppc_vcpu_run+0xc8/0xf0 [kvm]
	      [c0000000050aba50] .kvm_arch_vcpu_ioctl_run+0x5c/0x1a0 [kvm]
	      [c0000000050abae0] .kvm_vcpu_ioctl+0x478/0x730 [kvm]
	      [c0000000050abc90] .do_vfs_ioctl+0x4ec/0x7c0
	      [c0000000050abd80] .SyS_ioctl+0xd4/0xf0
	      [c0000000050abe30] syscall_exit+0x0/0x98

      Since this is a regression, this patch proposes a minimalistic
      and low-risk solution by blindly forcing the hardirq exit processing of
      softirqs on the softirq stack. This way we should reduce significantly
      the opportunities for task stack overflow dug by softirqs.

      Longer term solutions may involve extending the hardirq stack coverage to
      irq_exit(), etc...

      Reported-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: #3.9.. <stable@vger.kernel.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Consolidate do_softirq() arch overriden implementations

      All arch overriden implementations of do_softirq() share the following
      common code: disable irqs (to avoid races with the pending check),
      check if there are softirqs pending, then execute __do_softirq() on
      a specific stack.

      Consolidate the common parts such that archs only worry about the
      stack switch.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Optimize call to softirq on hardirq exit

      Before processing softirqs on hardirq exit, we already
      do the check for pending softirqs while hardirqs are
      guaranteed to be disabled.

      So we can take a shortcut and safely jump to the arch
      specific implementation directly.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Improve a bit softirq debugging

      do_softirq() has a debug check that verifies that it is not nesting
      on softirqs processing, nor miscounting the softirq part of the preempt
      count.

      But making sure that softirqs processing don't nest is actually a more
      generic concern that applies to any caller of __do_softirq().

      Do take it one step further and generalize that debug check to
      any softirq processing.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Justify the various softirq stack choices

      For clarity, comment the various stack choices for softirqs
      processing, whether we execute them from ksoftirqd or
      local_irq_enable() calls.

      Their use on irq_exit() is already commented.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Optimize softirq stack selection in irq exit

      If irq_exit() is called on the arch's specified irq stack,
      it should be safe to run softirqs inline under that same
      irq stack as it is near empty by the time we call irq_exit().

      For example if we use the same stack for both hard and soft irqs here,
      the worst case scenario is:
      hardirq -> softirq -> hardirq. But then the softirq supersedes the
      first hardirq as the stack user since irq_exit() is called in
      a mostly empty stack. So the stack merge in this case looks acceptable.

      Stack overrun still have a chance to happen if hardirqs have more
      opportunities to nest, but then it's another problem to solve.

      So lets adapt the irq exit's softirq stack on top of a new Kconfig symbol
      that can be defined when irq_exit() runs on the irq stack. That way
      we can spare some stack switch on irq processing and all the cache
      issues that come along.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

revert "softirq: Add support for triggering softirq work on softirqs"

      This commit was incomplete in that code to remove items from the per-cpu
      lists was missing and never acquired a user in the 5 years it has been in
      the tree.  We're going to implement what it seems to try to archive in a
      simpler way, and this code is in the way of doing so.

      Signed-off-by: Christoph Hellwig <hch@lst.de>
      Cc: Jan Kara <jack@suse.cz>
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[FILESYSTEM] *squashing updates*

switch do_fsync() to fget_light()

      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Move noop_backing_dev_info check from sync into writeback

      In principle, a filesystem may want to have ->sync_fs() called during sync(1)
      although it does not have a bdi (i.e. s_bdi is set to noop_backing_dev_info).
      Only writeback code really needs bdi set to something reasonable. So move the
      checks where they are more logical.

      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

quota: Split dquot_quota_sync() to writeback and cache flushing part

      Split off part of dquot_quota_sync() which writes dquots into a quota file
      to a separate function. In the next patch we will use the function from
      filesystems and we do not want to abuse ->quota_sync quotactl callback more
      than necessary.

      Acked-by: Steven Whitehouse <swhiteho@redhat.com>
      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

quota: Move quota syncing to ->sync_fs method

      Since the moment writes to quota files are using block device page cache and
      space for quota structures is reserved at the moment they are first accessed we
      have no reason to sync quota before inode writeback. In fact this order is now
      only harmful since quota information can easily change during inode writeback
      (either because conversion of delayed-allocated extents or simply because of
      allocation of new blocks for simple filesystems not using page_mkwrite).

      So move syncing of quota information after writeback of inodes into ->sync_fs
      method. This way we do not have to use ->quota_sync callback which is primarily
      intended for use by quotactl syscall anyway and we get rid of calling
      ->sync_fs() twice unnecessarily. We skip quota syncing for OCFS2 since it does
      proper quota journalling in all cases (unlike ext3, ext4, and reiserfs which
      also support legacy non-journalled quotas) and thus there are no dirty quota
      structures.

      CC: "Theodore Ts'o" <tytso@mit.edu>
      CC: Joel Becker <jlbec@evilplan.org>
      CC: reiserfs-devel@vger.kernel.org
      Acked-by: Steven Whitehouse <swhiteho@redhat.com>
      Acked-by: Dave Kleikamp <shaggy@kernel.org>
      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Reorder operations during sys_sync

      Change the order of operations during sync from

      for_each_sb {
	      writeback_inodes_sb();
	      sync_fs(nowait);
	      __sync_blockdev(nowait);
      }
      for_each_sb {
	      sync_inodes_sb();
	      sync_fs(wait);
	      __sync_blockdev(wait);
      }

      to

      for_each_sb
	      writeback_inodes_sb();
      for_each_sb
	      sync_fs(nowait);
      for_each_sb
	      __sync_blockdev(nowait);
      for_each_sb
	      sync_inodes_sb();
      for_each_sb
	      sync_fs(wait);
      for_each_sb
	      __sync_blockdev(wait);

      This is a preparation for the following patches in this series.

      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Make sys_sync writeout also block device inodes

      In case block device does not have filesystem mounted on it, sys_sync will just
      ignore it and doesn't writeout its dirty pages. This is because writeback code
      avoids writing inodes from superblock without backing device and
      blockdev_superblock is such a superblock.  Since it's unexpected that sync
      doesn't writeout dirty data for block devices be nice to users and change the
      behavior to do so. So now we iterate over all block devices on blockdev_super
      instead of iterating over all superblocks when syncing block devices.

      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Remove unnecessary flushing of block devices

      It is not necessary to write block devices twice. The reason why we first did
      flush and then proper sync is that
	for_each_bdev() {
	  write_bdev()
	  wait_for_completion()
	}
      is much slower than
	for_each_bdev()
	  write_bdev()
	for_each_bdev()
	  wait_for_completion()
      when there is bigger amount of data. But as is seen in the above, there's no real
      need to scan pages and submit them twice. We just need to separate the submission
      and waiting part. This patch does that.

      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Avoid unnecessary WB_SYNC_NONE writeback during sys_sync and reorder sync passes

      wakeup_flusher_threads(0) will queue work doing complete writeback for each
      flusher thread. Thus there is not much point in submitting another work doing
      full inode WB_SYNC_NONE writeback by writeback_inodes_sb().

      After this change it does not make sense to call nonblocking ->sync_fs and
      block device flush before calling sync_inodes_sb() because
      wakeup_flusher_threads() is completely asynchronous and thus these functions
      would be called in parallel with inode writeback running which will effectively
      void any work they do. So we move sync_inodes_sb() call before these two
      functions.

      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Create function for iterating over block devices

      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

fs/block-dev.c:fix performance regression in O_DIRECT writes to md block devices

      For regular file, write operaion used blk_plug function.But for block
      file,write operation did not use blk_plug.
      This patch is also for write-cache mode for block-device.

      Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
      Reviewed-by: NeilBrown <neilb@suse.de>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

Fix a crash when block device is read and block size is changed at the same time

      The kernel may crash when block size is changed and I/O is issued
      simultaneously.

      Because some subsystems (udev or lvm) may read any block device anytime,
      the bug actually puts any code that changes a block device size in
      jeopardy.

      The crash can be reproduced if you place "msleep(1000)" to
      blkdev_get_blocks just before "bh->b_size = max_blocks <<
      inode->i_blkbits;".
      Then, run "dd if=/dev/ram0 of=/dev/null bs=4k count=1 iflag=direct"
      While it is waiting in msleep, run "blockdev --setbsz 2048 /dev/ram0"
      You get a BUG.

      The direct and non-direct I/O is written with the assumption that block
      size does not change. It doesn't seem practical to fix these crashes
      one-by-one there may be many crash possibilities when block size changes
      at a certain place and it is impossible to find them all and verify the
      code.

      This patch introduces a new rw-lock bd_block_size_semaphore. The lock is
      taken for read during I/O. It is taken for write when changing block
      size. Consequently, block size can't be changed while I/O is being
      submitted.

      For asynchronous I/O, the patch only prevents block size change while
      the I/O is being submitted. The block size can change when the I/O is in
      progress or when the I/O is being finished. This is acceptable because
      there are no accesses to block size when asynchronous I/O is being
      finished.

      The patch prevents block size changing while the device is mapped with
      mmap.

      Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

fix O_SYNC|O_APPEND syncing the wrong range on write()

      It actually goes back to 2004 ([PATCH] Concurrent O_SYNC write support)
      when sync_page_range() had been introduced; generic_file_write{,v}() correctly
      synced
	      pos_after_write - written .. pos_after_write - 1
      but generic_file_aio_write() synced
	      pos_before_write .. pos_before_write + written - 1
      instead.  Which is not the same thing with O_APPEND, obviously.
      A couple of years later correct variant had been killed off when
      everything switched to use of generic_file_aio_write().

      All users of generic_file_aio_write() are affected, and the same bug
      has been copied into other instances of ->aio_write().

      The fix is trivial; the only subtle point is that generic_write_sync()
      ought to be inlined to avoid calculations useless for the majority of
      calls.

      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

blockdev: turn a rw semaphore into a percpu rw semaphore

      This avoids cache line bouncing when many processes lock the semaphore
      for read.

      New percpu lock implementation

      The lock consists of an array of percpu unsigned integers, a boolean
      variable and a mutex.

      When we take the lock for read, we enter rcu read section, check for a
      "locked" variable. If it is false, we increase a percpu counter on the
      current cpu and exit the rcu section. If "locked" is true, we exit the
      rcu section, take the mutex and drop it (this waits until a writer
      finished) and retry.

      Unlocking for read just decreases percpu variable. Note that we can
      unlock on a difference cpu than where we locked, in this case the
      counter underflows. The sum of all percpu counters represents the number
      of processes that hold the lock for read.

      When we need to lock for write, we take the mutex, set "locked" variable
      to true and synchronize rcu. Since RCU has been synchronized, no
      processes can create new read locks. We wait until the sum of percpu
      counters is zero - when it is, there are no readers in the critical
      section.

      Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

fs/block_dev.c:1644:5: sparse: symbol 'blkdev_mmap' was not declared

      blkdev_mmap() isn't used outside of fs/block_dev.c, mark it as
      static.

      Reported-by: Fengguang Wu <fengguang.wu@intel.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

Lock splice_read and splice_write functions

      Functions generic_file_splice_read and generic_file_splice_write access
      the pagecache directly. For block devices these functions must be locked
      so that block size is not changed while they are in progress.

      This patch is an additional fix for commit b87570f5d349 ("Fix a crash
      when block device is read and block size is changed at the same time")
      that locked aio_read, aio_write and mmap against block size change.

      Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

fs/buffer.c: make block-size be per-page and protected by the page lock

      This makes the buffer size handling be a per-page thing, which allows us
      to not have to worry about locking too much when changing the buffer
      size.  If a page doesn't have buffers, we still need to read the block
      size from the inode, but we can do that with ACCESS_ONCE(), so that even
      if the size is changing, we get a consistent value.

      This doesn't convert all functions - many of the buffer functions are
      used purely by filesystems, which in turn results in the buffer size
      being fixed at mount-time.  So they don't have the same consistency
      issues that the raw device access can have.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

blockdev: remove bd_block_size_semaphore again

      This reverts the block-device direct access code to the previous
      unlocked code, now that fs/buffer.c no longer needs external locking.

      With this, fs/block_dev.c is back to the original version, apart from a
      whitespace cleanup that I didn't want to revert.

      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

block: fix infinite loop in __getblk_slow

      Commit 080399aaaf35 ("block: don't mark buffers beyond end of disk as
      mapped") exposed a bug in __getblk_slow that causes mount to hang as it
      loops infinitely waiting for a buffer that lies beyond the end of the
      disk to become uptodate.

      The problem was initially reported by Torsten Hilbrich here:

	  https://lkml.org/lkml/2012/6/18/54

      and also reported independently here:

	  http://www.sysresccd.org/forums/viewtopic.php?f=13&t=4511

      and then Richard W.M.  Jones and Marcos Mello noted a few separate
      bugzillas also associated with the same issue.  This patch has been
      confirmed to fix:

	  https://bugzilla.redhat.com/show_bug.cgi?id=835019

      The main problem is here, in __getblk_slow:

	      for (;;) {
		      struct buffer_head * bh;
		      int ret;

		      bh = __find_get_block(bdev, block, size);
		      if (bh)
			      return bh;

		      ret = grow_buffers(bdev, block, size);
		      if (ret < 0)
			      return NULL;
		      if (ret == 0)
			      free_more_memory();
	      }

      __find_get_block does not find the block, since it will not be marked as
      mapped, and so grow_buffers is called to fill in the buffers for the
      associated page.  I believe the for (;;) loop is there primarily to
      retry in the case of memory pressure keeping grow_buffers from
      succeeding.  However, we also continue to loop for other cases, like the
      block lying beond the end of the disk.  So, the fix I came up with is to
      only loop when grow_buffers fails due to memory allocation issues
      (return value of 0).

      The attached patch was tested by myself, Torsten, and Rich, and was
      found to resolve the problem in call cases.

      Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
      Reported-and-Tested-by: Torsten Hilbrich <torsten.hilbrich@secunet.com>
      Tested-by: Richard W.M. Jones <rjones@redhat.com>
      Reviewed-by: Josh Boyer <jwboyer@redhat.com>
      Cc: Stable <stable@vger.kernel.org>  # 3.0+
      [ Jens is on vacation, taking this directly  - Linus ]
      --
      Stable Notes: this patch requires backport to 3.0, 3.2 and 3.3.
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

fs: vfat: reduce the worst case latencies

      When a block partition is mounted with FAT file system
      and MS_DIRSYNC option is used, some file system operations
      like create, rename shall sleep in caller's context until
      all the metadata have been committed to the non-volatile memory.
      Since this operation is blocking call for user context,
      the WRITE_SYNC option must be used instead of WRITE
      (async operation) which incur inherent latencies while
      flushing the meta-data corresponding to directory entries

      Change-Id: I41c514889873a39d564271db0a421e6c66e5ae33
      Signed-off-by: xiaogang <xiaogang@codeaurora.org>

writeback: Fix occasional slow sync(1)

      In case when system contains no dirty pages, wakeup_flusher_threads()
      will submit WB_SYNC_NONE writeback for 0 pages so wb_writeback() exits
      immediately without doing anything. Thus sync(1) will write all the
      dirty inodes from a WB_SYNC_ALL writeback pass which is slow.

      Fix the problem by using get_nr_dirty_pages() in
      wakeup_flusher_threads() instead of calculating number of dirty pages
      manually. That function also takes number of dirty inodes into account.

      Change-Id: I458027ae08d9a5a93202a7b97ace1f8da7a18a07
      CC: stable@vger.kernel.org
      Reported-by: Paul Taysom <taysom@chromium.org>
      Signed-off-by: Jan Kara <jack@suse.cz>

      # This is the 21st commit message:

      sync: don't block the flusher thread waiting on IO

      When sync does it's WB_SYNC_ALL writeback, it issues data Io and
      then immediately waits for IO completion. This is done in the
      context of the flusher thread, and hence completely ties up the
      flusher thread for the backing device until all the dirty inodes
      have been synced. On filesystems that are dirtying inodes constantly
      and quickly, this means the flusher thread can be tied up for
      minutes per sync call and hence badly affect system level write IO
      performance as the page cache cannot be cleaned quickly.

      We already have a wait loop for IO completion for sync(2), so cut
      this out of the flusher thread and delegate it to wait_sb_inodes().
      Hence we can do rapid IO submission, and then wait for it all to
      complete.

      Effect of sync on fsmark before the patch:

      FSUse%        Count         Size    Files/sec     App Overhead
      .....
	  0       640000         4096      35154.6          1026984
	  0       720000         4096      36740.3          1023844
	  0       800000         4096      36184.6           916599
	  0       880000         4096       1282.7          1054367
	  0       960000         4096       3951.3           918773
	  0      1040000         4096      40646.2           996448
	  0      1120000         4096      43610.1           895647
	  0      1200000         4096      40333.1           921048

      And a single sync pass took:

	real    0m52.407s
	user    0m0.000s
	sys     0m0.090s

      After the patch, there is no impact on fsmark results, and each
      individual sync(2) operation run concurrently with the same fsmark
      workload takes roughly 7s:

	real    0m6.930s
	user    0m0.000s
	sys     0m0.039s

      IOWs, sync is 7-8x faster on a busy filesystem and does not have an
      adverse impact on ongoing async data write operations.

      Signed-off-by: Dave Chinner <dchinner@redhat.com>
      Reviewed-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

      Change-Id: I9e55d65f5ecb2305497711d4688f0647d9346035

writeback: fix race that cause writeback hung

      There is a race between mark inode dirty and writeback thread, see the
      following scenario.  In this case, writeback thread will not run though
      there is dirty_io.

      __mark_inode_dirty()                                          bdi_writeback_workfn()
	      ...                                                       	...
	      spin_lock(&inode->i_lock);
	      ...
	      if (bdi_cap_writeback_dirty(bdi)) {
		  <<< assume wb has dirty_io, so wakeup_bdi is false.
		  <<< the following inode_dirty also have wakeup_bdi false.
		  if (!wb_has_dirty_io(&bdi->wb))
			  wakeup_bdi = true;
	      }
	      spin_unlock(&inode->i_lock);
									  <<< assume last dirty_io is removed here.
									  pages_written = wb_do_writeback(wb);
									  ...
									  <<< work_list empty and wb has no dirty_io,
									  <<< delayed_work will not be queued.
									  if (!list_empty(&bdi->work_list) ||
									      (wb_has_dirty_io(wb) && dirty_writeback_interval))
									      queue_delayed_work(bdi_wq, &wb->dwork,
										  msecs_to_jiffies(dirty_writeback_interval * 10));
	      spin_lock(&bdi->wb.list_lock);
	      inode->dirtied_when = jiffies;
	      <<< new dirty_io is added.
	      list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
	      spin_unlock(&bdi->wb.list_lock);

	      <<< though there is dirty_io, but wakeup_bdi is false,
	      <<< so writeback thread will not be waked up and
	      <<< the new dirty_io will not be flushed.
	      if (wakeup_bdi)
		  bdi_wakeup_thread_delayed(bdi);

      Writeback will run until there is a new flush work queued.  This may cause
      a lot of dirty pages stay in memory for a long time.

      Signed-off-by: Junxiao Bi <junxiao.bi@oracle.com>
      Reviewed-by: Jan Kara <jack@suse.cz>
      Cc: Fengguang Wu <fengguang.wu@intel.com>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>

      Change-Id: I973fcba5381881a003a035ffff48f64348660079

ext4: speed up truncate/unlink by not using bforget() unless needed

      Do not iterate over data blocks scanning for bh's to forget as they're
      never exist. This improves time taken by unlink / truncate syscall.
      Tested by continuously truncating file that is being written by dd.
      Another test is rm -rf of linux tree while tar unpacks it. With
      ordered data mode condition unlikely(!tbh) was always met in
      ext4_free_blocks. With journal data mode tbh was found only few times,
      so optimisation is also possible.

      Unlinking fallocated 60G file after doing sync && echo 3 >
      /proc/sys/vm/drop_caches && time rm --help

      X86 before (linux 3.6-rc4):
      real    0m2.710s
      user    0m0.000s
      sys     0m1.530s

      X86 after:
      real    0m0.644s
      user    0m0.003s
      sys     0m0.060s

      MIPS before (linux 2.6.37):
      real    0m 4.93s
      user    0m 0.00s
      sys     0m 4.61s

      MIPS after:
      real    0m 0.16s
      user    0m 0.00s
      sys     0m 0.06s

      Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: Andrey Sidorov <qrxd43@motorola.com>
      Signed-off-by: franciscofranco <franciscofranco.1990@gmail.com>

      Change-Id: Ie78a945cb82b3892eaf88701f2dc3b7726104fb5

fs: Asynchronous I/O latency to a solid-state disk greatly increased

      between the 2.6.32 and 3.0 kernels. By removing the plug from
      do_io_submit(), we observed a 34% improvement in the I/O latency.

      Unfortunately, at this level, we don't know if the request is to
      a rotating disk or not.

      # This is the 25th commit message:

      deadline: Allow 0ms deadline latency, increase the read speed

      Change a timer compare from after to after-equals, thus allowing
      0 timeout and making deadline schedule FIFO.

      Signed-off-by: xiaobing tu <xiaobing.tu@intel.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

block: Add bio_end_sector()

      Just a little convenience macro - main reason to add it now is preparing
      for immutable bio vecs, it'll reduce the size of the patch that puts
      bi_sector/bi_size/bi_idx into a struct bvec_iter.

      Signed-off-by: Kent Overstreet <koverstreet@google.com>
      CC: Jens Axboe <axboe@kernel.dk>
      CC: Lars Ellenberg <drbd-dev@lists.linbit.com>
      CC: Jiri Kosina <jkosina@suse.cz>
      CC: Alasdair Kergon <agk@redhat.com>
      CC: dm-devel@redhat.com
      CC: Neil Brown <neilb@suse.de>
      CC: Martin Schwidefsky <schwidefsky@de.ibm.com>
      CC: Heiko Carstens <heiko.carstens@de.ibm.com>
      CC: linux-s390@vger.kernel.org
      CC: Chris Mason <chris.mason@fusionio.com>
      CC: Steven Whitehouse <swhiteho@redhat.com>
      Acked-by: Steven Whitehouse <swhiteho@redhat.com>

block: Drop dead function blk_abort_queue()

      This function was only used by btrfs code in btrfs_abort_devices()
      (seems in a wrong way).

      It was removed in commit d07eb9117050c9ed3f78296ebcc06128b52693be,
      So, Let's remove the dead code to avoid any confusion.

      Changes in v2: update commit log, btrfs_abort_devices() was removed
      already.

      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: linux-kernel@vger.kernel.org
      Cc: Chris Mason <chris.mason@oracle.com>
      Cc: linux-btrfs@vger.kernel.org
      Cc: David Sterba <dave@jikos.cz>
      Signed-off-by: Asias He <asias@redhat.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

kernel:sched: LOAD_FREQ (4*HZ+61) avoids loadavg Moire

      LOAD_FREQ is (5*HZ+1) to avoid high load average when idle:
      http://kerneltrap.org/mailarchive/linux-kernel/2007/10/3/328568

      I suggest (4*HZ+61) for a better distribution.

      With some seconds based load (like SSL heartbeats)
      and LOAD_FREQ at (5*HZ+1) I see Moire patterns like inverse sawtooth,
      since 2 or 3 probes hit the jobs (load increases quickly),
      followed by several probes missing it.

      A 4.61 sec interval gives optimal distribution over when within a
      second a probe is taken, as .61 is close to golden ratio phi 1.618...
      (test in http://ripke.com/goldenratio.c).

      12*4.61 = 55.32 secs is still close to a minute,
      and 13*4.61=59.93 is even closer than the current 12*5.01=60.12
      (with exponents EXP_x adjusted to a ratio of 13 instead of 12).

block: Fix bad range check in bio_sector_offset

      DM would occasionally end up splitting data integrity-enabled requests
      incorrectly. The culprit was a bad range check in bio_sector_offset.

      Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

block: Expose kblock_schedule_delayed_work()

      This function is exported in blk-core.c to be used in other modules
      but it's definition in h file is missing.

      Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>

fs/sync: Make sync() satisfy many requests with one invocation

      Dave Jones reported RCU stalls, overly long hrtimer interrupts, and
      amazingly long NMI handlers from a trinity-induced workload involving
      lots of concurrent sync() calls (https://lkml.org/lkml/2013/7/23/369).
      There are any number of things that one might do to make sync() behave
      better under high levels of contention, but it is also the case that
      multiple concurrent sync() system calls can be satisfied by a single
      sys_sync() invocation.

      Given that this situation is reminiscent of rcu_barrier(), this commit
      applies the rcu_barrier() approach to sys_sync().  This approach uses
      a global mutex and a sequence counter.  The mutex is held across the
      sync() operation, which eliminates contention between concurrent sync()
      operations.  The counter is incremented at the beginning and end of
      each sync() operation, so that it is odd while a sync() operation is in
      progress and even otherwise, just like sequence locks.

      The code that used to be in sys_sync() is now in do_sync(), and sys_sync()
      now handles the concurrency.  The sys_sync() function first takes a
      snapshot of the counter, then acquires the mutex, and then takes another
      snapshot of the counter.  If the values of the two snapshots indicate that
      a full do_sync() executed during the mutex acquisition, the sys_sync()
      function releases the mutex and returns ("Our work is done!").  Otherwise,
      sys_sync() increments the counter, invokes do_sync(), and increments
      the counter again.

      This approach allows a single call to do_sync() to satisfy an arbitrarily
      large number of sync() system calls, which should eliminate issues due
      to large numbers of concurrent invocations of the sync() system call.

      Changes since v1 (https://lkml.org/lkml/2013/7/24/683):

      o	Add a pair of memory barriers to keep the increments from
	      bleeding into the do_sync() code.  (The failure probability
	      is insanely low, but when you have several hundred million
	      devices running Linux, you can expect several hundred instances
	      of one-in-a-million failures.)

      o	Actually CC some people who have experience in this area.

      Reported-by: Dave Jones <davej@redhat.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Alexander Viro <viro@zeniv.linux.org.uk>
      Cc: Christoph Hellwig <hch@lst.de>
      Cc: Jan Kara <jack@suse.cz>
      Cc: Curt Wohlgemuth <curtw@google.com>
      Cc: Jens Axboe <jaxboe@fusionio.com>
      Cc: linux-fsdevel@vger.kernel.org

      Signed-off-by: Paul Reioux <reioux@gmail.com>

epoll: optimize EPOLL_CTL_DEL using rcu

      Optimize EPOLL_CTL_DEL such that it does not require the 'epmutex' by
      converting the file->f_ep_links list into an rcu one. In this way, we can
      traverse the epoll network on the add path in parallel with deletes. Since
      deletes can't create loops or worse wakeup paths, this is safe.

      This patch in combination with the patch "epoll: Do not take global 'epmutex'
      for simple topologies", shows a dramatic performance improvement in
      scalability for SPECjbb.

      Tested-by: Nathan Zimmer <nzimmer@sgi.com>
      Signed-off-by: Jason Baron <jbaron@akamai.com>

      backported for Linux 3.4 for use on Android
      Signed-off-by: Paul Reioux <reioux@gmail.com>

epoll: Do not take global 'epmutex' for simple topologies

      When calling EPOLL_CTL_ADD for an epoll file descriptor that is attached
      directly to a wakeup source, we do not need to take the global 'epmutex',
      unless the epoll file descriptor is nested. The purpose of taking
      the 'epmutex' on add is to prevent complex topologies such as loops and
      deep wakeup paths from forming in parallel through multiple EPOLL_CTL_ADD
      operations. However, for the simple case of an epoll file descriptor
      attached directly to a wakeup source (with no nesting), we do not need
      to hold the 'epmutex'.

      This patch along with 'epoll: optimize EPOLL_CTL_DEL using rcu' improves
      scalability on larger systems. Quoting Nathan Zimmer's mail on SPECjbb
      performance:

      "
      On the 16 socket run the performance went from 35k jOPS to 125k jOPS.
      In addition the benchmark when from scaling well on 10 sockets to scaling well
      on just over 40 sockets.

      ...

      Currently the benchmark stops scaling at around 40-44 sockets but it seems like
      I found a second unrelated bottleneck.
      "

      Tested-by: Nathan Zimmer <nzimmer@sgi.com>
      Signed-off-by: Jason Baron <jbaron@akamai.com>

      backported to Linux 3.4 for use on Android
      Signed-off-by: Paul Reioux <reioux@gmail.com>

aio: convert the ioctx list to radix tree

      When using a large number of threads performing AIO operations the
      IOCTX list may get a significant number of entries which will cause
      significant overhead. For example, when running this fio script:

      rw=randrw; size=256k ;directory=/mnt/fio; ioengine=libaio; iodepth=1
      blocksize=1024; numjobs=512; thread; loops=100

      on an EXT2 filesystem mounted on top of a ramdisk we can observe up to
      30% CPU time spent by lookup_ioctx:

      32.51%  [guest.kernel]  [g] lookup_ioctx
	9.19%  [guest.kernel]  [g] __lock_acquire.isra.28
	4.40%  [guest.kernel]  [g] lock_release
	4.19%  [guest.kernel]  [g] sched_clock_local
	3.86%  [guest.kernel]  [g] local_clock
	3.68%  [guest.kernel]  [g] native_sched_clock
	3.08%  [guest.kernel]  [g] sched_clock_cpu
	2.64%  [guest.kernel]  [g] lock_release_holdtime.part.11
	2.60%  [guest.kernel]  [g] memcpy
	2.33%  [guest.kernel]  [g] lock_acquired
	2.25%  [guest.kernel]  [g] lock_acquire
	1.84%  [guest.kernel]  [g] do_io_submit

      This patchs converts the ioctx list to a radix tree. For a performance
      comparison the above FIO script was run on a 2 sockets 8 core
      machine. This are the results for the original list based
      implementation and for the radix tree based implementation:

      cores         1         2         4         8         16        32
      list        111025 ms  62219 ms  34193 ms  22998 ms  19335 ms  15956 ms
      radix        75400 ms  42668 ms  23923 ms  17206 ms  15820 ms  13295 ms
      % of radix
      relative      68%       69%       70%       75%       82%       83%
      to list

      To consider the impact of the patch on the typical case of having
      only one ctx per process the following FIO script was run:

      rw=randrw; size=100m ;directory=/mnt/fio; ioengine=libaio; iodepth=1
      blocksize=1024; numjobs=1; thread; loops=100

      on the same system and the results are the following:

      list        65241 ms
      radix       65402 ms
      % of radix
      relative    100.25%
      to list

      Cc: Andi Kleen <ak@linux.intel.com>

      Signed-off-by: Octavian Purdila <octavian.purdila@intel.com>
      modified for Mako hybrid from LKML

      Signed-off-by: Paul Reioux <reioux@gmail.com>
[FS: iov_iter:] *squashing updates*

iov_iter: move into its own file

      This moves the iov_iter functions in to their own file.  We're going to
      be working on them in upcoming patches.  They become sufficiently large,
      and remain self-contained, to justify seperating them from the rest of
      the huge mm/filemap.c.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Acked-by: Jeff Moyer <jmoyer@redhat.com>
      Cc: Zach Brown <zab@zabbo.net>

iov_iter: iov_iter_copy_from_user() should use non-atomic copy

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>

iov_iter: add copy_to_user support

      This adds iov_iter wrappers around copy_to_user() to match the existing
      wrappers around copy_from_user().

      This will be used by the generic file system buffered read path.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>

fuse: convert fuse to use iov_iter_copy_[to|from]_user

      A future patch hides the internals of struct iov_iter, so fuse should
      be using the supported interface.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Acked-by: Miklos Szeredi <mszeredi@suse.cz>
      Cc: fuse-devel@lists.sourceforge.net

iov_iter: hide iovec details behind ops function pointers

      This moves the current iov_iter functions behind an ops struct of
      function pointers.  The current iov_iter functions all work with memory
      which is specified by iovec arrays of user space pointers.

      This patch is part of a series that lets us specify memory with bio_vec
      arrays of page pointers.  By moving to an iov_iter operation struct we
      can add that support in later patches in this series by adding another
      set of function pointers.

      I only came to this after having initialy tried to teach the current
      iov_iter functions about bio_vecs by introducing conditional branches
      that dealt with bio_vecs in all the functions.  It wasn't pretty.  This
      approach seems to be the lesser evil.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako kernel from LKML

      Signed-off-by: faux123 <reioux@gmail.com>

iov_iter: add bvec support

      This adds a set of iov_iter_ops calls which work with memory which is
      specified by an array of bio_vec structs instead of an array of iovec
      structs.

      The big difference is that the pages referenced by the bio_vec elements
      are pinned.  They don't need to be faulted in and we can always use
      kmap_atomic() to map them one at a time.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>

iov_iter: add a shorten call

      The generic direct write path wants to shorten its memory vector.  It
      does this when it finds that it has to perform a partial write due to
      LIMIT_FSIZE.  .direct_IO() always performs IO on all of the referenced
      memory because it doesn't have an argument to specify the length of the
      IO.

      We add an iov_iter operation for this so that the generic path can ask
      to shorten the memory vector without having to know what kind it is.
      We're happy to shorten the kernel copy of the iovec array, but we refuse
      to shorten the bio_vec array and return an error in this case.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>

dio: Convert direct_IO to use iov_iter

      Change the direct_IO aop to take an iov_iter argument rather than an iovec.
      This will get passed down through most filesystems so that only the
      __blockdev_direct_IO helper need be aware of whether user or kernel memory
      is being passed to the function.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      heavily backported for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

dio: add bio_vec support to __blockdev_direct_IO()

      The trick here is to initialize the dio state so that do_direct_IO()
      consumes the pages we provide and never tries to map user pages.  This
      is done by making sure that final_block_in_request covers the page that
      we set in the dio.  do_direct_IO() will return before running out of
      pages.

      The caller is responsible for dirtying these pages, if needed.  We add
      an option to the dio struct that makes sure we only dirty pages when
      we're operating on iovecs of user addresses.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako kernel from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

aio: add aio_kernel_() interface

      This adds an interface that lets kernel callers submit aio iocbs without
      going through the user space syscalls.  This lets kernel callers avoid
      the management limits and overhead of the context.  It will also let us
      integrate aio operations with other kernel apis that the user space
      interface doesn't have access to.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

aio: add aio support for iov_iter arguments

      This adds iocb cmds which specify that memory is held in iov_iter
      structures.  This lets kernel callers specify memory that can be
      expressed in an iov_iter, which includes pages in bio_vec arrays.

      Only kernel callers can provide an iov_iter so it doesn't make a lot of
      sense to expose the IOCB_CMD values for this as part of the user space
      ABI.

      But kernel callers should also be able to perform the usual aio
      operations which suggests using the the existing operation namespace and
      support code.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

fs: create file_readable() and file_writable() functions

      Create functions to simplify if file_ops contain either a read
      or aio_read op, or likewise write or aio_write. We will be adding
      read_iter and write_iter and don't need to be complicating the code
      in multiple places.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

fs: use read_iter and write_iter rather than aio_read and aio_write

      File systems implementing read_iter & write_iter should not be required
      to keep aio_read and aio_write as well. The vfs should always call
      read/write_iter if they exist. This will make it easier to remove the
      aio_read/write operation in the future.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Alexander Viro <viro@zeniv.linux.org.uk>
      Cc: linux-fsdevel@vger.kernel.org

fs: add read_iter and write_iter to several file systems

      These are the simple ones.

      File systems that use generic_file_aio_read() and generic_file_aio_write()
      can trivially support generic_file_read_iter() and generic_file_write_iter().

      This patch adds those file_operations for 9p, adfs, affs, bfs, exofs, ext2,
      ext3, fat, f2fs, hfs, hfsplus, hostfs, hpfs, jfs, jffs2, logfs, minix, nilfs2,
      omfs, ramfs, reiserfs, romfs, sysv, and ufs.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Acked-by: Boaz Harrosh <bharrosh@panasas.com>

      ext4: add support for read_iter and write_iter

      use the generic_file_read_iter(), create ext4_file_write_iter() based on
      ext4_file_write(), and make ext4_file_write() a wrapper around
      ext4_file_write_iter().

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: linux-ext4@vger.kernel.org
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

ecrpytfs: Convert aio_read/write ops to read/write_iter

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Tyler Hicks <tyhicks@canonical.com>
      Cc: Dustin Kirkland <dustin.kirkland@gazzang.com>
      Cc: ecryptfs@vger.kernel.org

      # This is the 17th commit message:

      iov_iter: move into its own file (bug fix)

      I found the problem. iov_iter_shorten() wasn't setting i->count to the new
      value.

      This fixes it. I'll fix the patchset tomorrow.

      Thanks,
      Shaggy

fs: pull iov_iter use higher up the stack

      Right now only callers of generic_perform_write() pack their iovec
      arguments into an iov_iter structure.  All the callers higher up in the
      stack work on raw iovec arguments.

      This patch introduces the use of the iov_iter abstraction higher up the
      stack.  Private generic path functions are changed to operation on
      iov_iter instead of on raw iovecs.  Exported interfaces that take iovecs
      immediately pack their arguments into an iov_iter and call into the
      shared functions.

      File operation struct functions are added with iov_iter as an argument
      so that callers to the generic file system functions can specify
      abstract memory rather than iovec arrays only.

      Almost all of this patch only transforms arguments and shouldn't change
      functionality.  The buffered read path is the exception.  We add a
      read_actor function which uses the iov_iter helper functions instead of
      operating on each individual iovec element.  This may improve
      performance as the iov_iter helper can copy multiple iovec elements from
      one mapped page cache page.

      As always, the direct IO path is special.  Sadly, it may still be
      cleanest to have it work on the underlying memory structures directly
      instead of working through the iov_iter abstraction.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

tmpfs: add support for read_iter and write_iter

      Convert tmpfs do_shmem_file_read() to shmem_file_read_iter().
      Make file_read_iter_actor() global so tmpfs can use it too: delete
      file_read_actor(), which was made global in 2.4.4 for use by tmpfs.
      Replace tmpfs generic_file_aio_write() by generic_file_write_iter().

      Signed-off-by: Hugh Dickins <hughd@google.com>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>
rcu: Precompute RCU_FAST_NO_HZ timer offsets

When a CPU is entering dyntick-idle mode, tick_nohz_stop_sched_tick()
calls rcu_needs_cpu() see if RCU needs that CPU, and, if not, computes the
next wakeup time based on the timer wheels.  Only later, when actually
entering the idle loop, rcu_prepare_for_idle() will be invoked.  In some
cases, rcu_prepare_for_idle() will post timers to wake the CPU back up.
But all for naught: The next wakeup time for the CPU has already been
computed, and posting a timer afterwards does not force that wakeup
time to be recomputed.  This means that rcu_prepare_for_idle()'s have
no effect.

This is not a problem on a busy system because something else will wake
up the CPU soon enough.  However, on lightly loaded systems, the CPU
might stay asleep for a considerable length of time.  If that CPU has
a callback that the rest of the system is waiting on, the system might
run very slowly or (in theory) even hang.

This commit avoids this problem by having rcu_needs_cpu() give
tick_nohz_stop_sched_tick() an estimate of when RCU will need the CPU
to wake back up, which tick_nohz_stop_sched_tick() takes into account
when programming the CPU's wakeup time.  An alternative approach is
for rcu_prepare_for_idle() to use hrtimers instead of normal timers,
but timers are much more efficient than are hrtimers for frequently
and repeatedly posting and cancelling a given timer, which is exactly
what RCU_FAST_NO_HZ does.

Reported-by: Pascal Chapperon <pascal.chapperon@wanadoo.fr>
Reported-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Tested-by: Pascal Chapperon <pascal.chapperon@wanadoo.fr>
CK3 Tweaks

mm-kswapd_inherit_prio-1.patch

      When kswapd is awoken due to reclaim by a running task, set the priority of
      kswapd to that of the calling task thus making memory reclaim cpu activity
      affected by nice level.

mm-drop_swap_cache_aggressively.patch

      While it may be nice to have a copy of pages on swap once written there, the
      more garbage we leave in the swapspace the slower any further writes and
      reads to and from it are. Just free swapcache whenever we can.