[ARM: GENERIC *part2*] *squashing updates*

ARM: mm: remove IPI broadcasting on ASID rollover

      ASIDs are allocated to MMU contexts based on a rolling counter. This
      means that after 255 allocations we must invalidate all existing ASIDs
      via an expensive IPI mechanism to synchronise all of the online CPUs and
      ensure that all tasks execute with an ASID from the new generation.

      This patch changes the rollover behaviour so that we rely instead on the
      hardware broadcasting of the TLB invalidation to avoid the IPI calls.
      This works by keeping track of the active ASID on each core, which is
      then reserved in the case of a rollover so that currently scheduled
      tasks can continue to run. For cores without hardware TLB broadcasting,
      we keep track of pending flushes in a cpumask, so cores can flush their
      local TLB before scheduling a new mm.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Tested-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: mm: avoid taking ASID spinlock on fastpath

      When scheduling a new mm, we take a spinlock so that we can:

	1. Safely allocate a new ASID, if required
	2. Update our active_asids field without worrying about parallel
	  updates to reserved_asids
	3. Ensure that we flush our local TLB, if required

      However, this has the nasty affect of serialising context-switch across
      all CPUs in the system. The usual (fast) case is where the next mm has
      a valid ASID for the current generation. In such a scenario, we can
      avoid taking the lock and instead use atomic64_xchg to update the
      active_asids variable for the current CPU. If a rollover occurs on
      another CPU (which would take the lock), when copying the active_asids
      into the reserved_asids another atomic64_xchg is used to replace each
      active_asids with 0. The fast path can then detect this case and fall
      back to spinning on the lock.

      Tested-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: mm: use bitmap operations when allocating new ASIDs

      When allocating a new ASID, we must take care not to re-assign a
      reserved ASID-value to a new mm. This requires us to check each
      candidate ASID against those currently reserved by other cores before
      assigning a new ASID to the current mm.

      This patch improves the ASID allocation algorithm by using a
      bitmap-based approach. Rather than iterating over the reserved ASID
      array for each candidate ASID, we simply find the first zero bit,
      ensuring that those indices corresponding to reserved ASIDs are set
      when flushing during a rollover event.

      Tested-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7658/1: mm: fix race updating mm->context.id on ASID rollover

      If a thread triggers an ASID rollover, other threads of the same process
      must be made to wait until the mm->context.id for the shared mm_struct
      has been updated to new generation and associated book-keeping (e.g.
      TLB invalidation) has ben performed.

      However, there is a *tiny* window where both mm->context.id and the
      relevant active_asids entry are updated to the new generation, but the
      TLB flush has not been performed, which could allow another thread to
      return to userspace with a dirty TLB, potentially leading to data
      corruption. In reality this will never occur because one CPU would need
      to perform a context-switch in the time it takes another to do a couple
      of atomic test/set operations but we should plug the race anyway.

      This patch moves the active_asids update until after the potential TLB
      flush on context-switch.

      Cc: <stable@vger.kernel.org> # 3.8
      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7659/1: mm: make mm->context.id an atomic64_t variable

      mm->context.id is updated under asid_lock when a new ASID is allocated
      to an mm_struct. However, it is also read without the lock when a task
      is being scheduled and checking whether or not the current ASID
      generation is up-to-date.

      If two threads of the same process are being scheduled in parallel and
      the bottom bits of the generation in their mm->context.id match the
      current generation (that is, the mm_struct has not been used for ~2^24
      rollovers) then the non-atomic, lockless access to mm->context.id may
      yield the incorrect ASID.

      This patch fixes this issue by making mm->context.id and atomic64_t,
      ensuring that the generation is always read consistently. For code that
      only requires access to the ASID bits (e.g. TLB flushing by mm), then
      the value is accessed directly, which GCC converts to an ldrb.

      Cc: <stable@vger.kernel.org> # 3.8
      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7661/1: mm: perform explicit branch predictor maintenance when required

      The ARM ARM requires branch predictor maintenance if, for a given ASID,
      the instructions at a specific virtual address appear to change.

      From the kernel's point of view, that means:

	      - Changing the kernel's view of memory (e.g. switching to the
		identity map)
	      - ASID rollover (since ASIDs will be re-allocated to new tasks)

      This patch adds explicit branch predictor maintenance when either of the
      two conditions above are met.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7767/1: let the ASID allocator handle suspended animation

      When a CPU is running a process, the ASID for that process is
      held in a per-CPU variable (the "active ASIDs" array). When
      the ASID allocator handles a rollover, it copies the active
      ASIDs into a "reserved ASIDs" array to ensure that a process
      currently running on another CPU will continue to run unaffected.
      The active array is zero-ed to indicate that a rollover occurred.

      Because of this mechanism, a reserved ASID is only remembered for
      a single rollover. A subsequent rollover will completely refill
      the reserved ASIDs array.

      In a severely oversubscribed environment where a CPU can be
      prevented from running for extended periods of time (think virtual
      machines), the above has a horrible side effect:

      [P{a} denotes process P running with ASID a]

	      CPU-0		CPU-1

	      A{x}				[active = <x 0>]

	      [suspended]	runs B{y}	[active = <x y>]

					      [rollover:
					      active = <0 0>
					      reserved = <x y>]

			      runs B{y}	[active = <0 y>
					      reserved = <x y>]

					      [rollover:
					      active = <0 0>
					      reserved = <0 y>]

			      runs C{x}	[active = <0 x>]

	      [resumes]

	      runs A{x}

      At that stage, both A and C have the same ASID, with deadly
      consequences.

      The fix is to preserve reserved ASIDs across rollovers if
      the CPU doesn't have an active ASID when the rollover occurs.

      Cc: <stable@vger.kernel.org> # 3.9
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Catalin Carinas <catalin.marinas@arm.com>
      Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: 7768/1: prevent risks of out-of-bound access in ASID allocator

      On a CPU that never ran anything, both the active and reserved ASID
      fields are set to zero. In this case the ASID_TO_IDX() macro will
      return -1, which is not a very useful value to index a bitmap.

      Instead of trying to offset the ASID so that ASID #1 is actually
      bit 0 in the asid_map bitmap, just always ignore bit 0 and start
      the search from bit 1. This makes the code a bit more readable,
      and without risk of OoB access.

      Cc: <stable@vger.kernel.org> # 3.9
      Acked-by: Will Deacon <will.deacon@arm.com>
      Acked-by: Catalin Marinas <catalin.marinas@arm.com>
      Reported-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: tlb: don't perform inner-shareable invalidation for local TLB ops

      Inner-shareable TLB invalidation is typically more expensive than local
      (non-shareable) invalidation, so performing the broadcasting for
      local_flush_tlb_* operations is a waste of cycles and needlessly
      clobbers entries in the TLBs of other CPUs.

      This patch introduces __flush_tlb_* versions for many of the TLB
      invalidation functions, which only respect inner-shareable variants of
      the invalidation instructions when presented with the TLB_V7_UIS_FULL
      flag. The local version is also inlined to prevent SMP_ON_UP kernels
      from missing flushes, where the __flush variant would be called with
      the UP flags.

      This gains us around 0.5% in hackbench scores for a dual-core A15, but I
      would expect this to improve as more cores (and clusters) are added to
      the equation.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Reported-by: Albin Tonnerre <Albin.Tonnerre@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>

ARM: 7660/1: tlb: add branch predictor maintenance operations

      The ARM architecture requires explicit branch predictor maintenance
      when updating an instruction stream for a given virtual address. In
      reality, this isn't so much of a burden because the branch predictor
      is flushed during the cache maintenance required to make the new
      instructions visible to the I-side of the processor.

      However, there are still some cases where explicit flushing is required,
      so add a local_bp_flush_all operation to deal with this.

      Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: Will Deacon <will.deacon@arm.com>
      Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

ARM: nommu: add stub local_flush_bp_all() for !CONFIG_MMUU

      Since the merging of Will's tlb-ops branch, specifically 89c7e4b8bbb3
      (ARM: 7661/1: mm: perform explicit branch predictor maintenance when required),
      building SMP without CONFIG_MMU has been broken.

      The local_flush_bp_all function is only called for operations related to
      changing the kernel's view of memory and ASID rollover - both of which are
      irrelevant to an !MMU kernel.

      This patch adds a stub local_flush_bp_all() function to the other tlb
      maintenance stubs and restores the ability to build an SMP !MMU kernel.

      Signed-off-by: Jonathan Austin <jonathan.austin@arm.com>
      Acked-by: Will Deacon <will.deacon@arm.com>
byteorder: allow arch to opt to use GCC intrinsics for byteswapping

Since GCC 4.4, there have been __builtin_bswap32() and __builtin_bswap16()
intrinsics. A __builtin_bswap16() came a little later (4.6 for PowerPC,
48 for other platforms).

By using these instead of the inline assembler that most architectures
have in their __arch_swabXX() macros, we let the compiler see what's
actually happening. The resulting code should be at least as good, and
much *better* in the cases where it can be combined with a nearby load
or store, using a load-and-byteswap or store-and-byteswap instruction
(e.g. lwbrx/stwbrx on PowerPC, movbe on Atom).

When GCC is sufficiently recent *and* the architecture opts in to using
the intrinsics by setting CONFIG_ARCH_USE_BUILTIN_BSWAP, they will be
used in preference to the __arch_swabXX() macros. An architecture which
does not set ARCH_USE_BUILTIN_BSWAP will continue to use its own
hand-crafted macros.

Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
Acked-by: H. Peter Anvin <hpa@linux.intel.com>
binfmt_elf.c: use get_random_int() to fix entropy depleting.

Usually the entropy pool depletes very fast and stays in the 100-200 range.
After this patch, the entropy stays in the 600-1000 range without the risk of
depleting in normal workloads.

Change-Id: Ice0d0f14691046249d7b4a4d31af98ab80c6306b
introduce for_each_thread() to replace the buggy while_each_thread()

while_each_thread() and next_thread() should die, almost every lockless
usage is wrong.

1. Unless g == current, the lockless while_each_thread() is not safe.

   while_each_thread(g, t) can loop forever if g exits, next_thread()
   can't reach the unhashed thread in this case. Note that this can
   happen even if g is the group leader, it can exec.

2. Even if while_each_thread() itself was correct, people often use
   it wrongly.

   It was never safe to just take rcu_read_lock() and loop unless
   you verify that pid_alive(g) == T, even the first next_thread()
   can point to the already freed/reused memory.

This patch adds signal_struct->thread_head and task->thread_node to
create the normal rcu-safe list with the stable head.  The new
for_each_thread(g, t) helper is always safe under rcu_read_lock() as
long as this task_struct can't go away.

Note: of course it is ugly to have both task_struct->thread_node and the
old task_struct->thread_group, we will kill it later, after we change
the users of while_each_thread() to use for_each_thread().

Perhaps we can kill it even before we convert all users, we can
reimplement next_thread(t) using the new thread_head/thread_node.  But
we can't do this right now because this will lead to subtle behavioural
changes.  For example, do/while_each_thread() always sees at least one
task, while for_each_thread() can do nothing if the whole thread group
has died.  Or thread_group_empty(), currently its semantics is not clear
unless thread_group_leader(p) and we need to audit the callers before we
can change it.

So this patch adds the new interface which has to coexist with the old
one for some time, hopefully the next changes will be more or less
straightforward and the old one will go away soon.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Sergey Dyasly <dserrg@gmail.com>
Tested-by: Sergey Dyasly <dserrg@gmail.com>
Reviewed-by: Sameer Nanda <snanda@chromium.org>
Acked-by: David Rientjes <rientjes@google.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mandeep Singh Baines <msb@chromium.org>
Cc: "Ma, Xindong" <xindong.ma@intel.com>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
suspend: enable freeze timeout configuration through sys

At present, the value of timeout for freezing is 20s, which is
meaningless in case that one thread is frozen with mutex locked
and another thread is trying to lock the mutex, as this time of
freezing will fail unavoidably.
And if there is no new wakeup event registered, the system will
waste at most 20s for such meaningless trying of freezing.

With this patch, the value of timeout can be configured to smaller
value, so such meaningless trying of freezing will be aborted in
earlier time, and later freezing can be also triggered in earlier
time. And more power will be saved.
In normal case on mobile phone, it costs real little time to freeze
processes. On some platform, it only costs about 20ms to freeze
user space processes and 10ms to freeze kernel freezable threads.

Signed-off-by: Liu Chuansheng <chuansheng.liu@intel.com>
Signed-off-by: Li Fei <fei.li@intel.com>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
[FREEZER]: *squashing updates by Colin Cross (05/2013)*

freezer: add unsafe versions of freezable helpers for NFS

      NFS calls the freezable helpers with locks held, which is unsafe
      and will cause lockdep warnings when 6aa9707 "lockdep: check
      that no locks held at freeze time" is reapplied (it was reverted
      in dbf520a).  NFS shouldn't be doing this, but it has
      long-running syscalls that must hold a lock but also shouldn't
      block suspend.  Until NFS freeze handling is rewritten to use a
      signal to exit out of the critical section, add new *_unsafe
      versions of the helpers that will not run the lockdep test when
      6aa9707 is reapplied, and call them from NFS.

      In practice the likley result of holding the lock while freezing
      is that a second task blocked on the lock will never freeze,
      aborting suspend, but it is possible to manufacture a case using
      the cgroup freezer, the lock, and the suspend freezer to create
      a deadlock.  Silencing the lockdep warning here will allow
      problems to be found in other drivers that may have a more
      serious deadlock risk, and prevent new problems from being added.

      Signed-off-by: Colin Cross <ccross@android.com>
      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: add unsafe versions of freezable helpers for CIFS

      CIFS calls wait_event_freezekillable_unsafe with a VFS lock held,
      which is unsafe and will cause lockdep warnings when 6aa9707
      "lockdep: check that no locks held at freeze time" is reapplied
      (it was reverted in dbf520a).  CIFS shouldn't be doing this, but
      it has long-running syscalls that must hold a lock but also
      shouldn't block suspend.  Until CIFS freeze handling is rewritten
      to use a signal to exit out of the critical section, add a new
      wait_event_freezekillable_unsafe helper that will not run the
      lockdep test when 6aa9707 is reapplied, and call it from CIFS.

      In practice the likley result of holding the lock while freezing
      is that a second task blocked on the lock will never freeze,
      aborting suspend, but it is possible to manufacture a case using
      the cgroup freezer, the lock, and the suspend freezer to create
      a deadlock.  Silencing the lockdep warning here will allow
      problems to be found in other drivers that may have a more
      serious deadlock risk, and prevent new problems from being added.

      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Reviewed-by: Jeff Layton <jlayton@redhat.com>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

lockdep: remove task argument from debug_check_no_locks_held

      The only existing caller to debug_check_no_locks_held calls it
      with 'current' as the task, and the freezer needs to call
      debug_check_no_locks_held but doesn't already have a current
      task pointer, so remove the argument.  It is already assuming
      that the current task is relevant by dumping the current stack
      trace as part of the warning.

      This was originally part of 6aa9707099c (lockdep: check that
      no locks held at freeze time) which was reverted in
      dbf520a9d7d4.

      Original-author: Mandeep Singh Baines <msb@chromium.org>
      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: shorten freezer sleep time using exponential backoff

      All tasks can easily be frozen in under 10 ms, switch to using
      an initial 1 ms sleep followed by exponential backoff until
      8 ms.  Also convert the printed time to ms instead of centiseconds.

      Acked-by: Pavel Machek <pavel@ucw.cz>
      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: skip waking up tasks with PF_FREEZER_SKIP set

      Android goes through suspend/resume very often (every few seconds when
      on a busy wifi network with the screen off), and a significant portion
      of the energy used to go in and out of suspend is spent in the
      freezer.  If a task has called freezer_do_not_count(), don't bother
      waking it up.  If it happens to wake up later it will call
      freezer_count() and immediately enter the refrigerator.

      Combined with patches to convert freezable helpers to use
      freezer_do_not_count() and convert common sites where idle userspace
      tasks are blocked to use the freezable helpers, this reduces the
      time and energy required to suspend and resume.

      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Pavel Machek <pavel@ucw.cz>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: convert freezable helpers to freezer_do_not_count()

      Freezing tasks will wake up almost every userspace task from
      where it is blocking and force it to run until it hits a
      call to try_to_sleep(), generally on the exit path from the syscall
      it is blocking in.  On resume each task will run again, usually
      restarting the syscall and running until it hits the same
      blocking call as it was originally blocked in.

      Convert the existing wait_event_freezable* wrappers to use
      freezer_do_not_count().  Combined with a previous patch,
      these tasks will not run during suspend or resume unless they wake
      up for another reason, in which case they will run until they hit
      the try_to_freeze() in freezer_count(), and then continue processing
      the wakeup after tasks are thawed.

      This results in a small change in behavior, previously a race
      between freezing and a normal wakeup would be won by the wakeup,
      now the task will freeze and then handle the wakeup after thawing.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: convert freezable helpers to static inline where possible

      Some of the freezable helpers have to be macros because their
      condition argument needs to get evaluated every time through
      the wait loop.  Convert the others to static inline to make
      future changes easier.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

freezer: add new freezable helpers using freezer_do_not_count()

      Freezing tasks will wake up almost every userspace task from
      where it is blocking and force it to run until it hits a
      call to try_to_sleep(), generally on the exit path from the syscall
      it is blocking in.  On resume each task will run again, usually
      restarting the syscall and running until it hits the same
      blocking call as it was originally blocked in.

      To allow tasks to avoid running on every suspend/resume cycle,
      this patch adds additional freezable wrappers around blocking calls
      that call freezer_do_not_count().  Combined with the previous patch,
      these tasks will not run during suspend or resume unless they wake
      up for another reason, in which case they will run until they hit
      the try_to_freeze() in freezer_count(), and then continue processing
      the wakeup after tasks are thawed.

      Additional patches will convert the most common locations that
      userspace blocks in to use freezable helpers.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

binder: use freezable blocking calls

      Avoid waking up every thread sleeping in a binder call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

epoll: use freezable blocking call

      Avoid waking up every thread sleeping in an epoll_wait call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

select: use freezable blocking call

      Avoid waking up every thread sleeping in a select call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

futex: use freezable blocking call

      Avoid waking up every thread sleeping in a futex_wait call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Thomas Gleixner <tglx@linutronix.de>
      Acked-by: Darren Hart <dvhart@linux.intel.com>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

nanosleep: use freezable blocking call

      Avoid waking up every thread sleeping in a nanosleep call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

sigtimedwait: use freezable blocking call

      Avoid waking up every thread sleeping in a sigtimedwait call during
      suspend and resume by calling a freezable blocking call.  Previous
      patches modified the freezer to avoid sending wakeups to threads
      that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

af_unix: use freezable blocking calls in read

      Avoid waking up every thread sleeping in read call on an AF_UNIX
      socket during suspend and resume by calling a freezable blocking
      call.  Previous patches modified the freezer to avoid sending
      wakeups to threads that are blocked in freezable blocking calls.

      This call was selected to be converted to a freezable call because
      it doesn't hold any locks or release any resources when interrupted
      that might be needed by another freezing task or a kernel driver
      during suspend, and is a common site where idle userspace tasks are
      blocked.

      Acked-by: Tejun Heo <tj@kernel.org>
      Signed-off-by: Colin Cross <ccross@android.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

HACK: ARM: disable sleeping while atomic warning in do_signal

      ARM disables interrupts in do_signal, which triggers a warning in
      try_to_freeze, see details at https://lkml.org/lkml/2011/8/23/221.
      To prevent the warnings, add try_to_freeze_nowarn and call it from
      do_signal.

      Change-Id: If7482de21c386adc705fa1ac4ecb8c7ece5bb356
      Signed-off-by: Colin Cross <ccross@android.com>
[CPUFREQ] *optimizations*

*Reworked MSM-CPUfreq driver and ONDEMAND governor*

Signed-off-by: moddingg33k <moddingg33k@t-online.de>
[SCHED] *squashing updates*

sched: Fix hotplug vs. set_cpus_allowed_ptr()

      Lai found that:

	WARNING: CPU: 1 PID: 13 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x2d/0x4b()
	...
	migration_cpu_stop+0x1d/0x22

      was caused by set_cpus_allowed_ptr() assuming that cpu_active_mask is
      always a sub-set of cpu_online_mask.

      This isn't true since 5fbd036b552f ("sched: Cleanup cpu_active madness").

      So set active and online at the same time to avoid this particular
      problem.

      Fixes: 5fbd036b552f ("sched: Cleanup cpu_active madness")
      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Peter Zijlstra <peterz@infradead.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Michael wang <wangyun@linux.vnet.ibm.com>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
      Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Cc: Toshi Kani <toshi.kani@hp.com>
      Link: http://lkml.kernel.org/r/53758B12.8060609@cn.fujitsu.com
      Signed-off-by: Ingo Molnar <mingo@kernel.org>

sched: Improve scalability via 'CPU buddies', which withstand random perturbations

      Traversing an entire package is not only expensive, it also leads to tasks
      bouncing all over a partially idle and possible quite large package.  Fix
      that up by assigning a 'buddy' CPU to try to motivate.  Each buddy may try
      to motivate that one other CPU, if it's busy, tough, it may then try its
      SMT sibling, but that's all this optimization is allowed to cost.

      Sibling cache buddies are cross-wired to prevent bouncing.

      4 socket 40 core + SMT Westmere box, single 30 sec tbench runs, higher is better:

      clients     1       2       4        8       16       32       64      128
      ..........................................................................
      pre        30      41     118      645     3769     6214    12233    14312
      post      299     603    1211     2418     4697     6847    11606    14557

      A nice increase in performance.

      Signed-off-by: Mike Galbraith <efault@gmx.de>
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Link: http://lkml.kernel.org/r/1339471112.7352.32.camel@marge.simpson.net
      Signed-off-by: Ingo Molnar <mingo@kernel.org>

sched: Reorder 'struct lb_env' members to reduce its size

      Members of 'struct lb_env' are not in appropriate order to reuse compiler
      added padding on 64bit architectures. In this patch we reorder those struct
      members and help reduce the size of the structure from 96 bytes to 80
      bytes on 64 bit architectures.

      Suggested-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
      Signed-off-by: Prashanth Nageshappa <prashanth@linux.vnet.ibm.com>
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Link: http://lkml.kernel.org/r/4FE06DDE.7000403@linux.vnet.ibm.com
      Signed-off-by: Ingo Molnar <mingo@kernel.org>

sched: Reset loop counters if all tasks are pinned and we need to redo load balance

      While load balancing, if all tasks on the source runqueue are pinned,
      we retry after excluding the corresponding source cpu. However, loop counters
      env.loop and env.loop_break are not reset before retrying, which can lead
      to failure in moving the tasks. In this patch we reset env.loop and
      env.loop_break to their inital values before we retry.

      Signed-off-by: Prashanth Nageshappa <prashanth@linux.vnet.ibm.com>
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Link: http://lkml.kernel.org/r/4FE06EEF.2090709@linux.vnet.ibm.com
      Signed-off-by: Ingo Molnar <mingo@kernel.org>
touchscreen: adding fuzz to improve ovation sensitivity

Adding a fuzz factor to improve X/Y touch registration.
mm, mempolicy: remove per-process flag

PF_MEMPOLICY is an unnecessary optimization for CONFIG_SLAB users.
There's no significant performance degradation to checking
current->mempolicy rather than current->flags & PF_MEMPOLICY in the
allocation path, especially since this is considered unlikely().

Running TCP_RR with netperf-2.4.5 through localhost on 16 cpu machine with
64GB of memory and without a mempolicy:

	threads		before		after
	16		1249409		1244487
	32		1281786		1246783
	48		1239175		1239138
	64		1244642		1241841
	80		1244346		1248918
	96		1266436		1254316
	112		1307398		1312135
	128		1327607		1326502

Per-process flags are a scarce resource so we should free them up whenever
possible and make them available.  We'll be using it shortly for memcg oom
reserves.

Signed-off-by: David Rientjes <rientjes@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Jianguo Wu <wujianguo@huawei.com>
Cc: Tim Hockin <thockin@google.com>
Cc: Christoph Lameter <cl@linux.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
mm: per-thread vma caching

This patch is a continuation of efforts trying to optimize find_vma(),
avoiding potentially expensive rbtree walks to locate a vma upon faults.
The original approach (https://lkml.org/lkml/2013/11/1/410), where the
largest vma was also cached, ended up being too specific and random,
thus further comparison with other approaches were needed.  There are
two things to consider when dealing with this, the cache hit rate and
the latency of find_vma().  Improving the hit-rate does not necessarily
translate in finding the vma any faster, as the overhead of any fancy
caching schemes can be too high to consider.

We currently cache the last used vma for the whole address space, which
provides a nice optimization, reducing the total cycles in find_vma() by
up to 250%, for workloads with good locality.  On the other hand, this
simple scheme is pretty much useless for workloads with poor locality.
Analyzing ebizzy runs shows that, no matter how many threads are
running, the mmap_cache hit rate is less than 2%, and in many situations
below 1%.

The proposed approach is to replace this scheme with a small per-thread
cache, maximizing hit rates at a very low maintenance cost.
Invalidations are performed by simply bumping up a 32-bit sequence
number.  The only expensive operation is in the rare case of a seq
number overflow, where all caches that share the same address space are
flushed.  Upon a miss, the proposed replacement policy is based on the
page number that contains the virtual address in question.  Concretely,
the following results are seen on an 80 core, 8 socket x86-64 box:

1) System bootup: Most programs are single threaded, so the per-thread
   scheme does improve ~50% hit rate by just adding a few more slots to
   the cache.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 50.61%   | 19.90            |
| patched        | 73.45%   | 13.58            |
+----------------+----------+------------------+

2) Kernel build: This one is already pretty good with the current
   approach as we're dealing with good locality.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 75.28%   | 11.03            |
| patched        | 88.09%   | 9.31             |
+----------------+----------+------------------+

3) Oracle 11g Data Mining (4k pages): Similar to the kernel build workload.

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 70.66%   | 17.14            |
| patched        | 91.15%   | 12.57            |
+----------------+----------+------------------+

4) Ebizzy: There's a fair amount of variation from run to run, but this
   approach always shows nearly perfect hit rates, while baseline is just
   about non-existent.  The amounts of cycles can fluctuate between
   anywhere from ~60 to ~116 for the baseline scheme, but this approach
   reduces it considerably.  For instance, with 80 threads:

+----------------+----------+------------------+
| caching scheme | hit-rate | cycles (billion) |
+----------------+----------+------------------+
| baseline       | 1.06%    | 91.54            |
| patched        | 99.97%   | 14.18            |
+----------------+----------+------------------+

[akpm@linux-foundation.org: fix nommu build, per Davidlohr]
[akpm@linux-foundation.org: document vmacache_valid() logic]
[akpm@linux-foundation.org: attempt to untangle header files]
[akpm@linux-foundation.org: add vmacache_find() BUG_ON]
[hughd@google.com: add vmacache_valid_mm() (from Oleg)]
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: adjust and enhance comments]
Signed-off-by: Davidlohr Bueso <davidlohr@hp.com>
Reviewed-by: Rik van Riel <riel@redhat.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Michel Lespinasse <walken@google.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Tested-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[CPU HotPlug] Provide lockless versions of callback registration functions

CPU hotplug: Add lockdep annotations to get/put_online_cpus()

      Add lockdep annotations for get/put_online_cpus() and
      cpu_hotplug_begin()/cpu_hotplug_end().

      Cc: Ingo Molnar <mingo@kernel.org>
      Reviewed-by: Oleg Nesterov <oleg@redhat.com>
      Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

CPU hotplug: Provide lockless versions of callback registration functions

      The following method of CPU hotplug callback registration is not safe
      due to the possibility of an ABBA deadlock involving the cpu_add_remove_lock
      and the cpu_hotplug.lock.

	      get_online_cpus();

	      for_each_online_cpu(cpu)
		      init_cpu(cpu);

	      register_cpu_notifier(&foobar_cpu_notifier);

	      put_online_cpus();

      The deadlock is shown below:

		CPU 0                                         CPU 1
		-----                                         -----

	Acquire cpu_hotplug.lock
	[via get_online_cpus()]

						    CPU online/offline operation
						    takes cpu_add_remove_lock
						    [via cpu_maps_update_begin()]

	Try to acquire
	cpu_add_remove_lock
	[via register_cpu_notifier()]

						    CPU online/offline operation
						    tries to acquire cpu_hotplug.lock
						    [via cpu_hotplug_begin()]

				  *** DEADLOCK! ***

      The problem here is that callback registration takes the locks in one order
      whereas the CPU hotplug operations take the same locks in the opposite order.
      To avoid this issue and to provide a race-free method to register CPU hotplug
      callbacks (along with initialization of already online CPUs), introduce new
      variants of the callback registration APIs that simply register the callbacks
      without holding the cpu_add_remove_lock during the registration. That way,
      we can avoid the ABBA scenario. However, we will need to hold the
      cpu_add_remove_lock throughout the entire critical section, to protect updates
      to the callback/notifier chain.

      This can be achieved by writing the callback registration code as follows:

	      cpu_maps_update_begin(); [ or cpu_notifier_register_begin(); see below ]

	      for_each_online_cpu(cpu)
		      init_cpu(cpu);

	      /* This doesn't take the cpu_add_remove_lock */
	      __register_cpu_notifier(&foobar_cpu_notifier);

	      cpu_maps_update_done();  [ or cpu_notifier_register_done(); see below ]

      Note that we can't use get_online_cpus() here instead of cpu_maps_update_begin()
      because the cpu_hotplug.lock is dropped during the invocation of CPU_POST_DEAD
      notifiers, and hence get_online_cpus() cannot provide the necessary
      synchronization to protect the callback/notifier chains against concurrent
      reads and writes. On the other hand, since the cpu_add_remove_lock protects
      the entire hotplug operation (including CPU_POST_DEAD), we can use
      cpu_maps_update_begin/done() to guarantee proper synchronization.

      Also, since cpu_maps_update_begin/done() is like a super-set of
      get/put_online_cpus(), the former naturally protects the critical sections
      from concurrent hotplug operations.

      Since the names cpu_maps_update_begin/done() don't make much sense in CPU
      hotplug callback registration scenarios, we'll introduce new APIs named
      cpu_notifier_register_begin/done() and map them to cpu_maps_update_begin/done().

      In summary, introduce the lockless variants of un/register_cpu_notifier() and
      also export the cpu_notifier_register_begin/done() APIs for use by modules.
      This way, we provide a race-free way to register hotplug callbacks as well as
      perform initialization for the CPUs that are already online.

      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Acked-by: Oleg Nesterov <oleg@redhat.com>
      Acked-by: Toshi Kani <toshi.kani@hp.com>
      Reviewed-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Doc/cpu-hotplug: Specify race-free way to register CPU hotplug callbacks

      Recommend the usage of the new CPU hotplug callback registration APIs
      (__register_cpu_notifier() etc), when subsystems need to also perform
      initialization for already online CPUs. Provide examples of correct
      and race-free ways of achieving this, and point out the kinds of code
      that are error-prone.

      Cc: Rob Landley <rob@landley.net>
      Cc: Ingo Molnar <mingo@kernel.org>
      Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

---------------------------------------------------------------------------------------------------

CPU hotplug, perf: Fix CPU hotplug callback registration

arm, hw-breakpoint: Fix CPU hotplug callback registration

zsmalloc: Fix CPU hotplug callback registration

drivers/base/topology.c: Fix CPU hotplug callback registration

oprofile, nmi-timer: Fix CPU hotplug callback registration

profile: Fix CPU hotplug callback registration

mm, vmstat: Fix CPU hotplug callback registration

net/core/flow.c: Fix CPU hotplug callback registration

net/iucv/iucv.c: Fix CPU hotplug callback registration
[SOFTIRQ] *squashing updates*

mm: allow PF_MEMALLOC from softirq context

      This is needed to allow network softirq packet processing to make use of
      PF_MEMALLOC.

      Currently softirq context cannot use PF_MEMALLOC due to it not being
      associated with a task, and therefore not having task flags to fiddle with
      - thus the gfp to alloc flag mapping ignores the task flags when in
      interrupts (hard or soft) context.

      Allowing softirqs to make use of PF_MEMALLOC therefore requires some
      trickery.  This patch borrows the task flags from whatever process happens
      to be preempted by the softirq.  It then modifies the gfp to alloc flags
      mapping to not exclude task flags in softirq context, and modify the
      softirq code to save, clear and restore the PF_MEMALLOC flag.

      The save and clear, ensures the preempted task's PF_MEMALLOC flag doesn't
      leak into the softirq.  The restore ensures a softirq's PF_MEMALLOC flag
      cannot leak back into the preempted process.  This should be safe due to
      the following reasons

      Softirqs can run on multiple CPUs sure but the same task should not be
	      executing the same softirq code. Neither should the softirq
	      handler be preempted by any other softirq handler so the flags
	      should not leak to an unrelated softirq.

      Softirqs re-enable hardware interrupts in __do_softirq() so can be
	      preempted by hardware interrupts so PF_MEMALLOC is inherited
	      by the hard IRQ. However, this is similar to a process in
	      reclaim being preempted by a hardirq. While PF_MEMALLOC is
	      set, gfp_to_alloc_flags() distinguishes between hard and
	      soft irqs and avoids giving a hardirq the ALLOC_NO_WATERMARKS
	      flag.

      If the softirq is deferred to ksoftirq then its flags may be used
	      instead of a normal tasks but as the softirq cannot be preempted,
	      the PF_MEMALLOC flag does not leak to other code by accident.

      [davem@davemloft.net: Document why PF_MEMALLOC is safe]
      Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: Mel Gorman <mgorman@suse.de>
      Cc: David Miller <davem@davemloft.net>
      Cc: Neil Brown <neilb@suse.de>
      Cc: Mike Christie <michaelc@cs.wisc.edu>
      Cc: Eric B Munson <emunson@mgebm.net>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Christoph Lameter <cl@linux.com>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

softirq: reduce latencies

      In various network workloads, __do_softirq() latencies can be up
      to 20 ms if HZ=1000, and 200 ms if HZ=100.

      This is because we iterate 10 times in the softirq dispatcher,
      and some actions can consume a lot of cycles.

      This patch changes the fallback to ksoftirqd condition to :

      - A time limit of 2 ms.
      - need_resched() being set on current task

      When one of this condition is met, we wakeup ksoftirqd for further
      softirq processing if we still have pending softirqs.

      Using need_resched() as the only condition can trigger RCU stalls,
      as we can keep BH disabled for too long.

      I ran several benchmarks and got no significant difference in
      throughput, but a very significant reduction of latencies (one order
      of magnitude) :

      In following bench, 200 antagonist "netperf -t TCP_RR" are started in
      background, using all available cpus.

      Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
      IRQ (hard+soft)

      Before patch :

      # netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
      RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
      MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
      to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
      RT_LATENCY=550110.424
      MIN_LATENCY=146858
      MAX_LATENCY=997109
      P50_LATENCY=305000
      P90_LATENCY=550000
      P99_LATENCY=710000
      MEAN_LATENCY=376989.12
      STDDEV_LATENCY=184046.92

      After patch :

      # netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
      RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
      MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
      to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
      RT_LATENCY=40545.492
      MIN_LATENCY=9834
      MAX_LATENCY=78366
      P50_LATENCY=33583
      P90_LATENCY=59000
      P99_LATENCY=69000
      MEAN_LATENCY=38364.67
      STDDEV_LATENCY=12865.26

      Signed-off-by: Eric Dumazet <edumazet@google.com>
      Cc: David Miller <davem@davemloft.net>
      Cc: Tom Herbert <therbert@google.com>
      Cc: Ben Hutchings <bhutchings@solarflare.com>
      Signed-off-by: David S. Miller <davem@davemloft.net>

irq: Ensure irq_exit() code runs with interrupts disabled

      We had already a few problems with code called from irq_exit() when
      interrupted from a nesting interrupt. This can happen on architectures
      which do not define __ARCH_IRQ_EXIT_IRQS_DISABLED.

      __ARCH_IRQ_EXIT_IRQS_DISABLED should go away and we want to make it
      mandatory to call irq_exit() with interrupts disabled.

      As a temporary protection disable interrupts for those architectures
      which do not define __ARCH_IRQ_EXIT_IRQS_DISABLED and add a WARN_ONCE
      when an architecture which defines __ARCH_IRQ_EXIT_IRQS_DISABLED calls
      irq_exit() with interrupts enabled.

      Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linuxfoundation.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302202155320.22263@ionos

irq: Sanitize invoke_softirq

      With the irq protection in irq_exit, we can remove the #ifdeffery and
      the bh_disable/enable dance in invoke_softirq()

      Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Linus Torvalds <torvalds@linuxfoundation.org>
      Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302202155320.22263@ionos

      # This is the 5th commit message:

      irq: Remove IRQ_EXIT_OFFSET workaround

      The IRQ_EXIT_OFFSET trick was used to make sure the irq
      doesn't get preempted after we substract the HARDIRQ_OFFSET
      until we are entirely done with any code in irq_exit().

      This workaround was necessary because some archs may call
      irq_exit() with irqs enabled and there is still some code
      in the end of this function that is not covered by the
      HARDIRQ_OFFSET but want to stay non-preemptible.

      Now that irq are always disabled in irq_exit(), the whole code
      is guaranteed not to be preempted. We can thus remove this hack.

      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

irq: Don't re-enable interrupts at the end of irq_exit

      Commit 74eed0163d0def3fce27228d9ccf3d36e207b286
      "irq: Ensure irq_exit() code runs with interrupts disabled"
      restore interrupts flags in the end of irq_exit() for archs
      that don't define __ARCH_IRQ_EXIT_IRQS_DISABLED.

      However always returning from irq_exit() with interrupts
      disabled should not be a problem for these archs. Prior to
      this commit this was already happening anytime we processed
      pending softirqs anyway.

      Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Fix lockup related to stop_machine being stuck in __do_softirq.

      The stop machine logic can lock up if all but one of the migration
      threads make it through the disable-irq step and the one remaining
      thread gets stuck in __do_softirq.  The reason __do_softirq can hang is
      that it has a bail-out based on jiffies timeout, but in the lockup case,
      jiffies itself is not incremented.

      To work around this, re-add the max_restart counter in __do_irq and stop
      processing irqs after 10 restarts.

      Thanks to Tejun Heo and Rusty Russell and others for helping me track
      this down.

      This was introduced in 3.9 by commit c10d73671ad3 ("softirq: reduce
      latencies").

      It may be worth looking into ath9k to see if it has issues with its irq
      handler at a later date.

      The hang stack traces look something like this:

	  ------------[ cut here ]------------
	  WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x9c/0xa7()
	  Watchdog detected hard LOCKUP on cpu 2
	  Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
	  Pid: 23, comm: migration/2 Tainted: G         C   3.9.4+ #11
	  Call Trace:
	  <NMI>   warn_slowpath_common+0x85/0x9f
	    warn_slowpath_fmt+0x46/0x48
	    watchdog_overflow_callback+0x9c/0xa7
	    __perf_event_overflow+0x137/0x1cb
	    perf_event_overflow+0x14/0x16
	    intel_pmu_handle_irq+0x2dc/0x359
	    perf_event_nmi_handler+0x19/0x1b
	    nmi_handle+0x7f/0xc2
	    do_nmi+0xbc/0x304
	    end_repeat_nmi+0x1e/0x2e
	  <<EOE>>
	    cpu_stopper_thread+0xae/0x162
	    smpboot_thread_fn+0x258/0x260
	    kthread+0xc7/0xcf
	    ret_from_fork+0x7c/0xb0
	  ---[ end trace 4947dfa9b0a4cec3 ]---
	  BUG: soft lockup - CPU#1 stuck for 22s! [migration/1:17]
	  Modules linked in: ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc pktgen lockd sunrpc]
	  irq event stamp: 835637905
	  hardirqs last  enabled at (835637904): __do_softirq+0x9f/0x257
	  hardirqs last disabled at (835637905): apic_timer_interrupt+0x6d/0x80
	  softirqs last  enabled at (5654720): __do_softirq+0x1ff/0x257
	  softirqs last disabled at (5654725): irq_exit+0x5f/0xbb
	  CPU 1
	  Pid: 17, comm: migration/1 Tainted: G        WC   3.9.4+ #11 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
	  RIP: tasklet_hi_action+0xf0/0xf0
	  Process migration/1
	  Call Trace:
	  <IRQ>
	    __do_softirq+0x117/0x257
	    irq_exit+0x5f/0xbb
	    smp_apic_timer_interrupt+0x8a/0x98
	    apic_timer_interrupt+0x72/0x80
	  <EOI>
	    printk+0x4d/0x4f
	    stop_machine_cpu_stop+0x22c/0x274
	    cpu_stopper_thread+0xae/0x162
	    smpboot_thread_fn+0x258/0x260
	    kthread+0xc7/0xcf
	    ret_from_fork+0x7c/0xb0

      Signed-off-by: Ben Greear <greearb@candelatech.com>
      Acked-by: Tejun Heo <tj@kernel.org>
      Acked-by: Pekka Riikonen <priikone@iki.fi>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: stable@kernel.org
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

irq: Force hardirq exit's softirq processing on its own stack

      The commit facd8b80c67a3cf64a467c4a2ac5fb31f2e6745b
      ("irq: Sanitize invoke_softirq") converted irq exit
      calls of do_softirq() to __do_softirq() on all architectures,
      assuming it was only used there for its irq disablement
      properties.

      But as a side effect, the softirqs processed in the end
      of the hardirq are always called on the inline current
      stack that is used by irq_exit() instead of the softirq
      stack provided by the archs that override do_softirq().

      The result is mostly safe if the architecture runs irq_exit()
      on a separate irq stack because then softirqs are processed
      on that same stack that is near empty at this stage (assuming
      hardirq aren't nesting).

      Otherwise irq_exit() runs in the task stack and so does the softirq
      too. The interrupted call stack can be randomly deep already and
      the softirq can dig through it even further. To add insult to the
      injury, this softirq can be interrupted by a new hardirq, maximizing
      the chances for a stack overrun as reported in powerpc for example:

	      do_IRQ: stack overflow: 1920
	      CPU: 0 PID: 1602 Comm: qemu-system-ppc Not tainted 3.10.4-300.1.fc19.ppc64p7 #1
	      Call Trace:
	      [c0000000050a8740] .show_stack+0x130/0x200 (unreliable)
	      [c0000000050a8810] .dump_stack+0x28/0x3c
	      [c0000000050a8880] .do_IRQ+0x2b8/0x2c0
	      [c0000000050a8930] hardware_interrupt_common+0x154/0x180
	      --- Exception: 501 at .cp_start_xmit+0x3a4/0x820 [8139cp]
		      LR = .cp_start_xmit+0x390/0x820 [8139cp]
	      [c0000000050a8d40] .dev_hard_start_xmit+0x394/0x640
	      [c0000000050a8e00] .sch_direct_xmit+0x110/0x260
	      [c0000000050a8ea0] .dev_queue_xmit+0x260/0x630
	      [c0000000050a8f40] .br_dev_queue_push_xmit+0xc4/0x130 [bridge]
	      [c0000000050a8fc0] .br_dev_xmit+0x198/0x270 [bridge]
	      [c0000000050a9070] .dev_hard_start_xmit+0x394/0x640
	      [c0000000050a9130] .dev_queue_xmit+0x428/0x630
	      [c0000000050a91d0] .ip_finish_output+0x2a4/0x550
	      [c0000000050a9290] .ip_local_out+0x50/0x70
	      [c0000000050a9310] .ip_queue_xmit+0x148/0x420
	      [c0000000050a93b0] .tcp_transmit_skb+0x4e4/0xaf0
	      [c0000000050a94a0] .__tcp_ack_snd_check+0x7c/0xf0
	      [c0000000050a9520] .tcp_rcv_established+0x1e8/0x930
	      [c0000000050a95f0] .tcp_v4_do_rcv+0x21c/0x570
	      [c0000000050a96c0] .tcp_v4_rcv+0x734/0x930
	      [c0000000050a97a0] .ip_local_deliver_finish+0x184/0x360
	      [c0000000050a9840] .ip_rcv_finish+0x148/0x400
	      [c0000000050a98d0] .__netif_receive_skb_core+0x4f8/0xb00
	      [c0000000050a99d0] .netif_receive_skb+0x44/0x110
	      [c0000000050a9a70] .br_handle_frame_finish+0x2bc/0x3f0 [bridge]
	      [c0000000050a9b20] .br_nf_pre_routing_finish+0x2ac/0x420 [bridge]
	      [c0000000050a9bd0] .br_nf_pre_routing+0x4dc/0x7d0 [bridge]
	      [c0000000050a9c70] .nf_iterate+0x114/0x130
	      [c0000000050a9d30] .nf_hook_slow+0xb4/0x1e0
	      [c0000000050a9e00] .br_handle_frame+0x290/0x330 [bridge]
	      [c0000000050a9ea0] .__netif_receive_skb_core+0x34c/0xb00
	      [c0000000050a9fa0] .netif_receive_skb+0x44/0x110
	      [c0000000050aa040] .napi_gro_receive+0xe8/0x120
	      [c0000000050aa0c0] .cp_rx_poll+0x31c/0x590 [8139cp]
	      [c0000000050aa1d0] .net_rx_action+0x1dc/0x310
	      [c0000000050aa2b0] .__do_softirq+0x158/0x330
	      [c0000000050aa3b0] .irq_exit+0xc8/0x110
	      [c0000000050aa430] .do_IRQ+0xdc/0x2c0
	      [c0000000050aa4e0] hardware_interrupt_common+0x154/0x180
	      --- Exception: 501 at .bad_range+0x1c/0x110
		      LR = .get_page_from_freelist+0x908/0xbb0
	      [c0000000050aa7d0] .list_del+0x18/0x50 (unreliable)
	      [c0000000050aa850] .get_page_from_freelist+0x908/0xbb0
	      [c0000000050aa9e0] .__alloc_pages_nodemask+0x21c/0xae0
	      [c0000000050aaba0] .alloc_pages_vma+0xd0/0x210
	      [c0000000050aac60] .handle_pte_fault+0x814/0xb70
	      [c0000000050aad50] .__get_user_pages+0x1a4/0x640
	      [c0000000050aae60] .get_user_pages_fast+0xec/0x160
	      [c0000000050aaf10] .__gfn_to_pfn_memslot+0x3b0/0x430 [kvm]
	      [c0000000050aafd0] .kvmppc_gfn_to_pfn+0x64/0x130 [kvm]
	      [c0000000050ab070] .kvmppc_mmu_map_page+0x94/0x530 [kvm]
	      [c0000000050ab190] .kvmppc_handle_pagefault+0x174/0x610 [kvm]
	      [c0000000050ab270] .kvmppc_handle_exit_pr+0x464/0x9b0 [kvm]
	      [c0000000050ab320]  kvm_start_lightweight+0x1ec/0x1fc [kvm]
	      [c0000000050ab4f0] .kvmppc_vcpu_run_pr+0x168/0x3b0 [kvm]
	      [c0000000050ab9c0] .kvmppc_vcpu_run+0xc8/0xf0 [kvm]
	      [c0000000050aba50] .kvm_arch_vcpu_ioctl_run+0x5c/0x1a0 [kvm]
	      [c0000000050abae0] .kvm_vcpu_ioctl+0x478/0x730 [kvm]
	      [c0000000050abc90] .do_vfs_ioctl+0x4ec/0x7c0
	      [c0000000050abd80] .SyS_ioctl+0xd4/0xf0
	      [c0000000050abe30] syscall_exit+0x0/0x98

      Since this is a regression, this patch proposes a minimalistic
      and low-risk solution by blindly forcing the hardirq exit processing of
      softirqs on the softirq stack. This way we should reduce significantly
      the opportunities for task stack overflow dug by softirqs.

      Longer term solutions may involve extending the hardirq stack coverage to
      irq_exit(), etc...

      Reported-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: #3.9.. <stable@vger.kernel.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Consolidate do_softirq() arch overriden implementations

      All arch overriden implementations of do_softirq() share the following
      common code: disable irqs (to avoid races with the pending check),
      check if there are softirqs pending, then execute __do_softirq() on
      a specific stack.

      Consolidate the common parts such that archs only worry about the
      stack switch.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Optimize call to softirq on hardirq exit

      Before processing softirqs on hardirq exit, we already
      do the check for pending softirqs while hardirqs are
      guaranteed to be disabled.

      So we can take a shortcut and safely jump to the arch
      specific implementation directly.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Improve a bit softirq debugging

      do_softirq() has a debug check that verifies that it is not nesting
      on softirqs processing, nor miscounting the softirq part of the preempt
      count.

      But making sure that softirqs processing don't nest is actually a more
      generic concern that applies to any caller of __do_softirq().

      Do take it one step further and generalize that debug check to
      any softirq processing.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Justify the various softirq stack choices

      For clarity, comment the various stack choices for softirqs
      processing, whether we execute them from ksoftirqd or
      local_irq_enable() calls.

      Their use on irq_exit() is already commented.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

irq: Optimize softirq stack selection in irq exit

      If irq_exit() is called on the arch's specified irq stack,
      it should be safe to run softirqs inline under that same
      irq stack as it is near empty by the time we call irq_exit().

      For example if we use the same stack for both hard and soft irqs here,
      the worst case scenario is:
      hardirq -> softirq -> hardirq. But then the softirq supersedes the
      first hardirq as the stack user since irq_exit() is called in
      a mostly empty stack. So the stack merge in this case looks acceptable.

      Stack overrun still have a chance to happen if hardirqs have more
      opportunities to nest, but then it's another problem to solve.

      So lets adapt the irq exit's softirq stack on top of a new Kconfig symbol
      that can be defined when irq_exit() runs on the irq stack. That way
      we can spare some stack switch on irq processing and all the cache
      issues that come along.

      Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Paul Mackerras <paulus@au1.ibm.com>
      Cc: James Hogan <james.hogan@imgtec.com>
      Cc: James E.J. Bottomley <jejb@parisc-linux.org>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David S. Miller <davem@davemloft.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>

revert "softirq: Add support for triggering softirq work on softirqs"

      This commit was incomplete in that code to remove items from the per-cpu
      lists was missing and never acquired a user in the 5 years it has been in
      the tree.  We're going to implement what it seems to try to archive in a
      simpler way, and this code is in the way of doing so.

      Signed-off-by: Christoph Hellwig <hch@lst.de>
      Cc: Jan Kara <jack@suse.cz>
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[FILESYSTEM] *squashing updates*

switch do_fsync() to fget_light()

      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Move noop_backing_dev_info check from sync into writeback

      In principle, a filesystem may want to have ->sync_fs() called during sync(1)
      although it does not have a bdi (i.e. s_bdi is set to noop_backing_dev_info).
      Only writeback code really needs bdi set to something reasonable. So move the
      checks where they are more logical.

      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

quota: Split dquot_quota_sync() to writeback and cache flushing part

      Split off part of dquot_quota_sync() which writes dquots into a quota file
      to a separate function. In the next patch we will use the function from
      filesystems and we do not want to abuse ->quota_sync quotactl callback more
      than necessary.

      Acked-by: Steven Whitehouse <swhiteho@redhat.com>
      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

quota: Move quota syncing to ->sync_fs method

      Since the moment writes to quota files are using block device page cache and
      space for quota structures is reserved at the moment they are first accessed we
      have no reason to sync quota before inode writeback. In fact this order is now
      only harmful since quota information can easily change during inode writeback
      (either because conversion of delayed-allocated extents or simply because of
      allocation of new blocks for simple filesystems not using page_mkwrite).

      So move syncing of quota information after writeback of inodes into ->sync_fs
      method. This way we do not have to use ->quota_sync callback which is primarily
      intended for use by quotactl syscall anyway and we get rid of calling
      ->sync_fs() twice unnecessarily. We skip quota syncing for OCFS2 since it does
      proper quota journalling in all cases (unlike ext3, ext4, and reiserfs which
      also support legacy non-journalled quotas) and thus there are no dirty quota
      structures.

      CC: "Theodore Ts'o" <tytso@mit.edu>
      CC: Joel Becker <jlbec@evilplan.org>
      CC: reiserfs-devel@vger.kernel.org
      Acked-by: Steven Whitehouse <swhiteho@redhat.com>
      Acked-by: Dave Kleikamp <shaggy@kernel.org>
      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Reorder operations during sys_sync

      Change the order of operations during sync from

      for_each_sb {
	      writeback_inodes_sb();
	      sync_fs(nowait);
	      __sync_blockdev(nowait);
      }
      for_each_sb {
	      sync_inodes_sb();
	      sync_fs(wait);
	      __sync_blockdev(wait);
      }

      to

      for_each_sb
	      writeback_inodes_sb();
      for_each_sb
	      sync_fs(nowait);
      for_each_sb
	      __sync_blockdev(nowait);
      for_each_sb
	      sync_inodes_sb();
      for_each_sb
	      sync_fs(wait);
      for_each_sb
	      __sync_blockdev(wait);

      This is a preparation for the following patches in this series.

      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Make sys_sync writeout also block device inodes

      In case block device does not have filesystem mounted on it, sys_sync will just
      ignore it and doesn't writeout its dirty pages. This is because writeback code
      avoids writing inodes from superblock without backing device and
      blockdev_superblock is such a superblock.  Since it's unexpected that sync
      doesn't writeout dirty data for block devices be nice to users and change the
      behavior to do so. So now we iterate over all block devices on blockdev_super
      instead of iterating over all superblocks when syncing block devices.

      Reviewed-by: Christoph Hellwig <hch@lst.de>
      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Remove unnecessary flushing of block devices

      It is not necessary to write block devices twice. The reason why we first did
      flush and then proper sync is that
	for_each_bdev() {
	  write_bdev()
	  wait_for_completion()
	}
      is much slower than
	for_each_bdev()
	  write_bdev()
	for_each_bdev()
	  wait_for_completion()
      when there is bigger amount of data. But as is seen in the above, there's no real
      need to scan pages and submit them twice. We just need to separate the submission
      and waiting part. This patch does that.

      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Avoid unnecessary WB_SYNC_NONE writeback during sys_sync and reorder sync passes

      wakeup_flusher_threads(0) will queue work doing complete writeback for each
      flusher thread. Thus there is not much point in submitting another work doing
      full inode WB_SYNC_NONE writeback by writeback_inodes_sb().

      After this change it does not make sense to call nonblocking ->sync_fs and
      block device flush before calling sync_inodes_sb() because
      wakeup_flusher_threads() is completely asynchronous and thus these functions
      would be called in parallel with inode writeback running which will effectively
      void any work they do. So we move sync_inodes_sb() call before these two
      functions.

      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

vfs: Create function for iterating over block devices

      Signed-off-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

fs/block-dev.c:fix performance regression in O_DIRECT writes to md block devices

      For regular file, write operaion used blk_plug function.But for block
      file,write operation did not use blk_plug.
      This patch is also for write-cache mode for block-device.

      Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
      Reviewed-by: NeilBrown <neilb@suse.de>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

Fix a crash when block device is read and block size is changed at the same time

      The kernel may crash when block size is changed and I/O is issued
      simultaneously.

      Because some subsystems (udev or lvm) may read any block device anytime,
      the bug actually puts any code that changes a block device size in
      jeopardy.

      The crash can be reproduced if you place "msleep(1000)" to
      blkdev_get_blocks just before "bh->b_size = max_blocks <<
      inode->i_blkbits;".
      Then, run "dd if=/dev/ram0 of=/dev/null bs=4k count=1 iflag=direct"
      While it is waiting in msleep, run "blockdev --setbsz 2048 /dev/ram0"
      You get a BUG.

      The direct and non-direct I/O is written with the assumption that block
      size does not change. It doesn't seem practical to fix these crashes
      one-by-one there may be many crash possibilities when block size changes
      at a certain place and it is impossible to find them all and verify the
      code.

      This patch introduces a new rw-lock bd_block_size_semaphore. The lock is
      taken for read during I/O. It is taken for write when changing block
      size. Consequently, block size can't be changed while I/O is being
      submitted.

      For asynchronous I/O, the patch only prevents block size change while
      the I/O is being submitted. The block size can change when the I/O is in
      progress or when the I/O is being finished. This is acceptable because
      there are no accesses to block size when asynchronous I/O is being
      finished.

      The patch prevents block size changing while the device is mapped with
      mmap.

      Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

fix O_SYNC|O_APPEND syncing the wrong range on write()

      It actually goes back to 2004 ([PATCH] Concurrent O_SYNC write support)
      when sync_page_range() had been introduced; generic_file_write{,v}() correctly
      synced
	      pos_after_write - written .. pos_after_write - 1
      but generic_file_aio_write() synced
	      pos_before_write .. pos_before_write + written - 1
      instead.  Which is not the same thing with O_APPEND, obviously.
      A couple of years later correct variant had been killed off when
      everything switched to use of generic_file_aio_write().

      All users of generic_file_aio_write() are affected, and the same bug
      has been copied into other instances of ->aio_write().

      The fix is trivial; the only subtle point is that generic_write_sync()
      ought to be inlined to avoid calculations useless for the majority of
      calls.

      Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

blockdev: turn a rw semaphore into a percpu rw semaphore

      This avoids cache line bouncing when many processes lock the semaphore
      for read.

      New percpu lock implementation

      The lock consists of an array of percpu unsigned integers, a boolean
      variable and a mutex.

      When we take the lock for read, we enter rcu read section, check for a
      "locked" variable. If it is false, we increase a percpu counter on the
      current cpu and exit the rcu section. If "locked" is true, we exit the
      rcu section, take the mutex and drop it (this waits until a writer
      finished) and retry.

      Unlocking for read just decreases percpu variable. Note that we can
      unlock on a difference cpu than where we locked, in this case the
      counter underflows. The sum of all percpu counters represents the number
      of processes that hold the lock for read.

      When we need to lock for write, we take the mutex, set "locked" variable
      to true and synchronize rcu. Since RCU has been synchronized, no
      processes can create new read locks. We wait until the sum of percpu
      counters is zero - when it is, there are no readers in the critical
      section.

      Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

fs/block_dev.c:1644:5: sparse: symbol 'blkdev_mmap' was not declared

      blkdev_mmap() isn't used outside of fs/block_dev.c, mark it as
      static.

      Reported-by: Fengguang Wu <fengguang.wu@intel.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

Lock splice_read and splice_write functions

      Functions generic_file_splice_read and generic_file_splice_write access
      the pagecache directly. For block devices these functions must be locked
      so that block size is not changed while they are in progress.

      This patch is an additional fix for commit b87570f5d349 ("Fix a crash
      when block device is read and block size is changed at the same time")
      that locked aio_read, aio_write and mmap against block size change.

      Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

fs/buffer.c: make block-size be per-page and protected by the page lock

      This makes the buffer size handling be a per-page thing, which allows us
      to not have to worry about locking too much when changing the buffer
      size.  If a page doesn't have buffers, we still need to read the block
      size from the inode, but we can do that with ACCESS_ONCE(), so that even
      if the size is changing, we get a consistent value.

      This doesn't convert all functions - many of the buffer functions are
      used purely by filesystems, which in turn results in the buffer size
      being fixed at mount-time.  So they don't have the same consistency
      issues that the raw device access can have.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

blockdev: remove bd_block_size_semaphore again

      This reverts the block-device direct access code to the previous
      unlocked code, now that fs/buffer.c no longer needs external locking.

      With this, fs/block_dev.c is back to the original version, apart from a
      whitespace cleanup that I didn't want to revert.

      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

block: fix infinite loop in __getblk_slow

      Commit 080399aaaf35 ("block: don't mark buffers beyond end of disk as
      mapped") exposed a bug in __getblk_slow that causes mount to hang as it
      loops infinitely waiting for a buffer that lies beyond the end of the
      disk to become uptodate.

      The problem was initially reported by Torsten Hilbrich here:

	  https://lkml.org/lkml/2012/6/18/54

      and also reported independently here:

	  http://www.sysresccd.org/forums/viewtopic.php?f=13&t=4511

      and then Richard W.M.  Jones and Marcos Mello noted a few separate
      bugzillas also associated with the same issue.  This patch has been
      confirmed to fix:

	  https://bugzilla.redhat.com/show_bug.cgi?id=835019

      The main problem is here, in __getblk_slow:

	      for (;;) {
		      struct buffer_head * bh;
		      int ret;

		      bh = __find_get_block(bdev, block, size);
		      if (bh)
			      return bh;

		      ret = grow_buffers(bdev, block, size);
		      if (ret < 0)
			      return NULL;
		      if (ret == 0)
			      free_more_memory();
	      }

      __find_get_block does not find the block, since it will not be marked as
      mapped, and so grow_buffers is called to fill in the buffers for the
      associated page.  I believe the for (;;) loop is there primarily to
      retry in the case of memory pressure keeping grow_buffers from
      succeeding.  However, we also continue to loop for other cases, like the
      block lying beond the end of the disk.  So, the fix I came up with is to
      only loop when grow_buffers fails due to memory allocation issues
      (return value of 0).

      The attached patch was tested by myself, Torsten, and Rich, and was
      found to resolve the problem in call cases.

      Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
      Reported-and-Tested-by: Torsten Hilbrich <torsten.hilbrich@secunet.com>
      Tested-by: Richard W.M. Jones <rjones@redhat.com>
      Reviewed-by: Josh Boyer <jwboyer@redhat.com>
      Cc: Stable <stable@vger.kernel.org>  # 3.0+
      [ Jens is on vacation, taking this directly  - Linus ]
      --
      Stable Notes: this patch requires backport to 3.0, 3.2 and 3.3.
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

fs: vfat: reduce the worst case latencies

      When a block partition is mounted with FAT file system
      and MS_DIRSYNC option is used, some file system operations
      like create, rename shall sleep in caller's context until
      all the metadata have been committed to the non-volatile memory.
      Since this operation is blocking call for user context,
      the WRITE_SYNC option must be used instead of WRITE
      (async operation) which incur inherent latencies while
      flushing the meta-data corresponding to directory entries

      Change-Id: I41c514889873a39d564271db0a421e6c66e5ae33
      Signed-off-by: xiaogang <xiaogang@codeaurora.org>

writeback: Fix occasional slow sync(1)

      In case when system contains no dirty pages, wakeup_flusher_threads()
      will submit WB_SYNC_NONE writeback for 0 pages so wb_writeback() exits
      immediately without doing anything. Thus sync(1) will write all the
      dirty inodes from a WB_SYNC_ALL writeback pass which is slow.

      Fix the problem by using get_nr_dirty_pages() in
      wakeup_flusher_threads() instead of calculating number of dirty pages
      manually. That function also takes number of dirty inodes into account.

      Change-Id: I458027ae08d9a5a93202a7b97ace1f8da7a18a07
      CC: stable@vger.kernel.org
      Reported-by: Paul Taysom <taysom@chromium.org>
      Signed-off-by: Jan Kara <jack@suse.cz>

      # This is the 21st commit message:

      sync: don't block the flusher thread waiting on IO

      When sync does it's WB_SYNC_ALL writeback, it issues data Io and
      then immediately waits for IO completion. This is done in the
      context of the flusher thread, and hence completely ties up the
      flusher thread for the backing device until all the dirty inodes
      have been synced. On filesystems that are dirtying inodes constantly
      and quickly, this means the flusher thread can be tied up for
      minutes per sync call and hence badly affect system level write IO
      performance as the page cache cannot be cleaned quickly.

      We already have a wait loop for IO completion for sync(2), so cut
      this out of the flusher thread and delegate it to wait_sb_inodes().
      Hence we can do rapid IO submission, and then wait for it all to
      complete.

      Effect of sync on fsmark before the patch:

      FSUse%        Count         Size    Files/sec     App Overhead
      .....
	  0       640000         4096      35154.6          1026984
	  0       720000         4096      36740.3          1023844
	  0       800000         4096      36184.6           916599
	  0       880000         4096       1282.7          1054367
	  0       960000         4096       3951.3           918773
	  0      1040000         4096      40646.2           996448
	  0      1120000         4096      43610.1           895647
	  0      1200000         4096      40333.1           921048

      And a single sync pass took:

	real    0m52.407s
	user    0m0.000s
	sys     0m0.090s

      After the patch, there is no impact on fsmark results, and each
      individual sync(2) operation run concurrently with the same fsmark
      workload takes roughly 7s:

	real    0m6.930s
	user    0m0.000s
	sys     0m0.039s

      IOWs, sync is 7-8x faster on a busy filesystem and does not have an
      adverse impact on ongoing async data write operations.

      Signed-off-by: Dave Chinner <dchinner@redhat.com>
      Reviewed-by: Jan Kara <jack@suse.cz>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

      Change-Id: I9e55d65f5ecb2305497711d4688f0647d9346035

writeback: fix race that cause writeback hung

      There is a race between mark inode dirty and writeback thread, see the
      following scenario.  In this case, writeback thread will not run though
      there is dirty_io.

      __mark_inode_dirty()                                          bdi_writeback_workfn()
	      ...                                                       	...
	      spin_lock(&inode->i_lock);
	      ...
	      if (bdi_cap_writeback_dirty(bdi)) {
		  <<< assume wb has dirty_io, so wakeup_bdi is false.
		  <<< the following inode_dirty also have wakeup_bdi false.
		  if (!wb_has_dirty_io(&bdi->wb))
			  wakeup_bdi = true;
	      }
	      spin_unlock(&inode->i_lock);
									  <<< assume last dirty_io is removed here.
									  pages_written = wb_do_writeback(wb);
									  ...
									  <<< work_list empty and wb has no dirty_io,
									  <<< delayed_work will not be queued.
									  if (!list_empty(&bdi->work_list) ||
									      (wb_has_dirty_io(wb) && dirty_writeback_interval))
									      queue_delayed_work(bdi_wq, &wb->dwork,
										  msecs_to_jiffies(dirty_writeback_interval * 10));
	      spin_lock(&bdi->wb.list_lock);
	      inode->dirtied_when = jiffies;
	      <<< new dirty_io is added.
	      list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
	      spin_unlock(&bdi->wb.list_lock);

	      <<< though there is dirty_io, but wakeup_bdi is false,
	      <<< so writeback thread will not be waked up and
	      <<< the new dirty_io will not be flushed.
	      if (wakeup_bdi)
		  bdi_wakeup_thread_delayed(bdi);

      Writeback will run until there is a new flush work queued.  This may cause
      a lot of dirty pages stay in memory for a long time.

      Signed-off-by: Junxiao Bi <junxiao.bi@oracle.com>
      Reviewed-by: Jan Kara <jack@suse.cz>
      Cc: Fengguang Wu <fengguang.wu@intel.com>
      Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>

      Change-Id: I973fcba5381881a003a035ffff48f64348660079

ext4: speed up truncate/unlink by not using bforget() unless needed

      Do not iterate over data blocks scanning for bh's to forget as they're
      never exist. This improves time taken by unlink / truncate syscall.
      Tested by continuously truncating file that is being written by dd.
      Another test is rm -rf of linux tree while tar unpacks it. With
      ordered data mode condition unlikely(!tbh) was always met in
      ext4_free_blocks. With journal data mode tbh was found only few times,
      so optimisation is also possible.

      Unlinking fallocated 60G file after doing sync && echo 3 >
      /proc/sys/vm/drop_caches && time rm --help

      X86 before (linux 3.6-rc4):
      real    0m2.710s
      user    0m0.000s
      sys     0m1.530s

      X86 after:
      real    0m0.644s
      user    0m0.003s
      sys     0m0.060s

      MIPS before (linux 2.6.37):
      real    0m 4.93s
      user    0m 0.00s
      sys     0m 4.61s

      MIPS after:
      real    0m 0.16s
      user    0m 0.00s
      sys     0m 0.06s

      Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: Andrey Sidorov <qrxd43@motorola.com>
      Signed-off-by: franciscofranco <franciscofranco.1990@gmail.com>

      Change-Id: Ie78a945cb82b3892eaf88701f2dc3b7726104fb5

fs: Asynchronous I/O latency to a solid-state disk greatly increased

      between the 2.6.32 and 3.0 kernels. By removing the plug from
      do_io_submit(), we observed a 34% improvement in the I/O latency.

      Unfortunately, at this level, we don't know if the request is to
      a rotating disk or not.

      # This is the 25th commit message:

      deadline: Allow 0ms deadline latency, increase the read speed

      Change a timer compare from after to after-equals, thus allowing
      0 timeout and making deadline schedule FIFO.

      Signed-off-by: xiaobing tu <xiaobing.tu@intel.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

block: Add bio_end_sector()

      Just a little convenience macro - main reason to add it now is preparing
      for immutable bio vecs, it'll reduce the size of the patch that puts
      bi_sector/bi_size/bi_idx into a struct bvec_iter.

      Signed-off-by: Kent Overstreet <koverstreet@google.com>
      CC: Jens Axboe <axboe@kernel.dk>
      CC: Lars Ellenberg <drbd-dev@lists.linbit.com>
      CC: Jiri Kosina <jkosina@suse.cz>
      CC: Alasdair Kergon <agk@redhat.com>
      CC: dm-devel@redhat.com
      CC: Neil Brown <neilb@suse.de>
      CC: Martin Schwidefsky <schwidefsky@de.ibm.com>
      CC: Heiko Carstens <heiko.carstens@de.ibm.com>
      CC: linux-s390@vger.kernel.org
      CC: Chris Mason <chris.mason@fusionio.com>
      CC: Steven Whitehouse <swhiteho@redhat.com>
      Acked-by: Steven Whitehouse <swhiteho@redhat.com>

block: Drop dead function blk_abort_queue()

      This function was only used by btrfs code in btrfs_abort_devices()
      (seems in a wrong way).

      It was removed in commit d07eb9117050c9ed3f78296ebcc06128b52693be,
      So, Let's remove the dead code to avoid any confusion.

      Changes in v2: update commit log, btrfs_abort_devices() was removed
      already.

      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: linux-kernel@vger.kernel.org
      Cc: Chris Mason <chris.mason@oracle.com>
      Cc: linux-btrfs@vger.kernel.org
      Cc: David Sterba <dave@jikos.cz>
      Signed-off-by: Asias He <asias@redhat.com>
      Signed-off-by: Jens Axboe <axboe@kernel.dk>

kernel:sched: LOAD_FREQ (4*HZ+61) avoids loadavg Moire

      LOAD_FREQ is (5*HZ+1) to avoid high load average when idle:
      http://kerneltrap.org/mailarchive/linux-kernel/2007/10/3/328568

      I suggest (4*HZ+61) for a better distribution.

      With some seconds based load (like SSL heartbeats)
      and LOAD_FREQ at (5*HZ+1) I see Moire patterns like inverse sawtooth,
      since 2 or 3 probes hit the jobs (load increases quickly),
      followed by several probes missing it.

      A 4.61 sec interval gives optimal distribution over when within a
      second a probe is taken, as .61 is close to golden ratio phi 1.618...
      (test in http://ripke.com/goldenratio.c).

      12*4.61 = 55.32 secs is still close to a minute,
      and 13*4.61=59.93 is even closer than the current 12*5.01=60.12
      (with exponents EXP_x adjusted to a ratio of 13 instead of 12).

block: Fix bad range check in bio_sector_offset

      DM would occasionally end up splitting data integrity-enabled requests
      incorrectly. The culprit was a bad range check in bio_sector_offset.

      Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

block: Expose kblock_schedule_delayed_work()

      This function is exported in blk-core.c to be used in other modules
      but it's definition in h file is missing.

      Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>

fs/sync: Make sync() satisfy many requests with one invocation

      Dave Jones reported RCU stalls, overly long hrtimer interrupts, and
      amazingly long NMI handlers from a trinity-induced workload involving
      lots of concurrent sync() calls (https://lkml.org/lkml/2013/7/23/369).
      There are any number of things that one might do to make sync() behave
      better under high levels of contention, but it is also the case that
      multiple concurrent sync() system calls can be satisfied by a single
      sys_sync() invocation.

      Given that this situation is reminiscent of rcu_barrier(), this commit
      applies the rcu_barrier() approach to sys_sync().  This approach uses
      a global mutex and a sequence counter.  The mutex is held across the
      sync() operation, which eliminates contention between concurrent sync()
      operations.  The counter is incremented at the beginning and end of
      each sync() operation, so that it is odd while a sync() operation is in
      progress and even otherwise, just like sequence locks.

      The code that used to be in sys_sync() is now in do_sync(), and sys_sync()
      now handles the concurrency.  The sys_sync() function first takes a
      snapshot of the counter, then acquires the mutex, and then takes another
      snapshot of the counter.  If the values of the two snapshots indicate that
      a full do_sync() executed during the mutex acquisition, the sys_sync()
      function releases the mutex and returns ("Our work is done!").  Otherwise,
      sys_sync() increments the counter, invokes do_sync(), and increments
      the counter again.

      This approach allows a single call to do_sync() to satisfy an arbitrarily
      large number of sync() system calls, which should eliminate issues due
      to large numbers of concurrent invocations of the sync() system call.

      Changes since v1 (https://lkml.org/lkml/2013/7/24/683):

      o	Add a pair of memory barriers to keep the increments from
	      bleeding into the do_sync() code.  (The failure probability
	      is insanely low, but when you have several hundred million
	      devices running Linux, you can expect several hundred instances
	      of one-in-a-million failures.)

      o	Actually CC some people who have experience in this area.

      Reported-by: Dave Jones <davej@redhat.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Alexander Viro <viro@zeniv.linux.org.uk>
      Cc: Christoph Hellwig <hch@lst.de>
      Cc: Jan Kara <jack@suse.cz>
      Cc: Curt Wohlgemuth <curtw@google.com>
      Cc: Jens Axboe <jaxboe@fusionio.com>
      Cc: linux-fsdevel@vger.kernel.org

      Signed-off-by: Paul Reioux <reioux@gmail.com>

epoll: optimize EPOLL_CTL_DEL using rcu

      Optimize EPOLL_CTL_DEL such that it does not require the 'epmutex' by
      converting the file->f_ep_links list into an rcu one. In this way, we can
      traverse the epoll network on the add path in parallel with deletes. Since
      deletes can't create loops or worse wakeup paths, this is safe.

      This patch in combination with the patch "epoll: Do not take global 'epmutex'
      for simple topologies", shows a dramatic performance improvement in
      scalability for SPECjbb.

      Tested-by: Nathan Zimmer <nzimmer@sgi.com>
      Signed-off-by: Jason Baron <jbaron@akamai.com>

      backported for Linux 3.4 for use on Android
      Signed-off-by: Paul Reioux <reioux@gmail.com>

epoll: Do not take global 'epmutex' for simple topologies

      When calling EPOLL_CTL_ADD for an epoll file descriptor that is attached
      directly to a wakeup source, we do not need to take the global 'epmutex',
      unless the epoll file descriptor is nested. The purpose of taking
      the 'epmutex' on add is to prevent complex topologies such as loops and
      deep wakeup paths from forming in parallel through multiple EPOLL_CTL_ADD
      operations. However, for the simple case of an epoll file descriptor
      attached directly to a wakeup source (with no nesting), we do not need
      to hold the 'epmutex'.

      This patch along with 'epoll: optimize EPOLL_CTL_DEL using rcu' improves
      scalability on larger systems. Quoting Nathan Zimmer's mail on SPECjbb
      performance:

      "
      On the 16 socket run the performance went from 35k jOPS to 125k jOPS.
      In addition the benchmark when from scaling well on 10 sockets to scaling well
      on just over 40 sockets.

      ...

      Currently the benchmark stops scaling at around 40-44 sockets but it seems like
      I found a second unrelated bottleneck.
      "

      Tested-by: Nathan Zimmer <nzimmer@sgi.com>
      Signed-off-by: Jason Baron <jbaron@akamai.com>

      backported to Linux 3.4 for use on Android
      Signed-off-by: Paul Reioux <reioux@gmail.com>

aio: convert the ioctx list to radix tree

      When using a large number of threads performing AIO operations the
      IOCTX list may get a significant number of entries which will cause
      significant overhead. For example, when running this fio script:

      rw=randrw; size=256k ;directory=/mnt/fio; ioengine=libaio; iodepth=1
      blocksize=1024; numjobs=512; thread; loops=100

      on an EXT2 filesystem mounted on top of a ramdisk we can observe up to
      30% CPU time spent by lookup_ioctx:

      32.51%  [guest.kernel]  [g] lookup_ioctx
	9.19%  [guest.kernel]  [g] __lock_acquire.isra.28
	4.40%  [guest.kernel]  [g] lock_release
	4.19%  [guest.kernel]  [g] sched_clock_local
	3.86%  [guest.kernel]  [g] local_clock
	3.68%  [guest.kernel]  [g] native_sched_clock
	3.08%  [guest.kernel]  [g] sched_clock_cpu
	2.64%  [guest.kernel]  [g] lock_release_holdtime.part.11
	2.60%  [guest.kernel]  [g] memcpy
	2.33%  [guest.kernel]  [g] lock_acquired
	2.25%  [guest.kernel]  [g] lock_acquire
	1.84%  [guest.kernel]  [g] do_io_submit

      This patchs converts the ioctx list to a radix tree. For a performance
      comparison the above FIO script was run on a 2 sockets 8 core
      machine. This are the results for the original list based
      implementation and for the radix tree based implementation:

      cores         1         2         4         8         16        32
      list        111025 ms  62219 ms  34193 ms  22998 ms  19335 ms  15956 ms
      radix        75400 ms  42668 ms  23923 ms  17206 ms  15820 ms  13295 ms
      % of radix
      relative      68%       69%       70%       75%       82%       83%
      to list

      To consider the impact of the patch on the typical case of having
      only one ctx per process the following FIO script was run:

      rw=randrw; size=100m ;directory=/mnt/fio; ioengine=libaio; iodepth=1
      blocksize=1024; numjobs=1; thread; loops=100

      on the same system and the results are the following:

      list        65241 ms
      radix       65402 ms
      % of radix
      relative    100.25%
      to list

      Cc: Andi Kleen <ak@linux.intel.com>

      Signed-off-by: Octavian Purdila <octavian.purdila@intel.com>
      modified for Mako hybrid from LKML

      Signed-off-by: Paul Reioux <reioux@gmail.com>
[FS: iov_iter:] *squashing updates*

iov_iter: move into its own file

      This moves the iov_iter functions in to their own file.  We're going to
      be working on them in upcoming patches.  They become sufficiently large,
      and remain self-contained, to justify seperating them from the rest of
      the huge mm/filemap.c.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Acked-by: Jeff Moyer <jmoyer@redhat.com>
      Cc: Zach Brown <zab@zabbo.net>

iov_iter: iov_iter_copy_from_user() should use non-atomic copy

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>

iov_iter: add copy_to_user support

      This adds iov_iter wrappers around copy_to_user() to match the existing
      wrappers around copy_from_user().

      This will be used by the generic file system buffered read path.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>

fuse: convert fuse to use iov_iter_copy_[to|from]_user

      A future patch hides the internals of struct iov_iter, so fuse should
      be using the supported interface.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Acked-by: Miklos Szeredi <mszeredi@suse.cz>
      Cc: fuse-devel@lists.sourceforge.net

iov_iter: hide iovec details behind ops function pointers

      This moves the current iov_iter functions behind an ops struct of
      function pointers.  The current iov_iter functions all work with memory
      which is specified by iovec arrays of user space pointers.

      This patch is part of a series that lets us specify memory with bio_vec
      arrays of page pointers.  By moving to an iov_iter operation struct we
      can add that support in later patches in this series by adding another
      set of function pointers.

      I only came to this after having initialy tried to teach the current
      iov_iter functions about bio_vecs by introducing conditional branches
      that dealt with bio_vecs in all the functions.  It wasn't pretty.  This
      approach seems to be the lesser evil.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako kernel from LKML

      Signed-off-by: faux123 <reioux@gmail.com>

iov_iter: add bvec support

      This adds a set of iov_iter_ops calls which work with memory which is
      specified by an array of bio_vec structs instead of an array of iovec
      structs.

      The big difference is that the pages referenced by the bio_vec elements
      are pinned.  They don't need to be faulted in and we can always use
      kmap_atomic() to map them one at a time.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>

iov_iter: add a shorten call

      The generic direct write path wants to shorten its memory vector.  It
      does this when it finds that it has to perform a partial write due to
      LIMIT_FSIZE.  .direct_IO() always performs IO on all of the referenced
      memory because it doesn't have an argument to specify the length of the
      IO.

      We add an iov_iter operation for this so that the generic path can ask
      to shorten the memory vector without having to know what kind it is.
      We're happy to shorten the kernel copy of the iovec array, but we refuse
      to shorten the bio_vec array and return an error in this case.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>

dio: Convert direct_IO to use iov_iter

      Change the direct_IO aop to take an iov_iter argument rather than an iovec.
      This will get passed down through most filesystems so that only the
      __blockdev_direct_IO helper need be aware of whether user or kernel memory
      is being passed to the function.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      heavily backported for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

dio: add bio_vec support to __blockdev_direct_IO()

      The trick here is to initialize the dio state so that do_direct_IO()
      consumes the pages we provide and never tries to map user pages.  This
      is done by making sure that final_block_in_request covers the page that
      we set in the dio.  do_direct_IO() will return before running out of
      pages.

      The caller is responsible for dirtying these pages, if needed.  We add
      an option to the dio struct that makes sure we only dirty pages when
      we're operating on iovecs of user addresses.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako kernel from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

aio: add aio_kernel_() interface

      This adds an interface that lets kernel callers submit aio iocbs without
      going through the user space syscalls.  This lets kernel callers avoid
      the management limits and overhead of the context.  It will also let us
      integrate aio operations with other kernel apis that the user space
      interface doesn't have access to.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

aio: add aio support for iov_iter arguments

      This adds iocb cmds which specify that memory is held in iov_iter
      structures.  This lets kernel callers specify memory that can be
      expressed in an iov_iter, which includes pages in bio_vec arrays.

      Only kernel callers can provide an iov_iter so it doesn't make a lot of
      sense to expose the IOCB_CMD values for this as part of the user space
      ABI.

      But kernel callers should also be able to perform the usual aio
      operations which suggests using the the existing operation namespace and
      support code.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

fs: create file_readable() and file_writable() functions

      Create functions to simplify if file_ops contain either a read
      or aio_read op, or likewise write or aio_write. We will be adding
      read_iter and write_iter and don't need to be complicating the code
      in multiple places.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

fs: use read_iter and write_iter rather than aio_read and aio_write

      File systems implementing read_iter & write_iter should not be required
      to keep aio_read and aio_write as well. The vfs should always call
      read/write_iter if they exist. This will make it easier to remove the
      aio_read/write operation in the future.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Alexander Viro <viro@zeniv.linux.org.uk>
      Cc: linux-fsdevel@vger.kernel.org

fs: add read_iter and write_iter to several file systems

      These are the simple ones.

      File systems that use generic_file_aio_read() and generic_file_aio_write()
      can trivially support generic_file_read_iter() and generic_file_write_iter().

      This patch adds those file_operations for 9p, adfs, affs, bfs, exofs, ext2,
      ext3, fat, f2fs, hfs, hfsplus, hostfs, hpfs, jfs, jffs2, logfs, minix, nilfs2,
      omfs, ramfs, reiserfs, romfs, sysv, and ufs.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Acked-by: Boaz Harrosh <bharrosh@panasas.com>

      ext4: add support for read_iter and write_iter

      use the generic_file_read_iter(), create ext4_file_write_iter() based on
      ext4_file_write(), and make ext4_file_write() a wrapper around
      ext4_file_write_iter().

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: linux-ext4@vger.kernel.org
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

ecrpytfs: Convert aio_read/write ops to read/write_iter

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Tyler Hicks <tyhicks@canonical.com>
      Cc: Dustin Kirkland <dustin.kirkland@gazzang.com>
      Cc: ecryptfs@vger.kernel.org

      # This is the 17th commit message:

      iov_iter: move into its own file (bug fix)

      I found the problem. iov_iter_shorten() wasn't setting i->count to the new
      value.

      This fixes it. I'll fix the patchset tomorrow.

      Thanks,
      Shaggy

fs: pull iov_iter use higher up the stack

      Right now only callers of generic_perform_write() pack their iovec
      arguments into an iov_iter structure.  All the callers higher up in the
      stack work on raw iovec arguments.

      This patch introduces the use of the iov_iter abstraction higher up the
      stack.  Private generic path functions are changed to operation on
      iov_iter instead of on raw iovecs.  Exported interfaces that take iovecs
      immediately pack their arguments into an iov_iter and call into the
      shared functions.

      File operation struct functions are added with iov_iter as an argument
      so that callers to the generic file system functions can specify
      abstract memory rather than iovec arrays only.

      Almost all of this patch only transforms arguments and shouldn't change
      functionality.  The buffered read path is the exception.  We add a
      read_actor function which uses the iov_iter helper functions instead of
      operating on each individual iovec element.  This may improve
      performance as the iov_iter helper can copy multiple iovec elements from
      one mapped page cache page.

      As always, the direct IO path is special.  Sadly, it may still be
      cleanest to have it work on the underlying memory structures directly
      instead of working through the iov_iter abstraction.

      Signed-off-by: Dave Kleikamp <dave.kleikamp@oracle.com>
      Cc: Zach Brown <zab@zabbo.net>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>

tmpfs: add support for read_iter and write_iter

      Convert tmpfs do_shmem_file_read() to shmem_file_read_iter().
      Make file_read_iter_actor() global so tmpfs can use it too: delete
      file_read_actor(), which was made global in 2.4.4 for use by tmpfs.
      Replace tmpfs generic_file_aio_write() by generic_file_write_iter().

      Signed-off-by: Hugh Dickins <hughd@google.com>
      modified for Mako from LKML reference

      Signed-off-by: faux123 <reioux@gmail.com>
Asynchronous Fsync: initial extraction of Async Fsync from HTC

Signed-off-by: Paul Reioux <reioux@gmail.com>
Adjusted for Acclaim By Demetris
[MSM HotPlug] *squashing updates*

msm: hotplug/platsmp: Fix cpu invaldation pointer arithmetic

      These functions take two void pointers indicating where to start
      and end the invalidation and cache flush. The pointer arithmetic
      is considering the pen_release pointer as an int pointer so
      adding sizeof(pen_release) is adding 16 bytes to the pointer and
      specifying that as the end. This inadvertently invalidates data
      next to the pen_release variable which could be bad. Fix the
      pointer math to only invalidate or clean the region for the
      pen_release variable.

      Change-Id: If0cf0e5de7c9f4b8c70662b29957a1f347809124
      Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>

msm: hotplug: Change the pen variable in one place

      Other WFI based hotplug implementations set the pen variable to
      -1 in C code and do the appropriate flushing operations in one
      location. They also don't try to invalidate the cache on a
      spurious wakeup and report how many spurious wakeups there were.
      Let's do the same in our code so that we can remove our
      dependence on the dmac_{inv,flush}_range() functions (which are
      no longer upstream) and easily report spurious wakeups to the
      kernel logs.

      This does delay the time we set the pen back to its default
      value, but that doesn't actually matter because generic ARM
      hotplug code already waits for a completion before continuing.

      Change-Id: Ic36fbf744fe8abeda20ad9859a207b5e37c5e63c
      Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>

msm: hotplug: remove code related to cpu_killed

      Earlier the completion event cpu_killed was used to synchronize
      notify to hot plug framework that the CPU is dead. But now it is
      replaced with another completion event cpu_died and it is placed
      in the hotplug framework itself. So there is no need for a
      platform to implement a completion event.

      But in msm platform driver the completion event related code
      was not removed with the changes in hotplug framework. This
      patch removes the unused code.

      Change-Id: I4418e5e397821e39a7b6a6efa997c4a866e9b46b
      Signed-off-by: Venkat Devarasetty <vdevaras@codeaurora.org>

msm: hotplug: Block hotplug removal of CPU0

      Hotplug removal of CPU0 is not currently supported. Attempts to
      remove it may result in undefined behaviour. Explicitly block
      any such attempts before other hotplug notifiers get the chance
      to run by setting the notifier priority to INT_MAX.

      Change-Id: I9e1f5b4a0db554295a097220417f2bdf4c690f5f
      Signed-off-by: Matt Wagantall <mattw@codeaurora.org>

msm: pm2: Raise SPI interrupt instread of SGI to non boot CPUs

      Once APPS comes out of PC and mpdecision want to bring the non-boot
      cpus to online, first CORE0 raises the SPI interrupt(MPCORE) to bring
      the cores(1/2/3) online. While bringing the cores online, CORE0 raises
      the SPI interrupt instead of SGI. Because once APPS does a PC QGIC
      looses its state and none of the non-boot cpus are not capable of
      receiving the SGI interrupt until cores configure the QGIC PPIs&SGIs
      by themselves.

      Change-Id: Ic4aedeffbcd3fa2f0aa15e2e8ed7e25a03468f87
      Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>

msm: pm2: Call ACPU clock driver APIs at correct place

      On 8x25Q APPS core is powered up using external BUCK
      regulator. Configuring BUCK takes place using GPIOs
      lines which simulates I2C. In a case where APPS
      going to PC, it first informing the modem to take
      control of all APPS GPIOs and then it calls ACPU
      clock driver to set the frequency to TCXO. While
      setting the frequency ACPU clock driver is trying
      to use regulator GPIO which is currently owned by
      modem and infinitely waiting for response which
      causes UI freeze. Now move APPS frequency to TCXO
      before handing over GPIOs to modem.

      Change-Id: I603eca5fe675b8c6e1a2ea59ca60bb5fca8fa5d2
      Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>

msm: platsmp-8625: modify pen_release variable to volatile

      Since pen_release is a shared variable between cpus. By marking it
      as volatile, compiler won't cache that variable into registers and
      won't optimize the code which access pen_release variable.

      Change-Id: Ib5b7aa5932b35d64d05f9e4a4672df2bb9207c59
      Signed-off-by: tirupathireddy <tirupath@codeaurora.org>

      # This is the 8th commit message:

      msm: platsmp-8625: Add cold_boot_done as per_cpu variable

      cold_boot_done is declared as an ordinary variable, and useful for
      single secondary cpu, right now. It will not work in the scenarios
      where we have more than two cores in the system. Usage of per-cpu
      variable will make the code more generic for multi-core environment.

      Change-Id: I4b2480243b2adaadf5c5a2a3534f66faebd87553
      Signed-off-by: tirupathireddy <tirupath@codeaurora.org>

msm: pm2: Raise SPI interrupt instread of SGI to non boot CPUs

      Once APPS comes out of PC and mpdecision want to bring the non-boot
      cpus to online, first CORE0 raises the SPI interrupt(MPCORE) to bring
      the cores(1/2/3) online. While bringing the cores online, CORE0 raises
      the SPI interrupt instead of SGI. Because once APPS does a PC QGIC
      looses its state and none of the non-boot cpus are not capable of
      receiving the SGI interrupt until cores configure the QGIC PPIs&SGIs
      by themselves.

      Change-Id: Ic4aedeffbcd3fa2f0aa15e2e8ed7e25a03468f87
      Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>