softirq: reduce latencies

In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.

This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.

This patch changes the fallback to ksoftirqd condition to :

- A time limit of 2 ms.
- need_resched() being set on current task

When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.

Using need_resched() as the only condition can trigger RCU stalls,
as we can keep BH disabled for too long.

I ran several benchmarks and got no significant difference in
throughput, but a very significant reduction of latencies (one order
of magnitude) :

In following bench, 200 antagonist "netperf -t TCP_RR" are started in
background, using all available cpus.

Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
IRQ (hard+soft)

Before patch :

# netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=550110.424
MIN_LATENCY=146858
MAX_LATENCY=997109
P50_LATENCY=305000
P90_LATENCY=550000
P99_LATENCY=710000
MEAN_LATENCY=376989.12
STDDEV_LATENCY=184046.92

After patch :

# netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=40545.492
MIN_LATENCY=9834
MAX_LATENCY=78366
P50_LATENCY=33583
P90_LATENCY=59000
P99_LATENCY=69000
MEAN_LATENCY=38364.67
STDDEV_LATENCY=12865.26

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: David Miller <davem@davemloft.net>
Cc: Tom Herbert <therbert@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
rcu: *updating rcu APIs*

rcu: Avoid invoking RCU core on offline CPUs

      Offline CPUs transition through the scheduler to the idle loop one
      last time before being shut down.  This can result in RCU raising
      softirq on this CPU, which is at best useless given that the CPU's
      callbacks will be offloaded at CPU_DEAD time.  This commit therefore
      avoids raising softirq on offline CPUs.

      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Fix batch-limit size problem

      Commit 29c00b4a1d9e27 (rcu: Add event-tracing for RCU callback
      invocation) added a regression in rcu_do_batch()

      Under stress, RCU is supposed to allow to process all items in queue,
      instead of a batch of 10 items (blimit), but an integer overflow makes
      the effective limit being 1.  So, unless there is frequent idle periods
      (during which RCU ignores batch limits), RCU can be forced into a
      state where it cannot keep up with the callback-generation rate,
      eventually resulting in OOM.

      This commit therefore converts a few variables in rcu_do_batch() from
      int to long to fix this problem, along with the module parameters
      controlling the batch limits.

      Signed-off-by: Eric Dumazet <edumazet@google.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: <stable@vger.kernel.org> # 3.2 +

rcu: Fix day-one dyntick-idle stall-warning bug

      Each grace period is supposed to have at least one callback waiting
      for that grace period to complete.  However, if CONFIG_NO_HZ=n, an
      extra callback-free grace period is no big problem -- it will chew up
      a tiny bit of CPU time, but it will complete normally.  In contrast,
      CONFIG_NO_HZ=y kernels have the potential for all the CPUs to go to
      sleep indefinitely, in turn indefinitely delaying completion of the
      callback-free grace period.  Given that nothing is waiting on this grace
      period, this is also not a problem.

      That is, unless RCU CPU stall warnings are also enabled, as they are
      in recent kernels.  In this case, if a CPU wakes up after at least one
      minute of inactivity, an RCU CPU stall warning will result.  The reason
      that no one noticed until quite recently is that most systems have enough
      OS noise that they will never remain absolutely idle for a full minute.
      But there are some embedded systems with cut-down userspace configurations
      that consistently get into this situation.

      All this begs the question of exactly how a callback-free grace period
      gets started in the first place.  This can happen due to the fact that
      CPUs do not necessarily agree on which grace period is in progress.
      If a CPU still believes that the grace period that just completed is
      still ongoing, it will believe that it has callbacks that need to wait for
      another grace period, never mind the fact that the grace period that they
      were waiting for just completed.  This CPU can therefore erroneously
      decide to start a new grace period.  Note that this can happen in
      TREE_RCU and TREE_PREEMPT_RCU even on a single-CPU system:  Deadlock
      considerations mean that the CPU that detected the end of the grace
      period is not necessarily officially informed of this fact for some time.

      Once this CPU notices that the earlier grace period completed, it will
      invoke its callbacks.  It then won't have any callbacks left.  If no
      other CPU has any callbacks, we now have a callback-free grace period.

      This commit therefore makes CPUs check more carefully before starting a
      new grace period.  This new check relies on an array of tail pointers
      into each CPU's list of callbacks.  If the CPU is up to date on which
      grace periods have completed, it checks to see if any callbacks follow
      the RCU_DONE_TAIL segment, otherwise it checks to see if any callbacks
      follow the RCU_WAIT_TAIL segment.  The reason that this works is that
      the RCU_WAIT_TAIL segment will be promoted to the RCU_DONE_TAIL segment
      as soon as the CPU is officially notified that the old grace period
      has ended.

      This change is to cpu_needs_another_gp(), which is called in a number
      of places.  The only one that really matters is in rcu_start_gp(), where
      the root rcu_node structure's ->lock is held, which prevents any
      other CPU from starting or completing a grace period, so that the
      comparison that determines whether the CPU is missing the completion
      of a grace period is stable.

      Change-Id: Ic77e2bbe865059aa6f363db284473e9adcf8797e
      Reported-by: Becky Bruce <bgillbruce@gmail.com>
      Reported-by: Subodh Nijsure <snijsure@grid-net.com>
      Reported-by: Paul Walmsley <paul@pwsan.com>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Paul Walmsley <paul@pwsan.com>  # OMAP3730, OMAP4430
      Cc: stable@vger.kernel.org
      Signed-off-by: Jin Hong <jinh@codeaurora.org>

rcu: Stop rcu_do_batch() from multiplexing the "count" variable

      Commit b1420f1c (Make rcu_barrier() less disruptive) rearranged the
      code in rcu_do_batch(), moving the ->qlen manipulation to follow
      the requeueing of the callbacks.  Unfortunately, this rearrangement
      clobbered the value of the "count" local variable before the value
      of rdp->qlen was adjusted, resulting in the value of rdp->qlen being
      inaccurate.  This commit therefore introduces an index variable "i",
      avoiding the inadvertent multiplexing.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Fix detection of abruptly-ending stall

      The code that attempts to identify stalls that end just as we detect
      them is broken by both flavors of initialization failure.  This commit
      therefore properly initializes and computes the count of the number
      of reasons why the RCU grace period is stalled.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Prevent offline CPUs from executing RCU core code

      Earlier versions of RCU invoked the RCU core from the CPU_DYING notifier
      in order to note a quiescent state for the outgoing CPU.  Because the
      CPU is marked "offline" during the execution of the CPU_DYING notifiers,
      the RCU core had to tolerate being invoked from an offline CPU.  However,
      commit b1420f1c (Make rcu_barrier() less disruptive) left only tracing
      code in the CPU_DYING notifier, so the RCU core need no longer execute
      on offline CPUs.  This commit therefore enforces this restriction.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Make rcu_barrier() less disruptive

      The rcu_barrier() primitive interrupts each and every CPU, registering
      a callback on every CPU.  Once all of these callbacks have been invoked,
      rcu_barrier() knows that every callback that was registered before
      the call to rcu_barrier() has also been invoked.

      However, there is no point in registering a callback on a CPU that
      currently has no callbacks, most especially if that CPU is in a
      deep idle state.  This commit therefore makes rcu_barrier() avoid
      interrupting CPUs that have no callbacks.  Doing this requires reworking
      the handling of orphaned callbacks, otherwise callbacks could slip through
      rcu_barrier()'s net by being orphaned from a CPU that rcu_barrier() had
      not yet interrupted to a CPU that rcu_barrier() had already interrupted.
      This reworking was needed anyway to take a first step towards weaning
      RCU from the CPU_DYING notifier's use of stop_cpu().

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Reduce cache-miss initialization latencies for large systems

      Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper
      limit of 16 on the leaf-level fanout for the rcu_node tree.  This was
      needed to reduce lock contention that was induced by the synchronization
      of scheduling-clock interrupts, which was in turn needed to improve
      energy efficiency for moderate-sized lightly loaded servers.

      However, reducing the leaf-level fanout means that there are more
      leaf-level rcu_node structures in the tree, which in turn means that
      RCU's grace-period initialization incurs more cache misses.  This is
      not a problem on moderate-sized servers with only a few tens of CPUs,
      but becomes a major source of real-time latency spikes on systems with
      many hundreds of CPUs.  In addition, the workloads running on these large
      systems tend to be CPU-bound, which eliminates the energy-efficiency
      advantages of synchronizing scheduling-clock interrupts.  Therefore,
      these systems need maximal values for the rcu_node leaf-level fanout.

      This commit addresses this problem by introducing a new kernel parameter
      named RCU_FANOUT_LEAF that directly controls the leaf-level fanout.
      This parameter defaults to 16 to handle the common case of a moderate
      sized lightly loaded servers, but may be set higher on larger systems.

      Reported-by: Mike Galbraith <efault@gmx.de>
      Reported-by: Dimitri Sivanich <sivanich@sgi.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Document why rcu_blocking_is_gp() is safe

      The rcu_blocking_is_gp() function tests to see if there is only one
      online CPU, and if so, synchronize_sched() and friends become no-ops.
      However, for larger systems, num_online_cpus() scans a large vector,
      and might be preempted while doing so.  While preempted, any number
      of CPUs might come online and go offline, potentially resulting in
      num_online_cpus() returning 1 when there never had only been one
      CPU online.  This could result in a too-short RCU grace period, which
      could in turn result in total failure, except that the only way that
      the grace period is too short is if there is an RCU read-side critical
      section spanning it.  For RCU-sched and RCU-bh (which are the only
      cases using rcu_blocking_is_gp()), RCU read-side critical sections
      have either preemption or bh disabled, which prevents CPUs from going
      offline.  This in turn prevents actual failures from occurring.

      This commit therefore adds a large block comment to rcu_blocking_is_gp()
      documenting why it is safe.  This commit also moves rcu_blocking_is_gp()
      into kernel/rcutree.c, which should help prevent unwary developers from
      mistaking it for a generally useful function.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Add RCU_FAST_NO_HZ tracing for idle exit

      Traces of rcu_prep_idle events can be confusing because
      rcu_cleanup_after_idle() does no tracing.  This commit therefore adds
      this tracing.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Make RCU_FAST_NO_HZ use timer rather than hrtimer

      The RCU_FAST_NO_HZ facility uses an hrtimer to wake up a CPU when
      it is allowed to go into dyntick-idle mode, which is almost always
      cancelled soon after.  This is not what hrtimers are good at, so
      this commit switches to the timer wheel.

      Reported-by: Steven Rostedt <rostedt@goodmis.org>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Make RCU_FAST_NO_HZ account for pauses out of idle

      Both Steven Rostedt's new idle-capable trace macros and the RCU_NONIDLE()
      macro can cause RCU to momentarily pause out of idle without the rest
      of the system being involved.  This can cause rcu_prepare_for_idle()
      to run through its state machine too quickly, which can in turn result
      in needless scheduling-clock interrupts.

      This commit therefore adds code to enable rcu_prepare_for_idle() to
      distinguish between an initial entry to idle on the one hand (which needs
      to advance the rcu_prepare_for_idle() state machine) and an idle reentry
      due to idle-capable trace macros and RCU_NONIDLE() on the other hand
      (which should avoid advancing the rcu_prepare_for_idle() state machine).
      Additional state is maintained to allow the timer to be correctly reposted
      when returning after a momentary pause out of idle, and even more state
      is maintained to detect when new non-lazy callbacks have been enqueued
      (which may require re-evaluation of the approach to idleness).

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: List-debug variants of rcu list routines.

      * Make __list_add_rcu check the next->prev and prev->next pointers
	just like __list_add does.
      * Make list_del_rcu use __list_del_entry, which does the same checking
	at deletion time.

      Has been running for a week here without anything being tripped up,
      but it seems worth adding for completeness just in case something
      ever does corrupt those lists.

      Signed-off-by: Dave Jones <davej@redhat.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Replace list_first_entry_rcu() with list_first_or_null_rcu()

      The list_first_entry_rcu() macro is inherently unsafe because it cannot
      be applied to an empty list.  But because RCU readers do not exclude
      updaters, a list might become empty between the time that list_empty()
      claimed it was non-empty and the time that list_first_entry_rcu() is
      invoked.  Therefore, the list_empty() test cannot be separated from the
      list_first_entry_rcu() call.  This commit therefore combines these to
      macros to create a new list_first_or_null_rcu() macro that replaces
      the old (and unsafe) list_first_entry_rcu() macro.

      This patch incorporates Paul's review comments on the previous version of
      this patch available here:

      https://lkml.org/lkml/2012/4/2/536

      This patch cannot break any upstream code because list_first_entry_rcu()
      is not being used anywhere in the kernel (tested with grep(1)), and any
      external code using it is probably broken as a result of using it.

      Signed-off-by: Michel Machado <michel@digirati.com.br>
      CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      CC: Dipankar Sarma <dipankar@in.ibm.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Clarify help text for RCU_BOOST_PRIO

      The old text confused real-time applications with real-time threads, so
      that you pretty much needed to understand how this kernel configuration
      parameter worked to understand the help text.  This commit therefore
      attempts to make the help text human-readable.

      Reported-by: Jörn Engel <joern@purestorage.com>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Make __kfree_rcu() less dependent on compiler choices

      Currently, __kfree_rcu() is implemented as an inline function, and
      contains a BUILD_BUG_ON() that malfunctions if __kfree_rcu() is compiled
      as an out-of-line function.  Unfortunately, there are compiler settings
      (e.g., -O0) that can result in __kfree_rcu() being compiled out of line,
      resulting in annoying build breakage.  This commit therefore converts
      both __kfree_rcu() and __is_kfree_rcu_offset() from inline functions to
      macros to prevent such misbehavior on the part of the compiler.

      Signed-off-by: Jan Engelhardt <jengelh@medozas.de>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Document kernel command-line parameters

      Bring RCU's kernel command-line parameter documentation up to date.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Fixes to rcutorture error handling and cleanup

      The rcutorture initialization code ignored the error returns from
      rcu_torture_onoff_init() and rcu_torture_stall_init().  The rcutorture
      cleanup code failed to NULL out a number of pointers.  These bugs will
      normally have no effect, but this commit fixes them nevertheless.

      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Introduce rcutorture testing for rcu_barrier()

      Although rcutorture does invoke rcu_barrier() and friends, it cannot
      really be called a torture test given that it invokes them only once
      at the end of the test.  This commit therefore introduces heavy-duty
      rcutorture testing for rcu_barrier(), which may be carried out
      concurrently with normal rcutorture testing.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Direct algorithmic SRCU implementation

      The current implementation of synchronize_srcu_expedited() can cause
      severe OS jitter due to its use of synchronize_sched(), which in turn
      invokes try_stop_cpus(), which causes each CPU to be sent an IPI.
      This can result in severe performance degradation for real-time workloads
      and especially for short-interation-length HPC workloads.  Furthermore,
      because only one instance of try_stop_cpus() can be making forward progress
      at a given time, only one instance of synchronize_srcu_expedited() can
      make forward progress at a time, even if they are all operating on
      distinct srcu_struct structures.

      This commit, inspired by an earlier implementation by Peter Zijlstra
      (https://lkml.org/lkml/2012/1/31/211) and by further offline discussions,
      takes a strictly algorithmic bits-in-memory approach.  This has the
      disadvantage of requiring one explicit memory-barrier instruction in
      each of srcu_read_lock() and srcu_read_unlock(), but on the other hand
      completely dispenses with OS jitter and furthermore allows SRCU to be
      used freely by CPUs that RCU believes to be idle or offline.

      The update-side implementation handles the single read-side memory
      barrier by rechecking the per-CPU counters after summing them and
      by running through the update-side state machine twice.

      This implementation has passed moderate rcutorture testing on both
      x86 and Power.  Also updated to use this_cpu_ptr() instead of per_cpu_ptr(),
      as suggested by Peter Zijlstra.

      Reported-by: Peter Zijlstra <peterz@infradead.org>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>

rcu: Remove fast check path from __synchronize_srcu()

      The fastpath in __synchronize_srcu() is designed to handle cases where
      there are a large number of concurrent calls for the same srcu_struct
      structure.  However, the Linux kernel currently does not use SRCU in
      this manner, so remove the fastpath checks for simplicity.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Increment upper bit only for srcu_read_lock()

      The purpose of the upper bit of SRCU's per-CPU counters is to guarantee
      that no reasonable series of srcu_read_lock() and srcu_read_unlock()
      operations can return the value of the counter to its original value.
      This guarantee is require only after the index has been switched to
      the other set of counters, so at most one srcu_read_lock() can affect
      a given CPU's counter.  The number of srcu_read_unlock() operations
      on a given counter is limited to the number of tasks in the system,
      which given the Linux kernel's current structure is limited to far less
      than 2^30 on 32-bit systems and far less than 2^62 on 64-bit systems.
      (Something about a limited number of bytes in the kernel's address space.)

      Therefore, if srcu_read_lock() increments the upper bits, then
      srcu_read_unlock() need not do so.  In this case, an srcu_read_lock() and
      an srcu_read_unlock() will flip the lower bit of the upper field of the
      counter.  An unreasonably large additional number of srcu_read_unlock()
      operations would be required to return the counter to its initial value,
      thus preserving the guarantee.

      This commit takes this approach, which further allows it to shrink
      the size of the upper field to one bit, making the number of
      srcu_read_unlock() operations required to return the counter to its
      initial value even more unreasonable than before.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Flip ->completed only once per SRCU grace period

      This is an optimization of the SRCU grace period.  To guard against
      preempted readers with old values of the counter, it suffices to scan the
      old counters once more, then flip ->completed only one time.  The reason
      this works is that the old readers must have incremented the old set of
      counters (if they have not yet incremented, then their critical section
      starts after this grace period, so they may be safely ignored).

      This commit therefore optimizes the second flip out in favor of a simple
      rescan.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Improve SRCU's wait_idx() comments

      The safety of SRCU is provided byy wait_idx() rather than flipping.
      The flipping actually prevents starvation.

      This commit therefore updates the comments to more accurately and
      precisely describe what is going on.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

      # This is the 25th commit message:

      rcu: Implement a variant of Peter's SRCU algorithm

      This commit implements a variant of Peter's algorithm, which may be found
      at https://lkml.org/lkml/2012/2/1/119.

      o	Make the checking lock-free to enable parallel checking.
	      Parallel checking is required when (1) the original checking
	      task is preempted for a long time, (2) sychronize_srcu_expedited()
	      starts during an ongoing SRCU grace period, or (3) we wish to
	      avoid acquiring a lock.

      o	Since the checking is lock-free, we avoid a mutex in state machine
	      for call_srcu().

      o	Remove the SRCU_REF_MASK and remove the coupling with the flipping.
	      This might allow us to remove the preempt_disable() in future
	      versions, though such removal will need great care because it
	      rescinds the one-old-reader-per-CPU guarantee.

      o	Remove a smp_mb(), simplify the comments and make the smp_mb() pairs
	      more intuitive.

      Inspired-by: Peter Zijlstra <peterz@infradead.org>
      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Remove unused srcu_barrier()

      The old srcu_barrier() macro is now unused.  This commit removes it so
      that it may be used for the SRCU flavor of rcu_barrier(), which will in
      turn be needed to allow the upcoming call_srcu() to be used from within
      modules.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Improve srcu_readers_active_idx()'s cache locality

      Expand the calls to srcu_readers_active_idx() from srcu_readers_active()
      inline.  This change improves cache locality by interating over the CPUs
      once rather than twice.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Use single value to handle expedited SRCU grace periods

      The earlier algorithm used an "expedited" flag combined with a "trycount"
      counter to differentiate between normal and expedited SRCU grace periods.
      However, the difference can be encoded into a single counter with a cutoff
      value and different initial values for expedited and normal SRCU grace
      periods.  This commit makes that change.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Implement per-domain single-threaded call_srcu() state machine

      This commit implements an SRCU state machine in support of call_srcu().
      The state machine is preemptible, light-weight, and single-threaded,
      minimizing synchronization overhead.  In particular, there is no longer
      any need for synchronize_srcu() to be guarded by a mutex.

      Expedited processing is handled, at least in the absence of concurrent
      grace-period operations on that same srcu_struct structure, by having
      the synchronize_srcu_expedited() thread take on the role of the
      workqueue thread for one iteration.

      There is a reasonable probability that a given SRCU callback will
      be invoked on the same CPU that registered it, however, there is no
      guarantee.  Concurrent SRCU grace-period primitives can cause callbacks
      to be executed elsewhere, even in absence of CPU-hotplug operations.

      Callbacks execute in process context, but under the influence of
      local_bh_disable(), so it is illegal to sleep in an SRCU callback
      function.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Add rcutorture test for call_srcu()

      Add srcu_torture_deferred_free() for srcu_ops so as to test the new
      call_srcu().  Rename the original srcu_ops to srcu_sync_ops.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Move PREEMPT_RCU preemption to switch_to() invocation

      Currently, PREEMPT_RCU readers are enqueued upon entry to the scheduler.
      This is inefficient because enqueuing is required only if there is a
      context switch, and entry to the scheduler does not guarantee a context
      switch.

      The commit therefore moves the enqueuing to immediately precede the
      call to switch_to() from the scheduler.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Linus Torvalds <torvalds@linux-foundation.org>

rcu: Make exit_rcu() more precise and consolidate

      When running preemptible RCU, if a task exits in an RCU read-side
      critical section having blocked within that same RCU read-side critical
      section, the task must be removed from the list of tasks blocking a
      grace period (perhaps the current grace period, perhaps the next grace
      period, depending on timing).  The exit() path invokes exit_rcu() to
      do this cleanup.

      However, the current implementation of exit_rcu() needlessly does the
      cleanup even if the task did not block within the current RCU read-side
      critical section, which wastes time and needlessly increases the size
      of the state space.  Fix this by only doing the cleanup if the current
      task is actually on the list of tasks blocking some grace period.

      While we are at it, consolidate the two identical exit_rcu() functions
      into a single function.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Linus Torvalds <torvalds@linux-foundation.org>

rcu: Update RCU maintainership

      Split SRCU out and add Lai Jiangshan as SRCU co-maintainer.

      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Precompute RCU_FAST_NO_HZ timer offsets

      When a CPU is entering dyntick-idle mode, tick_nohz_stop_sched_tick()
      calls rcu_needs_cpu() see if RCU needs that CPU, and, if not, computes the
      next wakeup time based on the timer wheels.  Only later, when actually
      entering the idle loop, rcu_prepare_for_idle() will be invoked.  In some
      cases, rcu_prepare_for_idle() will post timers to wake the CPU back up.
      But all for naught: The next wakeup time for the CPU has already been
      computed, and posting a timer afterwards does not force that wakeup
      time to be recomputed.  This means that rcu_prepare_for_idle()'s have
      no effect.

      This is not a problem on a busy system because something else will wake
      up the CPU soon enough.  However, on lightly loaded systems, the CPU
      might stay asleep for a considerable length of time.  If that CPU has
      a callback that the rest of the system is waiting on, the system might
      run very slowly or (in theory) even hang.

      This commit avoids this problem by having rcu_needs_cpu() give
      tick_nohz_stop_sched_tick() an estimate of when RCU will need the CPU
      to wake back up, which tick_nohz_stop_sched_tick() takes into account
      when programming the CPU's wakeup time.  An alternative approach is
      for rcu_prepare_for_idle() to use hrtimers instead of normal timers,
      but timers are much more efficient than are hrtimers for frequently
      and repeatedly posting and cancelling a given timer, which is exactly
      what RCU_FAST_NO_HZ does.

      Reported-by: Pascal Chapperon <pascal.chapperon@wanadoo.fr>
      Reported-by: Heiko Carstens <heiko.carstens@de.ibm.com>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Heiko Carstens <heiko.carstens@de.ibm.com>
      Tested-by: Pascal Chapperon <pascal.chapperon@wanadoo.fr>

irq_work: Don't stop the tick with pending works

      Don't stop the tick if we have pending irq works on the
      queue, otherwise if the arch can't raise self-IPIs, we may not
      find an opportunity to execute the pending works for a while.

      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Acked-by: Steven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>

Signed-off-by: moddingg33k <moddingg33k@t-online.de>
time: RCU permitted to stop idle entry via softirq

The can_stop_idle_tick() function complains if a softirq vector is
raised too late in the idle-entry process, presumably in order to
prevent dangling softirq invocations from being delayed across the
full idle period, which might be indefinitely long -- and if softirq
was asserted any later than the call to this function, such a delay
might well happen.

However, RCU needs to be able to use softirq to stop idle entry in
order to be able to drain RCU callbacks from the current CPU, which in
turn enables faster entry into dyntick-idle mode, which in turn reduces
power consumption.  Because RCU takes this action at a well-defined
point in the idle-entry path, it is safe for RCU to take this approach.

This commit therefore silences the error message that is sometimes
produced when the going-idle CPU suddenly finds that it has an RCU_SOFTIRQ
to process.  The error message will continue to be issued for other
softirq vectors.

Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
sched: remove redundant update_runtime notifier

migration_call() will do all the things that update_runtime() does.
So it seems update_runtime() is a redundant notifier, remove it.

Furthermore, there is potential risk that the current code will catch
BUG_ON at line 687 of rt.c when do cpu hotplug while there are realtime
threads running because of enable runtime twice.

Signed-off-by: Neil Zhang <zhangwm@marvell.com>
Reviewed-on: http://git-master/r/215596
Reviewed-on: http://git-master/r/223067
Reviewed-by: Peter Boonstoppel <pboonstoppel@nvidia.com>
Tested-by: Peter Boonstoppel <pboonstoppel@nvidia.com>
Reviewed-by: Paul Walmsley <pwalmsley@nvidia.com>
Reviewed-by: Automatic_Commit_Validation_User
Reviewed-by: Diwakar Tundlam <dtundlam@nvidia.com>
sched: cpu_power: enable ARCH_POWER

Heteregeneous ARM platform uses arch_scale_freq_power function
to reflect the relative capacity of each core

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1341826026-6504-6-git-send-email-vincent.guittot@linaro.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
msm: kgsl: Remove extra interrupts when setting MMU state

The interrupts added to the ringbuffer on PTFLUSH and TLBUPDATE
were causing a major increase in the number of interrupts from the GPU.
This was leading to increase in power and loss of performance. Add a check
to turn off IOMMU clocks when going to SLEEP.

Change-Id: I41617dd3b7b3f7d9622523f2a1407b912dbd989e
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
mm: fix calculation of dirtyable memory

The system uses global_dirtyable_memory() to calculate number of
dirtyable pages/pages that can be allocated to the page cache.  A bug
causes an underflow thus making the page count look like a big unsigned
number.  This in turn confuses the dirty writeback throttling to
aggressively write back pages as they become dirty (usually 1 page at a
time).  This generally only affects systems with highmem because the
underflowed count gets subtracted from the global count of dirtyable
memory.

The problem was introduced with v3.2-4896-gab8fabd

Fix is to ensure we don't get an underflowed total of either highmem or
global dirtyable memory.

Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Signed-off-by: Puneet Kumar <puneetster@chromium.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Tested-by: Damien Wyart <damien.wyart@free.fr>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
arch: remove direct definitions of KERN_<LEVEL> uses

Add #include <linux/kern_levels.h> so that the #define KERN_<LEVEL> macros
don't have to be duplicated.

Signed-off-by: Joe Perches <joe@perches.com>
Cc: Kay Sievers <kay.sievers@vrfy.org>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Kay Sievers <kay@vrfy.org>
Acked-by: David Howells <dhowells@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

printk: add kern_levels.h to make KERN_<LEVEL> available for asm use

Separate the printk.h file into 2 pieces so the definitions can be used in
asm files.

Signed-off-by: Joe Perches <joe@perches.com>
Cc: Kay Sievers <kay.sievers@vrfy.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
exportfs: add FILEID_INVALID to indicate invalid fid_type

This commit adds FILEID_INVALID = 0xff in fid_type to
indicate invalid fid_type

It avoids using magic number 255
cgroups: disallow attaching kthreadd or PF_THREAD_BOUND threads

Allowing kthreadd to be moved to a non-root group makes no sense, it being
a global resource, and needlessly leads unsuspecting users toward trouble.

1. An RT workqueue worker thread spawned in a task group with no rt_runtime
allocated is not schedulable.  Simple user error, but harmful to the box.

2. A worker thread which acquires PF_THREAD_BOUND can never leave a cpuset,
rendering the cpuset immortal.

Save the user some unexpected trouble, just say no.

Signed-off-by: Mike Galbraith <mgalbraith@suse.de>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Li Zefan <lizefan@huawei.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
cgroup: *squashing updates from 01. April 2012*

cgroup: deprecate remount option changes

      This patch marks the following features for deprecation.

      * Rebinding subsys by remount: Never reached useful state - only works
	on empty hierarchies.

      * release_agent update by remount: release_agent itself will be
	replaced with conventional fsnotify notification.

      v2: Lennart pointed out that "name=" is necessary for mounts w/o any
	  controller attached.  Drop "name=" deprecation.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Lennart Poettering <mzxreary@0pointer.de>

cgroup: move cgroup_clear_directory() call out of cgroup_populate_dir()

      cgroup_populate_dir() currently clears all files and then repopulate
      the directory; however, the clearing part is only useful when it's
      called from cgroup_remount().  Relocate the invocation to
      cgroup_remount().

      This is to prepare for further cgroup file handling updates.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: build list of all cgroups under a given cgroupfs_root

      Build a list of all cgroups anchored at cgroupfs_root->allcg_list and
      going through cgroup->allcg_node.  The list is protected by
      cgroup_mutex and will be used to improve cgroup file handling.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: implement cgroup_add_cftypes() and friends

      Currently, cgroup directories are populated by subsys->populate()
      callback explicitly creating files on each cgroup creation.  This
      level of flexibility isn't needed or desirable.  It provides largely
      unused flexibility which call for abuses while severely limiting what
      the core layer can do through the lack of structure and conventions.

      Per each cgroup file type, the only distinction that cgroup users is
      making is whether a cgroup is root or not, which can easily be
      expressed with flags.

      This patch introduces cgroup_add_cftypes().  These deal with cftypes
      instead of individual files - controllers indicate that certain types
      of files exist for certain subsystem.  Newly added CFTYPE_*_ON_ROOT
      flags indicate whether a cftype should be excluded or created only on
      the root cgroup.

      cgroup_add_cftypes() can be called any time whether the target
      subsystem is currently attached or not.  cgroup core will create files
      on the existing cgroups as necessary.

      Also, cgroup_subsys->base_cftypes is added to ease registration of the
      base files for the subsystem.  If non-NULL on subsys init, the cftypes
      pointed to by ->base_cftypes are automatically registered on subsys
      init / load.

      Further patches will convert the existing users and remove the file
      based interface.  Note that this interface allows dynamic addition of
      files to an active controller.  This will be used for sub-controller
      modularity and unified hierarchy in the longer term.

      This patch implements the new mechanism but doesn't apply it to any
      user.

      v2: replaced DECLARE_CGROUP_CFTYPES[_COND]() with
	  cgroup_subsys->base_cftypes, which works better for cgroup_subsys
	  which is loaded as module.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: merge cft_release_agent cftype array into the base files array

      Now that cftype can express whether a file should only be on root,
      cft_release_agent can be merged into the base files cftypes array.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: relocate cftype and cgroup_subsys definitions in controllers

      blk-cgroup, netprio_cgroup, cls_cgroup and tcp_memcontrol
      unnecessarily define cftype array and cgroup_subsys structures at the
      top of the file, which is unconventional and necessiates forward
      declaration of methods.

      This patch relocates those below the definitions of the methods and
      removes the forward declarations.  Note that forward declaration of
      tcp_files[] is added in tcp_memcontrol.c for tcp_init_cgroup().  This
      will be removed soon by another patch.

      This patch doesn't introduce any functional change.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

      # This is the 7th commit message:

      cgroup: convert all non-memcg controllers to the new cftype interface

      Convert debug, freezer, cpuset, cpu_cgroup, cpuacct, net_prio, blkio,
      net_cls and device controllers to use the new cftype based interface.
      Termination entry is added to cftype arrays and populate callbacks are
      replaced with cgroup_subsys->base_cftypes initializations.

      This is functionally identical transformation.  There shouldn't be any
      visible behavior change.

      memcg is rather special and will be converted separately.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Paul Menage <paul@paulmenage.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: "David S. Miller" <davem@davemloft.net>
      Cc: Vivek Goyal <vgoyal@redhat.com>

memcg: always create memsw files if CONFIG_CGROUP_MEM_RES_CTLR_SWAP

      Instead of conditioning creation of memsw files on do_swap_account,
      always create the files if compiled-in and fail read/write attempts
      with -EOPNOTSUPP if !do_swap_account.

      This is suggested by KAMEZAWA to simplify memcg file creation so that
      it can use cgroup->subsys_cftypes.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: convert memcg controller to the new cftype interface

      Convert memcg to use the new cftype based interface.  kmem support
      abuses ->populate() for mem_cgroup_sockets_init() so it can't be
      removed at the moment.

      tcp_memcontrol is updated so that tcp_files[] is registered via a
      __initcall.  This change also allows removing the forward declaration
      of tcp_files[].  Removed.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Michal Hocko <mhocko@suse.cz>
      Cc: Balbir Singh <bsingharora@gmail.com>
      Cc: Glauber Costa <glommer@parallels.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Greg Thelen <gthelen@google.com>

cgroup: remove cgroup_add_file[s]()

      No controller is using cgroup_add_files[s]().  Unexport them, and
      convert cgroup_add_files() to handle NULL entry terminated array
      instead of taking count explicitly and continue creation on failure
      for internal use.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: relocate __d_cgrp() and __d_cft()

      Move the two macros upwards as they'll be used earlier in the file.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

      # This is the 12th commit message:

      cgroup: introduce struct cfent

      This patch adds cfent (cgroup file entry) which is the association
      between a cgroup and a file.  This is in-cgroup representation of
      files under a cgroup directory.  This simplifies walking walking
      cgroup files and thus cgroup_clear_directory(), which is now
      implemented in two parts - cgroup_rm_file() and a loop around it.

      cgroup_rm_file() will be used to implement cftype removal and cfent is
      scheduled to serve cgroup specific per-file data (e.g. for sysfs-like
      "sever" semantics).

      v2: - cfe was freed from cgroup_rm_file() which led to use-after-free
	    if the file had openers at the time of removal.  Moved to
	    cgroup_diput().

	  - cgroup_clear_directory() triggered WARN_ON_ONCE() if d_subdirs
	    wasn't empty after removing all files.  This triggered
	    spuriously if some files were open during directory clearing.
	    Removed.

      v3: - In cgroup_diput(), WARN_ONCE(!list_empty(&cfe->node)) could be
	    spuriously triggered for root cgroups because they don't go
	    through cgroup_clear_directory() on unmount.  Don't trigger WARN
	    for root cgroups.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Glauber Costa <glommer@parallels.com>

cgroup: implement cgroup_rm_cftypes()

      Implement cgroup_rm_cftypes() which removes an array of cftypes from a
      subsystem.  It can be called whether the target subsys is attached or
      not.  cgroup core will remove the specified file from all existing
      cgroups.

      This will be used to improve sub-subsys modularity and will be helpful
      for unified hierarchy.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: use negative bias on css->refcnt to block css_tryget()

      When a cgroup is about to be removed, cgroup_clear_css_refs() is
      called to check and ensure that there are no active css references.

      This is currently achieved by dropping the refcnt to zero iff it has
      only the base ref.  If all css refs could be dropped to zero, ref
      clearing is successful and CSS_REMOVED is set on all css.  If not, the
      base ref is restored.  While css ref is zero w/o CSS_REMOVED set, any
      css_tryget() attempt on it busy loops so that they are atomic
      w.r.t. the whole css ref clearing.

      This does work but dropping and re-instating the base ref is somewhat
      hairy and makes it difficult to add more logic to the put path as
      there are two of them - the regular css_put() and the reversible base
      ref clearing.

      This patch updates css ref clearing such that blocking new
      css_tryget() and putting the base ref are separate operations.
      CSS_DEACT_BIAS, defined as INT_MIN, is added to css->refcnt and
      css_tryget() busy loops while refcnt is negative.  After all css refs
      are deactivated, if they were all one, ref clearing succeeded and
      CSS_REMOVED is set and the base ref is put using the regular
      css_put(); otherwise, CSS_DEACT_BIAS is subtracted from the refcnts
      and the original postive values are restored.

      css_refcnt() accessor which always returns the unbiased positive
      reference counts is added and used to simplify refcnt usages.  While
      at it, relocate and reformat comments in cgroup_has_css_refs().

      This separates css->refcnt deactivation and putting the base ref,
      which enables the next patch to make ref clearing optional.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>

cgroup: make css->refcnt clearing on cgroup removal optional

      Currently, cgroup removal tries to drain all css references.  If there
      are active css references, the removal logic waits and retries
      ->pre_detroy() until either all refs drop to zero or removal is
      cancelled.

      This semantics is unusual and adds non-trivial complexity to cgroup
      core and IMHO is fundamentally misguided in that it couples internal
      implementation details (references to internal data structure) with
      externally visible operation (rmdir).  To userland, this is a behavior
      peculiarity which is unnecessary and difficult to expect (css refs is
      otherwise invisible from userland), and, to policy implementations,
      this is an unnecessary restriction (e.g. blkcg wants to hold css refs
      for caching purposes but can't as that becomes visible as rmdir hang).

      Unfortunately, memcg currently depends on ->pre_destroy() retrials and
      cgroup removal vetoing and can't be immmediately switched to the new
      behavior.  This patch introduces the new behavior of not waiting for
      css refs to drain and maintains the old behavior for subsystems which
      have __DEPRECATED_clear_css_refs set.

      Once, memcg is updated, we can drop the code paths for the old
      behavior as proposed in the following patch.  Note that the following
      patch is incorrect in that dput work item is in cgroup and may lose
      some of dputs when multiples css's are released back-to-back, and
      __css_put() triggers check_for_release() when refcnt reaches 0 instead
      of 1; however, it shows what part can be removed.

	http://thread.gmane.org/gmane.linux.kernel.containers/22559/focus=75251

      Note that, in not-too-distant future, cgroup core will start emitting
      warning messages for subsys which require the old behavior, so please
      get moving.

      Signed-off-by: Tejun Heo <tj@kernel.org>
      Acked-by: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Vivek Goyal <vgoyal@redhat.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Michal Hocko <mhocko@suse.cz>
      Cc: Balbir Singh <bsingharora@gmail.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>