ARM: 7587/1: implement optimized percpu variable access

Use the previously unused TPIDRPRW register to store percpu offsets.
TPIDRPRW is only accessible in PL1, so it can only be used in the kernel.

This replaces 2 loads with a mrc instruction for each percpu variable
access. With hackbench, the performance improvement is 1.4% on Cortex-A9
(highbank). Taking an average of 30 runs of "hackbench -l 1000" yields:

Before: 6.2191
After: 6.1348

Will Deacon reported similar delta on v6 with 11MPCore.

The asm "memory clobber" are needed here to ensure the percpu offset
gets reloaded. Testing by Will found that this would not happen in
__schedule() which is a bit of a special case as preemption is disabled
but the execution can move cores.

Signed-off-by: Rob Herring <rob.herring@calxeda.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
ARM: 7493/1: use generic unaligned.h

This moves ARM over to the asm-generic/unaligned.h header. This has the
benefit of better code generated especially for ARMv7 on gcc 4.7+
compilers.

As Arnd Bergmann, points out: The asm-generic version uses the "struct"
version for native-endian unaligned access and the "byteshift" version
for the opposite endianess. The current ARM version however uses the
"byteshift" implementation for both.

Thanks to Nicolas Pitre for the excellent analysis:

Test case:

int foo (int *x) { return get_unaligned(x); }
long long bar (long long *x) { return get_unaligned(x); }

With the current ARM version:

foo:
	ldrb	r3, [r0, #2]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 2B], MEM[(const u8 *)x_1(D) + 2B]
	ldrb	r1, [r0, #1]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 1B], MEM[(const u8 *)x_1(D) + 1B]
	ldrb	r2, [r0, #0]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D)], MEM[(const u8 *)x_1(D)]
	mov	r3, r3, asl #16	@ tmp154, MEM[(const u8 *)x_1(D) + 2B],
	ldrb	r0, [r0, #3]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 3B], MEM[(const u8 *)x_1(D) + 3B]
	orr	r3, r3, r1, asl #8	@, tmp155, tmp154, MEM[(const u8 *)x_1(D) + 1B],
	orr	r3, r3, r2	@ tmp157, tmp155, MEM[(const u8 *)x_1(D)]
	orr	r0, r3, r0, asl #24	@,, tmp157, MEM[(const u8 *)x_1(D) + 3B],
	bx	lr	@

bar:
	stmfd	sp!, {r4, r5, r6, r7}	@,
	mov	r2, #0	@ tmp184,
	ldrb	r5, [r0, #6]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 6B], MEM[(const u8 *)x_1(D) + 6B]
	ldrb	r4, [r0, #5]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 5B], MEM[(const u8 *)x_1(D) + 5B]
	ldrb	ip, [r0, #2]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 2B], MEM[(const u8 *)x_1(D) + 2B]
	ldrb	r1, [r0, #4]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 4B], MEM[(const u8 *)x_1(D) + 4B]
	mov	r5, r5, asl #16	@ tmp175, MEM[(const u8 *)x_1(D) + 6B],
	ldrb	r7, [r0, #1]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 1B], MEM[(const u8 *)x_1(D) + 1B]
	orr	r5, r5, r4, asl #8	@, tmp176, tmp175, MEM[(const u8 *)x_1(D) + 5B],
	ldrb	r6, [r0, #7]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 7B], MEM[(const u8 *)x_1(D) + 7B]
	orr	r5, r5, r1	@ tmp178, tmp176, MEM[(const u8 *)x_1(D) + 4B]
	ldrb	r4, [r0, #0]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D)], MEM[(const u8 *)x_1(D)]
	mov	ip, ip, asl #16	@ tmp188, MEM[(const u8 *)x_1(D) + 2B],
	ldrb	r1, [r0, #3]	@ zero_extendqisi2	@ MEM[(const u8 *)x_1(D) + 3B], MEM[(const u8 *)x_1(D) + 3B]
	orr	ip, ip, r7, asl #8	@, tmp189, tmp188, MEM[(const u8 *)x_1(D) + 1B],
	orr	r3, r5, r6, asl #24	@,, tmp178, MEM[(const u8 *)x_1(D) + 7B],
	orr	ip, ip, r4	@ tmp191, tmp189, MEM[(const u8 *)x_1(D)]
	orr	ip, ip, r1, asl #24	@, tmp194, tmp191, MEM[(const u8 *)x_1(D) + 3B],
	mov	r1, r3	@,
	orr	r0, r2, ip	@ tmp171, tmp184, tmp194
	ldmfd	sp!, {r4, r5, r6, r7}
	bx	lr

In both cases the code is slightly suboptimal.  One may wonder why
wasting r2 with the constant 0 in the second case for example.  And all
the mov's could be folded in subsequent orr's, etc.

Now with the asm-generic version:

foo:
	ldr	r0, [r0, #0]	@ unaligned	@,* x
	bx	lr	@

bar:
	mov	r3, r0	@ x, x
	ldr	r0, [r0, #0]	@ unaligned	@,* x
	ldr	r1, [r3, #4]	@ unaligned	@,
	bx	lr	@

This is way better of course, but only because this was compiled for
ARMv7. In this case the compiler knows that the hardware can do
unaligned word access.  This isn't that obvious for foo(), but if we
remove the get_unaligned() from bar as follows:

long long bar (long long *x) {return *x; }

then the resulting code is:

bar:
	ldmia	r0, {r0, r1}	@ x,,
	bx	lr	@

So this proves that the presumed aligned vs unaligned cases does have
influence on the instructions the compiler may use and that the above
unaligned code results are not just an accident.

Still... this isn't fully conclusive without at least looking at the
resulting assembly fron a pre ARMv6 compilation.  Let's see with an
ARMv5 target:

foo:
	ldrb	r3, [r0, #0]	@ zero_extendqisi2	@ tmp139,* x
	ldrb	r1, [r0, #1]	@ zero_extendqisi2	@ tmp140,
	ldrb	r2, [r0, #2]	@ zero_extendqisi2	@ tmp143,
	ldrb	r0, [r0, #3]	@ zero_extendqisi2	@ tmp146,
	orr	r3, r3, r1, asl #8	@, tmp142, tmp139, tmp140,
	orr	r3, r3, r2, asl #16	@, tmp145, tmp142, tmp143,
	orr	r0, r3, r0, asl #24	@,, tmp145, tmp146,
	bx	lr	@

bar:
	stmfd	sp!, {r4, r5, r6, r7}	@,
	ldrb	r2, [r0, #0]	@ zero_extendqisi2	@ tmp139,* x
	ldrb	r7, [r0, #1]	@ zero_extendqisi2	@ tmp140,
	ldrb	r3, [r0, #4]	@ zero_extendqisi2	@ tmp149,
	ldrb	r6, [r0, #5]	@ zero_extendqisi2	@ tmp150,
	ldrb	r5, [r0, #2]	@ zero_extendqisi2	@ tmp143,
	ldrb	r4, [r0, #6]	@ zero_extendqisi2	@ tmp153,
	ldrb	r1, [r0, #7]	@ zero_extendqisi2	@ tmp156,
	ldrb	ip, [r0, #3]	@ zero_extendqisi2	@ tmp146,
	orr	r2, r2, r7, asl #8	@, tmp142, tmp139, tmp140,
	orr	r3, r3, r6, asl #8	@, tmp152, tmp149, tmp150,
	orr	r2, r2, r5, asl #16	@, tmp145, tmp142, tmp143,
	orr	r3, r3, r4, asl #16	@, tmp155, tmp152, tmp153,
	orr	r0, r2, ip, asl #24	@,, tmp145, tmp146,
	orr	r1, r3, r1, asl #24	@,, tmp155, tmp156,
	ldmfd	sp!, {r4, r5, r6, r7}
	bx	lr

Compared to the initial results, this is really nicely optimized and I
couldn't do much better if I were to hand code it myself.

Signed-off-by: Rob Herring <rob.herring@calxeda.com>
Reviewed-by: Nicolas Pitre <nico@linaro.org>
Tested-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
introduce for_each_thread() to replace the buggy while_each_thread()

while_each_thread() and next_thread() should die, almost every lockless
usage is wrong.

1. Unless g == current, the lockless while_each_thread() is not safe.

   while_each_thread(g, t) can loop forever if g exits, next_thread()
   can't reach the unhashed thread in this case. Note that this can
   happen even if g is the group leader, it can exec.

2. Even if while_each_thread() itself was correct, people often use
   it wrongly.

   It was never safe to just take rcu_read_lock() and loop unless
   you verify that pid_alive(g) == T, even the first next_thread()
   can point to the already freed/reused memory.

This patch adds signal_struct->thread_head and task->thread_node to
create the normal rcu-safe list with the stable head.  The new
for_each_thread(g, t) helper is always safe under rcu_read_lock() as
long as this task_struct can't go away.

Note: of course it is ugly to have both task_struct->thread_node and the
old task_struct->thread_group, we will kill it later, after we change
the users of while_each_thread() to use for_each_thread().

Perhaps we can kill it even before we convert all users, we can
reimplement next_thread(t) using the new thread_head/thread_node.  But
we can't do this right now because this will lead to subtle behavioural
changes.  For example, do/while_each_thread() always sees at least one
task, while for_each_thread() can do nothing if the whole thread group
has died.  Or thread_group_empty(), currently its semantics is not clear
unless thread_group_leader(p) and we need to audit the callers before we
can change it.

So this patch adds the new interface which has to coexist with the old
one for some time, hopefully the next changes will be more or less
straightforward and the old one will go away soon.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Sergey Dyasly <dserrg@gmail.com>
Tested-by: Sergey Dyasly <dserrg@gmail.com>
Reviewed-by: Sameer Nanda <snanda@chromium.org>
Acked-by: David Rientjes <rientjes@google.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mandeep Singh Baines <msb@chromium.org>
Cc: "Ma, Xindong" <xindong.ma@intel.com>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
oom_kill: change oom_kill.c to use for_each_thread()

Change oom_kill.c to use for_each_thread() rather than the racy
while_each_thread() which can loop forever if we race with exit.

Note also that most users were buggy even if while_each_thread() was
fine, the task can exit even _before_ rcu_read_lock().

Fortunately the new for_each_thread() only requires the stable
task_struct, so this change fixes both problems.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Sergey Dyasly <dserrg@gmail.com>
Tested-by: Sergey Dyasly <dserrg@gmail.com>
Reviewed-by: Sameer Nanda <snanda@chromium.org>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mandeep Singh Baines <msb@chromium.org>
Cc: "Ma, Xindong" <xindong.ma@intel.com>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>
Acked-by: David Rientjes <rientjes@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
softirq: reduce latencies

In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.

This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.

This patch changes the fallback to ksoftirqd condition to :

- A time limit of 2 ms.
- need_resched() being set on current task

When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.

Using need_resched() as the only condition can trigger RCU stalls,
as we can keep BH disabled for too long.

I ran several benchmarks and got no significant difference in
throughput, but a very significant reduction of latencies (one order
of magnitude) :

In following bench, 200 antagonist "netperf -t TCP_RR" are started in
background, using all available cpus.

Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
IRQ (hard+soft)

Before patch :

# netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=550110.424
MIN_LATENCY=146858
MAX_LATENCY=997109
P50_LATENCY=305000
P90_LATENCY=550000
P99_LATENCY=710000
MEAN_LATENCY=376989.12
STDDEV_LATENCY=184046.92

After patch :

# netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=40545.492
MIN_LATENCY=9834
MAX_LATENCY=78366
P50_LATENCY=33583
P90_LATENCY=59000
P99_LATENCY=69000
MEAN_LATENCY=38364.67
STDDEV_LATENCY=12865.26

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: David Miller <davem@davemloft.net>
Cc: Tom Herbert <therbert@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
bdi: use deferable timer for sync_supers task

sync_supers task currently wakes up periodically for superblock
writeback. This hurts power on battery driven devices. This patch
turns this housekeeping timer into a deferable timer so that it
does not fire when system is really idle.

CRs-Fixed: 353700
Change-Id: Idc7953b5d0580546808bc5832291ca570837ee7f
Signed-off-by: Yong Wang <yong.y.wang@intel.com>
Signed-off-by: Xia Wu <xia.wu@intel.com>
Signed-off-by: Krishna Vanka <kvanka@codeaurora.org>
writeback: fix race that cause writeback hung

There is a race between mark inode dirty and writeback thread, see the
following scenario.  In this case, writeback thread will not run though
there is dirty_io.

__mark_inode_dirty()                                          bdi_writeback_workfn()
    ...                                                       	...
    spin_lock(&inode->i_lock);
    ...
    if (bdi_cap_writeback_dirty(bdi)) {
	<<< assume wb has dirty_io, so wakeup_bdi is false.
	<<< the following inode_dirty also have wakeup_bdi false.
	if (!wb_has_dirty_io(&bdi->wb))
		wakeup_bdi = true;
    }
    spin_unlock(&inode->i_lock);
								<<< assume last dirty_io is removed here.
								pages_written = wb_do_writeback(wb);
								...
								<<< work_list empty and wb has no dirty_io,
								<<< delayed_work will not be queued.
								if (!list_empty(&bdi->work_list) ||
								    (wb_has_dirty_io(wb) && dirty_writeback_interval))
								    queue_delayed_work(bdi_wq, &wb->dwork,
									msecs_to_jiffies(dirty_writeback_interval * 10));
    spin_lock(&bdi->wb.list_lock);
    inode->dirtied_when = jiffies;
    <<< new dirty_io is added.
    list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
    spin_unlock(&bdi->wb.list_lock);

    <<< though there is dirty_io, but wakeup_bdi is false,
    <<< so writeback thread will not be waked up and
    <<< the new dirty_io will not be flushed.
    if (wakeup_bdi)
	bdi_wakeup_thread_delayed(bdi);

Writeback will run until there is a new flush work queued.  This may cause
a lot of dirty pages stay in memory for a long time.

Signed-off-by: Junxiao Bi <junxiao.bi@oracle.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
writeback: fix occasional slow sync(1)

In case when system contains no dirty pages, wakeup_flusher_threads() will
submit WB_SYNC_NONE writeback for 0 pages so wb_writeback() exits
immediately without doing anything, even though there are dirty inodes in
the system.  Thus sync(1) will write all the dirty inodes from a
WB_SYNC_ALL writeback pass which is slow.

Fix the problem by using get_nr_dirty_pages() in wakeup_flusher_threads()
instead of calculating number of dirty pages manually.  That function also
takes number of dirty inodes into account.

Signed-off-by: Jan Kara <jack@suse.cz>
Reported-by: Paul Taysom <taysom@chromium.org>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
sync: don't block the flusher thread waiting on IO

When sync does it's WB_SYNC_ALL writeback, it issues data Io and
then immediately waits for IO completion. This is done in the
context of the flusher thread, and hence completely ties up the
flusher thread for the backing device until all the dirty inodes
have been synced. On filesystems that are dirtying inodes constantly
and quickly, this means the flusher thread can be tied up for
minutes per sync call and hence badly affect system level write IO
performance as the page cache cannot be cleaned quickly.

We already have a wait loop for IO completion for sync(2), so cut
this out of the flusher thread and delegate it to wait_sb_inodes().
Hence we can do rapid IO submission, and then wait for it all to
complete.

Effect of sync on fsmark before the patch:

FSUse%        Count         Size    Files/sec     App Overhead
.....
      0       640000         4096      35154.6          1026984
      0       720000         4096      36740.3          1023844
      0       800000         4096      36184.6           916599
      0       880000         4096       1282.7          1054367
      0       960000         4096       3951.3           918773
      0      1040000         4096      40646.2           996448
      0      1120000         4096      43610.1           895647
      0      1200000         4096      40333.1           921048

And a single sync pass took:

  real    0m52.407s
  user    0m0.000s
  sys     0m0.090s

After the patch, there is no impact on fsmark results, and each
individual sync(2) operation run concurrently with the same fsmark
workload takes roughly 7s:

  real    0m6.930s
  user    0m0.000s
  sys     0m0.039s

IOWs, sync is 7-8x faster on a busy filesystem and does not have an
adverse impact on ongoing async data write operations.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
timer: Fix mod_timer_pinned() header comment

The mod_timer_pinned() header comment states that it prevents timers
from being migrated to a different CPU.  This is not the case, instead,
it ensures that the timer is posted to the current CPU, but does nothing
to prevent CPU-hotplug operations from migrating the timer.

This commit therefore brings the comment header into alignment with
reality.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Steven Rostedt <rostedt@goodmis.org>
rcu: *updating rcu APIs*

rcu: Avoid invoking RCU core on offline CPUs

      Offline CPUs transition through the scheduler to the idle loop one
      last time before being shut down.  This can result in RCU raising
      softirq on this CPU, which is at best useless given that the CPU's
      callbacks will be offloaded at CPU_DEAD time.  This commit therefore
      avoids raising softirq on offline CPUs.

      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Fix batch-limit size problem

      Commit 29c00b4a1d9e27 (rcu: Add event-tracing for RCU callback
      invocation) added a regression in rcu_do_batch()

      Under stress, RCU is supposed to allow to process all items in queue,
      instead of a batch of 10 items (blimit), but an integer overflow makes
      the effective limit being 1.  So, unless there is frequent idle periods
      (during which RCU ignores batch limits), RCU can be forced into a
      state where it cannot keep up with the callback-generation rate,
      eventually resulting in OOM.

      This commit therefore converts a few variables in rcu_do_batch() from
      int to long to fix this problem, along with the module parameters
      controlling the batch limits.

      Signed-off-by: Eric Dumazet <edumazet@google.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: <stable@vger.kernel.org> # 3.2 +

rcu: Fix day-one dyntick-idle stall-warning bug

      Each grace period is supposed to have at least one callback waiting
      for that grace period to complete.  However, if CONFIG_NO_HZ=n, an
      extra callback-free grace period is no big problem -- it will chew up
      a tiny bit of CPU time, but it will complete normally.  In contrast,
      CONFIG_NO_HZ=y kernels have the potential for all the CPUs to go to
      sleep indefinitely, in turn indefinitely delaying completion of the
      callback-free grace period.  Given that nothing is waiting on this grace
      period, this is also not a problem.

      That is, unless RCU CPU stall warnings are also enabled, as they are
      in recent kernels.  In this case, if a CPU wakes up after at least one
      minute of inactivity, an RCU CPU stall warning will result.  The reason
      that no one noticed until quite recently is that most systems have enough
      OS noise that they will never remain absolutely idle for a full minute.
      But there are some embedded systems with cut-down userspace configurations
      that consistently get into this situation.

      All this begs the question of exactly how a callback-free grace period
      gets started in the first place.  This can happen due to the fact that
      CPUs do not necessarily agree on which grace period is in progress.
      If a CPU still believes that the grace period that just completed is
      still ongoing, it will believe that it has callbacks that need to wait for
      another grace period, never mind the fact that the grace period that they
      were waiting for just completed.  This CPU can therefore erroneously
      decide to start a new grace period.  Note that this can happen in
      TREE_RCU and TREE_PREEMPT_RCU even on a single-CPU system:  Deadlock
      considerations mean that the CPU that detected the end of the grace
      period is not necessarily officially informed of this fact for some time.

      Once this CPU notices that the earlier grace period completed, it will
      invoke its callbacks.  It then won't have any callbacks left.  If no
      other CPU has any callbacks, we now have a callback-free grace period.

      This commit therefore makes CPUs check more carefully before starting a
      new grace period.  This new check relies on an array of tail pointers
      into each CPU's list of callbacks.  If the CPU is up to date on which
      grace periods have completed, it checks to see if any callbacks follow
      the RCU_DONE_TAIL segment, otherwise it checks to see if any callbacks
      follow the RCU_WAIT_TAIL segment.  The reason that this works is that
      the RCU_WAIT_TAIL segment will be promoted to the RCU_DONE_TAIL segment
      as soon as the CPU is officially notified that the old grace period
      has ended.

      This change is to cpu_needs_another_gp(), which is called in a number
      of places.  The only one that really matters is in rcu_start_gp(), where
      the root rcu_node structure's ->lock is held, which prevents any
      other CPU from starting or completing a grace period, so that the
      comparison that determines whether the CPU is missing the completion
      of a grace period is stable.

      Change-Id: Ic77e2bbe865059aa6f363db284473e9adcf8797e
      Reported-by: Becky Bruce <bgillbruce@gmail.com>
      Reported-by: Subodh Nijsure <snijsure@grid-net.com>
      Reported-by: Paul Walmsley <paul@pwsan.com>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Paul Walmsley <paul@pwsan.com>  # OMAP3730, OMAP4430
      Cc: stable@vger.kernel.org
      Signed-off-by: Jin Hong <jinh@codeaurora.org>

rcu: Stop rcu_do_batch() from multiplexing the "count" variable

      Commit b1420f1c (Make rcu_barrier() less disruptive) rearranged the
      code in rcu_do_batch(), moving the ->qlen manipulation to follow
      the requeueing of the callbacks.  Unfortunately, this rearrangement
      clobbered the value of the "count" local variable before the value
      of rdp->qlen was adjusted, resulting in the value of rdp->qlen being
      inaccurate.  This commit therefore introduces an index variable "i",
      avoiding the inadvertent multiplexing.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Fix detection of abruptly-ending stall

      The code that attempts to identify stalls that end just as we detect
      them is broken by both flavors of initialization failure.  This commit
      therefore properly initializes and computes the count of the number
      of reasons why the RCU grace period is stalled.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Prevent offline CPUs from executing RCU core code

      Earlier versions of RCU invoked the RCU core from the CPU_DYING notifier
      in order to note a quiescent state for the outgoing CPU.  Because the
      CPU is marked "offline" during the execution of the CPU_DYING notifiers,
      the RCU core had to tolerate being invoked from an offline CPU.  However,
      commit b1420f1c (Make rcu_barrier() less disruptive) left only tracing
      code in the CPU_DYING notifier, so the RCU core need no longer execute
      on offline CPUs.  This commit therefore enforces this restriction.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Make rcu_barrier() less disruptive

      The rcu_barrier() primitive interrupts each and every CPU, registering
      a callback on every CPU.  Once all of these callbacks have been invoked,
      rcu_barrier() knows that every callback that was registered before
      the call to rcu_barrier() has also been invoked.

      However, there is no point in registering a callback on a CPU that
      currently has no callbacks, most especially if that CPU is in a
      deep idle state.  This commit therefore makes rcu_barrier() avoid
      interrupting CPUs that have no callbacks.  Doing this requires reworking
      the handling of orphaned callbacks, otherwise callbacks could slip through
      rcu_barrier()'s net by being orphaned from a CPU that rcu_barrier() had
      not yet interrupted to a CPU that rcu_barrier() had already interrupted.
      This reworking was needed anyway to take a first step towards weaning
      RCU from the CPU_DYING notifier's use of stop_cpu().

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Reduce cache-miss initialization latencies for large systems

      Commit #0209f649 (rcu: limit rcu_node leaf-level fanout) set an upper
      limit of 16 on the leaf-level fanout for the rcu_node tree.  This was
      needed to reduce lock contention that was induced by the synchronization
      of scheduling-clock interrupts, which was in turn needed to improve
      energy efficiency for moderate-sized lightly loaded servers.

      However, reducing the leaf-level fanout means that there are more
      leaf-level rcu_node structures in the tree, which in turn means that
      RCU's grace-period initialization incurs more cache misses.  This is
      not a problem on moderate-sized servers with only a few tens of CPUs,
      but becomes a major source of real-time latency spikes on systems with
      many hundreds of CPUs.  In addition, the workloads running on these large
      systems tend to be CPU-bound, which eliminates the energy-efficiency
      advantages of synchronizing scheduling-clock interrupts.  Therefore,
      these systems need maximal values for the rcu_node leaf-level fanout.

      This commit addresses this problem by introducing a new kernel parameter
      named RCU_FANOUT_LEAF that directly controls the leaf-level fanout.
      This parameter defaults to 16 to handle the common case of a moderate
      sized lightly loaded servers, but may be set higher on larger systems.

      Reported-by: Mike Galbraith <efault@gmx.de>
      Reported-by: Dimitri Sivanich <sivanich@sgi.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Document why rcu_blocking_is_gp() is safe

      The rcu_blocking_is_gp() function tests to see if there is only one
      online CPU, and if so, synchronize_sched() and friends become no-ops.
      However, for larger systems, num_online_cpus() scans a large vector,
      and might be preempted while doing so.  While preempted, any number
      of CPUs might come online and go offline, potentially resulting in
      num_online_cpus() returning 1 when there never had only been one
      CPU online.  This could result in a too-short RCU grace period, which
      could in turn result in total failure, except that the only way that
      the grace period is too short is if there is an RCU read-side critical
      section spanning it.  For RCU-sched and RCU-bh (which are the only
      cases using rcu_blocking_is_gp()), RCU read-side critical sections
      have either preemption or bh disabled, which prevents CPUs from going
      offline.  This in turn prevents actual failures from occurring.

      This commit therefore adds a large block comment to rcu_blocking_is_gp()
      documenting why it is safe.  This commit also moves rcu_blocking_is_gp()
      into kernel/rcutree.c, which should help prevent unwary developers from
      mistaking it for a generally useful function.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Add RCU_FAST_NO_HZ tracing for idle exit

      Traces of rcu_prep_idle events can be confusing because
      rcu_cleanup_after_idle() does no tracing.  This commit therefore adds
      this tracing.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Make RCU_FAST_NO_HZ use timer rather than hrtimer

      The RCU_FAST_NO_HZ facility uses an hrtimer to wake up a CPU when
      it is allowed to go into dyntick-idle mode, which is almost always
      cancelled soon after.  This is not what hrtimers are good at, so
      this commit switches to the timer wheel.

      Reported-by: Steven Rostedt <rostedt@goodmis.org>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Make RCU_FAST_NO_HZ account for pauses out of idle

      Both Steven Rostedt's new idle-capable trace macros and the RCU_NONIDLE()
      macro can cause RCU to momentarily pause out of idle without the rest
      of the system being involved.  This can cause rcu_prepare_for_idle()
      to run through its state machine too quickly, which can in turn result
      in needless scheduling-clock interrupts.

      This commit therefore adds code to enable rcu_prepare_for_idle() to
      distinguish between an initial entry to idle on the one hand (which needs
      to advance the rcu_prepare_for_idle() state machine) and an idle reentry
      due to idle-capable trace macros and RCU_NONIDLE() on the other hand
      (which should avoid advancing the rcu_prepare_for_idle() state machine).
      Additional state is maintained to allow the timer to be correctly reposted
      when returning after a momentary pause out of idle, and even more state
      is maintained to detect when new non-lazy callbacks have been enqueued
      (which may require re-evaluation of the approach to idleness).

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: List-debug variants of rcu list routines.

      * Make __list_add_rcu check the next->prev and prev->next pointers
	just like __list_add does.
      * Make list_del_rcu use __list_del_entry, which does the same checking
	at deletion time.

      Has been running for a week here without anything being tripped up,
      but it seems worth adding for completeness just in case something
      ever does corrupt those lists.

      Signed-off-by: Dave Jones <davej@redhat.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Replace list_first_entry_rcu() with list_first_or_null_rcu()

      The list_first_entry_rcu() macro is inherently unsafe because it cannot
      be applied to an empty list.  But because RCU readers do not exclude
      updaters, a list might become empty between the time that list_empty()
      claimed it was non-empty and the time that list_first_entry_rcu() is
      invoked.  Therefore, the list_empty() test cannot be separated from the
      list_first_entry_rcu() call.  This commit therefore combines these to
      macros to create a new list_first_or_null_rcu() macro that replaces
      the old (and unsafe) list_first_entry_rcu() macro.

      This patch incorporates Paul's review comments on the previous version of
      this patch available here:

      https://lkml.org/lkml/2012/4/2/536

      This patch cannot break any upstream code because list_first_entry_rcu()
      is not being used anywhere in the kernel (tested with grep(1)), and any
      external code using it is probably broken as a result of using it.

      Signed-off-by: Michel Machado <michel@digirati.com.br>
      CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      CC: Dipankar Sarma <dipankar@in.ibm.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Clarify help text for RCU_BOOST_PRIO

      The old text confused real-time applications with real-time threads, so
      that you pretty much needed to understand how this kernel configuration
      parameter worked to understand the help text.  This commit therefore
      attempts to make the help text human-readable.

      Reported-by: Jörn Engel <joern@purestorage.com>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Make __kfree_rcu() less dependent on compiler choices

      Currently, __kfree_rcu() is implemented as an inline function, and
      contains a BUILD_BUG_ON() that malfunctions if __kfree_rcu() is compiled
      as an out-of-line function.  Unfortunately, there are compiler settings
      (e.g., -O0) that can result in __kfree_rcu() being compiled out of line,
      resulting in annoying build breakage.  This commit therefore converts
      both __kfree_rcu() and __is_kfree_rcu_offset() from inline functions to
      macros to prevent such misbehavior on the part of the compiler.

      Signed-off-by: Jan Engelhardt <jengelh@medozas.de>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Reviewed-by: Josh Triplett <josh@joshtriplett.org>

rcu: Document kernel command-line parameters

      Bring RCU's kernel command-line parameter documentation up to date.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Fixes to rcutorture error handling and cleanup

      The rcutorture initialization code ignored the error returns from
      rcu_torture_onoff_init() and rcu_torture_stall_init().  The rcutorture
      cleanup code failed to NULL out a number of pointers.  These bugs will
      normally have no effect, but this commit fixes them nevertheless.

      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Introduce rcutorture testing for rcu_barrier()

      Although rcutorture does invoke rcu_barrier() and friends, it cannot
      really be called a torture test given that it invokes them only once
      at the end of the test.  This commit therefore introduces heavy-duty
      rcutorture testing for rcu_barrier(), which may be carried out
      concurrently with normal rcutorture testing.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Direct algorithmic SRCU implementation

      The current implementation of synchronize_srcu_expedited() can cause
      severe OS jitter due to its use of synchronize_sched(), which in turn
      invokes try_stop_cpus(), which causes each CPU to be sent an IPI.
      This can result in severe performance degradation for real-time workloads
      and especially for short-interation-length HPC workloads.  Furthermore,
      because only one instance of try_stop_cpus() can be making forward progress
      at a given time, only one instance of synchronize_srcu_expedited() can
      make forward progress at a time, even if they are all operating on
      distinct srcu_struct structures.

      This commit, inspired by an earlier implementation by Peter Zijlstra
      (https://lkml.org/lkml/2012/1/31/211) and by further offline discussions,
      takes a strictly algorithmic bits-in-memory approach.  This has the
      disadvantage of requiring one explicit memory-barrier instruction in
      each of srcu_read_lock() and srcu_read_unlock(), but on the other hand
      completely dispenses with OS jitter and furthermore allows SRCU to be
      used freely by CPUs that RCU believes to be idle or offline.

      The update-side implementation handles the single read-side memory
      barrier by rechecking the per-CPU counters after summing them and
      by running through the update-side state machine twice.

      This implementation has passed moderate rcutorture testing on both
      x86 and Power.  Also updated to use this_cpu_ptr() instead of per_cpu_ptr(),
      as suggested by Peter Zijlstra.

      Reported-by: Peter Zijlstra <peterz@infradead.org>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>

rcu: Remove fast check path from __synchronize_srcu()

      The fastpath in __synchronize_srcu() is designed to handle cases where
      there are a large number of concurrent calls for the same srcu_struct
      structure.  However, the Linux kernel currently does not use SRCU in
      this manner, so remove the fastpath checks for simplicity.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Increment upper bit only for srcu_read_lock()

      The purpose of the upper bit of SRCU's per-CPU counters is to guarantee
      that no reasonable series of srcu_read_lock() and srcu_read_unlock()
      operations can return the value of the counter to its original value.
      This guarantee is require only after the index has been switched to
      the other set of counters, so at most one srcu_read_lock() can affect
      a given CPU's counter.  The number of srcu_read_unlock() operations
      on a given counter is limited to the number of tasks in the system,
      which given the Linux kernel's current structure is limited to far less
      than 2^30 on 32-bit systems and far less than 2^62 on 64-bit systems.
      (Something about a limited number of bytes in the kernel's address space.)

      Therefore, if srcu_read_lock() increments the upper bits, then
      srcu_read_unlock() need not do so.  In this case, an srcu_read_lock() and
      an srcu_read_unlock() will flip the lower bit of the upper field of the
      counter.  An unreasonably large additional number of srcu_read_unlock()
      operations would be required to return the counter to its initial value,
      thus preserving the guarantee.

      This commit takes this approach, which further allows it to shrink
      the size of the upper field to one bit, making the number of
      srcu_read_unlock() operations required to return the counter to its
      initial value even more unreasonable than before.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Flip ->completed only once per SRCU grace period

      This is an optimization of the SRCU grace period.  To guard against
      preempted readers with old values of the counter, it suffices to scan the
      old counters once more, then flip ->completed only one time.  The reason
      this works is that the old readers must have incremented the old set of
      counters (if they have not yet incremented, then their critical section
      starts after this grace period, so they may be safely ignored).

      This commit therefore optimizes the second flip out in favor of a simple
      rescan.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Improve SRCU's wait_idx() comments

      The safety of SRCU is provided byy wait_idx() rather than flipping.
      The flipping actually prevents starvation.

      This commit therefore updates the comments to more accurately and
      precisely describe what is going on.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

      # This is the 25th commit message:

      rcu: Implement a variant of Peter's SRCU algorithm

      This commit implements a variant of Peter's algorithm, which may be found
      at https://lkml.org/lkml/2012/2/1/119.

      o	Make the checking lock-free to enable parallel checking.
	      Parallel checking is required when (1) the original checking
	      task is preempted for a long time, (2) sychronize_srcu_expedited()
	      starts during an ongoing SRCU grace period, or (3) we wish to
	      avoid acquiring a lock.

      o	Since the checking is lock-free, we avoid a mutex in state machine
	      for call_srcu().

      o	Remove the SRCU_REF_MASK and remove the coupling with the flipping.
	      This might allow us to remove the preempt_disable() in future
	      versions, though such removal will need great care because it
	      rescinds the one-old-reader-per-CPU guarantee.

      o	Remove a smp_mb(), simplify the comments and make the smp_mb() pairs
	      more intuitive.

      Inspired-by: Peter Zijlstra <peterz@infradead.org>
      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Remove unused srcu_barrier()

      The old srcu_barrier() macro is now unused.  This commit removes it so
      that it may be used for the SRCU flavor of rcu_barrier(), which will in
      turn be needed to allow the upcoming call_srcu() to be used from within
      modules.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Improve srcu_readers_active_idx()'s cache locality

      Expand the calls to srcu_readers_active_idx() from srcu_readers_active()
      inline.  This change improves cache locality by interating over the CPUs
      once rather than twice.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Use single value to handle expedited SRCU grace periods

      The earlier algorithm used an "expedited" flag combined with a "trycount"
      counter to differentiate between normal and expedited SRCU grace periods.
      However, the difference can be encoded into a single counter with a cutoff
      value and different initial values for expedited and normal SRCU grace
      periods.  This commit makes that change.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Implement per-domain single-threaded call_srcu() state machine

      This commit implements an SRCU state machine in support of call_srcu().
      The state machine is preemptible, light-weight, and single-threaded,
      minimizing synchronization overhead.  In particular, there is no longer
      any need for synchronize_srcu() to be guarded by a mutex.

      Expedited processing is handled, at least in the absence of concurrent
      grace-period operations on that same srcu_struct structure, by having
      the synchronize_srcu_expedited() thread take on the role of the
      workqueue thread for one iteration.

      There is a reasonable probability that a given SRCU callback will
      be invoked on the same CPU that registered it, however, there is no
      guarantee.  Concurrent SRCU grace-period primitives can cause callbacks
      to be executed elsewhere, even in absence of CPU-hotplug operations.

      Callbacks execute in process context, but under the influence of
      local_bh_disable(), so it is illegal to sleep in an SRCU callback
      function.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Add rcutorture test for call_srcu()

      Add srcu_torture_deferred_free() for srcu_ops so as to test the new
      call_srcu().  Rename the original srcu_ops to srcu_sync_ops.

      Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Move PREEMPT_RCU preemption to switch_to() invocation

      Currently, PREEMPT_RCU readers are enqueued upon entry to the scheduler.
      This is inefficient because enqueuing is required only if there is a
      context switch, and entry to the scheduler does not guarantee a context
      switch.

      The commit therefore moves the enqueuing to immediately precede the
      call to switch_to() from the scheduler.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Linus Torvalds <torvalds@linux-foundation.org>

rcu: Make exit_rcu() more precise and consolidate

      When running preemptible RCU, if a task exits in an RCU read-side
      critical section having blocked within that same RCU read-side critical
      section, the task must be removed from the list of tasks blocking a
      grace period (perhaps the current grace period, perhaps the next grace
      period, depending on timing).  The exit() path invokes exit_rcu() to
      do this cleanup.

      However, the current implementation of exit_rcu() needlessly does the
      cleanup even if the task did not block within the current RCU read-side
      critical section, which wastes time and needlessly increases the size
      of the state space.  Fix this by only doing the cleanup if the current
      task is actually on the list of tasks blocking some grace period.

      While we are at it, consolidate the two identical exit_rcu() functions
      into a single function.

      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Linus Torvalds <torvalds@linux-foundation.org>

rcu: Update RCU maintainership

      Split SRCU out and add Lai Jiangshan as SRCU co-maintainer.

      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

rcu: Precompute RCU_FAST_NO_HZ timer offsets

      When a CPU is entering dyntick-idle mode, tick_nohz_stop_sched_tick()
      calls rcu_needs_cpu() see if RCU needs that CPU, and, if not, computes the
      next wakeup time based on the timer wheels.  Only later, when actually
      entering the idle loop, rcu_prepare_for_idle() will be invoked.  In some
      cases, rcu_prepare_for_idle() will post timers to wake the CPU back up.
      But all for naught: The next wakeup time for the CPU has already been
      computed, and posting a timer afterwards does not force that wakeup
      time to be recomputed.  This means that rcu_prepare_for_idle()'s have
      no effect.

      This is not a problem on a busy system because something else will wake
      up the CPU soon enough.  However, on lightly loaded systems, the CPU
      might stay asleep for a considerable length of time.  If that CPU has
      a callback that the rest of the system is waiting on, the system might
      run very slowly or (in theory) even hang.

      This commit avoids this problem by having rcu_needs_cpu() give
      tick_nohz_stop_sched_tick() an estimate of when RCU will need the CPU
      to wake back up, which tick_nohz_stop_sched_tick() takes into account
      when programming the CPU's wakeup time.  An alternative approach is
      for rcu_prepare_for_idle() to use hrtimers instead of normal timers,
      but timers are much more efficient than are hrtimers for frequently
      and repeatedly posting and cancelling a given timer, which is exactly
      what RCU_FAST_NO_HZ does.

      Reported-by: Pascal Chapperon <pascal.chapperon@wanadoo.fr>
      Reported-by: Heiko Carstens <heiko.carstens@de.ibm.com>
      Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
      Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Tested-by: Heiko Carstens <heiko.carstens@de.ibm.com>
      Tested-by: Pascal Chapperon <pascal.chapperon@wanadoo.fr>

irq_work: Don't stop the tick with pending works

      Don't stop the tick if we have pending irq works on the
      queue, otherwise if the arch can't raise self-IPIs, we may not
      find an opportunity to execute the pending works for a while.

      Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
      Acked-by: Steven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>

Signed-off-by: moddingg33k <moddingg33k@t-online.de>
tick: Consolidate tick handling for high and low res handlers

Besides unifying code, this also adds the idle check before
processing idle accounting specifics on the low res handler.
This way we also generalize this part of the nohz code for
!CONFIG_HIGH_RES_TIMERS to prepare for the adaptive tickless
features.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
tick: Consolidate timekeeping handling code

Unify the duplicated timekeeping handling code of low and high res tick
sched handlers.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
time: RCU permitted to stop idle entry via softirq

The can_stop_idle_tick() function complains if a softirq vector is
raised too late in the idle-entry process, presumably in order to
prevent dangling softirq invocations from being delayed across the
full idle period, which might be indefinitely long -- and if softirq
was asserted any later than the call to this function, such a delay
might well happen.

However, RCU needs to be able to use softirq to stop idle entry in
order to be able to drain RCU callbacks from the current CPU, which in
turn enables faster entry into dyntick-idle mode, which in turn reduces
power consumption.  Because RCU takes this action at a well-defined
point in the idle-entry path, it is safe for RCU to take this approach.

This commit therefore silences the error message that is sometimes
produced when the going-idle CPU suddenly finds that it has an RCU_SOFTIRQ
to process.  The error message will continue to be issued for other
softirq vectors.

Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
sched: remove redundant update_runtime notifier

migration_call() will do all the things that update_runtime() does.
So it seems update_runtime() is a redundant notifier, remove it.

Furthermore, there is potential risk that the current code will catch
BUG_ON at line 687 of rt.c when do cpu hotplug while there are realtime
threads running because of enable runtime twice.

Signed-off-by: Neil Zhang <zhangwm@marvell.com>
Reviewed-on: http://git-master/r/215596
Reviewed-on: http://git-master/r/223067
Reviewed-by: Peter Boonstoppel <pboonstoppel@nvidia.com>
Tested-by: Peter Boonstoppel <pboonstoppel@nvidia.com>
Reviewed-by: Paul Walmsley <pwalmsley@nvidia.com>
Reviewed-by: Automatic_Commit_Validation_User
Reviewed-by: Diwakar Tundlam <dtundlam@nvidia.com>
sched: cpu_power: enable ARCH_POWER

Heteregeneous ARM platform uses arch_scale_freq_power function
to reflect the relative capacity of each core

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1341826026-6504-6-git-send-email-vincent.guittot@linaro.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched: Fix nohz_idle_balance()

On tickless systems, one CPU runs load balance for all idle CPUs.

The cpu_load of this CPU is updated before starting the load balance
of each other idle CPUs. We should instead update the cpu_load of
the balance_cpu.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Venkatesh Pallipadi <venki@google.com>
Cc: Suresh Siddha <suresh.b.siddha@intel.com>
Link: http://lkml.kernel.org/r/1347509486-8688-1-git-send-email-vincent.guittot@linaro.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
sched/nohz: Fix rq->cpu_load[] calculations

While investigating why the load-balancer did funny I found that the
rq->cpu_load[] tables were completely screwy.. a bit more digging
revealed that the updates that got through were missing ticks followed
by a catchup of 2 ticks.

The catchup assumes the cpu was idle during that time (since only nohz
can cause missed ticks and the machine is idle etc..) this means that
esp. the higher indices were significantly lower than they ought to
be.

The reason for this is that its not correct to compare against jiffies
on every jiffy on any other cpu than the cpu that updates jiffies.

This patch cludges around it by only doing the catch-up stuff from
nohz_idle_balance() and doing the regular stuff unconditionally from
the tick.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: pjt@google.com
Cc: Venkatesh Pallipadi <venki@google.com>
Link: http://lkml.kernel.org/n/tip-tp4kj18xdd5aj4vvj0qg55s2@git.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
writeback: Do not sort b_io list only because of block device inode

It is very likely that block device inode will be part of BDI dirty list
as well. However it doesn't make sence to sort inodes on the b_io list
just because of this inode (as it contains buffers all over the device
anyway). So save some CPU cycles which is valuable since we hold relatively
contented wb->list_lock.

Signed-off-by: Jan Kara <jack@suse.cz>
msm: kgsl: Remove extra interrupts when setting MMU state

The interrupts added to the ringbuffer on PTFLUSH and TLBUPDATE
were causing a major increase in the number of interrupts from the GPU.
This was leading to increase in power and loss of performance. Add a check
to turn off IOMMU clocks when going to SLEEP.

Change-Id: I41617dd3b7b3f7d9622523f2a1407b912dbd989e
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>