binfmt_elf.c: use get_random_int() to fix entropy depleting

Changes:
--------
v4->v3:
- s/random_stack_user()/get_atrandom_bytes()/
- Move this function to ahead of its use to avoid the predeclaration.

v3->v2:
- Tweak code comments of random_stack_user().
- Remove redundant bits mask and shift upon the random variable.

v2->v1:
- Fix random copy to check up buffer length that are not 4-byte multiples.

v3 can be found at:
http://www.spinics.net/lists/linux-fsdevel/msg59597.html
v2 can be found at:
http://www.spinics.net/lists/linux-fsdevel/msg59418.html
v1 can be found at:
http://www.spinics.net/lists/linux-fsdevel/msg59128.html

Thanks,
-Jeff

Entropy is quickly depleted under normal operations like ls(1), cat(1),
etc...  between 2.6.30 to current mainline, for instance:

$ cat /proc/sys/kernel/random/entropy_avail
3428
$ cat /proc/sys/kernel/random/entropy_avail
2911
$cat /proc/sys/kernel/random/entropy_avail
2620

We observed this problem has been occurring since 2.6.30 with
fs/binfmt_elf.c: create_elf_tables()->get_random_bytes(), introduced by
f06295b44c296c8f ("ELF: implement AT_RANDOM for glibc PRNG seeding").

/*
 * Generate 16 random bytes for userspace PRNG seeding.
 */
get_random_bytes(k_rand_bytes, sizeof(k_rand_bytes));

The patch introduces a wrapper around get_random_int() which has lower
overhead than calling get_random_bytes() directly.

With this patch applied:
$ cat /proc/sys/kernel/random/entropy_avail
2731
$ cat /proc/sys/kernel/random/entropy_avail
2802
$ cat /proc/sys/kernel/random/entropy_avail
2878

Analyzed by John Sobecki.

Signed-off-by: Jie Liu <jeff.liu@oracle.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Andreas Dilger <aedilger@gmail.com>
Cc: Alan Cox <alan@linux.intel.com>
Cc: Arnd Bergmann <arnn@arndb.de>
Cc: John Sobecki <john.sobecki@oracle.com>
Cc: James Morris <james.l.morris@oracle.com>
Cc: Jakub Jelinek <jakub@redhat.com>
Cc: Ted Ts'o <tytso@mit.edu>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Ulrich Drepper <drepper@redhat.com>
arm/crypto: Add optimized AES and SHA1 routines

Add assembler versions of AES and SHA1 for ARM platforms.  This has provided
up to a 50% improvement in IPsec/TCP throughout for tunnels using AES128/SHA1.

Platform   CPU SPeed    Endian   Before (bps)   After (bps)   Improvement

IXP425      533 MHz      big     11217042        15566294        ~38%
KS8695      166 MHz     little    3828549         5795373        ~51%

Signed-off-by: David McCullough <ucdevel@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Optimized ARM RWSEM algorithm

RWSEM implementation for ARM using atomic functions.
Heavily based on arch/sh/include/asm/rwsem.h

Signed-off-by: Ashwin Chaugule <ashwinc@codeaurora.org>
lib: Update RWSEM to Linux 3.10

Squashed commit of the following:

commit 28f9a942e4e1578c0e26e8211ba49bdf520de95c
Author: Davidlohr Bueso <davidlohr.bueso@hp.com>
Date:   Tue May 7 06:46:02 2013 -0700

    rwsem: no need for explicit signed longs

    Change explicit "signed long" declarations into plain "long" as suggested
    by Peter Hurley.

    Signed-off-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Reviewed-by: Michel Lespinasse <walken@google.com>
    Signed-off-by: Michel Lespinasse <walken@google.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 353ad88b6af95bc7bb127b3b5b1cfa36c72a7b67
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:46:00 2013 -0700

    rwsem: do not block readers at head of queue if other readers are active

    This change fixes a race condition where a reader might determine it
    needs to block, but by the time it acquires the wait_lock the rwsem has
    active readers and no queued waiters.

    In this situation the reader can run in parallel with the existing
    active readers; it does not need to block until the active readers
    complete.

    Thanks to Peter Hurley for noticing this possible race.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit e9149aa93e4e6919da4fe9e77a27f9c6a0159463
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:59 2013 -0700

    rwsem: implement support for write lock stealing on the fastpath

    When we decide to wake up readers, we must first grant them as many read
    locks as necessary, and then actually wake up all these readers.  But in
    order to know how many read shares to grant, we must first count the
    readers at the head of the queue.  This might take a while if there are
    many readers, and we want to be protected against a writer stealing the
    lock while we're counting.  To that end, we grant the first reader lock
    before counting how many more readers are queued.

    We also require some adjustments to the wake_type semantics.

    RWSEM_WAKE_NO_ACTIVE used to mean that we had found the count to be
    RWSEM_WAITING_BIAS, in which case the rwsem was known to be free as
    nobody could steal it while we hold the wait_lock.  This doesn't make
    sense once we implement fastpath write lock stealing, so we now use
    RWSEM_WAKE_ANY in that case.

    Similarly, when rwsem_down_write_failed found that a read lock was
    active, it would use RWSEM_WAKE_READ_OWNED which signalled that new
    readers could be woken without checking first that the rwsem was
    available.  We can't do that anymore since the existing readers might
    release their read locks, and a writer could steal the lock before we
    wake up additional readers.  So, we have to use a new RWSEM_WAKE_READERS
    value to indicate we only want to wake readers, but we don't currently
    hold any read lock.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 20addabcb540eb4b4e11d7e8af037543767e081d
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:58 2013 -0700

    rwsem: simplify __rwsem_do_wake

    This is mostly for cleanup value:

    - We don't need several gotos to handle the case where the first
      waiter is a writer. Two simple tests will do (and generate very
      similar code).

    - In the remainder of the function, we know the first waiter is a reader,
      so we don't have to double check that. We can use do..while loops
      to iterate over the readers to wake (generates slightly better code).

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit e56c8eea2628841a06a2420e2c6ca8b8c6bd4434
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:57 2013 -0700

    rwsem: skip initial trylock in rwsem_down_write_failed

    We can skip the initial trylock in rwsem_down_write_failed() if there
    are known active lockers already, thus saving one likely-to-fail
    cmpxchg.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 286040cc550df96bef6e97a5bc0231ded9a0187b
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:56 2013 -0700

    rwsem: avoid taking wait_lock in rwsem_down_write_failed

    In rwsem_down_write_failed(), if there are active locks after we wake up
    (i.e.  the lock got stolen from us), skip taking the wait_lock and go
    back to sleep immediately.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 6c7229b31a4c099157242c58f5d33ce728daff4b
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:55 2013 -0700

    rwsem: use cmpxchg for trying to steal write lock

    Using rwsem_atomic_update to try stealing the write lock forced us to
    undo the adjustment in the failure path.  We can have simpler and faster
    code by using cmpxchg instead.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 85dcaeae08b5738f8ea0b08734ab97e57bb9e8e2
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:54 2013 -0700

    rwsem: more agressive lock stealing in rwsem_down_write_failed

    Some small code simplifications can be achieved by doing more agressive
    lock stealing:

    - When rwsem_down_write_failed() notices that there are no active locks
      (and thus no thread to wake us if we decided to sleep), it used to wake
      the first queued process. However, stealing the lock is also sufficient
      to deal with this case, so we don't need this check anymore.

    - In try_get_writer_sem(), we can steal the lock even when the first waiter
      is a reader. This is correct because the code path that wakes readers is
      protected by the wait_lock. As to the performance effects of this change,
      they are expected to be minimal: readers are still granted the lock
      (rather than having to acquire it themselves) when they reach the front
      of the wait queue, so we have essentially the same behavior as in
      rwsem-spinlock.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 082a2e9c40e01d70c59ca4e3a9b552b81d573332
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:53 2013 -0700

    rwsem: simplify rwsem_down_write_failed

    When waking writers, we never grant them the lock - instead, they have
    to acquire it themselves when they run, and remove themselves from the
    wait_list when they succeed.

    As a result, we can do a few simplifications in rwsem_down_write_failed():

    - We don't need to check for !waiter.task since __rwsem_do_wake() doesn't
      remove writers from the wait_list

    - There is no point releaseing the wait_lock before entering the wait loop,
      as we will need to reacquire it immediately. We can change the loop so
      that the lock is always held at the start of each loop iteration.

    - We don't need to get a reference on the task structure, since the task
      is responsible for removing itself from the wait_list. There is no risk,
      like in the rwsem_down_read_failed() case, that a task would wake up and
      exit (thus destroying its task structure) while __rwsem_do_wake() is
      still running - wait_lock protects against that.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 6b3306dccd59d5dcf22e55a27827bccf51a796e1
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:52 2013 -0700

    rwsem: simplify rwsem_down_read_failed

    When trying to acquire a read lock, the RWSEM_ACTIVE_READ_BIAS
    adjustment doesn't cause other readers to block, so we never have to
    worry about waking them back after canceling this adjustment in
    rwsem_down_read_failed().

    We also never want to steal the lock in rwsem_down_read_failed(), so we
    don't have to grab the wait_lock either.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit c6b315c32d1ffbb33c75ab5e0f8ee27c89d206c4
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:51 2013 -0700

    rwsem: move rwsem_down_failed_common code into rwsem_down_{read,write}_failed

    Remove the rwsem_down_failed_common function and replace it with two
    identical copies of its code in rwsem_down_{read,write}_failed.

    This is because we want to make different optimizations in
    rwsem_down_{read,write}_failed; we are adding this pure-duplication
    step as a separate commit in order to make it easier to check the
    following steps.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 7d2869248ccd7971fc1b79e4bf22cb4be446faf4
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:50 2013 -0700

    rwsem: shorter spinlocked section in rwsem_down_failed_common()

    This change reduces the size of the spinlocked and TASK_UNINTERRUPTIBLE
    sections in rwsem_down_failed_common():

    - We only need the sem->wait_lock to insert ourselves on the wait_list;
      the waiter node can be prepared outside of the wait_lock.

    - The task state only needs to be set to TASK_UNINTERRUPTIBLE immediately
      before checking if we actually need to sleep; it doesn't need to protect
      the entire function.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 8c163f8ddf5c0bd018706dbdd44fb96a073401d0
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:49 2013 -0700

    rwsem: make the waiter type an enumeration rather than a bitmask

    We are not planning to add some new waiter flags, so we can convert the
    waiter type into an enumeration.

    Background: David Howells suggested I do this back when I tried adding
    a new waiter type for unfair readers. However, I believe the cleanup
    applies regardless of that use case.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 9a126edbed8ccfedda04d5c3fe256cd139d4bcca
Author: Alex Shi <alex.shi@intel.com>
Date:   Tue Feb 5 21:11:55 2013 +0800

    rwsem: Implement writer lock-stealing for better scalability

    Commit 5a505085f043 ("mm/rmap: Convert the struct anon_vma::mutex
    to an rwsem") changed struct anon_vma::mutex to an rwsem, which
    caused aim7 fork_test performance to drop by 50%.

    Yuanhan Liu did the following excellent analysis:

        https://lkml.org/lkml/2013/1/29/84

    and found that the regression is caused by strict, serialized,
    FIFO sequential write-ownership of rwsems. Ingo suggested
    implementing opportunistic lock-stealing for the front writer
    task in the waitqueue.

    Yuanhan Liu implemented lock-stealing for spinlock-rwsems,
    which indeed recovered much of the regression - confirming
    the analysis that the main factor in the regression was the
    FIFO writer-fairness of rwsems.

    In this patch we allow lock-stealing to happen when the first
    waiter is also writer. With that change in place the
    aim7 fork_test performance is fully recovered on my
    Intel NHM EP, NHM EX, SNB EP 2S and 4S test-machines.

    Reported-by: lkp@linux.intel.com
    Reported-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
    Signed-off-by: Alex Shi <alex.shi@intel.com>
    Cc: David Howells <dhowells@redhat.com>
    Cc: Michel Lespinasse <walken@google.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Anton Blanchard <anton@samba.org>
    Cc: Arjan van de Ven <arjan@linux.intel.com>
    Cc: paul.gortmaker@windriver.com
    Link: https://lkml.org/lkml/2013/1/29/84
    Link: http://lkml.kernel.org/r/1360069915-31619-1-git-send-email-alex.shi@intel.com
    [ Small stylistic fixes, updated changelog. ]
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

commit 77302ff69a93cfba32aaee98d2bb864ba8886e40
Author: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Date:   Fri Feb 1 18:59:16 2013 +0800

    rwsem-spinlock: Implement writer lock-stealing for better scalability

    We (Linux Kernel Performance project) found a regression
    introduced by commit:

      5a505085f043 mm/rmap: Convert the struct anon_vma::mutex to an rwsem

    which converted all anon_vma::mutex locks rwsem write locks.

    The semantics are the same, but the behavioral difference is
    quite huge in some cases. After investigating it we found the
    root cause: mutexes support lock stealing while rwsems don't.

    Here is the link for the detailed regression report:

      https://lkml.org/lkml/2013/1/29/84

    Ingo suggested adding write lock stealing to rwsems:

        "I think we should allow lock-steal between rwsem writers - that
         will not hurt fairness as most rwsem fairness concerns relate to
         reader vs. writer fairness"

    And here is the rwsem-spinlock version.

    With this patch, we got a double performance increase in one
    test box with following aim7 workfile:

        FILESIZE: 1M
        POOLSIZE: 10M
        10 fork_test

     /usr/bin/time output w/o patch                       /usr/bin/time_output with patch
     -- Percent of CPU this job got: 369%                 Percent of CPU this job got: 537%
     Voluntary context switches: 640595016                Voluntary context switches: 157915561

    We got a 45% increase in CPU usage and saved about 3/4 voluntary context switches.

    Reported-by: LKP project <lkp@linux.intel.com>
    Suggested-by: Ingo Molnar <mingo@kernel.org>
    Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
    Cc: Alex Shi <alex.shi@intel.com>
    Cc: David Howells <dhowells@redhat.com>
    Cc: Michel Lespinasse <walken@google.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Anton Blanchard <anton@samba.org>
    Cc: Arjan van de Ven <arjan@linux.intel.com>
    Cc: paul.gortmaker@windriver.com
    Link: http://lkml.kernel.org/r/1359716356-23865-1-git-send-email-yuanhan.liu@linux.intel.com
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

commit 95f8b6683b3f83e40a8ab0d93a9d155954b64396
Author: Andrei F <luxneb@gmail.com>
Date:   Wed Aug 21 21:49:39 2013 +0400

    ARM: rwsem: Update header from initial implementation

    Cleanup due to:

    c16a87ce063f79e0ec7d25ce2950e1bc6db03c72 rwsem: Cleanup includes
    1c8ed640d918290ddc1de5ada02ef6686a733c9f rwsem: Move duplicate struct rwsem declaration to linux/rwsem.h
    12249b34414dba7f386aadcf6be7ca36c6878300 rwsem: Move duplicate init macros and functions to linux/rwsem.h
    41e5887fa39ab272d9266a09cbefdef270e28b93 rwsem: Unify the duplicate rwsem_is_locked() inlines
    aac72277fda6ef788bb8d5deaa502ce9b9b6e472 rwsem: Move duplicate function prototypes to linux/rwsem.h
softirq: reduce latencies

Date	Thu, 03 Jan 2013 23:49:40 -0800

In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.

This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.

This patch changes the fallback to ksoftirqd condition to :

- A time limit of 2 ms.
- need_resched() being set on current task

When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.

Using need_resched() as the only condition can trigger RCU stalls,
as we can keep BH disabled for too long.

I ran several benchmarks and got no significant difference in
throughput, but a very significant reduction of latencies (one order
of magnitude) :

In following bench, 200 antagonist "netperf -t TCP_RR" are started in
background, using all available cpus.

Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
IRQ (hard+soft)

Before patch :

RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=550110.424
MIN_LATENCY=146858
MAX_LATENCY=997109
P50_LATENCY=305000
P90_LATENCY=550000
P99_LATENCY=710000
MEAN_LATENCY=376989.12
STDDEV_LATENCY=184046.92
After patch :

RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=40545.492
MIN_LATENCY=9834
MAX_LATENCY=78366
P50_LATENCY=33583
P90_LATENCY=59000
P99_LATENCY=69000
MEAN_LATENCY=38364.67
STDDEV_LATENCY=12865.26

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: David Miller <davem@davemloft.net>
Cc: Tom Herbert <therbert@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
jiffies conversions: Use compile time constants when possible

Date	Fri, 04 Jan 2013 13:15:43 -0800

Do the multiplications and divisions at compile time
instead of runtime when the converted value is a constant.

Make the calculation functions static __always_inline to jiffies.h.

Add #defines with __builtin_constant_p to test and use the
static inline or the runtime functions as appropriate.

Prefix the old exported symbols/functions with __

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: Fix clear NOHZ_BALANCE_KICK

I have faced a sequence where the Idle Load Balance was sometime not
triggered for a while on my platform, in the following scenario:

 CPU 0 and CPU 1 are running tasks and CPU 2 is idle

 CPU 1 kicks the Idle Load Balance
 CPU 1 selects CPU 2 as the new Idle Load Balancer
 CPU 2 sets NOHZ_BALANCE_KICK for CPU 2
 CPU 2 sends a reschedule IPI to CPU 2

 While CPU 3 wakes up, CPU 0 or CPU 1 migrates a waking up task A on CPU 2

 CPU 2 finally wakes up, runs task A and discards the Idle Load Balance
       task A quickly goes back to sleep (before a tick occurs on CPU 2)
 CPU 2 goes back to idle with NOHZ_BALANCE_KICK set

Whenever CPU 2 will be selected as the ILB, no reschedule IPI will be sent
because NOHZ_BALANCE_KICK is already set and no Idle Load Balance will be
performed.

We must wait for the sched softirq to be raised on CPU 2 thanks to another
part the kernel to come back to clear NOHZ_BALANCE_KICK.

The proposed solution clears NOHZ_BALANCE_KICK in schedule_ipi if
we can't raise the sched_softirq for the Idle Load Balance.

Change since V1:

- move the clear of NOHZ_BALANCE_KICK in got_nohz_idle_kick if the ILB
  can't run on this CPU (as suggested by Peter)

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1370419991-13870-1-git-send-email-vincent.guittot@linaro.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Git-commit: 873b4c6
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[smuckle@codeaurora.org: minor merge resolution for 3.4 in scheduler_ipi()]
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Change-Id: I3548612057cccc2ecc29429c129c44183083831f

Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
writeback: fix race that cause writeback hung

There is a race between mark inode dirty and writeback thread, see the
following scenario.  In this case, writeback thread will not run though
there is dirty_io.

__mark_inode_dirty()                                          bdi_writeback_workfn()
	...                                                       	...
	spin_lock(&inode->i_lock);
	...
	if (bdi_cap_writeback_dirty(bdi)) {
	    <<< assume wb has dirty_io, so wakeup_bdi is false.
	    <<< the following inode_dirty also have wakeup_bdi false.
	    if (!wb_has_dirty_io(&bdi->wb))
		    wakeup_bdi = true;
	}
	spin_unlock(&inode->i_lock);
	                                                            <<< assume last dirty_io is removed here.
	                                                            pages_written = wb_do_writeback(wb);
	                                                            ...
	                                                            <<< work_list empty and wb has no dirty_io,
	                                                            <<< delayed_work will not be queued.
	                                                            if (!list_empty(&bdi->work_list) ||
	                                                                (wb_has_dirty_io(wb) && dirty_writeback_interval))
	                                                                queue_delayed_work(bdi_wq, &wb->dwork,
	                                                                    msecs_to_jiffies(dirty_writeback_interval * 10));
	spin_lock(&bdi->wb.list_lock);
	inode->dirtied_when = jiffies;
	<<< new dirty_io is added.
	list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
	spin_unlock(&bdi->wb.list_lock);

	<<< though there is dirty_io, but wakeup_bdi is false,
	<<< so writeback thread will not be waked up and
	<<< the new dirty_io will not be flushed.
	if (wakeup_bdi)
	    bdi_wakeup_thread_delayed(bdi);

Writeback will run until there is a new flush work queued.  This may cause
a lot of dirty pages stay in memory for a long time.

Signed-off-by: Junxiao Bi <junxiao.bi@oracle.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
sched: Implement smarter wake-affine logic

The wake-affine scheduler feature is currently always trying to pull
the wakee close to the waker. In theory this should be beneficial if
the waker's CPU caches hot data for the wakee, and it's also beneficial
in the extreme ping-pong high context switch rate case.

Testing shows it can benefit hackbench up to 15%.

However, the feature is somewhat blind, from which some workloads
such as pgbench suffer. It's also time-consuming algorithmically.

Testing shows it can damage pgbench up to 50% - far more than the
benefit it brings in the best case.

So wake-affine should be smarter and it should realize when to
stop its thankless effort at trying to find a suitable CPU to wake on.

This patch introduces 'wakee_flips', which will be increased each
time the task flips (switches) its wakee target.

So a high 'wakee_flips' value means the task has more than one
wakee, and the bigger the number, the higher the wakeup frequency.

Now when making the decision on whether to pull or not, pay attention to
the wakee with a high 'wakee_flips', pulling such a task may benefit
the wakee. Also imply that the waker will face cruel competition later,
it could be very cruel or very fast depends on the story behind
'wakee_flips', waker therefore suffers.

Furthermore, if waker also has a high 'wakee_flips', that implies that
multiple tasks rely on it, then waker's higher latency will damage all
of them, so pulling wakee seems to be a bad deal.

Thus, when 'waker->wakee_flips / wakee->wakee_flips' becomes
higher and higher, the cost of pulling seems to be worse and worse.

The patch therefore helps the wake-affine feature to stop its pulling
work when:

	wakee->wakee_flips > factor &&
	waker->wakee_flips > (factor * wakee->wakee_flips)

The 'factor' here is the number of CPUs in the current CPU's NUMA node,
so a bigger node will lead to more pulling since the trial becomes more
severe.

After applying the patch, pgbench shows up to 40% improvements and no regressions.

Tested with 12 cpu x86 server and tip 3.10.0-rc7.

The percentages in the final column highlight the areas with the biggest wins,
all other areas improved as well:

	pgbench		    base	smart

	| db_size | clients |  tps  |	|  tps  |
	+---------+---------+-------+   +-------+
	| 22 MB   |       1 | 10598 |   | 10796 |
	| 22 MB   |       2 | 21257 |   | 21336 |
	| 22 MB   |       4 | 41386 |   | 41622 |
	| 22 MB   |       8 | 51253 |   | 57932 |
	| 22 MB   |      12 | 48570 |   | 54000 |
	| 22 MB   |      16 | 46748 |   | 55982 | +19.75%
	| 22 MB   |      24 | 44346 |   | 55847 | +25.93%
	| 22 MB   |      32 | 43460 |   | 54614 | +25.66%
	| 7484 MB |       1 |  8951 |   |  9193 |
	| 7484 MB |       2 | 19233 |   | 19240 |
	| 7484 MB |       4 | 37239 |   | 37302 |
	| 7484 MB |       8 | 46087 |   | 50018 |
	| 7484 MB |      12 | 42054 |   | 48763 |
	| 7484 MB |      16 | 40765 |   | 51633 | +26.66%
	| 7484 MB |      24 | 37651 |   | 52377 | +39.11%
	| 7484 MB |      32 | 37056 |   | 51108 | +37.92%
	| 15 GB   |       1 |  8845 |   |  9104 |
	| 15 GB   |       2 | 19094 |   | 19162 |
	| 15 GB   |       4 | 36979 |   | 36983 |
	| 15 GB   |       8 | 46087 |   | 49977 |
	| 15 GB   |      12 | 41901 |   | 48591 |
	| 15 GB   |      16 | 40147 |   | 50651 | +26.16%
	| 15 GB   |      24 | 37250 |   | 52365 | +40.58%
	| 15 GB   |      32 | 36470 |   | 50015 | +37.14%

Signed-off-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/51D50057.9000809@linux.vnet.ibm.com
[ Improved the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>

Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
sched: Micro-optimize the smart wake-affine logic

Smart wake-affine is using node-size as the factor currently, but the overhead
of the mask operation is high.

Thus, this patch introduce the 'sd_llc_size' percpu variable, which will record
the highest cache-share domain size, and make it to be the new factor, in order
to reduce the overhead and make it more reasonable.

Tested-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
Tested-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Acked-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Link: http://lkml.kernel.org/r/51D5008E.6030102@linux.vnet.ibm.com
[ Tidied up the changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>

Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
sched: Fix select_idle_sibling() bouncing cow syndrome

If the previous CPU is cache affine and idle, select it.

The current implementation simply traverses the sd_llc domain,
taking the first idle CPU encountered, which walks buddy pairs
hand in hand over the package, inflicting excruciating pain.

1 tbench pair (worst case) in a 10 core + SMT package:

  pre   15.22 MB/sec 1 procs
  post 252.01 MB/sec 1 procs

Signed-off-by: Mike Galbraith <bitbucket@online.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1359371965.5783.127.camel@marge.simpson.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
block: blk-core: add printk_ratelimit

On failure of requests the console is flooded with prints,
which result in a watchdog bark.
This patch limits the number of prints to console thus, avoiding
the bark.

CRs-Fixed: 458071
Change-Id: I614002cc1305ab7b5c6a64d278139291c1baa3c4
Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
mm/swap: fix race on swap_info reuse between swapoff and swapon

Date	Thu, 09 Jan 2014 13:39:55 +0800

swapoff clear swap_info's SWP_USED flag prematurely and free its resources
after that. A concurrent swapon will reuse this swap_info while its previous
resources are not cleared completely.

These late freed resources are:
- p->percpu_cluster
- swap_cgroup_ctrl[type]
- block_device setting
- inode->i_flags &= ~S_SWAPFILE

This patch clear SWP_USED flag after all its resources freed, so that swapon
can reuse this swap_info by alloc_swap_info() safely.

Signed-off-by: Weijie Yang <weijie.yang@samsung.com>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: move VFP init to an earlier boot stage

In order to use the NEON unit in the kernel, we should
initialize it a bit earlier in the boot process so NEON users
that like to do a quick benchmark at load time (like the
xor_blocks or RAID-6 code) find the NEON/VFP unit already
enabled.

Replaced late_initcall() with core_initcall().

Signed-off-by: Ard Biesheuvel <ard.biesheuvel at linaro.org>
Acked-by: Nicolas Pitre <nico at linaro.org>

Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: be strict about FP exceptions in kernel mode

The support code in vfp_support_entry does not care whether the
exception that caused it to be invoked occurred in kernel mode or
in user mode. However, neither condition that could trigger this
exception (lazy restore and VFP bounce to support code) is
currently allowable in kernel mode.

In either case, print a message describing the condition before
letting the undefined instruction handler run its course and trigger
an oops.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel at linaro.org>
Acked-by: Nicolas Pitre <nico at linaro.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: add support for kernel mode NEON

In order to safely support the use of NEON instructions in
kernel mode, some precautions need to be taken:
- the userland context that may be present in the registers (even
  if the NEON/VFP is currently disabled) must be stored under the
  correct task (which may not be 'current' in the UP case),
- to avoid having to keep track of additional vfpstates for the
  kernel side, disallow the use of NEON in interrupt context
  and run with preemption disabled,
- after use, re-enable preemption and re-enable the lazy restore
  machinery by disabling the NEON/VFP unit.

This patch adds the functions kernel_neon_begin() and
kernel_neon_end() which take care of the above. It also adds
the Kconfig symbol KERNEL_MODE_NEON to enable it.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>
ARM: crypto: add NEON accelerated XOR implementation

Add a source file xor-neon.c (which is really just the reference
C implementation passed through the GCC vectorizer) and hook it
up to the XOR framework.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Nicolas Pitre <nico@linaro.org>
ARM: 7835/2: fix modular build of xor_blocks() with NEON enabled

Commit 0195659 introduced a NEON accelerated version of the xor_blocks()
function, but it needs the changes in this patch to allow it to be built
as a module rather than statically into the kernel.

This patch creates a separate module xor-neon.ko which exports the NEON
inner xor_blocks() functions depended upon by the regular xor.ko if it
is built with CONFIG_KERNEL_MODE_NEON=y

Reported-by: Josh Boyer <jwboyer@fedoraproject.org>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
ARM: only allow kernel mode neon with AEABI

This prevents the linker erroring with:

arm-linux-ld: error: arch/arm/lib/xor-neon.o uses VFP instructions, whereas arch/arm/lib/built-in.o does not
arm-linux-ld: failed to merge target specific data of file arch/arm/lib/xor-neon.o

This is due to the non-neon files being marked as containing FPA data/
instructions (even though they do not) being mixed with files which
contain VFP, which is an incompatible floating point format.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Make Makefile compatible for patch