mmc: core: Allow changing bus frequency for SD/eMMC cards in runtime

Currently, bus frequency is set during the card initialization and
never changed until a new card is inserted. In some low power use
cases, scaling the clock frequencies while the card is in transfer
state would allow power savings. This change allows bus frequency
to be changed after the card initialization.

Change-Id: Iac064221199f69d162d91f5311becd735c15700a
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: core: Fix some driver hangs when dealing with broken devices

There are infinite loops in the mmc code that can be caused by bad
hardware.  The code will loop forever if the device never comes back
from program mode, R1_STATE_PRG, and it is not ready for data,
R1_READY_FOR_DATA.

A long timeout is added to prevent the code from looping forever.
The timeout will occur if the device never comes back from program
state or the device never becomes ready for data.

It's not clear whether the timeout will do more than log a pr_err()
and then start a fresh hang all over again.  We may need to extend
this patch later to perform some kind of reset of the device (is
that possible?) or rejection of new I/O to the device.

Change-Id: I1f3e5fddffb5134743fbfbcdef5e97803fd25fc6
Signed-off-by: Trey Ramsay <tramsay@linux.vnet.ibm.com>
Signed-off-by: Chris Ball <cjb@laptop.org>
Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
mmc: core: Add load based clock scaling support

The SD3.0/eMMC4.5/SDIO3.0 cards can support clock rates upto
200MHz (SDR104 or HS200 bus speed modes). For some workloads
like video playback it isn't necessary for these cards to run
at such high speed. Running at lower frequency, say 50MHz, in
such cases can still meet the deadlines for data transfers.
Scaling down the clock frequency dynamically has huge power
savings not only because the bus is running at lower frequency
but also has an advantage of scaling down the system core
voltage, if supported.

Provide an ondemand clock scaling support similar to cpufreq
ondemand governor having two thresholds, up_threshold and
down_threshold to decide whether to increase the frequency or
scale it down respectively. The sampling interval is in the
order of milliseconds and should be chosen by host drivers that
enable MMC_CAP2_CLK_SCALE capability to take advantage of clock
scaling. The sampling interval mainly depends on the the clock
switching delays and hence a host driver decision. If sampling
interval is too low frequent switching of frequencies can lead
to high power consumption and if sampling interval is too high,
the clock scaling logic would take long time to realize that the
underlying hardware (controller and card) is busy and scale up
the clocks.

Change-Id: I22a5054beec41b0b66b3bf030ddfcf284de448b3
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: core: Add sysfs entries for dynamic control of clock scaling

Add sysfs attributes to allow dynamic control of clock scaling
parameters. These attributes are used to enable/disable clock
scaling at runtime and change the up_threshold, down_threshold,
and polling_interval values. Complete documentation for these
sysfs entries are provided at "Documentation/mmc/mmc-dev-attrs.txt".

Change-Id: I4753c75c3b3eeea91e93bceba7af2535b793612e
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: core: claim mmc host while enabling clock scaling from userspace

A race condition can occur while enabling clock scaling and removal
of the card. If the card is removed just after the host->card check
is successful then there could be NULL pointer dereference later.
Claim mmc host before card availability check to avoid this race
condition.

Change-Id: I6339cad5857732ec6b66280940cd6b8b7ed30614
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: core: run clock scaling only in valid card state

Clock scaling requires the tuning and if the tuning command
is sent to the card in invalid state, tuning procedure will
fail.

With urgent request mechanism implemented, there is a chance
that clock scaling invoked when card is not in a proper state
to receive the tuning command.

This change checks the card state (by sending the status command)
before invoking the clock scaling request.

Change-Id: Icb71a8e74a9afdc70380a5901cd3d28931959e9c
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
mmc: core: Log MMC clock frequency transitions

Use kernel's ftrace support to capture MMC clock frequency
transitions which can be useful for debugging issues related
to power consumption.

Usage:
mount -t debugfs none /sys/kernel/debug
echo 1 > /sys/kernel/debug/tracing/events/mmc/mmc_clk/enable
cat /sys/kernel/debug/tracing/trace_pipe

Change-Id: I25c4ee39dcbe30e7665902a9f723a5a421b55ca3
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: core: Add support for notifying host driver while scaling clocks

Host drivers can participate in clock scaling by registering
->notify_load host operation, which allows host driver to carry out
platform specific operations for further power savings or increasing
throughput based on whether load is LOW or HIGH respectively. This
can be applicable to non-ultra high speed cards as well so remove
the check for ultra high speed cards to initialize clock scaling.

Change-Id: Icaab9520135e384f5470db68b2f25c5cdce5663a
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: msm_sdcc: Add support for dynamic bus clock vote

SDCC controller on some targets like 8660, 8960, 8064 and their
variants connects with Daytona Fabric (DFAB). To achieve max.
possible performance the DFAB clock is voted to 64MHz at the
cost of some power consumption. Since we now have dynamic clock
scaling for SD/eMMC clock we can also scale the DFAB clock
accordingly without impact on performance but have some power
savings in low throughput cases.

Change-Id: Ic26541d6845b65803aa813d672ccabdfc4658d9c
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: msm_sdcc: fix a type mismatch in msmsdccc_config_dma()

msmsdcc_prep_xfer() function returns a signed integer value but
caller function msmsdcc_config_dma() is storing this value in an unsigned
integer variable which is obviously not the correct thing to do.

CRs-Fixed: 476578
Change-Id: I39159e2f4f73c7f54de359cbe4faba5e902915ff
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
mmc: msm_sdcc: Enable clock scaling capability

Enable capability, MMC_CAP2_CLK_SCALE to allow mmc core
to scale the clocks at runtime based on load on card.

Clock scaling in MSM platforms also allow voltage scaling between
NOMINAL and SVS levels, thus allowing additional power savings.

Change-Id: Id1995ffea7a6c06a2e8b29b43da70894f35f6c0b
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
mmc: core: remove the polling for BKOPS completion

The original intention of polling for BKOPS completion was to give
the card enough time to perform the BKOPS before it is runtime suspended.
But as the BKOPS completion polling was happening in a different
context, it may race with card runtime/platform suspend which is quite
difficult to fix. So instead of BKOPS polling, let the runtime suspend
get deferred if the BKOPS is running on the card. Also if BKOPS is running
when platform suspend is triggered, stop the BKOPS before suspending the
card.

CRs-Fixed: 489523
Change-Id: I21e524dc2da37c4985c210abfaca00a28049c651
Signed-off-by: Maya Erez <merez@codeaurora.org>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
msm: kgsl: Don't do intensive memory recovery when allocating big pages

We don't want to incur too much overhead when allocating big pages
so don't attempt to retry, perform reclaim, or run memory compaction
on high-order allocations.

Change-Id: Ic0dedbadb354c6faea34abec36aee268ac0f2c34
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: msm_fb: Add range checks for fb cmap.

Add appropriate range checks to ensure valid access
through framebuffer cmap. This prevents illegal
access into memory.

Change-Id: I9e82a27a2d12f31d8150ea577514576824f8b1f5
Signed-off-by: Vishnuvardhan Prodduturi <vproddut@codeaurora.org>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
msm: SSR: Remove useless warning

This is a warning so that developers know to add more restart
order lists to SSR when a new chip is added. This is mostly
irrelevant now because we assume either entire SoC restart on SSR
or independent restart on SSR, not group restarts. Remove this
warning as it is mostly a reminder that nobody is listening for.

Change-Id: Icbf955cb18395d8d5d086b2167c5c329588b9256
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
PM / QoS: Fix deadlock during PM QoS vote to default value

pm_qos_work_fn() calls pm_qos_update_request() which does
cancel_delayed_work_sync() on the same pm_qos_work_fn()
causing deadlock. Fix this by updating the target with
default timeout value directly instead of calling
pm_qos_update_request().

CRs-Fixed: 526216
Change-Id: I7de2fb1c89f87b0ebf7427116b8920aec50d5b2b
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
lib: spinlock_debug: Avoid livelock in do_raw_spin_lock

The logic in do_raw_spin_lock attempts to acquire a spinlock
by invoking arch_spin_trylock in a loop with a delay between
each attempt. Now consider the following situation in a 2
CPU system:

1. Cpu 0 continually acquires and releases a spinlock in a
   tight loop; it stays in this loop until some condition X
   is satisfied. X can only be satisfied by another CPU.

2. Cpu 1 tries to acquire the same spinlock, in an attempt
   to satisfy the aforementioned condition X. However, it
   never sees the unlocked value of the lock because the
   debug spinlock code uses trylock instead of just lock;
   it checks at all the wrong moments - whenever Cpu 0 has
   locked the lock.

Now in the absence of debug spinlocks, the architecture specific
spinlock code can correctly allow Cpu1 to wait in a "queue"
(e.g., ticket spinlocks), ensuring that it acquires the lock at
some point. However, with the debug spinlock code, livelock
can easily occur due to the use of try_lock, which obviously
cannot put the CPU in that "queue".

Address this by actually attempting arch_spin_lock once it is
suspected that there is a spinlock lockup. If we're in a
situation that is described above, the arch_spin_lock should
succeed; otherwise other timeout mechanisms (e.g., watchdog)
should alert the system of a lockup.

Additionally, the delay between each trylock attempt adds up
to a total delay of 1 second. On some embedded architectures,
this is an unacceptably large value. Reduce this to a ~62 ms
delay.

CRs-Fixed: 393351
Change-Id: I56f0d8f5e483cd7e8167fc7a363b9de7777a0830
Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>
lib: spinlock_debug: increase spin dump timeout

On certain targets the worst case console buffer draining time can
exceed the aggressive spin dump timeout value. Relax the timeout value
to cater to these scenarios.

Change-Id: I7312ffe06bafd2dc7adf0f4e8b73a53fc5839235
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
arm: Remove no-longer-required RCU_NONIDLE wrapper

Commit 21111be8 "ARM: Fix negative idle stats for offline cpu" moved call
to cpu_die() to occur outside of rcu_idle_enter()/rcu_idle_exit() section.
As a result, the RCU_NONIDLE() wrapper to complete() call in
arch/arm/kernel/process.c:cpu_die() is no longer required (and is
technically incorrect to have). Removing RCU_NONIDLE() wrapper also removes
this warning seen during CPU offline:

[Note: Below message has been edited to fit 75-char per line limit.
Insignificant portions of warning message has been removed in each line.]

------------[ cut here ]------------
WARNING: at kernel/rcutree.c:456 rcu_idle_exit_common+0x4c/0xe0()
Modules linked in:
(unwind_backtrace+0x0/0x120) from (warn_slowpath_common+0x4c/0x64)
(warn_slowpath_common+0x4c/0x64) from (warn_slowpath_null+0x18/0x1c)
(warn_slowpath_null+0x18/0x1c) from (rcu_idle_exit_common+0x4c/0xe0)
(rcu_idle_exit_common+0x4c/0xe0) from (rcu_idle_exit+0xa8/0xc0)
(rcu_idle_exit+0xa8/0xc0) from (cpu_die+0x24/0x5c)
(cpu_die+0x24/0x5c) from (cpu_idle+0xdc/0xf0)
(cpu_idle+0xdc/0xf0) from (0x8160)
---[ end trace 61bf21937a496a37 ]---

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Change-Id: I721e6b20651674e6f6f584cf8d814af00b688c91
lib: Update RWSEM to Linux 3.10

Squashed commit of the following:

commit 28f9a942e4e1578c0e26e8211ba49bdf520de95c
Author: Davidlohr Bueso <davidlohr.bueso@hp.com>
Date:   Tue May 7 06:46:02 2013 -0700

    rwsem: no need for explicit signed longs

    Change explicit "signed long" declarations into plain "long" as suggested
    by Peter Hurley.

    Signed-off-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Reviewed-by: Michel Lespinasse <walken@google.com>
    Signed-off-by: Michel Lespinasse <walken@google.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 353ad88b6af95bc7bb127b3b5b1cfa36c72a7b67
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:46:00 2013 -0700

    rwsem: do not block readers at head of queue if other readers are active

    This change fixes a race condition where a reader might determine it
    needs to block, but by the time it acquires the wait_lock the rwsem has
    active readers and no queued waiters.

    In this situation the reader can run in parallel with the existing
    active readers; it does not need to block until the active readers
    complete.

    Thanks to Peter Hurley for noticing this possible race.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit e9149aa93e4e6919da4fe9e77a27f9c6a0159463
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:59 2013 -0700

    rwsem: implement support for write lock stealing on the fastpath

    When we decide to wake up readers, we must first grant them as many read
    locks as necessary, and then actually wake up all these readers.  But in
    order to know how many read shares to grant, we must first count the
    readers at the head of the queue.  This might take a while if there are
    many readers, and we want to be protected against a writer stealing the
    lock while we're counting.  To that end, we grant the first reader lock
    before counting how many more readers are queued.

    We also require some adjustments to the wake_type semantics.

    RWSEM_WAKE_NO_ACTIVE used to mean that we had found the count to be
    RWSEM_WAITING_BIAS, in which case the rwsem was known to be free as
    nobody could steal it while we hold the wait_lock.  This doesn't make
    sense once we implement fastpath write lock stealing, so we now use
    RWSEM_WAKE_ANY in that case.

    Similarly, when rwsem_down_write_failed found that a read lock was
    active, it would use RWSEM_WAKE_READ_OWNED which signalled that new
    readers could be woken without checking first that the rwsem was
    available.  We can't do that anymore since the existing readers might
    release their read locks, and a writer could steal the lock before we
    wake up additional readers.  So, we have to use a new RWSEM_WAKE_READERS
    value to indicate we only want to wake readers, but we don't currently
    hold any read lock.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 20addabcb540eb4b4e11d7e8af037543767e081d
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:58 2013 -0700

    rwsem: simplify __rwsem_do_wake

    This is mostly for cleanup value:

    - We don't need several gotos to handle the case where the first
      waiter is a writer. Two simple tests will do (and generate very
      similar code).

    - In the remainder of the function, we know the first waiter is a reader,
      so we don't have to double check that. We can use do..while loops
      to iterate over the readers to wake (generates slightly better code).

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit e56c8eea2628841a06a2420e2c6ca8b8c6bd4434
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:57 2013 -0700

    rwsem: skip initial trylock in rwsem_down_write_failed

    We can skip the initial trylock in rwsem_down_write_failed() if there
    are known active lockers already, thus saving one likely-to-fail
    cmpxchg.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 286040cc550df96bef6e97a5bc0231ded9a0187b
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:56 2013 -0700

    rwsem: avoid taking wait_lock in rwsem_down_write_failed

    In rwsem_down_write_failed(), if there are active locks after we wake up
    (i.e.  the lock got stolen from us), skip taking the wait_lock and go
    back to sleep immediately.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 6c7229b31a4c099157242c58f5d33ce728daff4b
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:55 2013 -0700

    rwsem: use cmpxchg for trying to steal write lock

    Using rwsem_atomic_update to try stealing the write lock forced us to
    undo the adjustment in the failure path.  We can have simpler and faster
    code by using cmpxchg instead.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Acked-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 85dcaeae08b5738f8ea0b08734ab97e57bb9e8e2
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:54 2013 -0700

    rwsem: more agressive lock stealing in rwsem_down_write_failed

    Some small code simplifications can be achieved by doing more agressive
    lock stealing:

    - When rwsem_down_write_failed() notices that there are no active locks
      (and thus no thread to wake us if we decided to sleep), it used to wake
      the first queued process. However, stealing the lock is also sufficient
      to deal with this case, so we don't need this check anymore.

    - In try_get_writer_sem(), we can steal the lock even when the first waiter
      is a reader. This is correct because the code path that wakes readers is
      protected by the wait_lock. As to the performance effects of this change,
      they are expected to be minimal: readers are still granted the lock
      (rather than having to acquire it themselves) when they reach the front
      of the wait queue, so we have essentially the same behavior as in
      rwsem-spinlock.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 082a2e9c40e01d70c59ca4e3a9b552b81d573332
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:53 2013 -0700

    rwsem: simplify rwsem_down_write_failed

    When waking writers, we never grant them the lock - instead, they have
    to acquire it themselves when they run, and remove themselves from the
    wait_list when they succeed.

    As a result, we can do a few simplifications in rwsem_down_write_failed():

    - We don't need to check for !waiter.task since __rwsem_do_wake() doesn't
      remove writers from the wait_list

    - There is no point releaseing the wait_lock before entering the wait loop,
      as we will need to reacquire it immediately. We can change the loop so
      that the lock is always held at the start of each loop iteration.

    - We don't need to get a reference on the task structure, since the task
      is responsible for removing itself from the wait_list. There is no risk,
      like in the rwsem_down_read_failed() case, that a task would wake up and
      exit (thus destroying its task structure) while __rwsem_do_wake() is
      still running - wait_lock protects against that.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 6b3306dccd59d5dcf22e55a27827bccf51a796e1
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:52 2013 -0700

    rwsem: simplify rwsem_down_read_failed

    When trying to acquire a read lock, the RWSEM_ACTIVE_READ_BIAS
    adjustment doesn't cause other readers to block, so we never have to
    worry about waking them back after canceling this adjustment in
    rwsem_down_read_failed().

    We also never want to steal the lock in rwsem_down_read_failed(), so we
    don't have to grab the wait_lock either.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit c6b315c32d1ffbb33c75ab5e0f8ee27c89d206c4
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:51 2013 -0700

    rwsem: move rwsem_down_failed_common code into rwsem_down_{read,write}_failed

    Remove the rwsem_down_failed_common function and replace it with two
    identical copies of its code in rwsem_down_{read,write}_failed.

    This is because we want to make different optimizations in
    rwsem_down_{read,write}_failed; we are adding this pure-duplication
    step as a separate commit in order to make it easier to check the
    following steps.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 7d2869248ccd7971fc1b79e4bf22cb4be446faf4
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:50 2013 -0700

    rwsem: shorter spinlocked section in rwsem_down_failed_common()

    This change reduces the size of the spinlocked and TASK_UNINTERRUPTIBLE
    sections in rwsem_down_failed_common():

    - We only need the sem->wait_lock to insert ourselves on the wait_list;
      the waiter node can be prepared outside of the wait_lock.

    - The task state only needs to be set to TASK_UNINTERRUPTIBLE immediately
      before checking if we actually need to sleep; it doesn't need to protect
      the entire function.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 8c163f8ddf5c0bd018706dbdd44fb96a073401d0
Author: Michel Lespinasse <walken@google.com>
Date:   Tue May 7 06:45:49 2013 -0700

    rwsem: make the waiter type an enumeration rather than a bitmask

    We are not planning to add some new waiter flags, so we can convert the
    waiter type into an enumeration.

    Background: David Howells suggested I do this back when I tried adding
    a new waiter type for unfair readers. However, I believe the cleanup
    applies regardless of that use case.

    Signed-off-by: Michel Lespinasse <walken@google.com>
    Reviewed-by: Rik van Riel <riel@redhat.com>
    Reviewed-by: Peter Hurley <peter@hurleysoftware.com>
    Acked-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

commit 9a126edbed8ccfedda04d5c3fe256cd139d4bcca
Author: Alex Shi <alex.shi@intel.com>
Date:   Tue Feb 5 21:11:55 2013 +0800

    rwsem: Implement writer lock-stealing for better scalability

    Commit 5a505085f043 ("mm/rmap: Convert the struct anon_vma::mutex
    to an rwsem") changed struct anon_vma::mutex to an rwsem, which
    caused aim7 fork_test performance to drop by 50%.

    Yuanhan Liu did the following excellent analysis:

        https://lkml.org/lkml/2013/1/29/84

    and found that the regression is caused by strict, serialized,
    FIFO sequential write-ownership of rwsems. Ingo suggested
    implementing opportunistic lock-stealing for the front writer
    task in the waitqueue.

    Yuanhan Liu implemented lock-stealing for spinlock-rwsems,
    which indeed recovered much of the regression - confirming
    the analysis that the main factor in the regression was the
    FIFO writer-fairness of rwsems.

    In this patch we allow lock-stealing to happen when the first
    waiter is also writer. With that change in place the
    aim7 fork_test performance is fully recovered on my
    Intel NHM EP, NHM EX, SNB EP 2S and 4S test-machines.

    Reported-by: lkp@linux.intel.com
    Reported-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
    Signed-off-by: Alex Shi <alex.shi@intel.com>
    Cc: David Howells <dhowells@redhat.com>
    Cc: Michel Lespinasse <walken@google.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Anton Blanchard <anton@samba.org>
    Cc: Arjan van de Ven <arjan@linux.intel.com>
    Cc: paul.gortmaker@windriver.com
    Link: https://lkml.org/lkml/2013/1/29/84
    Link: http://lkml.kernel.org/r/1360069915-31619-1-git-send-email-alex.shi@intel.com
    [ Small stylistic fixes, updated changelog. ]
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

commit 77302ff69a93cfba32aaee98d2bb864ba8886e40
Author: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Date:   Fri Feb 1 18:59:16 2013 +0800

    rwsem-spinlock: Implement writer lock-stealing for better scalability

    We (Linux Kernel Performance project) found a regression
    introduced by commit:

      5a505085f043 mm/rmap: Convert the struct anon_vma::mutex to an rwsem

    which converted all anon_vma::mutex locks rwsem write locks.

    The semantics are the same, but the behavioral difference is
    quite huge in some cases. After investigating it we found the
    root cause: mutexes support lock stealing while rwsems don't.

    Here is the link for the detailed regression report:

      https://lkml.org/lkml/2013/1/29/84

    Ingo suggested adding write lock stealing to rwsems:

        "I think we should allow lock-steal between rwsem writers - that
         will not hurt fairness as most rwsem fairness concerns relate to
         reader vs. writer fairness"

    And here is the rwsem-spinlock version.

    With this patch, we got a double performance increase in one
    test box with following aim7 workfile:

        FILESIZE: 1M
        POOLSIZE: 10M
        10 fork_test

     /usr/bin/time output w/o patch                       /usr/bin/time_output with patch
     -- Percent of CPU this job got: 369%                 Percent of CPU this job got: 537%
     Voluntary context switches: 640595016                Voluntary context switches: 157915561

    We got a 45% increase in CPU usage and saved about 3/4 voluntary context switches.

    Reported-by: LKP project <lkp@linux.intel.com>
    Suggested-by: Ingo Molnar <mingo@kernel.org>
    Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
    Cc: Alex Shi <alex.shi@intel.com>
    Cc: David Howells <dhowells@redhat.com>
    Cc: Michel Lespinasse <walken@google.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Anton Blanchard <anton@samba.org>
    Cc: Arjan van de Ven <arjan@linux.intel.com>
    Cc: paul.gortmaker@windriver.com
    Link: http://lkml.kernel.org/r/1359716356-23865-1-git-send-email-yuanhan.liu@linux.intel.com
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

commit 95f8b6683b3f83e40a8ab0d93a9d155954b64396
Author: Andrei F <luxneb@gmail.com>
Date:   Wed Aug 21 21:49:39 2013 +0400

    ARM: rwsem: Update header from initial implementation

    Cleanup due to:

    c16a87ce063f79e0ec7d25ce2950e1bc6db03c72 rwsem: Cleanup includes
    1c8ed640d918290ddc1de5ada02ef6686a733c9f rwsem: Move duplicate struct rwsem declaration to linux/rwsem.h
    12249b34414dba7f386aadcf6be7ca36c6878300 rwsem: Move duplicate init macros and functions to linux/rwsem.h
    41e5887fa39ab272d9266a09cbefdef270e28b93 rwsem: Unify the duplicate rwsem_is_locked() inlines
    aac72277fda6ef788bb8d5deaa502ce9b9b6e472 rwsem: Move duplicate function prototypes to linux/rwsem.h