Bump kernel version to 0.509_twrp_swappedSD

Change-Id: I1716dfaf469c8db2275ea4d4292ae39ee3a74236
block: urgent: Fix dispatching of URGENT mechanism

There are cases when blk_peek_request is called not from blk_fetch_request
thus the URGENT request may be started but the flag q->dispatched_urgent is
not updated.

Change-Id: I4fb588823f1b2949160cbd3907f4729767932e12
CRs-fixed: 471736
CRs-fixed: 473036
Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>
block: Remove "requeuing urgent req" error messages

It is possible for URGENT request to be requeued/reinserted if it was
fetched during the creation of a packed list. This end case is rare and is
not handled at the moment.
This patch changes the messages notifying of the above to debug level
(instead of error) in order to clear the dmesg log.

Change-Id: Ie8bc067e61559a6f702077b95c5dbcc426404232
Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>
block: urgent: Fix dispatching of URGENT mechanism

There are cases when blk_peek_request is called not from blk_fetch_request
thus the URGENT request may be started but the flag q->dispatched_urgent is
not updated.

Change-Id: I4fb588823f1b2949160cbd3907f4729767932e12
CRs-fixed: 471736
CRs-fixed: 473036
Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>

Conflicts:
	block/blk-core.c
block: Remove "requeuing urgent req" error messages

It is possible for URGENT request to be requeued/reinserted if it was
fetched during the creation of a packed list. This end case is rare and is
not handled at the moment.
This patch changes the messages notifying of the above to debug level
(instead of error) in order to clear the dmesg log.

Change-Id: Ie8bc067e61559a6f702077b95c5dbcc426404232
Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>
block: remove READ_META and WRITE_META

Replace all occurnanced of the undocumented READ_META with READ | REQ_META
and remove the unused WRITE_META define.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
block: separate priority boosting from REQ_META

Add a new REQ_PRIO to let requests preempt others in the cfq I/O schedule,
and lave REQ_META purely for marking requests as metadata in blktrace.

All existing callers of REQ_META except for XFS are updated to also
set REQ_PRIO for now.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Namhyung Kim <namhyung@gmail.com>
Signed-off-by: Jens Axboe <jaxboe@fusionio.com>

Conflicts:
	block/cfq-iosched.c

Change-Id: I9f5aadc73cc73a223c544e10fa3c4716c8c4f087
Squashed commit of the following:

commit 988a326fa22d6fabda04b5ec7a36fcc1b58a6bb2
Author: Lee Susman <lsusman@codeaurora.org>
Date:   Sun May 5 17:31:17 2013 +0300

    mm: pass readahead info down to the i/o scheduler

    Some i/o schedulers (i.e. row-iosched, cfq-iosched) deploy an idling
    algorithm in order to be better synced with the readahead algorithm.
    Idling is a prediction algorithm for incoming read requests.

    In this patch we mark pages which are part of a readahead window, by
    setting a newly introduced flag. With this flag, the i/o scheduler can
    identify a request which is associated with a readahead page. This
    enables the i/o scheduler's idling mechanism to be en-sync with the
    readahead mechanism and, in turn, can increase read throughput.

    Change-Id: I0654f23315b6d19d71bcc9cc029c6b281a44b196
    Signed-off-by: Lee Susman <lsusman@codeaurora.org>

commit 6e42d2a595838d1eb12c24e4cf1675e341f98eb1
Author: Tatyana Brokhman <tlinder@codeaurora.org>
Date:   Thu Jul 25 13:45:21 2013 +0300

    block: row: Remove warning massage from add_request

    Regular priority queues is marked as "starved" if it skipped a dispatch
    due to being empty. When a new request is added to a "starved" queue
    it will be marked as urgent.
    The removed WARN_ON was warning about an impossible case when a regular
    priority (read) queue was marked as starved but wasn't empty. This is
    a possible case due to the bellow:
    If the device driver fetched a read request that is pending for
    transmission and an URGENT request arrives, the fetched read will be
    reinserted back to the scheduler. Its possible that the queue it will be
    reinserted to was marked as "starved" in the meanwhile due to being empty.

    CRs-fixed: 517800
    Change-Id: Iaae642ea0ed9c817c41745b0e8ae2217cc684f0c
    Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>

commit 91ab76debc19a1fe5290369c4b4886dbb54ecbbc
Author: Tatyana Brokhman <tlinder@codeaurora.org>
Date:   Sun Jul 7 17:33:05 2013 +0300

    block: Add URGENT request notification support to CFQ scheduler

    When the scheduler reports to the block layer that there is an urgent
    request pending, the device driver may decide to stop the transmission
    of the current request in order to handle the urgent one. This is done
    in order to reduce the latency of an urgent request. For example:
    long WRITE may be stopped to handle an urgent READ.

    Change-Id: I3072b8a1423870fed9c04c28d93caaf9557a7b89
    Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>

commit be557c3f9b8944509293955da9be62bb292787e6
Author: Tatyana Brokhman <tlinder@codeaurora.org>
Date:   Tue Jul 2 14:43:13 2013 +0300

    block: row: change hrtimer_cancel to hrtimer_try_to_cancel

    Calling hrtimer_cancel with interrupts disabled can result in a livelock.
    When flushing plug list in the block layer interrupts are disabled and an
    hrtimer is used when adding requests from that plug list to the scheduler.
    In this code flow, if the hrtimer (which is used for idling) is set, it's
    being canceled by calling hrtimer_cancel. hrtimer_cancel will perform
    the following in an endless loop:
    1. try cancel the timer
    2. if fails - rest_cpu
    the cancellation can fail if the timer function already started. Since
    interrupts are disabled it can never complete.
    This patch reduced the number of times the hrtimer lock is taken while
    interrupts are disabled by calling hrtimer_try_co_cancel. the later will
    try to cancel the timer just once and return with an error code if fails.

    CRs-fixed: 499887
    Change-Id: I25f79c357426d72ad67c261ce7cb503ae97dc7b9
    Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>

commit e33e599a5527e24d2a41252937d625dac4e86cdb
Author: Lee Susman <lsusman@codeaurora.org>
Date:   Sun Jun 23 16:27:40 2013 +0300

    block: row-iosched idling triggered by readahead pages

    In the current implementation idling is triggered only by request
    insertion frequency. This heuristic is not very accurate and may hit
    random requests that shouldn't trigger idling. This patch uses the
    PG_readahead flag in struct page's flags, which indicates that the page
    is part of a readahead window, to start idling upon dispatch of a request
    associated with a readahead page.

    The above readehead flag is used together with the existing
    insertion-frequency trigger. The frequency timer will catch read requests
    which are not part of a readahead window, but are still part of a
    sequential stream (and therefore dispatched in small time intervals).

    Change-Id: Icb7145199c007408de3f267645ccb842e051fd00
    Signed-off-by: Lee Susman <lsusman@codeaurora.org>

commit fa709c01d23bcb5afe4651bca751327b5f4e22f0
Author: Tatyana Brokhman <tlinder@codeaurora.org>
Date:   Thu May 16 14:36:58 2013 +0300

    block: Remove "requeuing urgent req" error messages

    It is possible for URGENT request to be requeued/reinserted if it was
    fetched during the creation of a packed list. This end case is rare and is
    not handled at the moment.
    This patch changes the messages notifying of the above to debug level
    (instead of error) in order to clear the dmesg log.

    Change-Id: Ie8bc067e61559a6f702077b95c5dbcc426404232
    Signed-off-by: Tatyana Brokhman <tlinder@codeaurora.org>
locking, rwsem: Annotate inner lock as raw

There is no reason to allow the lock protecting rwsems (the
ownerless variant) to be preemptible on -rt. Convert it to raw.

In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
rwsem-spinlock: Implement writer lock-stealing for better scalability

We (Linux Kernel Performance project) found a regression
introduced by commit:

  5a505085f043 mm/rmap: Convert the struct anon_vma::mutex to an rwsem

which converted all anon_vma::mutex locks rwsem write locks.

The semantics are the same, but the behavioral difference is
quite huge in some cases. After investigating it we found the
root cause: mutexes support lock stealing while rwsems don't.

Here is the link for the detailed regression report:

  https://lkml.org/lkml/2013/1/29/84

Ingo suggested adding write lock stealing to rwsems:

    "I think we should allow lock-steal between rwsem writers - that
     will not hurt fairness as most rwsem fairness concerns relate to
     reader vs. writer fairness"

And here is the rwsem-spinlock version.

With this patch, we got a double performance increase in one
test box with following aim7 workfile:

    FILESIZE: 1M
    POOLSIZE: 10M
    10 fork_test

 /usr/bin/time output w/o patch                       /usr/bin/time_output with patch
 -- Percent of CPU this job got: 369%                 Percent of CPU this job got: 537%
 Voluntary context switches: 640595016                Voluntary context switches: 157915561

We got a 45% increase in CPU usage and saved about 3/4 voluntary context switches.

Reported-by: LKP project <lkp@linux.intel.com>
Suggested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Cc: Alex Shi <alex.shi@intel.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Anton Blanchard <anton@samba.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: paul.gortmaker@windriver.com
Link: http://lkml.kernel.org/r/1359716356-23865-1-git-send-email-yuanhan.liu@linux.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
rwsem: Implement writer lock-stealing for better scalability

Commit 5a505085f043 ("mm/rmap: Convert the struct anon_vma::mutex
to an rwsem") changed struct anon_vma::mutex to an rwsem, which
caused aim7 fork_test performance to drop by 50%.

Yuanhan Liu did the following excellent analysis:

    https://lkml.org/lkml/2013/1/29/84

and found that the regression is caused by strict, serialized,
FIFO sequential write-ownership of rwsems. Ingo suggested
implementing opportunistic lock-stealing for the front writer
task in the waitqueue.

Yuanhan Liu implemented lock-stealing for spinlock-rwsems,
which indeed recovered much of the regression - confirming
the analysis that the main factor in the regression was the
FIFO writer-fairness of rwsems.

In this patch we allow lock-stealing to happen when the first
waiter is also writer. With that change in place the
aim7 fork_test performance is fully recovered on my
Intel NHM EP, NHM EX, SNB EP 2S and 4S test-machines.

Reported-by: lkp@linux.intel.com
Reported-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Signed-off-by: Alex Shi <alex.shi@intel.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Anton Blanchard <anton@samba.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: paul.gortmaker@windriver.com
Link: https://lkml.org/lkml/2013/1/29/84
Link: http://lkml.kernel.org/r/1360069915-31619-1-git-send-email-alex.shi@intel.com
[ Small stylistic fixes, updated changelog. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
rwsem: make the waiter type an enumeration rather than a bitmask

We are not planning to add some new waiter flags, so we can convert the
waiter type into an enumeration.

Background: David Howells suggested I do this back when I tried adding
a new waiter type for unfair readers. However, I believe the cleanup
applies regardless of that use case.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: shorter spinlocked section in rwsem_down_failed_common()

This change reduces the size of the spinlocked and TASK_UNINTERRUPTIBLE
sections in rwsem_down_failed_common():

- We only need the sem->wait_lock to insert ourselves on the wait_list;
  the waiter node can be prepared outside of the wait_lock.

- The task state only needs to be set to TASK_UNINTERRUPTIBLE immediately
  before checking if we actually need to sleep; it doesn't need to protect
  the entire function.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: move rwsem_down_failed_common code into rwsem_down_{read,write}_failed

Remove the rwsem_down_failed_common function and replace it with two
identical copies of its code in rwsem_down_{read,write}_failed.
This is because we want to make different optimizations in
rwsem_down_{read,write}_failed; we are adding this pure-duplication
step as a separate commit in order to make it easier to check the
following steps.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: simplify rwsem_down_read_failed

When trying to acquire a read lock, the RWSEM_ACTIVE_READ_BIAS adjustment
doesn't cause other readers to block, so we never have to worry about waking
them back after canceling this adjustment in rwsem_down_read_failed().

We also never want to steal the lock in rwsem_down_read_failed(), so we
don't have to grab the wait_lock either.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: simplify rwsem_down_write_failed

When waking writers, we never grant them the lock - instead, they have
to acquire it themselves when they run, and remove themselves from the
wait_list when they succeed.

As a result, we can do a few simplifications in rwsem_down_write_failed():

- We don't need to check for !waiter.task since __rwsem_do_wake() doesn't
  remove writers from the wait_list

- There is no point releaseing the wait_lock before entering the wait loop,
  as we will need to reacquire it immediately. We can change the loop so
  that the lock is always held at the start of each loop iteration.

- We don't need to get a reference on the task structure, since the task
  is responsible for removing itself from the wait_list. There is no risk,
  like in the rwsem_down_read_failed() case, that a task would wake up and
  exit (thus destroying its task structure) while __rwsem_do_wake() is
  still running - wait_lock protects against that.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: more agressive lock stealing in rwsem_down_write_failed

Some small code simplifications can be achieved by doing more agressive
lock stealing:

- When rwsem_down_write_failed() notices that there are no active locks
  (and thus no thread to wake us if we decided to sleep), it used to wake
  the first queued process. However, stealing the lock is also sufficient
  to deal with this case, so we don't need this check anymore.

- In try_get_writer_sem(), we can steal the lock even when the first waiter
  is a reader. This is correct because the code path that wakes readers is
  protected by the wait_lock. As to the performance effects of this change,
  they are expected to be minimal: readers are still granted the lock
  (rather than having to acquire it themselves) when they reach the front
  of the wait queue, so we have essentially the same behavior as in
  rwsem-spinlock.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: use cmpxchg for trying to steal write lock

Using rwsem_atomic_update to try stealing the write lock forced us to
undo the adjustment in the failure path. We can have simpler and faster
code by using cmpxchg instead.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: avoid taking wait_lock in rwsem_down_write_failed

In rwsem_down_write_failed(), if there are active locks after we wake up
(i.e. the lock got stolen from us), skip taking the wait_lock and go back
to sleep immediately.

Signed-off-by: Michel Lespinasse <walken@google.com>
rwsem: skip initial trylock in rwsem_down_write_failed

We can skip the initial trylock in rwsem_down_write_failed() if there are
known active lockers already, thus saving one likely-to-fail cmpxchg.

Signed-off-by: Michel Lespinasse <walken@google.com>