sched: fix rq->lock recursion

Enabling SCHED_HRTICK currently results in rq->lock recursion and a hard
hang at bootup.  Essentially try_to_wakeup() grabs rq->lock and tries
arming a hrtimer via hrtimer_restart(), which deep down tries waking up
ksoftirqd, which leads to a recursive call to try_to_wakeup() and thus
attempt to take rq->lock recursively!!

This is fixed by having scheduler queue hrtimer via
__hrtimer_start_range_ns() which avoids waking up ksoftirqd.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Change-Id: I11a13be1d9db3a749614ccf3d4f5fb7bf6f18fa1
(cherry picked from commit 4ca1d04ea0bdc225cc7db302172f3375a63f44de)
Revert "sched_avg: add run queue averaging"

This reverts commit 3647f3cf1253c649b55e16982b112d62956ba842.
Revert "sched: add sysctl for controlling task migrations on wake"

This reverts commit 0eeabce5917d5197c01acac2f8c8070918d2e53b.
Revert "sched: add PF_WAKE_UP_IDLE"

This reverts commit 3f4321d21cb16eec34f3653c185f05e31cbe407a.
sched_avg: add run queue averaging.

Add code to calculate the run queue depth of a cpu and iowait
depth of the cpu.

The scheduler calls in to sched_update_nr_prod whenever there
is a runqueue change. This function maintains the runqueue average
and the iowait of that cpu in that time interval.

Whoever wants to know the runqueue average is expected to call
sched_get_nr_running_avg periodically to get the accumulated
runqueue and iowait averages for all the cpus.

Signed-off-by: Jeff Ohlstein <johlstei@codeaurora.org>
(cherry picked from commit 0299fcaaad80e2c0ac9aa583c95107f6edc27750)

Signed-off-by: Ram Kumar Chakravarthy Chebathini <rcheba@codeaurora.org>
(cherry picked from commit 5325d8a44f8296170cd8f42e245dd79f7b7347bf)

Change-Id: I92875789886c3f7c2457a063de4b3b6d8c893718
Signed-off-by: Sudhir Sharma <sudsha@codeaurora.org>
wake_up_process() should be never used to wakeup a TASK_STOPPED/TRACED task

wake_up_process() should never wakeup a TASK_STOPPED/TRACED task.
Change it to use TASK_NORMAL and add the WARN_ON().

TASK_ALL has no other users, probably can be killed.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
sched: provide per cpu-cgroup option to notify on migrations

On systems where CPUs may run asynchronously, task migrations
between CPUs running at grossly different speeds can cause
problems.

This change provides a mechanism to notify a subsystem
in the kernel if a task in a particular cgroup migrates to a
different CPU. Other subsystems (such as cpufreq) may then
register for this notifier to take appropriate action when
such a task is migrated.

The cgroup attribute to set for this behavior is
"notify_on_migrate" .

Change-Id: Ie1868249e53ef901b89c837fdc33b0ad0c0a4590
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Mekala Natarajan <mekalan@codeaurora.org>
sched: remove migration notification from RT class

Commit 88a7e37d265 (sched: provide per cpu-cgroup option to
notify on migrations) added a notifier call when a task is moved
to a different CPU. Unfortunately the two call sites in the RT
sched class where this occurs happens with a runqueue lock held.
This can result in a deadlock if the notifier call attempts to do
something like wake up a task.

Fortunately the benefit of 88a7e37d265 comes mainly from notifying
on migration of non-RT tasks, so we can simply ignore the movements
of RT tasks.

CRs-Fixed: 491370
Change-Id: I8849d826bf1eeaf85a6f6ad872acb475247c5926
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Mekala Natarajan <mekalan@codeaurora.org>
sched: Reset rq->next_interval before going idle

next_balance, the point in jiffy time scale when a cpu will next load
balance, could have been calculated when the cpu was busy. A busy cpu
will apply its sched domain's busy_factor (usually > 1) in computing
next_balance for that sched domain, which causes the (busy) cpu to load
balance less frequently in its sched domains. However when the same cpu
is going idle, its next_balance needs to be reset without consideration
of busy_factor. Failure to do so would not trigger nohz idle balancer on
that cpu for unnecessarily long time (introducing additional scheduling
latencies for tasks). Fix bug in scheduler which aims to reset
next_balance before a cpu goes idle (as per existing comment) but is
clearly not doing so.

Change-Id: I7e027a51686528c4092d770c7d33c874d38f5df4
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
sched: Fix SCHED_HRTICK bug leading to late preemption of tasks

SCHED_HRTICK feature is useful to preempt SCHED_FAIR tasks on-the-dot
(just when they would have exceeded their ideal_runtime). It makes use
of a a per-cpu hrtimer resource and hence alarming that hrtimer should
be based on total SCHED_FAIR tasks a cpu has across its various cfs_rqs,
rather than being based on number of tasks in a particular cfs_rq (as
implemented currently). As a result, with current code, its possible for
a running task (which is the sole task in its cfs_rq) to be preempted
much after its ideal_runtime has elapsed, resulting in increased latency
for tasks in other cfs_rq on same cpu.

Fix this by alarming sched hrtimer based on total number of SCHED_FAIR
tasks a CPU has across its various cfs_rqs.

Change-Id: I1f23680a64872f8ce0f451ac4bcae28e8967918f
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
sched: fix reference to wrong cfs_rq

Commit 7db16c8c (sched: Fix SCHED_HRTICK bug leading to late preemption
of tasks) introduced a bug in sched_slice() calculation by using wrong
cfs_rq for tasks. rq->cfs was incorrectly used as task's cfs_rq, rather
than the correct one to which they belonged.

Fix the bug by using correct cfs_rq for tasks.

Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
sched: re-calculate a cpu's next_balance point upon sched domain changes

Commit 55ddeb0f (sched: Reset rq->next_interval before going idle) reset
a cpu's rq->next_balance when pulled_task = 0, which will be true when
the cpu failed to pull any task, causing it go idle. However that patch
relied on next_balance being calculated as a result of traversing cpu's
sched domain hierarchy.

A cpu that is the only online cpu will however not be attached to any
sched domain hierarchy. When such a cpu calls into idle_balance(), we
will end up initializing next_balance to be 1sec away! Such a CPU will
defer load balance check for another 1sec, even though we may bring up
more cpus in the meantime requiring it to check for load imbalance more
frequently. This could then lead to increased scheduling latency for
some tasks.

This patch results in a cpu's next_balance being re-calculated when its
attaching to a new sched domain hierarchy.  This should let cpus call
load balance checks at the right time we expect them to!

Change-Id: I855cff8da5ca28d278596c3bb0163b839d4704bc
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
sched: Make sure to not re-read variables after validation

We could re-read rq->rt_avg after we validated it was smaller than
total, invalidating the check and resulting in an unintended negative.

Change-Id: I8543974aad539107768e9e513ca3a8c4cb79b2ff
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: David Rientjes <rientjes@google.com>
Link: http://lkml.kernel.org/r/1337688268.9698.29.camel@twins
Signed-off-by: Ingo Molnar <mingo@kernel.org>
CRs-Fixed: 497236
Git-commit: b654f7de41b0e3903ee2b51d3b8db77fe52ce728
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
sched: Convert BUG_ON()s in try_to_wake_up_local() to WARN_ON_ONCE()s

try_to_wake_up_local() should only be invoked to wake up another
task in the same runqueue and BUG_ON()s are used to enforce the
rule. Missing try_to_wake_up_local() can stall workqueue
execution but such stalls are likely to be finite either by
another work item being queued or the one blocked getting
unblocked.  There's no reason to trigger BUG while holding rq
lock crashing the whole system.

Convert BUG_ON()s in try_to_wake_up_local() to WARN_ON_ONCE()s.

Change-Id: I75fdaaf4dcaefcf3893e0404d98c8aa91f89934d
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Steven Rostedt <rostedt@goodmis.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20130318192234.GD3042@htj.dyndns.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 383efcd00053ec40023010ce5034bd702e7ab373
CRs-Fixed: 519942
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
sched: change WARN_ON_ONCE to WARN_ON in try_to_wake_up_local()

The WARN_ON_ONCE() calls at the beginning of try_to_wake_up_local()
were recently converted from BUG_ON() calls. If these hit it indicates
something is wrong and that may contribute to other system instability.
To eliminate the risk of an instance of one of these errors going
un-noticed because there was an earlier instance that occured long ago,
change to WARN_ON(). If there ever is a flood of these there are bigger
problems.

Change-Id: I392832e2b6ec24b3569b001b1af9ecd4ed6828e7
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
sched: Unthrottle rt runqueues in __disable_runtime()

migrate_tasks() uses _pick_next_task_rt() to get tasks from the
real-time runqueues to be migrated. When rt_rq is throttled
_pick_next_task_rt() won't return anything, in which case
migrate_tasks() can't move all threads over and gets stuck in an
infinite loop.

Instead unthrottle rt runqueues before migrating tasks.

Additionally: move unthrottle_offline_cfs_rqs() to rq_offline_fair()

Change-Id: If8a4a399f1a14b7f4789c1b205dcfadbde555214
Signed-off-by: Peter Boonstoppel <pboonstoppel@nvidia.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Turner <pjt@google.com>
Link: http://lkml.kernel.org/r/5FBF8E85CA34454794F0F7ECBA79798F379D3648B7@HQMAIL04.nvidia.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Git-commit: a4c96ae319b8047f62dedbe1eac79e321c185749
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
mm: change initial readahead window size calculation

Change the logic which determines the initial readahead window size
such that for small requests (one page) the initial window size
will be x4 the size of the original request, regardless of the
VM_MAX_READAHEAD value. This prevents a rapid ramp-up
that could be caused due to increasing VM_MAX_READAHEAD.

Change-Id: I93d59c515d7e6c6d62348790980ff7bd4f434997
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
mm: pass readahead info down to the i/o scheduler

Some i/o schedulers (i.e. row-iosched, cfq-iosched) deploy an idling
algorithm in order to be better synced with the readahead algorithm.
Idling is a prediction algorithm for incoming read requests.

In this patch we mark pages which are part of a readahead window, by
setting a newly introduced flag. With this flag, the i/o scheduler can
identify a request which is associated with a readahead page. This
enables the i/o scheduler's idling mechanism to be en-sync with the
readahead mechanism and, in turn, can increase read throughput.

Change-Id: I0654f23315b6d19d71bcc9cc029c6b281a44b196
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
mm: vmalloc: use const void * for caller argument

'const void *' is a safer type for caller function type. This patch
updates all references to caller function type.

Change-Id: If950cfcfc63911756ac3709c8bf6da10c8b98f1b
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: Kyungmin Park <kyungmin.park@samsung.com>
Reviewed-by: Minchan Kim <minchan@kernel.org>
Git-commit: 5e6cafc83e30f0f70c79a2b7aef237dc57e29f02
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
ARM: DMA: Fix struct page iterator in dma_cache_maint() to work with sparsemem

Subhash Jadavani reported this partial backtrace:
  Now consider this call stack from MMC block driver (this is on the ARMv7
  based board):

  [<c001b50c>] (v7_dma_inv_range+0x30/0x48) from [<c0017b8c>] (dma_cache_maint_page+0x1c4/0x24c)
  [<c0017b8c>] (dma_cache_maint_page+0x1c4/0x24c) from [<c0017c28>] (___dma_page_cpu_to_dev+0x14/0x1c)
  [<c0017c28>] (___dma_page_cpu_to_dev+0x14/0x1c) from [<c0017ff8>] (dma_map_sg+0x3c/0x114)

This is caused by incrementing the struct page pointer, and running off
the end of the sparsemem page array.  Fix this by incrementing by pfn
instead, and convert the pfn to a struct page.

Change-Id: I52966616d710420aa28529ba650e9177dcb0e731
Cc: <stable@vger.kernel.org>
Suggested-by: James Bottomley <JBottomley@Parallels.com>
Tested-by: Subhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>