epoll: optimize EPOLL_CTL_DEL using rcu

Date	Tue, 1 Oct 2013 17:08:10 +0000 (GMT)

Optimize EPOLL_CTL_DEL such that it does not require the 'epmutex' by
converting the file->f_ep_links list into an rcu one. In this way, we can
traverse the epoll network on the add path in parallel with deletes. Since
deletes can't create loops or worse wakeup paths, this is safe.

This patch in combination with the patch "epoll: Do not take global 'epmutex'
for simple topologies", shows a dramatic performance improvement in
scalability for SPECjbb.

Tested-by: Nathan Zimmer <nzimmer@sgi.com>
Signed-off-by: Jason Baron <jbaron@akamai.com>

backported for Linux 3.4 for use on Android
Signed-off-by: Paul Reioux <reioux@gmail.com>
epoll: Do not take global 'epmutex' for simple topologies

Date	Tue, 1 Oct 2013 17:08:14 +0000 (GMT)

When calling EPOLL_CTL_ADD for an epoll file descriptor that is attached
directly to a wakeup source, we do not need to take the global 'epmutex',
unless the epoll file descriptor is nested. The purpose of taking
the 'epmutex' on add is to prevent complex topologies such as loops and
deep wakeup paths from forming in parallel through multiple EPOLL_CTL_ADD
operations. However, for the simple case of an epoll file descriptor
attached directly to a wakeup source (with no nesting), we do not need
to hold the 'epmutex'.

This patch along with 'epoll: optimize EPOLL_CTL_DEL using rcu' improves
scalability on larger systems. Quoting Nathan Zimmer's mail on SPECjbb
performance:

"
On the 16 socket run the performance went from 35k jOPS to 125k jOPS.
In addition the benchmark when from scaling well on 10 sockets to scaling well
on just over 40 sockets.

...

Currently the benchmark stops scaling at around 40-44 sockets but it seems like
I found a second unrelated bottleneck.
"

Tested-by: Nathan Zimmer <nzimmer@sgi.com>
Signed-off-by: Jason Baron <jbaron@akamai.com>

backported to Linux 3.4 for use on Android
Signed-off-by: Paul Reioux <reioux@gmail.com>
epoll: do not take globals epmutex for simple topologies fix

- use `bool' for boolean variables
- remove unneeded/undesirable cast of void*
- add missed ep_scan_ready_list() kerneldoc

Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Davide Libenzi <davidel@xmailserver.org>
Cc: Eric Wong <normalperson@yhbt.net>
Cc: Jason Baron <jbaron@akamai.com>
Cc: Nathan Zimmer <nzimmer@sgi.com>
Cc: Nelson Elhage <nelhage@nelhage.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched/rt: Remove redundant nr_cpus_allowed test

Date	Fri, 4 Oct 2013 14:24:53 -0500

From: Shawn Bohrer <sbohrer@rgmadvisors.com>

In 76854c7e8f3f4172fef091e78d88b3b751463ac6 "sched: Use
rt.nr_cpus_allowed to recover select_task_rq() cycles" an optimization
was added to select_task_rq_rt() that immediately returns when
p->nr_cpus_allowed == 1 at the beginning of the function.  This makes
the latter p->nr_cpus_allowed > 1 check redundant and can be removed.
Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: Reduce overestimating rq->avg_idle

Date	Thu, 29 Aug 2013 13:05:34 -0700

When updating avg_idle, if the delta exceeds some max value, then avg_idle
gets set to the max, regardless of what the previous avg was. This can cause
avg_idle to often be overestimated.

This patch modifies the way we update avg_idle by always updating it with the
function call to update_avg() first. Then, if avg_idle exceeds the max, we set
it to the max.

Signed-off-by: Jason Low <jason.low2@hp.com>
Reviewed-by: Rik van Riel <riel@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>