epoll: optimize EPOLL_CTL_DEL using rcu

Date	Tue, 1 Oct 2013 17:08:10 +0000 (GMT)

Optimize EPOLL_CTL_DEL such that it does not require the 'epmutex' by
converting the file->f_ep_links list into an rcu one. In this way, we can
traverse the epoll network on the add path in parallel with deletes. Since
deletes can't create loops or worse wakeup paths, this is safe.

This patch in combination with the patch "epoll: Do not take global 'epmutex'
for simple topologies", shows a dramatic performance improvement in
scalability for SPECjbb.

Tested-by: Nathan Zimmer <nzimmer@sgi.com>
Signed-off-by: Jason Baron <jbaron@akamai.com>

backported for Linux 3.4 for use on Android
Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: pinkflozd <lk@varjanta.com>
epoll: Do not take global 'epmutex' for simple topologies

Date	Tue, 1 Oct 2013 17:08:14 +0000 (GMT)

When calling EPOLL_CTL_ADD for an epoll file descriptor that is attached
directly to a wakeup source, we do not need to take the global 'epmutex',
unless the epoll file descriptor is nested. The purpose of taking
the 'epmutex' on add is to prevent complex topologies such as loops and
deep wakeup paths from forming in parallel through multiple EPOLL_CTL_ADD
operations. However, for the simple case of an epoll file descriptor
attached directly to a wakeup source (with no nesting), we do not need
to hold the 'epmutex'.

This patch along with 'epoll: optimize EPOLL_CTL_DEL using rcu' improves
scalability on larger systems. Quoting Nathan Zimmer's mail on SPECjbb
performance:

"
On the 16 socket run the performance went from 35k jOPS to 125k jOPS.
In addition the benchmark when from scaling well on 10 sockets to scaling well
on just over 40 sockets.

...

Currently the benchmark stops scaling at around 40-44 sockets but it seems like
I found a second unrelated bottleneck.
"

Tested-by: Nathan Zimmer <nzimmer@sgi.com>
Signed-off-by: Jason Baron <jbaron@akamai.com>

backported to Linux 3.4 for use on Android
Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: pinkflozd <lk@varjanta.com>
epoll: do not take globals epmutex for simple topologies fix

- use `bool' for boolean variables
- remove unneeded/undesirable cast of void*
- add missed ep_scan_ready_list() kerneldoc

Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Davide Libenzi <davidel@xmailserver.org>
Cc: Eric Wong <normalperson@yhbt.net>
Cc: Jason Baron <jbaron@akamai.com>
Cc: Nathan Zimmer <nzimmer@sgi.com>
Cc: Nelson Elhage <nelhage@nelhage.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: pinkflozd <lk@varjanta.com>