Framework of Fastsocket module

The kernel module in Fastsocket is responsible to providing customized socket
interfaces via ioctl over a special device. This patch introduce the generic
framework of how this is done.

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Initialize Fastsocket FS

Fastsocket FS is a pseudo FS like sockfs. Sockets created by Fastsocket will be
attached to this FS.

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Create and close sockets via /dev/fastsocket

The socket init/fini routines are mostly a simplified copy of their original
version in the kernel. The most severely-contented path (inode and dcache
insertion) are skipped, as well as some other routines for advanced features,
e.g. security. Please carefully check updates to the original routines when
porting this patch.

The hottest locks in VFS has been fixed in recent kernels. Refer to the
following commits for details.

a209dfc vfs: dont chain pipe/anon/socket on superblock s_inodes list
4b93688 fs: improve scalability of pseudo filesystems

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Listen/accept/epoll_ctl on Fastsocket sockets

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Spawn CPU-pinned local sockets

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Establish percpu listen socket tables

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Ensure locality of active connections using RFD

This patch introduces Receive Flow Deliver(RFD) which ensures locality of active
connections by encoding processor IDs to TCP ports. On receiving a packet, RFD
takes the place of RPS and steers the packet according to the processor ID
encoded.

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Add an option to awake a single task on waitqueues at a time

With a large number of listeners on a single listen socket, waitqueues tend to
awake too many processes for a single incoming connection (known as the
thundering herd problem).

This patch provides an option WQ_FLAG_LOADBALANCE for waitqueues. When enabled,
this option makes waitqueues awake only one task at a time in a round robin
way. A file mode option FMODE_SINGLE_WAKEUP is also provided to mark files whose
events need be handled with WQ_FLAG_LOADBALANCE set.

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Cache epitem in files

This patch implements a cache of the epitem bound to a file in the file
structure in order to avoid overhead from epitem lookup. This works in most
cases for sockets, because sockets are rarely monitored by multiple processes
and thus only one epitem will be bound to sockets.

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Add statistics on percpu listen table at /proc/net/hash_stat

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Add statistics on packet steering at /proc/net/steer_stat

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Align bind hash buckets in cache

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Check loops before inserting an epitem

Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Fastsocket v1.1

Signed-off-by: Xiaofeng Lin <jerrylin.lxf@gmail.com>
Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Consult the epitem cache only if fast_epoll is enabled

Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Allocate security-related structures for inodes

Some security-related structures in inodes are required when SELinux is enforced
or permissive. When not properly initialized, dangling pointers to these
structures will lead to page faults and system crashes.

Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
epoll: optimize EPOLL_CTL_DEL using rcu

Nathan Zimmer found that once we get over 10+ cpus, the scalability of
SPECjbb falls over due to the contention on the global 'epmutex', which is
taken in on EPOLL_CTL_ADD and EPOLL_CTL_DEL operations.

Patch #1 removes the 'epmutex' lock completely from the EPOLL_CTL_DEL path
by using rcu to guard against any concurrent traversals.

Patch #2 remove the 'epmutex' lock from EPOLL_CTL_ADD operations for
simple topologies.  IE when adding a link from an epoll file descriptor to
a wakeup source, where the epoll file descriptor is not nested.

This patch (of 2):

Optimize EPOLL_CTL_DEL such that it does not require the 'epmutex' by
converting the file->f_ep_links list into an rcu one.  In this way, we can
traverse the epoll network on the add path in parallel with deletes.
Since deletes can't create loops or worse wakeup paths, this is safe.

This patch in combination with the patch "epoll: Do not take global 'epmutex'
for simple topologies", shows a dramatic performance improvement in
scalability for SPECjbb.

Signed-off-by: Jason Baron <jbaron@akamai.com>
Tested-by: Nathan Zimmer <nzimmer@sgi.com>
Cc: Eric Wong <normalperson@yhbt.net>
Cc: Nelson Elhage <nelhage@nelhage.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Davide Libenzi <davidel@xmailserver.org>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
CC: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
epoll: do not take global 'epmutex' for simple topologies

When calling EPOLL_CTL_ADD for an epoll file descriptor that is attached
directly to a wakeup source, we do not need to take the global 'epmutex',
unless the epoll file descriptor is nested.  The purpose of taking the
'epmutex' on add is to prevent complex topologies such as loops and deep
wakeup paths from forming in parallel through multiple EPOLL_CTL_ADD
operations.  However, for the simple case of an epoll file descriptor
attached directly to a wakeup source (with no nesting), we do not need to
hold the 'epmutex'.

This patch along with 'epoll: optimize EPOLL_CTL_DEL using rcu' improves
scalability on larger systems.  Quoting Nathan Zimmer's mail on SPECjbb
performance:

"On the 16 socket run the performance went from 35k jOPS to 125k jOPS.  In
addition the benchmark when from scaling well on 10 sockets to scaling
well on just over 40 sockets.

...

Currently the benchmark stops scaling at around 40-44 sockets but it seems like
I found a second unrelated bottleneck."

[akpm@linux-foundation.org: use `bool' for boolean variables, remove unneeded/undesirable cast of void*, add missed ep_scan_ready_list() kerneldoc]
Signed-off-by: Jason Baron <jbaron@akamai.com>
Tested-by: Nathan Zimmer <nzimmer@sgi.com>
Cc: Eric Wong <normalperson@yhbt.net>
Cc: Nelson Elhage <nelhage@nelhage.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Davide Libenzi <davidel@xmailserver.org>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Apply the same updates from epoll to Fastsocket

Signed-off-by: Junjie Mao <eternal.n08@gmail.com>
Fix a memory leak (as well as a potential crash)

Signed-off-by: Junjie Mao <eternal.n08@gmail.com>