ironport/shrapnel — pull request #15 by halayli (2012-04-12 07:36:22+00:00)
Title: Fix linux poller

The gist of this merge is the following commit:

https://github.com/halayli/shrapnel/commit/e07c1412c903a40caa1247ac2c1a783bc10fc92b

I'd like to have more eyes on it before I commit.

In short: epoll royally sucks.

The longer version: epoll doesn't allow 2 separate events to be
maintained per fd. If we register an fd for read, and then decide
to register it for write too, the later will overwrite the original
registered read event unless we register READ|WRITE. It seems
harmless but it's not. When we register for READ|WRITE and only
a READ event arrives, we first receive the READ event, and in the
next epoll_wait if a READ + WRITE event occurs we'll receive READ|WRITE
because the original registered event is READ|WRITE. What this means
is that we received READ twice here, once as READ and the second
time when a WRITE & READ occured, READ|WRITE. We were not expecting
the second READ event and w'ell have to ignore it.

This behavior happens because we are not passing EPOLLONESHOT. The
reason we don't want to do this is that the EPOLLONESHOT will get applied
to both events (read + write) when we only need it for one. Ex: If we
register for a READ event, followed by registering WRITE event
READ|WRITE|EPOLLONESHOT this will mean if either of the READ or WRITE
event occurs it will be EPOLLONESHOT and the second event will need
to be registered. I certainly don't want to track such events and
reregister them. Hence I am going with the first solution
by not use EPOLLONESHOT and ignore events I am not expecting.

If only epoll supports fd+event as a key rather than fd only.