dardok/Equalizer — comment on commit 987a6e04e2897c23c142e4a8811c46030d1733af
File:  line 0
By dardok:

No problem, I just hope it ends up being the fix because I'm not sure what else it could be (but that's not unusual!). Holding my breath!

This is my current analysis: the critical region is ibv_poll_cq (which is non-blocking). Both read() and write() call this to service both the send & receive completion queues. Previously, after read() decided the queue was empty, write() could take new events off the queue before read() rearms the CQ and the rearm would then have no effect. Now, we prevent write from taking those events off until after we rearm. Taking the events off does *not* disarm the notifier (unlike "read" from a readable socket) - ibv_get_cq_event does. So we should now ensure that the notifier stays "hot" until there *really* is no data in the buffer. ibv_req_notify_cq (aka rearm) is a topic of much discussion if you look it up.