Make sure that idle handler also catches connections which are idle for both reading and writing

This fixes problem of stale connections not getting closed after timeout.