ref/unref-ing socket as they become used/idle