Make sure SocketGrabber releases sockets early.

This fixes a Problem in combination with G_NonBlockingMultiThreadedSelectorServer.