Actually close down the connection when we time out.