Do not callback if we manually close the connection