Remove extra condition on calling disconnect callback.

We've closed the connection, EOF or error, who cares?