Avoid asserts to survive socket being closed while waiting