This is just to make sure if the connection is removed before sasl completes it gets cleaned up.