Cleanup and send close from server only once