Fix panic on unexpected conn close