Don't panic, handle closed better internally

Signed-off-by: Jonathan Rudenberg <jonathan@titanous.com>
Fix race in test, simplify