Use defers when doing cleanup

Should protect me more thoroughly against panics that are handled by
the client.