Revert "Rewrite conn destructor to avoid failure"

We're already unwrapping in `deref`, so this isn't really a help.

This reverts commit 51b24e60b8057d03570073b6e586e68660c581e4.