Fix deletion & dereferencing.

There were issues introduced by #154 that caused panics under certain edge cases. This pull request resolves all known outstanding issues and adds quite a few assertions to protect against this type of thing in the future.

/cc @mkobetic @snormore