Remove finalizer safety as it adds 2.5x slowdown

For now this is removed and no debug assistance is provided. Later this
may be replaced with weak (pre-finalizer) or phantom (post-finalizer)
reference collection.