Remove handle when dispose is called for the first time
Revert "Remove handle when dispose is called for the first time"

This reverts commit 05d23bb40c3e4d409e748a77bddc556ce0281dff.
Merge pull request #1 from sharwell/jbialobr_vNext

Only call AddHandle when we know the handle is valid
Handle the case when IsInvalid will be called for the first time after call to ~LibraryLifetimeObject
Make sure handles get registered, and also clean up invalid handles