Make sure handles get registered, and also clean up invalid handles