Don't worry about handles that are referenced during connection close.

refds #844