Destroy client handles after they're closed