Don't add handles of 0 to the handle set