Fix a bug with removal and unrelated orphan packages

They would still be put in the "to_keep" set, and trigger a solver failure