Only remove gem junk if it exists.
Merge pull request #28 from fedora-infra/feature/careful-with-the-gem-junk

Only remove gem junk if it exists.