Only remove gem junk if it exists.