I don't have any problem with Goliaths, to be honest. They're my favorite.
If you do make them more rare, I'd suggest even buffing them to make them MORE of a challenge.