If the players are trying to kick the ball while goalie is trying to control, I blow the whistle and signal the goalie to punt the ball. I had no injury goalie. Also if any player strikes the goalie while running in order to try to reach the ball, I immediatly yellow card, and if in my opinion it is reckless and out of control, then I red card them. The card would be for unsporting behaviour since nothing else really applies.
Soccer is a contact sport, but there are limitations.
In that case the red card could be given for serious foul play, if it was particulary violent or dangerous, and if a direct free kick is not punishment enough. Giving a yellow then a red is not really needed. Just giving my opinion on, and elaborating on, what you were saying. As long as you ar consistent with the call on both sides of the field then you wont have a problem. I was taught this philosophy by National Referees from all over.