There are going to be more people who just see the game in a different light. Precedents for ethical, self-regulating behavior in sports do exist. Most people play tennis without referees, so as a result the tradition evolved the same as golf. An honorable golfer or tennis player will make a call against himself. Each game has a penumbra of rules that are the traditions of the game.
Rules define several things.
They not only define what is fair, they really define what the game prizes or honors. Boxing prizes brute force. That's the nature of the game. Knock a person out. But it doesn't allow you to hit below the belt, so it still has rules.