This number is set by the 'time signature'. I'll use some graphs to explain, with time flowing left to right. Most music is written in 4's. Human brains seem to like the repetition of rhythms in groups of 4. They can change anytime at all but to be able to be enjoyed by the masses, a certain regularity and even predictability must be included.
Most people prefer to listen to music they can tap along to.
Most of us seem to have a deep-rooted understanding of rhythm, something that goes way back. Strange time signatures thwart that foot tapping. Symetry seems to be the prerequisite for that to happen naturally. This is the way I count it when I have to. Often it's done to follow a lyric or grab the attention.