The right size board depends on a riders weight, style, athletic ability, and experience. Let's look at them individually. Your weight may be the most significant factor in determining what size board you should ride. Big guys like to skim too, and they usually need bigger boards to do it. A heavier rider will be able to turn a large board more easily and often need the extra carry to get out to the waves.
Small guys can often get away with a smaller board.
Many board manufacturers offer weight recommendations for their different models. This is a good starting point, but the other factors can matter just as much, or more. Notice that many of the size recommendations have overlapping weights. Under 50 lbs. Small 50 - 80 lbs.