Possibly as big as any other single component. Someone with a small hand may need to switch to a thinner grip to accommodate their hand size. A big handed person as well may need a larger grip to properly seat the gun in their hand. What it all boils down to is personal preferences. I prefer a slightly oversized wood grip that does not interfere with the line of the gun.
Heavy calibers can present a problem when it comes to grip design.
The Casull, for instance, is a gun that doesn't seem to benefit from a rubber type grip, although probably some would disagree. Larry is an accomplished handgun hunter and silhouette competitor. His shop specializes in pistol-smithing, and he assisted me recently in getting my Super Blackhawk in shape for hunting. He told, in his opinion, on a heavy caliber that a wood grip is a better choice. The wood lets the gun slip slightly in your hand, thus absorbing some of the power and making the gun shoot more comfortably.