The answer is no! ABEC stands for Annular Bearings Engineers Committee. The rating is based on using the bearing in an engine or power tool. A skateboard wheel will never spin so fast that the rating is irrelevant for skateboarding.
What makes a bearing faster and efficient is the materials used and how precisely the bearing is manufactured.  The amount of balls in the bearing is for different reasons. The more balls you have, the stronger it is. More balls mean also more friction, and that means you probably go slower. So in this case, less is more (speed).