Regardless of how fast you plan to go, speed is affected first and foremost by the choice of lubricant. If we're going to talk about tolerances, the fit of your wheels and axles have a much greater effect on performance than ABEC rating. Wheels and axles for inline skates have extremely loose fits that allow you to press the bearings into the wheel by hand. This masks the benefits of a higher precision bearing by allowing it to slip on the axle or in the wheel. Slippage between the mating parts results in energy loss.
Lost energy is lost speed.
The two most common lubricants are grease and oil. Grease is basically oil with a thickener or soap. The thickener acts like a sponge to soak up the oil when not in use. As grease is thicker, it acts as a seal against dirt, but at the same time it can increase torque and slow down the bearing.