If you're injured all the time, you're a bad rider. Light people go slower on straight courses, but can hug turns on curvy courses and squeeze through small spaces between competitors. This can result in the spanking of big men. Lying on your back on a luge board, you cruise down a street with three to five competitors at speeds of 65 mph. We honestly are relatively safe.
Injuries usually tend to be a road-rash style thing.
Limbs that do break tend to be wrists and ankles. You can really control where you want to be. You can pull yourself out of a dangerous situation. Granted, there are always things out of your control, but I'm very rarely injured for how often I ride. If you're injured all the time, you're a bad rider.