A severely worn out CV joint can even disintegrate while you're driving and make the car undrivable. You may lose control of the vehicle entirely. It is not safe to drive with a damaged CV joint.