Not hitting the ball straight can lead to various issues, including landing in hazards, going off-course, and potentially increasing the number of strokes needed to complete a hole. It can also lead to frustration and a lack of confidence in your game.