Well, it depends. If your goal is to have a really strong low-bar back squat, then yeah, exclusively squatting with the safety bar will definitely limit your potential. You must practice the competition movements in order to improve on them, at least some of – if not most of – the time.
However, if you just want to be big and strong as you possibly can, there’s no reason you need to ever squat with a straight bar!