You should be concerned about persistent balance problems that affect daily activities or are accompanied by other symptoms like dizziness or vertigo.