Only try to add labels if there are some to add