Yes, if a large percentage of your links are spammy or have low domain authority, Google sees this as a negative indication of your own site's authority. This means Google may penalize your rankings.