No, your comments will not decrease. Even if your real followers unfollow you, their comments will remain.