No. When you remove a follower from your Instagram account, the person isn't notified. The only way they would find out is if they went to view your profile and noticed the active Follow button. Also, if your profile is set to private, they will be unable to see your posts or Stories.