In order to do so, go to the person’s profile on Facebook and block him. And then unblock him too. he shall get removed from your list of followers.