You can simply block them and then quickly unblock them. If you are not so unlucky, they will not see that you have blocked them in such a short time. In addition, this will make you both unfollow each other. In other words, the Twitter followers that you have removed from your followers list may found out that they are not following you anymore. However, they will not know that you have blocked them.