Remove unfollow handling code

Since it doesn't work anyway