Properly set follow and unfollow status