Check cached followers before autofollow