Remove reliance on crawlers. Instead, use the json from the sites.