Ahrefs crawls the web on a regular basis to collect data. They are discovering about 40 million new pages every 24 hours and updating the metrics of over 300 million pages daily. They are crawling about 5 million pages per minute. That's how Ahrefs is getting data.