Our crawler is a Screaming Frog, which is operated in the cloud. The crawls are started automatically on a monthly, weekly or daily basis, depending on the settings. The crawling results are automatically written to Google Big Query and then queried or displayed via Google Data Studio.