Crawler is designed for flexible crawling across different CMSs and content formats. Crawler is capable of finding and indexing content from any public-facing website. It uses on-page links and your site’s XML sitemap to find all of the pages within your domain and can also extract data from common file formats such as PDFs, PowerPoints, Excel spreadsheets, Word documents and similar rich text formats.