Basic crawler and downloader implementation. Can crawl and look for files with a specific extension, download them and add them to an index.