Crawls a given directory and recursively extracts text and metadata from files.