Implements a multi-threaded crawler for data collection, collation and processing.