Added log4j dependency
Buffer multiple lines of data rather than scanning lines individually
Use IntWritable rather than LongWritable for efficiency
Discover crawl segments by reading index file from S3