Updated multithread crawling for hadoop