integrated grequests, moved to lxml for performance and began using redis pipelining