adds documents per second and improves single document crawling.