single thread version without obeying robots.txt