progress and failed urls are written in the output directory