slurp down a whole HTTP directory, with parallel goodness