compiled list of downloaded links
altered archive.org blacklist
compiled list of downloaded links
really long extensions no longer allowed as in exteme cases they can result in a filename that is greater than the max allowed.
refactoring: moved some of BingSearchEngine's method to super class
Added ability to throttle the QueryThread. This will be important if we wish to run multiple downloader processes at once in order to maximise throughput without also overwhelming our network interface (or our threads performing the actual downloading), thereby hopefully avoiding cases where the server timesout due to our client being unresponsive.
MainDownloader now allows random seed to be specified from command line
updated README.md
Added Filter for MainDownloader class to allow us to partition downloads across a number of processes in order to further parallelise workload
updated README.md
refactoring: listAllTokenExpansions moved to IndexViewer
now download matains record of content-types from HttpGets