Package: paralleldownload
ParalleldownloadDownload large number of files extremely fast.A Python Package for Fast Parallel Download of Multiple Files.
It's Simple, easy and extremely fast since it uses, all the cores in your CPU to spin separate process for parallel download.
Paralleldownload usesrequestsas its only dependency, which is almost always present in all python environment.Python Package Index BadgesGithub BadgesPython Package Index Installpip install paralleldownloadUsage:Basic example:$paralleldownloadinput_url_file.txtIt downloads the files using the urls in the file. Each url in a line.The downloaded files is stored in current directory.Uses number of process equal to number of CPU Cores in the machine.Getting help, info, version and example:$paralleldownload[-h|-i|-v|-eg]These options will just print text and exits.-hPrints the help message.-iPrints information aboout the package.-vPrints current version of the package.-egPrints few example of how to use the this cli.Specify save directory:$paralleldownloadinput_url_file.txtdownloads_directoryThe downloaded files is stored indownloads_directorydirectory.You can provide absolute or relative paths for both url file and save directory.Use N Processes:$paralleldownloadinput_url_file.txt-p17Uses 17 processes to download files.Default is equal to the number of cpu core count in the machine.File names:$paralleldownloadinput_url_file.txt[-u|-n|-a]By default, it searches for the file name in the response. If found, it will use this name. Else it will use a uuid string as file name. [Yet to be implemented]-uAll downloaded files will be named as uuid strings[eg: 5b71113f-43be-40f5-b267-9b93919196aa.jpg]-nAll downloaded files will be named as sequential numbers[eg: 017.jpg]-aAll downloaded files will be named as sequential lowercase english alphabets[eg: exy.jpg]If extension is needed, they have to be manually provided.Specify Extension$paralleldownloadinput_url_file.txt-epngUses the provided extension in file names..(dot) is optional.If extension is needed, they must be provided when using[-u | -n | -a].DescriptionDo you want to download thousands of files at once but can't wait for sequential download?Today's machines have multiple CPU cores.
Most entry level machines have 4 Cores while higher end machines have around 8 Cores,
Some desktop processor even have 16 - 32 Cores.
But using just one Core for downloading files is not the best approach if you have hundreds or thousands of files to download.The rapid shift towards cloud technologies provide massive processing power, GigaBit network and faster writes to disk.
By properly making use of this processing power, bandwidth, memory and IO, we can make our life a bit easier.paralleldownload is one of such package.
It is a cli tool used to download thousands of files in short time.
It achieves it by spinning seperate process per CPU core and downloading parallely.Imagine a Machine with 24 Core CPU and Gigabit Network.
The process to download 1000 files, and it takes 1 second to fetch each file,Sequential downloadPython Interpreter does the following steps.Make request  ➜ open a file ➜ save the content to the fileAfter it is done, Python Interpreter repeats the steps 1000 times sequentially.It takes1000 filesx1 Second=1000 Secondsor~17 Minutesto download all files.➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜Parallel download28 Python Interpreter is spun up in all of 28 cores (one per core by default).Each Python Interpreter does the following steps.Make request  ➜ open a file ➜ save the content to the fileAfter it is done, Each Python Interpreter repeats the steps 42 times (1000 files / 24 process).➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜
➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜
[.. 21 more processes ..]
➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜ .. ➜➜➜It takes(1000 files/24 Processes)x1 Second=42 Secondsor~1 Minutesto download all files.Note:This is just a logical comparison.
The time is not a real world example, many factors affects the download speeed.
There will be a slight overhead in setting up the process and collecting the work.
So, it will not download all in under a minute.
The Overhead should be negligable with higher input.