Package: imgdl
Python package for downloading a collection of images from a list of
urls. It comes with the following features:Downloads are multithreaded usingconcurrent.futures.Relies on a persistent cache. Already downloaded images are not
downloaded again, unless you forceimgdlto do so.It can be used as a command line utility or as a python library.Normalizes images to JPG format + RGB mode after download.Generates thumbnails of varying sizes automatically.Can space downloads with a random timeout drawn from an uniform
distribution.InstallationpipinstallimgdlOr, from the root project directory:pipinstall.UsageHere is a simple example using the default configurations:fromimgdlimportdownloadurls=['https://upload.wikimedia.org/wikipedia/commons/9/92/Moh_%283%29.jpg','https://upload.wikimedia.org/wikipedia/commons/8/8b/Moh_%284%29.jpg','https://upload.wikimedia.org/wikipedia/commons/c/cd/Rostige_T%C3%BCr_P4RM1492.jpg']paths=download(urls,store_path='~/.datasets/images',n_workers=50)100%|███████████████████████████████████|3/3 [00:08<00:00,  2.68s/it]Images will be downloaded to~/.datasets/imagesusing 50 threads.
The function returns the list of paths to each image. Paths are
constructed as{store_data}/{SHA1-hash(url).jpg}. If for any reason a
download fails,imgdlreturns aNoneas path.Notice that if you invokedownloadagain with the same urls, it
will not download them again as it will check first that they are
already downloaded.paths=download(urls,store_path='~/.datasets/images',n_workers=50)100%|████████████████████████████████|3/3 [00:00<00:00, 24576.00it/s]Download was instantaneous! andimgdlis clever enough to return
the image paths.Here is the complete list of parameters taken bydownload:iterator: The only mandatory parameter. Usually a list of urls,
but can be any kind of iterator.store_path: Root path where images should be storedn_workers: Number of simultaneous threads to usetimeout: Timeout that the url request should toleratethumbs: If True, create thumbnails of sizes according to
thumbs_sizethumbs_size: Dictionary of the kind {name: (width, height)}
indicating the thumbnail sizes to be created.min_wait: Minimum wait time between image downloadsmax_wait: Maximum wait time between image downloadsforce:downloadchecks first if the image already exists onstore_pathin order to avoid double downloads. If you want to
force downloads, set this to True.Most of these parameters can also be set on aconfig.yamlfile found
on the directory where the Python process was launched. Seeconfig.yaml.exampleCommand Line InterfaceIt can also be used as a command line utility:$imgdl--helpusage:imgdl[-h][-oSTORE_PATH][--thumbsTHUMBS][--n_workersN_WORKERS][--timeoutTIMEOUT][--min_waitMIN_WAIT][--max_waitMAX_WAIT][-f]urlsBulkimagedownloaderfromalistofurlspositionalarguments:urlsTextfilewiththelistofurlstobedownloadedoptionalarguments:-h,--helpshowthishelpmessageandexit-oSTORE_PATH,--store_pathSTORE_PATHRootpathwhereimagesshouldbestored(default:~/.datasets/imgdl)--thumbsTHUMBSThumbnailsizetobecreated.Canbespecifiedasmanytimesasthumbssizesyouwant(default:None)--n_workersN_WORKERSNumberofsimultaneousthreadstouse(default:50)--timeoutTIMEOUTTimeouttobegiventotheurlrequest(default:5.0)--min_waitMIN_WAITMinimumwaittimebetweenimagedownloads(default:0.0)--max_waitMAX_WAITMaximumwaittimebetweenimagedownloads(default:0.0)-f,--forceForcethedownloadevenifthefilesalreadyexists(default:False)AcknowledgementsImages used for tests are from thewikimedia commons