Package: fastdownload
fastdownloadEasily download, verify, and extract archivesIf you have datasets or other archives that you want to make available to your users, and ensure they always have the latest versions and that they are downloaded correctly,fastdownloadcan help.InstallUsing pip:pip install fastdownload...or using conda:conda install -c fastai fastdownloadWhat's this about?The situation where you might want to usefastdownloadis where you have one or more URLs pointing at some archives you want to make available, and you want to ensure that your users download those archives correctly, have the latest version, and that it's as easy as possible for them to access the information in those archives.Your user just calls a single method,FastDownload.get, passing the URL required, and the URL will be downloaded and extracted to the directories you choose. The path to the extracted file is returned. If that URL has already been downloaded, then the cached archive or contents will be used automatically. However, if that size or hash of the archive is different to what it should be, then the user will be informed, and a new version will be downloaded.In the future, you may want to update one or more of your archives. When you do so,fastdownloadwill ensure your users have the latest version, by checking their downloaded archives against your updated file size and hash information.For instance,fastaiusesfastdownloadto provide access to datasets for deep learning.fastaiusers can download and extract them with a single command, using the return value to access the files. The files are automatically placed in appropriate subdirectories of a.fastaifolder in the user's homedir. If a dataset is updated, users are informed the next time they use the dataset, and the latest version is automatically downloaded and extracted for them.Usage: downloading filesWhen your users download an archive,fastdownloadwill automatically save it to a directory, check if the size and hash matches, and extract the contents. Minimal usage for downloading and extracting is:fromfastdownloadimportFastDownloadd=FastDownload()path=d.get('https://...')After this,pathwill contain the path where the extracted files are located. By default, archives are saved to{base}/archive, and extracted to{base}/data.{base}defaults to~/.fastdownload. If there is more than one file or folder in the root of the downloaded archive, then a new folder is created indatafor the contents.Instead ofget, usedownloadto download the URL without extracting it, orextractto extract the URL without downloading it (assuming it's already been downloaded to thearchivedirectory). All of these methods accept aforceparameter which will download/extract the archive even if it's already present.You can change any or all of thebase,archive, anddatapaths by passing them toFastDownload:d=FastDownload(base='~/.mypath',archive='downloaded',data='extracted')You can remove the cached archive file and/or the extracted contents withrm:d.rm('https://...')Usage: making archives available to downloadfastdownloadwill add a filedownload_checks.pyto your Python module which contains file sizes and hashes for your archives. The file is located in the same directory as a module you choose, e.g.:d=FastDownload(module=fastai.some_module)Then useupdateto create or update the size and hash for a URL:d.update('https://...')You will now find there is a file calleddownload_checks.pyin the same directory wherefastai.some_moduleis located, which contains a Python dict with the URL, size, and hash for this file. If you've downloaded this file before to yourarchivepath then it will be used, instead of downloading a new copy. Useget(force=True)first to download a new copy if even you have it in your archive.Config fileIf there is a file calledconfig.iniin yourbasedirectory, then keysarchiveanddatawill be used as the default values forFastDownload. The file should be inconfigparserformat. Here's a sampleconfig.ini:[DEFAULT]         
archive = downloaded
data = extractedIf there is no ini file present, one will be automatically created for for you using the details you pass toFastDownload.You can add any additional key/value pairs to the config file that you want. When you callFastDownload.getpassextract_keyto use a key other thandatafor choosing a location to extract to.