Package: curldl
IntroductionThecurldlPython module safely and reliably downloads files withPycURL, which in turn is a wrapper forlibcurlfile transfer library. The purpose ofcurldlis providing a straightforward API for downloading files with the following features:Multi-protocol support: protocol support is delegated tocurlin as protocol-neutral way as possible. This means that there is no reliance on HTTP-specific header and statuses, for example. If a feature like download resuming andif-modified-sincecondition is supported by the underlying protocol, it can be used bycurldl.If a partial download is abandoned, most chances are that it may be resumed later (supported for HTTP(S), FTP(S) and FILE protocols). A.partextension is added to the partial download file, and it is renamed to the target file name once the download completes.If the downloaded file exists, it is not downloaded again unless the timestamp of file on server is newer (supported for HTTP(S), FTP(S), RTSP and FILE protocols). Note that file download may be skipped before timestamp is considered due to file size matching the expected file size (see below).Downloads are configured relative to a base directory, and relative download path is verified not to escape the base directory directly, via symlinks, or otherwise.Downloaded file size and/or cryptographic digest(s) can be verified upon download completion. This verification, together with the relative path safety above, allows for easy implementation of mirroring scripts — e.g., when relative path, file size and digest are located in a downloaded XML file.Speed: since nativelibcurlwrites directly to the output stream file descriptor, there are no transfers of large chunks of data inside Python interpreter.UsageMost examples below use thecurldlwrapper script instead of Python code. Of course, in all cases it is easy to write a few lines of code with identical functionality — see the first example. Also, note that inline documentation is available for all functions.Simple DownloadThe following code snippet downloads a file and verifies its size and SHA-1 digest. A progress bar is shown onstderrwhile download is in progress.importcurldl,osdl=curldl.Curldl(basedir="downloads",progress=True)dl.get("https://kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gz","linux-0.01.tar.gz",size=73091,digests={"sha1":"566b6fb6365e25f47b972efa1506932b87d3ca7d"})assertos.path.exists("downloads/linux-0.01.tar.gz")If verification fails, the partial download is removed; otherwise it is renamed to the target file after being timestamped withlast-modifiedtimestamp received from the server.A similar result is achieved on command-line by running the CLI wrapper script, which is useful for quickly testingcurldlfunctionality:curldl-bdownloads-s73091-asha1-d566b6fb6365e25f47b972efa1506932b87d3ca7d\-p-ldebughttps://kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gzThe corresponding (redacted) log output:Saving download(s) to: linux-0.01.tar.gz
Creating directory: downloads
Downloading https://kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gz
            to downloads/linux-0.01.tar.gz.part
Finished downloading downloads/linux-0.01.tar.gz.part 0 -> 73,091 B
            (HTTPS 200: OK) [0:00:01]
Timestamping downloads/linux-0.01.tar.gz.part with 1993-10-30 00:00:00+00:00
Successfully verified file size of downloads/linux-0.01.tar.gz.part
Successfully verified SHA1 of downloads/linux-0.01.tar.gz.part
Partial download of downloads/linux-0.01.tar.gz passed verification
            (73091 / {'sha1': '566b6fb6365e25f47b972efa1506932b87d3ca7d'})
Moving downloads/linux-0.01.tar.gz.part to downloads/linux-0.01.tar.gzNote that renaming ofdownloads/linux-0.01.tar.gz.parttodownloads/linux-0.01.tar.gzis the very last action ofCurldl.get()method. If the target filename exists, the download succeeded and passed verification, if requested.Repeated DownloadRunning the same command again doesn't actually result in a server request since file size matches (digest is not checked since it would be time-prohibitive when mirroring large repositories):Saving download(s) to: linux-0.01.tar.gz
Skipping update of downloads/linux-0.01.tar.gz since it has
         the expected size 73,091 BWe can also request the same file without providing an expected size:importcurldldl=curldl.Curldl(basedir="downloads",progress=True)dl.get("ftp://ftp.hosteurope.de/mirror/ftp.kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gz","linux-0.01.tar.gz")In this case, the download is skipped due toIf-Modified-Sincecheck:Will update downloads/linux-0.01.tar.gz.part
     if modified since 1993-10-30 00:00:00+00:00
Discarding downloads/linux-0.01.tar.gz.part because
     it is not more recentNote that FTP protocol was used this time —curldlis entirely protocol-agnostic when using the underlyinglibcurlfunctionality.Resuming DownloadIf a download is interrupted, it will be resumed on the next attempt (which may also be a retry according to the configured retry policy). Here is what happens whenCtrl-Cis used to send a SIGINT signal to the Python process. This example also demonstrates how to construct a filename from a URL (CLI interface does the same when--outputswitch is omitted).importcurldl,os,urllib.parseasuparsedl=curldl.Curldl(basedir="downloads",progress=True)url="https://releases.ubuntu.com/22.04.2/ubuntu-22.04.2-live-server-amd64.iso"filename=os.path.basename(uparse.unquote(uparse.urlparse(url).path))dl.get(url,filename)The corresponding (redacted) log output:Downloading https://releases.ubuntu.com/22.04.2/ubuntu-22.04.2-live-server-amd64.iso
            to downloads/ubuntu-22.04.2-live-server-amd64.iso.part
ubuntu-22.04.2-live-server-amd64.iso:  13%|██▋                  | 244M/1.84G
            [00:06<00:38, 44.7MB/s] ^C KeyboardInterrupt:
Download interrupted downloads/ubuntu-22.04.2-live-server-amd64.iso.part 0 -> 259,981,312 B
            (42: Callback aborted / HTTPS 200: OK) [0:00:07]Attempting the same download again resumes the download:Resuming download of https://releases.ubuntu.com/22.04.2/ubuntu-22.04.2-live-server-amd64.iso
         to downloads/ubuntu-22.04.2-live-server-amd64.iso.part at 259,981,312 B
Finished downloading downloads/ubuntu-22.04.2-live-server-amd64.iso.part
         259,981,312 -> 1,975,971,840 B (HTTPS 206: Partial Content) [0:01:24]Note, however, that we didn't provide a size or digest for verification. Since the downloaded file is timestamped only once download completes, how doescurldlknow that the file wasn't changed on the server in the meantime? The answer is thatcurldlsimply avoids removing large partial downloads in such cases — see documentation foralways_keep_part_bytesconstructor parameter ofCurldl.Enabling Additional ProtocolsBy default,curldlenables the following protocols:HTTP(S)FTP(S)SFTPIn order to enable a different set of protocols, use theallowed_protocols_bitmaskconstructor argument. For instance, the code below downloads afile://URI:importcurldl,pycurl,pathlibprotocols=pycurl.PROTO_FILE|pycurl.PROTO_HTTPSdl=curldl.Curldl(basedir="downloads",allowed_protocols_bitmask=protocols)file_uri=pathlib.Path(__file__).absolute().as_uri()dl.get(file_uri,"current_source.py")To enable all protocols, useallowed_protocols_bitmask=pycurl.PROTO_ALL. Note, however, that there might be security repercussions.Escaping Base DirectoryAttempts to escape base directory are prevented, e.g.:importcurldl,osdl=curldl.Curldl(basedir=os.curdir)dl.get("http://example.com/",os.path.join(os.pardir,"file.txt"))The above results in:ValueError: Relative path ../file.txt escapes base path /home/user/curldlcurldlperforms extensive checks to prevent escaping the base download directory — seeFileSystemclass implementation and unit tests for details.InstallationThe only requirement forcurldlis Python 3.8+. Install the package as follows:pipinstallcurldlIf you encounter a build failure during installation ofpycurldependency, the following should help:On Linux, install one of:pycurlpackage from distribution repo — e.g., on Ubuntu runsudo apt install python3-pycurllibcurldevelopment files withsudo apt install build-essential libcurl4-openssl-devOn Windows, install an unofficialpycurlbuild since official builds are not available at the moment — e.g., fromChristoph Gohlke's packages, or useConda(see below).On Windows and macOS, useCondaorMinicondawithconda-forgechannel. For instance, see runtime dependencies in the followingtest environment.Overall,curldlis expected to not have any issues in any environment with Python 3.8+ (CPython or PyPy) — see theTestingsection below.TestingA simplified configuration matrix covered byCI/CD test + build pipelineat the time of writing this document is presented below:PlatformCPython 3.8CPython 3.9, PyPy 3.8+3.10CPython 3.10CPython 3.11Ubuntu-x64venv, conda, platformvenvvenv, platformvenv, condaWindows-x64venv, condavenv, condaWindows-x86venvvenvmacOS-x64condacondaIn the table:venv— virtual environment with all package dependencies andeditable package install; on Ubuntu includes tests with minimal versions of package dependencies;conda—Minicondawith package dependencies installed frommini-forgechannel, andcurldlas editable package install;platform— as many dependencies as possible satisfied via Ubuntu package repository, andcurldlaswheelinstall.The CI/CD pipeline succeeds only ifcurldlpackage successfully builds and passes all thepytesttest cases with 100%code coverage, as well asPylint,MypyandBanditstatic code analysis. Code style checks are also a part of the pipeline. Note that the testing code is also covered by these restrictions.DevelopmentPython EnvironmentThe following commands install and activate avenvenvironment in Linux, using the available Python 3 interpreter:./venv.shinstall-venv
.venv/bin/activatevenv.shis a conveniencevenvwrapper that also enables some additional Python checks; you can use it to run Python code, or just activate thevenvenvironment instead as shown above.Running TestsUsepytestin order to run all test cases:./venv.shpytestIn addition to the actual tests, pytest executesPylint,Mypy, code coverage and code formatting plugins (blackandisort). Thus, make sure that all new code is covered by tests.Testing withCondais possible as well — see theCI/CD pipeline executionfor details.Code FormattingReformat the code withblackandisortby running the following scripts:misc/scripts/run-black.sh
misc/scripts/run-isort.shThis is only necessary if the tests fail due to code formatting.Changelog EntriesUpon authoring a set of code or documentation changes, prepare a changelog fragment usingtowncrieras follows:towncriercreate-c"Extend package usage documentation"54.doc.mdThe command above creates a changelog fragment file indocs/changelog.d. See the output oftowncrier create --helpfor supported fragment types. Note also that54above is the number of GitHubpull request, which is used to format a pull request link when generating the combined changelog. So creation of new fragment needs to be doneafteropening a pull request.When releasing a new version, the changelog file can be updated as follows:towncrierbuild--version1.x.y[--draft]Add--draftoption for a dry run first, because otherwise fragment files will be removed, and changelog file extended with the new entries.ChangelogSee theChangelogfile for a summary of changes in each release.LicenseThis project is released under theGNU LGPL License Version 3or any later version.