Package: hexagonit.recipe.download
ContentsDetailed DocumentationSupported optionsSimple exampleSharing packages between buildoutsMD5 checksumsControlling the extraction processPartial extraction over existing contentExcluding package contentsOffline modeDownloading arbitrary filesRe-running install on updateChange History1.7.1 (2016-02-13)1.7 (2013-04-07)1.6 (2012-11-14)1.5.1 (2012-07-13)1.5.0 (2011-01-26)1.4.1 (2010-04-08)1.4.0 (2010-04-06)1.3.0 (2009-09-20)1.2.2 (2009-08-17)1.2.1 (2008-04-13)1.2.0 (2008-01-19)1.1.0 (2007-10-14)1.0.1 (2007-08-14)1.0.0ContributorsDownloadThe recipe downloads packages from the net and extracts them on the
filesystem. It is based on thegocept.downloadrecipe with a few
additional features.Github page:http://github.com/hexagonit/hexagonit.recipe.downloadClone URL: git://github.com/hexagonit/hexagonit.recipe.download.gitIssue tracker:http://github.com/hexagonit/hexagonit.recipe.download/issuesTravis build:Supported Python versions: 2.6, 2.7, 3.2, 3.3Supported zc.buildout versions: 1.x, 2.xDetailed DocumentationSupported optionsThehexagonit.recipe.downloadrecipe can be used to download and
extract packages from the net. It supports the following options:urlURL to the package that will be downloaded and extracted. The
supported package formats include.tar.gz,.tar.bz2, and.zip.
(Anything that the stdlibzipfileandtarfilemodules accept is OK; for example.warand.docxfiles.)
The value must be a full URL,
e.g.http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz.strip-top-level-dirSwitch to remove the top level directory from the
extracted archive. This will work only if the archive has exactly
one top level directory. Accepted values are ‘true’ or
‘false’. Defaults to ‘false’.ignore-existingSwitch to ignore existing files and/or directories. By
default, the extraction process fails if there is existing files
or directories matching the ones from the archive. Enabling this
option will skip these files/directories from the archive. When
this recipe is uninstalled the ignored files/directorieswill
notbe removed. Accepted values are ‘true’ or ‘false’. Defaults
to ‘false’.md5sumMD5 checksum for the package file. If available the MD5
checksum of the downloaded package will be compared to this value
and if the values do not match the execution of the recipe will
fail.destinationPath to a directory where the extracted contents of the package
will be placed. If omitted, a directory will be created under thebuildout['parts-directory']with the name of the section using
the recipe.download-onlyWhen set to ‘true’, the recipe downloads the file without trying
to extract it. This is useful for downloading non-tarball
files. Thestrip-top-level-diroption will be ignored if this
option is enabled. Defaults tofalse.modeSets the file mode on the downloaded file using the specified octal mode.
This option has effect only ifdownload-onlyis set totrue.
New in version 1.7.0.filenameAllows renaming the downloaded file when usingdownload-only= true.
The downloaded file will still be placed under thedestinationdirectory with the given filename. Ifdownload-only= falsethis
option will be ignored. By default the original filename will be used. New
in version 1.4.1.hash-nameWhen set to ‘true’, passes thehash_name=Truekeyword parameter to thezc.buildoutDownload utility which in turn uses MD5 hashes to name the
downloaded files. See the correspondingdocumentationfor details. Setting the parameter tofalsewill use the original
filename. Defaults totrue. New in version 1.4.0.excludesA list of newline separated path specifications to filter out files and
directories while unpacking. The path specifications may contain Unix
shell-style wildcards as implemented by thefnmatch.fnmatchfunction.For example, to limit the disk usage when downloading the Solr package the
following configuration may be used to exclude the documentation and
contrib from being unpacked:excludes =
  apache-solr-*/contrib/*
  apache-solr-*/docs/*on-updateWhen set totrue, the recipe will re-run itself when updated by the
buildout. Defaults tofalse.
New in version 1.7.0.The recipe uses the zc.buildoutDownload APIto perform the actual download which allows additional configuration of the
download process. You can use thedownload-cacheoption to optionally cache downloaded files.Simple example>>> import os.path
>>> testdata = join(os.path.dirname(__file__), 'testdata')
>>> server = start_server(testdata)
>>> mkdir(sample_buildout, 'downloads')In the simplest form we can download a simple package and have it
extracted in the parts directory.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... """.format(server=server))Ok, let’s run the buildout:>>> print(system(buildout))
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Extracting package to /sample-buildout/parts/package1Let’s take a look at the buildout parts directory now.>>> ls(sample_buildout, 'parts')
d package1The containing directory is named after our part name. Within this
directory are the contents of the extracted package.>>> ls(sample_buildout, 'parts', 'package1')
d package1-1.2.3-finalThe package contained a single top level directory. Let’s peek what’s inside.>>> ls(sample_buildout, 'parts', 'package1', 'package1-1.2.3-final')
- CHANGES.txt
- README.txt
d src>>> rmdir('downloads')Sharing packages between buildoutsUsing thedownload-cacheoption in the buildout allows you to
store the downloaded packages in central location on your
filesystem. Using the the same location for thedownload-cachein
multiple buildouts will effectively share the packages between them
and reduce the network traffic and storage requirements.Let’s create a directory to be used as the download cache.>>> cache = tmpdir('cache')And create a new buildout that sets thebuildout-cacheoption
accordingly.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = sharedpackage
... download-cache = {cache}
...
... [sharedpackage]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... """.format(cache=cache, server=server))Ok, let’s run the buildout:>>> print(system(buildout))
Uninstalling package1.
Installing sharedpackage.
Downloading http://test.server/package1-1.2.3-final.tar.gz
sharedpackage: Extracting package to /sample-buildout/parts/sharedpackageWe can see that the package was placed under the shared container
instead of the default location under the buildout directory. By default the
the filename of the downloaded package is hashed.>>> rmdir(cache, 'dist')
>>> ls(cache)
-  dfb1e3136ba092f200be0f9c57cf62ecWe can keep the original filename by setting thehash-nameparameter tofalse. For readability all the following examples will have hashing
disabled.MD5 checksumsThe downloaded package can be verified against an MD5 checksum. This will
make it easier to spot problems if the file has been changed.If the checksum fails we get an error.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = invalid
... hash-name = false
... """.format(server=server))>>> print(system(buildout))
Uninstalling sharedpackage.
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
While:
  Installing package1.
Error: MD5 checksum mismatch downloading 'http://test.server/package1-1.2.3-final.tar.gz'Using a valid checksum allows the recipe to proceed.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... hash-name = false
... """.format(server=server))>>> print(system(buildout))
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Extracting package to /sample-buildout/parts/package1Controlling the extraction processWe can also extract the archive to any arbitrary location and have the
top level directory be stripped, which is often a useful feature.>>> tmpcontainer = tmpdir('otherplace')
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = {dest}
... strip-top-level-dir = true
... hash-name = false
... """.format(server=server, dest=tmpcontainer))Rerunning the buildout now gives us>>> print(system(buildout))
Uninstalling package1.
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Extracting package to /otherplaceTaking a look at the extracted contents we can also see that the
top-level directory has been stripped.>>> ls(tmpcontainer)
- CHANGES.txt
- README.txt
d srcPartial extraction over existing contentBy default, the recipe will fail if the destination where the package
will be extracted already contains files or directories also included
in the package.>>> container = tmpdir('existing')
>>> existingdir = mkdir(container, 'src')>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = {dest}
... strip-top-level-dir = true
... hash-name = false
... """.format(server=server, dest=container))Running the buildout now will fail because of the existingsrcdirectory in the destination.>>> print(system(buildout))
Uninstalling package1.
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Extracting package to /existing
package1: Target /existing/src already exists. Either remove it or set ``ignore-existing = true`` in your buildout.cfg to ignore existing files and directories.
While:
  Installing package1.
Error: File or directory already exists.Setting theignore-existingoption will allow the recipe to
proceed.>>> rmdir(container)
>>> container = tmpdir('existing')
>>> existingdir = mkdir(container, 'src')>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = {dest}
... strip-top-level-dir = true
... ignore-existing = true
... hash-name = false
... """.format(server=server, dest=container))>>> print(system(buildout))
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Extracting package to /existing
package1: Ignoring existing target: /existing/src>>> ls(container)
- CHANGES.txt
- README.txt
d srcAlso note that when the recipe is uninstalled the ignored targets will
not be removed as they are not part of the output of this recipe. We
can verify this by running the buildout again with a different
destination.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... strip-top-level-dir = true
... ignore-existing = true
... hash-name = false
... """.format(server=server))>>> print(system(buildout))
Uninstalling package1.
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Extracting package to /sample-buildout/parts/package1Now when we look into the directory containing the previous buildout
we can see that thesrcdirectory is still there but the rest of
the files are gone.>>> ls(container)
d srcExcluding package contentsIt is possible to exclude selected contents of the downloaded package from
being extracted on the filesystem. The primary use case is to save disk space
in case the package contains content which is not necessary.Theexcludesoption allows multiple filtering definitions and supports
Unix shell-style wildcards for matching the package contents. The matching is
implemented usingfnmatchand done in a
case-insensitive manner. Each filtering definition must be given on a separate
line.In the following example we will exclude theCHANGES.txtfile and everything
under and including thesrcdirectory.>>> mkdir(sample_buildout, 'downloads')
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
... download-cache = downloads
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... hash-name = false
... excludes =
...     package1-*/CHANGES.txt
...     package1-*/src*
... """.format(server=server))Running the buildout will show how many files matched the configured excludes.>>> print(system(buildout))
Uninstalling package1.
Installing package1.
Downloading http://test.server/package1-1.2.3-final.tar.gz
package1: Excluding 3 file(s) matching the exclusion pattern.
package1: Extracting package to /sample-buildout/parts/package1Increasing the buildout verbosity with-vvwill show the individual files
that got excluded.>>> rmdir('parts', 'package1')
>>> print(system(buildout + ' -vv'))
<BLANKLINE>
...
Uninstalling package1.
...
Installing package1.
Searching cache at /sample-buildout/downloads/
Using cache file /sample-buildout/downloads/package1-1.2.3-final.tar.gz
package1: Excluding package1-1.2.3-final/CHANGES.txt
package1: Excluding package1-1.2.3-final/src
package1: Excluding package1-1.2.3-final/src/foo.txt
package1: Excluding 3 file(s) matching the exclusion pattern.
package1: Extracting package to /sample-buildout/parts/package1Viewing the unpacked package contents shows that the excluded paths are not
there.>>> ls(sample_buildout, 'parts', 'package1', 'package1-1.2.3-final')
- README.txtOffline modeIf the buildout is run in offline mode the recipe will still work if
the package is cached in the downloads directory. Otherwise the user
will be informed that downloading the file is not possible in offline
mode.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
... offline = true
... download-cache = downloads
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... strip-top-level-dir = true
... hash-name = false
... """.format(server=server))Let’s verify that we do have a cached copy in our downloads directory.>>> ls(sample_buildout, 'downloads')
d  dist
-  package1-1.2.3-final.tar.gz>>> print(system(buildout))
Uninstalling package1.
Installing package1.
package1: Extracting package to /sample-buildout/parts/package1When we remove the file from the filesystem the recipe will not work.>>> remove(sample_buildout, 'downloads', 'package1-1.2.3-final.tar.gz')
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package1
... offline = true
...
... [package1]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... hash-name = false
... """.format(server=server))>>> print(system(buildout))
Uninstalling package1.
Installing package1.
While:
  Installing package1.
Error: Couldn't download 'http://test.server/package1-1.2.3-final.tar.gz' in offline mode.Downloading arbitrary filesWe can download any file when setting thedownload-onlyoption totrue. This will simply place the file in thedestinationdirectory.>>> empty_download_cache(cache)
>>> downloads = tmpdir('my-downloads')
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... download-cache = {cache}
...
... [package]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = {dest}
... download-only = true
... """.format(server=server, dest=downloads, cache=cache))>>> print(system(buildout))
Installing package.
Downloading http://test.server/package1-1.2.3-final.tar.gzLooking into the destination directory we can see that the file was
downloaded but not extracted. Using thedownload-onlyoption will
work for any file regardless of the type.>>> ls(downloads)
-  package1-1.2.3-final.tar.gzAs seen above, withdownload-onlythe original filename will be preserved
regardless whether filename hashing is in use or not. However, the cached copy
will be hashed by default.The downloaded files may also be renamed to better reflect their purpose using
thefilenameparameter.>>> empty_download_cache(cache)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... download-cache = {cache}
...
... [package]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = {dest}
... download-only = true
... filename = renamed-package-1.2.3.tgz
... """.format(server=server, dest=downloads, cache=cache))>>> print(system(buildout))
Uninstalling package.
Installing package.
Downloading http://test.server/package1-1.2.3-final.tar.gz>>> ls(downloads)
-  renamed-package-1.2.3.tgzA mode can also be specified to set access permissions of the downloaded file.
This is the equivalent of thechmodshell command in Unix-like operating
systems, using octal mode only.>>> empty_download_cache(cache)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... download-cache = {cache}
...
... [package]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = {dest}
... download-only = true
... mode = 0654
... """.format(server=server, dest=downloads, cache=cache))>>> print(system(buildout))
Uninstalling package.
Installing package.
Downloading http://test.server/package1-1.2.3-final.tar.gz>>> oct(os.stat(os.path.join(downloads, 'package1-1.2.3-final.tar.gz')).st_mode)[-4:]
'0654'Variable substitionsmay be used
with thefilenameparameter to generate the resulting filename dynamically.>>> empty_download_cache(cache)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... download-cache = %(cache)s
... example = foobar-1.2.3
...
... [package]
... recipe = hexagonit.recipe.download
... url = %(server)spackage1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... destination = %(dest)s
... download-only = true
... filename = ${:_buildout_section_name_}-${buildout:example}.tgz
... """ % dict(server=server, dest=downloads, cache=cache))>>> print(system(buildout))
Uninstalling package.
Installing package.
Downloading http://test.server/package1-1.2.3-final.tar.gzIn this example we have used the section name and a value from the [buildout]
section to demonstrate the dynamic naming.>>> ls(downloads)
-  package-foobar-1.2.3.tgzRe-running install on updateSetting theon-updateflag totruewill re-run the install process on
buildout update.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = hexagonit.recipe.download
... url = {server}package1-1.2.3-final.tar.gz
... md5sum = 821ecd681758d3fc03dcf76d3de00412
... download-only = true
... on-update = true
... """.format(server=server))>>> print(system(buildout))
Uninstalling package.
Installing package.
Downloading http://test.server/package1-1.2.3-final.tar.gz>>> print(system(buildout))
Updating package.
Downloading http://test.server/package1-1.2.3-final.tar.gzChange History1.7.1 (2016-02-13)Do not copy source to target when ignore-existing is set #30
[mgrbyte]Fix utf-8 error with Python 3.4 #32
[thefunny42]1.7 (2013-04-07)New option,mode, to explicitly set the file mode of downloaded files whendownload-onlyis set totrue. Seehttps://github.com/hexagonit/hexagonit.recipe.download/pull/18[desaintmartin]Travis CI support, seehttps://travis-ci.org/hexagonit/hexagonit.recipe.download[dokai]Tox support for running tests.
[dokai]zc.buildout 2.x support. Latest versions of both 1.x and 2.x are supported.
[toumorokoshi]New option,on-update, which makes the recipe re-run itself on updates
when set totrue. Defaults tofalse. Seehttps://github.com/hexagonit/hexagonit.recipe.download/pull/16[toumorokoshi]1.6 (2012-11-14)py3k support
[iElectric]1.5.1 (2012-07-13)Ensure that the temporary extraction directory gets deleted even if an error
occurs later in the build. Previously it was possible for the temporary
directory to fill up without limit. Seehttps://github.com/hexagonit/hexagonit.recipe.download/pull/10for details.
[desaintmartin]The download cache is no longer configured automatically because this could
result in buildout errors if the default download cache directory was not
created in time. Seehttps://github.com/hexagonit/hexagonit.recipe.download/pull/9for details.
[miano]1.5.0 (2011-01-26)Implemented support for excluding paths while unpacking.
[fschulze]Reverted back to usingzope.testing.doctestbecause on Python 2.4 the__file__variable is not available using stdlibdoctest. [dokai]Fixed a spurious test failure because the results ofls()were
unpredictable due to random filename hashing. [dokai]Use the stdlibdoctestmodule instead ofzope.testing.doctest.
[dokai]1.4.1 (2010-04-08)Implemented support for specifying the filename of a downloaded file when
usingdownload-only= true. This closeshttp://github.com/hexagonit/hexagonit.recipe.download/issues#issue/3[dokai]Fixed the filename handling fordownload-only= true. The filename
hashing introduced in 1.4.0 resulted in the names of the downloaded files
to be hashed also. Now the original filename is preserved regardless
whether filenames are hashed in the download cache. This closeshttp://github.com/hexagonit/hexagonit.recipe.download/issues#issue/2[dokai]1.4.0 (2010-04-06)Changed the download policy to hash the downloaded filenames by default.
You can revert back to the original behavior by setting ahash-name= falseparameter in the section using the recipe. This closeshttp://github.com/hexagonit/hexagonit.recipe.download/issues#issue/1[dokai]1.3.0 (2009-09-20)Removed support for the deprecateddownload-directoryoption. [dokai]Refactored the download logic to use the Download API in
zc.buildout. We now require zc.buildout >= 1.4.0. [dokai]1.2.2 (2009-08-17)Merged the current trunk (revision 79982) from the Plone Collective
Subversion repository. The collective repository is now abandoned and the
Github repository is the canonical one. [dokai]Open files in binary mode when calculating MD5 checksums. This fixes a
bug with checksums on the Windows platform. Bug report and patch thanks to
Alexander Ivanov. [dokai]1.2.1 (2008-04-13)Rename the buildoutdownload-directoryoption todownload-cache(which is the name used by buildout) [thefunny]Added BBB support for thedownload-directoryoption. It will
emit a deprecation warning and set thedownload-cacheoption
accordingly. [dokai]1.2.0 (2008-01-19)Added thedownload-onlyoption to allow downloading arbitrary
files. [dokai]1.1.0 (2007-10-14)Refactored the install method so recipe subclasses can override the base
directory. For more info seecalculate_basemethod onRecipeclass.
[hexsprite]Recipe is now a new-style Python class so recipe subclasses can use super()
method to get some default behavior. [hexsprite]1.0.1 (2007-08-14)For consistency with other similar recipes, the recipe now sets alocationoption which may be read by other sections to learn
where the package was extracted. This is an alias for thedestinationoption.1.0.0First public release.ContributorsKai Lautaportti (dokai), Author -kai.lautaportti@hexagonit.fiJordan Baker (hexsprite) -jbb@scryent.comSylvain Viollon (thefunny)Alexander IvanovFlorian Schulze (fschulze)Cédric de Saint Martin (desaintmartin)Miano Njoka (miano)Domen Kožar (iElectric)Yusuke Tsutsumi (toumorokoshi)Download