Package: fetch
fetch stuff from the interwebsfetch.pyis
a single-file python module bundled as apackagefor easy installation
and python importing.  The purpose of fetch is to mirror remote
resources (URLs) to a local filesystem in order to synchronize and
update dependencies that are desired to be mirrored in this way.Formatfetchfetches from a manifest of the format:[URL] [Destination] [Type]A URL can contain a hash tag (e.g.http://example.com/foo#bar/fleem)
which is used to extract the subdirectories from a multi-directory
resource.TheTypeof the resource is used to dispatch to the included
Fetchers that take care of fetching the object.Manifests are used so that a number of resources may be fetched from a
particularfetchrun.ExampleAfter you checkout therepositoryand runpython setup.py develop, you should be able to runfetchon the example manifest:fetch example.txtThis will create atmpdirectory relative to the manifest and pull
down several resources to it.Fetchersfetchincludes several objects for fetching resources:file : fetch a single file
tar : fetch and extract a tarball
hg : checkout a mercurial repository
git : checkout a git repositoryThefilefetcher cannot have a hash tag subpath since it is a single
resource.Thoughfetchhas a set of fetchers included, you can pass an
arbitrary list intofetch.Fetch’s constructor.Version ControlThehgand thegitfetchers fetch from version control systems and
have additional options.  The only current option to the constructor
isexport, which is by default True.  Ifexportis True, then
the repository will be exported into a non-versioned structure.  If a
subpath is specified with a#in the URL, the repository will also
be exported.TODOusemanifestparser.inifiles versus another manifest
format: when I started work onfetch, I thought a
domain-specific manifest would be a big win.  But, now, maybe a.initype manifest looks about the same, and is something that
is already used.  The switch internally wouldn’t be that bad, but
iffetch.pyis used as a single file, it cannot have
“traditional” python dependencies. Sincemanifestparser.pyis
also a single file, andfetchis only usable with internet
access anyway, maybe therequirepattern could be used for this purposeclobber: generally, you will want the existing resource to be
clobbered, avoiding renames regarding upstream dependenciesoutputting only subpaths: often, you will not to fetch from the
whole manifest, only from certain subpaths of the manifest.  You
should be able to output a subset of what is to be mirrored based
on destination subpaths.  The CLI option--destis intended for
this purpose but currently unused.fetcher options: currentlyread_manifestsreads an unused
column intooptionswhen present in the form of a string likefoo=one,bar=twointo a dict like{'foo': 'one', 'bar': 'two'}.  This hasn’t been needed yet and
is unused.  If we want to have resource-specific options, we should
use this and make it work.  Otherwise it can be deleted.python package fetcher: often you will want to fetch a python
package as a resource.  This would essentially fetch the object
(using another fetcher) and take the (untarred) result ofpython setup.py sdistas a resource.  This will strip out files
that aren’t part of the python package. Unknowns include how to
specify the sub-fetcher. You could also include other
domain-specific fetchers as needed.note python 2.5+ specifics:fetchcurrently uses at leastos.path.relpathfrom python 2.5 and perhaps other 2.5+isms as
well. These should at least be documented and checked for if not
obviated. One such reimplementation is athttps://github.com/mozilla/mozbase/blob/master/manifestdestiny/manifestparser.py#L66Unsolved ProblemsA common story forfetchis mirroring files into a VCS repository
because the remote resources are needed as part of the repository and
there is no better way to retrieve and/or update them.  However, what
do you do if these remote resources are altered?  In an ideal
ecosystem, the fixes would be automatically triaged and triggered for
upstream inclusion, or the diffs from the upstream are kept in local
modifications (although vendor branches, etc, are more suitable for
the latter class of problems, and in general discouraged when a less
intrusive system of consuming upstream dependencies are available).Jeff Hammelhttp://k0s.org/mozilla/hg/fetch