Package: anypath
AnyPath makes it trivial to fetch remote resources and work with them locally.
It provides a normalized interface over different resources so that handling them is always consistent.withAnyPath('sftp://jane@host:/home/jane')aspath:path.joinpath('somefile.txt').open().read()Here AnyPath will copy the directory/home/janefrom a remote host via ssh to a local temporary directory.
It is then possible to work with the files locally. After we are done the temporary files are deleted.
Therefore AnyPath is useful if you want to fetch e.g. some config files or a small project directory from a remote location and work with it locally.InstallationDependenciesBasic UsagePersistanceProviders and optionsHttpSftpGitMercurialLocalChecking for dependenciesLimitationsContributingWriting a new PathProviderDecoratorsThe fetch methodLicenseInstallationTo install simply do:pip install anypathDependenciesBy default AnyPath does not install the dependencies for the different providers. You should install them as needed:ProviderDependenciesgitlocal git installationmercuriallocal mercurial installationhttppip install requestssftppip install paramikolocalNoneBasic UsageAnyPath usespathprovidersto handle different remote resources. The resources are then fetched to a new temporary directory where you can work with them.
The newly fetched ressources are wrapped in apathlib.Path.fromanypath.anypathimportAnyPath,path_providerfromanypath.pathprovider.httpimportHttpPathpath_provider.add(HttpPath)withAnyPath('http://example.org')aspath:path.open().read()First you register all the providers that you want to use (note: remember to install the dependencies per provider).
Now you can open any uri that has a scheme known to one of the registered providers.ProviderSchemesgitgit+http://git+https://git://mercurialhg+http://hg+https://httphttp://https://sftpsftp://ssh://localfile:///./You can use AnyPath either as a contextmanager (with AnyPath ...) or directly by callingfetch().
Beware that you will have to callclose()manually when not using the contextmanager to cleanup the temporary files.path_provider.add(HttpPath)ap=AnyPath('http://example.org')path=ap.fetch()# type: Pathpath.open().read()ap.close()PersistanceThe example so far was useful if you are only interested in the content of a fetched resource. They are created in a temporary folder, where you can work with them, and are deleted afterwards.
Sometimes however you may want to persist the remote resource outside of a temporary location.path_provider.add(HttpPath)withAnyPath('http://example.org',persist_dir='/your/local/path')aspath:path.open().read()Instead of copying the files manually you can specify apersist_dirwhen creating the AnyPath. The temporary resources will then be copied to that location.
As a result you will get thepersist_dirwrapped as anpathlib.Pathinstead of the temporary location and you can directly work with it.Providers and optionsWhile the defaults for fetching resources might be fine for many use cases there are many situations where you might want to pass some options to a provider.
You might for example want to do a POST with an HttpPath or pass credentials to a GitPath.Options are always passed as keyword arguments. Following you will find all providers and their available options.HttpThe options are passed to a requests.Request object, they behave the same and are named accordingly.AnyPath('http://example.org',method='GET',data=None,headers=None,params=None)OptionDescriptionmethodDefault: ‘GET’Specifies the HTTP method to be used as a string.E.g. POST, DELETE, PUTdataDefault: NoneThe body to attach to the request.If a dictionary is provided, form-encoding will take place.headersDefault: NoneA dictionary of headers to send in the request.paramsDefault: NoneA dictionary of URL parameters to append to the URL.SftpThe path for Sftp is expected to be in the formatsftp://user@host:/path/on/host, additional options can be set via arguments.AnyPath('sftp://user@localhost:/path/on/host',password=None,private_key=None,port=22)OptionDescriptionpasswordDefault: NoneThe password for the user.Also used if private_key is given,
and the key requires a passwordprivate_keyDefault: NoneThe path to the local private_key (as a string)
if it is used to loginportDefault: 22The ssh port to be used.GitNoneMercurialNoneLocalNoneChecking for dependenciesBy default dependencies are only checked right before the appropriate PathProvider is called, i.e., at the moment the remote resources should be fetched.
It is possible to check for dependencies as soon as all PathProviders are registered. There are two methods to do that,get_requirements()andcheck_requirements().get_requirements()only returns a dictionary of all dependencies (modules and executables) that would be needed, whilecheck_requirements()fully checks for all dependencies to be present and would raise an exception if they are not:>>>path_provider.add(HttpPath,SftpPath,GitPath)>>>path_provider.get_requirements(){'modules':['requests','paramiko'],'executables':['git']}If the requirements for HttpPath (the requests module) would not be met callingcheck_requirements()would raise an exception:>>>path_provider.add(HttpPath)>>>path_provider.check_requirements()...anypath.dependencies.NotInstalledError:Pythonmodulerequestsisnotinstalled.LimitationsYou might not want to use AnyPath if you are working with a huge remote resource.
Everything is fetched to your local machine, which might take some time and cost a lot of space if you try to work with a whole filesystemn of a remote host for example.
It is also not intended do do updates to the remote resource since there is no mechanism to write changes back to the remote.ContributingYou can contribute in any of the following areas, no matter if it is your first OSS contribution or your thousandths.
Contributions are welcome for example:If you find any issue or bug when using AnyPathIf you want to add to the documentation or fix incorrect or missing documentation.If you want to add features or work on the codebase in generalJust file an issue in the tracker first describing what you would like to do and then create a pull-request.Writing a new PathProviderCreating a new PathProvider requires writing a new class; using it requires registering it viapath_provider.add()The basic structure of a PathProvider looks like this:@pattern('protocol://')@required_executables('some_executable')@dependencies('some_py_module')classMyPath(BasePath):def__init__(self,protocol,path,persist_dir,some_option='default'):super().__init__(protocol,path,persist_dir)...@BasePath.wrappeddeffetch(self):...Here a PathProvider MyPath is created, it registers a protocol that it can handle, declares some requirements and has a fetch method which does the actual work.
The PathProvider must always inherit fromanypath.BasePath.DecoratorsThere are three class decorators available for a PathProvider:pattern('')This decorator is required. It lists all patterns a path can start with on which the PathProvider can act. In the example MyPath registers the patternprotocol://,
meaning whenever a path start withprotocol://MyPath will be called.A PathProvider can register more than one pattern, each pattern is passed as a single argument to the pattern decorator.
The HttpPath for example registershttp://andhttps://using@pattern('http://', 'https://').required_executables('')This decorator is optional. It lists all required executables which must be available on the system to perform the tasks of the PathProvider.
In the example MyPath specifies, thatsome_executablemust exists and be callable from within the program.AnyPath checks for the existence via checkingshutil.which(executable) is Nonewhereexecutableis the exectuable specified inrequired_executables().dependencies('')This decorator is optional. It lists all required python modules which must be available to be imported to perform the tasks of the PathProvider.
In the example MyPath specifies, thatsome_py_modulemust exists and be importable.AnyPath will import the module viaimportlib.import_module(module)wheremoduleis the module specified independencies().The fetch methodThe class must include afetchmethod which will be called to fetch the remote resources.
The fetch method must have the method decorator@BasePath.wrapped. Its main purpose is to call pre and post actions to fetching the resources.
Those actions are creating a temporary directory and persisting the temporary files if needed.LicenseAnyPath is licensed under “Mozilla Public License Version 2.0”. See LICENSE.txt for the full license.