Package: kghub-downloader
KG-Hub Downloader|Documentation|OverviewThis is a configuration based file caching downloader with initial support for http requests & queries against elasticsearch.InstallationKGHub Downloader is available to install via pip:pip install kghub-downloaderConfigureThe downloader requires a YAML file which contains a list of target URLs to download, and local names to save those downloads.For an example, seeexample/download.yamlAvailable options are:*url: The URL to download from. Currently supported:http(s)Google Cloud Storage (gs://)Google Drive (gdrive://orhttps://drive.google.com/...). The file must be publicly accessible.Amazon AWS S3 bucket (s3://)local_name: The name to save the file as locallytag: A tag to use to filter downloadsapi: The API to use to download the file. Currently supported:elasticsearchelastic search optionsquery_file: The file containing the query to run against the indexindex: The elastic search index for query* Note:Google Cloud Storage URLs require that you have set up your credentials as describedhere. You must:create a service accountadd the service account to the relevant bucketanddownload a JSON keyfor that service account.Then, set theGOOGLE_APPLICATION_CREDENTIALSenvironment variable to point to that file.Mirorring local files to Amazon AWS S3 bucket requires the following:Create an AWS accountCreate an IAM user in AWS: This enables getting theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYneeded for authentication. These two should be stored as environment variables in the user's system.Create an S3 bucket: This will be the destination for pushing local files.You can also include any secrets like API keys you have set as environment variables using{VARIABLE_NAME}, for example:----url:"https://example.com/myfancyfile.json?key={YOUR_SECRET}"localname:myfancyfile.jsonNote:YOUR_SECRETMUSTas an environment variable, and be sure to include the {curly braces} in the url string.UsageDownloader can be used directly in Python or via command lineIn Pythonfromkghub_downloader.download_utilsimportdownload_from_yamldownload_from_yaml(yaml_file="download.yaml",output_dir="data")Command Line$downloader[OPTIONS]ARGS╰ Download files listed in a download.yaml fileOPTIONSyaml_fileA string pointing to the download.yaml file, to be parsed for things to download.Defaults to./download.yamlignore_cacheIgnore cache and download files even if they exist [false]snippet_onlyDownloads only the first 5 kB of each uncompressed source, for testing and file checkstagsLimit to only downloads with this tagmirrorRemote storage URL to mirror download to. Supported buckets: Google Cloud StorageARGUMENTSoutput_dirA string pointing to where to write out downloaded files.Examples:$downloader--output_direxample_output--tagszfin_gene_to_phenotypeexample.yaml
$downloader--output_direxample_output--mirrorgs://your-bucket/desired/directory# Note that if your YAML file is named `download.yaml`,# the argument can be omitted from the CLI call.$downloader--output_direxample_outputDevelopmentInstallgitclonehttps://github.com/monarch-initiative/kghub-downloader.gitcdkghub-downloader
poetryinstallRun testspoetryrunpytestNOTE: The tests require gcloud credentials to be set up as described above, using the monarch github actions service account.