Package: hepdata-cli
HEPData-CLIAboutCommand line interface (CLI) and application program interface (API) to allow users to search, download from and upload toHEPData.The code is compatible with both Python 2 and Python 3. Inspiration fromarxiv-cli.Installation (for users)Install fromPyPIusingpip:$ pip install --user hepdata-cli
$ hepdata-cli --helpWith Python 3 (<3.7), if theLANGenvironment variable is not set, you might get an error like:RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.In this case, you will need to export a Unicode locale as described in theClick documentation.Installation (for developers)Install from GitHub in avirtual environment:$ git clone https://github.com/HEPData/hepdata-cli.git
$ cd hepdata-cli
$ python3 -m venv ~/venv/hepdata-cli
$ source ~/venv/hepdata-cli/bin/activate
(hepdata-cli) $ pip install -e '.[tests]'
(hepdata-cli) $ hepdata-cli --help
(hepdata-cli) $ pytest --cov=hepdata_cliUsageYou can use HEPData-CLI both as a command-line interface (CLI) to search, download and upload records from/to the HEPData database, or as a Python library to perform the same operations via its application program interface (API).CLI$ hepdata-cli [-v/--version, --help]
$ hepdata-cli [--verbose] find [QUERY] [-kw/--keyword KEYWORD] [-i/--ids IDTYPE]
$ hepdata-cli [--verbose] download [IDS] [-f/--file-format FORMAT] [-i/--ids IDTYPE] [-t/--table-name TABLE-NAME] [-d/--download-dir DOWNLOAD-DIR]
$ hepdata-cli [--verbose] fetch-names [IDS] [-i/--ids IDTYPE]
$ hepdata-cli [--verbose] upload [PATH-TO-FILE-ARCHIVE] [-e/--email YOUR-EMAIL] [-r/--recid RECORD-ID] [-i/--invitation-cookie COOKIE] [-s/--sandbox TRUE/FALSE] [-p/--password PASSWORD]The commandfindsearches theHEPDatadatabase for matches ofQUERY. The advanced search syntax from the website can be used.The commanddownloaddownloads records from the database (see options below).The commandfetch-namesreturns the names of the data tables in the records whose ids are supplied.The commanduploaduploads a file to the HEPData web site as either a sandbox or normal record.The argument[-kw/--keyword KEYWORD]filters the search result dictionary for specific keywords.
An exact match of the keyword is first attempted, otherwise partial matches are accepted.The argument[-i/--ids IDTYPE]acceptsIDTYPEequal toarxiv,hepdataorinspire.The argument[-f/--file-format FORMAT]acceptsFORMATequal tocsv,root,yaml,yoda,yoda1, orjson.
In the first four cases a .tar.gz archive is downloaded and unpacked as a directory, whereas in the last case a .json file is downloaded.The argument[-t/--table-name TABLE-NAME]accepts a string giving the table name as input.
In this case only the specified table is downloaded as a .csv, .root, .yaml, .yoda, .yoda1 or .json file.The argument[-d/--download-dir DOWNLOAD-DIR]specifies the directory to download the files.
If not specified, the default download directory is./hepdata-downloads.The argument[-e/--email YOUR-EMAIL]is the uploader's email, needed to associate the submission to their HEPData account.The argument[-i/--invitation-cookie COOKIE]must be supplied for non-sandbox submissions.
This can be found in the Uploader invitation email received at the beginning of the submission process.The argument[-s/--sandbox TRUE/FALSE]is a boolean to decide whether to upload to the sandbox or not.The argument[-p/--password PASSWORDis the password for the uploader's HEPData account (prompt if not specified).
Warning: do not store your password unencrypted in any code intended for shared use.Thehepdata-cli download/fetch-namesandhepdata-cli findcommands can be concatenated, if aIDTYPEis specified forfind.
It is also possible to concatenatearxiv download, formpypi/arxiv-cli, withhepdata-cli find, ifarxivis used asIDTYPE.APIEquivalently to the above, these commands can be invoked by the API (in fact, the CLI is just a wrapper around the API).fromhepdata_cli.apiimportClientclient=Client(verbose=True)client.find(query,keyword,ids)client.download(id_list,file_format,ids,table_name,download_dir)client.fetch_names(id_list,ids)client.upload(path_to_file,email,recid,invitation_cookie,sandbox,password)ExamplesExample 1 - a plain search:$ hepdata-cli --verbose find 'reactions:"P P--> LQ LQ X"'or equivalentlyclient.find('reactions:"P P--> LQ LQ X"')matches a single entry and returns full metadata dictionary.Example 2 - search with keyword:$ hepdata-cli --verbose find 'reactions:"P P--> LQ LQ"' -kw yearor equivalentlyclient.find('reactions:"P P--> LQ LQ"',keyword='year')matches four entries and returns their publication years, as a dictionary.Example 3 - search for ids of records:$ hepdata-cli --verbose find 'reactions:"P P--> LQ LQ"' -i hepdataor equivalentlyclient.find('reactions:"P P--> LQ LQ"',ids='hepdata')matches four entries and returns their hepdata ids, as a plain list.Example 4 - concatenate search with download using inspire ids:$ hepdata-cli --verbose download $(hepdata-cli find 'reactions:"P P--> LQ LQ"' -i inspire) -i inspire -f csvor equivalentlyid_list=client.find('reactions:"P P--> LQ LQ"',ids='inspire')client.download(id_list,ids='inspire',file_format='csv')downloads four .tar.gz archives containing csv files and unpacks them in the default./hepdata-downloadsdirectory.Example 5 - find table names in records:$ hepdata-cli fetch-names $(hepdata-cli find 'reactions:"P P--> LQ LQ"' -i hepdata) -i hepdataor equivalentlyid_list=client.find('reactions:"P P--> LQ LQ"',ids='hepdata')client.fetch_names(id_list,ids='hepdata')returns all table names in the four matching records.Example 6 - concatenate search with download from arxiv-cli:This example requiresarxiv-clito be installed, which is easily done via:$ pip install --user arxiv-cliNote that arxiv-cli installs an older version ofclickwhich changes the CLI command
in Example 5 above fromfetch-namestofetch_names.Then,$ arxiv download $(hepdata-cli find 'reactions:"P P--> LQ LQ"' -i arxiv)or equivalentlyimportarxiv_cliimporthepdata_cliarxiv_client=arxiv_cli.Client()hepdata_client=hepdata_cli.Client()id_list=hepdata_client.find('reactions:"P P--> LQ LQ"',ids='arxiv')arxiv_client.download(id_list)downloads two pdfs from the arXiv.Example 7 - upload record to the sandbox:$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -s Trueor equivalentlyclient.upload('/path/to/TestHEPSubmission.tar.gz',email='my@email.com',sandbox=True)The uploaded submission can then be found from yoursandbox.
You will be prompted for the password associated with your HEPData account.
If your account was created with CERN or ORCID authentication, you will first need toset a password.Example 8 - replace a record in the sandbox:$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -r 1234567890 -s Trueor equivalentlyclient.upload('/path/to/TestHEPSubmission.tar.gz',email='my@email.com',recid='1234567890',sandbox=True)Note that you must have uploaded the original sandbox record yourself and that you will be prompted for a password.Example 9 - upload a non-sandbox record:$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -r 123456 -i 8232e07f-d1d8-4883-bb1d-77fd9994ce4f -s Falseor equivalentlyclient.upload('/path/to/TestHEPSubmission.tar.gz',email='my@email.com',recid='123456',invitation_cookie='8232e07f-d1d8-4883-bb1d-77fd9994ce4f',sandbox=False)The uploaded submission can then be found from yourDashboard.
The invitation cookie is sent in your original invitation email.
You must have already claimed permissions by clicking the link in that email or from yourDashboard.
Again, you will be prompted for a password, which must besetif using CERN/ORCID login.
The password can alternatively be passed as an argument to the CLI (-p PASSWORD) or API (password=PASSWORD).
However, please be careful to keep your password secure, for example, by defining an encrypted environment variable for a CI/CD workflow.