Package: ctadata
Client for CTA Data Download Service at Swiss CTA DCDisclaimerThis is a prototype development only for Swiss CTA DC. Please report bugs here, and for any details please inquire the CTAO CH DC management.PurposeCTA data are/will be stored at CTAO-CH data center at CSCS.
These data should be accessible for selected external users.
In addition, these files should be available within interactive analysis platform at CSCS.
This client presents an API access to these services and data.Installation$pipinstall'ctadata>=0.2.6'Usage from within CTA CSCS JupyterHub platformFrom within CTA CSCS JupyterHub platform, selected authorized users are able to access the "data download service", like so:importctadataforurlinctadata.list_dir("cta/DL1/20241114/v0.1/"):if'datacheck'notinurland'.0100'inurland'11111'inurl:print("stored",ctadata.fetch_and_save_file(url)/1024/1024,"Mb")print("found keys",h5py.File(url.split("/")[-1]).keys())To download a file:ctadata.fetch_and_save_file_or_dir("lst/some-data-dir",recursive=True)or, in bash and recursively:ctadataget--recursivelst/some-data-dirUploading filesTo upload a file:ctadata.upload_file("latest","filelists/latest-file-list")The result is:{"path":"lst/users/volodymyr_savchenko_epfl_ch/filelists/latest-file-list","status":"uploaded","total_written":60098730}Note that for every user, the file is uploaded to their own directory constructed from the user name. The path specified is relative to this directory. If you need to move the files to common directories, please as support.  But you likely want to just share returned path to be used as so:ctadata.fetch_and_save_file("lst/users/volodymyr_savchenko_epfl_ch/filelists/latest-file-list")You can also use command line interface to do this:$cta-dataputlatest-file-listlatest-file-list-bla-blaBeware that all the files written are accessible to all CTAO members and all platform users.From outside (possibly another jupyterhub)You need to get yourself a jupyterhub token, it will be used to authenticate to the download service.If you are in the session, navigate to the hub control panel this way:Request a token:The rest is similar to the previous case:importosos.environ["CTADS_URL"]="DATA-DISTRIBUTING-JUPYTERHUB/services/downloadservice/"os.environ["JUPYTERHUB_API_TOKEN"]="INSERT-YOUR-TOKEN-HERE"forurlinctadata.list_dir("cta/DL1/20241114/v0.1"):if'datacheck'notinurland'.0100'inurland'11111'inurl:print("stored",ctadata.fetch_and_save_file(url)/1024/1024,"Mb")print("found keys",h5py.File(url.split("/")[-1]).keys())Webdav ClientIn order to make use of bare WebDAV interface of the storage,ctadataalso provides a configuredwebdav4client (seewebdav4for documentation).client=ctadata.webdav4_client()client.ls("/")client.uploadFile("example.txt","remote/example.txt")Please seeWebDAV4 documenationfor details on it's wide range of features.Delegating a proxy grid certificate to the PlatformIn order to make use of your own grid certificate to access CTA-CSCS storage from within CTA interactive platform it is necessary to upload you short-term proxy certificate to the platform.cta-dataprovides an easy way to do this:This tools also offers a way to upload your own time limited certificate to access the background webdav server.importctadatactadata.upload_certificate('yourcertificate.crt')Note that if you do not upload your own certificate, you can ask to make use of a shared robot certificate used for data syncing.