Package: tsdat-tools
Tsdat ToolsThis repository contains helpful scripts and notes for several tsdat-related tools.Some tools are available as jupyter notebooks, and others are available as a command-line utility.To get access to the command-line utilities, just run:pipinstalltsdat-toolsTo use all the other tools, we recommend cloning this repository.Data to YamlThe goal of this tool is to reduce the tediousness of writing tsdat configuration files for data that you can already
read and convert into anxr.Datasetobject in tsdat. It generates two output files:dataset.yamlandretriever.yaml, which are used bytsdatto define metadata and how the input variables should be mapped to output
variables.If your file is in one of the following formats, this tool can already do this for you. Formats supported out-of-box:netCDF: Files ending with.ncor.cdfwill use thetsdat.NetCDFReaderclasscsv: Files ending with.csvwill use thetsdat.CSVReaderclassparquet: Files ending with.parquetor.pqor.pqtwill use thetsdat.ParquetReaderclasszarr: Files/folders ending with.zarrwill use thetsdat.ZarrReaderclassUsageThen you can run the tool with:tsdat-toolsdata2yamlpath/to/data/file--input-configpath/to/current/dataset.yamlFull usage instructions can be obtained using the--helpflag:>>>tsdat-toolsdata2yaml--help

Usage:tsdat-toolsdata2yaml[OPTIONS]DATAPATH

╭─Arguments─────────────────────────────────────────────────────────────────────────────────────────────╮
│*datapathPATHPathtotheinputdatafilethatshouldbeusedtogeneratetsdatconfigurations.│
│[default:None]│
│[required]│
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─Options───────────────────────────────────────────────────────────────────────────────────────────────╮
│--outdirDIRECTORYThepathtothedirectorywhere│
│the'dataset.yaml'and│
│'retriever.yaml'filesshouldbe│
│written.│
│[default:.]│
│--input-configPATHPathtoadataset.yamlfiletobe│
│usedinadditionto│
│configurationsderivedfromthe│
│inputdatafile.Configurations│
│definedheretakepriorityover│
│auto-detectedpropertiesinthe│
│inputfile.│
│[default:None]│
│--helpShowthismessageandexit.│
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯This tool is designed to be run in the following workflow:Generate new ingest/pipeline from cookiecutter template (e.g.,make cookiescommand)Put an example data file for your pipeline in thetest/data/inputfolderClean up the autogenerateddataset.yamlfile.Add metadata and remove any unused variablesDon't add additional variables yet; just make sure that the info in the current file is accurateCommit your changes ingitor back up your changes so you can compare before & after the script runs.Run this script, passing it the path to your input data file and using the--input-configoption to tell it where
your cleaneddataset.yamlfile is. By default this will generate a newdataset.yamlfile in the current working
directory (location ofpwdon the command line), but you can also use the--outdiroption to specify the path where
it should write to.Review the changes the script made to each file. Note that it is not capable of standardizing units or other
metadata, so you will still need to clean those up manually.Continue with the rest of the ingest/pipeline development stepsExcel to YamlPlease consult the documentation in theexcel2yaml/README.mdfile for more information about
this tool.NetCDF to CSVPlease consult the documentation in thenetcdf2csv/README.mdfile for more information about
this tool.