Package: niet
nietGet data from YAML, JSON, and TOML file directly in your shell.How to easily parse and retrieve data from YAML file in our shell?The previous question, few years ago, led us to the development of niet.Indeed, at that time, we needed a way to store and retrieve data for our own
needs. We created niet to read those data. The goal was to develop a tools
that will allow us to standardize how we parse YAML locally or in our CI
pipelines. We wanted something reusable and easily distribuable. Niet was born.Over the years niet evolved to introduce the support of other formats like
TOML.Niet is likexmllintorjqbut for YAML, JSON and TOML data -
you can use it to slice and filter and map and transform structured data.You can easily retrieve data by using simple expressions or using
xpath advanced features to access non-trivial data.You can easily convert YAML format into JSON, or TOML formats and vice versa.Niet is writen in Python so you can install it from a package manager (from
PyPi) or directly by cloning this repository - no specific system rights are
needed to install it.Main FeaturesExtract elements by using xpath syntaxExtract values from JSON, YAML, and TOML formatAutomaticaly detect format (json/yaml)Read data from a web resourceRead data from file or pass data from stdinFormat output valuesFormat output to be reused by shellevalConvert YAML to JSON, or TOML and vice versaInstall or Update niet$pipinstall-UnietRequirementsPython 3.9 or higherSupported versionsSince niet 2.0 the support of python 2.7 have been dropped so if
if you only have python 2.7 at hands then you can use previous version (lower
to 2.0) but you should consider first that no support will be given on
these versions (no bugfix, no new feature, etc). If you report an issue or
or propose a new feature then they will be addressed only for current or
higher version.UsageHelp and options$niet--help
usage:niet[-h][-aADDITIONAL_OBJECTS[ADDITIONAL_OBJECTS...]][-f{json,yaml,toml,eval,newline,ifs,squote,dquote,comma}][-i][-oOUTPUT_FILE][-s][-v][--debug]object[file]ReaddatafromYAMLorJSONfile

positionalarguments:objectPathtoobject.Basedonjsmespathidentifiers(https://jmespath.org/specification.html#identifiers)Use'.'togetwholefile.(eg:a.b.c)fileOptionalJSONorYAMLlocalfilenameordistantwebresourceatrawformat.Ifnotprovidednietreadfromstdin

options:-h,--helpshowthishelpmessageandexit-aADDITIONAL_OBJECTS[ADDITIONAL_OBJECTS...],--additional-objectsADDITIONAL_OBJECTS[ADDITIONAL_OBJECTS...]Pathtoadditionalobjectstosearch.Hereyoucanpassalistofadditionalresearchs.Allowyoutocombineresearchsintothesamecommandcall.Theresearchswillbemadeontheoriginalfileaswiththe`object`parameter.Nietwilloutputtheresultssequentiallywithoutdelimiterbetweentheresults.Ifthe`--output`argumentisgivenbyuser,theresultsareappendedattheendofthefilesequentially.Basedonjsmespathidentifiers(https://jmespath.org/specification.html#identifiers)Use'.'togetwholefile.(eg:a.b.c)-f{json,yaml,toml,eval,newline,ifs,squote,dquote,comma},--format{json,yaml,toml,eval,newline,ifs,squote,dquote,comma}outputformat-i,--in-placePerformmodificationinplace.Willsoalterreadfile-oOUTPUT_FILE,--outputOUTPUT_FILEPrintoutputinafileinsteadofstdout(surchargedbyin-placeparameterifset)-s,--silentsilentmode,doesn'tdisplaymessagewhenelementwasnotfound-v,--versionprinttheNietversionnumberandexit(also--version)--debugActivatethedebugmode(basedonpdb)outputformats:jsonReturnobjectinJSONyamlReturnobjectinYAMLtomlReturnobjectinTOMLevalReturnresultinastringevaluablebyashellevalcommandasaninputnewlineReturnallelementsofalistinanewlineifsReturnallelementsofalistseparatedbyIFSenvvarsquoteAddsinglequotestoresultdquoteAdddoublequotestoresultcommaReturnallelementsseparatedbycommasWith Json from stdin$echo'{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}'|nietfizz.buzz12Fizz4Buzz
$echo'{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}'|nietfizz.buzz-fsquote'1''2''Fizz''4''Buzz'$echo'{"foo": "bar", "fizz": {"buzz": ["1", "2", "fizz", "4", "buzz"]}}'|niet.-fyaml
fizz:buzz:-'1'-'2'-fizz-'4'-buzz
foo:bar
$echo'{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet"fizz.buzz[2]"two
$echo'{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-fdquote"fizz.buzz[0:2]""zero""one"$echo'{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-fdquote"fizz.buzz[:3]""zero""one""two"With YAML fileConsider the yaml file with the following content:# /path/to/your/file.yamlproject:meta:name:my-projectfoo:barlist:-item1-item2-item3test-dash:valueYou candownload the previous examplelocally for testing purpose or use the command line for this:wgethttps://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yamlYou can retrieve data from this file by using niet like this:$niet".project.meta.name"/path/to/your/file.yaml
my-project
$niet".project.foo"/path/to/your/file.yaml
bar
$niet".project.list"/path/to/your/file.yaml
item1item2item3
$# assign return value to shell variable$NAME=$(niet".project.meta.name"/path/to/your/file.yaml)$echo$NAMEmy-project
$nietproject.'"test-dash"'/path/to/your/file.json
valueWith JSON fileConsider the json file with the following content:{"project":{"meta":{"name":"my-project"},"foo":"bar","list":["item1","item2","item3"],"test-dash":"value"}}You candownload the previous examplelocally for testing purpose or use the command line for this:wgethttps://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.jsonYou can retrieve data from this file by using niet like this:$niet"project.meta.name"/path/to/your/file.json
my-project
$niet"project.foo"/path/to/your/file.json
bar
$niet"project.list"/path/to/your/file.json
item1item2item3
$# assign return value to shell variable$NAME=$(niet"project.meta.name"/path/to/your/file.json)$echo$NAMEmy-project
$nietproject.'"test-dash"'/path/to/your/file.json
valueObject IdentifiersAn identifier is the most basic expression and can be used to extract a single
element from a JSON/YAML document. The return value for an identifier is
the value associated with the identifier. If the identifier does not
exist in the JSON/YAML document, than niet display a specific message and
return the error code1, example:$echo'{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}'|nietfizz.gogo
Elementnotfound:fizz.gogo
$echo$?1See therelated sectionfor more info on how to manage
errors withniet.Niet is based onjmespathto find results so for complexe research you can
refer to thejmespath specificationsto use identifiers properly.If you try to search for an identifier who use some dash you need to surround
your research expression with simple and double quotes, examples:$echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-fdquote'"foo-biz"'bar
$echo'{"key-test": "value"}'|niet'"key-test"'valueHowever,nietwill detect related issues and surround automatically your
identifier ifjmespathfail to handle it.Hence, the following examples will return similar results than the previous
examples:$echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-fdquotefoo-biz
bar
$echo'{"key-test": "value"}'|nietkey-test
valueIf your object is not at the root of your path, an example is available intests/sample/sample.json, then you need to only surround the researched
identifier like thisproject.'"test-dash"'{"project":{"meta":{"name":"my-project"},"foo":"bar","list":["item1","item2","item3"],"test-dash":"value"}}Example:nietproject.'"test-dash"'tests/sample/sample.jsonFurther examples withjmespathidentifiers.Additional objectsAdditional objects allow you to combine more than one query.
The--additional-objectsparameter accept a list of objects strings.
These objects are the same thing that the base object used to query
your inputs.This parameter allow to generate advanced output from your input, let see an
example:Consider the following yaml example:configuration:
  warehouse: warehouse-name
  database: database-name
  object_type:
      schema:
        schema1: "/path/to/schema1.sql"
        schema2: "/path/to/schema2.sql"The following command will allow us to generate an output constitued from the
results of the two objects used as query:$ niet ".configuration.object_type | keys(@)[0]" config.yaml
-a ".configuration.object_type.schema.[keys(@)[0], values(@)[0]]"The previous command will output:schema
schema1
/path/to/schema1.sqlThis output wouldn't be possible without combining the result of two queries,
the additional objects are made for that.Outputs of these additional objects are printed sequentially in the order they
are given in the command line.OutputStdoutBy default, niet print the output on stdout.Save output to a fileIt if possible to pass a filename using -o or --output argument to writes
directly in a file. This file will be created if not exists or will be
replaced if already exists.In-file modificationIt is possible to modify directly a file using -i or --in-place argument. This will replace
the input file by the output of niet command. This can be used to extract some data of a file or
reindent a file.Output formatsYou can change the output format using the -f or --format optional
argument.By default, niet detect the input format and display complex objects
in the same format. If the object is a list or a value, newline output
format will be used.Output formats are:ifssquotedquotenewlineyamljsontomlifsIfs output format print all values of a list or a single value in one line.
All values are separated by the content of IFS environment variable if defined,
space otherwise.Examples (consider the previousYAML file example):$IFS="|"niet.project.list/path/to/your/file.yaml-fifs
item1|item2|item3
$IFS=" "niet.project.list/path/to/your/file.yaml-fifs
item1item2item3
$IFS="@"niet.project.list/path/to/your/file.yaml-fifs
item1@item2@item3This is usefull in a shell for loop,
but your content must, of course, don't contain IFS value:OIFS="$IFS"IFS="|"foriin$(niet.project.list/path/to/your/file.yaml-fifs);doecho${i}doneIFS="${OIFS}"Previous example provide the following output:item1
item2
item3For single quoted seesquoteouput ordquotedouble quoted output with IFSsquoteSquotes output format print all values of a list or a single value in one line.
All values are quoted with single quotes and are separated by IFS value.Examples (consider the previousYAML file example):$# With the default IFS$niet.project.list/path/to/your/file.yaml-fsquote'item1''item2''item3'$# With a specified IFS$IFS="|"niet.project.list/path/to/your/file.yaml-fsquote'item1'|'item2'|'item3'dquoteDquotes output format print all values of a list or a single value in one line.
All values are quoted with a double quotes and are separated by IFS value.Examples (consider the previousYAML file example):$# With the default IFS$niet.project.list/path/to/your/file.yaml-fdquote'item1''item2''item3'$# With a specified IFS$IFS="|"niet.project.list/path/to/your/file.yaml-fdquote"item1"|"item2"|"item3"newlinenewlineoutput format print one value of a list or a single value per line.Thenewlineformat is mostly usefull with shell while read loops and
with script interactions.Example:whilereadvalue:doecho$valuedone<$(niet--formatnewlineproject.listyour-file.json)commacommaoutput format print results on the same line and separated by commas.Thecommaformat allow you to format your outputs to consume your results
with other commands lines interfaces. By example some argument parser
allow you to pass multi values for the same parameter (thebeagle commandper
example allow you torepeat the--repooption).Example of integration with beagle and shell:$OSLO_PROJECTS_URL=https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml
$beaglesearch\-flink\--repo$(niet"oslo.deliverables.*.repos[0]"${OSLO_PROJECTS_URL}-fcomma)'venv'The previous command will return all the links of files
who containsvenvon the openstack oslo's scope of projects (pbr,
taskflow, oslo.messaging, etc).Else another with a more reduced scope on openstack oslo's projects:$niet"oslo.deliverables.*.repos[0][?contains(@, \`oslo\`) == \`true\`]"\https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml\-fcomma
openstack/oslo-cookiecutter,openstack/oslo-specs,openstack/oslo.cache,
openstack/oslo.concurrency,openstack/oslo.config,openstack/oslo.context,
openstack/oslo.db,openstack/oslo.i18n,openstack/oslo.limit,openstack/oslo.log,
openstack/oslo.messaging,openstack/oslo.middleware,
openstack/oslo.policy,openstack/oslo.privsep,openstack/oslo.reports,
openstack/oslo.rootwrap,openstack/oslo.serialization,openstack/oslo.service,
openstack/oslo.tools,openstack/oslo.upgradecheck,openstack/oslo.utils,
openstack/oslo.versionedobjects,openstack/oslo.vmware,openstack/oslotestIn the previous example we retrieve only the projects repos who containsosloin their names, so other projects liketaskflow,pbr, etc will
be ignored.evalEval output format allow you to eval output string to initialize shell
variable generated from your JSON/YAML content.You can intialize shell variables from your entire content, example:$echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-feval.foo_biz="bar";fizz__buzz=(zeroonetwothree)$eval$(echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-feval.)$echo${foo_biz}bar
$echo${fizz__buzz}zeroonetwothree
$eval$(echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-feval'"foo-biz"');echo${foo_biz}bar
$echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-fevalfizz.buzzfizz_buzz=(zeroonetwothree);Parent elements are separated by__by example thefizz.buzzelement
will be represented by a variable namedfizz__buzz. You need to consider
that when you call your expected variables.Also you can initialize some shell array from your content and loop over in
a shell maner:$eval$(echo'{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}'|niet-fevalfizz.buzz)$forelin${fizz_buzz};doecho$el;donezero
one
two
threeyamlYAML output format force output to be in YAML regardless the input file format.jsonJSON output format force output to be in JSON regardless the input file format.tomlTOML output format force output to be in TOML regardless the input file format.Read data from a web resourceNiet allow you to read data (json/yaml/toml) from a web resource accessible by
using the HTTP protocole (introduced in niet 2.1).This can be done by passing an url to niet which refer to a raw content (json,
yaml, or toml).Here is some examples with theopenstack governance's projects data:$# List all the oslo projects repos (https://wiki.openstack.org/wiki/Oslo)$niet"oslo.deliverables.*.repos[0]"\https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml
openstack/automaton
openstack/castellan
...
openstack/debtcollector
...
openstack/futurist
openstack/oslo.cache
openstack/oslo.concurrency
openstack/oslo.config
openstack/oslo.context
openstack/oslo.db
openstack/oslo.i18n
openstack/oslo.limit
openstack/oslo.log
openstack/oslo.messaging
openstack/oslo.middleware
openstack/oslo.policy
...
openstack/oslo.service
openstack/osprofiler
openstack/pbr
...
openstack/stevedore
openstack/taskflow
openstack/tooz
openstack/whereto
$nietoslo.service\https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml
Commonlibraries
$# Get the openstack oslo's mission$nietoslo.mission\https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml
ToproduceasetofpythonlibrariescontainingcodesharedbyOpenStackprojects.
TheAPIsprovidedbytheselibrariesshouldbehighquality,stable,consistent,
documentedandgenerallyapplicable.
$eval$(nietoslo.service\https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml-feval)&&\test"${oslo_service}"="Common libraries"$# Get the name of the oslo PTL$eval$(nietoslo.ptl.name\https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml-feval)$echo"${oslo_ptl_name}"# now display your evaluated result$# Convert original distant yaml file into json$niet.https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml-fjsonFor further examples of filters and selections please take a look tothe jmespath's doc.Result not foundBy default when no results was found niet display a specific message and return
the error code1, example:$echo'{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}'|nietfizz.gogo
Elementnotfound:fizz.gogo
$echo$?1You can avoid this behavior by passing niet into a silent mode.Silent mode allow you to hide the specific message error but continue to return
a status code equal to1when the key was not found.You can use the silent mode by using the flag-s/--silent, example:$echo'{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}'|nietfizz.gogo-s
$echo$?1Deal with errorsWhen your JSON file content are not valid niet display an error and exit
with return code1You can easily protect your script like this:PROJECT_NAME=$(nietproject.meta.nameyour-file.yaml)if["$?"="1"];thenecho"Error occur${PROJECT_NAME}"elseecho"Project name:${PROJECT_NAME}"fiExamplesYou can try niet by using the samples provided with the project sources code.All the following examples use the sample file available in niet sources code
at the following locationtests/samples/sample.yaml.Sample example:# tests/samples/sample.yamlproject:meta:name:my-projectfoo:barlist:-item1-item2-item3Extract a single valueRetrieve the project name:$nietproject.meta.nametests/samples/sample.yaml
my-projectComplexe searchConsider the following content:$ cat /var/lib/libvirt/dnsmasq/virbr0.status
[
  {
    "ip-address": "192.168.122.113",
    "mac-address": "52:54:00:91:14:02",
    "hostname": "rhel79",
    "expiry-time": 1644251254
  },
  {
    "ip-address": "192.168.122.162",
    "mac-address": "52:54:00:23:37:ed",
    "hostname": "satellite",
    "expiry-time": 1644251837
  }
]Here we want to retrieve the value of the ip-address field when the hostname
is equal tosatellite. The following command will allow you to get this
value:$sed's/ip/_/g'/var/lib/libvirt/dnsmasq/virbr0.status|niet"[?hostname=='satellite'].ip"192.168.122.162You should notice that first we replace-by_by using the sed
command. We do that becausejmespath, the underlying library used byniet, poorly handle key that contain-. We chosen to replace all - by _ to avoid
any issues elsewhere on the fileHere is an exemple of an automated ssh connection in a kvm virtualised lab
environment by looking for vmname in dhcp file withnietand performing the
ssh connection to the server even if its ip changed.The ssh connection here can be performed with this command:ssh-oProxyCommand='nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='''%h'''].ip_address") %p'root@rhel79Tips - to ease that use you can for example set this.ssh/configentry:host lab-*
user root
ProxyCommand /usr/bin/nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='$(echo %h | cut -d'-' -f2 )'].ip_address") %pAnd then perform assh lab-rhel79or assh lab-satelliteto join all VMs
from your lab, by the hostname prefixed bylab-.Extract a list and parse it in shellDeal with list of items$forelin$(nietproject.listtests/samples/sample.yaml);doecho${el};doneitem1
item2
item3Also you canevalyournietoutput to setput some shell variables
that you can reuse in your shell scripts, the following example is similar to
the previous example but make use of the eval ouput format (-f eval):$eval$(niet-fevalproject.listtests/samples/sample.yaml)$forelin${project__list};doecho$el;donezero
one
two
threeExtract a complex object and parse it in shellExtract the object as JSON to store it in shell variable :$project="$(niet-fjson.projecttests/samples/sample.yaml)"Then parse it after in bash in this example:$niet.meta.name<<<$projectmy-projectTransform JSON into YAMLWith niet you can easily convert your JSON into YAML$niet.tests/samples/sample.json-fyaml
project:foo:barlist:-item1-item2-item3meta:name:my-projectTransform YAML into JSONWith niet you can easily convert your YAML into JSON$niet.tests/samples/sample.yaml-fjson{"project":{"meta":{"name":"my-project"},"foo":"bar","list":["item1","item2","item3"]}}Transform JSON into TOMLWith niet you can easily convert your JSON into TOML$niet.tests/samples/sample.json-ftoml[project]foo="bar"list=["item1","item2","item3"]test-dash="value"[project.meta]name="my-project"Transform YAML into TOMLWith niet you can easily convert your YAML into TOML$niet.tests/samples/sample.yaml-ftoml[project]foo="bar"list=["item1","item2","item3"]test-dash="value"[project.meta]name="my-project"Transform TOML into YAMLWith niet you can easily convert your TOML into YAMLniet.tests/samples/sample.toml-fyaml
project:foo:barlist:-item1-item2-item3meta:name:my-projecttest-dash:valueIndent JSON fileThis is an example of how to indent a JSON file :$niet.tests/samples/sample_not_indented.json{"project":{"meta":{"name":"my-project"},"foo":"bar","list":["item1","item2","item3"],"test-dash":"value"}}TipsYou can pass your search with or without quotes like this:$nietproject.meta.nameyour-file.yaml
$niet"project.meta.name"your-file.yamlYou can executenietstep by step by using the debug mode. It will allow
you to inspect your execution during your debug sessions.ContributeIf you want to contribute to nietplease first read the contribution guidelinesLicenceThis project is under the MIT License.See the license file for more details