Package: pyjawk
Python-based stream editor for json files.  It is a simple setup that
effectively works as a json-parsingawk, similar to jq, but allowing
in-place editing and output of json documents as well, and using Python as the
working language.  It supports colorized output.ContentsMotivationWhy not jq?InstallationUseFormatsThedataobjectDetails on IO and arguments in the REPLExamplesDumping some data to past.eeConverting data between formatsSelecting a part of a data-structure with evalsExtracting a value as a stringEasily embedding string data from stdin into a json structureRelocating an xml childExploring a structure in a REPLFixing Retroarch PlaylistsPlansMotivationThis program exists for fairly minor convenience, and mostly for my own use.
Whenever I end up needing to quickly edit some json, I find myself opening a
Python REPL, writing a bunch of obvious loading code to load in the json, work
on it a little bit, and then dump it back out to the relevant file.  Also,
whenever I end up needing to inspect json from a web page, I either curl it to a
file and then do the same, or use requests or something to pull it directly in
a Python REPL so I can properly inspect it, or I pipe it through Python’sjson.toolandless.This is meant to supplant those use cases entirely for my own uses.  If you find
it inconvenient to repeatedly undergo the busy work associated with working with
or inspecting json data, and especially if you are most familiar and comfortable
with the stream-editing way of doing things or spending time in a REPL, this
tool might make things a little more convenient for you.  It is also useful for
inspecting and converting between formats, such as between msgpack and json.Why not jq?jqis a really great tool for a lot of what
you would use this.  I wrote this because jq doesn’t provide the user with a
REPL to mangle data, and because Python is a much more powerful and flexible
language for the modification process, especially if you want to access the
filesystem or other I/O.jq is a powerful program with a lot of development, active maintenence,
maintainers, and its own filter language.  If that’s what you want, use that.
If you want a simple tool for loading json and working on it with python in
either a stream or REPL fashion, this is probably a better fit.Installationpip3install--userpyjawkNote that pyjawk is Python 3 only.UseDisplay the help text with-h.In all evaluated python,datarepresents the parsed input data.The program is passed an input string either through stdin or a-iargument,
and an output through-oor stdout.-farguments may pass in script
files that are run first.  Thedataobject is then serialized and output.-earguments are similar to-f, but run afterward and run as python
source text.-cmay be used to enable compact output and may be specified
multiple times for some output formats.  A positional parameter, if present, is
evaluated as a python expression and used to replace the data object.-Iand-Omay be used to set the input and output formats, respectively.-nand-Ndisable input and output respectively.If-r/--replis specified, instead of writing output after processing,
the function to write to the output is registered in the environment aswrite, the arguments structure is registered asargs, and aptpythonREPL is started up with the same environment.Multiple command line tools are available, but they all only set the default
input and output formats.FormatsjsonAvailable as the command line toolpyjawkSupports 3 levels of compactness.Outputs trailing newline except on highest compaction.Supports colorized output.yamlAvailable as the command line toolpyyawkSupports 3 levels of compactness.Outputs trailing newlineSupports colorized output.xmlAvailable as the command line toolpyxawkParses into axml.etree.ElementTree.Elementobject and dumps as xml
text.  Usesxml.etree.ElementTree.tostringto dump. and if uncompacted,
usesxml.dom.minidomto prettify.Supports 2 levels of compactness.Outputs trailing newlineSupports colorized output.pythonAvailable as the command line toolpypawkUsesevalto pull in objects, and eitherpprintorreprto dump,
depending on compactness.Supports 3 levels of compactness.Outputs trailing newline.Supports colorized output.msgpackAvailable as the command line toolpymawkstringAvailable as the command line toolpysawkSimply reads input into a string and outputs data as a string, usingstron it before dumping.Outputs trailing newline except when compaction is requested.bytesAvailable as the command line toolpybawkSimply reads input into bytes and outputs data as bytes.ThedataobjectThedataobject is much like the ones that are normally imported from the
libraries, except all dictionaries forjson,yaml, andmsgpackare
substituted for a subclass ofdictthat maintains order and gives attribute
access to the dictionary storage.  All of the following are equivalent:data["foo"]["bar"][0]["baz"]="spam"data.foo.bar[0].baz="spam"data.foo.bar[0]["baz"]="spam"data["foo"]["bar"][0].baz="spam"Note that constructing adictexplicitly will not automatically construct
this subclass.  You can do that by importingpyjawk.attrdict.AttrDict:data.foo={"bar":[{"baz":"spam"}],}# This will not work.data.foo.bar[0].baz="alot"# The dicts are still ordinary dicts, so this will workdata.foo["bar"][0]["baz"]="alot"frompyjawk.attrdictimportAttrDictasddata.foo=d(bar=[d(baz="spam")],)# Now this will workdata.foo.bar[0].baz="alot"Details on IO and arguments in the REPLIn the REPL, the program’s own argument namespace is available asargs.
Changing some of them is obvious (such asargs.output, which is just a
string, orargs.no_inputwhich is just a boolean), and some others are
perhaps non-obvious (args.compactis an integer specifying the number of
times it was present).  Some of the arguments don’t make any sense to work with
(such asargs.inputandargs.input_format, because those are already
finished by the time the REPL starts up).The REPL does not write the output by default.  To write the output with the
REPL, thewrite()function must be called explicitly.When you wish to use the REPL, stdin and stdout must be attached to a terminal.
This means that you need to be taking input from a file, not a pipe, and the
program may not be piped to anything else.  This is necessary because ptpython
needs stdin to be communicated with and stdout to communicate back to the user.
If you wish to pipe something into pyjawk for REPL use, you’ll have to use a
fifo, a temp file, or a process substitution as follows:# With process redirectionpyjawk-ri<(curl'https://httpbin.org/get?foo=bar&spam=spam')# With a temp filecurl'https://httpbin.org/get?foo=bar&spam=spam'>curltemp.jsonpyjawk-ricurltemp.json# With a fifomkfifocurl.fifocurl'https://httpbin.org/get?foo=bar&spam=spam'>curl.fifo&pyjawk-ricurl.fifoIn every case, the data in the repl is:>>>frompprintimportpprint>>>pprint(data){'args':{'foo':'bar','spam':'spam'},'headers':{'Accept':'*/*','Host':'httpbin.org','User-Agent':'curl/7.65.0'},'origin':'73.169.51.67, 73.169.51.67','url':'https://httpbin.org/get?foo=bar&spam=spam'}ExamplesDumping some data to past.ee$echo'{"a": "1", "b": null, "c": true, "d": false, "e": 7, "f": 8.5, "g": {"h": [1, 2, 3]}}'\|pyjawk'{"sections": [{"contents": str(data)}]}'\|curl-H'Content-Type: application/json'-H'X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'-XPOST--data-binary'@-'https://api.paste.ee/v1/pastes{"id":"umXKr","link":"https:\/\/paste.ee\/p\/umXKr","success":true}With this, you can also do any arbitrary string data, and also extract the link
from the output if you like:$echothisissometestdata\|pyjawk-Istring'{"sections": [{"contents": data}]}'\|curl-H'Content-Type: application/json'-H'X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'-XPOST--data-binary'@-'https://api.paste.ee/v1/pastes\|pyjawk-Ostring'data.link'https://paste.ee/p/iomJRConverting data between formats$echo'{"foo": "bar", "baz": ["spam", "Spam", {"SPAM?": "SPAM!"}]}'\|pyjawk-Oyamlbaz:-spam-Spam-SPAM?:SPAM!foo:barSelecting a part of a data-structure with evals$echo'{"foo": "bar", "baz": ["spam", "Spam", {"SPAM?": "SPAM!"}]}'\|pyjawk-c'data.baz[2]'{"SPAM?":"SPAM!"}Extracting a value as a string$echo'{"foo": "bar", "baz": ["spam", "Spam", {"SPAM?": "SPAM!"}]}'\|pyjawk-Ostring'data.baz[1]'SpamEasily embedding string data from stdin into a json structure$echo'this is a test string'\|pyjawk-Istring-Ojson-c'{"foo": data}'{"foo":"this is a test string\n"}Relocating an xml child$echo'<root><foo><bar>first</bar></foo><baz /></root>'\|pyxawk-e'foo = list(data)[0]; bar = list(foo)[0]; baz = list(data)[1]; baz.append(bar); foo.remove(bar)'<?xml version="1.0" ?><root><foo/><baz><bar>first</bar></baz></root>The-ecan also be specified separately:$echo'<root><foo><bar>first</bar></foo><baz /></root>'\|pyxawk-e'foo = list(data)[0]'-e'bar = list(foo)[0]'-e'baz = list(data)[1]'-e'baz.append(bar)'-e'foo.remove(bar)'Or just as a script file:$echo'<root><foo><bar>first</bar></foo><baz /></root>'\|pyxawk-frelocate.pyfoo=list(data)[0]bar=list(foo)[0]baz=list(data)[1]baz.append(bar)foo.remove(bar)Exploring a structure in a REPL$pyjawk-i<(echo'{"foo": "bar", "baz": ["spam", "Spam", {"SPAM?": "SPAM!"}]}')-r>>>data{'foo':'bar','baz':['spam','Spam',{'SPAM?':'SPAM!'}]}>>>write(){"foo":"bar","baz":["spam","Spam",{"SPAM?":"SPAM!"}]}>>>data=data.baz>>>write()["spam","Spam",{"SPAM?":"SPAM!"}]Fixing Retroarch PlaylistsIf you had an issue with the way that RetroArch generates its playlist files for
the Playstation (by default, it searches for .cue files, but not .bin), and
had something like this in /tmp/Roms/psx, all Sony PlayStation games:Alpha.bin
Alpha.cue
Bravo.bin
Charlie.bin
Delta.bin
Delta.cueYou might end up with a playlist file like this:{"version":"1.2","default_core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","default_core_name":"Sony - PlayStation (PCSX ReARMed)","label_display_mode":0,"right_thumbnail_mode":0,"left_thumbnail_mode":0,"items":[{"path":"/tmp/Roms/psx/Alpha.cue","label":"Alpha","core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","core_name":"Sony - PlayStation (PCSX ReARMed)","crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl"},{"path":"/tmp/Roms/psx/Delta.cue","label":"Delta","core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","core_name":"Sony - PlayStation (PCSX ReARMed)","crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl"}]}If you want the file to just have the bins, you can easily scan the directory
for these files and modify the json using this tool with this:$pyjawk-i'Sony - PlayStation.lpl'-o'Sony - PlayStation.lpl'-e'from pathlib import Path'-e'data.items = [{"path": str(path), "label": path.stem, "core_path": data.default_core_path, "core_name": data.default_core_name, "crc32": "00000000|crc", "db_name": "Sony - PlayStation.lpl"} for path in (Path("/tmp") / "Roms" / "psx").iterdir() if path.suffix == ".bin"]'Making the output{"version":"1.2","default_core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","default_core_name":"Sony - PlayStation (PCSX ReARMed)","label_display_mode":0,"right_thumbnail_mode":0,"left_thumbnail_mode":0,"items":[{"path":"/tmp/Roms/psx/Delta.bin","label":"Delta","core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","core_name":"Sony - PlayStation (PCSX ReARMed)","crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl"},{"path":"/tmp/Roms/psx/Charlie.bin","label":"Charlie","core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","core_name":"Sony - PlayStation (PCSX ReARMed)","crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl"},{"path":"/tmp/Roms/psx/Bravo.bin","label":"Bravo","core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","core_name":"Sony - PlayStation (PCSX ReARMed)","crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl"},{"path":"/tmp/Roms/psx/Alpha.bin","label":"Alpha","core_path":"/tmp/retroarch/cores/pcsx_rearmed_libretro.so","core_name":"Sony - PlayStation (PCSX ReARMed)","crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl"}]}That might look heavy up-front, but you can rewrite it as a script file with
simpler structure:frompathlibimportPathdata.items=[]forpathin(Path('/tmp')/'Roms'/'psx').iterdir():ifpath.suffix=='.bin':data.items.append({"path":str(path),"label":path.stem,"core_path":data.default_core_path,"core_name":data.default_core_name,"crc32":"00000000|crc","db_name":"Sony - PlayStation.lpl",})and run it with pyjawk as so:pyjawk-i'Sony - PlayStation.lpl'-o'Sony - PlayStation.lpl'-fscript.pyOr instead load it into a repl to work on it in real time with this:pyjawk-i'Sony - PlayStation.lpl'-o'Sony - PlayStation.lpl'-r>>>frompathlibimportPath>>>data.items=[]>>>forpathin(Path('/tmp')/'Roms'/'psx').iterdir():...ifpath.suffix=='.bin':...data.items.append({..."path":str(path),..."label":path.stem,..."core_path":data.default_core_path,..."core_name":data.default_core_name,..."crc32":"00000000|crc",..."db_name":"Sony - PlayStation.lpl",...})>>>write()>>>exit()Just make sure you callwrite()in the repl, or nothing will be written.PlansI don’t plan to add too much to this, as I want it to be useful but also as lean
and manageable as it possibly can be.  Things like HTTP input and output are
best left to other programs that can do it better, like curl, especially given
that this program can operate in a streamable fashion.This program needs some regression tests set up.