Package: qpyson
qpyson: Thin Commandline Tool to Explore, Transform, and Munge JSON using PythonThe JSON querying tool,jq, is a
really powerful tool. However, it’s sometimes a bit involved and has a
learning curve that requires digging into thejq
manualand familiarizing
yourself with a custom language.qpysonis a thin commandline tool to explore, transform, or munge JSON
using Python as the processing language.GoalsProcess (filter, map, general munging) of JSON files using PythonThin layer to process or apply transforms written in PythonProvide Python function as a string to the commandline or define
Python functions in an external fileCustom functions can be parameterized and configured from the
commandlineOutput results are emitted as JSON or in tabular form (usingtabulatefor quick viewing
from the commandlineNon-GoalsA replacement forjqNo custom DSL for filtering or querying (use Python directly)Does not support streaming (JSON files are loaded into memory)InstallationRecommended to install using avirtualenvorcondaenv to install.pip install qpysonQuick TourExample data from the Iris dataset.head examples/iris.json

[
  {"sepalLength": 5.1, "sepalWidth": 3.5, "petalLength": 1.4, "petalWidth": 0.2, "species": "setosa"},
  {"sepalLength": 4.9, "sepalWidth": 3.0, "petalLength": 1.4, "petalWidth": 0.2, "species": "setosa"},
  {"sepalLength": 4.7, "sepalWidth": 3.2, "petalLength": 1.3, "petalWidth": 0.2, "species": "setosa"},
  {"sepalLength": 4.6, "sepalWidth": 3.1, "petalLength": 1.5, "petalWidth": 0.2, "species": "setosa"},
  {"sepalLength": 5.0, "sepalWidth": 3.6, "petalLength": 1.4, "petalWidth": 0.2, "species": "setosa"},
  {"sepalLength": 5.4, "sepalWidth": 3.9, "petalLength": 1.7, "petalWidth": 0.4, "species": "setosa"},
  {"sepalLength": 4.6, "sepalWidth": 3.4, "petalLength": 1.4, "petalWidth": 0.3, "species": "setosa"},
  {"sepalLength": 5.0, "sepalWidth": 3.4, "petalLength": 1.5, "petalWidth": 0.2, "species": "setosa"},
  {"sepalLength": 4.4, "sepalWidth": 2.9, "petalLength": 1.4, "petalWidth": 0.2, "species": "setosa"},The commandline tool takes a function written as commandline string or
referenced in an external file as well as the JSON file to be processed.qpyson --help

qpyson: error: the following arguments are required: path_or_cmd, json_file
usage: qpyson [-f FUNCTION_NAME] [-n] [--indent INDENT] [-t]
              [--table-style TABLE_STYLE]
              [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}]
              [--help]
              path_or_cmd json_file

Util to use Python to process (e.g., filter, map) JSON files

positional arguments:
  path_or_cmd           Path to python file, or python cmd
  json_file             Path to JSON file

optional arguments:
  -f FUNCTION_NAME, --function-name FUNCTION_NAME
                        Function name (default: f)
  -n, --no-pretty       (Non-table) Pretty print the output of dicts and list
                        of dicts (default: False)
  --indent INDENT       (Non-table) Pretty print indent spacing (default: 2)
  -t, --print-table     Pretty print results (default: False)
  --table-style TABLE_STYLE
                        Table fmt style using Tabulate. See
                        https://github.com/astanin/python-tabulate#table-
                        format for available options (default: simple)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}
                        Log level (default: NOTSET)
  --help                Show this help message and exit (default: False)We can define a custom function to process the JSON dataset. By default
the function is namedfand can be customized by-for--function-namecommandline argument.qpyson "def f(d): return d[0]" examples/iris.json

{
  "sepalLength": 5.1,
  "sepalWidth": 3.5,
  "petalLength": 1.4,
  "petalWidth": 0.2,
  "species": "setosa"
}We can also write custom functions in a Python file.cat examples/iris_explore.py

def f(d):
    return d[0]


def f2(d, max_items: int = 10):
    return d[:max_items]


def f3(d, max_items: int = 5):
    return [x for x in d if x["species"] == "setosa"][:max_items]


def f4(d, sort_field: str, sort_direction: str = "asc", max_items: int = 5):
    reverse = not (sort_direction == "asc")
    d.sort(key=lambda x: x[sort_field], reverse=reverse)
    return d[:max_items]


def f0(d):
    # Identity operator
    return d


def first(d):
    return d[0]


def p(d, field: str = "sepalLength"):
    import pandas as pd

    df = pd.DataFrame.from_dict(d)
    sdf = df.describe()
    return sdf.to_dict()[field]Executing--helpwill show the output options.qpyson examples/iris_explore.py examples/iris.json --help

usage: qpyson [-f FUNCTION_NAME] [-n] [--indent INDENT] [-t]
              [--table-style TABLE_STYLE]
              [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}]
              [--help]
              path_or_cmd json_file

Util to use Python to process (e.g., filter, map) JSON files

positional arguments:
  path_or_cmd           Path to python file, or python cmd
  json_file             Path to JSON file

optional arguments:
  -f FUNCTION_NAME, --function-name FUNCTION_NAME
                        Function name (default: f)
  -n, --no-pretty       (Non-table) Pretty print the output of dicts and list
                        of dicts (default: False)
  --indent INDENT       (Non-table) Pretty print indent spacing (default: 2)
  -t, --print-table     Pretty print results (default: False)
  --table-style TABLE_STYLE
                        Table fmt style using Tabulate. See
                        https://github.com/astanin/python-tabulate#table-
                        format for available options (default: simple)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}
                        Log level (default: NOTSET)
  --help                Show this help message and exit (default: False)Executing functionf, yields:qpyson examples/iris_explore.py examples/iris.json 

{
  "sepalLength": 5.1,
  "sepalWidth": 3.5,
  "petalLength": 1.4,
  "petalWidth": 0.2,
  "species": "setosa"
}The output view can be changed to a table view using--print-tableor-t.qpyson examples/iris_explore.py examples/iris.json --print-table --table-style github

|   sepalLength |   sepalWidth |   petalLength |   petalWidth | species   |
|---------------|--------------|---------------|--------------|-----------|
|           5.1 |          3.5 |           1.4 |          0.2 | setosa    |A better example using functionf2defined iniris_explore.pyqpyson examples/iris_explore.py examples/iris.json  --function-name f2 --print-table

  sepalLength    sepalWidth    petalLength    petalWidth  species
-------------  ------------  -------------  ------------  ---------
          5.1           3.5            1.4           0.2  setosa
          4.9           3              1.4           0.2  setosa
          4.7           3.2            1.3           0.2  setosa
          4.6           3.1            1.5           0.2  setosa
          5             3.6            1.4           0.2  setosa
          5.4           3.9            1.7           0.4  setosa
          4.6           3.4            1.4           0.3  setosa
          5             3.4            1.5           0.2  setosa
          4.4           2.9            1.4           0.2  setosa
          4.9           3.1            1.5           0.1  setosaCustom functions can be defined with required or optional values (with
defaults) combined with Python 3 type annotations to generatecat examples/iris.py

# More examples that demonstrate generating commandline arguments


def f(d, sort_field: str, sort_direction: str = "asc", max_items: int = 5):
    reverse = not (sort_direction == "asc")
    d.sort(key=lambda x: x[sort_field], reverse=reverse)
    return d[:max_items]


def g(d, field: str = "sepalLength"):
    import pandas as pd

    df = pd.DataFrame.from_dict(d)
    sdf = df.describe()
    return sdf.to_dict()[field]And calling--helpwill show the custom function specific arguments
(e.g.,--max_itemsand--sort_direction)qpyson examples/iris.py examples/iris.json --help

usage: qpyson [-f FUNCTION_NAME] [-n] [--indent INDENT] [-t]
              [--table-style TABLE_STYLE]
              [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}]
              [--help] [--sort_field SORT_FIELD]
              [--sort_direction SORT_DIRECTION] [--max_items MAX_ITEMS]
              path_or_cmd json_file

Util to use Python to process (e.g., filter, map) JSON files

positional arguments:
  path_or_cmd           Path to python file, or python cmd
  json_file             Path to JSON file

optional arguments:
  -f FUNCTION_NAME, --function-name FUNCTION_NAME
                        Function name (default: f)
  -n, --no-pretty       (Non-table) Pretty print the output of dicts and list
                        of dicts (default: False)
  --indent INDENT       (Non-table) Pretty print indent spacing (default: 2)
  -t, --print-table     Pretty print results (default: False)
  --table-style TABLE_STYLE
                        Table fmt style using Tabulate. See
                        https://github.com/astanin/python-tabulate#table-
                        format for available options (default: simple)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}
                        Log level (default: NOTSET)
  --help                Show this help message and exit (default: False)
  --sort_field SORT_FIELD
                        sort_field type:<class 'str'> from custom func `f`
                        (default: _empty)
  --sort_direction SORT_DIRECTION
                        sort_direction type:<class 'str'> from custom func `f`
                        (default: asc)
  --max_items MAX_ITEMS
                        max_items type:<class 'int'> from custom func `f`
                        (default: 5)And calling with custom options yields:qpyson examples/iris.py examples/iris.json -t --max_items=3 --sort_direction=desc --sort_field sepalLength

  sepalLength    sepalWidth    petalLength    petalWidth  species
-------------  ------------  -------------  ------------  ---------
          7.9           3.8            6.4           2    virginica
          7.7           3.8            6.7           2.2  virginica
          7.7           2.6            6.9           2.3  virginicaAnother Example calling pandas underneath the hood to get a quick
summary of the data.qpyson examples/iris.py examples/iris.json -t -f g --field=sepalLength

  count     mean       std    min    25%    50%    75%    max
-------  -------  --------  -----  -----  -----  -----  -----
    150  5.84333  0.828066    4.3    5.1    5.8    6.4    7.9It’s also possible to create thin JSON munging tools for configuration
of systems or tools that take JSON as input.For example a JSON configuration template with defaults.cat examples/config_template.json

{
  "alpha":  1234,
  "beta": null,
  "gamma": 90.1234
}And a processing function,f.cat examples/config_processor.py

def f(dx, alpha: float, beta: float, gamma: float):
    """Simple example of config munging"""

    def _set(k, v):
        if v:
            dx[k] = v

    items = [("alpha", alpha), ("beta", beta), ("gamma", gamma)]

    for name, value in items:
        _set(name, value)

    dx["_comment"] = "Configured with qpyson"
    return dxRunning--helpwill show the supported configuration options.qpyson examples/config_processor.py examples/config_template.json --help

usage: qpyson [-f FUNCTION_NAME] [-n] [--indent INDENT] [-t]
              [--table-style TABLE_STYLE]
              [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}]
              [--help] [--alpha ALPHA] [--beta BETA] [--gamma GAMMA]
              path_or_cmd json_file

Util to use Python to process (e.g., filter, map) JSON files

positional arguments:
  path_or_cmd           Path to python file, or python cmd
  json_file             Path to JSON file

optional arguments:
  -f FUNCTION_NAME, --function-name FUNCTION_NAME
                        Function name (default: f)
  -n, --no-pretty       (Non-table) Pretty print the output of dicts and list
                        of dicts (default: False)
  --indent INDENT       (Non-table) Pretty print indent spacing (default: 2)
  -t, --print-table     Pretty print results (default: False)
  --table-style TABLE_STYLE
                        Table fmt style using Tabulate. See
                        https://github.com/astanin/python-tabulate#table-
                        format for available options (default: simple)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET}
                        Log level (default: NOTSET)
  --help                Show this help message and exit (default: False)
  --alpha ALPHA         alpha type:<class 'float'> from custom func `f`
                        (default: _empty)
  --beta BETA           beta type:<class 'float'> from custom func `f`
                        (default: _empty)
  --gamma GAMMA         gamma type:<class 'float'> from custom func `f`
                        (default: _empty)Now configuringalpha,betaandgamma.qpyson examples/config_processor.py examples/config_template.json --alpha 1.23 --beta 2.34 --gamma 3.45

{
  "alpha": 1.23,
  "beta": 2.34,
  "gamma": 3.45,
  "_comment": "Configured with qpyson"
}TestingTesting is currently done using RMarkdown using the make targetdoc.This should probably be ported to non-R based approach. However, this
current approach does keep the docs (e.g., README.md) up to date.Related JQ-ish toolshttps://github.com/dbohdan/structured-text-tools#json