Package: pg-jsonb-flattener
Thepg_jsonb_flatteneris a tool to ease selection of Postgresql table which
contains JSONB typed fields.Provided with a description of the data (a sort of jsonschema) contained in a
JSONB field, it is able to build a selectable sqlalchemy expression which can
be used as a plain Postgresql table for SELECT queries.StatusbranchCICoveragemasterdevelopUsageThe tool is used by instantiating aJsonbTableproviding it:a sqlaclehmy table description as the only positionnal argumentaJSON data descriptionfor eachJSONB fieldto flatten as named arguments
where the argument name is theJSONB fieldname and the argument value is
theJSON data description.The code below illustrates how the tool can be used to obtain aflat selectablefrom a data description written in a JSON file.fromsqlalchemyimportcreate_engine,MetaData,Tablefrompg_jsonb_flattenerimportJsonbTableENGINE=create_engine('some database URL')METADATA=MetaData()ONE_JSON_FIELD_TABLE=Table('one_json_field_table',METADATA,Column('field_one',JSONB))DATA_DESCRIPTION=json.load('path/to/data_description.json')jsonb_table=JsonbTable(ONE_JSON_FIELD_TABLE,field_one=DATA_DESCRIPTION)flat_selectable=jsonb_table.get_query()withengine.connect()asconn:result=conn.execute(flat_selectable).fetchall()Then, theflat_selectablevariable contains a selectable expression which acts
as a table where the JSONBfield_onefield has been replaced by several
scalar typed fields whose names are build from the original JSONB field name
and the path of the data in the data structure, with a double undercore as
separator.You can execute directly theflat_selectableto inspect what it contains
(this is what is done in the example to get theresultvariable) or use it as
a plain table to build more complex queries.See below examples for more details.Simple typesGiven the following data description:{
  "type": "object",
  "properties": {
    "integer": {"type": "integer"},
    "number": {"type": "number"},
    "string": {"type": "string"}
    "boolean": {"type": "boolean"}
  }
}flat_selectablewould provide the following fields:field_one__integerfield_one__numberfield_one__stringfield_one__booleanGiven the following JSON data in one record:{
  "integer": 42,
  "number": 3.141592,
  "string": "pouet",
  "boolean": false
}Theresultvariable would contain this:field_one__integerfield_one__numberfield_one__stringfield_one__boolean423.141592pouetfalseSimple objectsGiven the following data description:{
  "type": "object",
  "properties": {
    "object": {
      "type": "object",
      "properties": {
        "obj_int": {"type": "integer"},
        "obj_num": {"type": "number"},
        "obj_str": {"type": "string"}
      }
    }
  }
}flat_selectablewould provide the following fields:field_one__object__obj_intfield_one__object__obj_numfield_one__object__obj_strGiven the following JSON data in one record:{
  "object": {
    "obj_int": 42,
    "obj_num": 3.141592,
    "obj_str": "pouet"
  }
}Theresultvariable would contain this:field_one__object__obj_intfield_one__object__obj_numfield_one__object__obj_str423.141592pouetSimple arraysGiven the following data description:{
  "type": "object",
  "properties": {
    "int_arr": {
      "type": "array",
      "items": {"type": "integer"}
    },
    "num_arr": {
      "type": "array",
      "items": {"type": "number"}
    },
    "str_arr": {
      "type": "array",
      "items": {"type": "string"}
    }
  }
}flat_selectablewould provide the following fields:field_one__int_arr__itemfield_one__int_arr__item__indexfield_one__num_arr__itemfield_one__num_arr__item__indexfield_one__str_arr__itemfield_one__str_arr__item__indexNoteAs you can see each array field is suffixed by anitempart and each array
data is identified by an index which allows to retrieve the order of the
data in the JSONB array.Given the following JSON data in one record:{
  "int_arr": [42, 77, 1664],
  "num_arr": [3.141592, 1.618034, 37.2],
  "str_arr": ["pouet", "foo", "bar"]
}Theresultvariable would contain this:field_one__int_arr__itemfield_one__int_arr__item__indexfield_one__num_arr__itemfield_one__num_arr__item__indexfield_one__str_arr__itemfield_one__str_arr__item__index421NULLNULLNULLNULL772NULLNULLNULLNULL16643NULLNULLNULLNULLNULLNULL3.1415921NULLNULLNULLNULL1.6180342NULLNULLNULLNULL37.23NULLNULLNULLNULLNULLNULLpouet1NULLNULLNULLNULLfoo2NULLNULLNULLNULLbar3Array of objectsGiven the following data description:{
  "type": "object",
  "properties": {
    "array": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "obj_int": {"type": "integer"},
          "obj_num": {"type": "number"},
          "obj_str": {"type": "string"}
        }
      }
    }
  }
}flat_selectablewould provide the following fields:field_one__array__item__indexfield_one__array__item__obj_intfield_one__array__item__obj_numfield_one__array__item__obj_strGiven the following JSON data in one record:{
  "array": [
    {
      "obj_int": 42,
      "obj_num": 3.141592,
      "obj_str": "pouet"
    },
    {
      "obj_int": 77,
      "obj_num": 1.618034,
      "obj_str": "toto"
    }
  ]
}Theresultvariable would contain this:field_one__array__item__indexfield_one__array__item__obj_intfield_one__array__item__obj_numfield_one__array__item__obj_str1423.141592pouet2771.618034totoArrays in objectGiven the following data description:{
  "type": "object",
  "properties": {
    "object": {
      "type": "object",
      "properties": {
        "int_arr": {
          "type": "array",
          "items": {"type": "integer"}
        },
        "num_arr": {
          "type": "array",
          "items": {"type": "number"}
        },
        "str_arr": {
          "type": "array",
          "items": {"type": "string"}
        }
      }
    }
  }
}flat_selectablewould provide the following fields:field_one__object__int_arr__itemfield_one__object__int_arr__item__indexfield_one__object__num_arr__itemfield_one__object__num_arr__item__indexfield_one__object__str_arr__itemfield_one__object__str_arr__item__indexGiven the following JSON data in one record:{
  "object": {
    "int_arr": [42, 77, 1664],
    "num_arr": [3.141592, 1.618034, 37.2],
    "str_arr": ["pouet", "foo", "bar"]
  }
}Theresultvariable would contain this:field_one__object__int_arr__itemfield_one__object__int_arr__item__indexfield_one__object__num_arr__itemfield_one__object__num_arr__item__indexfield_one__object__str_arr__itemfield_one__object__str_arr__item__index421NULLNULLNULLNULL772NULLNULLNULLNULL16643NULLNULLNULLNULLNULLNULL3.1415921NULLNULLNULLNULL1.6180342NULLNULLNULLNULL37.23NULLNULLNULLNULLNULLNULLpouet1NULLNULLNULLNULLfoo2NULLNULLNULLNULLbar3Nested objectsGiven the following data description:{
  "type": "object",
  "properties": {
    "object": {
      "type": "object",
      "properties": {
        "nested": {
          "type": "object",
          "properties": {
            "obj_int": {"type": "integer"},
            "obj_num": {"type": "number"},
            "obj_str": {"type": "string"}
          }
        }
      }
    }
  }
}flat_selectablewould provide the following fields:field_one__object__nested__obj_intfield_one__object__nested__obj_numfield_one__object__nested__obj_strGiven the following JSON data in one record:{
  "object": {
    "nested": {
      "obj_int": 42,
      "obj_num": 3.141592,
      "obj_str": "pouet"
    }
  }
}Theresultvariable would contain this:field_one__object__nested__obj_intfield_one__object__nested__obj_numfield_one__object__nested__obj_str423.141592pouetNested arraysGiven the following data description:{
  "type": "object",
  "properties": {
    "nested_arrays": {
      "type": "array",
      "items": {
        "type": "array",
        "items": {"type": "string"}
      }
    }
  }
}flat_selectablewould provide the following fields:field_one__nested_arrays__item__indexfield_one__nested_arrays__item__item__indexfield_one__nested_arrays__item__itemGiven the following JSON data in one record:{
  "nested_arrays": [
    ["abc", "def"],
    ["tuw", "xyz"]
  ]
}Theresultvariable would contain this:field_one__nested_arrays__item__indexfield_one__nested_arrays__item__item__indexfield_one__nested_arrays__item__item11abc12def21tuw22xyzData “preservation”thepg_jsonb_flattenertool allows to specify chunks of JSON data not to be
flattened by providing a full name of data to preserve in a list passed atJsonbTableinstantiation via thepreservedoptional argument. See examples
below with nested arrays:jsonb_table=JsonbTable(ONE_JSON_FIELD_TABLE,field_one=DATA_DESCRIPTION,preserved=['field_one__nested_arrays__item'])flat_selectable=jsonb_table.get_query()withengine.connect()asconn:result=conn.execute(flat_selectable).fetchall()In this case, theflat_selectablewould provide the following fields:field_one__nested_arrays__item__indexfield_one__nested_arrays__itemTheresultvariable would contain this:field_one__nested_arrays__item__indexfield_one__nested_arrays__item1[“abc”, “def”]2[“tuw”, “xyz”]This also works for the nesting array and preserving thefield_one__nested_arrayswould result in the  following fields and result:field_one__nested_arraysfield_one__nested_arrays[[“abc”, “def”], [“tuw”, “xyz”]]Objects can also be preserved, see examples with nested objects.Preservingfield_one__object__nestedgives the following fields and result:field_one__object__nestedfield_one__object__nested{‘obj_int’: 42, ‘obj_num’: 3.141592, ‘obj_str’: ‘pouet’}Preservingfield_one__objectgives the following fields and result:field_one__objectfield_one__object__nested{‘nested’: {‘obj_int’: 42, ‘obj_num’: 3.141592, ‘obj_str’: ‘pouet’}}NoteOf course, it is possible to preserve objects in arrays and arrays in objects