Package: json-flattener
json-flattenerPython library for denormalizing/flattening lists of complex objects to tables/data frames, with roundtrippingNotebook ExampleEXAMPLE.ipynbDescriptionGiven YAML/JSON/JSON-Lines such as:-id:S001name:Lord of the Ringsgenres:-fantasycreator:name:JRR Tolkeinfrom_country:Englandbooks:-id:S001.1name:Fellowship of the Ringprice:5.99summary:Hobbits-id:S001.2name:The Two Towersprice:5.99summary:More hobbits-id:S001.3name:Return of the Kingprice:6.99summary:Yet more hobbits-id:S002name:The Culture Seriesgenres:-scificreator:name:Ian M Banksfrom_country:Scotlandbooks:-id:S002.1name:Consider Phlebasprice:5.99-id:S002.2name:Player of Gamesprice:5.99Denormalize usingjflcommand:jflflatten-Ccreator=flat-Cbooks=multivalued-iexamples/books1.yaml-oexamples/books1-flattened.tsvidnamegenrescreator_namecreator_from_countrybooks_namebooks_summarybooks_pricebooks_idcreator_genresS001Lord of the Rings[fantasy]JRR TolkeinEngland[Fellowship of the Ring|The Two Towers|Return of the King][Hobbits|More hobbits|Yet more hobbits][5.99|5.99|6.99][S001.1|S001.2|S001.3]S002The Culture Series[scifi]Ian M BanksScotland[Consider Phlebas|Player of Games][5.99|5.99][S002.1|S002.2]Convert back to JSON/YAML:jflunflatten-Ccreator=flat-Cbooks=multivalued-iexamples/books1.tsv-oexamples/books1.yamlThis library also allows complex fields to be directly serialized as json or yaml (the default is to append_jsonto the key). For example:jflflatten-Ccreator=json-Cbooks=json-iexamples/books1.yaml-oexamples/books1-jsonified.tsvidnamegenrescreator_jsonbooks_jsonS001Lord of the Rings[fantasy]{"name": "JRR Tolkein", "from_country": "England"}[{"id": "S001.1", "name": "Fellowship of the Ring", "summary": "Hobbits", "price": 5.99}, {"id": "S001.2", "name": "The Two Towers", "summary": "More hobbits", "price": 5.99}, {"id": "S001.3", "name": "Return of the King", "summary": "Yet more hobbits", "price": 6.99}]S002The Culture Series[scifi]{"name": "Ian M Banks", "from_country": "Scotland"}[{"id": "S002.1", "name": "Consider Phlebas", "price": 5.99}, {"id": "S002.2", "name": "Player of Games", "price": 5.99}]S003Book of the New Sun[scifi, fantasy]{"name": "Gene Wolfe", "genres": ["scifi", "fantasy"], "from_country": "USA"}[{"id": "S003.1", "name": "Shadow of the Torturer"}, {"id": "S003.2", "name": "Claw of the Conciliator", "price": 6.99}]S004Example with single book{"name": "Ms Writer", "genres": ["romance"], "from_country": "USA"}[{"id": "S004.1", "name": "Blah"}]S005Example with no books{"name": "Mr Unproductive", "genres": ["romance", "scifi", "fantasy"], "from_country": "USA"}See<iframe src="https://docs.google.com/presentation/d/e/2PACX-1vRyM06peU9BkrZbXJazuMlajw5s4Vbj5f0t0TE4hj_X9Ex_EASLSUZuaWUxYIhWbOC6CtPRtxrTGWQD/embed?start=false&loop=false&delayms=60000" frameborder="0" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>The primary use case is to go from a richnormalizeddata model (as python objects, JSON, or YAML) to a flatter representation that is amenable to processing with:Solr/LucenePandas/R DataframesExcel/Google sheetsUnix cut/grep/cat/etcSimple denormalized SQL database representationsThe target denormalized format is a list of rows / a data matrix, where each cell is either an atom or a list of atoms.MethodEach top level key becomes a columnif the key value is a dict/object, then flattenby default a '_' is used to separate the parent key from the inner keye.g. the composition ofcreatorandfrom_countrybecomescreator_from_countrycurrently one level of flattening is supportedif the key value is a list of atomic entities, then leave as isif the key value is a list of dicts/objects, then flatten each key of this inner dict into a liste.g. ifbooksis a list of book objects, andnameis a key on book, thenbooks_nameis a list of names of each bookorder is significant - the first element ofbooks_nameis matched to the first element ofbooks_price, etcAllow any key to be serialized as yaml/json/pickle if configuredCommand line usage (TODO)Usage from PythonDocumentation coming soon: see test folder for nowuse within LinkMLComparisonPandas json_normalizehttps://pandas.pydata.org/pandas-docs/version/0.25.0/reference/api/pandas.io.json.json_normalize.htmlJava json-flattenerhttps://github.com/wnameless/json-flattenerPythoncsvjsonhttps://csvjson.com/json2csv