Package: tidy-json-to-csv
tidy-json-to-csvConverts a subset of JSON to a set of tidy CSVs. Supports both streaming processing of input JSON and output of CSV, and so suitable for large files in memory constrained environments.What problem does this solve?Most JSON to CSV converters do not result in data suitable for immediate analysis. They usually output a single CSV, and to do this, result in some combination of:JSON inside CSV fields;values in lists presented as columms;data duplicated in multiple rows / a row's position in the CSV determines its context.Often these require subsequent manual, and so error-prone, data manipulation. This library aims to do all the conversion up-front, so you end up with a set oftidytables, which is often a great place from which to start analysis.Example input and outputThe JSON{"songs":[{"id":"1","title":"Walk through the fire","categories":[{"id":"1","name":"musicals"},{"id":"2","name":"television-shows"}],"comments":[{"content":"I love it"},{"content":"I've heard better"}],"artist":{"name":"Slayer"}},{"id":"2","title":"I could have danced all night","categories":[{"id":"1","name":"musicals"},{"id":"3","name":"films"}],"comments":[{"content":"I also could have danced all night"}],"artist":{"name":"Doolitle"}}]}maps to four files:songs.csv"id","title","artist__name"
"1","Walk through the fire","Slayer"
"2","I could have danced all night","Doolitle"songs__categories__id.csv"songs__id","categories__id"
"1","1"
"1","2"
"2","1"
"2","3"songs__comments.csv"songs__id","content"
"1","I love it"
"1","I've heard better"
"2","I also could have danced all night"categories.csv"id","name"
"1","musicals"
"2","television-shows"
"3","films"Installationpipinstalltidy-json-to-csvUsage: Convert JSON to multiple CSV files (Command line)catsongs.json|tidy_json_to_csvUsage: Convert JSON to multiple CSV files (Python)fromtidy_json_to_csvimportto_csvs# A save function, called by to_csvs for each CSV file to be generated.# Will be run in a separate thread, started by to_csvsdefsave_csv_bytes(path,chunks):withopen(f'{path}.csv','wb')asf:forchunkinchunks:f.write(chunk)defjson_bytes():withopen(f'file.json','rb')asf:chunk=f.read(65536)ifchunk:yieldchunkto_csvs(json_bytes(),save_csv_bytes,null='#NA',output_chunk_size=65536)Usage: Convert JSON to multiple Pandas data frames (Python)importioimportqueueimportpandasaspdfromtidy_json_to_csvimportto_csvsdefjson_to_pandas(json_filename):q=queue.Queue()classStreamedIterable(io.RawIOBase):def__init__(self,iterable):self.iterable=iterableself.remainder=b''defreadable(self):returnTruedefreadinto(self,b):buffer_size=len(b)whilelen(self.remainder)<buffer_size:try:self.remainder=self.remainder+next(self.iterable)exceptStopIteration:ifself.remainder:breakreturn0chunk,self.remainder=self.remainder[:buffer_size],self.remainder[buffer_size:]b[:len(chunk)]=chunkreturnlen(chunk)defsave_csv_bytes(path,chunks):q.put((path,pd.read_csv(io.BufferedReader(StreamedIterable(chunks),buffer_size=65536),na_values=['#NA'])))defjson_bytes():withopen(json_filename,'rb')asf:chunk=f.read(65536)ifchunk:yieldchunkto_csvs(json_bytes(),save_csv_bytes,null='#NA')dfs={}whilenotq.empty():path,df=q.get()dfs[path]=dfreturndfsdfs=json_to_pandas('songs.json')forpath,dfindfs.items():print(path)print(df)ConstraintsDenormalised input JSON is assumed, and the output is normalised. If a nested object has anidfield, it is assumed to be the primary key of a top-level table. All objects that have a nested object or arraymusthave anidfield that serves as its primary key in the final output. If present,idmust be thefirstkey in a map. All arrays must be arrays of objects rather than primitives.Althoughmostlystreaming, to support denormalised input JSON and to avoid repeating the same rows in normalised CSVs, an internal record of output IDs is maintained during processing.