Package: iterabledata
Iterable DataWork in progress. Documentation in progressIterable data is a Python lib to read data files row by row and write data files.
Iterable classes are similar to files or csv.DictReader or reading parquet files row by row.This library was written to simplify data processing and conversion between formats.Supported file types:BSONJSONNDJSON (JSON lines)XMLXLSXLSXParquetORCAvroPickleSupported file compression: GZip, BZip2, LZMA (.xz), LZ4, ZIPWhy writing this lib?Python has many high-quality data processing tools and libraries, especially pandas and other data frames lib. The only issue with most of them is flat data. Data frames don't support complex data types, and you mustflattendata each time.pyiterable helps you read any data as a Python dictionary instead of flattening data.
It makes it much easier to work with such data sources as JSON, NDJSON, or BSON files.This code is used in several tools written by its author. It's command line toolundatumand data processing ETL enginedatacrafterRequirementsPython 3.8+DocumentationIn progressUsage and examplesRead compressed CSV fileRead compressed csv.xz filefrom iterable.helpers.detect import open_iterable

source = open_iterable('data.csv.xz')
n = 0
for row in iterable:
    n += 1
    # Add data processing code here
    if n % 1000 == 0: print('Processing %d' % (n))Detect encoding and file delimiterDetects encoding and delimiter of the selected CSV file and use it to open as iterablefrom iterable.helpers.detect import open_iterable
from iterable.helpers.utils import detect_encoding, detect_delimiter

delimiter = detect_delimiter('data.csv')
encoding = detect_encoding('data.csv')

source = open_iterable('data.csv', iterableargs={'encoding' : encoding['encoding'], 'delimiter' : delimiter)
n = 0
for row in iterable:
    n += 1
    # Add data processing code here
    if n % 1000 == 0: print('Processing %d' % (n))Convert Parquet file to BSON compressed with LZMA using pipelineUses pipeline class to iterate through parquet file and convert its selected fields to JSON lines (NDJSON)from iterable.helpers.detect import open_iterable
from iterable.pipeline import pipeline

source = open_iterable('data/data.parquet')
destination = open_iterable('data/data.jsonl.xz', mode='w')

def extract_fields(record, state):
    out = {}
    record = dict(record)
    print(record)
    for k in ['name',]:
        out[k] = record[k]
    return out

def print_process(stats, state):
    print(stats)

pipeline(source, destination=destination, process_func=extract_fields, trigger_on=2, trigger_func=print_process, final_func=print_process, start_state={})Convert gzipped JSON lines (NDJSON) file to BSON compressed with LZMAReads each row from JSON lines file using Gzip codec and writes BSON data using LZMA codecfrom iterable.datatypes import JSONLinesIterable, BSONIterable
from iterable.codecs import GZIPCodec, LZMACodec


codecobj = GZIPCodec('data.jsonl.gz', mode='r', open_it=True)
iterable = JSONLinesIterable(codec=codecobj)        
codecobj = LZMACodec('data.bson.xz', mode='wb', open_it=False)
write_iterable = BSONIterable(codec=codecobj, mode='w')
n = 0
for row in iterable:
    n += 1
    if n % 10000 == 0: print('Processing %d' % (n))
    write_iterable.write(row)Examples and testsSeetestsfor example usage