Package: datachunks
datachunksData chunking for humans, particularly for data engineers. Makes batched data
processing less painful and a little bit more joyful.Installdatachunks requires Python 3.8 or newer. Install it from PyPI:$pipinstalldatachunksUsagedatachunks implements two chunking strategy:"Pull" strategy. Wrap your source stream withchunksgenerator and consume a chunked data stream."Push" strategy. Create a special "feeder" object that will send data chunks to a specified consumer function.First strategy is simple an sutable for most of applications, but second gives more flexibility in building non-trivial
in-memory processing pipelines.chunksandachunksfunctionsThese functions implement the "pull" chunking strategy resectively for synchronous and async/await apllications.fromdatachunksimportchunksforchunkinchunks(range(12),5):print(chunk)Expected output:[0, 1, 2, 3, 4]
[5, 6, 7, 8, 9]
[10, 11]Asynchronous version example:importasynciofromdatachunksimportachunksasyncdefarange(*args,**kwargs):foriinrange(*args,**kwargs):yieldiasyncdefachunks_demo():asyncforchunkinachunks(arange(12),5):print(chunk)asyncio.run(achunks_demo())Expected output:[0, 1, 2, 3, 4]
[5, 6, 7, 8, 9]
[10, 11]Using "push" strategy"Push" strategy is implemented inChunkingFeederandAsyncChunkingFeederobjects.Consider the situation we decided to process odd and even numbers separately. For this purpose we create two feeders
andputinto them odd and even values.fromdatachunksimportChunkingFeederwithChunkingFeeder(lambdac:print(f'evens:{c}'),5)asprint_evens_feeder,\ChunkingFeeder(lambdac:print(f'odds:{c}'),5)asprint_odds_feeder:foriinrange(25):ifi%2==0:print_evens_feeder.put(i)else:print_odds_feeder.put(i)Expected output:evens: [0, 2, 4, 6, 8]
odds: [1, 3, 5, 7, 9]
evens: [10, 12, 14, 16, 18]
odds: [11, 13, 15, 17, 19]
odds: [21, 23]
evens: [20, 22, 24]Additional features:It is guarandeed that all data is delivered to the callback functions after the context exit.It is possible to produce additional items in callback function. It allows to build flexible and even recursive data processing, but of course it is your responsibility to avoid infinite recursion.By defaultChunkingFeedercalls its chunk consumer synchronously. To use multithreading specify theworkers_numparameter.To use multiprocessing set themultiprocessingparameter toTruein addition toworkers_numparameter.TheAsyncChunkingFeederalso supports theworkers_numparameter, but does not supportmultiprocessing.ETL exampleConsider a simple ETL task: we have anorders.jsonlfile that we need to upload to some Mongo database. Sending objects one-by-one is too slow, and file is too big to opload it in one big batch. So we are going to split this data to chunks of reasonable size.Functionread_jsonlreads the file and yields objects one-by-one:importjsondefread_jsonl():withopen('orders.jsonl','r',encoding='utf-8')asjsonl:forjsoned_objinjsonl:ifjsoned_obj:yieldjson.loads(jsoned_obj)The following function pulls objects through chunks generator and send objects to some MongoDB:fromdatachunksimportchunksdeftransfer_orders(db_connection):forchunkinchunks(read_jsonl(),chunk_size=200):db_connection.orders.insert_many(chunk)After a while we decided to store purchase and sales orders into different MongoDB collections, so let's use two chunking feeders:fromdatachunksimportChunkingFeederclassTransferOrders():def__init__(self,db_connection):self.db_connection=db_connectiondefstore_purchase_orders(self,chunk):self.db_connection.purchase_orders.insert_many(chunk)defstore_sales_orders(self,chunk):self.db_connection.sales_orders.insert_many(chunk)defdo_transfer(self):withChunkingFeeder(self.store_purchase_orders,100,workers_num=1)aspurchase_feeder,\ChunkingFeeder(self.store_purchase_orders,500,workers_num=1)assales_feeder:fororderinread_jsonl():iforder.get('order_type')=='purchase':purchase_feeder.put(order)eliforder.get('order_type')=='sales':sales_feeder.put(order)deftransfer_orders(db_connection):TransferOrders(db_connection).do_transfer()Thepymongolibrary is thread-safe, so it makes sense to speed up our process by storing data in separate threads.