Package: chunkr
chunkrA python library for the purpose of chunking different types of data files, without having to store the whole file in memory.chunkr creates chunks from the source file with a user defined chunk size, then returns an iterator to loop over the resulting batches sequentially.The type of a resulting batch is PyArrow'sTabledue to PyArrow'sperformancein reading & writing data files.It's also possible to create a directory which contains the chunks as parquet files (currently only parquet is possible, new suggestions are welcomed), which will be cleaned up automatically when the user is done with the resulting files.Currently supported input formats: csv, parquetGetting startedpipinstallchunkrUsageIterate over resulting batchesCSV input:fromchunkrimportcreate_csv_chunk_iterwithcreate_csv_chunk_iter(path,chunk_size,storage_options,**extra_args)aschunk_iter:# process chunksforchunkinchunk_iter:# process chunk.to_pandas() or sthParquet:fromchunkrimportcreate_parquet_chunk_iterwithcreate_parquet_chunk_iter(path,chunk_size,storage_options,**extra_args)aschunk_iter:# process chunksforchunkinchunk_iter:# process chunk.to_pandas() or sthparameters:path (str): the path of the input (local, sftp etc, seefsspecfor possible inputs, not everything is supported though)chunk_size (int, optional): number of records in a chunk. Defaults to 100_000.storage_options (dict, optional): extra options to pass to the underlying storage e.g. username, password etc. Defaults to None.extra_args (dict, optional): extra options passed on to the parsing system, file type specificCreate a directory containing the chunks as Parquet filesCSV input:fromchunkrimportcreate_csv_chunk_dirwithcreate_csv_chunk_dir(input_filepath,output_dir,chunk_size,storage_options,write_options,exclude,**extra_args)aschunks_dir:# process chunk files inside dirpd.read_parquet(file)forfileinchunks_dir.iterdir()# the directory will be deleted when the context manager exitsor Parquet:fromchunkrimportcreate_csv_chunk_dirwithcreate_csv_chunk_dir(input_filepath,output_dir,chunk_size,storage_options,write_options,exclude,**extra_args)aschunks_dir:# process chunk files inside dirpd.read_parquet(file)forfileinchunks_dir.iterdir()# the directory will be deleted when the context manager exitsparameters:path (str): the path of the input (local, sftp etc, see fsspec for possible input)output_path (str): the path of the directory to output the chunks tochunk_size (int, optional): number of records in a chunk. Defaults to 100_000.storage_options (dict, optional): extra options to pass to the underlying storage e.g. username, password etc. Defaults to None.write_options (dict, optional): extra options for writing the chunks passed to PyArrow'swrite_table()function. Defaults to None.extra_args (dict, optional): extra options passed on to the parsing system, file specificNote: currently chunkr only supports parquet as the output chunk files formatAdditional examplesCSV inputSuppose you want to chunk a csv file of 1 million records into 10 parquet pieces, you can do the following:CSV extra args are passed to PyArrowsParsing Optionsfromchunkrimportcreate_csv_chunk_dirimportpandasaspdwithcreate_csv_chunk_dir('path/to/file','temp/output',chunk_size=100_000,quote_char='"',delimiter=',',escape_char='\\',)aschunks_dir:assert1_000_000==sum(len(pd.read_parquet(file))forfileinchunks_dir.iterdir())Parquet inputParquet extra args are passed to PyArrowsiter_batches()functionfromchunkrimportcreate_parquet_chunk_dirimportpandasaspdwithcreate_parquet_chunk_dir('path/to/file','temp/output',chunk_size=100_000,columns=['id','name'],)aschunks_dir:assert1_000_000==sum(len(pd.read_parquet(file))forfileinchunks_dir.iterdir())Reading file(s) inside an archive (zip, tar)reading multiple files from a zip archive is possible, for csv files in/folder_in_archive/*.csvwithin an archivecsv/archive.zipyou can do:fromchunkrimportcreate_csv_chunk_iterimportpandasaspdpath='zip://folder_in_archive/*.csv::csv/archive.zip'withcreate_csv_chunk_iter(path)aschunk_iter:assert1_000_000==sum(len(chunk)forchunkinchunk_iter)The only exception is when particularly reading a csv file from a tar.gz, there can beonly 1 csv filewithin the archive:fromchunkrimportcreate_csv_chunk_iterimportpandasaspdpath='tar://*.csv::csv/archive_single.tar.gz'withcreate_csv_chunk_iter(path)aschunk_iter:assert1_000_000==sum(len(chunk)forchunkinchunk_iter)but it's okay for other file types like parquet:fromchunkrimportcreate_parquet_chunk_iterimportpandasaspdpath='tar://partition_idx=*/*.parquet::test/parquet/archive.tar.gz'withcreate_parquet_chunk_iter(path)aschunk_iter:assert1_000_000==sum(len(chunk)forchunkinchunk_iter)Reading from an SFTP remote systemTo authenticate to the SFTP server, you can pass the credentials via storage_options:fromchunkrimportcreate_parquet_chunk_iterimportpandasaspdsftp_path=f"sftp://{sftpserver.host}:{sftpserver.port}/parquet/pyarrow_snappy.parquet"withcreate_parquet_chunk_iter(sftp_path,storage_options={"username":"user","password":"pw",})aschunk_iter:assert1_000_000==sum(len(chunk)forchunkinchunk_iter)Reading from a URLfromchunkrimportcreate_parquet_chunk_iterimportpandasaspdurl="https://example.com/1mil.parquet"withcreate_parquet_chunk_iter(url)aschunk_iter:assert1_000_000==sum(len(chunk)forchunkinchunk_iter)