Package: pybite
PyBiteChunk by chunk iteration made easierInstallationpip install pybiteMethodsiterate_byReturn a iterator of chunks for the iterableParametersiterable:iterAny iterable data e.g. list, tuple, dict, iter, ...chunk_size:intThe size of each chunkmap:callableoptional, defaultsNoneA map function for transform the data before dividing in chunkspersist_header:booloptional, defaultsFalseWhether to persist a header on each chunk or not.
If persist_header is True and header is None,
the firt line will be read as the actual header.header:stroptional, defaultsNoneA header to be written at the start of each file.
If persist_header is True and header is None,
the firt line will be read as the actual header.ReturnsiterNew iterable for the chunked dataExamples>>>iterate_by([1,2,3,4,5],2)iter([[1,2],[3,4],[5]])>>>iterate_by([1,2,3,4,5],2,map=lambdax:x*2)iter([[2,4],[6,8],[10]])>>>iterate_by(["numbers",1,2,3,4,5],2,persist_header=True)iter([["numbers",1,2],["numbers",3,4],["numbers",5]])iterate_file_by_linesReturn a iterator of file lines.Each line is read on-demand untill there is no more line to
be read.Parametersiterable:iterAny iterable data e.g. list, tuple, dict, iter, ...file_stream:str,io.StringIOA file path or io.StringIO instance to the file to be read.encoding:stroptional, defaultsNoneThe input file encoding.
The chunks will be saved using the same encoding.strip_end:booloptional, defaultsFalseFlag for strip the end of each line.
Same as callingline.rstrip().ReturnsiterNew iterable for the chunked dataExampleswithopen("test.txt","w",encoding="utf-8")asf:f.write("Symbols\nAyp\nBx\nCC\nDt")>>>iterate_file_by_lines("test.txt",encoding="utf-8")iter(["Symbols\n","Ayp\n","Bx\n","CC\n","Dt"])>>>iterate_file_by_lines("test.txt",encoding="utf-8",strip_end=True)iter(["Symbols","Ayp","Bx","CC","Dt"])split_file_by_linesSplit a file into multiple files and store them in the output path.The file is divided b lines into chunk files that later can be read individually
or joined back.Parametersfile_stream:str,io.StringIOA file path or io.StringIO instance to the file you want to split.output_path:strThe path to the directory where the chunks will be stored.lines:intThe quantity of lines on each chunk file.
If included the header the total will be lines + 1.encoding:stroptional, defaultsNoneThe input file encoding.
The chunks will be saved using the same encoding.persist_header:booloptional, defaultsFalseWhether to persist a header on each chunk or not.
If persist_header is True and header is None,
the firt line will be read as the actual header.header:stroptional, defaultsNoneA header to be written at the start of each file.
If persist_header is True and header is None,
the firt line will be read as the actual header.chunk_name_format:stroptional, defaults"04d"The format string for the chunk numbers in the output file names.ReturnslistList of paths to the written files.Exampleswithopen("test.txt","w",encoding="utf-8")asf:f.write("Symbols\nAyp\nBx\nCC\nDt")>>>split_by_lines("test.txt","out",2,encoding="utf-8")["out/test.chunk0000.txt","out/test.chunk0001.txt","out/test.chunk0002.txt"]>>>split_by_lines("test.txt","out",2,encoding="utf-8",persist_header=True)["out/test.chunk0000.txt","out/test.chunk0001.txt"]join_file_chunksJoin chunk files into a single line stream.Join the files created by split_by_lines into a iterable of
str lines and read ordered by name.
Chunks not found will throw an error if ignore_missing_chunks is not False.Avoid saving different files chunks into the same directory.Parametersfiles_path:strThe path to a directory containing the chunk files or a list of the path to the files.encoding:stroptional, defaultsNoneThe input file encoding.
The chunks will be saved using the same encoding.persisted_header:booloptional, defaultsFalseWhether a header was persisted on each chunk or not.
If persisted_header is True and header is None,
the firt line of teh first file will be read as the actual header.header:stroptional, defaultsNoneA header to be read at the start of the first file.
If persisted_header is True and header is None,
the firt line of teh first file will be read as the actual header.ignore_missing_chunks:booloptional, defaults"FalseFlag to ignore missing chunks.ReturnsiterAn iterable to the lines of the files (read in the order).split_file_by_linesSlice file by lines into an iterable of the sliced lines.Returns only cut lines in iterable str format.
It does not work with negative numbers.Parametersfiles_stream:strThe path and file to be cut.start:intInitial position of the cut.
Use positive Number.end:intEnd position of the cut.
Use Positive Number.encoding:stroptional, defaultsNoneThe input file encoding.
The chunks will be saved using the same encoding.persisted_header:booloptional, defaultsFalseWhether a header was persisted on each chunk or not.
If persisted_header is True and header is None,
the firt line of teh first file will be read as the actual header.header:stroptional, defaultsNoneA header to be read at the start of the first file.
If persisted_header is True and header is None,
the firt line of teh first file will be read as the actual header.ReturnsiterAn iterable with the cut portion of the fileExamplestest_data=["Name;Age;","Test;00;","Test;11;","John;22;","Test;33;","Test;44;","Test;55;","Test;66;","Test;77;","Test;88;",]withopen("./text.txt","w")asf:f.write("\n".join(test_data))>>>slice_file_by_lines(input_path,1,2,persist_header=True)iter(["Name;Age;\n","Test;11;\n","John;22;\n"])>>>slice_file_by_lines(input_path,1,2)iter(["Test;11;\n","John;22;\n"])TestTest are handled byPyTestand are included inside the foldertest.To run the test execute the command:pytest