Package: rs-document
RS DocumentA opinionated Rust implementation of various common functions of LangChain'sDocument modelas well asUnstructured.io's post processors.Why?I've been tinkering with different RAG projects and have landed on a set of processes that are pretty common between them. I was looking for a reason to try out the excellentmaturinbuild system that allows for rust to be brought in as a python package. Since my common processes involve alot of text processing over ususally a large number of documents, this seemed like a great project to get going with it.Installationpipinstallrs_documentUsageThe main function of this package is to quickly clean and split many documents.fromrs_documentimportDocument,clean_and_split_docs# Create a document with known attributescontent="A"*4000data={"Hello":"World"}doc=Document(page_content=content,metadata=data)# Run all cleaners on the documentdoc.clean()# Recursively split documentdoc.recursive_character_splitter(1000)# -> Produces list of documentsCleanersThe cleaners that are reimplemented fromUnstructured.ioare:clean_non_ascii_charsclean_bulletsclean_ligaturesclean_extra_whitespacegroup_broken_paragraphsnew_line_grouperauto_paragraph_grouperInstead of being standalone functions, I implemented them as methods on the Document class.There is also a.clean()method, which will run all of the cleaners.Thetest_cleaners.pymodule shows how they can be used.fromrs_documentimportDocumentdeftest_non_ascii_characters_cleanup()->None:doc=Document(page_content="\x88This text contains non-ascii characters!\x88",metadata={},)assert"\x88"indoc.page_contentdoc.clean_non_ascii_chars()assert(str(doc)=='Document(page_content="This text contains non-ascii characters!", metadata={})')assert"\x88"notindoc.page_contentdeftest_bullet_characters_cleanup()->None:doc=Document(page_content="●  This is an excellent point!",metadata={})assert"●"indoc.page_contentdoc.clean_bullets()assert(str(doc)=='Document(page_content="This is an excellent point!", metadata={})')assert"●"notindoc.page_contentdeftest_ligature_cleanup()->None:doc=Document(page_content="æ This is an excellent point!",metadata={})assert"æ"indoc.page_contentdoc.clean_ligatures()assert(str(doc)=='Document(page_content="ae This is an excellent point!", metadata={})')assert"æ"notindoc.page_contentdeftest_extra_whitespace_cleanup()->None:doc=Document(page_content="ITEM 1.     BUSINESS ",metadata={})doc.clean_extra_whitespace()assertstr(doc)=='Document(page_content="ITEM 1. BUSINESS", metadata={})'SplittersThere are two splitters:split_on_num_charactersrecursive_character_splitterSimilarly, they are implemented as methods on the doc class.deftest_splitting(document_fixture:Document)->None:split=document_fixture.split_on_num_characters(5)assertlen(split)==len(document_fixture.page_content)/5assertsplit[0].metadata=={"Hello":"World"}assertsplit[0].page_content=="AAAAA"A Note aboutrecursive_character_splitterThe recursive character splitter is modeled after LangChain's recursive character splitter,
but is absolutely not a 1:1 implementation. You will note that it doesn't take in a chunk overlap.
This is because I've implemented it to have an effective overlap of about 1/3 the chunk size,
as this is the number I've landed on being the most useful in my tinkering. Also, it
doesn't allow passing in seperators, because the default seperators seem to be the best
in every situation I've encountered. This makes the interface as simple as passing in achunk_size.clean_and_split_docs functionTo keep interfacing with this module as quick and easy in most of my projects as possible,
I've also implemented a wrapper functionclean_and_split_docs, which takes in a list of
documents (like what you'd get from a document loader) and achunk_size, and it give back
a list of clean and split documents.PerformanceI knew that rust has a leg up on text processing performance over python, but I wanted to
be sure that the performance improvements would be worth taking on another dependancy.I did some very scientific and rigorous testing on my personal machine, and cleaning and splitting
1,000 to 1,000,000 documents is somewhere between 40 and 75x faster with the rust implementation.I added some tests to ensure as this package improves that those performace gains won't be lost.The tests expect 25,000 documents to be processed per second, and for the rust version to be
minimum 25 times faster than the python version.