Package: semantic-text-splitter
semantic-text-splitterLarge language models (LLMs) can be used for many tasks, but often have a limited context size that can be smaller than documents you might want to use. To use documents of larger length, you often have to split your text into chunks to fit within this context size.This crate provides methods for splitting longer pieces of text into smaller chunks, aiming to maximize a desired chunk size, but still splitting at semantically sensible boundaries whenever possible.Get StartedBy Number of Charactersfromsemantic_text_splitterimportCharacterTextSplitter# Maximum number of characters in a chunkmax_characters=1000# Optionally can also have the splitter not trim whitespace for yousplitter=CharacterTextSplitter(trim_chunks=False)chunks=splitter.chunks("your document text",max_characters)With Huggingface Tokenizerfromsemantic_text_splitterimportHuggingFaceTextSplitterfromtokenizersimportTokenizer# Maximum number of tokens in a chunkmax_tokens=1000# Optionally can also have the splitter not trim whitespace for youtokenizer=Tokenizer.from_pretrained("bert-base-uncased")splitter=HuggingFaceTextSplitter(tokenizer,trim_chunks=False)chunks=splitter.chunks("your document text",max_tokens)With Tiktoken Tokenizerfromsemantic_text_splitterimportTiktokenTextSplitter# Maximum number of tokens in a chunkmax_tokens=1000# Optionally can also have the splitter not trim whitespace for yousplitter=TiktokenTextSplitter("gpt-3.5-turbo",trim_chunks=False)chunks=splitter.chunks("your document text",max_tokens)Using a Range for Chunk CapacityYou also have the option of specifying your chunk capacity as a range.Once a chunk has reached a length that falls within the range it will be returned.It is always possible that a chunk may be returned that is less than thestartvalue, as adding the next piece of text may have made it larger than theendcapacity.fromsemantic_text_splitterimportCharacterTextSplitter# Optionally can also have the splitter trim whitespace for yousplitter=CharacterTextSplitter()# Maximum number of characters in a chunk. Will fill up the# chunk until it is somewhere in this range.chunks=splitter.chunks("your document text",chunk_capacity=(200,1000))MethodTo preserve as much semantic meaning within a chunk as possible, a recursive approach is used, starting at larger semantic units and, if that is too large, breaking it up into the next largest unit. Here is an example of the steps used:Split the text by a given levelFor each section, does it fit within the chunk size?Yes. Merge as many of these neighboring sections into a chunk as possible to maximize chunk length.No. Split by the next level and repeat.The boundaries used to split the text if using the top-levelchunksmethod, in descending length:Descending sequence length of newlines. (Newline is\r\n,\n, or\r) Each unique length of consecutive newline sequences is treated as its own semantic level.Unicode Sentence BoundariesUnicode Word BoundariesUnicode Grapheme Cluster BoundariesCharactersSplitting doesn't occur below the character level, otherwise you could get partial bytes of a char, which may not be a valid unicode str.Note on sentences:There are lots of methods of determining sentence breaks, all to varying degrees of accuracy, and many requiring ML models to do so. Rather than trying to find the perfect sentence breaks, we rely on unicode method of sentence boundaries, which in most cases is good enough for finding a decent semantic breaking point if a paragraph is too large, and avoids the performance penalties of many other methods.InspirationThis crate was inspired byLangChain's TextSplitter. But, looking into the implementation, there was potential for better performance as well as better semantic chunking.A big thank you to the unicode-rs team for theirunicode-segmentationcrate that manages a lot of the complexity of matching the Unicode rules for words and sentences.