Package: semchunk
semchunksemchunkis a fast and lightweight pure Python library for splitting text into semantically meaningful chunks.Owing to its complex yet highly efficient chunking algorithm,semchunkis both more semantically accurate thanlangchain.text_splitter.RecursiveCharacterTextSplitter(seeHow It Works 🔍) and is also over 70% faster thansemantic-text-splitter(see theBenchmarks 📊).Installation 📦semchunkmay be installed withpip:pipinstallsemchunkUsage 👩‍💻The code snippet below demonstrates how text can be chunked withsemchunk:>>>importsemchunk>>>importtiktoken# `tiktoken` is not required but is used here to quickly count tokens.>>>text='The quick brown fox jumps over the lazy dog.'>>>chunk_size=2# A low chunk size is used here for demo purposes.>>>encoder=tiktoken.encoding_for_model('gpt-4')>>>token_counter=lambdatext:len(encoder.encode(text))# `token_counter` may be swapped out for any function capable of counting tokens.>>>semchunk.chunk(text,chunk_size=chunk_size,token_counter=token_counter)['The quick','brown fox','jumps over','the lazy','dog.']Chunkdefchunk(text:str,chunk_size:int,token_counter:callable,memoize:bool=True)->list[str]chunk()splits text into semantically meaningful chunks of a specified size as determined by the provided token counter.textis the text to be chunked.chunk_sizeis the maximum number of tokens a chunk may contain.token_counteris a callable that takes a string and returns the number of tokens in it.memoizeflags whether to memoise the token counter. It defaults toTrue.This function returns a list of chunks up tochunk_size-tokens-long, with any whitespace used to split the text removed.How It Works 🔍semchunkworks by recursively splitting texts until all resulting chunks are equal to or less than a specified chunk size. In particular, it:Splits text using the most semantically meaningful splitter possible;Recursively splits the resulting chunks until a set of chunks equal to or less than the specified chunk size is produced;Merges any chunks that are under the chunk size back together until the chunk size is reached; andReattaches any non-whitespace splitters back to the ends of chunks barring the final chunk if doing so does not bring chunks over the chunk size, otherwise adds non-whitespace splitters as their own chunks.To ensure that chunks are as semantically meaningful as possible,semchunkuses the following splitters, in order of precedence:The largest sequence of newlines (\n) and/or carriage returns (\r);The largest sequence of tabs;The largest sequence of whitespace characters (as defined by regex's\scharacter class);Sentence terminators (.,?,!and*);Clause separators (;,,,(,),[,],“,”,‘,’,',"and`);Sentence interrupters (:,—and…);Word joiners (/,\,–,&and-); andAll other characters.semchunkalso relies on memoization to cache the results of token counters and thechunk()function, thereby improving performance.Benchmarks 📊On a desktop with a Ryzen 3600, 64 GB of RAM, Windows 11 and Python 3.12.0, it takessemchunk24.41s seconds to split every sample inNLTK's Gutenberg Corpusinto 512-token-long chunks (for context, the Corpus contains 18 texts and 3,001,260 tokens). By comparison, it takessemantic-text-splitter1 minute and 48.01 seconds to chunk the same texts into 512-token-long chunks — a difference of 77.35%.The code used to benchmarksemchunkandsemantic-text-splitteris availablehere.Licence 📄This library is licensed under theMIT License.