Package: chunkipy
Chunkipychunkipyis an extremely useful tool for segmenting long texts into smaller chunks,
based on either a character or token count. With customizable chunk sizes and splitting strategies,chunkipyprovides flexibility and control
for various text processing tasks.Motivation and Featureschunkipywas created to address the need within the field of natural language processing (NLP)
to chunk text so that it does not exceed the input size ofneural networkssuch as BERT,
but it could be used for several other use cases.The library offers some useful features:Token estimation: unlike other text chunking libraries,chunkipyoffers the possibility of
providing a token estimator function, in order to build the chunks taking into account the tokenizer
that will use those chunks.Split text into meaningful sentences: in its default configuration,chunkipy,
in creating the chunks, avoids cutting sentences, and always tries to have a complete and syntactically correct sentence.
This is achieved through the use of thestanzalibrary, which utilizes semantic models to cut text
into meaningful sentences.Smart Overlapping:chunkipyoffers the possibility to define anoverlap_percentageand create overlapping chunks to
preserve the context along chunks.Flexibility in choosing split strategies: Additionally,chunkipyoffers complete flexibility
in choosing how to split, allowing users to define their own text splitting function or choose from a list
of pre-defined splitting strategies.By default,chunkipyusesstanzaare main text splitting method; however, ifstanzaproduces
sentences with a number of tokens greater than the chunk size, other split strategy are used.
Here the list of predefined strategies, sorted by priority (the first one is executed first,
if the chunk of text is larger than the chunk size, it is further split using a lower priority
strategy).PriorityNameEffect0split_by_sentencesIt usesstanzato split the text into meaningful sentences.1split_by_semicolonIt splits the text using the semicolon and space;as separator.2split_by_colonIt splits the text using the colon and space:as separator.3split_by_commaIt splits the text using the comma and space,as separator.4split_by_wordIt splits the text using the spaceas separator.InstallationYou can installchunkipyusing pip:pipinstallchunkipyUsageThe main class inchunkipyisTextChunker, which takes a text and splits it into chunks of a given size.
You can use the default settings or specify custom parameters for thechunk size,
whether to split bycharacters or tokens, thetokenizer functionto use (iftokensis set to True`), and the list ofsplit strategiesto apply.Here's an example of using it to chunk text that has a number of tokens
greater than the input size of BERT:fromchunkipyimportTextChunker,TokenEstimatorfromtransformersimportAutoTokenizertext="""Napoleon Bonaparte (born Napoleone Buonaparte; 15 August 1769 â€“ 5 May 1821), later known by his regnal name Napoleon I, was a French military commander and political leader who rose to prominence during the French Revolution and led successful campaigns during the Revolutionary Wars. He was the de facto leader of the French Republic as First Consul from 1799 to 1804, then Emperor of the French from 1804 until 1814 and again in 1815. Napoleon's political and cultural legacy endures to this day, as a highly celebrated and controversial leader. He initiated many liberal reforms that have persisted in society, and is considered one of the greatest military commanders in history. His wars and campaigns are studied by militaries all over the world. Between three and six million civilians and soldiers perished in what became known as the Napoleonic Wars.Napoleon was born on the island of Corsica, not long after its annexation by France, to a native family descending from minor Italian nobility. He supported the French Revolution in 1789 while serving in the French army, and tried to spread its ideals to his native Corsica. He rose rapidly in the Army after he saved the governing French Directory by firing on royalist insurgents. In 1796, he began a military campaign against the Austrians and their Italian allies, scoring decisive victories and becoming a national hero. Two years later, he led a military expedition to Egypt that served as a springboard to political power. He engineered a coup in November 1799 and became First Consul of the Republic.Differences with the United Kingdom meant France faced the War of the Third Coalition by 1805. Napoleon shattered this coalition with victories in the Ulm campaign, and at the Battle of Austerlitz, which led to the dissolution of the Holy Roman Empire. In 1806, the Fourth Coalition took up arms against him. Napoleon defeated Prussia at the battles of Jena and Auerstedt, marched the Grande ArmÃ©e into Eastern Europe, and defeated the Russians in June 1807 at Friedland, forcing the defeated nations of the Fourth Coalition to accept the Treaties of Tilsit. Two years later, the Austrians challenged the French again during the War of the Fifth Coalition, but Napoleon solidified his grip over Europe after triumphing at the Battle of Wagram.Hoping to extend the Continental System, his embargo against Britain, Napoleon invaded the Iberian Peninsula and declared his brother Joseph the King of Spain in 1808. The Spanish and the Portuguese revolted in the Peninsular War aided by a British army, culminating in defeat for Napoleon's marshals. Napoleon launched an invasion of Russia in the summer of 1812. The resulting campaign witnessed the catastrophic retreat of Napoleon's Grande ArmÃ©e. In 1813, Prussia and Austria joined Russian forces in a Sixth Coalition against France, resulting in a large coalition army defeating Napoleon at the Battle of Leipzig. The coalition invaded France and captured Paris, forcing Napoleon to abdicate in April 1814. He was exiled to the island of Elba, between Corsica and Italy. In France, the Bourbons were restored to power."""classBertTokenEstimator(TokenEstimator):def__init__(self):self.bert_tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")defestimate_tokens(self,text):returnlen(self.bert_tokenizer.encode(text))bert_token_estimator=BertTokenEstimator()print(f"Num of chars:{len(text)}")# --> Num of chars: 3149print(f"Num of tokens (using BertTokenEstimator):{bert_token_estimator.estimate_tokens(text)}")# --> Num of tokens (using BertTokenEstimator): 603# This creates an instance of the TextChunker class with a chunk size of 512,# using token-based segmentation and a custom tokenizer function bert_tokenizer.encode to count tokens.text_chunker=TextChunker(512,tokens=True,token_estimator=BertTokenEstimator())chunks=text_chunker.chunk(text)fori,chunkinenumerate(chunks):print(f"Chunk{i+1}, num of tokens:{bert_token_estimator.estimate_tokens(chunk)}->{chunk}")This code returns:Chunk 1, num of tokens 476: -> Napoleon Bonaparte (born Napoleone Buonaparte; 15 August 1769 â€“ 5 May 1821), later known by his regnal name Napoleon I, was a French military commander and political leader who rose to prominence during the French Revolution and led successful campaigns during the Revolutionary Wars. He was the de facto leader of the French Republic as First Consul from 1799 to 1804, then Emperor of the French from 1804 until 1814 and again in 1815. Napoleon's political and cultural legacy endures to this day, as a highly celebrated and controversial leader. He initiated many liberal reforms that have persisted in society, and is considered one of the greatest military commanders in history. His wars and campaigns are studied by militaries all over the world. Between three and six million civilians and soldiers perished in what became known as the Napoleonic Wars. Napoleon was born on the island of Corsica, not long after its annexation by France, to a native family descending from minor Italian nobility. He supported the French Revolution in 1789 while serving in the French army, and tried to spread its ideals to his native Corsica. He rose rapidly in the Army after he saved the governing French Directory by firing on royalist insurgents. In 1796, he began a military campaign against the Austrians and their Italian allies, scoring decisive victories and becoming a national hero. Two years later, he led a military expedition to Egypt that served as a springboard to political power. He engineered a coup in November 1799 and became First Consul of the Republic. Differences with the United Kingdom meant France faced the War of the Third Coalition by 1805. Napoleon shattered this coalition with victories in the Ulm campaign, and at the Battle of Austerlitz, which led to the dissolution of the Holy Roman Empire. In 1806, the Fourth Coalition took up arms against him. Napoleon defeated Prussia at the battles of Jena and Auerstedt, marched the Grande ArmÃ©e into Eastern Europe, and defeated the Russians in June 1807 at Friedland, forcing the defeated nations of the Fourth Coalition to accept the Treaties of Tilsit. Two years later, the Austrians challenged the French again during the War of the Fifth Coalition, but Napoleon solidified his grip over Europe after triumphing at the Battle of Wagram. Hoping to extend the Continental System, his embargo against Britain, Napoleon invaded the Iberian Peninsula and declared his brother Joseph the King of Spain in 1808.
Chunk 2, num of tokens 129: -> The Spanish and the Portuguese revolted in the Peninsular War aided by a British army, culminating in defeat for Napoleon's marshals. Napoleon launched an invasion of Russia in the summer of 1812. The resulting campaign witnessed the catastrophic retreat of Napoleon's Grande ArmÃ©e. In 1813, Prussia and Austria joined Russian forces in a Sixth Coalition against France, resulting in a large coalition army defeating Napoleon at the Battle of Leipzig. The coalition invaded France and captured Paris, forcing Napoleon to abdicate in April 1814. He was exiled to the island of Elba, between Corsica and Italy. In France, the Bourbons were restored to power.As you can see,chunkipycreated chunks smaller than the given input size,512, counted the tokens using BERT's tokenizer, and didn't cut sentences in half,
producing syntactically correct chunks of text.You can also useTextChunker'ssplit_text()method to split a text into smaller parts
without actually creating the chunks.
This can be useful, for example, when you want to apply further processing to
the text parts before creating the final chunks.OverlappingTextChunkerprovides you with theoverlapfunctionality.
The system aims to ensure that the last text parts of each segment do not exceed the maximum token limit defined for overlap.
For example, if thechunk_sizeis set to 100 and theoverlap_percentageis 0.1, the maximum number of overlapping tokens is 10.
Consequently, if there are sentences or text parts that fit within this token limit, they are added at the beginning of the next segment.
If not, they are skipped to maintain a good ratio between overlap and content.text_chunker=TextChunker(50,tokens=True,overlap_percent=0.3)# Set up test inputtext="In this unit test, we are evaluating the overlapping functionality."\"This is a feature of the TextChunker class, which is important for a proper context keeping. The "\"goal is to ensure that overlapping chunks are generated correctly. For this purpose, we have chosen a "\"long text that exceeds 100 tokens. By setting the overlap_percent to 0.3, we expect the "\"generated chunks to have an overlap of approximately 30%. This will help us verify the effectiveness "\"of the overlapping feature. The TextChunker class should be able to handle this scenario and "\"produce the expected results. Let's proceed with running the test and asserting the generated chunks "\"for proper overlap. "# Generate chunks with overlappingchunks=text_chunker.chunk(text)# Print the resulting chunksfori,chunkinenumerate(chunks):print(f"Chunk{i+1}:{chunk}")This would output:Chunk 1: In this unit test, we are evaluating the overlapping functionality. This is a feature of the TextChunker class, which is important for a proper context keeping. The goal is to ensure that overlapping chunks are generated correctly. For this purpose, we have chosen a long text that exceeds 100 tokens.
Chunk 2: For this purpose, we have chosen a long text that exceeds 100 tokens. By setting the overlap_percent to 0.3, we expect the generated chunks to have an overlap of approximately 30%. This will help us verify the effectiveness of the overlapping feature.
Chunk 3: This will help us verify the effectiveness of the overlapping feature. The TextChunker class should be able to handle this scenario and produce the expected results. Let's proceed with running the test and asserting the generated chunks for proper overlap.Provide a custom text split strategyIf you don't want to usestanzaor you need to use your custom text splitting logic,
you can provide it in the constructor, as shown in this example:fromchunkipyimportTextChunkerdefsplit_by_arrow(text):return[tfortintext.split("->")ift!=''and' ']text="This is a tokenized text -> with custom split strategy."# Create a TextChunker object with custom split strategytext_chunker=TextChunker(chunk_size=8,tokens=True,split_strategies=[split_by_arrow])chunks=text_chunker.chunk(text)# Print the resulting chunksfori,chunkinenumerate(chunks):print(f"Chunk{i+1}:{chunk}")This would output:Chunk 1: This is a tokenized text
Chunk 2: with custom split strategy.ContributingIf you find a bug or have a feature request, please open an issue onGitHub.
Contributions are welcome! Just fork the repository, create a new branch with your changes, and submit a pull request. Please make sure to write tests for your changes and to follow thecode style.LicenseTextChunker is licensed under theMIT License.