Repository: deepset-ai/haystack
Pull request: feat: add split_threshold to document splitter to avoid excessively short splits


### Related Issues

- fixes [#7275](https://github.com/deepset-ai/haystack/issues/7275)

### Proposed Changes:
Pass a split_threshold to the document splitter, if the last chunk is small than the threshold, attach it to the previous one.
This avoids having small chunks of 2 or 3 words that are not meaningful during the RAG.

### How did you test it?

I wrote a unit test


Problem statement:
Add split_threshold option to DocumentSplitter
**Is your feature request related to a problem? Please describe.**
If I'm splitting documents at a length of 1000, but I have a document that has a length of 1100, I'd rather not split the document.

**Describe the solution you'd like**
I'd like to be able to specify a split_threshold where a document (or the last chunk of a split document) must be larger than for it to be split.  Example:
```
splitter = DocumentSplitter(split_length=1000, split_overlap=200, split_threshold=1500);
```
This would only split a document (or the last piece of a document) if there are >= 1500 words remaining.

**Describe alternatives you've considered**
Manually checking the content length of a document before sending it to the splitter.  This would not prevent the last piece of a longer document from being split, nor is easily done in the middle of a pipeline.

Hints from issue discussion:

Commit message: feat: add split_threshold to document splitter to avoid excessively small splits

Patch to review:
diff --git a/haystack/components/preprocessors/document_splitter.py b/haystack/components/preprocessors/document_splitter.py
index 294f375e9e..1f2b0f0b6c 100644
--- a/haystack/components/preprocessors/document_splitter.py
+++ b/haystack/components/preprocessors/document_splitter.py
@@ -25,6 +25,7 @@ def __init__(
         split_by: Literal["word", "sentence", "page", "passage"] = "word",
         split_length: int = 200,
         split_overlap: int = 0,
+        split_threshold: int = 0,
     ):
         """
         Initialize the DocumentSplitter.
@@ -44,6 +45,7 @@ def __init__(
         if split_overlap < 0:
             raise ValueError("split_overlap must be greater than or equal to 0.")
         self.split_overlap = split_overlap
+        self.split_threshold = split_threshold
 
     @component.output_types(documents=List[Document])
     def run(self, documents: List[Document]):
@@ -74,7 +76,9 @@ def run(self, documents: List[Document]):
                     f"DocumentSplitter only works with text documents but document.content for document ID {doc.id} is None."
                 )
             units = self._split_into_units(doc.content, self.split_by)
-            text_splits, splits_pages = self._concatenate_units(units, self.split_length, self.split_overlap)
+            text_splits, splits_pages = self._concatenate_units(
+                units, self.split_length, self.split_overlap, self.split_threshold
+            )
             metadata = deepcopy(doc.meta)
             metadata["source_id"] = doc.id
             split_docs += self._create_docs_from_splits(
@@ -102,27 +106,34 @@ def _split_into_units(self, text: str, split_by: Literal["word", "sentence", "pa
         return units
 
     def _concatenate_units(
-        self, elements: List[str], split_length: int, split_overlap: int
+        self, elements: List[str], split_length: int, split_overlap: int, split_threshold: int
     ) -> Tuple[List[str], List[int]]:
         """
         Concatenates the elements into parts of split_length units keeping track of the original page number that each element belongs.
+
+        If the length of the current units is less than the pre-defined split_threshold, it does not create a new split. Instead, it concatenates the current units with the last split, preventing the creation of excessively small splits.
         """
-        text_splits = []
+
+        text_splits = []  # type: List[str]
         splits_pages = []
         cur_page = 1
         segments = windowed(elements, n=split_length, step=split_length - split_overlap)
         for seg in segments:
             current_units = [unit for unit in seg if unit is not None]
             txt = "".join(current_units)
-            if len(txt) > 0:
+            # check if length of current units is below split_threshold
+            if len(current_units) < split_threshold and len(text_splits) > 0:
+                # concatenate the last split with the current one
+                text_splits[-1] += txt
+            elif len(txt) > 0:
                 text_splits.append(txt)
                 splits_pages.append(cur_page)
-                processed_units = current_units[: split_length - split_overlap]
-                if self.split_by == "page":
-                    num_page_breaks = len(processed_units)
-                else:
-                    num_page_breaks = sum(processed_unit.count("\f") for processed_unit in processed_units)
-                cur_page += num_page_breaks
+            processed_units = current_units[: split_length - split_overlap]
+            if self.split_by == "page":
+                num_page_breaks = len(processed_units)
+            else:
+                num_page_breaks = sum(processed_unit.count("\f") for processed_unit in processed_units)
+            cur_page += num_page_breaks
         return text_splits, splits_pages
 
     @staticmethod
diff --git a/releasenotes/notes/document-splitter-accept-split-threshold-467abb9fcd1c316b.yaml b/releasenotes/notes/document-splitter-accept-split-threshold-467abb9fcd1c316b.yaml
new file mode 100644
index 0000000000..9638f39f60
--- /dev/null
+++ b/releasenotes/notes/document-splitter-accept-split-threshold-467abb9fcd1c316b.yaml
@@ -0,0 +1,5 @@
+---
+enhancements:
+  - |
+    DocumentSplitter now accepts split_threshold, if when chunking one of the chunk is smaller than threshold, it will be concatenated with the previous one.
+    This avoids having too small chunks that are not meaningful.
diff --git a/test/components/preprocessors/test_document_splitter.py b/test/components/preprocessors/test_document_splitter.py
index 417598ff01..3c052aed99 100644
--- a/test/components/preprocessors/test_document_splitter.py
+++ b/test/components/preprocessors/test_document_splitter.py
@@ -50,6 +50,21 @@ def test_split_by_word(self):
         assert result["documents"][0].content == "This is a text with some words. There is a "
         assert result["documents"][1].content == "second sentence. And there is a third sentence."
 
+    def test_split_by_word_with_threshold(self):
+        splitter = DocumentSplitter(split_by="word", split_length=15, split_threshold=10)
+        result = splitter.run(
+            documents=[
+                Document(
+                    content="This is a text with some words. There is a second sentence. And there is a third sentence."
+                )
+            ]
+        )
+        assert len(result["documents"]) == 1
+        assert (
+            result["documents"][0].content
+            == "This is a text with some words. There is a second sentence. And there is a third sentence."
+        )
+
     def test_split_by_word_multiple_input_docs(self):
         splitter = DocumentSplitter(split_by="word", split_length=10)
         result = splitter.run(


Code review comments:
diff_hunk: @@ -102,27 +106,34 @@ def _split_into_units(self, text: str, split_by: Literal["word", "sentence", "pa
         return units
 
     def _concatenate_units(
-        self, elements: List[str], split_length: int, split_overlap: int
+        self, elements: List[str], split_length: int, split_overlap: int, split_threshold: int
     ) -> Tuple[List[str], List[int]]:
         """
         Concatenates the elements into parts of split_length units keeping track of the original page number that each element belongs.
+
+        If the length of the current units is less than the pre-defined split_threshold, it does not create a new split. Instead, it concatenates the current units with the last split, preventing the creation of excessively small splits.
path: haystack/components/preprocessors/document_splitter.py
text: @user1:
```suggestion
        If the length of the current units is less than the pre-defined `split_threshold`, it does not create a new split. Instead, it concatenates the current units with the last split, preventing the creation of excessively small splits.
```

diff_hunk: @@ -102,27 +106,34 @@ def _split_into_units(self, text: str, split_by: Literal["word", "sentence", "pa
         return units
 
     def _concatenate_units(
-        self, elements: List[str], split_length: int, split_overlap: int
+        self, elements: List[str], split_length: int, split_overlap: int, split_threshold: int
     ) -> Tuple[List[str], List[int]]:
         """
         Concatenates the elements into parts of split_length units keeping track of the original page number that each element belongs.
+
+        If the length of the current units is less than the pre-defined split_threshold, it does not create a new split. Instead, it concatenates the current units with the last split, preventing the creation of excessively small splits.
         """
-        text_splits = []
+
+        text_splits = []  # type: List[str]
path: haystack/components/preprocessors/document_splitter.py
text: @user1:
```suggestion
        text_splits: List[str] = []
```