Meta-Chunking: Learning Efficient Text Segmentation via Logical
  Perception
Authors: name: Jihao Zhao; name: Zhiyuan Ji; name: Pengnian Qi; name: Simin Niu; name: Bo Tang; name: Feiyu Xiong; name: Zhiyu Li
Published: 2024-10-16 17:59:32

Abstract: Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline, which impacts the quality of knowledge-intensive
tasks. This paper introduces the concept of Meta-Chunking, which refers to a
granularity between sentences and paragraphs, consisting of a collection of
sentences within a paragraph that have deep linguistic logical connections. To
implement Meta-Chunking, we designed two strategies based on LLMs: Margin
Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform
binary classification on whether consecutive sentences need to be segmented,
making decisions based on the probability difference obtained from margin
sampling. The latter precisely identifies text chunk boundaries by analyzing
the characteristics of perplexity distribution. Additionally, considering the
inherent complexity of different texts, we propose a strategy that combines
Meta-Chunking with dynamic merging to achieve a balance between fine-grained
and coarse-grained text chunking. Experiments conducted on eleven datasets
demonstrate that Meta-Chunking can more efficiently improve the performance of
single-hop and multi-hop question answering based on RAG. For instance, on the
2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only
consuming 45.8% of the time. Our code is available at
https://github.com/IAAR-Shanghai/Meta-Chunking.