## 3 Methodology
### Entropy-Optimized Dynamic Text Segmentation
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**No** & **Method** & **Implementation Summary** & **Major Deficiency** \\ \hline
1 & Fixed-size chunking & Define a fixed character or word limit (e.g., 512 or 128 words per chunk). Split the text into & Semantic fragmentation, uneven & \\  & & equal-sized chunks. & & information density [34] \\ \hline
2 & Sliding window chunking & Define a window size (e.g., 256 words) and a step & & \\  & & size (e.g., 64 words). Slide the window across the & & \\  & & text to create overlapping chunks. & & sensitive [35; 36] \\ \hline
3 & Paragraph-based chunking & Split text at paragraph boundaries (e.g., line breaks). & Dependence on formatting quality, & \\  & & & uneven paragraph length [37] \\ \hline
4 & Sentence-based chunking & Split text at sentence boundaries (e.g., periods or & & \\  & & question marks). & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Some text segmentation methods. Using the chain rule of entropy, the conditional entropy can be decomposed into:  
\[H(W|c_{i})=H(W,c_{i})-H(c_{i}), \tag{4}\]  
The Equation (4) establishes the computational relationship between conditional entropy and joint entropy. The specific implementation algorithm of EDTS is shown in Algorithm 1. The input parameter _content_ represents the entire read-in text, and the output parameter _chunks_ represents the segments of the text after splitting. ```
inputs: content outputs: chunks
```  
1. Read in the entire text content;
2. Count the frequency of each word to form a word probability dictionary, where the key is the word and the value is the probability;
3. Use periods as delimiters to form chunks and the set \(S=[s_{1},s_{2},s_{3},\ldots,s_{m}]\);
4. Assume the current calculation is for sentence \(s_{i}\), the preceding sentence is \(s_{j-1}\), and the following sentence is \(s_{j+1}\);
5. Split \(s_{j}\) into a word list \(W=[w_{1},w_{2},w_{3},\ldots,w_{m}]\), using jieba segmentation;
6. Calculate the right boundary, use jieba segmentation to split \(s_{j+1}\) into a word list \(C=[c_{1},c_{2},c_{3},\ldots,c_{k}]\);
7. Calculate the joint entropy \(H(W,c_{1})\);
8. Calculate the entropy \(H(c_{1})\);
9. Calculate the conditional entropy \(H(W\wr c_{1})\), according to Formula (4);
10. Repeat steps (7) to (9) to obtain a sequence of conditional entropies \(H_{r}\) expanding word by word to the right, as well as the list of conditional entropy for the right boundary \(H_{r}\);
11. According to Formula (3), the boundary position on the right is determined by \(i_{r}=\arg\min(H_{r})\);
12. Repeat steps (5) to (11) to obtain the left boundary position \(i_{l}\) by expanding word by word to the left;
13. Form a word list for a chunk with the starting position at the left boundary \(i_{l}\) and the ending position at the right boundary \(i_{r}\);
14. Repeat steps (4) to (13) to traverse the sentences in \(S\) sequentially, wherein \(i_{l}\) in the first sentence is the first word of the entire text, and \(i_{r}\) in the last sentence is the last word of the entire text;
15. Form a chunk by taking the word at position \(i_{l}\) as the starting word and the word at position \(i_{r}\) as the ending word;
16. Traverse all statements in \(S\) and form the final output _chunks_. **Algorithm 1. EDTS Algorithm**  
EDTS aims to divide external knowledge texts into independent chunks. Using sentences as the central units, it dynamically expands the context window to the left and right. The optimal segmentation boundaries are determined by calculating the conditional entropy; the schematic diagram is shown in Figure 2. Specifically, the position with the minimum conditional entropy within the window is chosen as the segmentation point, ensuring that each text chunk has semantic completeness and high predictability. Finally, the text chunks are encoded as dense vectors and stored in a vector database.