## 3 Proposed Method
### Entropy Invariance of Attention Mechanisms  
By using attention mechanisms, it becomes possible to identify the most relevant segments of input data and concentrate on them. In the scaled dot-product attention mechanism, the scaling operation is used to ensure the entropy invariance of the attention distribution. This is achieved by multiplying the attention distribution by a scaling factor to adjust its magnitude, thus better controlling the learning efficiency and stability of the model. In the scaled dot-product attention, the input sequence is first mapped to query the vector \(Q\), key vector \(K\), and value vector \(V\). The dot product of \(Q\) and \(K\) is then calculated and scaled, and the scaled result is weighted and averaged with \(V\) to obtain the output of the self-attention mechanism. Specifically, assuming there are n input vectors, each with dimension d, the formula for calculating the scaled dot-product attention is  
\[\text{Attention}(Q,K,V)=\text{Softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}\cdot V \tag{3}\]  
The softmax function is used to normalize the attention distribution into a probability distribution. \(\sqrt{d}\) is the scaling factor used to prevent the dot-product result from being too large or too small, keeping the attention distribution within a suitable range and avoiding over-concentration, thus improving the model's robustness and generalization ability. In this paper, we introduce a method to maintain attention entropy invariance by using the scaling operation in the attention mechanism in the Transformer model. In Transformer, each attention head calculates attention for the input, and the results of multiple heads are fused to obtain the final output. During attention calculation, scaling the attention vector can maintain the entropy invariance of the attention distribution, better controlling the size and uniformity of the attention distribution. This paper addresses the shortage of disfluent speech data resources and the issue that the length of disfluent speech in practical scenarios is not fixed. We propose a model optimization method based on attention mechanism entropy invariance, which enables the model to generalize to speech data of different lengths, and achieve good results even when the length of fine-grained disfluent speech data used for training and testing is different. In this section, we optimize the model using attention mechanism entropy invariance to enhance its extrapolation ability, that is, the ability to infer the whole from the part, and enable a model trained on shorter disfluent speech data to be tested on longer data sequences without fine-tuning, achieving good results. Optimizing the extrapolation ability of the model length not only solves the problem of inconsistent lengths during training and prediction but also improves the model's generalization ability. Moreover, it allows an effective model to be trained on shorter data for use on longer data when disfluent speech data resources are limited, thus improving the practicality of the model. This paper references previous research on the entropy invariance of attention mechanisms [52; 53; 54] and, based on this perspective, redefines a new scaling factor for extrapolating length and introduces a constant \(\lambda\) to improve the length generalization ability of the attention mechanism. This enables the model to perform better in length extrapolation without changing its existing training performance. Information entropy is a measure of uncertainty, and in research, it is necessary to minimize uncertainty, i.e., minimize entropy. In our study, this uncertainty is considered the concentration of attention. Generally, training on longer data requires attention to be evenly distributed among all parts, making the attention more dispersed and resulting in greater entropy. Conversely, training on shorter data leads to more concentrated attention, resulting in smaller entropy.