## 3 Proposed Method
### Entropy Invariance of Attention Mechanisms
Entropy invariance mainly refers to reducing the correlation between this uncertainty and length, so that regardless of how the length changes, the entropy remains essentially unchanged. From the perspective of entropy invariance, the formula for scaled dot-product attention can be rewritten as Formula (4). \[a_{ij}=\frac{e^{\lambda q_{i}\cdot k_{j}}}{\sum_{j=1}^{n}e^{\lambda q_{i}\cdot k _{j}}} \tag{4}\]  
where \(q_{i}\) and \(k_{j}\) represent the \(i\)-th query vector and \(j\)-th key vector, respectively, of the input sequence, and \(q_{i}\cdot k_{j}\) is the dot product of the two vectors, which reflects the correlation between them. \(\lambda\) is the scaling factor, which is independent of \(q_{i}\) and \(k_{j}\). \(a_{ij}\) is the conditional distribution of a random variable, and the expression of entropy is shown in Formula (5). \[\mathcal{H}_{i}=-\sum_{j=1}^{n}a_{ij}\log a_{ij} \tag{5}\]  
Substituting \(a_{ij}\) into Formula (5) yields Formula (6). \[ \begin{split}\mathcal{H}_{i}&=-\sum_{j=1}^{n}a_{i, j}\log a_{i,j}\\ &=\log\sum_{j=1}^{n}e^{\lambda q_{i}\cdot k_{j}}-\lambda\sum_{j=1 }^{n}a_{ij}e^{\lambda q_{i}\cdot k_{j}}\\ &=\log n+\log\frac{1}{n}\sum_{j=1}^{n}e^{\lambda q_{i}\cdot k_{j }}-\lambda\sum_{j=1}^{n}a_{ij}e^{\lambda q_{i}\cdot k_{j}}\end{split}  \tag{6}\]  
Formula (6) comprises three terms, with the \(\log\frac{1}{n}\sum_{j=1}^{n}e^{\lambda q_{i}\cdot k_{j}}\) operation in the second term involving taking the exponential function first and then averaging. To simplify the computation, the mean-field approximation method can be used in place of this operation. This method involves first averaging the values and then exponentiating the result, thus resulting in Formula (7). \[\log\frac{1}{n}\sum_{j=1}^{n}e^{\lambda q_{i}\cdot k_{j}}\approx\log\exp\left( \frac{1}{n}\sum_{j=1}^{n}\lambda q_{i}\cdot k_{j}\right)=\log\lambda\overline{q_ {i}\cdot k_{j}} \tag{7}\]  
Since the softmax function emphasizes the position of the maximum value, an approximate value for the third term in Formula (6), \(\lambda\sum_{j=1}^{n}a_{ij}e^{\lambda q_{i}\cdot k_{j}}\), can be obtained, as shown in Formula (8). \[\lambda\sum_{j=1}^{n}p_{ij}e^{\lambda q_{i}\cdot k_{j}}\approx\lambda\max_{1 \leq j\leq n}\left(e^{\lambda q_{i}\cdot k_{j}}\right) \tag{8}\]  
It is important to note that the semi-quantitative estimations in Formulas (7) and (8) are used to determine the appropriate scaling factor for compensating for the impact of length on entropy rather than completely disregarding it. Therefore, by substituting Formulas (7) and (8) into Formula (6), we can obtain  
\[ \begin{split}\mathcal{H}_{i}&\approx\log n+\log \lambda\overline{q_{i}\cdot k_{j}}-\lambda\max_{1\leq j\leq n}\left(e^{ \lambda q_{i}\cdot k_{j}}\right)\\ &\approx\log n-\lambda\left(\max_{1\leq j\leq n}\left(e^{ \lambda q_{i}\cdot k_{j}}\right)-\log\lambda\overline{q_{i}\cdot k_{j}}\right) \end{split}  \tag{9}\]  
The concept of entropy invariance aims to minimize the impact of sequence length n on entropy. Based on Formula (9), we can set  
\[\lambda\propto\log n \tag{10}\]  
Therefore, based on the principle of entropy invariance and some reasonable assumptions, we can derive a new scaling factor and a corresponding scaled dot-product attention mechanism, as shown in Formula (11). Specifically, we compute the scaling factor by taking the logarithm of the length of the target sequence and multiplying it with the dot-product score matrix. This scaling factor rescales the values in the score matrix to better adapt to the length of the target sequence. Although this scaling factor is just a scalar, it scales every element in the score matrix, resulting in a scaled attention weight matrix. Finally, the weighted average of the input value vectors with the scaled attention weights generates the final context vector. \[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{\log nQK^{T}}{\sqrt{d}} \right)V \tag{11}\]  
In conclusion, this study obtained a new scaling factor, which makes the model more robust to changes in input length.