# A Multitask Cross-Lingual Summary Method Based on ABO Mechanism
## 3 Methods
### Reinforced Regularization Method  
Fully trained large language models typically have a high number of model parameters, which can provide substantial prior knowledge, but the limited data resources available for downstream tasks can result in overfitting during model fine-tuning. Therefore, it is necessary to introduce an appropriate regularization strategy to reduce overfitting and improve the model's generalization performance. The baseline model does not incorporate dropout regularization during the pre-training phase because it may inhibit the model's fitting effect and reduce its learning ability. However, during the fine-tuning phase, the model may suffer from overfitting when facing a single downstream task, so adding a more effective regularization method can significantly improve the fine-tuning performance of the model. As a result, a reinforced regularization method is added during fine-tuning. During training, performing forward calculations with different sampling processes for the same input \(x\) is equivalent to the same data sample going through two sub-networks and obtaining two different distributions, \(P_{1}\) and \(P_{2}\). The final weighting of the cross-entropy of the two components is shown in the following equation:  
\[L_{i}^{(CE)}=-logP_{\theta}(y_{i}|x_{i})-logP_{\theta}^{\prime}(y_{i}|x_{i}). \tag{1}\]  
To keep the output of the path network consistent across dropouts, the \(K\)L scatter is calculated. \[L_{i}^{(KL)}=\frac{1}{2}[KL(P_{\theta}(y_{i}|x_{i})\parallel P_{\theta}^{ \prime}(y_{i}|x_{i}))+KL(P_{\theta}^{\prime}(y_{i}|x_{i})\parallel P_{\theta} (y_{i}|x_{i}))]. \tag{2}\]  
The final loss after two loss calculations is the weighted sum of the two losses, which is calculated as shown below:  
\[L_{i}=L_{i}^{(CE)}+\alpha L_{i}^{(KL)}. \tag{3}\]  
In the equation above, \(\alpha\) represents the weight coefficient of the \(KL\) loss and is the only hyperparameter. By weighting the two losses with \(\alpha\), the model space is further regularized, compensating for the inconsistency of dropout in training and testing, and improving the model's generalization ability. The above method performs data augmentation while maintaining the input semantics, which boosts the confidence level of the primary categories but increases the training cost by adding numerous non-target classes in the prediction stage. To address this issue, we sparsify the output to provide a positive gain for the regular method via the category-invariant property. Typically, the softmax function is capable of mapping multiple neuron outputs to the \((0,1)\) interval, thereby enabling the numerical assignment of approximate probabilities. The common exponential form of softmax [26] is calculated as  
\[p_{i}=\text{softmax}(s_{i})=\frac{e^{s_{i}}}{\sum_{j=1}^{n}e^{s_{j}}}. \tag{4}\]  
In the above equation, \(s_{i}\) represents the score of an output result, \(p_{i}\) represents the corresponding probability, and \(n\) denotes the total number of output results. The softmax function transforms the set of \(n\) real-valued scores into a probability distribution. The decoder part uses softmax for two main functions: (1) calculating the normalized attention weights, and (2) computing the prediction probability distributions. However, traditional softmax cannot assign a probability of 0 to any predictor, making it impossible to exclude low-probability predictors. To address this, sparse softmax is utilized instead, where the hyperparameter \(k\) is manually set to fix the category and complete the initial screening of low-probability words. \[p_{i}= \begin{cases}\frac{e^{i}}{\sum_{l\in N_{k}}e^{i}}&i\in N_{k}\\ 0&i\
otin N_{k}\end{cases} . \tag{5}\]  
The output logits of the fully connected layer are denoted as \(S_{i}\). These logits are sorted from largest to smallest, and the set of subscripts of the first \(k\) elements is denoted as \(N_{k}\). At this point, sparse softmax only retains the probability values of the first \(k\) elements after sorting when calculating the probability.