## 3 Proposed Framework
### Extracting Domain-Independent Features Based on Pivot Words  
When there are fewer shared features between the source and target languages, the performance of cross-language OLD tends to decline. Effectively utilizing domain-specific features enhances detection capabilities. Typically, domain-specific pivot words in the dataset co-occur with domain-shared pivot words that exhibit similar offensive tendencies. To extract domain-specific features, it is essential to construct an auxiliary task closely related to the original task.  
Firstly, the PERL method proposed in the paper [60] is employed to select domain-shared feature words, known as pivots. Words that meet the criteria of being adjectives, adverbs, or verbs with high attention weights are chosen as pivots for attack judgment. Subsequently, pivots are concealed in the training data of both the source and target languages, and a special UNK tag is used to replace all the pivots in the original data.  
In BERT, the UNK (Unknown) label is employed to denote Out-Of-Vocabulary (OOV) terms, referring to words not encountered during the model's training, absent from the pre-trained vocabulary. Using the UNK label aids the model in better generalizing across different languages. Given BERT's capacity to grasp contextual information during pre-training, it is anticipated that the model can infer the semantic details of UNK-labeled words through context, thereby enhancing performance in tasks like cross-language text analysis. This approach also mitigates the model's reliance on pivot words for learning task labels, facilitating improved adaptation to new languages.  
The CLT training of mBERT, concealing pivot words and training a model for detecting offensive text, offers several advantages:  
1. Enhanced Generalization: By concealing pivot words, the trained model avoids over-reliance on specific vocabulary, thereby improving its ability to generalize to a broader range of offensive text and increasing overall applicability.
2. Reduced Overfitting: Over-dependence on pivot words during training may lead to overfitting, where the model performs well on training data but poorly on unseen data.  
Hiding these crucial words diminishes overfitting tendencies, enhancing robustness to new data.
3. Improved Contextual Understanding: Hiding pivot words compels the model to rely more on contextual information to discern text offensiveness. This fosters a better understanding of textual context, improving the model's capability to handle complex language structures and implicit information.
4. Enhanced Model Robustness: Concealing pivot words during training increases the likelihood of the model learning more universal and general features of offensive text, bolstering its robustness across different contexts.  
We introduce an auxiliary label, \(y_{\textit{off}}\), for each transformed data instance. If the original data instance contains at least one offensive pivot, label \(y_{\textit{off}}\) as 1; otherwise, label it as 0. The transformed data serves as the input to the feature extraction layer of the second network branch to generate feature representations, predicting the auxiliary label \(y_{\textit{off}}\). The loss function is denoted as \(L_{\textit{off}}\). By training on the prediction task of auxiliary labels, the second network branch can extract domain-specific features coinciding with domain-shared features.