## 3 Methodology
### Feature Extraction
#### 3.4.3 BERT Embedding  
BERT embeddings represent the words in a sentence as numerical, multidimensional vectors. These vectors capture both the meaning and structure of the words.  
BERT is a pre-trained natural language processing model built on transformer architecture, developed by Google to understand the bidirectional context of words within sentences. Unlike traditional models, such as Word2 Vec and GloVe, BERT represents words contextually [24], meaning that each word is presented differently depending on the surrounding words in a sentence.  
BERT's architecture (Figure 5) starts with the tokenization of words into smaller units, or subwords, using a method called WordPiece. This approach is applied when words are not part of BERT's vocabulary. For example, the word \"loving\" could be split into \"low\" and \"#ing\" if the model does not recognize the full word. Additionally, BERT uses two special tokens: [CLS] at the beginning of each sentence, which is used for classification tasks, and [SEP], which marks the end of sentences and helps distinguish between them. After tokenization, the text is processed by the BERT model, which has three main components: bidirectional encoding, multi-layered transformers, and a masked language model. Bidirectional encoding considers the context of words from both directions, helping to understand each word's meaning in its context. The multi-layered transformers apply a self-attention mechanism, allowing each word to connect with every other word in the sentence. Finally, the masked language model predicts masked tokens to learn deep contextual relationships between words. The next stage is fine-tuning, where the model is adapted to our specific dataset. The final output from BERT is a set of contextual embeddings for each word in the sentence.  
BERT offers several models in different languages. In this study, we focus on the English language using distilbert-base-uncased ([[https://huggingface.co/distilbert/distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased)]([https://huggingface.co/distilbert/distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased)), accessed on 17 November 2024), a distilled version of BERT. For Arabic, we use asafaya-bert-base-arabic ([[https://huggingface.co/asafaya/bert-base-arabic](https://huggingface.co/asafaya/bert-base-arabic)]([https://huggingface.co/asafaya/bert-base-arabic](https://huggingface.co/asafaya/bert-base-arabic)), accessed on 17 November 2024), a pre-trained BERT base model for Arabic. For French, we work with CamemBERT ([[https://huggingface.co/almanach/camembert-base](https://huggingface.co/almanach/camembert-base)]([https://huggingface.co/almanach/camembert-base](https://huggingface.co/almanach/camembert-base)), accessed on 17 November 2024), a BERT adaptation designed specifically for the French language. Although BERT models are pre-trained on raw text, we applied the same preprocessing steps used for other feature extraction techniques. This unified preprocessing strategy was adopted to ensure consistency and fairness in the comparative evaluation across all models. We note, however, that this approach may diverge from BERT's original training conditions and could influence its contextual embedding behavior, particularly in morphologically rich languages.