# RTACompensator: Leveraging AraBERT and XGBoost for Automated Road Accident Compensation
## 4 Research Methodology
### RTACompensator
#### 4.2.2 The Part of Responsibility Model  
After the initial compensation amount is determined, a script extracts the accident description from the police report. This description is then fed into the responsibility classifier model to determine the part of the responsibility. Finally, Equation (5) is applied to compute the final compensation amount. The part of the responsibility model was developed by integrating AraBERT embeddings with XGBoost. XGBoost is a tree-based ensemble algorithm that utilizes a gradient boosting framework [21]. It is highly scalable and effective in handling classification tasks [54]. Unlike traditional methods, XGBoost updates predictions based on pseudo-residuals, which are the gradients of the loss function, rather than the target values (\(\tilde{y}\)), as shown in Equation (6). Figure 13 describes the general architecture of XGBoost. \[\tilde{y}^{(K)}=\tilde{y}^{(K-1)}+f_{K}(x_{i}) \tag{6}\]  
where the following are used:  
* \(\tilde{y}^{(K)}\) is the predicted value at '**K**' stage (tree). * \(\tilde{y}^{(K-1)}\) is the predicted value at '**K**-**1**' stage (tree). * \(f_{K}\) is the predicted pseudo-residual at '**K**' stage (tree). Figure 12: Example of processing a sentence from the Arabic corpus of responsibility. XGBoost aims to optimize the objective function, as presented in Equation (7), which is the sum of training loss and regularization. \[Obj(\theta)=\sum_{i=1}^{n}l(y_{i},\tilde{y_{i}})+\sum_{k=1}^{K}\Omega(f_{k}) \tag{7}\]  
with  
\[\Omega(f)=\gamma T+\frac{1}{2}\lambda||w||^{2} \tag{8}\]  
where **K** are total number of trees, **n** are total number of rows in the dataset, \(l(y_{i},\tilde{y_{i}})\) presents the loss function, and \(\Omega(f_{k})\) is the regularization that reduces the model complexity. In addition, optimizing the training loss ensures the development of an accurate predictive model, while regularization addresses the bias-variance tradeoff. The regularization term controls model complexity, helping to avoid overfitting and achieve a simpler model. BERT is a language representation model designed to pretrain deep bidirectional representations from large unlabeled text corpora, enhancing the understanding of text sequences [22]. Since our corpora are in Arabic, the only language used in Moroccan courts, we focused on AraBERT, which is the Arabic version of BERT. AraBERT [20] retains the original BERT architecture, as shown in Figure 14, with stacked transformer encoder layers and is specifically tailored to Arabic text, offering bidirectional context awareness. Figure 14: BERT architecture. Figure 13: The general architecture of XGBoost. However, it has a maximum sequence length of 512 tokens, which presents a challenge. To manage this limitation, we used a head and tail truncation strategy that selects the first 128 tokens and the last 382 tokens. We then prepend and append the [CLS] and [SEP] special tokens, respectively, to fit the model's input requirements, as illustrated in Figure 15.  
To build the AraBERT embeddings, we used the Hugging Face Transformers library in PyTorch. We opted for the base version of AraBERT, which is pretrained on 70 million Arabic sentences, amounting to 24 GB of text data. This model features 12 encoder layers, 768 hidden dimensions, 12 attention heads, a maximum sequence length of 512 tokens, and a total of 110 million parameters. For each document, we calculated the average embeddings of all words as the aggregate representative embedding for the entire document. The integration of AraBERT and XGBoost effectively combines AraBERT's ability to understand the semantic structure of the input text with XGBoost's robust classification capabilities to determine the part of responsibility in road accident cases. AraBERT processes the input by tokenizing the text and generating contextual embeddings for each token, capturing the nuanced meanings of phrases relevant to responsibility allocation.