# Hate Speech Detection by Using Rationales for Judging Sarcasm
## 4 Experimental Conditions
### Metrics for Evaluation  
In the quest to uncover hate speech masked within sarcasm, a range of metrics covering diverse dimensions of hate expression was considered. Taking cues from the HateXplain paper's approach to hate speech classification, emphasis was placed on two primary types of metrics: performance-based and explainability-based.  
* _Performance-based metrics:_ Priority was given to these metrics, with accuracy, macro F1-score, and AUROC score reported for both the baseline and proposed method in accordance with established standards. These metrics serve to assess the classifier's performance in distinguishing among the three classes: hate speech, offensive speech, and normal speech;
* _Explainability-based metrics:_ We adhered to the framework outlined in the ERASER benchmark by to gauge the explainability aspect of the model. This assessment involved measuring the plausibility and faithfulness. Plausibility denotes the persuasiveness of the interpretation to humans, while faithfulness pertains to the accuracy with which it reflects the genuine reasoning process of the model [44; 45]. 1. _Plausibility:_ To assess plausibility, metrics for both discrete and soft selection were employed. Specifically, the Intersection-Over-Union (IOU) F1-score and token F1-score for the discrete case were reported, along with the Area Under the Precision-Recall curve (AUPRC) score for soft token selection [45]. The IOU metric was utilized for credit assignment in cases of partial matches. This metric, defined at the token-level, measures the overlap between two spans divided by the size of their union [45]. A prediction was considered a match if the overlap with any ground truth rationale exceeded 0.5. These partial matches were used to compute the IOU F1-score. Token-level precision and recall were also calculated, and these were used to derive token-level F1 scores (token F1). Additionally, to assess the plausibility of soft token scoring, the AUPRC was reported by sweeping a threshold over the token scores;
2. _Faithfulness:_ To assess faithfulness, two metrics were reported: comprehensiveness and sufficiency [45]. - _Comprehensiveness:_ This metric evaluates the extent to which the model's predictions are affected by the removal of predicted rationales. For each post \(x_{i}\), a contrast example \(\tilde{x}_{i}\) was created by removing the predicted rationales \(r_{i}\) from \(x_{i}\). Let \(m(x_{i})_{j}\) be the original prediction probability provided by a model \(m\) for the predicted class \(j\). Then, \(m(x_{i}|r_{i})_{j}\) represents the predicted probability of \(\tilde{x}_{i}\) (i.e., \(x_{i}\) without \(r_{i}\)) by the model \(m\) for class \(j\). Comprehensiveness was computed as \(m(x_{i})_{j}-m(x_{i}|r_{i})_{j}\). A higher value of comprehensiveness suggested that the rationales significantly influenced the prediction; - _Sufficiency:_ This metric measures the adequacy of the extracted rationales for the model to make a prediction. Sufficiency was calculated as \(m(x_{i})_{j}-m(r_{i})_{j}\), where \(m(r_{i})_{j}\) represents the predicted probability of the rationales \(r_{i}\) by the model \(m\) for class \(j\).