ToxiSight: Insights Towards Detected Chat Toxicity

Authors: zachary  yang
domenico  tullo
reihaneh  rabbany
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-16 05:21:00

Abstract: We present a comprehensive explainability dashboard designed for in-game chat toxicity. This dashboard integrates various existing explainable AI (XAI) techniques, including token importance analysis, model output visualization, and attribution to the training dataset. It also provides insights through the closest positive and negative examples, facilitating a deeper understanding and potential correction of the training data. Additionally, the dashboard includes word sense analysis—particularly useful for new moderators—and offers free-text explanations for both positive and negative predictions. This multi-faceted approach enhances the interpretability and transparency of toxicity detection models.

Main paper: 
 introduction: 
 Toxic and harmful speech in online platforms is an escalating concern, impacting the safety and inclusivity of digital spaces. Detecting and mitigating toxic speech is a critical task, where it has evolved significantly over the years. From early approaches that relied on traditional machine learning models with manually engineered features (Watanabe et al., 2018) , the field has progressed to the application of deep neural networks (Gambäck and Sikdar, 2017; Zhong et al., 2016; Gao and Huang, 2017; Fehn Unsvåg and Gambäck, 2018) and, more recently, the utilization of pre-trained language models (PLMs) (Yang et al., 2023) . These advancements have led to improved performance across a variety of NLP tasks, including toxicity detection.In parallel with these advances in detection, the importance of explainability in NLP models has grown. As models become more complex, the need to understand and interpret their decisions-especially in sensitive applications like toxicity detection-has led to the development of various explainability techniques. These techniques range from feature importance analysis (Ribeiro et al., 2016) and surrogate modeling (Ribeiro et al., 2016) to example-driven explanations and provenance-based methods (Pezeshkpour et al., 2019) . Visualizing these explanations (Bahdanau et al., 2014; Mullenbach et al., 2018) effectively is crucial for both refining models and making them accessible to moderators, particularly new moderators who need to understand the reasoning behind model predictions.In this context, we introduce ToxiSight, a multifaceted explainability dashboard designed to enhance the transparency and interpretability of toxicity detection models. ToxiSight integrates existing explainability approaches, including token importance, model output analysis, and attribution to the training dataset with closest positive and negative examples. Additionally, it offers word sense distribution insights and generates free-text explanations for both positive and negative predictions. This comprehensive approach not only aids in the detection and correction of toxic content but also serves as a valuable tool for new moderators, helping them to better understand the nuances of toxicity in online communications. 
methodology: 
 The development of the ToxiSight dashboard follows a structured approach that integrates various explainability techniques to create a comprehensive tool for understanding and analyzing the inferences made by toxicity detection models. This section outlines the steps involved in implementing the ToxiSight dashboard, detailing each module's function and purpose.The Chat Body module visualizes both the input to the detection model and its corresponding output. The specific chat message under analysis is prominently displayed, with any toxic spans highlighted Beneath the chat message, the dashboard shows the predicted label, indicating whether the content is classified as toxic or non-toxic. If the label is toxic, the module also identifies the target of the toxicity, if applicable. To the right of the chat message, a detailed distribution of the model's output probabilities across different toxicity classes is provided. This offers a granular view of the model's decision-making process.The Word Sense module enhances interpretability by analyzing specific words or phrases identified as potentially toxic. For the highlighted span, ToxiSight determines the most likely meaning of ambiguous or context-dependent words, which is crucial for understanding why certain words were flagged as toxic. The module leverages authoritative sources like Webster, Oxford, and crowdsourced sources such as Urban Dictionary to provide definitions, displaying multiple senses for each word ranked by relevance to the context, with ac-companying confidence scores. This analysis helps to clarify the specific usage and intent behind the language in the chat, contributing to a more nuanced understanding of the model's output.The Training Data Attribution module employs example-driven explainability by tracing the model's prediction back to its training data. This is achieved by identifying a positive example, a similar sample from the training dataset that closely matches the input and supports the same classification. Additionally, the module provides a counterexample, which is a contrasting sample that would have led to a different classification, such as nontoxic. If no suitable counter-example exists, this section may be empty, due to the absence of a sufficiently similar instance. These examples are identified through similarity measures, ensuring that the samples provided are the most relevant and informative for understanding the model's behavior.The Explanation Generation module leverages large language models (LLMs) such as LLaMA-3 and GPT-4o to generate free-text explanations that clarify the model's decision. The model is prompted to generate two types of explanations: one that justifies why the input was classified under the given label and another that explores why the input was not classified under an alternative label. These explanations are designed to help researchers and moderators understand the model's reasoning, providing both the justification for its decision and the plausible deniability of alternative classifications. This dual approach offers a balanced perspective, aiding in both the validation and critique of the model's outputs. 
results: 
 The implementation of the ToxiSight dashboard has significantly enhanced our understanding of toxicity detection models by providing detailed insights into their decision-making processes. Preliminary testing shows that the dashboard effectively highlights ambiguous cases, allowing for more informed moderation decisions. Furthermore, the integration of training data attribution and word sense analysis has improved the model's interpretability, enabling users to trace predictions back to specific examples and understand the contextual nuances that influence toxicity classification.While the ToxiSight dashboard offers a comprehensive tool for understanding toxicity detection models, there are some limitations to consider. First, the reliance on large language models (LLMs) for generating explanations may introduce biases inherent in the models themselves, potentially skewing the interpretations provided. Additionally, the example-driven approach for training data attribution depends heavily on the quality and diversity of the training dataset. If the dataset lacks representation of certain contexts or language variations, the attributions may be less reliable or informative. The dashboard also assumes that the most relevant word senses and training examples can be accurately identified, which may not always be the case, particularly in complex or highly nuanced conversations. Lastly, while the dashboard aids in interpreting model outputs, it does not guarantee improved performance or fairness in toxicity detection, and there may still be challenges in generalizing its insights across different domains or user groups. 
 
 Appendix:  


Decision: This submission presents a comprehensive explainability dashboard that integrates several existing explanation approaches for toxicity detection models. It is a good thematic fit for the workshop.