Understanding and Predicting Derailment in Toxic Conversations on GitHub
Authors: name: Mia Mohammad Imran; name: Robert Zita; name: Rebekah Copeland; name: Preetha Chatterjee; name: Rahat Rizvi Rahman; name: Kostadin Damevski
Published: 2025-03-04 02:01:37

Abstract: Software projects thrive on the involvement and contributions of individuals
from different backgrounds. However, toxic language and negative interactions
can hinder the participation and retention of contributors and alienate
newcomers. Proactive moderation strategies aim to prevent toxicity from
occurring by addressing conversations that have derailed from their intended
purpose. This study aims to understand and predict conversational derailment
leading to toxicity on GitHub.
  To facilitate this research, we curate a novel dataset comprising 202 toxic
conversations from GitHub with annotated derailment points, along with 696
non-toxic conversations as a baseline. Based on this dataset, we identify
unique characteristics of toxic conversations and derailment points, including
linguistic markers such as second-person pronouns, negation terms, and tones of
Bitter Frustration and Impatience, as well as patterns in conversational
dynamics between project contributors and external participants.
  Leveraging these empirical observations, we propose a proactive moderation
approach to automatically detect and address potentially harmful conversations
before escalation. By utilizing modern LLMs, we develop a conversation
trajectory summary technique that captures the evolution of discussions and
identifies early signs of derailment. Our experiments demonstrate that LLM
prompts tailored to provide summaries of GitHub conversations achieve 69%
F1-Score in predicting conversational derailment, strongly improving over a set
of baseline approaches.