Title: A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Safety, Adversarial Robustness

Abstract: Many safety post-training methods for large language models (LLMs) are designed to modify the model’s behaviour from producing unsafe answers to issuing refusals. 
However, such distribution shifts are often brittle and degrade performance on desirable tasks.
To address these pitfalls, we propose augmenting the model’s vocabulary with a special red flag token ($\langle\texttt{rf}\rangle$) and training the model to insert this token whenever harmful content is generated or imminent. 
This approach enables the model to explicitly learn the concept of harmfulness in its representations, with minimal impact on utility due to the marginal change in the generated distribution of natural language. 
Moreover, because the token is embedded in the model’s vocabulary, we can naturally leverage the LLMs' generalisation capabilities, such as in-context learning (ICL) and out-of-distribution generalisation to languages that are not formally supported (e.g., Japanese for Llama3). 
In particular, we demonstrate that through ICL alone, the model can learn to initiate reflective reasoning upon generating $\langle\texttt{rf}\rangle$ at inference, which steers the response away from harmful continuations or enables self-correction when the flag is raised falsely.
This approach is orthogonal and complementary to existing safety techniques—such as safety classifiers or standard safety training---and easier to evaluate in comparison to natural language refusals, as it does not require a human or automated judge to assess the harmlessness of the answers.

TL;DR: Teaching the model to generate a special red flag token at any point in the conversation when it turns harmful