Title: IF-Guide: Influence Function-Guided Detoxification of LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: influence functions

LLM toxicity
Abstract: We study how training data contributes to the emergence of toxic behaviors in large language models. Most prior work on reducing model toxicity adopts *reactive* approaches, such as fine-tuning pre-trained (and potentially toxic) models to align them with human values. In contrast, we propose a *proactive* approach—IF-Guide—that leverages influence functions to identify and suppress harmful tokens in the training data. To this end, we first show that standard influence functions are ineffective at discovering harmful training records. We then present a novel adaptation that measures token-level attributions from training data to model toxicity, along with techniques for selecting toxic training documents and a learning objective that can be integrated into both pre-training and fine-tuning. Moreover, IF-Guide does not rely on human-preference data, which is typically required by existing alignment methods. In our evaluation, we demonstrate that IF-Guide substantially reduces both explicit and implicit toxicity—by up to 10$\times$ compared to uncensored models, and up to 3$\times$ compared to baseline alignment methods such as DPO and RAD—across both pre-training and fine-tuning scenarios. IF-Guide is computationally efficient: a billion-parameter model is *not necessary* for computing influence scores; a million-parameter model—with 7.5$\times$ fewer parameters—can effectively serve as a proxy for identifying harmful data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies how to detoxify LLMs.  The work first examines the normal influence function and demonstrates that it will impact the model’s utility by misclassifying nontoxic examples.  Then the authors propose a new token-level influence function along with a new loss function.  Training with If-guide can significantly reduce the toxicity of LLMs in comparison with commonly used filtering methods and even outperform DPO and RAD in toxicity. The authors also show that models trained with If-guide are more robust to gcg attack.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nthe work demonstrates the limits of influence functions in detoxifying \n\nThe proposed If-guide has impressive performance in detoxifying \n\nThe proposed If- guide can be applied during pre-training or post-finetuning where the method is always effective in detoxifying\n\nWeakness:\n\nIf-guide may reduce the utility as a trade-off.  Although the authors design different components to avoid that, if-guide tends to given worse performance than other methods on influency.\n\nThe experiments do not clearly showcase the method's contributions. If-guide mainly consists of (1) a new influence function (2) a new loss function for training, which are proposed to address the performance degradation of using a naive influence function. Because naive influence function may return nontoxic but important examples.  However, the results show that **IF-Guide** is significantly more effective at detoxification compared to the baselines, rather than maintaining the utility. This makes it unclear what contributions those designed components truly bring. For example, toxificity filtering can be combined with proposed loss function. If this method is worse than If-Guide, it suggests the influence function indeed identify some really important tokens. Overall, despite the impressive performance, more discussion on explaining why the method works may be more helpful and interesting. \n\nThe overall framework of If-guide is very close to this related work [1] apart from the influence function part. This makes it more important to explain the role of influence function which is lacking in the current version.\n\n[1] Korbak, Tomasz, et al. \"Pretraining language models with human preferences.\" *International Conference on Machine Learning*. PMLR, 2023.\n\nQUESTIONS:\n1 It may be better if the reasons why the normal influence functions do not work are clearer. For example, comparing the Influence function with detoxify in detecting toxic examples.\n\n2 what does the DPO and RAD in table 1 refer to? Do they mean, e.g., DPO is applied to the base model trained without any filtering? A cleaner comparison should be IF-Guide, word/ toxicity filtering, and then DPO/RAD applied to those respective pre-trained models.  Current Table 1 is a bit confusing.\n\n3 instead of dropping examples, token-level attribution is proposed to improve the utility and retain benign knowledge. But If-guide is somehow always worse than direct training with filtering on the utility task. Can you provide some explanation or hypothesis? Is If-guide better than using naive influence function in the utility tasks?\n\n4 On the other hand,  toxicity is reduced more than by filtering methods or the conventional influence function. But the motivation of if-guide is to avoid degradation in utility, as you stated in section 3.2.  Could you provide some explanations? Do you try example filtering with your newly designed influence function? is the accuracy better than word filtering or detoxify?  Does suppressing toxic tokens instead of dropping them improve utility?   \n\n\nOthers:  \n\nThe legend of figure  5 seems wrong.\n\nTable 1 is not consistent. The `` word filtering'' is sometimes at different row.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
IF-Guide proactively reduces LLM toxicity by identifying and suppressing harmful training tokens using influence functions. It outperforms alignment baselines without human preference data and is computationally efficient, achieving up to 10× toxicity reduction using smaller proxy models.

STRENGTHS AND WEAKNESSES:
Strengths:

1. IF-Guide leverages influence functions to proactively identify and suppress harmful tokens during training, rather than relying on post hoc alignment. This helps prevent “training-time contamination” instead of applying reactive fixes.

2. Unlike methods such as DPO and RAD, IF-Guide does not depend on costly human preference or reward data, reducing reliance on manual annotations.

Weaknesses

1. Influence function theoretically involves the inversion of the Hessian matrix, which significantly increases computational complexity. It would be helpful to provide a comparison of the computational cost of the proposed method relative to baseline approaches, to better contextualize its efficiency.

2. The current experiments are conducted on base models with fewer than 1B parameters. It remains unclear how well the proposed method generalizes to larger-scale models (e.g., 7B or 13B). Further evaluation on such models would enhance the completeness of the study.

3. Analyzing the correlation and overlap of identified toxic tokens across different base models could be an interesting direction. It may offer insights into whether different model scales attend to similar or divergent harmful patterns.

QUESTIONS:
See Weakness.

LIMITATIONS:
See Weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper uses influence functions to identify which tokens in the training data is toxic and proposes methods to suppress that. Influence functions from Koh and Liang is used to identify the toxic tokens. however, they show that standard influence function based methods are not sufficient to uncover toxic tokens. They propose differential and token-level attribution to particularly choose toxic tokens ignoring the benign tokens.

STRENGTHS AND WEAKNESSES:
The paper presents the application of Influence Functions for identifying toxic tokens in the pretraining set. This is an important topic and paper presents a novel solution with practical considerations for applicability to 1 billion token pretraining set. The experimental section is strong with a large number of insightful experiments. The experiments show the effectiveness of the method in terms of low EMT and TP compared to the baselines. Section 4.5 also addresses the effect of proxy models for toxicity detection and application to a different model. Overall, I believe it is a strong paper.

The only concern I have is how well will this method scale to trillion token level pretraining. Although the paper outlines a method for speed-up, IF computation is typically slow especially at large scales? Also the models chosen are all <=1B tokens in size. Will this method scale for larger models? Can a smaller proxy model be used for suppressed training on large models?

QUESTIONS:
See the above section for questions

LIMITATIONS:
I could not find a limitations section in the paper. One potential limitation might be how to scale this method for larger pretraining corpuses and larger models.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes IF-GUIDE, a method for proactively reducing toxicity in Large Language Models (LLMs) by intervening during the training process to identify toxicity through influence functions and mitigate their effects on training. \n\nThe core idea is to use influence functions as a data attribution technique (a concept that has been studied in literature), to then identify specific training tokens that are causally responsible for toxic model outputs. \n\nThe authors introduce several modifications to standard influence functions, including differential and token-level attribution, to improve their effectiveness for this task. They then propose a novel training objective that penalizes the model for generating these identified toxic tokens. The paper presents extensive experiments across multiple models and benchmarks often outperforming standard alignment baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper addresses the critical issue of LLM safety. Its proactive stance, aiming to prevent the learning of harmful associations rather than suppressing them post-hoc, is a valuable and well-motivated research direction.\n\n- The paper's experiments are its greatest strength. The authors conduct a thorough evaluation across a range of models, datasets, and settings (pre-training and fine-tuning). The results consistently show that IF-GUIDE is effective at reducing both explicit and implicit toxicity, providing strong evidence for the method's practical utility.\n---\n\n### Weakness\n1. The ability of $Q_{tox}$ to identify all forms of toxicity across a pre-training dataset, say of size trillions of tokens is bottlenecked by the coverage over this sample set of 10k examples.\n\n2. Standard influence functions are flawed for LLMs due to factors like fitting errors—is a known issue in the literature. Concurrent work, such as Wu et al. (2024), tackles this exact problem with an alternative \"debias and denoise\" methodology. Furthermore, the idea of token-level data cleaning for SFT has been explored by Pang et al. (2025). The present paper fails to cite, discuss, or compare against these highly relevant works, which makes it difficult to assess the relative novelty and contribution of the proposed techniques.\n\n3. If a model is trained to suppress all toxic content, how does it recognize toxic content? \nWhile the authors' penalty-based objective is more nuanced than simple filtering, the paper does not empirically evaluate the model's ability to handle meta-level questions about toxicity or to refuse harmful instructions.\n\n4.  The authors argue their \"suppression\" approach is superior because filtering is imprecise and data-inefficient. However, this argument is less compelling when one considers that only a tiny fraction of a multi-trillion token dataset is likely to be filtered, potentially making the collateral damage negligible.\n\nQUESTIONS:\n## Some references to study and include:\n\n#### In Toxicity\n\nThe area of toxicity mitigation is vast. The authors may find it worth their time to extend the related works section.\n\n\n#### In IFs for LLMs\n\nFurthermore, there are the following relevant works in the use of IF for LLMs, some of which are close to the author's work. \n\n*   Choe, S. K., Ahn, H., Bae, J., Zhao, K., Kang, M., Chung, Y., ... & Xing, E. P. (n.d.). *Large-Scale Training Data Attribution with Efficient Influence Functions*.\n*   Li, Z., Zhao, W., Li, Y., & Sun, J. (2024). *Do Influence Functions Work on Large Language Models?*. arXiv preprint arXiv:2409.19998.\n*   Pang, J., Di, N., Zhu, Z., Wei, J., Cheng, H., Qian, C., & Liu, Y. (2025). *Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning*. arXiv preprint arXiv:2502.01968.\n*   Wu, K., Pang, L., Shen, H., & Cheng, X. (2024). *Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration*. arXiv preprint arXiv:2410.01285.\n*   Xia, M., Malladi, S., Gururangan, S., Arora, S., & Chen, D. (2024). *LESS: Selecting influential data for targeted instruction tuning*. arXiv preprint arXiv:2402.04333.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-05"}