Cascading Adversarial Bias from Injection to Distillation in Language
  Models
Authors: name: Harsh Chaudhari; name: Jamie Hayes; name: Matthew Jagielski; name: Ilia Shumailov; name: Milad Nasr; name: Alina Oprea
Published: 2025-05-30 17:41:58

Abstract: Model distillation has become essential for creating smaller, deployable
language models that retain larger system capabilities. However, widespread
deployment raises concerns about resilience to adversarial manipulation. This
paper investigates vulnerability of distilled models to adversarial injection
of biased content during training. We demonstrate that adversaries can inject
subtle biases into teacher models through minimal data poisoning, which
propagates to student models and becomes significantly amplified. We propose
two propagation modes: Untargeted Propagation, where bias affects multiple
tasks, and Targeted Propagation, focusing on specific tasks while maintaining
normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning
rate), student models generate biased responses 76.9% of the time in targeted
scenarios - higher than 69.4% in teacher models. For untargeted propagation,
adversarial bias appears 6x-29x more frequently in student models on unseen
tasks. We validate findings across six bias types (targeted advertisements,
phishing links, narrative manipulations, insecure coding practices), various
distillation methods, and different modalities spanning text and code
generation. Our evaluation reveals shortcomings in current defenses -
perplexity filtering, bias detection systems, and LLM-based autorater
frameworks - against these attacks. Results expose significant security
vulnerabilities in distilled models, highlighting need for specialized
safeguards. We propose practical design principles for building effective
adversarial bias mitigation strategies.