Title: Mitigating Text Degeneration via Token-Level Guidance For pruned Large Language Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Large Language model, Text Degnertaion, Pruning, Distillation

Abstract: Large language models (LLMs) suffer from substantial memory and inference costs, and pruning has emerged as a widely adopted strategy for compression. However, while pruning effectively reduces model size and latency, it often exacerbates undesirable side effects such as text degeneration, particularly repetition, even when perplexity remains largely intact. We observe that standard post-pruning fine-tuning is insufficient to suppress repetition, motivating the need for more targeted approaches. To address this issue, we propose two token-level guidance methods: FOCUS and PT. FOCUS adjusts token probabilities through token-weighted distillation, focusing on high-confidence regions to better align the student with the teacher while reducing the likelihood of undesirable tokens.
In contrast, PT employs contrastive training with negative and positive samples to explicitly encourage the generation of alternative tokens. Experiments on open-ended and instruction-based generation tasks demonstrate that our methods substantially reduce repetition and improve generation diversity, while causing only minimal impact on perplexity. Furthermore, our methods are compatible with other training strategies and consistently enhance their performance. Code will be available soon.

TL;DR: Surpressing text degeneration for pruned large language model