Scavenging Hyena: Distilling Transformers into Long Convolution Models
Authors: name: Tokiniaina Raharison Ralambomihanta, name: Shahrad Mohammadzadeh, name: Mohammad Sami Nur Islam, name: Wassim Jabbour, name: Laurence Liang

The rapid evolution of Large Language Models (LLMs), epitomized by
architectures like GPT-4, has reshaped the landscape of natural language
processing. This paper introduces a pioneering approach to address the
efficiency concerns associated with LLM pre-training, proposing the use of
knowledge distillation for cross-architecture transfer. Leveraging insights
from the efficient Hyena mechanism, our method replaces attention heads in
transformer models by Hyena, offering a cost-effective alternative to
traditional pre-training while confronting the challenge of processing long
contextual information, inherent in quadratic attention mechanisms. Unlike
conventional compression-focused methods, our technique not only enhances
inference speed but also surpasses pre-training in terms of both accuracy and
efficiency. In the era of evolving LLMs, our work contributes to the pursuit of
sustainable AI solutions, striking a balance between computational power and
environmental impact.