Catastrophic Token Leakage in Convolutional Sequence Modeling and How to Mitigate It

Authors: Naman Agarwal, Evan Dogariu, Xinyi Chen, Daniel Suo, Vladimir Feinberg, Elad Hazan
Primary area: foundation or frontier models, including LLMs
Keywords: Convolutional Models, LLMs, Language Modeling, Fast Fourier Transforms

Abstract:
Sequence models based on long convolutions have recently gained significant interest as an alternative to attention models in large scale language modeling due to fast training and inference enabled via the Fast Fourier Transform (FFT). Our work begins with an observation that sequence models based on FFT-based convolutions can have catastrophic leaking of future tokens. This striking failure of causality occurs due to numerical errors in the standard FFT. We provide a solution to the problem via Number-Theoretic FFTs which are executed solely on integers. Our method provably ensures no token leakage, providing a safe primitive for convolutional models in general. To align with current deep learning practice we provide a complete implementation using 32bit integers and leveraging standard integer matrix multiplications.

Decision: reject

Reviews and discussion:

Metareview: