Title: Memory-Efficient Training with In-Place FFT Implementation
Venue: NeurIPS 2025
Accepted: yes
Keywords: FFT

Inplace

Memory
Abstract: Fast Fourier Transforms (FFT) are widely used to reduce memory and computational costs in deep learning. However, existing implementations, including standard FFT and real FFT (rFFT), cannot achieve true in-place computation. In particular, rFFT maps an input of size $n$ to a complex output of size $\frac{n}{2}+1$, causing dimensional mismatch and requiring additional memory allocation.
We propose the first real-domain, fully in-place FFT framework (rdFFT) that preserves input-output dimensional consistency ($n \rightarrow n$). By leveraging butterfly operation symmetry and conjugate properties in the frequency domain, we design an implicit complex encoding scheme that eliminates intermediate cache usage entirely.
Theoretically, our method reduces memory usage by 50\% compared to rFFTs. Moreover, it enables zero-cache parameter updates by utilizing the derivative property of the Fourier transform to compute matrix inverses efficiently without intermediate storage. Experiments on multiple natural language understanding tasks demonstrate the method’s effectiveness in maintaining model performance while significantly lowering memory overhead, offering a promising direction for frequency-domain lightweight adaptation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to learn a new kind of Fast Fourier Transform (FFT) on real valued data which is in place and does not require any additional storage during computation making the operation more space efficient. Similar to the FFT, they leverage the recursive approach of a Butterfly graph and divide the computation of the real-valued FFT into subcomputations. However, their resulting frequency representation is only reported in real-valued numbers. Their evaluation showcases reduced memory usage for single layer training and model fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The proposed approach demonstrates significantly lower memory consumption compared to common FFT methods. This makes it highly relevant given the current trend of large models and input dimensions.\n- The proposed approach is interesting and relevant because, with growing model and dataset sizes, the use of frequency representation (e.g., for leveraging the convolution theorem) is becoming more popular. Additionally, the approach inherently lowers memory consumption. \n\nWeaknesses:\n1. The resulting models are evaluated solely on GPU consumption, but a performance evaluation of the finetuned models is missing. Consequently, it is hard to judge the actual functionality, meaning whether the proposed real-valued FFT works as intended. The authors should provide evaluations on the trained models.\n2. The applicability remains debatable. While the authors motivate their work with vision applications that leverage the frequency domain for efficient computation and enhanced model learning, the provided implementation appears to function only for 1D signals.\n3. Further, the separation into imaginary and real part into different array positions does imply that standard usage like the convolution theorem would need to be adapted. While this is discussed in the paper's limitations, no future strategy for tackling this problem is provided.\n\nQUESTIONS:\n1. Can the authors provide a full evaluation of the finetuned models' quality? Specifically, do performance numbers remain on par, or do they drop due to information loss?\n2. Given that the motivation for in-place FFT stems from vision tasks beyond 1D, how could the proposed approach be extended to 2D, 3D, or nD settings? Does the inherent split of real and imaginary parts preclude such extensions?\n3. What strategies might enable the proposed approach to regain the inherent interaction offered by complex numbers?\n\nLIMITATIONS:\nThe authors identify their paper's greatest limitation as the split into distinct real and imaginary numbers, which prevents direct use of complex forms. Consequently, as they state in lines 315-316, \"additional logic is required to decode the real-valued encoding into a usable complex form.\"\n\nI hope the authors can elaborate on how to adapt their approach to work like complex values and/or in settings beyond 1D.\n\nFurthermore, while I believe this work is a step in the right direction, a thorough evaluation of the resulting model quality is necessary to judge the effectiveness of the proposed approach.\n\nIf the authors manage to address these points, I would be happy to raise my score.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is introducing an in-place FFT framework that can be used in deep learning with reducing memory. The main contribution from the authors include 1.The rdFFT, a real-valued, fully in-place Fourier transform operator that eliminates memory space mismatches, offering improved practicality and usability for neural network applications compared to existing libraries. 2. A new memory layout design, develop a novel butterfly execution scheme for IFFT computation, and provide support for the bfloat16 data type. 3. Authors have integrated the proposed rdFFT into neural network models via circulant-structured parameter matrix and validate on real-world models. \nThe experimental results show the proposed method significantly reduces memory consumption.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper presents a novel in-place Fast Fourier Transform (FFT) operations for model training through the use of circulant-\nstructured parameter matrices, which have shown a significant memory reducing compared with other fft libraries. The method enables fully in-place FFT computation for real-valued input, with zero memory overhead and compatible with the Cooley–Tukey algorithm.\n2. The submission demonstrates a strong technical foundation with a robust experimental design and clear evidence supporting the claims presented in the paper. The experiments include different settings with the comprehensive memory analysis.\n3. The paper is well-structured and clearly written, making it accessible to a broader audience. \n4. This work have offered a promising direction for frequency-domain lightweight adaptation.The proposed method has the potential for wide adoption and could inspire future research in this area.\n\nWeakness:\n1. Other than the memory cost, accuracy impact is not fully investigated. Authors would also be curious about the accuracy/error results on the proposed fft operations. \n2. By storing the real and imaginary parts only, the impact of conjugate reconstruction would also provide more proof, such as computational overhead, error sensitivity, etc\n\nQUESTIONS:\n1. As well as the promising memory reducing results, is there any numeric accuracy comparision between standard fft, rfft and proposed method? or any end-to-end training result.\n2. By storing the real and imaginary parts only, is the impact of conjugate reconstruction investigated? such as computational overhead, error sensitivity, etc\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes rdFFT, a real-domain fully in-place fast Fourier transform operator for neural-network training. By interleaving the N + 2 real outputs of a conventional real-input FFT back into the original N slots and proving that this layout is preserved throughout the Cooley–Tukey computation, rdFFT performs both forward and backward passes without extra memory movement. A succinct CUDA kernel with native bf16 and FP16 support eliminates copy and padding operations, lowers peak GPU memory by roughly 25~30\\% versus rFFT, and keeps throughput within three percent on LLaMA-2-7B and RoBERTa-Large fine-tuning tasks running on a single A100 80 GB GPU, offering a lossless operator-level alternative that can complement or precede parameter-efficient adaptation methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n- Operator-level novelty: The interleaved layout compresses N + 2 real slots into N and is accompanied by a rigorous symmetry proof. \n- True in-place execution: Both forward FFT and backward IFFT run without auxiliary buffers, including gradient overwrite. \n- bf16 support: Existing cuFFT/FFTW kernels lack bf16; rdFFT supports it natively. \n\n## Weakness\n- No accuracy metrics: The paper assumes “lossless” equivalence but never reports validation accuracy/F1 on MRPC, GSM8K or any task, leaving the practical impact unverified.\n- No throughput metrics: The paper never discloses latency or training throughput, which seriously limits the practical relevance of the method. \n- No FP8 support: While the implementation covers FP32 and bfloat16, it does not support the FP8 formats now available on Hopper/Blackwell-class GPUs, leaving potential memory- and bandwidth gains untapped.\n\nQUESTIONS:\n- What is the end-to-end training throughput of LLaMA-2-7B compared with full-fine tuning, LoRA, and rFFT?\n- Please report validation accuracy/F1 for MRPC and GSM8K under full fine-tuning, LoRA, rFFT-circulant, and rdFFT-circulant.\n- Modern GPUs expose E4M3/E5M2 Tensor-Core paths. Can your in-place layout be extended to FP8, and what would be the numerical-stability considerations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces rdFFT, a real-domain, in-place Fast Fourier Transform (FFT) framework intended to reduce memory use in neural network training. The method uses butterfly operation symmetry and conjugate properties to create an implicit complex encoding scheme that operates within the original real-valued memory buffer. This approach is designed to address the memory-space mismatch in standard real FFT (rFFT) implementations. The paper reports that experiments on RoBERTa-large and LLaMA2-7B show reductions in peak memory usage compared to other FFT methods and LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper proposes what it claims is the first truly in-place FFT operator for real-valued data that resolves the input-output dimension mismatch of standard rFFT, contributing at the level of a fundamental arithmetic operator.\n\n2. The authors validate the method with both isolated single-layer analyses and full-model fine-tuning on large-scale models (LLaMA2-7B and RoBERTa-large), providing a breakdown of memory usage.\n\n3. The proposed rdFFT is benchmarked against standard complex FFT, rFFT in both full fine-tuning and LoRA. The paper's results indicate lower memory usage compared to these methods under the tested conditions.\n\n4. The implementation include BF16, which is important in modern LLMs while lacks proper implementation in traditional FFT libraries.\n\nWeaknesses:\n\n1. As I understand, this method only supports matrix sizes of powers of 2, which may limit the scope of it as many layers in modern LLMs are not of those sizes.\n\n2. Another important aspect, training speed, is not properly discussed in this paper.\n\nQUESTIONS:\n1. Could you benchmark on the speed of this operator?\n\n2. Could you discuss more on the kernel tuning results if possible (to see whether this operator fits a large range of matrix sizes)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}