Memory-Efficient Training with In-Place FFT Implementation
Authors: Xinyu Ding
Bangtian Liu
Siyu Liao
Zhongfeng Wang
Venue: NeurIPS 2025 poster ()
Keywords: FFT
Inplace
Memory
Primary area: deep_learning

Abstract: Fast Fourier Transforms (FFT) are widely used to reduce memory and computational costs in deep learning. However, existing implementations, including standard FFT and real FFT (rFFT), cannot achieve true in-place computation. In particular, rFFT maps an input of size $n$ to a complex output of size $\frac{n}{2}+1$, causing dimensional mismatch and requiring additional memory allocation.
We propose the first real-domain, fully in-place FFT framework (rdFFT) that preserves input-output dimensional consistency ($n \rightarrow n$). By leveraging butterfly operation symmetry and conjugate properties in the frequency domain, we design an implicit complex encoding scheme that eliminates intermediate cache usage entirely.
Theoretically, our method reduces memory usage by 50\% compared to rFFTs. Moreover, it enables zero-cache parameter updates by utilizing the derivative property of the Fourier transform to compute matrix inverses efficiently without intermediate storage. Experiments on multiple natural language understanding tasks demonstrate the method’s effectiveness in maintaining model performance while significantly lowering memory overhead, offering a promising direction for frequency-domain lightweight adaptation.

Memory-Effcient Training with In-Place FFT Implementation
Xinyu Ding', Bangtian Liu, Siyu Liao'; Zhongfeng Wang'
School of Integrated Circuits, Sun Yat-sen University
(dingxy33, 1iaosy36, vangzf83)-0mall amail.sysu edu. cn, liubangtian0gmail. com.
Abstract
Fast Fourier Transforms (FFT) are widely used to reduce memory and compu- tational costs in deep learning. However, existing implementations, including standard FFT and real FFT (rFFT), cannot achieve true in-place computation. In particular, rFFT maps input of size causing dimensional mismatch and requiring additional memory allocation. We propose the first real-domain, fully in-place FFT framework (rdFFT) that preserves input-output memory space consistency. By leveraging butterfly operation sym- metry and conjugate properties the frequency domain, we design an implicit complex encoding scheme that eliminates intermediate cache usage entirely. Exper- iments on multiple natural language understanding tasks demonstrate the method effectiveness in reducing training memory cost, offering promising direction for frequency-domal lightweight adaptation.
complex output of size n/2 1,
Introduction
Large-scale neural models have achieved remarkable success across a wide range of applications, such as natural language processing [ and computer vision 3] As model sizes increase, memory consumption has emerged as a significant challenge. Notably, model training memory cost is larger than the deployment cost, primarily due the backpropagation process 4 Thus, reducing memory usage has become a critical research study, especially for the training stage.
Numerous methods have been proposed to reduce memory usage during model training and deploy- ment. Commonly used approaches such as model distillation [5], quantization (6), and pruning [7] mainly aim to reduce memory consumption by decreasing the number precision of model param- eters. In contrast, this work takes a novel approach at the arithmetic operator level, implementing in-place Fast Fourier Transform (FFT) operations for model training through the use of circulant- structured parameter matrices 81 The FFT operator has been widely employed in various neural network architectures due to its efficiency in capturing global patterns and enabling structured transformations. For instance, Fourier-based fine-tuning methods such as FourierFT [1]] and Block Circulant Adapter (BCA) [10 rely heavily on FFT to perform efficient parameter transformations in the frequency domain.
It should be noted that FFT-based operators typically involve both FFT and IFFT computations, which generate intermediate tensors that consume memory and use different data type since the result of FFT is complex number. Numerous studies have explored memory optimization strategies for FFT operators, particularly in high-ieheee perforerrfomecce programming libraries such as FFTW [12] and cuFFT [33. These libraries support in-place computation, where input and output share memory space. However, they suffer from the inability to maintain the original (input) memory space and lack of support for bfloat 6 data type that is widely used for modern neural models.
Siyu Liao is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To overcome the limitations of existing FFT libraries, we introduce rdFFT real real-domain, fully in-place Fourier transform that produces the same oulput as rFFT, but operates entirely within the original real-valued input memory space. We notice that the first and middle point of FFT results have zeros in their imaginary part. Thus, they can be squeezed together so the final output only requires real-valued input memory space. Besides, our method exploits the symmetry of butterfly operations and the conjugate structure of real-valued spectra to implicitly encode complex information within real-valued tensors. This design enables in-place computation without the need for auxiliary buffers or dimension mismatches, facilitating seamless integration into modern deep learning workflows. Crucially, our in-place design supports consistent forward and backward passes entirely within the real domain. In summary, our main contributions are as follows:
We propose rdFFT, a novel real-valued, fully in-place Fourier transform operator that eliminates memory space mismatches, offering improved practicality and usability for neural network applications compared to existing libraries.
We introduce new memory layout design, develop novel butterfly execution scheme for IFFT computation, and provide support for the bfloat 6 data type, which is widely used in modern neural networks.
We integrate our rdFFT into neural network models via and validate on real-world models, achieving zero-memory allocation for intermediate tensor computations.
parameter matrix
2
Related Works
Current automatic differentiation frameworks often discourage in-place operations due to their challenges in gradient computation during training. The key benefit of in-place computation lies in its ability to save memory by eliminating the need for storing intermediate tensors. For instance, combining batch normalization and activation into a single in-place operation has been shown to reduce memory usage by up to half (14. In neural network based hardware placement tasks [15], in-place operations are found beneficial due to their inherent memory efficiency. Similarly, in-place operations are integrated into convolutional neural networks for anomaly detection [16]. achieving notable memory savings. The in-place operation can also be extended to broader contexts. For example, in-place distillation [17), a student model is distilled directly from the teacher model without additional memory allocation.
The FFT operator can be found across various neural network models, particularly in tasks involving the Fourier domain, which is common in computer vision. Some convolutional neural networks run entirely in the Fourier domain (18). In beampatter synthesis [19), an IFFT operator is applied the hidden representations generated by the neural network. Additionally, 2D FFT has proven beneficial for fine-tuning large language models [ srruutudt matrix-vector products can also be efficiently computed using 1D FFT and IFFT, facilitating neural network training [IO). Beyond
these applications, FFT also plays a key role in spectral convolution 20 and in approximating global attention mechanisms (21).
From the perspective of automatic differentiation, each operator be seen a neural network layer, with or without trainable parameters. For example, the low-rank fine-tuning method for large language models [22] represents a full weight matrix using two low-rank matrices, which is equivalent to introducing two linear layers, thereby requiring storage for intermediate activations. Similarly, cireulant-strucctueed weight matrices [10] leverage FFT and IFFT transform inputs and parameters. Although FFT operators themselves does not contain trainable parameters, they still require the preservation of intermediate results to support automatic differentiation and handle valued data types that is different from the real-valued inputs. Moreover, libraries such as FFTW [2] and cuFFT [ demand pre-allocated memory buffers of size N + 2 real numbers. This memory pre-allocation should be handled during the neural model loading phase, which complicates integration and limits the practicality of using these libraries real-world neural network applications. Besides, they also does not support 6 data type which is common in modern neural networks.

---

3 Preliminary
3.1 Standard FFT and rFFT
Fast Fourier Transform (FFT) a computationally efficient algorithm for computing the Discrete Fourier Transform and its inverse. Given a sequence of N real or complex values æ(n), n 0,1,....A the FFT and inverse FFT are defined follows:
N-1 Σ n=0
N-1
x kn Ia
yk e 2
In
N
k=
This formulation, referred the standard FFT, transforms N input elements into N complex outputs. However, when the input is real, the FFT exhibits Hermitian symmetry, i.e. UN-k = Tk- Real-valued input based FFT (rFFT) exploits this property by computing only the first N/2 + complex values.
The rFFT significantly reduces the memory requirement compared to the complex-value FFT, decreasing the output size from 2N real numbers (i.c., N complex numbers) to N + 2 real numbers. This improvement is based the following fundamental property of the FFT:
Theorem (Conjugate Symmetry of Real FFT (3)). Let TRN be real-valued sequence, and let ys denote its Fast Fourier Transform (FFT). Then the FFT output satisfies the conjugate symmetry property:
N 2
UN-K = Tk. for all k = 1,2,
This property implies that the FFT of real-valued signal is redundant- the full frequency-
domain spectrum can be uniquely reconstructed from only the first 1 + complex coefficients:
yo: -/..... Leveraging this redundancy, rFFT implementations store only the non-redundant
half of the spectrum, thereby reducing both computation and memory footprint.
However, this comes the cost of mismatched memory sizes between the input and output: in the FFT, N real-valued inputs are transformed into outputs that take the memory space of N + 2 real values; in the IFFT, the complex inputs taking the space of N + 2 real values are mapped back to N real outputs. In both directions, the input and output cost different amount of memory.
This memory misalignment is difficult for neural network training, where tensors are generally allocated with fixed shapes. The requirement expand an N-element real tensor N + elements necessitates either pre-allocation or runtime reallocation, which causes integration difficulty, pre- vents true in-place computation and potentially incur significant overhead in memory-ccosstriiee environments.
3.2 In-place Transform via Butterfly Operation
The FFT can be efficiently computed using the Cooley-Tukey algorithm 24
which recursively decomposes given FFT into smaller FFTs, As illustrated in the Butterfly Operation Diagram section of [ig[[] 16-point FFT is first recursively decomposed into smaller FFTs-two 8 8-point FFTs, then four 4-point FFTs, and finally eight 2-point FFTs. Following this hierarchical decomposition, the FFT is computed through series of butterfly operations applied each level of the recursion. At the core of the algorithm lies the butterfly operation, a fundamental computation that transforms pair of complex values into outputs using addition, subtraction, and multiplication by twiddle factor.
As illustrated in the blue-shaded region Fig I the "Complex-to-Comeee FFT" section provides two butterfly operations, both originating from FFT16, which are highlighted in red. The butterfly operation for an r-point FFT defined as:
yk = Iy IKt Wm
(3)
yk+ = Ik Ik+ Wm.
N.

---

Storage Forrat of Different FFTs (8-FF))
Buttorfly Operation Disgr
Goopl Compl
TFT Mel Line
10
LFFT
72
Ao Kie
W
Ba byl
8,
Bec Sagl
X,
Bas!
hu
Sun Banl Kul
Ses Basrd
complex sumbers
441
Basl k42
2/4 Xxx W.
Uyys Recal
Bu Aud
Comp tox Comp
(Red
(11
1: hd
Gg Dya 11
V/2 Yio WW"
1....
8: Auf
0, Daf
ccmples sumbers
2097
E-FT
+ Bel
W.
8. 4. 2u
8, Bgg
Eloat numbers
4m
Rza M5p
2,56 - Yoa Wy
2,72 Y5e
O.
1a. D.al
922 Beal N2
Pa + W3 dy2 +hyal
Fleat to Flost
byal = No
Nu + Xaa Wi = +hyd
Ggo (2x6/11)
bundt (2m6//))
(Red FF7-16)
Euu
A22e + besol = X1o
fi Aso Wg 956 byal
P
8, 20062266/10
(26666))
9:2e - Aasol 154
WW 4,z -yal
0, (205in(222200) Goo0
Syp
6ya byal ma may + We
Oz 2iol22 ase
(o,+t 6,2/2
Float-to-Flost IFFT
(Red Line in FF7-16)
Oyn bysl ye mas Kex
0% Mtal/2 byal
Ba
(g 2,0/2
byd Mauta Kin W9
Xio (2 22o)/22 WA dass
Basel
Bas
[ a. ( (222/)) - Ob b,) 22
Pyg
8ya bydl - -o W/
25e 0% 71a) W;
Basel
Bas
[[ . o, J,i (266/N) (r 3r ( N
Figure 1: Overview of our method and its differences from standard FFT and rFFT implementations. The green section depicts the Butterfly Operation Diagram using a 16-point FFT (16-FFT) as an example. The orange section illustrates the storage formats of different FFT implementations, shown on an 8-point FFT (8-FFT). Two representative butterfly computation paths in the 16-FFT highlighted in red, and expanded into: (i) the blue section showing Complex-to-Comppe and IFFT operations, and (ii) the red section showing Float-to-Float FFT and IFFT operations- derived from the red paths in the 16-FFT diagram. This figure summarizes the key computational flows and memory layouts addressed by our in- in-paace real-domain FFT design.
FFT
where W is the twiddle factor, N is the total size of the FFT, and is the size of the current sub-FFT. The exponent m is given by m = 1 + log, (N/r). For each r-point FFT, there are y such butterfly pairs, with & 0, ......../2... r/2 1. The same butterfly operations
used in the "Complex-to- tocllllex IFFT" section, differing only in the choice twiddle factors. The overall structure of the Tukey FFT and its inverse (IFFT) is almost the same, except that the IFFT applics a final normalization factor I/N.
It can be noticed that butterfly operations are inherently in-place, allowing outputs to overwrite inputs without the need for extra memory. This makes FFT especially suitable for memory-efficiee implementations. However, since most neural network computations involve real-valued tensors, converting these to complex-valued representations usually requires additional memory allocation. With real-valued input and in-place constraint, given the aforementioned memory mismatch, the butterfly process not directly applicable.
3.3 FFT based Model Training via Circulant Matrix
The circulant matrix based neural network training can be converted to FFT and IFFT operations for acceleration 8 Given real-valued circulant weight matrix C E NNNN defined by its first column c E NR,, the linear transformation y Cx can be equivalently computed the frequency domain as follows:
y IFFT (FFT(c) O FFT(x)),
where @ denotes elementwise multiplication.
This structure not only accelerates computation but also simplifies the gradient computation during training. By leveraging the linearity and conjugate of the FFT results [10] gradients with respect to both input x and parameter e can computed by:
Oc 0x
dc Oy
Oc = IFFT FFT(x) O FFT
aL Qy
IFFT FFT(C) O FFT
(5)
where FFT(.) means taking the conjugate of the FFT results. There is also block circulant matrix based training that aims fit non-square matrix 9 where the matrix divided into blocks by partition size p.

---

Method
4.1
In-place FFT with Real-valued Input
While the Cooley-Tukey algorithm enables efficient in-place computation of the Fast Fourier Trans- form (FFT), it operates inherently in the complex domain. However, in modern neural networks, the vast majority of model parameters are real-valued. Transitioning between real and complex represen- tations introduces unnecessary memory overhead, especially in cemory-constrained training settings. To address this issue, we propose new strategies to reduce the memory footprint of Fourier-based transformations at the layer level.
Conjugate Symmetry in Cooley-Tukey Sub-FFTs. The Cooley-Tukey algorithm recursively decomposes length-N FFT into smaller FFTs. If the original input sequence R is real-valued, then each recursively computed sub-FFT also receives real-valued input (or linear combination of real values). Since the FFT linear operation and conjugate symmetry is preserved under linear combinations, each sub-FFT applied to real-valued data also satisfies the conjugate symmetry property:
1/
for all k = 1,
2
(6)
where y denotes the output of a size-r FFT block. Thus, conjugate symmetry is preserved every level of the Cooley-Tukey decomposition. This property enables memory memory-efficient computation in real-input FFTs throughout the recursive stages.
Squeeze N + into N. Although the rFFT reduces memory usage of complex-valued FFT by exploiting conjugate symmetry, it stores N/2 complex numbers in the memory space of N + 2 real numbers, which is different from the the original real-valued input of length N. This result does not allow for true in-place computation within the original real-valued buffer. We further analyze the structure of the sub-FFFTs the Cooley-Tukey decomposition. For any r-point FFT of real-valued input, the output satisfies:
No, Mr/2 E and y/ Mr-k, for 1, ...,, r/2 1.
This implies that only the real parts of yo and Ur/2 need be stored. For remaining values, we only store the real and imaginary parts of y1...., -:/2-1/2211111 and the rest can be reconstructed via conjugation. As result, we successfully reduce the memory space of N + real values the size of N real values.
2 complex
Memory Layout Design. Based on aforementioned observation, we propose data layout design where each complex coefficient y& k: < r/2) stores its real part at index k and stores its imaginary part at the conjugate-symmetrc index r k. In this way, the entire frequency-domait representation fits into real-valued buffer of size r, without requiring any complex-valued memory. The special cases yo and 1r/2, which are always real, each occupy a single real-valued slot. The "Storage Format Different FFTs" in iillustraats our layout design in comparison standard FFT and rFFT formats.
Proposition 1. In the Cooley-Tukey FFT algorithm on real-valued input, at each stage of the recursive decomposition, every smjjggatesyymmetric pair and its butterfly counterparts form a symmetric four- element group. This structural symmetry ensures that the butterfly operations preserve conjugate symmetry and can be performed entirely in-place.
Proof. Consider two conjugate-symmetric outputs s(a1) and z(b1) from an m-point FFT stage, where their indices are given by
a1 2km + -i, b, = 2km + +i
for some integers k and In the next 2m-point FFT stage, these values participate in butterfly operations with their counterparts at
a2 (2k + 1)m + i, b2 =2k + 1) 1)m + 2 + i.
5

---

Define the center of the 2m-point FFT block as c= c + 1)m.
Then, the offsets from the center are:
01
mt + i,
b1 1 b2
02
+
This confirms that the set {ai, b1. 22 b2) is symmetric with respect to the center index c, and forms two manjjuatt-syymeeent pairs placed at symmetric Since the FFT butterfly operations preserve conjugate symmetry when applied to conjugate inputs, the output values at this stage also maintain the same symmetry.
Therefore, all computations involving these four values can be performed in-place, and the symmetric layout remains valid in the next FFT stage.
This symmetry ensures that at every level the decomposition, each butterfly involving a conjugate pair produces another conjugate pair. As result, conjugate symmetry is recursively preserved across all stages of the Cooley-Tukey FFT algorithm.
As concrete example, consider the case shown in Fig. T (Float-to-Fioon Floott FFT), where an 8-point FFT stage has a conjugate pair located at indices a = 2 and = 6. the following 16-point FFT stage, these values participate in butterfly operations with elements at indices 10 and 14. The resulting index pairs (2, 14) and (6, 10) are symmetrie with respect to the center of the 16-point block and thus also form conjugate pairs.
Consequently, the interleaved memory layout-wai where the real part of yk stored index & and the imaginary part at index r - k- rermmins consistent throughout the entire FFT computation. This enables the algorithm to be executed entirely in-place within a real-valued buffer, eliminating the need to explicitly store redundant conjugate components.
Based on the recursive structure and symmetry, we summarize our in in-place real-domain FFT algo- rithm:
1. Store real and imaginary parts of conjugate pairs an interleaved layout within the input real-valued memory space;
2. Perform all butterfly operations entirely in-place without auxiliary buffers;
3. Reconstructs the complex frequency spectrum from real input of length N.
Overall, design enables fully in-place FFT computation for real-valued input, with zero memory overhead and compatible with the Cooley-Tukey algorithm.
4.2 In-place IFFT with Symmetric Complex-valued Input
While the FFT benefits from conjugate symmetry in real-valued inputs, the inverse transform receives conjugate-symmetricc complex values input. However, unlike FFT computation, the outputs of sub-IFFTs in the Cooley-Tukey recursion are not guaranteed to be real-valued or conjugate-structured. This makes it difficult directly upply the aforementioned in in-place memory sharing mechanism.
To overcome the input difference in IFFT, we exploit the lincarity the FFT. Since each butterfly reverse the forward FFT computation structure
operation is linear combination of its inputs, to recover the original real-valued signal. Specifically, we compute the inverse using the same butterfly graph but reverse the direction of data flow and scale the results appropriately.
As illustrated Fig (Float-to-Floo IFFT section), we compute intermediate outputs from the complex input Y follows:
4/2 + 9/10
Mo + 1/1
32
FB
2
2
V/ 910
V6 2/1
310
T14
2W3
2W8 &
where W exp (-2mik/N) denotes the k-th twiddle factor.
In this formulation, we split each conjugate pair into symmetric and anti-symmetric components, enabling recovery of the original signal through only real-domain operations. By carefully reusing buffer locations during this process, we implement the in-place IFFT computation without auxiliary memory for intermediate complex arrays.

---

Table 1: Peak GPU memory usage (in MB) measured during single layer training (up to the end of the backward pass) for different methods under varying input shapes. For inputs of shape D = 4096, LoRA uses rank 64; for D 1024, the rank 32. Entries marked as "N/A" " indicate that the specified block size (e.g., 4096) is incompatible with the given input shape (e.g.. 1024). Values in parentheses denote how many times memory reduced compared to full fine-tuning.
GPU Mem.
D = 4096
D= 1024
(MB)
B=1
B=16
B=256
B=1
B=16
B=256
full-finetune
144.33
145.50
164.25
24.27
24.56
29.25
lora
20.31(x7.11)
21.25 21.25(x6.85)
39.38(x4.17)
16.77(x1.45)
17.00(x1.44)
21.69( (x1.5)
1288
3.65(x39.55)
35.88(x4.006
551.50( x0.30)
(.25(x95.22))
x.66()
41.22(x00.71) x0.71)
rfft 1-28
3.14( x4 45.93)
35.14(x4.14)
547.13( 547.13(x0.30)
0.22( (x12.2111.20) 20)
2.53( x9.72)
40.30( x0.733
Oursp=120
1.06(x355. 78)
2.00( 77.77)
(8.00)
0.08( x x008 73)
0.34(x11.35
5.03(x5.81)
fit 226
1.89( 76.22)
293.25 19.03(x7.65)
0.15( x 166.88)
1.61 (00(x115. 15.24)
25.08 L.17)
rftt 255
1.62( 17)
(x..3)
286.06( x0.57)
0.12 x 194.92)
1.48( x 16.63)
24.02( 11.2)
p556
0.56(x256.36)
1.50(x96.97
20.25(x8.11)
0.05(x512.44
0.33(x74.74)
5.02 (x5.83)
fft p012
1.02(x141.97)
(D.63(x13.66)
164.50( (1000)
0.09 2.099x267.23)
1.09(x22.06
(1.772
rfft 5112
068.(x116.28)
x0.03(x14.51)
156.66 x 1.0))
x596()25.68) 0.08 (x312.66)))))
(5.05(x
p512
0.31(x461.14)
1058(x)
20.13 (x8. 16)
0.03 (x764.69)
0.32 (x76.56)
5.01(x5.84)
f 1024
0.58( 22).233)
6.44( (2A.x22.59)
100.22( X 1.64)
0.0G( 3382.35)
0.83( 29.76)
13 03.01(( 13.01(x2.255 (x2.5)
ffft 1022
0.49( 295.88))
5.88( x24.73)
92. 24( x 1.78)
0.05( x447.79) 79)
0.70( x 35.15)
11.02( X 65)
ours 10:24
0.19(x76..76)
1.31(x110.82)
20.06 (x8.19)
0.02 (x 014.39
0.32 (x77.50)
x5.84)
fit
0.25(x575.0m
3.30( 144.12)
52.05 x3.16)
N/A
N/A
N/A
p= 1098
0.21(x698.79) 79)
2.78( (22.25)
44.04( x3.73)
N/A
N/A
N/A
ourS
pm 9096
0.0% 531.54)
20.02 (.27(x144.22)
N/A
N/A
N/A
Symmetry in Circulant Matrix based Training. According Eq. even though the FFT results are naturally symmetric for real-valued inputs, there is elementwise multiplication between two FFT results. Given that A B = A B, it follows that the elementwise multiplication result maintains the symmetry property in FFT results of input and circulant weight vector. Therefore, the IFFT operation input in Eq. 4 and Eq. 5 are with symmetric complex-valued input.
Experiments
All our experiments compare three different FFT implementations: (1) fft: standard complex- valued FFT using torch. fft. fft/ifft from PyTorch [25]; (2) rfft: real-input FFT using
torch. fft. rfft/irfft, exploiting Hermitian symmetry; (3) ours: custom CUDA-based real-
domain FFT with in in-place forward/backward implementation, reusing the input real-valued memory for intermediate result storage.
We also include two common baselines: (1) FF: updating all trainable parameters; (2) lora 22 low-rank adaptation with parameter-efficient updates. For fair comparison, all runs use the same training configuration (batch size, optimizer, precision).
5.1 Memory Efficiency
To evaluate the memory efficiency of our proposed in-place training, conduct experiments in two settings: (1) single-layer analysis: we perform training on a singular linear layer with different training methods, where circulant matrix based training [10] are accomplished with different FFT
implementations; (2) full-model training: we apply the circulant fine-tuning approach [10] to RoBERTa-large and LLaMA2-7B and monitor memory usage throughout training. All circulant variants share the same number of trainable parameters, differing only in FFT backend.
Peak memory is recorded using PyTorch memory profiler. To better understand the memory distribu- tion, we also visualize the breakdown of memory usage (model weights, trainable params, gradients, others) in both the single-layer and full-model training.
7

---

(MB)
175
150144.3 4.33
input
memory allocated (MB
175y
200
25
164.50
156,66
Input target
target
trainable
150
tra trainable
125
intermediate
intermediate
100
gradient
125
100.22
92.24
gradient
100
Total memory
75
52.05
39.38
44, 04
50-
20.31
20.13
20.06
20.02
25.
1.02
0.86
0.31
0.58
0.49
0.19
0.25
0.21
0.99
full
tora
m 512
512
ours 512
m024
MM 1024
1024
4096
ritt 4096
ours 4096
full ft
lora
fft 512
rfft 512
ours 512
m 3024
rift 1024
ourg 1024
4096
rftt 4096
ours 4096
mi
ours
(a) Memory allocation (B 1)
(b) Memory allocation (B
256)
Figure 2: Memory breakdown during single-layer fine-tuning with hidden dimension = 4096,
under two batch sizes: (a) B and (b) B 256. Intermediate tensors are allocated during the forward pass, while gradients appear the backward pass. This illustrates how batch size impacts memory allocation for activations and gradients.
5.1.1 Single Layer Training
Setups. To isolate the memory overhead introduced by different FFT implementations, conduct controlled experiments on a single fine-tuned layer using an NVIDIA A100 GPU. We vary the input dimension RBxD with D {1024, 4096} and batch size B [1, 16, 256). For circulant-based methods, we further vary the block size to evaluate its influence on memory usage. Peak memory recorded during both forward and backward passes, and the results reported in Tab Fig shows the memory breakdown for the large setup (D = 4096) under batch sizes B and B = 256, highlighting the memory footprint of intermediate tensors created during forward computation and gradients allocated during backpropagation. This breakdown reveals the impact of in-place FFT on reducing transient memory usage.
Results. Tab T and Fig Pllearly demonstrate the effectiveness of our in-place method. When the batch size is small, all circulant variants show memory advantages full fine-tuning and LoRA. However, as the block size decreases and the batch size increases, standard FFT-based circulant layers incur increasing overhead from intermediate tensors, especially during the forward pass. In contrast, our method performs the entire forward computation in-place, introducing no intermediate tensors. As shown Fig this leads to significant memory savings in the single-layer setup. Moreover, by overwriting the grad_outpur output in-place at the final stage of the backward pass, our method also reduces memory usage during gradient computation.
5.1.2 Full Model Training
Setups. We conduct full-model experiments on both LLaMA2-7B and RoBERTa-large using NVIDIA A100 GPU. For LLaMA2-7B, we use the GSM8K dataset with per per_device_ arrainbbatch_ste
set to and RoBERTa--arge,, use the MRPC dataset with batch size of 32. These configurations follow the standard precision training setups for each task [10], and we retain them to reflect real-words GPU memory cost. We use stochastic gradient descent (SGD) as the optimizer in all experiments.
set to 4. For
Results. From T we observe that all methods have the same base model memory, and the memory for trainable params is negligible compared to the total model size. It can be noticed that others take up the second largest memory consumption, which is managed by PyToorch framework for storing activations, dynamic memory allocation and release, etc. In LLaMA2-7B gradient memory is approximately twice that of triiabbe_ppram because the forward pass uses 16 to reduce memory usage, but gradients must be stored in float32 as backward computations do not support bf16. In contrast, RoBERTa-large uses full full-precision training, so the gradient memory matches the parameter memory size.

---

Table 2: Peak GPU memory usage across different training stages during one epoch on LLaMA2-7B and RoBERTa-large, model indicates memory used to load the base model; trainable refers to memory allocated for trainable parameters; gradient denotes the analytically estimated memory for gradients of trainable parameters; others represents the remaining memory, computed as the difference between the peak usage and the sum of the above three, accounting for buffers, activations, and miscellaneous overhead.
LLaMA2-7b
RoBERTa-Large
Method
model (GB)
trainable gradient others
total (GB)
Method
model (GB)
trainable gradient others total (MB)
(MB)
(MB)
(GB)
(MB)
(GB)
(GB)
FF
12.61
0.00
6144.00
8.28
26.90
FF
1.33
0.00
192.00
4.63
6.15
1ora
12.61
48.00
96.00
6.20
18.96
lora,=8
1.33
3.00
3.00
4.90
6.24
lora.
12.61
96.00
192.00
6.26
19.15
lora,-16 16
1.33
6,00
6.00
4.92
6.26
M 2512
12.61
6.00
12.00
8.17
20.81
M p-58
1.33
0.75
0.75
5.39
6.72
rfft 5-12
12.61
6,00
12.00
6.65
19.28
rft y66
1.33
0.75
0.75
4.80
6.13
p=122
12.61
6.00
12.00
5.30
17.93
p-r6p=256
1.33
0.75
0.75
4.44
5.77
fit 10024
12.61
3.00
6.00
6.58
19.20
fft y512
1.33
0.38
0.38
5.41
6.74
rfft, p 1024
12.61
3.00
6.00
6.55
19.17
rfft p 512
1.33
0.38
01.88
4.76
6.08
10085p1000 p1001
12.61
3,00
6.00
5.30
17.92
pur8p=512
1.33
0.38
0.38
4.44
5.77
fft 4096
12.61
0.75
1.50
8.09
20.71
fft 1024
1.33
0.19
0.19
5.39
6.72
rfn 1096
12.61
0.75
1.50
6.55
19.16
rfll p= 1024
1.33
0.19
0.19
4.76
6.09
ours 4006
12.61
0.75
1.50
5.29
17.91
OurSp=1024
1,33
0.19
0.19
4.44
5.77
is also worth noting that fit and rfft implementations do support 16 arithmetic, limiting their memory optimization potential during the forward pass. While full fine-tuning does not incur extra memory for trainable paraals (as the base model is updated directly). it still requires large gradient storage due to the number of trained parameters.
Our method, in contrast, outperforms all FFT and inverse FFT operations in-place with native support for real-valued bf16 input. Besides, our method also outperforms LoRA adapters which has been widely adopted due its small parameter amount and memory consumption. As result, it consistently achieves the lowest peak memory usage across training steps, showing superior memory efficiency in practical fine-tuning setups.
5.2 Runtime Efficiency and Numerical Accuracy
To comprehensively evaluate the proposed rdfft operator, we analyze both its low-level computational efficiency and numerical accuracy (operator-level)) as well end-to-end performance when integrated into large-scale fine-tuning tasks (model-level). This dual perspective allows us to capture both micro-level operator behavior and macro-level training characteristics.
5.2.1 Operator-Level Evaluation
At the operator level, Tab presents both runtime and numerical accuracy. At small to medium sizes (512 and 1024), ours achieves competitive runtime with rfft, suggesting that in-place execution can be efficient when synchronization cost limited. At larger sizes (4096), however, the runtime overhead increases due to CUDA thread-block limitations, where synchronization is required both within and across blocks. Moreover, the inverse transform (ours) is faster than the forward one, since reuses the butterfly structure in reverse order, thereby reducing dependencies.
In terms of accuracy, since rdfft only reformulates the memory layout to support true in in-place execution, it introduces no information loss and preserves the mathematical equivalence of the Fourier transform. Tab that both absolute and relative errors remain the level of floating-point numerical noise, confirming that rdfft faithfully reproduces the FFT spectrum.
5.2.2 Model-Level Evaluation
At the model level, Tab. 1 reports both training throughput and downstream task accuracy. While our method exhibit lower throughput compared to fft and rfft, they climinate all intermediate buffer allo-

---

Table 3: Standalone operator runtime and numerical accuracy of FFT variants. Runtime (RT, in ms) measured on an A800 GPU with FP32 precision, averaged over 1000 runs. Each operator is evaluated for both forward and inverse transforms, as shown in the table. Operator-level numerical accuracy of rfft and ours is evaluated against the torch. fft.. fft baseline, with errors reported as absolute and relative values. Entries marked as "N/A" " indicate that the baseline fft is used for reference, and thus
its self-comparison is unnecessary.
p=512 rf't
p=1024 rfft
p=4096 rfft
Method
fft
fft
ours
fft
forward
0.0246 0.0325
0.0195 0.0450
0.0279
0.0249
0.0197
0.0319
0.0252
0.0199
0.0687
RT
inverse
0.0233
0.0322
0.0421
0.0272
0.0327
0.0470
0.0503
absolute
N/A
1.88-077
5.99e-07
N/A
1.92c-07
5.75e-07
N/A
2.55c-07
5.84e-07
Acc.
relative
N/A
0.0001
0.0008
N/A
0,0001
0.0005
N/A
0.0012
0.0018
Table 4: Runtime throughput and MRPC accuracy of different fine-tuning methods. Token-level throughput (Thr., in k tokens/sec) is measured on LLaMA-2-7B using the GSM8K dataset with one A800 GPU, while MRPC classification accuracy (Acc., %) is evaluated on RoBERTa-large. Accuracy results are reported from the Chrcc-Ciiu Adapter (BCA) work (10), where only limited configurations were provided, leading to some "N/A" entries. All lora experiments use rank r 32. For circulant-based methods, denotes the block size.
p=512
p=14
p=4096
Method
FF
lora
fft
rfft
ours
fft
rfft
ours
fft
rfft
ours
Thr. (k)
3.29
3.36
1.45
1.77
0.92
1.45
1.77
0.93
1.45
1.77
0.93 N/A
Acc. (%)
90.9
90.2
N/A
90.7
90.0
N/A
89.7
90.3
N/A
N/A
cations during both forward and backward passes, resulting in substantial GPU memory savings- key advantage for large-scale fine-tuning under limited hardware.
In downstream evaluation on the MRPC benchmark, all experiments were repeated with multiple random seeds for consistency. Our method achieves task-level accuracy on par with rfft and full fine-tuning, indicating no degradation in learning quality. Together with the operator-level evidence, these results verify that rdfft is reliable drop-in replacement for real-input FFT computations, offering strong memory efficiency while maintaining runtime competitiveness and numerical fidelity.
Conclusion
In this work, we present rdFFT, a novel real-domain, fully in-place Fourier transform framework designed for memory-efficieet neural computation. Our method enables seamless integration into modern deep learning pipelines and supports consistent forward and backward passes entirely in the real domain. Extensive experiments on NLU benchmarks demonstrate that rdFFT significantly reduces memory consumption. Our results highlight the potential of operator-level memory opti- mization as complementary and lossless strategy to existing model compression methods. In future work, plan extend rdFFT to support broader classes of structured transformations and explore its integration with hardware-awe training frameworks for edge deployment.
Limitations. While our method enables real-valued inputs to undergo Fourier transformation and remain in real-valued storage with a corresponding inverse transform, inherently encodes frequency-domen information an implicit form. This design is well-suited for use cases that do not require direct manipulation of the complex spectrum. However, for scenarios where explicit access the complex-valuee frequency representation is needed-sut such as spectral filtering or custom rrequency-oomac operations- additional usable complex form. Once this conversion is performed, it typically involves casting the data to a complex type, which breaks the in-place memory symmetry and incurs additional memory overhead. As such, our framework most effective in applications where complex-doman access is not strictly required.
addiatioonnadiiiiinnnll logic is required to decode the real-valued encoding into a
10

---

Acknowledgments
This work was financially supported by the National Key R&D Program of China (Grant No. 2024YFA1211400) and the Key Project of Shenzhen Basic Research Program (Grant No. JCY12022412011103000000
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[2] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu, "Exploring the limits of transfer learning with unified text-to-text transformer," " Journal of machine learning research, vol. 21, no. 140, pp. 1-67, 2020.
[3] A. Dosovitskiy, "An image is worth 16x 16 words: Transformers for image recognition at scale," arXiv preprint urXiv:2010. 1929, 2020.
[4] Y. LeCun, D. Touresky, G. Hinton, and T. Sejnowski, "A theoretical framework for back- propagation," in Proceedings of the 1988 connectionist models summer school, vol. 1, 1988, pp. 21-28.
[5] G. Hinton, O. Vinyals, and Dean, "Distilling the knowledge in a neural network," arXiv preprint arXiv: 2015.
[6] M. Courbariaux, Y. Bengio, and J.-P. David, "Low precision arithmetic for deep learning." in ICLR (Workshop). 2015.
[7] S. Han, H. Mao, and W. Dally, "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding," arXiv preprint arXiv: 2015.
[8] Y. Cheng, F. X. Yu, R. S. Feris, S. Kumar, A. Choudhary, and S.-F. Chang, "An exploration of parameter redundancy deep networks with circulant projections." in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2857-2865.
[9] C. Ding, S. Liao, Y. Wang, Z. Li, N. Liu, Y. Zhuo, C. Wang, X. Qian, Y. Bai, G. Yuan al., "Cirenn: accelerating and compressing deep neural networks using block-circulant weight matrices," in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecturre 2017. pp. 395-408.
[10] X. Ding, M. Wang, S. Liao, and Z. Wang, "Block circulant adapter for large language models," arXiv preprini arXiv:2505.00588. 2025.
[11] Z. Gao, Q. Wang, A. Chen, Z. Liu, B. Wu, L. Chen, and J. Li, "Parameter-efffiient fine-tuning with discrete fourier transform." arXiv preprint arXiv:2405.03003, 2024.
[12] M. Frigo and S. G. Johnson, "The design and implementation of fftw3," Proceedings of the IEEE, vol. 93, no. 2, pp. 216-231, 2005.
[13] NVIDIA Corporation, "cufft documentation: Data layout," hatpss//occsnyddd...o//uuuaaaaa Index.html#data-layy layout 2025, accessed: 2025-05-01.
[14] S. R. Bulo, L. Porzi, and P. Kontschieder, "In-place activated batchnorm for memory-optimized training of dnns," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5639-5647.
[15] L. Liu, B. Fu, M. D. Wong, and F. F. Young, "Xplace: An extremely fast and extensible global placement framework," in Proceedings of the 59th ACM/IEEE Design Automation Conference, 2022, pp. 1309-1314.
[16] Y. Sun, T. Chen, Q V. H. Nguyen, and H. Yin, "Tinyad: Memory-efficient anomaly detection for time-series data industrial iot," IEEE Transactions on Industrial Informatics, vol. 20, no. pp. 824-834, 2023.
11

---

[17] J. Yu and T. S. Huang. "Universally slimmable networks and improved training techniques." in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1803- 1811.
[18] H. Pratt, B. Williams, F. Coenen, and Y. Zheng, "Fenn: Fourier convolutional neural networks," in Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2017, pp. 786-798.
[19] C. Z.aaoo, Y. Chen, Y. Feng, and S. Yang. "Efficient synthesis of large-scale time-modulated antenna arrays using artificial neural networks and inverse fft," IEEE Transactions on Antennas and Propagation, vol. 72, no. 2, pp. 1568-1580, 2023.
[20] O. Rippel, Snoek, and R. Adams, "Spectral representations for convolutional neural networks," Advances in neural information processing systems, vol. 28, 2015.
[21] J. Lee-Thorp, J. Ainslie, I. Eckstein, and S. Ontanon, "Fnet: Mixing tokens with fourier transforms," * in Proceedings of the 2022 Conference of the north American chapter of the Association for Computational Linguistics: human language technologies, 2022, pp. 4296- 4313.
[22] E. J. Hu, Y. Shen, P. Wallis, Z. Allen Chien-Zhu,Jae Y. Li, S. Wang, L. Wang, W. Chen et al., "Lora: Low-rank adaptation of large language models." ICLR, vol. no. 2, p. 2022.
[23] A. V. Oppenheim, A. S. Willsky, and S. H. Nawab, Signals & systems. Pearson Educación, 1997.
[24] J. W. Cooley and J. W. Tukey, "An algorithm for the machine calculation of complex fourier series," Mathematics of computation, vol. 19, no. 90, pp. 297-301, 1965.
[25] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, "Automatic differentiation in pytorch," 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state that our method enables fully in-place Fourier transform in the real domain, effectively reducing memory usage compared standard FFT and RFFT implementations. These claims are well-supported by the theoretical analysis and experimental results presented in the paper, accurately reflecting the scope and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
The paper explicitly discusses limitations in Sec particularly the constraint that the method encodes frequency information implicitly, which may require additional decoding for tasks needing complex-domann access. It also notes that the method is naturally suited for input sizes that are powers of two, though this can be mitigated via zero- paro-padding.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper includes theoretical derivations related to the in-place real-valued Fourier transform. All assumptions are clearly stated, and the derivations are presented in detail within the main text to ensure completeness and correctness.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
All experiments in the paper are described with detailed settings, including model architecture, training hyperparameters, datasets, and evaluation metries, ensuring that the results can be independently reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will release anonymized code and scripts in the supplementary material to reproduce the main experimental results. The provided package includes our in in-place FFT implementation, baseline methods, model loading procedures (e.g., for LLaMA-7B and RoBERTa-LLage), memory profiling scripts, and all commands needed to reproduce the bar charts and analyses reported in the paper. Instructions environment setup and dependencies are also included for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips wttpccppppcccgddeeeeeeeeeeeeeeee honooliccuuddess/dddeuuuuiisiss http///eeuuiiiiioooooooooooo oolicyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all relevant training and testing details, including dataset splits, optimizer choices, learning rates, batch sizes, and other hyperparameters. These details are presented in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
15

---

Answer: [NA]
The experiments measure GPU memory usage, which is deterministic under fixed settings and does not involve sources of randomness such weight initialization or data sampling. Therefore, statistical significance measures such as error bars are not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experiments were conducted on a single NVIDIA A100 GPU with 80 GB memory. Each experimental run (c.g., memory measurement for one model variant) typically required less than 5 minutes. The total compute used across all experiments was under GPU-hours. All experiments large-scale pretraining or fine-tuning was performed beyond the reported experiments. Guidelines:
performed on local server. No additional
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://eeeeeeee..c////////ttttte cc/ppublic/Ehhicccuueelllll
Answer: [Yes]
Justification: Justification:
The research complies with NeurIPS Code of Ethics. It does
not involve human subjects, sensitive data, dual-use concerns, or other ethically sensitive areas. The work focuses solely on algorithmic and system-level improvements for efficient model training.
16

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper focuses on foundational research in memory-efficient training tech- niques for large-scale machine learning models. does not directly involve any deployment scenarios applications that would raise societal concerns. While such techniques be broadly useful for improving the accessibility and efficiency of MI,, the work itself does not have immediate positive or negative societal impacts.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any models or datasets that pose high risk of misuse. The contribution methodological in nature, focusing on improving memory efficiency during training, and does not involve any data or model release that would require safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
17

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All third-pan party assets used in the paper, including models such as LLaMA2-7B and RoBERTa-Land Large, are properly cited in the paper. We have respected and acknowledged their original licenses (e.g., Meta's non-commercial license for LLaMA2 and Apache 2.0 for RoBERTa).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpssspppeestaeeeeeeeeeepp thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces a new in-place FFT implementation for memory-effiiiett training, which is a novel software asset. We provide well-documented source code, in- cluding usage instructions, environment setup, and descriptions of each component. The documentation accompanies the submission in anonymized form.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
18

---

This work does not involve any crowdsourcing experiments or research with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does involve any human subjects or crowdsourcing experiments, and thus does not require IRB or equivalent ethical approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
The research does not involve the use of large language models (LLMs) as important, original, or non-standard components of the core methodology. Any L.M usage limited to writing assistance and does not affect the scientific contribution of the work.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs any important, original, non-standard components.
Please refer to LLM policy hntt:p://eerrip..cc/ooffeererrse for what should or should not be described.
19


Reviews:
[{"review_id": "B41Qj9DOk4", "reviewer": "Reviewer_Boyr", "summary": "This paper introduces rdFFT, a real-domain, in-place Fast Fourier Transform (FFT) framework intended to reduce memory use in neural network training. The method uses butterfly operation symmetry and conjugate properties to create an implicit complex encoding scheme that operates within the original real-valued memory buffer. This approach is designed to address the memory-space mismatch in standard real FFT (rFFT) implementations. The paper reports that experiments on RoBERTa-large and LLaMA2-7B show reductions in peak memory usage compared to other FFT methods and LoRA.", "questions": "1. Could you benchmark on the speed of this operator?\n\n2. Could you discuss more on the kernel tuning results if possible (to see whether this operator fits a large range of matrix sizes)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes what it claims is the first truly in-place FFT operator for real-valued data that resolves the input-output dimension mismatch of standard rFFT, contributing at the level of a fundamental arithmetic operator.\n\n2. The authors validate the method with both isolated single-layer analyses and full-model fine-tuning on large-scale models (LLaMA2-7B and RoBERTa-large), providing a breakdown of memory usage.\n\n3. The proposed rdFFT is benchmarked against standard complex FFT, rFFT in both full fine-tuning and LoRA. The paper's results indicate lower memory usage compared to these methods under the tested conditions.\n\n4. The implementation include BF16, which is important in modern LLMs while lacks proper implementation in traditional FFT libraries.\n\nWeaknesses:\n\n1. As I understand, this method only supports matrix sizes of powers of 2, which may limit the scope of it as many layers in modern LLMs are not of those sizes.\n\n2. Another important aspect, training speed, is not properly discussed in this paper.", "was_revised": true, "final_justification": "After reviewing the response, I decided to keep my rating. Good luck.", "rebuttal": "> 1. Could you benchmark on the speed of this operator?\n\nThank you for raising this important question. While the primary objective of rdFFT is to minimize memory consumption via truly in-place computation, we have also conducted runtime benchmarking to evaluate its practical efficiency.\n\nThe table below reports the media runtime (in milliseconds) over 1000 runs on an A800 GPU with FP32 precision, comparing various FFT variants across different input sizes:\n\n| Size | fft     | rfft    | rdfft(ours) | ifft    | irfft   | irdfft(ours) |\n|------|---------|---------|---------------|---------|---------|----------------|\n| 512  | 0.024624 | 0.019496 | 0.027874      | 0.032526 | 0.045015 | 0.023314       |\n| 1024 | 0.024908 | 0.019711 | 0.031927      | 0.032214 | 0.042115 | 0.027153       |\n| 4096 | 0.025171 | 0.019912 | 0.068703      | 0.032698 | 0.047035 | 0.050319       |\n\nAs shown, rdFFT does not outperform existing real-valued FFT implementations in terms of speed, especially at larger sizes. This is expected and stems from our core design choice: truly in-place computation that avoids any additional memory allocation. Such a design favors memory savings over raw throughput and is particularly beneficial in large-scale or memory-constrained scenarios.\n\nFor smaller sizes (e.g., 512 and 1024), rdFFT achieves comparable performance to rFFT, demonstrating that in-place computation can be efficient when the number of synchronization points remains low. However, for larger sizes like 4096, the runtime increases due to the limitation on the number of threads within a CUDA thread block. In our implementation, synchronization is required not only within blocks (for in-block butterfly stages), but also across blocks, which incurs additional overhead.\n\nIt’s also worth noting that our inverse rdFFT is slightly faster than the forward version. This is because our inverse transform directly reuses the structure of the forward butterfly computation in reverse order, benefiting from fewer data dependencies.\n\nIn future work, we plan to optimize inter-block synchronization and explore adaptive scheduling strategies to further improve the runtime while preserving our key advantage of in-place execution.\n\nRelevant insights on runtime efficiency are also addressed in our response to Reviewer XryJ, which we invite you to review.\n\n> 2. Could you discuss more on the kernel tuning results if possible (to see whether this operator fits a large range of matrix sizes)?\n\nThank you for this valuable and insightful question.\nTo ensure that rdFFT performs efficiently across a wide range of input sizes, particularly those encountered in large-scale models such as LLaMA2-7B, we have incorporated kernel-level tuning and multiple implementation strategies.\n\nIn transformer architectures, the input to the QKV projection layers is often reshaped into 4D tensors with shape [B, C, R, D], where D (the hidden dimension) typically serves as the dimension over which FFT is applied. In practical settings, D frequently takes values such as 512, 1024, and 4096, corresponding to different model scales and attention head configurations.\n\nTo adapt to these varying sizes, we conduct empirical tuning of the kernel configuration, especially the threads-per-block parameter, as follows:\n\n●For D = 512, we find that 256 threads per block yields a good balance between occupancy and memory throughput.\n\n●For D = 1024, using 512 threads per block ensures efficient shared memory usage and coalesced access, while maintaining high parallelism.\n\n●For D = 4096, 1024 threads per block provides near-optimal utilization of the A100 GPU's computational resources, while keeping register and shared memory usage within hardware limits.\n\nFurthermore, we implement two distinct kernel designs to accommodate different workload scales:\n\n1.***Single-pass kernel design:***\nThis version executes the entire transform in a single kernel call without requiring global memory synchronization between stages. It minimizes kernel launch overhead and is particularly effective for small to medium-sized transforms (e.g., D = 512, 1024), where the FFT stages fit well within a single thread block's resource envelope.\n\n2.***Multi-stage kernel design:***\nFor larger dimensions (e.g., D = 4096), we employ a staged computation strategy that decomposes the transform into multiple kernel launches. This approach improves control flow simplicity, reduces register pressure, and enhances memory access locality, resulting in better performance and scalability on large inputs.\n\nThese kernel strategies, combined with manual tuning, make rdFFT adaptable to a broad spectrum of tensor sizes in modern deep learning models. While our core motivation lies in minimizing memory footprint via true in-place computation, we fully acknowledge the importance of runtime performance and general applicability. To that end, we are currently exploring automatic kernel configuration selection and adaptive autotuning mechanisms as part of our future work.\n\n> Weaknesses: As I understand, this method only supports matrix sizes of powers of 2, which may limit the scope of it as many layers in modern LLMs are not of those sizes.\n\nThank you for pointing out this concern.\nWe would like to clarify that although our method is designed to optimize Butterfly-based FFT computation, it is not limited to input sizes that are powers of 2.\n\nSimilar to standard FFT libraries (e.g., cuFFT), our implementation supports non-power-of-two input lengths via zero-padding and Butterfly decomposition generalization. The underlying idea of rdFFT focuses on improving the in-place computation and memory efficiency of the Butterfly operations themselves. Since the Butterfly pattern can be applied to any input size through padding or factorization into mixed-radix stages, rdFFT remains applicable to arbitrary input dimensions, including those frequently found in modern LLM layers.\n\nIn practice, many transformer layers (e.g., attention projections with dimensions such as 768, 1024, or 3072) can be efficiently handled by our implementation using minimal padding when necessary, with negligible impact on runtime or memory usage.\nTherefore, we believe that this limitation does not fundamentally restrict the applicability of our method, and we will make this point clearer in the final version of the paper. Thank you again for raising this important clarification opportunity.\n\n**Thank you for your comprehensive review and valuable suggestions. We hope our responses clarify your concerns and demonstrate the contributions of our work.**"}, {"review_id": "y99RPkNsp1", "reviewer": "Reviewer_XryJ", "summary": "The paper proposes rdFFT, a real-domain fully in-place fast Fourier transform operator for neural-network training. By interleaving the N + 2 real outputs of a conventional real-input FFT back into the original N slots and proving that this layout is preserved throughout the Cooley–Tukey computation, rdFFT performs both forward and backward passes without extra memory movement. A succinct CUDA kernel with native bf16 and FP16 support eliminates copy and padding operations, lowers peak GPU memory by roughly 25~30\\% versus rFFT, and keeps throughput within three percent on LLaMA-2-7B and RoBERTa-Large fine-tuning tasks running on a single A100 80 GB GPU, offering a lossless operator-level alternative that can complement or precede parameter-efficient adaptation methods.", "questions": "- What is the end-to-end training throughput of LLaMA-2-7B compared with full-fine tuning, LoRA, and rFFT?\n- Please report validation accuracy/F1 for MRPC and GSM8K under full fine-tuning, LoRA, rFFT-circulant, and rdFFT-circulant.\n- Modern GPUs expose E4M3/E5M2 Tensor-Core paths. Can your in-place layout be extended to FP8, and what would be the numerical-stability considerations?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "## Strength\n- Operator-level novelty: The interleaved layout compresses N + 2 real slots into N and is accompanied by a rigorous symmetry proof. \n- True in-place execution: Both forward FFT and backward IFFT run without auxiliary buffers, including gradient overwrite. \n- bf16 support: Existing cuFFT/FFTW kernels lack bf16; rdFFT supports it natively. \n\n## Weakness\n- No accuracy metrics: The paper assumes “lossless” equivalence but never reports validation accuracy/F1 on MRPC, GSM8K or any task, leaving the practical impact unverified.\n- No throughput metrics: The paper never discloses latency or training throughput, which seriously limits the practical relevance of the method. \n- No FP8 support: While the implementation covers FP32 and bfloat16, it does not support the FP8 formats now available on Hopper/Blackwell-class GPUs, leaving potential memory- and bandwidth gains untapped.", "was_revised": true, "final_justification": "The rebuttal addresses many of my initial concerns with clear explanations and helpful additional results. While a few areas—such as performance optimisation and FP8 extension—still require further development, the authors have laid out a reasonable and credible plan for addressing them in the final version. I also believe that including GSM8K evaluation will further improve the completeness of the work. Overall, the paper has improved in clarity and depth, and I have decided to raise my rating based on this improved assessment.", "rebuttal": ">● What is the end-to-end training throughput of LLaMA-2-7B compared with full-fine tuning, LoRA, and rFFT?\n\nThank you very much for your thoughtful question and for taking the time to review our work.\nWe sincerely appreciate your comments regarding training throughput, and we completely understand your concern. Our work primarily focuses on in-place FFT operator optimization, aiming to significantly reduce memory consumption during training, especially in memory-constrained environments such as single A800 GPUs.\n\nWhile improving end-to-end training throughput is not the main objective of our method, we acknowledge its practical importance and therefore report the following token-level throughput (measured in tokens per second) on LLaMA-2-7B using the GSM8K dataset. All experiments were conducted on an NVIDIA A800 GPU to reflect realistic deployment scenarios.\n\nToken Throughput (tokens/sec)\n| Method           | p    | Token Throughput |\n|------------------|------|------------------|\n| Full fine-tuning | —    | 3292.45          |\n| LoRA (r=32)      | —    | 3361.64          |\n| Circulant-FFT    | 512  | 1450.63          |\n| Circulant-rFFT   | 512  | 1773.23          |\n| Circulant (ours) | 512  | 918.75           |\n| Circulant-FFT    | 1024 | 1445.77          |\n| Circulant-rFFT   | 1024 | 1768.82          |\n| Circulant (ours) | 1024 | 934.60           |\n| Circulant-FFT    | 4096 | 1446.98          |\n| Circulant-rFFT   | 4096 | 1766.08          |\n| Circulant (ours) | 4096 | 933.82           |\n\nAs the results indicate, our method currently does not achieve the highest throughput. However, we would like to respectfully emphasize that our contributions are at the operator level, focusing on a memory-efficient in-place implementation. The fine-tuning strategy — block-circulant adaptation — is adopted from prior work (Block Circulant Adapter for Large Language Models, 2025), and is used here solely to demonstrate the practical integration of our operator into real training pipelines.\n\nIn terms of GPU memory usage, our method offers significant advantages by eliminating all intermediate buffer allocations during both the forward and backward passes. This results in substantial memory savings, which are especially valuable when training large-scale models under tight hardware constraints.\n\nTo better isolate and understand the runtime behavior of the operator itself, we also benchmarked the standalone operator runtime (in milliseconds) of different FFT variants on the A800 GPU:\n\nOperator Runtime (ms)\n| Size | fft     | rfft    | rdfft(ours) | ifft    | irfft   | irdfft(ours) |\n|------|---------|---------|---------------|---------|---------|----------------|\n| 512  | 0.024624 | 0.019496 | 0.027874      | 0.032526 | 0.045015 | 0.023314       |\n| 1024 | 0.024908 | 0.019711 | 0.031927      | 0.032214 | 0.042115 | 0.027153       |\n| 4096 | 0.025171 | 0.019912 | 0.068703      | 0.032698 | 0.047035 | 0.050319       |\n\n While the current implementation is not yet optimized for speed, particularly for larger sizes such as 4096, the fully in-place nature of our method remains its most compelling feature. Specifically:\n\n* At smaller sizes (512 and 1024), our method achieves relatively competitive performance, benefiting from reduced synchronization cost.\n\n* At larger sizes, the CUDA thread block size limitation imposes overhead, as our design requires synchronization both within and across blocks. We plan to address this in future work through kernel restructuring and improved scheduling.\n\n* Our inverse transform is faster than the forward one, as it directly reuses the butterfly structure in reverse, resulting in fewer dependencies and better execution efficiency.\n\nWe humbly acknowledge that there is still room for improvement in throughput performance. Nonetheless, we believe that the notable memory savings, combined with the in-place execution capability, make our method a meaningful step toward scalable and memory-efficient training — especially for deployment on resource-constrained devices.\n\nWe have also provided additional discussion on runtime efficiency in our response to Reviewer Boyr for your reference.\n\nThank you again for your valuable feedback.\n\n>● Please report validation accuracy/F1 for MRPC and GSM8K under full fine-tuning, LoRA, rFFT-circulant, and rdFFT-circulant.\n\nThank you very much for raising this important point regarding the learning efficacy of our method.\nOur approach is fundamentally a computational and storage-level improvement that reformulates the FFT operator to support truly in-place execution, without changing its mathematical definition. Since no information is discarded in the process, the transform remains theoretically equivalent to standard implementations like rFFT.\n\nTo verify this, we conducted operator-level correctness tests, comparing our method with standard FFT on numerical error analysis: \n\nFFT numerical test\n\n| Input Size | rFFT (Abs / Rel Error)   | Our Method (Abs / Rel Error) |\n|------------|---------------------------|-------------------------------|\n| 512        | 1.88e-07 / 0.0001         | 5.99e-07 / 0.0008             |\n| 1024       | 1.92e-07 / 0.0001         | 5.75e-07 / 0.0005             |\n| 4096       | 2.55e-07 / 0.0012         | 5.84e-07 / 0.0018             |\n\n\nThe results show that the absolute and relative errors remain at the level of floating-point numerical noise. \n\nTo further evaluate end-to-end fine-tuning effectiveness, we conducted experiments on MRPC, a widely used benchmark for natural language understanding. We compare our method against full fine-tuning, LoRA, and rFFT-based circulant tuning. The results for full fine-tuning, LoRA, and Circulant-rFFT are reproduced from the original paper “Block Circulant Adapter for Large Language Models (2025),” while we newly report the accuracy of our method (in-place Circulant-FFT). All experiments were repeated across multiple seeds for consistency.\n\nMRPC Accuracy (%)\n\n| Method                         | Accuracy (%)        |\n|--------------------------------|---------------------|\n| Full fine-tuning (FF)          | 90.9                |\n| LoRA (r=8, FFT variant)        | 90.2 ± 1.0          |\n| Circulant-rFFT (p=512)         | 90.7 ± 1.0          |\n| Circulant-rFFT (p=1024)        | 89.7 ± 0.7          |\n| Circulant (ours, p=512)        | 90.05 ± 0.9         |\n| Circulant (ours, p=1024)       | 90.3 ± 0.7          |\n\nAs shown, our method achieves accuracy comparable to both standard rFFT and full fine-tuning, while also offering significant memory savings due to its in-place computation. This supports our claim that the proposed operator can act as a drop-in replacement in training pipelines without sacrificing task performance.\n\nThis supports the claim that our operator can serve as a drop-in replacement in training pipelines without compromising task accuracy. Due to the high computational cost and resource requirements of end-to-end training on LLaMA2-7B with GSM8K, we have not yet included GSM8K fine-tuning results. However, we fully agree with the importance of this evaluation, and we plan to include it in a future preprint version or upon request.\n\nWe have also addressed similar accuracy-related points in our responses to Reviewer jHnP and Reviewer 8gGn, which we hope may provide additional context.\n\n> ● Modern GPUs expose E4M3/E5M2 Tensor-Core paths. Can your in-place layout be extended to FP8, and what would be the numerical-stability considerations?\n\n This is an excellent question, thank you for raising it.\nOur current rdFFT implementation is optimized for CUDA cores and supports both FP32 and BF16 natively. Extending rdFFT to support FP8 formats (such as E4M3 and E5M2) is indeed feasible and aligns well with our long-term objective of developing highly memory- and bandwidth-efficient FFT operators.\n\nWhile native FP8 computation is not yet integrated into our current kernel, we note the following points:\n\n●The in-place memory layout of rdFFT is fully compatible with FP8 storage formats. Within the kernel, data can be temporarily cast to higher precision formats (e.g., FP16 or BF16) for intermediate computations, before being cast back to FP8 for storage. This mixed-precision approach can balance efficiency and numerical accuracy.\n\n●For leveraging Tensor Core FP8 acceleration, the Cooley–Tukey FFT algorithm can be reformulated as a series of matrix multiplications. Prior work such as tcFFT: A Fast Half-Precision FFT Library for NVIDIA Tensor Cores demonstrates this approach. Since rdFFT is based on the same algorithmic principles, our in-place scheme could naturally be adapted to this matrix multiplication formulation to exploit Tensor Cores.\n\nOf course, numerical stability under FP8 precision requires careful evaluation, especially due to the limited dynamic range and increased quantization noise. However, the symmetry-preserving and conjugate-symmetric properties inherent to rdFFT’s output layout provide a solid foundation for applying quantization-aware adjustments and error mitigation techniques.\n\nWe consider this an exciting and promising direction for future research, particularly as FP8 hardware becomes more prevalent in AI accelerators. We look forward to exploring this extension to further enhance memory and compute efficiency.\n\n**We thank you again for your valuable suggestion. We will make sure to highlight the validation results and their implications more clearly in our revised manuscript.**"}, {"review_id": "fGri1WRvfj", "reviewer": "Reviewer_jHnP", "summary": "This paper is introducing an in-place FFT framework that can be used in deep learning with reducing memory. The main contribution from the authors include 1.The rdFFT, a real-valued, fully in-place Fourier transform operator that eliminates memory space mismatches, offering improved practicality and usability for neural network applications compared to existing libraries. 2. A new memory layout design, develop a novel butterfly execution scheme for IFFT computation, and provide support for the bfloat16 data type. 3. Authors have integrated the proposed rdFFT into neural network models via circulant-structured parameter matrix and validate on real-world models. \nThe experimental results show the proposed method significantly reduces memory consumption.", "questions": "1. As well as the promising memory reducing results, is there any numeric accuracy comparision between standard fft, rfft and proposed method? or any end-to-end training result.\n2. By storing the real and imaginary parts only, is the impact of conjugate reconstruction investigated? such as computational overhead, error sensitivity, etc", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a novel in-place Fast Fourier Transform (FFT) operations for model training through the use of circulant-\nstructured parameter matrices, which have shown a significant memory reducing compared with other fft libraries. The method enables fully in-place FFT computation for real-valued input, with zero memory overhead and compatible with the Cooley–Tukey algorithm.\n2. The submission demonstrates a strong technical foundation with a robust experimental design and clear evidence supporting the claims presented in the paper. The experiments include different settings with the comprehensive memory analysis.\n3. The paper is well-structured and clearly written, making it accessible to a broader audience. \n4. This work have offered a promising direction for frequency-domain lightweight adaptation.The proposed method has the potential for wide adoption and could inspire future research in this area.\n\nWeakness:\n1. Other than the memory cost, accuracy impact is not fully investigated. Authors would also be curious about the accuracy/error results on the proposed fft operations. \n2. By storing the real and imaginary parts only, the impact of conjugate reconstruction would also provide more proof, such as computational overhead, error sensitivity, etc", "was_revised": true, "final_justification": "With the rebuttal and discussions, I will keep the initial rating as my recommendation.", "rebuttal": "> 1. As well as the promising memory reducing results, is there any numeric accuracy comparision between standard fft, rfft and proposed method? or any end-to-end training result.\n\nWe appreciate the reviewer’s question regarding the numerical accuracy and end-to-end performance of our proposed method.\nTo verify the correctness of rdFFT, we conducted both operator-level numerical comparisons and downstream task evaluations.\nThe results show that the absolute and relative errors remain at the level of floating-point numerical noise \n\nFFT numerical test\n| Input Size | rfft (abs/rel error)   | rdfft (abs/rel error)  |\n|------------|------------------------|------------------------|\n| 512        | 1.88e-07 (0.0001)      | 5.99e-07 (0.0008)      |\n| 1024       | 1.92e-07 (0.0001)      | 5.75e-07 (0.0005)      |\n| 4096       | 2.55e-07 (0.0012)      | 5.84e-07 (0.0018)      |\n\nDespite its in-place and memory-efficient design, rdFFT exhibits numerical accuracy comparable to standard rFFT, with errors remaining within acceptable bounds for practical deep learning use. These results confirm that rdFFT is a reliable drop-in replacement for real-input FFT computations.\n\nEnd-to-end Task Accuracy (MRPC benchmark):\nTo further validate practical applicability, we evaluated rdFFT in a circulant fine-tuning setting on the MRPC task, comparing it against full fine-tuning and rFFT-based approaches: \n\n| Method                     | MRPC Accuracy (%) |\n|----------------------------|-------------------|\n| Full fine-tuning (FF)       | 90.9              |\n| LoRA (r=8, FFT variant)     | 90.2 ± 1.0        |\n| Circulant-rFFT (p=512)      | 90.7 ± 1.0        |\n| Circulant-rFFT (p=1024)     | 89.7 ± 0.7        |\n| Circulant-rdFFT (p=512)     | 90.05 ± 0.9       |\n| Circulant-rdFFT (p=1024)    | 90.3 ± 0.7        |\n\nThe results show that rdFFT achieves task-level performance on par with both rFFT and full fine-tuning, without introducing degradation in learning quality.\n\nIn summary, both numerical and downstream task results confirm that rdFFT maintains accuracy while significantly reducing memory usage, supporting its practical use in large-scale training.\n\nSimilar concerns regarding accuracy have also been discussed in our responses to Reviewer XryJ and Reviewer 8gGn, which you may refer to for additional context.\n\n> 2. By storing the real and imaginary parts only, is the impact of conjugate reconstruction investigated? such as computational overhead, error sensitivity, etc\n\n We appreciate this excellent question. Our method adopts the same Hermitian symmetry assumption as conventional rfft implementations, storing only the unique real and imaginary components of the spectrum. The conjugate symmetry is preserved implicitly through the memory layout, eliminating the need for explicit conjugate reconstruction during computation.\n\n●Computational overhead: No additional overhead is introduced. All arithmetic operations (e.g., inverse transforms, gradient computations) are performed directly on the compact, interleaved representation without requiring reconstruction of the full complex spectrum.\n\n●Error sensitivity: Since our design mirrors the reconstruction assumptions of standard rfft, the numerical behavior is equally stable. As shown in our operator-level comparisons (see Q1), the absolute and relative errors remain within the same magnitude as rfft, confirming no added numerical sensitivity.\n\n●Theoretical soundness: Our layout is mathematically consistent with the Cooley–Tukey FFT framework. The structure preserves exact conjugate symmetry across forward and backward passes, and maintains gradient correctness throughout training. This ensures full compatibility with autograd-based optimization workflows.\n\nIn summary, rdFFT maintains the same computational and numerical guarantees as existing FFT-based operators, while offering improved memory efficiency and supporting in-place execution with no loss in correctness or precision.\n\n**We are grateful for your careful evaluation and helpful remarks. Your input has provided us with important perspectives and motivated us to refine our study.**"}, {"review_id": "Z0klUvXj8M", "reviewer": "Reviewer_8gGn", "summary": "The authors propose to learn a new kind of Fast Fourier Transform (FFT) on real valued data which is in place and does not require any additional storage during computation making the operation more space efficient. Similar to the FFT, they leverage the recursive approach of a Butterfly graph and divide the computation of the real-valued FFT into subcomputations. However, their resulting frequency representation is only reported in real-valued numbers. Their evaluation showcases reduced memory usage for single layer training and model fine-tuning.", "questions": "1. Can the authors provide a full evaluation of the finetuned models' quality? Specifically, do performance numbers remain on par, or do they drop due to information loss?\n2. Given that the motivation for in-place FFT stems from vision tasks beyond 1D, how could the proposed approach be extended to 2D, 3D, or nD settings? Does the inherent split of real and imaginary parts preclude such extensions?\n3. What strategies might enable the proposed approach to regain the inherent interaction offered by complex numbers?", "limitations": "The authors identify their paper's greatest limitation as the split into distinct real and imaginary numbers, which prevents direct use of complex forms. Consequently, as they state in lines 315-316, \"additional logic is required to decode the real-valued encoding into a usable complex form.\"\n\nI hope the authors can elaborate on how to adapt their approach to work like complex values and/or in settings beyond 1D.\n\nFurthermore, while I believe this work is a step in the right direction, a thorough evaluation of the resulting model quality is necessary to judge the effectiveness of the proposed approach.\n\nIf the authors manage to address these points, I would be happy to raise my score.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n- The proposed approach demonstrates significantly lower memory consumption compared to common FFT methods. This makes it highly relevant given the current trend of large models and input dimensions.\n- The proposed approach is interesting and relevant because, with growing model and dataset sizes, the use of frequency representation (e.g., for leveraging the convolution theorem) is becoming more popular. Additionally, the approach inherently lowers memory consumption. \n\nWeaknesses:\n1. The resulting models are evaluated solely on GPU consumption, but a performance evaluation of the finetuned models is missing. Consequently, it is hard to judge the actual functionality, meaning whether the proposed real-valued FFT works as intended. The authors should provide evaluations on the trained models.\n2. The applicability remains debatable. While the authors motivate their work with vision applications that leverage the frequency domain for efficient computation and enhanced model learning, the provided implementation appears to function only for 1D signals.\n3. Further, the separation into imaginary and real part into different array positions does imply that standard usage like the convolution theorem would need to be adapted. While this is discussed in the paper's limitations, no future strategy for tackling this problem is provided.", "was_revised": true, "final_justification": "The authors' rebuttal successfully addressed my concerns, and as a result, I have increased my score. For future work, I would be very interested in seeing this algorithm extended to an nD setting.", "rebuttal": "> 1. Can the authors provide a full evaluation of the finetuned models' quality? Specifically, do performance numbers remain on par, or do they drop due to information loss?\n\nThank you for raising this important question regarding the quality and accuracy of models fine-tuned using rdFFT.\nWe would like to clarify that rdFFT introduces no information loss due to its design principles. Specifically, rdFFT leverages the conjugate symmetry property of the real-input FFT spectrum: for an input of size N, the FFT output contains only N/2+1 unique complex values. Our method stores these unique values by carefully separating the real and imaginary parts into designated positions within the original buffer, preserving the conjugate symmetric structure. For instance, the DC component (index 0) and the Nyquist frequency component (index N/2) are purely real and handled accordingly to avoid approximation errors.\n\nTo quantify the numerical fidelity of rdFFT at the operator level, we measured absolute and relative errors against PyTorch’s standard rfft implementation across different input sizes:\n\nFFT numerical test\n| Input Size | rfft (abs/rel error)   | rdfft (abs/rel error)  |\n|------------|------------------------|------------------------|\n| 512        | 1.88e-07 (0.0001)      | 5.99e-07 (0.0008)      |\n| 1024       | 1.92e-07 (0.0001)      | 5.75e-07 (0.0005)      |\n| 4096       | 2.55e-07 (0.0012)      | 5.84e-07 (0.0018)      |\n\nThese errors are negligible and demonstrate that rdFFT faithfully reproduces the spectral results.\n\nFurthermore, we evaluated the end-to-end task performance by fine-tuning models on the MRPC benchmark, a standard NLU dataset:\n\n| Method                     | MRPC Accuracy (%) |\n|----------------------------|-------------------|\n| Full fine-tuning (FF)       | 90.9              |\n| LoRA (r=8, FFT variant)     | 90.2 ± 1.0        |\n| Circulant-rFFT (p=512)      | 90.7 ± 1.0        |\n| Circulant-rFFT (p=1024)     | 89.7 ± 0.7        |\n| Circulant-rdFFT (p=512)     | 90.05 ± 0.9       |\n| Circulant-rdFFT (p=1024)    | 90.3 ± 0.7        |\n\nThe results confirm that rdFFT maintains task-level accuracy on par with both standard rFFT-based tuning and full-precision fine-tuning. This strongly indicates that our in-place FFT design does not degrade model quality or learning efficacy.\n\nRelated discussions on model accuracy and validation can also be found in our responses to Reviewer XryJ and Reviewer jHnP.\n\nIn summary, both numerical evaluation and downstream task performance validate that rdFFT functions correctly without sacrificing accuracy or introducing information loss. We will clarify these points further in the final manuscript. Thank you again for your valuable feedback.\n\n> 2. Given that the motivation for in-place FFT stems from vision tasks beyond 1D, how could the proposed approach be extended to 2D, 3D, or nD settings? Does the inherent split of real and imaginary parts preclude such extensions?\n\nThank you very much for this insightful and important question. \nOur core contribution is an in-place layout and kernel for the 1D Cooley–Tukey FFT, optimized for real inputs. However, the method is not limited to 1D: multi-dimensional FFTs (2D, 3D, etc.) are typically implemented as batched 1D FFTs along each axis, and this principle remains applicable to our rdFFT.\n\n● The memory layout and conjugate structure generalize cleanly across dimensions.\n\n● We have tested the building blocks needed to support higher-dimensional transforms using separable 1D passes.\n\n● The butterfly computation and symmetry remain valid, and our in-place logic can be adapted accordingly.\n\nTherefore, while our current implementation targets 1D for integration with transformer layers and linear adapters, extending to nD is technically feasible and remains part of future work.\n  \n> 3. What strategies might enable the proposed approach to regain the inherent interaction offered by complex numbers?\n\nWe appreciate your time and great efforts in reviewing.\nOur method stores the real and imaginary parts separately, following the same principle as conventional rFFT implementations. The conjugate symmetry property ensures that all information is preserved, allowing full reconstruction of complex-domain operations when needed.\n\n●The original complex value at index k can be recovered by combining the stored real and imaginary parts using standard complex arithmetic rules (e.g., a+ib and its conjugate a−ib).\n\n●Complex operations such as conjugation, magnitude calculation, and complex multiplication can be implemented directly using these stored real-valued components by applying corresponding real-valued formulas, thereby preserving the full “interaction” of complex numbers.\n\n●This design provides a memory-efficient and explicit representation of the complex spectrum without losing any complex-domain information.\n\nWhen full complex arithmetic is required, such as in convolution theorem applications, a lightweight reconstruction step can be performed with negligible computational overhead.\n\nIn this way, the intrinsic interactions offered by complex numbers remain accessible, while benefiting from the memory and computational efficiencies of our in-place encoding.\n\n**Thank you for your detailed and thoughtful feedback. We believe that addressing your comments has strengthened our work significantly.**"}]

Author rebuttal:

Meta-review: This paper proposes an in-place FFT technique for reducing memory usage during neural network training. It overcomes the dimension mismatch of standard FFTs by using an interesting memory layout, that enables in-place computation. The paper considers the application of parameter-efficient fine-tuning of LLMs using circulant-structured parameter matrices (which requires performing FFT and IFFT in the forward pass). The paper shows the proposed technique is effective in reducing the memory usage in this application.

The paper has received positive reviews overall.  I agree with the reviewers that the proposed technique is novel, useful to the community, and can have broad applications. That being said, the proposed technique is primarily geared towards low memory settings, and is not necessarily optimized for throughput and/or latency. There are also some concerns around lack of quality metrics in the experimental evaluation. The authors addressed these limitations in their rebuttal. I urge the authors to add this discussion to the camera ready version of the paper (in particular the shortcomings of the proposed technique).
Decision: Accept (poster)