Irrational Complex Rotations Empower Low-bit Optimizers
Authors: Zhen Tian
Xin Zhao
Ji-Rong Wen
Venue: NeurIPS 2025 poster ()
Keywords: Irrational Complex Rotation
Training-oriented Quantization
Memory-efficient Optimization
Primary area: deep_learning

Abstract: In this paper, we propose a novel optimizer state compression algorithm, namely \textbf{$\pi$-Quant}, which leverages the properties of irrational numbers (\eg $\pi$) for memory-efficient training. 
The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme.
Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. 
To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity.
We evaluate $\pi$-Quant on  a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 41.8\% decrease in GPU memory usage, all while maintaining full accuracy. \textcolor{blue}{We have submitted the code in supplementary materials}.

Irrational Complex Rotations Empower Low-bit Optimizers
Zhen Tian
Wayne Xin Zhao"
Byne Beijing
GSAI, Renmin University of China Beijing batmanfly@gmaill. com
chenyuwuximm@mmailcco com
Ji-Rong Wen
GSAI, Renmin University of China Beijing
jrwen@ruc.eds edu. cn
Abstract
this paper, we propose a novel optimizer state compression algorithm, namely r-Quant, which leverages the properties of irrational numbers (e.g.. лт) for memory- efficient training. The core idea based on our mathematical findings, which show that a pair of parameters the complex rotation scheme. Building on this insight, we map the parameters into complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate π--uant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters 3.32-bit, achieving 41.8% decrease in GPU memory usage, all while maintaining full accuracy. The code is provided hhttps://gg giighh.t.... www.eooo...../// Bonn/RUCAIBoo/Pi--..
be represented by single rotation angle using
Introduction
The rapid growth large-scale AI models (e-g. large language models [1) has led to significant im- provements in various tasks. However, larger models often require substantially more computational power and memory resources to train. Typically, most models are trained with momentum- based optimizers such Adam (2), which require the storage of or optimizer momentum for each individual parameter. As the parameter scale increases, the optimizer states consume the majority of memory, with first-order and second-order momentum tensors in the Adam accounting for 66% of parameter usage 3 Therefore, finding a solution to develop memory-efficient optimizer is critical challenge.
propose using lower-precisiee representations for the optimizer states, shown in Table [ They primarily analyze the frequently occurring values of the optimizer states, and then use binary search to quantize all state elements to these values. While previous attempts have demonstrated the promise of this approach, two major challenges remain. First, most deep learning frameworks (e.g., Pytorch (5). TensorFlow [6]) do not support such search quantization operations, these methods require additional compilation of GPU-supported operators. As result, these methods are not directly compatible with devices other than GPUs (e.g.. CPU or TPU). Second, these methods specifically designed for certain bit-width quantization, making
Considering the above issues, a number of studies
WWanee Xin Zhao (batmanfty @ @gmail.com) is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

difficult to adapt them to other bit-widths. Since these predefined precisions may not be well-suited all tasks, they can lead to performance degradation in certain cases.
Table I: Comparing different quantization methods, where n is the parameter size, and m is the precision size.
Metric
Bub (2021)
Lpmm (2023)
--Qaat (ours)
Bit-walth
8 O(n log m) O(n log m)
3.32
Complexity Full Acc.
O(n)
To address these issues, we aim to develop a precise and theoretically srorerically-guaranteedd compres- sion algorithm that can flexibly reduce the memory usage of optimizer states within typical model optimization framework. Our key idea is grounded in an important mathematical property: v((x,,) [)<=,))}² y² I 4). 30 E I + iy + e _ This enables the precise representation of parameter pair (x, y) with a single complex rotation angle (i.e., 0), by harnessing the properties of irrational numbers (i.e., πτ. As result, the parameter scale is halved with no loss of precision. Further, we can quantize the rotation angles to achieve even greater memory reduction. To implement our idea, fundamental challenge is integrating the proposed complex rotation scheme into the optimization process. Since the optimizers are often designed for real-valued parameters, they are not directly compatible with our complex transformation. Further, it is also difficult to find an efficient solution for accurately computing the rotation angles of the corresponding parameters during optimization.
Motivated by our mathematical findings, in this paper, we propose novel optimizer state compression approach with irrational complex rotation, named -uuuant. The major contribution of x--uant is the introduction of precise theoretical framework for parameter compression, resulting in lower memory costs during model training. Specifically, N-Quant involves two key techniques. First, it develops a new representation mechanism that maps the parameters in a complex space. On this basis, the complex-value valued parameters are transformed into single rotation angles according to our proposed mathematical formula, which precisely halves the parameter scale. Notably, employs an efficient system of geometric equations that can compute the precise rotation angles with linear complexity. Second, it features an effective quantization algorithm that reduces the precision of the rotation angles during training, all while preserving full accuracy. Our theoretical framework is particularly robust for numerically sensitive parameters, offering lower quantization errors compared to prior methods. Generally, т qu-Qt can be seamlessly integrated into existing optimization pipeline, which largely reduces the training memory footprint, with minimal impact on the training speed.
The main contributions are summarized as follows:
We propose effective optimizer state compression approach, named T- 7T-Qunt, by introducing a new mathematical formula. This method transforms a parameter pair into a univariate rotation equation with irrational coefficients, enabling precise compression by halving the parameter scale. To the best of our knowledge, we are the first to leverage the properties of irrational numbers for model compression.
We propose an efficient system of geometric equations for optimizing the compression process, which is capable of computing the precise rotation angle with linear complexity. Additionally, develop an effective quantization algorithm that reduces the representation precision the rotation angles, while maintaining full accuracy.
Experimental results show that π r-Quant an effective quantizer, capable of reducing the width of optimizer states to 3.32 bits, e.g. it can reduce the training memory of TinyLlama from 19,47 G to 11.32 G, with comparable accuracy. Besides, consistently outperforms several state-of-the-art quantization methods on wide range of tasks, highlighting the effectiveness of our approach.
2 Preliminaries
In this section, we give a brief introduction to the quantization techniques, and then describe the complex rotation technique used in our approach.
Training-Orienter Quantization. Quantization a compression technique that maps high-precision values to finite set of discrete values. Formally, given the quantization range D, quantization

---

approaches typically design a mapping function [0. 1] 1] D. where k represents the bit-width of the quantization. The quantization methods can be generally divided into inference-oriented and training-oriente according to their application scenarios. Typically, inference-orienter methods often conduct post-training quantization of the backbone network (e.g. dense layers) for reducing inference costs. However, these methods are not practical to be applied into training pipeline, since they fail to effectively fit the non-uniform distributed parameters, they often suffer from significant accuracy loss during optimization.
In contrast, training-oriented methods 7 focus on reducing the precision of optimizer states to achieve lower training memory. For better fitting the non-uniform parameter distributions, their basic idea is to sample high-free high-frequency elements
construct non-uniform quantization range D, e.g., D = {0.12, 0.25, 0.5, 1.0}. Due to such non non-uniformity, these methods cannot directly compute the quantization indices and typically require a search- based method to perform the quantization, i.e, X!NT arg minge p ||0 X | In practice, these methods often require compiling additional GPU kernels to perform parallel searching. Besides, the certain bit-width setting of D cannot be extended to other bit-widths. As a training-oriented method, our approach focus on reducing the memory of optimizers, which pushes the lowest achievable bit-width (4-bit) down to 3.32-bit, with higher accuracy and no need for search process.
Complex Rotation. In mathematics, complex rotation is mathematical operation that applies rotation in the complex plane. It is represented by multiplying a complex number of the form i.e. cos 0 + isin 0, where i is the imaginary unit (i² = -1) and 0 is the rotation angle. According to Euler's formula, a complex number can be written as:
cos 0 - isin 0 E.
This equation establishes mapping from ((,, ()² more profound theorem, which shows that arbitrary pair of (x. y) can be represented by an angle @ in this complex rotation scheme, As such, it provides a new solution to compress the parameters.
x²2 + y² 1) to 0 R. In paper, we reveal
3 Methodology
In this section, we introduce the proposed 7-- rT-uuant method (illustrated in Figure D for achieving memory-effcient parameter optimization. Our method is grounded in key mathematical property: each real vector can be split into two components, which can then be uniquely represented by rotation angle. Building on this idea, we propose a geometric approach for performing this transformation, enabling the computation of the rotation angle with linear complexity. Moreover, we design quantization pipeline implement our method, which significantly reduces the parameter scale and can be seamlessly integrated into existing optimization algorithms. We begin by introducing the theoretical framework of π Quant, followed by a description of its practical application pipeline.
Forward
(teep Compression
Step 2: Quantization
Para.
Input Parameter
251446 First 2 digits
Scale
Concat
Real
Imaginary
Restore
Memory
X
+
iY
= 10 x m + g
Geom.
1
IA
C
B
3.322 bit
Quant
Trans
5
B
Split
061524
Output:
ino
2TT Last À digits
Para.
Gradient Descent
Figure 1: The overall framework of x -uuunt.
3.1 Parameter Compression with Irrational Rotations
The key point of our approach is a novel data compression theorem, which enables the precise representation of two variables using univariate rotation equation with an irrational number as
3

---

the coefficient. We first present the mathematical foundation behind this theorem, followed by the instantiation of our idea.
Data Compression with Irrational Numbers. Existing machine learning frameworks typically represent model parameters using seaating-point numbers (e.g. FP32) with limited precision. In this representation, previous work [8] often employs truncation-like methods to uniformly quantize high-precision parameters to lower precision. The underlying assumption that the parameters follow uniform distribution within the quantization range. However, due to the mismatch between this assumption and the actual distribution of the parameters, most quantization methods encounter significant errors, as parameters are more likely follow non-uniform distribution (e.g.. Gaussian distribution). In contrast to prior approaches, we introduce a novel data compression technique that compresses parameters from the perspective of irrational numbers. Formally, we present the following theorem:
Theorem 3.1. Given complex number + iy C that satisfies ||ж|| s 2. For any deviation e > 0, there exists an angle 8 E I. such that
1 e e S.
where: is the imaginary unit, and āt denotes any irrational number (e.g., the circle ratio πT).
Specifically, it shows that any vector of two real values x and y can be represented by a real-valued angle 0 in this complex scheme. This property extends to any parameter tensor T, which can be split and linearized into two cqual-sized matrices, i.e., (X, Y)= = (T) corresponding to the real part represent T by specific angle vector O
and imaginary part, respectively. In this manner, using half the number of real values. A proof of this theorem is provided in Appendix
Geometric Solutions. Finding the exact solution to Eq. (2) is the key challenge of our approach. Intuitively, one might consider using a search- based method (e.g., KDTree), where the r and y values corresponding to all 9 angles within the desired precision are stored in cached array, and the array is then searched for the coordinates closest to the target (ær, y). However, this approach often incurs On log m) complexity and presents challenges in performing parallel searches GPU. As shown in Figure [. to efficiently solve the Eq. (2) we propose a geometric solution that enables faster implementation:
Lemma 3.2. Given the input complex number x + iy, the solution 0 of Eq. 2 can be given by the
following system of geometric equations:
arctan(y/
= arccos / x2 + /22/2) 0 = B B+ 2mm, E 7.
0 = a + 3 + 2kT, E Z
We prove this lemma in Appendix B As such, the task of finding the exact value of is transformed into: given (I, y). determine the specific value of the integer m, such that there exists an integer k satisfying the above system of equations. By further simplifying the system of Eqs. (3), our main objective becomes solving the following equation:
{m+} = {2],
(4)
Ω a(1 ④) +) (1)
(5)
2T
where {-} returns the fractional part of a input number. Here, we can first compute the exact value of Q with linear complexity. However, directly solving Eq. (4) is very difficult. Intuitively, we can enumerate all possible values of m and then search for the longest match with the fractional part of the target Q. Nevertheless, the time complexity of the search process is still extremely high and not practical. As our solution, we selectively mask certain digits of æ. Specifically, the value of स in our approach is constructed as:
0. 000 1 10000- 3589793238
(6)
A limes
A times
same a5

---

where À is a quantization hlper-paramette We can guarantee that ā is still an irrational number, as it can be obtained by adding π to some rational numbers. In this way, we can approximate m the first X decimal places of f::
~ [[2) x 10a]
(7)
where [.] represents the floor function. We give an intuitive explanation of our construction in Appendix This solution guarantees that the error does not exceed 1 The entire process maintains linear complexity.
3.2
Optimization with Rotation Quantization
In the previous section, we discussed how to compress a pair real-value parameters using a single rotation variable 0. To further reduce the memory usage, we aim to use a low-precise representation for quantizing the corresponding Next, we will introduce the details of our framework.
Quantize the Rotation Angles. As introduced in Section 3.1 our approach use the complex rotation scheme to represent parameter pairs. One limitation of this method is that it requires the magnitudes of x and y (i.e, ||x² + y²ll) to be no more than 4. To address this, we first scale and normalize the parameters to ensure they lie within the representable range. Formally, given the split input matrices (X,Y) Split(T), the scaled tensor can be computed by simple method:
X /wW Y/w
(8)
where is the maximum absolute value of the numbers in both X and Y. Using this method,
the value ranges of X and Y are both transformed to [-1, 1]. satisfying the mapping conditions in Theorem 3.1 Note that can also constrain them to larger range (e.g.. [-v2, 22) 22) by multiplying w by a cocfficient. Empirically, this setting has already achieved a good trade-off in quantization deviation, which we will discuss in Section 3.3 Afterwards, we compress the and Y to a unique
θ following the method introduced in Section 3.1 For further quantizing 0, we retain À decimal places of a - 3 (See Eq. (3), with 10 as the interval:
2x10-A 0
= 10 +
a x 1
2m
In this way, we can represent each 9 with 2 digits, where the first À digits correspond to the m and the remaining À digits correspond to the g. As such, we set varied À to achieve different bit-widths. Since the is set based on the decimal system, on average, each parameter requires A logg 10 Ra 3.32X bits for storage. In particular, the entire process remains linear complexity and supports parallel computation on GPUs, making it practical in the large scale model optimization. We present detailed procedure for the quantization process in Algorithm [
Through the rotation transformation (Eq. ), the parameter scale is reduced by half. After quantiza- tion, the precision of the angle representation can be further reduced. In practice, we can compress the precision of an angle into an integer ranging from 0 to 99, achieving the same quantization effect as 3.32-bit (See Table Next discuss how to apply our algorithm during the training process.
Optimization. The main advantage of our algorithm lies in its ability to reduce the optimization memory. Since only small part of parameters necessitate immediate computation during training (3), we only restore this subset of parameters, while the remainder continues to reside in memory in their angular representation. As shown in Fi in our framework, the parameters stored in memory are and the corresponding w (See Algorithm []]. When the parameters are involved in computing, apply the inverse operations of the quantization process activate them:
= x 2T 10
cos + cos #0, ğ sin 0 + sin
(10)
x = ž w, = y w.
5

---

Algorithm 1: Quantization of the Rota- tion
Algorithm 2: Adam with x u-Qua (Dif= ferences highlighted)
Angles
1: Input: Parameter T, size: n
1: Input: learn rate a, decay rates $1, A2, Initialize: mg (O)
2: Split T into two tensors: X and Y
3: Compute w = max([X Y))
Vo (0), & = 0 {using Alg.
4: Scale X and Y according to Eq.
for each iteration & 1,2,.... ................................
5: Compute a and ( using Eq. (
4: Compute gradient V
6: Compute S2 according to Eq.
m < Restore(m;--
(Eq.
7: Compute m based on Eq.
6: my + - + (1
31 Vo
8: Calculate C from Eq. (
0t-1 1 Resstore 11-1)
(Eq. 10
9: Output: Quantized parameter θ size:
8:
01 + -,20--1 + +111 (2) (v.))
n/2 Scale Factor w, size:
m, 1-31
m 84 x- I-33
10:
0: f- 01-1 a
11:
my Quant m.)) (Alg.
12:
Vt <- Quant(u,) (Alg.
13: end for
14: Output: Optimized parameters 0T
In this way, we can easily recover the parameter tensor X, Y before quantization, and we then concatenate them to reconstruct the original T = Concat( X, Y). Similar to existing quantization works 71, the activated parameters maintain high precision ensure the accuracy of the gradients and predictions. Since only small subset of parameters is involved in computation most of the time, the temporary storage overhead incurred by these high-precision activated parameters is negligible. The detailed algorithm is presented in Algorithm
Like most training- orrinnte quantization methods "Restore" computations for each iteration, which results in higher latency compared to the original Adam. Specifically, given the parameter size n, the additional complexity of π Quant is ((n) per iteration, which can be negligible compared to the overall time complexity.
Qu-nn introduces additional "Quant" and
3.3 Discussion
In this part, we discuss the advantage of n--uunnt and compare our work with existing work.
Model Merits. Our method has the following merits:
Quantization Precision. We analyze the quantization accuracy in π n--Quant, and demonstrate its superiority over traditional approaches. Specifically, the main deviation of method comes from the approximation of the m and g in Eq. (7) and Eq. (9) respectively. Formally, given the deviation of AO in Eq. (9) the upper bound of quantization error for a specific point (z, y) is given by:
||
A0
(1 #) sin 0' Fy'll < A0,
(11)
A
A6
(1 T) cos 0' + nz' A0,
where 8' E 10, + A We give proof in Appendix C It indicates that our method can reduce the quantization error in the corresponding angle 9. Further, given the precision width X, the average quantization error can be given by:
(12) The proof is in Appedix D This indicates that the quantization error increases as the bit width decreases. Note that 2- (1 + R) < n, which further demonstrates that our method has lower average quantization error compared to traditional methods (i.e., 10->). In practice, we reduce the bit-width of optimizer states down to bit while maintaining full accuracy (See Section 4.1)
TE ((A))= E(ay)) 0 + -) -1=
Non-uniformity, Non-uniformity is an essential property that ensures the accuracy of quantization for optimization, as the parameters of neural networks typically exhibit non-uniform distribution (e.g., Gaussian distribution). In Figure 2 we plot the quantization error distribution between traditional

---

Traditional Quantization
Rotation Quantization
Table 2: Comparison of different quantization meth-
ods.
Methods
Bit-width
Seurch
Distribution
Unit
LincarQuant
Hexxible
No
Uniform
Single
Bub.Adam
Yos
Men-Uniform
Single
Lymm.
Non. Mon-Uniform
Single
a 7-Quuntt (ours)
3.32 (Plexible)
No
Non-Uniform
Pair
Figure 2: The error (top) and precision distribution (bottom) between x z-Quant and traditional methods.
quantization methods and our approach. shown, the errors of traditional methods are uniformly distributed across the parameter space. In contrast, our method produces smaller errors near the zero point, with larger errors at the periphery. Additionally, we plot the accuracy distribution for both traditional methods and our approach. It is evident that our method allocates more precision around the zero point to better capture the parameters, while assigning less precision to the range of larger values. Since most parameters and momentum in large models follow standard distribution with mean close to zero, our method is more effective at quantizing these parameters.
Novelty and Differences. In Table 2 compare our method with optimizer state quantization methods. To the best of knowledge, it is the first attempt that leverages the properties of irrational numbers for data compression, grounded in new mathematical formula. This approach transforms complex-valued parameter into single rotation angle, which can precisely halve the parameter scale of wide range of modules such optimizer states and model weights. By the proposed system of calculations, possessing superior
geometric equations, our approach only requires tinear-complest theoretical properties (e.g.. -on-uniffrmitt)) with need of complicated search process. In contrast to traditional quantization methods 3 77. which perform numerical compression for each single element, propose conduct the quantization for each pair unit of parameters, by reducing the precision of the rotation angles. As such, it provides a way to better capture the correlations between different elements, making possible further reduce precision to lower bit-widths. In practice, x r-Quant can reduce the precision of optimizer states to 3.32-bit, which can be flexibly extended into variable bit-widths, with minimal impact on the training speed (See Table 33.. In general, our method provides precise and theoretically-grondde solution for compressing the parameter scale for optimization.
Experiment
4.1 Language Modeling Evaluation Under Different Bit-Widths
Experimental Settings. We compare our approach with state-of-the-at training-oriented quantization methods, including Linear- Quant [9] with different bit-widths, Sbit-Adam (Bnb) [4]. 4bit-Adam (Lpmm) 3. and standard FP32 precision. Following the previous work (TO, we continually pre-train the TinyLlama-1 1. .B checkpoint segments, with a context window of 2048. We report the test perplexity in the Proof-pile dataset, and evaluate the trained LLM across four public benchmarks from Huggingface [13), i.e. ARC-Challes Chh-Chalee Hellaswag, Lambada and PIQA. The training settings are provided Appendix F
[ for 400 steps the PG-19 12 dataset, chunked into 64k
Results. As shown in Table we can observe that Linear- Quant 9 exhibits the poorest performance among all compared methods, with its training process being highly unstable and frequently resulting in catastrophic failures. Besides, Bnb 7 exhibits remarkable robustness, achieving performance metrics on par with the full-precision baseline. However, Lpmm Adam 3 shows a notable degrada- tion in performance metrics, even though its training loss is not much different from full precision. Notably, π ru-Qn achieves lossless performance in both 13.28-bit and bit settings, with metrics surpassing those of the full-precision baseline. Remarkably, method reduces state bit-widthe
7

---

Table 3: Validation of effectiveness under different bandwidths (enclosed in parentheses"0"> The backbone model is Tamy "N/A" denotes the "nan loss" (training collapse). Error bar provided in Table 6 Results of more LM backbone are reported in Table g
Type
Approach
Memory / Step Time
Train
Test PPL
ARC-et
Hellaswag
Lambada
PIQA 1
Avg.
Full
FP32
19.47 GB /
2,423
5.17
32.51
58.65
56.55
73.18
55.22
FP16 (16)
s
10*
24.57
49.02
High
Lincar Quant (16)
15.53 GB / 19.79
NAA
NIA
NIA
NIA
NIA
NIA
N/A
n-Quant (13.28)
15.40
33.02
73.23
55.45
Lincur Quant (8)
13.44 GB 14.97
N/A
NIA
NAA
NAA
N/A
N/A
N/A
Medium
Bub (8)
13.44 10.47
2.421
33.02
73.19
Quant (6.64)
13.58 GB/ s
2.421
5.16
33.02
58.75
56.76
73.06
55.40
pdmm (4)
12.30 10.66
2,422
5.41
31.23
58.47
55.19
72.85
54.44
Low
x- Quant (3.32)
11.32 GB/ 1
2,422
5.14
33.33
58.45
56.05
73.34
55.34
to 3.32 bits while maintaining similar convergence trend (Figure 3 and performance metrics. These results show that our rotation-based quantization method can effectively reduce the quantization loss, serving as a robust replacement for full full-precision optimizers.
Besides, report the training memory usage and the optimization step time in Table 3 As for training memory, we can observe that Bnb achieves a 30.8% reduction in memory usage compared to the FP32 Adam; Lpmm can further reduce memory usage due to its lower precision representation. In comparison, our method consumes the least memory, achieving a 41.9% reduction in memory usage. As for training speed, quantization-based methods introduce additional quantization and de-quantization operations during training, leading to relatively higher latency. Notably, Bnb and Lpmm Adam have compiled GPU binary search kernels, achieving comparable quantization latency. Since our method requires calculating parameter angles during Quant and Restore stages, resulting in additional computational overhead, which is a limitation of our approach.
4.2 Experimental Analysis
4.2.1 Accuracy Other Downstream Tasks
Experimental Settings. We evaluate π- r--uant on five tasks on long-range-arena (LRA) bench- mark [14), including Listops [15], Text classification on IMDb review dataset [16), Document Retrieval on AAN dataset [17,, Pathfinder [8], and Image classification on CIFAR-10 19]. Besides, we introduce two Seq2Seq tasks, including text summarization
the Samsum 20 and sequential
recommendation on the Movielens [21] Follow the work [22), we use the 2-yeyer transformer as the backbone, and the training settings are provided in Appendix F We report the accuracy for the LRA tasks, and report the belu scores and recommendation metrics for the other tasks.
The experimental results shown in Table We can observe that Bab 4 shows significant accuracy drops on Image, Retrieval, Listops, Summary and RecSys tasks, but surpasses full- full-precision on Text and PathFinder tasks. Besides, Lpmm (3 shows suboptimal performance across most tasks, with only comparable results on PathFinder. These results suggest that the predefined quantization precision in these methods is not always optimal across different tasks. In comparison, observe that T- r--uant outperforms the original Adam across all tasks, suggesting that the rotation quantization may represent a superior choice optimization settings.
Results.
4.2.2 Ablation Study
We conduct ablation studies explore the impact of settings, As introduced in Section 3.1 the design of स influences the accuracy of solving for 8 using Eq. 77 To verify this, we change the value of ग as two forms: (1) as truncated rational number, i.e., कं 0.001 and (2) setting its integer part to other values, e.g., 52 3.00100035.... Specifically, we focus on analyzing quantization errors for two distributions: Gaussian (i.e., (O, 1)) and uniform distributions (i.e., U0, 1)). As shown in Figure our method achieves lower error in fitting Gaussian distributions, revealing its non-uniform fitting capability. In addition, म has lower error than मं as its decimal portion provides error compensation. Besides, we find that setting the integer part of π to zero yields minimal error.
8

---

Table 4: Performance on other learning tasks. The
10
bar is presented in Table 7
10
Task
Metric
Adam
Bub
Lymm
T-Quant
Text
Apc.
Loss
0.634
61.83
64.68
0.630
01.14
0.635
0.631
64,68
m-Quuntt LINTSS
Image
Loss
1,966
1.763
1.771
1,745
INTTI
Retrieval
Acc. ?
Loss
43.61
M0.70
45.21
79.92
45.93
79.19
46.32
80.79
100
Listops
Loss
1,700
1.715
1.741
1.646
PathFinder
Ace.
69.15
7038
72,00
71.02
109
Loss
53.91
49.91
Blue-
13.34
13:71
Text
Bluc-2
8.30
7.68
7.95
8.72
Sumunary
Blac-3
4,03
3.87
3,99
4.74
50
100
150
200
250
300
350
400
1.61
Iteration
Recall 10
28.19
27.75
28.01
28.76
Figure 3: Loss comparison of FP32 and TT-Qunnt.
RecSys
Ndcg@ 101
Mirts 10
15.51
11.66
11.58
15.34
15:41
11.53
15,85
11.94
Hire
Gradient Descent Himmeblau Function
0.06
bita
0.05
ZZA
m
0.04
Poldci
0.03
0.02
0.01
U10. 1)
N(0.1)
Figure 4: Ablation Study of the setting.
Figure 5: Visualizing the optimization process.
4.2.3
Visualizing the Gradient Update Process
To have an intuitive understanding of approach, visualize our approach's 's gradient descent through simple task: minimizing the Himmelblau's 's function (f(z, y) (x² + y 11) 2 (z (x+ 7)?). As shown in Figure we establish multiple starting points and compared the training processes across different optimizers, including Adam with varying bit-widths, SGD, and our proposed method. We can observe that Adam's path is typically shorter than SGD's, showing that Adam's 's momentum can accelerate convergence. Our method achieves the shortest paths across all initialization points, particularly start point where it successfully finds the correct minimum while Adam becomes trapped in local saddle point. These findings show that our rotation method can enhance convergence speed in gradient descent and facilitates escape from local minima and saddle points.
Related Work
Model Quantification. LLMs have achieved outstanding performance across various tasks. However, the extensive parameters in LLMSs results in substantial memory usage, significantly increasing the cost of deploying them real-world scenarios. To this end, a number quantization methods [23) have been proposed for reducing the deployment cost. Typically, there are two classes of approaches, namely inference-oriente quantization and training-orientt quantization. Among them, inference-oricn oriented work [2 30 primarily focuses on quantizing the neural weights for achieving faster inference. Through extensive experiments on LLaMA, their approach demonstrates promising results for weight and activation quantization. Despite the progress, these approaches are not suitable to be applied for the training phase, as their precision uniformly across the quantization range, which cannot adapt the non-uniformly distributed parameters. Unlike inference-oriented approaches, -orinnte quantization methods [7 focus on compressing the optimizer states for achieving low training memory. Specifically, they propose to sample the frequecyy elements and then use

---

surch-baseed approach to quantize the optimizer states to these values. However, these predefined elements cannot adapt well all tasks, which may lead to degraded performance in certain tasks.
Optimization with Complex Rotations. Traditional neural networks are typically modeled in a real vector space, which exhibits certain constraints when conducting mathematical operations such as exponentiation and vector rotations. To enhance the capability of representation learning, a number of approaches (31)36) have proposed conducting representation learning complex vector space. Typically, rotation techniques in complex space have been widely applied in the modeling of language models. Specifically, RoPE 34 employ complex rotation to inject the position information
into the attention mechanism; xPos 37 employs two-dimensional pairwise rotation technique to enhance the positional embeddings in Transformers. Unlike these work, we attempt to utilize complex rotations based on irrational numbers for quantization, allowing it to exhibit non-uniform distribution for parameter optimization.
6 Conclusion
In this paper, we proposed a novel optimizer state compression approach 7T-uunn. Different from prior work, r-Quant compressed the parameters using a complex rotation scheme, by leveraging the properties of irrational numbers according to our proposed mathematical findings. In π- n-Quant, the two dimensional parameter pairs were efficiently converted into a single rotation angle with our proposed system of geometric equations, which precisely halved the parameter scale with linear complexity. Further, π r-Quant introduced an effective quantization algorithm, which reduced the precision of the rotation angles, all while maintaining full accuracy. In principle, π r-Quant possessed lower quantization error and demonstrated effective replacement of full precision optimizers. As the further work, we will consider reducing the quantization complexity, and testing the capacity of π- r-Quant in more compression scenarios, e.g., KV cache compression in the large language models.
non-uniform precision distribution, which served as an
Acknowledgement
This work was partially supported by National Natural Science Foundation of China under Grant No. 92470205 and 62222215. Xin Zhao the corresponding author.
References
[1] Wayne Xin Zhao. Kun Zhou, Junyi Li, Tianyi Tang, Xiaolci Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[2] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[3] Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 06115136-11711 2023.
[4] Tim Dettmers. 8-bit approximations for parallelism in deep learning. arXiv preprint arXiv:1511.04566 2015.
[5] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, ler-heerformance deep learning library. Advances in neural information processing systems, 32, 2019.
[6] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. [TensorFlow system for (Large-Seale)) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283, 2016.
10

---

[7] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021.
[8] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq; Accurate post-training quantization for generative pre-trained transformers. arXiv preprini arXiv:2210. 2022.
[9] Vincent Vanhoucke, Andrew Senior, Mark Z Mao, al. Improving the speed of neural networks on cpus, In Proc. deep learning and unsupervised feature learning NIPS workshop, volume 1, page 4, 2011.
[10] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007/ 2023.
[1]] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprini arXiv:2401.02.0 000285, 2024.
[12] Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[13] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[14] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers, 2020.
[15] Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning, 2018.
[16] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
[17] Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. The ACL Anthology network corpus. In Min- Yen Kan and Simone Teufel, editors, Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DLL) pages 54-61, Suntec City, Singapore, August 2009. Association for Computational Linguistics.
[18] Drew Linsley, Junkyung Kim, Vijay Veerabadran, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated-recurrent units, 2019.
[19] Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.
[20] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum cor- pus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint 12X1v 2019.
[21] Maxwell Harper and Joseph Konstan. The movielens datasets: History and context, Acm transactions on interactive intelligent systems (tiis, 5(4):1-19, 2015.
[22] Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nystr)." " om method. Advances Neural Information Processing Systems, 34:2122- 2135, 2021.
[23] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference. In Low-Power Com- puter Vision, pages 291-326. Chapman and Hall/CRC, 2022.
11

---

[24] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarr weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[25] Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.0000 000044 2023.
[26] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless IIm weight compression. arXiv preprint arXiv:2306.03078 2023.
[27] Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. Aptq: Attention- aware post-training mixed-precision quantization for large language models. In Proceedings of the sl ACM/IEEE Design Automation Conference, pages 1-6, 2024.
[28] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 13355-13364, 2024.
[29] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantiza- tion aware training for large language models. arXiv preprint arXiv:2305. 17888, 2023.
[30] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang. Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang. Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310. 1453, 2023.
[31] Zhen Tian, Ting Bai, Wayne Xin Zhao, Ji-Rong Wen, and Zhao Cao. Eulemnt: Adaptive feature interaction learning via euler's formula for etr prediction. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 1376-1385. 2023.
[32] Zhen Tian, Wayne Xin Zhao, Changwang Zhang, Xin Zhao, Zhongrui Ma. and Ji-Rong Wen. Fulerforme: Sequential user behavior modeling with complex vector attention. In Proceedings the the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1628, 2024.
[33] Zhen Tian, Yuhong Shi, Xiangkun Wu, Wayne Xin Zhao, and Ji-Rong Wen. Rotative factoriza- tion machines. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2912-2923, 2024.
[34] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 127063, 2024.
[35] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is a wave: Phase-aware vision mlp. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 09935-10944, 2022.
[36] Xianming Li and Jing Li. Angle-optimized embeddings. arXiv preprint 17X17 2023.
[37] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A lengthh-etrrppolttbble transformer. arXiv preprint arXiv:2212. 10554, 2022.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwiicccguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tppr/////uriss.../// http///urrr.cccpuu/////e http://cscccppttttttttttttteeeea
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
17

---

Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
18

---

Justification: [Yes]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreerp for what should or should not be described.
19

---

A Proof of Theorem 3.1
Similar to the original proof, to approximate 2 = x + iy, we construct geometric relationships following Figure 6 thereby deriving the system of geometric equations in Eq.
arctan(y/at $ = arccos( v2 + + /2/2)
3 2mm E Z
(13)
#O + 3 + 2kT, k E Z
Similarly, we can obtain the Ω to be fitted as:
a(1 - B + x)
Ω
(14)
2x
Within this geometric framework, the proof of the original theorem be transformed into proving that for given e > 0, there exists an integer m satisfying [mf] - {2H
1logıo c]- We only need to find the integer such that the first M decimal digits of the fractional parts of mıt and Ω are identical. In fact, if take as normal irrational number (normal number means that any numerical pattern can be found in the digits of тT..
Let M = 1
Remark T [(2} 10a4 and S FT(T), where the function FT Z+ N maps digit sequence to its first occurrence index in r's decimal expansion. For example, 3.1415926535...
the mapped values is shown the following:
T (Digit Sequence)
(T)
Positions
3.[1]415...
4
3.141[5|92...
41
2
3.1[41]592...
159
159
In this way, we set m 108-1, which ensures that the first M decimal digits of the fractional parts of mm and Q are identical, i.e.
[ma) - {2H) < 10-M < <e/10< < e.
(15)
B Proof of Lemma 3.2
12
y)
B
int
e
a
-axis
0
e10
m
X-axis
4
Figure Geometric relationship + iy e0 + 8ng
20

---

As shown in Figure 6 according to the definition of tangent function, we have:
a = arctan
(16)
Furthermore, since the magnitudes of e0 and eix0 are both 1, according to the cosine theorem, we have:
22 + 3² +
x + y²
B arccos
arccos(
(17)
2 V 1.2 + 122
2
In particular, we also know that the triangle formed by e c and Ozy is an isosceles triangle. According to the angular relationship, we can easily deduce that:
a 3 = 0 + 2mm,, E Z.
(18)
Additionally, since / and le are parallel, we have:
a: + 3 = T + 2kT, 1: E Z.
(19)
Therefore, Lemma 3.2 is proved by combing Eq. T6 Eq. 19
C Proof of Eq.
Recall that x = cos f + cos 70, and we have:
|r
|| cos(@ + A0) cos 0 + (o + A6)) cos R0|
9440
(sin 0 sin de
0+20
(sin 0 + ssin #0) d0
<
Let 0' = argmax,
(sin 0 + क sin #0)
we have:
Ar A0
sin 0' sin 00
A0
(1 π) sin 0' (sin o' + sin #
(20)
A0
πt) sin 0' +
Since ||''ll < (See Eq.
8 thus we have:
Ar A0l (1 -) + # A6
(21)
Similarly, we can prove that Ay < A6 (11 #) cos 0' + + < A0.
D Proof of Eq. 12
Note that the angle space is [0, 1m remark M 2 1 and have:
21

---

1 Add
E(Ax)
10 10->11 || 0 + sin T0
M
10
AM
||sin 01100 + ! A
#| sinsin T0
M
(22)
10 10-A
2. MM
2. M π
+
M
π
2 (1
+ T)
10-A
π
Similarly, have:
1 PM
E(Ay)
M J
10 10->1| ||scos8 cos T
10-A
M % /M
|| cls 01 d0 + L M
#|| cos T d0)
(23)
10-2 A
2 M
2 M
+
M
π
2 (1 #)
10
E Explanation of Eq.
Note that ग can be written as: 10 10
0.3589793238:
0. 0.0
0000
3589793238
(24)
10
10-2A
Since m retains the first À decimal places of Ω we have:
m π
m 10
+ m 10 2A 0.3589793238
(25)
first À decimal of f
compensation part
In this setting, we guarantee that the difference between # and Ω does not exceed A further understand the construction of m, we present an illustrative example with Ω 1.97525751858 If we set À = 4, then [(2) x 10a = 9752 (first 4 decimal of
Q). We have:
m ग = 0. 9752 350076636569 = {2].
(26)
10-1 compensation
The error is 0.000022510916...
F More Experimental Results
22

---

Table
setting training pipeline.
Dataset
PG19
LRA
Samsum
MovieLens
Learning Rate
2e-5
le-4
3e-5
le-3
Learning Rate Schedule
Linear
Linear
Linear
Weight Decay
0.0
0.0
0.0
0.0
Batch size
64
256
2048
81
0.9
0.9
0.9
0.9
0.95
0.95
0.95
0.95
Table 6: Standard Deviations of LM task
Metric
ARC-c
Hellaswag
Lambada
PIQA
Std (+)
1.37
0.49
0.69
1.04
Table Standard Deviations of Other Machine Learning Tasks
Task Std (+)
Text Acc
Image Acc
Retrieval Acc Listops Acc
PathFinder Acc
BLEU-1 0.0640
BLEU-2 0.0862
0.0531
0.0604
0.0410
0.1352
0.3272
Task Std (+)
BLEU-3
BLEU-4
Recall @ 10
NDCG@ 10 0.2927
MRR @ 10
Hit 10
0.0780
0.0462
0.3585
0.1716
0.3585
Table 8: Model Comparison of Other Backbone LMs
Model
Method
Memory (GB)
Step Time (s)
ARC-e
Hellaswag
Lambada
PIQA
FP32
6.34
5.22
22.95
31.14
32.56
62.51
Bnb.Adam (8-bit)
4.58
5.86
23,05
31.14
32.06
62.37
GPT-2
Lpmm.Adan (4-bit)
3.39
5.37
21.67
30.82
33.02
60.14
T- n-Quant (3.32-bit)
3.07
6.14
22.83
31.14
32.75
62,23
FP32
4.93
1.61
23.49
50.27
Bnb.Adam (8-bit)
4.22
1.67
23.40
50.30
RoBERTa
Lpmm.Adan Adam (4-bit)
3.79
1.64
22.73
48.53
x n-Quant (3.32-bit)
3.57
1.83
23.37
50.27
FP32
55.11
19.27
50.51
76.21
73.55
79.16
Anh.Adam (8-bit)
42.71
19.44
50.55
75.95
73.55
78.96
LLaMA-7B
Lpmm. Adam (4-bit)
36.42
19.35
48.63
74.99
73.02
77.36
--Quant (3.32-bit)
32.87
20.15
50,32
76.48
73.40
79.07
Note: Full reproducibility details are in the supplementary code, '_" denotes the NaN encountered in the output.
23


Reviews:
[{"review_id": "r5YRc7AL4K", "reviewer": "Reviewer_xC83", "summary": "This paper provides the technique for training time compression. The basic idea is pretty neat. Represent numbers in the complex relying on the mathematical formation that X plus Y can be represented as a complex number with one rotation angle. Thus representing two numbers with one angle. The paper has a good mathematical proof , relatively detailed, experiment analysis and evaluation. It also goes into substantial depth to show the angle can be compressed into a small number of bits.", "questions": "Can you please explain intuitively and ideally with some more quantitative data why accuracy improves when you have fewer training bits and especially since you do a form of lossy compression? Basically why do you outperform fp32. Or have I misunderstood your data.\n\nThis may have been in the paper already - can you comment on into an training time and how optimized your GPU kernels are and whether the transformation into the rotations introduces computation latencies that can be addressed in future work", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nGood mathematical proof.\nDetailed training, evaluation and test accuracy on downstream tasks.\nNovel idea.\nDetailed treatment of the number of it’s needed to support the rotation.\n\n\nWeaknesses.\n\nThe main weakness for me in this paper is somewhat poor description for why accuracy improves and perplexity decreases When the number of bits used during training is reduced. I don’t understand intuitively why you outperform fp32. This could simply be a stability  issue. It could be something in their experimental infrastructure. I found the lack of explanation confusing.", "was_revised": true, "final_justification": "The authors provided clarifications to my questions which are helpful. I will keep my score as Accept.", "rebuttal": "Dear Reviewer xC83,\n\nThanks very much for your valuable feedback and insightful suggestions! We list our response to your concerns as follows. If you have further questions, please feel free to let us know. We will continue to try our best to answer for you.\n\nQ1: Can you please explain intuitively and ideally with some more quantitative data why accuracy improves when you have fewer training bits and especially since you do a form of lossy compression? Basically why do you outperform fp32. Or have I misunderstood your data.\n\nA1: We appreciate the chance to elaborate on the accuracy of our approach.\nThe core of our approach is rotation quantization, which allocates precision non-uniformly across the parameter space (as visualized in Fig. 2). \nThis design directly aligns with the statistical properties of neural network parameters: most parameters and optimizer states follow a Gaussian-like distribution, clustering around zero, with only a small fraction having large magnitudes:\n\n- For small-magnitude parameters (near zero, where most values lie), our method retains high precision. \nThis preserves critical information for gradient updates, as these parameters are sensitive to small errors.\n\n- For large-magnitude parameters (rare but impactful), rotation quantization uses coarser steps. We observe that the slight overestimation of large values (due to coarser steps/deviations in high-magnitude regions, see Fig. 2) can act as a mild \"push\" in gradient descent, accelerating convergence by reinforcing impactful updates.\nThis dynamic is particularly evident in Fig. 5, where our method’s optimization path is shorter than FP32’s, escaping local minima faster. Such acceleration is more pronounced on smaller datasets, where convergence speed plays a larger role in final performance.\n\nSpecially, this phenomenon is not unique to our method. \nFor example, Bnb’s 8-bit Adam also occasionally outperforms the FP32 baseline (Table 3), likely due to similar non-uniform quantization benefits.\nIntuitively, the improvement in accuracy is more likely to be observed when working with smaller models. \nFor large-scale models, our approach performs comparably to FP32 optimizers, with convergence trends that are nearly aligned (as shown in Fig. 3).\nSpecifically, the error bar is provided in Table 6 and Table 7.\nSince the core focus of our method is to maintain accuracy on par with existing methods while reducing memory usage through lossy compression, achieving accuracy parity with FP32 when optimizing large models is already in line with our expectations.\nThank you again for your valuable comments. \nWe will incorporate a more detailed discussion on accuracy in the revised manuscript. We are also grateful for your suggestion and would like to list you as one of our contributors.\n\n---\n\nQ2: Can you comment on into an training time and how optimized your GPU kernels are and whether the transformation into the rotations introduces computation latencies that can be addressed in future work\n\nA2: Thank you for your suggestion. Below we provide a granular profiling of Algorithm 1’s execution time, measured on an A100 GPU. \nThe results validate the efficiency of individual components:  \n\n| **Step**                     | **Operation Description**                          | **Time (ms)** | **Percentage** |\n|------------------------------|----------------------------------------------------|---------------|----------------|\n| **Line 2**                   | Splitting $\\mathbf{T}$ into $\\textbf{X}$, $\\textbf{Y}$     | 0.067         | 2.83%          |\n| **Line 3**                   | Computing max value   | 0.4583        | 19.35%         |\n| **Line 4**                   | Scaling $\\tilde{\\textbf{X}}$, $\\tilde{\\textbf{Y}}$         | 0.1998        | 8.43%          |\n| **Line 5**                   | Calculating $\\alpha, \\beta$ (Eq. 3)               | 0.5345        | 22.57%         |\n| **Line 6**                   | Computing ${\\Omega}$ (Eq. 4)                   | 0.6633        | 28.01%         |\n| **Line 7**                   | Solving for $m$ (Eq. 6)                           | 0.1349        | 5.70%          |\n| **Line 8**                   | Generating ${\\Theta}$ (Eq. 8)                 | 0.2630        | 11.11%         |\n| **Other**                    | Memory allocation and data transfer               | 0.0474        | 2.00%          |\n| **Total Time**               | —                                                  | **2.3682**    | **100.00%**    |\n\nThe profiling reveals that 50.58% of the execution time is concentrated in angle computations (Lines 5-6), particularly the $\\arctan$ and $\\arccos$ operations in Eq. 3. \nWhile our current PyTorch implementation uses ATen's general-purpose kernels, we can leverage the constrained input domain $[-1,1]$ and monotonic properties to deploy optimized polynomial approximations. Specifically, we can implement custom CUDA kernels using low-order Taylor expansion (e.g., $\\arctan(u) \\approx u - \\frac{u^3}{3} + \\frac{u^5}{5}$ for $|u|<1$, similar for $\\arccos(u)$).\nIn this way, we can consolidate and simplify the polynomial expressions to directly merge Eq. 3 and Eq. 4 as a bivariate polynomial in terms of x and y, thereby reducing the original three-step computation to a single operation.\n\nOn the other hand, we recognize significant untapped parallelism in our reference implementation. The current sequential execution (provided in supplementary code) overlooks inherent concurrency opportunities—for instance, the max-value computation (Line 3) and tensor splitting (Line 2) have no data dependency and can run concurrently. By constructing a detailed data dependency graph (DAG), we will reorganize operations into parallel streams: \n\n- Independent steps like scaling and normalization will execute in dedicated CUDA streams.\n\n- Element-wise trig operations will maximize warp occupancy through batched SIMD execution.\n\n- Memory-bound steps (e.g., ${\\Omega}$ generation) will overlap computation with data transfers. \n\nThese approaches could largely reduce total latency, particularly benefiting the scaling and angle computation phases that dominate runtime. We are implementing these optimizations and will include benchmark comparisons in the camera-ready version."}, {"review_id": "YKEV0siquu", "reviewer": "Reviewer_FyEQ", "summary": "This paper proposes $\\Pi-Quant$, an optimizer state compression method that uses irrational complex rotations to achieve low-bit quantization for optimizer states in neural networks. The main innovation is a mathematical representation of a pair of real numbers using a single rotation angle, allowing halving the amount of optimizer parameters. The authors further propose quantizing the resulting representation angle with high precision and low error.", "questions": "I would like the authors to address the concerns raised in the weaknesses section. Especially theoretical concerns and paper limiations.\nFurthermore, the authors marked \"yes\" in the statistical significance section of their paper checklist. I could not see where in the paper this is addressed, and request the authors the justify their answer.", "limitations": "As previously mentioned, paper limiations and societal implications are not sufficiently discussed. Futhermore, the authors only marked \"yes\" or \"no\" in their paper checklist, without acutally justifying any of their answers. This makes some of the answers on the checklist difficult to accept.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strenghs:\n1. Under the experimentation brought in the paper, which I find to be sufficient in both diversity of experiments and chosen cohort of baselines, this paper shows promising results in terms of compression potential. The proposed method manages to drop below the 4-bit parameter bit-width known from existing methods, while also introducing favorable (linear) runtime in quantization.\n\n2. The proposed method relies on elegant, mathematically-sound and geometrically interpretable foundations, which I believe make this method both clear and extnedible for future work.\n\n3. Presentation is notably clear - contributions are clearly stated and motivated. Theoretical foundations are well-structured by the appropriate lemmas and theorems.\n\nWeaknesses:\nI believe the main contributions of the paper, while solid, are mostly empirical. I find some of the theoretical analysis lacking, or at least requiring further clarification.  \nThere is a theoretical gap in theorem 3.1 - To my understanding, the arguments in appendix A only establish density, not surjectivity. The function $f(\\theta)$ is bounded and non-periodic, and thus fills a dense subset of a region in $\\mathbb{R}^2$, but not necessarily every point exactly. This means the existence of an exact $\\theta$ for an arbitrary point in the disk is not guaranteed by their argument — only that it can be approximated arbitrarily closely. Additionally, the function is not injective, so uniqueness of $\\theta$ is not sufficiently justified. Unless the authors can indicate an error in my reasoning, the implications of theorem 3.1 are overstated. \n\nFurthermore, while the notion of using irrational numbers for compressed representation is innovative and seems to work well in practice, this reliance on irrational numbers renders $\\Pi-Quant$ dependant on the representation error of irrational numbers, that cannot be fully represented in a computer. $\\bar{\\pi}$ is ultimately a rational approximation of an irrational number.\nThe non-periodicity of $f(\\theta) = e^{i\\theta} + e^{i \\bar{pi} \\theta}$ is only approximate, not absolute — the function may exhibit long-term pseudo-periodicity. The angle $\\theta$ is quantized to a finite precision controlled by a hyperparameter $\\lambda$ which determines how many digits are preserved. This could mean that means the actual values of $x+iy$ that are recoverable may deviate from their theoretical values, particularly for small $\\lambda$ or very large vectors. To my understanding, the claim that \"every parameter pair can be encoded precisely\" only holds in theory. \nWhile the implications of the points I raised may or may not be minor in practice, and while I do not believe they pose grounds for rejection, these concers should be addressed by the authors.\n\n\nLastly, the paper is missing a more-extensive discussion of limiations, which I believe is essential for publication. Another minor comment is that figure 1 is overly crowded. It does not take much room in the paper, and I propose making its contents more spacious to make it more readable.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' responses to both my concerns and those raised by other reviewers, and I find them convincing. This is reflected in my final assessment. I strongly encourage the authors to incorporate all clarifications and improvements from their responses into the final version of the paper.", "rebuttal": "Dear Reviewer FyEQ,\n\nThanks very much for your careful reading and insightful suggestions of our paper! We appreciate the time and efforts you have dedicated to reviewing our work. We list our response to your concern in the following three aspects. If you have further questions, please feel free to let us know. We will continue to try our best to answer for you.\n\nQ1. Some of the theoretical analysis lacking, or at least requiring further clarification.\n\nA1: Your insights are truly profound—we recognize the theoretical limitations in our initial formulation.\nWe have refined it as follows:\n\n**Revised Theorem 1**: \n> Given a complex number $z = x + iy \\in \\mathbb{C}$ that satisfies $\\|z\\| \\leq 2$. For any deviation $\\epsilon > 0$, there exists an angle $\\theta \\in \\mathbb{R}$, such that  \n$$\n\\left\\| z - \\left( e^{i\\theta} + e^{i\\pi\\theta} \\right) \\right\\| \\leq \\epsilon.\n$$\n\n**Proof**:\n\nSimilar to the original proof, to approximate \n$z = x +i y$, we construct geometric relationships following Fig. 7, thereby deriving the system of geometric equations in Eq. (3):\n\n$\\alpha = \\arctan\\left(\\frac{y}{x}\\right),  \\beta = \\arccos\\left(\\frac{\\sqrt{x^2 + y^2}}{2}\\right)$\n\n$\\theta = \\alpha - \\beta + 2m\\pi, m \\in \\mathbb{Z}$\n\n${\\pi}\\theta = \\alpha + \\beta + 2k\\pi, k \\in \\mathbb{Z} $\n\nSimilarly, we can obtain the $\\Omega$ to be fitted as:\n\n$\\Omega = \\frac{\\alpha(1 - {\\pi}) + \\beta(1 + {\\pi})}{2 \\pi}$.\n\nWithin this geometric framework, the proof of the original theorem can be transformed into proving that:\n\n>  For given $\\epsilon > 0$, there exists an integer $m$ satisfying $| [ m\\pi ] - [ \\Omega ]| < \\epsilon$. (Here $[ \\cdot ]$ returns the fractional parts of input value.)\n\nLet $M = 1- \\lfloor\\log_{10}\\epsilon\\rfloor$. We only need to find the integer $m$ such that the first M decimal digits of the fractional parts of $m\\pi$ and $\\Omega$ are identical.\nIn fact, if we take $\\pi$ as a **normal** irrational number (where 'normal number' means that any numerical pattern can be found in the digits of $\\pi$ [1]).\n\nRemark $T = \\lfloor [\\Omega] \\times 10^{M} \\rfloor$ and $S = \\mathcal{F}{\\pi}(T)$, where the function $\\mathcal{F}{\\pi}: \\mathbb{N} \\to \\mathbb{N}$ maps a digit sequence to its first occurrence index in $\\pi$'s decimal expansion. For example, $\\pi = 3.1415926535...$, the mapped values is shown as the following:\n\n| $T$ (Digit Sequence) | $\\mathcal{F}_{\\pi}(T)$ | Verification (Positions) |\n|----------------------|------------------------|--------------------------|\n| `1`                  | 1                      | 3.`1`415... |\n| `5`                  | 4                      | 3.141`5`92...          |\n| `41`                 | 2                      | 3.1`41`592...          |\n| `159`                | 3                      | 3.14`159`265...         |\n\n\nIn this way, we set $m = 10^{S - 1}$, which ensures that the first M decimal digits of the fractional parts of $m\\pi$ and $\\Omega$ are identical, i.e., \n\n$|[ m\\pi ] - [ \\Omega ]| < 10^{-M} \\leq {\\epsilon} / {10} < \\epsilon$.\n\nIn the context of practical numerical computation where values are constrained by finite precision or resolution (e.g., FP32 with $\\epsilon \\approx 10^{-7}$), the dense coverage guarantees that every machine-representable point can be fitted by some $\\theta \\in \\mathbb{R}$. **Therefore, the modification of the theory does not affect the correctness of our proposed compression/quantization algorithm**.\n\nWe sincerely appreciate you identifying this critical theoretical gap in our original formulation, which has significantly strengthened our work. We will explicitly correct this limitation and modify our theorem accordingly in the revised manuscript. Furthermore, we would be honored to recognize your contribution in the acknowledgments section as a key theoretical contributor to this important refinement.\n\n\n[1] Bailey D H, Crandall R E. Random generators and normal numbers[J]. Experimental Mathematics, 2002, 11(4): 527-546.\n\n---\n\nQ2: The representation error of irrational numbers, that cannot be fully represented in a computer. $\\pi-Quant$ is ultimately a rational approximation of an irrational number. The non-periodicity of  is only approximate, not absolute — the function may exhibit long-term pseudo-periodicity.\n\nA2: \nThank you for this astute observation. You are absolutely correct in noting that the representation of irrational numbers (e.g., $\\pi$) in our framework is inherently an approximation, as computers can only represent finite-precision values. The theoretical guarantee of dense coverage holds in the limit of infinite precision, but in practice, our $\\theta$ angles are represented with finite bits, introducing unavoidable approximation errors.\n\nActually, our method does not rely on infinite-bit precision approximation. It is grounded in a rotation-based quantization framework that leverages the unique properties of complex rotations in their mathematical structure, coupled with careful alignment to neural network dynamics, to mitigate the drawbacks of lossy compression. Specifically, our scheme minimizes quantization artifacts through two interconnected mechanisms, both substantiated by our theoretical analysis and empirical results:\n\n- First, the **non-uniform precision allocation** (Fig. 2) is deliberately designed to align with the sensitivity of parameters during training. As discussed in Section 3.3, neural network parameters and optimizer states follow a Gaussian-like distribution, with most values clustering near zero—where small perturbations can significantly impact gradient stability and convergence. Our rotation quantization capitalizes on this by allocating finer precision (smaller quantization steps) to values near zero (preserving critical information for accurate gradient updates) and coarser precision to large-magnitude values. This approach stands in contrast to uniform precision, which wastes bits on large, insensitive parameters and can introduce stability issues due to unnecessary precision in noisy regions. Quantitatively, Eq. 10 shows that our quantization error is smaller than $\\Delta\\theta$, resulting in smaller average errors compared to conventional uniform quantization, especially for the small-magnitude parameters that dominate gradient dynamics.  \n\n- Second, our **pair-wise quantization into the rotation space** (Section 3.1) preserves critical structural information lost in element-wise quantization. Unlike traditional methods that quantize each parameter independently (ignoring correlations between parameters), our rotation-based approach explicitly models the covariance between parameter pairs $(x, y)$ by mapping them to a single rotation angle $\\theta$. \nIntuitively, most parameter pairs share similar distributions, and the rotation-based mapping acts as a form of dimensionality reduction that retains critical correlations. \nThis provides a new perspective for further reducing the bit-width of quantization methods.\n\n\nOverall, despite being a form of lossy compression, our method maintains stability and performance parity with FP32 because the introduced errors are structured to align with the model’s learning dynamics—regularizing large-magnitude parameters where noise is less detrimental (Section 3.3). As shown in Table 3 and Fig. 2, experimental results in convergence trajectories and perplexity values comparable to FP32, validating its effectiveness for large language models.\nYour feedback helps clarify the need to better distinguish theoretical guarantees from practical implementation, which we will address in the revised manuscript. Thank you again for this valuable insight.\n\n---\n\nQ3: Lastly, the paper is missing a more-extensive discussion of limiations, which I believe is essential for publication. Another minor comment is that figure 1 is overly crowded. It does not take much room in the paper, and I propose making its contents more spacious to make it more readable.\n\nA3: We sincerely appreciate your valuable feedback and will implement both suggestions comprehensively in the revised version. Specifically, we will add a dedicated \"Limitations\" section to the conclusion (Section 5) to thoroughly discuss computational overhead, sensitivity to extreme parameter distributions, and applicability boundaries beyond momentum-based optimizers. Simultaneously, we will completely redesign Figure 1 by expanding layout spacing, separating geometric equations from the pipeline diagram into distinct panels, and incorporating color-coded flow markers to enhance visual clarity and readability. Thank you for these concrete suggestions—they significantly strengthen our paper’s rigor and accessibility, and all changes will be explicitly highlighted in the revised manuscript.\n\n---\n\nQ4: Furthermore, the authors marked \"yes\" in the statistical significance section of their paper checklist. I could not see where in the paper this is addressed, and request the authors the justify their answer.\n\nA4: We sincerely appreciate your meticulous attention to the statistical significance reporting in our paper checklist. Regarding the statistical significance section requirement—\"Does the paper report error bars suitably and correctly defined...?\" We confirm that such information is provided in Table 6 (std for language modeling tasks) and Table 7 (std for downstream tasks) in the Appendix. All error bars are calculated over ten independent trials.\n\nTo further ensure reproducibility, we included detailed code in supplementary materials that documents all statistical evaluation protocols, including random seed management, hyperparameter configurations. We acknowledge that this critical information should have been more prominently highlighted in the main text and will add explicit references to Tables 6-7 in the revised manuscript. Thank you for reinforcing the importance of rigorous statistical reporting—this refinement will enhance the paper's scholarly integrity."}, {"review_id": "QKnnRVbjLu", "reviewer": "Reviewer_pNT4", "summary": "The paper proposes to quantize the parameters of a the neural network optimizer into lower-bit representation by using the properties of complex rotation function. Parameter tensors T are split into X and Y matrices, and then approximately mapped into angular rotation in complex plane via elementwise function $z(\\theta) = x + iy = e^{i\\theta}+e^{i \\bar{\\pi} \\theta} $ where $\\bar{\\pi}$ is specially chosen irrational number. The paper proposes a solution of finding the approximate $\\theta$ given x and y pair that can be controlled by the design of the $\\bar{\\pi}$ number. Then, using this quantization technique, the authors re-desing Adam optimizer to store optimizers states in in angular form and de-quantizing them just-in-time when these parameter are being used/updates. The overhead of these transformations seems to be marginal (within 10% of the actual forward/backward times) and numerical results supports that the transformation has good approximation power (allowing to store the optimizer's state with 4bit parameter precision).", "questions": "Please address my concerns regarding the correctness of the theorem 3.1. \n\nAlso, I was wondering whether the authors applied the efficient-parameter changes to other optimizers (i.e., SGD with momentum)? I am curious whether same practical results will hold with other optimizers (i.e., is this generic enough), or for some reason it should be paired with Adam.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The proposed idea is quite novel, and the practical results make the method to stand out. However, I do have some concerns regarding the statements within the theorems. \n\nIt seems that Theorem 3.1 is wrong. It was a while for me since I took any classes in geometric measure theory, but the function in equation 2 is Lipschits continues, and maps 1D interval into 2D. As such, its  Hausdorff dimension is at most 1, meaning it cannot be a space-filling curve, i.e., we cannot map every point within the circle into some \\theta. Please take a look at Theorem/Propositon 2.3 in Fractal Geometry: Mathematical Foundations and Applications by Kenneth Falconer. Again I might be wrong in my analysis, but this is a serious impediment for the acceptance of the paper if true. \n\nPractically speaking, I don't think we need the theorem 3.1 because floating point representations is anyways a discretization of the space (albeit in tiny intervals), and probably the function 2 can map any fp32 number within disc to some theta with good approximation. However, we need a precise and correct statement regarding this in the paper. I will put week reject for now, but once this issue is (hopefully) clarified, I am willing to put strong accept.", "was_revised": true, "final_justification": "Authors clarified and answered all my questions. I think this is a technically solid paper (if all discussions will be synthesized back into the paper)", "rebuttal": "Dear Reviewer pNT4,\n\nThanks very much for your careful reading and insightful suggestions of our paper! We appreciate the time and efforts you have dedicated to reviewing our work. We list our response to your concern in the following three aspects.\nIf you have further questions, please feel free to let us know. We will continue to try our best to answer for you.\n\n\nQ1: It seems that Theorem 3.1 is wrong. It was a while for me since I took any classes in geometric measure theory, but the function in equation 2 is Lipschits continues, and maps 1D interval into 2D. As such, its Hausdorff dimension is at most 1, meaning it cannot be a space-filling curve.\n\nA1: Thank you for your profound insight into the geometric measure theory implications of our mapping function. You are absolutely correct that our original claim of exact surjectivity in Theorem 3.1 was mathematically unsound. We have revised the theorem and its interpretation as follows:\n\n**Revised Theorem 1**: \n> Given a complex number $z = x + iy \\in \\mathbb{C}$ that satisfies $\\|z\\| \\leq 2$. For any deviation $\\epsilon > 0$, there exists an angle $\\theta \\in \\mathbb{R}$, such that  \n$$\n\\left\\| z - \\left( e^{i\\theta} + e^{i\\pi\\theta} \\right) \\right\\| \\leq \\epsilon.\n$$\n\n**Proof**:\n\nSimilar to the original proof, to approximate \n$z = x +i y$, we construct geometric relationships following Fig. 7, thereby deriving the system of geometric equations in Eq. (3):\n\n$\\alpha = \\arctan\\left(\\frac{y}{x}\\right),  \\beta = \\arccos\\left(\\frac{\\sqrt{x^2 + y^2}}{2}\\right)$\n\n$\\theta = \\alpha - \\beta + 2m\\pi, m \\in \\mathbb{Z}$\n\n${\\pi}\\theta = \\alpha + \\beta + 2k\\pi, k \\in \\mathbb{Z} $\n\nSimilarly, we can obtain the $\\Omega$ to be fitted as:\n\n$\\Omega = \\frac{\\alpha(1 - {\\pi}) + \\beta(1 + {\\pi})}{2 \\pi}$.\n\nWithin this geometric framework, the proof of the original theorem can be transformed into proving that:\n\n>  For given $\\epsilon > 0$, there exists an integer $m$ satisfying $| [ m\\pi ] - [ \\Omega ]| < \\epsilon$. (Here $[ \\cdot ]$ returns the fractional parts of input value.)\n\nLet $M = 1- \\lfloor\\log_{10}\\epsilon\\rfloor$. We only need to find the integer $m$ such that the first M decimal digits of the fractional parts of $m\\pi$ and $\\Omega$ are identical.\nIn fact, if we take $\\pi$ as a **normal** irrational number (where 'normal number' means that any numerical pattern can be found in the digits of $\\pi$ [1]).\n\nRemark $T = \\lfloor [\\Omega] \\times 10^{M} \\rfloor$ and $S = \\mathcal{F}{\\pi}(T)$, where the function $\\mathcal{F}{\\pi}: \\mathbb{N} \\to \\mathbb{N}$ maps a digit sequence to its first occurrence index in $\\pi$'s decimal expansion. For example, $\\pi = 3.1415926535...$, the mapped values is shown as the following:\n\n| $T$ (Digit Sequence) | $\\mathcal{F}_{\\pi}(T)$ | Verification (Positions) |\n|----------------------|------------------------|--------------------------|\n| `1`                  | 1                      | 3.`1`415... |\n| `5`                  | 4                      | 3.141`5`92...          |\n| `41`                 | 2                      | 3.1`41`592...          |\n| `159`                | 3                      | 3.14`159`265...         |\n\n\nIn this way, we set $m = 10^{S - 1}$, which ensures that the first M decimal digits of the fractional parts of $m\\pi$ and $\\Omega$ are identical, i.e., \n\n$|[ m\\pi ] - [ \\Omega ]| < 10^{-M} \\leq {\\epsilon} / {10} < \\epsilon$.\n\nIn the context of practical numerical computation where values are constrained by finite precision or resolution (e.g., FP32 with $\\epsilon \\approx 10^{-7}$), the dense coverage guarantees that every machine-representable point can be fitted by some $\\theta \\in \\mathbb{R}$. **Therefore, the modification of the theory does not affect the correctness of our proposed compression/quantization algorithm**.\nWe will updated Section 3.1 and Appendix A to reflect this rigorous framing.\nYour expertise significantly improved our work’s mathematical integrity – we are deeply grateful, and would also like to acknowledge you as a theoretical contributor in the Acknowledgments section of our paper.\n\n[1] Bailey D H, Crandall R E. Random generators and normal numbers[J]. Experimental Mathematics, 2002, 11(4): 527-546.\n\n---\n\nQ2: Practically speaking, I don't think we need the theorem 3.1 because floating point representations is anyways a discretization of the space (albeit in tiny intervals), and probably the function 2 can map any fp32 number within disc to some theta with good approximation.\n\nA2: You are absolutely right that floating-point representations inherently discretize the space.\nThe theoretical guarantee of dense coverage holds in the limit of infinite precision, but in practice, our $\\theta$ angles are represented with finite bits, introducing unavoidable approximation errors.\nDespite the limitation, our rotation function (Eq. 2) can map most FP32 number within the disc to some $\\theta$ with sufficient approximation, even without strict theoretical guarantees of dense coverage. \n\nTheorem 3.1 was originally included to provide a mathematical foundation for why rotation-based quantization *can* minimize lossy compression artifacts—by illustrating that complex rotations (in the theoretical limit) offer dense coverage of the target space, justifying the intuition behind our design. However, we agree that its practical relevance is secondary to the empirical and structural strengths of our method, which do not depend on \"perfect\" dense coverage.  \n\nAs emphasized in our earlier response, our core contributions lie in two practical mechanisms: (1) non-uniform precision allocation aligned with parameter sensitivity (Fig. 2), which reduces unnecessary bit usage for large, error-tolerant values; and (2) pair-wise quantization that preserves covariance between parameters (Section 3.1), which is capable of capturing structural information lost in point-wise methods. These design choices drive our method’s ability to maintain FP32-comparable performance with fewer bits (Table 3, Fig. 2).  \n\nWe appreciate your feedback highlighting the need to reframe Theorem 3.1 as a motivating intuition rather than a strict practical requirement. In the revised manuscript, we will clarify this distinction, emphasizing that our method’s effectiveness stems from its alignment with neural network dynamics.  Thank you again for helping strengthen the clarity of our work.  \n\n---\n\nQ3: Also, I was wondering whether the authors applied the efficient-parameter changes to other optimizers (i.e., SGD with momentum)? I am curious whether same practical results will hold with other optimizers (i.e., is this generic enough), or for some reason it should be paired with Adam.\n\nA3: Thank you for this valuable question. We confirm that our method is generic and can be applied to other optimizers (e.g., SGD with momentum).\nTo illustrate, we conducted additional experiments on TinyLlama using SGD with momentum. The results show nearly identical performance to FP32 SGD:  \n\n| Metric     | FP32 SGD      | $\\pi$-Quant SGD   |  \n|------------|---------------|---------------|\n| **Mem**    | 14.48 G       | 10.184 G |  \n| **PPL**    | 5.79          | 5.79          |  \n| **ARC-C**  | 31.70         | 31.83     |  \n| **Hellaswag** | 57.70       | 57.43         |  \n| **Lambada**| 56.73         | 56.94     |  \n| **PIQA**   | 73.22         | 73.03         |  \n\nNotably, adapting our method to other optimizers or tasks requires only minor modifications following the code structure in Appendix B. \nYou can directly replace the SGD optimizer's momentum update steps with our rotation-based transformation as outlined in supplementary material. \nWe will release the full experimental code for these extensions after the submission period, ensuring reproducibility.\nThank you again for your insight."}, {"review_id": "atp2LFrj6o", "reviewer": "Reviewer_QC8y", "summary": "The paper introduce a method called π-Quant which leverages irrational complex rotations to enable memory-efficient training of neural networks. By compressing the real-valued parameter pair (x,y) into a single rotation angle θ using Euler’s formula, π-Quant reduce the memory from 4 bits to 3.32 bits while maintain the full model accuracy", "questions": "please see weakness sections", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Pros\n1. They propose a geometric equation for solving rotation angle θ efficiently which only require linear-time complexity.\n2. They achieve 3.32 bit quantization of optimizer states while maintain the full accuracy, which is also supported by mathematical result instead of purely experimental.\n\nCons\n1. They introduce additional restoration steps in each iteration, which can increase the training time and complexity compared to baseline optimizers.\n2. The example given in Appendix E does not follow what they claimed. \n3. Even though for each transformation step, the approximation error is small, but it is unclear what would happen when all these error accumulated from repeated quant/restore cycles.", "was_revised": true, "final_justification": "Thank you for your response. I am in favor of accepting the paper.", "rebuttal": "Dear Reviewer QC8y,  \n\nThanks very much for your valuable feedback and insightful suggestions! We list our response to your concerns as follows. If you have further questions, please feel free to let us know. We will continue to try our best to answer for you.\n\nQ1: They introduce additional restoration steps in each iteration, which can increase the training time and complexity compared to baseline optimizers.\n\nA1: Thank you for highlighting this important consideration. We acknowledge that the additional restoration steps in our algorithm introduce marginal overhead compared to baseline optimizers.\n\nTo quantify this, we conducted a granular profiling of Algorithm 1’s execution time on an A100 GPU, breaking down the runtime of each component. As shown in the table below, the total execution time per iteration is 2.3682 ms, with the majority attributed to angle-related computations (Lines 5–6: 22.57% + 28.01% = 50.58%), primarily involving $\\arctan$ and $\\arccos$ operations.  \n\n| **Step**                     | **Operation Description**                          | **Time (ms)** | **Percentage** |\n|------------------------------|----------------------------------------------------|---------------|----------------|\n| **Line 2**                   | Splitting $\\mathbf{T}$ into $\\textbf{X}$, $\\textbf{Y}$     | 0.067         | 2.83%          |\n| **Line 3**                   | Computing max value                                | 0.4583        | 19.35%         |\n| **Line 4**                   | Scaling $\\tilde{\\textbf{X}}$, $\\tilde{\\textbf{Y}}$         | 0.1998        | 8.43%          |\n| **Line 5**                   | Calculating $\\alpha, \\beta$ (Eq. 3)               | 0.5345        | 22.57%         |\n| **Line 6**                   | Computing ${\\Omega}$ (Eq. 4)                   | 0.6633        | 28.01%         |\n| **Line 7**                   | Solving for $m$ (Eq. 6)                           | 0.1349        | 5.70%          |\n| **Line 8**                   | Generating ${\\Theta}$ (Eq. 8)                 | 0.2630        | 11.11%         |\n| **Other**                    | Memory allocation and data transfer               | 0.0474        | 2.00%          |\n| **Total Time**               | —                                                  | **2.3682**    | **100.00%**    |\n\nNotably, this overhead is manageable and can be significantly reduced with targeted optimizations. Our current implementation uses PyTorch’s general-purpose ATen kernels, but we are developing custom CUDA optimizations to streamline the most costly steps:  \n1. Leveraging the monotonicity and constrained input domain of trigonometric operations (e.g., $\\arctan$, $\\arccos$) to replace them with low-order polynomial approximations, merging Eqs. 3 and 4 into a single operation.  \n2. Exploiting parallelism through a data dependency graph (DAG) that runs independent steps (e.g., tensor splitting and max-value computation) concurrently in separate CUDA streams.  \n\nThank you again for your attention to this detail. We will include updated benchmarks with these optimizations in the revised manuscript.  \n\n---\n\nQ2: The example given in Appendix E does not follow what they claimed.\n\nA2: We sincerely appreciate the reviewer's meticulous examination. Appendix E aims to illustrate the bound $|\\{m \\cdot \\bar{\\pi}\\} - \\{\\Omega\\}| \\leq 10^{-\\lambda}$ through a concrete example. Upon rechecking, we confirm the example is mathematically consistent but requires clarification in presentation. Here is the detailed verification:\n\n- Given: $\\Omega = 1.97525751858\\cdots$, $\\lambda = 4$\n- Compute $m$:  \n  $\n  m = \\lfloor \\{\\Omega\\} \\times 10^{\\lambda} \\rfloor = \\lfloor 0.97525751858 \\times 10^4 \\rfloor = \\lfloor 9752.5751858 \\rfloor = 9752\n  $\n- Compute $m \\cdot \\bar{\\pi}$:  \n  $\n  \\bar{\\pi} = 10^{-4} + 10^{-8} \\cdot 0.3589793238 = 0.000100003589793238$ \n\n    0.975235007663657  $\n  m \\cdot \\bar{\\pi} = 9752 \\times 0.000100003589793238 = 0.975235007663657\n  $\n- Claimed error:  \n  $\n  |m \\cdot \\bar{\\pi} - \\{\\Omega\\}| = |0.975235007663657 - 0.97525751858| = 0.000022510916343 < 0.0001\n  $\n\n We thank the reviewer for highlighting this and will restructure Appendix E to provide explicit step-by-step derivations and visual illustrations, ensuring readers gain intuitive understanding of the algorithm's foundational operations.\n\n---\n\n Q3: Even though for each transformation step, the approximation error is small, but it is unclear what would happen when all these error accumulated from repeated quant/restore cycles.\n\nA3: We thank the reviewer for raising this important concern about error accumulation during repeated quantization/restoration cycles. Our theoretical analysis and extensive experiments confirm that approximation errors do not compound over training iterations. To directly address your question, we conducted extended 20,000-step training on TinyLlama and recorded the loss trajectory compared to FP32:\n\n**Avg. Loss Comparison Over Extended Training (TinyLlama)**  \n| Step | FP32 Loss      | π-Quant Loss   |  \n|--------------|----------------|----------------|  \n| 2k            | 2.71435644     | 2.72284928     |  \n| 4k            | 2.69537313     | 2.69423438     |  \n| 6k            | 2.72945515     | 2.71732398     |  \n| 8k            | 2.72485786     | 2.72879837     |  \n| 10k           | 2.70670259     | 2.71383749     |  \n| 12k           | 2.69904825     | 2.69482937     |  \n| 14k           | 2.69350642     | 2.69380413     |  \n| 16k           | 2.69184519     | 2.68949827     |  \n| 18k           | 2.68973141     | 2.67384239     |  \n| 20k           | 2.69378422     | 2.69178723     |  \n\nWe observe that as training progresses, the loss curves of $\\pi$-Quant and FP32 remain nearly identical, with no signs of performance degradation.\n\nActually, our extensive experiments have systematically validated the robustness of our method across diverse model scales and task types.\n\n- For model sizes, we tested configurations ranging from models like TinyLlama to GPT-2, and further to larger models like Llama-7B. As detailed in Table 3 and Table 8, even under extremely low-bit settings (e.g., 3.32-bit quantization), the perplexity gap between our quantized models and full-precision (FP32) counterparts remains within 0.05, with downstream task accuracy differences stably below ~1%.\n\n- In terms of task diversity, we evaluated performance across language modeling (LM), classification tasks including text and image, and other benchmarks (Table 4). Our method consistently maintains performance comparable to FP32. These consistent results across models and tasks strongly demonstrate the robustness of our approach.\n\nThank you again for your insightful feedback, which has helped us emphasize the reliability of our method."}, {"review_id": "s7MqH2YnhD", "reviewer": "Reviewer_V7cB", "summary": "The paper proposes a novel optimizer state compression algorithm named \\pi-Quant, which aims to solve the high memory consumption problem caused by optimizer states (such as Adam's momentum) in large-scale AI model training. It can precisely represent a pair of real parameters (x, y) as a single rotation angle \\theta, thus halving the number of parameters. To efficiently solve for the rotation angle \\theta, the paper also designs a system of geometric equations with linear complexity.", "questions": "See 'Weaknesses' for details.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. This paper introduces a novel and elegant approach by integrating the properties of irrational numbers with Euler's formula and complex rotations to address parameter compression in deep learning, thereby providing a fresh perspective for the field.\n2. The paper's approach is grounded in rigorous mathematical theorems and lemmas. By providing detailed proofs in the appendix, the authors theoretically ensure the method's viability, thereby strengthening the credibility of the paper.\n3. Compared to quantization methods that require complex search operations (e.g., Bnb and Lpmm) , the geometric solution proposed by \\pi-Quant has O(n) linear complexity and does not require compiling additional GPU kernels, making it easier to integrate into existing deep learning frameworks.\n\nWeaknesses\n\n1. The paper's main limitation, as acknowledged by the authors, is its computational overhead. A further discussion on how this cost scales with model size would be beneficial. For instance, does this speed disadvantage become more pronounced for larger models?\n2. In Figure 2 (top), it is not clear what the colors and the horizontal and vertical axes represent.\n3. The figure number on line 292 is incorrect; it should be Figure 5. Additionally, in Figure 5, \"Adam (32, 8, 4) bits\" is represented by a single color, which seems to imply that the optimizer paths for these three bit-widths are identical. If this is the case, it should be stated. If not, they should be differentiated with different colors.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer V7cB,\n\nThanks very much for your careful reading and insightful suggestions of our paper! We appreciate the time and efforts you have dedicated to reviewing our work. We list our response to your concern in the following three aspects.\nIf you have further questions, please feel free to let us know. We will continue to try our best to answer for you.\n\nQ1: The paper's main limitation, as acknowledged by the authors, is its computational overhead. A further discussion on how this cost scales with model size would be beneficial. For instance, does this speed disadvantage become more pronounced for larger models?\n\nA1: We appreciate the reviewer's insightful inquiry regarding the computational overhead scaling of our approach.\nTo address this, we conducted experiments across different model sizes, measuring the optimizer time per iteration for both FP32 and our method:\n\n| Model          | Step Time (FP32) | Step Time (ours) | Impr. |\n|---------------|-------------------|-----------------------|---------------|\n| RoBERTa | 1.61              | 1.83                  | +13.7%        |\n| GPT-2         | 5.22              | 6.14                  | +17.6%        |\n| TinyLlama-1.1B| 9.77              | 10.56                 | +8.1%         |\n| LLaMA-7B      | 19.27             | 20.15                 | +4.6%         |\n\nNotably, as model size increases, the proportional overhead of our method decreases. For smaller models like RoBERTa, our overhead is ~13.7%, while for larger models like Llama-7B, it drops to 4.6%. \n\nThis trend arises because larger models involve more parameters and a greater volume of forward computations, such as attention mechanisms and feed-forward layers. Importantly, the time complexity of these core model operations grows faster with increasing parameter count than that of the optimizer, which operates at $O(n)$ complexity. As a result, the relative contribution of the optimizer’s runtime to the total training time diminishes as model size increases, making the overhead of our method less pronounced for larger models.\nThus, the speed disadvantage becomes less pronounced as model size grows, which aligns with our goal of practicality for large-scale models.\n\n---\n\nQ2: In Figure 2 (top), it is not clear what the colors and the horizontal and vertical axes represent.\n\n\nA2: We appreciate the reviewer’s feedback regarding Figure 2 (top). The visualization depicts the **quantization error distribution** of our rotation-based compression scheme across the complex plane, where each point $(x, y)$ denotes a unique **parameter pair**:\n\n- **Horizontal Axis (x)** represents the real component of complex numbers ($x$ in $z = x + iy$), corresponding to the first parameter in each pair.  \n- **Vertical Axis (y)** represents the imaginary component $y$ in $z = x + iy$, corresponding to the second parameter.\n- **Color Mapping**: Induces the magnitude of quantization error ($||z_{\\text{original}} - z_{\\text{reconstructed}}\\|_2$). The color scale ranges from deep blue (near-zero error) to yellow (high error), with warmer colors indicating larger reconstruction discrepancies after rotation-based quantization and restoration.  \n\nThe original labeling was insufficient. In the revision, we will add explicit axis labels, and include a color-bar legend, and annotate characteristic zones (e.g., origin, max-error points) with error values.\nThese adjustments will ensure readers intuitively grasp how geometric properties of complex rotations govern error behavior—a cornerstone of our methodology. Thank you for helping us improve the exposition.\n\n---\n\nQ3: The figure number on line 292 is incorrect; it should be Figure 5. Additionally, in Figure 5, \"Adam (32, 8, 4) bits\" is represented by a single color, which seems to imply that the optimizer paths for these three bit-widths are identical. If this is the case, it should be stated. If not, they should be differentiated with different colors.\n\nA3: Thank you so much for your meticulous attention to detail and valuable feedback. We sincerely apologize for the inaccuracies in Figure 5 and the incorrect figure number reference on line 292; the line 292 reference will be corrected to \"Figure 5\" in the revised manuscript, and we deeply appreciate you catching this oversight. Regarding the representation of \"Adam (32, 8, 4) bits\" in Figure 5, while the three bit-widths do have distinct optimization paths, the differences between them are relatively small. \n\nIn our initial attempt to avoid visual clutter, we merged their trajectories, but we now recognize that this was an inappropriate choice as it masks meaningful distinctions. To address this, we will redraw Figure 5 with separate, distinct colors for each bit-width, ensuring the unique paths of the three configurations are clearly visible and easy to interpret. This revision will provide a more accurate and informative visualization of how varying bit-widths impact optimization dynamics, and we are grateful for your guidance in making this improvement. Thank you again for your thoughtful input, which continues to help us refine our work."}]

Author rebuttal:

Meta-review: This paper proposes $\pi$-Quant, an optimizer state compression method that leverages irrational complex rotations to represent a pair of parameters with a single rotation angle in the complex plane. A system of geometric equations enables linear-time computation of these angles, and extensive experiments show that $\pi$-Quant reduces parameter precision to 3.32 bits, cutting GPU memory usage by 41.8% without accuracy loss. Reviewers concur on the importance of the problem and the novelty and mathematical elegance of the approach, noting its rigorous theoretical grounding (bridging number theory and optimization) and thorough empirical validation. Overall, the work sets a strong benchmark for memory-efficient training.
Decision: Accept (poster)