Title: Irrational Complex Rotations Empower Low-bit Optimizers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Irrational Complex Rotation

Training-oriented Quantization

Memory-efficient Optimization
Abstract: In this paper, we propose a novel optimizer state compression algorithm, namely \textbf{$\pi$-Quant}, which leverages the properties of irrational numbers (\eg $\pi$) for memory-efficient training. 
The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme.
Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. 
To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity.
We evaluate $\pi$-Quant on  a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 41.8\% decrease in GPU memory usage, all while maintaining full accuracy. \textcolor{blue}{We have submitted the code in supplementary materials}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel optimizer state compression algorithm named \\pi-Quant, which aims to solve the high memory consumption problem caused by optimizer states (such as Adam's momentum) in large-scale AI model training. It can precisely represent a pair of real parameters (x, y) as a single rotation angle \\theta, thus halving the number of parameters. To efficiently solve for the rotation angle \\theta, the paper also designs a system of geometric equations with linear complexity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. This paper introduces a novel and elegant approach by integrating the properties of irrational numbers with Euler's formula and complex rotations to address parameter compression in deep learning, thereby providing a fresh perspective for the field.\n2. The paper's approach is grounded in rigorous mathematical theorems and lemmas. By providing detailed proofs in the appendix, the authors theoretically ensure the method's viability, thereby strengthening the credibility of the paper.\n3. Compared to quantization methods that require complex search operations (e.g., Bnb and Lpmm) , the geometric solution proposed by \\pi-Quant has O(n) linear complexity and does not require compiling additional GPU kernels, making it easier to integrate into existing deep learning frameworks.\n\nWeaknesses\n\n1. The paper's main limitation, as acknowledged by the authors, is its computational overhead. A further discussion on how this cost scales with model size would be beneficial. For instance, does this speed disadvantage become more pronounced for larger models?\n2. In Figure 2 (top), it is not clear what the colors and the horizontal and vertical axes represent.\n3. The figure number on line 292 is incorrect; it should be Figure 5. Additionally, in Figure 5, \"Adam (32, 8, 4) bits\" is represented by a single color, which seems to imply that the optimizer paths for these three bit-widths are identical. If this is the case, it should be stated. If not, they should be differentiated with different colors.\n\nQUESTIONS:\nSee 'Weaknesses' for details.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduce a method called π-Quant which leverages irrational complex rotations to enable memory-efficient training of neural networks. By compressing the real-valued parameter pair (x,y) into a single rotation angle θ using Euler’s formula, π-Quant reduce the memory from 4 bits to 3.32 bits while maintain the full model accuracy

STRENGTHS AND WEAKNESSES:
Pros
1. They propose a geometric equation for solving rotation angle θ efficiently which only require linear-time complexity.
2. They achieve 3.32 bit quantization of optimizer states while maintain the full accuracy, which is also supported by mathematical result instead of purely experimental.

Cons
1. They introduce additional restoration steps in each iteration, which can increase the training time and complexity compared to baseline optimizers.
2. The example given in Appendix E does not follow what they claimed. 
3. Even though for each transformation step, the approximation error is small, but it is unclear what would happen when all these error accumulated from repeated quant/restore cycles.

QUESTIONS:
please see weakness sections

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to quantize the parameters of a the neural network optimizer into lower-bit representation by using the properties of complex rotation function. Parameter tensors T are split into X and Y matrices, and then approximately mapped into angular rotation in complex plane via elementwise function $z(\\theta) = x + iy = e^{i\\theta}+e^{i \\bar{\\pi} \\theta} $ where $\\bar{\\pi}$ is specially chosen irrational number. The paper proposes a solution of finding the approximate $\\theta$ given x and y pair that can be controlled by the design of the $\\bar{\\pi}$ number. Then, using this quantization technique, the authors re-desing Adam optimizer to store optimizers states in in angular form and de-quantizing them just-in-time when these parameter are being used/updates. The overhead of these transformations seems to be marginal (within 10% of the actual forward/backward times) and numerical results supports that the transformation has good approximation power (allowing to store the optimizer's state with 4bit parameter precision).\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed idea is quite novel, and the practical results make the method to stand out. However, I do have some concerns regarding the statements within the theorems. \n\nIt seems that Theorem 3.1 is wrong. It was a while for me since I took any classes in geometric measure theory, but the function in equation 2 is Lipschits continues, and maps 1D interval into 2D. As such, its  Hausdorff dimension is at most 1, meaning it cannot be a space-filling curve, i.e., we cannot map every point within the circle into some \\theta. Please take a look at Theorem/Propositon 2.3 in Fractal Geometry: Mathematical Foundations and Applications by Kenneth Falconer. Again I might be wrong in my analysis, but this is a serious impediment for the acceptance of the paper if true. \n\nPractically speaking, I don't think we need the theorem 3.1 because floating point representations is anyways a discretization of the space (albeit in tiny intervals), and probably the function 2 can map any fp32 number within disc to some theta with good approximation. However, we need a precise and correct statement regarding this in the paper. I will put week reject for now, but once this issue is (hopefully) clarified, I am willing to put strong accept.\n\nQUESTIONS:\nPlease address my concerns regarding the correctness of the theorem 3.1. \n\nAlso, I was wondering whether the authors applied the efficient-parameter changes to other optimizers (i.e., SGD with momentum)? I am curious whether same practical results will hold with other optimizers (i.e., is this generic enough), or for some reason it should be paired with Adam.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes $\\Pi-Quant$, an optimizer state compression method that uses irrational complex rotations to achieve low-bit quantization for optimizer states in neural networks. The main innovation is a mathematical representation of a pair of real numbers using a single rotation angle, allowing halving the amount of optimizer parameters. The authors further propose quantizing the resulting representation angle with high precision and low error.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghs:\n1. Under the experimentation brought in the paper, which I find to be sufficient in both diversity of experiments and chosen cohort of baselines, this paper shows promising results in terms of compression potential. The proposed method manages to drop below the 4-bit parameter bit-width known from existing methods, while also introducing favorable (linear) runtime in quantization.\n\n2. The proposed method relies on elegant, mathematically-sound and geometrically interpretable foundations, which I believe make this method both clear and extnedible for future work.\n\n3. Presentation is notably clear - contributions are clearly stated and motivated. Theoretical foundations are well-structured by the appropriate lemmas and theorems.\n\nWeaknesses:\nI believe the main contributions of the paper, while solid, are mostly empirical. I find some of the theoretical analysis lacking, or at least requiring further clarification.  \nThere is a theoretical gap in theorem 3.1 - To my understanding, the arguments in appendix A only establish density, not surjectivity. The function $f(\\theta)$ is bounded and non-periodic, and thus fills a dense subset of a region in $\\mathbb{R}^2$, but not necessarily every point exactly. This means the existence of an exact $\\theta$ for an arbitrary point in the disk is not guaranteed by their argument — only that it can be approximated arbitrarily closely. Additionally, the function is not injective, so uniqueness of $\\theta$ is not sufficiently justified. Unless the authors can indicate an error in my reasoning, the implications of theorem 3.1 are overstated. \n\nFurthermore, while the notion of using irrational numbers for compressed representation is innovative and seems to work well in practice, this reliance on irrational numbers renders $\\Pi-Quant$ dependant on the representation error of irrational numbers, that cannot be fully represented in a computer. $\\bar{\\pi}$ is ultimately a rational approximation of an irrational number.\nThe non-periodicity of $f(\\theta) = e^{i\\theta} + e^{i \\bar{pi} \\theta}$ is only approximate, not absolute — the function may exhibit long-term pseudo-periodicity. The angle $\\theta$ is quantized to a finite precision controlled by a hyperparameter $\\lambda$ which determines how many digits are preserved. This could mean that means the actual values of $x+iy$ that are recoverable may deviate from their theoretical values, particularly for small $\\lambda$ or very large vectors. To my understanding, the claim that \"every parameter pair can be encoded precisely\" only holds in theory. \nWhile the implications of the points I raised may or may not be minor in practice, and while I do not believe they pose grounds for rejection, these concers should be addressed by the authors.\n\n\nLastly, the paper is missing a more-extensive discussion of limiations, which I believe is essential for publication. Another minor comment is that figure 1 is overly crowded. It does not take much room in the paper, and I propose making its contents more spacious to make it more readable.\n\nQUESTIONS:\nI would like the authors to address the concerns raised in the weaknesses section. Especially theoretical concerns and paper limiations.\nFurthermore, the authors marked \"yes\" in the statistical significance section of their paper checklist. I could not see where in the paper this is addressed, and request the authors the justify their answer.\n\nLIMITATIONS:\nAs previously mentioned, paper limiations and societal implications are not sufficiently discussed. Futhermore, the authors only marked \"yes\" or \"no\" in their paper checklist, without acutally justifying any of their answers. This makes some of the answers on the checklist difficult to accept.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper provides the technique for training time compression. The basic idea is pretty neat. Represent numbers in the complex relying on the mathematical formation that X plus Y can be represented as a complex number with one rotation angle. Thus representing two numbers with one angle. The paper has a good mathematical proof , relatively detailed, experiment analysis and evaluation. It also goes into substantial depth to show the angle can be compressed into a small number of bits.

STRENGTHS AND WEAKNESSES:
Strengths

Good mathematical proof.
Detailed training, evaluation and test accuracy on downstream tasks.
Novel idea.
Detailed treatment of the number of it’s needed to support the rotation.


Weaknesses.

The main weakness for me in this paper is somewhat poor description for why accuracy improves and perplexity decreases When the number of bits used during training is reduced. I don’t understand intuitively why you outperform fp32. This could simply be a stability  issue. It could be something in their experimental infrastructure. I found the lack of explanation confusing.

QUESTIONS:
Can you please explain intuitively and ideally with some more quantitative data why accuracy improves when you have fewer training bits and especially since you do a form of lossy compression? Basically why do you outperform fp32. Or have I misunderstood your data.

This may have been in the paper already - can you comment on into an training time and how optimized your GPU kernels are and whether the transformation into the rotations introduces computation latencies that can be addressed in future work

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-09