Title: Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets
Venue: NeurIPS 2025
Accepted: yes
Keywords: landscape analysis

modular addition; gradient dynamics; reasoning; symmetry; representation learning
Abstract: We prove rich algebraic structures of the solution space for 2-layer neural networks with quadratic activation and $L_2$ loss, trained on reasoning tasks in Abelian group (e.g., modular addition). Such a rich structure enables \emph{analytical} construction of global optimal solutions from partial solutions that only satisfy part of the loss, despite its high nonlinearity. We coin the framework as \ours{} (\emph{\underline{Co}mposing \underline{G}lobal \underline{S}olutions}). Specifically, we show that the weight space over different numbers of hidden nodes of the 2-layer network is equipped with a semi-ring algebraic structure, and the loss function to be optimized consists of \emph{sum potentials}, which are ring homomorphisms, allowing partial solutions to be composed into global ones by ring addition and multiplication. Our experiments show that around $95\%$ of the solutions obtained by gradient descent match exactly our theoretical constructions. Although the global solutions constructed only required a small number of hidden nodes, our analysis on gradient dynamics shows that overparameterization asymptotically decouples training dynamics and is beneficial. We further show that training dynamics favors simpler solutions under weight decay, and thus high-order global solutions such as perfect memorization are unfavorable. The code is open sourced\footnote{\url{https://github.com/facebookresearch/luckmatters/tree/yuandong3/ssl/real-dataset}}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the algebraic structure of the loss minimum for a two-layer network with quadratic activations trained to do modular arithmetic.  They find the weight space has the structure of a semi-ring and the loss can be decomposed in terms of ring homomorphisms.  Using this ring structure, the authors construct explicit weight values which minimize the loss and show empirically that gradient descent usually converges to these solutions, favoring lower order (fewer non-zero neurons) solutions.  The authors also prove mode connectivity implying gradient flow will favor lower order solutions under L2 regularization.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The theoretical analysis in this paper is inventive, rigorous, and insightful.  Despite the simple setting, the authors are able to explicitly describe the minimum.  \n- The algebraic structure they describe for the weight space is very interesting on its own, but it also provides insight to gradient dynamics. \n- The experiments show that GD actually converges to these minima most of the time. \n- The decomposition of the loss into SPs and the leveraging of the ring structure to create an algorithm to produce solutions is clever.  Moreover, from my (incomplete) understanding of Thm 6, this decomposition is relevant to finding the full solution via gradient descent as well since as q grows, the SPs are \"decoupled.\"   \n- I found Thm 5 interesting since it provides a theoretical backing for mode connectivity (in this simple setting) and theoretical justification for preference for simpler more generalizable solutions.  There is minimal theoretical progress in this area and this work provides an interesting and different perspective.\n- The limitations are clear and the future work is nicely described.  I particularly appreciate that point that its useful to put different widths in the same framework.  I think the idea to develop new training algorithms based on this seems like a longshot, given that we don't know how this generalizes, but I appreciate the idea.  \n\n## Weaknesses \n- The current work is limited to a fairly toy problem (modular arithmetic / abelian group composition) and a small network (2-layer) with an uncommon and simple activation \\sigma(x)=x^2.  The authors acknowledge this limitation and suggest taylor approximations can be used to generalize their work to other activations.  While I am noting this limitation and the fact that the insights here may not scale to larger networks or other problems, I still think these choices make for a reasonable trade-off.  The paper is able to reach very insightful conclusions  in this restricted setting which makes these restrictions reasonable. \n- The notation and indexing could be a big cleaner.  Overall, this is not so easy to solve since there is a lot to keep track of, but I wasn't a huge fan of using {a,b,c} as index values.   For example w_{aj} is a 2-dim tensor, but the group index is actually suppressed here and \"a\" refers to the layer but as a layer name, not an index.  The index value for layer is p.   \\mathbb{I} is not defined, but I can infer its the characteristic function.  Also, in Thm 6 J is not defined.  \n- There is some informal terminology would I appreciate if it were made more precise.  For example, L234 \"mixed\", L243 \"solution solution\".  The argmax on L190, it took me a bit to understand the arg is considered \\mathbf{u}, so maybe clarify that. L174: \"once we reach 0/1-sets\" -- Can you spell out precisely what this means?  I believe I understand L193-L194 but it could be clearer.   \n- The related works section on algebraic structures in ML is pretty short.  Are there no other instances?\n\nQUESTIONS:\n- Can you help me understand Thm 6? What is J and what does it mean for the dynamics of the SPs to be decoupled? \n- In some sense, cross-entropy loss would make more sense for the task.  I assume L2 loss was chosen to make the analysis work? \n- Is it possible to say how much of the minimum Lemma 1 describes?\n- For Fig. 2 why do the weights for z_a z_b and z_c look so similar?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to study how neural networks with one set of hidden neurons learns to perform computations on Abelian groups - with modular addition being a notable instantiation of the problem which has received recent interest in the community. The primary result of this work shows that global solutions can be obtained by composing local solutions which, in the case of neural networks, means that portions of the network's hidden layers will each perform meaningful parts of a larger computation which results in the globally optimal solution. Experiments on modular addition show that the vast majority of neural networks converge to these composable solutions rather than learning monolithic solutions, corresponding to memorisation.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n## Quality and Significance\nThe experimental results seem to support the broad points of the theory and I agree that it is important to the community that we begin to understand how neural networks solve more algorithm tasks. The fact that modular addition has been a recent task of interest to the mechanistic interpretability community speaks to this works potential significance. Additionally, the generality of the work in considering tasks on an abelian group further increases its applicability.\n\n# Weaknesses\n# Clarity\nIn general I find this work to be extremely unclear. Grammatically the text itself is difficult to read and contains long sentences which end abruptly. The abstract for example has the following sentence: \"Specifically, we show that the weight space over different numbers of hidden nodes of the 2-layer network is equipped with a semi-ring algebraic structure, and the loss function to be optimized consists of sum potentials, which are ring homomorphisms, allowing partial solutions to be composed into global ones by ring addition and multiplication\". Unfortunately this problem persists throughout the writing. Secondly, almost no effort is made to limit the jargon which is very specific to the group theory tasks being considered. For example in the introduction it says: As a result, our theoretical framework, named CoGS (i.e., Composing Global Solutions), successfully constructs two distinct types of Fourier-based global solutions of per-frequency order 4 (or “2 ×2”) and order 6 (or “2 ×3”), and a global solution of order d2 that correspond to perfect memorization\". At this point in the text the notion of \"per-frequency order\" had not been discussed at all. This is particularly problematic in this case as it is the sentence highlighting the contribution of the proposed CoGS framework. A few other examples in the early paragraphs of points I found vague (to try highlight my point and make things more actionable):\n- Lines 67 and 68: \"Existing works on expressibility (Li et al., 2024; Liu et al., 2022) gives explicit weight construction of neural networks weights (e.g., Transformers)\". I have no idea what it means to give a weights construction of neural network weights.\n- Lines 110 and 111: \"We exclude $\\phi_0 ≡ 1 $ because the constant bias term has been filtered out by the top-down gradient from the loss function\". What does it mean for the bias term to be filtered out by the gradient?\n- Figure 1: While Figure 1 is helpful to try contextualise everything (and I found myself returning to this figure often), the numbers in the caption are not present in the image.\n\nConsidering the clarity of the mathematics. Notation is not always introduce well and there a typos which make the already dense notation even more taxing. For example on Lines 109 and 110 $\\phi_{-k}$ is used to be the complex conjugate of $\\phi_k$ but the notation for complex conjugates had already been introduced. Another example is on Line 101, where $P_1^\\perp$ is introduced with little justification for its need. Also, Definition 5 seems to try to define concatenation twice but using different notation each time. I also find it quite unhelpful to refer to notation as nouns in text, especially when the notation is given no explicit intuition to help ground the variable. The worst example of this is the \"$r$ terms\" which are defined mathematically on Line 115 and then referenced as entities thereafter. This serves a particular disservice to Definition 1. Lastly, the absence of proof sketches (or even references in the main text to proves in the appendix) makes it very difficult to understand the contributions of the work - especially when coupled with the complex notation and needless definitions in some cases. For example, it is surprising that Definition 2 and 3 were deemed necessary but then the Kernel of $r$ was left as a footnote and without formal definition at all (and it seems necessary to connect the geometry of the weights to the loss, no?). Another example here is that Sum Potentials are referenced four times before Definition 6 and after Theorem 1 where it is necessary to appreciate the point of the theorem (as is stated on Lines 45 and 46).\n\n## Quality\nI have three main concerns for quality. The first two are quick to summarise: firstly I do not feel the assumptions of the theory or work are clearly stated. This could be a consequence of the other clarity issues, but I find it surprising that no assumptions seem to be mentioned (especially when prior work had such strong assumptions such as infinite width). Secondly, there is little acknowledge of the limitations of this work. The problem setup and architecture is very specific and the use of one-hot encodings is a big limitation (other works in this space allow for an initial embedding layer even). In addition, the experiments show that gradient descent follows the theoretical construction in most cases, but this is on one instantiation of a reasoning task on an Abelian group. There is no guarantee any other task would similarly follow the theory but the work seems to see the modular arithmetic example as conclusive. This leads me to my final concern, I think the claims around the theory matching experiments is too imprecise and the in some cases the text over claims how close this fit is. For example the abstract says the experiments fit the theory 95% of the time, but then Lines 246 and 247 actually make it clear that \"Although characterizing the full gradient dynamics is beyond the scope of this paper, we theoretically characterize some rough behaviors below\". This is then followed by Lines 277 to 282 which claim exact matches between the theoretical construction and empirical solutions. None of those three statements seems to be consistent to me.\n\nI hope my comments here at least serve to constructively guide the exposition of the paper. While the work itself seems promising, the paper as it stands feels rushed.\n\nQUESTIONS:\nIt is difficult to phrase my concerns as questions, but I would be happy to continue the discussion on my above points during the rebuttals phase. It would help if the authors could clarify any mistakes I may have made in my comments above, and answer any of the smaller questions mixed in there.\n\nLIMITATIONS:\nI don't feel the limitations of this work were accurately stated and I have raised this as a weakness for quality above.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces CoGS (Composing Global Solutions), a theoretical framework that identifies and characterizes an algebraic structure that exists over global solutions to group multiplication tasks with 2-layer neural networks. The authors identify a sufficient condition for global solutions, construct examples of global solutions that do (Z_F6) and do not (Z_F4/6) satisfy the sufficient conditions, and demonstrate how partial solutions can be combined to form global solutions. The work supports its theoretical framework with empirical experiments, finding that up to 95% of neural networks trained with gradient descent can be decomposed into solutions that match the provided constructions. The authors provide a theoretical statement of Occam's Razor in the CoGS setting, proving that gradient dynamics prefer simpler solutions that have lower order under the proposed algebraic structure.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The work is original. The idea of identifying algebraic structure over neural network solutions is novel, as far as I am aware. Furthermore, the work is related to an important problem in the field, understanding grokking and how NNs solve arithmetic tasks.\n- Theoretical analysis is novel and comprehensive. The authors provide sufficient conditions for global solutions, propose constructed solutions, and analyze gradient dynamics.\n- Empirical results are comprehensive and convincing. The fact that the authors identify compositions of their constructions within neural networks trained with gradient descent strongly supports their proposed framework.\n\nWeaknesses:\n- The work focuses on 2-layer neural networks with quadratic activations, L2 loss, and abelian group multiplication tasks. This setting is already interesting, but the work would be strengthened if the framework were extendable to further settings.\n\nQUESTIONS:\n- Lemma 1 provides sufficient conditions for global optimality, and the authors identify global solutions that do not satisfy them (e.g. Z_F4/6). Does the CoGS framework characterize the full set of global solutions, or are there also classes of zero-loss solutions that exist outside the algebraic framework?\n- Relatedly, were the authors able to identify what solutions the neural networks were converging to in the 2-5% of cases for which gradient descent did not converge to a composition of your constructions?\n- How generalizable are the work's results/framework to changes in activation (e.g. ReLU) or loss (e.g. cross-entropy)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoGS (Composing Global Solutions), a theoretical framework that reveals a rich algebraic structure in the weight space of 2-layer neural networks with quadratic activation trained on group multiplication task. By showing that the weights form a commutative semi-ring and that the L2 loss decomposes into sum potentials, CoGS enables global solutions to be analytically composed from partial ones. The authors construct low-order Fourier-based and perfect memorization solutions and demonstrate that approximately 95% of solutions found by gradient descent match these theoretical forms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The paper introduces a rigorous algebraic framework (CoGS) that reveals a previously unrecognized semi-ring structure in the weight space of 2-layer neural networks and interprets the L2 loss through ring homomorphisms. This perspective provides clear, sufficient conditions for identifying global optima and offers a systematic way to construct them analytically.\n- The theoretical predictions are validated by experiments showing that about 95% of gradient descent solutions exactly match the proposed constructions, with minimal reconstruction error. This high degree of empirical alignment reinforces the practical relevance of the theoretical insights.\n- The paper provides good explanations for why gradient descent with weight decay prefers low-order, structured solutions over high-order ones like perfect memorization. It links this preference to topological and algebraic connectivity between solutions of different orders.\n\n**Weaknesses**\n- The theoretical framework is developed specifically for 2-layer networks with quadratic activations trained on group operations (e.g., modular addition), assuming full knowledge of the data's algebraic structure. This limits its applicability to real-world settings where such structure is unknown or more complex. \n- While CoGS successfully constructs a family of global solutions, it does not cover all possible global optima.\n\nQUESTIONS:\nOverall, I find this to be a strong theoretical paper with a meaningful contribution to the algebraic aspects of neural network learning. However, there are a few points I would like the authors to clarify:\n- The empirical results focus primarily on synthetic group multiplication tasks. To what extent do the authors believe the key findings could generalize to real-world settings where the underlying structure is implicit or unknown? It would strengthen the work to either discuss this more explicitly or explore empirical validations beyond group-structured tasks.\n- The paper briefly touches on grokking in the conclusion. Given that this is a well-known behavior that usually occurs during training, can the authors comment more specifically on how the CoGS framework can be extended to explain this phenomenon?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}