Harmonic Loss Trains Interpretable AI Models
Authors: David D. Baek
Ziming Liu
Riya Tyagi
Max Tegmark
Venue: Accepted by TMLR ()

Abstract: In this paper, we introduce harmonic loss as an alternative supervisory signal for training neural networks and large language models (LLMs). Harmonic loss differs from standard cross-entropy loss by (a) replacing the usual SoftMax normalization with a scale-invariant HarMax function and (b) computing logits via Euclidean distance rather than a dot product. Harmonic loss enables improved interpretability and faster convergence, owing to its scale invariance and finite convergence point by design, which can be interpreted as a class center. We first validate the performance of harmonic models across algorithmic, vision, and language datasets. Through extensive experiments, we demonstrate that models trained with harmonic loss perform better than standard models by: (a) enhancing interpretability (i.e. geometry of representations), (b) requiring less data for generalization, and (c) reducing grokking. Moreover, we compare a GPT-2 model trained with harmonic loss to the standard GPT-2, illustrating that the harmonic model develops more interpretable representations. We hope our work will inspire future research exploring various methods to improve the geometry of representations, paving the way toward building more interpretable AI models.

Published in Transactions on Machine Learning Research (12/2025)
Harmonic Loss Trains Interpretable Al Models
David D. Back
dback edu
Massachusetts Institute of Technology
Ziming Liu'
amliu@mit. edu
Massachusetts Institute of Technology
Riya Tyagi
riyaby@mil. edu
Massachusetts Institute of Technology
Max Tegmark
tegmark@mi.. edu
Massachusetts Institute of Technology
Reviewed on OpenReview: https:// openrevee............e. net/ forum? 2d=2pSZ7pNoos
Abstract
In this paper, introduce harmonic loss alternative supervisory signal for training neural networks and large language models (LLMs). Harmonic loss differs from standard cross-entropy loss by (a) replacing the usual SoftMax normalization with scale-invariant HarMax function and (b) computing logits via Euclidean distance rather than a dot product. Harmonic loss enables improved interpretability and faster convergence, owing to its scale invariance and finite convergence point by design, which can be interpreted as class center. We first validate the performance of harmonic models across algorithmic, vision, and language datasets. Through extensive experiments, we demonstrate that models trained with harmonic loss perform better than standard models by: (a) enhancing interpretability (i.e. geometry of representations), (b) requiring less data for generalization, and (c) reducing grokking. Moreover, we compare a GPT-2 model trained with harmonic loss to the standard GPT-2, illustrating that the harmonic model develops more interpretable representations. We hope our work will inspire future research exploring various methods to improve the geometry of representations, paving the way toward building more interpretable AT models.
1 Introduction
As machine learning models become powerful, it has become increasingly important to thoroughly understand the behavior of neural networks. One particularly intriguing characteristic of neural networks is their ability to generalize: Empirical evidence shows that. neural networks can perform well on unseen data not explicitly encountered during training dNovak et al. 2018). This remarkable ability stems from the networks' capacity Iearn gencralizable representations and algorithms through training. However, current models face three key challenges when it comes to generalization:
(1) Lack of interpretability: Neural networks often lack interpretability, which is critical issue in high-stakes applications like healthcare, finance, and autonomous systems. While multiple rescarch efforts have advanced our insight into inner workings of LLMs (Bereska and Gavves 2024), we are still far from fully explaining their outputs. Ultimately, is crucial to design systems that are interpretable by design. Otherwise, it is challenging to diagnose errors, ensure fairness, or build trust in a model's decisions.
(2) Low data efficiency: Generalization often requires vast and diverse training data. This raises critical question: can models generalize effectively with less data? This issue is especially relevant in domains where
"Equa contribution.

---

Published in Transactions on Machine Learning Research (12/2025)
data availability is scarce, such rare discase diagnosis specialized scientific fields. Previous approaches for improving neural network generalization include efficient data sampling (Li et al. 2024a) and modifications to the training procedure to accelerate training (Wang et al.) 2024). However, these methods focus on optimizing existing training procodures rather than addressing the core issues in model design.
(3) Delayed generalization (grokking): Models sometimes experience a phenomenon known as "grokking," " (Power et al. 2022; Liu et al., 2022a) where there is a noticeable delay between the convergence of the training loss and the convergence of the test loss. This gap is problematic because: (i) it complicates determining the optimal point to stop training in order to achieve generalization, and (ii) it necessitates extended computation time and resources to continue training until grokking occurs.
A contributing factor may be how probabilities are induced from logits. In particular, Softmax's temperature- sensitive normalization interacts with scale and margin in ways that can complicate optimization and geometry. Thus, we propose harmonic loss as an alternative. Harmonic loss has two desirable mathematical propertics that enable faster convergence and improved interpretabilily: (1) scale invariance, and (2) a finite convergence point, which can be interpreted as a class center. We also present range of empirical evidences that harmonic loss can reduce grokking, match or improve data efficiency, and produce more structured representations relative to standard cross-cntropy.
Furthermore, we compare a GPT-2 model trained with harmonic loss to the standard GPT-2 and show that the harmonic model develops more interpretable representations. While its scalability to even larger models remains to be evaluated in future works, we believe that the scale-invariant formulation of the harmonic loss is crucial for enhancing representation geometry. We hope our work will inspire more future works on developing training methods that yicld interpretable representations.
The remainder of this paper is organized as follows: We first review the relevant literature in Section 2 Section 3 introduces the principles underlying harmonic loss and explains why it is preferable to cross-entropy loss in terms of generalization and interpretability. Section details a comprehensive set of experiments on algorithmic datasets, illustrating that models trained with harmonic loss have nunerous desirable properties that are absent in standard models. In Section 5 we demonstrate the performance of harmonic models on the vision task of MNIST digit classification. In Section 0 we extend our analysis to large models, illustrating that the advantages of harmonic loss also hold at scale. We present ablation experiments in Section 7 We conclude the paper in Section
2 Related Works
Representations and Mechanistic Interpretability: Numerous studies have shown that LLMs can form conceptual representations across spatial (Gurnee and Tegmark 2023), temporal [Li ct al.
2021 and
color domains (Abdou el al.] 2021). The structure of such representations includes one-dimensional concepts
(Gurnee and Tegmark] 2023 Marks and Tegmark 2023
Heinzerling and Inui 2024 Park et al.] 2024a).
well as multi-dimensional representations such lattices MMichaud et al., 2024 Li et al.) 2024b Park
et al. 2024b) and circles (Liu et al. 2022al Engels et al. 2024 While the structure of these representations
correlates with certain geometric patterns, significant unexplained variance frequently remains, necessitating efforts to improve the interpretability of neural network representations.
Previous research has shown that loss functions can influence how a model learns to represent data, affecting its abilities in unique ways (Li et al. 2024c Bosco et al.] 2024 Sudre et al. 2017 Demir et al. 2023; Salehi et al., 2017 Bommidi et al.) 2023 [eeer 2024). We refer readers to (Alshammari et al. 2025) and WWng et al. 2022) for comprehensive survey of different loss functions used in machine learning. Our harmonic loss offers an alternative supervisory signal in standard supervised learning by (a) replacing the usual SoftMax normalization with scale-invariant HarMax function and (b) computing logits via Euclidean distance rather than a dol product. While it bears resemblance to contrastive loss, which also encourages maximal separation between different classes by using Euclidean distance as a metric, contrastive learning mothods are not inherently supervised: they typically append cross-cntropy layer to generate logits, thus reintroducing SoftMax (and its drawbacks). We also show in Section 7 that using Euclidean distance alone is insufficient to fully replicate harmonic loss's capabilities. Furthermore, directly leveraging Fuclidean
Loss Functions:
2

---

Published in Transactions on Machine Learning Research (12/2025)
(a) Representation space before unembedding
(b) Toy I: two points
cross-entropy loss
harmonic loss (ours)
W
(correct label)
(1.I)
1211
(-34)
non-diverging 4
faster loss
2/
decrease!
d
(11,1)
<ie0
training steps
training steps
W3
(c) Toy 2: five points
cross-entropy loss
harmonic loss (ours)
Cross-entropy loss vs Harmonic loss
logit y; = W) 'X
logit d, = 1|w; xI|2
(0.1)
cannot pick Aonue Maae
exp(y;
1/d
(-1.0) (0.0) (1.0)
non-diverging 4
prob p; = Σ, exp(x;)
prob p; = Σ, 1/d
can pick
loss l = llog P.
loss l' log P.
(0,-1)
3M0 training stops
660
1989
N0k aide 1600
training steps
Figure 1: Cross-entropy loss versus harmonic loss (ours). (a) Definitions. Cross-entropy loss leverages the inner product as the similarity metric, whereas the harmonic loss uses Euclidean distance. (b) Toy case with two points (classes). Both the loss and l2 weight norm converge faster for the harmonic loss. (c) Toy case with five points (classes). Harmonic loss can pick out the red point in the middle. By contrast, the cross-entropy loss cannot, since the red point is not linearly separable from other points. Weight matrices are also more interpretable with harmonic loss than with cross-entropy loss.
distance-basse supervised learning has been relatively underexplored in language modeling outside of simple tasks like sentence sentiment classification (Xu et al. 2023 We present more comprehensive comparison of harmonic loss with other loss functions in Appendix D We also note that after completing this project, we became aware of a prior work that explored an idea similar to ours, which the authors term inverse distance weighting attention (McCarter) 2023), although their empirical evaluation is limited scope and does not. comprehensively examine the benefits of employing inverse distance weighting.
Our harmonic loss is closely related to the prototype learning literature, which aims to learn vectors that represent each class region in the feature space. A family of algorithms designed for such prototype-bare based classification is known as Learning Vector Quantization (LVQ) (Kohonen) 1990 Sato
Prototype Learning:
and Yamada 1995; Nova and Estévez 2014). Recently, pro.otype-baseee methods have been ex to modern deep learning settings (Sharma et al., 2023 Guerriero et al., 2018 Chen et al.) 2019 Deng et al.) 2009), showing particular promisc improving interpretability in image classification tasks. this paper, take a modest step toward exploring loss functions that are suitable for learning prototypes across wide range of domains, including algorithmie, vision, and language modeling tasks.
3 Harmonic Loss
We first review cross-entropy loss and present the harmonic loss, visualized in Figure 1 (a). Denote the unmmbedding matrix as W NxV (N is the embedding dimension, V the vocabulary size), and the penultimate representation (the representation prior to the unembedding matrix) as R.V.
Cross-entropy loss:
Logits y are defined as the matrix multiplication, i.e., y T RV (ignoring biases), y w; æ, where w; is the tth column of W. Probability p can be obtained by applying SoftMax to y, i.e.,
exp(i:)
= SoftMax(y): = Σ, exp(u))
Suppose the real class label c, then loss = -log Pe. For notational simplicity, we call linear layer combined with the cross-entropy loss cross-entropy layer.
3

---

Published in Transactions on Machine Learning Research (12/2025)
The harmonic logit d is the l2 distance between wi and a, ie., d, I|w, =2l2. We interpret w; as keys and at as query, so smaller de means a higher probability of pr. We define harmonic max (HarMar) as
Harmonic loss:
p: = HarMax(d); 1/d. Σ,
(2)
where n (harmonic exponent) a hyperparameter that controls the heavy-tailedness of the probability distribution. If the true class label is c, then loss = -log Pe. For notational simplicity, we call a layer combined with the harmonic loss the harmonic layer. Since the last step of both losses is the same (f = llog p), comparing their values is meaningful. They only differ in the ways of computing probabilities from representations
A reasonable choice of n is n ~ min( v D2),, where Dj is the embedding dimension, and D2 represents the intrinsic dimensionality of the underlying data. In LLMs, D2 could be approximated as D2 s dembed, where dembed is the embedding dimension. This approximation arises from considcring an embodding initialized from D-dimensional Gaussian distribution. The squared distance between two points, normalized by the number of dimensions D, is on the order of 1 1 O To ensure that the harmonic distance [1///)
remains constant as we scale D, we require n ~ vD, since lima- (1 + + = e. We also show the empirical impact of the exponent on the learned representations in Appendix E
Toy cases: To provide intuition about what advantages the harmonic loss has over the cross-cntropy loss, we consider two toy cases in 2D, as shown in Figure I (c). In each toy case, we train the cross-entropy layer and the harmonic layer with the Adam optimizer. Toy case 1: क = (1, (1,1) and æ2 = (-1, belong to two different classes. The harmonic layer produces a faster loss decrease, because the harmonic loss only requires de + (converging point is finite) to get p. + 1. By contrast, cross-entropy loss requires y + oo (converging point infinite) to get p; + 1. The harmonic loss already produces l2 weight norm that plateaus to a constant, while the cross-entropy loss leads to increasing 12, diverging towards infinity. Toy case 2: There are 5 points in 2D, each of which belong to a different class. In particular, the red point (0, 0) is surrounded by the other four points, i.e., cannot be linearly separated. The cross-entropy layer indeed cannot perform well on this task, manifested by high loss platcau. By contrast, the harmonic layer can drive the loss down to machine precision. Similar to case 1, the harmonic layer has a plateauing l2 while the cross-entropy layer has an ever growing 12. We also observe that the weights of the harmonic layer correspond to x, which is more interpretable than the weights of the cross-entropy layer.
From these two loy cases, we understand the advantages of harmonic loss: (1) nonlinear separability: in case 2, the red dot can be classified correctly even though it is not linearly separable. (2) fast convergence: The fact that the converging point is finite leads both to faster loss decay, and plateauing (non-diverging) lz. (3) scale invariance: Harmonic loss is scale-invariant, i.e., d: + ad, leaves Pe (hence loss) invariant, whereas Yi + az: would produce a different cross-entropy loss. (4) interpretability: the weight vectors correspond to class centers. We present the formal proof of these properties in Appendix
Benefits of harmonic loss:
Notes on interpretability: Measuring interpretability is inherently challenging in the absence of ground- truth representations. Hence, we propose two principled indicators of interpretability throughout the Sparse, low-dimensional representations enhance interpretability by concentrating semantics. We measure this via cumulative explained variance in PCA projections.
paper: (1) Compression:
(2) Geomelry: In general models, we hypothesize that parallelogram-liic units with multiple one-dimensiona semantic directions enable compositional reasoning; This enables vector arithmetic such as man woman = king queen, and supports laithful feature attribution. We measure this via parallelogram loss in Section
4 Algorithmic Experiments
A Agooithmic tasks are good benchmarks for interpretability since they are well-defined mathematically. However, training neural networks on these tasks is non-trivial due to grokking (delayed generalization) (Power
Note that when we say "cross-entropy loss." we da only refer = -log p. but rather refer to the whole pipeline including penultimate representation, logit, probability, and loss.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Seed
Seed
Seed 105
Seed
Seed 210
Seed
Seed bia
Seed
50%
Standars HUP
100%
100%
EV 100%
In i-Conneec Learthng
45%
EV: 42%
EV: 30%
46%
DV
47%
EV: 44%
DV. 45%
EV:
Add Standard MUP
EM 100%
EV
EV
5A%
EV 100%
EV 100%
100%
EM 100%
100%
Modular Addi Harmonic
Equlvalence classes Standard
EV: 64%
IV. 92%
EV. 0P%
IV: 13%
DV 50%
EV: 63%
EV. 91%
DV 07%
EV: 66%
CV. 79%
Equivalance Harmonic Mu"
Genwalogy Learring
43%
EV. 50%
EW: 95%
BV. SAYs
EV. 53%
EV: 54%
EV. 35%
52%
EV. SA%
EV. 47%
Harmonis MUP
39% EV.
32%
35%
EV: 35%
41%
EV
Fermutatton Groupe
34%
33%
28%
61%
Harmanic MUP
Figure Visualization of the top two principal components of the embeddings synthetic experiments. The title of each subplot shows the explained variance by the first two principal components. Each row corresponds to pair of a dataset and model, while each column represents the embeddings from different training runs with varying seeds. Groups of consecutive two rows belong to the same dataset, with models arranged in the order: {Standard MLP, Harmonic MLPJ. The datasets are ordered as follows: {In-Context Learning, Genealogy Learning, Equivalence Classes, Modular Addition, and Permutation Groups). X and Y axis spans are equal.
et al. 2022 and the existence of multiple algorithms (Zhong al. 2024), etc. We will show that harmonic models learn better representations, are more data-efficient, and exhibit less grokking.
4.1 Models and Datasets
Models:
We compare four models:
1. Standard MLP: Tokens are embedded into 16-dimensional embeddings, which are then concatenated and used as the input. The model consists of two hidden layers with widths of 100 and 16, respectively. The SiLU activation function used.
2. Standard Transformer: Tokens are embedded into 16-dimensional embedding, with learnable positional embedding added. The input passes through two transformer decoder layers, each comprising two attention heads and an MLP with a hidden dimension of 64.
3. Harmonic MLP: Standard MLP with harmonic unembedding layer of exponent

---

Published in Transactions on Machine Learning Research (12/2025)
4. Harmonic Transformer: Standard Transformer with an harmonic unembedding layer of exponent
=
We trained the MLP models for 7000 epochs and the transformers for 10000 epochs. For all four models, we used the AdamW optimizer with a learning rate of 2 x 10 10-3, weight decay of 10-2, and an Lg regularization on the embeddings with strength 0.01.
Datasets: We trained the four models above using the lollowing five datasets, and analyzed their performance as well as the resulting representations:
In- Context Learning: In a 5x5 integer lattice, given three points on the lattice, the model is trained to predict the fourth point that would form parallclogram with the others. This task exemplifies in-context reasoning in LLMs, mirroring the classic man:woman:kiig requiring the model to complete the relational pattern such as 'man is to woman as king is to queen' based on the given context.
analogy by
2. Modular Addition: Given two integers v, the model is trained to predict (ær + v) mod 31.
3. Equivalence Classes: Given two integers 0 S I, 40, the model is trained to predict if
= mod 5.
4. Genealogy Learning: In complete binary tree with 127 nodes, given subject and relation, the model is trained to predict the corresponding object. The relation can be one of the following: parent, grandparent, or sibling.
5. Permutation Composition: Given two permutations and in S4, the model is trained to predict On this dataset, we trained standard and harmonic transformers with an L2 regularization of 0.005, as we found this configuration lod to more complete training.
4.2 Representation Faithfulness
Figure 2 shows the plot of the top two principal components of the models' embeddings for MLP tasks. We show the complete embedding visualization for all tasks in Appendix A Overall, harmonic loss representations are cleaner and more organized than their cross-entropy counterparts. We found near-perfect circle representations for the modular addition task, clear tower-like structure for tree learning, and neat clusters for permutation composition. We examine the representations task by task:
1. In-context Learning: Standard models' representations are either imperfect lattices or exhibit unexplained variance in higher dimensions, whereas harmonic models almost always perfectly (100%) recover the underlying 2D lattice structure across different random seeds.
2. Modular Addition: Harmonic MLPs consistently recover a perfect 2D circular representation in almost all runs, whereas tstandard MLPs often fail to do Harmonic transformer has a similar success rate to the standard transformer in constructing circles, but the explained variance captured by the first two principal components is generally much higher, indicating that harmonic models discover more compact representations with fewer uninterpretable components,
3. Equivalence Classes: While both standard and harmonic models are able to identify the underlying groups, standard models' representation tends to be more "clongated', or not completely grouped, compared to its harmonic counterpart. This could be attributed to the fact that cross-entropy loss does not have an incentive to reduce irrelevant variations to zero.
4. Genealogy Learning: Only Harmonic MIP recovers the underlying tree representation.
5. Permutation Composition: Harmonic MLP generally produces better-separated clusters. A particularly clean representation that appears multiple times contains 6 clusters of 4 permutations, where each cluster is a coset of the subgroup (e, (3)) (13) (14)(23)) or one of conjugates.
the harmonic transformer, permutations commonly organize into clusters that are cosets of (e, (13), (14), (34), (134), (143)) or one of its conjugates, subgroups isomorphic to S3 (one element, in this case 2, never permutes).

---

Published in Transactions on Machine Learning Research (12/2025)
Standand MLP
Harmaric MUP
Slandard Vaassformrr
Harmanic Tranffomerr
(a)
In- Dn-Centext Learning
Modular Addition
Equivalence Classes
Genealegy Learning
Permutation Groups
0-8
0.4
Principal Companen: Ind ox
(b)
In-Centext Learinn
Modular Addition
Equivalence Classes
Genealogy Learning
Permutation Groups
Ast-Acursyy
0.2
0.4
0.8
0.0
0.4
0.8 0.0
0.4
0.8 0.0
0.4
0.8
0.0
0.4
0.8
Thain Fraction
Harmonic
Wrarsfomer
Harmenic Transformer
(c)
In-Context
Addition Equivalence
Genealogy Learning Permutation
3000
Epocts Bad Acc D.
2000
2000
Figure (a) Cumulative explained variance vs. principal components (mean over 20 soeds). Harmonic representations are more compact than standard counterparts. (b) Test accuracy function of Train Fraction (fixed seed). Harmonic models generalize faster with less data than standard counterparts. (c) Epochs to Test Ace 0.9 vs. Epochs to Train Ace >> 0.9 for 20 consecutive epochs, for 20 different random seeds. y = x line represents no grokking, where train and test accuracy improve simultancously. Points closer to the y-axis indicate greater degree of grokking.
Figure 8 a) further demonstrates that harmonic representations tend to be more compact than standard models, with fewer uninterpretable components. In particular, harmonic models trained for in-context learning achieve 100% explained variance using only the first two principal components.
4.3 Data Efficiency in Training
Figure bb) shows the test accuracy as a function of train data fraction for our synthetic experiments, indicating how much data is necessary in order for the model to be generalizable. We observe that harmonic models require comparable or much less amount of data to generalize, compared to their cross-entropy counterparts. Such improvement especially notable for in-context learning, where harmonic models generalize nearly immediately.
4.4 Reduced Grokking
Grokking refers to the phenomenon of delayed gemeralization (Power el. al. 2022 for example, it lakes 10 steps to reach perfect accuracy on the training data, but it takes 10a steps to generalize to the test data. Grokking pathological phenomenon that we want to avoid (Liu al. (022b). We find that harmonic loss overall reduces grokking, seen in Figure 3(... Points on the y === line represent models which trained without grokking, with train and test accuracy improving together. This improvement is particularly evident.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Standard
Standard weight d
Harmonic weight decay=0
MNIST: Standard Weight Visualization
weight decay=0
1.0
Accuracy
train
train test
train test
0.0
test
1000
1000 Epach
Epoch
Epoch
MNIST: Harmonic Weight Visualization
25
26%
58%
100%
-2
PCD
PCD
PCO
Figure 4: Left: Case study on modular addition. Standard MLP trained for modular addition without weight decay often fails to generalize. Generalization only achieved with the addition of strong weight decay; however, (a) significant grokking occurs, and (b) while the first two principal components form an approximate circle, they explain far less than the total variance. In contrast, the harmonic model trained for modular addition gencralizes quickly without grokking. Morcover, the embedding forms perfoct 2D circle. EV in the plot represents the explained variance by the first two principal components of the embedding, Right: Visualization of model weights trained for MNIST. Yellow cells show values less than 0.01. Both models achieved xs 92.5% test accuracy.
in learning modular addition and permutation composition: while the standard MLP exhibits severe grokking, most data points for the harmonic MLP lie much closer to the y = æ line.
4.5 Case Study: Modular Addition
study and analyze why the harmonic MLP encourages more interprelable representations and better generalization compared to the standard MLP. The standard MLP trained for modular addition without weight decay often fails to generalize, as shown in Figure g Generalization is only achieved with the addition of strong weight docay; however, (a) significant grokking occurs, as depicted in Figure and (b) while the first two principal components form an approximate circle, they explain far less than the total variance, leaving significant unexplained variance. In contrast, the harmonic model trained for modular addition generalizes quickly without grokking. Furthermore, the embedding forms perfect circle, as shown in Figure 4
In this section, study modular addition
The better formation of circle and improved generalization in harmonic MLP can be attributed to the properties of harmonic loss, as explained in Section 3 To drive the probability to 1, the standard cross-entropy loss requires driving the representation to infinity -i.., making the logit infinite. In contrast, harmonic loss achieves this by driving the harmonic logit to zero, which easily accomplished by learning w; = æ Equation 2 The existence of such finite converging point may be responsible for the (a) faster convergence, (b) better generalization, and (c) circular geometry observed in embeddings.
5 MNIST Experiments
For vision tasks, convolutional neural networks are shown to be (at least somewhat) interpretable by demonstrating "edge detoctors", "wheel detoctors", etc. (Olah et al. 2020).. In this section, we show that the harmonic loss can lead to more interpretable network for the MNIST dataset when it comes to training fully connected networks. As proof of concept, compare one-layer neural networks trained using cross-entropy
8

---

Published in Transactions on Machine Learning Research (12/2025)
train loss
val loss
accumulated fractions (5 PC)
asandard, GPT H.GPT
capibalie
english-gemen
Jength
country-cppitl
0.52
0.2-
45-
5:5
6.80
4.-
4.2
2,00
synonym
french
prra
6.75
0.50
10-
0.35
9953
2003 240
staps
steps
paraleigtom peralistogiam
peraliilggian
paraineoggaa
saralliigran
present-past
rank=D
rank=1
rank=2
rank=10
rank=100
rank=200
rank=300
rank=500
standard
harmonic
piged Tigiond
ded
Figure 5: GPT2 experiments: (Top left) loss curves. Harmonic GPT achieves a slightly lower loss compared to standard GPT. (Top right) cumulative distribution function with respect to parallelogram loss, for twelve function-vector tasks. Harmonic GPT consistently shows lower parallelogram losses (i.e., better parallelograms). (Bottom) Parallelograms (Ist and 2nd principal component) with quality ranked in descending order from left to right. Harmonic GPT tends to produce parallelograms that are more 'rectangular', while standard GPT produces flat. 'parallelograms.
loss and harmonic loss. The input images are first flattened and passed through a 784 x 10 linear layer obtain the logits. The models were trained with batch size of 64, learning rate of 0.001, and for 10 epochs, achieving 92.50% test accuracy for cross-entropy loss and 92.49% test accuracy for harmonic loss.
Figure shows that the harmonic model's weights are more interpretable than those of the standard model. Consistent with its core principle, the harmonic model's 's weights almost perfectly align with class centers (images of each number). They also assign near-zero values to peripheral pixels, unlike the model trained with cross-entropy loss, which lacks an incentive to reduce irrelevant background weights to exactly zero.
6 GPT2 Experiments
Many mechanistic interpretability works have been dedicated to understanding large language models. For example, probing and attribution methods are good post hoc analysis tools. Despite their (partial) success, these tools are not creating interpretable models in the first place but are trying to find needles in the haystack. more interpretable. By using
We argue that would be nicer if we could pre-train the language models harmonic loss in training, we can produce language model that can "grow' crystal-like representations, while having comparable performance with standard one (trained with the cross-entropy loss).
We pre-train GPT-2 small model (128M, based on NanoGPT) OpenWeb The embedding matrix and the unembedding matrix are tied (share the same weights). We use 8 V100 GPUs, choose block size 1024, batch size 480 blocks. We use the Adam Optimizer with 3 = 0.9, 82 0.95. For the harmonic loss, we choose n = /768- xe 28, following the discussion on harmonic exponent in Section 3 For standard (harmonic) GPT, we use a lincar warmup learning rate schodule for 2k (1k) steps to maximum learning rate 10-4 (6 10-3), and a cosine decay schedule from 2k to 10k, ending at Ir 3 x 10-5 (3 x 10-4). As shown in Figure top left, Harmonic GPT shows faster converging initially (partially due to larger learning rates), and

---

Published in Transactions on Machine Learning Research (12/2025)
Seed D 120
Seed
Seed 1.5
Seed 157 200
Seed
Seed
Seed BCs
Seed 157
100
32.4%
hamono
47
Ju m0
Fvr no
Pv. 000.0
Fv top me
IV ton
Fo 16 13
Fv. 59.24
TV M1
TV. 75
12 wwl.mua
40
23)
100.0%
ditgroo harmaak
+2
92.0%
Ey 11.8
Ev. 90.0%
as s0 20
Ed 48 os
Ev 45
Ev. 43 5%
Figure 6: Learned embeddings on the lattice and modular addition tasks. Each pane shows the 5x5 class embeddings after training (numbers denote class Dss). Columns vary random seeds; the four left columns are for in-context learning, and the four right columns are for modular addition task. Rows correspond to loss functions: (top) full harmonic loss (E2 logits HarMax), (2nd) &g logits + SoftMax, (3rd) dot-product logits + HarMax, (bottom) standard cross-entropy layer. Here, HarMax suecessfully recovers both the lattice and circular structure.
that only E2 distance paired with
converges to similar performance in the end (at 10k steps). The final validation losses are 3.159 (standard) and 3.146 (harmonic). From training loss curves, harmonic GPT also seems to have smaller fluctuations. This suggests the effectiveness of the harmonic loss on real-world models.
To testify the interpretability of the learned embeddings, we take twelve function-vector tasks from [Todd ct al.] 2023). Each datasot contains many input-output pairs that have a certain relation. For example, the "present- pressent-aat'' dataset contains pairs like: jump-jumped, fastene, win-won, etc. To construct parallelograms, we can draw two different pairs from the dataset, obtaining quadruples like (jump, jumped, fasten, fastened) which are expected to form parallelograms. Each word is tokenized into tokens; if multiple tokens are obtained, we use the last token. We project token embeddings onto the first two principal components. The quadruple (i,j. m, n) has 2D PC embeddings (E,, E.,, Em, E,); we define the parallelogram loss lpara to be
para E, + E. - E, Emll/o.
33)
a scale factor that normalizes the loss (Es + a leaves (para invariant). We obtain 10000 quadruples, measuring the parallelogram qualities by computing their parallelogram losses. We plot their cumulative distribution function in Figure 5 in the top right: for every task, the harmonic GPT produces lower parallelogram loss (better parallelograms) than standard GPT. We show the parallelograms obtained in the present-past task Figure bot. The parallelograms are ranked with quality in descending order from left to right. The harmonic GPT tends to produce visually appealing parallelograms that are more 'rectangular', while standard GPT produces flat 'parallelograms' Discussion about internal representations is included in Appendix C
where o a= EK E&
7 Ablation Experiments
Harmonic loss makes two major modifications to the standard cross-cntropy loss: (i) compute logits via l distances, and (i) use HarMax function as shown in Eq. equation 2 To tease apart their individual contributions, we perform set of targeted ablations in which one component is replaced at time while the
10

---

Published in Transactions on Machine Learning Research (12/2025)
remainder of the training pipeline is left unchanged. Specifically, we train MLP models on the in- learning and modular addition tasks with the ablated loss functions.
Results are shown in Figure G In in-context learning tasks, we observe that including either HarMax or & logits alone is sufficient to replicate the full performance of Harmonic Loss. However, for modular addition tasks, both HarMax and & logits are essential achieve the full performance. While incorporating only one component enhances the quality of the circular representation, the explained variance remains significantly below 100%. Overall, both HarMax and lg logits play critical roles in improving representations.
8 Conclusions
In this paper; we int roduced harmonic loss an alternative the standard cross-entropy loss for training neural networks and large language models (LLMs). Across the evaluated tasks, found that models trained with harmonic loss perform better than standard models by: (a) reducing grokking, (b) requiring less data for generalization, and (c) improving interpretability. We also compared a GPT-2 model trained with harmonic loss to the standard GPT-2, illustrating that the harmonic loss-trained model develops more interpretable representations. Further study is needed to explore the scalability and applicability of our findings to even larger models.
Limitations: As discussed carlier in the manuscript, measuring interpetabilityy is inherently challenging in the absence of ground-truth representations. For example, although harmonic loss slightly outperforms cross- entropy loss the ImageNet benchmark (see Appendix G). it is difficult to define ground-truth representations of images, beyond the qualitative observation that semantically similar images tend to cluster in the representation space. This makes it hard to quantify progress in interpretability. Nevertheless, we conjecture that the scale-invariant formulation improves the parallelogram-like geometry of model representations as demonstrated in this paper, which may facilitate identifying semantic feature directions. Further investigation is required to refine our understanding of the extent which representation geometry affects interpreta billty. Lastly, note that harmonic loss might not be optimal in all instances. For instance, expect cross-entropy loss to be better for tasks requiring linear separability, since they are more suitable for finding the linear separation boundary compared to distance-bace losses.
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study, arXiv preprint arXiv: arXim:1802.08700 2018.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for safety-a review. arXiv preprint ar arXiv:2404.1488,, 2024.
Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Connor Holmes, Cheng Li, and Yuxiong He. Deepspeed data efficiency: Improving deep learning model quality and training efficiency via efficient data sampling and routing. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 38, pages 18490-18498, 2024a.
Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, Mingyuan Zhou, et al. Patch diffusion: Faster and more data-efficient training of diffusion models. Advances in neural information processing systems, 36, 2024.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXie preprint arXiv:2201.0217 2022.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651 34663, 2022a.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint ar trXi:2310002200 2023.
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint ftt:2106.0073" 2021.
Mostafa Abdou, Artur Kulmizev, Daniel Hersheovich, Stella Frank, Ellie Pavlick, and Anders Segaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprint 0rXiv:2109.06129 2021.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent lincar structure large language model representations of true/ true/false false datasets. arXie preprinl arXiv:2310.06824 2023.
Benjamin Heinzerling and Kentaro Inui. Monotonic representation of numeric properties in language models. arXiv preprint 20X 2024.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arxiv preprint ar (xxx:2406.015000 2024a.
Eric J Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability. arXiv preprint 002.::240..05110 2024.
Yuxiao Li, Eric J Michand, David D Back, Joshua Engels, Xiaoqing Sun, and Max Togmark. The goometry of concepts: Sparse auloencoder feature structure. arXiv preprint ar arxiv:24 10..:2/10.197100 2024b.
Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Mar- tin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. arXiv preprint arXiv:2501.0007 2024b.
Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. Xiv preprint 10xi:22055.1886 2024.
Xue Li, Qi-Liang Sun, Yanfci Zhang, Jian Sha, and Man Zhang. Enhancing hydrological extremes prediction accuracy: Integrating diverse loss functions in transformer models. Environmental Modelling & Software, 177:106042, 2024c.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Edoardo Bosco, Giovanni Magenes, and Giulia Matrone. Echocardhographic image segmentation with vision transformers: A comparative analysis of different loss functions. Tn 2024 IEEE International Symposium on Medical Measurements and Applications (MeMeA), pages 1 6. LEEE, 2024.
Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Mullimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 9, pages 240-248. Springer, 2017.
Andac Demir, Elie Massaad, and Bulent Kiziltan. Topology-aware focal loss for 3d image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 580 589, 2023.
Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3d fully convolutional deep networks. Tn International workshop on machine learning in medical imaging, pages 379 387. Springer, 2017.
Bala Saíbabu Bommidi, Kiran Teeparthi, and Vishalteja Kosana. Hybrid wind speed forecasting using icoemdan and transformer model with novel loss function. Energy, 265:126383, 2023.
Pedro Seber. Predicting o-glcnacylation sites in mammalian proteins with transformers and rmns trained with a new loss function. arXiv preprint arXi::2402. 171911 2024.
Shaden Alshammari, John Hershey, Axel Feldmann, William T Freeman, and Mark Hamilton. I-con: A unifying framework for representation learning, arXiv preprinl ar Xrv:22500 16929, 2025.
Qi Wang, Yue Ma, Kun Zhao, and Yingjic Tian. A comprehensive survey of loss functions in machine learning. Annals of Data Science, 9(2):187-2)) 212, 2022.
Lingling Xu, Haoran Xie, Zongxi Li, Fu Lee Wang, Weiming Wang, and Qing Li. Contrastive learning models for sentence representations. ACM Transactions on Intelligent Systems and Technology, 14(4):1-34 34, 2023.
Calvin McCarter. Inverse distance weighting attention. orXiv preprint ar Xiv:23300 18805, 2023.
Teuvo Kohonen. Improved versions of learning vector quantization. In 1990 ijenn international joint conference on Neural networks, pages 545 550. LEEE, 1990.
Atsushi Sato and Keiji Yamada. Generalized learning vector quantization. Advances in neural information processing systems, 8, 1995.
David Nova and Pablo A Estévez. A review of learning vector quantization classifiers. Newral Computing and Applications, 25(3):511-524, 2014.
Saurabh Sharma, Yongqin Xian, Ning Yu, and Ambuj Singh. Learning prototype classifiers for long-tailed recognition. arXiv preprinl or orXi:22022000.000 2023.
Samantha Guerriero, Barbara Caputo, and Thomas Mensink. Deepnem: Deep nearest class mean classifiers. 2018.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recogritton, pages 248 255. leee, 2009.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural works Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint trXi::2210.01117, 2022b.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024-00) 2020.
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint 1rXi::2310.52213, 2023.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. arXiv preprint "'Xiu:107.09561, 2017.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311003666 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot,
Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances neural informalion processing systems, 1118661-18673) 2020.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint 1015: 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631 1642, 2013.
14

---

Published in Transactions on Machine Learning Research (12/2025)
A
Full Representation Visualization
Figure 7
shows the visualization of representations for all models and datascts.
SOL
H
Figure Visualization of the top two principal components of the embeddings in synthetic experiments. The title of each subplot shows the explained variance by the first two principal components. Each row corresponds to a pair of a dataset and a model, while each column represents the embeddings from different training runs with varying seeds. Groups of four rows belong to the same datasct, with models arranged in the order: {Standard MLP, Harmonic MLP, Standard Transformer, Harmonic Transformer). The datasets are ordered as follows: (Tn-Context Learning, Genealogy Learning, Equivalence Classes, Modular Addition, and Permutation Groups)
15

---

Published in Transactions on Machine Learning Research (12/2025)
parallelogram loss (2 PC),
Output of Block 6
antonym
jev_item
capitalize
english-german
word_length
country-capital
1500
standard
harmonic
count
1000
500
synonym
next item
english-french
singular-plural
english-spanish
present- pastt
1000
750
count
500
250
10-1
100
10-1
100
10-1
10"
10-1
10
10-1
100
10-1
100
parallelogram loss
parallelogram loss
parallelogram loss
parallelogram loss
parallelogram loss
parallelogram loss
Figure 8: Harmonic loss (harmonic) and cross-entropy loss (standard) induce qualitatively different representa- tions in the intermediate layer of GPT2. We show the distribution of parallelogram loss for the parallelogram dataset. Harmonic loss has more perfect parallelograms (spike close to zero loss) but demonstrates heavier tail.
B Identifying Coset Structure in Permutation Representations
To explore the coset structure in permutation representations of S4, we began by enumerating its subgroups. Using this enumeration, we computed all possible left and right cosets of each subgroup in S4, yielding 28 distinct left cosets and 28 distinct right cosets.
Among these cosets, two pairs are equivalent, since we consider two of the four normal subgroups of S4: the alternating group A4 and the Klein-4 group. To focus on meaningful structures, the trivial subgroup and the entire group were excluded from further analysis.
The coset partitions were then compared using the silhouette score, metric for evaluating the quality of clustering. This comparison helped identify the partition with the most structured coset organization, which is likely the structure that the model has captured during training. We then color the representation according to the best-clustered partition, with each coset being a different color.
C Analyzing GPT2 hidden representations
In Section we have shown that GPT2 trained with the harmonic loss has nicer structures in its embeddings (i.e., parallelograms) than that trained with the standard cross-entropy loss. We now show that intermediate representations (output Block 6) induced by the harmonic loss are also qualitatively different from those of the cross-entropy loss. In Figure g the harmonic loss produces more perfect parallelograms (spike around zero parallelogram loss) but also displays a heavier tail for the parallelogram loss. The heavy tail is due to the heavy-tailedness of the harmonic loss (power law), as opposed to the cross-entropy loss (exponential). It remains to be understood if such heavy-tailedness is feature or bug for the harmonic loss, but the more perfect parallelograms are probably a good thing, or this al least suggests that imposing the harmonic loss at the end of the network can have noticeable influences the intermediate representations. In Figure g we also notice that for the Captalize datasct, the lowercase and uppercase words tend to overlap in the first. two PCs with the harmonic loss, but not with the cross-entropy loss. This again suggests the qualitative difference between the harmonic loss and the cross-entropy loss.
16

---

Published in Transactions on Machine Learning Research (12/2025)
capitalize
rank
rank=1
rank=2
rank=10
rank=100
rank 200
rank 300
rank
standard
coget
000
Numggrin
harmonic
Figure 9: Visualization of layer 6 representations projected onto the first two principal components, for the capitalize dataset. The harmonic loss (bottom) tends to collapse corresponding lower-case and upper-case words, while the cross-cntropy loss (top) places them at different locations.
D Comparison of Harmonic Loss to Alternative Loss Functions
We briefly contrast the harmonic layer (lg logits + HarMax) with three popular loss families. Throughout, lot be an example embodding, Wy the weight of the correct class y, and w; those of incorrect classes.
(a) Contrastive
InfoNCE.
A generic form is
exp ((xxx+))//
Lconte =- -lo exp exp(s(x,xx )/T) E, s(x,s(xxxxf//))
It enforces only relative ordering x sxx, + m, entire constellations drift or rotate. In contrast, harmonic loss pulls every example directly toward a fixed class anchor wy and repels it from all others, yielding a stable, globally referenced geometry.
(b) Margin-based SoftMax. marge-margin variants add a fixed gap A to every class boundary, s(x, wy) s(x, w:) + A. Because A is global, semantically close classes (e.g. dog cat) forced far apart unrelated ones (dog vs. airplane). Harmonic loss adapts separation dynamically: x ||x w.ll-", so related concepts converge while unrelated ones diverge, yiclding meaningful hicrarchies (c.g. the FAMIL-TRREE task).
(c) Spherical / cosine losses. These constrain embeddings the unit hypersphere and optimise angular margins: Caph - log Σ. con While scale-invariant in angular space, they ignore absolute Euclidean proximity; our tasks (lattice, modular-add) benefit from the latter, explaining the poorer alignment of spherical loss.
We also run some experiments contrasting harmonic loss with loss (a) contrastive loss and (c) spherical loss for the in-context learning and modular addition tasks. Results for MLP and Transformer models are in Figure 10 and Figure respectively.
E Sweeping HarMax Exponent Value
We perform experiments sweeping the HarMax exponent value for the in-contexi learning and modular addition tasks. Results are displayed in Figure 12 and Figure [3] We note that varying n has minor impacts on lattice quality, with the default choice n=1 having the highest explained variances. Based on the modular addition task, our overall takeaway is that MLPs prefer the default n=1, while explained variance and circular structure for Transformer representations may improve with slightly larger exponent.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Sead 105 DV. 90. 7s
Seed CV. 98,8%
Seed
Secd
Seed 105
Sead 157
Evi eg 2
De 90 4
Ey $4.3%
Dv 92 3%
ty 92 6%
wly wis
bb M4
EV. 99..%%
EMI 00.59
EV. 09.01%
EV. 20.895
BV. 90.7%
Ev. 00.7%
EV.99 756
EV. 99.7%
1000 U.A. 4.R82C
spherccal
100 ms
100
10a
23p
wl
d7 wy 22
harmanic
do
#B
de
20 45
Figure 10: Results for MLP models. Rows show harmonic, D l2 +SoftMax, and standard losses (top to bottom). Harmonic loss achieves the best reconstruction across sceds.
Seed Ev 9%
33.9%
Bv: 32,2%
EV. 41.3%
Evi 44.25%
EM 44.4%
Ey: 38.3%
Ev. 41.3%
contractive
EV. 99 4%
the un ons
FV. 90.7%
EV. 98.4%
290 34, on,, soold
Ev 0s 0%
tv 2% Mee
29. 3%
spherccll
Ey: 08.5%
BV 00.0%
Ev. 09.015
EV: 00
BA 05.00
Ev. 30.3%
EV. 55 256
Ev. 65.0%
2) de
Figure 11: Results for Transformer models. Same ordering Fig. 10
F Properties of Harmonic Loss: Proofs
Theorem (Finite Convergence of Harmonic Loss). Consider a classification model with K classes and weight vectors w,y..., WK Rdd (no bias). Let 1 be the training set, with yi {l,..., K}. The cross-entropy loss is given by
n
exp(Wy: I44)
LCE(W) = Σ In K
1 exp(w1 x4)
The harmonic loss (with exponent 3 > 0) is given by
||, Wy, 11
Ln(W) ΣØ=1
| w. -B
If the training data is linearly seporable (i.e. there exists W such that for all i, Wy. 24 wy 24 for # y1),
then:
18

---

Published in Transactions on Machine Learning Research (12/2025)
390032
Figure 12: Effect of the harmonic exponent n on lattice in-context learning. We sweep n E {1...., 103. Columns 1-4: Carmonic-MLPP columns 5-8: Harmonic Transformer. The learned x lattice remarkably stable; n=1 already provides crisp and interpretable geometry.
Lcr(W) has no finite minimum. In fact, for any weight matrix W that classifies all training points correctly, one can decrease LCE further by scaling to larger norm. Thus the infomum of LCE is but it is approached only as [[W]| + oc.
Ln(W) allains a (global) ménimoum some fivile W. Once the weights are large enough classify all training points correctly (t.e. -| wp, min || - for all i), increasing the norm of W does not reduce Lu- In particular, Lu is scale-invariants scaling all wa and all x, by common factor leaves the loss unchanged. Consequently, Lu has finite global minimizer.
Proof. For the cross-entropy loss LCE, suppose W classifies all training examples correctly. Then for each Wy. F; > maxjæy. Wy- Pz. Consider scaling W by factor > 1: replace each wk with twx. Then Wy. and wi- are both multiplied by I. The SoftMax probability of the true class yi becomes
exp(wyy x22)
Pw (yilax)) E, exp(w, 2z))
Under scaling tW, this becomes
exp(t Wyy Iz) exp(t xz)
Paw(y:lus)
19

---

Published in Transactions on Machine Learning Research (12/2025)
20
Figure 13: Effect of the harmonic exponent n on modular addition. Columns 4: Harmonic MLP, columns 5-8: Harmonic Transformer. MLPs remain stable across seeds, whereas Transformers are more sensitive yet form tighter circles at higher n; n=1 works well for MLPs, while larger n may benefit Transformers.
Since Wy, -, the largest logit for sample i, This holds for all so LCe(tW) approached only in the limit ||W]| ->
we have Paw (ys (z,)) + and thus In (aw(y, 0. 0 + cc. Therefore, no finite W minimizes LOE; the infimum 0 is
For Lu, once W is such that each training point is correctly classified by its nearest prototype (i.e. ||a;- < ||æ; wyll for all j # v.), increasing the norms ||w || further will not improve the loss. In fact, if every 24 is closer to its correct Wy, than to any other wy, then the harmonic probabilitics
|| Wg. 11
Σ ||: wyl1-8
remain unchanged under a uniform scaling: if we replace x; by Cr; and Wik by cuk, then ||er; cuk || c||æ;-wwll, so the scaling factors cancel. Therefore, once correct classification is achieved, no further reduction in loss is obtained by increasing [[W]], and Lu achieves its minimum at finite W.
Theorem 2 (PAC-Bayesian Generalization Bound of Harmonic Loss). Assume all training examples lie within a ball. of radius R in inpul space, i.e. || S R for all Suppose weight matrix W achieves a distance margin of x > on the training set, meaning that for every training sample (24, vi) and ony other class j + yr.
| Wy, + ||are wyll-
20

---

Published in Transactions on Machine Learning Research (12/2025)
Then, with probability at least 1 8, the generalization (test) error of the harmonic classifier satisfies
R WII
In(1/5)
(nw S O (z.g)=D
Pr
YV
where hw(2) denotes the predicted class and n is the rumber of training samples.
particular, [WI| is finile for harmonic loss (by Theorem 1 and lypically much smaller than the weight norm of the solution obtained with rros--enropyy loss. Thus, the hormonic classifier has tighter generulization bound.
Proof. Applying the standard PAC-Bayes margin bounds (see e.g. with probability at least 1 5,
Neyshabur al. 2017 one obtains that
RIW | TVn
[m(1/5) n
Pr(hw(() 4) 0o
Since the harmonic loss yields a solution with finite |[W]|, the bound is finite. In contrast, the cross- entropy solution would have |[]] + oo even when achicving zero training error, rendering a similar bound meaningless.
Thcorem (Interprctable Representations of Harmonic Loss). At critical point (in particular, global minimumm) of the harmonic loss, each weighl vector wy becomes on interprelable class cender for class k. Specifically, the stationarity condition implies
Σ kryckk
with a; 2 O, Σ a
Wk =
aq 24
1:y==
d.e. Wh is a convex combination of the training examples of class k. Consequently, we represents the center point of its closs, leading to more interpretable representations compared to cross-entropy loss.
Proof. Differentiate the harmonic loss with respect to Wk. For simplicity, denote
Ir. wkll 8
ΣØ= 10, wgl
For samples 24 with yi k, the derivative takes the form
ŌLh
B
Σ =
wx (wk x.) v. + terms from with yi o k.
Owk
At critical point, the total derivative vanishes. Rearranging the stationarity conditions (and noting that the repulsive forces from other classes tend to balance out overall on average due to long distance) yields
Eny || I2 + E 1: # we
Exy | Σ Een #f I|ay 1I
Since wi is closer to class-k examples than to others, the weights sum. Define
|| for i with y: = dominate the
It wa !
0;
Σ
Zigi=
Σ
Ejughh
Then wi can be written as a convex combination
Σ
Σ 7k
aj
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table Validation accuracy on ImageNet using different loss functions.
Loss
Top-1
Val Acc
Top-5 Val Acc
Cross-Entron (Our Impl.)
74.17%
91.88%
Harmonic Loss (Our Impl.)
75.08%
92.12%
Cross-Entropy (Khosla et al.
2020
77.6%
95.3%
Supervised Contrastive Loss
Khosla ct al. 2020)
78.7%
94.3%
Table Probing F1 score on SST-2 and CoLA datasets.
Model
SST2 (Layer 0)
CoLA (Layer 0)
SST2 (Layer 6)
CoLA (Layer 6)
Cross-Entropy Entropy
76.2 + 1.5%
73.9 +1. %
79.9 + 1.3 % 79.9 + 1.7 %
78.2 = 1.7%
Harmonic
77.9 + 1.1%
74.3 + 1.0 %
77.1 + 4.2%
many practical settings, the contribution from Ty with yj * k is negligible, so wy is nearly a convex combination solely of class-k samples. By construction, a; 2 0 and the weights sum to This shows that wy is an interpretable vector representing its class center. In contrast, for cross-entropy loss the stationary condition does not yield similar expression for we combination of data points.
Under cross-entropy loss, the weight vectors usually end up pointing to the average direction of class elements, due to its use of the dot product. However, they do not have closed-form formula like the harmonic loss above, and the weight vectors are not linear combinations of all class feature directions. We believe that enforcing such linear combination structure plays a crucial role in enhancing interpretability - it directly aligns with the Linear Representation Hypothesis Park et al. 2023), and natively supports compositional generalization.
Remark:
G
Additional Benchmark Results
G.1 ImageNet
ImageNet DDng el al. 2009 is a large-scale visual dataset commonly used in object recognition research. We compare the performance standard cross entropy loss and harmonic loss on ImageNet. We trained ResNet-50 with AutoAugment data augmentation method for 90 epochs, starting with a learning rate of 0.1, which was reduced by factor of 10 at epochs 10, 30, 60, and 80. The training results are presented in Table [ We have also implemented our own cross-entropy training pipeline, and compared them with existing results in (Khosla et al. 2020). In our implementation, the harmonic model modestly outperformed the standard model.
G.2 SST2 and GLUE
We also compare the standard GPT2 and harmonic GPT2 with the GLUE benchmark below. We evaluate two tasks, COLA (linguistic acceptibility) (Warstadt et al. 2018) and SST2 (sentence sentiment classification) [Socher et al. 2013 We train a 1-layer MLP probe with hiddon dimension 16 that takes the model's residual stream representation as an input, and outputs the label. Table 2 shows the Fl score of the probe on validation dataset.
22


Reviews:
[{"review_id": "39aEwtERkh", "reviewer": "Reviewer_Xt82", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Introduces harmonic loss, motivated as a variation of cross entropy loss with\n\n1. the class specific weights becoming prototypes which the penultimater representation have to hit, replacing the standard inner product which calls for only inner product alignment $s_i =\\langle w_i,x\\rangle \\Leftrightarrow s _i=\\Vert w_i -x\\Vert_2$\n    \n2. the softmax nromalization replaced by a power normalized i.e. $ p_i =\\frac{\\exp(s_i)}{\\sum_j\\exp(s_j)}$ becomes $p_i=\\frac{s_i^{-k}}{\\sum_j s_j^{-k}}$ with $k\\geq 1$\n    \n\nClaimed theoretical benefits are scale invariance, nonlinear separability and faster loss decay due to finite converging point and interpretability since the weights become class protoyps. The paper also claims higher interpretability, measured in terms of compressability (measured as explained variance via PCA) and geometric composability (ala man-woman = king -queen)  measured via parallelogram loss.\n\nEmpirical evidence is given by experiments on\n\n- toy examples\n    \n    - complete a parallelogram on a 5x5 lattice given 3 points by predicting the 4th\n        \n    - given 2 integers x and y, predict x+y mod 31\n        \n    - given two integer smaller then 40, preict whether x=y mod 5\n        \n    - in a binary tree, predict parent,grandparent or sibling given a point (n-d tree relation prediction)\n        \n    - permutation composition given two permutation x any in S4 predict x chained with y (trained with $L_2$ regulariaztion of $5e-3$ instead\n        \n    \n    comparing\n    \n\n- “standard” MLP (input tokenized to 16 dimemional embeddings, concatenated, fed to a 2 hidden layer MLP with $d=64$ ,meaning for vocabulary size $V$ and max length $L$  we have $W\\in\\mathbb{R}^{V\\times 16}, W_1 \\in \\mathbb{R}^{(16L)\\times 100},W_1 \\in \\mathbb{R}^{100\\times 16}$  , with $SiLU$ as activation function)\n    \n- “standard” transformer with learnable positional embedding, two layers transformer decocder layers with 2 attention heads, of residual embedding dimension $16$ and MLP hidden dimension $64$\n    \n- harmonic MLP/transformer: same architectures, trained with harmonic instead of CE loss\n    \n    MLPs are trained for $7e3$  epochs, transformer of for $10e3$ epochs , using AdamW with $\\eta=2e-3$ and weight decay$1e-2$, as well as $L_2$ regularization of the embeddings of $1e-2$\n    \n\nAdditoinally, authors investigate\n\n- MNIST, using a linear model with learning rate 1e-3, bathc size 64 for 10 epochs\n    \n- openwebtext using GPT2-small, using nanogpt with differing hyperparameters between harmonic and standard loss\n    \n- imagenet (discussed in limitations and appendix)\n    \n\nAnd run ablations.\n\n---\n\nstrengths: \n\n1. interesting idea, haven't seen *exactly* this in literature yet\n2. exposition well done\n3. decent amount of experiments\n\nweaknesses:\n\n1. claims to strong for the evidence presented\n2. lack of related work to position in the literature", "claims_and_evidence_yn": "No", "claims_explanation": "1. There is a slight misclaim about scale invariance: harmax is only *uniformly* scale invariant, i.e. scaling all points by a constant doesn’t change the loss.  I think this is simply a slip of the tongue as it were,Softmax exhibits temperature scaling, which has a probabilistic interpretation,and should be more carefully discussed. \n    \n2. The experiments are not enough to present clear evidence for the paper claims for me for the following reasons\n    \n    1. the toy examples are constructed to map cleanly onto cases where k-nn /protoype matching will be aligned with the proble domain, if one were to construct examples where the structure maps onto cosine similarity the claims about interpretability could be reversed.\n        \n        1. this can be justified if presented transparently, so this can be salvaged if the claims are reduced by talking about what property makes a dataset aligned with one or the other (e.g. in the case of the graph there are graphs recoverable exclusively via KNN and exclusively  via inner products )\n            \n    2. all examples have a hidden penalty for the CE loss, by using $L_2$ regularization. this will mean that all norms are biased to be small, and close to another, which means the softmax is forced to smooth out contributions - effectively including a type of adversarial temperature scaling since the toy examples are all about recovering precise relations\n        \n3. on high dimensional datasets, where interpretability is actually required, the interpretability becomes much less pronounced (Imagenet, GPT-2) since no statistical analysis on distinguishability or actual *uses* of interpretability are performed (an experiment that could be performed is providing recourse or explanations, leveraging the compressible nature)\n    \n4. the related work section is missing comparisons to numerous prior art in prototype learning e.g.  [https://openaccess.thecvf.com/content_cvpr_2018/html/Yang_Robust_Classification_With_CVPR_2018_paper.html](https://openaccess.thecvf.com/content_cvpr_2018/html/Yang_Robust_Classification_With_CVPR_2018_paper.html)  , hyperbolic learning [https://www.sciencedirect.com/science/article/abs/pii/S0893608024003447?casa_token=AlAmi5gc2qUAAAAA:D57YCcv_stdEj9I40eECuJaLzwhHzEona_0wOA9QUO15ldy-TFtAVr1rrt-YAI0P4n-2NfgMAiBk](https://www.sciencedirect.com/science/article/abs/pii/S0893608024003447?casa_token=AlAmi5gc2qUAAAAA:D57YCcv_stdEj9I40eECuJaLzwhHzEona_0wOA9QUO15ldy-TFtAVr1rrt-YAI0P4n-2NfgMAiBk)  and zero shot learning [https://proceedings.neurips.cc/paper/2020/hash/fa2431bf9d65058fe34e9713e32d60e6-Abstract.html](https://proceedings.neurips.cc/paper/2020/hash/fa2431bf9d65058fe34e9713e32d60e6-Abstract.html) (these are meant to illustrate, not be exact citations). There is novelty to the presented paper (none of the works I found use exactly this loss, specifically the power normalization, and the paper isn’t doing hyperbolic representations but only penalizes large norms, which is a softer penalty to be close or within the hyperball, but not a strict constraint). Spherical/cosine losses are briefly discussed in appendix D, I think the paper needs to be more accurately placed in the literature\n    \n5. Hyper parameters appear cherry picked and are not justified\n    \n\nIf these issue can be fixed, I think the papers *idea* is potentially interesting, but the study needs to be made more rigorous", "audience_yn": "Yes", "audience_explanation": "I found it interesting, and I'm in the audience. Additionally, I've provided citations that show that there is research in this type of loss already, and interpretability is a hot topic.", "requested_changes": "critical\n\n1. add a more thorough literature review\n2. add more adversarial experiments to show limitations of the work more clearly\n3. harmonize and/or justify hyperparameters of experiments performed to show it's a fair comparison\n4. scope claims more precisely to the evidence provided \n5. use statistical significance to decide benefits and equality", "broader_impact_concerns": "None", "additional_comments": "Whenever I say \"cherry picked\" or \"not fair\" it is not meant to insinuate malice by the authors, it's simply the most compact way I can convey the idea. I hope the authors can improve their potentially very interesting study based on the requests", "extra_scores": {}, "extra_text": {}}, {"review_id": "W18xmdhZnZ", "reviewer": "Reviewer_hJ7b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces harmonic loss, a proposed alternative to the standard cross-entropy loss function. Its methodology differs from the standard approach in two main ways: it uses Euclidean distance to compute logits, and it normalizes these logits with a scale-invariant \"HarMax\" function instead of SoftMax.\n\nThe authors put forward three primary contributions stemming from this approach:\n1.  **Improved Model Interpretability**: It is argued that models trained with harmonic loss learn representations with more organized geometric structures, where the learned weights correspond to class centers.\n2.  **Greater Data Efficiency**: The paper shows that models can generalize from less training data compared to standard training.\n3.  **Mitigation of Grokking**: The authors provide evidence that harmonic loss reduces the effect of delayed generalization, where test performance improves long after training performance has converged.\n\n**Key Strengths:**\n* The paper presents a novel and well-motivated modification to a fundamental component of neural network training.\n* The claims are evaluated across a reasonably wide range of tasks, including algorithmic, vision, and language domains.\n* The qualitative evidence presented, particularly the visualizations of learned embeddings and model weights, helps to illustrate the method's effect on model internals.\n\n**Key Weaknesses:**\n* While the paper references Large Language Models (LLMs), the language model experiment is conducted on a GPT-2 small (128M parameter) model. By contemporary standards, this is not considered a large-scale model, which limits the direct applicability of the findings to the current class of LLMs.\n* The evidence for improved interpretability is clearest on tasks with well-defined ground-truth structures. On more unstructured tasks, the evaluation relies on proxies such as geometric analogies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In my view, the claims made in the submission are adequately supported by the provided evidence.\n\n* **Interpretability**: This claim is supported by visualizations and metrics across the different experiments. For the algorithmic tasks, the results in Figures 2 and 3(a) show that the harmonic models tend to recover the known data-generating structures more cleanly than the baseline models. The MNIST experiment in Figure 4 shows that the model's weights resemble the digit classes, which aligns with the paper's claims. For the language model, the analysis of parallelogram loss provides quantitative support for the idea that the embeddings learn a more consistent geometric structure for semantic relationships.\n\n* **Data Efficiency**: Figure 3(b) shows a consistent trend where harmonic models require a smaller fraction of the training data to reach high test accuracy compared to the baselines.\n\n* **Reduced Grokking**: The plots in Figure 3(c) effectively illustrate a reduction in the gap between the convergence of training and testing accuracy for the harmonic models, which supports the claim of mitigating grokking.\n\nThe ablation study in Section 7 also lends credibility to the design of the loss function by showing that both of its primary components contribute to the observed effects.", "audience_yn": "Yes", "audience_explanation": "I believe the findings would be of interest to the TMLR audience. The work addresses several topics of relevance to the machine learning community.\n\nResearchers in areas like **interpretability and representation learning** would likely be interested in a method that aims to create more structured and understandable models by modifying the training objective itself. The paper's investigation into the **fundamentals of training dynamics**, particularly its connection to the grokking phenomenon, is also a topic of current research interest. Given that the paper proposes an alternative to a near-universal standard (cross-entropy with SoftMax), it is likely to be of broad interest to those who design, train, and analyze neural networks.", "requested_changes": "1.  **Experimental Validation of Downstream Interpretability**: The claim of improved interpretability in unstructured domains currently rests on the analysis of parallelogram structures. To provide more direct evidence of the practical benefits of this improved geometry, I would suggest a small-scale experiment applying a post-hoc feature attribution method to the GPT-2 models. For instance, an analysis of how attributions for a few key predictions differ between the standard and harmonic models could reveal whether the latter's representations lead to more semantically coherent or localized explanations.\n\n2.  **Analysis of Euclidean Distance Scaling in High-Dimensional Spaces**: The use of Euclidean distance as the core logit is a key aspect of this work. However, as models scale, their embedding and vocabulary dimensions can grow substantially. It is a known phenomenon that in very high-dimensional spaces, the discriminative power of L2 distance can diminish. To address how this might affect the proposed method's scalability, the paper would be stronger with an empirical analysis of the distance distributions from the existing GPT-2 experiments.\n\n3.  **Validation on a Contemporary Large Language Model**: To fully substantiate the claims regarding LLMs, the authors should consider either toning down the claims in the abstract and introduction or, preferably, validating their method on a more contemporary, large-scale model (e.g., in the 1B+ parameter class). Demonstrating that the observed benefits hold at a scale that is more representative of today's LLMs would significantly increase the impact of this work and provide stronger support for its broader claims.", "broader_impact_concerns": "I have no broader impact concerns regarding this work. The research is foundational and aimed at improving the transparency and reliability of AI models. By making models more interpretable, this work contributes positively to the goal of building safer and more trustworthy AI systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G9v4RHYJXh", "reviewer": "Reviewer_Q8dE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces harmonic loss as an alternative to cross-entropy loss for training neural networks and LLMs. The key innovation involves: replacing SoftMax with a scale-invariant HarMax function, and computing logits via Euclidean distance rather than dot products. The authors claim this leads to improved interpretability, faster convergence, and reduced grokking across algorithmic, vision, and language tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides solid experimental evidence across multiple domains. The algorithmic experiments clearly demonstrate reduced grokking and improved data efficiency. The representation visualizations (Figure 2) convincingly show that harmonic models learn more interpretable structures. The GPT-2 experiments with parallelogram analysis provide quantitative evidence of improved geometric properties.", "audience_yn": "Yes", "audience_explanation": "This work addresses fundamental questions in machine learning about interpretability, data efficiency, and training dynamics that are of broad interest to the TMLR community. The interpretability problem is particularly relevant given current concerns about understanding large neural networks. The grokking phenomenon and data efficiency improvements have practical implications for training neural networks.", "requested_changes": "Would Strengthen the Work:\n1. Computational Complexity Analysis: Add discussion of computational overhead comparing L2 distance computation vs. dot products, especially for large vocabulary sizes. This is essential for practical adoption.\n2. Additional Case Discussion: Investigate specific scenarios where harmonic loss might fail or be suboptimal (e.g., highly imbalanced datasets, extremely sparse high-dimensional data, tasks requiring linear separability).", "broader_impact_concerns": "No significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5QmITe1u7i", "reviewer": "Reviewer_2V7y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces harmonic loss as an alternative to the standard cross-entropy loss. This method brings two main innovations: (1) Replacing the SoftMax function with a scale-invariant HarMax function; (2) Computing logits using Euclidean distance instead of the dot product. The authors argue that this structure provides scale invariance, a finite convergence point, faster training, better generalization with less data, reduced groping, and more interpretable representations. These claims are supported through experiments on algorithmic tasks, MNIST, GPT-2 pretraining, and ablation studies.\n\nStrengths\n* The harmonic loss idea is well-motivated and presented, with strong arguments against certain structural limitations of SoftMax.\n* Properties such as finite minimum and scale invariance are explained both intuitively and formally.\n* The work spans from algorithmic datasets (in-context learning, modular addition, etc.) to real-world settings (MNIST, GPT-2).\n* PCA explained variance and parallelogram loss are used as systematic indicators of representation interpretability.\n* Contributions of HarMax and L2 logits are analyzed separately.\n\nWeaknesses\n* While PCA variance and parallelogram loss are useful, more comprehensive or user-centered measures (human evaluation, task-based interpretability) are missing.\n* Results on GPT-2 small (128M) are promising, but applicability and stability for larger LLMs (1B+, 10B+) remain unknown.\n* The “faster convergence” claim is shown qualitatively in plots but lacks quantitative data on wall-clock time, GPU hours, or energy savings.\n* While there is some discussion in Appendix D, the paper does not include full-scale empirical comparisons to strong baselines such as focal loss, margin-based SoftMax, or supervised contrastive loss.\n* The main experimental wins are on tasks with well-defined mathematical structure (lattice, modular addition). It is unclear if the advantage persists in noisy, complex real-world tasks.\n* Although a heuristic formula for the HarMax exponent is provided, a deeper analysis of its sensitivity across data types and model sizes is lacking.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission provides extensive experimental evidence across multiple domains, algorithmic tasks, MNIST, and GPT-2 pretraining, to support its main claims. The paper combines qualitative representation analyses (e.g., PCA visualizations, coset structures, parallelogram geometry) with quantitative metrics (explained variance, generalization curves, grokking plots) that consistently show improvements over cross-entropy baselines. Ablation studies further isolate the contributions of HarMax and L2 distance, strengthening causal interpretation.\n\nThat said, some claims, particularly those related to large-scale applicability, training efficiency gains, and interpretability in noisy real-world settings, are supported mainly by small- to mid-scale experiments and qualitative observations. While the presented evidence is convincing for the tested settings, broader benchmarks would be needed to confirm generality.", "audience_yn": "Yes", "audience_explanation": "The work addresses a core component of modern machine learning, the loss function, and proposes a novel alternative to cross-entropy with theoretical grounding, clear interpretability motivations, and empirical performance gains. TMLR’s audience, which includes researchers in optimization, representation learning, interpretability, and large language models, would likely find the findings relevant. In particular, the demonstrated ability of harmonic loss to reduce grokking, improve representation geometry, and potentially enable more interpretable LLM embeddings aligns well with current research interests in trustworthy and efficient AI.", "requested_changes": "The author may consider to:\n\nComplement PCA and parallelogram loss with human evaluation studies or semantic alignment measures.\nEven a limited-scale comparison on a 1B+ parameter LLM would strengthen claims of scalability.\nReport actual time-to-accuracy and resource usage for both harmonic and standard losses.\nInclude empirical benchmarks against focal loss, large-margin SoftMax, and supervised contrastive loss.\nAnalyze optimal HarMax exponent values for diverse tasks (vision, language, graph) and model sizes.\nInclude experiments with noisy labels, domain shift, or open-ended NLP tasks to demonstrate robustness.", "broader_impact_concerns": "I do not see any major unaddressed ethical risks that would require a separate Broader Impact Statement beyond standard practice. The proposed harmonic loss is a general training objective and does not inherently introduce new privacy, fairness, or misuse concerns compared to existing supervised learning methods.\n\nThat said, the work’s interpretability claims could have downstream societal implications. If harmonic loss indeed produces more interpretable models, this could positively affect transparency, trust, and accountability in high-stakes domains such as healthcare or finance. Conversely, if interpretability is overstated or misunderstood, it might create a false sense of trust in models that remain opaque in practice. It would be beneficial for the authors to explicitly acknowledge this risk and clarify that their interpretability metrics are proxies, not guarantees of real-world explainability.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)