Isometric Representations in Neural Networks Improve Robustness
Authors: Kosio Beshkov
Jonas Verhellen
Mikkel Elle Lepperød
Venue: Submitted to ICLR 2023 (2023)
Keywords: Isometry
Deep Learning
Robustness
Adversarial Attacks
Continuous Representation
Classification

Abstract: Artificial and biological agents are unable to learn given completely random and unstructured data. The structure of data is encoded in the distance or similarity relationships between data points. In the context of neural networks, the neuronal activity within a layer forms a representation reflecting the transformation that the layer implements on its inputs. In order to utilize the structure in the data in a truthful manner, such representations should reflect the input distances and thus be continuous and isometric. Supporting this statement, recent findings in neuroscience propose that generalization and robustness are tied to neural representations being continuously differentiable. However, in machine learning, most algorithms lack robustness and are generally thought to rely on aspects of the data that differ from those that humans use, as is commonly seen in  adversarial attacks.

During cross-entropy classification,  the metric and structural properties of network representations are usually broken both between and within classes. This side effect from training can lead to instabilities under perturbations near locations where such structure is not preserved. One of the standard solutions to obtain robustness is to train specifically by introducing perturbations in the training data. This leads to networks that are particularly robust to specific training perturbations but not necessarily to general perturbations. While adding ad hoc regularization terms to improve robustness has become common practice, to our knowledge, forcing representations to preserve the metric structure of the input data as a stabilising mechanism has not yet been introduced.

In this work, we train neural networks to perform classification while simultaneously  maintaining the metric structure within each class, leading to continuous and isometric within-class representations. We show that such network representations turn out to be a beneficial component for making accurate and robust inferences about the world. By stacking layers with this property we provide the community with an network architecture that facilitates hierarchical manipulation of internal neural representations. Finally, we verify that our isometric regularization term improves the robustness to adversarial attacks on MNIST. 

Under review as a conference paper at ICLR 2023
ISOMETRIC REPRESENTATIONS IN NEURAL NET- WORKS IMPROVE ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Artificial and biological agents are unable to learn given completely random and unstructured data. The structure of data encoded in the distance or similarity relationships between data points. In the context of neural networks, the neuronal activity within a layer forms a representation reflecting the transformation that the layer implements in a truthful manner, such representations should reflect the input distances and thus be continuous and isometric. Supporting this statement, recent findings in neuroscience propose that generalization and robustness are tied to neural repre- sentations being continuously differentiable. However, in machine learning, most algorithms lack robustness and are generally thought to rely on aspects of the data that differ from those that humans use, as commonly seen in adversarial attacks.
its inputs. In order to utilize the structure in the data
During cross-entropy classification, the metric and structural properties of net- work representations side effect from training can lead to instabilities under perturbations near locations where such structure is not preserved. One of the standard solutions to obtain ro- bustness is to train specifically by introducing perturbations in the training data. This leads to networks that are particularly robust to specific training perturbations but not necessarily to general perturbations. While adding ad hoc regularization terms to improve robustness has become common practice, to our knowledge, forcing representations to preserve the metric structure of the input data as sta- bilising mechanism has not yet been introduced.
usually broken both between and within classes. This
In this work, we train neural networks to perform classification while simultane- ously maintaining the metric structure within each class, leading continuous and isometric within-class representations. We show that such network representations turn out be beneficial component for making accurate and robust inferences about the world. By stacking layers with this property we provide the community with an network architecture that facilitates hierarchical manipulation of internal neural representations. Finally, we verify that our isometric regularization term improves the robustness to adversarial attacks on MNIST.
INTRODUCTION
Using neuroscience as an inspiration to enforce properties in machine learning has roots dating back the birth of artificial neural networks (MeCulloch & Pitts 1943 Rosenblau 1958). One way to study natural and artificial neural networks is to look at how they transform specific structural properties of input data. The output of such a transformation is typically called neural, or latent, representation, and it carries information about the computational role of a brain region or network layer (Kriegeskorte) 2008 Kriegeskorte & Diedrichsen 2019: [Begiio al. 2013). Different prop- erties of representations are helpful in different ways for both organisms and artificial agents, Some examples of this are efficient coding Barlow et al. (1961). mixed selectivity (Rigotti et al. 2013), sparse coding (Olshausen & Field 2004). response normalization (Carandini & Heeger] 2012). et- ficiency and smoothness (Stringer et al. 2019) and expressivity (Poole et al. 2016 Raghu et al. 2017) among others.
For example, one subsection of theories related to efficient coding proposes that neural circuits should generate discontinuous and high-dimensional representations pack the most information

---

Under review as a conference paper at ICLR 2023
On the other hand, empirical results point out that neural circuits generate low dimensional smooth representations of the data (Gao & Ganguli 2015 Gao et al. 2017). This apparent contradiction has already been rigorously discussed in inSiiinnger et al. al. (2019). According to the work of Stringee et al.] neural circuits try to be as efficient possible while smoothly mapping inputs. Without this smoothness constraint, infinitesimal perturbations of input stimuli could drastically change the output, thereby making such circuits non-robust some perturbations. Given the empirical support, seems likely for these properties to hold in early sensory systems and thus to be important for broad class of machine leamnng algorithms.
possible into network (Barlow al.
1961 Simoncelli & Olshausen 2001
Organisms seem particularly robust to random input perturbations. However, artificial models suffer from lack of robustness to adversarial attacks Goodfellow et al. (2014). One argument for why (2020). in which the authors use topological ap- this happens can be deduced from Naitzat et al.
proach based on persistent homology (Carlsson 2009 Edelsbrunner & Harer (022), to study the
mappings realized by neural networks performing classification task. They claim that, in classi- fication problems, neural networks implement structure-breaking
noo-moomeomopphic mappings, and argued above, models implementing such mappings are unlikely to be robust. There many ways to improve the robustness of network (Madry et al. [017] [Silaa & Najafirad [2020] Xu
et al. 2020). Nevertheless, of particular interest to are strategies that try solve this problem by restricting the properties of the mapping realized by network. Examples of this are Jacobian reg-
ularization (Hoffman al. (2119). spectral regularization (Miyato al. 2018 Nassar et al. 2020).
Lipschitz continuity (Virmaux & Scaman 22118; Liu et al. 2022), topological regularization (Chen
al. 2019) and manifold regularization (Jin & Rinard 2020) among others.
While regularities in neural representations help with robustness, they do not necessarily guarantee that the input and output representations will have the same metric relationships, thereby reflecting the actual structure of the data. To our knowledge, there are still no methods to preserve the class metric structure while allowing for robust classification. To achieve this behavior, we create a neural network model with, what we call, Locally Isometric Layers (LIIs) and study the representations generated by training such networks. Furthermore, we extend these networks to generate represen- tations in a hierarchical manner, which makes them helpful in performing classification at different resolutions. Finally, we train LILs on MNIST and show that the isometry condition leads to an im- provement in network robustness to both the Fast Gradient Sign Method (FGSM) and the Projected Gradient Descent (PGD) adversarial attacks.
2 BACKGROUND AND METHODS
In this section, we summarise the mathematical background of some the different mappings that neural network implement and introduce LILs. We treat both training and test data as being sampled from a manifold M and the neural network N as a set of maps N = {f [f! M + R), with l denoting the layer and i the index of neuron. Another way to interpret the action of neural network, which will become useful later on, is to define its mapping layer-wise.
Definition 1: A Neural Nenwork N with L layers acting on manifold M is set of functions ( 1 H- where n, is the number of neurons in a particular layer.
To perform classification,
usually tries to train network to realize particular function Φ M + ReE, which holds easily separable representations of the data. After this, a simple linear layer can be used for the final classification. This procedure requires the specification of cost function. Here we will consider the following example:
C(X, () T [+(X)]
NZ IIG DM - Go G = =oSE + LisO,
(I)
where X = (xI,..., NN} } are the inputs, T (ti,..,,N) are the desired labels, o is the Softmax function, ||. II is the Frobenius norm, DM and Dx are the distance matrices in the input and output space generated by d(z,, xyy and d(;();;, ), ((x))) respectively and G is an indexing matrix. The distance functions can be any metric, but in this work, we stick to the Euclidean distance. Given partition of the training set = Lk Vr. VA C. M, the indexing matrix G is defined in the following way:
2

---

Under review as a conference paper at ICLR 2023
B
M RM
M Li
Localy Isometric Layer (LIL)
Isametry
Local Isomeeryy
Non-lsomelry
V1
V2
2/2
V
Classiller layer
Figure 1: A) Visualization of the three types of mappings. B) The architecture of Locally Isometric layer (LIL). Green nodes correspond to an input layer, purple to hidden layers and orange to classification layer.
G(Z,, = {1
xs E Vk,
(2)
otherwise.
This indexing term forces the network to maintain the local distance relationships between the input points the output layer. The metric preservation is enforced within each class separately, which helps the network find representations that easy classify while also preserving the relational information of the data. This second loss term works hand in hand with the first loss term, which is just the usual cross-entropy loss regularly used in classification problems and helps to separate the different classes. This procedure might seem reminiscent of triplet loss, which is widely used in contrastive learning (Hadsell et al. 2006 Mao al., 2019). The vital difference is that instead of minimizing the distance between points in the same class, it enforces the distances between them to be the same as in the input data.
In this context, it is also important to mention the work of Sengupta et al. 2018), where the au- thors imposed similar metric constraint on ReLU networks to explain the emergence of localized receptive fields. However, they were not attempting to solve classification tasks and, as result, did not adapt their loss function only to preserve metric relationships within the class. A similar loss function, without the local within-class indexing term G, is also one of the main components behind some unsupervised dimensionality reduction algorithms (Tenenbaum et al. (2000). especially those based multidimensional scaling. The main difference between rithms that they are used to project data to a low-dimensional space in which it can be visualized. Additionally, they usually not realized by neural networks and are not used for classification.
method from such algo-
2.1 THREE TYPES OF MAPPINGS
One describe the mappings of neural networks in many ways (Homik 1991 Bianchini & Scarselli 2014 Guss & Salakhutdinov] 2018 Yang & Salman 2019), but the main property which of interest to us is that of metric preservation. Using such description,
end up with three types of mappings: global isometries, local isometries, and non-isometries. Visual representations of these different mappings
given in figure [[ A definition of these concepts in order.
Definition 2: Given a metric space (X,d) and a mapping F X
+ +. is an isometry (i.e. (F(())F())) If this property only holds locally, meaning
preserves the metric) if d(z, y) d(æ, y) = d()((), F(y)) for I, y V, where V C. X, we call it a local or piecewise isometry.
To get these different regimes, we can weigh the loss in the following way,
f aLCSE BLiso,
(3)
3

---

Under review as a conference paper at ICLR 2023
with 3 = 0. only maintain the standard cross-entropy term and obtain discontinuous representa- tions. For 3 > O, one gets locally isometric maps. The case with global isometry can be obtained by setting G.y = 1, Vi, j since then, the isometry loss is enforced for all points independent of their class. We will refer to layers whose weights are updated with 3 > 0 as Locally Isometric Layers (LII.). As mentioned before, by partitioning the data manifold M into different subsets, the network enforces isometry on different local patches or at different resolutions.
2.2 EXPERIMENTS
Our initial experiments were done relatively low dimensional data that we could use small and tractable neural networks made up of 4 layers consisting of 20 neurons with hyperbolic tangent activation functions. So for both the entangled rings and torus task, each LIL had the following feedforward architecture [D.20.20.00.0.c.
where D is the dimension of the dataset the LIL re- ceives and C is the number of neurons in the final classifying layer. The networks in the toy datasets were trained for 10000 epochs with a batch size equal to the number of training points using the Adam optimizer (Kingma & Ba 2014) in PyTorch (Paszke et al.) 2019). The choice of a large batch size guarantees that each time the gradient is computed, there will be more points in the same class, and thus the distance relationships between them will contribute more to the loss. these networks are trained solely by stochastic gradient descent with a batch size of 1, the isometric term will not contribute anything to the loss, which would be equivalent to only using the cross-entropy term.
When training on MNIST, we slightly increased the network size to 100 neurons and did not use any hicrarchical structure as that is not present in the labeling of MNIST. The classification was done with single LIL of 4 layers with 100 neurons each. We used a batch size of 100 for five epochs for the training. To test the robustness obtained by adding the isometric loss term, we trained six networks with the following B parameters- adversarial attacks with a step size varying logarithmically from 0.01 to 1. For the PGD attacks, we fixed the Loo ball size to 0.5 and took ten steps with a varying step size in the same range.
1..0000,,0001..1 10100.1.,, 1,111, We performed L FGSM
3 RESULTS
3.1 DISENTANGLING ENTANGLED DATA
Inspired by (Naitzat et al. 2020), we generated a dataset in which the two classes take the shape of two topologically entangled rings in 3-dimensional embedding space, which means that there is no way to separate the two without cutting one of them apart. However, the rings are not entangled in the same way in higher dimensional spaces, as one can move one of the rings in the direction of the fourth dimension, thereby putting it in a different subspace. Given that neural networks project to spaces of much higher dimension, an operation like this should be simple to implement. After training a network with LII.,, we find mapping that disentangles the two manifolds while preserving their topological structure. This solution is very different from the one achieved by only using the cross-entropy loss, where as expected, the topological structure is lost - see figure 2
In order to visualize the rings in the network's 's last layer, we use the UMAP dimensionality reduction algorithm (McInnes et al. 2018). In addition, we show the distance distributions between the points in each ring after passing through the two networks, which heavily supports the idea that the LIL implementation manages to preserve the within-class distance relationships almost perfectly 2B. In contrast, using the cross-entropy loss results in no preservation of structure and instead brings points closer together, as indicated by the exponential shape of the orange histograms, which appears even when a log scale is used.
3.2 ISOMETRIC MAPPINGS ARE ROBUST
As seen in the previous section, imposing isometry as condition the output layer of neural network leads to a within-class continuous mapping. We propose that this is a result of the following properties:
Definition 3: A functin is called K- K-Lipschitz continuous if there exists a constant K 2 for which d(F(x), F())) & Kd(a.y) y).
4

---

Under review as a conference paper at ICLR 2023
A
B
Distance distributions
Without LIL
Blue class
207
Cond
30
10a
Intia data
102
Orange class
30"
20*
102
301
With LIL
Figure 2: A): The initial dataset and UMAP projections (number of neighbors=40 and minimal distance = 0.25) of the last layer with and without LII... B): The distance distributions between the embeddings in the last layer for each of the two rings, trained solely with CSE (orange) and with the isometric term included (light green).
With this definition in mind, one can see that:
Proposition: Isometric mappings are 1-Lipschitz continuous.
Given the definition of isometry: d(x,8) ((F(x),(y))) Simply let K 1 + and obtain the inequality (1 + e)d(x, y) 2 (())) The smallest value for which the equality holds is which is the Lipschitz constant.
Proof:
Such mapping has additional desirable properties like being almost everywhere differentiable due to Rademacher's theorem (Heinonen et al., 2001) and having derivatives bounded by the Lipschitz constant.
Proposition: K-Lipschitz continuous mappings have bounded derivative.
Proof: Rearrange the inequality from the previous proposition d(F()).F()) d(x.y)
< K. Then take
I + 5, this leads to the derivative /| I K.
All of these properties improve robustness adversarial attacks since they enforce the gradients of the loss with respect to the data to be bounded. To see this consider the gradient computed in a fast gradient sign method (FGSM).
(a VaaLosE + V.BLiso.
By expanding these terms and applying the chain rule, we obtain the following:
(o()(xx)) (T(5)))
(o(((x)))
V.CCE = -a o(((I:)) 04(x))
S a o(((z))) Ob(x;)
K,
(5)
dr
BV.Lis0 =
233
Σ H1 8>1
43G N2
Σ HiyJesk. 1>1
(6)
N2
Here have compressed our notation and redefined the distance matrix difference H d(æ,, x.) d(ó(x,), ((x,)), the difference between the mapping at two different points as Jr,
o(m,)-((z,) (((,,, and the difference of the gradient evaluated at two different points as (#xx,,,, ((x;)] (Z,) - (2,)) To see a more detailed derivation of equation 6 see Appendix A.I
Thus, by exploiting the isometric property, with a bounded derivative, which is expected to improve robustness because it keeps gradients with respect the data small.
enforce neural networks implement mappings

---

Under review as a conference paper at ICLR 2023
A
B
FGSM
PGD
1.0
1.0
CSE
0.001
0.8
B 0.01
0.8
Clss-iiction rate
0.1.1
Claveitatiion rate
0.8
B=1
0.8
== 10
0.4
0.4
0.2
0.2
0.0
0.0
10
10
10"
10
Figure 3: A) Robustness of LIL an FGSM adversarial attack. B) Robustness a PGD adver- sarial attack with a Lo bounding neighborhood of size 0.5.
3.3 ROBUSTNESS TO ADVERSARIAL ATTACKS ON MNIST
To show this empirically, we trained the LIL model on MNIST, as described in 2.2 We see stark improvement in robustness to both FGSM and PGD attacks as the constant controlling the importance of the isometry condition is increased. The results are shown for FGSM and PGD in figure 3, panels A and B, respectively. However, this improvement does not come without a cost; see table [] showing that increasing the importance of isometry leads to degradation in terms of performance. At the same time, smaller values keep the performance around the cross-entropy baseline. Thus models which perform mappings very close to isometry are much more robust, but they pay the price for that by having sub-optimal performance in the absence of perturbations.
Table MNIST test performance as a function of the 33 parameter which controls to what extent the isometry property contributes to the loss.
3=0
/=0.001
B=0.01
3=0.1 0.9665
3=1 0.9588
3=10
0.9701
0.9688
0.9686
0.8816 16816
3.4 LOCALLY ISOMETRIC LAYERS AND HIERARCHICCA REPRESENTATIONS
Another way to use LILS by stacking them in order to glue or eut subsets the data manifold, as shown in figure 4A. The first LIL implements some function + .M- - R which splits the original manifold into C' submanifolds (corresponding to the number of classes) while maintaining
the distances between the points. After the first split is performed, the following L.II operates on the representation already split by the previous layer, giving a new partition M Lle Vs- Depending on the choice of the partition, each following LIL can be used either glue or cut the original manifold. For the gradients not to interfere, we impose the condition that for each LIL, gradients are backpropagated only their preceding LIL, but this assumption can be relaxed. In this work, we also use a shared classification layer, for which we only need to train the linear projection from each L.I.. However, the same result can be achieved by appending separate classification layers to each L.I...
pieces of
In this final example case, we use this stacked LII. model to look at a simple torus split into cylin- drical regions We sampled 1600 points from the standard three-dimensional parametrization of the torus and added a small amount of noise v (O, 0.001) to it. We set the 3 = 100, which is relatively high. We used only one batch, consisting of all the sampled points from torus. This way, the isometric condition is applied fully in all epochs. This is a rather simple example case, and the network has to perform the task by separating the different regions of the torus into locally isometric submanifolds figure 4 Because, this case, we make use of the hierarchical version of our model, the regions of the torus are partitioned with different degrees of coarseness across

---

Under review as a conference paper at ICLR 2023
A
= L
LIL
Partition
M Live
LIL
Partition
LIL
M
Live
Classifter layer
Partition
Figure 4: A) Stacked LILs, implementing different locally isometric maps depending on the parti- tioning (labeling) of the input data. B) UMAP visualizations (number of neighbors 40, minimal distance 0.25) of the representations classification task in hierarchically stacked LILs.
layers. The stacked LIL network manages to separate the classes well while maintaining a high classification rate 0.99).
DISCUSSION
4.1 ADVANTAGES OF ISOMETRIC MAPPINGS
In this work, developed method for classification while maintaining the local metric relation- ships between points the input data. This type of mapping is beneficial for the truthful represen- tation of structural relationships in data and for improved robustness to targeted perturbations like those used in adversarial attacks. We used a Euclidean metric to represent the structural relationships between data points. However, the approach can also be extended other metries more suitable the data being analyzed. In our toy dataset experiments, we show that the LIL implementation completes a classification task without destroying the local structure within each class.
At the same time, standard neural networks trained with solely cross-entropy perform non- homeomorphic mappings. This result contrasts the reasoning in Naitzat et al. (2020), which argues that non- -omm-oommmoppicc mappings are necessary to disentangle topologically entangled data, simpli- fying the topology the data and allowing for separation by hyperplanes. Using LILs, comparable classification performance is made possible while the topology and even the metric structure of the initial data manifold are preserved within each class throughout the network. From mathematical disentangle data in higher-dimensien spaces obtained with
perspective, this is expected a projection of wide neural network layers.
4.2 ROBUSTNESS TO ADVERSARIAL ATTACKS
Our results on MNIST indicate clear improvement robustness adversarial attacks. One pos- sible explanation for this improvement is that the isometry condition imposes small Lipschitz constant on the function mapping data or stimuli to neural representations. This makes the deriva- tives of such mappings bounded. As result, perturbation strategies that rely computing gradients with respect to data, like those employed in FGSM and PGD, will be weaker. One might also use similar reasoning explain the increased robustness observed when adding either smoothness or more direct Lipschitz constraints, as seen in Chen et al.) 2209%; Hoffman et al. (2019): Jin & Rinard (020): Nassar al. (2020); [Liu et al. 2022 among others.
On the other hand, it is unclear what the Lipschitz constant of a traditional cross cross-entron entropy trained neural network is, despite attempts provide estimate (Virmaux & Scaman 2018). If such

---

Under review as a conference paper at ICLR 2023
networks turned out have smaller Lipschitz constant almost everywhere except few breaking points where the topological structure is not maintained, they would still be expected to be robust. In that case, there would have to be different explanation for the improvement in robustness that we observe.
We also point out that using the LIL strategy with large 3 values decreases classification perfor- mance. This implies that, at least in this example case, there is a trade-off between performance and robustness. This trade-off has also been found in other work (Tsipras al. 2018) and might be a necessity or a feature of how we achieve robustness. In any case, it is an interesting problem to explore in future work.
4.3 HIERARCHICAL SPLITTING OF REPRESENTATIONS
Finally, we have also allowed our model to take advantage of hierarchical structure in labeled data. Hierarchical classification has been attempted previously (Srivastava & Salakhutdinov 2013 Deng
al. 2014, Wehrmann al. 2018 Scieur & Kim 202I). However, to our knowledge, this has not been done while also maintaining the local within class structure of the data. At least intuitively, by mapping the data to the first class partition a locally isometric manner, one guarantees that the fol- lowing more fine-gra fne-grained classification network will work with a truthful representation of the data. While we have not explored this intuition further in this work, we believe that this something that would be interesting
investigate in hierarchically structured datasets like Imagenet (Krizhevsky 2017) or the many of the other publicly available and more specialized alternatives (Wei al.) 2021 in the future. Another possible application of these concepts is in the area of dimensionality reduction and data visualization. For example, one can use hand-labeling or unsupervised hicrar- chical clustering algorithms to obtain plausible data labels. Given such labeling, LII.s be used to project the data to a low dimensional space which the different classes are separated while the metric structure maintained.
ct al.
CONCLUSION
To summarize, we have extended the existing regularization methods by introducing a local isome- try condition that keeps the structure within classes consistent across network layers. Nevertheless, such a mapping still allows for different classes to be separated, which is a necessary feature of clas- sification. Additionally, one, due to the isometry property, it is capable of continuously untying entangled data. We have also shown that in virtue of such properties, gradients with respect the input data are bounded, which expected to improve robustness to adversarial attacks. This is empirically demonstrated in simu- lations on MNIST. Finally, we propose that our LIL model can be stacked improve hierarchical classification.
have explored some peculiar features of this type of regularization. For
ACKNOWLEDGMENTS
REFERENCES
Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01), 1961.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1788 2013.
Monica Bianchini and Franco Scarselli. On the Complexity of Neural Network Classifiers: A Com- parison Between Shallow and Deep Architectures. IEEE Transactions on Neural Networks and Learning Systems, 25(8):1553-15 1555, August 2014, ISSN 2162-237X, 2162-2388. doi: 10. 109/ INNLS.2013.229936 URL http: http::/
ieee wtgg//ccmeett//9976800
Matteo Carandini and David Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1):51-62,2) 2012.
8

---

Under review as a conference paper at ICLR 2023
Gunnar Carlsson. Topology and data. Bulletin of the American Mathematiccal Society, 46(2):255- 308, January 2009. ISSN 0273-0979. doi: 02L.0000/0027330999990-----22220 URL http:
// //ww //www ams org/
Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. topological regularizer for classifiers via persistent homology. In The 22nd International Conference Artificial Intelligence and Statis- fics, pp. 2573-2582 PMIR, 2019.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In European conference on computer vision, pp. 48-64. Springer, 2014.
Herbert Edelsbrunner and John L Harer. Computational topology: an introduction. American Mathematical Society, 2022.
Peiran Gao and Surya Ganguli. On simplicity and complexity in the brave new world of large-scale neuroscience. Current opinion in neurobiology, 32:148-1555 155, 2015.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. theory of multineuronal dimensionality, dynamics and measurement. preprint, Neuroscience, November 2017. URL http::://bbbookt...oo///////////
214262
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, 2014.
William H. Guss and Ruslan Salakhutdinov. On Characterizing the Capacity of Neural Networks
using Algebraic Topology. arXiv: [es, math, stat], February 2018. URL http:
100.// 04443
arXiv: 1802.04443.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Juha Heinonen et al. Lectures on analysis on metric spaces. Springer Science & Business Media, 2001.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv:1908.022 027229, 2019.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2):251-257, 1991.
Charles Jin and Martin Rinard. Manifold regularization for locally stable deep neural networks. arXiv preprint arXiv:2003.00 2003.00000, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint
arXiv:1412.6900 2014.
Nikolaus Kriegeskorte. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2008. ISSN 16625137. doi: 10.3389/neuro.06 004.2008. URL
frontiers wwo.ttiiessii.orrgattillllleee/e
06. 16.004.2008 abstract
Nikolaus Kricgeskorte and Jöm Diedrichsen. Peeling the onion of brain representations. Annual review of neuroscience, 42(1)::07-433) 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90 2017.
Hsuch- Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. arXiv preprint 220::220220034455 2022.

---

Under review as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rXiv:1706.060 2017.
Chengzhi Mao, Ziyuan Zhong, Junfeng Yang. Carl Vondrick, and Baishakhi Ray. Metric learning for adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115-133, 1943.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rX1::180..03326, 2018. 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprini arXiv: 1rxxv:/802.0555: 2018.
Gregory Naïtzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. J. Mach. Learn. Res., 21(184):1-40, 2020.
Josue Nassar, Piotr Aleksander Sokol, Sue Yeon Chung, Kenneth D. Harris, and Memming Park.
On I/n neural representation and robustness. arXiv:2012.0472. [cs], December 2020. URL http://arxiv.ot
wttpor/gbabb/////a http:///arxxv.orrrab//0001.
04729
arXiv: 2012.04729.
Bruno Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neuro- biology, 14(4):481-485 2004.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the pressive power deep neural networks. In international conference on machine learning, pp- 2847-2854. PMLR, 2017
Mattia Rigotti, Omri Barak, Melissa R. Warden, Xiao-Jing Wang, Nathaniel D. Daw, Earl Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585-590, May 2013. ISSN 0028-0836, 1476-4687. doi: 10.1038/nature12166) URL http:// wwwp///wwww/.m// nature. com/ artiaaattilesaaa htt.:iiccee//atteee116000
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization
in the brain. Psychological review, 65(6):386, 1958.
Damien Scieur and Youngsung Kim. Connecting sphere manifolds hierarchically for regularization. In International Conference on Machine Learning, pp. 9399-9409. PMLR, 2021.
Anirvan Sengupta, Cengiz Pehlevan, Mariano Tepper, Alexander Genkin, and Dmitri Chklovskii. Manifold-tiling localized receptive fields are optimal in similarity- prresevii neural networks. Advances in neural information processing systems, 31, 2018.
Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning ad- versarial robustness: A survey. arXiv preprint arXiv:2007.00553 2020.
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Anmual review of neuroscience, 24(1):113311266, 2001.
Nitish Srivastava and Russ R Salakhutdinov. Discriminative transfer learning with tree- brsed priors. Advances in neural information processing systems, 26, 2013.
10

---

Under review as a conference paper at ICLR 2023
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D. Harris. High-dimensional geometry of population responses in visual cortex -SI Nature, 571(7765):
361-365, July 2019. ISSN 0028-0836, 1476-4687. doi: 10.1088641156-099-33346
URL
http:: wwwp//wwwww..mm nar.ur..u..eee.e...e. wwt.a//rticleeee eeseee//555 /tte/4415866 01 019-1111110 019-1346-55
Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. A Global Geometric Framework
for Nonlinear Dimensionality Reduction. Science, 290c5500):2319 2-2323, December 2000. URL https::/ /www
ISSN 9036-8075, 1095-9203. doi: 10.1126/sciencee290.33333 290.::eieeneee2...000... 290.0500..205500211. eccienee... wttp:///g//ooo/ooooo 1126/ wtt///scennce.. 290. 5500 5500.2319)
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustmess may be at odds with accuracy, 2018. URL httt::://arxi..ogg/bbb//////////
12152
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. Hierarchical multi-label classification net- works. In International conference on machine learning, pp. 5075-5084. PMLR, 2018.
Xiu-Shen Wei, Yi-Zhe Song. Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge Belongie. Fine-grained image analysis with deep learning: A survey, IEEE Transac- fions Puttern Analysis and Machine Intelligence, 2021.
Han Xu, Yao Ma Hao-Chen Liu, Dehayan Deb, Hui Liu, Ji-Liang Tang, and Anil K Jain. Adversar- ial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing, 17(2):151-178, 2020.
Greg Yang and Hadi Salman. fine- fine-grained spectral perspective on neural networks. arXiv preprint arXiv: / 1977. 10599, 2019.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DERIVATION FOR LIPSCHITZ BOUND ON THE ISOMETRIC LOSS GRADIENT.
We start be writing down the isometric loss term:
3 N2 |I DM G D
Liso
When taking the gradient we can rewrite it in terms of the distance generating functions d(æ,, zz)) and d()(r;), ((r))) Furthermore for simplicity of notation we will abbreviate (xx,)' to øf. where I is one of the basis components (eg. pixel of an image). Lastly for the purpose of readability, we will omit writing out the element-win multiplication operator for the indexing matrix G. This gives the following:
3G N
VVsiO = V
Σ
Tz) d(oi, 2z))²
(8)
1
By expanding the squared brackets and applying the chain and product rules, the expression be- comes:
28G N
d(z; 2;) Vadkts)))
( + d(o,, &, (9)
rearranging these terms we get:
23G N2
Σ 4
v - V V.d(o;, -100 - v.d(r,,
23G
Σ
[[d(x;, o( - Vzd(oi o;)]}.
N²
(10)
In order to continue it helps to compute the derivatives of the distance functions with respect to the basis in which the input data is described, this gives:
x - d.
vad(I, d(zr,21) S - de(Is) vad(on 2b) d( Ø2)
(11)
Ood 6.
with el being the standard basis components. Substituting these expressions and taking advantage of the symmetry in the distance functions rewrite the expression as:
23G
o of as(x)) x - 2/
Σ ((d(a,,))) d(0i, D
d(oi, Φÿ) Ox ed)
N
d(I;, mz)
2 xt
of 0f d0(x;) dbi, dxd
+(d(s,, d(pr, 0b1)) D
(12)
d(ai,, xz))
23G
of 0
do(z,)
0σ(z;)
(1) d(ór 0;)1
Ox )el
N
d(di bb)
Ox
12

---

Under review as a conference paper at ICLR 2023
At this point it helps to compress notation further. We redefine the difference between the two distance matrices as Ht, = [[) d(o,, Qy), the difference between the mapping at two 0.- and the difference of the derivative at two different points as different points as Ji,j
[dr, Ø,] = V o - v (inspired by the notation used in Lie brackets). Given these new identifi- cations we obtain the final form of the gradient as:
CisC
23G N
Σ ド23
HajJeslor Uz]-
(13)
Given that for an isometry the norm of the gradient V < 1, the difference between the gradients evaluated at any two points is bounded by 2K. With this in mind we arrive the final inequality, showing that the isometric loss gradient is bounded:
23G N
43G N2
Hag o,
Σ HigKigg.
(14)
i>j
13


Reviews:
[{"review_id": "jWbHuMd4Re2", "reviewer": "Reviewer_78Vo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to learn isometric representations in neural networks to preserve the structure in the output end. Then locally isometric layer is proposed to learn the required output representation, which is shown to be robust on MNIST compared with the cross-entropy training baseline.", "summary_of_the_review": "The paper has a major flaw in experiment as mentioned in the Weaknesses, so I tend to reject the paper.", "strength_and_weaknesses": "Strength:\n\n1. The Locally Isometric Layer (LIL) is proposed to avoid the structure-breaking property in neural network classification, which sounds interesting. \n\n2. The proposed method is clearly presented. \n\n\nWeaknesses:\n\n1. The reason why LIL makes the NN more robust is not well-explored. In Section 4.2, there is a discussion on the relationship between Lipschitz constant and the LIL, but no empirical study is provided to compare the performance of LIL to other methods that also reduces Lipschitz constant, such as [1,2,3].\n\n2. The experiment is only carried out with MNIST, which is a quite simple dataset. To show the effectiveness of LIL, the experiment on CIFAR10 is at least needed. \n\n3. The experiment result on MNIST is quite poor if compared with existing benchmarks such as [4]. The robust accuracy under $\\epsilon=0.3$ constraint is over 80%, while Fig. 3 only shows an accuracy of ~20% when $\\epsilon=0.3$, which is a huge gap. Thus, I seriously doubt the effectiveness of LIL.\n\n[1] Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. A topological regularizer for classifiers via persistent homology. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2573–2582. PMLR, 2019.\n\n[2] Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv:1908.02729, 2019\n\n[3] Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. arXiv preprint arXiv:2202.08345, 2022\n\n[4] https://github.com/MadryLab/mnist_challenge", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jL7Ml1RSwc", "reviewer": "Reviewer_RnHw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an isometry aware training approach that improves robustness. The main intuition is that if the latent representations (pre final layer in this case), are (locally) Lipschitz, the network will be robust to adversarial perturbations. The algorithm enforces this Lipschitz constraint using a regularizer which penalizes any deviation in pairwise distance matrices of the outputs and inputs. The paper presents evidence for this in form of experiments on several synthetic datasets (constructed with specific topographic properties) and MNIST. ", "summary_of_the_review": "Overall, the paper needs some additional work -- especially with better experiments -- to be acceptable. While local Lipschitzness does imply robustness, it is really hard to execute for larger datasets. I am willing to change my opinion if the authors show some evidence of their approach succeeding on CIFAR-10.", "strength_and_weaknesses": "**Strengths**\n1. The paper is well motivated with principled arguments about why the locally isometric network should work. \n2. The writing is concise, clear, and well organized.\n\n**Weaknesses**\n\n1. The evaluation is very sparse compared to the standards accepted by the community. Most adversarial robustness papers test their approach on a variety of networks, and datasets. In addition, the attacks used here are not enough to ensure robustness (see Tramer et al (On adaptive attacks to adversarial example defenses), Carlini et al. (On evaluating adversarial robustness) for a more detailed discussion). \n2. In addition, the paper does not present any comparisons with corresponding methods (adversarial training, TRADES, MART) which all present counters to PGD type attacks.\n3. Another issue would be scalability. While the synthetic data and MNIST are mostly well separated, real world image data is both high dimensional and non-separable. My conjecture is that the second term of the loss will lead to significant accuracy losses for CIFAR-10 and Imagenet. I suggest the authors update the paper with these experiments if possible.\n\nMinor concern: While the idea of enforcing local Lipschitzness is inspired, it is not new. Several works [1,2,3] have approached the problem of adversarial robustness by constraining local robustness. The actual implementation differs in each case, however, the overall approach is similar. \nRefs:\n1. Mao et al., Metric Learning for Adversarial Robustness, Neurips 2019\n2. Finlay et al, Lipschitz regularized Deep Neural Networks generalize and are adversarially robust\n3. Yang et al, Adversarial Robustness Through Local Lipschitzness", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a9RisW_Va2L", "reviewer": "Reviewer_aCCN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to combine the existing neural network's loss with an isometry-preserving loss, which they argue would allow for preserving the geometry, which the authors complement with some theoretical results. \n\nThe two main experiments aim to show that the resulting model is more robust to adversarial attacks and leads to better accuracy when the geometry is preserved across layers. \n\nThe paper is somewhat reminiscent of [1], combining similarity-preserving objectives and standard objective function layer-wise. The empirical results nonetheless are very promising but too sparse to be of real value. \n\n[1] Nøkland, Arild, and Lars Hiller Eidnes. \"Training neural networks with local error signals.\" International conference on machine learning. PMLR, 2019.\n", "summary_of_the_review": "The paper is interesting and the current experiments are ok. However, there is a lot that is missing to be able to actually confirm the claims made by the authors. We would like more thorough evaluations of the models. It is too premature to be accepted as a publication given that it is not possible to clearly evaluate the approach. ", "strength_and_weaknesses": "The paper is well written, and the problem is very interesting to the community. The clarity made the paper enjoyable to read. \n\n\nThe paper, however, falls short in various aspects. \n\n1) the experimental evaluations are too sparse to evaluate the models. It is tested on one dataset per task. As mentioned earlier, this paper is reminiscent of that of Nøkland et al. 2019, but not cited in the paper. \n2) the models consider have various parameters, and none of their influences are thoroughly studied. \n3) as mentioned by the authors, competing models exist, yet the current model is not compared to those. \n\nThe idea is clear and interesting, but the claims are not well-supported. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "00bL96nDTxa", "reviewer": "Reviewer_1G3T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to regularize neural networks using isometric properties. The paper presents some intriguing ideas along with some preliminary results that would be worth exploring further.", "summary_of_the_review": "This is a submission with an intriguing idea. The results are premature.\n\n\n**** Edits after the rebuttal:\n\nI appreciate the authors' responses to my critiques.  In its current form, I consider the study to be premature. Performing additional research along the directions the authors raised in their response will likely make the study stronger.", "strength_and_weaknesses": "Strength: The idea is relatively straightforward and intuitive. Some preliminary experimental results are provided. \n\nWeakness:\nHowever, the results in the current form are too premature. There is also a fundamental concern with the premise of the proposed approach.  \nMaking the representation isometric will certainly increase the robustness to perturbation in the input as it directly constrains the Lipschitz continuity. However, this does not touch on the key issue of robustness. Making a neural network robust to image classification tasks requires the network to be sensitive to the on-manifold perturbations (those that change the perceptual properties, e.g., the identity of an object), while being insensitive to the nuisance variability (e.g., increasing the contrast of the stimulus). For the approach proposed by the authors, increasing the contrast of the stimulus or a global scaling of the image would pose a major challenge to the network for solving the image classification task. The preliminary results in Table 1 are consistent with this interpretation and there is a corresponding drop in the performance of the network (as acknowledged by the authors). \n\nOther comments:\nThe second paragraph in the Introduction is a misinterpretation of the existing neuroscience literature. The results of Stringer et al are inconsistent with the classic efficient coding idea. The representation they proposed is closer to the “fractional” representation. Traditional neural population coding models with broadly tuning neurons will create representations that are more consistent with the isometric idea. The paper below seems to contain an additional neuroscience example that might be useful for motivating the study: Gao et al. \"On Path Integration of Grid Cells: Group Representation and Isotropic Scaling.\" Advances in Neural Information Processing Systems 34 (2021): 28623-28635. But they’re dealing with a low-dimensional problem, i.e., physical space.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject