Atomized Deep Learning Models
Authors: Yi-Lin Tuan
Zih-Yun Chiu
William Yang Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deep learning models often tackle the intra-sample structure, such as the order of words in a sentence and pixels in an image, but have not pay much attention to the inter-sample relationship. In this paper, we show that explicitly modeling the inter-sample structure to be more discretized can potentially help model's expressivity. We propose a novel method, Atom Modeling, that can discretize a continuous latent space by drawing an analogy between a data point and an {\it atom}, which is naturally spaced away from other atoms with distances depending on their intra structures. Specifically, we model each data point as an atom composed of electrons, protons, and neutrons and minimize the potential energy caused by the interatomic force among data points. Through experiments with qualitative analysis in our proposed Atom Modeling on synthetic and real datasets, we find that Atom Modeling can improve the performance by maintaining the inter-sample relation and can capture an interpretable intra-sample relation by mapping each component in a data point to electron/proton/neutron.

Under review as a conference paper at ICLR 2023
ATOMIZED DEEP LEARNING MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep learning models often tackle the intra-sample structure, such as the order of words in sentence and pixels in image, but have not pay much attention to the inter-sample relationship. In this paper, we show that explicitly modeling the inter- sample structure be more discretized can potentially help model's expressivity. We propose a novel method, Atom Modeling, that can discretize a continuous latent space by drawing an analogy between a data point and an atom, which is naturally spaced away from other atoms with distances depending on their intra structures. Specifically, we model each data point as an atom composed of electrons, protons, and neutrons and minimize the potential energy caused by the interatomic force among data points. Through experiments with qualitative analysis in our proposed Atom Modeling synthetic and real datasets, we find that Atom Modeling improve the performance by maintaining the inter-sample relation and can capture interpretable antrassample relation by mapping each component in data point to plecrrom/proton/neutroo.
INTRODUCTION
Multiple widely used neural networks are composed of two parts: the first part projects data points into another space, and the other part of the model does further regression/laasiffctionn upon this space. By transforming raw data features another potentially more tractable space, deep learning models have recently shown potential in many areas, ranging from dialogue systems (Vinyals & Le
2015
López et al. [2077] Chen et al.] 2017)), medical image analysis (Kononenko) [001] Ker et al.
2017 Erickson et al. 2017 7 Litjens et al.) 2017 Razzak et al. 2018 Bakator & Radosav 2018 to
roboties (Peters al. 2003
Kober et al. 2013] Pierson & Gashler (007deinnerraa al. 2018).
One major challenge in deep learning is to model better intra- and inter-sample structures for complex data features. Recent works often model the intra-sample structure by considering the order and adjacency of the input features, for instance, positional encoding for texts/speech in Transform- ers Vaawaan et al. 2017) and kernel width for images in convolution neural networks (LeCun et al. 2015). Regarding the inter-sample structure, literature often assumes that a dataset can be represented in continuous space and an interpolation of two embeddings might be meaningful (Bowman et al. 2016 Chen et al., 2016), while the data might be naturally discrete (van den Oord et al., [2017). Moreover, the mainstream relies on a non-fully transparent optimization function to reorganize the space. Therefore, in this work, we would like to explore how to explicitly, dynamically rearrange the space (inter-sample structure) by leveraging the intra-sample structures.
Inspired by Atomic Physics, where atom is the smallest unit of matter and meanwhile discretely distributed, we propose that we can model data point as an atom. As illustrated in the left of Figure an atom in Bohr model (Bohr 1913). an often adopted concept in Physics (Haliday et al.) 2013) and Chemistry (Brown) 2009). contains a dense nucleus, which is composed of the positively charged protons and uncharged neutrons, surrounded by orbiting negatively charged electrons with mucleus radius. Further, multiple atoms can have interatomic forces, composed of attractive and repulsive forces, that make the atoms distant away (non-zero). Such interatomic forces are also the reason for the atoms to form molecules, crystals and metals in our observable life.
In this paper, we propose Atom Modeling, a science- and theoretically- based method that explicitly model the intra-sample relation via atomic structure and the inter-sample relation via interactomic forces. Specifically, we consider data point as an atom and let a model automatically learn the mapping of each component in a data point an electron, proton, or a neutron. We then estimate

---

Under review as a conference paper at ICLR 2023
h())
9
increase s(h)
dynamically distance away the latent space
Figure I:
An illustration our motivation, where a neural network is seen to composed of two groups of functions h(+) and g(-). For a fixed capacity of g(-), e.g., linear function, the left latent space is not able to be separated. If the lower left part is distanced away, the same g(-) can split them.
interatomic forces with the learned subatomic particles, nucleus radius and atomic spacing. Finally, the model is optimized to minimize the potential energy induced by the interatomic forces and maintain the balance total charges and the number of electrons, protons, and neutrons. This method is not only found effective, but also easy to implement in tens of lines for any model architecture.
We validate the effects of Atom Modeling on synthetic data and real data in the domains of text and image classification well as convolutional neural networks and transformers. The empirical results show that Atom Modeling can consistently improve the performance accross data amounts, domains, and output complexity. The analyses demonstrate that Atom Modeling can capture intra- sample structures with interpretable meanings of subatomic particles a data point, while forms an snter-sample structure that increases the model expressivity.
Our contributions are:
We propose to look into the problem of discrete representation in deep learning models.
We propose Atom Modeling, simple method Atomic Physics for machine learning where the distances among data points (inter-sample structure) naturally depend on their intra-sample structure.
We empirically demonstrate that Atom Modeling improve the performance across different setups and provides an interpretable atomic structure of a data point.
MOTIVATION
We motivated by property of the hidden layer of neural network, and the property of the naturally existing atoms.
2.1 DISCRETE REPRESENTATION
A neural network be represented composition of functions f(:) IN a fN-1 o... f.)).
where each function is one of its N layers. When a neural network f(-) is seen as two groups of functions fx) g h(x)), the first half h(-) = fa-1 o... o (-) encodes the input into hidden space and the second half g(-) fN o fN o fn+1(-) transforms the latent into the output space.
o
We consider that in a situation when the model capacity of the second half c(g) is fixed. The projected latent space by the first half functions are hence important to the final output of the model. For easier mathematical description, we denote a quantization of the simplicity of the encoded hidden space as s(h). The whole model capacity e(f) is therefore bounded by: min(s(h), o(g)) S e(f) S max(s(h), e(g))
(1)
Our intuition is that if the distance and the shape of two classes are hard to separate, the simplicity s(h) is small. Then the second half functions with a model capacity equal to a linear function cannot separate them. An example is shown in Figure If the points' distances are larger, they can be split by the same linear function. That is, the space simplicity s(h) can be promoted by a more discrete latent space, one of the bounds of the model capacity c(f) can be improved.
However, only naively increasing the distances among all the points, the space will be unboundly enlarged and the relative positions might not be changed much. A method that focuses more separating points that are nearby but with different properties (c.g.. the lower left side in the two plots of Figure I is desired.
2

---

Under review as a conference paper at ICLR 2023
Atomic Structure
Text
Learned Analogy
orbit
nucieus radius
The movie is fantastic
Atom Modeing
neutron context
proton key context
movie fantastic
electron assistant text
The motivation of Atom Modeling, by which model will automatically learn mapping between the atomic structure (left; Bohr model (Bohr) 1913)) and the features a data point (middle; the tokens in a piece text). A learned analogy (right) can be that less variant main contexts are like the neutrons; key contexts are like the protons; assistant texts that often co-occur with the protons are like the electrons.
Figure 2:
2.2 ATOMIC PHYSICS
Atom is an unit in the nature that composes molecules, crystals, metals in the nature and helps determine their properties (Halliday et al. 2013; Brown 2009). An atom consists three types of subatomic particles: electrons, protons, and neutrons. Among them, an electron has a negative charge, proton has positive charge, and neutron has no charge. In terms of their masses, scientists have empirically measured that an electron has approximately 1/1836 mass of a proton and a proton is slightly lighter than neutron (Mohr al. 2008).
The atomic structure of Bohr model (Bohr 1913) is shown Figure [ where the protons and neutrons form nucleus that occupies small volume of the atom; the electrons orbit around the nucleus with a nucleus radius. Such atoms have two primary types of forces: intraatomic and interatomic forces. The intraatomic forces include the strong nucleus force that forms protons and neutrons into a nucleus, while the interatomic forces bind two atoms and avoid them having zero distance.
The major motivating property of atoms are that the distance between two atoms related to their atomic structures, such the number of the protons and electrons. As shown in Figure 3c the intuition is that only atoms that are already close to each other and have similar structures, e.g., they all positively charged, will be distanced away.
This property can be desirable for us to discretize data representation in neural networks. In this work, (1) assume that there is intraatomic force in the mapping from data point the atomic structure and (2) borrow the idea of the interatomic forces to explicitly model the inter-sample structure.
ATOM MODELING
Our proposed method include two parts: the intra-sample structure modeling and the inter-sample structure regulation.
As illustrated in Figure 2 we regard each component data point as subatomic particle. The mapping from each component to subatomic particle forms the intra-sample structures, and these intra-sample structures are jointly learned with the dynamic distance function among data points, ie., the inter-sample structures. Atom modeling has property to increase the distance among data points with similar intra-sample structures while not affecting data points with different intra-sample structures. We provide this proof in the Appendix.
3.1 INTRA-SAMPLE STRUCTURE
simple atomic structure Bohr 1913) includes three subatomic particles, protons, electrons, and neutrons. This structure has several properties, such charges, masses, mucleus radius and the distances between two particles. We will show that these properties can simply determined by a learnalle charge associated with each component in data point.
3

---

Under review as a conference paper at ICLR 2023
Charges. We assign charge, R, to each component of data point. We set the range of q to be [-1,1] S R since in nature, the charge of the subatomic particles, electron, neutron, and proton, respectively -1, O, and +1. To obtain @ in neural network, we first assume model encodes each component of data point into an /i-dimensional embedding e E IR one of its hidden layer (the n-th layer in section (.T). Then without additional parameters, can transform arbitrary one dimension in e, which is denoted as el E I, into the charge by (e 1, where (.) indicates the Sigmoid function.
Masses. The empirical results in Atomic Physics show that electrons have limited masses while protons and neutrons masses are in a similar level (Mohr al. (008). Following these results, we approximate the mass E R of each component in data point as shown in Figure Bal i... the mass is about one when the charge larger than or equal to O; the mass linearly reduced to zero when the charge is negative. Our analogy of mass is mathematically defined as = 1 max( q. 0).
Nucleus. An atomic nucleus is composed of protons and neutrons, which occupy nearly all the mass of atom. Therefore, we approximate the position of the atomic nucleus by weighted average of each subatomic particle with its mass as the weight. The position of the atomic nucleus μu E h then formulated as yxP |A| Eie where the i-th component of a data point (atom) A, and |A|| is the number of components. eP is the position of each component in the embedding space. Without additional parameters, we take the remaining dimensions of (without Ed) as the position in the embedding space of each component in data point, so eP E h-
Nucleus Radius. Since the outermost particles an atom are electrons, we heuristically approxi- mate the nucleus radius as the average distance from electrons to the nucleus. The nucleus radius r E R is hence formulated as
[ EieA Ile (1- m,) - uP where
-lp denotes p-norm.
Distances. Considering the geometry of atomic structure, as illustrated in Figure 3b we approximate the distances between pair of particles in different atoms (atoms A1, AA2 by d:j 1| p.P 2 if they are homoelectricity (gag; > O, where i E A1 and E A2). If they are heteroelectricity (q:4, < 0), the distance is approximated by dij = ||pP; pP 2lpp + (ri + r2) The nucleus position uP and nucleus radius with subscripts and 2 corresponds atoms Ai and Aa respectively.
Balance of Charges and Number of Particles. Moreover, we follow the findings in Atomic Physics that an atom tends to be electrically neutral for stability. Meanwhile, the number of neutrons is usually about the same as the number of protons. Therefore, we propose two losses, Loharge and Lnpeutrona regularize the charges and number of neutrons.
Leharge
=5 E9)
iEA
(2)
Σ 9² 3 11)2
Linutrons
The idea is that Leharge is the mean square eтror between the total charge and zero, and Ineutrons is an approximation of the total number of charged particles (protons and electrons; E+e ). The charged particles should occupy about 2/3 of the total number of particles in atom (lAA). That is, the remaining 1/3 is the number of neutrons.
3.2
STRUCTURE
After assigning a charge for each component in a data point, we can take advantage of the derived characteristics, i.c., mass, nucleus, radius, and distance, to compute the interatomic forces for building the inter-sample structure.
As illustrated in Figure 3c of two parts: repulsive force, which pushes away two homoelectric particles, and cohesive force, which pulls closer two heteroeletric particles. The combination of the two forces will compose curve of their potential energy with respect to the atom spacing. Such curve has a Balance Point which results in the lowest potential energy and often locates short distance between two atoms but
widely accepted understanding is that the interatomic force is composed
4

---

Under review as a conference paper at ICLR 2023
Mass
11
4 all
Repultdvs Force
Potental Energe
Intentomic Forre
110
Charge
Ochosivo Force
d, 12
allp
r2)/2
(a) The approximated masses of (b) The approximated distances charges.
Distance
particles.
(c) The effect interatomic force.
Figure 3: The atomic analogy of (a) masses and (b) forces; (c) typical resulting potential energy curve with respect to the distance among two atoms induced by the interatomic force, which is combination of repulsive and cohesive forces.
must be larger than zero. When distance closer to zero, the potential energy increases dramatically, thus naturally preventing two atoms become "continuous" (with zero distance).
In Atom Modeling, compute the potential energy caused by Columb Forces (Halliday al. 2013 for randomly sampled particle pairs drawn from two data points (A), and take the summation. That we aim minimize the loss defined as:
by Σ
qolj
de
During implementation, for computational efficiency. we randomly pair components from data points in the same batch to compute Ly. Therefore, the order of complexity does not increase.
3.3 OPTIMIZATION
During training, take weighted of the three losses, b Leharge and Lncutrons- and add them to the original loss function Lors, e.g., cross-entropy loss. The model will then be trained by minimizing the following loss function:
== + Ej Ls + Ceharge Loharge + Cneutrons Lineutrons
empirically observe that the coefficients Cy, Ccharges and Cneatrons do not require careful tuning and can often be the same number.
3.4 THEORETICAL RESULTS
Data points with similar intra-sample structures have longer balance distance; ones with more opposite intra-sample structures can have shorter balance distance. Specifically, the balance distance is proportional to VK+1 where l indicates the difference between the intra-sane aatra-sample structures of two data points and satisfies k >> 1.
Theorem 3.1.
R-1"
We provide this proof in the Appendix.
EXPERIMENTS
In order to validate the effects of Atom Modeling, we conduct extensive experiments across multiple datasets synthetic, text (Warstadt et al. 2019; (Sheng & Uthus 2020 Graßer et al.) 2018) and image domains (Parkhi et al. 2012 Nilsback & Zisserman 2008 Deng et al.) 2009) with two model architectures, transformer (Vaswani et al. 2017 Devlin et al. 2019 and convolution neural network (He et al. 2016). We compare our method with two commonly p-norm with p and 2, as designed baselines, since to the best of our knowledge, we have not seen prior works discuss this type of method. We also answer the question: "what are the learned intra- and inter-structures?" by qualitative analyses in the next section.
distance functions,
5

---

Under review as a conference paper at ICLR 2023
4.1 SYNTHETIC EXPERIMENT
To gain more insights of Atom Modeling, we design the following synthetic experiment. In this way. can visualize the inter-sample structure in the latent space without further projection, which can change the structure.
Setup- Data Generation. Inspired by the complexity of real data, e.g., texts and images, propose synthetic dataset where each data point has multiple components. We design each data point to contain five 2-dimensional vectors, where each vector (input feature) is sampled from Gaussian Mixture Model composed two Gaussian distributions, en and C1. If most of the input features in a data point is sampled from co. we label the data point as O; otherwise, label it The generated dataset is plotted in Figure
The synthetic dataset. The left is the two Gaussian distributions where input features are sampled from; each point in the right is the average of the five input features in each data point.
Figure 4:
Setup Model Architecture. For fair comparison and easy visual- ization, all four methods are trained on the same neural network with two linear layers. The first layer (2x3) is fed with the data points of size (5x2), and its output will be of size (5x3) in hidden space. We then take the first dimension of this output as the weight (5x1) and the rest as the embeddings of the five input features (5x2). Hence, obtain a wwo-dimensional vector. We take this vector as the embedding of a data point and feed it into the second layer (2x2) to classify the data points 0 or 1.
can compute the weighted sum of the five embeddings and
Results. As shown in Figure 3 across 10 random runs, the same model trained with only cross-entropy loss achieves an average 87% accuracy and has a relatively low variance. With the addition of increasing 1-norm or 2-norm distance among data points in the latent space, the highest accuracy can achieve about 90% accuracy. How- ever, since the variances simultancously increased, the
Civst frtrey
+2ram
0.54
0.06
0.00
3.30
0.2
0.44
336
average accuracy is only enhanced bit to 87%-88%. If apply Atom Modeling, the average accuracy is signif- icantly improved to 92% and with a median 96%. Overall, observe that distancing away data point in the latent space can enhance the chance improve the model performance. However, 1-norm and 2-norm distances treat every data points the same, hence the learned data point positions in the latent space can be relatively the same and only the scale is enlarged. In the other hand, Atom Modeling does not treat every data points equally but focuses on increasing the data points that should not be too closed, e.g., points the neighborhood but in different classes.
Figure 5: The testing results of four dif- ferent training methods.
4.2 TEXT CLASSIHCCATION
We validate the effect of Atom Modeling on text classification, which is an often seen real application where the data points composed of multiple, ordered tokens. The number of tokens data points are various.
Setup. We conduct experiments on three datasets with different scale and number of classes, CoLAA (Wang et al. 2018 Warstadr et al. 2019), Poem (Sheng & Uthus 2020). and Drugs (Gräßer ct al. 2018). CoLA set of about 10k sentences annotated with classes about whether it is grammatical English sentence based on Linguistic theory. Poem is corpus about 1k examples for sentiment analysis of classic poems with verses annotated with four classes: negative, no impact, positive, mixed (both negative and positive). Drugs is set of about 100k pieces of reviews and each is diagnosed to be one of 50 conditions, e.g.. "depression" """"ai""" """aa"",""acce""" "acne".
We fine-tune a pretrained language model, BERT- mase-uncasse (Devlin al. 2019), with cross- entropy loss. To implement --norm/2- norm distance regulation on BERT, we take the mean of the n-th layer hidden states of all tokens as the embedding of the data point. To atomize the n-th hidden

---

Under review as a conference paper at ICLR 2023
CoLA
Poem
Drugs
Acc.
MCC
Acc.
F1
Acc.
F1
BERT
83.6=0.4
60.0+1.0
86.7+2.5
60.3:4.5
83.5.1.33
79.5:0.22
+1-norm
83.6=0.3
59.8=0.8
87.3+3.1
62.0#3.2
80.8+1.3
75.7+2.2
+2-norm
83.7=0.4
60.1
87.3#4.0
61.8#3.5
84.5#0.2
80.7+1..3
+Atom 8..1
61.31...
88.6.1.9
62.7.2244
84.7.10.4
80.5
Table I:
Results of text classification. We report the mean and the standard deviation (the value after +) of each evaluation metric for three random runs.
layer of the BERT model (n = 1 here), we utilize the first dimension in the latent space of each token as its charge ea, Therefore 1-normm/-norrm/Aoonn Modeling are implemented upon BERT without extra parameters.
Results. In Table similar to the results of synthetic experiment, observe that in most cases Atom Modeling achieves the best results. The method 2-norm distance has a more on-par performance with Atom Modeling on the Drugs dataset. Oppositely, the 1 1-norm distance method performs similarly or slightly better on CoLA and Poem, but worse than only using cross-entropy on Drugs. One difference from the synthetic experiment is that the utilized transformer architecture does not directly operates on the data points embeddings, but on the components embeddings. In this way, the benefits of Atom Modeling might not be fully leveraged. However, we can still observe the improvements via the jointly learned components embeddings across different number of classes and dataset scales.
4.3 IMAGE CLASSIFICATION
We evaluate Atom Modeling on image classification, which is an often seen real application where the data points are composed of pixels with specific arrangement.
Setup. We conduct experiments on three datasets with different scale and number of classes, Oxford- Pets (Parkhi et al. 2012). Oxford-Flowers 102 (Nilsback & Zisserman 2008), and ImageNet (Deng) ct al. 2009). Dxford-III' Pets consists of 37 cats and dogs species with roughly 200 images for cach class. Oxford Flowers- 102 consists of 102 flower categories. Each class consists of between 40 and 258 images. ImageNet consists of about 1M images in 1000 classes. We resize all the images to 32x32 and apply the same data preprocessing method for these three datasets. Note that the resizing will make the tasks more challenging since some details a image might disappear.
Based on previous implementation of ResNet18 (He et al., 2016), we first concatenate the output channels of the first convolution layer and take each corresponding position of a pixel as its embedding To implement 1-norm/2-norm distance regulation, we take the mean of the pixel embeddings as the embedding of the data point. To atomize this layer in ResNet, we utilize the first dimension of each embedding its charge e9. Therefore 1-norm/2-norm/Atoon Modeling are implemented upon ResNet without extra parameters.
Results. As listed in Table 2 similar the results of both the synthetic experiment and text classification, we observe that Atom Modeling achieves the highest accuracy in all cases. In most cases, the 2-norm distance method slightly improves the results trained with only cross-entropy loss. However, the 1- --norm distance method performs poorly in image classification. We also found that the performance improvements are larger on Oxford-IIIT Pets and Oxford Flowers- 102. We conjecture that this is because these two datasets are more fine- grained classification tasks, since some categories of pets and flowers are not easy to distinguish.
QUANTITATIVE AND QUALITATIVE ANALYSES
As our motivations about the discrete representation and the property atomic physics, we are curious about (1) the impact of the atomized n-th layer, (2) the learned latent space/inter-sample
7

---

Under review as a conference paper at ICLR 2023
Oxford-III Pet
Oxford Flowers- 102
ImageNet
dev
test
dev
test
ResNet
21.58=0.99
20.98=1.111
59.15
56.73
48.07=0.09
+l-norm
20.41=1.40
19.98=11.09
1..20=1.42
54.67
47.76=00.20
-2-morm
22.75=:0.41 23.41:10557
22.39=00.50
60.72++0.54
58.01
47.90=0.20
+Atom
0.87
1..78
59.08
48.23
Table 2: Results of image classification. We report the mean and the standard deviation (the value after =)) of each evaluation metric for three random runs.
Cross-Entropy
+1-norm
+2-nom
+atom
Figure 7: The visualization the learned inter-sample structure of the synthetic data.
structure, and (3) the learned mapping from data point components subatomic particleshintra-sample structures.
5.1 ATOMIZED THE n-TH HIDDEN LAYER
M
The n-th layer (in Section 2.1 split model into the first part that projects data point into the latent space, and the second part that transforms the latent space to the output. Due to the goal of Atom Modeling to reorganize the latent space for the second part of the model can do better transformation, which hidden layer reorganized might affect the final results. In Figure which to apply Atom Modeling in different layer of BERT model CoLA, we observe that discretizing the first layer generally has the best performance. Recall that were using complex model architecture, which does not directly operate on the data point embedding, but the components embeddings. We conjecture that this complexity of the model architecture requires a larger capacity of the second part of the model.
83.5
Figure 6: The results of atom- izing different hidden layer in a BERT model on CoLA.
5.2 VISUALIZATION OF LEARNED
STRUCTURE
After training with different methods, the projection of the same data points to the same latent space will have different relative positions. As plotted in Figure 7 we visualized four random runs of the learned inter-sample structures of models trained with only cross-entropy loss, 1-norm/2-nome norm distance, and Atom Modeling on the synthetic experiment. We found that training with only the cross-entropy loss results in the embeddings entangled in the middle and has long and thin shape. When applying additional I-norm distance, the shape is similar to cross-entropy but the scale of both axes is enlarged about five times. When using 2-norm distance, the shape is more similar to the original dataset. However, there are still multiple overlaps and the scale is enlarged up to ten times. In the end, found that Atom Modeling not only spreads out the distribution be like the original dataset, but also distance away middle ones as our motivation. Meanwhile, the scale is remained about the same the scale learned by cross-entropy.
8

---

Under review as a conference paper at ICLR 2023
told
anything ryinn
water
mgro
appeared
knows
computer
-10
onto
-03
met
03
vigiin
charges
algo
caria
dep
Figure 8: The charges of sampled words in CoLA. The y-axis values randomly generated. Most key contents are automatically learned to be pro- tons, assistant texts are learned to be electrons.
Figure 9: The charges (second row) of figures in Oxford Flowers- 102. The first row and the third row are the original images and the ones multiplied with charges respectively.
5.3 VISUALIZATION OF LEARNED INTRA-SAMPLE S STRUCTURES
In Atom Modeling, the inter- sample structure depends on the learned intra-sample structure. We look into the learned intra- satrle structure in texts and images to have an understanding of how the charges distributed in real world among words and pixels. For texts, we first extract the charges of a token with different context and then take an average. We plot the charges of randomly sampled 15 words of CoLA in Figure 8 We found that in both CoLA and STSB datasets, most often seen words such as "more" and " "also" are learned to be electrons, while protons are often Nouns with specific meanings, such "water" and "violin". This phenomenon validates our expectation that the key contexts are mapped to protons and assistant texts that often co-appear with the key contexts are mapped to electrons. For images, the charges of every pixels are plotted in Figure 9 We observe that the charge distribution is often like a relief sculpture, where electrons are mapped to shadows on found
object edges and protons are the crucial parts for a model classify the image. Moreover, that the charge distribution sometimes recover some missing details in the processed images. For example, in the right most of Figure 9 hidden in the dark.
the charge distribution draws the other petals that originally
6 RELATED WORK AND DISCUSSION
Recent works have tried to connect Physics with machine learning and data science (Raissi et al. 2017
Karpatne et al. 2017| Bar-Sinai et al. 2019 Raissi et al., 2019 Sitzmann al., 2020 Krishnapriyan
ct | | Giladi et al. 2021 Takeishi & Kalousis 2021). For computer vision, Hasani et al. (2021)
and Ding al. 2021) model visual reasoning by considering Physical equations such velocity.
Tang et al. (2021) proposes to split images into amplitude and phase as waves to improve the MLP architecture. For natural language processing, there is a surge of researches on the utility of Quantum Physics in texts (Wu et al. (021). For example, ZZhang et al. (2020) applies Quantum Physics to do sentiment analysis for emotions in dialogues. Atom Modeling ing also explores the potential impact from Nature science to machine learning. However, the goal of Atom Modeling is to approach the question: "how to promote the discrete nature of data points in a deep learning model?"
Some will find similarity between Atom Modeling and regularization (Tibshirani) 1996 Nie et al.
2010, Gulrajani et al. 2017) and kernel methods MMuller et al. 2001 Keerthi & Lin 2003 Hofmann al. 2008). The main differences from them are that first Atom Modeling constrains the underlying data representation rather than the model parameters. Second, Atom Modeling aims to enhance the expressivity of the inter-sample structure in dataset and specifically model the discrete nature in an unchanged dimensional space, instead of mapping from lower to higher dimensional space.
7 CONCLUSION
In this paper, we propose Atom Modeling, a science- and theory-based approach to promote a discrete representation in a deep learning model. Moreover, by drawing an analogy between data and subatomic components, Atom Modeling learns interpretable intra-sample structure for each data point. An important property is that model trained with Atom Modeling learns inter-sample structure (distances among data points) depending on the intra-sample structure. Leveraging this property can potentially improve model capacity. Moving forward, we see that Atom Modeling provides a possible way to improve the current neural network and hope it provides a base for future works to consider more about the discrete nature.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mihalj Bakator and Dragica Radosav. Deep learning and medical diagnosis: review of literature. Multimodal Technologies and Interaction, 2(3):47, 2018.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven
discretizations for partial differential equations. Proceedings of the National Academy of Sciences,
11663))15334-15549 2019.
Niels Bohr. I. on the constitution of atoms and molecules. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 26(151):1-255 1913.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, 2016
Theodore Lawrence Brown. Chemistry: the central science. Pearson Education, 2009.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2)255-55, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 205..25.222 Ieee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dy- namic visual reasoning by learning differentiable physics models from video and language. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 887-898- Curran Asso- cc/ppap//////222//111l ciates, Inc., 2021. URL Papeees.....a222-ap...... Paper. 0
Bradley J Erickson, Panagiotis Korfiatis, Zeynettin Akkus, and Timothy L Kline. Machine learning for medical imaging. Radiographics, 37(2):505-515, 2017.
Niv Giladi, Zvika Ben-Haim, Sella Nevo, Yossi Matias, and Daniel Soudry. Physics- aware downsampling with deep learning for scalable flood modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1378-1389. Curran Asso-
ciates, Inc., 2021. URL ltttps:/ httpps://prooeedissss. neurips wtt httppp////////ape////////// 02255774777d37740011111122266663
pdd
Felix Gräßer, Surya Kallumadi, Hagen Malberg, and Sebastian Zaunseder. Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning. Proceedings of the 2018 International Conference Digital Health, pp. 121-125, 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances neural information processing systems, 30, 2017.
David Halliday, Robert Resnick, and Jearl Walker. Fundamentals of physics. John Wiley & Sons, 2013.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time- constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7657-7666, 2021.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Thomas Hofmann, Bernhard Schölkopf, and Alexander Smola. Kernel methods in machine learning. The annals of statistics,
2008.
Anuj Karpatne, Gowtham Atluri, James H Faghmous, Michael Steinbach, Arindam Banerjee, Auroop Ganguly, Shashi Shekhar, Nagiza Samatova, and Vipin Kumar. Theory-guidee data science: new paradigm for scientific discovery from data. IEEE Transactions on knowledge and data engineering, 291)) 2-2311, 2017.
Sathiya Keerthi and Chih-Jen Lin. Asymptotic behaviors of support vector machines with gaussian kernel. Neural computation, 15(7): (117):6667-6899 2003.
Justin Ker, Lipo Wang, Jai Rao, and Tchoyoson Lim. Deep learning applications in medical image analysis. leee Access, 6:9375-9389, 2017.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 12:11):1 2013.
Igor Kononenko. Machine learning for medical diagnosis: history, state of the art and perspective.
Arlificial Intelligence in medicine, 23(1)89--009 2001.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney, Characterizing possible failure modes in physics-informed neural networks. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 26548-26560, Curran Asso- ciates, Inc., 2021. URL htttttss://proceeeiiiii dF438e5206F3 00026660000000444242222222523333 Pappaaaarr.. .ddf
neur neurips ips cc//paptt//2221/11ee.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553):436 444, 2015.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara Sánchez. A survey on deep learning medical image analysis. Medical image analysis, 42:60-88, 2017.
Gustavo López, Luis Quesada, and Luis A Guerrero. Alexa vs. siri vs. cortana vs. google assistant: a comparison of speech-based natural user interfaces. In International conference on applied human factors and ergonomics, pp. 241-250. Springer, 2017.
Peter Mohr, Barry N Taylor, and David B Newell. Codata recommended values of the fundamental physical constants: 2006. Journal of Physical and Chemical Reference Data, -03)).633-1284 2008.
K-R Muller, Sebastian Mika, Gunnar Ratsch, Koji Tsuda, and Bernhard Scholkopf. An introduction to kernel-based learning algorithms. IEEE transactions on neural networks, 12(2):181-20 2001.
Feiping Nie, Heng Huang, Xiao Cai, and Chris Ding. Efficient and robust feature selection via joint 12, 1-norms minimization. Advances in neural information processing systems, 23, 2010.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Jan Peters, Sethu Vijayakumar, and Stefan Schaal. Reinforcement learning for humanoid robotics. In Proceedings of the third FREE-RS international conference on humanoid robots, pp. 1-20, 2003.
Harry A Pierson and Michael S Gashler. Deep learning in robotics: a review of recent research. Advanced Robotics, 31(16))821-838 2017.
11

---

Under review as a conference paper at ICLR 2023
Maziar Raissi, Paris Perdikaris, and George Em Kariiadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561, 2017.
Maziar Raissi, Paris Perdikaris, and George Kariiadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-7077 2019.
Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. Deep learning for medical image process- ing: Overview, challenges and the future. Classification in BioApps, pp. 323-350, 2018.
Emily Sheng and David C Uthus. Investigating societal biases in a poetry composition system. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pp. 93-106, 2020.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcrofi, Picter Abbeel, Wolfram Burgard, Michael Milford, et al. The limits and potentials of deep learning for robotics. The International journal of robotics research, 37(4-5):405-44 -200000--00000 2018.
Naoya Takeishi and Alexandros Kalousis. Physics-integrated variational autoencoders for robust and interpretable generative modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), ), Advances in Neural Information Processing Systems, volume 34, pp. 14809- Curran Associates, Inc., 2021. URL Utt/tttss: htt.:ooceeeinngg..ur........p..s www.ips. cc/ pdf.
Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is a wave: Phase-aware vision mlp. arXiv preprint arXiv:2111.1294..02 2021.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical
Society: Series B (Methodological), 58(1 (00(1))2:6)) 2:67-2288, 1996.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL /ttpsttp::/pppccoeeeeee/ 9ce2e 6d039 9233222bcaa3399 pddf
neurips cc/ppaprr/2017/111
7a98a 3al.eeee776..ceeeeeeee2233333333
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Oriol Vinyals and Quoc Le. A neural conversational model. arXiv preprint arXiv: 05666666008,,, 2015.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-64) 2019. doi: 10.1 tacl a 00290. URL URL.ttps:: http///aaannclllyorrrr/////
Sixuan Wu, Jian Li, Peng Zhang, and Yue Zhang. Natural language processing meets quantum physies: A survey and categorization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3172-3182, 2021.
Yazhou Zhang, Dawei Song, Xiang Li, Peng Zhang, Panpan Wang, Lu Rong, Guangliang Yu, and Bo Wang. A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis. Information Fusion, 62:14-31, 2020.
12

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREM 3.1
A1A2
1 = Σ
dij
iEA
(5)
Σ
9i9p
+
Σ
9:9
11
122 lp
12
lellp 1
2:gy >0
2:gy
<0
Set d12 = | - H2l1p and
F
2
iL1
En -g9 + +E 1 -9gdd
(6)
0d12
dld + 8)2
Set Egg o0 digj and C2 = Eqigee0 9
Σ -q + Σ qqqdd
44420
444,
(-c1 + o2)
dc, c
(7)
Gτ )² - 0
---c + (d)
)² - (-c1 + 02)
- + C2
C1 + 02
(d
)² of +
02)
72
C1
(8)
01 + 02
(-01 + 02)
(-01 + 0g
C1C2
d = d= + C2
1 V (-a+
(9)
If c2. the right hand side divided by zero is undefined.
C O2 and recall that d || || 2 0, then 01-02
S / (-ort This results in of < C102.
contradict the premise e₁ > C2- This equation can satisfied only when c1 < C2. balance point exists in atvaro f.
Suppose that 02 = kej with t > 1, then,
+ Va12
C2 C1 C1 vke
(10)
Vie + k - ar
(k 1)c1
which is monotonically decreasing given 1.
13


Reviews:
[{"review_id": "wguhT_nvTi", "reviewer": "Reviewer_hSD9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript maps some concepts of the Atom to the hidden space of a hidden output of a NN. The hidden features are transferred to the concept of Atom and regularized with charge, neutrons, and forces as an additional loss. The proposed method is evaluated on several machine learning tasks with respect to some naive baselines.", "summary_of_the_review": "The main concerns are the insufficient evaluation (baselines and weird datasets) and the novelty of the proposed method with respect to many other similar works (disentanglement learning, auto-encoder, and contrastive learning).", "strength_and_weaknesses": "Strength\n\n1 Bringing the concepts of atoms into the neural network is interesting.\n\nWeaknesses\n\n1 The setup of some experiments is strange. For example, the image from ImageNet is resized to 32x32. It does not make much sense since it will make the input much noise and make the comparison unconvincing.\n\n2 The writing can be improved a lot. Many parts of the manuscript are unclear. E.g.\n- What’s the meaning of $\\circ$ in the first line of section 2.1?\n\n- What's the meaning of the simplicity of a model?\n\n- The meaning of k is unclear. What's the meaning of \"difference between the intra-sample structures of two data point\"?\n\n3 The evaluations are quite insufficient. \n- There is no sensitiveness analysis with respect to the three weighting factors of the losses, although the authors claim that they have little influence on the performance.\n\n- There is no comparison with the optimization method along the line of contrastive learning which can also reduce the intra-class distance and enlarge the inter-class distance. The baseline models are too weak to show the effectiveness of the proposed method.\n\n4 There are many related works missing, making the novelty of the proposed method unclear. For example, the regularization of the latent space is quite standard in the field of disentanglement learning, auto-encoder, and contrastive learning. It is unclear what's the difference between the proposed method and these related works. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "tzwo6ZVFSW", "reviewer": "Reviewer_vH4E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces additional loss terms that impose structure on the representation networks learn. The definition of those terms is inspired by atom and interactions between atoms (masses, charges, etc.), and can loosely be thought of as regularizing the representation space by separating the embeddings.\n\nThe authors evaluate their method on several datasets, and show that their approach leads to more meaningful embeddings.", "summary_of_the_review": "Overall, the paper presents a novel and intriguing method for learning relationships within samples and between samples, but the methodology of the paper is unclear in many places, the theoretical grounding is not adequately justified, and the results are not clearly significant enough to demonstrate the effectiveness of the method as a whole.", "strength_and_weaknesses": "Strengths:\n- This is a very novel and creative idea. Applying inspiration from physics to machine learning can lead to impactful methods.\n- The method does not seem to add much complexity or overhead to the training process (though this appears to not be explicitly evaluated).\n\nWeaknesses:\n- The results seem fairly unconvincing. The baselines being compared to appear to have been implemented by the authors themselves, and the dataset choices seem odd. It would be better to start with an externally published and tuned baseline, implement this paper's proposed losses on those models, evaluate on the same datasets, and compare the results. Doing this process for both a recent image and text based model (e.g. ALBERT) seems appropriate.\n- The theoretical grounding and justification for this approach seem shaky and could be made more clear. It's not obvious why all the additional structure around atomic modeling is more useful than other ways of affecting the representation space (for example, disentangled representation learning or contrastive losses). This should also ideally be compared to other modern representation learning techniques. Combined with the results not having externally validated baselines, it's hard to understand the significance of the proposed novelty.\n- The clarity of the paper could be improved significantly, especially since this paper introduces many equations and approximations. See comments in the section below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8WQ6VNIcaJ", "reviewer": "Reviewer_KJc9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper post-processes the representation of deep neural networks from a physical perspective. This process considers not only the relationship between different components within a sample point, but also interaction between sample pairs. The proposed method:\n\n- promotes the discrete nature of sample points in a deep learning model.\n- explicitly models the feature interaction between sample pairs.\n- is demonstrated to improve the capacity of deep models.\n", "summary_of_the_review": "Overall I think this paper provides an interesting perspective on feature post-processing for neural networks that considers both intra-sample and inter-sample feature interactions. However there is much room for improvement in the writing of the article, see the Weakness section above. I therefore tend to give a reject.", "strength_and_weaknesses": "**[Strength]**\n\n- The modeling of feature interactions from an atomic perspective is novel.\n- Considering cross-sample interactions complements the current single-input-single-output mainstream neural networks.\n- The proposed module has been generally validated on different models and datasets.\n\n**[Weaknesses]**\n\n- The discussion of related work is very inadequate. There are many existing fields that consider inter-sample feature interactions (e.g., contrastive learning), but the authors do not expand on them.\n- Figure 1 shows a good motivation -- considering the distance relationship between **different sample points** makes sense. This drives us to implement feature interactions across data samples. However, using Atom Modeling to do this is not as intuitive and, for example, not as clean as Supervised Contrastive Learning [R1]. I don't think Atom Modeling is well motivated from Figure 1, and the description of this section may need to be rewritten.\n- The experimental configuration on image classification does not match the mainstream research. In most recent work evaluated on Pets, Flowers or ImageNet datasets, the image resolution is typically set to 224 and their performance far exceeds that of this paper. These facts make the conclusions of this work on the image classification task less convincing.\n\n-----------\n[R1] Khosla, Prannay, et al. \"Supervised contrastive learning.\" Advances in Neural Information Processing Systems 33 (2020): 18661-18673.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8-4azmG_wP", "reviewer": "Reviewer_qaGR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a very interesting view of deep latent space. The authors introduce a idea inspired by phisics, and propose a new model regularization method. The new regularization method promotes distance between data points based on they intra-sample properties learned from the task. The idea is validated on synthesized toy experiments and real-world classification task.", "summary_of_the_review": "My main concerns are:\n\n1. This paper fails to position itself under a more general context. Therefore it is hard to evaluate the contribution. \n\n2. The motivations behind extending atomic physics to real-world data modalities remain unclear to me. \n\n3. Experiments are weak. ", "strength_and_weaknesses": "**Strengh**\n\n1. The paper is generally well-organized, and clearly presented\n\n2. The background of atomic physics is concise and helpful. \n\n\n**Weakness and questions**\n\n1.  In the abstract, the claim '*have not pay much attention to the inter-sample relationship*' seems even wrong to me. For example, research on graph neural networks is mainly on sample-level representations. And there is extensive research on regularizing the deep feature space based on intra-sample or inter-class properties. \n\n2. *we show that explicitly modeling the intersample structure to be more discretized can potentially help model’s expressivity*. I cannot see a very clear motivation for why a more discretized feature space can help improve model expressiveness. My main concern is based on the fact that well atomic physics is a well-established research, it does not necessarily apply to all data modalities that deep neural networks are dealing with. And I fail to find sufficient discussions on the motivation behind extending atomic physics to data we use in deep learning. In short, I do not see a clear connection between atom-level physics and modalities like images and text. \n\n3. The previous concern is also confirmed by the experimental results. In my opinion, most of the performance improvements on all the real-world datasets cannot be considered significant. \n\n4. This paper fails to position itself under a more general context. The method is only compared against some simple alternatives like 1-norm 2-norm. There are potentially more studies on feature geometry that can be included. For example, center loss [1] and orthogonal low-rank geometry [2]. \n\n5. The experiment settings, such as 32*32 ImageNet classification are not standard. If it is due to hardware constraints, then standard low-resolution experiments like CIFAR-10 and CIFAR-100 can be more convincing.  \n\n6. I personally do not recommend calling this method *science- and theory-based*. It is a science-inspired method. But the theory provided in this paper does not support the effectiveness. For example, the proof in Theorem 3.1 describes the mathematical properties of the new regularization term, but does not explain why it should work on real-world data. \n\n[1] A Discriminative Feature Learning Approach for Deep Face Recognition, ECCV 2016\n\n[2] OLE: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning, CVPR 2018", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3ytMZ4nDTg", "reviewer": "Reviewer_vXfj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to structure the latent space of a deep neural network following the Bohr model of atoms. In this latent space, each datapoint is to be embedded as an 'atom', i.e. a set of particles represented by their position and their charge.\nUsing these values, the authors can then compute losses to make sure the atoms respect physical constraints. Inside an atom, the sum of the charges of all particles is pushed towards 0, while the sum of squared charges is pushed towards 2/3 of the number of particles. Moreover, in this latent space, pairs of atoms are moved so that they minimize their potential energy, making sure that atoms of similar structures cannot stay too close to each other.\nThese losses are added to the original classification loss to solve the task.\n", "summary_of_the_review": "The proposed approach is suprising and interesting. It could be very well suited to embed sets and to discover structure within those sets.\nHowever, it is also difficult to build an intuition of its inner workings from the provided results.\n\nConsidering how complex and difficult to analyze the proposed approach is, it needs a strong motivation to justify, as well as a extensive evaluation of the properties we can find in the latent space.\n", "strength_and_weaknesses": "Strengths:\n- The learned embeddings seem to have interesting properties. It could be interesting to embed sets and discover their inner structure.\n- Experiments have been conducted in very different settings in terms of data (toy dataset, image classification, text classification) or architecture (MLP, ConvNet, Transformer)\n- The descriptions are clear and straightforward\n\nWeaknesses:\n- One important point that might not have enough emphasis is that the model is targeted to data that could be represented by a set of embeddings. Images as collections of pixel embeddings, or sentences as a set of word embeddings. Making it clear early on would help readability.\n- The motivation is also not really convincing. The main goal seems to be that embeddings shouldn't lie too close one to another so that a decision frontier is easy to find. I would question the soundness of the assumption (what about generalization then for instance?), but even when taking it at face value, the proposed solution seems like a very contrived and unpractical way to solve this specific problem.\n- Given the peculiar constraint on the latent space, its analysis is insufficient. We would need more than a few qualitative samples to understand what is happening. For instance, what kind of properties of the data do they hope will be captured in the intra-atom structure, and conversely, in the inter-atom relationship? We can infer some answers from the experiments, but they need to be set beforehand and explained in the introduction, and then explicited in the results.\n- Why are the image experiments performed in 32x32? It seems far to small in for any reasonnable benchmark.\n- Many geometrical properties and emergent spatial structures work in a 3-dimensional space and but break in higher dimensions. However, in the proposed model, the atoms are in a large (h-1)-dimensional space. The illustrative experiments they perform are in 2-D space as well. Have the authors considered this limit to their analogy?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject