Closed-Form Interpretation of Neural Network Latent Spaces with Symbolic Gradients

## Abstract

Abstract It has been demonstrated in many scientific fields that artificial neural networks like autoencoders or Siamese networks encode meaningful concepts in their latent spaces.
However, there does not exist a comprehensive framework for retrieving this information in a human-readable form without prior knowledge.
In order to extract these concepts, we introduce a framework for finding closed-form interpretations of neurons in latent spaces of artificial neural networks.
The interpretation framework is based on embedding trained neural networks into an equivalence class of functions that encode the same concept. We interpret these neural networks by finding an intersection between the equivalence class and human-readable equations defined by a symbolic search space.
The approach is demonstrated by retrieving invariants of matrices and conserved quantities of dynamical systems from latent spaces of Siamese neural networks.

## 1 Introduction

The current AI revolution is driven by artificial neural networks (ANNs), particularly deep learning models. These models have enabled machines to achieve superhuman performance in a variety of tasks, such as image recognition, language translation, game playing, and even generating human-like text. However, this remarkable power comes at the expense of interpretability, often referred to as the ”black box” problem. The representational capacity of artificial neural networks relies on interactions between possibly billions of neurons. While each single neuron is easy to describe mathematically, as networks become larger, it becomes increasingly difficult to understand how these interactions give rise to a neural network’s overall prediction.

The black-box nature of neural networks can be acceptable in applications where prediction is the primary goal. However, in science, where the goal is not just prediction but also understanding the underlying phenomena, interpretability is crucial. Moreover, in medicine, it is important to understand why an AI system has made a particular diagnosis or treatment recommendation to avoid risks of dangerous or ethically questionable decisions Jin et al. (); Amann et al. (). AI interpretability in the law domain is crucial for understanding and explaining how automated decisions are made, which helps ensure transparency and accountability. It also allows for the identification and correction of biases, compliance with regulations, and maintains the integrity of legal processes Hacker et al. (); Bibal et al. ().

In many scientific applications of neural networks, it can be verified that neural networks often learn meaningful concepts, similar to those that humans use, to describe certain phenomena Ha and Jeong (); Desai and Strachan (); Nautrup et al. () . Unfortunately, without a method to distill this learned concept in a human-interpretable form, the only way to reveal it is by directly comparing it to a set of candidates that the researcher is already aware of. Clearly, it is not possible to make new discoveries in this way.

To address this problem, symbolic regression techniques have been proposed to interpret neural networks by deriving closed-form expressions that represent the underlying concepts learned by these networks Cranmer et al. (); Mengel et al. (). These approaches involve exploring the space of potential mathematical expressions to identify those that best replicate the predictions of a neural network. Unfortunately, such methods are limited to interpreting output neurons of neural networks performing regression, where the concept that is recovered is the global function learned by the neural network.

Neural networks applied to perform scientific discovery are often tasked with solving problems that cannot be formulated under the umbrella of regression. Further, it is often necessary to interpret a simpler sub-concept encoded in hidden layers. For these reasons, it is desirable to have a framework capable of interpreting concepts encoded in arbitrary intermediate neurons of artificial neural networks.

Prominent artificial scientific discovery methods are based on networks like autoencoders Wetzel (); Iten et al. (); Miles et al. (); Frohnert and van Nieuwenburg () or Siamese networks Wetzel et al. (); Patel et al. (); Han et al. (). These networks can conceive meaningful concepts inside their latent spaces without explicit training information in the form of labeled targets. The crucial obstacle is the lack of tools that enable the recovery of such concepts without prior knowledge of them. Removing this bottleneck would hence allow scientists the discovery of potentially unknown scientific insights.

In this publication, we describe a framework that can be employed to interpret any single neuron within an artificial neural network in closed form. We demonstrate the power of our framework by rediscovering the explicit formulas of matrix invariants and conserved quantities from the latent spaces of Siamese networks.

Concepts encoded in neurons in hidden layers are in general not stored in a human-readable form, but they get distorted and transformed in a highly non-linear fashion. Hence, the interpretation method is based on constructing an equivalence class around a certain neuron that contains all functions encoding the same concept as the target neuron. In practice, we interpret the neuron by searching a closed-form representative function contained in this equivalence class.

## 2 Related Work

The current manuscript contributes to the domain of artificial neural network interpretability with the focus of enabling new scientific discovery through latent space models. Almost all of the neural network interpretability research works in science manage to confirm or deny whether neural networks learn certain known scientific concepts. While verifying a neural network is an important task, it is unsuitable for gaining novel scientific insights. There has been limited progress toward revealing scientific insights in symbolic form from artificial neural networks that do not require previous knowledge of the underlying concept beforehand Wetzel and Scherzer (); Cranmer et al. (); Miles et al. (); Liu and Tegmark (). These cases are rare examples where the underlying concept is encoded in a linear manner or other properties of the concept simplify the interpretation problem.
While there are no unified approaches to interpreting latent space models, it might in principle be possible to build such models based on architectures with symbolic layers Martius and Lampert (); Sahoo et al. (); Dugan et al. (); Liu et al. ()

Our article aims to interpret existing latent space models. We extend an interpretation framework Wetzel () originally developed to interpret neural network classifiers to interpret neural network latent spaces.

The interpretation method relies on efficiently searching the space of symbolic equations which can be achieved by genetic search algorithms which form the backend of many symbolic regression algorithms. These include Eureqa Schmidt and Lipson (), Operon C++Burlacu et al. (), PySINDy Kaptanoglu et al. (), FeynBroløs et al. (), Gene-pool Optimal Mixing Evolutionary Algorithm Virgolin et al. (), GPLearn Stephens () and PySRCranmer (). Other symbolic regression algorithms include deep symbolic regression uses recurrent neural networks Petersen et al. (), symbolic regression with transformersKamienny et al. (); Biggio et al. () or AI Feynman Udrescu and Tegmark ().

An overview of interpretable scientific discovery with symbolic Regression can be found inMakke and Chawla (); Angelis et al. ().

Figure: Figure 1: (a) Most existing approaches attempt to interpret a neural network latent space by comparing the latent with known candidate concepts. In this case, it is necessary to have the correct concept at hand, which is unsustainable for scientific discovery. (b) Our method requires only a dataset and a trained neural network to be used in conjunction with a symbolic search algorithm, which then discovers a closed-form expression describing the concept encoded in the network’s latent space.
Refer to caption: x1.png

## 3 Method

### 3.1 Siamese Neural Networks

Siamese neural networks (SNN) Baldi and Chauvin (); Bromley et al. () were originally introduced to solve fingerprint recognition and signature verification problems. SNNs consist of two identical sub-networks with shared parameters, each receiving distinct inputs which are then projected to an embedding space. These projections are then compared by a distance metric, which joins the sub-networks together at their output. Inputs belonging to the same class should obtain high similarity, while those belonging to different classes should obtain low similarity. Such a framework allows for generalization to infinite-class classification problems. The distance metric $d(\cdot)$ is chosen according to the specific problem at hand, and in our case we use the squared Euclidean distance.

The network $f$ can be trained effectively using a contrastive or triplet loss Schroff et al. (), wherein a set of triplets are supplied to the energy function,

$$ $\displaystyle\mathcal{L}(A,P,N)=\max(d(f(A),f(P))-d(f(A),f(N))+\alpha,0).$ $$

The anchor $A$ is the ground truth class, the positive sample $P$ is of the same class as $A$, whereas the negative sample $N$ is of a different class. Instead of using a twin network, this setup requires a triplet of identical networks, each still sharing the same weights. The triplet loss is minimized when the distance between the anchor and positive sample is minimized in the embedding space, while the distance between the anchor and negative sample is maximized. The margin parameter $\alpha$ is a positive constant which encourages separation between positive and negative samples, as $\alpha=0$ would mean that the loss could be trivially minimized by projecting all samples to the same location. Finally, the $\max(\cdot)$ operation ensures that the distance between positive and negative samples remains finite.

It has been shown that in scientific settings SNNs can be trained to learn conserved quantities and symmetry invariants of the underlying system. For this purpose, training data is collected where data points belonging to the same class are defined through a connection via trajectories obeying laws of motion (conserved quantities) or a desired symmetry group (symmetry invariants) Wetzel et al. ().

The architecture of the sub-network $f$ depends on the underlying data. In our case, we implement $f$ as a fully-connected network.

### 3.2 Interpretation Framework

A neural network which is trained to perform a task requiring knowledge of some concept $g$ does not necessarily encode it in the same, human-readable format that we use. Instead, it is more likely that it learns some function $f$, which is related to $g$ by some highly nonlinear and uninterpretable transformation $\phi$,

$$ $\displaystyle f(\mathbf{x})=\underbrace{\phi}_{\text{uninterpretable transformation}}(\underbrace{g(\mathbf{x})}_{\text{closed form concept}}).$ (1) $$

Different realizations of neural networks $f$ might learn the same concept $g$ and therefore contain the same information. More formally, these realizations are all members of the following equivalence class:

$$ $\displaystyle\tilde{H}_{g}=\left\{f(\mathbf{x})\in C^{1}(D\subset\mathbb{R}^{n },\mathbb{R})\mid\exists\text{ invertible }\phi\in C^{1}(\mathbb{R},\mathbb{R} ):f(\mathbf{x})=\phi(g(\mathbf{x}))\right\}.$ (2) $$

While each network uses a unique invertible transformation $\phi$, they are functionally equivalent in that they learn the same concept from the data. At this point, we ask the question, whether it is possible to identify the concept $g$ without knowing the function $\phi$.

$$ $\displaystyle g(\mathbf{x})=\phi^{-1}\left(f(\mathbf{x})\right).$ (3) $$

In order to avoid the necessity of knowing $\phi$, we rewrite the equivalence class (2) such that membership can be defined without explicit information about $\phi$. Since all $f\in\tilde{H}_{g}$ are required to be continuously differentiable, we can show that the gradients of the two functions $f$ and $g$ point in the same direction,

$$ $\displaystyle\nabla f(\mathbf{x})=\phi^{\prime}(g(\mathbf{x}))\cdot\nabla g( \mathbf{x})\quad\text{where}\quad\|\phi^{\prime}(g(\mathbf{x}))\|>0.$ (4) $$

Here we used that $\phi$, by construction, is invertible. Since $\phi^{\prime}(g(\mathbf{x}))$ is merely a scaling factor, this equation allows us to define a new equivalence class $H_{g}\subseteq\tilde{H}_{g}$,

$$ $\displaystyle H_{g}=\left\{f(\mathbf{x})\in C^{1}(D\subset\mathbb{R}^{n}, \mathbb{R})|\frac{\nabla f(\mathbf{x})}{\left\|\nabla f(\mathbf{x})\right\|}= \frac{\nabla g(\mathbf{x})}{\left\|\nabla g(\mathbf{x})\right\|}\vee\nabla f( \mathbf{x})=\nabla g(\mathbf{x})=0,\forall\mathbf{x}\in\mathcal{D}\right\}.$ (5) $$

In the next section, we discuss how to obtain a closed-form expression which is a member of (5).

### 3.3 Symbolic Search

Symbolic regression is a regression analysis technique that has traditionally been used to find closed-form expressions that approximate the relation between target and input variables for a given dataset. Typically, this is done by employing a genetic algorithm, which evolves a population of candidate formulas using genetic operations like selection, crossover, and mutation, aiming to find the least complex tree of operators $T$ that maps inputs $X$ to outputs $Y$. In the context of neural network interpretation, symbolic regression is employed to convert a complex model into an interpretable tree representation.

In our case, we search for a simple tree $T$ which represents a function $g\in H_{g}$. However, instead of performing regression on a set of prediction targets to find the best fitting function, we search for an analytical expression whose normalized gradients are as close as possible to those of $f$. Because of this difference, we refer to this approach as symbolic search instead of symbolic regression. Note that this requires that $T$ consists of operators that yield a differentiable function.

The objective function we choose is the mean-squared-error (MSE), which measures the distance between the normalized gradients $g_{T}(\mathbf{x})=\frac{\nabla T(\mathbf{x})}{\|\nabla T(\mathbf{x})\|}$, and $g_{f}(\mathbf{x})=\frac{\nabla f(\mathbf{x})}{\|\nabla f(\mathbf{x})\|}$,

$$ $\displaystyle\text{MSE}(g_{T}(X),g_{f}(X))=\frac{1}{n}\sum_{i=1}^{n}\left\|g_{ T}(\mathbf{x}_{i})-g_{f}(\mathbf{x}_{i})\right\|^{2}.$ (6) $$

### 3.4 Algorithms

Implementing our framework involves three main algorithms which summarize the preceding sections:

- 1.
Train the model $F_{\theta}$ to learn the invariant. See algorithm 1.
- 2.
Choose a neuron to interpret. This neuron computes $f_{\theta^{\prime}}(\mathbf{x})$, where $\theta^{\prime}\subseteq\theta$. Treat this as the latent model and compute its gradient with respect to the input, i.e., $\nabla_{\mathbf{x}}f_{\theta^{\prime}}(\mathbf{x})$. See algorithm 2.
- 3.
Apply symbolic search to find a symbolic tree $T$ whose gradients point in the same direction as $f_{\theta}^{\prime}$. See algorithm 3.

Figure: ALGORITHM 1 Training a Siamese Neural Network to Learn an Invariant

Figure: ALGORITHM 2 Extracting the Gradients from the Siamese Network

Figure: ALGORITHM 3 Symbolic Search

## 4 Experiments

**Table 1: Matrix Invariants**
| Exp. No. | Name | $d$ | Transformation | Invariant | Analytical Expression | Retrieved Expression |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | $2\times 2$ | 4 | Similarity Transformation | Trace | $A_{11}+A_{22}$ | $\frac{A_{11}+A_{22}}{-0.878}$ |
| 2 |  |  |  | Determinant | $A_{11}A_{22}-A_{12}A_{21}$ | $A_{12}A_{21}-A_{11}A_{22}$ |
| 3 | $3\times 3$ | 9 | Similarity Transformation | Trace | $A_{11}+A_{22}+A_{33}$ | $A_{11}+A_{22}+A_{33}$ |
| 4 | $3\times 3$ Antisymmetric |  |  | Sum of Principal Minors | $A_{12}^{2}+A_{23}^{2}+A_{13}^{2}$ | $A_{12}A_{21}+A_{23}A_{32}+A_{13}A_{31}$ |
| 5 | $4\times 4$ | 16 | Similarity Transformation | Trace | $A_{11}+A_{22}+A_{33}+A_{44}$ | $A_{11}+A_{22}+A_{33}+A_{44}$ |
| 6 |  | 6 | Lorentz Transformation | Inner Product | $E_{1}B_{1}+E_{2}B_{2}+E_{3}B_{3}$ | $E_{1}B_{1}+E_{2}B_{2}+E_{3}B_{3}$ |

**Table 2: Potentials**
| Experiment No. | $d$ | Potential $V$ | Invariant | Analytical Expression | Retrieved Expression |
| --- | --- | --- | --- | --- | --- |
| 7 | 1 | $\frac{1}{2}x^{2}$ | Energy | $\frac{1}{2}v^{2}+\frac{1}{2}x^{2}$ | $v^{2}+x^{2}$ |
| 8 |  | $\sin(x)$ |  | $\frac{1}{2}v^{2}+\sin(x)$ | $v^{2}+\sin(x)+\sin(x)$ |
| 9 |  | $\frac{1}{2}x^{2}+\frac{1}{4}x^{4}$ |  | $\frac{1}{2}v^{2}+\frac{1}{2}x^{2}+\frac{1}{4}x^{4}$ | $v\cdot v+x\cdot x+0.513(x\cdot x)^{2}$ |
| 10 |  | $\frac{1}{2}x^{2}+\exp(x+1)$ |  | $\frac{1}{2}v^{2}+\frac{1}{2}x^{2}+\exp(x+1)$ | $\text{square}(v)+x\cdot x+\exp(x+1.684)$ |
| 11 | 2 | $-r^{-2}$ | Angular Momentum | $x_{1}v_{2}-x_{2}v_{1}$ | $x_{2}v_{1}-x_{1}v_{2}$ |

**Table 3: Spacetime**
| Experiment No. | $d$ | Transformation | Invariant | Analytical Expression | Retrieved Expression |
| --- | --- | --- | --- | --- | --- |
| 12 | 4 | Lorentz Transformation | Spacetime Interval | $t^{2}-x_{1}^{2}-x_{2}^{2}-x_{3}^{2}$ | $x_{1}\cdot x_{1}+\text{square}(x_{2})-((t\cdot t)-\text{square}(x_{3}))$ |

We retrieve the invariants of matrices and various physical systems using our method. We consider invariants of matrices under similarity and Lorentz transformations. Additionally, we investigate dynamical systems characterized by a variety of potentials, as well as the invariants in Minkowski spacetime.

### 4.1 Datasets and Training

#### 4.1.1 Invariants Under the Similarity Transformation

In experiments 1-3, 5 in Table 1, we search for the trace and determinant of matrices under the similarity transformation. Each data point is a triplet consisting of three matrices of dimension $n$: an anchor matrix $A$, a positive example $P$, and a negative example $N$. The anchor is sampled by generating a random matrix.

The positive example shares one or more invariants with the anchor. In the case of the similarity transformation, these invariants are the trace and determinant. To this end, we sample a $n\times n$ invertible matrix $M$ and apply the similarity transformation $P=MAM^{-1}$. The negative example should not share invariants with the anchor, which is trivially achieved by sampling another matrix $N$, which is almost certainly characterized by different invariants.

In practice, we find that the neural network prefers to learn the trace. To discover a second invariant, such as the determinant, we sample triplets in which all matrices have the same trace. The network can no longer rely on the trace to identify similar matrices, or to distinguish between dissimilar ones, as the trace no longer provides any useful information for this task. Instead, an alternative invariant must be learned, which in this case is the determinant. This can be done for any number of invariants: upon discovery of the first one, it can be made constant across the entire dataset to force the neural network to learn another.

#### 4.1.2 Invariants of Antisymmetric Matrices

For antisymmetric matrices in experiment 4, we prepare our dataset in the same way as we describe in 4.1.1. We first sample an antisymmetric $3\times 3$ matrix for the anchor $A$, followed by a similarity transformation for the positive sample $P$. Finally, we sample a new antisymmetric matrix for the negative sample $N$. While both the anchor and negative samples are antisymmetric, the positive sample does not inherit this property under the transformation $P=MAM^{-1}$ when $M$ is not orthonormal, because antisymmetry is not preserved under a general change of basis. Hence, we use all 9 entries of the matrix as input, although we acknowledge that one could easily enforce that $M$ is orthonormal, in which case only 3 inputs would be needed from each of $A$, $P$, and $N$.

We attempt to retrieve (8), which is invariant under the similarity transformation. Since we use the antisymmetric anchor matrix $A$ as input when computing $\nabla_{\mathbf{x}}f(\mathbf{x})$, we expect that the result of symbolic search would simplify to (10).

#### 4.1.3 Invariants Under the Lorentz Transformation

In experiment 6, we apply the Lorentz transformation to the field strength tensor $F_{\mu\nu}$, which gives rise to the Lorentz invariants X and Y. Since the antisymmetry of $F_{\mu\nu}$ is preserved under the Lorentz transformation, each member of a triplet is antisymmetric, so we only use the 6 off-diagonal entries above (or equivalently below) the main diagonal as our input to the neural network. The anchor is a vector of these 6 entries from $F_{\mu\nu}$,

#### 4.1.4 Potentials

The experiments in Table 2 correspond to motion in a potential, where we simulate trajectories by randomly sampling initial positions and velocities, and subsequently evolve these systems according to Hamilton’s equations.
For each triplet $(x_{A},x_{P},x_{N})$, the anchor $x_{A}$ and positive sample $x_{P}$ are measurements at two different points along the same trajectory, while the negative sample $x_{N}$ is sampled from a different trajectory. The network must determine whether or not two measurements belong to the same particle.

#### 4.1.5 Spacetime

In experiment 12 in Table 3, each triplet again consists of an anchor $x_{A}$, a positive sample $x_{P}$, and a negative sample $x_{N}$. The anchor is a randomly sampled four-vector representing an event in Minkowski spacetime. The positive sample is generated by applying a Lorentz transformation to the anchor, ensuring that the spacetime interval remains invariant. The negative sample, on the other hand, is another randomly generated four-vector that does not share the same spacetime interval as the anchor, allowing the neural network to distinguish between vectors that do and do not preserve this invariant.

### 4.2 Results

Figure: Figure 2: The latent space encodings of Siamese neural network applied to different data sets are compared with the corresponding ground truth concept for each data point. In all cases, it is possible to see a clear correlation. However, this correlation is mostly non-linear causing other interpretation methods to fail.
Refer to caption: x2.png

We summarize the results of our experiments in Tables 1–3. For each experiment, we use the method outlined in section 3 to obtain a set of predicted expressions, which we present as a Pareto plot in Figure 4 and Figure 3. We identify the solution that most closely matches the correct expression, denoted as retrieved expression in tables 1–3. It is interesting to note that the correct expression is often the one corresponding to the steepest drop in the loss. We note that in some cases, the network learned a polynomial approximation to the desired expression, which we rectified by increasing the sampling range used to produce the dataset.

All obtained solutions match the correct expressions. It is possible for the symbolic search algorithm to instead return a solution that matches the ground truth one up to a piecewise invertible transformation, although we do not observe this in our experiments. We also note that the symbolic search algorithm may approximate the correct solution, or add simplifications to it. For example, the solution denoted by the striped green bar in Figure 4 (c) a uses $\exp(x_{1}\cdot x_{1})\approx 1+x^{2}+\frac{x^{4}}{2}$, which matches the correct solution up to the fourth order in $x$.
In Figure 4 (d), the expression $2\exp(x+1)$ was simplified to $\exp(x+1+\ln(2))\approx exp(x+1.684)$.

Figure: Figure 3: The Pareto front for experiments involving conserved quantities, summarizing the results of the symbolic gradient-based interpretation framework to find a candidate concept that is contained in the corresponding neural network latent space. Several possible equations are plotted in order of decreasing Mean Square Error (MSE) and increasing complexity. The red bar indicates the candidate that resembles the ground truth concept, which is often found at the point of steepest change of the Pareto front.
Refer to caption: x3.png

Furthermore, because the network’s latent layer consists of only a single neuron, we can directly compare the value it encodes for inputs $\mathbf{X}$ to the true underlying concept $g(\mathbf{X})$. We plot these quantities against each other in Figure 2. Note that these correlation plots are not a necessary component of our interpretation framework. We use them only to highlight the non-linear manner in which the neural network encodes the concept. In most experiments, the values encoded in the latent space are highly correlated with some well-known concept. In fact, the correlation plots for the trace are almost linear, which is expected as they can trivially be learned by a single-layer neural network with no non-linearities. In such cases, it may be possible to use other methods to interpret the neural network. However, most invariants are significantly more complex, and the neural network will encode them in a non-linear manner, in which case most other interpretation methods will fail.

Figure: Figure 4: The Pareto front for experiments involving matrices, summarizing the results of the symbolic gradient-based interpretation framework to find a candidate concept that is contained in the corresponding neural network latent space. Several possible equations are plotted in order of decreasing Mean Square Error (MSE) and increasing complexity. The red bar indicates the candidate that resembles the ground truth concept, which is often found at the point of steepest change of the Pareto front. The striped green bar denotes a solution that approximates the correct one up to the fourth order.
Refer to caption: x4.png

## 5 Conclusions

In this manuscript, we develop a framework to interpret any single neuron in neural network latent spaces in the form of a symbolic equation. It is based on employing symbolic search to find a symbolic tree that exhibits the same normalized gradients as the examined latent space neuron. We justify this procedure by defining an equivalence class of functions encoding the same concept, in which the membership criterion is that all members have parallel gradients at every point on the data manifold. Through this procedure, we enable the extraction of concepts encoded by latent space models.

We demonstrate the power of our approach by interpreting Siamese networks tasked with discovering invariants of matrices and conserved quantities of dynamical systems. We are able to uncover the correct equations in all of our examples. It is important to note that the symbolic search algorithm sometimes made clever approximations. For example, the anharmonic potential was summarized by an exponential function whose Taylor expansion agrees to fourth order in $x$. Further, the approach simplified expressions, for example, the term $2\exp(x+1)$ was transformed into $\exp(x+1+\ln(2))\approx\exp(x+1.684)$.

It is impossible to compare our results to other methods because our approach is the only general method that allows for the extraction of concepts encoded in latent spaces in closed form. As we have seen, sometimes the latent space encodings are approximately linearly correlated with the human-readable ground truth concept. In these cases, it is possible to retrieve the expression with traditional symbolic or polynomial regression. However, this is not the general case. It is important to note that there might exist publication bias towards linear encodings, since non-linear encodings cause traditional interpretations to fail.

The pathways to scientific understanding via interpretable machine learning might lead down different roads. On one side there are inherently interpretable ML models, like PCA or support vector machines. On the other side, there are powerful artificial neural networks, which are difficult to interpret. Further, there is a middle ground implementing layers resembling symbolic calculations inside artificial neural networks. Until recently, none of these approaches was able to truly discover human-readable concepts from latent space models. We hope that through our approach many scientists will understand the potential discoveries that their latent space models might make.

## Appendix A Preliminaries

### A.1 Matrix Invariants

#### A.1.1 Invariants of Rank-Two Tensors Under Similarity Transformations

For any $n\times n$ matrix $A$, we can compute the similarity transform $B=MAM^{-1}$, where $M$ is any invertible matrix. Using the cyclic property of trace,

$$ $\displaystyle\text{tr}(B)=\text{tr}(MAM^{-1})=\text{tr}(AM^{-1}M)=\text{tr}(A).$ $$

Furthermore,

$$ $\displaystyle\det(B)=\det(MAM^{-1})=\det(M)\det(A)\frac{1}{\det(M)}=\det(A).$ $$

Hence, both the trace and determinant are invariant under this basis change. It is straightforward to see that the following expression, called the sum of principle minors, is also basis-invariant:

$$ $\displaystyle\text{tr}(\text{tr}(A)^{2}-\text{tr}(A^{2})).$ $$

Together, these three comprise the principal invariants of rank-two tensors:

$$ $\displaystyle I_{1}=\text{tr}(A),$ (7) $\displaystyle I_{2}=\text{tr}(\text{tr}(A)^{2}-\text{tr}(A^{2})),$ (8) $\displaystyle I_{3}=\det(A).$ (9) $$

#### A.1.2 3 × 3 3 3 3\times 3 3 × 3 Antisymmetric Matrices

In the case of antisymmetric $n\times n$ matrices of odd size, the number of principal invariants reduces to one. The trace of an antisymmetric matrix is 0 0, so $I_{1}=\text{tr}(A)=0$, and any antisymmetric square matrix of odd size $n$ must have at least one zero-eigenvalue, so $I_{3}=\det(A)=0$.
We treat the case of a $3\times 3$ antisymmetric matrix, in which case $I_{2}$ can be written in terms of its entries as,

$$ $\displaystyle I_{2}=A_{11}A_{22}+A_{22}A_{33}+A_{11}A_{33}-A_{12}A_{21}-A_{23} A_{32}-A_{13}A_{31}.$ $$

Since the diagonal elements of $A$ are 0 0 and $A_{ij}=-A_{ji}$, the expression for $I_{2}$ is simplified:

$$ $\displaystyle I_{2}=A_{12}^{2}+A_{23}^{2}+A_{13}^{2}.$ (10) $$

#### A.1.3 Invariants of the Field Strength Tensor Under the Lorentz Transformation

Under Lorentz transformations, the invariants of the electromagnetic field strength tensor $F_{\mu\nu}$ are preserved. The tensor $F_{\mu\nu}$ is antisymmetric, meaning $F_{\mu\nu}=-F_{\nu\mu}$. Its invariants include the scalar $\mathbf{B}\cdot\mathbf{E}$ and the quantity $\frac{1}{2}F_{\mu\nu}F^{\mu\nu}$. Specifically,

$$ $\mathbf{B}\cdot\mathbf{E}=\det(F_{\mu\nu})$ $$

and

$$ $\left|\mathbf{B}\right|^{2}-\left|\mathbf{E}\right|^{2}=\frac{1}{2}F_{\mu\nu}F ^{\mu\nu}.$ $$

These invariants highlight the consistency of electromagnetic properties across different inertial frames.

### A.2 Space-Time Interval

In special relativity, Minkowski spacetime is a four-dimensional continuum that combines three spatial dimensions with one time dimension. This framework allows for a unified description of space and time, where events are described by four coordinates $(t,x,y,z)$, and the separation between events is invariant under Lorentz transformations. The distance between events in this spacetime is determined by the Minkowski metric, which is defined by the metric tensor,

$$ $\displaystyle\eta_{\mu\nu}=\text{diag}(-1,1,1,1)$ $$

This metric defines a scalar product for any two four-vectors $x$ and $y$ as,

$$ $\displaystyle\langle x,y\rangle=\eta_{\mu\nu}x^{\mu}y^{\nu}=x^{\mu}y_{\mu}$ $$

where $x^{\mu}$ and $y^{\mu}$ are the components of the four-vectors $x$ and $y$. The spacetime interval $s$, which remains constant under Lorentz transformations, is given by,

$$ $\displaystyle\langle x,x\rangle=-t^{2}+x^{2}+y^{2}+z^{2}=s^{2}$ $$

The Lorentz group, which consists of transformations that preserve this scalar product in Minkowski spacetime, is denoted as,

$$ $\displaystyle O(3,1)=\left\{\Lambda\in M(\mathbb{R}^{4})\mid\langle\Lambda x, \Lambda y\rangle=\langle x,y\rangle,\forall x,y\in\mathbb{R}^{4}\right\}$ $$

### A.3 Dynamical Systems

In dynamical systems involving motion in a potential, conservation principles are fundamental. In one-dimensional systems, energy is invariant, meaning the total energy—comprising both kinetic and potential components—remains constant in an isolated system. In two-dimensional systems, both energy and angular momentum are conserved, provided the potential is central (i.e., depends only on the radial distance). These invariants are crucial for understanding and modeling dynamical behaviors in both 1D and 2D contexts.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: I would appreciate if the authors could address the questions I raised in the Weaknesses section, at least by adding explanation and clarification, but preferably by further experiments.
soundness: 3
strengths: To identify what a neuron does in a trained network and, therefore, open the "black box" of neural networks is of significant importance. Specifically, The ability to express the internal operation of networks by standard mathematical formulas in general cases is highly desirable.     The method proposed is relatively straightforward to understand, and represents a notable attempt to address this issue.
summary: The authors proposed a framework to find mathematical interpretations of what a neuron does in mathematical equations readable to humans.  It relied on searching for an equivalence class of functions that encodes the same concept as the target neuron, and a symbolic search process guided by minimizing the difference in gradients. Experiments were conducted to show the proposed method can indeed identify the invariant mathematical operations among various inputs.
weaknesses: 1. The tests were to identify relatively straightforward invariance and express it with mathematical symbols. However, it is not clear if real tasks completed by neural networks, e.g., recognition of a complex object or making a decision in dynamic environments, can be understood in that way. It is not clearly explained how the proposed method can be used in more complex tasks. 
2. Only one specific network structure was tested experimentally, without showing that it could work for other networks.
3. In the experiments, a single neuron that carries out the complete network output was analyzed, which did not demonstrate effectiveness in more complex situations.  For example, if we want to understand what a hidden layer containing multiple neurons does, how could this be done with the proposed method? More generally, it is more important to understand what is the operation carried out by a population of neurons, rather than by a single neuron.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Questions on correctness: 

I’ve checked the maths carefully, and it seems to me that the model assumptions are conflated with the model properties. 
If we closely follow section 3.2, Eq.1 states that the model more likely learns the function as a composition of an uninterpretable transformation function $\phi$ and a closed form concept function $g$. The authors do not support that it actually the case. Then imagine that the function $g$ which maps inputs $x$, $x’$ into the same point, $g(x)  = g(x’)$. Then $\phi(g(x))=\phi(g(x’)$. But the end-to-end learnt counterpart of function $h$, with certain parameterisation, may learn to differentiate between the two. Therefore, it should be clearly stated that the authors assume that the function $h(x) = \phi(g(x))$, not that it’s likely the case.

Then the authors change the notation again, saying that actually one is dealing with a special type of  $h$, which is a neural network $f$. It looks confusing. Instead I understand that one can just define it in the very beginning that they assume that the neural network $f$ can be approximately decomposed into $f(x) \approx \phi(g(x))$ (this will be a kind of a concept bottleneck model, as per (Koh et al, 2020)). 

Then the authors say that they “ can show that the gradients of the two functions f and g point in the same direction”, but it is in no way obvious from the text that $\phi’ (g(x)$ would indeed be non-negative (or is it another assumption?). Furthermore, I am not sure what is $\phi’$? It is also unclear how the equivalence class from Eq. 5 folds into the rest of the narrative.

Questions on reproducibility and clarity:

The experimental section looks entirely unclear. I understand that the authors might assume that it would follow from the released code, but the paper needs to be self-sufficient in explanation and reproducibility and at least define the experimental setting and what should the readers refer to to reproduce the results. 

The authors state in line 233 that they are using genetic algorithm, but it is unclear where one can find a settings of such algorithm and what particular genetic algorithm they are referring to. There is no citation for the genetic algorithm either. There is no description of the neural network the authors use, nor there is any clarity about the dataset. The dataset should be described as well, with the links to the dataset and the procedure for reproducing the setting, also I cannot see in the text what are the input data the authors are using. Also, it does not relate to the claim that the authors are able to interpret the latent-space neurone as it is unclear while neurones are being interpreted.  

Also, is it possible to present any like-for-like comparison with other symbolic learning methods, perhaps Cranmer et al, 2020 or other related work aiming at symbolic interpretability?

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang, Concept Bottleneck Models, ICML 2020
soundness: 2
strengths: - Significance: The symbolic interpretation is an important aspect of interpreting  opaque neural network representations.
summary: The paper proposes a novel method for finding closed-form interpretation for closed-form expression interpretation of neurons in the artificial neural networks. The authors demonstrate the method’s efficiency on retrieving matrix invariants  and conserved quantities of dynamic systems within the latent spaces.
weaknesses: - Motivation: the authors should clearly state what the proposed interpretation aims for and how, by whom and for what scenarios it could be used.
 - reproducibility and clarity: see the questions below, in summary, the authors need to significantly change the text to reach the necessary standard of clarity
- correctness: see the questions below
- novelty: it is important to highlight how the proposed solution contrasts with the other symbolic interpretation methods such as 	Cranmer et al., 2020; Mengel et al., 2023, and why these methods cannot achieve such interpretation

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the "weaknesses" section. 

Besides, it is clear that the proposed method is not applicable or scalable to all neural networks. Therefore, I would like the paper to explicitly discuss what types of neural networks and latent spaces it is able or unable to handle.
soundness: 2
strengths: 1. **Closed-form interpretation**. it is interesting that the proposed method produces closed-form interpretation for neurons. 

2. **Good performance on transformations**. the empirical results on similarity and Lorentz transformations are reasonable.

3. **Straightforward method**. The gradient-based method is easy to comprehend, despite some issues in the theory part.
summary: The paper presents a framework to interpret latent spaces of neural networks. The approach addresses the challenge of extracting human-readable concepts encoded in neurons within neural networks, particularly autoencoders and Siamese networks. The method constructs an equivalence class of functions that represent the same concept (under some assumptions) and retrieves interpretable mathematical expressions using symbolic search. It demonstrates its effectiveness in recovering invariants of matrices and conserved quantities under similarity and Lorentz transformations.
weaknesses: 1. **Interpretation limited to scalar outputs**. The proposed method assumes $f(\boldsymbol{x})$ and $g(\boldsymbol{x})$ to be in $C^{1}(\mathbb{R}^{n},\mathbb{R})$. This could limit the method's ability to interpret high-dimensional latent spaces, where multi-dimensional vector-valued functions are more common. In such cases, while the proposed method could be used to interpret each element in the output vector separately, this could (1) lose relations between elements, and (2) be inefficient.

2. **Issue with equivalence between $\tilde{H}_g$ and $H_g$**. The current assumption that $g \in C^{1}(\mathbb{R},\mathbb{R})$ does not fully account for cases where $\phi'(g(x))$ is negative. In such scenarios, the gradient directions of $f$ and $g$ may point in opposite directions. Therefore, there may exist functions in $\tilde{H}_g$ but not in $H_g$. 

3. **Class equivalence does not ensure correct interpretation**. Even if the issue with $\phi$ is fixed, $\tilde{H}_g = H_g$ only implies that the class of ground truth function of interest $\tilde{H}_g$ and the class of gradient-based interpretation $H_g$ are the same. This does not guarantee that the gradient-based method can recover the exact underlying $g(\boldsymbol{x})$. In practice, especially in complex latent spaces such as those used in visual tasks, one specific output could be explained by various combinations of input vectors. The proposed method could converge on an approximation or an equivalent function, but not necessarily the exact underlying concept.

4. **Scalability of symbolic searching**. Genetic algorithms are known to scale poorly as the complexity of the search space increases. 
 Also, there is no empirical study in the paper showing that the proposed method works for the latent space of complicated tasks, such as VAE-based image generation. 

5. (minor) **Assumption of $f$ and $g$ being continuously differentiable**. This may or may not hold for all neural networks. For example, networks with ReLU activations are not continuously differentiable. It is unclear to me how this will affect the behaviour of the proposed method.

Meta-review: The papers takes up on a *very* interesting problem that I personally feel quite close to, that is how to interpret neural network latent spaces with symbolic equations. I believe all reviewers are intrigued, however, given the novelty of this problem, there is a gap between what is claimed and what is demonstrated. This was also reaffirmed in the discussions after the rebuttal, where it was pointed out that perhaps it would be better if the claims would be restricted to the physical equation discovery, with possible addition of the impact statement which shows how the method can be expanded to other areas. This is something that I also agree with. There was also a remark on that the problem with interpreting network subpopulation is not addressed as the authors only hypothesise that it could be done through disentanglement but do not demonstrate this.

As a side note, and regarding the scalability, you could perhaps take into account recent works on mechanistic neural networks that also learn symbolic expressions in latent spaces, and are computationally efficient (Pervez et al., Mechanistic Neural Networks for Scientific Machine Learning, ICML 2025).

Decision: Reject