Local Identifiability of Deep ReLU Neural Networks: the Theory

Abstract: Is a sample rich enough to determine, at least locally, the parameters of a neural network? To answer this question, we introduce a new local parameterization of a given deep ReLU neural network by fixing the values of some of its weights. This allows us to define local lifting operators whose inverses are charts of a smooth manifold of a high dimensional space. The function implemented by the deep ReLU neural network composes the local lifting with a linear operator which depends on the sample. We derive from this convenient representation a geometrical necessary and sufficient condition of local identifiability. Looking at tangent spaces, the geometrical condition provides: 1/ a sharp and testable necessary condition of identifiability and 2/ a sharp and testable sufficient condition of local identifiability. The validity of the conditions can be tested numerically using backpropagation and matrix rank computations.

Reviews and author responses:

 Thank you for increasing your score. We appreciate it.

**On changes in the manuscript**

Indeed, we will add a simplified version of our responses on related works and paper motivation to the Introduction, especially the discussion in Response 1.b. We have posted a comment entitled "*Summary of changes in the manuscript*" to summarize all the changes that we will make, following the discussions with you and the other two reviewers.


**On Response 2.a**

Step 1 consists in studying the invariants $\sim$ associated to the activation function $\sigma$, satisfying $\theta \sim \theta' \Rightarrow f_\theta = f_{\theta'}$. These invariants prevent indentifiability even when the entire network function $f_{\theta}$ is observed, thus a fortiori they prevent indentifiability from a finite sample, when $f_{\theta}(X)$ is observed. We focus more specifically on the local invariants, for which there exist $\theta,\theta'$ arbitrarily close to each other such that $\theta \neq \theta'$ and $\theta \sim \theta'$. 
To remove the local invariants, we construct a reparameterization $\rho_\theta: \mathbb{R}^D \rightarrow \mathbb{R}^E \times \mathbb{R}^B \ , \ \tau \mapsto \rho_\theta(\tau)$, which restricts the parameter space, that is $D \leq |E| + |B|$. In the ReLU case, $D = |F_\theta| + |B|$, as in the definition (9) of $\rho_\theta$ in Section 3. This reparameterization satisfies for any $\tau, \tau'$, 
$$ \rho_\theta(\tau) \sim \rho_\theta(\tau')\quad  \Longrightarrow \quad \tau = \tau'.$$

We propose here two classical examples of activation functions other than ReLU, and give an idea of what Step 1 could be in those cases.

* For $\tanh$, two well-known invariants are the permutation of neurons (which is common to all feedforward networks and does not depend on the activation $\sigma$) and sign-flips, that consist in changing simultaneously the sign of all inward weights, outward weights and bias of a hidden neuron (this invariant is specific to odd activation functions, such as $\tanh$). The results in [2, 10, 38] in our bibliography indicate that these are the only generic invariants for $\tanh$. Contrary to the rescalings in the case of ReLU, none of these two invariants are local: for almost all parameters $\theta \in \mathbb{R}^E \times \mathbb{R}^B$, there exists a small ball $B(\theta, \epsilon)$ around $\theta$ such that any parameter $\theta' \neq \theta$ obtained applying permutations and sign-flips to $\theta$ falls outside of $B(\theta, \epsilon)$. Since we are working locally around $\theta$, these invariants can be avoided and in that case, no restriction in the parameters would thus be needed. In other words, we would have $D = |E| + |B|$ and the reparameterization $\rho_\theta$ would be trivial: $\rho_\theta(\tau) = \tau$.

* For leaky ReLU (that is $\sigma(x) = x$ if $x \geq 0$ and $\sigma(x) = ax$ if $x < 0$, with $0 < a < 1$) we have the same invariants as for ReLU: the permutations and rescalings. In that case, Step 1 could be similar as for ReLU, using the same reparameterization $\rho_\theta$ as in our article. 

**On Response 3.b-c**

For Response 3.b-c, we just realized that there was a typo in our response. The sentence 

"*In particular, for simplicity when the network has only one **input**, the right-hand-side of the equation in Proposition 1 is a scalar product, which is interpreted as a sum over paths in the network graph*" 

should have been

"*In particular, for simplicity when the network has only one **output**, the right-hand-side of the equation in Proposition 1 is a scalar product, which is interpreted as a sum over paths in the network graph*". 

We are sorry about this typo. What we meant is that the right-hand-side of the equation in Proposition 1 is a product between
a line vector (left) and a matrix (right), with
dimensions $(1 \times \mathcal{P}) \times (\mathcal{P} \times V_L)$. When there is only one output, then $V_L = 1$ and we obtain simply a scalar product of dimension $(1 \times \mathcal{P}) \times (\mathcal{P} \times 1)$. Considering the case $V_L = 1$ was just a way for us to have an easier-to-read discussion in our response (we hope that it would indeed had been easier to read if there had not been the typo). The theory in the paper holds for general values of $V_L$.

Hence, in summary, the theory in the paper holds for general numbers of inputs and outputs.

Regarding the linear representation (using the lifting operator), this is not an assumption we make on the neural networks considered in the paper. This is a tool for analyzing them. The outputs of the ReLU network functions considered are non-linear with respect to the parameters $\theta$, even locally around a fixed $\theta_0$. For instance we can see at the bottom of page 3 that there are products between the values of $\theta$ corresponding to different layers (with also applications of the ReLU function). 

We do hope that these two responses on your comments on Response 3.b-c are appropriate. 

 Thank you for your response. We agree and we will make this suggested change. At the end of Section 2 we will formally define local identifiability, as in Theorem 7(i). In Theorem 7, it will be explicitly stated that (i) is the definition of local identifiability (with a similar formulation as in Theorems 8 and 9).

 Thank you for your response. Indeed, in the next version of the manuscript, we will include all these elements.
We have posted a comment entitled "*Summary of changes in the manuscript*" to summarize all the changes that we will make, following the discussions with you and the other two reviewers.

 We agree with the discussion of the distinction between local and global identifiability. We remark that we will clarify this distinction in the manuscript.
We have posted a comment entitled *Summary of changes in the manuscript* to summarize all the changes that we will make, following the discussions with you and the other two reviewers.

 Again, we thank the three reviewers for their comments and suggestions.
Following the author-reviewer discussion, we list here the main changes to the manuscript that we are happy to make.

* Section 1.1 (context and motivation) will define local and global identifiability more explicitly and explain their distinction. It will also define stability and inverse stability more clearly, as discussed in a response to Reviewer jUtV. It will also include the comments on overparametrization given in a response to Reviewer Pq5e. Additionally, it will provide motivations on robustness and reproducibility, as discussed in a response to Reviewer jUtV.

* Section 1.2 (existing work) will make the shortcomings of the existing literature more explicit: no testable identifiability conditions for any finite sample $X$ and for deep neural networks. We will also explain better that existing stability and stable recovery results are also local or make unrealistic assumptions, as discussed in a response to Reviewer Pq5e. In general, we will make the description of existing results more precise, as discussed in a response to Reviewer jUtV. Finally, we will discuss the reference [SII21], as stated in a response to Reviewer JEdY.

* In Section 2, we will explain why $\phi$ is called a lifting operator. We will also announce the goal of reaching the linear representation of Proposition 1 at the beginning. This way, we will better explain and motivate the notations (in particular the graph-related ones), as discussed in responses to Reviewers jUtV and JEdY. 

* At the end of Section 2 we will add the formal definition of local identifiability that is used in this article, and which corresponds to Theorem 7(i). We hope this will make the general reading easier.

* At the beginning of Section 3, we will add an explanation on the role of tangent spaces and why it is thus useful to prove that $\Sigma_1^*$ is a smooth manifold, motivating better the section.

* We will clarify the idea of fixing parameters to prevent rescaling, as discussed in the responses to Reviewers Pq5e and jUtV.
 
* It will be explicitly stated that Theorem 7(i) is the definition of local identifiability.

* We will move Figure 2 to Section 1.3 as suggested by Reviewer jUtV. 


 The reviewer appreciates point-to-point responses from the authors! Here are some follow-ups.

1. The supplementary explanation of related works and paper motivation is clear. The reviewer suggested adding a simplified version in the Introduction and related works, especially the discussion in Response 1. b.

2. For Response 2. a, can the authors explain more specifically about the activation in Step 1? For example, making one example of another common activation function and illustrating what remains the same and what is different in sufficient conditions will be helpful.

3. For Responses 3. b - c, I understand that linear representation (using features/lifting operator $\phi$) and a single input are convenient and appropriate for theoretical analysis. Based on the authors' explanation, will the extension to more general forms be an extended work? Or, can the authors include some discussions (e.g., a paragraph after the problem definition or at the end of the main part) about how to generalize the theory?

Based on the responses, the reviewer would like to increase the score.

 I noticed that two of the reviews seemed to miss the distinction between (global) identifiability and local identifiability. Global identifiability says that the parameters that produce the function behavior on inputs in X are unique (up to equivalence) whereas local identifiability says only that there exists an arbitrarily small neighborhood around the parameters where the parameters are unique. (See Theorem 7(i) where the definition of local identifiability appears in the paper.)

For example, the summary from Reviewer jUtV says “The main focus is the uniqueness of parameters in a ReLU-activated NN. Proofs are provided to support the conditions of such uniqueness.”, and the summary from Reviewer JEdY says “This study investigates the identifiability of deep ReLU nets, that is, the existence of a tractable inverse map from finite input-response examples {(x,f_\theta(x))} to the parameter \theta, and presents necessary and sufficient conditions of the identifiability.”

This paper presents a local identifiability result. (More specifically, the sufficient condition is for local identifiability. The necessary condition is necessary for local identifiability but this implies it is also necessary for global identifiability, since global identifiability implies local identifiability.)

This is an important distinction (proving global identifiability is a stronger theoretical result than proving local identifiability). Global identifiability gives a result about uniqueness and would be required for the result to apply to the motivations around privacy/security whereas local identifiability is more of a result about the geometry of the relationship between parameter space and function space.

 Also regarding making the definition of local identifiability more clear, in the original submission the definition appears in Theorem 7(i), but it isn't labeled explicitly as being the definition of local identifiability. It would improve clarity if you labeled explicitly that part (i) of that theorem is the definition of local identifiability and that part (ii) is the condition you are defining and showing is equivalent. You allude to this in the text above the statement of Theorem 7, but it's not explicit. (This might seem obvious to the authors after working on this paper for a long time, but it was hard to understand this as a reader and it's a key result of the paper.)

 Thank you to the authors for the thorough response to all three reviews and the answers to my questions. You already mention that you plan to include some of the information in this response in the paper, but in particular I think the paper would benefit from including:
- The comments on overparameterization
- The additional comments on motivation and where this paper falls in the literature. What you wrote in the paragraphs starting with "There exists a literature studying theoretically the identifiability..." and ending with "...although it is a local result" were worthwhile motivations.
- A short explanation of why \phi is called a lifting operator
- The clarification about the idea of fixing some parameters to prevent rescaling

 Dear reviewers,

We would like to thank you again for the time spent writing such constructive reviews, which raised some very interesting questions that we were glad to discuss.

Please feel free to let us know if our responses address your concerns. Given that the author-reviewer discussion period will end shortly, we would be happy to answer any additionnal question or to clarify any point of the rebuttal in the remaining time.

 Thank you for the time spent reading the paper, for your overall positive feedback and for making us aware of the reference [SII21], which is definitely related to the paper. We have done our best to provide appropriate responses below.

**About difficulty to follow notations**

Following this comment and also comments from Reviewer jUtV, we will add explanations on the notations to the paper. On the right-hand-side of the equation in Proposition 1, we will indeed explain that it corresponds to $\sum_p \alpha_p(x,\theta) \phi_{p,v} (\theta)$. We will explain that this corresponds to summing over paths in the graph of the neural network, and we will in general discuss and motivate better the use of graphs and paths, also following a comment from Reviewer jUtV. 

**About numerical examples**

Numerical experiments are ongoing. The question of ghosts is interesting (see the answer in next section) and the testable conditions provided in this paper should allow to study them numerically. 

We agree with the remarks regarding the lack of numerical examples. Unfortunately, the paging of these experiments lengthens even more an article of already 38 pages, and we chose to develop the numerical aspect in a separate, thorough study. Several problematics are to be explored: the efficient implementation of the conditions (and particularly $C_N$), the sharpness of the conditions $C_N$ and $C_S$, how to choose the samples to identify a given network, how many samples are needed in practice, the influence of depth/width, the question of ghosts, etc.

**On the null space and the reference [SII21]**

Thank you again for pointing out this reference. It is very interesting and seems indeed linked to the question of identifiability. We will include it in our literature review as a counter-example to identifiability. There are positive results of identifiability, for instance for ReLU networks with one hidden layer, which are well understood, as described in [28].

Let us consider your example when $W_2 \sigma( W_1 x + b) = 0$ for all $x$, characterize it with [28] and then see how our results deal with it. It will be easier if we denote by $a_i$ the lines of $W_1$ and by $d_i$ the columns of $W_2$, so that 
$$
W_2 \sigma( W_1 x + b) = \sum_{i} d_i \sigma(a_i x + b_i).
$$
The results in [28] show that for each hidden neuron $i$, there are only two possibilities:
* the unit $x \mapsto d_i \sigma (a_i x + b_i )$ is a trivially degenerate one satisfying $a_i = 0$ or $d_i = 0$, 
* the unit is cancelled by other units $j$ which have an alignment with it: there exists $\lambda \in \mathbb{R}$ such that $(a_j, b_j) = \lambda ( a_i, b_i )$.

In other words, a network with no degenerate units and no particular alignment between two units implements a nonzero function. 

Coming back to our present work with this characterization, to say it shortly, the degenerate cases are contained in the set $S$ which has Lebesgue measure zero and that we exclude early without loss of generality. When it comes to aligned units, two such units $v_i$ and $v_j$ would have same activation patterns, that is $a_{v_i}(x, \theta) = a_{v_j}(x, \theta)$ for all $x$ (see the beginning of section 2.2), and that would prevent the operators $\alpha(X,\theta)$ and $\Gamma(X,\theta)$ from having full rank. The sufficient condition $C_S$ would not be satisfied.

For deep networks, less trivial ghosts can appear, for instance the case $x \mapsto W_3 \sigma(W_2 \sigma(W_1 x + b_1) + b_2 )$ when $W_2$ and $b_2$ have only zero or negative coefficients, and arbitrary values for the remaining parameters. Our test could allow to detect the presence of ghosts and better understand this phenomenon.

There is certainly more to say on the matter, but we hope to have provided some insight to how such cases would be dealt with.


 *(continuation of the previous comment)*

**f.** Fixing the weights is not a restriction we put on the neural networks that we study, or on our identifiability results. In particular, our definition of identifiability in i) in Theorem 7 does not consider any fixed weights. Instead, fixing some weights is a mathematical tool both for stating some of the identifiability conditions and for the proofs. 

The high-level idea is that for each $\theta$, we construct a local reparametrization of the neural network output vector function around $\theta$. This reparameterization removes redundancy in the parameters due to the scaling invariance. It enables to provide an atlas for the smooth manifold $\Sigma^*_1$.

More precisely, for each fixed parameter vector $\theta$,
we consider a sufficiently small ball around $\theta$. Let us call, say, $D$ the dimension of $\theta$. Then we consider the function giving the neural network output $u \mapsto f_{\theta + u}(X)$, for $u$ close to zero. Local identifiability can be interpreted as studying this function around $u=0$. 

Let us remark that we can select $N_1+ \dots + N_{L-1}$ scalar scaling parameters (each in a neighborhood of $1$), and use them to "rescale" $\theta + u$ as in Definition 3, leaving $f_{\theta+u}(X)$ unchanged. Locally and intuitively, this means that there are $N_1+ \dots + N_{L-1}$ linear combinations of $u$ which leave $f_{\theta+u}(X)$ invariant, which makes the study of $u \mapsto f_{\theta + u}(X)$ more complex. It is thus convenient to remove these degrees of freedom for $u$ that leave $f_{\theta+u}(X)$ unchanged. We do this by fixing $N_1+ \dots + N_{L-1}$ components of $u$ to $0$ as indicated in (8) (this is what we call fixing weights). We show that this gives the same local image space for $u \mapsto f_{\theta + u}(X)$ around $0$, with only $D - N_1- \dots - N_{L-1}$ components of $u$ that are free to move. 

Also, combining Theorem 9 and Proposition 10 shows that a sufficient condition for local identifiability is that the Jacobian matrix of  $u \mapsto f_{\theta + u}(X)$ at $0$ is full column rank, where only $D - N_1- \dots - N_{L-1}$ components of $u$ that are free to move. 
Hence, fixing $N_1+ \dots +N_{L-1}$ components of $u$ around $0$ (equivalently $N_1+ \dots +N_{L-1}$ components of $\theta$) indeed provides a nice sufficient condition for local identifiability. 

We will add the above explanations to the paper.
We will also insist more clearly on the fact that for each $\theta$, there are different components that are fixed.
We will highlight that this mathematical technique of fixing weights does not put any limitations on the neural network functions that are studied.  

**g.** We will define the term affine space formally.
An affine space is a set of the form {$u + v; v \in V$}, where $V$ is a vector space (composed of vectors or matrices in the paper) and $u$ is a fixed element of $V$. We will also define "Ker", which is indeed the kernel. That is, if $M$ is a linear map (or matrix) from a linear space $U$ to a linear space $V$, then $\mathrm{Ker}M =$ {$u \in U ; Mu = 0$}. 

**h.** We are grateful for the comment on Figure 2. We will move it to Section 1.3, as suggested.

**Conclusion of the responses** 

We thank you again for the comments. As mentioned along them, they will enable us to improve the exposition in the paper. 
We are happy to answer any other follow-up questions (if any) during the discussion period. 
If all doubts were resolved, considering that you have pointed out the importance of the topic and originality of the results, we would be happy if you would consider raising your score.



 *(continuation of the previous comment)*

**3. Details of the mathematical derivation and proofs need to be clarified.**

We propose below some explanations for the points that were not clear. We will rephrase and detail these points in the article to take your remarks into account. 

**a and e.** Local identifiability can be characterized by the nature of the intersection between the image $\Sigma_1^*$ of the lifting operator $\phi$ and the affine space $N(X, \theta)$ around $\phi(\theta)$, as stated in Theorem 7. However, the non linear nature of $\phi$ and thus of its image $\Sigma_1^*$ makes it difficult to test the intersection in practice. Showing that $\Sigma_1^*$ is a smooth manifold allows to consider tangent spaces and thus to linearize the problem. The question becomes then the one of the intersection between two affine spaces, which can be numerically tested with rank computations, as in Theorems 8 and 9.

**b.** Modeling a neural network by a graph enables to consider paths on the graph, which correspond to products of weight and bias parameters. Summing over very large numbers of paths occurs naturally with the lifting operator $\phi$ that enables to write the neural network output as a linear combination of the lifted parameters (Proposition 1). Obtaining this linearity is key for our statements and proofs, and we find that interpreting the components of $\phi$ as paths is the most convenient solution for clarity. In particular, for simplicity when the network has only one input, the right-hand-side of the equation in Proposition 1 is a scalar product, which is interpreted as a sum over paths in the network graph. We do not see more convenient ways to describe this sum, as the number of terms in the sum is very large (exponentially large as a function of the numbers of neurons per layer). We will add this explanation to the paper.

We remark that using this graph representation is common for neural networks in our opinion, in two ways. First, it is used for instance in textbooks or in practice for visualization and pedagogical reasons.
Second, it is relatively common for theoretically analysing deep neural networks as we do, for instance in [37]. 

**c.** This comment seems to us related to the previous one on using graph notations. At the beginning of Section 2, we will anticipate on Proposition 1 and Lines 145-147 (the linear representation) to motivate the graph notations. We will also interpret some of the most technical formulas in light of this linear representation. 

**d.** We agree with the suggestion. Local identifiability was defined mathematically only in Theorem 7. We will define local identifiability mathematically in Section 2 (Theorem 7 was in Section 4). 

*(continued in the next comment)*




 *(continuation of the previous comment)*

**2. Concerns about the general technical contents.**

**a. About ReLU activation.** Studying specifically the ReLU activation has become very standard in the machine learning community, with for example the recent references [29,33,37,39] of the paper. It is also our understanding that the ReLU activation function is quite prominent in practice. 

In this paper, the ReLU activation is crucial, since it is piecewise linear, and thus enables to obtain local linear representation with the lifting operator (Proposition 1).
Furthermore, the invariance of the network function to rescaling of the weights (Definition 3 and (7); see also our response to your question 3.f. below about fixing weights) is specific to the ReLU activation function. Hence, not only the proofs but also the nature of our identifiability conditions crucially depend on the ReLU activation function.

Note that, combining Theorem 9 and Proposition 10, our sufficient condition for indentifiability can be viewed in two steps. In Step 1, we locally reparameterize the neural network to remove the redundancy due to scaling (Definition 3). In Step 2, the sufficient condition is that the Jacobian matrix, after reparameterization, is full column rank. We think that this sufficient condition can be extended to more general activation functions. Step 1 would need to be specific to the activation function (depending on the activation, there could be more or less invariance than for ReLU). Step 2 would be unchanged for other activation functions.

We will clarify the important role of the ReLU activation function in the manuscript, and also mention the extension prospect of the sufficient  condition for identifiability. 

**b. On impact for applications.** The theory of this paper contributes to identifiability of ReLU-based neural network parameters. When global identifiability holds (for which we provide necessary conditions, but unfortunately no sufficient conditions as of now), the neural network framework becomes more aligned with standard parametric statistics. Furthermore, global identifiability means that the global minimum of the empirical risk (when $X$ is a learning sample) can be unique. In this case, there will typically be less variability of the practical optimization result, with respect to optimization parameters (choice of the algorithm, number of epochs,...) and to stochasticity (for stochastic optimizers). 

Hence our motivation for studying identifiability theoretically is that it can provide a practical framework for neural networks where the user has more understanding and control. This is important in our opinion, since a common critic of deep neural networks is their "black box" nature, and the fact that their predictions can be difficult to interpret and to reproduce.
On this point, we remark that identifiability is all the more beneficial when the empirical risk minimization has guarantees to converge to the global minimum (that can be unique with global identifiability). Indeed, in this case the neural network training becomes almost completely reproducible. These convergence guarantees do exist for neural networks with one hidden layer, thanks to suitable initializations of local descent algorithms [43]. Obtaining similar guarantees for deeper networks is largely open.

In the paper, we had not put this discussion on the benefit of identifiability in practice forward, because it is not really in agreement with the recent ideas of implicit regularization, double descent and benign overfitting, that tend to show that for neural networks, minimal prediction errors (e.g. $Q^2$ test performance) can be obtained with overparametrization and non-identifiability. Nevertheless, identifiability still can be beneficial, even if it has a small cost for prediction performance compared to the overparameterized regime. Notably, during a double descent phenomenon, for a fixed training set $X$ and for a critical number of parameters above which the neural network becomes unindentifiable, the test error may be low enough, even if non-minimal, so that the user may be interested in not increasing the number of parameters, in order to retain identifiability. 

In conclusion, to motivate better the broader impact of this work, as suggested by the reviewer, we will discuss in the introduction the benefit of identifiability for understanding and control for users of deep neural networks.

*(continued in the next comment)*

 Thank you for your careful reading of the paper, for pointing out the importance of the topic and the originality of our work, and for your questions and comments. We have done our best to address them below. They will definitely enable us to improve the paper's exposition. 

**1. The motivation and the relevance to existing work need to be clarified.**

We will improve and clarify the introduction and the discussion of the state of the art and our contributions to take into account your remarks.

**a.** We agree that the different notions of identifiability, of inverse stability and of stable recovery as well as their connections could be detailed more for more clarity, and will modify the introduction accordingly. To develop here:

* Identifiability is necessary for inverse stability, which itself is necessary for stable recovery. 

* Identifiability states that for a given network architecture, if the functions implemented by two networks coincide (on a given domain or on a finite sample) then their parameters are equal (up to elementary operations).

* Inverse stability states in addition that if the functions implemented by the two networks are close (on the domain or on the finite sample) for some distance, then their parameters are close. In that sense, inverse stability is a stable extension of identifiability to small perturbations.

* Stable recovery is the ability to recover in practice an approximation of the parameters of a network from its function (again, on a given domain or on a finite sample). Identifiability and inverse stability are necessary to stable recovery: if these are not satisfied, one could reconstruct a networks that fits perfectly with the behavior of the original network and still not find the true parameters.

* To the best of our knowledge, existing results of inverse stability and stable recovery apply to networks with only one hidden layer (e.g. [5, 13, 41, 42, 43, 44]) and/or make unrealistic assumptions (such as using the identity as activation function [21, 22, 23], or having access to the population risk instead of the empirical risk [5]).

* Our identifiability results, which apply to deep neural networks, with a realistic activation function, and only assume the knowledge of the function implemented by the network on a finite sample, are a step towards inverse stability and stable recovery for deep ReLU networks. Furthermore, the tools developed in this paper could be useful for that purpose. Our intuition suggests for instance that one could obtain conditions of inverse stability by showing the inverse stability of $\phi$ and by requiring the intersection between $\Sigma_1^*$ and $N(X, \theta)$, illustrated in Figure 2, to be transverse.

**b.** The results mentioned in lines 47-52 are diverse but generally consist in showing, for different choices of activation functions, that the parameters of a network are entirely determined by the function it implements, provided some conditions are satisfied by the architecture or the parameters of the network. They are mainly theoretical and all have the same limitation in assuming full knowledge of the function implemented by the network, which is impossible in practice. We will highlight this better in the article, and specify the vague formulations.

More generally for the entire Section 1.2, the limitation of the previous works is that none of them provide conditions for (even local) identifiability that are testable for any finite sample $X$ and address deep networks. Indeed, some address only networks with one hidden layer (e.g. [13,42,43]), some study identifiability from the entire function rather than its values at a finite sample [33], and finally [37] does not enable to test any finite sample $X$.

This paper is thus the first to overcome this limitation, providing testable conditions for any finite sample $X$ for deep neural networks. This was mentioned in Section 1.3, a bit indirectly. We will highlight this clearly in Section 1.2.

*(continued in the next comment)*

 *(continuation of the previous comment)*

Regarding the Privacy/IP motivations, we agree that there are other ways to prevent attacks and do not claim our results to be more than an additional tool at the disposal of defenders, and on the opposite side, when trying to recover the parameters of a network, a tool for monitoring the results.

As mentioned in Section 1.1, we can also think of other motivations of identifiability for obtaining theoretical guarantees of robustness: in empirical risk minimization, identifiability can allow to show the unicity of the global minimum and the independence of the result of the training process to the choice of the optimizer/to the stochastic parameters, though in that context we can encounter the limitation due to overparameterization that you highlighted in the review. We also discuss this motivation in more detail in our answer 2.b to reviewer jUtV.

We will improve the clarity and exhaustiveness of the practical significance and motivations in the introduction, based on your comment and this response. 

**On the 4 questions**

**$\bullet$** We agree with the suggestion. Local identifiability was defined explicitly only in Theorem 7. We will define local identifiability in the introduction in words, and also define it mathematically in Section 2 (Theorem 7 was in Section 4).

**$\bullet$** As suggested, we will say "geometric" instead of "geometrical".

**$\bullet$** $\phi$ is called a lifting operator because it corresponds to injectively map a smaller dimensional object (the vector of network parameters) into a (much) larger dimensional one. The benefit is that we transform a non-linear function on the smaller dimensional space into a linear function on the larger dimensional one. The term lifting is commonly used with this meaning in compressed sensing and dictionary learning, for instance in [Candes, E. J., Eldar, Y. C., Strohmer, T., Voroninski, V. (2015). Phase retrieval via matrix completion. SIAM review, 57(2), 225-251] (PhaseLift). We will add this explanation to the paper.

**$\bullet$** We agree that the idea of fixing some parameters to prevent rescaling can be explained more clearly, in particular the sentence mentioned that ends in "and we consider its value to be fixed from now on". In a response to a similar question of Reviewer jUtV, 
we provide an (hopefully clearer) explanation. We do not duplicate this entire response here for the sake of brevity. Its summary is: By fixing some parameters we mean that for each $\theta$, we construct a local reparametrization of the neural network output vector function $\tilde{\theta} \mapsto f_{\tilde{\theta}}(X)$ around $\theta$. This reparameterization removes redundancy in the parameters due to the scaling invariance (Definition 3). In particular, it enables to provide an atlas for the smooth manifold $\Sigma^*_1$, and to state and prove our identifiability results. This local reparameterization does not put any limitations on the neural network functions that are studied, it is only a mathematical tool for proving our results, and also to state some of them.

 Thank you for the time spent reading the paper, for your overall positive feedback, and for your constructive comments. We have done our best to address your comments and questions below. 

**On the significance and practical motivations**

Thank you for your remarks regarding the motivations and significance of our work. Let us try to give a more convincing answer. 

The question you raise about overparameterization is very interesting and we understand it in two different ways.

* In the optimization process, neural networks are indeed generally overparameterized in the sense that there are more parameters than learning samples. For this reason, there is in general no identifiability with respect to the learning data. However, in our results $X$ does not need to be made of learning samples, it can be any list of inputs such as queries of an already trained network, and these are not limited in number. In particular, one can query a trained network of fixed size until the number of queries is large enough so that the network is not overparameterized. As an illustration, in existing attacks in the literature which query a network in order to reconstruct its architecture and weights, such as [7], the number of queries is typically more than hundreds of times the number of parameters to be reconstructed.

* When it comes to overparameterization of neural networks as models in general, in the sense that different parameterizations can encode the same function, there indeed exist invariants in the parameters. Two invariants are well-known and common to all ReLU networks: the permutation of neurons in a hidden layer, and the positive rescalings. For that reason, it is true that one cannot hope to identify the exact parameters of a ReLU neural network, and existing results on identifiability or stable recovery rather aim to identify them modulo these permutations and rescalings. 

    The question of the existence of other invariants than these two is more complex and at the core of the work around identifiability. In the case of ReLU networks with one hidden layer for instance, [28] shows that under some mild conditions, there are no more invariants: the parameters are identifiable up to permutation and positive rescaling. The case of deep ReLU networks seems more contrasted: in some situations, permutations and rescalings remain the only invariants (as shown in [29, 4]), but in some other situations, additional invariants certainly appear. The big picture for deep networks is still unknown, and by providing testable conditions, even only for *local* identifiability, we also hope to help in this exploration.

We agree that there remains a distance between theory and practice around these questions, and we can explain why we think this paper could interest the NeurIPS community. There exists a literature studying theoretically the identifiability, stability or stable recovery of the parameters of neural networks. There is a gap between these articles and the practice: only one hidden layer for most of them, unrealistic assumptions on the network (such as the identity as activation function), unrealistic assumptions on the information available (population risk instead of empirical risk), etc. Furthermore, a lot of stable recovery results are often local as well in the sense that they assume a preliminary initialization close to the true parameters before applying their algorithms. These articles nevertheless contribute to develop our understanding of these questions. A few examples of such papers that were published in NeurIPS are:

* Yuanzhi Li and Yang Yuan (2017). “Convergence analysis of two-layer neural networks with ReLU activation”. In: Advances in neural information processing systems,
pp. 597–607

* Mahdi Soltanolkotabi (2017). “Learning ReLUs via gradient descent”. In: Advances in neural information processing systems, pp. 2007–2017 

* Pranjal Awasthi, Alex Tang, and Aravindan Vijayaraghavan (2021). “Efficient algorithms for learning depth-2 neural networks with general ReLU activations”. In: Advances in Neural Information Processing Systems 34, pp. 13485–13496

Our work humbly aims to add a contribution to this landscape and to get closer to real-life applications by providing a result for deep networks, with realistic activation function, a finite sample, and not much assumptions on the architecture and parameters (the sets $S$ and $\Delta_X$ that we exclude have zero Lebesgue measure) - although it is a local result. 



*(continued in the next comment)*

 We would like to thank again all three reviewers for their constructive feedback and the time spent reading the paper. In order to ease the reading of our individual responses, please find a short summary of them.

There were various requests for clarifications and discussions, to which we tried to answer as precisely as possible. In addition, we noted in particular three comments on (a) the idea of fixing weights, (b) the practical relevance of this work and (c) its originality. 

(a) In our reply to Reviewer jUtV we explain that fixing weights does not restrict the class of networks that we consider in any way, and is a mathematical tool based on a local reparameterization to remove the scaling invariance of the parameterization for ReLU networks. 

(b) In our replies to Reviewers Pq5e and jUtV, we have reinforced the motivation on privacy, and explained that identifiability provides more control on the training procedure of neural networks. This may be considered as an important motivation, given that a common critic of deep neural networks is their "black box" nature and the difficulty to reproduce their results. Our results, similarly as several ones published in NeurIPS  and other main machine learning conference proceedings (see our responses and the list of references of the paper), are small but hopefully useful steps towards a more complete mathematical understanding of deep neural networks.

(c) The originality of our work is summarized as follows: it is the first to provide testable identifiability conditions for any finite sample $X$ for deep neural networks. Existing works, that we discuss in Section 1.2 and in the response to Reviewer jUtV, fall short of addressing all these criteria (for instance only considering networks with one hidden layer or studying identifiability from the entire network function rather than only a sample).

In the individual responses, we have listed the modifications to the paper that we are happy to commit to do. They seem very feasible to implement to us, given the additional content page for the camera ready version, the possibility to add appendix content, and the time left for the camera ready version.

 This paper is a purely theoretical contribution (no code or experiments) on parameter identifiability for a Deep ReLU network. The main contribution is to give criteria for when the behavior of the function parameterized by the network on a finite set of points induces local identifiability of the parameters.

Whereas (global) identifiability shows that the parameters that generate that function are unique, local identifiability only tells you that the parameters are unique within an arbitrarily small neighborhood (they are the unique parameters in that neighborhood that produce that behavior of the function).

The motivations come from security, privacy, and intellectual property.

To summarize the main steps in the argument:

- The paper considers a deep fully-connected ReLU network with parameters $\theta$ that outputs a function $f_{\theta}(X)$ on a finite set of inputs $X$.
- It is well-known that some operations on the parameters do not change the function, including permutation and rescaling. Positive rescaling (when you multiply all the output weights for a single neuron by a positive constant and divide the input weights by the same constant) changes the parameters without changing the function.
- They decompose the function implemented by the network into a composition of the lifting operator (which only depends on the parameters, not on x), and a piecewise constant operator that depends on both the parameters and the inputs.
- They exclude a measure zero set they call $S$ from the parameter space. The excluded set S is the finite set of parameters that lie at the edges of the piecewise constants.
- They show the lifting operator is invariant to rescaling. If the parameters are equivalent, then the lifting operators match. If the lifting operators match, then the parameters are equivalent modulo rescaling.
- After excluding the set $S$, the lifting operator maps the parameter space (minus S) to a smooth manifold $\Sigma_1^*$ with a certain dimension.
- Next they fix some of the weights so that they can’t be rescaled. This removes some of the degrees of freedom for the parameters. (I didn’t fully understand this step.)
- This allows them to give a geometric condition (that the smooth manifold and an affine space intersect at a single point) that is equivalent (necessary and sufficient) to local identifiability.
- However, in practice, if I gave you a network and a finite sample, you wouldn’t know whether this geometric criteria was satisfied. So they continue to give computable criteria based on the rank of quantities that depend on both X and $\theta$.
- The computable criteria aren’t as tight as the geometric criteria. The geometric criterion is necessary and sufficient for local identifiability. There are two different computable criteria - one is necessary for identifiability and the other is sufficient for local identifiability.
- They explain how to numerically compute the sufficient criterion, including giving the big-O complexity of the runtimes. They leaving efficient computation of necessary criterion as future work. The paper is reasonably well-written although I'm not sure I gained a lot of intuition from reading it. There are no experiments to evaluate. I read the main text of the paper carefully but did not read any of the supplementary material and did not check if the proofs are correct.

My main comment is that I'm unsure of the significance of this paper. I don’t fully understand the motivation for studying identifiability of neural network parameters from their function behavior, since neural networks are overparameterized by design and in practice this appears to be important for optimization. Further, the assumptions required for this theory are so far from practice, and even then only yield a result on *local* identifiability (identifiability within arbitrarily small neighborhoods of the parameter space). The motivations given in the paper around security/privacy/IP don't seem compelling because anyone wanting to protect the exact values of parameters could just use a more complex and more overparameterized architecture. However, my background is biased in that I have worked more on the problem of identifying which function a neural network parameterizes given observed data, along the lines of Nonlinear ICA. - Suggestion: It took me a while to figure out what the definition of local identifiability is. You should state this more clearly.
- Suggestion: I think you should say “geometric” instead of “geometrical”.
- Question: Why is it called a “lifting operator”? You could explain the intuition for this naming.
- Suggestion: The section (beginning of Section 3) on fixing some parameters to prevent rescaling was confusing to me. Particularly the sentence that ends in "and we consider its value to be fixed from now on". What does it mean to be fixed? What does "from now on" mean? I think the paper is clear about its limitations. I don't have concerns about negative societal impact.

 This paper targets an interesting question of neural network basis. The main focus is the uniqueness of parameters in a ReLU-activated NN. Proofs are provided to support the conditions of such uniqueness. The reviewer has the following concerns about the work. This work starts with a question to doubt the uniqueness of NN in terms of the parameters, i.e., weights and bias. The authors try to provide the conditions (necessary and sufficient) of the uniqueness for a ReLU-activated NN. To the best of the review’s knowledge, this work is new. For confidential/security concerns, the topic is of great importance and needs more attention in the community.
While the problem is important, the review has some concerns and questions about the work.

1.	The motivation and the relevance to existing work need to be clarified. 

a.	For example, the authors claim not solving the inverse stability problem in Sec. 1.1. It is confusing. What is the connection between your work and inverse stability? The review didn’t find a discussion in the following paper. What are the insights generated from this work that can help with inverse stability?

b.	In Sec. 1.2, although the authors provide a literature review of the parameter uniqueness of NNs, some details are missing. For example, in Row 48: “interesting results on identifiability of networks,” what are the interesting results? In Row 51: “under some conditions,” what are the conditions that narrow down the implementation. For this entire section, what is the limitation(s) in previous work so that yours is necessary?


2.	Concerns about the general technical contents.

a.	Through the paper reading, the review would like to know why does the paper focus on ReLU activation? Can the theorems and proofs be generalized to other activation functions?

b.	From this review’s point of view, it’s better to include the discussion of how the theory can be used for the theoretical justification of ReLU-based or other neural networks in massive applications. They are essential and very helpful in related research and could emphasize the broader impact of this work.


3.	Details of the mathematical derivation and proofs need to be clarified. 

a.	The authors mention tangent spaces in the abstract and introduction for proof. Why should tangent space be used? 

b.	Could the author include some explanations of why graph modeling has to be used for NN representation? Is it easier to represent the weights on the edges, like in Figure 1? 

c.	It’s easy to get lost in heavy math formulas. For example, it is confusing what is the purpose of defining several notations in Sec.2.1-2.2 until the illustration in Row 145-147 on page 4.

d.	Then, the last sentence of Sec.2.2 makes it clear that the purpose of \phi is to show the local dependency of f_{\theta}(X), and this is related to the local identifiability. The reviewer suggests that the authors should define local identifiability at the beginning of Sec. 2, which should be a mathematical definition other than the one in Sec. 1.1.

e.	For Sec. 3, could the authors explain why the smooth manifold of image is essential for proving local identifiability? It seems to be a key intermediate result but has not been mentioned much.

f.	How do you generalize the results from fixing weights to free weights? The case now may be too narrow. It is worth more discussion as the freedom of weights combination and function composition are the magic of deep NN. 

g.	The definition of affine space needs to be clarified. Does “Ker” mean kernel in this paper?

h.	The visualization in Figure 2 is intuitive and helpful. I suggested the authors put it around Sec. 1.3, where the intersection is first mentioned in Contribution. Otherwise, the description is confusing to follow.
 Please see the questions and suggestions in the above comments. Please see the limitations and suggestions in the above comments.

 This study investigates the identifiability of deep ReLU nets, that is, the existence of a tractable inverse map from finite input-response examples {(x,f_\theta(x))} to the parameter \theta, and presents necessary and sufficient conditions of the identifiability. Strength:
In general, identification (of parameters) is a challenging problem. This can be expected by considering Fourier series, instead of neural networks, which have been investigated much longer than neural networks. That is, it is difficult to estimate the frequencies (or the parameters) of the original signal from a finite number of irregularly sampled input-output examples. Even in the case of regular sampling, classical issues such as aliasing are known to occur. Therefore, I consider the necessary and sufficient conditions to be nontrivial and interesting.

Weakness:
It would be better if the authors have presented a few (simple) numerical examples. Please refer to the questions section below for more details.

I found the notations are sometimes hard to follow. For example, I could not immediately understand that the equation in Proposition 1 is an abbreviation of \sum_p \alpha_p(x,\theta) \phi_{p,v}(\theta).
 Comments and Suggestions:
I believe that identification is possible by appropriately constraining the solution space. On the other hand, however, the following paper has pointed out the existence of a “null space” in the parameters of neural networks in a constructive manner:

[SII21] S.Sonoda, I.Ishikawa, M.Ikeda, Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space, arXiv:2106.04770.

The null space presented there does not result from the permutation of neurons, but rather construct a nontrivial arrangement where multiple neurons cancel each other out. That is, there exist W_2, W_1, and b_1 such that W_2, W_1, b_1 \neq 0, but W_2 \sigma( W_1 x + b_1 ) = 0 for all x. As long as such null space exists, identification is generally hopeless. As such, it would be more practical if the authors did not only describe the conditions under which identification is possible, but also provided a specific discussion on how the non-identifiable cases are avoided. (And my current guess is that either we can show that “(W_2, W_1, b_1) \sim 0” (in the sense of equivalence relation defined in Eq.5) or Proposition 5 drops such cases. In the former case, I further guess that identification is much harder when the activation function is not ReLU.)
 Yes they have in Sections 5 and 6.


Meta-review: Ratings: 6/5/5.
Confidence: 3/4/3.
Discussion among reviewers: No.

This paper provides results on local identifiability of deep ReLU networks. Identifiability of neural networks is an important theoretical topic, with practical implications such as reproducability, and we think the NeurIPS community would find the material interesting. Although the result is only for fairly specific assumptions that typically don't hold in practice, and only on local identifiability, the reviewers generally agree that the material is well presented. I think the result could lead as a stepping stone towards more general results, and I think that the results are worthy of presentation at NeurIPS.

My recommendation is to accept, assuming the list of promised updates to the paper, as detailed by the author, are executed.

Decision: Accept