Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper is a purely theoretical contribution (no code or experiments) on parameter identifiability for a Deep ReLU network. The main contribution is to give criteria for when the behavior of the function parameterized by the network on a finite set of points induces local identifiability of the parameters.

Whereas (global) identifiability shows that the parameters that generate that function are unique, local identifiability only tells you that the parameters are unique within an arbitrarily small neighborhood (they are the unique parameters in that neighborhood that produce that behavior of the function).

The motivations come from security, privacy, and intellectual property.

To summarize the main steps in the argument:

- The paper considers a deep fully-connected ReLU network with parameters $\theta$ that outputs a function $f_{\theta}(X)$ on a finite set of inputs $X$.
- It is well-known that some operations on the parameters do not change the function, including permutation and rescaling. Positive rescaling (when you multiply all the output weights for a single neuron by a positive constant and divide the input weights by the same constant) changes the parameters without changing the function.
- They decompose the function implemented by the network into a composition of the lifting operator (which only depends on the parameters, not on x), and a piecewise constant operator that depends on both the parameters and the inputs.
- They exclude a measure zero set they call $S$ from the parameter space. The excluded set S is the finite set of parameters that lie at the edges of the piecewise constants.
- They show the lifting operator is invariant to rescaling. If the parameters are equivalent, then the lifting operators match. If the lifting operators match, then the parameters are equivalent modulo rescaling.
- After excluding the set $S$, the lifting operator maps the parameter space (minus S) to a smooth manifold $\Sigma_1^*$ with a certain dimension.
- Next they fix some of the weights so that they can’t be rescaled. This removes some of the degrees of freedom for the parameters. (I didn’t fully understand this step.)
- This allows them to give a geometric condition (that the smooth manifold and an affine space intersect at a single point) that is equivalent (necessary and sufficient) to local identifiability.
- However, in practice, if I gave you a network and a finite sample, you wouldn’t know whether this geometric criteria was satisfied. So they continue to give computable criteria based on the rank of quantities that depend on both X and $\theta$.
- The computable criteria aren’t as tight as the geometric criteria. The geometric criterion is necessary and sufficient for local identifiability. There are two different computable criteria - one is necessary for identifiability and the other is sufficient for local identifiability.
- They explain how to numerically compute the sufficient criterion, including giving the big-O complexity of the runtimes. They leaving efficient computation of necessary criterion as future work.
strengths_and_weaknesses: The paper is reasonably well-written although I'm not sure I gained a lot of intuition from reading it. There are no experiments to evaluate. I read the main text of the paper carefully but did not read any of the supplementary material and did not check if the proofs are correct.

My main comment is that I'm unsure of the significance of this paper. I don’t fully understand the motivation for studying identifiability of neural network parameters from their function behavior, since neural networks are overparameterized by design and in practice this appears to be important for optimization. Further, the assumptions required for this theory are so far from practice, and even then only yield a result on *local* identifiability (identifiability within arbitrarily small neighborhoods of the parameter space). The motivations given in the paper around security/privacy/IP don't seem compelling because anyone wanting to protect the exact values of parameters could just use a more complex and more overparameterized architecture. However, my background is biased in that I have worked more on the problem of identifying which function a neural network parameterizes given observed data, along the lines of Nonlinear ICA.
questions: - Suggestion: It took me a while to figure out what the definition of local identifiability is. You should state this more clearly.
- Suggestion: I think you should say “geometric” instead of “geometrical”.
- Question: Why is it called a “lifting operator”? You could explain the intuition for this naming.
- Suggestion: The section (beginning of Section 3) on fixing some parameters to prevent rescaling was confusing to me. Particularly the sentence that ends in "and we consider its value to be fixed from now on". What does it mean to be fixed? What does "from now on" mean?
limitations: I think the paper is clear about its limitations. I don't have concerns about negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper targets an interesting question of neural network basis. The main focus is the uniqueness of parameters in a ReLU-activated NN. Proofs are provided to support the conditions of such uniqueness. The reviewer has the following concerns about the work.
strengths_and_weaknesses: This work starts with a question to doubt the uniqueness of NN in terms of the parameters, i.e., weights and bias. The authors try to provide the conditions (necessary and sufficient) of the uniqueness for a ReLU-activated NN. To the best of the review’s knowledge, this work is new. For confidential/security concerns, the topic is of great importance and needs more attention in the community.
While the problem is important, the review has some concerns and questions about the work.

1.	The motivation and the relevance to existing work need to be clarified. 

a.	For example, the authors claim not solving the inverse stability problem in Sec. 1.1. It is confusing. What is the connection between your work and inverse stability? The review didn’t find a discussion in the following paper. What are the insights generated from this work that can help with inverse stability?

b.	In Sec. 1.2, although the authors provide a literature review of the parameter uniqueness of NNs, some details are missing. For example, in Row 48: “interesting results on identifiability of networks,” what are the interesting results? In Row 51: “under some conditions,” what are the conditions that narrow down the implementation. For this entire section, what is the limitation(s) in previous work so that yours is necessary?


2.	Concerns about the general technical contents.

a.	Through the paper reading, the review would like to know why does the paper focus on ReLU activation? Can the theorems and proofs be generalized to other activation functions?

b.	From this review’s point of view, it’s better to include the discussion of how the theory can be used for the theoretical justification of ReLU-based or other neural networks in massive applications. They are essential and very helpful in related research and could emphasize the broader impact of this work.


3.	Details of the mathematical derivation and proofs need to be clarified. 

a.	The authors mention tangent spaces in the abstract and introduction for proof. Why should tangent space be used? 

b.	Could the author include some explanations of why graph modeling has to be used for NN representation? Is it easier to represent the weights on the edges, like in Figure 1? 

c.	It’s easy to get lost in heavy math formulas. For example, it is confusing what is the purpose of defining several notations in Sec.2.1-2.2 until the illustration in Row 145-147 on page 4.

d.	Then, the last sentence of Sec.2.2 makes it clear that the purpose of \phi is to show the local dependency of f_{\theta}(X), and this is related to the local identifiability. The reviewer suggests that the authors should define local identifiability at the beginning of Sec. 2, which should be a mathematical definition other than the one in Sec. 1.1.

e.	For Sec. 3, could the authors explain why the smooth manifold of image is essential for proving local identifiability? It seems to be a key intermediate result but has not been mentioned much.

f.	How do you generalize the results from fixing weights to free weights? The case now may be too narrow. It is worth more discussion as the freedom of weights combination and function composition are the magic of deep NN. 

g.	The definition of affine space needs to be clarified. Does “Ker” mean kernel in this paper?

h.	The visualization in Figure 2 is intuitive and helpful. I suggested the authors put it around Sec. 1.3, where the intersection is first mentioned in Contribution. Otherwise, the description is confusing to follow.

questions: Please see the questions and suggestions in the above comments.
limitations: Please see the limitations and suggestions in the above comments.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This study investigates the identifiability of deep ReLU nets, that is, the existence of a tractable inverse map from finite input-response examples {(x,f_\theta(x))} to the parameter \theta, and presents necessary and sufficient conditions of the identifiability.
strengths_and_weaknesses: Strength:
In general, identification (of parameters) is a challenging problem. This can be expected by considering Fourier series, instead of neural networks, which have been investigated much longer than neural networks. That is, it is difficult to estimate the frequencies (or the parameters) of the original signal from a finite number of irregularly sampled input-output examples. Even in the case of regular sampling, classical issues such as aliasing are known to occur. Therefore, I consider the necessary and sufficient conditions to be nontrivial and interesting.

Weakness:
It would be better if the authors have presented a few (simple) numerical examples. Please refer to the questions section below for more details.

I found the notations are sometimes hard to follow. For example, I could not immediately understand that the equation in Proposition 1 is an abbreviation of \sum_p \alpha_p(x,\theta) \phi_{p,v}(\theta).

questions: Comments and Suggestions:
I believe that identification is possible by appropriately constraining the solution space. On the other hand, however, the following paper has pointed out the existence of a “null space” in the parameters of neural networks in a constructive manner:

[SII21] S.Sonoda, I.Ishikawa, M.Ikeda, Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space, arXiv:2106.04770.

The null space presented there does not result from the permutation of neurons, but rather construct a nontrivial arrangement where multiple neurons cancel each other out. That is, there exist W_2, W_1, and b_1 such that W_2, W_1, b_1 \neq 0, but W_2 \sigma( W_1 x + b_1 ) = 0 for all x. As long as such null space exists, identification is generally hopeless. As such, it would be more practical if the authors did not only describe the conditions under which identification is possible, but also provided a specific discussion on how the non-identifiable cases are avoided. (And my current guess is that either we can show that “(W_2, W_1, b_1) \sim 0” (in the sense of equivalence relation defined in Eq.5) or Proposition 5 drops such cases. In the former case, I further guess that identification is much harder when the activation function is not ReLU.)

limitations: Yes they have in Sections 5 and 6.
review_title: null

Meta-review: metareview: Ratings: 6/5/5.
Confidence: 3/4/3.
Discussion among reviewers: No.

This paper provides results on local identifiability of deep ReLU networks. Identifiability of neural networks is an important theoretical topic, with practical implications such as reproducability, and we think the NeurIPS community would find the material interesting. Although the result is only for fairly specific assumptions that typically don't hold in practice, and only on local identifiability, the reviewers generally agree that the material is well presented. I think the result could lead as a stepping stone towards more general results, and I think that the results are worthy of presentation at NeurIPS.

My recommendation is to accept, assuming the list of promised updates to the paper, as detailed by the author, are executed.
confidence: Less certain
Decision: Accept