Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper claims that, through Canonical Correlation Analysis on the representations learnt by popular deep models, we can show that the representations learnt on the same dataset, with same model using different sets of parameters, the representations are linearly identifiable. 

This seems like a nice thing to know. But I however am not sure about few things. 
1) I think the paper doesn't really discuss the implications of this enough. That is, I would like to see few example applications where we take advantage of this fact. As is, this contribution in my opinion remains as a cute thing to know. 
2) It would be interesting to how much would change in initializations would effect the CCA curves. That is, if I initialize my network within a wider range, the identifiability result would still hold in practice? This would be interesting to add in my opinion. 
3) Ideally I would like to see if this statement hold for different hyperparameters also. 

Overall, the conclusion is interesting and I think that it could be published. 
 
review_title: Seems like a good paper, but not clear to me what are the practical benefits

final_score: rating: 4: Ok but not good enough - rejection
review_content: %%% post-rebuttal %%

The authors replied to my comments related to the diversity condition in 3.2 and Theorem 1. Their answers did not fully clarify my concerns or misunderstandings, and it seems the authors didn't make any changes in that regard in the revised version. Except if I missed something in the review system, they did not answer my other comments.

%%%%%%%%%%%%%%%


The paper discusses identifiability of learnt representations in supervised settings and under "canonical discriminative" modelling. It provides conditions under which two learnt representations are equivalent up to a linear transform. The paper discusses an important topic (related in particular to explicability of DNNs) and appears to provide valuable results but is difficult to follow. The paper is packed with results and difficult to read as important details are missing. A longer journal version would be more appropriate.

Comments:

- I got lost from the paragraph "Diversity condition". I don't understand the meaning of y_A and y_B. What do subscripts A and B refer to ? Then what do y_A^(i) and y_B^(i) refer to ? What are they exactly samples of ? Using illustrative examples would be very useful there. For example, what do they mean in classification where y is a label ?

- Because I didn't understand the meaning of y_A and y_B, I failed to understand the impact of Theorem 1. In particular, I don't understand why Eq. (5) holds in the proof.

- I am mildly familiar with the "canonical discriminative" formulation. Can all DNN tasks but represented as such ? How does it relate to more traditional forms of deep learning, such as based on minimisation of cross entropy or quadratic loss ? I assume that the sum in Eq. (1) is an integral when y is continuous.

- I got the idea of using CCA for comparing two different representations but the exact definition of C_i is unclear. Is it a vector or matrix ? Besides, why using only a subset B of D ?

- In the experiments you basically compare two learnt representations and show they are essentially similar up to a linear transform. However it's not clear how the two representations have been obtained. Is it only a matter of different initialisations ?

- I didn't understand how are the labels constructed in Section 5.1. Do you simply divide the unit circle in K=18 vectors that form the mean of each cluster ? I didn't get the "model misspecification" argument and the need to use DNN on such a simple task.

- "This show that increasing model size correlates strongly with increase in linear similarity of learned representations." I understand why this can be true when increasing number of samples but I don't understand why this should be expected with model capacity. What is it in Theorem 1 that reflects this ?


Minor:
- homogenise use of "Equation (x)" and "equation (x)"
- missing compiled reference at the beginning of Section 4
- it looks like you are using two different fonts for f_theta in the paper. Or are those different variables ?
- many typos: "share parameters", "but OF the network", "a the", ...


review_title: Valuable results but a longer version would be more suitable. Important details missing.

final_score: rating: 7: Good paper, accept
review_content: This paper investigated the identifiability of the learned representations in pre-trained DNN models that fall into a general class of function space,  defined by the canonical mathematical form.
The identifiability of learned representations, in this paper, is defined as the representations are reproducible on the same data
distribution, regardless of the randomness in the training procedure such as the random initialization of parameters and the stochastic optimization procedure.

The authors first proved that, in the limit of infinite data, learned representations in this family are asymptotic identifiable in function space up to a linear transformation, with the additional assumption of the diverse condition. 

They further showed that this property is applicable to several state-of-the-art pre-trained models, including CTC, BERT and GPT-2 and GPT-3.

At last, the authors conducted experiments to empirically investigate the linear identifiability of DNN models in a practical setting: finite data and partial optimization.  They adapted Canonical Correlation Analysis (CCA) (and SVCCA for high-dimensional space) to measure the linear similarity between two learned representations. Results from three sets of experiments, including classification, self-supervised learning for images (CTC) and for texts (GPT-2). Results show that the learned representations, after mapping through the optimal linear transformation from CCA, have a strongly linear relationship.

Overall, this paper is well-written and well-motivated. From the theoretical aspect, this paper proved the linear identifiability of a large class of DNN models in an ideal setting. From the empirical aspect, they provided experimental results to demonstrate the theorem in a practical setting.

review_title: Official Review for Linear Identifiability

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors address the model identifiability in a general setting that can be adapted to several recent deep learning models (DNN supervised learning, CPC, BERT and GPT). Since model parameters (NN weights) are not identifiable, the authors hypothesize that vector f and g can be identifiably up to a linear transformation. Although the purpose of the work is appealing, there are some issues related to the current structure of the paper, proposed theory and its relationship to the provided experiments. See my detailed comments and questions below:
-	Figure 1: This figure is not referenced in the text. It should be referenced as a motivating fact.
-	Figure 1: It is mentioned in the Appendix that the subset of words was selected. How these subsets were chosen? Randomly? I think this is important to clarify.
-	Theoretical claims: This figure clearly illustrate a strong relationship between f’ and f*, which is close to a linear mapping, but it is not exactly linear. The main result of the paper is that if diversity condition is met, then models are linearly identifiable, but nothing is said about approximately linearly identifiable models as the ones shown in Figure 1. One question that should be addressed is how a perturbation in the diversity condition is translated to the identifiability property. It would be good that the authors add some theory about it.
-	It is suggested that eq. (1) holds for arbitrary supervised learning problem but, in the Appendix, the equivalence is not shown for this case. Also, the example provided in Fig. 2 seems to be a very arbitrary case. Is it possible to write any supervised learning problem as eq. (1)? Could you please include its derivation?
-	Is the form of eq. (1) already used as a general approach for different ML problems as the authors claim? If so, could you please cite relevant references?
-	Diversity condition means that such invertible matrices L’ and L* can be constructed from data samples. What can be said if those matrices are invertible but ill-conditioned, i.e. smallest singular value close to zero?
-	For supervised learning the condition implies that K>= M+1. Is it not a very restrictive condition? For example, it is possible to apply this analysis to a simple 2-classes supervised classification setting.
-	Agreement between theory and experimental results. With exception of the first experiment (supervised learning) the rest of the experiments show that by increasing iterations, dataset size and number of hidden units, then functions f and g tends to be linearly related (but not closely linear related). I don’t see why the main theoretical result of the paper (Theorem 1) is related to the experimental observations. I disagree with the authors claim that “these experiments validate Theorem 1”
review_title: Interesting experiments but some concerns on the theoretical claims

Meta-review: This paper presents novel results on linear identifiability in discriminative models, with three of the four reviewers arguing for acceptance. The paper went through an extensive round of edits, which incorporated detailed responses to issues raised by the reviewers. 

While this paper would be a nice contribution to the conference, some reviewer concerns remain unresolved, so we encourage the authors to revise and resubmit to a future venue.
Decision: Reject