On Linear Identifiability of Learned Representations
Authors: Geoffrey Roeder
Luke Metz
Diederik P Kingma
Venue:  (2021)
Keywords: identifiability analysis
deep learning
representation learning
probabilistic discriminative models

Abstract: Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances in nonlinear Independent Components Analysis, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.

Under review as a conference paper at ICLR 2021
ON LINEAR IDENTIFLABILITY OF LEARNED REPRE- SENTATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
Identifiability is desirable property of statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of repre- sentation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances nonlinear Independent Components Analysis, we aim rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of of- the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.
INTRODUCTION
An increasingly common methodology in machine learning is to improve performance on a primary down-stream task by first learning a High--iimessionl representation of the data on a related, proxy task. In this paradigm, training a model reduces to fine-tuning the learned representations for optimal performance on a particular sub-task (Erhan et al., 2010). Deep neural networks (DNNs). as flexible function approximators, have been surprisingly successful in discovering effective high-dimensional representations for use in downstream tasks such as image classification (Sharif Razavian al., 2014), text generation (Radford et al., 2018 Devlin et al. 2018), and sequential decision making (Oord et al, 2018).
When learning representations for downstream tasks, it would be useful if the representations were reproducible, in the sense that every time a network relearns the representation function on the same data distribution, they were approximately the same, regardless of small deviations in the initialization of the parameters or the optimization procedure. In some applications, such as learning real-world causal relationships from data, such reproducible learned representations
crucial for accurate and robust inference (Johansson et al., 2016) Louizos et al. (017). A rigorous way to achieve reproducibility
choose a model whose representation function identifiable in function space. Informally speaking, identifiability in function space is achieved when, in the limit of infinite data, there exists single, global optimum function space. Interestingly, Figure exhibiis learned representation functions that appear to be the same up to linear transformation, even on finite data and optimized without convergence guarantees (see Appendix A. for training details).
In this paper, we account for Figure T by making precise the relationship it exemplifies. We prove that a large class of discriminative and autoregressive models are identifiable in function space, up to lincar transformation. Our results extend recent advances in the theory of nonlincar Independent Components Analysis (ICA), which have recently provided strong identifiability results for generative models of data (Hyvärinen al. 2018 Khemakhem et al. 2019 2020 Sorrenson al., 2020). Our key contribution is to bridge the gap between these results and discriminative models, commonly used for representation learning (e.g., (Hénaff al. 2019 Brown et al. (0200).
The rest of the paper is organized as follows. In Section we describe a general discriminative model family, defined by its canonical mathematical form, which generalizes many supervised, self-

---

Under review as a conference paper at ICLR 2021
Figure 1: Left and Middle: Two learned DNN representation functions fø, (B), fg. (B) visualized on held-out data B. The DNNs are word embedding models Mnih and Teh (2012) trained on the Billion Word Dataset (Chelba et al., (013) (see Appendix A.] for code release and training details). Right: Afe, (B) and (B), where A is a linear transformation learned after training. The overlap exhibits linear identifibility (see Section B) different representation functions, learned on the same data distribution, live within linear transformations of each other function space.
supervised, and contrastive learning frameworks. In Section we prove that learned representations in this family have an asymptotic property desirable for representation learning: equality up to linear transformation. In Section we show that this family includes a number of highly performant models, statc-of- -et----rr-tee-rttt- at publication for their problem domains, including CPC (Oord et al. 2018), BERT (Devlin et al.) 2018), and GPT-2 and GPT-3 (Radford et al. 2018 2019, Brown et 2020). Section investigates the actually realizable regime of finite data and partial optimization, showing that representations learned by members of the identifiable model family approach equality up to linear transformation as function of dataset size, neural network capacity, and optimization progress.
2 MODEL FAMILY AND DATA DISTRIBUTION
The learned embeddings of DNN are function not only of the parameters, but also the network architecture and size of dataset (viewed as a sample from the underlying data distribution). This renders any analysis in full generality challenging. To make such analysis tractable, in this section, we begin by specifying set of assumptions about the underlying data distribution and model family that must hold for the learned representations to be similar up linear transformation. These assumptions are, in fact, satisfied by a number of already published, highly performant models. We establish definitions in this section, and discuss these existing approaches in depth in Section 4
Data Distribution distribution PD y, S) over random variables x, y and S with the following properties:
We assume the existence of generalized dataset in the form of empirical
The random variable x is an input variable, typically high-dimensional, such as text an image.
The random variable y is target variable whose value the model predicts. In case of object classification, this would be some semantically meaningful class label. However, in our model family, y may also be high-dimensional context variable, such text, image, sentence fragment.
S is set containing the possible values of given so (D
S.
Note that the set of labels S is not fixed, but random variable. This allows supervised, contrastive, and self-supervised learning frameworks to be analyzed together: the meaning of S encodes the task. For supervised classification, S is deterministic and contains class labels. For self-supervised pretraining, contains randomly-sampled high-dimensional variables such as image embeddings. For deep metric learning (Hoffer and Ailon 2015 Sohn 2016), the S contains one positive and negative samples of the class to which x belongs.
Canonical Discriminative Form Given data distribution as above, a generalized discriminative model family may be defined by its parameterization of the probability of a target variable y conditioned observed variable and set S that contains only the true target label y, but
2

---

Under review as a conference paper at ICLR 2021
also collection of distractors y':
exp(fo(x) go(y))
po po(y Eyres (x) go
The codomain of the functions fx(x) and (y is R" and the domains vary according to modelling task. For notational convenience both parameterized by 0 O, but f and g may use disjoint parts of 0, meaning that they do not necessarily share parameters.
With F and G we denote the function spaces of fe and go respectively. Our primary domain of interest is when fo and go are highly flexible function approximators, such as DNNs. This brings certain analytical challenges. In neural networks, different choices of parameters O can result in the same functions fg and ge. hence the map e - x g is many-to-one. In the context of representation learning, the function fe typically viewed as a nonlinear feature extractor, e.g., the learned representation of the input data. While other choices meet the membership conditions for the family defined by the canonical form of Equation (I), in the remainder, we will focus on DNNs the remainder. We next present a definition of identifiability suitable for DNNs, and prove that members of the above family satisfy under additional assumptions.
MODEL IDENTIFIABILITY
In this section, we derive identifiability conditions for models in the family defined in Section
3.1 IDENTIFABILITY IN PARAMETER SPACE
Identifiability analysis answers the question of whether is theoretically possible to learn the parameters of statistical model exactly. Specifically, given some estimator O' for model parameters 0", identifiability the property that, for any {0', 0- C
Pø Pa
0'=
Models that do not have this property are said to be non-identifiable. This happens when different
values (0', )*) C O can give rise to the model distribution pø (y |xx S) po (y xx S). In such a case, observing an empirical distribution po- (y x, S), and fitting model pe (y x S) to perfectly does guarantee that O' 0".
Neural networks exhibit various symmetries in parameter space such that there is almost al always a many-to-one correspondence between choice of 0 and resulting probability function pa- A simple example in neural networks is that one can swap the (incoming and outgoing) connections of two neurons in a hidden layer. This changes the value of the parameters, but does not change the network's 's function. Thus, when representation functions fo or ge are parameterized as DNNs, equation 2 is not satisfiable.
3.2 IDENTIRIABILITY IN FUNCTION SPACE
For reliable and efficient representation learning, we want learned representations fg from two identifiable models to he sufficiently similar for interchangeable use downstream tasks. The most general property we wish preserve among learned representations is their ability to discriminate among statistical patterns corresponding to categorical groupings. In the model family defined in Section [2] the data and context functions fo and ge parameterize pe p |x, S), the probability of label assignment, through normalized inner product. This induces a hyperplane boundary, for discrimination, in joint space of learned representations for data x and context y. Therefore, in the following, we will derive identifiability conditions up to a linear transformation, using a notion of similarity in parameter space inspired by Hyvärinen al. (2018).
Definition 1. Let & be pairwise relation O defined as:
for (x) = Afg- (x)
6' 0
+ ge (y) Bg (y)
(3)
where and are invertible M x M matrices. See Appendix for proof that A is an equivalence relation. In the remainder, we refer to identifiability up to the equivalence relation & as E-identifialle or linearly identifable.
3

---

Under review as a conference paper at ICLR 2021
3.3 LINEAR IDENTIFIABILITY OF LEARNED REPRESENTATIONS
We next present a simple derivation of the >-identifiability of members of the generalized discrimina- tive family defined in Section 2 This result reveals sufficient conditions under which a discriminative probabilistic model pe (y) x, S) has a useful property: the learned representations of the input x and
target random variables y for any two pairs of parameters (0', O*) are related as O 0", that is, for (x) = Afe- (x) and ger (y) Bga- (y).
We first review the notation for the proof, which is introduced in detail in Section We then highlight an important requirement on the diversity of the data distribution, which must be satisfied for the proof statement hold. We prove the result immediately after.
Notation. The target random variables y. associated with input random variables may be class labels (as in supervised classification), or they could be stochastically generated from datapoints as, e.g., perturbed image patches (as in self-supervised learning). We account for this additional stochasticity as set-valued random variable S, containing all possible values of y conditioned on some x. For brevity, we will use shorthands that drop the parameters 0: p' == P Pg = fg-, = for gg == ge
Diversity condition. We assume that for any (0', 0*) for which it holds that p' = p", and for any given x, by repeated sampling S ~ (S)x and picking y E S, we can construct set of M distinct tuples { -
Y ) M such that the matrices L' and L' are invertible, where L' consists (
of columns (y (1) g (yl) (1) )), and L consists of columns g" (y g" (y ie (l...., M..
See Section 3.4 for detailed discussion.
Theorem 1. Under the diversity condition, models in the family defined by Equation ① are linearly identifiable. That is, for any 0', 0" E O, and f", ".,"g
"g'' "f,""""
,p, p'pp' defined as in Section
p' d'=p"
0' 2 0
(4)
To establish the result, we proceed by directly constructing an invertible linear transformation that satisfies Definition [
Consider YA; Yn E S. The likelihood ratios for these points
y 8 S
p' S)
(5)
D" py x, S)
are equal. Substituting our model definition from equation I we find:
exp(f'(x) g(()))) exp(f* (x) gx) (y)))
(6)
exp(f'(x) gy)))) exp(f=(x) (yn))
where the normalizing constants cancelled out on the left- and right-hand sides. Taking the logarithm, this simplifies to:
(g'(y)) g'(y))) F( = (g" (f
Note that this equation true for any triple YA, for which pD( YB,yB) >0.
We next collect M distinct tuples (y . (1) so that by repeating Equation (7) M times and by the
diversity condition noted above, the resulting difference vectors are linearly independent. We collect these vectors together the columns of (M x -immmension matrices L' and L", forming the following system of M linear equations:
I' LL
L
Since L' and L are invertible, we rearrange:
= (I'')(LLL' (x)).
(8)
Hence, (x( = Af (x) where = (L'L'-I). This completes the first half the proof. See Appendix Ce the second half of the proof, which is similar, and handles the function g.
4

---

Under review as a conference paper at ICLR 2021
3.4 DISCUSSION: WHEN DOES THE DIVERSITY CONDITION HOLD?
Theorem D is constructive proof of existence that exhibits invertible (M x M) matrices L' and L'. We require the diversity condition hold in order to guarantee invertibility. Such requirement is similar to the conditions in earlier work on nonlinear ICA such as (Hyvärinen et al.) (2018). as discussed in Section 6 Informally, this means that there needs to be sufficient number of
possible values E S. In the case of supervised classification with K classes, S is fixed and of size K. Then, we need K 2 M + 1 in order to generate M difference vectors go(((1) go (y) j - 2,.... M 1. In case of self-supervised or deep metric learning, where S and y may be algorithmically generated from x, this requirement is easy to satisfy, as there will typically be diversity of values of y. The same holds for language models with large vocabularies. However, for supervised classification with a small number of classes, this requirement on the size of S may be restrictive, as we discuss further in Section 4
Note that by placing the diversity requirement on the number of classes K, we implicitly assumed that the context representation function ge has the following property: the M difference vectors span the range of go. This is mild assumption in the context of DNNs: for random initialization and iterative weight updates, this property follows from the stochasticity of the distribution used to initialize the network. Briefly, a set of M + unique points y (l) such that the M vectors ()() (o((())). .....,........................ M + 1 are not linearly independent has measure zero. For other choices of go. care must be taken to ensure this condition is satisfied.
What can be said when L' and L' are ill-conditioned, that is, the ratio between maximum and minimum singular value Guma (L)
amni (I) (dropping superscripts when a statement apply both) is large? In the context a data representation matrix such as L, this implies that there exists at least one column Lj of L and constants 2S for k # j such that [|e, Ekzz AgLal12 2l2 for small In other words, some column is nearly linear combination of the others. This implies, in turn, that there exists some
tuple (y(*), M (()) such that the resulting difference vector L, ge g (a) go ( can nearly (in the sense above) be written as a linear combination of the other columns. Such near singularity is in this case a function of the choice of samples y that yield the difference vectors. The issue could be handled by resampling different data points until the condition number of the matrices is satisfactory. This amounts to strengthening the diversity condition. We leave more detailed analysis to future work, as the result will depend on the choice of architectures for f and g.
4 EXAMPLES OF LINEARLY IDENTIFIABLE MODELS
The form of Equation ① is already used as a general approach for a variety of machine learning problems. We present non-exhaustive sample of such publications, chosen to exhibit the range of applications. Many of these approaches were state-of-the-arr at the time of their release: Con- trastive Predictive Coding (Hénaff et al. 2019), BERT (Devlin et al. 2018). GPT-2 and GPT-3 (Radford et al., 2018 2019 Brown et al. 2020). XLNET (Yang et al. 2019). and the triplet loss for deep metric learning (Sohn) 2016). In this section, we discuss how to interpret the functional components of these frameworks with respect to the generalized data distribution of Section and canonical parameterization of Equation (T. See Appendix D for reductions to the canonical form of Equation
Supervised Classification. Although the scope of this paper is identifiable representation learning, under certain conditions, standard supervised classifiers can learn identifiable representations as well. In this case, the number of classes must be strictly greater than the feature dimension, as noted in Section 3.4 We simulate such a model in Section 5. to show evidence of its linear identifiabiligg We stress that representation learning as pretraining for classification conditions on label diversity are met, rather than relying on the supervised classifier itself to generate identifiable representations. This paradigm is discussed in the next subsection.
way to ensure that the
Representations learned during supervised classification can be linearly identifiable under the follow- ing model specification. The input random variables x represent some data domain to be classified, such as images word embeddings. The target variables y represent label assignments for x, typically semantically meaningful. These are often encoded these as the standard basis vectors ey. "one-hot encoding." The set S contains all K possible values of y. In this case, notice that S is
5

---

Under review as a conference paper at ICLR 2021
not stochastic: the empirical distribution Pp(S) x) is modelled Dirac measure with all proba- bility mass on the set S = (0.......) -X1 (using integers, here, to represent distinct labels). The representation function f (x) of classifier is often implemented as DNN that maps from the input layer to the layer just prior to the model logits. The context map go(y) is given by the weights in the final, linear projection layer, which outputs unnormalized logits. Concretely, go Weey where W E RMM M is learnable weight matrix. In order satisfy the diversity condition, the dimension M of the number of classes K must be strictly greater than the dimension of the learned representation M, that is, |S| M + 1. Finally, the output of the final, linear projection layer is normalized through a Softmax function, yielding the parameterization of Equation
Self-Supervised Pretraining for Image Classificationn. Self-supervised learning is a framework that first pretrains DNN before deploying on some other, related task. The pretraining task often takes the form of Equation IT and meets the sufficient conditions to be linearly identifiable. A paradigmatic example is Contrastive Predictive Coding (CPC) (Oord et al.] 2018 CPC is general pretraining framework, but we focus for the sake of clarity on its use in image models here. CPC as applied to images involves: (1) preprocessing an image into augmented patches, (2) assigning labels according to which image the patch came from, and then (3) predicting the representations of the patches whether below, to the right, to the left, or above a certain level (Oord et al. 2018).
The context function of CPC, (y))), encodes particular position in the sequence of patches, and the representation function, fx). is an autoregressive function of the previous & patches, according to some predefined patch ordering. Given some the collection of all patches from the sequence, from a given minibatch of images, is the set S ~ (D xx), where the randomness enters via the patch preprocessing algorithm. Since the preprocessing phase is part of the algorithm design, it is straightforward to make it sufficiently diverse (enough transformations of enough patches) so as to meet the requirements for the model to be linearly identifiable.
Multi-task Pretraining for Natural Language Generation. Autoregressive language models, such as (Mikolov et al. 00; Dai and Le 2013) and more recently GPT-2 and GPT-3 (Radford et al. 2018, | Brown et al.) 2020). are typically also instances of the model family of Equation [ Data points x are the past tokens, (xx is nonlinear representation of the past estimated by either an LSTM (Hochreiter and Schmidhuber 1997) or an autoregressive Transformer model (Vaswani et al., 2017 y is the next token, and w,- go i) is a learned representation of the next token, often implemented as a simple look-up table, as in supervised classification.
BERT (Devlin et al. 2018) is also member of the linearly identifiable family. This model pretrains word embeddings through a denoising autoencoder-Hee (Vincent et al. (2008) architecture. For a given sequence of tokenized text, some fixed percentage of the symbols are extracted and set aside, and their original values set to a special null symbol, "corrupting" the original sequence. The pretraining task in BERT is to learn a continuous representation of the extracted symbols conditioned on the remainder of the text. A transformer (Vaswani et al., 2017) function approximator is used to map from the corrupted sequence into continuous space. The transformer network is the fx(x) function of Equation] I
The context map ge (y) is a lookup map into the learned basis vector for each token.
5 EXPERIMENTS
The derivation in Section 3 shows that, for models in the general discriminative family defined in Section 2 the functions fo and ge are identifiable up to linear transformation given unbounded data and assuming model convergence. The question remains as how close model trained on finite data and without convergence guarantees will approach this limit. One subtle issue is that poor architecture choices (such as too few hidden units, or inadequate inductive priors) or insufficient data samples when training can interfere with model estimation and thereby linear identifiability of the learned representations, due to underfitting. In this section, we study this issue over a range of models, from low-dimensional language embedding and supervised classification (Figures I and 2 respectively) to GPT-2 (Radford et al. 2219), an approximately 1.5 * 10--parameter generative model of natural language (Figure 4p) See Appendix A and the code release for details needed to reproduce.
Through these experiments, we show that (1) in the small dimensional, large data regime, linearly identifiable models yield learned representations that lie approximately within linear transformation

---

Under review as a conference paper at ICLR 2021
of each other (Figures I and 2 predicted by Theorem I and (2) in the high dimensional, large data regime, linearly identifiable models yield learned representations that exhibit a strong trend towards linear identifiability. The learned representations approach a linear transformation of each other monotonically, as a function of dataset sample size, neural network capacity (number of hidden units), and optimization progress. In the case of GPT-2, which has benefited from substantial tuning by engineers to improve model estimation, we find strong evidence of linear identifiability.
Measuring linear similarity between learned representations. How can we measure whether pairs of learned representations live within linear transformation of each other in function space? We adapt Canonical Correlation Analysis (CCA) (Hotelling) 1936) for this purpose, which finds the optimal linear tramsformations to maximize correlation among two random vectors. On randomly selected held-out subset C D of the training data we compute fg, (B) and fg, (B) for two models with parameters 0₁ and 02 respectively. Assume without loss of generality that fo, (B3 and fo. (B) are centered. CCA finds the optimal linear transformations C' and D such that the pairwise correlations Pj between the 7th columns of C fg, (B) and D fø, (B) logether in p. If after linear transformation the two matrices are aligned, the mean of will be I; if they are instead uncorrelated, then the mean of p will be O. We use the mean of p as a proxy for the existence of linear transformation between fe, (B) and føz (15.. For DNNs, it is a well known phenomenon that most of the variability low-dimensional subspace, leaving many noisy, random dimensions (Morcos et al. 2018). Such random noise can result in spurious high correlations CCA. A solution to this problem is to apply Principal Components Analysis (PCA) (Pearson 1901) to each of the two matrices fg, (B) and fo, (B), projecting onto their top-k principal components, before applying CCA. This technique is known as SVCCA (Raghu et al. 2017).
maximized. We collect correlations
learned representation tends to concentrate in
5.1 SIMULATION STUDY: CLASSIFICATION BY DNNS
1.0
Trantremmol 2
150
Nean COA
0
Input
-150
0.0
(a)
Input
(c)
Hodel
Figure 2: Deep Supervised Classification. (a) Data distribution for linearly identifiable K-way classification problem. (b) Mean (centered) CCA between the learned representations over the course of training. After aрprox. 4000 iterations, CCA finds linear transformation that rotate the learned representations into alignment, up to optimization error. (c) Learned representations after transformation via optimal linear transformation. The first dimension of the first model's feature space is plotted against the first dimension of second. The learned representations have a nearly linear relationship, modulo estimation noise.
We report first on a simulation study of linearly identifiable K-way classification, where all assump- tions and sufficient conditions of Theorem Mare guaranteed to be met. We generated synthetic data distribution with the properties required by Section 2 and chose DNNs that had sufficient capacity to learn specified nonlinear relationship between inputs and targets y. In short, the data distribution PD (x, y, S) consists of inputs x sampled from a 2-D Gaussian with
3. The targets y were assigned among K = 18 classes according to their radial position (angle swept out by a ray fixed at the origin). The number of classes K was chosen to ensure K > dim fo + 1, the diversity condition. See Appendix [DI for more details.
To evaluate linear similarity, trained two randomly initialized models of pD |x, S). Plots show (x), the data representation function, on random x. Figure 2p shows that the mean CCA increases to its maximum value over training, demonstrating that the feature spaces converge to the same solution up to a linear transformation modulo model estimation noise. Similarly, Figure 2 shows that the learned representations exhibit strongly linear relationship.
7

---

Under review as a conference paper at ICLR 2021
5.2
SULE-SUPERVISED LEARNING FOR IMAGE CLASSIHICATION
0.9
CCA caefficient value
0.8
0.8
Moan CCA
Mean CCA
2500
5000
0.6
0.7
0.4
15000
02
30000
0.6
45000
0.5
0.5
10 20 30
40
50
60
10a
10a
10
10a
10a
10
(a)
(b)
OCA coefficient index
(c)
Dataset Size
(d)
Hidden Units
Figure 3: Self-Supervised Representation Learning. Error bars are computed over 5 pairs of models. (a) Input data. Two patches are taken (one from top half, and one from the bottom half) of an image at random. Using a contrastive loss, we predict the identity of the bottom patch encoding from the top. (b) Linear similarity of learned representations at checkpoints (see legend). As models converge, linear similarity increases. (c) Linear similarity as we increase the amount of data for fo and go. Error bars are computed over pairs of models. (d) As we increase model size, linear similarity after convergence increases for both lg and go.
pricipal
256
760 attcipal
Foll danieer
Figure 4: Text Embeddings by GPT-2. GPT-2 results. Representations of the last hidden layer (which is identifiable), addition to three earlier layers (not necessarily identifiable) for four GPT-2 models. For each representation layer, SVCCA is computed over all pairs of models, over which correlation coefficients were averaged. SVCCA was applied with 16, 64, 256 and 768 principal components. The learned representations in the last, identifiable layer more correlated than representations learned in preceding layers.
We next investigate high-dimensional, self- supervuee representation learning on CIFAR-10 (Krizhevsky et al., 2009) using CPC (Oord et al., 2018:: Hénaff et al., 2019). For given input image, this model predicts the identity of a bottom image patch representation given a top patch representation (Figure 3a.) Here, S comprises the true patch with set of distractor patches from across the current minibatch. For each model we define both for and ge as a 3-layer MLP with 256 units per layer (except where noted otherwise) and fix output dimensionality of 64.
In Figure 3p, CCA coefficients are plotted over the course of training. As training progresses, alignment between the learned representations increases. In Figure 3.. artificially limited the size of the dataset, and plot mean correlation after training and convergence. This shows that increasing availability of data correlates with closer alignment. In Figure 31, we fix dataset size and artificially limit the model capacity (number hidden units) to investigate the effect of model size on the learned representations, varying the number of hidden units from 64 to 8192. This show that increasing model capacity correlates with increase in alignment of learned representations.
5.3
GPT-2
Finally, we report on a study of GPT-2 (Radford al. 2019), a massive-scale language model. The identifiable representation is the set of features just before the last linear layer of the model. We use pretrained models from HuggingFace (Wolf et al. 2019). HuggingFace provides four different versions of the GPT-2: gpt2, 2, gpt2 12--medium um, gpt 2 large and gpt2 x1, which differ mainly in the hyper-parameters that determine the width and depth of the neural network layers. For approximately 2000 input sentences, per timestep. for each model, we extracted representations at the last layer (which is identifiable) in addition to the representations per timestep given by three earlier layers in the model. Then, performed SVCCA on each possible pair of models, on each of
8

---

results from Under review as a conference paper at ICLR 2021
the four representations. SVCCA was performed with 16, 64, 256 and 768 principal components, computed by applying SVD separately for each representations of each model. We chose 768 as the largest number of principal components, since that is the representation size for the smallest model in the repository (gpt.2). We then averaged the CCA correlation coefficients across the pairs of models. Figure shows the results. The results align well with our theory, namely that the representations at the last layer are more linearly related than the representations at other layers of the model.
5.4 INTERPRETATION AND SUMMARY
Theorem D establishes linear identifiability as an asymptotic property of a model that holds in the limit of infinite data and exact estimation. The experiments of this section have shown that for linear identifiable models, when the dimensionality is small relative to dataset size (Figures D and 2). the learned embeddings are closely linearly related, up noise. Problems of model estimation and sufficient dataset size are more pronounced in high dimensions. Nevertheless, in GPT- 2, representations among different trained models do in fact approach a mean correlation coefficient of 1.0 after training (Figure 4 blue line), providing strong evidence of linear identifiability.
6 RELATED WORKS
Prior to Hyvärinen and Morioka (0166), identifiability analysis uncommon in deep learning. We build on advances in the theory of nonlinear ICA (Hyvärinen and Morioka 2016 Hyvärinen et al. 2018 Khemakhem et al., 2019). In this section, carefully distinguish prior and concurrent works. Our diversity assumption is similar to diversity assumptions in these earlier works, while differing on certain conditions. The main difference is that their results apply to related but distinct families of models compared to the general discriminative family outlined in this paper. Arguably most related is Theorem 3 of Hyvärinen al. (2018) and its proof, which shows that class of contrastive discriminative models will estimate, up to an affine transformation, the true latent variables of a nonlinear ICA model. The main difference with our result that they additionally assume: (1) that the mapping between observed variables and latent representations is invertible; and (2) that the discriminative model is binary logistic regression exhibiting universal approximation (Hornik et al. 1989), estimated with a contrastive objective. In addition, (Hyvärinen) ctt al. 2018) does not present conditions for affine identifiability for their version of the context representation function g. It should be noted that Theorem 1 in (Hyvärinen et al., 2018) provides potential avenue for further generalization of our theorem] to discriminative models with non-linear interaction between f and g.
Concurrent work (Khemakhem et al. 2020) has expanded the theory of identifiable nonlinear ICA to class of conditional energy-based models (EBMs) with universal density approximation capability, therefore imposing milder assumptions than previous nonlinear ICA results. Their version of affine identifiability is similar our result of linear identifiability in Section 3.2) The main differences are that Khemakhem et al. (2020) focus in both theory and experiment on EBMs. This allows for alternative versions of the diversity condition, assuming that the Jacobians of their versions of f or are full rank. This is only possible if x or y are assumed continuous valued; note that we do not make such an assumption. Kheemaheem et al. (2020) also presents an architecture for which the conditions provably hold, in addition to sufficient conditions for identifiability up to element- scaling, which we did not explore in this work. While we build on these carlier results, we are, to the best of our knowledge, the first to apply identifiability analysis to state-of-the- the-tar discriminative and autoregressive generative models.
7 CONCLUSION
We have shown that representations learned by large family of discriminative models are identifiable up to linear transformation, providing novel perspective representation learning using DNNs. Since identifiability is a property of a model class, and identification realized in the asymptotic limit of data and compute, we perform experiments in the more realistic setting with finite datasets and finite compute. Our empirical results show that as the representational capacity of the model and dataset size increases, learned representations indeed tend towards solutions that are equal up to only a linear transformation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Bradbury. R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, and S. Wanderman- milne. Jax: Composable transformations of Python+NumPy programs, 2018. URL Http: //Github. Com/ Saem/Google/Jax
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, and Others. Language Models Few-Shot Learners. Arxiv Preprini
Sari:20005114665 2020.
C. Chelba, T. Mikolov, M. Schuster, Q. Ge. T. Brants, P. Koehn, and t. Robinson. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. Arxiv Preprint Arxiv: 2013.
A. M. Dai and Q. V. Le. Semi-Supervised Sequence Learning. In Advances in Neural information Processing Systems, pages 5079-3087, 2015.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Arxiv Preprint Arxiv:1810.04800 2018.
D. Erhan, Y. Bengio. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. Why Does Unsuper- vised Pre-training Help Deep Learning? Journal of Machine Learning Research, 1FFee)):25-666, 2010.
O. J. Hénaff, A. Razavi, C. Doersch, Eslami, and A. D. Oord. Data-Efficient Image Recognition with Contrastive Predictive Coding, Arxiv Preprint Arxiv: Arxiv:1905.099 190..09995 2009272,20110. 2019.
S. Hochreiter and J. Schmidhuber. Long Short-Tere Memory. Neural Computation, 98):1735-1788) 1997.
E. Hoffer and N. Ailon. Deep Metric Learning Using Triplet Network. In International Workshop On Similarity-Basee Pattern Recognition, pages 84-92. Springer, 2015.
K. Hornik, M. Stinchcombe, and H. White. Multilayer Feedforward Networks are Universal Approx- imators. Neural Networks, 2(5):359-33) -6699,666660 1989.
H. Hotelling. Relations Between Two Sets of Variates. Biometrika, 28(3/4):321-371 1936.
A. Hyvärinen and H. Morioka. Unsupervised Feature Extraction by Time-Contrastivn Learning and Nonlinear ICA. In Advances in Neural information Processing Systems, pages 3765-3773, 2016.
A. Hyvärinen, H. Sasaki, and R. E. Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. Arxiv Preprint Arxiv:1805.0863 2018.
Johansson, Shalit, and Sontag. Learning representations for counterfactual inference. In International conference on machine learning, pages 3020-3029, 2016.
I. Khemakhem, D. P. Kingma, and A. Hyvärinen. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. Arxiv Preprint (tti::/997.044000 2019.
I. Khemakhem, R. P. Monti, D. Kingma, and A. Hyvärinen. ICE-BeeM: Identifiable Conditional Energy- Deep Models. Arxiv Preprint Arxv: 2020.
D. P. Kingma and Ba. Adam: A Method for Stochastic Optimization. Arxiv Preprint Arxiv:1412.6988 2014.
A. Krizhevsky, G. Hinton, and Others. Learning Multiple Layers of Features from Tiny Images. 2009.
J. Liu, M. Saleh, Pot, B. Goodrich, R. Sepassi, L.. Kaiser, and Shazeer. Generating Wikipedia by Summarizing Long Sequences. Arxiv Preprint Arxiv:1801. 118801101990 10198, 2018.
C. Louizos, U. Shalit, J. M. Mooij, D. Sontag, R. Zemel, and M. Welling. Causal effect inference with deep variable models. In Advances in Neural Information Processing Systems, pages 6446-6456, 2017.
10

---

Under review as a conference paper at ICLR 2021
T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur. Recurrent Neural Network Based Language Model. Eleventh Anmual Conference of The international Speech Communication Association, 2010.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural information Processing Systems, pages 3111-3119, 2013.
A. Mnih and E. Hinton. A Scalable Hierarchical Distributed Language Model. In Advances in Neural information Processing Systems, pages 1081-1088 2009.
A. Mnih and Y. W. Teh. A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. Arxiv Preprint A2xiv:/006.660 6426, 2012.
A. S. Morcos, M. Raghu, and S. Bengio. Insights on Representational Similarity in Neural Networks with Canonical Correlation, 2018
A. V. Oord, Li, and O. Vinyals. Representation Learning with Contrastive Predictive Coding. Arxiv Preprint Arxiv: 1807.03748, 2018.
K. Pearson. LIII On Lines and Planes of Closest Fit to Systems of Points in Space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11))559-572, 1901.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving Language Understanding by Generative Pre-training. 2018
A. Radford, Wu, R. Child, D. Luan, Amodei, and Sutskever. Language Models are Unsuper- vised Multitask Learners. Openai Blog, 1(8), 2019.
M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and interpretability. In Advances in Neural information Processing Systems, pages 6076-6085, 2017.
A. Sharif Razavian, H. Azizpour, Sullivan, and S. Carlsson. CNN Features Off-the-Shelf: An Astounding Baseline for Recogmition. In Proceedings of The leee Conference On Computer Vision and Pattern Recognition Workshops, pages 806-813, 2014.
K. Sohn. Improved Deep Metric Learning with Multi-class N-Pair Loss Objective. Advances in Neural information Processing Systems, pages 1857-1865, 2016.
P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by Nonlinear ICA with General Incompressible-foow Networks (Gin). Arxiv:2001.04877 [Cs, Stat], Jan. 2020.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All You Need. In Advances Neural information Processing Systems, pages 5998- 6008, 2017.
P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of The 25th international Conference On Machine Learning, pages 1096-1103, 1103, 2008.
T. Wolf, I. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew. Huggingface's Transformers: State-of-the-ard Natural Language Processing. Arxiv, Abs/1910.03771, 2019.
Z. Yang, Dai, Y. Yang. Carbonell, R. Salakhutdinov, and Q. V. Le. XLNET: Generalized Autoregressive Pretraining for Language Understanding. Arxiv Preprint
2019,
11

---

Under review as a conference paper at ICLR 2021
A REPRODUCING EXPERIMENTS AND FIGURES
In this section, present training and optimization details needed to reproduce empirical validation of Theorem [] experiments that investigate the result in the small and massive scale regines, for Figure and GPT-2 (ANONYMIZED).
We also published notebooks and check- sroeck-pointed weights for two crucial
A.1 FIGURE
We provide Jupyter notebook and model checkpoints for reproducing Figure 1 Please refer to this for hyperparameter settings. In short, we implemented model (Mnih and Teh 2012) in the family of Section 2 and trained on the Billion Word dataset (Chelba et al. |l.] [2013). This is illustrative of the property of Theorem I because the relatively modest size of the parameter space (see notebook) and massive dataset minimizes model convergence and data availability restrictions, e.g., approaches the asymptotic regime.
The word embedding space is 2-D for ease of visualization. We randomly selected subset of words, mapped them into their learned embeddings, and visualized them as points in the left and middle panes. We then regress pane one onto pane two in order to learn the best linear transformation between them. Note that if the two are linear transformations of each other, regression will recover that transformation exactly.
A.2 SIMULATION STUDY: CLASSIFICATION BY DNNs
For this experiment, we want ensure that the chosen model can fit the data distribution exactly. Controlling this removes one possible factor that could prevent linear identifiability of learned representations despite the model formally having that property. We do this by making sure that the process that generates the dataset matches the model chosen to learn the relationships between inputs and labels.
This is achieved through the following algorithm. We first randomly assign initialization labels based on angular position, then two neural networks for and go to predict the final labels, using the discrininative model of Equation (1) and Appendix D.I] Both fo- and go- 4-hidden-layer MLPs with two 64 unit layers and one 2-D bottle neck layer. After training these representation functions to convergence, generated new batch of points and used the trained networks to predict the ground truth labels y.
Finally, to conduct experiments, we chose for and ge to be the same architecture as fg- and ge- This ensures that the supervised classifier we attempted to learn would using the function approximators for and ge would be able to capture the true data generating process, e.g, would not fail due to too few hidden units, or too complex a relationship between targets and inputs.
Remaining training details 10-4 for 5 10 iterations. To make the classification problem more challenging, we additionally add 20 input dimensions of random noise the data. The Adam optimizer Kingma and Ba (2014) with learning rate of 3 10 is used.
follows. We optimize weights using Adam with learning of
A.3 SELF-SUPERVISED LEARNING FOR IMAGE CLASSIFICATION
To compute linear similarity between representations, we train two independent models in parallel. For each model we define both fo and go as a 3-layer fully connected neural network with 2 units per layer and fixed output dimensionality of 26, We define our model following Equation 11. where S is the set of the other image patches from the current minibatch and optimize the objective of (Hénaff et al. 2019). We augment both sampled patches independently with randomized brightness, saturation, hue, and contrast adjustments. following the recipe of (Hénaff et al., 2019). We train on the CIFAR 10 dataset (Krizhevsky et al. (2009) with batchsize 28, using the Adam optimizer with a learning rate of 10-4 4 and the JAX (Bradbury et al., 2018) software package. For each model, early stop based on a validation loss failing to improve further.
Additional details about the experiments that generated Figure
12

---

Under review as a conference paper at ICLR 2021
Figure 3
Patches are sampled randomly from training images.
Figure 3 b. For each model, we train for at most * 10 iterations, early stopping when necessary
based on validation loss.
Figure c.
For each model, we train for most 10 iterations, early stopping when necessary based on validation loss.
Figure
Error bars show standard error computed over 5 pairs of models after 1.5 10 training iterations.
A.4
GPT-2
We include all details through a notebook in the code release. Pretrained GPT-2 weights as specified in the main text are publicly available from HuggingFace Wolf et al. (2019).
A.5 REMARK ON EFFECT OF INITIALIZATION AND HYPERPARAMETERS OF MODELS
One question that may be of interest is whether initialization affects whether learned representations will be within a linear transformation of each other. This depends on whether the optimization routines (like Adam, AdaGrad, etc.) are robust to wider initialization within certain range. If so, model convergence will be unaffected. However, this cannot make up for poor initialization poor optimization: just as in any deep neural network, a poor initialization and inadequate optimizer will interfere with learning the model parameters. In the case of linearly identifiable model, means that the learned representations would not live within linear transformation each other (up to noise from model fitting), since the models have failed to converge to reasonable solution for the task at hand.
When the hyperparameters of a DNN are changed, this changes the class of functions that the network can represent (i.e., the size and stride convolution filters will change which input pixels could be correlated in deeper layers). Typically, hyperparameters based on held-out data. We did so our experiments also. We expect that such a tuning procedure would yield hyperparameters that are good possible for the model to be optimized, allowing sufficient optimization so that the linear identifiability of the learned representations is realized. If the hyperparameters are sufficiently bad and optimization suffers, this will interfere with model fitting, and with linear identifiability of the learned representations also.
carefully tuned using cross validation
B PROOF THAT LINEAR SIMILARITY IS AN EQUIVALENCE RELATION
We claim that E is an equivalence relation. It suffices to show that it is reflexive, transitive, and symmetric.
Proof. Consider some function ge and some 0', 0',0" 000'00 0 C Suppose 0' A 0-. Then, there exists an invertible matrix B such that gø (x) Bge- (x). Since ge- (x) B B-1 (x), & is symmetric. Reflexivity follows from setting ge- to gø and B to the identity matrix. To show transitivity, suppose
also that 0+ A 0i, Then, there exists an invertible C such that ge- (x) = Cgar (x). Since ge A ge-,
B-1 gor (x) Cger (x). Rearranging terms, go (x) BCgar (x), that 0' A Ot as required.
C SECTION 3.2 FUNCTION g
CONTINUED: CASE OF CONTEXT REPRESENTATION
Our derivation of identifiability of ga is similar to the derivation of fe. The primary difference is that the normalizing constants in Equation (6) do not cancel out. First, note that can rewrite Equation | as:
po po S) (x)(
13

---

Under review as a conference paper at ICLR 2021
where:
(x, S) [x fo;
(10)
[y = [o)
(11)
(x, S) = log Σ
exp(fo(x) go(y)))
(12)
y'ES
Below, we will show that for the model family defined in Section
po Po go (y) B go (y),
(13)
where B is an invertible (M x -dimensioonl matrix, concluding the proof of the linear identifiability of models in the family defined by Equation
We adopt the same shorthands as in the main text.
C.I DIVERSITY CONDITION
We assume that for any (0', (*) C for which holds that p' = p", and for any given y, there exist M such that (x .y, S > 0, and such that the ((M+ +1) x (M)
M+1 tuples (x((), ) matrices M' and M are invertible, where M' consists of columns (((((), S()), and M consists of columns f* (x((), S())..
This is similar to the diversity condition of Section 3.2 but milder, since typical dataset will have multiple x for each y.
C.2 PROOF
With the data distribution pr y. S). for a given y. there exists a conditional distribution pr Sly). Let (x be sample from this distribution. From equation and the statement prove, it follows that:
p' p'(y|x, S) = p" (ylx, S)
(14)
Substituting in the definition of our model from equation 9 we find:
exp(f' (x, (y)) = exp(f" (x. S) g (y)).
(15)
which, evaluating logarithms, becomes
S) g (y) F* (x, S) (y),
(16)
which is true for any triple (x where Py x, S) > O.
From M' and M (Section C.I and equation 16 we form a linear system of equations, collecting the M + relationships together:
M g''g((y) M Tg"
(17)
g = Ag" (y),
(18)
where A (M' invertible (M 11 (M + 1) matrix.
It remains to show the existence of an invertible M x M matrix B such that
g' (y) = Bg((y)
(19)
We proceed by constructing B from A. Since A is invertible, there exist. j elementary matrices (Ei,..., E,} such that their action R = E, E converts A to (non-unique) row echelon form. Without loss of generality, we build R such that the a1 entry of A is the first pivot, leading to the particular row echelon form:
01,1
01,2
01.3
01.xx1
0
22.2
02,3
mx
RA
0
03.3
02
(20)
0
om m1
14

---

Under review as a conference paper at ICLR 2021
where di. indicates that the corresponding entry in R.A may differ from A due to the action of R.
Applying R to Equation 17 we have
Rg (y) RAg (y).
(21)
We now show that removing the first row and column of RAA and R generates matrices of rank M. Let RA and R denote the (M x M) submatrices formed by removing the first row and column of R. and R. respectively.
Equation 20 shows that RA has a pivot in each column, and thus has rank M. To show that R is invertible, we must show that removing the first row and column reduces the rank of R. E Ej by exactly 1. Clearly, each Eg is invertible, and their composition is invertible. We must show the same for the composition Eg.
There are three cases to consider, corresponding to the three unique types of elementary matrices. Each elementary matrix acts on A by either (1) swapping rows i and j. (2) replacing row j by a multiple of itself, or (3) adding multiple of row to row j. We denote elementary matrix types by superscripts.
In Case (1), E is an identity matrix with row i and row swapped. For Case (2), Ef is an identity matrix with the j. j.jth entry replaced by some For each E and E; in R, where 1 S k, I j. know that the indices j. 2 2, because we chose the first entry of the first row of A to be the pivot, and hence do not swap the first row, replace the first row by itself multiplied by constant. This implies that removing the first row and column of E and E removes a pivot entry in the 1) position, and removes zeros elsewhere. Hence, the (M x M) submatrices E and Ej are elementary matrices with rank M.
For Case (3), E has some value E R in the th entry, and 1s along the diagonal. In this case, we may find a non-zero entry in some E so that, e.g., the second row has a pivot at position (2, 2). Without loss of generality, suppose = j = 2 and let be some nonzero constant. Removing the first row and column of E removes this m also. Nevertheless, E Las, the rank M identity matrix. For any other E < i S M + j, 2 2 because we chose 01,1 as the first pivot, and hence do not swap the first row, or replace the first row by itself multiplied by a constant. In both cases, removing the first row and first column creates an Eg that is a rank M elementary matrix.
We have shown by the above that R is a composition of rank M matrices. We now construct the matrix B by removing the first entries of g' and g" and removing the first row and first column of R and RAA in Equation (equation 21). Then, we have
Rg' = RAg"
(22)
g g' R 'RAg'
(23)
Choosing B R 'RA proves the result.
D REDUCTIONS TO CANONICAL FORM OF EQUATION
In the following, we show membership in the model family of Equation D using the mathematical notation of the papers under discussion in Section Note that each subsection will change notation to match the papers under discussion, which varies quite widely. We employ the following colour-coding scheme to aid in clarity:
log (y)p)) =fo
go(y)
log y'eS
exp(fo(x) (y))),
where (x) is generalized to a data representation function, go (y) is generalized to a context representation function, and Ey'es exp(ff go(y")) is some constant
15

---

Under review as a conference paper at ICLR 2021
D.1 SUPERVISED CLASSIFICATION
Supervised classifiers commonly employ neural network feature extractor followed by a linear projection of the output of this network into space of unnormalized logits. All the layers prior to the logits are the representation function fg, and the final projection layer is the context map go(y = Wi where w; the i-th column of a weight matrix W. The set S in this case contains human-chosen labels and has no stochasticity. The loss function is the negative log-likelihone of the data under categorical distribution with softmax parameterization:
S
log po (y i|p:: S)
(x)
log
exp (x) w0,1
Supervised classification thus an member family defined in Section It exhibits the simplest functional form for the function while allowing f to be arbitrarily complicated.
D.2 CPC
Consider a sequence of points X,. We wish to learn the parameters & to maximize the k-step ahead predictive distribution p(X{+*|Xx X,, Φ). In the image patch example, each patch center i, is indexed by t. Each X, is mapped to sequence of feature vectors 2 = fe(x;) An autoregressive model, already updated with the previous latent representations 2<t-1, transforms the 2, into "context" latent representation c₂ g^R(==g)) Instead of predicting future observations steps ahead, X++k- directly through generative model Ph (x++ (c)). Oord al. (2018) model a density ratio order to preserve the mutual information between X and
Objective Let X = {xi...., } be a set of N random samples containing one positive sample from p(x++*/cc (c)) and N samples from the proposal distribution p(x(++). Oord et al. (2018) define the following link function: 4 (X1+1. cx) & exp (z(+1 Wgct). Then, CPC optimizes
La (x
exp
Zi+k
W&C.
Ex
log
Ex log
(24)
Ex,ex (x), ct)
Σx exp (z, WgC.
Substituting in the definition of l makes equation
24 identical to the model family (Equation
D.3 AUTOREGRESSIN LANGUAGE MODELS (E.G. GPT-2)
Let 2 = fui...., Un / be corpus of tokens. Autoregressive language models maximize a log-
likelihood L(U) Σt log Plu u4-1; O), Concretely, the conditional density is mod- elled as
log (1ill;-----1:3
W, h
log Σ exp W,hh,)
where h; is the m x 1 output of function approximator (e.g. Transformer decoder (Liu et al. 2018 and W. the i'th row of the [U]] x m token embedding matrix.
D.4 BERT
Consider sequence of text _ [TT) Some proportion of the symbols in extracted into a vector x, and then set in x to special null symbol, "corrupting" the original sequence. This operation generates the corrupted sequence x. The representational learning task to predict conditioned on x, that is, to maximize w.r.t. 0:
log Pa (x(x)= R3
log po
Σ t=1
Ha(x); (e(x))
log Σ exp (Ho(x)) e(x'))
16

---

Under review as a conference paper at ICLR 2021
where H is transformer, is a lookup table, and my 1 if symbol 2; is masked. That is, corrupted symbols are "reconstructed" by the model, meaning that their index is predicted. As noted in Yang al. (2019), BERT models the joint conditional probability p(X(x) factorized that each masked token is separately reconstructed. This means that the log likelihood approximate instead of exact.
D.5 QUICKTHOUGHT VECTORS
Let and g be functions that take sentence input and encode it into fixed length vector. Let be given sentence. and Setat be the set of sentences appearing in the context of for a fixed context size. Let Seand be the set of candidate sentences considered for given context sentence Satzt E Setrt. Then, Seand contains valid context sentence Sctat well as many other non-context sentences. Seand used for the classification objective. For any given sentence position in the context of (for example, the preceding sentence), the probability that candidate sentence Soand E Seand is the correct sentence for that position is given by
log (Scandd s, Seand) fe
go Scand)
log Σ
exp (fe(s) go (s(an)
s' S'ESoand
D.6 DEEP METRIC LEARNING
The multi-clase N-pair loss in Sohn
2016) is proportional to
log N
1 N
log
1 Σ
exp(fo(x;) fo(u;)
(o)(o(()TTTo(() - fo(x;) T
1=1
which can be simplified as
log
(fo fo(w;) Too))
log
Eラn K (x;( (u())) (xx) (Tx)aaTea T (v())
exp(fo((x) fo(v))}
log
K Eド= expl exp (f)()))) (o(3))}
Setting N to and evaluating the log gives
fg x fo
exp(fo(x;) To,()))
K
which is Equation where fo go-
D.7 NEURAL PROBABILISTIC LANGUAGE MODELS (NPLMs)
Figure T shows results from neural probabilistic language model as proposed in Mnih and Teh (012)). Mnih and Teh (2012) propose using log-bilinear model (Mnih and Hinton (009) which, given some context h. learns context word vectors Tw and target word vectors Tw- Two different embedding matrices are maintained, in other words: one to capture the embedding of the word and the other the context. The representation for the context vector, é, is then computed as the linear combination of the context words and context weight matrix C, so that q = >m Cirm,, The score for the match between the context and the next word is computed as a dot product, e.g.. se (w h) = q qu and substituting into the definition of P/ (w), we see that
log Pd (w)
qwe
log Σ cxp (ÿ qw.)
We have absorbed the per-token baseline offset into the que defined in Mnih and Tch 2012), forming the vector &u whose i'th entry is (qw); (qw)s bw/(())
17

---

Under review as a conference paper at ICLR 2021
shows that Mnih and Tch (2012) is member of the model family.
Interestingly, a touchstone work in the area of NPLMs, Word2Vec (Mikolov et al. 2013), does not fall under the model family due to additional nonlinearity applied to the score of Mnih and Tch (2012).
18


Reviews:
[{"review_id": "bmLCUaeLMH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors address the model identifiability in a general setting that can be adapted to several recent deep learning models (DNN supervised learning, CPC, BERT and GPT). Since model parameters (NN weights) are not identifiable, the authors hypothesize that vector f and g can be identifiably up to a linear transformation. Although the purpose of the work is appealing, there are some issues related to the current structure of the paper, proposed theory and its relationship to the provided experiments. See my detailed comments and questions below:\n-\tFigure 1: This figure is not referenced in the text. It should be referenced as a motivating fact.\n-\tFigure 1: It is mentioned in the Appendix that the subset of words was selected. How these subsets were chosen? Randomly? I think this is important to clarify.\n-\tTheoretical claims: This figure clearly illustrate a strong relationship between f’ and f*, which is close to a linear mapping, but it is not exactly linear. The main result of the paper is that if diversity condition is met, then models are linearly identifiable, but nothing is said about approximately linearly identifiable models as the ones shown in Figure 1. One question that should be addressed is how a perturbation in the diversity condition is translated to the identifiability property. It would be good that the authors add some theory about it.\n-\tIt is suggested that eq. (1) holds for arbitrary supervised learning problem but, in the Appendix, the equivalence is not shown for this case. Also, the example provided in Fig. 2 seems to be a very arbitrary case. Is it possible to write any supervised learning problem as eq. (1)? Could you please include its derivation?\n-\tIs the form of eq. (1) already used as a general approach for different ML problems as the authors claim? If so, could you please cite relevant references?\n-\tDiversity condition means that such invertible matrices L’ and L* can be constructed from data samples. What can be said if those matrices are invertible but ill-conditioned, i.e. smallest singular value close to zero?\n-\tFor supervised learning the condition implies that K>= M+1. Is it not a very restrictive condition? For example, it is possible to apply this analysis to a simple 2-classes supervised classification setting.\n-\tAgreement between theory and experimental results. With exception of the first experiment (supervised learning) the rest of the experiments show that by increasing iterations, dataset size and number of hidden units, then functions f and g tends to be linearly related (but not closely linear related). I don’t see why the main theoretical result of the paper (Theorem 1) is related to the experimental observations. I disagree with the authors claim that “these experiments validate Theorem 1”", "extra_scores": {}, "extra_text": {}}, {"review_id": "JwsivZGcCuB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigated the identifiability of the learned representations in pre-trained DNN models that fall into a general class of function space,  defined by the canonical mathematical form.\nThe identifiability of learned representations, in this paper, is defined as the representations are reproducible on the same data\ndistribution, regardless of the randomness in the training procedure such as the random initialization of parameters and the stochastic optimization procedure.\n\nThe authors first proved that, in the limit of infinite data, learned representations in this family are asymptotic identifiable in function space up to a linear transformation, with the additional assumption of the diverse condition. \n\nThey further showed that this property is applicable to several state-of-the-art pre-trained models, including CTC, BERT and GPT-2 and GPT-3.\n\nAt last, the authors conducted experiments to empirically investigate the linear identifiability of DNN models in a practical setting: finite data and partial optimization.  They adapted Canonical Correlation Analysis (CCA) (and SVCCA for high-dimensional space) to measure the linear similarity between two learned representations. Results from three sets of experiments, including classification, self-supervised learning for images (CTC) and for texts (GPT-2). Results show that the learned representations, after mapping through the optimal linear transformation from CCA, have a strongly linear relationship.\n\nOverall, this paper is well-written and well-motivated. From the theoretical aspect, this paper proved the linear identifiability of a large class of DNN models in an ideal setting. From the empirical aspect, they provided experimental results to demonstrate the theorem in a practical setting.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IiJ4fWRXFUn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper claims that, through Canonical Correlation Analysis on the representations learnt by popular deep models, we can show that the representations learnt on the same dataset, with same model using different sets of parameters, the representations are linearly identifiable. \n\nThis seems like a nice thing to know. But I however am not sure about few things. \n1) I think the paper doesn't really discuss the implications of this enough. That is, I would like to see few example applications where we take advantage of this fact. As is, this contribution in my opinion remains as a cute thing to know. \n2) It would be interesting to how much would change in initializations would effect the CCA curves. That is, if I initialize my network within a wider range, the identifiability result would still hold in practice? This would be interesting to add in my opinion. \n3) Ideally I would like to see if this statement hold for different hyperparameters also. \n\nOverall, the conclusion is interesting and I think that it could be published. \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2rgCejHx-c", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "%%% post-rebuttal %%\n\nThe authors replied to my comments related to the diversity condition in 3.2 and Theorem 1. Their answers did not fully clarify my concerns or misunderstandings, and it seems the authors didn't make any changes in that regard in the revised version. Except if I missed something in the review system, they did not answer my other comments.\n\n%%%%%%%%%%%%%%%\n\n\nThe paper discusses identifiability of learnt representations in supervised settings and under \"canonical discriminative\" modelling. It provides conditions under which two learnt representations are equivalent up to a linear transform. The paper discusses an important topic (related in particular to explicability of DNNs) and appears to provide valuable results but is difficult to follow. The paper is packed with results and difficult to read as important details are missing. A longer journal version would be more appropriate.\n\nComments:\n\n- I got lost from the paragraph \"Diversity condition\". I don't understand the meaning of y_A and y_B. What do subscripts A and B refer to ? Then what do y_A^(i) and y_B^(i) refer to ? What are they exactly samples of ? Using illustrative examples would be very useful there. For example, what do they mean in classification where y is a label ?\n\n- Because I didn't understand the meaning of y_A and y_B, I failed to understand the impact of Theorem 1. In particular, I don't understand why Eq. (5) holds in the proof.\n\n- I am mildly familiar with the \"canonical discriminative\" formulation. Can all DNN tasks but represented as such ? How does it relate to more traditional forms of deep learning, such as based on minimisation of cross entropy or quadratic loss ? I assume that the sum in Eq. (1) is an integral when y is continuous.\n\n- I got the idea of using CCA for comparing two different representations but the exact definition of C_i is unclear. Is it a vector or matrix ? Besides, why using only a subset B of D ?\n\n- In the experiments you basically compare two learnt representations and show they are essentially similar up to a linear transform. However it's not clear how the two representations have been obtained. Is it only a matter of different initialisations ?\n\n- I didn't understand how are the labels constructed in Section 5.1. Do you simply divide the unit circle in K=18 vectors that form the mean of each cluster ? I didn't get the \"model misspecification\" argument and the need to use DNN on such a simple task.\n\n- \"This show that increasing model size correlates strongly with increase in linear similarity of learned representations.\" I understand why this can be true when increasing number of samples but I don't understand why this should be expected with model capacity. What is it in Theorem 1 that reflects this ?\n\n\nMinor:\n- homogenise use of \"Equation (x)\" and \"equation (x)\"\n- missing compiled reference at the beginning of Section 4\n- it looks like you are using two different fonts for f_theta in the paper. Or are those different variables ?\n- many typos: \"share parameters\", \"but OF the network\", \"a the\", ...\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents novel results on linear identifiability in discriminative models, with three of the four reviewers arguing for acceptance. The paper went through an extensive round of edits, which incorporated detailed responses to issues raised by the reviewers. 

While this paper would be a nice contribution to the conference, some reviewer concerns remain unresolved, so we encourage the authors to revise and resubmit to a future venue.
Decision: Reject