Learning relevant features for statistical inference
Authors: Cédric Bény
Venue:  (2020)
Keywords: unsupervised learning
non-parametric probabilistic model
singular value decomposition
fisher information metric
chi-squared distance

Abstract: We introduce an new technique to learn correlations between two types of data.
The learned representation can be used to directly compute the expectations of functions over one type of data conditioned on the other, such as Bayesian estimators and their standard deviations. 
Specifically, our loss function teaches two neural nets to extract features representing the probability vectors of highest singular value for the stochastic map (set of conditional probabilities) implied by the joint dataset, relative to the inner product defined by the Fisher information metrics evaluated at the marginals.
We test the approach using a synthetic dataset, analytical calculations, and inference on occluded MNIST images. 
Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective.
We also explore using this approach to discover salient independent features of a single dataset. 

Under review as a conference paper at ICLR 2020
LEARNING RELEVANT FEATURES FOR STATISTICAL IN- FERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Given two views of data, we consider the problem of finding the features of one view which can be most faithfully inferred from the other. We find that these are also the most correlated variables in the sense of deep canonical correlation anal- ysis (DCCA). Moreover, we show that these variables can be used to construct non-parametric representation of the implied joint probability distribution. This representation can be used to compute the expectations of functions over one view of data conditioned the other, such as Bayesian estimators and their standard deviations. We test the approach using inference on occluded MNIST images, and show that our representation contains multiple modes. Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared the cross-entropy ob- jective. We also explore using this approach to discover salient independent vari- ables of single dataset.
INTRODUCTION
Given samples (I1,31),..... (2i, (.), from an unknown joint probability distribution p(z, y). we want to construct useful representation of the conditional probabilities p(aly) and p(y(zr), so that we that we can infer one view from the other on new data.
For instance, and y could be past and future histories of dynamical data, visual and auditory inputs, actions and their effects, etc.
Following the approach introduced in Bény & Osborne 2013 2015b) in the context of quantum information theory, we look at the problem as follows: the conditional distributions p(y|æ) can be thought of representing noisy communication channel (stochastic map). This channel is linear map between spaces of typically ludicrously large dimensions (the spaces of all probability distributions over y). We want pair of small subspaces which best represent the channel. Specifically, we look for those vectors representing probability distributions over x which lose least distinguishabiiity under the channel, where the distinguishability measured by the x2 divergence.
We show in Section 3 that this is equivalent to performing a certain singular value decomposition of the channel (seen an operator in Hilbert space) and keep only the components with the largest singular values. Moreover, the full singular value decomposition is equivalent to the decomposition in terms of canonical variables introduced Lancaster (1958). namely,
D
p(u]z) = p(v) Σ man(()))
(I)
where u4 and uy real non-linear functions such that E(uu,) 8.g, B((uvy) dej, and 0 In E ... S m < 7/0 = 1 are the singular values.
The practical advantage of this representation for inference (or prediction) is that it reduces the evaluation of conditional expectations to that of empirical averages over the (unconditional) marginal p(y).
As observed in Michaeli et al. (0016), the span of the first i canonical variables u4 By is what is learned by the deep canonical correlation analysis (DCCA) (Andrew al. 2013). Indeed, these

---

Under review as a conference paper at ICLR 2020
variables those which maximize the correlations E(u,2;) subject the same constraints as above. (This reduces to CCA (Hoteliing 1936) when u, by are linear maps).
In this work, besides establishing this new pntemmattoon-hcoretiee interpretation of canonical vari- ables and DCCA, we experiment with using this representation for performing inference on new data. Moreover, propose strategy for extracting disentangled variables from the canonical variables, inspired by analytical solutions.
2 RELATED WORK
This general problem (of building an effective representation of the conditional probability distribu- tions implied by joint samples) covers many existing approaches different contexts. For instance, if the variables has few possible states, then it reduces to a classification problem, usually solved by minimizing the crossentropy between predicted distribution and the one-hot encoding of the classes.
When y has a large number of states, or is fundamentally continuous, existing approaches usually do not model the whole conditional distribution, but either provide the average (regression), or approximately sample from it.
The main class of methods which allows for sampling from the conditional distributions varia- tional: a deterministic neural networks produces the parameters of analytical classes probabilities. This includes variational autoencoders (iingggm & Welling (2013) (e.g..Itten al. (20T8); Wang al. 2016 ) and approaches based on the minimum description length principle such as Gregor et al. 2013
Alternatively, it may also be possible to use adversarial training Goodfellow et al. (2014), by using a conditional Mirza& & Osindero (0114) version of an energy- -base GAN Zhao al. (2016).
By contrast, our approach doesn't it require the training of generative model. Instead, conditional expectations are constructed as linear combinations of unconditional empirical averages over the training data.
Previous approaches to equipping CCA DCCA with information-tthereed interpretation have explored different directions. For instance, in Wang et al. (2116). the authors generalize a probabilis- tic interpretation for CCA in terms of gaussian distributions, which leads to variational approach. In Painsky et al. (2018), additional constraints on the mutual information between the data and the learned variables are added the optimizations.
A previous attempt at designing numerical solution for our singular value problem can be found in Bény (2018b). In that work, the relevant variables were represented through a PCA kernel produced by Monte Carlo sampling, but wasn't 't practical.
THEORY
We formalize the problem by assuming that data was sampled from unknown joint distribution p(x, y) over two random variables X and Y.
Let Vx and Vy denote the linear spaces spanned by all probability distributions over X and re- spectively. Here we assume that X and Y take finitely many values for simplicity, but this formalism can be straightforwardly extended
dimittediimessionll vector spaces.
We will need inner products on Vx and Vy, to make them into real Hilbert spaces. We use the Fisher information metrics evaluated the points p(z) and p(y) respectively (marginals of p(z, ((), that
is,
Σ (()) p(x)
and (v,V'yy Σ v(y)v(()) p(y)
(,,
for any vectors μ., nl' Vx and v, v/ E Vy. Below we also call the marginals px and py respectively when omitting their arguments (px (æ) = p(x) and py(u) (v) p(y)).
2

---

Under review as a conference paper at ICLR 2020
These inner products allow to define the x2 divergence:
(4 Px = (q Px. px)x:
(3)
which a measures of statistical distinguishability between q and px- Specifically, it quantifies how easy it is reject the null hypothesis that the state is px when it is actually q, based the empirical distribution obtained from independent samples. It is also the lowest order approximation of the Kullback-Lebber divergence.
The joint distribution p(z,y/ y) yields conditional distributions plylæ) and p(alyy) These can be un- derstood as the components (or kernels) of stochastic maps A Vx Vy and N Vx Vy respectively. Explicitely, if Vx and P/ E Vy, then the images M()) E Vy and (²) Vx are defined by
(v) Σ p(ula))u()) and N')()()) Σ p(alu)/(u)
These stochastic maps N and perform inference of one variable given some (possibly imperfect) knowledge about the other, with priors given by the marginals p(r) or p(y) of p(a, y) depending on the direction of the inference. Importantly, N is the transpose of N terms of the inner products defined above:
(v.N(n)) = (V),
(5)
We now have the tools to address the problem mentioned in the introduction. The distinguishability
between q E Vx and px after the action of the channel N is x²(V(g). py) since py NVxx)) Hence we want to find the distributions, which maximize the relevance Bény & Osborne (2013).
n x(((()).rr) ((()) (g.
(1,
where = 9 Px- The inner-product formulation makes it clear that this amounts to finding the eigenvector with largest eigenvalue for the symmetric map N which is also the singular vector with largest singular value for A. On can then go on to find the eigenvector with next largest eigenvalue and so on, which are automatically orthogonal.
In practice, the inner products are more tractable to compute if we express elements E Vx and Vy in terms of variables f and g as μ fx and PY g. or
μ(x) = (zx) and v(y) = p())
for all v, Indeed, this yields simply
==)')x=(fff) and (u,4)) E(gg').
We are now in measure make the connection with DCCA Andrew et al. (2013). Indeed, the aims of DCCA is to maximize the correlations corr(f.g) = E(fg) over function f(x) and g(y) such that E(f²) Egg²) 1. But, using μ = fx and Py we have
E(fg) = (v)N(
which is maximized by the left- and right- singular vectors μ and v of N with largest singular value. Given all the singular vectors 14 Px and v4 = -yg: with singular values 7)s we obtain the representation
N Σ mani
which, using more standard notation and the Kronecker delta 6z, yields Eq.
p(vl)) = (((())())=
p(z) Σ muill))((T)) -p()) Σ (1)()))((
where p4 Px and v4 Py
3

---

Under review as a conference paper at ICLR 2020
3.1 NON-DIAGONAL FORM AND RELEVANT VARIABLES
For the purpose of the optimization and inference, we do not need to full diagonal decomposition, but just functions f. pu/Px and 9j 1///Y,, i.j 1...., kg. which have the same span as the canonical variables u, and Ey respectively for i.j = 1, 1,...,ku ko (assuming that 7...... 7,, are the largest singular vector). Below we refer f. and 9j as the kg most relevant variables.
Because these functions may not be orthogonal, we need the covariance matrices
Kg = (4ss = B(Jsf), Lis = (Us, vy)x E(gg))) Asj = (v,, = -E(gsf)
Nij denote the components of N the sense that (fx = Σ, NopY 9s, then, using our inner products to isolate Nij. we obtain
-.-1A. A. Similarly, the components of N are N K AT This implies that the sum of the square of the singular values of N restricted to the spans of the vectors fx fs and gyg, for all i. which is what we want to maximize, just given by
Σ o = Tr Tr A
This is the DCCA objective. Below we use the objective function C= - (N*N), for the cosmetic reason that its optimal value is zero.
Moreover, the corresponding truncated representation of the conditional distribution is
p(yl=) = N ~ p(v)
(L LL' (1)))
where we used the fact that the components of 62 are 6, = k.)
Of course, This approach can produce faithful representation of the correlations only if N is actually close to being of rank kn (see Appendix B for more precise statement). we interpret the relevant subspace
space of probability over latent variable, this means that our latent variables have at most kn discrete states.
However, even if the rank ko corner of A is a good approximation, this strategy allows us to nevertheless do the correct inference on certain random variables, namely those which are in the span of the canonical variables!
Indeed, the exact conditional expectation of gb is (assuming D is the actual rank of N),
Σ (a(())p(la)= El AK AK-
W1=
D (
Ro
Σ (af,
(AK
where the last truncation exact if k < ky due to the assumption that the basis fa and 9j have the same span as the Ko largest right and left singular vectors of N respectively.
For instance, if p(z, y) is Gaussian, the canonical variables can be computed analytically, as in Lancaster (1958) or Beny (0188) in the multivariate case. Solutions for other distributions were also computed in Eagleson 1964).
Notably, for any two dimensional Gaussian, the space of k most relevant variables is simply spanned by the moments fn (2) z and gn) y for = O,..., k 1. Hence, in this case the first moments can be inferred exactly using only the & - 1 most relevant variables (See Appendix C
4 ALGORITHM
Let us explicit the algorithm resulting from the above analysis.
4

---

Under review as a conference paper at ICLR 2020
We assume that given independent samples (21, V1), (I1, (x1, y2),... from the otherwise un- known joint distribution p(z, y)-
We first perform DCCA (Andrew et al. 2013). That is, we need two independent deterministic feed- forward neural networks. The first maps x to a set of kn real-valued variables f1 The second maps different set of ko variables (1()), Iko (y).
The parameters of the neural networks are to be set to minimize the objective function
ko Tr (K- A A')
(7)
where the matrices K, L, can be approximated over a mini-batch (In, Mn). ). 12 = I,..., via
N N (Vnn) Au Σ (ym))f(Z..) (8)
Kij Σ (ZW)))
Lij
1
Σ
We found that, provided the batch size is sufficiently large compared to lo (about 10 times in our experience), this can be minimized using ADAM or direct gradient descent. However, to guarantee stability when using large ko. we needed to explicit the gradient of the objective function in order force the of the Moore-Penrose pseudo-inverses for K-1 and T,-1 in both the forward and backward passes, in addition to using 64 bits floats in these computations.
Once the relevant variables have been learned, we still need use the training data in second step. Indeed, suppose that we wish to use our model to infer the value of some function (2x), i.e. compute its approximate expectation value in terms the conditional distribution + p(xly). Then we need to store, for each variable j l....,kou ku, the quantities
Naau
1
9,
((xn)fs(zn))
(9)
-Nuull
where the average is to be taken on the full training batch (of size Nfa1). The same can be done exchanging x with y and fj with gj for the reverse inference.
For instance, if data point is composed of real components suchuch as pixel color components for an image-and we are interested in the estimator which minimize the expected 12 distance to the predicted values of these components, then need the expectation values of the components (xx) = x for all a, and possibly higher moments to gain more knowledge about the shape of the posterior distribution, such as the second moments O'(x) x², etc.
Inference can then performed with new data using
No Σ 5/=1
t Σ (())))
(10)
Moreover, the accuracy of these predictions does not depend the rank ko if θ is taken in the span of the relevant variables, i.c., ((z) c(f(())))) for which θ, -he c.Kig
The reverse inference formulas are obtained simply by the exchanges K L. A AT, and ++ St.
5 EXPERIMENTS
In all our experiments, we used the ADAM optimizer with learning rate 0.001. We used the Flux package (Innes) 2018) for Julia, well as Tensorflow.
As usual the data is divided into a training set and a testing set. No aspect the testing set used during training. The loss function refers Eq. ( In order to monitor overfitting, compute "test loss" and a "training loss". The test loss is computed from the trained variables using only the test data, and accordingly, the training loss is computed purely using the training data.
Moreover, when performing inference on test data using Eq. (I0), we use the covariances A, L, K and expectations θ (Equ. ( built from the training data only.
5

---

become sig- Under review as a conference paper at ICLR 2020
5.1 INFERENCE ON OCCLUDED MNIST
In this experiment, we the left and right halves of the MNIST digit images correlated variables X and Y. The goal is to obtain the expected left halves given the right halves, or vice versa.
The training set was augmented by random small rotations and displacements make the task more ambiguous, as we want to explore the uncertainty in the prediction.
The relevant variables were represented by two convolutional neural networks of identical architec- ture. They are composed of four convolutional layers and one fully connected layer, an architecture that performs well for supervised learning on this dataset. For ease of implementation, these CNN have the whole image as input, but with either half zeroed (same value as black pixels). Half-width CNNs with proper padding at the cut perform similarly.
After training, we used the training dataset to also compute the expected pixel gray value as well as their covariance for each relevant variable using Eq. (9).
These were used into Eq. (T0) to compute the mean pixel gray values and their covariances over the conditional probability X given Y on test data. This mean the Bayesian estimator for the 12 distance between half images, i.e., it should minimize the expected distance dæ over the conditional distribution, where (z. y) Σ, (a: y; where x; E [0, 1] is the value of the ith pixel. (This is equivalent to minimizing the mean square error).
The results on a randomly selected subset test digits shown Fig. D For each example, we also computed the images obtained by adding plus minus one standard deviation along the direction of greatest variance in the space of relevant variables. This reveals the main ambiguities (such as between 8 and 3 and which share similar right half).
The graph of the singular values shows that the rank cutoff of 200 too low to capture all of the relevant variables (the sudden drop at the end is not robust to increase in the cutoff), but the sults are reasonable nevertheless. This shows that our representation of the conditional distributions contains valuable information besides the simple mean.
5.2 SUPERVISED LEARNING
In the context of a supervised classification task, one of the dataset (the labels) is of sufficiently low dimensionality that we can complete hasis over its probability space our relevant variables, such as the standard one-hot encoding of labels. This serves as good first sanity test for our approach. Surprisingly, we find that converges faster than standard approaches, and without the need for regularization.
Let the variable stands for the labels, with values {1....,k) k). The probability space consists of vectors with & real components. The canonical basis corresponds to the one-hot encoding g)) 5,j (Kronecker delta). All we need neural network to encode & variables /I,.... fA on X. After learning the most relevant variables fi, we apply the reverse of Eq. (1O) for function ( and use the maximum component of expected value ÿ to infer the labels from the data.
Let us refer to this procedure as DCCI (Deep Canonical Correlations based Inference).
We tested this approach on the MNIST and CIFARIO datasets, and compared the results to the standard cross-entropy objective (Fig.
We plotted the accuracy as function of the epoch rather than clock time which would depend on many factors. But the time per epoch is roughly the same for each approaches in the above experi- ments. Indeed, the training time is dominated by the forward and backward evaluations of the neural networks which identical. (However, the time it takes evaluate our objective nificant for much larger number of labels k. since it involves the inversion of matrices of dimension k. This is in addition the fact that greater dimension would require also larger batches.)
We found that, without regularization, simply changing the objective from cross- cross-entropy to DCCI provided large improvement both of convergence speed and final accuracy for both models.

---

Under review as a conference paper at ICLR 2020
4
8
3
5
5
5
9
1
test loss
training loss
O
O
0
6
5
5
5
3
3
8
10a
5
5
5
7
2
3
9
B
B
8
83
7 3
7 22 9
10
9
7
O
3
3
3
10
2
2
5
2
9
epoch
O
7
7
4
L.
6
K
9
4
7
7
7
kg =500
O
8
3
8
3
3
kg
4
0
0
4
4
4
6
4
3
3
3
9
7
9
0
O
5
5 4 5
6
4
+
0
2
2
8
3
3
4
4
9
P
P
2
3
3
3
5
5
5
8
8
8
5 O
5
0.6
100
150
200
250
O
7
8
8
0
0
singular values
Figure I: Left mosaic: the left halves of the MNIST digits in random sample from the test set inferred from the right half, with cutoff kg = 200. Three images are shown for each digit. Within each triplet. the middle image represents the mean over pixel intensity of the inferred condition distribution. while left and right images corresponds to plus and minus one standard deviation from the mean in the direction of largest covariance (in the space of half-images). interesting example is highlighted. Top-right: loss per epoch for ko 200. Bottom-right: the singular values for different values of the cutoff ko, after 150 epochs each case.
particularly
Inaccuracy on MNIST (%)
Inaccuracy on CIFAR10 (%)
2.0
40
CE
CE DCCI
CE, Dropout
1.5
DCCI
30
CE, BatchNorm
DCCI, BatchNorm
1.0
20-
0.5
0.0
0
30
40
50
10
20
30
epoch
epoch
Figure 2: Loss and inaccuracy (error rate) on test sets for two classification tasks. The models were trained either using the cross-entropy (CE) layers. On the MNIST dataset, we used an "all CNN" network, and for the CIFAR 10 dataset we used short VGG variation with 10 convolutions and 3 fully connected layers. In the regularized form, post-activation Batchnorm layers were placed after each convolutional layers on the VGG network. What shown is the mean over 10 independent runs for MNIST and runs for CIFARIO. The shaded area spans the standard deviation. ADAM with default parameters was used in all cases. No data augmentation was used except for horizontal flips for CIFARIO (resulting in epochs of 100,000 images).
approach (DCCI), with without regularization

---

Under review as a conference paper at ICLR 2020
Figure 3: Top-left: First 20 relevant variables (on X) determined by DCCA for system where X consists of two coordinates uniformly sampled over a circle and surrounding ring, and Y consists of the same points but shifted by small normally distributed vector. The variables are arranged from left-to-right and top-to-bottom in order of decreasing relevance. Top-right: the same variables multiplied by the marginal px. Bottom row: introducing a gap ring allows for a monotonous function of the angle to serve as second most relevant variable (instead of the sine/cosine couple). Hence the angle is automatically "disentangled" from the other variables. (Mid-gray represents the value 0).
On MNIST, DCCI alone also outperformed cross-entropy with dropout. (Dropout did not yield any improvement in conjunction with DCCI). However, adding batch-normalization layers on the CIFAR example, erased any distinction between DCCI and cross-entropy.
5.3 STRUCTURE OF THE RELEVANT VARIABLES
We mentioned in Section 3 that if p(x, y) is two-dimensional Gaussian distribution with mean, then the most relevant variables of X are the first powers of X itself, independently of the covariance matrix. This implies that the canonical variables are the Hermite polynomials in X (which results from applying the Gram-Schmidt procedure to the basis
A similar property holds for multivariate Gaussians, namely, the less relevant singular values polynomials in the more relevant ones. If this is true more generally, it should be possible to further compress and organize the latent space extracted with DCCA by finding minimal of generators, which ought to also be in the span of the most relevant variables.
We applied DCCA to a synthetic dataset to explore this idea, shown in Fig. In this case, we actually performed final SVD to obtain the unique uncorrelated canonical varialles, and ordered them by decreasing relevance (their respective singular values).
Here, X consists of two real numbers, distributed uniformly within ring and a disk. The variable Y is obtained by adding a random Gaussian shift to with small standard deviation. The more relevant variables ought to be those which are more robust to such small random displacement. This formalizes the idea that we are interested in extracting "large-scale" variables Bény (2018b).
We would expect the relevant independent variables to be: the binary variable indicating whether the point is in the disk or the ring and the angle around the ring, followed by the radial component in the ring, and finally the Cartesian coordinates inside the disk. This is precisely what see in Fig. 3
Indeed if we put aside for now the fact that the angle itself is not directly represented besides the constant function, the two most relevant variables are the sine and cosine of the angle, followed by the binary variable separating the disk from the ring.
But these variables ought to span the space of probabilities over the relevant variables, not just the variables themselves. Hence the next six variables are sines and cosines of smaller wavelength,
8

---

Under review as a conference paper at ICLR 2020
reconstruction error vs latent dimension
5 7 8 G
2 $ S 534
9 5 g
2
3
9 3
5 0
5 7 0 0
3 5 9 7 4
3 8
6 6 62 3 0 1 772
0 17244
3 5
2
5 0 10332
77 1179/
5 5 55844
7 à
8
8 1 7
5
10
9 19306
3 5009
6 3 8 9
10
0 0830/
3 3 07
Q 9
6 3 9 4 8
4 8 5
Y 0 0544
9 6
10
8 850/9
9 79252
6 9 6 377
2 1
100.0
10
10 00
10..2 1.25
10
3 6 17
4 33502
latent dimension
19
Figure 4: Left: Best mean squared error for images reconstructed from the most relevant variables, function of (the latent dimension). This logarithmic plot shows that improvements stop once the dimension reaches 19 (where the two lines cross). Right: images produced by the generator from latent variables sampled according the best Gaussian fit in latent space, for feature subspaces of dimensions 2, 8 and 19.
which can encode probability distributions which are increasingly more precisely localized, down to a precision (wavelength) comparable with the diameter the inner disk. Accordingly, next two most relevant variables are the Cartesian coordinates inside the disk. This is followed by additional moments of the angle, down to wavelength equal to the ring's s thickness, at which point we see the radius in the ring appear.
As mentioned, we see that the angle itself is not represented, likely because discontinuous. However, shown also in Fig. creating gap in the ring allows for the angle emerge as most relevant variable. This suggests that this approach may able to automatically learn intrinsic coordinates of the latent variable manifold.
5.4 INDEPENDENT VARIABLES AND GENERATIVE MODEL.
we postulate that the independent (or disentangled) relevant latent variables can be found in the linear span of the relevant variables, composed of two parts. Firstly, a linear layer maps the relevant variables to small number of outputs (equal to the latent dimension). The purpose of this linear layer is find the independent variables. These latent variables are then processed by an arbitrarily complex generative network to produce a possible value of the variable X. As objective function, we may an appropriate measure of similarity between the output and the data element from which the variables were obtained.
attempt extract them by optimizing neural network
We tested this idea as follows. We took X consist of the MNIST digits, and produced by randomly permuting neighboring pixels in the image, until the mean displacement per pixel is of order In addition, we added independent Gaussian noise to the pixel values. (Hence the noise map ' simulates the coarse- coarse-graining channel introduced in Bény & Osbore (2015))).
As in the previous experiment, we do so to implement our intuition that the more relevant variables ought to be the ones which are of larger scale, or more robust to local perturbations.
The relevant variables of the clean images were produced by the same convolutional neural network in Section 5.2 while the variables of the coarse- coarse-grained images were extracted by network of the same geometry, but with half the number of filters and neurons.
We extracted the 1000 most relevant out of 1200 learned variables in this way. (The least relevant variables in this system happen to be highly dependent on the total number variables and hence cannot be trusted to be correct). As second step. trained linear layer coupled to network composed fully-comnected layers of 800 hidden neurons each. We refer to the number of output neurons in the first linear layer as the lalent dimension.

---

Under review as a conference paper at ICLR 2020
As imput, this network received the variables extracted from MNIST images using the above convo- lutional neural net (after it was fully trained using DCCA), and was trained to minimize the mean square error between its output and the original MNIST digit.
The resulting best mean square errors are shown in Fig. as function of the latent dimension. Here see distinet change of polynomial scaling law at dimension 19. Increasing the dimension further provides no improvement. This behaviour is compatible with our hypothesis that the extra variables are just functions of those first twenty variables (functions which are effectively re-implemented by the generative network).
Images generated by sampling from Gaussian approximation of the latent distribution for different latent dimensions are shown in Fig. Below dimension 20, most generated image can be recognized as specific digit.
6 OUTLOOK
We studied the classical (non-quantum) form of the theory introduced in Bény & Osborne] (2013). and found that the relevant observables that theory are just the most correlated canonical variables in the sense of DCCA Andrew et al. (2013), and can be learned effectively using standard machine learning methods.
This point of views DCCA provided us with several new insights. The first is that the learned relevant variables provide a useful representation joint probability distribution. We showed that performing inference using this representation Our experiments on halves of MNIST also show that the conditional distribution we obtain can effectively represent the uncertainty in the prediction of high-dimensional data.
outperform crossentropy in predicting classes.
second insight relates to the interpretation of the canonical variables as spanning directions in the space of probability distributions. As suggested by the gaussian solutions and our experiment on synthetic data, we postulate that the canonical variables are functions of a small number of independent generators contained in their span. This hypothesis is supported by our experiment on MNIST, but further work is required to find a way to cleanly extract these variables.
We have yet to explore the potential applications of one of the salient aspect of this approach to inference, the fact that the canonical variables learned using DCCA are also those which can be most reliably predicted, irrespective of the value of the cutoff. To see why this potentially significant, we observe that central feature of scientific exploration is that we are not so concerned with making predictions about some given variables, as much as we are with discovering variables which can be predicted.
Another important feature this approach is the fact that the resulting model allows for the direct evaluation of the expectation values in the posterior distribution without sampling. In particular this allows for the evaluation of credible intervals. Hence it should be especially suited to scientific applications where the ability quantify uncertainty is essential.
Finally, the relationship that we established with theory of quantum origin points towards a potential quantum generalization of DCCA that would apply quantum data, classical measurements of quantum systems.
ACKNOWLEDGMENTS
We would like thank Joël Bény and Raban Iten for helpful suggestions. We are also in- debted to an anonymous ICLR2020 referee for pointing out the connection between our approach and DCCA. This work was supported by the National Research Foundation of Korea (NRF-
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference machine learning, pp. 1247-1255, 20133
10

---

Under review as a conference paper at ICLR 2020
C. Bény and T. J. Osbore. Information geometric approach to the renormalisation group. Phys. Rev. A, 92:022330, 2015a. doi: 10.110/PhysReev422202222 (arXiv:1206.7004)
Cédric Bény. Coarse-grainee distinguishabilityy of field interactions. (arXiv:1509002299
Quantum, 2:67, 2018a.
Cédric Bény. Inferring relevant features: from qft to pea. International Journal of Quantum Infor- mation, 16: 840012, 2018b. (arXiv:102..0556))
Cédric Bény and Tobias J Osborne. Renormalisation as an inference problem. (arXiv: 2013.
Cédric Bény and Tobias J Osborne. The renormalisation group via statistical inference. New J. Phys., 17:083005, 2015b. doi: 10.11088/1377-225011/88880000 (arXiv:1402.4949)
GK Eagleson. Polynomial expansions of bivariate distributions. The Annals of Mathematical Statis- tics, 35(8): 1964.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- frika, (3):211-215)) 1936.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 6672-2680, 2014.
Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregres- sive networks. arXiv preprint arXiv: 8499, 2013.
Harold Hotelling.
Relations between two sets of variates. Biometrika, 28(3/4):321-377 1936.
Mike Innes. Flux: Elegant machine learning with julia. Journal of Open Source Software, 2018. doi: 10.21 101/100000ss0000.2.
Raban Iten, Tony Metger, Henrik Wilming, Lídia Del Rio, and Renato Renner. Discovering physical concepts with neural networks. (arXiv:1807.10300) 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. (arXiv:1312.611 2013.
HO Lancaster. The structure bivariate distributions. The Annals of Mathematical Statistics, 29 (3):719-736, 1958.
Tomer Michaeli, Weiran Wang, and Karen Livescu. Nonparametric canonical correlation analysis. In International Conference on Machine Learning, pp. 1967-1976, 2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 2014.
M. Ohya and D. Petz. Quantum entropy and its use. Springer Verlag, 2004.
Amichai Painsky, Meir Feder, and Naftali Tishby. An information-thherrtiic framework for non- linear canonical correlation analysis. arXiv preprint arXiv:1810.133: 13259, 2018.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis. arXiv preprint arXiv: 2016.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy- based generative adversarial network. arXiv preprint 1rX1v:1609.03//2 2016.

---

Under review as a conference paper at ICLR 2020
A EXTRA INFORMATION ABOUT THE ALGORITHM
A.1 ALTERNATIVE INTERPRETATION OF THE OBJECTIVE
If we write Pij == ff( and Gij = gj(M (V4) for the value of our variables on the dataset, then K $ FFF.L= -= N G and A = N
1 G F. The DCCA objective can then be written as
Tr Tr(K- - L A) Tr (PQ)
where P FF and Q G G) are the projectors the ranges of F and respectively. Hence, we are maximizing the overlap between those ranges (which represent pos- sible linear combinations of datapoints, respectively determined from variables of one the other correlated views.)
A.2 HEURISTIC
Batch size-In In our experiments, we observed that the batch size during training needs to be an order of magnitude larger than the number of variables (rank cutoff). When the batch size was too small, learning seemed to converge normally in terms of training and test loss, but resulted in variables which yield dramatically different losses when evaluated on larger batches, and yield spurious predictions.
Constant variables- The loss function C takes value between and ky- 1 because the constant variable always has relevance 1. The constant variable could be enforced priori rather than learned, which, due the objectíve, automatically forces the learned variables to have zero expectation values (be orthogonal to the constant variable). This might have advantages in certain circumstances, but in experiments we found that this sometime hindered convergence.
Invertibility issues-The covariance matrices K and L can be ill-conditioned, potentially causing the gradient to "explode" because of the inverses K-1 L-1 involved in the loss function. This can be avoided either by using the Moore-Penrose pseudo-inverse, or by replacing K K-1 by (K + e1) in the loss for some small positive number e, and likewise for L-1
Symmetries in the loss function-Thh loss only depends on the span the variables fa and gy. hence it has a very large group of symmetries. In particular, it is invariant under a change of the norm each variable independently from each other. Because of that, is preferable not to have linear last layer. Using a hyperbolic tangent as last nonlinearity worked in our experiments.
Regularization In all our tests, dropout had no beneficial effect. In fact, objective seems to already provide a form of regularization, as shown Section 5.2
B THEORY IN MORE DETAILS
We consider two correlated random variables X and Y with joint probability distribution p(z, y)- We assume that we are able to numerically evaluate expectations with respect to this distribu- tion, for instance because we can sample from We want to use this ability order to com- pute expectations with respect to the conditional distributions Pxly (zly) p(x, v)xxx) and Py]x(v)z) p(x, /// (v), where (z) = Σ, p(z, y) and py (3) = E.p,, y) are the marginals of p. Below sometime remove the subscripts X, X|Y or Y|X if there is no ambi- guity.
For instance, suppose generated samples of given ær through explicit knowledge of PYJx Then the evaluation of expectations with respect to Px is the subject of Bayesian inference. However, this is generally done in context where the variable X has low dimensionality and parameterizes a hand-crafted model. Our approach, however, is free such a model and the variable X can be of very high dimensionality.
12

---

Under review as a conference paper at ICLR 2020
B.1 INNER PRODUCT ON PROBABILITY VECTORS
In order to define our strategy, we need to equip the spaces of probability distributions for X and Y with inner product structure. Let focus on X, and assume that it takes discrete values avoid unnecessary technicalities. The set of probability vectors is convex subset of the real linear space Vx R". Let equip this space with the product
(u, p')x == E (())
(11)
Px(x)
for any P. 1¹ E Vx- We also write || (1,, p)xx Importantly, this depends explicitly on the fixed probability vector (x),,, which we took be the marginal of p(z, y). If px has full support, this makes Vx into a real inner product space. The same can be done for the variable Y, yielding the inner product (e, ,")y for v, Vy.
Had we interpreted y and pí as tangent vectors to Vx, considered as a manifold, this would be the Fisher information (Riemannian) metric, as in Bény Osborne (2015b). But this quantity is also meaningful for finite vectors: the induced norm distance between and any probability vector is the x2 diverrgene::
x (g. = (g px. px)
(12)
The set of conditional probability distributions PY xx form stochastic map, i.e., a linear map N Vx + Vr, u + (H), where
(())())= = Σ (x)x)))))u(()
(13)
for any j E Vx-
It is straightforward to check that the stochastic map V defined by
(²) Σ (x))(()))
(14)
is the transpose N of N with respect to the inner products we defined (Ohya & Petz 2004), i.c.,
for all v Vy and μ E Vx,
(u,N(u)) = (v),
(15)
Also, we observe that ((xx) PY and N' = Px.
B.2 EIGEN-RELEVANCE DECOMPOSITION
We can use the inner products on Vx and Vy to define singular value decomposition of the stochas- tic map N. That is, there an orthonormal family uI,..., uu Vx and an orthonormal family U....., Ux of Vy, such that
N(ux) リイ
(16)
for j = 1,...,k. For each j. Mi is singular value of N, whose square we call the relevance of the vector vy. Moreover 7, E [0, 1] since the x2 divergence is contractive under any stochastic map. Given that V is the transpose of N::
N N = Uy
(17)
Equivalently, Uj is an eigenvector of N N and vj is an eigenvectors of M o N both with eigenvalue 17.
Because N maps px py, we always have the dual eigenvectors un px and up Py with eigenvalue 1.
B.3 LOW-RANK APPROXIMATION
Typically, the dimension & of the space of probabilities is more than astronomically large. For instance, if the values of X consists of small 256 gray level images of 28 x 28 pixels, then k
13

---

Under review as a conference paper at ICLR 2020
25628* 10 108888_ However, in many case, only very few of these dimensions may be relevant for the purpose of inferring other variables.
The core of our approach is to approximate .N' and N by restricting them to the span of the first ko eigenvectors 3j and v/ with largest singular values 7z. That is, if we order the singular values 7j. j 1,... ..... k in decreasing order, we propose use the approximations
No(m) = Σ mu(u p)x²,
(18)
(V)
Σ
(19)
JSko
(20)
V and A respectively, for some kg typically much smaller than and any μ E Vx. e Vy.
We denote the components of No and Nó by ((y(>)) and q( (2). e.g.,
No) = Σ (()1=))(x))
(21)
Since No and Ng are adjoint, we can define g(2, y) = ( (v) = ((ylz)px(x)) Although the marginals of g(x, v) are the probability distributions px and py, the numbers q(z, y) are necessarily positive.
The quality of this approximation for given ko does not directly depend the dimensionality of X and Y, but only on the amount of correlations between the two variables. Our aim is to use ko small enough that the components of No and Ng can be computed explicitly.
Theorem 1. No is the map of rank ki which minimizes the average distance
(9(x, 3) 8))²
[o()))o666.)) - -N())6
p(z)p(u)
(22)
Proof. The low rank approximation No minimizes the distance ||No - -NIlF where
| =T (M' M)
(23)
is the Hilbert-Schhee (or Frobenius) norm (Eckart & Young 1936 This follows from the fact
that this is also the 12-norm of the vector of singular values of M. Let us find the explicit form of the trace. Each possible value of the variable X is associated with probability distribution (x) = 1 when x = y and zero otherwise. These distributions form an orthogonal basis of Vx, and have norms (8z, dz) 1/px(x) (x). Therefore,
Tr (M M.x(=)(0..MMMM....)
Σ Mx(2))IM(6511 a
B.4 RELEVANT VARIABLES
We express the elements μ E Vx and Vy in terms of the marginals px and py simple products:
p(x) px (x) and v(y) = pv (z)g(v) for all v, y. where f and g are real functions of and y.
(24)
The inner products then simply become correlations among variables. Using also μ = px f and v' = pyg'. we obtain
(1,1(x Σ (xx())(0)())
(25)
(,, Epr(w)ou)o'e))) gg'
(26)
14

---

Under review as a conference paper at ICLR 2020
These are simple expectation values with respect to which assumed is the type of quantity can evaluate for arbitrary functions f'
Since N is self-adjoint in terms of this inner product. its eigenvectors u, orthogonal, and hence the corresponding variables a, defined by (z(x) = px (x) are uncorrelated. Indeed,
J = (ui, Uz) x O,
(27)
for all i. Moreover, accounting for the cigenvector un px (corresponding the constant feature 40 (z) = 1 for all r))
a = 0
(28)
for of Hence we trivially have
ajaj a.a,
(29)
for all i.j # O.
Likewise for the eigenvectors of ~N 1f) = py (y)b,b())
b,b, = (oi, 21)r = 0 = bb,b.
(30)
for all i.j * O.
Importantly, this does mean that the variables U1, u2.... nor U1 02 ... are "disentangled", i.e., they are not statistically independent. These variables represent components in the space of proba- bility vectors, rather than the "sample" space. They should be understood spanning a subspace of the space of functions over the relevant independent variables. We discuss this in more detail in Section 5.3
B.5 CORNERS OF N AND LOSS FUNCTION
The final piece of puzzle need, is the ability express the components (corners) of N and A in the span of possible non-orthogonal families of variables.
Let therefore consider arbitrary families f1,..., Sto and 91,-..,9kg of variables, which respectively represent the vectors px fj E Vx and py E Vy.
Firstly, need matrices representing the components of the inner products on Vx and Vy. Those are the symmetric matrices
= fifs,
(31)
Lag (pYD:.PYg,l = DiGj
(32)
The components No of N are defined by
N(px Σ NOpY
(33)
Taking the inner product with Py obtain
N(px = Σ NeyLai
(34)
The left-hand side can be computed using Equ. 13 It the matrix
Asj =
(py 9c.
py mww (x)
(y
(35)
Σ
ple, u)gn (v) 9bJ5
3.9
Therefore, in matrix notation, Equ.
34
LN, or
N 1-1 A.
(36)
15

---

Under review as a conference paper at ICLR 2020
The components N₁ of N* are obtained by just swapping X and Y, yielding
N= = =
(37)
Hence the singular values of the corner of N defined by the variables fj and gj are just the square- root of the eigenvalues of the matrix N N K-1 L-1A. In order to find the variables ʃj and gj with the same span as the first kn eigenvectors Vjj UJ, we just need to maximize all the eigenvalues of N N. A simple way to this use (minus) the trace of N* as loss function, since it the sum of the square of the singular values. We call 'rr (N*N the relevance of the subspaces defines by the variables f ad 9i for all i. This yields the loss/cost function:
C= kb Tr (N ku - Tr (K L A).
(38)
Once optimal variables have been found, can obtain the components of the eigenvectors in the span of Skk through standard numerical diagonalization of N* N.
B.6 INFERENCE
The variables minimizing C' can be used to infer one variable from the other. For instance, given y. the inferred probability distribution over given by (xly) (6,))) where (y is when y y and zero otherwise. In order compute this, first need the components of the distribution 6w in terms of the family "yyy 9ko i.e., the real numbers (6y); such that
6g(3/) py Σ (6)))))))))))
(39)
where (r,pyó,)y 0 for all i. Taking the inner product with py 95. obtain
ko
(y) Σ
(6y)
(40)
where the left hand side is also just
(prgi = (z(
(41)
Therefore the components by are explicitly
(5y) = -C
(42)
follows that
Σ
(43)
Then, for instance, the expected inferred value of X is
Σ
( (v) (xx(x)
(44)
Gj
For the inference of Y from Φ, we have
(y)x( Σ NEa(K- (1)
(45)
16

---

Under review as a conference paper at ICLR 2020
C ANALYTICAL EXAMPLE
When p(a, y) any multivariate Gaussian distribution, everything can be computed analytically. Let us consider here the one-dimensional case. We use p(z) x exp (-x0//2220) /2/²) and the conditional p(ylæ) x exp (-(v - 2)2 /20)) That is, y is equal to x but with some added Gaussian noise. This gives
(xlr(zly) exp (- (x (w=
2- ( where
(46)
o² 72
It was show in Lancaster (958), that the most relevant subspace of dimension kn on the variable X simply spanned by the variables
fn(x) z",
(47)
ko 1. Similarly for Y;
gn (v) = y²"
(48)
This independence of the relevant variables on the detailed parameters of p is a general property of Gaussian joint distributions.
This means, for instance, that the most relevant feature 1) for predicting the value of X given Y= is simply Y itself. The higher order variables have to do with inferring extra aspects of the probability distribution over X.
A set of orthogonal variables can be obtain from the Gram- Sean--Shmiit procedure, which, if done from small large n much necessarily yield the cigenvectors un and Vy- For illustration purpose, let us work with the non-orthogonal vectors fn and In. keeping only the first ko 3 vectors.
The three matrices (correlators) we need can be easily computed:
1
2
1
72 02
K-
0
22
==
0
+2
0²
0
(49)
2
0
3r
T2
+
0
3(7²
+ 02)3
72
A =
0
72
(50)
72
0
(02 + 372)
We obtain
0
,2 72)
= A L
0
0
(51)
The eigenvalues of M can be read on the diagonal, and the corresponding eigenvectors are (1, (1,0,)). (0, 1, 0) and (-72.00 0, 1). which means that the eigenfunctions are in order uig (z) 1, u1 [z) and Uz
-
Because we are working with continuous variables, the true rank of A infinite, even for any finite cutoff on the singular values. Nevertheless, it is instructive to see how the approximate inference fares for rank kg 3. Given the value y for Y. the inferred distribution over X is
(T) = (x)
Σ (K-1A- )ugh'"
(52)
j.k=
The approximately inferred first and second moments of X is given by integrating the above times (resp. x²2) over æ. We obtain
T = 7 and 22 + - 7)
(53)
which are actually exact: they are equal to the first two moments of X overpxij as given in Eq. 446
In fact, it is easy to see that this would be true for the first ko 1 moments had we kept the kn most relevant variables.
17


Reviews:
[{"review_id": "rklS9UB6YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided.\n\nMy decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren't very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here.\n\n***\n\nDecision-related suggestions/questions:\n\n* Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting.\n\n* The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful. \n\n* There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples:\n\t* In the third paragraph under theory, \"...linear spaces spanned by the probability distributions...\" should probably be \"...linear spaces spanned by all probability distributions...\" (?)\n\t* The following sentence is a run-on. \n\t* In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n).\n\t* The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1.\n\t* Introducing a \"Bayesian estimator for an l^2 distance\" w/o explanation. What does this mean?\n\n* How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it?\n\n* Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features?\n\n* Is there an interpretation for why both spaces require the same feature dimension, k_0?\n\n***\n\nComments not related to decision:\n\n* It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: \"...produce a useful model of correlations... for the task of data generation...\"\n\n* Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option.\n\n* The last statement of the paragraph under (3) needs a reference.\n\n* It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of \"faster convergence\" should be demonstrated in wall-clock time.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyggZ2a-5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to find a map between two feature spaces to maximize correlation between them and to use the resulting map for inference. A theoretical exposition is given and some empirical results are provided showing that the approach speeds up convergence on supervised MNIST and can be used for image completion (again on MNIST).\n\nThe paper should be rejected for the following reasons. First, the approach looks very similar to deep CCA, but the connection is never mentioned. This connection needs to be clarified. The objective function needs to be clearly stated and related to the loss function in eq. (7). In particular, I would suggest to give a clear definition of the problem before delving into the theory in section 2. In its current version, it is difficult to assess how the parts of section 2 relate to the overall objective. The paper severely lacks in relation to relevant related work. Half(!) of the 14 referenced papers are by the author himself. This can be verified since the double blind review process is compromised as the paper links to code in the author’s public github account. Finally, the empirical results are quite incomplete. It is not clear how the results compare to generative methods like VAEs, which are referenced as a motivation for this work in this work.\n\nThe improvement in convergence from RFA for supervised learning is interesting and this aspect deserves more analysis. It would be useful to look at the total amount of computation required to reach a given loss. I also wonder how this differs from simply mapping the output of the first network to a low-rank space via PCA. Is the dual-view really necessary in this case since the information content in the label space must be very limited, beyond simple class balance statistics?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklaGd7KKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes how the correlation between two different types of data can be extracted from learned representations. The proposed metric can also be used as an alternative to cross entropy loss. The paper provides analytical calculations as well as real data sets simulations/experiments. However there are significant draw-becks:\n\n1) Similarity and Metric Learning is a booming area in machine learning with several different directions focusing on different problems. The paper fails to locates itself in the literature, how it compares itself into other techniques (both analytically and experimentally). \n\n2) The proposed technique seems to be very similar to SVD of learned representations. Connection to quantum field theory is well established but more simpler comparisons to SVD and other spectral techniques are not provided in metric learning.\n\n3) Novelty is not clear. There are interesting experiments in disentangled feature feature extraction and data generation. However, they are mostly proof of concept and lack of baselines. It is not clear what problem this technique solves better compared to other existing solutions. \n\nPaper is mostly written clearly. I do suggest putting Appendix A1 to the main paper though.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes an approach for estimating cross-correlations between model outputs, related to deep CCA. Authors note that the procedure improves results when applied to supervised learning problems.

The reviewers have pointed out the close connection to previous work on deep CCA, and the author(s) have agreed. The reviewers agree that the paper has promise if properly expanded both theoretically and empirically.
Decision: Reject