Learning relevant features for statistical inference

1 INTRODUCTION . Given samples ( x1 , y1 ) , . . . , ( xn , yn ) from an unknown joint probability distribution p ( x , y ) , we want to construct a useful representation of the conditional probabilities p ( x|y ) and p ( y|x ) , so that we that we can infer one view from the other on new data . For instance , x and y could be past and future histories of dynamical data , visual and auditory inputs , actions and their effects , etc . Following the approach introduced in Bény & Osborne ( 2013 ; 2015b ) in the context of quantum information theory , we look at the problem as follows : the conditional distributions p ( y|x ) can be thought of as representing a noisy communication channel ( stochastic map ) . This channel is a linear map between spaces of typically ludicrously large dimensions ( the spaces of all probability distributions over x or y ) . We want a pair of small subspaces which best represent the channel . Specifically , we look for those vectors representing probability distributions over x which lose least distinguishability under the channel , where the distinguishability is measured by the χ2 divergence . We show in Section 3 that this is equivalent to performing a certain singular value decomposition of the channel ( seen as an operator in Hilbert space ) and keep only the components with the largest singular values . Moreover , the full singular value decomposition is equivalent to the decomposition in terms of canonical variables introduced in Lancaster ( 1958 ) , namely , p ( y|x ) = p ( y ) D∑ i=1 ηiui ( x ) vi ( y ) , ( 1 ) where ui and uj are real non-linear functions such that E ( uiuj ) = δij , E ( vivj ) = δij , and 0 < ηD ≤ · · · ≤ η1 ≤ η0 = 1 are the singular values . The practical advantage of this representation for inference ( or prediction ) is that it reduces the evaluation of conditional expectations to that of empirical averages over the ( unconditional ) marginal p ( y ) . As observed in Michaeli et al . ( 2016 ) , the span of the first k canonical variables ui , vj is what is learned by the deep canonical correlation analysis ( DCCA ) ( Andrew et al. , 2013 ) . Indeed , these variables are those which maximize the correlations E ( uivi ) subject to the same constraints as above . ( This reduces to CCA ( Hotelling , 1936 ) when ui , vj are linear maps ) . In this work , besides establishing this new information-theoretical interpretation of canonical variables and DCCA , we experiment with using this representation for performing inference on new data . Moreover , we propose a strategy for extracting disentangled variables from the canonical variables , inspired by analytical solutions . 2 RELATED WORK . This general problem ( of building an effective representation of the conditional probability distributions implied by joint samples ) covers many existing approaches in different contexts . For instance , if the variables y has few possible states , then it reduces to a classification problem , usually solved by minimizing the crossentropy between a predicted distribution and the one-hot encoding of the classes . When y has a large number of states , or is fundamentally continuous , existing approaches usually do not model the whole conditional distribution , but either provide the average ( regression ) , or approximately sample from it . The main class of methods which allows for sampling from the conditional distributions are variational : a deterministic neural networks produces the parameters of analytical classes of probabilities . This includes variational autoencoders Kingma & Welling ( 2013 ) ( e.g. , Iten et al . ( 2018 ) ; Wang et al . ( 2016 ) ) , and approaches based on the minimum description length principle such as Gregor et al . ( 2013 ) . Alternatively , it may also be possible to use adversarial training Goodfellow et al . ( 2014 ) , by using a conditional Mirza & Osindero ( 2014 ) version of an energy-based GAN Zhao et al . ( 2016 ) . By contrast , our approach doesn ’ t require the training of a generative model . Instead , conditional expectations are constructed as linear combinations of unconditional empirical averages over the training data . Previous approaches to equipping CCA or DCCA with an information-theoretic interpretation have explored different directions . For instance , in Wang et al . ( 2016 ) , the authors generalize a probabilistic interpretation for CCA in terms of gaussian distributions , which leads to a variational approach . In Painsky et al . ( 2018 ) , additional constraints on the mutual information between the data and the learned variables are added to the optimizations . A previous attempt at designing a numerical solution for our singular value problem can be found in Bény ( 2018b ) . In that work , the relevant variables were represented through a PCA kernel produced by Monte Carlo sampling , but wasn ’ t practical . 3 THEORY . We formalize the problem by assuming that our data was sampled from an unknown joint distribution p ( x , y ) over two random variables X and Y . Let VX and VY denote the linear spaces spanned by all probability distributions over X and Y respectively . Here we assume thatX and Y take finitely many values for simplicity , but this formalism can be straightforwardly extended to infinite-dimensional vector spaces . We will need inner products on VX and VY , to make them into real Hilbert spaces . We use the Fisher information metrics evaluated at the points p ( x ) and p ( y ) respectively ( marginals of p ( x , y ) ) , that is , 〈µ , µ′〉X : = ∑ x µ ( x ) µ′ ( x ) p ( x ) and 〈ν , ν′〉Y : = ∑ y ν ( y ) ν′ ( y ) p ( y ) ( 2 ) for any vectors µ , µ′ ∈ VX and ν , ν′ ∈ VY . Below we also call the marginals pX and pY respectively when omitting their arguments ( pX ( x ) ≡ p ( x ) and pY ( y ) ≡ p ( y ) ) . These inner products allow us to define the χ2 divergence : χ2 ( q , pX ) = 〈q − pX , q − pX〉X , ( 3 ) which a measures of statistical distinguishability between q and pX . Specifically , it quantifies how easy it is to reject the null hypothesis that the state is pX when it is actually q , based on the empirical distribution obtained from independent samples . It is also the lowest order approximation of the Kullback-Leibler divergence . The joint distribution p ( x , y ) yields conditional distributions p ( y|x ) and p ( x|y ) . These can be understood as the components ( or kernels ) of stochastic maps N : VX → VY and N ∗ : VX → VY respectively . Explicitely , if µ ∈ VX and ν ∈ VY , then the images N ( µ ) ∈ VY and N ∗ ( ν ) ∈ VX are defined by N ( µ ) ( y ) = ∑ x p ( y|x ) µ ( x ) and N ∗ ( ν ) ( x ) = ∑ y p ( x|y ) ν ( y ) . ( 4 ) These stochastic mapsN andN ∗ perform inference of one variable given some ( possibly imperfect ) knowledge about the other , with priors given by the marginals p ( x ) or p ( y ) of p ( x , y ) depending on the direction of the inference . Importantly , N ∗ is the transpose of N in terms of the inner products defined above : 〈ν , N ( µ ) 〉Y = 〈N ∗ ( ν ) , µ〉X . ( 5 ) We now have the tools to address the problem mentioned in the introduction . The distinguishability between q ∈ VX and pX after the action of the channel N is χ2 ( N ( q ) , pY ) since pY = N ( pX ) . Hence we want to find the distributions p which maximize the relevance Bény & Osborne ( 2013 ) . η ( q ) = χ2 ( N ( q ) , pY ) χ2 ( q , pX ) = 〈N ( µ ) , N ( µ ) 〉Y 〈µ , µ〉X , ( 6 ) where µ = q − pX . The inner-product formulation makes it clear that this amounts to finding the eigenvector with largest eigenvalue for the symmetric map N ∗N , which is also the singular vector with largest singular value forN . On can then go on to find the eigenvector with next largest eigenvalue and so on , which are automatically orthogonal . In practice , the inner products are more tractable to compute if we express elements µ ∈ VX and ν ∈ VY in terms of variables f and g as µ = pXf and ν = pY g , or µ ( x ) = p ( x ) f ( x ) and ν ( y ) = p ( y ) g ( y ) for all x , y . Indeed , this yields simply 〈µ , µ′〉X = E ( ff ′ ) and 〈ν , ν′〉Y = E ( gg′ ) . We are now in measure to make the connection with DCCA Andrew et al . ( 2013 ) . Indeed , the aims of DCCA is to maximize the correlations corr ( f , g ) = E ( fg ) over function f ( x ) and g ( y ) such that E ( f2 ) = E ( g2 ) = 1 . But , using µ = pXf and ν = pY g , we have E ( fg ) = 〈ν , N ( µ ) 〉Y , which is maximized by the left- and right- singular vectors µ and ν ofN with largest singular value . Given all the singular vectors µi = pXfi and νi = pY gi with singular values ηi , we obtain the representation N ( µ ) = ∑ i ηiνi〈µi , µ〉X , which , using a more standard notation and the Kronecker delta δx , yields Eq . 1 : p ( y|x ) = N ( δx ) ( y ) = 1 p ( x ) ∑ i ηiνi ( y ) µi ( x ) = p ( y ) ∑ i ηivi ( y ) ui ( x ) , where µi = pXui and νi = pY vi . 3.1 NON-DIAGONAL FORM AND RELEVANT VARIABLES . For the purpose of the optimization and inference , we do not need to full diagonal decomposition , but just functions fi = µi/pX and gj = νj/pY , i , j = 1 , . . . , k0 , which have the same span as the canonical variables ui and vj respectively for i , j = 1 , . . . , k0 ( assuming that η1 , . . . , ηk0 are the largest singular vector ) . Below we refer to fi and gj as the k0 most relevant variables . Because these functions may not be orthogonal , we need the covariance matrices Kij = 〈µi , µj〉X = E ( fifj ) , Lij = 〈νi , νj〉X = E ( gigj ) , Aij = 〈νi , N ( µj ) 〉Y = E ( gifj ) . If Nij denote the components of N in the sense that N ( pXfj ) = ∑ iNijpY gi , then , using our inner products to isolate Nij , we obtain N = L−1A . Similarly , the components of N ∗ are N∗ij = K−1A > . This implies that the sum of the square of the singular values of N restricted to the spans of the vectors pXfi and pY gj for all i , j , which is what we want to maximize , is just given by k0∑ i=1 η2i = Tr ( N ∗N ) = Tr ( K−1A > L−1A ) . This is the DCCA objective . Below we use the objective function C = k0 − Tr ( N∗N ) , for the cosmetic reason that its optimal value is zero . Moreover , the corresponding truncated representation of the conditional distribution is p ( y|x ) = N ( δx ) ( y ) ' p ( y ) k0∑ i , j=1 ( L−1AK−1 ) ijgi ( y ) fj ( x ) , where we used the fact that the components of δx are δj = ∑ iK −1 ji fi ( x ) . Of course , This approach can produce a faithful representation of the correlations only if N is actually close to being of rank k0 ( see Appendix B for a more precise statement ) . If we interpret the relevant subspace as a space of probability over latent variable , this means that our latent variables have at most k0 discrete states . However , even if the rank k0 corner of N is a not a good approximation , this strategy allows us to nevertheless do the correct inference on certain random variables , namely those which are in the span of the canonical variables ! Indeed , the exact conditional expectation of gk is ( assuming D is the actual rank of N ) , ∑ y gk ( y ) p ( y|x ) = D∑ i , j=1 ( L−1AK−1 ) ijE ( gkgi ) fj ( x ) = D∑ j=1 ( AK−1 ) kjfj ( x ) = k0∑ j=1 ( AK−1 ) kjfj ( x ) , where the last truncation is exact if k ≤ k0 due to the assumption that the basis fi and gj have the same span as the k0 largest right and left singular vectors of N respectively . For instance , if p ( x , y ) is Gaussian , the canonical variables can be computed analytically , as in Lancaster ( 1958 ) or Bény ( 2018a ) in the multivariate case . Solutions for other distributions were also computed in Eagleson ( 1964 ) . Notably , for any two dimensional Gaussian , the space of k most relevant variables is simply spanned by the moments fn ( x ) = xn and gn ( y ) = yn for n = 0 , . . . , k − 1 . Hence , in this case the first k moments can be inferred exactly using only the k + 1 most relevant variables ( See Appendix C ) .

Summary: The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided.