Dimensionality Reduction for Representing the Knowledge of Probabilistic Models

1 INTRODUCTION . Dimensionality reduction is an important problem in machine learning tasks to increase classification performance of learned models , improve computational efficiency , or perform visualization . In the context of visualization , high-dimensional representations are typically converted to two or threedimensional representations so that the underlying relations between data points can be observed and interpreted from a scatterplot . Currently , a major source of high-dimensional representations that machine learning practitioners have trouble understanding are those generated by deep neural networks . Techniques such as PCA or t-SNE ( Van der Maaten , 2014 ) are typically used to visualize them , e.g. , in Law et al . ( 2017 ) ; Snell et al . ( 2017 ) . Moreover , Schulz et al . ( 2015 ) proposed a visualization technique that represents examples based on their predicted category only . However , none of these techniques exploit the fact that deep models have soft probabilistic interpretations . For instance , the output of deep classifiers typically employs softmax regression , which optimizes classification scores across categories by minimizing cross entropy . This results in soft probabilistic representations that reflect the confidence of the model in assigning examples to the different categories . Many other deep learning tasks such as semantic segmentation ( Long et al. , 2015 ) or boundary/skeleton detection ( Xie & Tu , 2015 ) also optimize for probability distributions . In this paper , we experimentally demonstrate that the soft probability representations learned by a neural network reveal key structure about the learned model . To this end , we propose a dimensionality reduction framework that transforms probability representations into a low-dimensional space for easy visualization . Furthermore , our approach improves generalization . In the context of zero-shot learning where novel categories are added at test time , deep learning approaches often learn high-dimensional representations that over-fit to training categories . By learning low-dimensional representations that match the classification scores of a high-dimensional pre-trained model , our approach takes into account inter-class similarities and generalizes better to unseen categories than standard approaches . Proposed approach : We propose to exploit as input representations the probability scores generated by a high-dimensional pre-trained model , called the teacher model or target , in order to train a lower-dimensional representation , called the student . In detail , our approach learns low-dimensional student representations of examples such that , when applying a specific soft clustering algorithm on the student representations , the predicted clustering scores are similar to the target probability scores . Contributions : This paper makes the following contributions : ( 1 ) We propose the first dimensionality reduction approach optimized to consider some soft target probabilistic representations as input . ( 2 ) By exploiting the probability representations generated by a pre-trained model , our approach reflects the learned semantic structure better than standard visualization approaches . ( 3 ) We experimentally show that our approach improves generalization performance in zero-shot learning . ( 4 ) We theoretically analyze the statistical properties of the approach and provide a finite sample error upper bound guarantee for it . 2 DIMENSIONALITY REDUCTION FOR PROBABILISTIC REPRESENTATIONS . ( DRPR ) Our method , called Dimensionality Reduction of Probabilistic Representations ( DRPR , pronounced “ Derper ” ) , is given probability distribution representations generated from high-dimensional data as target . Its goal is to learn a low-dimensional representation such that the soft clustering scores predicted by a soft clustering algorithm are similar to the target . If the targets are probability distributions generated by a pre-trained classifier , we want the low-dimensional space to reflect the relationships between categories interpreted by the classifier . The position of each example in the low-dimensional space should then reflect the ambiguity of the classifier for the example ( see Fig . 1 ) . We summarize in Section 2.1 the soft clustering algorithm that is used by DRPR in the low-dimensional space , the algorithm is detailed in Banerjee et al . ( 2005 ) . The general learning algorithm of DRPR is introduced in Section 2.2 . 2.1 FAMILY OF EFFICIENT SOFT CLUSTERING ALGORITHMS . Probability density : We consider that we are given a set of n vectors f1 , · · · , fn ∈ V concatenated into a single matrixF = [ f1 , · · · , fn ] > ∈ Vn . In the following , we consider V = Rd , and Vn = Rn×d . The goal is to partition n examples into k soft clusters . Each cluster Cc with c ∈ { 1 , · · · , k } has a center µ̂c ∈ V and its corresponding probability density is pc ( fi ) = exp ( −d ( fi , µ̂c ) ) b ( fi ) , where d is a regular Bregman divergence ( Banerjee et al. , 2005 ; Bregman , 1967 ) and b : V → R+ is a uniquely determined function that depends on d and ensures that the integral of the density over V is 1 ( e.g. , b ( fi ) = √ 1/ ( 2π ) d if d is the squared Euclidean distance ) . For simplicity , we consider that d is the squared Euclidean distance . The density pc ( fi ) decreases as the divergence between the example fi and the center µ̂c increases . Bregman Soft Clustering problem ( BSCP ) : The BSCP ( Banerjee et al. , 2005 ) is defined as that of learning the maximum likelihood parameters Γ = { µ̂c , π̂c } kc=1 of a mixture model p ( fi|Γ ) =∑k c=1 π̂c exp ( −d ( fi , µ̂c ) ) b ( fi ) where π̂c is the prior probability that fi is generated by Cc . To partition the n examples into k clusters , Banerjee et al . ( 2005 ) apply the EM algorithm to maximize the likelihood parameter estimation problem for mixture models formulated as : maxΓ ∏n i=1 p ( fi|Γ ) Assignment matrix : Partitioning the n observations in F into k soft clusters is equivalent to determining some soft assignment matrix Ŷ ∈ ( 0 , 1 ) n×k in the set Yn×k of matrices whose rows are positive and sum to 1 . Formally , Yn×k is written Yn×k : = { Ŷ ∈ ( 0 , 1 ) n×k : Ŷ 1k = 1n } where 1k ∈ { 1 } k is the k-dimensional vector containing only 1 . For a given value of Γ , the element Ŷic = p ( Cc|fi ) ∈ ( 0 , 1 ) is the posterior probability , or responsibility of Cc for fi . The higher the value of Ŷic , the more likely fi belongs to cluster Cc . Local maximum condition : Once the BSCP has converged to a local maximum of maxΓ ∏n i=1 p ( fi|Γ ) , the following equations are all satisfied : ( E-step ) ∀i , c , p ( Cc|fi ) = Ŷic = π̂c exp ( −d ( fi , µ̂c ) ) ∑k m=1 π̂m exp ( −d ( fi , µ̂m ) ) ( 1 ) ( M-step ) ∀c , µ̂c = ∑n i=1 Ŷicfi∑n i=1 Ŷic and ∀c , π̂c = 1 n n∑ i=1 Ŷic ( 2 ) Eq . ( 1 ) corresponds to the E-step of the EM algorithm that computes p ( Cc|fi ) = Ŷic when the parameters F and Γ are given . Eq . ( 2 ) corresponds to the M-step , which has a simple form since the likelihood is a regular exponential family function ( Banerjee et al. , 2005 ) . The M-step may be computationally expensive for other types of exponential family distributions ( Banerjee et al. , 2005 , Section 5.2 ) . It is worth noting that these optimality conditions do not depend on the function b used to define pc ( fi ) , so b can be ignored . 2.2 LEARNING THE MAPPING TO THE LOW-DIMENSIONAL SPACE . Section 2.1 explains how to perform soft clustering on some fixed representation F . We now describe how to learn F so that the soft clustering scores predicted by the BSCP match those of the target . We assume that we are given the probability representation yi ∈ [ 0 , 1 ] k as target for each training example fi . These representations are concatenated into a single matrix Y = [ y1 , · · · , yn ] > ∈ Yn×k which is the target of our method for F . DRPR learns the representation of F ∈ Vn such that the soft assignment matrix obtained from applying the BSCP on F is close to Y . We first give the formulation of our prediction function in Eq . ( 3 ) . Our dimensionality reduction problem is given in Eq . ( 4 ) . Prediction function : Let us assume that we are given the dataset matrixF = [ f1 , · · · , fn ] > ∈ Vn = Rn×d , the centersM = [ µ1 , · · · , µk ] > ∈ Vk = Rk×d and the priors π = [ π1 , · · · , πk ] > ∈ ( 0 , 1 ) k. As in Eq . ( 1 ) , we define our prediction function ψ ( F , M , π ) = Ψ ∈ Yn×k as the soft assignment matrix predicted by the BSCP given F , M and π . The elements of the matrix Ψ are then computed as follows : ∀i , c , Ψic = πc exp ( −d ( fi , µc ) ) ∑k m=1 πm exp ( −d ( fi , µm ) ) ( 3 ) Optimization problem : DRPR learns the representation of F so that the predicted assignment matrix ψ ( F , M , π ) = Ψ is similar to Y . Given the optimal condition properties of the BSCP stated in Section 2.1 , the optimal values of M and π also depend on Ψ and are therefore variables of our dimensionality reduction problem that we formulate : min F , M , π ∆n ( ψ ( F , M , π ) , Y ) ( 4 ) The function ∆n ( Ψ , Y ) is an empirical discrepancy loss between the predicted assignment matrix Ψ and the target assignment matrix Y . Since the rows of Ψ and Y represent probability distributions , we formulate ∆n as the average KL-divergence between the rows of Y and the rows of Ψ . Let ψ > i and y > i be the i-th rows of Ψ and Y , respectively , we formulate ∆n ( Ψ , Y ) = 1 n ∑n i=1DKL ( yi‖ψi ) . Note that the choice of the discrepancy loss ∆n is independent of the chosen Bregman divergence d. Moreover , the number of classes k has an impact on the number of clusters in the low-dimensional space but is not related to the dimensionality d of the model . DRPR considers that each class c ∈ { 1 , · · · , k } is represented by one cluster prototype µc ∈ Rd . Terminology : In our experiments , the target ( or teacher ) is the assignment matrix Y ∈ Yn×k that contains the probability scores generated by a pre-trained neural network . It corresponds to the output of a classifier trained with softmax regression in the visualization experiments , and to the matrices Y1 and Y2 described in Section 4.2 . The goal is then to learn F , M and π so that ( 1 ) F , M and π reach the BSCP optimality conditions given in Section 2.1 and ( 2 ) Ψ = ψ ( F , M , π ) is similar to Y . Visualization : DRPR can be used for visualization since many models ( e.g. , usual neural networks ) have probabilistic interpretations w.r.t . the k training categories . In our visualization task , the matrices M and π are not provided , whereas the target matrix Y is given . By using the optimality conditions Algorithm 1 Dimensionality Reduction of Probabilistic Representations ( DRPR ) input : Set of training examples ( e.g. , images ) in X and their target probability scores ( e.g. , classification scores w.r.t . k training categories ) , nonlinear mapping gθ parameterized by parameters θ , number of iterations t 1 : for iteration 1 to t do 2 : Randomly sample n training examples x1 , · · · , xn ∈ X and create target assignment matrix Y ∈ Yn×k containing the target probability scores y1 , · · · , yn ( i.e. , Y = [ y1 , · · · , yn ] > ∈ Yn×k ) 3 : Create matrix F ← [ f1 , · · · , fn ] > ∈ Vn such that ∀i , fi = gθ ( xi ) 4 : Create matrix of centers M ← diag ( Y > 1n ) −1Y > F and prior vector π ← 1nY > 1n 5 : Update the parameters θ by performing a gradient descent iteration of ∆n ( ψ ( F , M , π ) , Y ) ( i.e. , Eq . ( 4 ) ) 6 : end for output : nonlinear mapping gθ in Eq . ( 2 ) , we can write the desired values of M and π as a function of F and Y : at each iteration , for some current value of F , the optimal values M = diag ( Y > 1n ) −1Y > F and π = 1nY > 1n are computed and F is updated via gradient descent . DRPR is illustrated in Algorithm 1 in the case where F is the output of a model gθ parameterized by θ , e.g. , gθ can be a neural network . However , we represent F as non-parametric embeddings in our visualization experiments to have an equitable comparison to non-parametric baselines . The learning algorithm then modifies the matrix F at each iteration . If the priors πc are all equal , then the priors are updated in step 4 as follows : π ← 1k1k . Zero-shot learning : DRPR can be used to improve zero-shot learning generalization since highdimensional models may overfit to training categories and the goal of zero-shot learning is to generalize to novel categories . In the considered zero-shot learning task , the variable F concatenates image representations ( outputs of a neural network ) in the same way as step 3 of Algorithm 1 , and the variable M concatenates category representations extracted from text ( outputs of another neural network ) . Both F and M are of different nature and are therefore computed as concatenating the outputs of two distinct neural networks taking different sources as input . To optimize Eq . ( 4 ) , both neural networks are trained jointly by fixing the other neural network during backpropagation . Choice of divergence : In our experiments , we consider the squared Euclidean distance d ( fi , µ̂c ) = ‖fi − µ̂c‖22 . However , DRPR can be used with any regular Bregman divergence ( Banerjee et al. , 2005 ) . The algorithm is then identical , with the exception of the chosen divergence d to compute the prediction in Eq . ( 3 ) . Convergence and scalability : Although our problem has 3 variables ( F , M and π ) , we use the optimal properties of the BSCP to write them as a function of each other ( see step 4 of Algo 1 ) . Eq . ( 4 ) is then an optimization problem w.r.t . only one variable F . Since the problem is differentiable wrt F and unconstrained , it is easy to optimize by gradient descent ( e.g. , back-propagation when training neural networks ) . Moreover , our loss is nonnegative , hence lower-bounded . It is worth noting that we do not apply multiple iterations of the EM algorithm at each gradient descent iteration , as we first use the optimal properties of BSCP to obtain closed-form formulations of M and π , and then compute ψ ( F , M , π ) = Ψ to minimize our problem in Eq . ( 4 ) . Unlike t-SNE and many iterative DR problems , the complexity of DRPR is linear in n ( instead of quadratic ) and linear in k , which makes it efficient and scalable . Our visualization experiments take less than 5 minutes to do 105 iterations while t-SNE takes 1 hour to do 1000 iterations . PCA , which has an efficient closed-form solution , is still much faster . Our approach is thus simple to optimize , hence scalable , and it generalizes to a large family of Bregman divergences . Gradient interpretation : We now interpret the gradient of our optimization problem w.r.t . examples . To simplify its formulation , we consider that all the priors π1 , · · · , πk are equal and the matrix M does not depend on F , which is the case in our zero-shot learning task . When d is the squared Euclidean distance , all the priors are equal and M does not depend on F , the gradient of Eq . ( 4 ) w.r.t . fi is : 2 n k∑ c=1 YicΨic [ k∑ m=1 exp ( ‖fi − µc‖22 − ‖fi − µm‖22 ) ] [ −µc + k∑ m=1 Ψimµm ] ( 5 ) One can observe that its magnitude depends on both the target scores Yic ∈ ( 0 , 1 ) and the predicted responsibilities Ψic . The gradient tries to make the vector fi closer to each centroid µc while separating it from all the centroids µm depending on their predicted scores Ψim ∈ ( 0 , 1 ) . Statistical guarantee : We analyze the statistical property of the algorithm in Appendix A. Theorem 1 provides a finite sample upper bound guarantee on the quality of the minimizer of the empirical discrepancy of Eq . ( 4 ) . We show that it is upper bounded by the minimum of the true expected discrepancy , and an estimation error term of O ( n−1/2 ) ( under certain conditions and with high probability ) . We defer the detail to the appendix .

Summary: Authors propose a method of embedding training data examples into low-dimensional spaces such that mixture probabilities from a mixture model on these points are close to probability predictions from the original model in terms of KL divergence. Authors suggest two use-cases of such an approach: 1) data visualization, and 2) zero-shot learning. For the visualization use-case, authors compare against other dimensionality reduction methods with qualitative analysis on a synthetic problem, as well as evaluation metrics such as Neighborhood-Preservation Ratio and Clustering Distance Preservation Ratio. For zero-shot use-case, they take pre-trained models on two zero-shot tasks, and improve the accuracy by using probability outputs from pre-trained models as target.