Target-Embedding Autoencoders for Supervised Representation Learning

1 INTRODUCTION . Representation learning deals with uncovering useful underlying structures of data , and autoencoders ( Hinton & Salakhutdinov , 2006 ) have been a staple in a variety of problems . While much research focuses on its use in unsupervised or semi-supervised settings with such diverse objectives as sparsity ( Ranzato et al. , 2007 ) , generation ( Kingma & Welling , 2013 ) , and disentanglement ( Chen et al. , 2018 ) , autoencoders are also useful in purely supervised settings—in particular , adding an auxiliary feature-reconstruction task to supervised classification problems has been shown to empirically improve generalization ( Le et al. , 2018 ) ; in the linear case , the theoretically quantifiable benefit matches that of simplistic norm-based regularization ( Bousquet & Elisseeff , 2002 ; Rosasco & Poggio , 2009 ) . In this paper , we consider the inverse problem setting where the target space Y is high-dimensional ; for instance , consider the multi-label classification tasks of object tagging , text annotation , and image segmentation . This is in contrast to the vast majority of works designed to tackle a high-dimensional feature space X ( where commonly |X | |Y| , such as in standard classification problems ) . In this setting , the usual ( and universal ) strategy of learning to reconstruct features ( Weston et al. , 2012 ; Kingma et al. , 2014 ; Le et al. , 2018 ) may not be most useful : learning latent representations that encapsulate the variation withinX does not directly address the more challenging problem of mapping back up to a higher-dimensional Y . Instead , we argue for leveraging intermediate representations that are compact and more easily predictable from features , yet simultaneously guaranteed to be predictive of targets . In the process , we provide a unified theoretical perspective on recent applications of autoencoders to label-embedding in static , high-dimensional classification problems ( Yu et al. , 2014 ; Girdhar et al. , 2016 ; Yeh et al. , 2017 ) . Extending into the temporal setting , we further empirically demonstrate the generality of target-embedding for recurrent , multi-variate sequence forecasting . Our contributions are three-fold . First , we motivate and formalize the target-embedding autoencoder ( TEA ) framework : a general approach applicable to any underlying architecture . Second , we provide a theoretical learning guarantee in the linear case by demonstrating uniform stability ; specifically , we obtain an O ( 1/N ) bound on instability by analogizing the benefit of the auxiliary reconstruction task to a form of regularization—without incurring additional bias from explicit shrinkage . Finally , we extend empirical validation of this approach beyond the domain of static classification : using the task of multivariate disease trajectory forecasting as case study , we experimentally validate the advantage that TEAs confer on both linear and nonlinear architectures using real-world datasets with both continuous and discrete targets . To the best of our knowledge , we are the first to formalize and quantify the theoretical benefit of autoencoder-based target-representation learning in a purely supervised setting , and to extend its application to the domain of multivariate sequence forecasting . 2 TARGET-EMBEDDING AUTOENCODERS . Let X and Y be finite-dimensional vector spaces , and consider the supervised learning problem of predicting targets y ∈ Y from features x ∈ X . With a finite batch of N training instances D = { ( xn , yn ) } Nn=1 , the objective is to learn a mapping h : X → Y that generalizes well to new samples from the same distribution . The vast majority of existing work consider the setting—most commonly , classification—where |X | |Y| ; under this scenario , autoencoders are often used to first transform the input into some lower-dimensional representation z ∈ Z amenable to the downstream task . Doing so involves adding an auxiliary reconstruction loss ` r to the primary prediction loss ` p. Formally , solutions of this form—in supervised and semi-supervised settings alike—consist of a shared forward model φ : X → Z , a reconstruction function r : Z → X , and a prediction function d : Z → Y during training ( where notation d reflects the downstream nature of the prediction task ) . Denote x̃ = r ( φ ( x ) ) and ŷ = d ( φ ( x ) ) ; then the complete loss function takes the following form , L = 1 N ∑N n=1 [ ` p ( ŷn , yn ) + ` r ( x̃n , xn ) ] ( 1 ) In contrast , we focus on settings where the target space Y is high-dimensional , and where possibly |Y| > |X | . In this case , we argue that learning to reconstruct the input is not necessarily most beneficial . In a simple classification problem , autoencoding inputs leverages the hypothesis that a reconstructive representation is also likely discriminative . In our setting , however , the more immediate problem is the high-dimensional structure of Y ; in particular , there is little guarantee that intermediate representations trained to encapsulate x are easily mapped back up to higher-dimensional targets . Our goal is to make use of intermediate representations that are both predictable from features as well as predictive of targets . A target-embedding autoencoder ( TEA ) —versus what we shall term a feature-embedding autoencoder ( FEA ) —flips the model architecture around by learning an embedding of target vectors instead , which a predictor then learns a mapping into . This involves an encoder e : Y → Z , an upstream predictor u : X → Z , and a shared forward model θ : Z → Y . Denote ỹ = θ ( e ( y ) ) and ŷ = θ ( u ( x ) ) ; the complete loss function is now of the following form , L = 1 N ∑N n=1 [ ` p ( ŷn , yn ) + ` r ( ỹn , yn ) ] ( 2 ) Abstractly , the general idea of target space reduction is not new ; in particular , it has been present in various solutions in the domain of multi-label classification ( see Section 4 and Appendix B for discussions of related work ) . Here we focus on target-embedding autoencoders ; they leverage the assumption that variations in ( high-dimensional ) target space are driven by a compact and predictable set of factors . By construction , learning to reconstruct directly in output space ensures that latent representations are predictive of targets ; at the same time , jointly training with the prediction loss ensures that latent representations are predictable from features . Instead of learning representations for mapping out of ( downstream ) , here we learn representations for mapping into ( upstream ) ; the shared forward model handles the rest . See Figure 1 for high-level diagrams of TEAs versus FEAs . Training and Inference . Figure 2 gives block diagrams of component functions and objectives in ( a ) FEAs and ( b ) TEAs during training ( see Algorithm 1 in Appendix C for pseudocode ) . Training occurs in three stages . First , the autoencoder is trained ( to learn representations ) : the parameters of e and θ are learned on the reconstruction loss . Second , the prediction arm is trained to regress the learned embeddings ( generated by the encoder ) : the parameters of u are learned ( on the latent loss ) while the autoencoder is frozen . Finally , all three components are jointly trained on both prediction and reconstruction losses ( Equation 2 ) : parameters of the predictor , embedding , and shared forward model are trained simultaneously . Note that during training , the forward model receives two types of latents as input : encodings of true targets , as well as encodings predicted from features . At inference time , the target-embedding arm is dropped , leaving the learned hypothesis h = θ ◦ u for prediction . Figure 4 ( Appendix C ) provides step-by-step block diagrams of both training and inference in greater detail . We emphasize that TEAs—as is the case with FEAs—specify a general framework independent of the implementation details of each component . For instance , the solutions to applications in Yu et al . ( 2014 ) , Yeh et al . ( 2017 ) , and Girdhar et al . ( 2016 ) can be abstractly regarded as linear and nonlinear instances of this framework , with domain-specific architectures ( see Section 4 and Appendix B for more detailed discussions ) . Linear TEAs—which we study in greater detail in Section 3—involve parameterizations ( Θ , Wu , We ) of ( θ , u , e ) consisting of single hidden layers with linear activation . 3 STABILITY-BASED LEARNING GUARANTEE . Two questions are outstanding . The first is theoretical . We are motivated by the prior that variations in target space are driven by a lower-dimensional set of underlying factors . In this context , can we say something more rigorous about the benefit of TEAs ? In this section , we take the first step in showing that jointly learning target representations improves generalization performance in the supervised setting . Specifically , we demonstrate that linear TEAs are characterized by uniform stability , from which theoretical guarantees are known to follow . The second question is empirical . We noted above that certain applications of label-embedding to classification can be interpreted through this framework . Does the benefit extend beyond its static , feedforward instantiations—into the temporal setting for multi-variate sequence forecasting , with both continuous and discrete targets ? In Section 5 , we first validate our theoretical findings with linear models and sensitivities , as well as extending our empirical analysis to the realm of recurrent , nonlinear models for both regression and classification . Consider a linear TEA , where the upstream predictor is parameterized by Wu ∈ R|Z|×|X| , targetembedding by We ∈ R|Z|×|Y| , and shared forward model by Θ ∈ R|Y|×|Z| , where |Z| < |Y| . The complete loss function is given by L = 1N ∑N n=1 [ ` p ( ΘWuxn , yn ) + ` r ( ΘWeyn , yn ) ] following Equation 2 . Interpreting the jointly learned autoencoding component as an auxiliary task , we show that the TEA algorithm for learning the shared forward model Θ is uniformly stable with respect to the domain of the supervised prediction task . To establish our notation , first recall the following : Definition 1 ( Generalization Bound ) Given a learning algorithm D 7→ hD that returns hypothesis hD , let R ( hD ) = ∫ ` ( hD ( x ) , y ) dµ ( x , y ) denote the risk , and R̂ ( hD ) = 1N ∑N n=1 ` ( hD ( xn ) , yn ) denote the empirical risk , where ` is some loss function . A generalization bound is a probabilistic bound on the defect that takes the following form : R ( hD ) − R̂ ( hD ) ≤ with some confidence 1− δ . Definition 2 ( Uniform Stability ) Let Di denote a modification of batch D where the i-th training instance ( xi , yi ) is replaced by an independent and identically distributed example ( x′i , y ′ i ) . A learning algorithm is said to be γ-uniformly stable with respect to the loss function ` if ∀D ∈ ( X ×Y ) N , ∀i ∈ { 1 , ... , n } , ∀ ( x , y ) , ( x′i , y′i ) ∈ X ×Y : | ` ( hD ( x ) , y ) − ` ( hDi ( x ) , y ) | ≤ γ . Uniform stability holds if the minimum value of γ converges to zero as batch size N increases without limit . Uniform stability can be used to derive algorithm-dependent generalization bounds . In particular , Bousquet & Elisseeff ( 2002 ) first showed that the defect of a γ-uniformly stable algorithm is is less than O ( ( γ + 1/N ) √ N log ( 1/δ ) ) with probability ≥ 1 − δ. Feldman & Vondrak ( 2018 ) recently demonstrated an improved bound of O ( √ ( γ + 1/N ) log ( 1/δ ) ) . Here , we show uniform stability for linear TEAs , where γ is O ( 1/N ) —by which a tight generalization bound follows immediately . Before we begin , we introduce two additional tools : c-strong convexity and σ-admissibility . Note that these conditions are standard and easily satisfied—for instance , by the quadratic loss function ; for more context see for example Bousquet & Elisseeff ( 2002 ) , Liu et al . ( 2016 ) , and Mohri et al . ( 2018 ) . Definition 3 ( c-Strong Convexity ) Differentiable loss function ` is c-strongly convex if ∀h , h′ ∈ H : 〈h′ ( x ) − h ( x ) , ∇ ` ( h′ ( x ) , y ) − ∇ ` ( h ( x ) , y ) 〉 ≥ c‖h′ ( x ) − h ( x ) ‖22 for some c ∈ R+ , where ∇ ` ( h ( x ) , y ) denotes the gradient with respect to h ( x ) , and 〈· , ·〉 denotes the dot product operation . Definition 4 ( σ-Admissibility ) Loss function ` is σ-admissible with respect to the underlying hypothesis classH if ∀h , h′ ∈ H : | ` ( h′ ( x ) , y ) − ` ( h ( x ) , y ) | ≤ σ‖h′ ( x ) − h ( x ) ‖2 for some σ ∈ R+ . To obtain uniform stability , we make two assumptions—both analogous to prior work arguing from the benefit of learning shared models between tasks . Liu et al . ( 2016 ) deals with learning multiple tasks in general , and Le et al . ( 2018 ) deals with reconstructing inputs in what we describe as FEAs . Now in multi-task learning , the separate tasks are usually chosen due to some prior relationship between them . In the case of Assumption 1 in Liu et al . ( 2016 ) and Assumption 5 in Le et al . ( 2018 ) , this is assumed to come from similarities in feature structures across tasks ; hence their assumptions of cross-representability are made in feature space . ( Note that this restricts primary and auxiliary features to be elements of the same space ) . Our setting is contrary : the inputs to primary and auxiliary tasks come from different spaces , but are trained to produce similar labels through a compact , shared latent space ; hence our assumption of cross-representability will be made in this latent space instead . Assumption 1 ( Representative Vectors ) There exists a representative subset of target vectors B = { b1 , ... , bM } ⊂ { y1 , ... , yN } such that the latent representation of any individual ( x , y ) can be linearly reconstructed from that of the representative subset with small error , i.e . Wpx =∑M m=1 αmWebm+η and Wey = ∑M m=1 βmWebm+η for some coefficients αm , βm ∈ R , where ΣMm=1α 2 m ≤ r2α , ΣMm=1β2m ≤ r2β for some rα , rβ ∈ R+ , and η is a small error satisfying ‖η‖2 ≤ ε . Remark 1 This assumption is comparatively mild , even for ε = 0 . Note that in Liu et al . ( 2016 ) the features for separate tasks come from different examples in general , and the similarity of their distributions within X is simply assumed . Here , each pair of inputs to the prediction and reconstruction tasks comes from the same instance , and similarity within Z is explicitly enforced through the ( joint ) training objective . In addition , observe that the assumption will hold with zero error as long as the number of independent latent vectors is at least |Z| . Furthermore , unlike the scenarios in Liu et al . ( 2016 ) and Le et al . ( 2018 ) we do not require that the input domains of the two tasks be identical . Therefore for ease of exposition , we assume going forward that ε = 0 ( see Remark 6 in Appendix A ) . Remark 2 A comparison with Assumption 4 in Le et al . ( 2018 ) sheds additional light on why we expect TEAs to be beneficial where |Y| > |Z| , in contrast with the ( more typical ) scenario |X | > |Z| |Y| . Critically , the technique in Le et al . ( 2018 ) banks on the fact that the prediction arm projects the latent into a lower-dimensional target space . Conversely , Assumption 1 here relies on the fact that the encoding arm maps into the latent space from a higher-dimensional target space ( rendering cross-representability therein reasonable ) . The distinction is crucial : we certainly do not expect any benefit from autoencoding trivially low-dimensional vectors ! Note also that here the representative vectors are taken from Y ; to take them from X instead would be unreasonable . For any compressive autoencoder , we generally expect if some subset { b1 , ... , bM } ⊂ { y1 , ... , yN } spans Y that { Web1 , ... , WebM } then also span Z in order to be maximally reconstructive . The same can not be said of subsets { c1 , ... , cM } ⊂ { x1 , ... , xN } that span X—for instance , take |X | ≤ |Z| . In addition to being representative in terms of latent values , the set of representative points also needs to be representative in terms of the reconstruction error . First , let L′ denote the counterpart to L where the i-th sample ( xi , yi ) is replaced by some new instance ( x′i , y ′ i ) ∈ X × Y ; that is , L′ = 1N [ ` p ( ΘWux ′ i , y ′ i ) + ` r ( ΘWey ′ i , y ′ i ) + ∑N n=1 ; n 6=i ( ` p ( ΘWuxn , yn ) + ` r ( ΘWeyn , yn ) ) ] . Then , let Θ∗ , Θ′∗ denote the optimal parameters corresponding to the two losses L and L ′ . Assumption 2 ( Representative Errors ) LetL′r contain the reconstruction errors of the dataset without the i-th sample : L′r = ( 1/N ) Σ N n=1 ; n 6=i ` r ( ΘWeyn , yn ) , and let L B r denote the reconstruction error of the representative subset : LBr = ( 1/M ) Σ M m=1 ` r ( ΘWebm , bm ) . Then there exists some a > 0 such that for any small κ > 0 : LBr ( Θ∗ ) − LBr ( κΘ′∗ + ( 1− κ ) Θ∗ ) + LBr ( Θ′∗ ) − LBr ( κΘ∗ + ( 1− κ ) Θ′∗ ) ≤ a [ L′r ( Θ∗ ) − L′r ( κΘ′∗ + ( 1− κ ) Θ∗ ) ] + a [ L′r ( Θ′∗ ) − L′r ( κΘ∗ + ( 1− κ ) Θ′∗ ) ] . That is , the difference in reconstruction error LBr between the two points Θ∗ , Θ ′ ∗ is upper bounded by some constant factor of the corresponding difference in reconstruction error L′r at the two points . Importantly , note that this does not require that the values of the errors L′r and L B r themselves be similar , only that their differences be similar . This assumption is identical to Assumption 6 in Le et al . ( 2018 ) , and plays an identical role : we make use of LBr —which is only dependent on M—to allow the bound to decay with N ; this is in contrast with the generic multi-task analysis of Liu et al . ( 2016 ) , which—if applied directly to TEAs ( as with FEAs ) —would give a bound that does not decay with N . Theorem 1 ( Uniform Stability ) Let ` p and ` r be σp-admissible and σr-admissible loss functions , and let ` r be c-strongly convex . Then under Assumptions 1 and 2 , the following inequality holds , | ` p ( Θ′∗Wux , y ) − ` p ( Θ∗Wux , y ) | ≤ 2 ( σ2pr 2 α + σpσrrαrβ ) aM cN Proof . Appendix A. Corollary 1 ( Generalization Bound ) Consider the same conditions as in Theorem 1 ; that is , let ` p and ` r be σp-admissible and σr-admissible losses , and let ` r be c-strongly convex . Then under Assumptions 1 and 2 , the defect is less than O ( √ ( 1/N ) log ( 1/δ ) ) with probability at least 1− δ . Proof . Follows immediately from Theorem 1 ( above ) and either of the following ( similar results hold for both ) : Theorem 1.2 ( Feldman & Vondrak , 2018 ) , and Theorem 12 ( Bousquet & Elisseeff , 2002 ) . In supervised learning , it is often easy to make an argument—on an intuitive level—for the “ regularizing ” effect of additional loss terms . In contrast , this analysis allows us to unambiguously identify and quantify the benefit of the embedding component as a regularizer ( see Remark 4 in Appendix A ) . Remark 3 In the linear label space reduction framework of Yu et al . ( 2014 ) , uniform convergence is also shown to hold via norm-based regularization . Specifically for uniform stability , a similar bound can also be achieved by adding a strongly convex term to the objective , such as Tikhonov—and ` 2—regularization ( Bousquet & Elisseeff , 2002 ; Rosasco & Poggio , 2009 ; Shalev-Shwartz et al. , 2010 ) . Here , however , the joint reconstruction task leverages a different kind of bias—precisely , the assumption that there exist compact and predictable representations of targets . Therefore the significance of this analysis is that we achieve an equivalent result independent of explicit regularization .

Summary: 1. Summary: In this paper, the authors proposed a Target-Embedding Autoendocer (TEA) model for supervised representation learning. Different from the traditional feature embedding autoencoder model, TEA tries to learn a compact latent representation that can reconstruct the target vector. Hypothetically, this model should be especially useful when the target vector has a much higher dimension than the feature vector. The authors analyzed the proved some characteristics of this framework and conducted empirical experiments on three datasets to prove its effectiveness.