Target-Embedding Autoencoders for Supervised Representation Learning
Authors: Daniel Jarrett
Mihaela van der Schaar
Venue:  (2020)
Keywords: autoencoders
supervised learning
representation learning
target-embedding
label-embedding

Abstract: Autoencoder-based learning has emerged as a staple for disciplining representations in unsupervised and semi-supervised settings. This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets---encoding the prior that variations in targets are driven by a compact set of underlying factors. As our theoretical contribution, we provide a guarantee of generalization for linear TEAs by demonstrating uniform stability, interpreting the benefit of the auxiliary reconstruction task as a form of regularization. As our empirical contribution, we extend validation of this approach beyond existing static classification applications to multivariate sequence forecasting, verifying their advantage on both linear and nonlinear recurrent architectures---thereby underscoring the further generality of this framework beyond feedforward instantiations.

Published as a conference paper at ICLR 2020
TARGET-EMBEDDING AUTOENCODERS FOR SUPERVISED REPRESENTATION LEARNING
Daniel Jarrett Department of Mathematics University of Cambridge, UK daniel. wtt.:jjaarttttttts.mommmm cam. ac. uk
Mihaela van der Schaar University of Cambridge, UK University of California, Los Angeles, USA mv4720cam. ac. mihael a@ee. ucla edu:
ABSTRACT
Autoencoder-based learning has emerged as staple for disciplining representations unsupervised and semi-supervised settings. This paper analyzes a framework for improving generalization in purely supervised setting, where the target space is high- diggsdimensi.na.. We motivate and formalize the general framework of target- embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well the prior that variations in targets are driven by
predictive of targets- a compact set of underlying factors. As our theoretical contribution, we provide a guarantee of generalization for linear TEAS by demonstrating uniform stability, in- terpreting the benefit of the auxiliary reconstruction task as a form of regularization. As empirical contribution, ing static classification applications to multivariate sequence forecasting, verifying their ad advantage both linear and nonlinear recurrent architectures tehitectures-thhrebyy under- scoring the further generality of this framework beyond feedforward instantiations.
extend validation of this approach beyond exist-
INTRODUCTION
Representation learning deals with uncovering useful underlying structures of data, and autoencoders (Hinton & Salakhutdinov 2006) have been a staple in a variety of problems. While much research focuses on its use in unsupervised or semi--upern supervisd settings with such diverse objectives as sparsity (Ranzato et al., (007), generation (Kingma & Welling 20133), and disentanglement (Chen al.) 2018), autoencoders are also useful in purely supervised settings-in particular, adding an auxiliary feattre-reoonstruutiion task to supervised classification problems has been shown to empirically im- prove generalization (Le et al. 2018); in the linear case, the theoretically quantifiable benefit matches that of simplistic norm-based based regularization (Bousquet& & Elisseeff, 2002 Rosasco & Poggio 2009).
In this paper, we consider the inverse problem setting where the target space ) is high-dimensional: for instance, consider the multi-label classification tasks of object tagging, text annotation, and image segmentation. This is in contrast to the vast majority of works designed to tackle digh-diimnsional feature space X' (where commonly | > [)] such as standard classification problems). In this setting, the usual (and universal) strategy of learning reconstruct features (Weston et al. 2012 Kingma et al.) 2014, Le et al.] 2018) may not be most useful: learning latent representations that encapsulate the variation within X does not directly address the more challenging problem of mapping back up to a higher-dimensional y. Instead, we argue for leveraging intermediate representations that are compact and more easily predictable from features, yet simultaneously guaranteed be predictive of targets. In the process, we provide unified theoretical perspective on recent applications of autoencoders to label-ember embeldin in static, high -igh--imensional classification problems (Yu et al.) 2014) Girdhar et al. 2016: Yeh et al., 2017). Extending into the temporal setting, we further empirically demonstrate the generality of target-embedding for recurrent, multi-variate sequence forecasting.
Our contributions are three-fold. (TEA) framework: a general approach applicable to any underlying architecture. Second, we provide theoretical learning guarantee in the linear case by demonstrating uniform stability; specifically, we obtain an O(1/N) bound on instability by analogizing the benefit of the auxiliary reconstruction w task to form of regularization without incurring additional bias from explicit shrinkage. Finally, we extend empirical validation of this approach beyond the domain of static classification: using the task of multivariate disease trajectory forecasting case study, experimentally validate the
First, we motivate and formalize the target-embedding autoencoder

---

Published as a conference paper at ICLR 2020
X'
Semand pereitiiennn (plstced)
embbed
Z
predict
y
X
predict
Z
Sorwand
rncotstatic
A'
(etateel)
Y
Feature- Eineeuii-i Autoencader
(b) Target Embedding Autoencoder
Figure 1: (a) Feature-embedding and (b) Target-embedding autoencoders. Solid lines correspond the (prim- ary) prediction task; dashed lines to the (auxiliary) reconstruction task. Shared components are involved in both.
advantage that TEAS confer on both linear and nonlinear architectures using real-world datasets with both continuous and discrete targets. To the best our knowledge, we are the first to formalize and quantify the theoretical benefit of autoencoder-basee supervised setting, and to extend its application to the domain of multivariate sequence forecasting.
sngee-representation learning in purely
TARGET-EMBEDDIN AUTOENCODERS
Let X' and y be finite-dimensional vector spaces, and consider the supervised learning problem of predicting targets E y from features x X. With finite batch of N training instances the objective is to learn a mapping h X y that generalizes well to new
D D= (x... Yn) samples from the same distribution. The vast majority of existing work consider the setting- most commonly, classification -eeer | > \ under this scenario, autoencoders are often used to first transform the input into some lower-dimensional representation task. Doing so involves adding an auxiliary reconstruction loss e to the primary prediction loss lp-
Z amenable to the downstream
Formally, solutions of this form in supervised and semi-supervised settings alike- alike-consist of shared forward model Φ : X + Z, a reconstruction function r: Z + X, and a prediction function d: Z + y' during training (where notation d reflects the downstream nature of the prediction task). Denote x = r()(x)) and = d(o(x)); then the complete loss function takes the following form,
L= 1 EN (ep( + ly,, x,,) N
In contrast, we focus on settings where the target space y is high- digenssiilii and where possibly | > |.Y). this case, we argue that learning to reconstruct the input is not necessarily most beneficial. In simple classification problem, autoencoding inputs leverages the hypothesis that reconstructive representation also likely discriminative. our setting, however, the more immediate problem is the high-dimensional structure of ;; in particular, there is little guarantee that intermediate representations trained to encapsulate x are easily mapped back up to nigher-dimensiona targets.
Our goal is to make use of intermediate representations that are both predictable from features as well as predictive of targets. A turget-embedding autoencoder (TEA)-versee) versus what we shall autoencoder (FEA) flips the model architecture around by learning an emhedding of target vectors instead, which a predictor then learns a mapping into. This involves an encoder y Z, an upstream predictor u X 2, and shared forward model 0 Z - y Denote ý = 0(e(y)) and ý = (x(())); the complete loss function is now of the following form,
term a
1
L= L= N IM (Cp(5n. + l, yn]]
Abstractly, the general idea of target space reduction is not new; in particular, it has been present in various solutions in the domain of multi-label classification (see Section 4 and Appendix B for discussions of related work). Here we focus on assumption that variations in (high-dimensional) target space are driven by a compact and predictable set of factors. By construction, learning to reconstruct directly in output space ensures that latent representations are predictive of targets; at the same time, jointly training with the prediction loss ensures that latent representations are predictable from features. Instead of learning representations for mapping out of (downstream), here we learn representations for mapping into (upstream); the shared forward model handles the rest. See Figure T for high-level diagrams of TEAS versus FEAS.
autoencoders; they leverage the
Training and Inference. Figure [ gives block diagrams of component functions and objectives in (a) FEAS and (b) TEAS during training (sce Algorithm 1 in Appendix Cf pseudocode). Training occurs in three stages. First, the autoencoder is trained (to learn representations): the parameters of e and learned the reconstruction loss. Second, the prediction arm is trained to regress the learned
2

---

Published as a conference paper at ICLR 2020
Feature
Feature Vector
Target
86g
X
Hl, 3
2
7
0 4
2 = u(x: Ww)
cly; W.)
86p
8e,
plp
Latent Code E 2
ae,
OW.
Latent Code
OW.
a d
Ow,
E 2
de p
d(2; Wa)
8 (2: W,
y 0i2: o)
A(Z o
7
Prediction
Reconstruction
Prediction EV
Reconstruction
E y
EX
E y
P
Feature-Embedding Autoencoder
(b) Target-Embedding Autoencoder
Figure 2: Functions and objectives (a) FEAS and (b) TEAS. Blue and red identify supervised and representa- tion learning components, FEAS are parameterized by (Ø, Wa, W,) of (ø, d, r), and TEAS by (Ø, W., W.) of (0, u, e). Solid lines indicate forward propagation of data; dashed lines indicate backpropagation of gradients.
embeddings (generated by the encoder): the parameters of u are learned (on the latent loss) while the autoencoder is frozen. Finally, all three components are jointly trained on both prediction and reconstruction losses (Equation 2): parameters of the predictor, embedding. and shared forward model are trained simultaneously. Note that during training, the forward model receives two types of latents as input: encodings of true targets, as well as encodings predicted from features. At inference time, the target-embedding arm is dropped, leaving the learned hypothesis h 0 u for prediction. Figure (Appendix C provides step-by-step block diagrams of both training and inference greater detail.
We emphasize that TEAS is the case with FEAS specify a general framework independent of the implementation details each component. For instance, the solutions to applications in Yu et al. (0144) Ych et al. (2017), and Girdhar al. (2016) can be abstractly regarded as linear and nonlinear instances of this framework, with domain-specific architectures (see Section f and Appendix B for more detailed discussions). Linear TEAS which we study in greater detail in Section involve parameterizations (O, W.. We) of (0, u, E) consisting of single hidden layers with linear activation.
3 STABILITY-BASHT LEARNING GUARANTEE
Two questions are outstanding. The first is theoretical. We are motivated by the prior that variations in target space are driven by lower-dimensional set of underlying factors. In this context, can we say something more rigorous about the benefit TEAS? this section, we take the first step showing that jointly learning target representations improves generalization performance in the supervised setting. Specifically, we demonstrate that linear TEAS are characterized by uniform stability, from which theoretical guarantees known to follow. The second question is empirical. We noted above that certain applications of label-embedding to classification can be interpreted through this framework. Does the benefit extend beyond its static, feedforward instantiations into the temporal setting for multi-variate sequence forecasting, with both continuous and discrete targets? In Section 5 we first validate our theoretical findings with linear models and sensitivities, as well as extending our empirical analysis to the realm of recurrent, nonlinear models for both regression and classification.
Consider a linear TEA, where the upstream predictor is parameterized by W. E x| target- embedding by W. E R! and shared forward model by E RD' where |Z| < (D). The complete loss function is given by L = 1 Σ 14p (OW Xx,,, vn) + e, (oW eVn. y.)] following Equation 2. Interpreting the jointly learned autoencoding component as an auxiliary task, we show that the TEA algorithm for learning the shared forward model @ is uniformly stable with respect to the domain of the supervised prediction task. To establish our notation, first recall the following:
Definition (Generalization Bound)
Given learning algorithm D + hp that returns hypothesis hvp. let R(hp) = J (x)o(x). y y)d((x, y) denote the risk, and R(hp) = 1 denote the empirical risk, where is some loss function. A generalization bound is a probabilistic bound the defect that takes the following form: R(hr)) R((p)) & with some confidence 1 8.
N EN E(hp (X,) ).))
3

---

Published as a conference paper at ICLR 2020
Definition 2 (Uniform Stability) Let D' denote a modification of batch D where the i-th training instance ys) is replaced by an independent and identically distributed example yy). A learning algorithm is said to be y-uniformly stable with respect to the loss function E if VD E
(X x y)y, Vi E [1...,n), v(x. (x, E X x ((((bp(x) (x). - -x(kp=(x), y)1 S 7. Uniform
stability holds the minimum value of converges to zero as batch size N increases without limit.
generalization bounds. In particular, Bousquet & Elisseefff (2002) first showed that the defect of 7-uniformly stable algorithm is is less
Uniform stability can be used to derive aggorithm-dependee
than ((7 + v/ N (1)6) with probability 2 1 8. Feldman & Vondrak 20118) recently log(1/))) Here, we show uniform stability demonstrated an improved bound of O(v(7 + I/N)
for linear TEAS, where y is ((1/N)
which tight generalization bound follows immediately. Before we begin, we introduce two additional tools: c-strong convexity and a r-admissibility. Note that these conditions are standard and easily satisfied for instance, by the quadratic loss function; for more context see for example Bousquet& & Elisseeff (002)) Liu et al. (2016), and Mohri et al. (2018).
Definition 3 (c-Strong Convexity) Differentiable loss function E is c-strongly convex if Yh E H. : :hh(((xx (h'(x) - fx)x,, Ve(N'(x), - Vl(h(x), y)) 2 cl]h'(x) 10--hh h(x)]lB for some c E Rt, where V((((xx, y) denotes the gradient with respect to h(x), and (:,-) denotes the dot product operation.
Definition 4 (o-Admissibilitt) Loss function E is a-admissible with respect to the underlying hy- pothesis class H if vh, h E H: [((h(()),) - y)) S (x) - 22 some a E R
To obtain uniform stability, we make two assumptions both analogous to prior work arguing from the benefit of learning shared models between tasks. [Liu et al. (2016) deals with learning multiple tasks in general, and Le et al. (2018) deals with reconstructing inputs in what we describe as FEAS. Now in multi-task learning, the separate tasks are usually chosen due to some prior relationship between them. In the case of Assumption 1 in [ii et al. (2016) and Assumption in [e et al. (2018). this is assumed to come from similarities in feature structures across tasks; hence their assumptions of cross- roooesetrrsett are made in feature space. (Note that this restricts primary and auxiliary features to be elements of the same space). Our setting is contrary: the inputs primary and auxiliary tasks come from different spaces, but are trained to produce similar labels through a compact, shared latent space; hence our assumption of cross-repreentabbiityy will be made in this latent space instead.
Assumption (Representative Vectors) There exists a representative subset of target vectors B (b.,... bu } C. {y1, (y..... YN. such that the latent representation of any individual (x,y) can be linearly reconstructed from that of the representative subset with small error, i.e. W, M
Em= am W.b, +1 and W.y = = Pm W Webm for some coefficients Omm 3 E R. where 32 I r for some Ta, T3 R+, and 1) is small error satisfying ||nl2 S E. DM 1==1
M
m= an < r2.
Remark This assumption is comparatively mild, even for e = O. Note that in Liu et al. 2016) the
features for separate tasks come from different examples in general, and the similarity of their distri- butions within X' is simply assumed. Here, each pair of inputs to the prediction and reconstruction tasks comes from the same instance, and similarity within 2 is explicitly enforced through the (joint) training objective. In addition, observe that the assumption will hold with zero error as long as the number of independent latent vectors is at least [Z]. Furthermore, unlike the scenarios in Liu et al. (2016) and Le et al. (2018) we do not require that the input domains of the two tasks be identical. Therefore for ease of exposition, we assume going forward that e = 0 (see Remark 6 in Appendix (A).
Remark 2 A comparison with Assumption 4 in Le et al. 2018) sheds additional light on why we expect TEAS to be beneficial where | > 2), in contrast with the (more typical) scenario | > | >> | Critically, the technique in Le et al. (2018) banks the fact that the prediction arm projects the latent into a lower-dimensional target space. Conversely, Assumption here relies on the fact that the encoding arm maps into the latent space from highher-dimensional target space (rendering cross-representabiliy therein reasonable). The distinction is crucial: we certainly do not expect any benefit from autoencoding trivially dimeensniional vectors! Note also that here the representative vectors are taken from )) to take them from X' instead would be unreasonable. For any compressive autoencoder, we generally expect if some subset (b)..., b..] C (y....., yN) spans ) that [W.bi.-. ..., W. W.by) } then also span 2 in order to be maximally reconstructive. The same cannot be said of subsets {CI, cM3 C {xi...., x that span x instance, take |X| < (Z].
In addition to being representative in terms of latent values, the set of representative points also needs to be representative in terms of the reconstruction error. First, let T.' denote the counterpart
4

---

Published as a conference paper at ICLR 2020
to L where the i-th sample (x, is replaced by some instance (x) E &' x ): that is, cy, y1) + E En= (l, aXn: + e (OW y.n,, Vn)]]. Then, let .. O denote the optimal parameters corresponding to the two losses L and L'.
L' = N 1lpp (OW,,
y/) + (O,,x;;
Assumption (Representative Errors) Let I contain the reconstruction errors of the dataset with- out the i-th sample: L', = (1/N) Imn (OW e.n,; Yn), and let L denote the reconstruction
error of the representative subset: L, (1/ -- L (eW, e mm bm)) Then there exists some L (0') L," Lxx (xO.+
a > 0 such that for any small k > 0: L O L, (
x) 0 (n.. + (1-))=)))
(1 - x) e)) S a [e, T,, (xO,, (1 K) ) + a (e.
That is, the difference in reconstruction error L. between the two points ②.. O is upper bounded by some constant factor of the corresponding difference in reconstruction error L' at the two points. Importantly, note that this does not require that the values of the errors L' and L themselves be similar, only that their differences be similar. This assumption is identical to Assumption 6 in [e et al. (2018), and plays an identical role: we make use of I.5 which is only dependent on M to allow the bound to decay with N; this is in contrast with the generic multi-task analysis Liu et al. (2016), which-if applied directly to TEAS (as with wous give a bound that does not decay with N.
Theorem (Uniform Stability) Let lp and ey be op op-admissible and o, aadiissiie loss functions, and let e, be e-strongly convex. Then under Assumptions I and 2, the following inequality holds,
2( 2 + +pporaaT Jar3)aM
1e wx,,) (,
cN
Proof. Appendix A
Corollary (Generalization Bound) Consider the same conditions as in Theorem 1; that is, let lp and e, be oy -y-adiissible and o--udmissible losses, and let &, be c-strongly convex. Then under Assumptions and 2, the defect is less than O (1//)) (og()/))) with probability at least 1 8.
Proof. Follows immediately from Theorem (above) and either of the following (similar results hold for both): Theorem 1.2 (Feldman & Vondrak 2018), and Theorem 12 (Bousquet & Elisseeff 2002).
In supervised learning, it is often easy to make an argument-on an intuitive level-for the "regular- izing" effect of additional loss terms. In contrast, this analysis allows us unambiguously identify and quantify the benefit of the embedding component as a regularizer (see Remark 4 in Appendix (A]
Remark 3 In the linear label space reduction framework of Yu et al. (2014), uniform convergence is also shown to hold via norm-based regularization. Specifically for uniform stability, a similar bound can also be achieved by adding a strongly convex term to the objective, such as Tikhonov and Eg regularization (Bousquet& & Elisseeff 22022, Rosasco & Poggio 2009; Shwaarz-Shwarzz et al., 2010). Here, however, the joint reconstruction task leverages a different kind of precisely,slly the assumption that there exist compact and predictable representations of targets. Therefore the signifi- cance of this analysis is that we achieve an equivalent result independent of explicit regularization.
4 RELATED WORK
Our work straddles three threads of research: (1) supervised representation learning with autoencod- ers, (2) label space reduction for multi-label classification, and (3) stability-based learning guarantees. Appendix provides a much expanded treatment, and presents summary tables for additional context.
Supervised representation learning. While great deal of research is devoted to uncovering useful underlying structures of data through autoencoders -itieerrrr--hhhhheeee various properties such as sparsity (Ranzato et al. 2007) and disentanglement (Chen et al., 2018), among many others (T'schannen et al. 2018)-the the goal of better representations often for the henefit of downstream tasks. Semi-supervised autoencoders jointly optimized on partially-lablled data can obtain compact representations that improve prediction (Weston et al.) 2012. Kingma et al. 2014: Ghifary et al., 2016). Furthermore, auxiliary reconstruction is also useful in a purely supervised setting: rather than focusing how specific architectures better structure unlabeled data, Le et al. (2018) show the simple benefit of Ceature-reconsttuctton on supervised classification a special case of what we describe FEAS.
In contrast, we focus on target-representation learning in the supervised setting, and analyze its benefit under the prior that high-dimensional targets are driven by compact and predictable set of factors.
5

---

Published as a conference paper at ICLR 2020
We take inspiration from the empirical study of Girdhar al. (2016), where latent representations of 3D objects are jointly trained to be predictable from 2D images. Their setup can be viewed as specific instance of TEAS with (nonlinear) convolutional components, with a minor variation in training: in the joint stage, predictors continue to regress the learned embeddings, and gradients only backpropagate from latent space (instead of target space). Unlike the symmetry of our losses (which we require for our analysis above), their common decoder is only shared indirectly (and predictions made indirectly). As it turns out, this does not appear matter for performance (see Section [ In comparable
Mostajabi et al. (2018), , a two-stage procedure is used for semantic Pegmentation-losse to the first two stages in TEAS; in contrast to our emphasis on joint training, they study the benefit of a frozen embedding branch in parallel with direct prediction. More broadly related to target-embedding. Dalca et al. (2018) build anatomical priors for biomedical segmentation in unsupervised settings.
Multi-labe classification. The general idea of target space dimension reduction has been explored for multi- label classification problems (commonly, annotation based on bags of features). These first derive a reduced label space, then subsequently associate inputs to it; methods include compressed sensing (Hsu et al. 2009),, principal components (Tai & Lin 2010), maximum-margin coding (Zhang & Schneider 2012), and landmarking (Balasubramanian & Lebanon 2012). Closer to our theme of joint learning. Chen & Lin (202)) first propose simultaneously minimizing encoding and prediction errors via an SVD formulation. Using generic empirical risk minimization, Yu et al. (2014) formulate the problem as a linear model with a low-rank constraint. While this captures an intuition (similar to ours) of restricted latent factors, their error bounds require norm-based regularization (unlike ours).
Recently, Ych et al. (0177) generalize the label-embedding approach to autoencoders. This flexibly accommodates custom losses to exploit correlations, as well as deep learning for nonlinearities. Our work is related to this line of research, although we operate at a higher level of abstraction, with a significant difference in focus. Their problem is multi-label classification, and their starting point is binary relevance (i.e. label by label). During reduction, they worry about specific losses that capture dependencies within and among spaces. contrast, we worry about autoencoding at all-thar is, we focus on the effect of joint reconstruction on learning the prediction model. Problems can be of any form: classification or regression, and our starting point is direct prediction (i.c. no reconstruction).
Stability and learning guarantees. Generalizability via hypothesis stability is first studied in Rogers) & Wagner (1778) and Devroye & Wagner (1979); unlike arguments based on the complexity of the search space (Vapnik & Chervonenkis 1971; 1984 Koltchinskii 2001). these account for how the algorithm depends on the data. [Bousquet& & Elisseeff (2002) first formalize the notion of uniform stability sufficient for learnability, and Fellman & Vondrak (208) use ideas related to
differential privacy (Bassily et al.] 2016) for further improvement. Separately, while there is a wealth of research dimensionality reduction and autoencoders (Singh al.) 2009: MMhrr et al. 2015
Gottlibb et al. 2016; Epstein & Meir 2019), they either operate in the semi semi-supervised setting, or focus on the benefit of feature representations (not targets) and also do not consider joint learning.
The benefit of jointly learning multiple tasks through a common operator (Caruana 1997) is explored with VC-based (Baxter 2000) and Rademacher complexity- based (Maurer) 2006 Maurer et al. 2016) analyses. Recently, Liu et al. (2016) show that the algorithm for learning the shared model in multi-task setting is uniformly stable. While our argument is based on theirs, we are not interested in generic bound for all tasks; closer to Lo et al. (2018), we focus on the primary prediction task, and leverage the auxiliary reconstruction task for stability. Similarly, we arrive at an ((1/N) on instability without an explicit regularization term as in Bousquet & Elisseeff (2002). Unlike them, however, the fundamental distinction of our setting is that y is high-dimensional (but where the underlying factors are assumed compact); in this sense our focus is the mirror opposite of theirs.
5 EXPERIMENTS AND DISCUSSION
So far, we have formalized general target-autoencoding framework for supervised learning, and quantified the benefit via uniform stability. Our overall goal in this section is to explore this benefit in simple controlled setting, such that we can identify and isolate its utility on the prediction task, and investigate any sensitivities of interest. By way of preface, we emphasize two observations from above: (1) In the static, multi-llbeel classification setting, the gain from label-embedding has been studied, including the autvencoder approach of Yeh et al. (0177) 2017)-which which can be viewed as an instantiation of TEAS with sophisticated refinements. (2) The benefit of turget-uutonccoding is also

---

Published as a conference paper at ICLR 2020
Table I: Dataset statistics and input/output dimensions used in experiments
Num.
Samp.
Target type
Static dim.
Temp. dim. (history)
Temp, dim. Window Window
Effective Effective
Dataset
patients
freq.
(forecast) (history) (forecast)
|X|
y
UKCF 10,000
yr.
Binary
11
43
34
3
4
140
136
ADNI
1,700
6 Continuous
11
26
24
4
115
192
MIMIC
22,000
4 hr.
Mixed
26
361
361
5
5
1,831
1,805
The effective input dimension [x') is computed as the dimension of static data plus the product of the width of the historical window (of temporal information) with its dimension; the effective target dimension [J] is sim- ilarly computed as the product of the width the forecast window (of temporal information) with its dimension.
demonstrated using nonlinear, convolutional architectures in Girdhar et al. (2016) which is also an instantiation of TEAs, also noting significant gains. Therefore a natural question of interest is:
Does the utility of target-embedding extend to (nonlinear) recurrent models with sequ- ential data for general, high-dimensional targets (i.e. regression and/or classification)?
Disease Trajectories. In this section, we take the first step in answering this question-as our empirical contribution, we extend validation of target-embeddingg autoencoders the domain of mullivariate sequence forecasting, exploring its utility on linear and nonlinear sequence-to-sequence architectures. What makes a good testbed? In particular, the progression of diseases (and their markers) is dighndimensiona in presentation; at the same time. their evolution is often driven by latent biological dynamics (Szczesniak et al. 20177) Pascooll et al. 2017 Alaa & van der Schaar 2019).
|
With the increasing importance of early diagnosis and timely intervention in healthcare, the ability to forecast individual disease trajectories ()) in the presence of limited windows of information (X)
has become increasingly desirable (Donohue et al. 2014 Pham et al. 2017 Bhagwat et al. 2018
Datasets. We use three datasets in our experiments. The first consists of cohort of patients enrolled in the UK Cystic Fibrosis registry (UKCF), which records follow-up trajectories for over 10,000 patients. We are interested in forecasting future trajectories for the 11 possible infections and 23 possible comorbidities (all binary variables) recorded at each follow-up. using past trajectories and basic demographics as input. The second consists of patients in the Alzheimer's 's Disease Neuroimaging Initiative stuudy (ADNI), which tracks disease progression for over 1,700 patients. We are interested in forecasting the evolution of the 8 primary biomarkers and 16 cognitive tests (all continuous variables) measured at each visit, using past measurements and basic demographics as input. The third consists of a cohort of patients in intensive care units from the Medical Information Mart for Intensive Care (MIMIC), which records physiological data streams for over 22.000 patients. Likewise, we are interested in forecasting future trajectories for the 361 most frequently measured variables such as vital signs and lab tests (both binary and continuous variables). again using past measurements and basic demographics as input. See Appendix D for more information on datasets.
Experimental Setup. In each instance, the prediction input a precedent window of up to width Wx. and the prediction target is the succedent window of width Wy. For UKCF (Wa, w.y) = (3, 4) at 1-year resolution, for ADNI (4, 8) at 6-month resolution, and for MIMIC (5, 5) at 4-hour (resampled) resolution. All models are implemented in Tensorflow. Linear models consist of a single hidden layer with no nonlincarity; for the nonlinear case, we implement an RNN model for each component using GRUs See Appendix D for additional detail model implementation and configuration. For evaluation, we measure the mean squared error (MSE) for continuous targets (averaged across variables), and the area under the precision-recall curve (PRC) and area under the receiver operating characteristic curve (ROC) for binary targets (averaged across variables). We use cross-validation on the training set for hyperparameter tuning, selecting the setting that gives the lowest validation loss averaged across folds. For each model and dataset, we report the average and standard error of each performance metric across 10 different experiment runs, each with a different random train-ter test split.
Note that forecasting high- -igh-dimensional disease trajectories is challenging. and input information is deliberately limited (as often the case in medical practice); the desired targets are similar or higher- dimensional than the inputs (see Table [T. This obviously results in an inherently difficult prediction problem- but which makes for a good candidate setting to test the utility of represtrpprssnntation learning. RNN autoencoders have previously been proposed for learning representations of inputs (i.e. FEAS instantiated with RNNs) to improve classification (Dai & Le 2013). prediction (Lyu et al.. 2018), generation (Srivastava et al. 2015),, and clustering (Baytas et al. 2017); similarly, their mission is not in excessively optimizing specific architectural novelties to match state-of-the aoffthe-art art, but rather in exploring the benefit of the autoencoding framework. Here, we learn representations of targets.
7

---

Published as a conference paper at ICLR 2020
Table 2: Summary results for TEA and comparators on linear model with UKCF (Bold indicates best)
Base
REG
FEA
TEA
F/TEA
PRC(1)
0.322 + 0.099*
0.347 + 0.085*
0.351 = 0.079*
0.450 + 0.035
0.414 + 0.028*
PRC(C)
0.416 + 1.100*
0.433 + 0.083*
0.455 = 0.087*
0.559 + 0.060
0.520 +: 0.052
Roc(1)
0.689 0.089*
0.710
0.072*
0.720 = 0.073
0.767 0.026
0.766 0.023
Roc(c)
0.679 0.091
0.700 + 0.075
0.713 = 0.075
0.767
0.042
0.755 0.037
The two-sample t-test for difference in means is conducted on the results. An asterisk indicates statistically significant difference in means (p-value < (0.05) relative to the TEA result. PRC and Roc metrics are reported separately for variables representing infections (I) and comorbidities (C). Sec Tables 10 for extended results.
Table 3: Summary results for TEA and comparators on nonlinear (RNN) model (Bold indicates best)
UKCF (Binary Targets)
ADNI (Continuous Targets)
MIMIC (Mixed Targets)
PRC(1)
PRC(C)
MsF(B)
MsF(C)
PRC
MsF
Base REG
0.411 #0.035*
0.497
0.361 0.105
0.153:+0.011 0.142:+0.028**
0.005 0.410 0.483 0.457
0.518
0.096:1..014
0.060
0.143
0.051
FEA
0.004
0.092:1.012**
0.056110.08
0.0022 0.034
TEA
0.583+00.02
0.003
0.330-:0.066
0.150
F/TBA
0.576110.071
0.073110.010
0.338
0.166
1.0001
The two-sample t-test for difference in means is conducted on the results. An asterisk indicates statistically significant difference in means (p-value < 0.05) relative to the TEA result. For UKCF, only PRC metrics for infections (I) and comorbidities (C) are shown due to space limitation; for ADNI, MSE metrics are reported
separately for targets representing biomarkers (B) and cognitive tests (C). See Tables 13 14
18 and 21
5.1
MAIN RESULTS
Overall Benefit. First, we examine the overall utility of TEAS. To verify the linear case first, Table summarizes the performance of TEA and alternate setups on UKCF The temporal axis is flattened to simulate ordinary static multi-label classification, and the base case is direct prediction (Base) that is, absent any auxiliary representation learning regularization. Next, we allow for &g Ez-reuularization over direct prediction (REG), as well as over all other methods. FEAS differ only by the added feature- reconstruciinn, and TEAS only by the tanget-reconstrucctin;; as additional sensitivity, we also implement a combined approach (F/TEA). More generally, we also wish to examine the benefit of TEA for the nonlinear case: Table 3 summarizes analogous results where component functions are implemented with GRU networks; results are shown for all datasets. Ceteris paribus, we observe that sugeetrepressnttttio learning has notable positive impact on performance. Interestingly, learning representations of inputs does not yield significant benefit, and the hybrid approach (F/TEA) is worse than TEA; this suggests that forcing intermediate representation to encode both features and targets may be overly constraining. (Note that for the linear model, the instances are restricted to those for which the full input window is available; as consequence, the results for linear and nonlinear cases are not directly comparable). Figures 4 (Appendix ( and (Appendix D) give training diagrams for all comparators. Additional experiment results (by model, timestep, metric) are in Appendix E 1-2.
Source of Gain. There are two (related) interpretations of TEAS. First, we studied the regularization view in Section 3 this concerns the benefit of joint training using both prediction and reconstruction losses. Ceteris paribus, we expect performance to improve purely by dint of the jointly trained TEa objective. Second, the reduction view says that TEAS decompose the (difficult) prediction problem into two (smaller) tasks: the autoencoder learns a compact representation of y, and the predictor learns to map x to z. This makes the potential benefit of staged training (Section and Appendix C intuitively clear, and suggests an alternative that of simply training the autoencoder and predictor arms in two stages-à la Mostajabi et al.) (2018). As a general framework, TEAS is a combination of both ideas: all three components are jointly trained in a third stage- à la Girdhar et al. (2016). We now account for the improvement in performance due to these two sources of benefit; Table 4 does so for the linear case (on UKCF), and Table for the more general nonlinear case (on all datasets). The "No Joint" setting isolates the benefit from staged training only. This is analogous to basic unsupervised pretraining (though using targets), and corresponds to omitting the final joint training stage in Algorithm 1. The "No Staged" setting isolates the benefit from joint training only (without pretraining the autoencoder or predictor), and corresponds to omitting the first two training stages in Algorithm 1. The "Neither" setting is equivalent to vanilla prediction (REg) without leveraging either of the advantages. We observe that while both sources of benefit are individually important, neither setting performs quite as well when both are combined. See Appendix H 1-2 for extended results.
8

---

Published as a conference paper at ICLR 2020
Table 4: Summary source of gain and TEA variants on linear model with UKCF (Bold indicates best)
Neither
No Joint
No Staged
Tea
TEA(L)
TEA(LP)
PRC(1)
0.347 + 0.085
0.402 +: 0.026
0,431 = 0.031
0.450 + +00335
0.435 + 0.031
0.454 + 0.036
PRC(C)
0.433 + 0.083
0.507 0.040
0.543 = 0.054
0.559 + 0.060
0.544 + 0.053
0.560 + 0.061
Roc(1)
0.710
0.072
0.747 1 0.022
0.764 = 0.022
0.767 0.026
0.759 100025
0.768
0.028
Roc(c)
0.700 0.075
0.744 + 0.038
0.766 = 0.038
0.767 + 0.042
0.760 0.042
0.767 + 0.042
"No Joint" omits final joint training, and "No Staged" skips (pre)-training stages. PRC and Roc metrics are repo-
rted separately for targets representing infections (I) & comorbidities (C). See Tables 1 for extended results.
Table 5: Summary source of gain and TEA variants on nonlinear (RNN) model (Bold indicates best)
UKCF (Binary Targets)
ADNI (Continuous Targets)
MIMIC (Mixed Targets)
PRC(1)
PRC(C)
MsF(n)
MSF(C)
PRC
MSE
Neither
0.415+00.30
0.518+0.055
0.096=0.014
0.360+00066
0.152 0.109
+0.000
No Joint
0.435
0.574
0.092=00014
0.353
0.183=00038
+00011
No Staged
0,424:10.031
0.043
0.106=0.022
0.167 0.363
0.050
TEA
0.045
0.083
0.063=0.00
0.330=:0.066
0.239
0.150
TEA(L)
0.047
0.581 +0.074
0.058
0.330+00.076
0.249
0.149+0..00
TEA(LP)
0.583 0.480
0.012
0.229 1.0039
0.151 10.011
"No Joint" omits final joint training, and "No Staged" skips (pre)-training stages. For UKCF, only PRC metrics for infections (I) and comorbidities (C) are shown due to space limitation; for ADNI, MSE metrics are reported
separately for targets representing biomarkers (B) and cognitive tests (C). See Tables 16 20 and 23
Variations. Having established the utility of target-embedding, we can ask whether variations on the same theme perform similarly. In particular, the embeddings in the empirical studies of Girdhar al. (2016) and Yeh et al. (2017) are jointly learned via the reconstruction loss f,- and latent loss Lg- that is. the prediction arm continues regress learned embeddings during the joint training stage (Figure in Appendix D). The principle is similar, although (as noted in Section 4 the primary task is therefore learned indirectly; this is in contrast to the vanilla TEA setup, where the primary task is learned directly via the prediction loss Lp. Tables and also compare the performance of well as hybrid variant (TEA(LP)) for which both latent and prediction losses are trained jointly with the reconstruction loss (Figure Je). Perhaps as expected, observe that performance across all three variants more or less identical, affirming the general benefit of target-representttion learning. Again, see Appendix E
vanilla TEAS with this indirect variant (TEA(L)).
1-2 for extended results.
5.2 SENSITIVITIES
Regularization. Of course, rengetrrpressnttioonn learning is not replacement for other regulariza- tion strategies; it is an additional tool that can be used parallel where appropriate. Figure B)) shows the performance of TEA and REG with various coefficients on fz-regularization. By itself, introducing & f-reeulariizttion does improve performance up to a certain point, beyond which the additional shrinkage bias incurred begins to be counterproductive; this is not surprising. Interestingly, introducing sargctrrpresenntatio learning appears to leverage an orthogonal bias: it consistently improves prediction performance regardless of level of shrinkage. This is a practical result of the the- oretical observation in Remark 3: while prior works obtain stability through explicit Ez-regularivation, the benefit from larget-embedding relies on different bias entirely, which allows us to combine them. While increasing the strength of either form of regularization reduces variability in results (see also below), excessive bias of either alone degrades performance. See Appendix E 3 for full results.
Strength of Prior. Target-embedding attempts to leverage the assumption that there exist compact and predictable representations of targets. Realistically (e.g. due to measurement noise), of course, this will not hold perfectly. In our experiments, we set the ratio of prediction and reconstruction losses to be 1 for TEA (as well as FEA and F/TFA): that is, the "strength-of-prron coefficient on E, is 0.5. In order to isolate the effect of À during joint training, we observe the performance of TEAS with joint training only (i.e. removing the confounding effect of staged training). For large values of X, we expect the reconstruction task to dominate in priority, which is (under imperfect prior) not beneficial for the ultimate prediction task-in in general, a hidden representation that is most reconstructive is not necessarily also what is most predictable). For small values of X, the setup begins to resemble direct prediction. Figure ((bb) verifies our intuition. Note that in the extreme case of X = 1, predictions no better than random (see Roc~ 0.5). See Appendix F 3 for full results.
9

---

Published as a conference paper at ICLR 2020
0.
08
84
lagal wnoaadig
0.5
0.5
0.4
04
04
PRC
PRC
0.3
03
033
0.2
Lb predesct
0.1
411
189
201
0.
03
0.8
0.7
R0000
0.5
predeces
044
36-5
30c 3e-3
201
0.5
0.8
0.20
27 20% 50%
10%
(a) Sensitivities on
(b) Sensitivities
(c) Sensitivities N
Figure 3: Sensitivities on éz-regularization coefficient v. strength-of-prior coefficient X. and training size N for direct prediction (REG) and with target-embedding (TEA) on linear model with UKCF. For sensitivities on we perform joint training only, so that we isolate the effect of the joint reconstruction task (i.c. removing the confounding effect of staged training). Standard errors are indicated with shaded regions. For full results, see for sensitivites on N.
Tables 25 28
for sensitivities v, Tables 29 30 for sensitivities on X. and Tables 31 34
Sample Complexity. Figure (e shows the performance of TEA and REG under various levels of data scarcity. The benefit conferred by TEAS is significant especially when the amount of training data N is limited. Importantly, note that we are operating strictly within the context of supervised learning: unlike in semi-supe settings, here we not just restricting access to paired data; we are restricting access to data per se. (Though beyond the scope of this paper, we expect that extending TEAS to semi-supervised learning with additional unpaired data would yield larger gains). Here, without the luxury of learning from unpaired data, we highlight the comparative advantage purely from the addition of farget-representation learning. Again, see Appendix F 3 for full results.
5.3 DISCUSSION
By way of conclusion, we emphasize the importance of our central assumption: that there exist compact and predictable representations of the (high-dimensional targets. This is critical: target- embedding not useful where this is not true. Now obviously, learning representations of targets unnecessary if the output dimension is trivially small (e.g. if the target is single classification label), or if the problem itself trivially easy (e-g. if direct prediction is already perfect). Also obvious is the situation where representations cannot possibly be compact (e.g. if all output dimensions are independent of each other), in which case any model with a compressive (bottleneck) representation as intermediate target may make little sense to begin with. Perhaps less obvious is that we cannot assume that the goals of prediction and reconstruction are always aligned. Just as in learning feature- embeddings (for downstream classification), what is most reconstructive may not necessarily encode what is most discriminative; so too in learning target-embeddinns (for upstream prediction), what most reconstructive may not necessarily encode what is most predictable. In the case of disease trajectories, it is medical knowledge that permits this assumption with some confidence. Appendix 4 gives an extreme (synthetic) counterexample where this prior is outright false-i.e. prediction and reconstruction are directly at odds. While certainly contrived, it serves as caveat about assumptions.
Using the deliberately challenging setting of disease trajectory forecasting with limited information, we have illustrated the nontrivial utility of ruresstepressnttatio learning in a controlled setting with baseline models. While we appreciate that component models the wild may be more tailored, this setting better allows us to identify and isolate the potential utility of angee-auuoonccoinn per se. In addition to verifying our intuitions for the linear case, we have extended empirical validation of aarget-attencooiing to (nonlinear) eequence-to-sqqueme we explored the sources of gain from joint and staged training, as well as various sensitivities of interest. Where the prior holds, target-embedding autoencoders are potentially applicable any high-ul digh-dimensional prediction task beyond static classification and imaging applications, and exploring its utility for other specific nomain-architecture may be practical direction for future research.
recurrent architectures; along the way,
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This work was supported by Alzheimer's Research UK (ARUK), the US Office of Naval Research (ONR), and the National Science Foundation (NSF): grant numbers ECCSI462245, ECCS1533983, and ECCS! 407712. We thank the UK Cystic Fibrosis Trust, the Alzheimer's s Disease Neuroimaging Initiative, and the MIT Lab for Computational Physiology respectively for making the UKCF, ADNI, and MIMIC datasets available for research. We thank the reviewers for their helpful comments.
REFERENCES
Ahmed M. Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease progression. In 2019 Conference on Neural Information Processing Systems, 2019.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. International Conference on Machine Learning. 2018.
Krishnakumar Balasubramanian and Guy Lebanon. The landmark selection method for multiple output prediction. Proceedings of the 29th International Conference on Machine Learning. 2012.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. peeetrallynnormalizee margin bounds for neural networks. In Advances in Neural Information Processing Systems, pp. 6240-6249, 2017.
Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp. (046-1059, ACM, 2016.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. Patient subtyping via time-aware lstm networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 74.74.2 ACM, 2017.
Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Learning Theory and Kernel Machines, pp. 567-580. Springer, 2003.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE trans. on pattern analysis and machine intelligence, 35(8):1798-182 1828, 2013.
Nikhil Bhagwat, Joseph D Viviano, Aristotle N Voineskos, M Mallar Chakravarty, Alzheimer's Disease Neuroimaging Initiative, et Modeling and prediction of clinical symptom trajectories in alzheimer's 's disease using longitudinal data. PLoS computational biology, 14(9):e1006376 2018.
Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateck Jain. Sparse local embeddings for extreme multi-label classification. In Advances in Neural Information Processing Systems, pp. 730-738. 2015.
Kush Bhatia, Kunal Dahiya, Himanshu Jain, Yashoteja Prabhu, and Manik Varma. The ex- treme classification repository, 2019. URL XMLRepository. htttttoo...mmm.l.l..
Wei Bi and James Kwok. Efficient multi-label classification with many labels. In International Conference on Machine Learning, pp. 405-413, 2013.
Konstantinos Bousmalis, George Trigeorgis. Nathan Silberman, et al. Domain separation networks. In Advances neural information processing systems, pp. 343-351, 2016.
Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of machine learning research, 2(Mar):499-526, 2002.
Matthew Boutell, Jiebo Luo, Xipeng Shen. and Christopher M Brown. Learning multi-label scene classification. Patfern recognition, 3707):157-1177 2004.
11

---

Published as a conference paper at ICLR 2020
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
Rui M. Castro and Robert D. Nowak. General Bounds for Bounded Losses, in Statistical Learning Theory. 2018.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Yao-Nan Chen and Hsuan-Tien Lin. Feature-aware label space dimension reduction for multi-label classification. In Advances in Neural Information Processing Systems, pp. 1529-1537, 2012.
Moustapha M Cisse, Nicolas Usunier, Thierry Artieres, and Patrick Gallinari. Robust bloom filters for large multilabel classification tasks. In Advances in Neural Information Processing Systems, pp. 1851-1859, 2013.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pp. 3079-3087, 2015.
Adrian V Dalca, John Guttag, and Mert R Sabuncu. Anatomical priors in convolutional networks for unsupervised biomedical segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9290-9299, 2018.
Luc Devroye and Terry Wagner. Distribution-fres inequalities for the deleted and holdout error estimates. IEFE Transactions on Information Theory, 25(2):202-207, 1979.
Michael Donohue, Hélène Jacqmin-Gadee Gadda, Mélanie Le Goff, et al. Estimating long-term multivari- ate progression from short-term data. Alzheimer's 'ss & Dementia, 1065):8400-$400, 2014.
Baruch Epstein and Ron Meir. Generalization bounds for unsupervised and semi-supervised learning with autoencoders. arXiv preprint WxXiv:1902.01449 2019.
Vitaly Feldman and Jan Vondrak. Generalization bounds for uniformly stable algorithms. In Advances in Neural Information Processing Systems, pp. 9747-9757. 2018.
Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Meneía, and Klaus Brinker. Multilabel classifi- cation via calibrated label ranking. Machine learning, 133-53333-153, 2008.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstuction-caasiffation networks for unsupervised domain adaptation. In European Confer- ence on Computer Vision, pp. 597-613, Springer, 2016.
Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In European Conference on Computer Vision, pp. 484-499. Springer, 2016.
Lee-Ad Gottlieb, Aryeh Kontorovich, and Robert Krautbgamer. Adaptive metric dimensionality reduction. Theoretical Computer Science, 620:105-118, 2016.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(578)::0045000 2006.
Daniel J Hsu, Sham M Kakade, John Langford, and Tong Zhang. Multi-label prediction via com- pressed sensing. In Advances neural information processing systems, pp. 772-780, 2009.
Ashish Kapoor, Raajay Viswanathan, and Prateek Jain. Multilabel classification using bayesian compressed sensing. In Advances in Neural Information Processing Systems, pp. 2645-2653, 2012.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
12

---

Published as a conference paper at ICLR 2020
Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. JEEE Transactions on Information Theory, 47(5):1902-1999 1914, 2001.
Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. In Advances in Neural Information Processing Systems, pp. 107-117, 2018.
Xin Li and Yuhong Guo. Multi-label classification with feature-aware lineli label space transfor- mation. In Twenty-Fourd Fourth International Joint Conference on Artificial Intelligence, 2015.
Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang. Multi-label classification via feature- aware implicit label space encoding, In International conference on machine learning, 2014.
Tongliang Liu, Dacheng Tao, Mingli Song, and Stephen J Maybank. Algoribmn-dependet gen-
eralization bounds for multi-task learning, IEEE transactions on pattern analysis and machine intelligence, 39(2):227-244 2016.
Gábor Lugosi and Miroslaw Pawlak. On the posterior-probabiiiee estimate of the error rate of nonparametric classification rules. IEEE Trans. on Information Theory, 40(2):475 4814 1994.
Xinrui Lyu, Matthias Hüser, Stephanie Hyland, George Zerveas, and Gunnar Rätsch. Improving clinical predictions through unsupervised time series representation learning. NeurlPS 2018 Machine Learning for Health (ML4H) workshop, 2018
Andreas Maurer. Bounds for linear multi-task learning. Journal of Machine Learning Research, 17 20060
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredeee The benefit of multitask representation learning. The Journal of Machine Learning Research, 17(1):2853-288) 2016.
Colin McDiarmid. On the method of bounded differences. Surveys in combinatorics, 141(1):148-188, 1989.
Mehryar Mohri, Afshin Rostamizadeh, and Dmitry Storcheus. Generalization bounds for supervised dimensionality reduction. Feature Extraction: Modern Questions and Challenges, pp. 226-241, 2015.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Mohammadreza Mostajabi, Michael Maire, and Gregory Shakhnarovich. Regularizing deep networks by modeling and predicting label structure. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5629-5638, 2018.
Sayan Mukherjee, Partha Niyogi, Tomaso Poggio, and Ryan Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25(1-3):161-199 2006.
Siddharth Narayanaswamy, Brooks Paige, Mil-Willem Van Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, pp- 5925-5935. 2017.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. International Conference on Learning Representa- tions 2018, 2017.
Ozan Oktay, Enzo Ferrante, Konstantinos Kamnitsas, Mattias Heinrich, Wenjia Bai, Jose Caballero, Stuart A Cook, Antonio De Marvao, Timothy Dawes, Declan P O'Regan, et al. Anatomically constrained neural networks (acnns): application to cardiac image enhancement and segmentation. IEEE transactions medical imaging, 37(2):384-395, 2017.
13

---

Published as a conference paper at ICLR 2020
Tharick A Pascoal, Sulantha Mathotaarachchi, Monica Shin, Andrea L Benedet, Sara Mohades, Seqian Wang, Tom Beaudry, Min Su Kang, Jean-Paul Soucy, Aurelie Labbe, et al. Synergistic interaction between amyloid and tau predicts the progression to dementia. Alzheimer's 's & Dementia, (3(6):44465533 2017.
Trang Pham, Trayen Tran, Dinh Phung, and Svetha Venkatesh. Predicting healthcare trajectories from medical records: A deep learning approach. Journal of biomedical informatics, 69:218-222 2017.
David Pollard. Convergence of stochastic processes. Springer Science & Business Media, 1984.
Piyush Rai, Changwei Hu, Ricardo Henao, and Lawrence Carin. Large-scale bayesian multi-label learning via topic-based label embeddings. In Advances in Neural Information Processing Systems, pp. 3222-3230, 2015.
Mare' Aurelio Ranzato and Martin Szummer. cumi-supervised learning of compact document representations with deep networks. In Proceedings of the 25th international conference on Machine learning, pp. 792-799, ACM, 2008.
Mare' Aurelio Ranzato, Christopher Poultney, Sumit Chopra, Yann I. Cun, et al. Efficient learning of sparse representations with energy-based model. In Advances in neural information processing systems, pp. 1137-1144, 2007.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances neural information processing systems, pp. 3546- 3554, 2015.
Philippe Rigollet.
Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research, 1369))::3391132 2007.
William H Rogers and Terry J Wagner. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statisties, pp. 506,514, 1978.
Lorenzo Rosasco and Tomaso Poggio. Stability of tikhonov regularization. MIT 9.520 Staristical Machine Learning, 2009.
M. Seeger. Learning with labeled and unlabeled data. Technical report, Institute for ANC, Edinburgh, 2000.
Shai Shalev- Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11(Oct):663--267) 2010.
Aarti Singh, Robert Nowak, and Jerry Zhu. Unlabeled data: Now it helps, now doesn't. In Advances in neural information processing systems, pp. 1513-1520, 2009.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852, 2015.
Rhonda D Szczesniak, Dan Li, Weiji Su, Cole Brokamp, John Pestian, Michael Seid, and John P Clancy. Phenotypes of rapid cystic fibrosis lung disease progression during adolescence and young adulthood. American journal of respiratory and critical care medicine, 196(4):471-47) 2017.
Farbound Tai and Hsuan-Tiee Lin. Multilabel classification with principal label space transformation. International Conference on Machine Learning workshop on Learning from Multi-Label Data 2010.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. NewrIPS 2018 Bayesian Deep Learning workshop, 2018.
Grigorios Tsoumakas, Ioannis Katakis, and loannis Vlahavas. Mining multi-label data. In Data mining and knowledge discovery handbook, pp. 667-685. Springer, 2009.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
14

---

Published as a conference paper at ICLR 2020
Vladimir Vapnik and Alexey Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theoretical Probabibility and its Applicactions, 77:264-280, 01 1971. doi: 10.1 10.1137/1116025
Kaixiang Wang, Ming Yang, Wanqi Yang, and YiLong Yin. Deep correlation structure preserved label space embedding for multi-label classification. In Asian Conference on Machine Learning, pp. 1-16, 2018.
Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639-655, Springer, 2012.
Baoyuan Wu, Zhilei Liu, Shangfei Wang. Bao-Gang Hu, and Qiang Ji. Multi-label learning with missing labels. In 22nd International Conference on Pattern Recognition, pp. 1964-1968 IEEE, 2014.
Yan Yan, Glenn Fung, Jennifer G Dy, and Romer Rosales. Medical coding classification by leveraging inter-code relationships. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 193-202. ACM, 2010.
Chih-Kusa Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Hsiang-FF Fu Yu, Prateek Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In International conference on machine learning, pp. 593-601, 2014.
Yi Zhang and Jeff Schneider. Multi-label output codes using canonical correlation analysis. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 873-882, 2011.
Yi Zhang and Jeff Schneider. Maximum margin output coding. Proceedings of the 29th International Conference on Machine Learning, 2012.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep genera- tive models. In Proceedings of the 34th International Conference Machine Learning-Voit Volume 70, pp. 4091-4099. JMLR. org, 2017.
Wen-Ji Zhou, Yang Yu, and Min-Ling Zhang. Binary linear compression for multi-label classification. In ISCAI, pp. 3546-3555 3552,, 2017.
Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. Supervised representa- tion learning: Transfer learning with deep autoencoders. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
15

---

Published as a conference paper at ICLR 2020
A PROOF OF THEOREM
Definition (Bregman Distance) Let E be some differentiable, strictly convex loss. For any h h' H. the Bregman distance associated with is given by De(M' = E(h') {(h) (h' h, Ve(h)). Additivity and non-negativity are easy to see; that is, De(h'((h) Z 0 for any h h' E H. and if L = & + lg and both & and Eg are convex functions, then De [|h) Dr, (h'llh) + Dr, (h)))).
Theorem (Uniform Stability) Let lp and e, be op admissibee and admiiiiiiee loss functions, and let e, be c-strongly convex. Then under Assumptions and 2, the following inequality holds,
2(02 2 + tpD
(O w.,,,,,y) ep O.
cN
Proof. The overall strategy consists of three steps in sequence. First, we want to bound the delta in prediction loss under O and O using the of representative vectors B. Second, we bound the resulting expression in terms of the Bregman divergence of the complete loss functions L and L' under O₂ and O'. Third, we express the divergence back terms of the original expression itself (consisting of representative vectors), which allows us to solve for bound on that expression. Finally, combining the results from the three steps completes the proof. We begin with the left-hand term,
(0, W. w. ( W W. y)) S opli(O', - O.)
Gp
Σ am (Gm O.)
S Spro
Σ (( O.) W. bm
where the first incquality follows from the fact that ép is -p-adiissible, the equality follows from Assumption 1 for some coefficients Om R, and the second inequality follows from the Cauchy- Schwarz inequality. As our second step, the goal is to upper-bound the term under the square-root,
M
Σ ((ee' O.) W.bnl 2
((0', W. Ve,(O..Wbb.. bm) Vl.(O. (.(.e.WW.. W.b bm))
M
Σ (0', O., (e, bm. bm bb W Ve,(O, W W.bm. bm b WT)
M [Drs (r. Dy. e. [
where the first inequality follows from the fact that e, is c-strongly convex, and the final equality follows from the definition of Bregman divergence (the standalone loss terms cancel). We want an expression in terms of the loss functions and L', which will subsequently allow obtain a bound expressed back in terms of the set of representative vectors. Focusing on the term in the brackets,
(r.(('.((.) Dr.
= O. (..((e))).. (0. -', VL.((.))
= lim
[..) + [1- ((-
< lim
((-- -1 - (1-x)++1)))
k-0-
[- (e' O. VI ee e, VL,
= a Dr, ( O.) +D, ( [.)
< s 0.) + DT ( 00))
(5)
16

---

Published as a conference paper at ICLR 2020
where the first and last equalities follows from the definition of Bregman divergence, and the second equality from the definition of directional derivatives. The first inequality follows from Assumption 2; for the second inequality, note that L', consists of a strict subset of the set of strictly convex losses that I. consists of, and similarly for T." Therefore by additivity and non-negativity of the Bregman distance, we have that Pzg [..) S Dz ( @.) and Dr: (+. ( s Dr. (o, Our third and final step is to go back and bound this term again using the set of representative vectors,
Dr + (2.
L(O)) (( L( -
[L(O') ['())) (L())) [0.)
1 W.
- [6.0,.....,))+(((00...
N
N
1
1
N (6, w. (0
N
T (II(6', 9.) w. 2 [0 ).)
N
or
(II(e' (...Weel Weyill |0 .. W.y N
2(ppTa + (rra)
M
Σ II - 0.) ).) W.bmll
(6)
losses L and L' are zero at the respective optimal models O, and O, The first inequality follows from the fact that lp and E, are op op-admissible and admissiis respectively, and the second inequality follows from Assumption and the Sauwhy-Schwarz inequality. Now, combining Equations 4, 5, and allows to write
where for first equality note that by construction the gradients
- (.).-..)..b.lll 2( orrs)aM cN
which-by substitution into Equation complettes the proof.
Remark 4 Formulating the autoencoding component as auxiliary task allows to unambiguously interpret its benefit as a regularizer. Specifically, the complete loss can be summarized and rewritten
L()) - L. + R, (+) + RO):;;; that is, the TEA objective is a combination of the primary prediction loss (e) - n Σm= Ep (eW DWW... Xn, yn. plus additional regularization, where Rd (9)
N E bm, bm.) N
M (O) and (2) 3 Ew e, (OW...... y.,
L O
In particular, the proof of Theorem relies on R3 (e) to upper-bound instability (Appendix (A.. This precisely identifies the regularizer in question, while Theorem quantifies its generalization benefit.
Remark 5 Technicality: Moving from uniform stability (Theorem I) to generalization bounds (Corollary 1) requires that the loss function not take on arbitrarily large values (see e.g. Bousquet & Elisseeff (2002). In practice. the label space itself is often bounded (see e.g. Assumption 1 in Le et al. (2018);; then the problem effectively reduces to that with bounded loss function. For example, consider a regression setting using the quadratic loss function, where the target data lie within [-U, U|: then the loss function bounded to be within [0, 4 See Castro & Nowak (0118).
Earlier we assumed = 0 for ease of exposition; carrying around the extra (es) term (from Equations and 6) is not particularly illuminating. Generalizing to E of 0 can be done in similar manner to Le et al. (2018), with the additional assumptions of bounded spaces and that & decreases as 1/N (which they note is reasonable, since the more samples in the data, the more likely the cross-representatiiiyy assumption will hold with low error). Again, note that long as the number of independent latent vectors is at least Zl). Similarly. Liu et al. (2016) consider 0, noting in any that they can increase N to obtain small || their analysis for detail.
Remark 6 Technicality:
holds as
17

---

Published as a conference paper at ICLR 2020
B EXPANDED RELATED WORK
In this paper, we motivate and analyze general autoencoder-haset target-representaiinn learning technique in the supervised setting, quantifying the generalization benefit via an argument from uniform stability, as well as verifying its practical utility, As such, our work lies at the intersection of three threads of research: (I) supervised representation learning using autoencoders, (2) label space reduction for multi-laber classification, as well as (3) algorithmic stability-basee learning guarantees.
B.1 Supervised Representation Learning using Autoencoders
Table 6: Autoencoder-Basse Supervised Representation Learning
Work
Contribution
Setting
Type
Emhedding
Jointly optimized classific- ation and embedding Variational inference generative modeling Disentangled latent representations
Semi-supervised
Weston 2012
Deterministic
Features
learning
Semi-supervised
Kingma al.
2014
Probabilistic
Features
learning
Narayanaswamy al.
Semi-supervised
Probabilistic
Features
(2117)
learning
Input- and output-encoding for transfer learning
Semi-supervised
Zhuang et al.
2015
Deterministic
Features
learning
Paired autoencoders for transfer learning
Semi-supervised
Bousmalis al.
2016
Deterministic
Features
learning
Jointly optimized
Semi-supervised learning
Ghifary al. 2016
Deterministic
Features
feature-embeddinng
Jointly optimized
Supervised
Lc ct al. 2018
Deterministic
Features
feature-embedding
learning
Generative model using learned target priors Jointly optimized target- embedding (Indirect)
Unsupervised, Unpaired
Dalca et al.
2018
Probabilistic
Targets
Supervised learning
Girdhar
2016
Deterministic
Targets
Jointly optimized target- embedding (Direct)
Supervised Icarning
(Ours)
Deterministic
Targets
Autoencoder- based representation learning (Hinton & Salakhutdinov 2006 has long played impor-
tant roles in unsupervised and semi-supervised settings. Various inductive biases have been proposed
to promote representations that are sparse (Ranzato al. 2007), discrete (van den Oord et al. 2017).
2017). among others. For more thorough overview of various methods, we refer the reader to Bengio et al. (013) and Tschannen et al. 22018).
factorized (Chen al.) 2018)), hierarchical (Zhao et al.
The goal of better representations is often for the benefit of downstream tasks. Semi-supervised autoencoders-ttrann
on partially-labeled data-cam be jointly optimized to obtain compact rep- resentations that improve generalization supervised tasks (Ranzato & Szummer 2008 Weston et al. 2012). This naturally extends to representations that are generative (Kingma et al. 2014).
disentangled Narayaaaswamy al., 2017). or hierarchical (Rasmus et al.) 2015). In addition, semi-supervised autoencoders enable transfer learning across different domains via cointly-trained reconstruction-elasiiiie
networks Ghifary al. (2016), private-shared partitioned representations (Bousmalis et al, 2016). or by augmenting models with label-encoding layers (Zhuang et al. 2015).
Although less studied, more closely related our work is the use of autoencoders in purely supervised setting: rather than focusing on how specific architectural novelties may better structure unlabeled data, Le et al. (2018) instead study the generalization benefit by the simple addition of reconstruction the supervised classification task-a special case of what we describe as FEASS. Now, all aforementioned studies operate on the basis of autoencoding features (for an explicit implicit downstream prediction task). In this paper, instead focus on autoencoder-based purge--rpresennatton learning (using TEAS) in the supervised setting, and and immortantly--nallle the theoretical and empirical benefits of the approach. Unlike in simple classification (for which FEAS make sense), we are motivated by problems with nigh-dimensional output spaces, but where operate under the assumption of more compact and predictable set of underlying factors.
We take inspiration from the empirical investigation of Girdhar et al. (016)), where latent repre- sentations of 3D objects (targets) are jointly trained to be predictable from 2D images (features);
18

---

Published as a conference paper at ICLR 2020
Table 7: Label Space Dimension Reduction via Label Embedding
Work
Contribution
Learning
Problem
Coding with compressed sensing
Multi-label
Hsu et al.
2009
Separate
(random projections)
classification
Coding with principal label space transformation (PC-based projections) Coding with maximum margin (between prediction distances)
Multi-label
Tai & Lin 2010)
Separate
classification
Zhang & Schneider
Multi-labo
Separate
(2011)
classification
Balasubramanian &
Landmark selection of labels via group-spars learning
Multi-label
Separate
Lebanon (0012)
classification
Landmark selection labels via randomized sampling
Multi-label
Bi & Kwok
2013
Separate
classification
Feature-aware principal label space
Multi-label
Chen & Lin
2012
Joint
transformation for embedding
classification
Generic empirical risk minimization formulation and bounds Autoencoder embedding with canonical correlation analysis Autoencoder component as (Implicit) regularization for learning predictor Autoencoder component (Explicit) regularization for learning predictor
Multi-label
Yu ct al.
2014
Joint
classification
Multi-label
Ych et al. (017
Joint
classification
General; Semantic
Mostajabi al.
2018
Separate
segmentation
General; Sequence
(Ours)
Joint
forecasting
Oktay et al. (017)) deploy similar approach for medical image segmentation. On the other hand, both cases the supervised task truncated: the predictors are trained to regress the unsupervised embeddings (instead of ground-truth targets), and gradients only backpropagate from the latent space (instead of the target space). This means that their common decoder function is only shared indirectly (and predictions made indirectly), versus the symmetric and simultancously optimized forward model proposed for TEAS an important distinction that our analysis relies on obtain uniform stability. In Mostajabi et al. (2088) , a two-stage procedure is used for semantic segmentation loosely comparable to the first two stages in TEAS; in contrast to our emphasis on joint training, they study the benefit of a frozen embedding branch in parallel with direct prediction. More broadly related to target-embedding. Dalca et al. (2018) build anatomical priors for biomedical segmentation in unsupervised settings.
B.2 Label Space Reduction for Multi-Label Classification
Label space dimension reduction comprises techniques that focus specifically on multi-label clas- sification. Early approaches to multi-label classification employ simplistic transformations such as label power-sets (Boutell et al. 2004), binary relevance (Tsoumakas et al. 2009), and label rankings (Fürnkranz et al., 2008); these are computationallyy inefficient, and do not capture interdependencies between labels. In contrast, label-embedding methods first derive latent label space with reduced dimensionality, and subsequently associate inputs to that latent space instead. Encodings have been projections (Tai & Lin 2010), canonical correlation analysis (Zhang & Schneider 2011). as well as maximum-margin coding (Zhang & Schneider 2012). A parallel thread of research has focused on selecting representa- tive and reconstructive subsets of labels through group-sparse learning (Balasubramanian & Lebanon 2012) and randomized sampling (Bi & Kwok 2013). Various extensions of label-embedding tech-
obtained via random projections (Hsu al. 2009)) principal components-basee
niques abound, such as using bloom filters (Cisse et al. 2013), nearest- neirest-neighhors (Bhatia et al. 2015). handling missing data (Wu et al. 2014). as well as using binary compression (Zhou et al. 2017).
Closer our theme of joint learning by utilizing both features and targets, Chen & Lin (2012) first proposed simultaneously minimizing the encoding error (from labels) and prediction error (from features) through an SVD formulation. Towards more moreflexible learning, [in et al. (2014) did away with explicitly specified encoding functions, proposing to learn code matrices directly-making no assumptions whatsoever. Unifying several prior methods, Yu et al. (2014) cast label-embedding within the generic empirical risk minimization framework as learning a linear model with a low-rank constraint: this perspective captures the generic intuition of restricted number of latent factors, and admits generalization bounds based on norm bandd regularization. Recently, Ych et al. (2017) generalized the label-embedding approach autoencoders; this formulation flexibly allows the
19

---

Published as a conference paper at ICLR 2020
Table 8: Generalizability, Multiple Tasks, and Algorithmic Stability
Work
Contribution
Setting
Facus
Learning
Bousquet & Elisseeff
Uniform stability generalization
Supervised, general
Single task
(2002)
Feldman & Vondrak
Improve on Bousquet & Elisseeff's bound VC-dimension for generalization
Supervised, general
Single task
2018))
Multi-tan task learning
Baxter
2000
All tasks
Jointly
Rademacher complexity for generalization
Multi-task
Maurer
2006
All tasks
Jointly
learning
Uniform stability generalization
Multi-task
Liu et al.
2016
All tasks
Jointly
learning
Rademacher complexity for generalization
Feature-
Mohri ct al.
2015
Primary task
Separately
embedding
Non-contractiveness and suniviuperviiion Uniform stability for generalization Uniform stability generalization
Feature-
Epstein & Meir 2019
Primary task
Separately
embedding
Feature-
Le al.
2018
Primary task
Jointly
embedding
Target-
(Ours)
Primary task
Jointly
embedding
addition of specific losses exploit correlations-a tactic also used in Wang et al. (2018) with multi-dimensional scaling. Furthermore, nonlinearities can be handled by deep learning in component functions, unlike earlier approaches limited kernel methods (Lin al. 2014 Li & Guo 2015).
Our work is related to this general autoencoder approach to lahel-embedding, although there are significant differences in focus. In particular, we operate higher level of abstraction. Label- embedding techniques worry about label reduction, and about specific loss functions that aim to preserve dependencies within and among spaces; their problem is one of multi-label classification, and their baseline is binary relevance. In contrast, we worry about autoencoding at all -tat is, we focus on the regularizing effect of the reconstruction loss on learning the prediction model; our baseline direct prediction, and the output can be of any form (classification regression). In light of the sizable performance improvement of the autoencoder based model of Ych et al. (2017) over comparators using direct prediction, work can be regarded as more generalized analysis of the contribution of the autoencoding component. Moreover, unlike the uniform convergence-basee analysis in Yu al. (2014), our bound does not rely explicit norm-based regularization-instiee the embedding task itself as an intrinsic form regularization to derive our stability- based guarantee.
interpret
Finally, also worth mentioning is the field of extreme multi- classification (Bhatia al. 2015), for which probabilistic methods such as Rai et al. (2015) and Kapoor et al. (2012) present sophisticated approaches to extremely high-dimensional classification
wibhe advantages in performance
and use cases. In light of the medical relevance of our experimental setting, we point out the applica- tion of Yan et al. (2010) to medical coding. See Bhatia et al. (2019) for more detailed overview.
B.3 Algorithmic Stability and Learning Guarantees
Generalizability is central machine learning, and its analysis via hypothesis stability is first studied in Rogers & Wagner (1978) and Devroye & Wagner (1979). Unlike arguments based on the complexity of the search space (Vapnik & Chervonenkis 197 1971]Pollar 1984; Koltchinskij (001), stability- based approaches account for how the model produced by the algorithm depends on the data. Based on concentration inequalities (McDiarmid 1989), improved bounds are developed in Lugosi Pawlak (1994) by estimating posterior error probabilities. The landmark work of Bousquet & Elisseefff (2002) first formalizes the notion of uniform stability sufficient for learnability, obtaining relatively strong bounds for several regularization algorithms, Feldman & Vondrak (2018) recently use ideas related to differential privacy (Bassily et al.) (2016) for further improved bounds without additional assumptions. For further context, see Mukherjee et al. (0066) and Shalev-Shhaaae et al. (0110).
For semi-supervised representation learning, Rigollet (2007) first introduces the notion cluster excess-risk and convergence, formalizing the clustering criterion for unlabeled features be useful (Seeger) 2000). Based on the clustering assumption, [Singh et al.] (2009) develops a finite sample analysis quantify the performance improvement from unlabeled features. Focusing on autoencoders.
20

---

Published as a conference paper at ICLR 2020
Epstein & Meir (2019) adapt recent margin, norm, and compression- results for deep networks (Bartlett et al., 2017 Neyshabur et al. 2017 Arora et al. (018). and relate generalization of feature
reconstructions to the benefit of additional unlabeled features for the primary classification task.
In the context of supervised problems, Mohri et al. (2013) and Gottlieb et al. (2016) analyze the generalization properties of dimensionality reduction techniques for features with respect to downstream task; however, rather than joint training, the primary task is optimized subsequently over the learned representations. Taking a joint, multi-task approach (Caruana 1997), Baxter (2000) first leverages the inductive bias of a common optimal hypothesis class to obtain a VC-based generalization bound. Maurer (2006) and Maurer et al. (016)) argue from Rademacher complexity to illustrate the benefit of the common operator; however, they only consider the task misk-averaged benefit, whereas we want to focus specifically on the primary task. There been some work on generalization for each task (Ben-David & Schuller, 2003), but limited to binary classification-ccnnee contrary to our setting.
Arguing from stability, our approach related to Liu et al. (2016) in showing that the algorithm for learning the shared model in multi-task setting is uniformly stable. Our analysis also resembles Le ctt al. (2018) in the more specific setting where the bound for the primary prediction task obtained with assistance from the auxiliary reconstruction loss; unlike Liu et al. (2016), we are not interested in a generic bound for all tasks. Again, however, the fundamental (and motivating) difference of work stems from the (inverted) problem setting and resulting framework. In the vast majority of works, the primary task one of classification (or more generally X > [)/), where feature- embeddings make sense to learn. Instead, attend the setting in which y is high-dimensional
(but where the underlying factors are assumed to be compact). In this setting, we argue (theoretically and empirically) that target-embeddings make more sense to learn in an auxiliary reconstruction task.
C EXPANDED AL.GORITHM DETAIL.
In the following, Algorithm gives pseudocode for TEA training. Figure 4 gives detailed block diagrams component functions and objectives corresponding to each training stage (and variant).
Algorithm Pseudocode for TEA Training
Input: D = (xx,, }
J== learning rate e,, minibatch size N₂ Parameters θ W.. W. of components 0, u,
Output:
1: Initialize: θ, W., W.
2: while not converged do
b Stage 1: Learn Target-Embedding
3: Sample ((xn; yn) ) N, Lid.
D
n=1
4: for n (1,..., N₂ }
do 5: Zn t e(yn: We)
> Encode
6: Yn x 1ln +)
Decode
7:
end for
8:
Ir
N. DN,, eo (5n,
9: W W.
15 Vw, L,p
10:
θ + VeL
11: end while
12: while not converged do
Stage 2: Regress Embeddings
yn) 13: Sample {x)) }
lid. D
14: for E (I, N₂} do
15: 2n - u(xn; W..)
Predict
16:
2, - e(yn; W.)
D Encode
17:
end for
18: Lg f- N. (
(2,, kn) 19: W. f- W V.
20:
end while
21: while not converged do
P Stage 3: Joint Training
22: Sample (x)) iid. D
21

---

Published as a conference paper at ICLR 2020
23:
for {1, ..., N₂) do
24:
on u(X;; W.r
Predict
25:
Zn e('ni We)
> Encode
26:
Yn (2ni )
Decode
27:
Yn < OZni )))
Decode
28:
end for
Lp f- 1 E pup Xn)
29:
N.
30:
L, f- 1 D.,,
N. m====1 (8 Vn)
31:
W. < W,
" - -VW.. Tup
32:
W. t W. VV Lop
33:
④ Ve Tp + Tarl
34: end while
35: return θ W.: We
Feature Vector E E
Target Vector E E
Feature Vector
Vector
2'
E E
Oe
W.)
JW
Latent Code
OW
xxx w
ely; W.)
Z
Latent Code E
Oe,
y 8(2; θ)
0
Prediction 6 cy
Reconstruction
Prediction y
Reconstruction
E }
E
(n) Training (Stage 1)
(b) Training (Stage 2)
Target
Feature Vector
Target Vector
X
E
X'
Of,
de,
f
u(x: W.)
cly: W.)
ZW.
JW.
06
0
aW
Latent Code
aw
2 u(x; W..)
Z el; W.)
Z
Latent Code
E
0Ep
8(2; +
8(2: o)
0(z
d
Prediction
Reconstruction
Prediction
Reconstruction
O2,
E
E
E
oW
(c) Training (Stage 3), TEA
(d) Training (Stage 3), TEX(L)
Figure 4: TEAS consist of shared forward model upstream predictor and target-embedding function e, parameterized by (O, W.. We). Blue and red respectively identify the supervised and representation learning components each arrangement. Solid lines indicate forward propagation of data; dashed lines indicate backpropagation of gradients. (a) First, the autoencoding components e,
are trained to learn target representations. (b) Next, using the inputs, the prediction arm u trained to regress the learned embeddings generated by the encoder. trained on both prediction and reconstruction losses. (d) In the indirect variant (Girdhar et al. 2016).
(c) Finally, all three components are jointly
22

---

Published as a conference paper at ICLR 2020
Feature Vector
Target Vector
Feature Vector
Target Vector E>
E
0e
0e,
UW
OW
2 u(x; W..)
xx; W..
ey: W.)
Latent Code Z
Latent Cude
Z
Dt 2
(2:
y A( +
De,
Prediction
Reconstruction
Dt,
Prediction
Reconstruction
ow.
E y
y
OW.
EV
&p
Training (Stage 3). TEA(LP)
(1) Inference Time
Figure 4: (continued) the predictor continues regress the learned embeddings, and the latent loss backpropagates through both u and e (TEA(L)). (e) The TEA(LP) variant combines the previous two: both the latent loss and prediction loss are trained jointly together with the reconstruction loss. (f) At inference time, the target-embedding arm is dropped, leaving the hypothesis h A u for prediction.
D ADDITIONAL EXPERIMENT DETAIL
The UK Cystic Fibrosis registry records follow-up trajectories for over 10,000 patients over the period from 2008 and 2015 with total of over 60,000 hospital visits. Fach patient is associated with 90 variables over time, which includes data on treatments and diagnoses for 23 possible comorbidities (e.g. ABPA, diabetes, hypertension, pancreatitis), cepacia, klebsiella pneumoniac), as well as static demographic information (c.g. gender, genetics, smoking status). Using both static and temporal information in precedent window, we forecast the future trajectories for the diagnoses of infections and comorbidities (all binary variables) recorded at each follow-up. The Alzzeimer's Disease Neuroimaging Initiative study tracks disease progression for over 1, 1,700 patients over the period from 2004 to 2016 with total of over 10,000 (bi-annual)
possible infections (e.g. aspergillus, burkholderia
Feature Vector E
Feature Vector X
Oep
Ofp,
8e,
aw.
0Φ
axx W.)
2
x(
Latent Code Z
0C
Latent Code 2
Ve,
a
aw
aw,
OW.
y d(2; W
d(; Wa)
X = r(a; W,)
Prediction
Prediction
Reconstruction
E y
E )
E X
6p
(a) Base; REG
(b) FEA
Figure 5: Component functions and training objectives for comparators in experiments. Blue and red respectively identify the supervised and representation lcarning components in each arrangement. Solid lines indicate forward propagation of data; dashed lines indicate backpropagation of gradients. (a) The baseline is direct prediction (with (REG) and without (Base) fzrreullarization, which simply corresponds to removing the autoencoder; here we explicitly identify some intermediate hidden layer to preserve visual correspondence with the autoencoder models, but note that the "latent code" is strictly speaking a misnomer-an nothing being encoded here. (b) FEAS consist a shared forward
23

---

Published as a conference paper at ICLR 2020
Feuture Vector E X
Target Vector EY
Feature Vector E Et
Target Vector E EY
p
04
x(x; W.)
2 ely; We)
$
cly; We) dlry UW.
dlp
He,
aw.
Latent Code
OW
06p 8
Latent Code
Olix Ow,
arry ae
E 2
ai
((2
)
y fl2 O)
y
0(2: 22
θ
r W.
0 2 2)
Prediction
Reconstruction
Prediction
Reconstruction
Reconstruction
E
E
y
6 X
Ep
Exx
(c) TEA
(d) E/TEA
Figure 5: (continued) model ø, downstream predictor d, and feature reconstructor r, parameterized by ( Wa, W.). (c) TEAS consist of shared forward model 0, upstream predictor and target- embedding function e, parameterized by (O, W.., W.). (d) As an additional sensitivity, F/TEAS combine the previous two, forcing intermediate representations to encode both features and targets.
clinical visits. We focus on the 8 primary quantitative biomarkers (e.g. entorhinal cortex, fusiform gyrus, hippocampus), cognitive tests (e.g. ADAS11, CDR sum boxes, mini mental state exam), as well as static demographic information (e.g. apolipoprotein E4, education level, ethnicity); we omit the remaining variables, for which the rate of missingness is over 50%. Using a precedent window, we forecast the future the evolution of the primary quantitative biomarkers and cognitive test results (all continuous variables) measured at each visit. The Medical Information Mart for Intensive Care records physiological data streams patients admitted intensive units after 2008. We use over 22,000 patients with a total of over 500,000 measurements (resampled at 4 hour intervals). We focus on the most frequently measured vital signs and lab tests (e.g. heart rate, oxygen saturation, respiratory rate) recorded over time (with categorical variables binarized, this gives a total of 361 variables), as well as static demographic information (e.g. admission type, gender, location, marital status); we omit the remaining variables, for which the rate of missingness is over 50%. Using a precedent window, we forecast the subsequent window of those variables. For each dataset, sequences are randomized at the patient level in order to obtain splits for training (and validation) and testing.
We implement all models using Tensorflow. For the linear model, each component (encoder, decoder, and predictor) consists of single layer, no bias term, and linear activation; static (demographic) and temporal data are concatenated and flattened for both features and targets. For the nonlinear case, we implement each component as an RNN using GRUs with the number of hidden layers < E [1,2] where the number of hidden units is equal to the temporal feature dimension, and tanh is used for activation; the dimension of the latent space is (therefore) equal to the hidden state dimension. (For even larger hidden capacities, the increased number of parameters rapidly degrades performance). Static (demographic) information is incorporated as a mapping into the initial state for recurrent cells. Training is performed using the ADAM optimizer with a learning rate of v E (3e-5, 3e 3 3,- 3e-2). Models are trained until convergence up maximum of 10,000 iterations with a minibatch size of N, E {32, 64, ]: the empirical loss is computed on the validation set every 50 iterations of training, and convergence determined on the basis of that error. Checkpointing is implemented every 50 iterations, and the best model parameters are restored (upon convergence) for use on the testing set. For all models except "Base", we allow the opportunity to select among the fz-regularization coefficients E {0, 5. 5e 3e.3,-4,5 3e-3,,3.0 3e 2)- We set the strength-of -frirror coefficient provide sensitivities on X for TEA in our experiments). For hyperparameter tuning (S, 1; v, N.), we use cross- cooss-valdattion on the training set using 20 iterations random search, selecting the setting that gives the lowest validation loss averaged across folds. For fair comparison (so as to isolate the effect of supervised representation learning over and above direct prediction), we apply the same setting chosen for REG for FRA, F/TRA, and all variants of TEA; therefore the only difference is the presence
0.5 for FEA, F/TEA, as well as all variants of TEA (however, we do
24

---

Published as a conference paper at ICLR 2020
absence of each autoencoding component. For each model and dataset, the experiment is repeated for a total of 10 times (each with a different random split of data into training and held-out testing sets); all results reported as means and standard errors of each performance metric across runs.
E
ADDITIONAL EXPERIMENT RESULTS
E.1 Results for Linear Models
Table Extended results for TEA and comparators on linear model with UKCF (Bold indicates best)
Base
REG
FEA
TEA
F/TEA
PRC(1)
0.340 0.118
0.370 0.101
0.374 1 0.091
0.497 1 0.016
0.454 1 0.010
0.325 + 0.099*
0.350 + 0.085*
0.353 0.077
0.459 1 0.017
0.419 1 0.012
0.314 +: 0
0.337 +: 0.076*
0.071 0.342
0.432 1 0.015
0.399 0.013
4
0.307 = 0.082*
0.328 0.071
0.333 0.067
0.413 + 0.010
0.386
+ 0.009*
PRC(C)
0.445 1 =
0.467 0
0.499 10
0.653 1 0.009
0.599 0.019
0.418 1 0.099
0.435 0
0.457 0.083*
0.566 1 0.010
0.524 0.017
0.403 = 0.082
0.418 0.067*
0.436 0.0099
0.521 0.008
0,487 0.015*
0.398 - 0.073
0.412 + 0.060*
0.426 0.061
0.498 + 0.009
0.471 0.013*
ROC(I)
0.713 = 0.104
0.737 0
0.750 0.078
0.806 + 0.007
0.801 0.007
0.688 = 0.089*
0.709 0
0.718 0.068
0.771 + 0.007
0.765 0.009
0,677 = 0.080*
0.697 = 0
0.706 0.066
0.750 = 0.007
0.749 0.009
0.677 0
0.696 + 0.063
0.707 0.068
0.740 = 0.008
0.748 + 0.006
Roc(C)
0.713 = 0
0.741 + 0
0.756 + 0.084*
0.829 + 0.006
0.810 + 0.008
0.686 0.091
0.707 0.072*
0.719 0.071
0.775 + 0.008
0.762 1 0.008
0.668 0.077*
0.686 0.061
0.699 0.063
0.743 + 0.005
0.735 + 0.007
0.651 0.070*
0.667 + 0.056*
0.679 + 0.057*
0.720 + 0.006
0.712 + 0.009
PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C). The two-sample r-test for difference in is conducted on the results. asterisk result used to indicate a statistically significant difference in means (p-value 0.05) relative to the TEA result.
the comparator
Table 10: Summary results for TEA and comparators on lincar model with UKCF (Bold indicates hest)
Base
REG
FEA
TeA
F/TEA
PRC(1)
0.322 0.099*
0.347 0.085*
0.351 = 0.079*
0.450
0.035
0.414 *
PRC(C)
0.416 1010**
0.433 + 0.083*
0.455
0.087*
0.559 + 0.060
0.520 +: 0.052
Roc(1)
0.689 + 0.089*
0.710 + 0.072*
0.720 = 0.073
0.767 + 0.026
0.766 + 0.023
Roc(C)
0.679 + 0.091 0.700 + 0.075*
0.713 = 0.075
0.767 + 0.042
0.755 + 0.037
PRC and Roc evaluations reported separately for targets representing infections (T) and comorbidities (C). two-sample r-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate statistically sigmificant difference in means (p-value < 0.05) relative to the TEA result. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
Table 11: Extended source of gain and variants on linear model with UKCF (Bold indicates best)
No Joint
No Staged
Tea
TEA(L)
TEA(LP)
PRC(1)
0.434 + 0.017
0.472 0.016
0.497 + 0.016
0.474 + 0.018
0.502 + 0.015
0.408 + 00016
0.439 0.016
0.459 0.017
0.443 +.0018
0.463 + 0.016
0.390 0.015
0.415 00015
0.432 0.015
0.420
0.435 + 0.013
0.377 0.014
0.399 10011
0.413
0.402 100000
0.415 1 0.009
PRC(C)
0.563 0.025
0.620 0.030
0.653 0.009
0..012
0.655 0.008
0.511 0.018
0.550 00002
0.566 0.010
0.550
0.566 + 0.010
0.484 0.014
0,512 10001
0.521 0.008
0.511 0.000
0.521 1 0.008
4
0.469 1 0.014
0.491 10015
0.498 1 0.009
0.490 1 0.009
0.499 1 0.009
Roc(1)
0.779 1 0.008
0.799 0.007
0.806 0.007
0.797 1 0.007
0.809 1 0.005
0.750 + 0.008
0.765 0.00
0.771 4: 0.007
0.763 + 0.008
0.772 1 0.007
25

---

Published as a conference paper at ICLR 2020
0.733 0.008
0.750
0.008 0.744 0.005
0.750 1 0.007
0.744 1 1.007
0.750 1 0.008
0.725 0.007
0.740 0.008
0.734 + 0.006
0.740 + 0.007
Roc(c)
0.799 0.012
0.821 0.011
0.829 1 0.006
0.823 1 0.000
0.830 + 0.005
0.751 0.012
0.774 0.009
0.775 0.008
0.769 - 0.006
0.775 = 0.006
0.723 0.010
0.746 0.007
0.743 + 0.005
0.737 + 0.005
0.742 + 0.005
4
0.702 0.011
0.722 = 0.007
0.720 + 0.006
0.713 = 0.008
0.720 + 0.006
PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C). The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining. though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).
Table 12: Summary source of gain and variants on linear model with UKCF (Bold indicates best)
No Joint
No Staged
TeA
TEA(1)
TEA(LP)
PRC(1)
0.402 + 0.026
0,431 + 0.031
0.450 = 0.035
0.435 0.031
0.454 0.036
PRC(C)
0.507 0.040
0.543 0.054
0.559 = 0.060
0.544 1 0.053
0.560 0.061
Roc(1)
0.747 + 0.022
0.764 +: 0.022
0.767
0.026
0.759 + 0.025
0.768 +: 0.028
Roc(c)
0.744 = 0.038
0.766 0.038
0.767
0.042
0.760 0.042
0.767 = 0.042
PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbiditics (C).
The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining. though using largets); the "No Staged" setting isolates the benefit from joint training only (without pretraining). Results grouped over the temporal axis; note that the variance between splits is artifact of this grouping.
E.2 Results for Recurrent Models
Table 13: Extended results for TEA and comparators on RNN model with UKCF (Bold indicates best)
Base
REG
FEA
TEA
F/TEA
PRC(1)
0,451 0
0.456 + 0
0.448 1 0.026
0.549 1 0.014
0.509 1 0.014
0.417 0
0.420 0.013*
0.416 1 0
0.490 1 0.012
0.463 0.011
0.395 0.019
0.400 0
0.395 0.020*
0.457 = 0.010
0.437 0.013*
0.380 0.017
0.385 0.010
0,380 0.017*
0.434 = 0.007
0.417 000013
PRC(C)
0.561 + 0
0.592 +: 0.029*
0.598
0.029*
0.695 + 0.010
0.685 + 0.015
0.504 = 0
0.523 = 0.022*
0.527 0.021
0.591 + 0.014
0.584 0.018
0.471 = *
0.488 + 0
0,489 0.017*
0.537 = 0.007
0.530 0.530 0.017
0.453 = *
0.469 0.017*
0.470 0.016*
0.510 + 0.007
0.015
ROc(1)
0.788 + 0
0.791 0.009*
0.794 + 0.014
0.827 1 0.007
0.818 1 0.006*
0.753 0
0.757 0
0.758 0.017
0.783 1 0.008
0.778 0.009
0.736 0.013
0.741 1
0.740 +
0.760 0.007
0.757 1.010
0.725 0.012
0.731 +0.011
0.727 00014
0.748 + 0.008
0.744 0.010
Roc(c)
0.794 1 0.022
0.809 0.015
0.808
0.012
0.838 0.007
0.834 1 0.007
0.750 0.017
0.761 0.010
0.761 *
0.782 + 0.007
0.781
+ 0.008
0.723 = 0.013*
0.733 + 0.007*
0.735 0.010*
0.752 - 0.006
0.751 + 0.009
0.699 = 0.009*
0.709 + 0
0.711 0.010*
0.726 + 0.006
0.724 + 0.008
PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbiditics (C).
The two-sample r-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value 0.05) relative to the TEA result.
Table 14: Summary results for TEA and comparators on RNN model with UKCF (Bold indicates best)
Base
REG
FEA
TeA
F/TEA
PRC(1)
0.411 + 0.035*
0.415 + 0.030*
0,410 = 0.033*
0.483 + 0.045
0.457 + 0.037
PRC(C)
0.497 1 0.057*
0.518 1 0.052*
0.521 _ 0.054*
0.583 1 0.072
0.576 1
Roc(1)
0.750 0.028*
0.755 0.025
0.755 - 0.029
0.779
0.031
0.774 0.003
Roc(C)
0.742 0.038
0.753 0.039
0.754 = 0.037
0.774 + 0.042
0.772 0.004
26

---

Published as a conference paper at ICLR 2020
PRC and RoC evaluations reported separately for targets representing infections (I) and comorbidities (C). The two-sample Hees for difference in means is conducted on the results. asterisk next to the comparator result used to indicate statistically significant difference means (p-value < 0.05) relative to the TEA result. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
Table 15: Extended source of gain and variants on RNN model with UKCF (Bold indicates best)
No Joint
No Staged
TEA
TEA(L)
TEA(LP)
PRC(I)
0,511 0.014
0.468 0.015
0.549 100114
0.553 1 0.009
0.545 1 0.011
0.461 0.013
0,429 00011
0.490 0.012
0.492 + 0.013
0.487 0.012
0.434 0.016
0.407 00011
0.457 0.010
0.457 = 0.013
#0014
0.414 0.009
0.392 0.011
0.434 0.007
0.432 + 0.008
PRC(C)
0.682 0.011
0.633 #0.32
0.695 0.010
0.697 0.010
0.695 0.012
0.581 0.012
0.549 0..25
0.591 #0014
0.589 0.013
0.592 0.011
0.530 + 0.010
0.506 0.018
0.537 0.007
0.534 + 0.009
0.538 0.009
0.504 + 0.009
0.484 00015
0.510 + 0.007
0.505 + 0.008
0.509 0.010
Roc(1)
0.816 = 0.004
0.795 =0.008
0.827 + 0.007
0.825 + 0.005
0.822 +.010
0.774 + 0.010
0.759 0.007
0.783 + 0.008
+ 1782+00000
0.778 + 0.007
0.749 + 0.010
0.744 00008
0.760 + 0.007
0.758 0.006
0.758 0.006
0.732 + 0.008
0.735 0.008
0.748 0.008
0.739 0.007
0.745 0.004
Roc(c)
0.830 = 0.008
0.816 00011
0.838 + 0.007
0.839 + 0.008
0.839 + 0.007
0.775 0.010
0.767 0.009
0.782 0.007
0.784 + 0.010
0.782 + 0.005
0.743 = 0.009
0.735 = 0.007
0.752 + 0.006
0.751 - 0.010
0.751 - 0.007
0.721 0.006
0.712 0.007
0.726 0.006
0.724 1 0.007
0.726 + 0.006
PRC and Roc evaluations reported separately for targets representing infections (I) and comorbidities (C). "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining. though using targets); the "No Staged" selting isolates the benefit from joint training only (without pretraining).
Table 16: Summary source gain and variants on RNN model with UKCF (Bold indicates best)
No Joint
No Staged
TEA
TEA(L)
TEA(LP)
PRC(1)
0.455 0.039
0.424 + 0.031
0.483
0.045
0.483 +: 0.047
0.480 0.044
PRC(C)
0.574 0.069
0.543 0.061
0.583
0.072
0.581 0.074
0.583 0.072
Roc(1)
0.768 1 0.033
0.758 1 0.024
0.779
0.031
0.776 1 0.033
0.776 1 0.030
Roc(c)
0.767 0.042
0.758 1: 0.040
0.774
0.042
0.774 1. 0.044
0.774 1: 0.043
PRC Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C).
The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining. though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining). Results grouped the temporal axis; note that the variance between splits is artifact of this grouping.
Table 17: Extended results for TEA and comparators on RNN model with ADNI (Bold indicates best)
Base
REG
FEA
TEA
F/TEA
MsF(B)
1
0.095
0
0.088 0.010
0.082 + 0.007*
0.057 + 0.007
0.065 + 0.008
2
0.097 0
0.089 0
0.084 1 0.008*
0.057 1 0.007
0.066 1 0.007
0.100 0
0.092 0.010
0.087 0.008
0.059 1 0.007
0.068 1 0.007
4
0.104
1 0
0.095 =
0.091 0
0.061 1 0.007
0.071 11 0
5
0.105
*
0.097 0
0.093 *
0.062 = 0.008
0.073 = 0
6
0.109 0
0.097 *
0.065 = 0.008
0.076 0.076+00.099
0.112 *
0.103 =
0.101 *
0.068 + 0.009
0.080 0.009*
0.115 0
0.106 0
0.105 0.013*
0.072 + 0.011
0.083 0.010
MSE(C)
0.275 0
0.270
0.265 #
0.239 + 0.015
0.243 0.013
0.300 0
0.295 0
0.290 +
0.265 + 0.014
0.273 0.014
3
0..23 + *
0.320 +
0.314
0.287 + 0.014
0.297 0.015
4
0.358 +
0.354 +0.018
0.352 +0.01
0.322 + 0.015
0.333 +0.01
5
0.371 0
0.370
0.367 0
0.341 - 0.019
10.021
0.393 0.033
0.393 0.032
0.391 10.344
0.366 1 0.026
0.374 10022
7
0.417 0.043
0.419 10.000
0.417 10.044
0.394 1 0.035
0.399 10003
27

---

Published as a conference paper at ICLR 2020
8 0.453 1 0.058 0.455 1 0.054 0.454 1 0.062 0.430 1 0.048 0.435 1 0.057
MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C). The two-sample r-test for a difference in mcans is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (n-valuc 0.05) relative to the TEA result.
Table I8: Summary results for TEA and comparators on RNN model with ADNI (Bold indicates best)
Base
REG
FeA
TEA
F/TEA
MSE(B)
| 0.105 + 0.018
0.096 0.014
0.092 + 0.012
0.063 + 0.010
0.073 0.010
MsF(C)
0.361 + 0.064 0.360 + 0.066
0.356 + 0.068
0.330 + 0.066
0.338 0.067
MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C). The two-sample 1-tsst for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate statistically significant difference in means (p-value < 0.05) relative to the TEA result. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
Table 19: Extended source of gain and variants on RNN model with ADNI (Bold indicates best)
No Joint
No Staged
TEA
TEA(L)
TEA(LP)
MSE(B)
1
0.081 + 0.011
0.098 + 0.018
0.057 + 0.007
0.049 + 0.009
0.057 + 0.009
2
0.084 0.011
0.098 = 0.018
0.057 0.007
0.051 = 0.010
0.058 0.008
3
0.087 0.011
0.101 0.019
0.059 0.007
0.054 + 0.011
0.059 0.008
0.090 0.011
0.105
0.061 0.007
0.056 = 0.011
0.062 0.008
5
0.092 0.011
0.001
0.062 0.008
0.059 + 0.011
0.064 0.009
6
0..99 0.012
0.023
0.065 0.008
0.063 + 0.011
0.068 0.010
7
0.100
0.025
0.068 +0000
0.066 + 0.010
0.001
8
0.10410.004
0.117
0.072 1.011
0.070 1 0.011
0.076 0.013
MSE(C)
1
0.258 0.016
0.001
0.239 +0015
0.231 = 0.020
00006
0.285 0.016
0.297
0.265 #0.014
0.258 + 0.021
0.266 00018
0.311 1.011
1..08
0.287 10014
0.282 1 0.021
0.287 1.018
4
0.346 0.019
0.356
0.322 +0015
0.319 1 0.021
0.321 0.022
5
0.363 0.024
0.373 0.004
0.341 0.009
0.337 + 0.024
0.338 0.026
6
0.389 0.033
0.397 0.031
0.366 0.006
0.366 0.030
0.362 = 0.033
0.416 0.041
0.424 0.040
0.394 0.035
0.401 = 0.043
0.390 0.044
8
0.454 0.059
0.462 0.053
0.430 0.048
0.447 = 0.064
0,427 - 0.063
MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C). The "No Joint" setting isolates the benefit from staged training only (analogous basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).
Table 20: Summary source of gain and variants RNN model with ADNI (Bold indicates best)
No Joint
No Staged
TeA
TEA(L)
TEA(LP)
MSE(B)
0.092 0.014
0.106 + 0.022
0.063 + 0.010
0.058 = 0.012
0.064
0.012
MSE(C)
0.353 1 0.070
0.363 + 0.067
0.330 + 0.066
0.330 0.076
0.329 +: 0.068
MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C). The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining. though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining). Results are grouped over the temporal axis; note that the variance between splits an artifact of this grouping.
Table 21: Extended results for TEA and comparators RNN model with MIMIC (Bold indicates best)
Basc
REG
FEA
TeA
F/TEA
PRC 1
0.159
0
0.159 0.022
0.162 1 0.036*
0.293 + 0.031
0.193 1 0
0.148 *
0.149 0
0.150 = 0.030*
0.254 - 0.025
0.174 + 0.018*
3
0.139
*
0.141 0.015*
0.142 = 0.027*
0.230 0.021
0.162 0.015
4
0.133
0
0.135 = 0.012*
0.135 = 0.024*
0.214 = 0.018
0.153 0.012
5
0.129
+ 0.019*
0.130 + 0.01
0.130 1 0.022*
0.203 + 0.015
0.147 0.011
Roc
1
0.699
+ 0.049*
0.704 + 0.028*
0.709 = 0.060*
0,801 = 0.018
0.745 +0021
28

---

Published as a conference paper at ICLR 2020
2
0.701 0.044
0.707 0.025
0.705 1 0.050*
0.778 0.015
0.740 10.018
0.690 0.0441
0.696 = 0.024
1693=00004
0.758 0.016
0.726
0.681 +
0.688 = 0.023*
0.684 = 0.042*
0.745 = 0.015
0.715
0.679 0.0377 5
0.685 = 0.023*
0,680 = 0.043*
0.736 = 0.012
0.713 0.019
MSE
0.141 00007
0.140 = 0.006
0.138 = 0.010
0.137 = 0.008
0.139 0.007
2
0.159 0.010
0.159 = 0.007
0.160 = 0.008
0.154 = 0.009
0.162 + 0.007*
0.156 0.000
0.155 0.007
0.156 = 0.008
0.158 = 0.008
0.158 0.008
0.154 0.008
0.153 0.008
0.153 + 0.008
0.153 0.009
0.155 + 0.009
0.154 0.010
0.152 + 0.010
0.152 + 0.010
0.150 0.011
0.155 + 0.010
The two-sample r-test for a difference in means is conducted on the results. An asterisk next to the comparator result used to indicate statistically significant difference in means (p-value 0.05) relative the TEA result.
Table 22: Summary results for TEA and comparators on RNN model with MIMIC (Bold indicates best)
Base
REG
FEA
Tea
F/TEA
PRC
0.142 = 0.028*
0.143 0.019*
0.144 0.030*
0.239 + 0.039
0.166 + 0.023*
Roc
0.690 = 0.043*
0.696 + 0.026*
0.694 = 0.050*
0.763 + 0.028
0.728 + 0.023*
MSE
0.153 = 0.011
0.152 0.010
0.152 = 0.012
0.150 - 0.012
0.154 + 0.011
The two-sample 1-test for difference in means is conducted on the results. An asterisk next to the comparator result used to indicate statistically significant difference means (p-value 0.0S) relative to TEA result. Results are grouped over the temporal axis; note that the variance between splits an artifact of this grouping.
Table 23: Extended source of gain and variants on RNN model with MIMIC (Bold indicates best)
No Joint
No Staged
TEA
TEA(L)
TEA(LP)
PRC
0,216 0.044
0.194 0.020
0.293 1 0.031
0.310
0.047
0,280 0.033
2
0.194 1 10035
0.175 1.0.17
0.254 1 0.025
0,265 1 0.035
0.242 1.0226
3
0.178 0.030
0.163 0.013
0.230 0.021
0.239 = 0.030
0,221 0.022
4
0.168 0000
0.154
0.001
0.222 = 0.026
0.206 0.020
0.160 0.0024
0.148 0.000
0.203 0.015
0.210 0.023
0.195 0.018
Roc
0.756 00.42
0.742 0.002
0.801 0.018
0.807 0.025
0.791 0.018
0.741 #0.031
0.738 #.022
0.778 0.015
0.783 0.017
0.773 0.012
3
0.726 +0.01
0.724 10.019
0.758 = 0.016
0.761 = 0.019
0.756 0.013
4
0.715 0.030
0.715 00001
0.745 + 0.015
0.747 - 0.017
0.742 0.001
5
0.710 0.031
0.711 0.018
0.736 + 0.012
0.741 = 0.019
0.736 0.014
MSE
1
0.138 0.008
0.137 0.007
0.137 = 0.008
0.136 = 0.006
0.137 0.008
2
0.158 0.007
0.156 0.012
0.154 = 0.009
0.153 = 0.007
0.154 +
0.155 0.009
0.156 0.010
0.158 + 0.008
0.156 0.010
0.158 0.008
0.152 0.008
0.153 0.009
0.153 - 0.009
0.151 0.011
0.154 + 0.008
5
0.151 0.000
0.150 0.009
0.150 1 0.011
0.147 0.011
0.150 1 0.008
The "No Joint" setting isolates the benefit from staged training only (analogous basic unsupervised pretraining, though using targets); the "No Staged" setting isolates benefit from joint training only (without pretraining).
Table 24: Summary source of gain and variants RNN model with MIMIC (Bold indicates best)
No Joint
No Staged
Tea
TEA(L))
TEA(LPP)
PRC
0.183
0.038
0.167 + 0.022
0.239 0.039
0.249 + 0.049
0.229 + 0.039
Roc
0.730 = 0.038
0.726 + 0.023
0.763
0.028
0.768 + 0.031
0.759 0.025
MSE
0.151 + 0.011
0.150 - 0.012
0.150 0.012
0.149 1 0.012
0.151 0.011
"No Joint" setting isolates the benefit from staged training only (analogous basic unsupervised pretraining. though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining). Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
E.3 Results for Sensitivities
29

---

Published as a conference paper at ICLR 2020
Table 25: Extended o-Sensitivities for REG linear model with UKCF (Bold indicates best)
0
30-5
3e-4
3
3c-2
PRC(1)
1
0.340 0.118
0.355 0.114
0.370 + 0.101
0.320 0.034
0.163 0.003
2
0.325 0.099
0.338 0.096
0.350 + 0.085
0.309 0.026
0.176 0.004
3
4.314:00008.
0.327 0.086
0.337 + 0.076
0.300
0.182 0.005
4
0.307 0.082
0.318 0.080
0.328 + 0.071
0.293 + 0.023
0.184 0.004
PRC(C)
1
0.445 0.131
0.460 0.11
0.467 = 0.106
0.0034
0.240 0.000
2
0.418 + 0.099
0.428 0.095
0,435 = 0.081
0.025
0.260 0.005
3
0.403 0.082
0
0.418 + 0.067
0.0021
0.272 #0.006
0.398 + 0.073 4
0.406 0.070
0.412 + 0.060
0.281 0.007
Roc(1)
I
0.713 0.104
0.724 0.100
0.737 = 0.081
0.022
0.527 0.006
2
0.688 + 0.089
0.697 0.086
0.709 + 0.070
0.693 0.022
0.532 0.008
0.677 + 0.080
0.686 0.078
0.697 + 0.065
0.683 0.021
0.543 0.007
0.677 0.07 4
0.686 0.074
0.696 + 0.063
0.681 0.019
0.555 0.007
Roc(C)
1
0.713 0.110
0.727 0.110
0.741 + 0.088
0.716:00022
0,512 0.010
2
0.686 + 0.091
0.696 0.086
0.707 + 0.072
0.686 0.022
0.523 0.008
3
0.668 0.077
0.678 0.074
0.686 + 0.061
0.668 0.021
0.530 0.013
4
0.651 1 0.070
0.660 0.066
0.667 1 0.056
0.652 0.018
0.527 1.015
The v coefficient controls the strength of fz-regularization applied on top of the original loss function minimized. PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C).
Table 26: Summary v-Sensitivities for REG on linear model with UKCF (Bold indicates hest)
= 0
3e-5
3e
3e-3
3e
PRC(1)
0.322 +: 0.099
0.335 + 0.096
0.347 + 0.085
0.305 + 0.029
0.176 = 0.009
PRC(C)
0.416 +: 0.100
0.426 + 0.097
0.433 + 0.083
0.408 + 0.028
0,263 = 0.016
ROc(1)
0.689 + 0.089
0.698 0.087
0.710 + 0.072
0.693 0.025
0.540 0.013
Roc(c)
0.679 0.091
0.690 0.087
0.700 + 0.075
0.681 0.032
0.523 0.013
The y coefficient controls the strength of fz-regularization applied on top of the original loss function minimized. PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C). Results grouped over the temporal axis; that the variance between splits artifact of this grouping.
Table 27: Extended y-Sensitivities for TEA on linear model with UKCF (Bold indicates hest)
3e
3c
3e 3
3e-2
PRC(1)
1
0.484 + 0.020
0.489 0.019
0.497 + 0.016
0.442 + 0.005
0.174 + 0.004
2
0.450 0.016
0.453 0.016
0.459 + 0.017
0.414 + 0.007
0.186 + 0.005
3
0.424 0.014
0.426 00011
0.432 1 0.015
0.394 0.009
0.192 0.005
4
0.405 0.012
0.407 10.100
0.413 0.010
0.381 + 0.008
0.193 0.004
PRC(C)
1
0.641 0.021
0.644 0.001
0.653 + 0.009
0.612 + 0.009
0.276 0.007
2
0.561 1.001
0,562 100011
0.566 1 0.010
0.544 1 0.007
0.293 0.008
3
0.519 0.000
0.519
0.521 1 0.008
0.508 0.005
0.302 0.008
4
0.495 1 0.008
0.0007
0.498 1 0.009
0.489 0.007
0.309 0.000
Roc(1)
1
0.800 0.015
0.803 0.015
0.806 1 0.007
0.779 0.007
0.555 0.005
2
0.765 0.009
0.767 0.011
0.771 1 0.007
0.751 0.007
0.557 0.007
0.746 0.008
0.000
0.750 + 0.007
0.735
1 0.007
0.565 0.006
+.7000 4
0.737 000007
0.740 + 0.008
0,727 0.005
0.575 0.006
RoC(C)
0.825 + 0.011
0.826
0.010
0.829 + 0.006
0.819 + 0.006
0.560 0.011
2
0.772 + 0.010
0.774
0.009
0.775 + 0.008
0.771 + 0.006
0.564 + 0.009
3
0.742
+ 0.004
0.744 0.004
0.743 + 0.005
0.741 + 0.004
0.566 0.012
4
0.718 + 0.006
0.720 0.006
0.720 + 0.006
0.717 1 0.008
0.561 0.015
The y coefficient controls the strength of ezrreuularization applied on top of the original loss function minimized.
PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C).
30

---

Published as a conference paper at ICLR 2020
Table 28: Summary o-Sensitivities for TEA linear model with UKCF (Bold indicates best)
3e-5
1 = 3e-4
V 3e-3
3e-2
PRC(1)
0.441 0.033
0.444 0.034
0.450 + 0.035
0.408
0.024
0.186 + 0.000
PRC(C)
0.554 + 0.057
0.555 + 0.058
0.559 + 0.060
0.538 + 0.047
0.295 = 0.015
Roc(1)
0.762 1 0.026
0.763 1: 0.028
0.767 1 0.026
0.748 1 0.021
0.563 + 0.010
Roc(C)
0.764 1 0.041
0.766 1 0.041
0.767 1: 0.042
0.762 1 0.039
0,563 1 0.012
The coefficient controls the strength of fz-regularization applied top of the original loss function minimized. PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C). Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
Table 29: Extended X-Sensitivities for TEA on linear model with UKCF (Bold indicates best)
= 0
x= 0.01
= 0.1
= 0.5
À A= 0.9
À = 0.99
X x=
PRC(1)
0.370 0.101
0.383 1.100
0,383
0.412 0.0091
0.472
100016
0,461 1:0012
0,327 1.0011
0.150 1 0.008
2 0.350 0.085
0.361
0.090
0.006
0.439
0.431 0.016
0.323 +0.011
0.162 0.008
3 0.337 0.076
0.347
0.368 0.088
0.415 0.015
0.316 0.000
0.168 0.007
4 0.328 00071
00007
0.357
00004
0.399
0.01
0.005
0.307
#0.011
0.170
PRO(C)
0.467 +: 0.16
0.481
0..110
0.528 0.100
0.620
+: 0,030
0.016
0.433
0.011
0.236 0.012
2 0.435 + 0.081
0.445
0.481
0.007
0.550 0.002
0.55
0.427
+00010
0.249 0.010
3 0.418 0.067
0.427
1.0000
0.456
1.006
0.512 10.017
0.51611..00
0.421
1.0009
0.259 1.011
4 0.412 + 0.060
0.062
0.445
0.057
0.491
00.05
0.494
0.009
0.415
0.266 0.011
Roc(1)
I 0.737 0.081
0.742
0.089
0.764 0.067
0.799 0.007
0.791
+ 0.004
0,708
0.008
0.499 + 0.013
2 0.709 0.070
0.713
0.07
0.733 0.058
0.765 = 0.00
0.760
= 0.008
0.694
0.007
0.502 = 0.014
3 0.697 1. 0.065
0,701
1.0071
0,719 0.054
0.750
1 0.008
0,746
0.000
0.690
1.000
0,500 1 0.014
4 0.696 1 0.063
0.699 1 0.068
0.715 0.053
0.744
0.005
0.741
+ 0.006
0.690
0.000
0.501 0.015
Roc(c) I 0.741 = 0.088
0.747 = 0.092
0.775 0.075
0,821 + 0.011
0.819 : 0.006
0.725
0.014
0.493 0.034
2 0.707 1 0.072
0.712 0.006
0.735 1 0.061
0.774 1 0.009
0.774 1 0.007
0,704
0.012
0.496 1 0.027
3 0.686 + 0.061
0.691
+ 0.067
0,711
+ 0.052
0.746 + 0.007
0.745
+ 0.003
0.690
0.011
0.497 = 0.028
4 0.667 + 0.056 0.672 0.059
0.689 1 0.048
0.722 1 0.007
0.721 + 0.005 0.675 0.014 0.497 1 0.025
The À coefficient controls the strength of prior-i.e. the tradeoff' between the prediction and reconstruction loss.
PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C).
Table 30: Summary X-Sensitivities for TEA on linear model with UKCF (Bold indicates best)
== 0
>= 0.01
>= 0.1
À = 0.5
À = 0.9
A= 0.99
À >= 1
PRC(1)
[0.347 +: 0.085
0.357
+: 0.090
0.381
+: 0.078
0.431
+: 0.031
0.424 + 0.027
0.318 0.013
0.162 0.001
PRC(C)
00.433 +: 0.083
0.443 1: 0.087
0.477
1: 0.084
0.543 1: 0.054
0.546 1: 0.050
0.424
0.012 0.252 0.016
Roc())
0.7110 1 0.072
0.714
1 0.078
0.733
1 0.061
0.764
1. 0.022
0.759
T 0.020
0.695 1 0.010 0.501 10.014
Roc(c) 0.700 1 0.075 0.705 + 0.080 0.727 1: 0.068 0.766 1: 0.038 0.765 1: 0.037 0.698 0.022 0.496 0.029
The À coefficient controls the strength of prior-i.c. the tradcoff between the prediction and reconstruction loss. PRC and RoC evaluations are reported separately for targets representing infections (I) and comorbidities (C). Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
Table 31: Extended N N-Sensitivities for REG on linear model with UKCF (Bold indicates best)
T
N x 1%
N x 5%
N x 20%
N x 50%
N' x 100%
PRC(1)
0.160 0.010
0.176 0.024
0.199 0.044
0.325 0.114
0.370 0.101
2
0.172 0.009
0.187 0.021
0.207 0.037
0,312 0.095
0.350 0.085
3
0.180 0.011
0.192 + 0.020
0.209 0.032
0.303 +: 0.085
0,337 + 0.076
0.182 0.009 4
0.193 + 0.020
0.208 + 0.028
0.295 0.078
0.328 + 0.071
PRC(C)
I 0.246 +0.011
0.268 0.034
0.293 0.046
0.421 10.119
0.467 + 0.106
2
0.263 0.011
0.283 0.022
0.304 0.036
0.401 0.091
0.435 + 0.081
3
0.2755 0.013
0.292 00028
0.313 0.0032
0.006
0.418 0.067
4
0.003
0.302 00025
0.319 0.029
0.384 + 0.066
0.412 0.060
Roc())
1
0.512 0.024
0.553 0006
0.598 0.057
0.705 + 0.090
0.737 0.081
0..025 2
0.557 0.040
0.594 0.049
0.681 + 0.077
0.709 0.070
31

---

Published as a conference paper at ICLR 2020
3
0.521 0.022
0.549 1.037
0.590 0.045
0.671 0.071
0.697 0.065
0.519 0.024
0.546 0.037
0.590 0.039
0.669 0.068
0.696 0.063
Roc(c)
1
0.507 1 0.026
0.539 0.049
0.591 1 0.054
0.702 1:0101
0.741 1: 0.088
0.520 0.024
0.546 0.040
0.587 0.041
0.082
0.707 0.072
3
0.526
= 0.023
0.549 = 0.038
0.586 + 0.036
0.659 0.070
0.686
0.061
4
0.528 + 0.027
0.550 + 0.039
0.581 + 0.034
0.642 = 0.062
0.667 + 0.056
The proportion data N used is randomly restricted, showing performance under various levels of data scarcity. PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C).
Table 32: Summary N- V-Sensitivities for REG on linear model with UKCF (Bold indicates best)
1%
N x 5%
N x 20%
x 50%
x 100%
PRC(1)
0.173 0.013
0.187 1 0.022
0.206 + 0.036
0.309 0.094
0.347
0.085
PRC(C)
0.267 0.019
0.286 0.031
0.307 1 0.038
0.399 0.092
0.433
0.083
Roc(1)
0,517 + 0.025
0.551 0.041
0.593 = 0.048
0.682 0.078
0.710 = 0.072
RoC(C)
0.520 + 0.026 0.546 = 0.042
0.586 + 0.042
0.669 + 0.083
0.700 = 0.075
proportion data N used is randomly restricted, showing performance under various levels PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbiditics (C). Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
scarcity.
Table 33: Extended N-Sensitivities for TEA on lincar model with UKCF (Bold indicates best)
N x 1%
N x 5%
N x 20%
N x 50%
N x 100%
PRC(1)
1
0.199 + 0.016
0.342 0.040
0.453 + 0.020
0,486 + 0.017
0.497 0.016
2
0.205 0.013
0,336 + 0.031
0.421 0.017
0.448 + 0.015
0.459 + 0.017
3
0.212 0..011 1.0018
0.322 0.022
0.398 0.015
0.420 0.0114
0.432 1 0.015
0.016 4
0.308 0.025
0.381 0.011
1.0011
0.413 1 0.010
PRC(C)
1
0.287 0.021
0.400
0.610
0.653 0.009
2
0.292 0.018
0.445 0.041
0.535 0.014
0.0112
0.566 + 0.010
0.302 0.014
0.424
0.495 0.011
0
0.521 0.008
0.311 + 0.017 4
0.417 0.030
0.478 0.014
0.493 0.010
0.498 + 0.009
Roc())
0.570 + 0.022
0.698 +0022
0.776 + 0.008
0.797
0.007
0.806 0.007
2
0.565 0.027
0.684 0.019
0.744 0.010
0.763 0.007
0.771 + 0.007
3
0.568 0.027
0.663 00001
0.723 +0.111
0..00
0.750 + 0.007
4
0.564 0.026
0.652 0.020
0.707 10.010
0.740 1 0.008
Roc(c)
1
0.581 0.017
0.715 0.032
0.795 0.009
0.822 0.0007
0.829 + 0.006
2
0.565 0.015
0.684 0.019
0.745 0.006
0.768 0.007
0.775 1 0.008
0.570 1 0.015
0.668 1.018
0.718 0.007
0.737 0.005
0.743
0.005
4
0.570 1 0.020
0.653 100015
0.698 1 0.008
0.715 1 0.008
0.720 0.006
The proportion of data N used is randomly restricted, showing performance under various levels of data scarcity. PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbidities (C).
Table 34: Summary N N-Sensitivities for TEA linear model with UKCF (Bold indicates best)
1%
N 5%
N 20%
x 50%
x 100%
PRC(1)
0.207 0.017
0.327 + 0.034
0.413 + 0.031
0.439 0.035
0.450
0.035
PRC(C)
0.298 + 0.020
0.439 + 0.047
0.530 + 0.053
0.551 + 0.058
0.559 + 0,060
Roc(1)
0.567 1 0.026
0.674 1: 0.029
0.738 1 0.027
0.757 1: 0.027
0.767 : 0.026
Roc(C)
0.572 1.0.18
0.680 1 0.032
0.739 1 0.038
0.760 1 0.041
0.767 1 0.042
The proportion of data N used is randomly restricted, showing performance under various levels of data scarcity. PRC and Roc evaluations are reported separately for targets representing infections (I) and comorbiditics (C). Results grouped over the temporal axis; note that the variance between splits is artifact of this grouping.
E.4 Contrived Negative Example
32

---

Published as a conference paper at ICLR 2020
Here with give contrived example where the goals of prediction and reconstruction directly odds with each other. Specifically, we show setup where the "and" part of "compact and predictable" representations is impossible- that is, a compact sugget-representttion that is more reconstructive ends up being less predictable. Consider the following (true) data generating process: Latent variables P, u each consist of 10 independent dimensions, where Pt (O, 1) and u, -N(0,1 1). Feature vectors x are also of length 10, and are linear in p. Target vectors = lyp Yul are of length 50, where yp is of length 10 and linear in p and yu of length 40 and linear in u. See Figure 7 Note that the input features are inadequate for predicting all targets; our choice of lettering denotes elements that are in principle predictable ("P"), and those that are in principle unpredictable ("U").
dim
Yu
dim = 40
dim = 10
P
yr
dim 10
dim 10
Figure 6: Data generating process for synthetic example. Latent vectors p, u linearly generate feature vectors x and target vectors y. In this situation, yr is in principle predictable from x, while yu is impossible to predict.
Suppose this data generating process is unknown to us. First, consider what happens with direct prediction: A linear model would learn to predict yp well, while predictions of yu would be no better than random. So far, good. Now given the feature and target dimensions, consider the (not unreasonable) choice of TEAS with a latent dimension During reconstruction, we naturally get more bang for our buck by encoding more of the (highly compressible) yu instead of yp; yet yu is entirely useless to encode, as it is not predictable from inputs anyway. Reconstructing well is therefore directly odds with predicting well. This is certainly an extremely contrived scenario; nevertheless, without sufficient domain knowledge, it serves as a caveat that- as with feature- embteddi paradigms- target-embedding is only good as its assumptions.
10. This is an obvious problem:
Prodiction Emor y)
0.3
MSE
Reconstructoo on
Enor (V)
0.1
Prediction
Sensitivities on
Figure 7: Synthetic scenario where prediction is directly at odds with reconstruction. The prior (that we can lev- erage compact and predictable representations of targets) is hugely incorrect in this case; as a result, not only is angee-intoencoding not beneficial it is positively harmful. For TEAS, observe that the strength-of of-rrior coefficient À increases, the overall prediction error actually increases (while the reconstruction error decreases).
Table 35: Results for TEA and comparators on linear model for negative example (Bold indicates best)
Base
REG
FEA
TEA
F/TEA
MSE
0.266 + 0.045
0.266
0.045
0.266 + 0.045
0.285 = 0.047
0.280 + 0.04
MSE(U)
0.333 1: 0.056
0.333 + 0.056
0.333 1: 0.056
0.334 1: 0.056
0.334 1: 0.056
MSE(P)
0.000 +: 0.000
0.000
0.000
0.000 + 0.000
0.087 + 0.028
0.061 0.022
MsE metrics further reported separately for targets that in principle predictable (P) and unpredictable (U).
33

---

Published as a conference paper at ICLR 2020
Table 36: X-Sensitivities for TEA linear model for negative example (Bold indicates best)
0.01
À 0.1
0.5
À = 0.9
MSE
0.266 + 0.045
0.267 + 0.045
0.271 + 0.046
0.285 + 0.047
0.292 + 0.051
MSE(U)
0.333 + 0.056
0.333 + 0.056
0.334 + 0.056
0.334 + 0.056
0.334 0.056
MSE(P)
0.000 - 0.000
0.003 1 0.005
0.022 1 0.009
0.087 1 0.028
0.125 0.063
MSE metrics are further reported separately for targets that are in principle predictable (P) and unpredictable (U).
E,5 Results from Open Discussion
One can also ask the (purely empirical) question of how much each model degrades on out-of- distribution data-without additional training to fine-tune the model to the new data. In this context, we actually have no reason to expect TEAS to degrade any more or less than comparators. For thoroughness, we show an additional experiment as an example for sensitivity analysis (using UKCF) as follows: Each model is trained (only) on male patients and tested (only) on female patients, and vice versa. The average results on held-out samples from in-distribution data and out-of-distribution data then allow us to compute the net degradation (i.e. negative difference), which is reported below. While TEAS individually perform better overall on both in mn-distribution and out-of-distribution samples, none of the differences in the amounts of degradation between models are statistically significant:
Table 37: Performance degradation for TEA and comparators on linear model with UKCF (Bold indicates best)
Base
REG
FEA
TeA
F/TEA
Roc(1)
0.019 = 0.015
0.020 + 0.014
0.019 0.01
0.020 = 0.016
0.017 + 0.014 0.019 + 0.018
Roc(c)
0.025 + 0.015
0.029 + 0.015
0.013 = 0.020
PRC(I)
0.022 = 0.020
0.018 + 0.021
0.021 0.022
0.033 = 0.022
0.027 0.022
PRC(C)
0.026 = 0.021
0.029 + 0.018
0.026 1
0.018 = 0.023
0.021 0.019
PRC and RoC evaluations are reported separately for targets representing infections (I) and comorbidities (C). The two-sample r-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate statistically significant difference means (p-value 0.05) relative to the TEA result. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
Finally, given the staged training in Algorithm 1, it should be clear that the order cannot be changed. Stage requires the encoder to already be trained to provide the requisite embeddings, so it must be preceded by Stage 1. Therefore the only relevant possibilities are: (1) Stages 1-2 by themselves, without Stage 3; this is simply the "No Joint" setting. (2) Stage 3 by itself, without Stages 1-2; this simply the "No Staged" setting. (3) None of the stages altogether; this is simply the "Neither" setting. (4) Stages 1, 2, and 3 in order; this is simply Algorithm I itself. The only remaining possibility is to have Stage 3 precede Stages 1-2. This makes little sense, since when the reconstruction loss is trained by itself it is likely to "undo" the result of joint training. For thoroughness, we run an additional sensitivity experiment (using UKCF) to confirm this. The following corresponds to the left half of Table 4, with the additional column on the right (and the other columns labeled to reflect the training stages). Verifying our intuitions, the setting "3-1-2" behaves almost identically to the setting "1-2":
Table 38: Performance by training stages for TEA linear model with UKCF (Bold indicates best): column headers indicate the sequence of training stages executed (note that "1-2-3" simply corresponds Algorithm 1)
None
1-2
3
1-2-3
3-1-2
Roc(1)
0.710 +.002
0.747 1: 0.022
0.764 + 0.022
0.767 1: 0.026
0.749 0.022
ROC(C)
0.700 +
0.744 1 0.038
0.766 1 0.038
0.767 1 0.042
0.747 1 0.037
PRC(1)
0.347 + 0.085
0.402 + 0.026
0.431 + 0.031
0.450 + 0.035
0.404 + 0.027
PRC(C)
0.433 1 00.03
0.507 1 0.040
0.543 1 0.054
0.559 1 0.060
0.512 1 0.042
PRC and Roc evaluations reported separately for targets representing infections (I) and comorbidities (C). The two-sample 1-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate statistically significant difference in means (p-value < 0.05) relative to the TEA result. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
34


Reviews:
[{"review_id": "ryx7d3w75H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary: In this paper, the authors proposed a Target-Embedding Autoendocer (TEA) model for supervised representation learning. Different from the traditional feature embedding autoencoder model, TEA tries to learn a compact latent representation that can reconstruct the target vector. Hypothetically, this model should be especially useful when the target vector has a much higher dimension than the feature vector. The authors analyzed the proved some characteristics of this framework and conducted empirical experiments on three datasets to prove its effectiveness.\n2. Overall assessment: The motivation of this paper is well justified. It's easy to follow and fun to read, even for a person who is not an expert in this area, like me. However, there still exist some problems in this paper. It needs more improvement to get published in a competitive conference like ICLR.\n3. Comments:\n3.1 Datasets used in this paper cannot fully prove the effectiveness of this framework. These datasets are all from very similar domains. The dimension of target vectors is comparable to that of feature vectors. In my view, it's necessary to test on more different types of datasets to prove the usefulness of a model, especially if it is a general framework like TEA.\n3.2 Models used in this paper are relatively simple. Demonstrate the performance of TEA on more advanced models and more difficult tasks can deliver more insights to the community.\n3.3 No state-of-the-art models are used in experiments. It's very likely that some existing work has already adopted the idea of target embedding. There also exist much other work on dealing with high dimensional target vector problem. How are the performances of these models? What is the advantage of the proposed framework over these existing work?\n3.4 The source of gain part on page 8 should contain more explanations and analysis. This part is one of the most important parts of this paper. It can provide quite valuable insights to readers. I hope the author can expand it.\n3.5 More details about training and inference are needed. The authors only use a few sentences to describe their three staged training process. I still have some questions left after reading it, such as how do you train the shared parts in TEA? Do you update its parameters in all stages? What the effect of the order of training? What will happen if I change it?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rkecJZjucH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an extremely well-written and well-motivated paper. The idea of target-embedding autoencoders is extremely relevant for problems where the dimension of the label space is as large (or larger) than the dimension of the input features. The experiments are thorough, the theoretical guarantees are extremely well thought of and derived. The applications to modelling the progression of cystic fibrosis and Alzheimer's are extremely useful and timely.  I vote for a strong accept for this paper. \n\nI would like to see some references to the extreme multi-label classification problems (http://manikvarma.org/downloads/XC/XMLRepository.html) and some of the other probabilistic approaches attempted in this domain (please see https://papers.nips.cc/paper/5770-large-scale-bayesian-multi-label-learning-via-topic-based-label-embeddings and the references and citations). ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1x_NdV6FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces the idea of target embedding autoencoders for supervised prediction, designed to learn intermediate latent representations jointly optimized to be both predictable from features and predictive of targets. This is meant to help with generalization and has certain theoretical guarantees. \n\nIt is an interesting problem setting to consider where  Y is high dimensional instead of X. More examples of this would be useful to provide in the intro. I think this is crucial to understand where this method might be useful. \n\nFigure 1 is super informative and very nice!\n\nSection 2: \nWhy do we expect that this paradigm of autoencoder based regularization “generalizes” better?\nI like the explicit and honest discussion of prior work in this section. \nOne question is how important is the choice of reconstruction loss function - L2, vs max likelihood gaussian, vs L1, vs cross entropy, etc for performance?\nAnother question: how bad is performance if the learning is done stagewise - first the Y-Z-Y^ representation is learned and then the X->Z predictor is learned. \nIf something is out of distribution, how easy are TEA based learners to finetune?\nOverall the idea seems reasonable - if the targets have some common set of factors, just predict those instead of predicting the full target value which might be harder to get right. It’s just a question of whether this holds true in many domains and how well this reconstruction loss generalizes across problems?\n\nSection 3: \n“We havenoted that TEA components can in principle be instantiated by any architecture. Does its benefit extend beyond the commonly-studied domain of static classification?” -> not clear what this means? Does this mean this algorithm has been proposed before or is it that it can ALSO work on non static classification tasks? Not clear how to situate this claim\n\nThe theoretical section seems to follow largely from Le et al, but with important distinctions on dimensionalities of various spaces involved. I wonder if the authors can comment on how often Assumption 1 and 2 are actually satisified?\n\nRelated Work: \nIs the main difference between Yu 2014 and this just in the norm based regularization? I don’t think so, can this be made more clear. This seems also fairly related to Yeh, is it just a generalization of that paradigm? Or is there more to it? In light of the contribution of Yeh, this seems like slightly more marginal of a contribution? Is the main points of contribution the theoretical analysis and the extended experiments to sequence data rather than static classification?\n\nThe results do seem to show a signficant benefit as compared to FEA or base models. It also seems like this is applicable across multiple disease datasets. Do the authors think that this could be applicable to other domains altogether? Would it be quick to run a comparison on these?\n\nGenerally seems like a well grounded and meaningful contribution with many improvements. Would be curious to see applications to other datasets and also some improvements/clarifications noted above?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gkm4ET5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines target-embedding autoencoders (TEAs) in theory and practice. TEAs autoencode the output (rather than input) space and find a mapping from the input to the latent representation of the output. The forward pass of the decoder (for the output space) is shared by the input-to-output computation.\n\nTarget-embedding autoencoders (TEAs) have previously been proposed and used in practice (though not necessarily by the \"TEA\" name).  The paper's presentation is confusing on this matter, at it claims to be the first to \"motivate and formalize\" TEAs; I do not believe it is appropriate to claim such a contribution in light of prior work. [Girdhar et al.] clearly utilizes a target-embedding autoencoder (see [Girdhar et al.] Figure 2). In addition, more recent published work clearly utilizes TEAs (though not named as such) as the centerpiece of their approaches. See, for example:\n\n[A] Adrian V. Dalca, John Guttag, Mert R. Sabuncu. Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation. CVPR, 2018.\n\n[B] Mohammadreza Mostajabi, Michael Maire, Gregory Shakhnarovich. Regularizing Deep Networks by Modeling and Predicting Label Structure. CVPR, 2018.\n\nFigure 2 of [A] and Figure 1 of [B] both clearly depict applying target-embedding autoencoders on semantic image segmentation problems. [B] operates in the same supervised representation-learning setting proposed here. Notably, [B] utilizes staged training -- learning the autoencoder first -- as discussed in Section 2 of the submitted paper, and finds that to be important for achieving a regularization effect.\n\nThe real applications explored by [A] and [B] are perhaps more challenging than the datasets used in experiments here.  The concluding sentence of the paper,\"Target-representation learning is potentially applicable to any high-dimensional prediction task, and exploring its utility for specific domain-architectures may be a practical direction for future research\" should be changed -- prior work has already successfully utilized TEAs in the specific domain of image segmentation.\n\nGiven that the paper has missed (not cited) highly related published work that applies TEAs in practice, a rewrite of Section 4 is required. In the appendix, Table 6, Table 7 and Section B.1 also need significant updates. The proposed approach is no longer a unique entry in Table 6 or 7 -- e.g. [B] already contributed \"autoencoder component as regularization for learning predictor\" (Table 7). Additionally, toy experiments in Section 5 appear less significant a contribution when multiple full-scale systems already employ TEAs.\n\nThis paper's theoretical analysis does appear to set it apart from prior work. However, theorems are developed for an extremely limited context (linear TEAs) and it is unclear whether or how they might extend to practical use cases (i.e. TEAs that are nonlinear, deep neural networks).\n\n---\n\nThe extensive author response and updated paper address many of my original concerns.  I have updated my overall rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a general view of supervised learning models that are jointly trained with a model for embedding the labels (targets), which the authors dub target-embedding autoencoders (TEAs).  Similar models have been studied before, but this paper unifies the idea and studies more carefully various components of it.  It provides a proof for the specific case of linear models and a set of experiments on disease trajectory prediction tasks.  The reviewer concerns were addressed well by the authors and I believe the paper is now strong.  It would be even stronger if it included more tasks (and in particular some "typical" tasks that more of the community is focusing on), and the theoretical part is to my mind not a major contribution, or at least not as large as the paper implies, because it analyzes a much simpler model than anyone is likely to use TEAs for.
Decision: Accept (Talk)