Title: TARGET-EMBEDDING AUTOENCODERS FOR SUPERVISED REPRESENTATION LEARNING

Abstract: Autoencoder-based learning has emerged as a staple for disciplining representations in unsupervised and semi-supervised settings. This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of targetembedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets-encoding the prior that variations in targets are driven by a compact set of underlying factors. As our theoretical contribution, we provide a guarantee of generalization for linear TEAs by demonstrating uniform stability, interpreting the benefit of the auxiliary reconstruction task as a form of regularization. As our empirical contribution, we extend validation of this approach beyond existing static classification applications to multivariate sequence forecasting, verifying their advantage on both linear and nonlinear recurrent architectures-thereby underscoring the further generality of this framework beyond feedforward instantiations.

1. INTRODUCTION:

Representation learning deals with uncovering useful underlying structures of data, and autoencoders (Hinton & Salakhutdinov, 2006) have been a staple in a variety of problems. While much research focuses on its use in unsupervised or semi-supervised settings with such diverse objectives as sparsity (Ranzato et al., 2007) , generation (Kingma & Welling, 2013) , and disentanglement (Chen et al., 2018) , autoencoders are also useful in purely supervised settings-in particular, adding an auxiliary feature-reconstruction task to supervised classification problems has been shown to empirically improve generalization (Le et al., 2018) ; in the linear case, the theoretically quantifiable benefit matches that of simplistic norm-based regularization (Bousquet & Elisseeff, 2002; Rosasco & Poggio, 2009) .

In this paper, we consider the inverse problem setting where the target space Y is high-dimensional; for instance, consider the multi-label classification tasks of object tagging, text annotation, and image segmentation. This is in contrast to the vast majority of works designed to tackle a high-dimensional feature space X (where commonly |X | |Y|, such as in standard classification problems). In this setting, the usual (and universal) strategy of learning to reconstruct features (Weston et al., 2012; Kingma et al., 2014; Le et al., 2018) may not be most useful: learning latent representations that encapsulate the variation within X does not directly address the more challenging problem of mapping back up to a higher-dimensional Y. Instead, we argue for leveraging intermediate representations that are compact and more easily predictable from features, yet simultaneously guaranteed to be predictive of targets. In the process, we provide a unified theoretical perspective on recent applications of autoencoders to label-embedding in static, high-dimensional classification problems (Yu et al., 2014; Girdhar et al., 2016; Yeh et al., 2017) . Extending into the temporal setting, we further empirically demonstrate the generality of target-embedding for recurrent, multi-variate sequence forecasting.

Our contributions are three-fold. First, we motivate and formalize the target-embedding autoencoder (TEA) framework: a general approach applicable to any underlying architecture. Second, we provide a theoretical learning guarantee in the linear case by demonstrating uniform stability; specifically, we obtain an O(1/N ) bound on instability by analogizing the benefit of the auxiliary reconstruction task to a form of regularization-without incurring additional bias from explicit shrinkage. Finally, we extend empirical validation of this approach beyond the domain of static classification: using the task of multivariate disease trajectory forecasting as case study, we experimentally validate the for wa rd (sh ar ed ) em be d predict Y < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > (b) Target-Embedding Autoencoder (a) Feature-Embedding Autoencoder advantage that TEAs confer on both linear and nonlinear architectures using real-world datasets with both continuous and discrete targets. To the best of our knowledge, we are the first to formalize and quantify the theoretical benefit of autoencoder-based target-representation learning in a purely supervised setting, and to extend its application to the domain of multivariate sequence forecasting.

2. TARGET-EMBEDDING AUTOENCODERS:

Let X and Y be finite-dimensional vector spaces, and consider the supervised learning problem of predicting targets y ∈ Y from features x ∈ X . With a finite batch of N training instances D = {(x n , y n )} N n=1 , the objective is to learn a mapping h : X → Y that generalizes well to new samples from the same distribution. The vast majority of existing work consider the setting-most commonly, classification-where |X | |Y|; under this scenario, autoencoders are often used to first transform the input into some lower-dimensional representation z ∈ Z amenable to the downstream task. Doing so involves adding an auxiliary reconstruction loss r to the primary prediction loss p .

Formally, solutions of this form-in supervised and semi-supervised settings alike-consist of a shared forward model φ : X → Z, a reconstruction function r : Z → X , and a prediction function d : Z → Y during training (where notation d reflects the downstream nature of the prediction task). Denote x = r(φ(x)) and ŷ = d(φ(x)); then the complete loss function takes the following form,

EQUATION

In contrast, we focus on settings where the target space Y is high-dimensional, and where possibly |Y| > |X |. In this case, we argue that learning to reconstruct the input is not necessarily most beneficial. In a simple classification problem, autoencoding inputs leverages the hypothesis that a reconstructive representation is also likely discriminative. In our setting, however, the more immediate problem is the high-dimensional structure of Y; in particular, there is little guarantee that intermediate representations trained to encapsulate x are easily mapped back up to higher-dimensional targets.

Our goal is to make use of intermediate representations that are both predictable from features as well as predictive of targets. A target-embedding autoencoder (TEA)-versus what we shall term a feature-embedding autoencoder (FEA)-flips the model architecture around by learning an embedding of target vectors instead, which a predictor then learns a mapping into. This involves an encoder e : Y → Z, an upstream predictor u : X → Z, and a shared forward model θ : Z → Y. Denote ỹ = θ(e(y)) and ŷ = θ(u(x)); the complete loss function is now of the following form, embeddings (generated by the encoder): the parameters of u are learned (on the latent loss) while the autoencoder is frozen. Finally, all three components are jointly trained on both prediction and reconstruction losses (Equation 2): parameters of the predictor, embedding, and shared forward model are trained simultaneously. Note that during training, the forward model receives two types of latents as input: encodings of true targets, as well as encodings predicted from features. At inference time, the target-embedding arm is dropped, leaving the learned hypothesis h = θ • u for prediction. Figure 4 (Appendix C) provides step-by-step block diagrams of both training and inference in greater detail.

EQUATION

We emphasize that TEAs-as is the case with FEAs-specify a general framework independent of the implementation details of each component. For instance, the solutions to applications in Yu et al. (2014) , Yeh et al. (2017) , and Girdhar et al. (2016) can be abstractly regarded as linear and nonlinear instances of this framework, with domain-specific architectures (see Section 4 and Appendix B for more detailed discussions). Linear TEAs-which we study in greater detail in Section 3-involve parameterizations (Θ, W u , W e ) of (θ, u, e) consisting of single hidden layers with linear activation.

3. STABILITY-BASED LEARNING GUARANTEE:

Two questions are outstanding. The first is theoretical. We are motivated by the prior that variations in target space are driven by a lower-dimensional set of underlying factors. In this context, can we say something more rigorous about the benefit of TEAs? In this section, we take the first step in showing that jointly learning target representations improves generalization performance in the supervised setting. Specifically, we demonstrate that linear TEAs are characterized by uniform stability, from which theoretical guarantees are known to follow. The second question is empirical. We noted above that certain applications of label-embedding to classification can be interpreted through this framework. Does the benefit extend beyond its static, feedforward instantiations-into the temporal setting for multi-variate sequence forecasting, with both continuous and discrete targets? In Section 5, we first validate our theoretical findings with linear models and sensitivities, as well as extending our empirical analysis to the realm of recurrent, nonlinear models for both regression and classification. 

= 1 N N n=1 [ p (ΘW u x n , y n ) + r (ΘW e y n , y n )]

following Equation 2. Interpreting the jointly learned autoencoding component as an auxiliary task, we show that the TEA algorithm for learning the shared forward model Θ is uniformly stable with respect to the domain of the supervised prediction task. To establish our notation, first recall the following:

Definition (Generalization Bound) Given a learning algorithm D → h D that returns hypothesis h D , let R(h D ) = (h D (x), y)dµ(x, y) denote the risk, and R(h D ) = 1 N N n=1 (h D (x n ), y n ) denote the empirical

risk, where is some loss function. A generalization bound is a probabilistic bound on the defect that takes the following form: R(h D ) -R(h D ) ≤ with some confidence 1δ.

Definition 2 (Uniform Stability) Let D i denote a modification of batch D where the i-th training instance (x i , y i ) is replaced by an independent and identically distributed example (x i , y i ). A learning algorithm is said to be γ-uniformly stable with respect to the loss function

if ∀D ∈ (X × Y) N , ∀i ∈ {1, ..., n}, ∀(x, y), (x i , y i ) ∈ X × Y : | (h D (x), y) -(h D i (x), y)| ≤ γ.

Uniform stability holds if the minimum value of γ converges to zero as batch size N increases without limit.

Uniform stability can be used to derive algorithm-dependent generalization bounds. In particular, Bousquet & Elisseeff (2002) first showed that the defect of a γ-uniformly stable algorithm is is less than O((γ + 1/N ) N log(1/δ)) with probability ≥ 1δ. Feldman & Vondrak (2018) recently demonstrated an improved bound of O( (γ + 1/N ) log(1/δ)). Here, we show uniform stability for linear TEAs, where γ is O(1/N )-by which a tight generalization bound follows immediately. Before we begin, we introduce two additional tools: c-strong convexity and σ-admissibility. Note that these conditions are standard and easily satisfied-for instance, by the quadratic loss function; for more context see for example Bousquet & Elisseeff (2002 ), Liu et al. (2016 ), and Mohri et al. (2018) .

Definition 3 (c-Strong Convexity) Differentiable loss function is c-strongly convex if ∀h, h ∈ H : h (x) -h(x), ∇ (h (x), y) -∇ (h(x), y) ≥ c h (x) -h(x) 2

2 for some c ∈ R + , where ∇ (h(x), y) denotes the gradient with respect to h(x), and •, • denotes the dot product operation.

Definition 4 (σ-Admissibility) Loss function is σ-admissible with respect to the underlying hypothesis class

H if ∀h, h ∈ H : | (h (x), y) -(h(x), y)| ≤ σ h (x) -h(x) 2 for some σ ∈ R + .

To obtain uniform stability, we make two assumptions-both analogous to prior work arguing from the benefit of learning shared models between tasks. Liu et al. ( 2016) deals with learning multiple tasks in general, and Le et al. ( 2018) deals with reconstructing inputs in what we describe as FEAs. Now in multi-task learning, the separate tasks are usually chosen due to some prior relationship between them. In the case of Assumption 1 in Liu et al. (2016) and Assumption 5 in Le et al. (2018) , this is assumed to come from similarities in feature structures across tasks; hence their assumptions of cross-representability are made in feature space. (Note that this restricts primary and auxiliary features to be elements of the same space). Our setting is contrary: the inputs to primary and auxiliary tasks come from different spaces, but are trained to produce similar labels through a compact, shared latent space; hence our assumption of cross-representability will be made in this latent space instead.

Assumption 1 (Representative Vectors) There exists a representative subset of target vectors B = {b 1 , ..., b M } ⊂ {y 1 , ..., y N } such that the latent representation of any individual (x, y) can be linearly reconstructed from that of the representative subset with small error, i.e.

W p x = M m=1 α m W e b m + η and W e y = M m=1 β m W e b m + η for some coefficients α m , β m ∈ R, where Σ M m=1 α 2 m ≤ r 2 α , Σ M m=1 β 2 m ≤ r 2

β for some r α , r β ∈ R + , and η is a small error satisfying η 2 ≤ ε. Remark 1 This assumption is comparatively mild, even for ε = 0. Note that in Liu et al. (2016) the features for separate tasks come from different examples in general, and the similarity of their distributions within X is simply assumed. Here, each pair of inputs to the prediction and reconstruction tasks comes from the same instance, and similarity within Z is explicitly enforced through the (joint) training objective. In addition, observe that the assumption will hold with zero error as long as the number of independent latent vectors is at least |Z|. Furthermore, unlike the scenarios in Liu et al. (2016) and Le et al. (2018) we do not require that the input domains of the two tasks be identical. Therefore for ease of exposition, we assume going forward that ε = 0 (see Remark 6 in Appendix A). 2018) banks on the fact that the prediction arm projects the latent into a lower-dimensional target space. Conversely, Assumption 1 here relies on the fact that the encoding arm maps into the latent space from a higher-dimensional target space (rendering cross-representability therein reasonable). The distinction is crucial: we certainly do not expect any benefit from autoencoding trivially low-dimensional vectors! Note also that here the representative vectors are taken from Y; to take them from X instead would be unreasonable. For any compressive autoencoder, we generally expect if some subset {b 1 , ..., b M } ⊂ {y 1 , ..., y N } spans Y that {W e b 1 , ..., W e b M } then also span Z in order to be maximally reconstructive. The same cannot be said of subsets {c 1 , ..., c M } ⊂ {x 1 , ..., x N } that span X -for instance, take |X | ≤ |Z|.

In addition to being representative in terms of latent values, the set of representative points also needs to be representative in terms of the reconstruction error. First, let L denote the counterpart to L where the i-th sample (x i , y i ) is replaced by some new instance

(x i , y i ) ∈ X × Y; that is, L = 1 N [ p (ΘW u x i , y i ) + r (ΘW e y i , y i ) + N n=1;n =i ( p (ΘW u x n , y n ) + r (ΘW e y n , y n ))].

Then, let Θ * , Θ * denote the optimal parameters corresponding to the two losses L and L .

Assumption 2 (Representative Errors) Let L r contain the reconstruction errors of the dataset without the i-th sample: L r = (1/N )Σ N n=1;n =i r (ΘW e y n , y n ), and let L B r denote the reconstruction error of the representative subset:

L B r = (1/M )Σ M m=1 r (ΘW e b m , b m ).

Then there exists some a > 0 such that for any small κ > 0 :

L B r (Θ * ) -L B r (κΘ * + (1 -κ)Θ * ) + L B r (Θ * ) -L B r (κΘ * + (1 -κ)Θ * ) ≤ a [L r (Θ * ) -L r (κΘ * + (1 -κ)Θ * )] + a [L r (Θ * ) -L r (κΘ * + (1 -κ)Θ * )].

That is, the difference in reconstruction error L B r between the two points Θ * , Θ * is upper bounded by some constant factor of the corresponding difference in reconstruction error L r at the two points. Importantly, note that this does not require that the values of the errors L r and L B r themselves be similar, only that their differences be similar. This assumption is identical to Assumption 6 in Le et al. (2018) , and plays an identical role: we make use of L B r -which is only dependent on M -to allow the bound to decay with N ; this is in contrast with the generic multi-task analysis of Liu et al. (2016) , which-if applied directly to TEAs (as with FEAs)-would give a bound that does not decay with N .

Theorem 1 (Uniform Stability) Let p and r be σ p -admissible and σ r -admissible loss functions, and let r be c-strongly convex. Then under Assumptions 1 and 2, the following inequality holds,

| p (Θ * W u x, y) -p (Θ * W u x, y)| ≤ 2(σ 2 p r 2 α + σ p σ r r α r β )aM cN Proof. Appendix A.

Corollary 1 (Generalization Bound) Consider the same conditions as in Theorem 1; that is, let p and r be σ p -admissible and σ r -admissible losses, and let r be c-strongly convex. Then under Assumptions 1 and 2, the defect is less than O( (1/N ) log(1/δ)) with probability at least 1δ.

Proof. Follows immediately from Theorem 1 (above) and either of the following (similar results hold for both): Theorem 1.2 (Feldman & Vondrak, 2018) , and Theorem 12 (Bousquet & Elisseeff, 2002) .

In supervised learning, it is often easy to make an argument-on an intuitive level-for the "regularizing" effect of additional loss terms. In contrast, this analysis allows us to unambiguously identify and quantify the benefit of the embedding component as a regularizer (see Remark 4 in Appendix A).

Remark 3 In the linear label space reduction framework of Yu et al. (2014) , uniform convergence is also shown to hold via norm-based regularization. Specifically for uniform stability, a similar bound can also be achieved by adding a strongly convex term to the objective, such as Tikhonov-and 2 -regularization (Bousquet & Elisseeff, 2002; Rosasco & Poggio, 2009; Shalev-Shwartz et al., 2010) . Here, however, the joint reconstruction task leverages a different kind of bias-precisely, the assumption that there exist compact and predictable representations of targets. Therefore the significance of this analysis is that we achieve an equivalent result independent of explicit regularization.

4. RELATED WORK:

Our work straddles three threads of research: (1) supervised representation learning with autoencoders, (2) label space reduction for multi-label classification, and (3) stability-based learning guarantees.

Appendix B provides a much expanded treatment, and presents summary tables for additional context.

Supervised representation learning. While a great deal of research is devoted to uncovering useful underlying structures of data through autoencoders-with various properties such as sparsity (Ranzato et al., 2007) and disentanglement (Chen et al., 2018) , among many others (Tschannen et al., 2018) -the goal of better representations is often for the benefit of downstream tasks. Semi-supervised autoencoders jointly optimized on partially-labeled data can obtain compact representations that improve prediction (Weston et al., 2012; Kingma et al., 2014; Ghifary et al., 2016) . Furthermore, auxiliary reconstruction is also useful in a purely supervised setting: rather than focusing on how specific architectures better structure unlabeled data, Le et al. (2018) show the simple benefit of feature-reconstruction on supervised classification-a special case of what we describe as FEAs.

In contrast, we focus on target-representation learning in the supervised setting, and analyze its benefit under the prior that high-dimensional targets are driven by a compact and predictable set of factors.

We take inspiration from the empirical study of Girdhar et al. (2016) , where latent representations of 3D objects are jointly trained to be predictable from 2D images. Their setup can be viewed as a specific instance of TEAs with (nonlinear) convolutional components, with a minor variation in training: in the joint stage, predictors continue to regress the learned embeddings, and gradients only backpropagate from latent space (instead of target space). Unlike the symmetry of our losses (which we require for our analysis above), their common decoder is only shared indirectly (and predictions made indirectly). As it turns out, this does not appear to matter for performance (see Section 5). In Mostajabi et al. (2018) , a two-stage procedure is used for semantic segmentation-loosely comparable to the first two stages in TEAs; in contrast to our emphasis on joint training, they study the benefit of a frozen embedding branch in parallel with direct prediction. More broadly related to target-embedding, Dalca et al. (2018) build anatomical priors for biomedical segmentation in unsupervised settings.

Multi-label classification. The general idea of target space dimension reduction has been explored for multi-label classification problems (commonly, annotation based on bags of features). These first derive a reduced label space, then subsequently associate inputs to it; methods include compressed sensing (Hsu et al., 2009) , principal components (Tai & Lin, 2010) , maximum-margin coding (Zhang & Schneider, 2012) , and landmarking (Balasubramanian & Lebanon, 2012) & Wagner (1978) and Devroye & Wagner (1979) ; unlike arguments based on the complexity of the search space (Vapnik & Chervonenkis, 1971; Pollard, 1984; Koltchinskii, 2001) , these account for how the algorithm depends on the data. Bousquet & Elisseeff (2002) first formalize the notion of uniform stability sufficient for learnability, and Feldman & Vondrak (2018) use ideas related to differential privacy (Bassily et al., 2016) for further improvement. Separately, while there is a wealth of research on dimensionality reduction and autoencoders (Singh et al., 2009; Mohri et al., 2015; Gottlieb et al., 2016; Epstein & Meir, 2019) , they either operate in the semi-supervised setting, or focus on the benefit of feature representations (not targets) and also do not consider joint learning.

The benefit of jointly learning multiple tasks through a common operator (Caruana, 1997) is explored with VC-based (Baxter, 2000) and Rademacher complexity-based (Maurer, 2006; Maurer et al., 2016) analyses. Recently, Liu et al. (2016) show that the algorithm for learning the shared model in a multi-task setting is uniformly stable. While our argument is based on theirs, we are not interested in a generic bound for all tasks; closer to Le et al. ( 2018), we focus on the primary prediction task, and leverage the auxiliary reconstruction task for stability. Similarly, we arrive at an O(1/N ) on instability without an explicit regularization term as in Bousquet & Elisseeff (2002) . Unlike them, however, the fundamental distinction of our setting is that Y is high-dimensional (but where the underlying factors are assumed compact); in this sense our focus is the mirror opposite of theirs.

5. EXPERIMENTS AND DISCUSSION:

So far, we have formalized a general target-autoencoding framework for supervised learning, and quantified the benefit via uniform stability. Our overall goal in this section is to explore this benefit in a simple controlled setting, such that we can identify and isolate its utility on the prediction task, and investigate any sensitivities of interest. By way of preface, we emphasize two observations from above: (1) In the static, multi-label classification setting, the gain from label-embedding has been studied, including the autoencoder approach of Yeh et al. ( 2017)-which can be viewed as an instantiation of TEAs with sophisticated refinements.

(2) The benefit of target-autoencoding is also The effective input dimension |X | is computed as the dimension of static data plus the product of the width of the historical window (of temporal information) with its dimension; the effective target dimension |Y| is similarly computed as the product of the width of the forecast window (of temporal information) with its dimension. demonstrated using nonlinear, convolutional architectures in Girdhar et al. (2016) -which is also an instantiation of TEAs, also noting significant gains. Therefore a natural question of interest is:

• Does the utility of target-embedding extend to (nonlinear) recurrent models with sequential data for general, high-dimensional targets (i.e. regression and/or classification)?

Disease Trajectories. In this section, we take the first step in answering this question-as our empirical contribution, we extend validation of target-embedding autoencoders to the domain of multivariate sequence forecasting, exploring its utility on linear and nonlinear sequence-to-sequence architectures. What makes a good testbed? In particular, the progression of diseases (and their markers) is high-dimensional in presentation; at the same time, their evolution is often driven by latent biological dynamics (Szczesniak et al., 2017; Pascoal et al., 2017; Alaa & van der Schaar, 2019) .

With the increasing importance of early diagnosis and timely intervention in healthcare, the ability to forecast individual disease trajectories (Y) in the presence of limited windows of information (X ) has become increasingly desirable (Donohue et al., 2014; Pham et al., 2017; Bhagwat et al., 2018) .

Datasets. We use three datasets in our experiments. The first consists of a cohort of patients enrolled in the UK Cystic Fibrosis registry (UKCF), which records follow-up trajectories for over 10,000 patients. We are interested in forecasting future trajectories for the 11 possible infections and 23 possible comorbidities (all binary variables) recorded at each follow-up, using past trajectories and basic demographics as input. The second consists of patients in the Alzheimer's Disease Neuroimaging Initiative study (ADNI), which tracks disease progression for over 1,700 patients. We are interested in forecasting the evolution of the 8 primary biomarkers and 16 cognitive tests (all continuous variables) measured at each visit, using past measurements and basic demographics as input. The third consists of a cohort of patients in intensive care units from the Medical Information Mart for Intensive Care (MIMIC), which records physiological data streams for over 22,000 patients. Likewise, we are interested in forecasting future trajectories for the 361 most frequently measured variables such as vital signs and lab tests (both binary and continuous variables), again using past measurements and basic demographics as input. See Appendix D for more information on datasets.

Experimental Setup. In each instance, the prediction input is a precedent window of up to width w x , and the prediction target is the succedent window of width w y . For UKCF (w x , w y ) = (3, 4) at 1-year resolution, for ADNI (4, 8) at 6-month resolution, and for MIMIC (5, 5) at 4-hour (resampled) resolution. All models are implemented in Tensorflow. Linear models consist of a single hidden layer with no nonlinearity; for the nonlinear case, we implement an RNN model for each component using GRUs. See Appendix D for additional detail on model implementation and configuration.

For evaluation, we measure the mean squared error (MSE) for continuous targets (averaged across variables), and the area under the precision-recall curve (PRC) and area under the receiver operating characteristic curve (ROC) for binary targets (averaged across variables). We use cross-validation on the training set for hyperparameter tuning, selecting the setting that gives the lowest validation loss averaged across folds. For each model and dataset, we report the average and standard error of each performance metric across 10 different experiment runs, each with a different random train-test split.

Note that forecasting high-dimensional disease trajectories is challenging, and input information is deliberately limited (as is often the case in medical practice); the desired targets are similar or higherdimensional than the inputs (see Table 1 ). This obviously results in an inherently difficult prediction problem-but which makes for a good candidate setting to test the utility of target-representation learning. RNN autoencoders have previously been proposed for learning representations of inputs (i.e. FEAs instantiated with RNNs) to improve classification (Dai & Le, 2015) , prediction (Lyu et al., 2018) , generation (Srivastava et al., 2015) , and clustering (Baytas et al., 2017) ; similarly, their mission is not in excessively optimizing specific architectural novelties to match state-of-the-art, but rather in exploring the benefit of the autoencoding framework. Here, we learn representations of targets. The two-sample t-test for difference in means is conducted on the results. An asterisk indicates statistically significant difference in means (p-value < 0.05) relative to the TEA result. For UKCF, only PRC metrics for infections (I) and comorbidities (C) are shown due to space limitation; for ADNI, MSE metrics are reported separately for targets representing biomarkers (B) and cognitive tests (C). See Tables 13-14, 17-18, and 21-22. 

5.1. MAIN RESULTS:

Overall Benefit. First, we examine the overall utility of TEAs. To verify the linear case first, Table 2 summarizes the performance of TEA and alternate setups on UKCF. The temporal axis is flattened to simulate ordinary static multi-label classification, and the base case is direct prediction (Base)-that is, absent any auxiliary representation learning or regularization. Next, we allow for 2 -regularization over direct prediction (REG), as well as over all other methods. FEAs differ only by the added feature-reconstruction, and TEAs only by the target-reconstruction; as an additional sensitivity, we also implement a combined approach (F/TEA). More generally, we also wish to examine the benefit of TEA for the nonlinear case: Table 3 summarizes analogous results where component functions are implemented with GRU networks; results are shown for all datasets. Ceteris paribus, we observe that target-representation learning has a notable positive impact on performance. Interestingly, learning representations of inputs does not yield significant benefit, and the hybrid approach (F/TEA) is worse than TEA; this suggests that forcing intermediate representation to encode both features and targets may be overly constraining. (Note that for the linear model, the instances are restricted to those for which the full input window is available; as a consequence, the results for linear and nonlinear cases are not directly comparable). Figures 4 (Appendix C ) and 5 (Appendix D) give training diagrams for all comparators. Additional experiment results (by model, timestep, metric) are in Appendix E.1-2.

Source of Gain. There are two (related) interpretations of TEAs. First, we studied the regularization view in Section 3; this concerns the benefit of joint training using both prediction and reconstruction losses. Ceteris paribus, we expect performance to improve purely by dint of the jointly trained TEA objective. Second, the reduction view says that TEAs decompose the (difficult) prediction problem into two (smaller) tasks: the autoencoder learns a compact representation z of y, and the predictor learns to map x to z. This makes the potential benefit of staged training (Section 2 and Appendix C) intuitively clear, and suggests an alternative-that of simply training the autoencoder and predictor arms in two stages-à la Mostajabi et al. (2018) . As a general framework, TEAs is a combination of both ideas: all three components are jointly trained in a third stage-à la Girdhar et al. (2016) . We now account for the improvement in performance due to these two sources of benefit; Table 4 does so for the linear case (on UKCF), and Table 5 for the more general nonlinear case (on all datasets). The "No Joint" setting isolates the benefit from staged training only. This is analogous to basic unsupervised pretraining (though using targets), and corresponds to omitting the final joint training stage in Algorithm 1. The "No Staged" setting isolates the benefit from joint training only (without pretraining the autoencoder or predictor), and corresponds to omitting the first two training stages in Algorithm 1. The "Neither" setting is equivalent to vanilla prediction (REG) without leveraging either of the advantages. We observe that while both sources of benefit are individually important, neither setting performs quite as well as when both are combined. See Appendix E.1-2 for extended results. Variations. Having established the utility of target-embedding, we can ask whether variations on the same theme perform similarly. In particular, the embeddings in the empirical studies of Girdhar et al. (2016) and Yeh et al. ( 2017) are jointly learned via the reconstruction loss r and latent loss z -that is, the prediction arm continues to regress learned embeddings during the joint training stage (Figure 4 (d), in Appendix D). The principle is similar, although (as noted in Section 4) the primary task is therefore learned indirectly; this is in contrast to the vanilla TEA setup, where the primary task is learned directly via the prediction loss p . Tables 4 and 5 also compare the performance of vanilla TEAs with this indirect variant (TEA(L)), as well as a hybrid variant (TEA(LP)) for which both latent and prediction losses are trained jointly with the reconstruction loss (Figure 4 (e). Perhaps as expected, we observe that performance across all three variants are more or less identical, affirming the general benefit of target-representation learning. Again, see Appendix E.1-2 for extended results.

5.2. SENSITIVITIES:

Regularization. Of course, target-representation learning is not a replacement for other regularization strategies; it is an additional tool that can be used in parallel where appropriate. Figure 3(a) shows the performance of TEA and REG with various coefficients ν on 2 -regularization. By itself, introducing 2 -regularization does improve performance up to a certain point, beyond which the additional shrinkage bias incurred begins to be counterproductive; this is not surprising. Interestingly, introducing target-representation learning appears to leverage an orthogonal bias: it consistently improves prediction performance regardless of level of shrinkage. This is a practical result of the theoretical observation in Remark 3: while prior works obtain stability through explicit 2 -regularization, the benefit from target-embedding relies on a different bias entirely, which allows us to combine them. While increasing the strength of either form of regularization reduces variability in results (see also below), excessive bias of either alone degrades performance. See Appendix E.3 for full results.

Strength of Prior. Target-embedding attempts to leverage the assumption that there exist compact and predictable representations of targets. Realistically (e.g. due to measurement noise), of course, this will not hold perfectly. In our experiments, we set the ratio of prediction and reconstruction losses to be 1 : 1 for TEA (as well as FEA and F/TEA); that is, the "strength-of-prior" coefficient λ on r is 0.5. In order to isolate the effect of λ during joint training, we observe the performance of TEAs with joint training only (i.e. removing the confounding effect of staged training). For large values of λ, we expect the reconstruction task to dominate in priority, which is (under an imperfect prior) not beneficial for the ultimate prediction task-in general, a hidden representation that is most reconstructive is not necessarily also what is most predictable). For small values of λ, the setup begins to resemble direct prediction. Figure 3 (b) verifies our intuition. Note that in the extreme case of λ = 1, predictions are no better than random (see ROC∼ 0.5). See Appendix E.3 for full results. Here, without the luxury of learning from unpaired data, we highlight the comparative advantage purely from the addition of target-representation learning. Again, see Appendix E.3 for full results.

5.3. DISCUSSION:

By way of conclusion, we emphasize the importance of our central assumption: that there exist compact and predictable representations of the (high-dimensional) targets. This is critical: targetembedding is not useful where this is not true. Now obviously, learning representations of targets is unnecessary if the output dimension is trivially small (e.g. if the target is a single classification label), or if the problem itself is trivially easy (e.g. if direct prediction is already perfect). Also obvious is the situation where representations cannot possibly be compact (e.g. if all output dimensions are independent of each other), in which case any model with a compressive (bottleneck) representation as an intermediate target may make little sense to begin with. Perhaps less obvious is that we cannot assume that the goals of prediction and reconstruction are always aligned. Just as in learning featureembeddings (for downstream classification), what is most reconstructive may not necessarily encode what is most discriminative; so too in learning target-embeddings (for upstream prediction), what is most reconstructive may not necessarily encode what is most predictable. In the case of disease trajectories, it is medical knowledge that permits this assumption with some confidence. Appendix E.4 gives an extreme (synthetic) counterexample where this prior is outright false-i.e. prediction and reconstruction are directly at odds. While certainly contrived, it serves as a caveat about assumptions.

Using the deliberately challenging setting of disease trajectory forecasting with limited information, we have illustrated the nontrivial utility of target-representation learning in a controlled setting with baseline models. While we appreciate that component models in the wild may be more tailored, this setting better allows us to identify and isolate the potential utility of target-autoencoding per se. In addition to verifying our intuitions for the linear case, we have extended empirical validation of target-autoencoding to (nonlinear) sequence-to-sequence recurrent architectures; along the way, we explored the sources of gain from joint and staged training, as well as various sensitivities of interest. Where the prior holds, target-embedding autoencoders are potentially applicable to any high-dimensional prediction task beyond static classification and imaging applications, and exploring its utility for other specific domain-architectures may be a practical direction for future research. A PROOF OF THEOREM 1

Definition 5 (Bregman Distance) Let be some differentiable, strictly convex loss. For any h, h ∈ H, the Bregman distance associated with is given by D

(h h) = (h ) -(h) -h -h, ∇ (h) .

Additivity and non-negativity are easy to see; that is, D (h h) ≥ 0 for any h, h ∈ H, and if = 1 + 2 and both 1 and 2 are convex functions, then

D (h h) = D 1 (h h) + D 2 (h h).

Theorem 1 (Uniform Stability) Let p and r be σ p -admissible and σ r -admissible loss functions, and let r be c-strongly convex. Then under Assumptions 1 and 2, the following inequality holds,

| p (Θ * W u x, y) -p (Θ * W u x, y)| ≤ 2(σ 2 p r 2 α + σ p σ r r α r β )aM cN

Proof. The overall strategy consists of three steps in sequence. First, we want to bound the delta in prediction loss under Θ * and Θ * using the set of representative vectors B. Second, we bound the resulting expression in terms of the Bregman divergence of the complete loss functions L and L under Θ * and Θ * . Third, we express the divergence back in terms of the original expression itself (consisting of representative vectors), which allows us to solve for a bound on that expression. Finally, combining the results from the three steps completes the proof. We begin with the left-hand term,

| p (Θ * W u x, y) -p (Θ * W u x, y)| ≤ σ p (Θ * -Θ * )W u x 2 = σ p M m=1 α m (Θ * -Θ * )W e b m 2 ≤ σ p r α M m=1 (Θ * -Θ * )W e b m 2 2

(3)

where the first inequality follows from the fact that p is σ p -admissible, the equality follows from Assumption 1 for some coefficients α m ∈ R, and the second inequality follows from the Cauchy-Schwarz inequality. As our second step, the goal is to upper-bound the term under the square-root,

EQUATION

where the first inequality follows from the fact that r is c-strongly convex, and the final equality follows from the definition of Bregman divergence (the standalone loss terms cancel). We want an expression in terms of the loss functions L and L , which will subsequently allow us to obtain a bound expressed back in terms of the set of representative vectors. Focusing on the term in the brackets,

EQUATION

where the first and last equalities follows from the definition of Bregman divergence, and the second equality from the definition of directional derivatives. The first inequality follows from Assumption 2; for the second inequality, note that L r consists of a strict subset of the set of strictly convex losses that L consists of, and similarly for L . Therefore by additivity and non-negativity of the Bregman distance, we have that

D L r (Θ * Θ * ) ≤ D L (Θ * Θ * ) and D L r (Θ * Θ * ) ≤ D L (Θ * Θ * ).

Our third and final step is to go back and bound this term again using the set of representative vectors,

EQUATION

where for first equality note that by construction the gradients of the losses L and L are zero at the respective optimal models Θ * and Θ * . The first inequality follows from the fact that p and r are σ p -admissible and σ r -admissible respectively, and the second inequality follows from Assumption 1 and the Cauchy-Schwarz inequality. Now, combining Equations 4, 5, and 6 allows us to write

M m=1 (Θ * -Θ * )W e b m 2 2 ≤ 2(σ p r α + σ r r β )aM cN (7)

which-by substitution into Equation 3-completes the proof.

Remark 4 Formulating the autoencoding component as an auxiliary task allows us to unambiguously interpret its benefit as a regularizer. Specifically, the complete loss can be summarized and rewritten as L(Θ) = L p (Θ) + R 1 (Θ) + R 2 (Θ); that is, the TEA objective is a combination of the primary prediction loss

L p (Θ) = 1 N N n=1 p (ΘW u x n , y n ) plus additional regularization, where R 1 (Θ) = 1 N M m=1 r (ΘW e b m , b m ) = M N L B r (Θ) and R 2 (Θ) = 1 N N n=1 r (ΘW e y n , y n ) -M N L B r (Θ).

In particular, the proof of Theorem 1 relies on R 1 (Θ) to upper-bound instability (Appendix A). This precisely identifies the regularizer in question, while Theorem 1 quantifies its generalization benefit.

Remark 5 Technicality: Moving from uniform stability (Theorem 1) to generalization bounds (Corollary 1) requires that the loss function not take on arbitrarily large values (see e.g. Bousquet & Elisseeff (2002) ). In practice, the label space itself is often bounded (see e.g. Assumption 1 in Le et al. ( 2018)); then the problem effectively reduces to that with a bounded loss function. For example, consider a regression setting using the quadratic loss function, where the target data lie within [-U, U ]; then the loss function is bounded to be within [0, 4U 2 ]. See Castro & Nowak (2018).

Remark 6 Technicality: Earlier we assumed ε = 0 for ease of exposition; carrying around the extra O(ε) term (from Equations 3 and 6) is not particularly illuminating. Generalizing to ε = 0 can be done in a similar manner to Le et al. ( 2018), with the additional assumptions of bounded spaces and that ε decreases as 1/N (which they note is reasonable, since the more samples in the data, the more likely the cross-representativity assumption will hold with low error). Again, note that ε = 0 holds as long as the number of independent latent vectors is at least |Z|). Similarly, Liu et al. ( 2016) consider ε = 0, noting in any case that they can increase N to obtain a small η 2 ; see their analysis for detail.

B EXPANDED RELATED WORK:

In this paper, we motivate and analyze a general autoencoder-based target-representation learning technique in the supervised setting, quantifying the generalization benefit via an argument from uniform stability, as well as verifying its practical utility. As such, our work lies at the intersection of three threads of research: (1) supervised representation learning using autoencoders, (2) label space reduction for multi-label classification, as well as (3) algorithmic stability-based learning guarantees. The goal of better representations is often for the benefit of downstream tasks. Semi-supervised autoencoders-trained on partially-labeled data-can be jointly optimized to obtain compact representations that improve generalization on supervised tasks (Ranzato & Szummer, 2008; Weston et al., 2012) . This naturally extends to representations that are generative (Kingma et al., 2014), disentangled (Narayanaswamy et al., 2017) , or hierarchical (Rasmus et al., 2015) . In addition, semi-supervised autoencoders enable transfer learning across different domains via jointly-trained reconstruction-classification networks Ghifary et al. (2016) , private-shared partitioned representations (Bousmalis et al., 2016) , or by augmenting models with label-encoding layers (Zhuang et al., 2015) .

B.1 Supervised Representation Learning using Autoencoders:

Although less studied, more closely related to our work is the use of autoencoders in a purely supervised setting: rather than focusing on how specific architectural novelties may better structure unlabeled data, Le et al. (2018) instead study the generalization benefit by the simple addition of reconstruction to the supervised classification task-a special case of what we describe as FEAs. Now, all aforementioned studies operate on the basis of autoencoding features (for an explicit or implicit downstream prediction task). In this paper, we instead focus on autoencoder-based target-representation learning (using TEAs) in the supervised setting, and-importantly-analyze the theoretical and empirical benefits of the approach. Unlike in simple classification (for which FEAs make sense), we are motivated by problems with high-dimensional output spaces, but where we operate under the assumption of a more compact and predictable set of underlying factors.

We take inspiration from the empirical investigation of Girdhar et al. (2016) , where latent representations of 3D objects (targets) are jointly trained to be predictable from 2D images (features); 2017) deploy a similar approach for medical image segmentation. On the other hand, in both cases the supervised task is truncated: the predictors are trained to regress the unsupervised embeddings (instead of ground-truth targets), and gradients only backpropagate from the latent space (instead of the target space). This means that their common decoder function is only shared indirectly (and predictions made indirectly), versus the symmetric and simultaneously optimized forward model proposed for TEAs-an important distinction that our analysis relies on to obtain uniform stability. In Mostajabi et al. ( 2018), a two-stage procedure is used for semantic segmentation-loosely comparable to the first two stages in TEAs; in contrast to our emphasis on joint training, they study the benefit of a frozen embedding branch in parallel with direct prediction. More broadly related to target-embedding, Dalca et al. (2018) build anatomical priors for biomedical segmentation in unsupervised settings.

B.2 Label Space Reduction for Multi-Label Classification:

Label space dimension reduction comprises techniques that focus specifically on multi-label classification. Early approaches to multi-label classification employ simplistic transformations such as label power-sets (Boutell et al., 2004) , binary relevance (Tsoumakas et al., 2009) , and label rankings (Fürnkranz et al., 2008) ; these are computationally inefficient, and do not capture interdependencies between labels. In contrast, label-embedding methods first derive a latent label space with reduced dimensionality, and subsequently associate inputs to that latent space instead. Encodings have been obtained via random projections (Hsu et al., 2009) , principal components-based projections (Tai & Lin, 2010) , canonical correlation analysis (Zhang & Schneider, 2011) , as well as maximum-margin coding (Zhang & Schneider, 2012) . A parallel thread of research has focused on selecting representative and reconstructive subsets of labels through group-sparse learning (Balasubramanian & Lebanon, 2012) and randomized sampling (Bi & Kwok, 2013) . Various extensions of label-embedding techniques abound, such as using bloom filters (Cisse et al., 2013) , nearest-neighbors (Bhatia et al., 2015) , handling missing data (Wu et al., 2014) , as well as using binary compression (Zhou et al., 2017) .

Closer our theme of joint learning by utilizing both features and targets, Chen & Lin (2012) first proposed simultaneously minimizing the encoding error (from labels) and prediction error (from features) through an SVD formulation. et al., 2014; Li & Guo, 2015) .

Our work is related to this general autoencoder approach to label-embedding, although there are significant differences in focus. In particular, we operate at a higher level of abstraction. Labelembedding techniques worry about label reduction, and about specific loss functions that aim to preserve dependencies within and among spaces; their problem is one of multi-label classification, and their baseline is binary relevance. In contrast, we worry about autoencoding at all-that is, we focus on the regularizing effect of the reconstruction loss on learning the prediction model; our baseline is direct prediction, and the output can be of any form (classification or regression). In light of the sizable performance improvement of the autoencoder-based model of Yeh et al. ( 2017) over comparators using direct prediction, our work can be regarded as a more generalized analysis of the contribution of the autoencoding component. Moreover, unlike the uniform convergence-based analysis in Yu et al. ( 2014), our bound does not rely on explicit norm-based regularization-instead, we interpret the embedding task itself as an intrinsic form of regularization to derive our stability-based guarantee.

Finally, also worth mentioning is the field of extreme multi-label classification (Bhatia et al., 2015) , for which probabilistic methods such as Rai et al. (2015) and Kapoor et al. (2012) present sophisticated approaches to extremely high-dimensional classification problems-with advantages in performance and use cases. In light of the medical relevance of our experimental setting, we point out the application of Yan et al. ( 2010) to medical coding. See Bhatia et al. (2019) for a more detailed overview.

B.3 Algorithmic Stability and Learning Guarantees:

Generalizability is central to machine learning, and its analysis via hypothesis stability is first studied in Rogers & Wagner (1978) and Devroye & Wagner (1979) . Unlike arguments based on the complexity of the search space (Vapnik & Chervonenkis, 1971; Pollard, 1984; Koltchinskii, 2001) , stability-based approaches account for how the model produced by the algorithm depends on the data. Based on concentration inequalities (McDiarmid, 1989) , improved bounds are developed in Lugosi & Pawlak (1994) by estimating posterior error probabilities. The landmark work of Bousquet & Elisseeff (2002) first formalizes the notion of uniform stability sufficient for learnability, obtaining relatively strong bounds for several regularization algorithms, and Feldman & Vondrak (2018) recently use ideas related to differential privacy (Bassily et al., 2016) for further improved bounds without additional assumptions. For further context, see Mukherjee et al. (2006) and Shalev-Shwartz et al. (2010) .

For semi-supervised representation learning, Rigollet (2007) first introduces the notion of cluster excess-risk and convergence, formalizing the clustering criterion for unlabeled features to be useful (Seeger, 2000) . Based on the clustering assumption, Singh et al. (2009) develops a finite sample analysis to quantify the performance improvement from unlabeled features. Focusing on autoencoders, Epstein & Meir (2019) adapt recent margin, norm, and compression-based results for deep networks (Bartlett et al., 2017; Neyshabur et al., 2017; Arora et al., 2018) , and relate generalization of feature reconstructions to the benefit of additional unlabeled features for the primary classification task.

In the context of supervised problems, Mohri et al. (2015) and Gottlieb et al. (2016) analyze the generalization properties of dimensionality reduction techniques for features with respect to a downstream task; however, rather than joint training, the primary task is optimized subsequently over the learned representations. Taking a joint, multi-task approach (Caruana, 1997) , Baxter (2000) first leverages the inductive bias of a common optimal hypothesis class to obtain a VC-based generalization bound. Maurer (2006) and Maurer et al. (2016) argue from Rademacher complexity to illustrate the benefit of the common operator; however, they only consider the task-averaged benefit, whereas we want to focus specifically on the primary task. There has been some work on generalization for each task (Ben-David & Schuller, 2003) , but limited to binary classification-contrary to our setting.

Arguing from stability, our approach is related to Liu et al. (2016) in showing that the algorithm for learning the shared model in a multi-task setting is uniformly stable. Our analysis also resembles Le et al. (2018) in the more specific setting where the bound for the primary prediction task is obtained with assistance from the auxiliary reconstruction loss; unlike Liu et al. ( 2016), we are not interested in a generic bound for all tasks. Again, however, the fundamental (and motivating) difference of our work stems from the (inverted) problem setting and resulting framework. In the vast majority of works, the primary task is one of classification (or more generally |X | |Y|), where featureembeddings make sense to learn. Instead, we attend to the setting in which Y is high-dimensional (but where the underlying factors are assumed to be compact). In this setting, we argue (theoretically and empirically) that target-embeddings make more sense to learn in an auxiliary reconstruction task.

C EXPANDED ALGORITHM DETAIL:

In the following, Algorithm 1 gives pseudocode for TEA training. 

∼ D:

4:

for n ∈ {1, ..., N s } do 5:

z n ← e(y n ; W e ) Encode 6:

ỹn ← θ(z n ; Θ) Decode 7:

end for

8: L r ← 1 Ns Ns n=1 r (ỹ n , y n ) 9:

W e ← W e -ψ∇ We L r 10: 

Θ ← Θ -ψ∇ Θ L

L p ← 1 Ns Ns n=1 p (ŷ n , x n ) 30: L r ← 1 Ns Ns n=1 r (ỹ n , y n ) 31: W u ← W u -ψ∇ Wu L p 32:

W e ← W e -ψ∇ We L r 33: clinical visits. We focus on the 8 primary quantitative biomarkers (e.g. entorhinal cortex, fusiform gyrus, hippocampus), 16 cognitive tests (e.g. ADAS11, CDR sum of boxes, mini mental state exam), as well as static demographic information (e.g. apolipoprotein E4, education level, ethnicity); we omit the remaining variables, for which the rate of missingness is over 50%. Using a precedent window, we forecast the future the evolution of the primary quantitative biomarkers and cognitive test results (all continuous variables) measured at each visit. The Medical Information Mart for Intensive Care records physiological data streams patients admitted to intensive care units after 2008. We use over 22,000 patients with a total of over 500,000 measurements (resampled at 4 hour intervals). We focus on the most frequently measured vital signs and lab tests (e.g. heart rate, oxygen saturation, respiratory rate) recorded over time (with categorical variables binarized, this gives a total of 361 variables), as well as static demographic information (e.g. admission type, gender, location, marital status); we omit the remaining variables, for which the rate of missingness is over 50%. Using a precedent window, we forecast the subsequent window of those variables. For each dataset, sequences are randomized at the patient level in order to obtain splits for training (and validation) and testing.

Θ ← Θ -ψ∇ Θ [L p + L r ] 34: end while 35: return Θ, W u , W e

We implement all models using Tensorflow. For the linear model, each component (encoder, decoder, and predictor) consists of a single layer, no bias term, and linear activation; static (demographic) and temporal data are concatenated and flattened for both features and targets. For the nonlinear case, we implement each component as an RNN using GRUs with the number of hidden layers ζ ∈ {1, 2} where the number of hidden units is equal to the temporal feature dimension, and tanh is used for activation; the dimension of the latent space is (therefore) equal to the hidden state dimension. (For even larger hidden capacities, the increased number of parameters rapidly degrades performance). Static (demographic) information is incorporated as a mapping into the initial state for recurrent cells. Training is performed using the ADAM optimizer with a learning rate of ψ ∈ {3e-5, 3e-4, 3e-3, 3e-2}. Models are trained until convergence up to a maximum of 10,000 iterations with a minibatch size of N s ∈ {32, 64, 128}; the empirical loss is computed on the validation set every 50 iterations of training, and convergence is determined on the basis of that error. Checkpointing is implemented every 50 iterations, and the best model parameters are restored (upon convergence) for use on the testing set. For all models except "Base", we allow the opportunity to select among the 2 -regularization coefficients ν ∈ {0, 3e-5, 3e-4, 3e-3, 3e-2}. We set the strength-of-prior coefficient λ = 0.5 for FEA, F/TEA, as well as all variants of TEA (however, we do provide sensitivities on λ for TEA in our experiments). For hyperparameter tuning (ζ, ψ, ν, N s ), we use cross-validation on the training set using 20 iterations of random search, selecting the setting that gives the lowest validation loss averaged across folds. For fair comparison (so as to isolate the effect of supervised representation learning over and above direct prediction), we apply the same setting chosen for REG for FEA, F/TEA, and all variants of TEA; therefore the only difference is the presence or absence of each autoencoding component. For each model and dataset, the experiment is repeated for a total of 10 times (each with a different random split of data into training and held-out testing sets); all results are reported as means and standard errors of each performance metric across runs. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).

E ADDITIONAL EXPERIMENT RESULTS:



E.1 Results for Linear Models:



E.2 Results for Recurrent Models:

Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).

Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining). The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).

Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. The ν coefficient controls the strength of 2-regularization applied on top of the original loss function minimized. PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).

E.3 Results for Sensitivities:

Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. The ν coefficient controls the strength of 2-regularization applied on top of the original loss function minimized. PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).

Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. The λ coefficient controls the strength of prior-i.e. the tradeoff between the prediction and reconstruction loss. PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C). The λ coefficient controls the strength of prior-i.e. the tradeoff between the prediction and reconstruction loss. PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).

Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping. 

E.5 Results from Open Discussion:

One can also ask the (purely empirical) question of how much each model degrades on out-ofdistribution data-without additional training to fine-tune the model to the new data. In this context, we actually have no reason to expect TEAs to degrade any more or less than comparators. For thoroughness, we show an additional experiment as an example for sensitivity analysis (using UKCF) as follows: Each model is trained (only) on male patients and tested (only) on female patients, and vice versa. The average results on held-out samples from in-distribution data and out-of-distribution data then allow us to compute the net degradation (i.e. negative difference), which is reported below. While TEAs individually perform better overall on both in-distribution and out-of-distribution samples, none of the differences in the amounts of degradation between models are statistically significant: Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.

Finally, given the staged training in Algorithm 1, it should be clear that the order cannot be changed.

Stage 2 requires the encoder to already be trained to provide the requisite embeddings, so it must be preceded by Stage 1. Therefore the only relevant possibilities are: (1) Stages 1-2 by themselves, without Stage 3; this is simply the "No Joint" setting. (2) Stage 3 by itself, without Stages 1-2; this is simply the "No Staged" setting. (3) None of the stages altogether; this is simply the "Neither" setting.

(4) Stages 1, 2, and 3 in order; this is simply Algorithm 1 itself. The only remaining possibility is to have Stage 3 precede Stages 1-2. This makes little sense, since when the reconstruction loss is trained by itself it is likely to "undo" the result of joint training. For thoroughness, we run an additional sensitivity experiment (using UKCF) to confirm this. The following corresponds to the left half of Table 4 , with the additional column on the right (and the other columns labeled to reflect the training stages). Verifying our intuitions, the setting "3-1-2" behaves almost identically to the setting "1-2": Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.

:



Figures and Tables:

Figure 1: (a) Feature-embedding and (b) Target-embedding autoencoders. Solid lines correspond to the (primary) prediction task; dashed lines to the (auxiliary) reconstruction task. Shared components are involved in both.

Figure 2: Functions and objectives in (a) FEAs and (b) TEAs. Blue and red identify supervised and representation learning components. FEAs are parameterized by (Φ, W d , Wr) of (φ, d, r), and TEAs by (Θ, Wu, We) of (θ, u, e). Solid lines indicate forward propagation of data; dashed lines indicate backpropagation of gradients.

Consider a linear TEA, where the upstream predictor is parameterized by W u ∈ R |Z|×|X | , targetembedding by W e ∈ R |Z|×|Y| , and shared forward model by Θ ∈ R |Y|×|Z| , where |Z| < |Y|. The complete loss function is given by L

Remark 2 A comparison with Assumption 4 in Le et al. (2018) sheds additional light on why we expect TEAs to be beneficial where |Y| > |Z|, in contrast with the (more typical) scenario |X | > |Z| |Y|. Critically, the technique in Le et al. (

Figure 3: Sensitivities on 2-regularization coefficient ν, strength-of-prior coefficient λ, and training size N for direct prediction (REG) and with target-embedding (TEA) on linear model with UKCF. For sensitivities on λ, we perform joint training only, so that we isolate the effect of the joint reconstruction task (i.e. removing the confounding effect of staged training). Standard errors are indicated with shaded regions. For full results, see Tables 25-28 for sensitivities on ν, Tables 29-30 for sensitivities on λ, and Tables 31-34 for sensitivites on N .Sample Complexity. Figure3(c) shows the performance of TEA and REG under various levels of data scarcity. The benefit conferred by TEAs is significant especially when the amount of training data N is limited. Importantly, note that we are operating strictly within the context of supervised learning: unlike in semi-supervised settings, here we are not just restricting access to paired data; we are restricting access to data per se. (Though beyond the scope of this paper, we expect that extending TEAs to semi-supervised learning with additional unpaired data would yield larger gains). Here, without the luxury of learning from unpaired data, we highlight the comparative advantage purely from the addition of target-representation learning. Again, see Appendix E.3 for full results.

Figure 4 gives detailed block diagrams of component functions and objectives corresponding to each training stage (and variant). Algorithm 1 Pseudocode for TEA Training Input: D = {(x n , y n )} N n=1 , learning rate ψ, minibatch size N s Output: Parameters Θ, W u , W e of components θ, u, e 1: Initialize: Θ, W u , W e 2: while not converged do Stage 1: Learn Target-Embedding 3: Sample {(x n , y n )} Ns n=1 i.i.d.

Figure 5: (continued) model φ, downstream predictor d, and feature reconstructor r, parameterized by (Φ, W d , W r ). (c) TEAs consist of a shared forward model θ, upstream predictor u, and targetembedding function e, parameterized by (Θ, W u , W e ). (d) As an additional sensitivity, F/TEAs combine the previous two, forcing intermediate representations to encode both features and targets.

Dataset statistics and input/output dimensions used in experiments

Table content: <table><tr><td colspan="2">Num. Samp. Target</td><td colspan="7">Static Temp. dim. Temp. dim. Window Window Effective Effective</td></tr><tr><td>Dataset patients freq.</td><td>type</td><td colspan="5">dim. (history) (forecast) (history) (forecast)</td><td>|X |</td><td>|Y|</td></tr><tr><td>UKCF 10,000 1 yr.</td><td>Binary</td><td>11</td><td>43</td><td>34</td><td>3</td><td>4</td><td>140</td><td>136</td></tr><tr><td colspan="3">ADNI 1,700 6 m. Continuous 11</td><td>26</td><td>24</td><td>4</td><td>8</td><td>115</td><td>192</td></tr><tr><td>MIMIC 22,000 4 hr.</td><td>Mixed</td><td>26</td><td>361</td><td>361</td><td>5</td><td>5</td><td>1,831</td><td>1,805</td></tr></table>

Summary results for TEA and comparators on linear model with UKCF (Bold indicates best)The two-sample t-test for difference in means is conducted on the results. An asterisk indicates statistically significant difference in means (p-value < 0.05) relative to the TEA result. PRC and ROC metrics are reported separately for variables representing infections (I) and comorbidities (C). See Tables 9-10 for extended results.

Table content: <table><tr><td/><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC(I)</td><td>0.322 ± 0.099*</td><td>0.347 ± 0.085*</td><td>0.351 ± 0.079*</td><td>0.450 ± 0.035</td><td>0.414 ± 0.028*</td></tr><tr><td>PRC(C)</td><td>0.416 ± 0.100*</td><td>0.433 ± 0.083*</td><td>0.455 ± 0.087*</td><td>0.559 ± 0.060</td><td>0.520 ± 0.052</td></tr><tr><td>ROC(I)</td><td>0.689 ± 0.089*</td><td>0.710 ± 0.072*</td><td>0.720 ± 0.073</td><td>0.767 ± 0.026</td><td>0.766 ± 0.023</td></tr><tr><td>ROC(C)</td><td>0.679 ± 0.091*</td><td>0.700 ± 0.075*</td><td>0.713 ± 0.075</td><td>0.767 ± 0.042</td><td>0.755 ± 0.037</td></tr></table>

Summary results for TEA and comparators on nonlinear (RNN) model (Bold indicates best) ±0.072 0.063±0.010 0.330 ±0.066 0.239±0.039 0.150 ±0.012 F/TEA 0.457±0.037 0.576 ±0.071 0.073±0.010* 0.338 ±0.067 0.166±0.023* 0.154 ±0.011

Table content: <table><tr><td/><td colspan="2">UKCF (Binary Targets)</td><td colspan="2">ADNI (Continuous Targets)</td><td colspan="2">MIMIC (Mixed Targets)</td></tr><tr><td/><td>PRC(I)</td><td>PRC(C)</td><td>MSE(B)</td><td>MSE(C)</td><td>PRC</td><td>MSE</td></tr><tr><td>Base</td><td colspan="6">0.411±0.035* 0.497 ±0.057* 0.105±0.018* 0.361 ±0.064 0.142±0.028* 0.153 ±0.011</td></tr><tr><td>REG</td><td colspan="6">0.415±0.030* 0.518 ±0.052* 0.096±0.014* 0.360 ±0.066 0.143±0.019* 0.152 ±0.010</td></tr><tr><td>FEA</td><td colspan="6">0.410±0.033* 0.521 ±0.054* 0.092±0.012* 0.356 ±0.068 0.144±0.030* 0.152 ±0.012</td></tr><tr><td>TEA</td><td colspan="2">0.483±0.045 0.583</td><td/><td/><td/></tr></table>

Summary source of gain and TEA variants on linear model with UKCF (Bold indicates best) PRC(I) 0.347 ± 0.085 0.402 ± 0.026 0.431 ± 0.031 0.450 ± 0.035 0.435 ± 0.031 0.454 ± 0.036 PRC(C) 0.433 ± 0.083 0.507 ± 0.040 0.543 ± 0.054 0.559 ± 0.060 0.544 ± 0.053 0.560 ± 0.061 ROC(I) 0.710 ± 0.072 0.747 ± 0.022 0.764 ± 0.022 0.767 ± 0.026 0.759 ± 0.025 0.768 ± 0.028 ROC(C) 0.700 ± 0.075 0.744 ± 0.038 0.766 ± 0.038 0.767 ± 0.042 0.760 ± 0.042 0.767 ± 0.042 "No Joint" omits final joint training, and "No Staged" skips (pre)-training stages. PRC and ROC metrics are reported separately for targets representing infections (I) & comorbidities (C). See Tables 11-12 for extended results.

Table content: <table><tr><td>Neither</td><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr></table>

Summary source of gain and TEA variants on nonlinear (RNN) model (Bold indicates best)

Table content: <table><tr><td/><td colspan="2">UKCF (Binary Targets)</td><td colspan="2">ADNI (Continuous Targets)</td><td colspan="2">MIMIC (Mixed Targets)</td></tr><tr><td/><td>PRC(I)</td><td>PRC(C)</td><td>MSE(B)</td><td>MSE(C)</td><td>PRC</td><td>MSE</td></tr><tr><td>Neither</td><td colspan="6">0.415±0.030 0.518±0.052 0.096±0.014 0.360 ±0.066 0.143±0.019 0.152 ±0.010</td></tr><tr><td>No Joint</td><td colspan="6">0.455±0.039 0.574±0.069 0.092±0.014 0.353 ±0.070 0.183±0.038 0.151 ±0.011</td></tr><tr><td colspan="7">No Staged 0.424±0.031 0.543±0.061 0.106±0.022 0.363 ±0.067 0.167±0.022 0.150 ±0.012</td></tr><tr><td>TEA</td><td colspan="6">0.483±0.045 0.583±0.072 0.063±0.010 0.330 ±0.066 0.239±0.039 0.150 ±0.012</td></tr><tr><td>TEA(L)</td><td colspan="6">0.483±0.047 0.581±0.074 0.058±0.012 0.330±0.076 0.249±0.049 0.149 ±0.012</td></tr><tr><td>TEA(LP)</td><td colspan="6">0.480±0.044 0.583±0.072 0.064±0.012 0.329±0.068 0.229±0.039 0.151 ±0.011</td></tr></table>

Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. IEEE Transactions on Information Theory, 47(5):1902-1914, 2001. Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. In Advances in Neural Information Processing Systems, pp. 107-117, 2018. Xin Li and Yuhong Guo. Multi-label classification with feature-aware non-linear label space transformation. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015. Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang. Multi-label classification via featureaware implicit label space encoding. In International conference on machine learning, 2014. Tongliang Liu, Dacheng Tao, Mingli Song, and Stephen J Maybank. Algorithm-dependent generalization bounds for multi-task learning. IEEE transactions on pattern analysis and machine intelligence, 39(2):227-241, 2016. Gábor Lugosi and Miroslaw Pawlak. On the posterior-probability estimate of the error rate of nonparametric classification rules. IEEE Trans. on Information Theory, 40(2):475-481, 1994. Xinrui Lyu, Matthias Hüser, Stephanie L Hyland, George Zerveas, and Gunnar Rätsch. Improving clinical predictions through unsupervised time series representation learning. NeurIPS 2018 Machine Learning for Health (ML4H) workshop, 2018. Mohammadreza Mostajabi, Michael Maire, and Gregory Shakhnarovich. Regularizing deep networks by modeling and predicting label structure. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5629-5638, 2018. Sayan Mukherjee, Partha Niyogi, Tomaso Poggio, and Ryan Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25(1-3):161-193, 2006. Vladimir Vapnik and Alexey Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theoretical Probabibility and its Applicactions, 17:264-280, 01 1971. doi: 10.1137/1116025. Kaixiang Wang, Ming Yang, Wanqi Yang, and YiLong Yin. Deep correlation structure preserved label space embedding for multi-label classification. In Asian Conference on Machine Learning, pp. 1-16, 2018. Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semisupervised embedding. In Neural Networks: Tricks of the Trade, pp. 639-655. Springer, 2012. Baoyuan Wu, Zhilei Liu, Shangfei Wang, Bao-Gang Hu, and Qiang Ji. Multi-label learning with missing labels. In 22nd International Conference on Pattern Recognition, pp. 1964-1968. IEEE, 2014. Yan Yan, Glenn Fung, Jennifer G Dy, and Romer Rosales. Medical coding classification by leveraging inter-code relationships. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 193-202. ACM, 2010. Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In Thirty-First AAAI Conference on Artificial Intelligence, 2017. Hsiang-Fu Yu, Prateek Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In International conference on machine learning, pp. 593-601, 2014.

Table content: <table><tr><td>Siddharth Narayanaswamy, T Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah</td></tr><tr><td>Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations</td></tr><tr><td>with semi-supervised deep generative models. In Advances in Neural Information Processing</td></tr><tr><td>Systems, pp. 5925-5935, 2017.</td></tr><tr><td>Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-</td></tr><tr><td>normalized margin bounds for neural networks. International Conference on Learning Representa-</td></tr><tr><td>tions 2018, 2017.</td></tr></table>

Autoencoder-Based Supervised Representation Learning

Table content: <table><tr><td>Work</td><td>Contribution</td><td>Setting</td><td>Type</td><td>Embedding</td></tr><tr><td>Weston et al. (2012)</td><td>Jointly optimized classific-ation and embedding</td><td>Semi-supervised learning</td><td>Deterministic</td><td>Features</td></tr><tr><td>Kingma et al. (2014)</td><td>Variational inference for generative modeling</td><td>Semi-supervised learning</td><td>Probabilistic</td><td>Features</td></tr><tr><td>Narayanaswamy et al. (2017)</td><td>Disentangled latent representations</td><td>Semi-supervised learning</td><td>Probabilistic</td><td>Features</td></tr><tr><td>Zhuang et al. (2015)</td><td>Input-and output-encoding for transfer learning</td><td>Semi-supervised learning</td><td>Deterministic</td><td>Features</td></tr><tr><td>Bousmalis et al. (2016)</td><td>Paired autoencoders for transfer learning</td><td>Semi-supervised learning</td><td>Deterministic</td><td>Features</td></tr><tr><td>Ghifary et al. (2016)</td><td>Jointly optimized feature-embedding</td><td>Semi-supervised learning</td><td>Deterministic</td><td>Features</td></tr><tr><td>Le et al. (2018)</td><td>Jointly optimized feature-embedding</td><td>Supervised learning</td><td>Deterministic</td><td>Features</td></tr><tr><td>Dalca et al. (2018)</td><td>Generative model using learned target priors</td><td>Unsupervised, Unpaired</td><td>Probabilistic</td><td>Targets</td></tr><tr><td>Girdhar et al. (2016)</td><td>Jointly optimized target-embedding (Indirect)</td><td>Supervised learning</td><td>Deterministic</td><td>Targets</td></tr><tr><td>(Ours)</td><td>Jointly optimized target-embedding (Direct)</td><td>Supervised learning</td><td>Deterministic</td><td>Targets</td></tr></table>

Label Space Dimension Reduction via Label Embedding

Table content: <table><tr><td>Work</td><td>Contribution</td><td>Learning</td><td>Problem</td></tr><tr><td>Hsu et al. (2009)</td><td>Coding with compressed sensing (random projections)</td><td>Separate</td><td>Multi-label classification</td></tr><tr><td>Tai &amp; Lin (2010)</td><td>Coding with principal label space transformation (PC-based projections)</td><td>Separate</td><td>Multi-label classification</td></tr><tr><td>Zhang &amp; Schneider (2011)</td><td>Coding with maximum margin (between prediction distances)</td><td>Separate</td><td>Multi-label classification</td></tr><tr><td>Balasubramanian &amp; Lebanon (2012)</td><td>Landmark selection of labels via group-sparse learning</td><td>Separate</td><td>Multi-label classification</td></tr><tr><td>Bi &amp; Kwok (2013)</td><td>Landmark selection of labels via randomized sampling</td><td>Separate</td><td>Multi-label classification</td></tr><tr><td>Chen &amp; Lin (2012)</td><td>Feature-aware principal label space transformation for embedding</td><td>Joint</td><td>Multi-label classification</td></tr><tr><td>Yu et al. (2014)</td><td>Generic empirical risk minimization formulation and bounds</td><td>Joint</td><td>Multi-label classification</td></tr><tr><td>Yeh et al. (2017)</td><td>Autoencoder embedding with canonical correlation analysis</td><td>Joint</td><td>Multi-label classification</td></tr><tr><td>Mostajabi et al. (2018)</td><td>Autoencoder component as (Implicit) regularization for learning predictor</td><td>Separate</td><td>General; Semantic segmentation</td></tr><tr><td>(Ours)</td><td>Autoencoder component as (Explicit) regularization for learning predictor</td><td>Joint</td><td>General; Sequence forecasting</td></tr><tr><td>Oktay et al. (</td><td/><td/><td/></tr></table>

Generalizability, Multiple Tasks, and Algorithmic Stability

Table content: <table><tr><td>Work</td><td>Contribution</td><td>Setting</td><td>Focus</td><td>Learning</td></tr><tr><td>Bousquet &amp; Elisseeff (2002)</td><td>Uniform stability for generalization</td><td>Supervised, general</td><td>Single task</td><td>-</td></tr><tr><td>Feldman &amp; Vondrak (2018)</td><td>Improve on Bousquet &amp; Elisseeff's bound</td><td>Supervised, general</td><td>Single task</td><td>-</td></tr><tr><td>Baxter (2000)</td><td>VC-dimension for generalization</td><td>Multi-task learning</td><td>All tasks</td><td>Jointly</td></tr><tr><td>Maurer (2006)</td><td>Rademacher complexity for generalization</td><td>Multi-task learning</td><td>All tasks</td><td>Jointly</td></tr><tr><td>Liu et al. (2016)</td><td>Uniform stability for generalization</td><td>Multi-task learning</td><td>All tasks</td><td>Jointly</td></tr><tr><td>Mohri et al. (2015)</td><td>Rademacher complexity for generalization</td><td>Feature-embedding</td><td>Primary task</td><td>Separately</td></tr><tr><td>Epstein &amp; Meir (2019)</td><td>Non-contractiveness and semi-supervision</td><td>Feature-embedding</td><td>Primary task</td><td>Separately</td></tr><tr><td>Le et al. (2018)</td><td>Uniform stability for generalization</td><td>Feature-embedding</td><td>Primary task</td><td>Jointly</td></tr><tr><td>(Ours)</td><td>Uniform stability for generalization</td><td>Target-embedding</td><td>Primary task</td><td>Jointly</td></tr></table>

Extended results for TEA and comparators on linear model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>τ</td><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC(I)</td><td colspan="5">1 0.340 ± 0.118* 0.370 ± 0.101* 0.374 ± 0.091* 0.497 ± 0.016</td><td>0.454 ± 0.010*</td></tr><tr><td/><td colspan="5">2 0.325 ± 0.099* 0.350 ± 0.085* 0.353 ± 0.077* 0.459 ± 0.017</td><td>0.419 ± 0.012*</td></tr><tr><td/><td colspan="5">3 0.314 ± 0.088* 0.337 ± 0.076* 0.342 ± 0.071* 0.432 ± 0.015</td><td>0.399 ± 0.013*</td></tr><tr><td/><td colspan="5">4 0.307 ± 0.082* 0.328 ± 0.071* 0.333 ± 0.067* 0.413 ± 0.010</td><td>0.386 ± 0.009*</td></tr><tr><td>PRC(C)</td><td colspan="5">1 0.445 ± 0.131* 0.467 ± 0.106* 0.499 ± 0.110* 0.653 ± 0.009</td><td>0.599 ± 0.019*</td></tr><tr><td/><td colspan="5">2 0.418 ± 0.099* 0.435 ± 0.081* 0.457 ± 0.083* 0.566 ± 0.010</td><td>0.524 ± 0.017*</td></tr><tr><td/><td colspan="5">3 0.403 ± 0.082* 0.418 ± 0.067* 0.436 ± 0.069* 0.521 ± 0.008</td><td>0.487 ± 0.015*</td></tr><tr><td/><td colspan="5">4 0.398 ± 0.073* 0.412 ± 0.060* 0.426 ± 0.061* 0.498 ± 0.009</td><td>0.471 ± 0.013*</td></tr><tr><td>ROC(I)</td><td colspan="4">1 0.713 ± 0.104* 0.737 ± 0.081* 0.750 ± 0.078</td><td>0.806 ± 0.007</td><td>0.801 ± 0.007</td></tr><tr><td/><td colspan="5">2 0.688 ± 0.089* 0.709 ± 0.070* 0.718 ± 0.068* 0.771 ± 0.007</td><td>0.765 ± 0.009</td></tr><tr><td/><td colspan="4">3 0.677 ± 0.080* 0.697 ± 0.065* 0.706 ± 0.066</td><td>0.750 ± 0.007</td><td>0.749 ± 0.009</td></tr><tr><td/><td colspan="3">4 0.677 ± 0.076* 0.696 ± 0.063</td><td>0.707 ± 0.068</td><td>0.740 ± 0.008</td><td>0.748 ± 0.006*</td></tr><tr><td colspan="6">ROC(C) 1 0.713 ± 0.110* 0.741 ± 0.088* 0.756 ± 0.084* 0.829 ± 0.006</td><td>0.810 ± 0.008*</td></tr><tr><td/><td colspan="5">2 0.686 ± 0.091* 0.707 ± 0.072* 0.719 ± 0.071* 0.775 ± 0.008</td><td>0.762 ± 0.008*</td></tr><tr><td/><td colspan="4">3 0.668 ± 0.077* 0.686 ± 0.061* 0.699 ± 0.063</td><td>0.743 ± 0.005</td><td>0.735 ± 0.007*</td></tr><tr><td/><td colspan="5">4 0.651 ± 0.070* 0.667 ± 0.056* 0.679 ± 0.057* 0.720 ± 0.006</td><td>0.712 ± 0.009*</td></tr></table>

Summary results for TEA and comparators on linear model with UKCF (Bold indicates best) evaluations are reported separately for targets representing infections (I) and comorbidities (C). The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td/><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC(I)</td><td>0.322 ± 0.099*</td><td>0.347 ± 0.085*</td><td>0.351 ± 0.079*</td><td>0.450 ± 0.035</td><td>0.414 ± 0.028*</td></tr><tr><td>PRC(C)</td><td>0.416 ± 0.100*</td><td>0.433 ± 0.083*</td><td>0.455 ± 0.087*</td><td>0.559 ± 0.060</td><td>0.520 ± 0.052</td></tr><tr><td>ROC(I)</td><td>0.689 ± 0.089*</td><td>0.710 ± 0.072*</td><td>0.720 ± 0.073</td><td>0.767 ± 0.026</td><td>0.766 ± 0.023</td></tr><tr><td>ROC(C)</td><td>0.679 ± 0.091*</td><td>0.700 ± 0.075*</td><td>0.713 ± 0.075</td><td>0.767 ± 0.042</td><td>0.755 ± 0.037</td></tr><tr><td>PRC and ROC</td><td/><td/><td/><td/><td/></tr></table>

Extended source of gain and variants on linear model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>τ</td><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>PRC(I)</td><td colspan="2">1 0.434 ± 0.017</td><td>0.472 ± 0.016</td><td>0.497 ± 0.016</td><td>0.474 ± 0.018</td><td>0.502 ± 0.015</td></tr><tr><td/><td colspan="2">2 0.408 ± 0.016</td><td>0.439 ± 0.016</td><td>0.459 ± 0.017</td><td>0.443 ± 0.018</td><td>0.463 ± 0.016</td></tr><tr><td/><td colspan="2">3 0.390 ± 0.015</td><td>0.415 ± 0.015</td><td>0.432 ± 0.015</td><td>0.420 ± 0.014</td><td>0.435 ± 0.013</td></tr><tr><td/><td colspan="2">4 0.377 ± 0.014</td><td>0.399 ± 0.011</td><td>0.413 ± 0.010</td><td>0.402 ± 0.010</td><td>0.415 ± 0.009</td></tr><tr><td>PRC(C)</td><td colspan="2">1 0.563 ± 0.025</td><td>0.620 ± 0.030</td><td>0.653 ± 0.009</td><td>0.626 ± 0.012</td><td>0.655 ± 0.008</td></tr><tr><td/><td colspan="2">2 0.511 ± 0.018</td><td>0.550 ± 0.022</td><td>0.566 ± 0.010</td><td>0.550 ± 0.008</td><td>0.566 ± 0.010</td></tr><tr><td/><td colspan="2">3 0.484 ± 0.014</td><td>0.512 ± 0.017</td><td>0.521 ± 0.008</td><td>0.511 ± 0.006</td><td>0.521 ± 0.008</td></tr><tr><td/><td colspan="2">4 0.469 ± 0.014</td><td>0.491 ± 0.015</td><td>0.498 ± 0.009</td><td>0.490 ± 0.009</td><td>0.499 ± 0.009</td></tr><tr><td>ROC(I)</td><td colspan="2">1 0.779 ± 0.008</td><td>0.799 ± 0.007</td><td>0.806 ± 0.007</td><td>0.797 ± 0.007</td><td>0.809 ± 0.005</td></tr><tr><td/><td colspan="2">2 0.750 ± 0.008</td><td>0.765 ± 0.009</td><td>0.771 ± 0.007</td><td>0.763 ± 0.008</td><td>0.772 ± 0.007</td></tr></table>

Summary source of gain and variants on linear model with UKCF (Bold indicates best) PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C). The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).

Table content: <table><tr><td/><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>PRC(I)</td><td>0.402 ± 0.026</td><td>0.431 ± 0.031</td><td>0.450 ± 0.035</td><td>0.435 ± 0.031</td><td>0.454 ± 0.036</td></tr><tr><td>PRC(C)</td><td>0.507 ± 0.040</td><td>0.543 ± 0.054</td><td>0.559 ± 0.060</td><td>0.544 ± 0.053</td><td>0.560 ± 0.061</td></tr><tr><td>ROC(I)</td><td>0.747 ± 0.022</td><td>0.764 ± 0.022</td><td>0.767 ± 0.026</td><td>0.759 ± 0.025</td><td>0.768 ± 0.028</td></tr><tr><td>ROC(C)</td><td>0.744 ± 0.038</td><td>0.766 ± 0.038</td><td>0.767 ± 0.042</td><td>0.760 ± 0.042</td><td>0.767 ± 0.042</td></tr></table>

Extended results for TEA and comparators on RNN model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>τ</td><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC(I)</td><td colspan="5">1 0.451 ± 0.027* 0.456 ± 0.016* 0.448 ± 0.026* 0.549 ± 0.014</td><td>0.509 ± 0.014*</td></tr><tr><td/><td colspan="5">2 0.417 ± 0.024* 0.420 ± 0.013* 0.416 ± 0.022* 0.490 ± 0.012</td><td>0.463 ± 0.011*</td></tr><tr><td/><td colspan="5">3 0.395 ± 0.019* 0.400 ± 0.013* 0.395 ± 0.020* 0.457 ± 0.010</td><td>0.437 ± 0.013*</td></tr><tr><td/><td colspan="5">4 0.380 ± 0.017* 0.385 ± 0.010* 0.380 ± 0.017* 0.434 ± 0.007</td><td>0.417 ± 0.013*</td></tr><tr><td>PRC(C)</td><td colspan="5">1 0.561 ± 0.056* 0.592 ± 0.029* 0.598 ± 0.029* 0.695 ± 0.010</td><td>0.685 ± 0.015</td></tr><tr><td/><td colspan="5">2 0.504 ± 0.039* 0.523 ± 0.022* 0.527 ± 0.021* 0.591 ± 0.014</td><td>0.584 ± 0.018</td></tr><tr><td/><td colspan="5">3 0.471 ± 0.028* 0.488 ± 0.018* 0.489 ± 0.017* 0.537 ± 0.007</td><td>0.530 ± 0.017</td></tr><tr><td/><td colspan="5">4 0.453 ± 0.023* 0.469 ± 0.017* 0.470 ± 0.016* 0.510 ± 0.007</td><td>0.504 ± 0.015</td></tr><tr><td>ROC(I)</td><td colspan="5">1 0.788 ± 0.018* 0.791 ± 0.009* 0.794 ± 0.014* 0.827 ± 0.007</td><td>0.818 ± 0.006*</td></tr><tr><td/><td colspan="5">2 0.753 ± 0.015* 0.757 ± 0.011* 0.758 ± 0.017* 0.783 ± 0.008</td><td>0.778 ± 0.009</td></tr><tr><td/><td colspan="5">3 0.736 ± 0.013* 0.741 ± 0.012* 0.740 ± 0.016* 0.760 ± 0.007</td><td>0.757 ± 0.010</td></tr><tr><td/><td colspan="5">4 0.725 ± 0.012* 0.731 ± 0.011* 0.727 ± 0.014* 0.748 ± 0.008</td><td>0.744 ± 0.010</td></tr><tr><td colspan="6">ROC(C) 1 0.794 ± 0.022* 0.809 ± 0.015* 0.808 ± 0.012* 0.838 ± 0.007</td><td>0.834 ± 0.007</td></tr><tr><td/><td colspan="5">2 0.750 ± 0.017* 0.761 ± 0.010* 0.761 ± 0.009* 0.782 ± 0.007</td><td>0.781 ± 0.008</td></tr><tr><td/><td colspan="5">3 0.723 ± 0.013* 0.733 ± 0.007* 0.735 ± 0.010* 0.752 ± 0.006</td><td>0.751 ± 0.009</td></tr><tr><td/><td colspan="5">4 0.699 ± 0.009* 0.709 ± 0.009* 0.711 ± 0.010* 0.726 ± 0.006</td><td>0.724 ± 0.008</td></tr><tr><td colspan="7">PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).</td></tr><tr><td colspan="7">The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator</td></tr><tr><td colspan="7">result is used to indicate a statistically significant difference in means (p-value &lt; 0.05) relative to the TEA result.</td></tr></table>

Summary results for TEA and comparators on RNN model with UKCF (Bold indicates best) evaluations are reported separately for targets representing infections (I) and comorbidities (C). The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result. Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.

Table content: <table><tr><td/><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC(I)</td><td>0.411 ± 0.035*</td><td>0.415 ± 0.030*</td><td>0.410 ± 0.033*</td><td>0.483 ± 0.045</td><td>0.457 ± 0.037</td></tr><tr><td>PRC(C)</td><td>0.497 ± 0.057*</td><td>0.518 ± 0.052*</td><td>0.521 ± 0.054*</td><td>0.583 ± 0.072</td><td>0.576 ± 0.071</td></tr><tr><td>ROC(I)</td><td>0.750 ± 0.028*</td><td>0.755 ± 0.025</td><td>0.755 ± 0.029</td><td>0.779 ± 0.031</td><td>0.774 ± 0.030</td></tr><tr><td>ROC(C)</td><td>0.742 ± 0.038</td><td>0.753 ± 0.039</td><td>0.754 ± 0.037</td><td>0.774 ± 0.042</td><td>0.772 ± 0.042</td></tr></table>

Extended source of gain and variants on RNN model with UKCF (Bold indicates best) PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C). The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining, though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).

Table content: <table><tr><td/><td>τ</td><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>PRC(I)</td><td colspan="2">1 0.511 ± 0.014</td><td>0.468 ± 0.015</td><td>0.549 ± 0.014</td><td>0.553 ± 0.009</td><td>0.545 ± 0.011</td></tr><tr><td/><td colspan="2">2 0.461 ± 0.013</td><td>0.429 ± 0.011</td><td>0.490 ± 0.012</td><td>0.492 ± 0.013</td><td>0.487 ± 0.012</td></tr><tr><td/><td colspan="2">3 0.434 ± 0.016</td><td>0.407 ± 0.011</td><td>0.457 ± 0.010</td><td>0.457 ± 0.013</td><td>0.455 ± 0.014</td></tr><tr><td/><td colspan="2">4 0.414 ± 0.009</td><td>0.392 ± 0.011</td><td>0.434 ± 0.007</td><td>0.432 ± 0.008</td><td>0.433 ± 0.009</td></tr><tr><td>PRC(C)</td><td colspan="2">1 0.682 ± 0.011</td><td>0.633 ± 0.032</td><td>0.695 ± 0.010</td><td>0.697 ± 0.010</td><td>0.695 ± 0.012</td></tr><tr><td/><td colspan="2">2 0.581 ± 0.012</td><td>0.549 ± 0.025</td><td>0.591 ± 0.014</td><td>0.589 ± 0.013</td><td>0.592 ± 0.011</td></tr><tr><td/><td colspan="2">3 0.530 ± 0.010</td><td>0.506 ± 0.018</td><td>0.537 ± 0.007</td><td>0.534 ± 0.009</td><td>0.538 ± 0.009</td></tr><tr><td/><td colspan="2">4 0.504 ± 0.009</td><td>0.484 ± 0.015</td><td>0.510 ± 0.007</td><td>0.505 ± 0.008</td><td>0.509 ± 0.010</td></tr><tr><td>ROC(I)</td><td colspan="2">1 0.816 ± 0.004</td><td>0.795 ± 0.008</td><td>0.827 ± 0.007</td><td>0.825 ± 0.005</td><td>0.822 ± 0.010</td></tr><tr><td/><td colspan="2">2 0.774 ± 0.010</td><td>0.759 ± 0.007</td><td>0.783 ± 0.008</td><td>0.782 ± 0.008</td><td>0.778 ± 0.007</td></tr><tr><td/><td colspan="2">3 0.749 ± 0.010</td><td>0.744 ± 0.008</td><td>0.760 ± 0.007</td><td>0.758 ± 0.006</td><td>0.758 ± 0.006</td></tr><tr><td/><td colspan="2">4 0.732 ± 0.008</td><td>0.735 ± 0.008</td><td>0.748 ± 0.008</td><td>0.739 ± 0.007</td><td>0.745 ± 0.004</td></tr><tr><td colspan="3">ROC(C) 1 0.830 ± 0.008</td><td>0.816 ± 0.011</td><td>0.838 ± 0.007</td><td>0.839 ± 0.008</td><td>0.839 ± 0.007</td></tr><tr><td/><td colspan="2">2 0.775 ± 0.010</td><td>0.767 ± 0.009</td><td>0.782 ± 0.007</td><td>0.784 ± 0.010</td><td>0.782 ± 0.005</td></tr><tr><td/><td colspan="2">3 0.743 ± 0.009</td><td>0.735 ± 0.007</td><td>0.752 ± 0.006</td><td>0.751 ± 0.010</td><td>0.751 ± 0.007</td></tr><tr><td/><td colspan="2">4 0.721 ± 0.006</td><td>0.712 ± 0.007</td><td>0.726 ± 0.006</td><td>0.724 ± 0.007</td><td>0.726 ± 0.006</td></tr></table>

Summary source of gain and variants on RNN model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>PRC(I)</td><td>0.455 ± 0.039</td><td>0.424 ± 0.031</td><td>0.483 ± 0.045</td><td>0.483 ± 0.047</td><td>0.480 ± 0.044</td></tr><tr><td>PRC(C)</td><td>0.574 ± 0.069</td><td>0.543 ± 0.061</td><td>0.583 ± 0.072</td><td>0.581 ± 0.074</td><td>0.583 ± 0.072</td></tr><tr><td>ROC(I)</td><td>0.768 ± 0.033</td><td>0.758 ± 0.024</td><td>0.779 ± 0.031</td><td>0.776 ± 0.033</td><td>0.776 ± 0.030</td></tr><tr><td>ROC(C)</td><td>0.767 ± 0.042</td><td>0.758 ± 0.040</td><td>0.774 ± 0.042</td><td>0.774 ± 0.044</td><td>0.774 ± 0.043</td></tr><tr><td colspan="6">PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).</td></tr></table>

Extended results for TEA and comparators on RNN model with ADNI (Bold indicates best) separately for targets representing quantitative biomarkers (B) and cognitive tests (C). The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td>τ</td><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td colspan="5">MSE(B) 1 0.095 ± 0.014* 0.088 ± 0.010* 0.082 ± 0.007* 0.057 ± 0.007</td><td>0.065 ± 0.008*</td></tr><tr><td colspan="5">2 0.097 ± 0.015* 0.089 ± 0.010* 0.084 ± 0.008* 0.057 ± 0.007</td><td>0.066 ± 0.007*</td></tr><tr><td colspan="5">3 0.100 ± 0.015* 0.092 ± 0.010* 0.087 ± 0.008* 0.059 ± 0.007</td><td>0.068 ± 0.007*</td></tr><tr><td colspan="5">4 0.104 ± 0.016* 0.095 ± 0.011* 0.091 ± 0.008* 0.061 ± 0.007</td><td>0.071 ± 0.008*</td></tr><tr><td colspan="5">5 0.105 ± 0.017* 0.097 ± 0.012* 0.093 ± 0.008* 0.062 ± 0.008</td><td>0.073 ± 0.008*</td></tr><tr><td colspan="5">6 0.109 ± 0.017* 0.100 ± 0.013* 0.097 ± 0.009* 0.065 ± 0.008</td><td>0.076 ± 0.009*</td></tr><tr><td colspan="5">7 0.112 ± 0.019* 0.103 ± 0.014* 0.101 ± 0.011* 0.068 ± 0.009</td><td>0.080 ± 0.009*</td></tr><tr><td colspan="5">8 0.115 ± 0.021* 0.106 ± 0.016* 0.105 ± 0.013* 0.072 ± 0.011</td><td>0.083 ± 0.010*</td></tr><tr><td colspan="5">MSE(C) 1 0.275 ± 0.013* 0.270 ± 0.013* 0.265 ± 0.011* 0.239 ± 0.015</td><td>0.243 ± 0.013</td></tr><tr><td colspan="5">2 0.300 ± 0.015* 0.295 ± 0.013* 0.290 ± 0.012* 0.265 ± 0.014</td><td>0.273 ± 0.014</td></tr><tr><td colspan="5">3 0.323 ± 0.018* 0.320 ± 0.015* 0.314 ± 0.013* 0.287 ± 0.014</td><td>0.297 ± 0.015</td></tr><tr><td colspan="5">4 0.358 ± 0.019* 0.354 ± 0.018* 0.352 ± 0.017* 0.322 ± 0.015</td><td>0.333 ± 0.018</td></tr><tr><td colspan="5">5 0.371 ± 0.024* 0.370 ± 0.023* 0.367 ± 0.023* 0.341 ± 0.019</td><td>0.350 ± 0.021</td></tr><tr><td colspan="2">6 0.393 ± 0.033</td><td>0.393 ± 0.032</td><td>0.391 ± 0.034</td><td>0.366 ± 0.026</td><td>0.374 ± 0.028</td></tr><tr><td colspan="2">7 0.417 ± 0.043</td><td>0.419 ± 0.040</td><td>0.417 ± 0.044</td><td>0.394 ± 0.035</td><td>0.399 ± 0.038</td></tr></table>

Summary results for TEA and comparators on RNN model with ADNI (Bold indicates best) The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td/><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>MSE(B)</td><td>0.105 ± 0.018*</td><td>0.096 ± 0.014*</td><td>0.092 ± 0.012*</td><td>0.063 ± 0.010</td><td>0.073 ± 0.010*</td></tr><tr><td>MSE(C)</td><td>0.361 ± 0.064</td><td>0.360 ± 0.066</td><td>0.356 ± 0.068</td><td>0.330 ± 0.066</td><td>0.338 ± 0.067</td></tr><tr><td colspan="6">MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C).</td></tr></table>

Extended source of gain and variants on RNN model with ADNI (Bold indicates best)

Table content: <table><tr><td>τ</td><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td colspan="2">MSE(B) 1 0.081 ± 0.011</td><td>0.098 ± 0.018</td><td>0.057 ± 0.007</td><td>0.049 ± 0.009</td><td>0.057 ± 0.009</td></tr><tr><td colspan="2">2 0.084 ± 0.011</td><td>0.098 ± 0.018</td><td>0.057 ± 0.007</td><td>0.051 ± 0.010</td><td>0.058 ± 0.008</td></tr><tr><td colspan="2">3 0.087 ± 0.011</td><td>0.101 ± 0.019</td><td>0.059 ± 0.007</td><td>0.054 ± 0.011</td><td>0.059 ± 0.008</td></tr><tr><td colspan="2">4 0.090 ± 0.011</td><td>0.105 ± 0.019</td><td>0.061 ± 0.007</td><td>0.056 ± 0.011</td><td>0.062 ± 0.008</td></tr><tr><td colspan="2">5 0.092 ± 0.011</td><td>0.106 ± 0.021</td><td>0.062 ± 0.008</td><td>0.059 ± 0.011</td><td>0.064 ± 0.009</td></tr><tr><td colspan="2">6 0.097 ± 0.012</td><td>0.110 ± 0.023</td><td>0.065 ± 0.008</td><td>0.063 ± 0.011</td><td>0.068 ± 0.010</td></tr><tr><td colspan="2">7 0.100 ± 0.013</td><td>0.113 ± 0.025</td><td>0.068 ± 0.009</td><td>0.066 ± 0.010</td><td>0.072 ± 0.011</td></tr><tr><td colspan="2">8 0.104 ± 0.014</td><td>0.117 ± 0.027</td><td>0.072 ± 0.011</td><td>0.070 ± 0.011</td><td>0.076 ± 0.013</td></tr><tr><td colspan="2">MSE(C) 1 0.258 ± 0.016</td><td>0.274 ± 0.017</td><td>0.239 ± 0.015</td><td>0.231 ± 0.020</td><td>0.241 ± 0.016</td></tr><tr><td colspan="2">2 0.285 ± 0.016</td><td>0.297 ± 0.017</td><td>0.265 ± 0.014</td><td>0.258 ± 0.021</td><td>0.266 ± 0.018</td></tr><tr><td colspan="2">3 0.311 ± 0.017</td><td>0.321 ± 0.018</td><td>0.287 ± 0.014</td><td>0.282 ± 0.021</td><td>0.287 ± 0.018</td></tr><tr><td colspan="2">4 0.346 ± 0.019</td><td>0.356 ± 0.020</td><td>0.322 ± 0.015</td><td>0.319 ± 0.021</td><td>0.321 ± 0.022</td></tr><tr><td colspan="2">5 0.363 ± 0.024</td><td>0.373 ± 0.024</td><td>0.341 ± 0.019</td><td>0.337 ± 0.024</td><td>0.338 ± 0.026</td></tr><tr><td colspan="2">6 0.389 ± 0.033</td><td>0.397 ± 0.031</td><td>0.366 ± 0.026</td><td>0.366 ± 0.030</td><td>0.362 ± 0.033</td></tr><tr><td colspan="2">7 0.416 ± 0.041</td><td>0.424 ± 0.040</td><td>0.394 ± 0.035</td><td>0.401 ± 0.043</td><td>0.390 ± 0.044</td></tr><tr><td colspan="2">8 0.454 ± 0.059</td><td>0.462 ± 0.053</td><td>0.430 ± 0.048</td><td>0.447 ± 0.064</td><td>0.427 ± 0.063</td></tr><tr><td colspan="6">MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C).</td></tr><tr><td colspan="6">The "No Joint" setting isolates the benefit from staged training only (analogous to basic unsupervised pretraining,</td></tr><tr><td colspan="6">though using targets); the "No Staged" setting isolates the benefit from joint training only (without pretraining).</td></tr></table>

Summary source of gain and variants on RNN model with ADNI (Bold indicates best)

Table content: <table><tr><td/><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>MSE(B)</td><td>0.092 ± 0.014</td><td>0.106 ± 0.022</td><td>0.063 ± 0.010</td><td>0.058 ± 0.012</td><td>0.064 ± 0.012</td></tr><tr><td>MSE(C)</td><td>0.353 ± 0.070</td><td>0.363 ± 0.067</td><td>0.330 ± 0.066</td><td>0.330 ± 0.076</td><td>0.329 ± 0.068</td></tr><tr><td colspan="6">MSE evaluations reported separately for targets representing quantitative biomarkers (B) and cognitive tests (C).</td></tr></table>

Extended results for TEA and comparators on RNN model with MIMIC (Bold indicates best)The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td/><td>τ</td><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC</td><td>1</td><td>0.159 ± 0.034*</td><td colspan="3">0.159 ± 0.022* 0.162 ± 0.036* 0.293 ± 0.031</td><td>0.193 ± 0.023*</td></tr><tr><td/><td>2</td><td>0.148 ± 0.028*</td><td colspan="3">0.149 ± 0.018* 0.150 ± 0.030* 0.254 ± 0.025</td><td>0.174 ± 0.018*</td></tr><tr><td/><td>3</td><td>0.139 ± 0.024*</td><td colspan="3">0.141 ± 0.015* 0.142 ± 0.027* 0.230 ± 0.021</td><td>0.162 ± 0.015*</td></tr><tr><td/><td>4</td><td>0.133 ± 0.021*</td><td colspan="3">0.135 ± 0.012* 0.135 ± 0.024* 0.214 ± 0.018</td><td>0.153 ± 0.012*</td></tr><tr><td/><td>5</td><td>0.129 ± 0.019*</td><td colspan="3">0.130 ± 0.011* 0.130 ± 0.022* 0.203 ± 0.015</td><td>0.147 ± 0.011*</td></tr><tr><td>ROC</td><td>1</td><td>0.699 ± 0.049*</td><td colspan="3">0.704 ± 0.028* 0.709 ± 0.060* 0.801 ± 0.018</td><td>0.745 ± 0.021*</td></tr></table>

Summary results for TEA and comparators on RNN model with MIMIC (Bold indicates best)The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td/><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>PRC</td><td>0.142 ± 0.028*</td><td>0.143 ± 0.019*</td><td>0.144 ± 0.030*</td><td>0.239 ± 0.039</td><td>0.166 ± 0.023*</td></tr><tr><td>ROC</td><td>0.690 ± 0.043*</td><td>0.696 ± 0.026*</td><td>0.694 ± 0.050*</td><td>0.763 ± 0.028</td><td>0.728 ± 0.023*</td></tr><tr><td>MSE</td><td>0.153 ± 0.011</td><td>0.152 ± 0.010</td><td>0.152 ± 0.012</td><td>0.150 ± 0.012</td><td>0.154 ± 0.011</td></tr></table>

Extended source of gain and variants on RNN model with MIMIC (Bold indicates best)

Table content: <table><tr><td>τ</td><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>PRC</td><td>0.216 ± 0.044</td><td>0.194 ± 0.020</td><td>0.293 ± 0.031</td><td>0.310 ± 0.047</td><td>0.280 ± 0.033</td></tr><tr><td/><td>0.194 ± 0.035</td><td>0.175 ± 0.017</td><td>0.254 ± 0.025</td><td>0.265 ± 0.035</td><td>0.242 ± 0.026</td></tr><tr><td/><td>0.178 ± 0.030</td><td>0.163 ± 0.013</td><td>0.230 ± 0.021</td><td>0.239 ± 0.030</td><td>0.221 ± 0.022</td></tr><tr><td/><td>0.168 ± 0.027</td><td>0.154 ± 0.011</td><td>0.214 ± 0.018</td><td>0.222 ± 0.026</td><td>0.206 ± 0.020</td></tr><tr><td/><td>0.160 ± 0.024</td><td>0.148 ± 0.010</td><td>0.203 ± 0.015</td><td>0.210 ± 0.023</td><td>0.195 ± 0.018</td></tr><tr><td>ROC</td><td>0.756 ± 0.042</td><td>0.742 ± 0.022</td><td>0.801 ± 0.018</td><td>0.807 ± 0.025</td><td>0.791 ± 0.018</td></tr><tr><td/><td>0.741 ± 0.031</td><td>0.738 ± 0.021</td><td>0.778 ± 0.015</td><td>0.783 ± 0.017</td><td>0.773 ± 0.012</td></tr><tr><td/><td>0.726 ± 0.031</td><td>0.724 ± 0.019</td><td>0.758 ± 0.016</td><td>0.761 ± 0.019</td><td>0.756 ± 0.013</td></tr><tr><td/><td>0.715 ± 0.030</td><td>0.715 ± 0.019</td><td>0.745 ± 0.015</td><td>0.747 ± 0.017</td><td>0.742 ± 0.011</td></tr><tr><td/><td>0.710 ± 0.031</td><td>0.711 ± 0.018</td><td>0.736 ± 0.012</td><td>0.741 ± 0.019</td><td>0.736 ± 0.014</td></tr><tr><td>MSE</td><td>0.138 ± 0.008</td><td>0.137 ± 0.007</td><td>0.137 ± 0.008</td><td>0.136 ± 0.006</td><td>0.137 ± 0.008</td></tr><tr><td/><td>0.158 ± 0.007</td><td>0.156 ± 0.012</td><td>0.154 ± 0.009</td><td>0.153 ± 0.007</td><td>0.154 ± 0.007</td></tr><tr><td/><td>0.155 ± 0.009</td><td>0.156 ± 0.010</td><td>0.158 ± 0.008</td><td>0.156 ± 0.010</td><td>0.158 ± 0.008</td></tr><tr><td/><td>0.152 ± 0.008</td><td>0.153 ± 0.009</td><td>0.153 ± 0.009</td><td>0.151 ± 0.011</td><td>0.154 ± 0.008</td></tr><tr><td/><td>0.151 ± 0.009</td><td>0.150 ± 0.009</td><td>0.150 ± 0.011</td><td>0.147 ± 0.011</td><td>0.150 ± 0.008</td></tr></table>

Summary source of gain and variants on RNN model with MIMIC (Bold indicates best)

Table content: <table><tr><td/><td>No Joint</td><td>No Staged</td><td>TEA</td><td>TEA(L)</td><td>TEA(LP)</td></tr><tr><td>PRC</td><td>0.183 ± 0.038</td><td>0.167 ± 0.022</td><td>0.239 ± 0.039</td><td>0.249 ± 0.049</td><td>0.229 ± 0.039</td></tr><tr><td>ROC</td><td>0.730 ± 0.038</td><td>0.726 ± 0.023</td><td>0.763 ± 0.028</td><td>0.768 ± 0.031</td><td>0.759 ± 0.025</td></tr><tr><td>MSE</td><td>0.151 ± 0.011</td><td>0.150 ± 0.012</td><td>0.150 ± 0.012</td><td>0.149 ± 0.012</td><td>0.151 ± 0.011</td></tr></table>

Extended ν-Sensitivities for REG on linear model with UKCF (Bold indicates best)The ν coefficient controls the strength of 2-regularization applied on top of the original loss function minimized. PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).

Table content: <table><tr><td>τ</td><td>ν = 0</td><td>ν = 3e-5</td><td>ν = 3e-4</td><td>ν = 3e-3</td><td>ν = 3e-2</td></tr><tr><td>PRC(I)</td><td>0.340 ± 0.118</td><td>0.355 ± 0.114</td><td>0.370 ± 0.101</td><td>0.320 ± 0.034</td><td>0.163 ± 0.003</td></tr><tr><td/><td>0.325 ± 0.099</td><td>0.338 ± 0.096</td><td>0.350 ± 0.085</td><td>0.309 ± 0.026</td><td>0.176 ± 0.004</td></tr><tr><td/><td>0.314 ± 0.088</td><td>0.327 ± 0.086</td><td>0.337 ± 0.076</td><td>0.300 ± 0.024</td><td>0.182 ± 0.005</td></tr><tr><td/><td>0.307 ± 0.082</td><td>0.318 ± 0.080</td><td>0.328 ± 0.071</td><td>0.293 ± 0.023</td><td>0.184 ± 0.004</td></tr><tr><td colspan="2">PRC(C) 0.445 ± 0.131</td><td>0.460 ± 0.125</td><td>0.467 ± 0.106</td><td>0.426 ± 0.034</td><td>0.240 ± 0.004</td></tr><tr><td/><td>0.418 ± 0.099</td><td>0.428 ± 0.095</td><td>0.435 ± 0.081</td><td>0.409 ± 0.025</td><td>0.260 ± 0.005</td></tr><tr><td/><td>0.403 ± 0.082</td><td>0.412 ± 0.078</td><td>0.418 ± 0.067</td><td>0.399 ± 0.021</td><td>0.272 ± 0.006</td></tr><tr><td/><td>0.398 ± 0.073</td><td>0.406 ± 0.070</td><td>0.412 ± 0.060</td><td>0.397 ± 0.019</td><td>0.281 ± 0.007</td></tr><tr><td>ROC(I)</td><td>0.713 ± 0.104</td><td>0.724 ± 0.100</td><td>0.737 ± 0.081</td><td>0.715 ± 0.022</td><td>0.527 ± 0.006</td></tr><tr><td/><td>0.688 ± 0.089</td><td>0.697 ± 0.086</td><td>0.709 ± 0.070</td><td>0.693 ± 0.022</td><td>0.532 ± 0.008</td></tr><tr><td/><td>0.677 ± 0.080</td><td>0.686 ± 0.078</td><td>0.697 ± 0.065</td><td>0.683 ± 0.021</td><td>0.543 ± 0.007</td></tr><tr><td/><td>0.677 ± 0.076</td><td>0.686 ± 0.074</td><td>0.696 ± 0.063</td><td>0.681 ± 0.019</td><td>0.555 ± 0.007</td></tr><tr><td colspan="2">ROC(C) 0.713 ± 0.110</td><td>0.727 ± 0.105</td><td>0.741 ± 0.088</td><td>0.716 ± 0.025</td><td>0.512 ± 0.010</td></tr><tr><td/><td>0.686 ± 0.091</td><td>0.696 ± 0.086</td><td>0.707 ± 0.072</td><td>0.686 ± 0.022</td><td>0.523 ± 0.008</td></tr><tr><td/><td>0.668 ± 0.077</td><td>0.678 ± 0.074</td><td>0.686 ± 0.061</td><td>0.668 ± 0.021</td><td>0.530 ± 0.013</td></tr><tr><td/><td>0.651 ± 0.070</td><td>0.660 ± 0.066</td><td>0.667 ± 0.056</td><td>0.652 ± 0.018</td><td>0.527 ± 0.015</td></tr></table>

Summary ν-Sensitivities for REG on linear model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>ν = 0</td><td>ν = 3e-5</td><td>ν = 3e-4</td><td>ν = 3e-3</td><td>ν = 3e-2</td></tr><tr><td>PRC(I)</td><td>0.322 ± 0.099</td><td>0.335 ± 0.096</td><td>0.347 ± 0.085</td><td>0.305 ± 0.029</td><td>0.176 ± 0.009</td></tr><tr><td>PRC(C)</td><td>0.416 ± 0.100</td><td>0.426 ± 0.097</td><td>0.433 ± 0.083</td><td>0.408 ± 0.028</td><td>0.263 ± 0.016</td></tr><tr><td>ROC(I)</td><td>0.689 ± 0.089</td><td>0.698 ± 0.087</td><td>0.710 ± 0.072</td><td>0.693 ± 0.025</td><td>0.540 ± 0.013</td></tr><tr><td>ROC(C)</td><td>0.679 ± 0.091</td><td>0.690 ± 0.087</td><td>0.700 ± 0.075</td><td>0.681 ± 0.032</td><td>0.523 ± 0.013</td></tr></table>

Extended ν-Sensitivities for TEA on linear model with UKCF (Bold indicates best)The ν coefficient controls the strength of 2-regularization applied on top of the original loss function minimized. PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C).

Table content: <table><tr><td>τ</td><td>ν = 0</td><td>ν = 3e-5</td><td>ν = 3e-4</td><td>ν = 3e-3</td><td>ν = 3e-2</td></tr><tr><td>PRC(I)</td><td>0.484 ± 0.020</td><td>0.489 ± 0.019</td><td>0.497 ± 0.016</td><td>0.442 ± 0.005</td><td>0.174 ± 0.004</td></tr><tr><td/><td>0.450 ± 0.016</td><td>0.453 ± 0.016</td><td>0.459 ± 0.017</td><td>0.414 ± 0.007</td><td>0.186 ± 0.005</td></tr><tr><td/><td>0.424 ± 0.014</td><td>0.426 ± 0.013</td><td>0.432 ± 0.015</td><td>0.394 ± 0.009</td><td>0.192 ± 0.005</td></tr><tr><td/><td>0.405 ± 0.012</td><td>0.407 ± 0.010</td><td>0.413 ± 0.010</td><td>0.381 ± 0.008</td><td>0.193 ± 0.004</td></tr><tr><td colspan="2">PRC(C) 0.641 ± 0.021</td><td>0.644 ± 0.019</td><td>0.653 ± 0.009</td><td>0.612 ± 0.009</td><td>0.276 ± 0.007</td></tr><tr><td/><td>0.561 ± 0.013</td><td>0.562 ± 0.011</td><td>0.566 ± 0.010</td><td>0.544 ± 0.007</td><td>0.293 ± 0.008</td></tr><tr><td/><td>0.519 ± 0.008</td><td>0.519 ± 0.007</td><td>0.521 ± 0.008</td><td>0.508 ± 0.005</td><td>0.302 ± 0.008</td></tr><tr><td/><td>0.495 ± 0.008</td><td>0.496 ± 0.007</td><td>0.498 ± 0.009</td><td>0.489 ± 0.007</td><td>0.309 ± 0.009</td></tr><tr><td>ROC(I)</td><td>0.800 ± 0.015</td><td>0.803 ± 0.015</td><td>0.806 ± 0.007</td><td>0.779 ± 0.007</td><td>0.555 ± 0.005</td></tr><tr><td/><td>0.765 ± 0.009</td><td>0.767 ± 0.011</td><td>0.771 ± 0.007</td><td>0.751 ± 0.007</td><td>0.557 ± 0.007</td></tr><tr><td/><td>0.746 ± 0.008</td><td>0.747 ± 0.008</td><td>0.750 ± 0.007</td><td>0.735 ± 0.007</td><td>0.565 ± 0.006</td></tr><tr><td/><td>0.736 ± 0.006</td><td>0.737 ± 0.007</td><td>0.740 ± 0.008</td><td>0.727 ± 0.005</td><td>0.575 ± 0.006</td></tr><tr><td colspan="2">ROC(C) 0.825 ± 0.011</td><td>0.826 ± 0.010</td><td>0.829 ± 0.006</td><td>0.819 ± 0.006</td><td>0.560 ± 0.011</td></tr><tr><td/><td>0.772 ± 0.010</td><td>0.774 ± 0.009</td><td>0.775 ± 0.008</td><td>0.771 ± 0.006</td><td>0.564 ± 0.009</td></tr><tr><td/><td>0.742 ± 0.004</td><td>0.744 ± 0.004</td><td>0.743 ± 0.005</td><td>0.741 ± 0.004</td><td>0.566 ± 0.012</td></tr><tr><td/><td>0.718 ± 0.006</td><td>0.720 ± 0.006</td><td>0.720 ± 0.006</td><td>0.717 ± 0.008</td><td>0.561 ± 0.015</td></tr></table>

Summary ν-Sensitivities for TEA on linear model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>ν = 0</td><td>ν = 3e-5</td><td>ν = 3e-4</td><td>ν = 3e-3</td><td>ν = 3e-2</td></tr><tr><td>PRC(I)</td><td>0.441 ± 0.033</td><td>0.444 ± 0.034</td><td>0.450 ± 0.035</td><td>0.408 ± 0.024</td><td>0.186 ± 0.009</td></tr><tr><td>PRC(C)</td><td>0.554 ± 0.057</td><td>0.555 ± 0.058</td><td>0.559 ± 0.060</td><td>0.538 ± 0.047</td><td>0.295 ± 0.015</td></tr><tr><td>ROC(I)</td><td>0.762 ± 0.026</td><td>0.763 ± 0.028</td><td>0.767 ± 0.026</td><td>0.748 ± 0.021</td><td>0.563 ± 0.010</td></tr><tr><td>ROC(C)</td><td>0.764 ± 0.041</td><td>0.766 ± 0.041</td><td>0.767 ± 0.042</td><td>0.762 ± 0.039</td><td>0.563 ± 0.012</td></tr></table>

Extended λ-Sensitivities for TEA on linear model with UKCF (Bold indicates best)

Table content: <table/>

Summary λ-Sensitivities for TEA on linear model with UKCF (Bold indicates best) PRC(I) 0.347 ± 0.085 0.357 ± 0.090 0.381 ± 0.078 0.431 ± 0.031 0.424 ± 0.027 0.318 ± 0.013 0.162 ± 0.011 PRC(C) 0.433 ± 0.083 0.443 ± 0.087 0.477 ± 0.084 0.543 ± 0.054 0.546 ± 0.050 0.424 ± 0.012 0.252 ± 0.016 ROC(I) 0.710 ± 0.072 0.714 ± 0.078 0.733 ± 0.061 0.764 ± 0.022 0.759 ± 0.020 0.695 ± 0.010 0.501 ± 0.014 ROC(C) 0.700 ± 0.075 0.705 ± 0.080 0.727 ± 0.068 0.766 ± 0.038 0.765 ± 0.037 0.698 ± 0.022 0.496 ± 0.029

Table content: <table><tr><td>λ = 0</td><td>λ = 0.01</td><td>λ = 0.1</td><td>λ = 0.5</td><td>λ = 0.9</td><td>λ = 0.99</td><td>λ = 1</td></tr></table>

Extended N -Sensitivities for REG on linear model with UKCF (Bold indicates best)

Table content: <table><tr><td/><td>τ</td><td>N × 1%</td><td>N × 5%</td><td>N × 20%</td><td>N × 50%</td><td>N × 100%</td></tr><tr><td>PRC(I)</td><td colspan="2">1 0.160 ± 0.010</td><td>0.176 ± 0.024</td><td>0.199 ± 0.044</td><td>0.325 ± 0.114</td><td>0.370 ± 0.101</td></tr><tr><td/><td colspan="2">2 0.172 ± 0.009</td><td>0.187 ± 0.021</td><td>0.207 ± 0.037</td><td>0.312 ± 0.095</td><td>0.350 ± 0.085</td></tr><tr><td/><td colspan="2">3 0.180 ± 0.011</td><td>0.192 ± 0.020</td><td>0.209 ± 0.032</td><td>0.303 ± 0.085</td><td>0.337 ± 0.076</td></tr><tr><td/><td colspan="2">4 0.182 ± 0.009</td><td>0.193 ± 0.020</td><td>0.208 ± 0.028</td><td>0.295 ± 0.078</td><td>0.328 ± 0.071</td></tr><tr><td colspan="3">PRC(C) 1 0.246 ± 0.011</td><td>0.268 ± 0.034</td><td>0.293 ± 0.046</td><td>0.421 ± 0.119</td><td>0.467 ± 0.106</td></tr><tr><td/><td colspan="2">2 0.263 ± 0.011</td><td>0.283 ± 0.027</td><td>0.304 ± 0.036</td><td>0.401 ± 0.091</td><td>0.435 ± 0.081</td></tr><tr><td/><td colspan="2">3 0.275 ± 0.013</td><td>0.292 ± 0.028</td><td>0.313 ± 0.032</td><td>0.390 ± 0.076</td><td>0.418 ± 0.067</td></tr><tr><td/><td colspan="2">4 0.286 ± 0.013</td><td>0.302 ± 0.025</td><td>0.319 ± 0.029</td><td>0.384 ± 0.066</td><td>0.412 ± 0.060</td></tr><tr><td>ROC(I)</td><td colspan="2">1 0.512 ± 0.024</td><td>0.553 ± 0.046</td><td>0.598 ± 0.057</td><td>0.705 ± 0.090</td><td>0.737 ± 0.081</td></tr><tr><td/><td colspan="2">2 0.516 ± 0.025</td><td>0.557 ± 0.040</td><td>0.594 ± 0.049</td><td>0.681 ± 0.077</td><td>0.709 ± 0.070</td></tr></table>

λ-Sensitivities for TEA on linear model for negative example (Bold indicates best) further reported separately for targets that are in principle predictable (P) and unpredictable (U).

Table content: <table><tr><td/><td>λ = 0</td><td>λ = 0.01</td><td>λ = 0.1</td><td>λ = 0.5</td><td>λ = 0.9</td></tr><tr><td>MSE</td><td>0.266 ± 0.045</td><td>0.267 ± 0.045</td><td>0.271 ± 0.046</td><td>0.285 ± 0.047</td><td>0.292 ± 0.051</td></tr><tr><td>MSE(U)</td><td>0.333 ± 0.056</td><td>0.333 ± 0.056</td><td>0.334 ± 0.056</td><td>0.334 ± 0.056</td><td>0.334 ± 0.056</td></tr><tr><td>MSE(P)</td><td>0.000 ± 0.000</td><td>0.003 ± 0.005</td><td>0.022 ± 0.009</td><td>0.087 ± 0.028</td><td>0.125 ± 0.063</td></tr><tr><td colspan="2">MSE metrics are</td><td/><td/><td/><td/></tr></table>

Performance degradation for TEA and comparators on linear model with UKCF (Bold indicates best) PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C). The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td/><td>Base</td><td>REG</td><td>FEA</td><td>TEA</td><td>F/TEA</td></tr><tr><td>ROC(I)</td><td>0.019 ± 0.015</td><td>0.020 ± 0.014</td><td>0.019 ± 0.015</td><td>0.020 ± 0.016</td><td>0.017 ± 0.014</td></tr><tr><td>ROC(C)</td><td>0.025 ± 0.015</td><td>0.029 ± 0.015</td><td>0.024 ± 0.014</td><td>0.013 ± 0.020</td><td>0.019 ± 0.018</td></tr><tr><td>PRC(I)</td><td>0.022 ± 0.020</td><td>0.018 ± 0.021</td><td>0.021 ± 0.022</td><td>0.033 ± 0.022</td><td>0.027 ± 0.022</td></tr><tr><td>PRC(C)</td><td>0.026 ± 0.021</td><td>0.029 ± 0.018</td><td>0.026 ± 0.019</td><td>0.018 ± 0.023</td><td>0.021 ± 0.019</td></tr></table>

Performance by training stages for TEA on linear model with UKCF (Bold indicates best); column headers indicate the sequence of training stages executed (note that "1-2-3" simply corresponds to Algorithm 1) PRC and ROC evaluations are reported separately for targets representing infections (I) and comorbidities (C). The two-sample t-test for a difference in means is conducted on the results. An asterisk next to the comparator result is used to indicate a statistically significant difference in means (p-value < 0.05) relative to the TEA result.

Table content: <table><tr><td/><td>None</td><td>1-2</td><td>3</td><td>1-2-3</td><td>3-1-2</td></tr><tr><td>ROC(I)</td><td>0.710 ± 0.072</td><td>0.747 ± 0.022</td><td>0.764 ± 0.022</td><td>0.767 ± 0.026</td><td>0.749 ± 0.022</td></tr><tr><td>ROC(C)</td><td>0.700 ± 0.075</td><td>0.744 ± 0.038</td><td>0.766 ± 0.038</td><td>0.767 ± 0.042</td><td>0.747 ± 0.037</td></tr><tr><td>PRC(I)</td><td>0.347 ± 0.085</td><td>0.402 ± 0.026</td><td>0.431 ± 0.031</td><td>0.450 ± 0.035</td><td>0.404 ± 0.027</td></tr><tr><td>PRC(C)</td><td>0.433 ± 0.083</td><td>0.507 ± 0.040</td><td>0.543 ± 0.054</td><td>0.559 ± 0.060</td><td>0.512 ± 0.042</td></tr></table>

Peer review (iclr2020):
This work introduces the idea of target embedding autoencoders for supervised prediction, designed to learn intermediate latent representations jointly optimized to be both predictable from features and predictive of targets. This is meant to help with generalization and has certain theoretical guarantees. It is an interesting problem setting to consider where Y is high dimensional instead of X. More examples of this would be useful to provide in the intro. I think this is crucial to understand where this method might be useful. Figure 1 is super informative and very nice! Section 2: Why do we expect that this paradigm of autoencoder based regularization “generalizes” better? I like the explicit and honest discussion of prior work in this section. One question is how important is the choice of reconstruction loss function - L2, vs max likelihood gaussian, vs L1, vs cross entropy, etc for performance? Another question: how bad is performance if the learning is done stagewise - first the Y-Z-Y^ representation is learned and then the X->Z predictor is learned. If something is out of distribution, how easy are TEA based learners to finetune? Overall the idea seems reasonable - if the targets have some common set of factors, just predict those instead of predicting the full target value which might be harder to get right. It’s just a question of whether this holds true in many domains and how well this reconstruction loss generalizes across problems? Section 3: “We havenoted that TEA components can in principle be instantiated by any architecture. Does its benefit extend beyond the commonly-studied domain of static classification?” -> not clear what this means? Does this mean this algorithm has been proposed before or is it that it can ALSO work on non static classification tasks? Not clear how to situate this claim The theoretical section seems to follow largely from Le et al, but with important distinctions on dimensionalities of various spaces involved. I wonder if the authors can comment on how often Assumption 1 and 2 are actually satisified? Related Work: Is the main difference between Yu 2014 and this just in the norm based regularization? I don’t think so, can this be made more clear. This seems also fairly related to Yeh, is it just a generalization of that paradigm? Or is there more to it? In light of the contribution of Yeh, this seems like slightly more marginal of a contribution? Is the main points of contribution the theoretical analysis and the extended experiments to sequence data rather than static classification? The results do seem to show a signficant benefit as compared to FEA or base models. It also seems like this is applicable across multiple disease datasets. Do the authors think that this could be applicable to other domains altogether? Would it be quick to run a comparison on these? Generally seems like a well grounded and meaningful contribution with many improvements. Would be curious to see applications to other datasets and also some improvements/clarifications noted above?