Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper examines target-embedding autoencoders (TEAs) in theory and practice. TEAs autoencode the output (rather than input) space and find a mapping from the input to the latent representation of the output. The forward pass of the decoder (for the output space) is shared by the input-to-output computation.

Target-embedding autoencoders (TEAs) have previously been proposed and used in practice (though not necessarily by the "TEA" name).  The paper's presentation is confusing on this matter, at it claims to be the first to "motivate and formalize" TEAs; I do not believe it is appropriate to claim such a contribution in light of prior work. [Girdhar et al.] clearly utilizes a target-embedding autoencoder (see [Girdhar et al.] Figure 2). In addition, more recent published work clearly utilizes TEAs (though not named as such) as the centerpiece of their approaches. See, for example:

[A] Adrian V. Dalca, John Guttag, Mert R. Sabuncu. Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation. CVPR, 2018.

[B] Mohammadreza Mostajabi, Michael Maire, Gregory Shakhnarovich. Regularizing Deep Networks by Modeling and Predicting Label Structure. CVPR, 2018.

Figure 2 of [A] and Figure 1 of [B] both clearly depict applying target-embedding autoencoders on semantic image segmentation problems. [B] operates in the same supervised representation-learning setting proposed here. Notably, [B] utilizes staged training -- learning the autoencoder first -- as discussed in Section 2 of the submitted paper, and finds that to be important for achieving a regularization effect.

The real applications explored by [A] and [B] are perhaps more challenging than the datasets used in experiments here.  The concluding sentence of the paper,"Target-representation learning is potentially applicable to any high-dimensional prediction task, and exploring its utility for specific domain-architectures may be a practical direction for future research" should be changed -- prior work has already successfully utilized TEAs in the specific domain of image segmentation.

Given that the paper has missed (not cited) highly related published work that applies TEAs in practice, a rewrite of Section 4 is required. In the appendix, Table 6, Table 7 and Section B.1 also need significant updates. The proposed approach is no longer a unique entry in Table 6 or 7 -- e.g. [B] already contributed "autoencoder component as regularization for learning predictor" (Table 7). Additionally, toy experiments in Section 5 appear less significant a contribution when multiple full-scale systems already employ TEAs.

This paper's theoretical analysis does appear to set it apart from prior work. However, theorems are developed for an extremely limited context (linear TEAs) and it is unclear whether or how they might extend to practical use cases (i.e. TEAs that are nonlinear, deep neural networks).

---

The extensive author response and updated paper address many of my original concerns.  I have updated my overall rating.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This is an extremely well-written and well-motivated paper. The idea of target-embedding autoencoders is extremely relevant for problems where the dimension of the label space is as large (or larger) than the dimension of the input features. The experiments are thorough, the theoretical guarantees are extremely well thought of and derived. The applications to modelling the progression of cystic fibrosis and Alzheimer's are extremely useful and timely.  I vote for a strong accept for this paper. 

I would like to see some references to the extreme multi-label classification problems (http://manikvarma.org/downloads/XC/XMLRepository.html) and some of the other probabilistic approaches attempted in this domain (please see https://papers.nips.cc/paper/5770-large-scale-bayesian-multi-label-learning-via-topic-based-label-embeddings and the references and citations). 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 1. Summary: In this paper, the authors proposed a Target-Embedding Autoendocer (TEA) model for supervised representation learning. Different from the traditional feature embedding autoencoder model, TEA tries to learn a compact latent representation that can reconstruct the target vector. Hypothetically, this model should be especially useful when the target vector has a much higher dimension than the feature vector. The authors analyzed the proved some characteristics of this framework and conducted empirical experiments on three datasets to prove its effectiveness.
2. Overall assessment: The motivation of this paper is well justified. It's easy to follow and fun to read, even for a person who is not an expert in this area, like me. However, there still exist some problems in this paper. It needs more improvement to get published in a competitive conference like ICLR.
3. Comments:
3.1 Datasets used in this paper cannot fully prove the effectiveness of this framework. These datasets are all from very similar domains. The dimension of target vectors is comparable to that of feature vectors. In my view, it's necessary to test on more different types of datasets to prove the usefulness of a model, especially if it is a general framework like TEA.
3.2 Models used in this paper are relatively simple. Demonstrate the performance of TEA on more advanced models and more difficult tasks can deliver more insights to the community.
3.3 No state-of-the-art models are used in experiments. It's very likely that some existing work has already adopted the idea of target embedding. There also exist much other work on dealing with high dimensional target vector problem. How are the performances of these models? What is the advantage of the proposed framework over these existing work?
3.4 The source of gain part on page 8 should contain more explanations and analysis. This part is one of the most important parts of this paper. It can provide quite valuable insights to readers. I hope the author can expand it.
3.5 More details about training and inference are needed. The authors only use a few sentences to describe their three staged training process. I still have some questions left after reading it, such as how do you train the shared parts in TEA? Do you update its parameters in all stages? What the effect of the order of training? What will happen if I change it?
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This work introduces the idea of target embedding autoencoders for supervised prediction, designed to learn intermediate latent representations jointly optimized to be both predictable from features and predictive of targets. This is meant to help with generalization and has certain theoretical guarantees. 

It is an interesting problem setting to consider where  Y is high dimensional instead of X. More examples of this would be useful to provide in the intro. I think this is crucial to understand where this method might be useful. 

Figure 1 is super informative and very nice!

Section 2: 
Why do we expect that this paradigm of autoencoder based regularization “generalizes” better?
I like the explicit and honest discussion of prior work in this section. 
One question is how important is the choice of reconstruction loss function - L2, vs max likelihood gaussian, vs L1, vs cross entropy, etc for performance?
Another question: how bad is performance if the learning is done stagewise - first the Y-Z-Y^ representation is learned and then the X->Z predictor is learned. 
If something is out of distribution, how easy are TEA based learners to finetune?
Overall the idea seems reasonable - if the targets have some common set of factors, just predict those instead of predicting the full target value which might be harder to get right. It’s just a question of whether this holds true in many domains and how well this reconstruction loss generalizes across problems?

Section 3: 
“We havenoted that TEA components can in principle be instantiated by any architecture. Does its benefit extend beyond the commonly-studied domain of static classification?” -> not clear what this means? Does this mean this algorithm has been proposed before or is it that it can ALSO work on non static classification tasks? Not clear how to situate this claim

The theoretical section seems to follow largely from Le et al, but with important distinctions on dimensionalities of various spaces involved. I wonder if the authors can comment on how often Assumption 1 and 2 are actually satisified?

Related Work: 
Is the main difference between Yu 2014 and this just in the norm based regularization? I don’t think so, can this be made more clear. This seems also fairly related to Yeh, is it just a generalization of that paradigm? Or is there more to it? In light of the contribution of Yeh, this seems like slightly more marginal of a contribution? Is the main points of contribution the theoretical analysis and the extended experiments to sequence data rather than static classification?

The results do seem to show a signficant benefit as compared to FEA or base models. It also seems like this is applicable across multiple disease datasets. Do the authors think that this could be applicable to other domains altogether? Would it be quick to run a comparison on these?

Generally seems like a well grounded and meaningful contribution with many improvements. Would be curious to see applications to other datasets and also some improvements/clarifications noted above?


review_title: Official Blind Review #1

Meta-review: The paper presents a general view of supervised learning models that are jointly trained with a model for embedding the labels (targets), which the authors dub target-embedding autoencoders (TEAs).  Similar models have been studied before, but this paper unifies the idea and studies more carefully various components of it.  It provides a proof for the specific case of linear models and a set of experiments on disease trajectory prediction tasks.  The reviewer concerns were addressed well by the authors and I believe the paper is now strong.  It would be even stronger if it included more tasks (and in particular some "typical" tasks that more of the community is focusing on), and the theoretical part is to my mind not a major contribution, or at least not as large as the paper implies, because it analyzes a much simpler model than anyone is likely to use TEAs for.
Decision: Accept (Talk)