Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:

The paper presents a novel approach to improve the accuracy of regression models that are learned from a skew dataset. The proposed approach consists of two parts, namely, (i) adversarial network for forcing output distributions and (ii) regularization based on an adversarial autoencoder. Experiments suggest that the proposed approach increases the accuracy of the regression model for all the four datasets considered in the paper.


##########################################################################

Pros: 

(1) In practice, the distribution of reported data could be different from the true distribution. The proposed approach is potentially useful for analyzing such data.

(2) This paper provides comprehensive experiments using four datasets. All the experiments agree that the proposed approach increases the accuracy of the regression model and outperforms some existing approaches in terms of RMSE.

(3) The paper is clearly written and the motivation for the study is well explained in the introduction. Figure 1 is helpful to understand the architecture of the proposed approach.


##########################################################################

Cons:

(1) The proposed approach is based on the assumption that labeled data are skewed and unlabeled data follow the assumed true distribution. I am not sure how realistic this assumption is. For instance, the examples discussed in Introduction appear to be the ones in which both labeled and unlabeled data are skewed. Also, in the experiments given in Section 4, all the datasets are not originally skewed, but they are artificially changed to skewed datasets. It would be convincing to provide a real example in which the assumption of the proposed approach holds.

(2) In the second paragraph of Section 4.3, it is claimed that the proposed approach relies strongly on the assumed true distribution. At the same time, it is also mentioned that the proposed approach is not sensitive to error in the assumed true distribution, and is appropriate for practical applications. These two claims seem contradictory to me. More explanations about these statements would be helpful.

(3) It would be ideal if some theoretical results are presented to support the proposed approach. For example, is it possible to provide any theoretical background to support the claim that the proposed approach is not sensitive to error in the assumed true distribution?


##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above.


---

### Updates:

I thank the authors for their response. Some of my concerns are addressed. However, unfortunately, I still think the assumption of the proposed approach is too strong to have broad applications. I will keep my original score.
review_title: Regression method for skewed data

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a semi-supervised learning approach to improve the regression model trained on output-skewed data. The key assumption is that, though the training outputs can be skewed, it is easy to estimate the true distribution of the output. The proposed model that combines an AAE that generates the output distribution, and an adversarial model that enforces the distribution of the predicted output to resemble the true distribution of the output. On several real datasets, the ablation study shows the proposed model can improve the regression accuracy.

The paper is in general well-written. However, the proposed method is straightforward, and a few important questions need to be addressed and/or clarified. 

(1)	 The paper assumes that the training data are often highly skewed (intentionally) but the true distribution of the output can be easily estimated or obtained. That seems to imply that ---  it is easy to fix the data skewness by simply collecting more labeled data. Then why should we use the proposed approach? Can you provide a concrete application where getting more labeled examples is very costly yet estimating the label distribution is easy and cheap? 
(2)	The paper did not explain where the target distribution in the adversarial part of AAE comes from. R_{enc} generates the latent features, which are aimed to fool the discriminator. But which target distribution do you believe the latent features should (approximately) follow?  How does it connect to using the information of the unlabeled data or the true output distribution?
(3)	The evaluation is done by creating skewed training data from existing data. This is more like a simulation. Can you incorporate a real-world application, e.g., drug design (mentioned by the paper), to showcase the performance and usefulness of the proposed approach? 

review_title: Straightfoward method with several problems

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed to learn a regression model using "skewed data", which is defined as the subset of training samples with true target above certain threshold. The model consists of two components. First, the input x was mapped to its latent space through encoder R_enc. The latent representation was further mapped to the predicted output through regressor network R_post. The predictive distribution was forced to match the true target distribution p(y) through an adversarial network. Second, the latent space representations were also forced to match the true target distribution p(y). Experimental results on synthetic benchmark data showed the proposed approach performed better than naively applying regression model on the skewed data.

However, defining the "skewed data" as the subset of training samples whose target values are above certain threshold is an overly strong assumption. In practice, it's more likely that values within certain ranges were over (or under) sampled, leading to mismatch between the empirical target distribution and the true target distribution. This strong assumption makes the proposed approach not applicable to most real-world problems.

Assuming p(y) can be reliably estimated from the training data also seemed problematic. It's not clear how p(y) was estimated from the labeled dataset, where only samples with true target above certain threshold are available.
review_title: The assumption is too strong to be useful in practice

final_score: rating: 6: Marginally above acceptance threshold
review_content: Premise and Contributions:
This work presents a machine learning scenario where the training data is biased due to collection artifacts. The training data does not represent the true real world distribution. In order to overcome this obstacle, the work presents the following contributions:
- A semi-supervised method that uses unlabeled data (that follows the real distribution) in order to improve predictions.
- This method contains an adversarial network that forces the regression output distribution to be similar to the assumed true label distribution.
- This method contains an adversarial autoencoder that propagates the information from the assumed true distribution to the latent vector of the regression model.

Strengths:
- The scenario seems plausible.
- The proposed method seems sound. Although I have a concern with the AAE regularization.
- The experiments are convincing in my opinion, but there are some issues that I talk about below.
- The paper is pretty clear. Although, I believe the scenario is not explained in a way that is simple enough to understand instantly. But of course after reading the paper it becomes clear. Maybe there could be some improvements in the abstract and the introduction. Also I noticed that the introduction repeats the abstract, there could be some more precise explanations instead of this. For example (Kim et al. 2020)?

Issues:
- Section 3.2: "To be a useful feature for Rpost, latent vectors should be arranged in a similar way to p(y), which possesses information about how the labeled dataset is skewed." - p(y) is a distribution of labels right? How can the latent vectors have same distribution as a distribution of labels. I do not understand this. 
- Is there generality with respect to different types of bias for this method? Not just biasing the dataset with labels above $\theta$
- How is the assumed true label distribution selected? What happens when it is very different from the true label distribution? Very interesting experiments could be undertaken here.
- In Section 3.1: "The model is concurrently trained on $D_l$ for regression, and on $D_u$ to force the regression outputs to have similar distribution to p(y)" - How is the network "trained" on $D_u$ is there are no labels? Or is it just running a forward pass?
- A real world application on a dataset that actually presents the issues of this specific scenario would make the value of the contributions much more convincing, instead of only testing on synthetic datasets. But I do not know about the availability of such datasets.
- Experiments: For the ablation study, how many samples are used for the "Only AAE regularization" and "Only forcing output distribution"?

My biggest concern, and a question to other reviewers and the authors: Are there any other comparable methods to the proposed method, or other methods that solve this task in comparable scenarios? Should any other method be included in the experiments section for comparison purposes?

Current decision: 5 - until I read other reviews and understand some of the moving parts better.

UPDATE:

After reading the author's updated paper and comments I have decided to improve my rating to a 6 since some of my concerns have been eased. The most important concern being eased was the type of bias was too constrained at first and now there are experiments with a more unconstrained version of bias that is more convincing.

Overall, I would say that future versions of the paper could look into a task and dataset that are close to their domain of applicability and where they can contribute an increase in performance. That would strengthen the case of this paper. I think a rating of 6 is fair for this version of the paper and I thank the authors for their efforts in updating the work and addressing my concerns directly and efficiently.
review_title: Review

Meta-review: This paper addresses the real-world problem of semi-supervised learning where the distribution from which the labeled examples are drawn is different from the distribution from which the unlabeled examples are drawn.  The task is motivated by structure-activity prediction for drug design (quantitative structure activity prediction, or QSAR).  Examples represent molecules, and we wish to predict a real-valued measure of binding affinity.  Exactly the general problem of data skew arose with exactly this task for example in one of the KDD Cup 2001 tasks.  While the authors here mention that labeled data may be focused more on active molecules (those with a high continuous-valued response), in the KDD Cup 200`1 data the reverse was true, and the unlabeled test data were skewed to higher activity level.  I say all this to agree with the authors about the real-world nature of the problem they address.  Also, some reviewers felt more empirical evaluation was needed, so that may be an additional data set for the authors to consider using.

Reviewer concerns including that the approach was simplistic, the empirical results were insufficient, and the claims were oversold.  The author replies and revisions, and the discussion, moved the reviews to be more favorable but still not strong enough to justify acceptance yet.  Nevertheless, the consensus is that the paper addresses an important problem and the revisions are headed in the right direction to make a strong future paper, and that the authors should be encouraged to continue this work.
Decision: Reject