Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed to address a compound problem where missing data and distribution shift are both at play.  The paper goes on to describe some heuristic methods that resemble the gradient reversal methods due to Ganin et al for handling both problems. 

The novel part of the paper over DANNs is the joint, end-to-end training of latent representations for missing data,  While it is sloppy with terminology, the paper is overall reasonably easy to follow although it might mislea a novice reader and sufficient details are provided to replicate their results.  

The major problem here is the problem appears to be underspecified, and its not clear under what conditions if any the proposed methods are valid. Moreover it’s not clear to what extent the experimental results should ameliorate these concerns. 

If the data is not missing at random then there is presumably confounding. The authors dance around this topic, just asserting that they are handling non-stochastic missing data but do not say precisely what is assumed about the relationship between the observed and missing data. 

In short the paper addresses an under-specified problem with a heuristic technique based upon domain-adversarial nets which have recently been shown have a number of fundamental flaws. It's never made clear under what assumptions this proposed procedure is valid and the paper misrepresents the prior work on lable shift, including the theoretically sound work, e.g.:

"we assume covariate shift as in most UDA papers e.g. Ben-David et al. (2010); Ganin & Lempitsky (2015).”
>>>  Ben-David 2010 is not about covariate shift ….

Some minor thoughts:

“some components of the target data are systematically absent”
>>> 	Not clear what “component” means at this point

“We propose a way to impute non-stochastic missing data”
>>> 	What does this mean? Is non-stochastic, not missing at random? What is the pattern of missing-ness conditioned on? What assumption, if any, is made? 

“This key property allows us to handle non-stochastic missing data,” 
>>> 	again what precisely does this mean?

“Consider that x has two components (x_1, x_2)…”
>>>	sloppy  notation:
	“Source features” x_s = (x_S1, x_S2) are always available 


I read the author's reply but do not believe that the responses are satisfactory. The authors do not address the primary concerns clearly and do not point to specific improvements in the draft that might cause me to change my mind.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: *Summary.* The paper presents and addresses the problem of performing domain adaptation when the target domain is systematically (i.e., not the result of a stochastic process) missing subsets of the data. The issue is motivated by applications where one modality of data becomes unavailable in the target domain (e.g., when deciding which ads to serve to new users, the predictor may have access to behavior across other websites but not on a specific merchant's website). The proposed method learns to map source and target data to a latent space where the representations for the source and target are aligned, the missing components of the target can be inferred, and classification can be performed successfully. These are achieved by adversarial/optimal transport loss on source and target features, a mean-squared error and adversarial loss on latent generation/imputation, and a cross entropy loss on source label prediction, respectively. Experiments are performed on digits and click-through rate (CTR) prediction and include a thorough set of baselines/oracles for comparison.

*Review.* While the problem statement is novel, I am unconvinced that the advertising experiment includes both a domain adaptation and imputation problem. I describe this in detail below. For this reason, I am giving the paper a weak reject.

*Questions that impacted rating.*
1. Ads experiment: From my understanding, the source domain is the traffic of users who have interacted with (clicked through to?) a specific partner and the target domain is the traffic of the users who have not interacted with that specific partner. The data that needs to be imputed is the click through rate for target users with that specific partner. In this case, it is not obvious to me why there is a domain shift between these two groups of users. This would imply that the traffic of source users and target users is different for other partners. I don't see why this would need to be true. Could the authors provide an explanation as to why this is the case (e.g., by showing that CTRs differ with other (partner, publisher) pairs between source and target). From my understanding, Table 5 only shows CTR averaged across all users in each domain, but does not show that the CTRs differ between source and target users for contexts/(partner, publisher) pairs (i.e., the results in table 5 could be due to the fact that the prior distribution over context is different for source and target users).

*Additional notes. Immaterial to rating.*
1. I personally felt that the motivation for UDA vs imputation in the first paragraph was a bit muddled. I think sticking to one example would make the motivation more clear to the reader. E.g., explain the prediction problem for medical imaging (which I assume is disease diagnosis, but it is not stated explicitly), describe how some medical imaging may be missing for certain patients (imputation), then explain that there may be noise across different medical imaging systems (UDA), then list the other applications where this arises with citations (e.g., These phenomena have also been documented in advertising applications [1], ...).
2. I was surprised by the difference between Adaptation-Partial and the other two train/test conditions in Figure 2 when p=30%. Out of curiosity, do the authors have an explanation for this discrepancy? I would have predicted that, if most of the information necessary for prediction was available in the remaining 70% of the image that the performance of these cases would be very similar.  I think it would be helpful to see the accuracy on the source domain and the labeled target domain to better understand that result.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The submission describes an approach for unsupervised domain adaptation in a setting where some parts of the target data are missing.

Both UDA approaches as well as data completion approaches have a sizable research history, as laid out in the related work section (Section 5). The novelty here comes from the properties that a) domain adaptation and data imputation are handled in a joint manner, b) the missing data in the target domain is non-stochastic, and c) imputation is performed in a latent space. This maps to a fairly specific, but realistic enough set of real-world problems; the authors give an image recognition as well as an advertising prediction related problem as experimental examples.

The submission is overall well written and easy to understand. I'd rate the novelty as medium (smart combination of existing methods), but the exemplary experimental evaluation elevates it to more than a systems paper.

The method is described clearly in Section 3, and the joint training makes sense. I notice that not all hyperparameters ({lambda_adv, lambda_mse}, {lambda_1, lambda_2, lambda_3}) are truly needed. lambda_adv and lambda_1 could be canonically set to 1 for such a loss minimization problem, so why are the extraneous parameters included?

In addition to Section 3, the experimental evaluation on two very different data sets in Section 4 is highly detailed and describes the insights clearly, both qualitatively and quantitatively. I'm happy that mean standard deviations are reported on an acceptable experiment sample set size.
Regarding the different approaches: I'm wondering whether the higher performance of the ADV approach over OT (or the parameter hunger of OT over ADV) is only due to the tuning of the network architectures, or whether this is due to the approximations described in B.1.
The ablation study in Section 4.4 is interesting w.r.t. the trade-off it shows between stable, consistent, "average" results from an MSE loss term, vs. high-variance (and on average better) results when a choice of mode is forced using an adversarial loss term.

Minor comments:
- In Table 2, I am not sure what the first row ('Naive') refers to. As far as I can tell, it is not referenced in the text.
- I would move Section 5 (related work) to right after the introduction, as is common in conference papers and makes for smoother reading.
- Section 5.2: type "impainting" -> "inpainting"
- Appendix, section 'Pre-processing': It seems to me that there is a clear assumption made that the target set is balanced, since training happens with a balanced source set. Is this realistic in practical scenarios? There is work on DA with unequal class distributions between domains.

In summary, I can clearly recommend this submission for publication.
review_title: Official Blind Review #3

Meta-review: This paper addresses the problem of performing unsupervised domain adaptation when some target domain data is missing is a potentially non-stochastic way. The proposed solution consists of applying a version of domain adversarial learning for adaptation together with an MSE based imputation loss learned using complete source data. The method is evaluated on both the standard digit recognition datasets and a real-world advertising dataset. 

The reviewers had mixed recommendations for this work, with two recommending weak reject and one recommending acceptance. The key positive point from R3 who recommended acceptance was that this work addresses a new problem statement which may be of practical importance. The other two reviewers expressed concerns over the contribution of the work and the validity of the problem setting. Namely, both R2 and R4 had significant confusion over the problem specification and/or under what conditions the proposed setting is valid. 

It is a difficult decision for this paper as there is a core disagreement between the reviewers. All reviewers seem to agree that the proposed solution is a combination of prior methods in a new way to address the specific problem setting of this work.  However, the reviewers differ in precisely whether they determine the proposed problem setting to be valid and justified. Due to this discrepancy, the AC does not recommend acceptance at this time. If the core contribution is to be an application of existing techniques to a new problem statement than that should be clarified and motivated further. 

Decision: Reject