Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary
-------
This paper presents a novel theoretical analysis for unsupervised domain adaptation by revisiting the \lambda joint error term charactering adaptability in the seminal analysis of Ben-David et al. They propose to replace it by considering discrepancy between information on constrained class hypothesis and a possible discrepancy term that related the learned model with the class A cross-margin discrepancy is proposed in the multiclass context. They extend their approach by proposing to reweight differently some errors with an expected performance of target models on source and an additional one where they distinguish the performance with respect to accuracy on pseudo-labeled target data. Their approach leads to an adversarial-based loss function to optimize which is evaluated on two visual domain adaptation tasks. 

Evaluation
--------
The idea is novel and I find the discussion on the considered restriction on the hypothesis class interesting. The experimental evaluation is interesting with good results reported on the two problems considered. I think that some parts could be improved in terms of presentation, in particular The paper contains many typos that make sometimes the reading difficult.

Other comments
------------

-The comparison with other existing bounds is interesting. I think the authors should expand this by also taking into consideration the bound of Mansour et al., COLT'09 which has some links with the proposed approach (check their comparison to Ben-David's bound). 
The links with this bound and the one of Ben-David could also be summarized in an appendix, I would like to see a better characterization of the cases where the proposed bound is better and worse. 

-About original proposal. I am wondering if the authors could discuss the relationship between the value of \gamma and the expressiveness of the considered model. If the model is powerful enough, \gamma should certainly be large. In a context of a training with "Learning without forgetting" strategy, the performance on source could maintained in a high standard. 
Note that in this context, the classic joint error of Ben-David et al. can be considered as rather small.

-About the alternative proposal: I am a bit skeptical on the ration \eta and (1-\eta) for accuracy on source and pseudo-labeled data respectively. Indeed, since the pseudo-labels are obtained from a classifier that make use a lot of source information, one may think that their performance is rather related. So using to correlated ratios would probably be more relevant here, but maybe the authors can bring some arguments against.

-In the experimental evaluation, the tuning of the different parameters, in particular \gamma and \mu, is not particularly discussed and there is probably an issue. I tend to think that these values are rather difficult to assess. A discussion on this point would be welcomed.

-I am not sure to understand the optimization problem (8), (9) and (18), in particular the term after the "s.t.": I would expect an inequality somewhere, otherwise everything can be added to the general objective function. If there is an alternate optimization scheme, this should be mentioned explicitly.

-Table 1 and Table 2: use a third identifier different from the second for the last version of your method.

-Please check the typos.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose an approach based on an upper bound on target domain error for the task
of unsupervised domain adaptation (where one does not have access to
any labels in the target domain). The upper bound makes possible the
penalization of mixing samples from different classes together during
distribution matching.  Empirical results on image classification in a
few benchmarks show the significant promise of the approach.


The paper makes a good attempt at explaining the ideas/concepts, but
still remains very  unpolished (including English usage issues), and it is
hard to follow in a number of places. A few such, a sampling, are mentioned below with
some suggestions. In my opinion, 
not ready for publication.

* abstract: '.. to address the problem for unsupervised domain
  adaptation.' >> 'to address a major problem facing many unsupervised
  domain adaptation techniques'.

* page 2, several rewordings in: "The reason is obvious, as marginal
  distributions being matched for source and target, it is possible
  that samples from different classes are aligned together, where the
  joint error becomes non-negligible since no hypothesis can classify
  source and target at the same time." For instance, a partial
  rewording: '.. when an attempt is made to match marginal
  distributions of source and target domains, samples from different
  classes can be mixed together'. Also, I wouldn't use "The reason is
  obvious"...

Finally, here it is a good place to refer to Figure 1.



* page 2: 'our proposal can degrade to some other methods' >> '.. can reduce
  to several other methods ..'



* in lines 1 and 2 on pg 3, the simplification from line 1 to 2,
  explain the major reasoning (perhaps in the appendix if you don't
  have space): there are number of terms added and subtracted (which
  is understandable), but hard to keep track of what simplifications
  are being carried and where the terms move (too many terms)...

* 'the following theorem holds' >> 'the following bound holds' (or
  inequality, etc.)

* Is derivation 3 from triangle inequality?  (add that explanation to
  line 3)

* hard to parse (missing pronoun): 'the above upper bound in minimized when h=f_s thus
  equivalent to ..'

* 'is capable of' >> 'is capable to do so' (and at this point, is not
  yet clear how the bound helps avoid the problems with distribution
  matching.. )

* Figure 1 (and subsequent figures): suggest put A, B and A' and B'
  for the two classes and domains, inside the circles, so it's easier to see what
  the source and target domain classes are (dotted boundary is hard to discern).


* on top of pg 4: couldn't the feature extractor make the max-player
  stronger too? ( In "... since the max-player taking two parameters
  f1 , f2 is too strong, we introduce a feature extractor g to make
  the min-player stronger.. " ..   )


review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper introduces a new upper bound of unsupervised domain adaptation, which takes the adaptability term lambda into consideration. The new theory can be expanded into a novel algorithm. Experiments on domain adaptation datasets demonstrate improvement over previous state-of-the-art methods.
The authors propose to incorporate lambda into adversarial feature learning. Specifically, the authors assume that f_s and f_t are from some hypothesis space H. Then relaxing f_s and f_t to f_1 and f_2, we can turn the problem into a minimax game between f_1, f_2 and feature extractor g. To further implement their method, the authors propose to constrain f_1 and f_2 with source accuracy and target pseudo label accuracy. Based on the margin theory, the authors also introduce the cross margin discrepancy, which increase the reliability of adversarial adaptation.
The paper is well-written and the contributions are stated clearly. The attempt to incorporate lambda into feature learning is really interesting.

However, I have several concerns:
*The proposed theory of equation (4), (5), and (6) is problematic. h is the hypothesis which belongs to a hypothesis class H. f_s and f_t are true labeling functions, and do not necessarily belong to the hypothesis space H. In this sense, the inequality of equation (4) does not hold. Problems of equation (5) and (6) are similar. The authors do realize that the supremum term can be arbitrarily large and put constraints to f_1 and f_2. But no matter what hypothesis class we are using, it generally does not contain the true labeling functions, and what we can do is only approximating them. Thus, in spite of the good performance of the proposed method, the proposed upper bound is not reliable.
*Lack of experimental results on the role of f_1 and f_2. The proposed method demonstrates good performance, but the manuscript does not provide some experimental results on the source of performance gain. In particular, how is f_1 and f_2 changed during training? Are they substantially different from the h’in MCD and MDD? Besides, how does each part contribute to the performance gain? Is it from the novel loss function or just the new adversarial adaptation method itself? A proper ablation study would be helpful. 

review_title: Official Blind Review #1

Meta-review: Given two distributions, source and target, the paper presents an upper bound on the target risk of a classifier in terms of its source risk and other terms comparing the risk under the source/target input distribution and target/source labeling function. In the end, the bound is shown to be minimized by the true labeling function for the source, and at this minimum, the value of the bound is shown to also control the "joint error", i.e., the best achievable risk on both target and source by a single classifier. 

The point of the analysis is to go beyond the target risk bound presented by Ben-David et al. 2010 that is in terms of the discrepancy between the source and target and the performance of the source labeling function on the target or vice versa, whichever is smaller. Apparently, concrete domain adaptation methods "based on" the Ben-David et al. bound do not end up controlling the joint error. After various heuristic arguments, the authors develop an algorithm for unsupervised domain adaptation based on their bound in terms of a two-player game.

Only one reviewer ended up engaging with the authors in a nontrivial way. This review also argued for (weak) acceptance. Another reviewer mostly raised minor issues about grammar/style and got confused by the derivation of the "general" bound, which I've checked is ok. The third reviewer raised some issues around the realizability assumption and also asked for better understanding as to what aspects of the new proposal are responsible for the improved performance, e.g., via an ablation study.

I'm sympathetic to reviewer 1, even though I wish they had engaged with the rebuttal. I don't believe the revision included any ablation study. I think this would improve the paper. I don't think the issues raised by reviewer 3 rise to the level of rejection, especially since their main technical concern is due to their own confusion. Reviewer 2 argues for weak acceptance. However, if there was support for this paper, it wasn't enough for reviewers to engage with each other, despite my encouragement, which was disappointing.
Decision: Reject