Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: (Summary)
This paper tackles the cross-task and cross-domain transfer and adaptation problems. The authors propose learning to output a probability distribution over k-clusters and designs a loss function which encourages the distributions from the similar pairs of data to be close (in KL divergence) and the distributions from dissimilar pairs of data to be farther apart (in KL divergence). What's similar vs dissimilar is trained with a binary classifier.

(Pros)
1. The citations and related works cover fairly comprehensive and up-to-date literatures on domain adaptation and transfer learning.
2. Learning to output the k class membership probability and the loss in eqn 5 seems novel.

(Cons)
1. The authors overclaim to be state of the art. For example, table 2 doesn't compare against two recent methods which report results exactly on the same dataset. I checked the numbers in table 2 and the numbers aren't on par with the recent methods. 1) Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks, Bousmalis et al. CVPR17, and 2) Learning Transferrable Representations for Unsupervised Domain Adaptation, Sener et al. NIPS16. Authors selectively cite and compare Sener et al. only in SVHN-MNIST experiment in sec 5.2.3 but not in the Office-31 experiments in sec 5.2.2.
2. There are some typos in the related works section and the inferece procedure isn't clearly explained. Perhaps the authors can clear this up in the text after sec 4.3.

(Assessment)
Borderline. Refer to the Cons section above.
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: pros:
This is a great paper - I enjoyed reading it. The authors lay down a general method for addressing various transfer learning problems: transferring across domains and tasks and in a unsupervised fashion. The paper is clearly written and easy to understand. Even though the method combines the previous general learning frameworks, the proposed algorithm for  LEARNABLE CLUSTERING OBJECTIVE (LCO) is novel, and fits very well in this framework.  Experimental evaluation is performed on several benchmark datasets - the proposed approach outperforms state-of-the-art for specific tasks in most cases. 

cons/suggestions: 
- the authors should discuss in more detail the limitations of their approach: it is clear that when there is a high discrepancy between source and target domains, that the similarity prediction network can fail. How to deal with these cases, or better, how to detect these before deploying this method?
- the pair-wise similarity prediction network can become very dense: how to deal with extreme cases?
review_title: The authors propose a novel framework for task-/domain- transfer in an unsupervised setting (no labels for target data). The idea of defining the similarity based-clustering within domain adaptation /transfer learning framework is novel. Experiments are thorough and show clear benefits of the proposed learning strategy.  

final_score: rating: 7: Good paper, accept
review_content: The authors propose a method for performing transfer learning and domain adaptation via a clustering approach. The primary contribution is the introduction of a Learnable Clustering Objective (LCO) that is trained on an auxiliary set of labeled data to correctly identify whether pairs of data belong to the same class. Once the LCO is trained, it is applied to the unlabeled target data and effectively serves to provide "soft labels" for whether or not pairs of target data belong to the same class. A separate model can then be trained to assign target data to clusters while satisfying these soft labels, thereby ensuring that clusters are made up of similar data points. 

The proposed LCO is novel and seems sound, serving as a way to transfer the general knowledge of what a cluster is without requiring advance knowledge of the specific clusters of interest. The authors also demonstrate a variety of extensions, such as how to handle the case when the number of target categories is unknown, as well as how the model can make use of labeled source data in the setting where the source and target share the same task.

The way the method is presented is quite confusing, and required many more reads than normal to understand exactly what is going on. To point out one such problem point, Section 4 introduces f, a network that classifies each data instance into one of k clusters. However, f seems to be mentioned only in a few times by name, despite seeming like a crucial part of the method. Explaining how f is used to construct the CCN could help in clarifying exactly what role f plays in the final model. Likewise, the introduction of G during the explanation of the LCO is rather abrupt, and the intuition of what purpose G serves and why it must be learned from data is unclear. Additionally, because G is introduced alongside the LCO, I was initially misled into understanding was that G was optimized to minimize the LCO. Further text explaining intuitively what G accomplishes (soft labels transferred from the auxiliary dataset to the target dataset) and perhaps a general diagram of what portions of the model are trained on what datasets (G is trained on A, CCN is trained on T and optionally S') would serve the method section greatly and provide a better overview of how the model works.

The experimental evaluation is very thorough, spanning a variety of tasks and settings. Strong results in multiple settings indicate that the proposed method is effective and generalizable. Further details are provided in a very comprehensive appendix, which provides a mix of discussion and analysis of the provided results. It would be nice to see some examples of the types of predictions and mistakes the model makes to further develop an intuition for how the model works. I'm also curious how well the model works if, you do not make use of the labeled source data in the cross-domain setting, thereby mimicking the cross-task setup.

At times, the experimental details are a little unclear. Consistent use of the A, T, and S' dataset abbreviations would help. Also, the results section seems to switch off between calling the method CCN and LCO interchangeably. Finally, a few of the experimental settings differ from their baselines in nontrivial ways. For the Office experiment, the LCO appears to be trained on ImageNet data. While this seems similar in nature to initializing from a network pre-trained on ImageNet, it's worth noting that this requires one to have the entire ImageNet dataset on hand when training such a model, as opposed to other baselines which merely initialize weights and then fine-tune exclusively on the Office data. Similarly, the evaluation on SVHN-MNIST makes use of auxiliary Omniglot data, which makes the results hard to compare to the existing literature, since they generally do not use additional training data in this setting. In addition to the existing comparison, perhaps the authors can also validate a variant in which the auxiliary data is also drawn from the source so as to serve as a more direct comparison to the existing literature.

Overall, the paper seems to have both a novel contribution and strong technical merit. However, the presentation of the method is lacking, and makes it unnecessarily difficult to understand how the model is composed of its parts and how it is trained. I think a more careful presentation of the intuition behind the method and more consistent use of notation would greatly improve the quality of this submission.

=========================
Update after author rebuttal:
=========================
I have read the author's response and have looked at the changes to the manuscript. I am satisfied with the improvements to the paper and have changed my review to 'accept'. 
review_title: New method with a rough presentation

Meta-review: Pros
-- A novel formulation for cross-task and cross-domain transfer learning.
-- Extensive evaluations.

Cons
-- Presentation a bit confusing, please improve.

The paper received positive reviews from reviewers. But the reviewers pointed out some issues with presentation and flow of the paper. Even though the revised version has improved, the AC feels that it can be improved further. For example, as pointed out by reviewers, different parts of the model are trained using different losses and / or are pre-trained. It would be worth clarifying that. It might help if the authors include a pseudocode / algorithm block to the final version of the paper.
Decision: Accept (Poster)