Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper addresses the question of identifying 'shared features' in neural networks trained on different datasets.  Concretely, suppose you have two datasets X1, X2 and you would like to train auto-encoders (with potential augmentation with labeled examples) for the two datasets. One could work on the two separately; here, the authors propose sharing some of the weights to try and exploit/identify common features between the two datasets. The authors formalize by essentially looking to optimize an auto-encoder that take inputs of the form (x1, x2) and employing architectures that allow few nodes to interact with both x1,x2. The authors then try to minimize an appropriate loss function by standard methods. 

The authors then apply the above methodology to transfer learning between various datasets. The empirical results here are interesting but not particularly striking; the most salient feature perhaps is that the architectures and training algorithms are perhaps a bit simpler but the overall improvements over existing methods are not too exciting. 
review_title: The paper proposes a model for allowing various deep neural network architectures to share weights (parameters) across different datasets. The authors then apply the framework to transfer learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: 

The paper focuses on learning common features from multiple domains data in a unsupervised and supervised learning scheme. Setting this as a general multi task learning, the idea consists in jointly learning autoecnoders, one for each domain, for the multiples domain data in such a way that parts of the parameters of the domain autoencoder are shared. Each domain/task autoencoder  then consists in a shared part and a private part. The authors propose a variant of the model in the case of supervised learning and end up with a general architecture for multi-task, semi-supervised and transfer learning.

The presentation of the paper is good and the paper is easy to follow and explores the rather intuitive and simple idea of sharing parameters between related tasks.

Experimental show some interesting results. First unsupervised experiments on Mnist data show improved MSe of joint autoecnoders but are these differences really significant (e.g. from 0.56 to 5.52) ?  Moreover i am not sure to understand the meaning of separation criterion computed on t-sne of hidden representations. Results of Table 1 show improved reconstruction performance (MSE?) of joint auto encoders over independent ones for unrelated pairs such as airplane and horses. I a not sure ti understand why this improvement occurs even with very different classes.    The investigation on the depth where sharing should occur is quite interesting and related to the usual idea of higher transferable property low level features. Results on transfer are the most interesting ones actually but do not seem to improve so much over baselines. 


review_title: An appealing architecture for domain adaptation, multitask, and transfer learning but without strong enough results 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The work proposed a generic framework for end-to-end transfer learning / domain adaptation with deep neural networks. The idea is to learn a joint autoencoders, containing private branch with task/domain-specific weights, as well as common branch consisting of shared weights used across tasks/domains, as well as task/domain-specific weights.  Supervised losses are added after the encoders to utilize labeled samples from different tasks. Experiments on the MNIST and CIFAR datasets showed improvements over baseline models. Its performance is comparable to / worse than several existing deep domain adaptation works on the MNIST, USPS and SVHN digit datasets.

The structure of the paper is good, and easy to read.  The idea is fairly straight-forward. It reads as an extension of "frustratingly easy domain adaptation" to DNN (please cite this work). Different from most existing work on DNN for multi-task/transfer learning, which focuses on weight sharing in bottom layers, the work emphasizes the importance of weight sharing in deeper layers. The overall novelty of the work is limited though. 

The authors brought up two strategies on learning the shared and private weights at the end of section 3.2. However, no follow-up comparison between the two are provided. It seems like most of the results are coming from the end-to-end learning. 

Experimental results:
section 4.1: Figure 2 is flawed. The colors do not correspond to the sub-tasks. For example, there are digits 1, 4 in color magenta, which is supposed to be the shared branch of digits of 5~9. Vice versa. 
From reducing the capacity of JAE to be the same as the baseline, most of the improvement is gone. It is not clear how much of the improvement will remain if the baseline model gets to see all the samples instead of just those from each sub-task. 

section 4.2.1: The authors demonstrate the influence of shared layer depth in table 2. While it does seem to matter for tasks of dissimilar inputs, have the authors compare having a completely shared branch or sharing more than just a single layer?

The authors suggested in section 4.1 CIFAR experiment that the proposed method provides more performance boost when the two tasks are more similar, which seems to be contradicting to the results shown in Figure 3, where its performance is worse when transferring between USPS and MNIST, which are more similar tasks vs between SVHN and MNIST. Do the authors have any insight?
review_title: Review

Meta-review: Thank you for submitting you paper to ICLR. ICLR. The consensus from the reviewers is that this is not quite ready for publication. In particular, the experimental results are promising, but further work is required to fully demonstrate the efficacy of the approach.
Decision: Reject