Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1. p. 5, 3.3, What is the purpose of the second paragraph? I'm not sure I understand the link between this one and the previous one.

2. p. 6, Table 1, What is the evaluation protocol for the Office dataset? DAN results are for the transductive setting; RevGrad results differ from the ones in the ICML paper (also transductive setting)

3. The authors are using Inception BN as a base model while the other competing methods are built on top of AlexNet. Could that happen that those methods would also benefit from having a stronger base model. Is it possible to conduct a fair comparison?
review_title: Questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overall I think this is an interesting paper which shows empirical performance improvement over baselines. However, my main concern with the paper is regarding its technical depth, as the gist of the paper can be summarized as the following: instead of keeping the batch norm mean and bias estimation over the whole model, estimate them on a per-domain basis. I am not sure if this is novel, as this is a natural extension of the original batch normalization paper. Overall I think this paper is more fit as a short workshop presentation rather than a full conference paper.

Detailed comments:

Section 3.1: I respectfully disagree that the core idea of BN is to align the distribution of training data. It does this as a side effect, but the major purpose of BN is to properly control the scale of the gradient so we can train very deep models without the problem of vanishing gradients. It is plausible that intermediate features from different datasets naturally show as different groups in a t-SNE embedding. This is not the particular feature of batch normalization: visualizing a set of intermediate features with AlexNet and one gets the same results. So the premise in section 3.1 is not accurate.

Section 3.3: I have the same concern as the other reviewer. It seems to be quite detatched from the general idea of AdaBN. Equation 2 presents an obvious argument that the combined BN-fully_connected layer forms a linear transform, which is true in the original BN case and in this case as well. I do not think it adds much theoretical depth to the paper. (In general the novelty of this paper seems low)

Experiments:

- section 4.3.1 is not an accurate measure of the "effectiveness" of the proposed method, but a verification of a simple fact: previously, we normalize the source domain features into a Gaussian distribution. the proposed method is explicitly normalizing the target domain features into the same Gaussian distribution as well. So, it is obvious that the KL divergence between these two distributions are closer - in fact, one is *explicitly* making them close. However, this does not directly correlate to the effectiveness of the final classification performance.

- section 4.3.2: the sensitivity analysis is a very interesting read, as it suggests that only a very few number of images are needed to account for the domain shift in the AdaBN parameter estimation. This seems to suggest that a single "whitening" operation is already good enough to offset the domain bias (in both cases shown, a single batch is sufficient to recover about 80% of the performance gain, although I cannot get data for even smaller number of examples from the figure). It would thus be useful to have a comparison between these approaches, and also a detailed analysis of the effect from each layer of the model - the current analysis seems a bit thin.
review_title: An interesting paper that shows improvements, but I am not sure about its technical advantage

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a simple domain adaptation technique in which batch normalization is performed separately in each domain.


Pros:

The method is very simple and easy to understand and apply.

The experiments demonstrate that the method compares favorably with existing methods on standard domain adaptation tasks.

The analysis in section 4.3.2 shows that a very small number of target domain samples are needed for adaptation of the network.


Cons:

There is little novelty -- the method is arguably too simple to be called a “method.” Rather, it’s the most straightforward/intuitive approach when using a network with batch normalization for domain adaptation.  The alternative -- using the BN statistics from the source domain for target domain examples -- is less natural, to me. (I guess this alternative is what’s done in the Inception BN results in Table 1-2?)

The analysis in section 4.3.1 is superfluous except as a sanity check -- KL divergence between the distributions should be 0 when each distribution is shifted/scaled to N(0,1) by BN.

Section 3.3: it’s not clear to me what point is being made here.


Overall, there’s not much novelty here, but it’s hard to argue that simplicity is a bad thing when the method is clearly competitive with or outperforming prior work on the standard benchmarks (in a domain adaptation tradition that started with “Frustratingly Easy Domain Adaptation”).  If accepted, Sections 4.3.1 and 3.3 should be removed or rewritten for clarity for a final version.
review_title: trivially simple yet effective

final_score: rating: 4: Ok but not good enough - rejection
review_content: Update: I thank the authors for their comments. I still think that the method needs more experimental evaluation: for now, it's restricted to the settings in which one can use pre-trained ImageNet model, but it's also important to show the effectiveness in scenarios where one needs to train everything from scratch. I would love to see a fair comparison of the state-of-the-art methods on toy datasets (e.g. see (Bousmalis et al., 2016), (Ganin & Lempitsky, 2015)). In my opinion, it's a crucial point that doesn't allow me to increase the rating.

This paper proposes a simple trick turning batch normalization into a domain adaptation technique. The authors show that per-batch means and variances normally computed as a part of the BN procedure are sufficient to discriminate the domain. This observation leads to an idea that adaptation for the target domain can be performed by replacing population statistics computed on the source dataset by the corresponding statistics from the target dataset.

Overall, I think the paper is more suitable for a workshop track rather than for the main conference track. My main concerns are the following:

1. Although the main idea is very simple, it feels like the paper is composed in such a way to make the main contribution less obvious (e.g. the idea could have been described in the abstract but the authors avoided doing so). 

2. (This one is from the pre-review questions) The authors are using much stronger base CNN which may account for the bulk of the reported improvement. In order to prove the effectiveness of the trick, the authors would need to conduct a fair comparison against competing methods. It would be highly desirable to conduct this comparison also in the case of a model trained from scratch (as opposed to reusing ImageNet-trained networks).

review_title: Final review

Meta-review: The paper performs domain adaptation using a very simple trick inspired by BatchNorm. The paper received below margin scores. The reviewers both, liked the simplicity of the approach, and at the same time felt that the contribution was too thin. Given the high bar of ICLR, this paper falls short.
Decision: Reject