Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The motivation is to increase accuracy of CNNs with unseen (unknown) distribution shifts. To this end, instead of the recent approach of training-time self-supervision, they adopted it for test-time [limited novelty]. More precisely, for a classification task, they considered two headed neural network with one head for main classification task and another for an auxiliary task (e.g. predicting rotation degree of rotated images). The feature extractor (up to k layers) is shared between these tasks thus updated using the two tasks, while two heads are updated according to each task. In test-time, the samples (drawn from shifted data distributions) are being used for updating the shared feature extractor through the auxiliary task. They have investigated their test-time approach in a series of experiments in online and offline settings on synthetic shift in data distribution of image classification tasks. The distributional shifts are synthesized adding some non-adversarial perturbation to clean images. The authors proof that their test-time approach can lead to lower error rate on given test-time samples when the underlying learning model is a linear regression model. 
The authors incorrectly interchange OOD with domain shifts in their manuscript. The common usage of OOD set is to capture novel samples that are not properly following the training set distribution, e.g. from different concepts than the set of classes given in the training set. With an OOD set, a robust model dealing with it should be able to detect whether an instance is in-distribution or OOD, making a special decision for the latter case (e.g., rejecting the instance).
In domain shift, we look at the scenario where test objects are the same as used for training the model (i.e., correspond to one of the classes the model is processing), but might be perturbed or coming having a different distribution (e.g. SVHN for MNIST or vice versa). The approaches for domain shift concern to improve robustness of CNNs to such shifts in data distribution. Accordingly, the title of the paper inaccurately reflects of the claim of the paper and is misleading, this paper is not on learning with out-of-distribution instances.
The other important point is about catastrophic forgetting phenomena in online setting of their approach, which was not addressed thoroughly in the paper. How not to forget what the model has previously learnt a test-time training? I see this somewhat has been empirically shown in Fig 2 with accuracy on the original data, but what is the mechanism not to forget what have being learned so far?
Besides generalization enhancement, the advantage of test-time self-supervision over training-time joint self-supervised is not clear for the readers, particularly considering the problem of the pitfall of catastrophic forgetting phenomena in test-time training. This pitfall does not exist for training-time joint self-supervised approach. What are the advantages of this approach?
The claims (about synthetic shift in distribution shift) are well supported in a series of experiments, where the distribution shifts are synthesized using adding some non-adversarial perturbation to clean images. However, the other common experiments on real shifts in distribution (e.g. SVHN for MNIST and vice versa, and those performed in Volpi et al  2018) are missing, which can help the paper to being better supported and justified for its practicality.
[Volpi et al  2018]: Generalizing to Unseen Domains via Adversarial Data Augmentation, NIPS 2018
I found the paper rather difficult to follow and not very coherent in its organization. The main idea is fundamentally simple, but it is still difficult to get it from the text. It needed me 2-3 readings before really getting the point of the paper.
** Update ** I read other reviews and comments. The answer of the authors to my comments are somehow satisfactory, especially the point of changing from "out-of-distribution" to "domain shift", which avoid some confusion. I upgraded my rating to a "weak accept".
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes test time training, a method that uses an auxiliary task to provide a kind of loose supervision during test time. I say loose supervision because the theory suggests this is only useful when the gradients of the main and auxiliary losses are positively correlated.

I'm not sure I totally believe that this is a method of out-of-distribution generalization, but rather it helps adjust for corruptions and modest dataset shifts which is an important problem itself. I suspect test-time training is fundamentally better suited for the latter because in general the assumption of correlation between the loss gradients cannot hold in the test if we allow for large shifts (like the airplane class in the video experiment). I note that the authors are aware of this limitation (page 5, last paragraph).

My main problem with this paper is that the more fine-tuned labels get (like the density of a tumor), the harder it gets to create auxiliary tasks. This will be a significant problem when the samples at test-time only share the highest level common characteristics with the true dataset; (like rotations do not impact or density of tumor, like face detection and similar fine-feature-based tasks).

That said, I do appreciate the experimental results which show promise; especially the CIFAR-10.1 results. So I'm inclined toward accepting this paper. I would be more so inclined if the authors could provide a reasonable categorization of tasks where this method is expected to be applicable. I ask this because, for practitioners, it is near impossible to verify the positive correlation of gradients assumption. If there are high-level targets that the distribution and/or task must satisfy that can act as indicators for applying this method, I believe that would be a valuable addition to the paper.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors propose a method for adapting model parameters by doing self-supervised training on each individual test example. They show striking improvements in out-of-domain performance across a variety of image classification tasks while preserving in-domain performance; the latter is a marked difference from other robustness procedures which tend to sacrifice in-domain performance for out-of-domain (or adversarial) performance. These results are exciting, and I believe that this proposed test-time training method will spur a significant amount of further research into similar approaches.

The paper is well-written and the experiments are thorough, so I have no major concerns. Some remaining questions about the proposed approach are:

1) How sensitive is test-time training to hyperparameters like splitting the parameters at the right location (i.e., the particular partitioning of $\theta$ into $\theta_e$, $\theta_s$, and $\theta_m$), or to the learning rate? Is there a good way to pick these hyperparameters, given that evaluation is on an out-of-domain distribution that we assume we do not have access to at training time? The paper proposes a particular split of parameters and a particular learning rate and number of steps (which differs for standard vs. online training). How were those chosen?

2) How does test-time training compare to methods that assume access to the test distribution? I understand that a big benefit is that test-time training does not need to see the entire distribution (unlike standard domain adaptation approaches). But in cases where we do get to see parts of the test distribution -- say some unlabeled examples from it, or even some labeled examples -- how does test-time training compare? For example, should we see test-time training as providing the benefits of domain adaptation even when we're unable to access the unlabeled test distribution; or should we see it as doing something beyond what standard domain adaptation methods do, even when we have access to the unlabeled test distribution?

Minor comments, no need to respond:
a) There are several minor typos in the paper, e.g.: p1, "prediciton"; eqn 8, v; p8, "address"; p9, "orcale"; appendix A, missing ref.
b) The discussion in Appendix A seems a bit speculative and opinionated. For example, it is not obvious that one has to fall back on the space of all possible models in order to represent test-time training with a single gradient step as a fixed model. The discussion is useful but in my subjective opinion could be toned down; the experiments and discussion in the main paper are strong and less speculative.

===

Edit: Thank you for the response. The discussion about hyper-parameters makes sense. My rating edit comes from the realization that the performance improvements obtained are almost entirely from the "online" version, which gets to see the test distribution. So I think the baselines are in lacking in that sense: as a straw man baseline for example, one could simply run the normal model on half of the test set, and then use those observed test examples to do some other sort of domain adaptation training.
review_title: Official Blind Review #1

Meta-review: The paper is on a new approach approach to transductive learning. Reviewers were a bit on the fence. Their most important objection is that the performance improvements that the authors report almost entirely come from the "online" version, which basically gets to see the test distribution.  That contribution is nevertheless, in itself, potentially interesting, but I was surprised not to see comparison with simple transductive learning from semi-supervised learning, learning with cache, or domain adaptation, e.g., using knowledge of the target distribution to reweigh the training sample, or [0], on using an adversary to select a distribution consistent with sample statistics. I encourage the authors to add more baselines, analyze differences with existing approaches, and, if their approach is superior to existing approaches, resubmit elsewhere. 

[0] http://papers.nips.cc/paper/5458-robust-classification-under-sample-selection-bias.pdf
Decision: Reject