Test-Time Training for Out-of-Distribution Generalization
Authors: Yu Sun
Xiaolong Wang
Zhuang Liu
John Miller
Alexei A. Efros
Moritz Hardt
Venue:  (2020)
Keywords: out-of-distribution
distribution shifts

Abstract: We introduce a general approach, called test-time training, for improving the performance of predictive models when test and training data come from different distributions. Test-time training turns a single unlabeled test instance into a self-supervised learning problem, on which we update the model parameters before making a prediction on the test sample. We show that this simple idea leads to surprising improvements on diverse image classification benchmarks aimed at evaluating robustness to distribution shifts. Theoretical investigations on a convex model reveal helpful intuitions for when we can expect our approach to help.

Under review as a conference paper at ICLR 2020
TEST-TIME TRAINING FOR GENERALIZATION UNDER DISTRIBUTION SHIFTS
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce general approach, called test-time training, for improving the per- formance of predictive models when training and test data come from different distributions. 'Test-time training turns single unlabeled test instance into self- supervised learning problem, on which we update the model parameters before making prediction on this instance. We show that this simple idea leads sur- prising improvements on diverse image classification benchmarks aimed at eval- uating robustmess to distribution shifts. Theoretical investigations on a convex model reveal helpful intuitions for when we can expect our approach to help.
INTRODUCTION
Supervised learning remains notoriously weak at generalization under distribution shifts. Unless training and test data are drawn from the same distribution, even seemingly minor differences turn out to defeat ness are but few existing paradigms that anticipate differences of sorts between the training and test distribution. In this work, we explore new take on generalization, without any mathematical structure or data available at training-time to anticipate the distribution shifts.
models. Transfer learning, domain adaptation and adversarial robust-
We start from simple observation. The unlabeled test instance æ presented at test-time gives us a hint about the distribution from which was drawn. Our approach, called test-time training. allows the model parameters 0 to depend on the test instance but not its unknown label y. This variable decision boundary 0 ((x) is powerful in theory (see Appendix A). and raises new challenges in practice that we only begin to address here.
Our algorithm creates a self-supervised learning problem based on this single test instance æ, updat- ing at test-time before making prediction. Self-supervised learning uses auxiliary task that automatically creates labels from unlabeled inputs. For the visual data we work with, the task rotates an image x by multiple of 90 degrees, and assigns the angle the label (Gidaris et al. 2018).
Our approach can also be easily modified to work outside the standard supervised learning setting. If several test instances arrive in batch, we can the entire batch for test-time training. If test instances arrive online sequentially, we obtain further improvements by keeping the state of the parameters. After all, prediction is rarely single event. The online setting can be the natural mode of deployment in practice, and shows the strongest improvements.
We experiment with test-time training for generalization under distribution shifts in the context of object recognition on several benchmarks. These include images with diverse types of corruption various levels (Hendrycks & Dietterich) (209)). video frames of moving objects (Shankar al., 2019), and new test set of unknown shifts collected by |Recht et al. (2018). Our algorithm does not hurt on the original distribution, but makes substantial improvements under distribution shifts.
In all our experiments, we compare with strong baseline (labeled joint training) that uses both supervised and self-supervised learning at training-time, but keeps the model fixed at test-time. Very recent work shows that additional training-tine self-supervision improves robustness Henddyycks et al. (019b). The joint training baseline we use corresponds to an improved implementation of their work. We are also inspired by Sun et al. (2019), which performs joint training with self- supervised task in the context of unsupervised domain adaptation, simultancously on the source and target domain. A comprehensive review of related work follows in section 5)

---

Under review as a conference paper at ICLR 2020
We complement the empirical results with theoretical investigations in section 4 of when test-time training is expected to help on a convex model, and establish an intuitive sufficient condition, which, roughly speaking, is to have correlated gradients between the loss functions of the two tasks.
2
METHOD
Next describe the algorithmic details of our method. To setup notations, consider standard K- layer neural network with parameters 92 for layer k. The stacked parameter vector = (f)..... 0x) specifies the entire model for classification task with loss function 4m (1r, y; 0) the test instance (I, v)- We call this the main task, as indicated by the subscript of the loss function.
We assume to have training data (I1, yn.,................ (Xi, Un) drawn i.i.d. from distribution P. Standard empirical risk minimization corresponds to solving the optimization problem:
min
(1t, Vo: 0).
(1)
Our method requires a self-supervised auxiliary task with loss function (x)). In this paper, we choose the rotation prediction task (Gidaris et al., 2018), which has been demonstrated to be simple and effective at feature learning for neural networks. The task simply rotates æ on the image plane by one of 0, 90, 180 and 270 degrees and have the model predict the angle of rotation classification problem. Other self-supervised tasks in Section 5) might also be used for our method.
four-way
The auxiliary task shares some of the model parameters O₂ (01...., .xx) up to a certain {1...., K.. We think of those ń. layers as a shared feature extractor. The auxiliary task uses its own task-specific parameters O,- - (0', (0--11-... 0x). We call the unshared parameters O, the self- supervised task branch, and 0m = (0 x)) the main lask branch. Pictorially, the joint architecture Y -ttruucturee with a shared bottom and two branches. For our experiments, the self- supervised task branch has the exact same architecture as the main branch, except for the output dimensionality of the last layer due to the different number of classes the two tasks.
Training is done in the fashion of multi-task learning (Caruana 1997); the model is trained on both tasks on the same data drawn from P. Losses for both tasks are added together, and gradients are taken for the collection of all parameters. The joint training problem is therefore
min 0 O, Σ (x, yi: 0m, 0z) + (z,; 0s).
Now we describe the standard version of test-time training on a single test instance x. Simply put, test-time training finetunes the shared feature extractor O. by minimizing the auxiliary task loss on x. This can be formulated as
min 1, Os, 0z).
(3)
The model then makes a prediction using the updated parameters (zx) = (0 0m). Empirically, the difference is negligible between minimizing Equation 3 over 0_ versus over both 0₂ and 0,. Theoretically, there exists a difference only when optimization is done with more than one step of gradient descent.
Denote O the (approximate) minimizer Equation
In the standard version of our method, the optimization problem in Equation is always initialized with parameters 0 = (0.., O.) obtained by minimizing [Equation 2 on data from P. After making a prediction on x. 0% discarded. Outside of the standard supervised learning setting, when the test instances arrive online sequentially, the online version of test-time training solves the same optimization problem as in [Equation to update the shared feature extractor 0g. However, on test instance Ni. 0 is instead initialized with ((x)-1) updated the previous instance 22-1- This allows 0 (x+) to take advantage of the distributional information available in øi,..., It-1 as well as Zt.
Test-time training naturally benefits from standard data augmentation techniques. On each test in- stance I, we perform the exact same set of random transformations as used for data augmentation during training to form batch only containing these augmented copies of for test-time training.
2

---

Under review as a conference paper at ICLR 2020
Baseine
Joint training
Test-lime training
Error (%)
40
training online
30
20
10
original
gauss
shot
impulse
defocus
gloss
motion
zoom
show
frast
tog
bright
contrast
elastic
pizelate
Ipeg
Figure 1: Test error (%) on CIFAR- 10-C, level 5. See subsection 3.1 for details.
3 EMPIRICAL RESULTS
We experiment with both versions of our method (standard and online) on three kinds of benchmarks for distribution shifts, presented here in the order of visually low to high-level, which is roughly also the order of easy to hard. Our code will be released once the paper is accepted.
Network details. Our architecture and hyper- pareerr-rrmeeeers are consistent across all experiments. We use Residual Networks (ResNets) (He et al., 2016b), which are constructed differently for CIFAR- 10 (26-layer) and ImageNet P (18-layer), ResNets on ImageNet have four groups, each containing convolutional layers with the same number of channels and size of feature maps; splitting point is the end of the third group. ResNets on CIFAR- 10 have three groups; our splitting point is the end of the second group. In addition, Batch Normalization (BN), a popular module in deep networks, is ineffective when training on small batches, for which the estimated batch statistics are less accurate (loff & Szegedy 2015). This technicality hurts test-time training since each batch only contains (augmented) copies of single image. Therefore our networks instead use Group Normalization (GN) (Wu & He 2018), which achieves similar performance as BN on large batches without hurting on small ones. Results with BN are shown in Appendix E for completeness.
Optimization details. For Equation 2 optimization hyper-parameters are set to the default 3 in standard practice (Huang et al., 2016 He et al. 2016a). For Eqqaaion 3] we use stochastic gradient descent (SGD) with the learning rate set to that of the last epoch during training, which is 0.001 in all our experiments. Following standard practice (He et al. 2018 Liu et al. 2018) known to improve performance when finetuning, we do not use weight decay or momentum. For the standard version, we take ten gradient steps, using batches independently generated by the same image. For online we take only one step. The computational aspects of our method are discussed in Appendix C Following standard practice, the transformations used for data augmentation are random crop with padding and random horizontal flip for CIFAR-10 (Guo et al.] 2017at Huang al. 2016). and random resized crop and random horizontal flip for ImageNet (Loffe & Szegedy 2015]] He et al. 2016a). Specifically, these transformations do not contain information about the distribution shifts.
In all the tables and figures, baseline refers to the plain ResNet model (using GN, unless otherwise specified); joint training refers to the model jointly trained both the main task and the self- supervised task, fixed at test-time as in Hendrycks et al. (2019b); test-time training refers to the standard version described /ection 2] and test-time training online refers to the online version that does not discard (zz,) for Ie arriving sequentially from the same distribution. Performance for test-time training online is calculated, just like the others, as the average over the entire test set; we always shuffle the test set before test-time training online to avoid ordering artifacts.
CCIARR-I (Krizhevsky& Hinton. 2009) is a standard object recognition dataset with 10 classes of objects in natural scenes. The standard train/ test split has 50,000 / 10,000 images, each of size 32-by-32 pixels.
'The ImageNet 2012 classification dataset (Russakovsky et al. 2015) for object recognition has images from 1, 1.00 classes, 1.2 million for training and 50,000) for validation. Following standard practice (He et al. 2016a b) Huang et al. 2016), the validation set used as the test set.
Namely, we use stochastic gradient descent (SGD) with weight decay and momentum; learning rate starts
0.1 and dropped by factor of ten at two scheduled milestones, to 0.01 and 0.001.
3

---

Under review as a conference paper at ICLR 2020
60
Joint training
Accuracy (%)
Test-Eme training
training
20
0
original
gauss
shot
impulse
deforus
glass
mation
zoom
snow
Frost
fog
bright
contrast
elastic
pixelate
ipeg
Original
Gaussian Noise
Defocus Blur
Accuracy 196)
76
Sliding window average
33
27
Accuracy (%)
28
32
30
Sliding wi window average
70
24
68
21
22
66
20
64
60
15
12
Sliding window average
118
16
Zoom Blur
Fog
Elastic Transform
54
Mow M ManMM
51
Accuracy 196)
36
196)
51
Accuracy (96)
48
45
34
42
hy
32
42
39
30
39
36
28
Sliding dow average
Sliding window average
33
Sliding window average
30
20000 40000
20000 Number samples
40000
20000 40000
Number of samples
Number of samples
Test accuracy (%) ImageNet-C, level 5. The lower panel shows the accuracy of the online version as the average over sliding window of 100 samples; test-time learning online generalizes better as more samples are tested on, without hurting on the original distribution. We use accuracy instead of error here because the baseline performance is very poor with most corruptions. See lcolsssscttion 3.1 for details.
Figure 2:
3.1 COMMON CORRUPTIONS
Hendrycks & Dietterich (0199) propose to benchmark robustness of neural networks on 15 types of corruptions from four broad categories: noise, blur, weather and digital. Each corruption type comes in five levels of severity, with level 5 the most severe (details and sample images in Appendix D). The corruptions are algorithmically simulated to mimic real-world corruptions as much as possible on copies of the test set for both CIFAR-IO and ImageNet. According the authors, training should be done on the original training set, and the diversity of corruption types should make it difficult for any method to work well across the board if it relies too much corruption specific knowledge.
Our results on the level 5 corruptions (most severe) are shown in Figure 9 Due space constraints, our results on levels 1-4 are shown in [nppendixx E. Across all five levels and 15 corruption types, both versions of test-time training always improve over the baseline by large margin. The standard version of test-time training always improves over joint training, and the online version often improves very significantly (>> 10%) over joint training and never hurts by more than 0.2%. Specifically, test-time training online contributes >24% on the three noise types and 38% on pixelation. For the seemingly unstable setup of a learning problem that abuses a single image, this kind of consistency is rather surprising.
CIFAR-1O-C.
The baseline ResNet-26 has error 8.9% on the original test set of CIFAR-10. The joint training baseline actually improves performance on the original 8.1%. Most surprisingly, unlike many other methods that tradeoff original performance with robustness, test-time training further improves on the original test set by 0.2% consistently over many independent trials. This indicates that our method does not choose between specificity and generality.
Separate from our method, it is interesting to note that joint training consistently improves over the baseline, as discovered by [enddyykk et al. (2019b). Hendrycks & Dietterich (2019) have also experimented with various other training methods on this benchmark, and point to Adversarial Logit Pairing (ALP) (Kannan al. 2018) the most effective. Results of this additional baseline
4

---

Under review as a conference paper at ICLR 2020
Accuracy (%)
Airplane
Bird
Car
Dog
Cat
Horse
Ship
Average
Baseline
67.9
35.8
42.6
14.7
52.0
42.0
66.7
41.4
Joint training
70.2
36.7
42.6
15.5
52.0
44.0
66.7
42.4
Test-time training
70.2
39.2
42.6
21.6
54.7
46.0
77.8
45.2
TTT online
70.2
39.2
42.6
22.4
54.7
46.0
77.8
45.4
Table I: Test accuracy (%) on our video classification dataset, adapted from Shankar et al. (2019).
We report accuracy for each class and the average over all samples. See subsection 3.2 for details.
all levels of CIFAR-10-C are shown in Appendix E, along with its implementation details. While surprisingly robust under some of the most severe corruptions (especially the three noise types), ALP incurs much larger error (by a factor of two) on the original distribution and some corruptions (e.g. all levels of contrast and fog), and hurts performance significantly when the corruptions are not as severe (especially on levels 1-3); this kind of tradeoff is to be expected for methods based on adversarial training, but not test-time training.
We use accuracy instead of error for this dataset because the baseline severely underperforms with most corruptions. The general trend is roughly the same as on CIFAR-1O-C. Test-time training (standard version) always improves over the baseline and joint training, while the online version only hurts on the original by 0.1% over the baseline, but dramatically improves (by a factor of more than three) on many of the corruption types.
ImageNet-C. Our results on the level 5 corruptions (most severe) are shown in Figure 2
In the lower panel of Figure 2] we visualize how the accuracy (averaged over sliding window) of the online version changes as more images are tested on. Due to space constraints, we show this plot on the original test set, well as every third corruption type, following the same order as in the original paper. On the original test set, there is no visible change in performance after updating on the 50,000 samples. With corruptions, accuracy has already risen significantly after 10,000 samples, but is still rising towards the end of the 50,000 samples, indicating room for additional improvements if more samples were available. Without looking single label, test-time training online behaves as if we were training on the test set from the appearance of these plots.
3.2 VIDEO CLASSIFICATION
The ImageNet Video Classification (VID) dataset was de- veloped by (Shankar et al. (2019) from the Video Detection dataset of ILSVRC2015 (Russakovsky et al.) 2015), to
Method
Accuracy (%)
Baseline
62.7
demonstrate how deep learning models for object recog- nition trained on ImageNet (still images) fail to adapt well to video frames 4 Without any modification for videos, we apply our method to VID on top of the same ImageNet model as in the previous subsection. Our results are shown Again, we use accuracy instead of error be- cause the baseline performance poor.
Joint training
63.5
Test-time training
63.8
TTT online
64.3
in Table 2
Table 2: Test accuracy (%) on VID, See subsection 3.2 for details.
In addition, we take the seven classes in VID that overlap with CIFAR- and rescale those video frames the size of CIFAR-I0 10 images, as new test set for the model trained on CIFAR-IO in the previous subsection. Again, we apply our method to this dataset without any modification. Our results are shown in Table I] with breakdown for each class. Noticing that test-time training does not improve the airplane class, we inspect some airplane samples, and observe that most of them have black margins on the sides, which provide a trivial hint for the rotation prediction task. In addition, for airplanes captued in the sky, it is often impossible even for humans to tell if an image is rotated. This shows that our method requires the self-supervised task to be both well defined and non-trivial on the new domain.
4 The VID dataset contains 1 1109 sets of video frames; each set forms a short video clip where all the frames are similar to anchor frame. Our results are reported the anchor frames. To map the 1000 ImageNet classes the 30 VID classes, we the max-conversion function in Shankar al. 2019).
5

---

Under review as a conference paper at ICLR 2020
3.3 CIFAR-10.1: A NEW TEST SET WITH UNKNOWN DISTRIBUTION SHIFTS
CIFAR-10.1 (Recht et al. 2018) is a new test set of size 2000 modeled after CIFAR-IO, with the exact same classes and image dimensionality, following the dataset creation process documented by the original CIFAR-10 paper as closely as possible. The purpose is to investigate the distribution shifts present between the two test sets, and the effect on object recognition. All models tested by the authors suffer a large performance drop on 10.1 comparing to CIFAR-10, 10, even though there is no human noticable difference, and both have the same human accuracy. This demonstrates how insidious and ubiquitous distribution shifts are, even when researchers strive to minimize them.
The distribution shifts from CIFAR- 10 to CIFAR-10.1 pose an extremely difficult problem, and nobody has made a success- ful attempt to improve the performance of an existing model on this new test set. probably because 1) Researchers cannot even identify the distribution shifts, let alone describe them with mathematics. 2) The samples in CIFAR-10.1 are only revealed at test-time; and even if revealed during training, the distribution shifts too subtle, and the sample size is too small, for domain adaptation algorithms (Recht et al.) 2018).
Method
Error (%)
Baseline
17.4
Joint training 16.7
Test-time training
15.9
Table 3: Test error (%) on CIFAR-
10.1. See subsectionnnnn 3.3 for details.
On the original CIFAR-10 test set, our baseline has error 8.9%, and with joint training 8.1%; com- paring to the first two rows of Table 3) both suffer the typical performance drop (by a factor of two). Test-time training yields an improvement of 0.8% (relative improvement of 4.8%) over joint train- ing. We recognize that this improvement is still small comparing to the performance drop, but see it as an encouraging first step for this very difficult problem.
TOWARDS UNDERSTANDING TEST-TIME TRAINING
This section contains our preliminary study of when and why test-time training is expected to work. For convex models, we prove that positive gradient correlation between the loss functions leads to better performance on the main task after test-time training. Equipped with this insight, we then empirically demonstrate that gradient correlation governs the success of test-time training on the deep learning model discussed in Section 3
Before stating our main theoretical result, we first illustrate the general intuition with a toy model. Consider a regression problem where x E IRd denotes the input, y1 E IR denotes the label, and the objective is the square loss (9 222 22 for prediction ÿ. Consider two layer linear network parametrized by E hxd and e E RR (where h stands for the hidden dimension). The prediction according to this model is Ax, and the main task loss is
(x, yı: A,
(yi v ATr)²
(4)
In addition, consider a self-supervised regression task that also uses the square loss and automatically generates a label ys for x. Let the self-supervised head be parameterized by w E TRA.. Then the self- supervised task loss is
1 (12 w Ar)
(1, y2; A, w) =
(5)
Now we apply test-time training to update the shared feature extractor by one step of gradient descent on lg, which we can compute with y/2 known. This gives
A' + A - n (y2 w Az) (-wr")
(6)
where A' is the updated matrix and 17 is the learning rate. If we set 17 = nn* where
1/1 2 Ar
7 = (12 w Ax)
v wr TT
then with some simple algebra, it is easy to see that the main task loss Im (I. y1: cretely, test-time training drives the main task loss down to zero with single gradient step for carefully chosen learning rate. In practice, this learning rate is unknown since depends on the unknown y1- However, since our model is convex, long as 17" is positive, it suffices to v to
Con-

---

Under review as a conference paper at ICLR 2020
Level Level Level Lovel
35
Level 5
Level 4
Level 3
Improvement (9))
Improvement (1%
25
Lovd 2
Level
Level
20-
15
10
10
20
30
50
10
20
30
40
50
60
Gradient product
Gradient inner product
Figure 3: Scatter plot of the inner product between the gradients (on the shared feature extractor 0.) of the main task lm and the self-supery supervised task le, and the improvement in test error (%) from test- -st-time training, for the standard (left) and online (right) version. Each point is the average 75 test sets, from 15 types corruptions over five levels as described in insubsection 3.1] The blue lines and bands are the best linear fits and 99% confidence intervals. The linear correlation coefiicients are 0.93 and 0.89 respectively, indicating strong positive correlation between the two quantities, as suggested by Theorem |
test and each scatter plot has
be a small positive constant (see Lemma Tp. If x + O, one sufficient condition for 1)" to be positive (when neither loss is zero) to have
sign y1 r. Ax) sign (u2 w Ax) TAA))
and w 0.
For our toy model, both parts in Equation 8 have an intuition interpretation. The first part says that the mistakes should be correlated, in the sense that predictions from both tasks are mistaken in the same direction. The second part, T un > says that the decision boundaries on the feature space should be correlated. In fact, these two parts hold iff (vlm (A), VI, > (see Lemma 2 To summarize, if the gradients have positive correlation, test-time training guaranteed to reduce the main task loss. Our main theoretical result extends this to general smooth and convex loss functions.
Theorem 1. Let Lm (I. y; 1) denote the main task loss on test instance x. with parameters 0, and (x; 0) the self-supervised task loss that only depends on x 5 Assume that for all x. y, bm (2z y, 0) is differentiable, convex and B-smooth in O and both | mm (I, y; 0) , ( 0)l S G for all O. With a fixed learning rate rj = BG2 for every x. y such that
(Vlm 0), Vlo(=;0)) >
(9)
we have
bm (x. Wr 0) > 1m (x. y, ((x)),
(10)
where 0(x) = 12; 0) i.e. test-time training with one step of gradient descent.
determining factor of The proof is in Appendix B.2 Theorem 1 reveals gradient correlation
the success of test-time training in the smooth and convex case. In Figure 3] we empirically show that insight also holds for non-convex loss functions, the deep learning model and across the diverse set of corruptions considered in sectiinn 3) stronger gradient correlation clearly indicates higher performance improvements over the baseline.
RELATED WORK
Our work has been influenced by the successes and limitations of many related fields. Each of these fields contains maybe hundreds of interesting works, which we unfortunately do not have enough time and space to acknowledge in this draft. We apologize for the omissions and are happy to include additional citations upon request.
Learning on test instances. Concurrent work Bau et al. (2019) improves photo manipulation with generative adversarial network, by adapting its image prior the statistics of the input image. Shocher et al.) (2018) performs super-resolution on an image, by learning to recover the original
'Because the main task branch and the self-supervised branch are kept fixed at test-time, we do not explicitly describe their parameters, and include them implicitly in the loss functions.
7

---

Under review as a conference paper at ICLR 2020
image itself from its downsampled version. An older work,Jain & Learned- (2011) improves Viola-Jones (Viola et al. 2001) for face detection, by bootstrapping the more difficult faces in an image from the easter ones with high confidence in the same image. The online version of our algorithm is inspired by Mullapudi et al. (2018), which makes video segmentation more efficient by using student model that learns online from teacher model. The idea of online updates has also been used in Kalal et al. (2011) for tracking and detection. Zhu et al. (2019), a concurrent work in echocardiography, improves the deep learning model that tracks myocardial motion and cardiac blood flow with sequential updates. Lastly, we share the philosophy of transductive learning [Vapnik] 2013] Gammerman et al., 1998), but have little in common with their classical algorithms; concurrent work Tripuraneni & Mackey (2019) theoretically explores this for linear prediction.
Adversarial robustness studies the robust risk: Rp.a (0) = -z..g- max 1xx+ [[x++00 6, y: #) where 7 is some loss function, and A is the set of perturbations; A is often chosen as the L, ball, for p E (1, 2, x0) }. Many popular algorithms formulate and solve this as a robust optimization problem (Goodfellow et al.] [2014] [Madry et al. 2017, Sinha al.) 2017: Raghunathan et al., [0118] Wong
& Kolter 2017] Croce et al. 2018), and the most well known technique is adversarial training.
Another line of work is based on randomized smoothing (Cohen et al. [209 Salman et al. 2019).
while some other approaches such as input transformations (Guo et al. 201 Song et al. 2017) are
shown to be less effective (Athalye et al. 2018). There are two main problems in this field. First, all the approaches mentioned above can be seen as smoothing the decision boundary. This establishes theoretical tradeoff between accuracy and robustness (Tsipras et al. 2018 Zhang et al. 2019), which we also observe empirically with our adversarial training baseline in Section 3 Intuitively, the more diverse A is, the less effective this one- sere-boudddy-fitte filt-alll approach can be for a particular element of A. Second, adversarial methods rely heavily on the mathematical structure of A, which might not accurately model perturbations in the real world. Therefore, generalization remains hard outside of the A we know in advance or can mathematically model, especially for non-adversarial distribution shifts. Empirically, Kang et al. (2019) shows that robustness for one A might not transfer to another, and training on the Los ball actually hurts robustness on the L1 ball.
Non-adversarial robustness studies the effect of corruptions, perturbations, outrof-distribution examples, and real-worle distribution shifts (Hendrycks et al., [2019a] bJ 2018 [endrrycks & Gimpel, 2016). Geirhos et al. (2018) show that training on images corrupted by Gaussian noise makes deep learning models recover super-human performance on this particular noise type, but cannot improve performance on another those corrupted by another noise type e.g. salt-and-pepper noise.
Unsupervised domain adaptation (a.k.a. transfer learning) studies the problem of distribution shift (from P to Q), when unlabled data from Q is available at training -iime time (Tzeng et al. 2017 Ganin et al. 2016 Gong et al. 2012; Long et al., 2016; Chen et al.) 2018: 2011] Hoffman et al., 2017 Csurka 2017: Long et al. 2015). We are inspired by this very active and successful com- munity, especially (Sun et al.] 2019), and believe that progress in this community can motivate new algorithms in the framework of test-time learning. Our update rule can be viewed as performing one-sample unsupervised domain adaptation on the fly 0 On the other hand, test-time learning comes from realizing the limitations of the unsupervised domain adaptation setting, that outside of the specific target distribution where data is available for training, generalization is still elusive. Previous works make the source and target distributions broader with multiple and evolving sources
and targets without fundamentally address this problem (Hoffman et al. 2018 2012 2014).
Self-supervised learning studies how to create labels from the data, by designing ingenious tasks that contain semantic information without human annotations, such as context prediction (Doersch et al. 20155.. solving jigsaw puzzles (Noroozi & Favaro 2016). colorization (Larsson et al. 2017 Zhang et al. 2016). noise prediction (Bojanowski & Joulin 2017), and feature clustering Caron et al. 2018). Self-supervision has also been used on videos (Wang & Guptal 2015) Wang et al.; 2019). ). Particularly relevant to our work, work,[Asano et al. (209)) shows that self-supervised learning on only a single image, surprisingly, can produce low-level features that generalize well. In addition,
'Note that typical unsupervised domain adaptation algorithms,such as those based on distributional discrep- ancy, adversarial learning. co co-training and generative modeling, might not work in our framework because the concept of target distribution, which has heen so deeply rooted and heavily relied on, becomes ill-defined when there is only one sample from the target domain.
8

---

Under review as a conference paper at ICLR 2020
Hendrycks al. (2019b) proposes that jointly training main task and self-supervised task (our joint training baseline in |ection 3) can improve robustness of the main task. The same idea used in few-shot learning (Su et al., 2019), and domain generalization (Carlucci et al. 2019).
Domain generalization studies when meta distribution generates multiple environment distribu- tions, some of which are available during training (source), while others are used for testing (target) (Li al.; 2018, Shankar et al. 2018. Muandet al. 2013 Balaji et al., 2018 Ghifary al. 2015 Motiian et al. 20177] Li et al. 2017a Gan et al., 2016). With only a few environments, information
on the meta distribution is often too scarce be helpful, and with many environments, we are back to the i.i.d. setting where each environment can be seen as sample, and a strong baseline is to sim- ply train on all the environments (Li et al. 2019). The setting of domain generalization is limited by the inherent tradeoff between specificity and generality of fixed decision boundary, and the fact that generalization is again elusive outside of the meta distribution i.e. the actual
Continual learning (a.k.a. learning without forgetting) studies when a model is made to learn a sequence of tasks, and not forget about the task at the beginning (Li & Hoiem 2017 Lopez-Paz & Ranzato 2017, Kirkpatrick et al. 2017 Santoro et al., 2016). Test-time training does not care about forgetting the past test instances since they have already been evaluated on; and if past instance comes up by any chance, it would go through test-time training again. In addition, the impact of forgetting the training set is minimal, because both tasks have been jointly trained. This is in contrast to continual learning, when the tasks are trained one-by-one from scratch.
Few (one)-shot learning studies extremely small training sets (maybe for some categories) (Snell al.
2017 Vinyals et al. 2016 Fei-Fei al. 2006) Ravi & Larochelle 2016 Li 2017b Finn
et al. 2017 Gidaris & Komodakis) 2018). Our update rule can be viewed as performing one-
self- supervise learning and can potentially be improved by progress few-shot learning.
Online learning (a.k.a. online optimization) is a well-studied area of learning theory (Shalev- Shwartz et al., 2012 Hazan et al. 2016). The basic setting repeats the following: receive Zr, predict Dt. receive y/, from a worst-case oracle and learn. Final performance is evaluated using the regret, colloquially how much worse than the best fixed model in hindsight. It is easy to see how our setting differs, even for the online version. We learn before predicting Vt. but never receive any ye that is evaluated on, thus do not need consider the worst-case oracle the regret.
CONCLUSION
The idea of test-time training also makes sense for other tasks such as segmentation and detection, and in other fields such speech recognition and natural language processing. More generally, we hope this paper can encourage researchers to abandon the self-imposed constraint of a fixed decision boundary test-time, or even altogether, the artificial divison between training and testing.
REFERENCES
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. Surprising effectiveness of few-image unsupervised feature learning. arXiv preprint arXiv:1904.1313 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: a 1802.0002000 2018.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regulariizaton. In Advances in Neural Information Processing Systems, pp. 998-1008, 2018.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Man Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (TOG), 38(4):59, 2019.

---

Under review as a conference paper at ICLR 2020
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Proceedings of the 34th International Conference on Machine Learning-Voo Volume 70, pp. 517-526. JMLR. org, 2017.
Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference Com- puter Vision and Patlern Recognition, pp. 2229-2238, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Rich Caruana. Multitask learning. Machine learning, 28(1))41-75, 1997.
Minmin Chen, Kilian Q Weinberger, and John Blitzer. Co-training for domain adaptation. Ad- vances in neural information processing systems, pp. 2456-2464, 2011.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial deep averaging networks for cross- coossll sentiment classification. Transactions of the Association for Computational Linguisties, 6:557-570, 2018.
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: 1902.0298,, 2019.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu net- works via maximization of linear regions. arXiv preprint arXiv:1810.0748) 2018.
Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey, arXiv
preprint arXiv: 2017.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv: 1902.07633 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on paltern analysis and machine intelligence, 28(4):594-6) 111, 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1133 JMLR. org, 2017.
Alexander Gammerman, Volodya Vovk, and Vladimir Vapnik. Learning by transduction. In Pro- ceedings of the Fourteenth conference Uncertainty in artificial intelligence, pp. 148-155. Mor- gan Kaufmann Publishers Inc., 1998.
Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain gen- eralization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2016
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. works. The Journal of Machine Learning Research, 17(1):2096- 2030, 2016.
training of neural net-
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Informa- tion Processing Systems, pp. 7538-7550, 2018
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generaliza- tion for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
10

---

Under review as a conference paper at ICLR 2020
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0078, 2018.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2066-2073, IEEE, 2012.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning- Voluue 70, pp. 1321-1330. JMLR. org, 2017a.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. arXiv preprint 1rXiv:1711.00/1 2017b.
Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Opti- mization, 2(3-4):157-325, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint arXiv: 188.088,, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv: 1203,,, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution 2016.
examples in neural networks. arXiv preprint
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in neural information processing systems, pp. 10456-10465, 2018
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Improving model robustness and uncertainty estimates with self-supervised learning. 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. arXiv preprint arXiv: /906. /2340, 2019b.
Judy Hoffmman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering latent domains for multi- source domain adaptation. In European Conference on Computer Vision, pp. 702-715. Springer, 2012.
Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evoly- ing visual domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 867-874, 2014.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiw:1711.03213. 2017.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang, Algorithmss and theory for multiple-source adaptation. In Advances in Neural Information Processing Systems, pp. 8246-8256, 2018.
11

---

Under review as a conference paper at ICLR 2020
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646 661. Springer, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint orXivv/502.03/ 2015.
Vidit Jain and Erik Leamnd--Mller. Online domain adaptation of pre-trained cascade of classifiers. In CVPR 1, pp. 577-584, IEEE, 2011.
Zdenek Kalal, Krystian Mikolajezyk, and Jiri Matas. TTacking-lemingg-eetttiioo.....i JEEE transac- tions on pattern analysis and machine intelligence, 34(7):1409-1422, 2011.
Daniel Kang. Yi Sun, Tom Brown, Dan Hendrycks, and Jacob Steinhardt. Transfer of adversarial
robustness between perturbation types. arXiv preprint arXiv: wrXiv:1905.01034 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv: arXiv:/803.06:: /803.06333, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska BaraanBaakkaaa et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3351-3352, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5542-5550, 20
Da Li, Jianshu Zhang, Yongxin Yang. Cong Liu, Yi-Zhe Song, and Timothy Hospedales. Episodic training for domain generalization. arXiv preprint arXiv:1902.00111 2019.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 2017b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( 40(12):2935-2947 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1810v21810.05270, 2018.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv:/502. 150.00279, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, pp. 136- 144, 2016.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 17X.::77060668. 2017.
12

---

Under review as a conference paper at ICLR 2020
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5715-5725, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. arXiv preprint arXiv:1812.0220 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference Computer Vision, pp. 69-84. Springer, 2016.
Aditi Raghumathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. arXiv preprint arXiv: 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do eifar-10 classifiers generalize to cifar- 10? arXiv preprint (0XXv::8806000455/1 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 115())211-2522 2015. doi: 10. 1007/s 1263-015-0816-y.
Hadi Salman, Greg Yang. Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. arXiv preprint 1rXiv:1906.04584 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory augmented neural networks. In International conference on machine learn- ing. pp. 1842-1850, 2016.
Shai Shalev-Shwartz al. Online learning and online convex optimization. Foundations and Trends@ in Machine Learning, 4(2):107-19, 2012.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXivv1804.10745. 2018.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Lud- wig Schmidt. A systematic framework for natural perturbations from videos. arXiv preprint arXiv:1906.021666 2019.
Assaf Shocher, Nadav Cohen, and Michal Irani. zero-shot super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3118-3166, 2018.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint 1701::1700.10571,,000 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 00077-4077,200
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend:
Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv: 10766, 2017.
Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. Boosting supervision with self-supervision for few-shot learning, arXiv preprint arXiv:I906. 07079, 2019.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through seff-supervision. arXiv preprint, 2019.
13

---

Under review as a conference paper at ICLR 2020
Nilesh Tripuraneni and Lester Mackey, arXiv:1908.0 02341, 2019.
Debiasing linear prediction.
arXiv preprint
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint arXiv: 12152, 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Paul Viola, Michael Jones, et al. Rapid object detection using boosted cascade of simple features. CVPR (1), 1(511-518)) 2001.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV,, 2015.
Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the cycle- consistency of time. In CVPR, 2019.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint 4rXiv:/11.0008: 2017.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Erie P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprini 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Wentao Zhu, Yufang Huang, Mani A Vannan, Shizhen Liu, Daguang Xu, Wei Fan, Zhen Qian, and Xiaohui Xie. Neural multi-scale self-supervised registration for echocardiogram dense tracking. arXiv preprint ar 1906v.1906007330 2019.
14

---

Under review as a conference paper at ICLR 2020
A INFORMAL DISCUSSION ON OUR VARIABLE DECISION BOUNDARY
In Infsection we claim that in traditional supervised learning @ gives a fixed decision boundary, while our gives variable decision boundary. Here we informally discuss this claim.
Denote the input space X' and output space y. A decision boundary is simply mapping / + y. Let + be model class e-g Id Now consider family of parametrized functions ge X + V where 0 E O In the context of deep learning, g is the neural network architecture and 0 contains the parameters. We say that is a fixed decision boundary w.r.t. 9 and if there exists 0 E s.t. (3) = go (x) for every E X, and a variable decision boundary if for every I E X, there exists + s.t. f(x) = go (z). Note how selection of can depend on x for a variable decision boundary, and cannot for fixed one. It is then trivial verify that our claim is true under those definitions.
A critical reader might say that with an arbitrarily large model class, can't every decision boundary be fixed? Yes, but this is not the end of the story. Let d = dim(X') dim()"), and consider the enormous model class O' = Rd which is capable of representing all possible mappings between X' and V. Let ger simply be the mapping represented by 0' O'. A variable decision boundary w.r.. g and & then indeed must be a fixed decision boundary w.r.t. g' and O', but we would like to note two things. First, without any prior knowledge, generalization in O' is impossible with any finite amount of training data; reasoning about g' and O' is most likely not productive from an algorithmic point of view, and the concept of a variable decision boundary is avoid such reasoning. Second, selecting 0 based on x for a variable decision boundary can be thought of as "training" on all points R. however, "training" only happens when necessary. for the that it actually encounters.
Altogether, the concept of a variable decison boundary is different from what can be described by traditional learning theory. A formal discussion beyond the scope of this paper.
B
PROOFS
Here we prove the theoretical results covered section
B.1
PROOFS FOR THE TOY PROBLEM
The following setting applies to the two lemmas; this is simply the setting of our toy problem, reproduced here for ease of reference. Consider a two layer linear network parametrized by A E Raxd (shared) and v, Rh (fixed) for the two heads, respectively. Denote x E Rdd the input and y1, 1/2 E R the labels for the two tasks, respectively, For the main task loss
(A: v) (y1 Ax)²
(11)
and the self-supervised task loss
1
1. A: w) =
(y2 w T Ax)
(12)
2
test-time learning yields an updated matrix
A' +- A 7 (1/2 w Ax) (-
(13)
where 17 is the learning rate.
Lemma 1. Following the exposition section F denote
(y) v Ax)
(14)
(u2 w T Ax)u" u wxx T x:
Assume 17" E [e, oo) for some 0. Then for any 1 E (0, we guaranteed an improvement on the main loss i.e. (A') < 1m (A).
Proof. From the exposition of section 4 we know that
Lm (A 1) VI, 0,
15

---

Under review as a conference paper at ICLR 2020
which can also be derived from simple algebra. Then by convexity, have
- 1m (11
!
2 (A (V(((A)))
mm (A + [)V)
A
(15)
1f
rf
11
4m (A) +0
(16)
"1*
mm (m(A)
(17)
< 1m (A),
(18)
where the last inequality uses the assumption that 4m (A) > O, which holds because vf" > 0. Lemma 2. Define U) = vec (U) vec (V) iee the Frobenious inner product, then
sign (17*) sign ((Vlm (A), vI,
(19)
Proof. By simple algebra, (Vlm (A)) ((y) T Az) (-vz) () (y/2 - w Ax) (-wx') "))
(20)
(y1 T Ax) (V2 w T Ax)) Tr (rV T "xx )
(21)
(31 v Az) (y2 w *Ax) 2 wx x,
(22)
which has the same sign as 17"
B.2 PROOF OF THEOREM 1
For any 17, by smoothness and convexity.
Im (x, yı (x))) (x, y: (x;
(23)
T23 11; 4m (I, vi ( 0), Vl,(I,
(24)
Denote
17 (Vlm y: 0). (r,(,,0)
B (z, 0112
Then Equation 23 becomes
Vhn (x. y: 0), (1, 0))
(n
y; (=
17' (I; 0)) S mm (x, y; 0)
(25)
2 [1; 0)|
And by our assumptions on the gradient norm and gradient inner product.
(2, y; 0) - Lm (x, y; 0 (1, ()) 2 23G²
(26)
Because cannot observe 17" in practice, we instead use fixed learning rate 1/ z0z. stated in Theorem I Now we argue that this fixed learning rate still improves performance on the main task.
By our assumptions, 17" 2 अ so 77 E (0, 17"). Denote g = Vl, (x; 0), then by convexity 4m.
Im (,. ((x)) = mm (z, y: 0 - -gg)
(27)
1]
1 (0 17 nr'g) r/²
= mm
x, y!
+
(28)
17"
")
n
Σ
1
Im (I, yr 0) +
(I, y: 0 - n"
(29)
17a
Combining with Equation 26 we have
2²
Im 0x <
27
Im (I, y: 0)
17 17¹
[m((:; y: 0) 23G
(30)
17"
2
= -m (x,, y:
0)
(31)
11" 2
Since n/n2 > 0, we have shown that Im (xr, yy 0) Im (I., y: (s:)) 0.
(32)
16

---

Under review as a conference paper at ICLR 2020
C COMPUTATIONAL ASPECTS OF OUR METHOD
At test-time, our method is 2xbatch.size number_of of -iteratiions times slower than regular test- ing, which only performs a single forward pass for each sample. As the first work on test-time learning, this paper is not as concerned about computational efficiency as improving robustness, but here we provide two potential solutions that might be useful, but have not been thoroughly verified. The first is to use the thresholding trick on I., introduced as solution for the small batches problem /sctionn 2. For the models considered our experiments, roughly 80% of the test instances fall below the threshold,
test-time learning can only be performed on the other 20% without much effect on performance, because those 20% contain most of the samples with wrong predictions. The second is to reduce the number of iiterations of test-time updates. For the online version, the number of terations is already 1, so there is nothing to do. For the standard version, we have done some preliminary experiments setting number of iiterations learning rate to 0.01 (instead of 0.001), and observing results almost as good as the standard hyper- parameter setting. A more in depth discussion efficiency is left for future works, which might, during training, explicitly make the model amenable to fast updates.
(instead of 10) and
D SAMPLE IMAGES FROM THE COMMON CORRUPTIONS BENCHMARK
Gaussian Noise
Shot Noise
Impuise Noise
Defocus Blur Frosted Glass Blu
Motion Blur
Zoom Blur
Frost
Fog
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure 4: Sample images from the Common Corruptions Benchmark, taken from the original paper by Hendrycks & Dietterich (201)).
E ADDITIONAL RESULTS ON THE COMMON CORRUPTIONS DATASET
For table aethetics, we use the following abbreviations: B for baseline, JT for joint training, TTT for test-time training standard version, and TTTO for test-time training online version.
We have also abbreviated the names of the corruptions. The full names are, order: original test set, Gaussian noise, shot noise, impulse noise, defocus blur, glass blue, motion blur, zoom blur, snow, frost, fog. brightness, contrast, elastic transformation, pixelation, and JPEG compression.
E. RESULTS USING BATCH NORMALIZATION
As discussed in section Batch Normalization (BN) ineffective for small batches, which are the inputs for test-time training (both standard and online version) since there is only one sample available when forming each batch; therefore, our main results are based on a ResNet using Group Normalization (GN). Here we provide results of our method CIFAR- 10-C level 5, with ResNet using Batch Normalization (BN). These results are meant to be merely a point of reference for the curious readers, instead of our technical contributions.
17

---

Under review as a conference paper at ICLR 2020
In the early stage of this project, have experimented with two potential solutions to the small batches problem with BN. The naive solution is to fix the BN layers during test-time training. but this diminishes the performance gains since there are fewer shared parameters. The better solution, adopted for the results below, is hard example mining: instead of updating on all inputs, we only update inputs that incur large self-supervised task loss l. where the large improvements might counter the negative effects of inaccurate statistics.
Baseline
60
training Test-time training
Error (%)
0
original
gAuss
shot
Impulse
defocus
glass
mation
zoom
show
frast
fog
bright
contrast
elastic
pixelate
ipeg
Figure 5: Test error (%) on CIFAR R0-CC, level 5, ResNet-26 with Batch Normalization.
orig
gauss
shot
impul
defoc
glass
motn
zoom
snow
frost
fog
brit
contr
elast
pixel
jpeg
B
7.9
63.9
58.8
64.3
46.3
54.6
41.6
45.9
31.9
44.0
37.5
13.0
69.2
33.8
61.4
31.7
JT
7.5
70.7
65.6
67.2
43.1
55.4
40.9
42.7
30.3
44.5
42.5
12.7
58.6
30.7
62.6
31.9
TTT
7.9
47.9
45.2
54.8
27.6
50.4
31.5
30.9
28.7
34.3
26.9
12.6
35.2
30.6
51.2
31.3
Table 4: Test error (%) on CIFAR-10-C, level 5, ResNet-26 with Batch Normalization.
Test-time training (standard version) is still very effective with BN. In fact, some the improve- ments are quite dramatic, such as on contrast (34%), defocus blue (18%) and Gaussian noise (22% comparing to joint-training training, and 16% comparing to the baseline). Performance on the original dis- tribution still almost the same, and the original error with BN is in fact slightly lower than with GN, and takes half as many epochs to converge.
We did not further experiment with BN because of two reasons: 1) The online version does not work with BN, because the problem with inaccurate batch statistics is exacerbated when training online for many (e.g. 10000) steps. 2) The baseline error for almost every corruption type significantly higher with BN than with GN. Although unrelated to the main idea of our paper, we make the interesting note that GN significantly improves model robustness.
E.2 ADDITIONAL BASELINE: ADVERSARIAL LOGIT PAIRING
Hendrycks & Dietterich 2019) point to Adversarial Logit Pairing (ALP) (Kannan et al.] 2018) as an effective method for improving model robustness to corruptions and perturbations, as an additional bascline on all benchmarks based on CIFAR- 10 (using GN), following the training procedure in Kannan et al. 2018) and their recommended hyper-parameters. The implementation of the adversarial attack comes from the codebase of Ding et al.) (2019). We did not run ALP on ImageNet because the two papers we reference for this method, Kannan et al. (2018) and Hendrycks & Dietterich (2019), did not run on ImageNet or make any claim or recommendation.
As discussed in subsection 3.1
though it designed to defend against adversarial attacks. We take ALP
18

---

Under review as a conference paper at ICLR 2020
E.3
RESULTS ON CIFAR- -0-C AND IMAGENET-C, LEVEL 5
The following two tables correspond to the bar plots in section 3
org
gauss
shot
impul
defoc
glass
motn
700m
snow
frost
fog
brit
contr
clast
pixel
ipeg
B
8.9
50.5
47.2
56.1
23.7
51.7
24.3
26.3
25.6
34.4
28.1
13.5
25.0
27.4
55.8
29.8
JT
8.1
49.4
45.3
53.4
24.2
48.5
24.8
26.4
25.0
32.5
27.5
12.6
25.3
24.0
51.6
28.7
TTT
7.9
45.6
41.8
50.0
21.8
46.1
23.0
23.9
23.9
30.0
25.1
12.2
23.9
22.6
47.2
27.2
TTTO
8.2
25.8
22.6
30.6
14.6
34.4
18.3
17.1
20.0
18.0
16.9
11.2
15.6
21.6
18.1
21.2
ALP
16.5
22.7
22.9
28.3
25.0
25.6
27.4
23.1
25.2
27.2
64.8
21.7
73.6
23.0
20.2
18.9
Table 5: Test error (%) on CIFAR-IO-C, level 5, ResNet-26.
orig
gauss
shot
impul
defoc
glass
motn
200m
snow
frost
fog
brit
contr
clast
pixel
ipeg
B
68.9
1.3
2.0
1.3
7.5
6.6
11.8
16.2
15.7
14.9
15.3
43.9
9.7
16.5
15.3
23.4
JT
69.1
2.1
3.1
2.1
8.7
6.7
12.3
16.0
15.3
15.8
17.0
45.3
11.0
18.4
19.7
22.9
TTT
69.0
3.1
4.5
3.5
10.1
6.8
13.5
18.5
17.1
17.9
20.0
47.0
14.4
20.9
22.8
25.3
TTTO
68.8
26.3
28.6
26.9
23.7
6.6
28.7
33.4
35.6
18.7
47.6
58.3
35.3
44.3
47.8
44.3
Table 6: Test accuracy (%) ImageNet-C, level 5, ResNet-18.
E.4
RESULTS ON CIFAR-1O-C, LEVELS 1-4
The following bar plots and tables levels 1 1-4 of CIFAR-1O-C. The original distribution is the same for all levels, so are our results on the original distribution.
Base Ine
joint training
Test-lime training
Error (%)
cist-ttim training online
30-
20
10
0
original
gauss
shot
Impuise
defocus
glass
motion
zoom
snow
frost
tog
bright
contrast
elastic
pirelate
Ipeg
Figure 6: Test error (%) on CIFAR- 10-C, level 4. See subsection 3.1 for details.
orig
gauss
shot
impul
defoc
glass
motn
700m
snow
frost
fog
brit
contr
clast
pixel
ipeg
B
8.9
46.4
39.2
44.8
15.3
$2.5
19.1
20.5
21.3
26.9
13.3
10.5
13.7
20.8
35.3
26.9
JT
8.1
45.0
38.3
42.2
16.4
50.2
20.7
20.5
21.1
25.4
14.1
10.0
14.7
19.0
33.2
25.1
TTT
7.9
41.5
35.4
39.8
15.0
47.8
19.1
18.4
20.1
24.0
13.5
10.0
14.1
17.7
29.4
24.5
TTTO
8.2
22.9
20.0
23.9
11.2
35.1
15.6
13.8
18.6
15.9
12.3
9.7
11.9
16.7
13.6
19.8
ALP
16.5
21.3
20.5
24.5
20.7
25.9
23.7
21.4
24.2
23.9
42.2
17.5
53.7
22.1
19.1
18.5
Table 7: Test error (%) on MIFAR--0-CC level 4, ResNet-26.
Baseline
Joint training
Test-time training
Error (%)
30
Test: time training onlne
20
10
0
original
gauss
shot
Impuise
defocus
glass
mation
zoom
snow
frost
log
bright
contrast
elastic
pixelate
Ipeg
Figure 7: Test emorr
on CIFAR- 10-C, level 3. See subsection 3.1 for details.
19

---

Under review as a conference paper at ICLR 2020
ong
gauss
shot
impul
defoc
glass
motn
Z00m
snow
frost
fog
brit
contr
elast
pixel
ipeg
B
8.9
42.2
35.1
30.7
12.2
41.7
18.6
17.5
19.0
25.3
10.8
9.7
11.6
15.3
21.7
24.6
JT
8.1
40.2
34,4
29.9
12.2
37.9
20.8
17.3
18.4
25.0
11.4
9.2
12.0
15.2
20.8
22.8
TTT
7.9
37.2
31.6
28.6
11.5
35.8
19.1
15.8
17.8
23.3
11.0
9.1
11.6
14.3
18.9
22.3
TTTO
8.2
21.3
17.7
17.9
9.0
23.4
15.3
12.5
16.4
15.8
10.9
9.0
10.7
12.8
12.2
18.7
ALP
16.5
20.0
19.3
20.5
19.2
21.2
24.0
20.5
20.9
24.2
30.1
16.6
39.6
20.9
17.8
18.0
Table 8: Test error (%) on CIFAR-1O-C, level 3, ResNet-26.
40
Baselie
Joint training
Test-time training
Error (%)
30
Test time training online
20-
10-
0
original
gauss
shot
impulis
defocus
glass
motion
zoom
snow
frost
log
bright
contrast
elastic
picelate
lpeg
Figure 8: Test error (%) on CIFAR- 10-C, level 2. See subsection 3.1 for details.
onig
gauss
shot
impul
defoc
glass
motn
z0om
snow
frost
fog
brit
contr
elast
pixel
ipeg
B
8.9
31.7
22.6
24.3
9.9
42.6
14.9
14.7
21.7
18.4
9.8
9.1
10.0
13.1
17.1
22.4
JT
8.1
31.0
22.6
23.4
9.1
39.2
16.4
14.2
21.2
17.5
9.4
8.3
10.6
12.8
15.9
20.5
TTT
7.9
28.8
20.7
23.0
9.0
36.6
15.4
13.1
20.2
16.9
9.2
8.3
10.2
12.5
14.8
19.7
TTTO
8.2
16.8
13.8
15.5
8.5
23.4
13.3
11.5
16.8
12.7
9.4
8.4
9.7
12.4
11.5
17.0
ALP
16.5
18.0
17.2
19.0
17.8
20.7
21.2
19.3
19.0
20.1
22.4
16.3
29.2
20.3
17.4
17.8
Table 9: Test error (%) on CIFAR-1O-C, level 2, ResNet-26.
Baseline
40
Joint training
rest-time training
Error (%)
Test time training online
20
10
0
original
gauss
shot
Impuise
defocus
glass
motion
zoom
snow
frast
tog
bright
contrast
elastic
pixelate
Ipeg
Figure 9: Test error (%) on CIFAR- 1R-C, level 1. See subsection 3.1 for details.
orig
gauss
shot
impul
defoc
glass
motn
z0om
snow
frost
fog
brit
contr
clast
pixel
ipeg
B
8.9
21.7
17.1
17.0
9.0
44.0
12.1
13.9
14.3
13.4
9.2
8.9
9.0
13.2
12.0
17.3
JT
8.1
20.4
16.6
16.9
8.2
40.5
12.2
13.0
13.1
12.3
8.4
8.1
8.5
12.9
11.3
15.9
TTT
7.9
19.1
15.8
16.5
8.0
37.9
11.7
12.2
12.8
11.9
8.2
8.0
8.3
12.6
11.1
15.5
TTTO
8.2
13.8
11.9
12.2
8.5
24.4
10.5
11.5
12.4
10.7
8.5
8.3
8.6
12.4
10.7
14.4
ALP
17.0
16.8
17.6
16.8
20.9
18.7
19.0
17.3
17.5
17.4
16.1
18.4
20.4
17.0
17.2
Table 10: Test error (%) on CIFAR- level 1, ResNet-26.
20

---

Under review as a conference paper at ICLR 2020
Figure 10: Sample Images from the VID dataset in subsection 3.2 adapted to CIFAR-10. Each row shows eight sample images from one class. The seven classes shown are, in order: airplane, bird, car, dog, cat, horse, ship.
21


Reviews:
[{"review_id": "SkgD9svKFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes test time training, a method that uses an auxiliary task to provide a kind of loose supervision during test time. I say loose supervision because the theory suggests this is only useful when the gradients of the main and auxiliary losses are positively correlated.\n\nI'm not sure I totally believe that this is a method of out-of-distribution generalization, but rather it helps adjust for corruptions and modest dataset shifts which is an important problem itself. I suspect test-time training is fundamentally better suited for the latter because in general the assumption of correlation between the loss gradients cannot hold in the test if we allow for large shifts (like the airplane class in the video experiment). I note that the authors are aware of this limitation (page 5, last paragraph).\n\nMy main problem with this paper is that the more fine-tuned labels get (like the density of a tumor), the harder it gets to create auxiliary tasks. This will be a significant problem when the samples at test-time only share the highest level common characteristics with the true dataset; (like rotations do not impact or density of tumor, like face detection and similar fine-feature-based tasks).\n\nThat said, I do appreciate the experimental results which show promise; especially the CIFAR-10.1 results. So I'm inclined toward accepting this paper. I would be more so inclined if the authors could provide a reasonable categorization of tasks where this method is expected to be applicable. I ask this because, for practitioners, it is near impossible to verify the positive correlation of gradients assumption. If there are high-level targets that the distribution and/or task must satisfy that can act as indicators for applying this method, I believe that would be a valuable addition to the paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BklX60fKtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for adapting model parameters by doing self-supervised training on each individual test example. They show striking improvements in out-of-domain performance across a variety of image classification tasks while preserving in-domain performance; the latter is a marked difference from other robustness procedures which tend to sacrifice in-domain performance for out-of-domain (or adversarial) performance. These results are exciting, and I believe that this proposed test-time training method will spur a significant amount of further research into similar approaches.\n\nThe paper is well-written and the experiments are thorough, so I have no major concerns. Some remaining questions about the proposed approach are:\n\n1) How sensitive is test-time training to hyperparameters like splitting the parameters at the right location (i.e., the particular partitioning of $\\theta$ into $\\theta_e$, $\\theta_s$, and $\\theta_m$), or to the learning rate? Is there a good way to pick these hyperparameters, given that evaluation is on an out-of-domain distribution that we assume we do not have access to at training time? The paper proposes a particular split of parameters and a particular learning rate and number of steps (which differs for standard vs. online training). How were those chosen?\n\n2) How does test-time training compare to methods that assume access to the test distribution? I understand that a big benefit is that test-time training does not need to see the entire distribution (unlike standard domain adaptation approaches). But in cases where we do get to see parts of the test distribution -- say some unlabeled examples from it, or even some labeled examples -- how does test-time training compare? For example, should we see test-time training as providing the benefits of domain adaptation even when we're unable to access the unlabeled test distribution; or should we see it as doing something beyond what standard domain adaptation methods do, even when we have access to the unlabeled test distribution?\n\nMinor comments, no need to respond:\na) There are several minor typos in the paper, e.g.: p1, \"prediciton\"; eqn 8, v; p8, \"address\"; p9, \"orcale\"; appendix A, missing ref.\nb) The discussion in Appendix A seems a bit speculative and opinionated. For example, it is not obvious that one has to fall back on the space of all possible models in order to represent test-time training with a single gradient step as a fixed model. The discussion is useful but in my subjective opinion could be toned down; the experiments and discussion in the main paper are strong and less speculative.\n\n===\n\nEdit: Thank you for the response. The discussion about hyper-parameters makes sense. My rating edit comes from the realization that the performance improvements obtained are almost entirely from the \"online\" version, which gets to see the test distribution. So I think the baselines are in lacking in that sense: as a straw man baseline for example, one could simply run the normal model on half of the test set, and then use those observed test examples to do some other sort of domain adaptation training.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gWRWfMcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation is to increase accuracy of CNNs with unseen (unknown) distribution shifts. To this end, instead of the recent approach of training-time self-supervision, they adopted it for test-time [limited novelty]. More precisely, for a classification task, they considered two headed neural network with one head for main classification task and another for an auxiliary task (e.g. predicting rotation degree of rotated images). The feature extractor (up to k layers) is shared between these tasks thus updated using the two tasks, while two heads are updated according to each task. In test-time, the samples (drawn from shifted data distributions) are being used for updating the shared feature extractor through the auxiliary task. They have investigated their test-time approach in a series of experiments in online and offline settings on synthetic shift in data distribution of image classification tasks. The distributional shifts are synthesized adding some non-adversarial perturbation to clean images. The authors proof that their test-time approach can lead to lower error rate on given test-time samples when the underlying learning model is a linear regression model. \nThe authors incorrectly interchange OOD with domain shifts in their manuscript. The common usage of OOD set is to capture novel samples that are not properly following the training set distribution, e.g. from different concepts than the set of classes given in the training set. With an OOD set, a robust model dealing with it should be able to detect whether an instance is in-distribution or OOD, making a special decision for the latter case (e.g., rejecting the instance).\nIn domain shift, we look at the scenario where test objects are the same as used for training the model (i.e., correspond to one of the classes the model is processing), but might be perturbed or coming having a different distribution (e.g. SVHN for MNIST or vice versa). The approaches for domain shift concern to improve robustness of CNNs to such shifts in data distribution. Accordingly, the title of the paper inaccurately reflects of the claim of the paper and is misleading, this paper is not on learning with out-of-distribution instances.\nThe other important point is about catastrophic forgetting phenomena in online setting of their approach, which was not addressed thoroughly in the paper. How not to forget what the model has previously learnt a test-time training? I see this somewhat has been empirically shown in Fig 2 with accuracy on the original data, but what is the mechanism not to forget what have being learned so far?\nBesides generalization enhancement, the advantage of test-time self-supervision over training-time joint self-supervised is not clear for the readers, particularly considering the problem of the pitfall of catastrophic forgetting phenomena in test-time training. This pitfall does not exist for training-time joint self-supervised approach. What are the advantages of this approach?\nThe claims (about synthetic shift in distribution shift) are well supported in a series of experiments, where the distribution shifts are synthesized using adding some non-adversarial perturbation to clean images. However, the other common experiments on real shifts in distribution (e.g. SVHN for MNIST and vice versa, and those performed in Volpi et al  2018) are missing, which can help the paper to being better supported and justified for its practicality.\n[Volpi et al  2018]: Generalizing to Unseen Domains via Adversarial Data Augmentation, NIPS 2018\nI found the paper rather difficult to follow and not very coherent in its organization. The main idea is fundamentally simple, but it is still difficult to get it from the text. It needed me 2-3 readings before really getting the point of the paper.\n** Update ** I read other reviews and comments. The answer of the authors to my comments are somehow satisfactory, especially the point of changing from \"out-of-distribution\" to \"domain shift\", which avoid some confusion. I upgraded my rating to a \"weak accept\".", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is on a new approach approach to transductive learning. Reviewers were a bit on the fence. Their most important objection is that the performance improvements that the authors report almost entirely come from the "online" version, which basically gets to see the test distribution.  That contribution is nevertheless, in itself, potentially interesting, but I was surprised not to see comparison with simple transductive learning from semi-supervised learning, learning with cache, or domain adaptation, e.g., using knowledge of the target distribution to reweigh the training sample, or [0], on using an adversary to select a distribution consistent with sample statistics. I encourage the authors to add more baselines, analyze differences with existing approaches, and, if their approach is superior to existing approaches, resubmit elsewhere. 

[0] http://papers.nips.cc/paper/5458-robust-classification-under-sample-selection-bias.pdf
Decision: Reject