Test Time Adaptation via Conjugate Pseudo-labels
Authors: Sachin Goyal
Mingjie Sun
Aditi Raghunathan
J Zico Kolter
Venue: NeurIPS 2022 Accept (2022)
Keywords: Test Time Adaptation
Domain Adaptation

Abstract: Test-time adaptation (TTA) refers to adapting neural networks to distribution shifts, specifically with just access to unlabeled test samples from the new domain at test-time. Prior TTA methods optimize over unsupervised objectives such as the entropy of model predictions in TENT (Wang et al., 2021), but it is unclear what exactly makes a good TTA loss. In this paper, we start by presenting a surprising phenomenon: if we attempt to $\textit{meta-learn}$ the ``best'' possible TTA loss over a wide class of functions, then we recover a function that is $\textit{remarkably}$ similar to (a temperature-scaled version of) the softmax-entropy employed by TENT. This only holds, however, if the classifier we are adapting is trained via cross-entropy loss; if the classifier is trained via squared loss, a different ``best'' TTA loss emerges.
To explain this phenomenon, we analyze test-time adaptation through the lens of the training losses's $\textit{convex conjugate}$.  We show that under natural conditions, this (unsupervised) conjugate function can be viewed as a good local approximation to the original supervised loss and indeed, it recovers the ``best'' losses found by meta-learning. This leads to a generic recipe than be used to find  a good TTA loss for $\textit{any}$ given supervised training loss function of a general class. Empirically, our approach dominates other TTA alternatives over a wide range of domain adaptation benchmarks. Our approach is particularly of interest when applied to classifiers trained with $\textit{novel}$ loss functions, e.g., the recently-proposed PolyLoss (Leng et al., 2022) function, where it differs substantially from (and outperforms) an entropy-based loss. Further, we show that our conjugate based approach can also be interpreted as a kind of self-training using a very specific soft label, which we refer to as the $\textit{conjugate pseudo-label}$. Overall, therefore, our method provides a broad framework for better understanding and improving test-time adaptation. Code is available at https://github.com/locuslab/tta_conjugate.

Test-Time Adaptation via Conjugate Pseudo-labels
Sachin Goyal"!
Mingjee Sun"! Aditi Raghunathan' Carnegie Mellon University, BBosc Center for Al {sachingo, mingjies, raditi, [kolter@ccs cmu. edu
Zico Kolter
Abstract
Test-time adaptation (TTA) refers to adapting neural networks to distribution shifts, with access to only the unlabeled Prior TTA methods optimize over unsupervised objectives such as the entropy of model predictions in TENT (50). but it unclear what exactly makes good TTA loss. In this paper, we start by presenting a surprising phenomenon: if we attempt to meta-learn the "best" possible TTA loss over wide class of functions, then we recover a function that is remarkably similar to (a temperature-scaled version of) the softmax-entropy employed by TENT. This only holds, however, if the classifier we are adapting trained via cross-entropy loss; if the classifier is trained via squared loss, different "best" TTA loss emerges. To explain this phenomenon, we analyze test-time adaptation through the lens of the training losses's 's convex conjugate. We show that under natural conditions, this (unsupervised) conjugate function can be viewed a good local approximation to the original supervised loss and indeed, it recovers the "best" losses found by meta-learning. This leads to a generic recipe that be used to find good TTA loss for any given supervised training loss function a general class. Empirically, our approach consistently dominates other TTA alternatives over wide range of domain adaptation benchmarks. Our approach is particularly of interest when applied to classifiers trained with novel loss functions, e.g., the recently-proposed proposed PolyLoss [25] function, where it differs substantially from (and outperforms) an entropy- based loss. Further, we show that our conjugate based approach can also be interpreted as a kind of self-training using very specific soft label, which we refer as the conjugate pseudo-label. Overall, our method provides broad framework for better understanding and improving test-time adaptation. Code is available https: tttps://githule ctt.cuou..ccmmocuc.laaaaaaaaaa Eta_conjugate
samples from the new domain at lest-time.
1 Introduction
Modern deep networks perform exceeding well on new test inputs that are close to the training distribution. However, this performance dramatically decreases test inputs drawn from a different distribution. While there a large body of work on improving the robustness of models, most robust training methods are highly specialized to the setting they cater to. For e.g., they assume pre-specified perturbations, subpopulations, and spurious correlations, or access to unlabeled data from the target distribution, and most methods offer close to no improvement on general distribution shifts beyond what they were trained for 112 21
In practice, it often cumbersome (or even impossible) to precisely characterize all possible distri- bution shifts model could encounter and then train accordingly. Instead, model already trained on some source data must be able to adapt at test-time to new inputs from a different domain. This
setting of test-time adaptation (TTA) has gained interest in recent years [6] 47 50 [54). TTA is typically accomplished by updating the source model parameters via few steps of optimization on an unsupervised objective involving the new test sample from the target distribution. The choice
Equal Contribution
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

of this unsupervised objective, which we call the TTA loss, dictates the success of the adaptation procedure. [47] uses self-supervised objective on the test sample, [50] uses the entropy of model predictions, and several follow-ups have proposed variants or alternatives [40] [5]. However, it remains unclear to how choose or guide the selection of this TTA loss, and thus far the choice of these losses has remained largely heuristic in nature.
In this work, we begin by presenting set of intriguing experiments where we attempt to learn the "best" TTA loss for a given source classifier and distribution shift. We parameterize the TTA loss by another neural network whose parameters are learnt via meta-learning B3 9 where we differentiate through the adaptation process to find the TTA loss that achieves the best adaptation on distribution shifts. Surprisingly, we ultimately learn a TTA loss that looks remarkably similar to (a temperature-scalt version of) the softmax- entrrpp loss, which was already proposed by [50]. Why did we recover the commonly used softmax-entropp loss despite the fact that the procedure is capable of learning very general class of losses and the meta-learning process could potentially specialize to both the source classifier and the distribution shift of interest? Furthermore, we find that this pattern only holds when the loss used to train the source classifier is cross-entropy loss; when different loss such as squared loss is used instead, the meta-learning procedure recovers a TTA loss that itself looks more like a negative squared error, and is very different from the softmax-entropy loss (Section 3
In order to explain this phenomenon, we propose to consider TTA through the lens of the convex conjugate function. Specifically, given a hypothesis function ((x)) and label y, several common losses (cross-entropp and the squared loss amongst them, but not limited to these) can be written in the form C(h(r), y) = =f((=)) - y (xx) for some function f. In these cases, we show that "natural" TTA loss for such classifiers is precisely the (negation of) the convex conjugate evaluated at the gradient of h, CTTA = f (V (h(x)), where f* is the convex conjugate of f. This framework not only recovers the results of our meta-learning experiments, but also justifies why some specific choices of TTA loss in the previous literature work well (e.g., this framework recovers TENT's choice of softmax-entropp for cross-entropy-trainee classifier). Moreover, it also provides a broad framework for what the TTA loss should be when the source model is trained using various different loss functions (for example the
PolyLoss [25] (9]) as is becoming increasingly common in machine learning. Further, we show that our proposed conjugate adaptation loss is in fact a kind of self-training with pseudo-labels [42) a classic approach in machine learning. Various formulations of the pseudo-label have been proposed in the literature, and our conjugate analysis provides a general recipe for the "correct" choice of soft pseudo-labels given by ((rr) = (h(())) We thus refer to these as conjugate pseudo-labels (Conjugate PL,s), and believe our work provides a broad framework for understanding adaptation with unlabeled data in general.
Finally, we empirically verify the effectiveness of our proposed conjugate adaptation loss across several datasets and training losses, such as cross-entropy and squared loss, along with the recently- proposed PolyLoss (25) (which itself has shown higher standard test accuracy on a wide range of vision tasks). Over all models, datasets and training losses, we find our proposed conjugate useudo-labeling consistently outperforms prior TTA losses and improves TTA performance over the current state of the art.
2 Background and preliminaries.
Test-time adaptation. We are interested in mapping an input x E IRd to a label y E y. We learn a model he Rd + RD' parameterized by 0 that maps an input z to predictions (z)).. We assume access to a trained source model and adapt at test-time over the test input, before making the final prediction. This is the standard test-time adaptation (TTA) setting 50). During TTA, we update the model parameters on an unsupervised objective C(x, ha). For example, in TENT [50), this loss is the entropy of the peftmax-normalinn observe a batch of test inputs and we take a gradient step towards optimizing the TTA loss on this test batch. As is standard, we measure the average online performance of models across all steps (number of test batch inputs seen) in the adaptation process.
predictions of the model. At each time step of adaptation,
Meta learning the loss function. In order to explore the existence of different TTA losses, we employ the meta-learning procedure where we attempt to learn the TTA loss. We use similar procedure as prior work on meta-learning loss functions [3] 37 and parameterize the loss function via a neural network ms RD + IR that takes in the model predictions/logits and outputs a loss value. We want to learn parameter è such that when we update 0 via the loss function my. our final

---

performance is optimal. In order to do so, let z be the unlabeled test samples to adapt to, and y be the corresponding labels. We update & and @ alternatively as follows.
0++1 t 0¹ - Omg (hae (x)) db+11 t 0² (1')
(1)
00
OΦt
where L is some supervised surrogate loss function such cross-entropy. Please refer Appendix A3 for further details regarding meta-learning setup. Note that the meta-learning process above assumes access to labels y of test inputs. In this paper, do propose meta-learning the TTA loss as an approach. Rather, we use meta-lcarning to explore what the "best" TTA losses look like. We discuss our findings from this exploration in the next section.
3 Test-time Adaptation via Meta-Learnt Losses
The objective used in TENT is the softmax- softmax-entropy of the model predictions which essentially makes the classifier more confident in its current predictions. The same can be achieved by various other loss formulations such as those mentioned in 40). With so many possible choices for the loss function, what should use for TTA? In this section, we attempt to answer this empirically and present some intriguing observations.
Meta Loss for Classifier trained with Cross Entropy
Meta Loss Classifier trained with Squared Loss
meta (enor 12.99%)
entropy (enror 13. 14%)
24
rited quadi Jeroor 13.200)
riteed 12.08%)
M A
loss value
loss value
10
15-
-2-
16-
14
-10
10 20
2.0
-0.5
0.0
0.5
1.10
1.5
2.0
prediction score
prediction score
(a)
(b)
Figure 1: Visualization of meta loss (blue) by varying one input prediction score. (a) For cross-entropy loss trained model, the learnt meta loss can be approximated with scaled softmax-entropy function (dashed red). (b) When the source model is trained with a squared loss for classification, the learnt meta loss (blue) can be fitted closely with quadratic function (dashed red), shown in |Figure 1b The range (max/min) of the prediction score (logit) in x-axis is chosen to cover the empirical range of the predicted logits.
We learn the TTA loss parameterized by a neural network via meta-learning as described in Section Our source classifier is a ResNet-26 trained on CIFAR- 10 and we adapt to distribution shifts in CIFAR-10-C. We use the 4 labeled validation noises in CIFAR-10- to learn the meta-loss network parameters and we denote the resulting learnt loss function by meta-TTA loss. We then adapt the source classifier to the test set of 15 corruptions by optimizing the meta-TTA loss.
Experiment 1.
Observations. First, find that TTA using meta-TTA loss performs better than TENT (12.35% vs 13.14%), suggesting that there are better TTA losses than previous losses based on softmax-entrun entropy.
However, on examining this meta-TTA loss, we find surprising observation. Figure Ia (blue curve) visualizes the learnt meta-loss over model predictions as vary single class prediction with the rest fixed, Qualitatively, the learnt meta-loss looks very similar to softmax-entropy in one dimension. In fact, we can fit it closely with scaled softmax-entropy function (dashed red curve): a (sooftmax( (ho (z)) (T)) where a is a magnitude parameter and T is a temperature scaler. We want to test if the meta-loss is basically learning the softmax-entropi function. Hence, perform dest-time adaptation with the fitted softmax- entropp function instead (dashed red curve) and achieve an error of 12.32%, essentially recovering the performance of meta-TTi
3

---

Despite the ability to represent many different loss functions and potentially specialize to the CIFAR- 10-C setting, the meta-loss procedure gave back the standard entropy objective. Do we always recover a loss that looks like coftmax-entropy
In an attempt to isolate when we get back the entropy objective, we vary several things. We tried different architectures for the souree classifier, different losses C during the meta- learning process I and different training losses for the source classifier.
Experiment 2.
Results. We observed that we consistently recovered the temperature scaled softmax- entropp
function in all cases except when we varied the training loss for the source classifier (Appendix [A. I0) On using the squared loss function [18,, a strikingly different meta- TTA loss emerges. Figure Ib (blue curve) shows the learnt meta-loss (13. 3.48% error) for this network. Here again, the meta-TTA loss outperforms entropy (14.57%) but it is not simply due a scaling factor. The loss now looks like the negative squared error (red curve). Like previously, we tried fitting quadratic loss directly to the meta loss in Figure 1b, and this time we even slightly outperformed the meta-TTA loss.
To summarize, we used a lectn-le procedure to search for the "best" TTA loss, where the loss itself was parameterized by a neural network that could potentially represent arbitrarily complex loss functions. However, we ended up with loss functions displaying remarkable structure: across different architectures and different variants of meta- meta-learning. for a classifier trained with cross-entropy, the meta-TTA loss was temperature scaled softmax-entrops and for a classifier trained with squared loss, the meta-TTA loss was a negative squared loss. This is interesting from both a practical and conceptual standpoint where the "best" TTA loss depends on the loss used to train the source classifier in clean fashion. We attempt to understand and explain this phenomenon in the next section.
4 Conjugate Pseudo Labels
Results in the previous section raise an obvious question: why does softmax-entropy as used in TENT seem to be the "best" possible test time adaptation loss for classifiers trained via cross-entropy (at least, best in the sense that meta-learning consistently recovers something which essentially mimics softmax-entropy, even though meta-loss is parameterized by a neural network and hence could learn much more complex functions specific to the model and the particular shift)? And why, alternatively, does a quadratic TTA loss seem to perform best when the classifier is trained via squared loss?
In this section, we offer an explanation of this phenomenon via the construct of the convex conjugate function M. As we will see, our method recovers softmax-entropp and quadratic loss as the "natural" objectives for classifiers trained via cross-entropy and squared loss respectively. Furthermore, for classifiers trained via other loss functions, as is becoming increasingly common in deep learning, our approach naturally suggests corresponding test-time adaptation losses, which we show in the next section to comparatively outperform alternatives. Thus, we argue that our framework overall provides compelling recipe for specifying the "correct" method for TTA for a large class of possible losses.
4.1 Losses and the convex conjugate
We begin by formally considering loss functions between hypothesis output (a(x)) (e.g., the logit outputs of classifier, or the direct prediction of a regressor) and target y that take the following form
f(ho(z)) ((h(()) ( - y" ho(z)
for some function fü when there is no risk of confusion, we will use h in place of he for simplicity of notation. While not every loss can be expressed in such a form, this captures a wide variety of
common losses (possibly scaled by constant value). For example, cross-entropy loss corresponds to the choice (h) = log Σ, (h,) and where y denotes a one-hot encoding of the class label; similarly, squared loss corresponds the choice (h) 1111112
When training an over wer--arameterized classifier, we can roughly view the training process as (approxi- mately) attaining the minimum over hypotheses h for each training example
min- 6
((ho(T:), ~
min C(h, v1)
(3)

---

where t is the number of training samples. However, in the case of losses in the form (2) the minimization over in this form represents very specific and well-known optimization problem: is known as the convex conjugate [1] of the function f
min [(,, v) min(f(h -y = h
h
where f denotes the convex conjugate of f. f a convex function in y (and indeed, is convex regardless of whether or not f is convex). Furthermore, for the case that f is convex differentiable, the optimality condition of this minimization problem is given by Vf(h - y, also have that
(v( f (V
(5)
where 0 refers to the optimal classifier (used interchangeably with hvv) ). Putting this all together, we can state (admittedly, in rather informal manner) that under the assumption that 0'pr is chosen so as to approximately minimize the empirical loss on the source data the over-parameterizee setting,
we have that for inputs
C(geen (x.). ~ Σ ==1
("V (x+)))
(6)
i.e., the empirical loss can be approximated by the (negative) conjugate applied to the gradient of the f. at least a region close to the optimal 0pp that minimizes the empirical loss. But the later expression has the notable benefit that it does not require any label yi in order to compute the loss, and thus can be used as a basis for TTA on target domain of the hypothesis function hive.
Definition (conjugate adaptation loss) Consider a loss function that takes the form given in 2 used for truining a hypothesis hig in the over-parumeterized regime. define the conjugate adaptation loss Counj (uai(he(2)) (z)( RD1 R as follows.
(ha())) f- f'
(f((t))))(((((())))) ho(x).
4.2 Recovery of existing test-time adaptation strategies
The interesting aspect to this formalism is that when applied to classifiers trained with cross-entropy, it recovers exactly the TENT approach to TTA : minimizing the softmax-entropy of (x)) And indeed, this loss also recovered when using meta-learning to learn the "optimal" test-time adaptation loss. To see this, note that for cross-entropy, we have that f(h) = log Σ, cxp(h,),
Cross-entropy
exp(h**) and the conjugate function giving the optimality condition y = f(( = Σ, exp(N(")
(*(y) {
E, ve Liullgg if E
(8)
2
otherwise
In other words,
( - = =-E, exp(h:) log exp(hi) E3 exp(h;) Ej exp(h;)
(9)
ie. softmax-entropy of the model prediction, which is exactly the TTA loss that TENT uses.
Squared loss For the squared loss, we have that /(h) 1 1 leading to the optimality condition
y h and conjugate function (v) ||ll Hence, the adaptation loss in this case would be
simply given by (hmm((h - ft (z)()))) ||//1/3 which is also what observed in
the meta-learning experiments discussed in Section
4.3 Conjugate pseudo-labels
We now emphasize that by the nature of our approximations, there is an additional simple interpre- tation of the conjugate loss: it is also equal to the original loss (2) applied to the "psuedo-labels" CPL
Vg (z) = (ho( where CPL refers to conjugate pseudo-labels, i.e.,
C (hTn = - -f (V = f(h (x))
(((hho()))Tho(())) = ((h Vz(he(z))).
(10)
5

---

This property is known as the Fenchel- Young inequality, that is f(x) + (*( 2 x u holding with equality when u fz)). In other words, our conjugate adaptation loss is precisely equivalent to self-training under the specific soft pseudo-labels given by g V (=)) And indeed, for many cases, this may be a more convenient form to compute than explicitly computing the conjugate function at all. For this reason, we refer our method that of conjugate veudo-labels.
In the case of cross-entropy loss, this approach then corresponds exactly to self-training using labels given by the softmax applied to the current hypothesis. We must emphasize, however, that while conjugate formulation indeed has this "simple" form for the case of cross-entropy loss, the real advantage comes in that it provides the "correct" pseudo-label for use with other losses, which may result in pseudo-labels different from the "common" softmax operation.
Example: conjugate pseudo-labels for PolyLoss. PolyLoss 25 is recentlyproposed simple alternative to gross-entrop loss than has been shown to improve performance across a wide variety of compute tasks. This loss is given by the form
poty Chaty(ha((e) (ha( v) (eo(ho()) te. of (ol)
(11)
We note that this can be put exactly into our conjugate form (equation 2 by writing the loss in slightly more involved fashion, which we refer to as the expanded conjugate form
poly (x). y) = f(ho(z)) - v² g(ho(x)).
(12)
where f is the log-sun log-sum-exp function as before, and g(h) = h c(1 softmax(h)), In order to formally put this into the form of the previous loss function (equation 2 we can simply define an alternative hypothesis as the function hi (x) = ((h)) and then define PolyLoss in the conjugate form as
pplly (hb(2)) v) =
-
(13)
Typically, however, is easier to simply operate on the expanded conjugate form, which yields the optimality condition for the pseudo-label V fh = Dg(h" )0% (I), where D is the Jacobian operator. For the case of PolyLoss, this leads the conjugate pseudo-label of the following form: yg
JPPL = (I + ediag(2) Ezz
z softmax(heer))).
Test-time adaptation. Finally, we note that the above discussion doesn't 't actually address any topics related to test-time adaptation to OOD data, but merely provides generic characterization of a self- training procedure for generic loss functions of the form 2). However, the application to TTA on OOD data fairly straightforward: as long as the learnt source parameters 0 is a reasonable approximation to the true optimal 6ePl on the shifted domain, self-training with the conjugate pscudo-labels provides reasonable proxy for fine-tuning the network on the true OOD loss. We emphasize that, common to most approaches for TTA there are still some amount of design decisions that must be put in place; these are detailed in Section 5.1 In practice, benefits (across all baselines) from an additional "temperature" scaling, i.e., applying the TTA loss to ho(x)/TT /T for some fixed temperature T', although it requires a held-out validation dataset for tuning T. However, we should emphasize that truly unsupervised TTA would require making an informed guess for the value of these hyper-parameters The full procedure for test time adaptation via conjugate pseudo-labels is shown in Algorithm
observe OOD generalization typically
Algorithm 1 Conjugate pseudo-labeling (Conjugate PL)
Input: Source classifier 0g trained using loss ((ha(z), v) = f(h((z)) - ho(x)) Ty N batches of test data Diest = T2, NN
Hyperparams: learning rate 7) and temperature T.
Let (x))) def he(x)/TT be the temperature scaled predictor.
Let 90 (IP)(r)) denote the conjugate pseudo-label function PP v([(ho
for n = 0, 1......N N 1 do
On+1 = On - v
(TVCCho(z)) ((())))) 50 (xm)(x)))
[Self-training with conjugate pseudo-labels]

---

Experiments
In this section, we empirically evaluate the effectiveness and generality of the proposed conjugate pseudo-labeling procedure (Algorithm for test-time adaptation on a variety of datasets.
5.1 Setup
Datasets. We evaluate on the three common corruption benchmarks: adapting classifier trained on CIFAR-10 to CIFAR-1O-C, CIFAR- 100 to LLFAR-100-C and ImageNet to ImageNet-C [15]. Following the previous works [4 50], report the averaged across corruptions the highest severity for CIFAR- 1LFAR--10110-C and averaged across corruptions and severity level for ImageNet-C. We also evaluate three domain adaptation datasets: adapting classifier trained on SVHN to MNIST, an ImageNet classifier to ImageNet-R [6] and adapting from synthetic to real data in VISDA-C 38
ResNet-26 [4] as the source classifier architecture for CIFAR- 10/100 experiments. ResNet- 18 for SVHN to MNIST and ResNet-50 for ImageNet and source synthetic data VisDA-C. We consider source classifiers trained via the following loss functions: the de-facto cross cross-cntropy, recently proposed polyloss [25] and squared loss [18
Models and Training losses. Following previous works on TTA 50]
Baselines. Our proposed conjugate pseudo-label is the classic approach of self-training with a specific form of pseudo-labels. In self-training, we replace the label y with a pseudo-label 77x)) and adapt by optimizing the loss function C(ha (x), j(x)). Note that we could either instantaneously update the pseudo-labels using the current classifier, or generate pseudo-labels once with just the source classifier. Instantaneous updates have been shown work better for domain adaptation 7 440, and we perform instantancous updates for all methods. While we propose using y jCPl
(ha (z)) (See Section 4.3). we compare the standard pseudo-labels used in the literature:
(i) the "hard" pseudo-label (hard PL) where j(x) = arg max; (ho(#)); is the most likely class as predicted by hg. As common the self-training literature, we perform confidence thresholding. (ii) The "soft" pseudo-label (soft PI.) where j(x) is obtained by applying softmax function to the model predictions hg(a).
We also compare with the following recently proposed test-time adaptation methods.
Entropy Minimization (ENT) 500 minimizes the entropy of model predictions.
Robust Pseudo-Label 40 where minimize robust classification loss, Capt q -1-
where = urgmax;p(f) ()) and E [0, 1].
MEMO [54] minimizes entropy of model's outputs across different augmentations of a test input.
We implement batch version, where we see multiple test points at once, for fair comparisons.
fine-tune by updating the learnable scale and shift parameters of the batch normalization layers across all adaptation losses. For each batch, batch normalization statistics is also updated, as suggested in [41. We report performance at the end of one round of test-time adaptation over the entire test set.
TTA methodology. Following [50 and 40
We tune the learning rate (LR) and temperature (T) on the validation noises in the corruption
benchmark by grid-search. LR is selected from {1e -Le-', le 1 .... 1e 1e=+) and T from 21, .2...5...... 5}. All the experiments have been performed on A6000 GPU's. On domain adaptation benchmarks, where there is no held-out target domain, the same hyperparameter tuning protocol across all methods. We single out temperature as a very important hyperparameter, as we discuss in the results below.
set T to be and use the LR suggested by [6 50). We use
5.2 Results on classifiers trained with cross-entropy
We study the effectiveness of our proposed conjugate pseudo-labels when the source classifier is trained via cross-entrops loss. In this case, baselines Softmax PL and ENT are the same as Conjugate PL. Thus we omit them in our results. Table reports the performance of various TTA methods.
When the source classifier is trained via cross-entropy, our conjugate pscudo-label algorithm exactly corresponds to entropy minimization with additional temperature scaling. Entropy minimization as
7

---

Temperature (T)
Conjugate PL (ENT)
Dalasel
Hard
Robust PL
MEMO
13.95 (100.6)
13.97 (10.04)
12.60 (#:0.04)
13.07 (+0.05)
CIFAR-IO-C
13.95 (+0.066)
12.85 (+0.04)
12.51 (+0.01)
12.51 (+0.03)
X
45.22 (#:0,4)
39.80 (#0,18)
38.52 (#0.16)
41.15 (#0.225)
CIFAR-100-C /
45.22 (=:0.4
36.37 (=00100
37.38 (+0.06)
36.10 (#0.07)
45.43 (++0005)
45.68 (+0.01)
48.91(+0.03)
45.82(+00.0))
ImageNet-C:
45.43 (+0.05)
45.61 (10.01) 48.91(=0.04)
45.36(=0.01)
Table 1: Mean errors when adapting to corruptions using a source classifier trained via cross- entropy loss. Here, conjugate pseudo-labeling becomes softmax- -entrop minimization. With the right temperature scaling, softmax-entropp minimization matches or outperforms other approaches. Prior reported gains of other methods over softmax-entropy minimization disappear when we use temperature scaling. For additional context, the source classifier errors without adaptation are: CIFAR-10-C (29.54%), CIFAR-100-C (62.26%), ImageNet-C (61.8%%)
proposed in prior work [50] does not tune the temperature parameter, and some newer objectives such as robust PL or MEMO outperform vanilla entropy minimization. For example, on CIFAR-100-C, vanilla ENT obtaines 41. 41.15% average error, while robust PL improves this to 39.80% and MEMO to 38.52% However, with the right temperature scaling, entropy minimization obtains 36.10 10% error which outperforms the newer objectives (with and without temperature scaling). A similar observation holds for CIFAR-I0- 10-C and ImageNet-C as well. Essentially, the gains over vanilla entropy minimization vanish when we do temperature scaling, and entropy minimization (i.e. conjugate pseudo-labeling corresponding to cross-entropy) turns out to be the best objective after all.
5.3 Results on classifiers trained with polyloss and squared loss
In the case of cross-entropy, conjugate pseudo-labeling reduces to the familiar notion of entropy minimization. We now explore the performance our method on different loss functions where the conjugate pseudo-labels differ substantially from entropy minimization (section 4.3). Table 2 presents the results on the corruption benchmarks and Table the results on the other domain adaptation datasets for source classifiers trained with PolyLoss.
Conjugate PL. (Ours)
Dataset
Hard
Robust PL.
ENT
MEMO
Sofimax PL.
13.81(+0,12)
(4,23(+0.02)
13.46(+0.06)
13.23(=0.07)
14.64(+0.11)
13.02(+0009))
CIFAR-10-C
13.81(=0.12)
12.45(+0.05)
12.23(=0.06)
12.33(=0.04)
12.26(+0.04)
12.08(=0.05)
42.86(10.11) 40.47(10.05)
40. 40.12(11.008)
39.90(10.05)
41.00(10.11)
38.17(10..1)
CIFAR-100-C
40.47(=0.00)
39.80(+0.088
38.23(11005)
39.23(=0.04)
37.04(10.00)
36.83(=0008)
ImageNet-C
45.4((+0221) 45.44(+0.21)
46.27(+0003)
46. 46.10(=#0033
48.21(=0.055)
44.63(+0.03)
44.01(=0.01)
46.27(++0.03)
45.50(++0.22)
48.21(+0.04)
44.45(+0.03)
44.01(+0.01)
Table 2: Mean errors when adapting to corruptions using source classifier trained via recently proposed Poly-1 Loss (25). Conjugate
consistently outperforms all previous ap- proaches, For additional context, source classifier errors without adaptation CIFAR-10-C (30.22%), CIFAR-100-C (63.91 %) and ImageNet-C (62.18%)
First, we note that, across all datasets Table 2 and Table our conjugate PL approach outperforms all other TTA losses. With polyloss classifiers, entropy miniiization is no longer the best method on CIFAR-100-C. entropy minimization achieves 38. 23% error while our conjugate PL achieves 36.83%. We see similar consistent gains on CIFAR-10-C, ImageNet-C, ImageNet-R and VisDA-C. On digit adaptation tasks from SVHN MNIST/USPS/MMNIITM where there larger shift between source and target, the gains are especially pronounced. Figure 2 compares how the task loss (polyloss c = 6) on the test data decreases
adapt the model through conjugate PL and other baselines. We use
CIFAR-10 10-C as an example. Observe that our proposed conjugate PL indeed reduces the task loss the most among other baselines.
8

---

Source
Hard
Robust
Softmax
Conjugate PL
Task Evaluated Test Data
Dataset
Entropy
Eror
Pt.
P..
Ours
Hard
SVHN - MNIST
28.33
20.21
19.73
14.28
16.54
10.73
Task Loss
Conpupate
SVHN - USPS
31,58
23.32
26.12
23.12
24.07
21.62
SVHN
MNISTM
61.69
50,73
51.35
49.33
50.47
47,59
ImageNer-R
64.19
58.52
59.46
58.25
56.62
55.63
ViDD-CC
58,13
40.45
45.44
44.11
39.63
38.42
17
Iterations
Table Target error when adapting models trained via polyloss on source domains across different domain adaptation bench- Figure 2: Task Loss (PolyLoss e marks. Conjugate pseudo-labeling offers consistent and substan- evaluated on CIFAR-10-4 test tial gains over previous approaches across three datasets.
data during test-time adaptation.
Furthermore, on CIFAR-IO-C and ImageNet-C, we find that adapting polyloss classifiers via conjugate PL improves the performance over all methods applied to cross-entropy trained source classifiers. For e-g., ImageNet-C, the performance improves from 45.34% 44.01%. %. However, this is only true when using the proposed conjugate PL. we just did softmax entrrpp minimization (even with temperature scaling), the final adapted performance of a polyloss classifier (45.5%) is in fact worse than that of a tross-entropy classifier (45.34%). Our results suggest that as we develop new training losses that improve the source classifiers, it important to adapt via conjugate pseudo-labeling to reap the maximum gains.
Similarly, we experiment with the case when the source classifier is trained using squared loss on the C1 and CIFAR-100 datasets, and observe consistent gains using the proposed conjugate pseudo-labels over the basclines. For example, on LIFAR-10-C. TTA using conjugate PL gives and error of 12.87%, outperforming basclines like ENT (13.24%) and Softmax PL (31.81%). Table 5 in
Appendix A.7 shows the detailed results.
Comparing Table aan Table 2 we see that the relative ordering between the various baselines differs. This is further evidence that the adaptation loss has to depend the training loss, and we believe conjugate pseudo-late leeull-label approach captures this appropriately by offering consistent gains across the various settings we experimented with.
6 Related Works
Test-time adaptation methods. In recent years, the setting of test-time adaptation has gained lot of interest with host of different approaches proposed in the literature. One family of TTA approaches update the source classifier by minimizing an unsupervised loss on the target
distribution 4 6 20 22 35 36
40
43
44
50
51 54]. TENT [50 proposes to minimize the propose
entropy of model predictions at test time. Several follow ups like 6
35| 40
54
alternative TTA objectives, e.g. robust pseudo-labelling 40, likelihood ratio loss 35
entropy of
marginal probability averaged across augmentations (54) and self-supervised contrastive losses 6 49)
However, most of these objectives are heuristically designed or chosen. In this paper, we provide a principled approach of designing unsupervised objectives for TTA.
Another family of approaches for test-time adaptation such as auxiliary self-supervised task (e.g. rotation prediction [47], masked autoencoders (10) to update model parameters on each test sample. Crucially, these methods require modifying the source model training by augmenting the supervised training objective with an auxiliary self-supervised loss. Hence it cannot be applied to typical standard classifiers that are trained by minimizing supervised loss the source data.
leverage an
Source-free domain adaptation. A very related setting to test- time adaptation is source- domain adaptation, where a trained source classifier must be adapted to a target distribution of interest, although the entire target unlabeled data is available at once. SHOT (28] proposes to optimize the source hypothesis (i.e. feature extractor) with combination of entropy minimization, diversity and self self-training on pseudo-labeld on the unlabeled target data. (53) promotes feature clustering on features from target distributions. [24] (26] use generative modeling estimate the underlying source distributions for enforcing feature invariance. Such approaches typically require multiple epochs over the target data and cannot be casily adopted to work in an online fashion.

---

Unsupervised domain adaptation. The most canonical setting of domain adaptation involves access to labeled source data and unlabeled target data, all during training. The availability of source and target data during training lends itself to approaches that "align" the source and target representations
in some way: (3 33
match distribution statistics, [TT] uses a discriminator, [46] uses self-supervised learning. However, such approaches require access to source data which might not always be feasible due data privacy and efficiency issues.
45
48
Pseudo-labels and self-training. Self-training a classic idea for leveraging unlabeled data, developed first for the semi-supervised setting. Self-training generates pseudo-labels on the unlabeled data, allowing us use any "supervised" loss on this pseudo-labeled data. Self-training has shown promising results in various settings like semi-supervised learning [19] and improving adversarial robustness [5]. Self-training has also been gaining attention in the setting of unsupervised domain adaptation [28] 39], where pseudo-labela generated on the unlabeled data from target domain is used
to supervise the adaptation process. 7 23 52 provide theoretical insights into how self-trainin with pseudo-labels can help under distribution shift. TENT 50) (i.e entropy minimization) can be viewed as a form of self-training with instantaneous softmax pseudo-labels. Our work provides general framework for the choice of soft pseudo- labels based on the conjugate analysis of the source training objective. Some prior works like [7 27 30 55 56 have documented the improvement in performance when using instantaneous pseudo-labels over pre- pre-computed pseudo-labels, and thus lend further support to the benefits of our proposed conjugate pseudo-labeling approach. The experiment results presented in this work supporting conjugate pscudo-labels suggest that conjugate pseudo-labels is a promising direction of pseudo-labeling labeling in a broader context.
7 Conclusion, Limitations and Future Directions
In this work, proposed general test-time adaptation loss, based on the convex conjugate formulation which in turn was motivated by the intriguing meta learning experiments. The fact that meta-learning recovers the proposed loss hints at some kind of optimality of the loss. In Section 4 we prove that for a broad set of loss functions, the proposed (unsupervised) conjugate loss is close to the oracle supervised loss. However, this still does not completely answer what the optimal test-time adaptation loss is and why.
The meta-learning framework in this work was constrained to learn functions over the logits of each individual input. It can be expanded to more involved setups, where we consider functions over the intermediate representations too and also consider learning functions over a batch of input while accounting for their interactions.
Beyond the choice of the adaptation loss itself, achieving good test-time adaptation generally involves several heuristics like updating only the batch norm parameters [50]. While our work was motivated by the loss function, via the meta-learning experiments, we discovered that temperature scaling is another important
that improves the performance of all previous baselines as well. At a high level, test-time adaptation has to be appropriately regularized to prevent the updates over batches from taking the model too far: updating only a few batch norm parameters is one way to do that, and perhaps temperature scaling provides similar beneficial regularization effect by making the network predictions on unlabeled inputs less confident. Understanding the role of these heuristics more concretely is an interesting direction for future work. It also remains an open problem to understand under what sort of real-world distribution shifts would self-training based approaches would help.
Finally, it is also worth extending and applying the conjugate pseudo-labeling to other settings like semi-supervised learning.
8 Acknowledgments
We thank Shubhang Bhatnagar and Asher Trockman for helping with running the ImageNet experi- ments. We thank Zhili Feng for useful feedback. Sachin Goyal and Mingjie Sun were supported by funding from the Bosch Center for Artificial Intelligence. Aditi Raghunathan was supported by an Open Philanthropy AI Fellowship.
10

---

References
[1] http:///aaawapppeeaaaaagaataaa//
[2] Pratyay Banerjee, Tejas Gokhale, and Chitta Baral. Self-supervised test-time learning for reading comprehension. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
[3] Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-learning via learned loss. arXiv preprint arXiv:1906.05337, 2019.
[4] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[5] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Un- labeled data improves adversarial robustness. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bor Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL RtLLtttps///proceeddae neurips Paaer.pp.
[6] Dian Chen, Dequan Wang. Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[7] Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. In Advances in Neural Information Processing Systems, 2020.
[8] Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In Proceedings of the 39th International Conference Machine Learning (ICML), 2022.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
[10] Yossi Gandelsaman, Yu Sun, Xinlei Chen, and Alexei A. Efros Test-time training with masked autoencoders. In Advances in Neural Information Processing Systems, 2022.
[11] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Nomain-adversarian training of neural networks. Journal of Machine Learning Research, 17(59):1-33 2016.
[12] Ishaan Gulrajani and David Lapez-Paz. In search of lost domain generalization. In International Conference Learning Representations, 2021.
[13] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenya, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[15] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[16] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[17] Yosuke Higuchi. Niko Moritz, Jonathan Le Roux, and Takaaki Hori Advancing momentum pseudo-labeling with conformer and initialization strategy. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2022.

---

[18] Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. In International Conference on Learning Representations. 2021.
[19] Dong hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.
[20] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model model-agnostic domain generalization. In Advances in Neural Information Processing Systems, 2021.
[21] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: benchmark of in-the-wild distribution shifts. Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
[22] Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Robustifying vision transformer without retraining from scratch by test-time dass-connitional feature alignment. In International Joint Conference on Artificial Intelligence, 2022.
[23] Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In Proceedings of the 37 th International Conference on Machine Learning (ICML), 2020.
[24] Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In IEEE Winter Conference Applications of Computer Vision (WACV), 2021.
[25] Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: A polynomial expansion perspective of classification loss functions. In International Conference on Learning Representations, 2022.
[26] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[27] Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for semi-supervised few-shot classification. In Advances in Neural Information Processing Systems, 2019.
[28] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th International Conference Machine Learning (ICML), 2020.
[29] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
[30] Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In Advances in Neural Information Processing Systems, 2021.
[31] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurles Taylor Mordan, and Alexandre Alahi. Tut+: When does sulf-super test-time training fail or thrive? In Advances in Neural Information Processing Systems, 2021.
[32] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2013.
[33] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
[34] Xuan Luo, Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. In SIGGRAPH, 2020.
12

---

[35] Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-Time Adaptation Distribution Shift by Confidence Maximization and Input Transformation. arXiv preprint arXiv: 2106.14994 2021.
[36] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng. Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
[37] Junhyuk Oh, Matteo Hessel, Wojciech M. Czamecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. In Advances in Neural Information Processing Systems, 2020.
[38] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge, 2017.
[39] Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[40] Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Vincent Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. If your data distribution shifts, use self- learning, 2022. URL httpps: www.://eooommmdddoeeee.e.
[41] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems, 2020.
[42] H. Scudder. Probability of error of some adaptive pattern- recognition machines. IEEE Transac- tions on Information Theory, 1965.
[43] Manli Shu, Weili Nie, De-An Huang. Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, 2022.
[44] Prabhu Teja Sivaprasad and François Fleuret. Test time adaptation through perturbation robust- ness. arXiv preprint arXiv: 2/ 10232, 2021.
[45] Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. arXiv preprint arXiv: 16/2.0/939, 2016.
[46] Yu Sun, Eric Tzeng. Trevor Darrell, and Alexei A. Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv: 190:. 11855 2019.
[47] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
[48] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint 1rXiv:1412.34777 2014.
[49] Dequan Wang, Shaoteng Liu, Sayna Ebrahimi, Evan Shelhamer, and Trevor Darrell. On-target adaptation. arXiv preprint arXiv: 2109.01087, 2021.
[50] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[51] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[52] Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. In International Conference Learning Representations, 1021.
13

---

[53] Shiqi Yang, Yaxing Wang, Joost van de Weijer. Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[54] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, 2022.
[55] Yang Zou, Zhiding Yu, B. K. Vijaya Kumar, and Jinsong Wang. Domain adaptation for semantic segmentation via class-balanced self-training. European Conference on Computer Vision, 2018.
[56] Yang Zou, Zhiding Yu, Xiaofeng Liu, B. V. K. Vijaya Kumar, and Jinsong Wang. Confidence regularized self-training. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We add the limitations of our work in the end. See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7 We do not foresee any specific negative societal impact of our work, apart from general guidelines valid for deep learning in general.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL.? [Yes] We will be submitting the code along with the supplementary material. We will be adding detailed readme for replicating the experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the implementation details like the gridsearch and validation data have been mentioned in the paper, see section [3.] Further, we will be adding exact saper-parameterss used for all the datasets the appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] All the results in the paper have the standard deviations reported, obtained by running the experiment times with random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We will be adding the code to replicate experiments in this paper
(d) Did you discuss whether and how consent
obtained from people whose data you's 're using/curating? [N/A] We use all the publicly available datasets
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "raCnVBoxDC", "reviewer": "Reviewer_JZZE", "summary": "The authors propose an approach for soft-labeling test data for test-time adaptation (TTA) purposes based on \"conjugate pseudolabels\". Motivated by the result that metalearning the TTA objective given test time labels roughly recovers the (temperature-scaled) training objective, the authors show that if the training objective can be expressed as a conjugate function $L(x,y) = f(h(x)) - y^Th(x)$ ($x$ inputs, $y$ labels), then the optimial test time loss can be approximated by $L(y)_{conj} = -f^*(\\nabla f(h(x))$, (f* the conjugate function of f), which is independent of the label, and implies the \"conjugate pseudo-labels\" (the usual softmax of h(x) for CE loss). TTA results on CIFAR-10-C,CIFAR-100-C, and ImageNet-C demonstrate that conjugate pseudo-labels consistently outperform other PL approaches.", "questions": "See S&W section.", "limitations": "See S&W section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: 7\nQuality: 7\nClarity: 7\nSignificance: 7\n\n- The paper is well written and the approach and results are interesting.\n- Using conjugate PLs for polyloss (novel) delivers on-par or better results than existing methods\n- The conjugate PL approach is adequately developed in the paper, but could benefit significantly from a section discussing current limitations, the more general case of $L(y)_{conj}$ not independent of $y$, the dual view, and future research directions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TNOzB9HXvG4", "reviewer": "Reviewer_U1Uv", "summary": "Paper proposes a general way of obtaining test-time adaptation loss which is used for more robust predictions under distribution shifts. First, this loss is learnt using meta-learning and the insights obtained from the same is used to obtain the proposed TTA loss (related to the conjugate of the training loss). The proposed loss is then reinterpreted as self-training with pseudolabels. ", "questions": "1. I do not agree with the paper’s claims about the “best” TTA loss as:\n- “Best” loss is not defined theoretically; Equation (6) is an approximation and it is unclear how close an approximation it is. \n- The scale of the “best” TTA loss from meta-learning seems to vary with task loss (Figure 5 & 6) which suggests that Equation (6) that has no scale is not the “best”. (My concern with temperature scaled TTA loss is below; point 2).\n- “Best” TTA loss likely depends on the type of distribution shifts. The meta-learning experiments are done on CIFAR10 with noises/corruptions. But the theory does not discuss what kind of distribution shifts are allowed. For example, is the only requirement that learnt $\\theta^\\star$ is (close to) optimal for target domain as well? Could we observe different meta-learnt loss if the distribution shifts were stronger, say shifts in illumination etc.?\n\n2. My other main concern is regarding the temperature scaling. \n\n- The paper does not provide a reasonable theoretical justification of using a temperature parameter since the approximation in Equation (6) doesn’t seem to demand it. It seems to be needed empirically though, as seen from Figure 1a and results in Section 5, but it is not clear why. \n\n- The need for temperature scaling creates an issue that held-out labeled validation data from target domain is required to tune it. Further, the paper also uses validation target domain data to tune the learning rate as well. This setting is easier than the standard TTA setting and many other domain adaptation methods could be more applicable. For example, I believe the relevant prior work TENT [1] does not assume the presence of this validation data and does no tuning. \n\n\n- While the paper claims that conjugate PL is much less sensitive to tuning T, I am not convinced it is. Table 1 shows that proposed approach + cross-entropy training loss is never best without temperature scaling and becomes best with temperature scaling. It would be helpful to see if similar trends appear on other domain adaptation datasets like SVHN->MNIST, etc. with the cross-entropy loss. In Table 2, while conjugate PL outperforms other loss functions, the sensitivity to temperature scaling is still comparable with other methods (compare ENT or MEMO). \n\n3. Minor:\n- More detail regarding Equation 1 will help the reader unfamiliar with meta learning loss functions. For example, the role of $\\mathcal{L}$, clarifying its difference from the source training loss.\n\n- Section 4.1 should include a reference for the optimization problem and its solution. \n\n- $h^\\star$ is not defined; I assume it is the optimal function. It is better to use another symbol to avoid confusion with conjugate symbol. At several places (like equations 8 and 9), $f^\\star$ is used instead of $f^*$.\n\n**AFTER REBUTTAL**: The authors have adequately addressed my concerns and updated the paper. I have increased my score. \n\n\nReferences:\n\n[1] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.\n\n", "limitations": "It would be helpful to clarify the following limitations better: What kind of distribution shifts are allowed? When is the proposed loss \"best\" with/without temperature scaling? ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. Paper is novel and useful as it provides a general framework to perform test-time adaptation for any training loss. Further, it provides an informal theory on why existing TTA loss (softmax-entropy) is a good TTA loss. \n\n2. Paper is clearly written and well-motivated through preliminary experiments. \n\n3. Experiments show advantage of the proposed training procedure, particularly in more recent non-standard loss functions.\n\n**Summary of weaknesses (details provided later):**\n\n1. The paper’s claims about the proposed loss being “best” TTA loss are not substantiated theoretically. \n\n2. The use of the temperature parameter is not motivated theoretically; it seems like an empirical fix. \n\n3. Tuning temperature parameter requires validation data from some target domains which is not the gold-standard TTA setting. Conjugate PL seems to be sensitive to temperature tuning (e.g., in Table 1 temperature scaling is required to outperform baselines). \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QyW0P-HWZSt", "reviewer": "Reviewer_M6iL", "summary": "Test-time training is an emerging and promising approach for building robust machine learning models to distribution shifts. As the model can not access ground-truth labels,  the most prior test-time adaptation literatures design the unsupervised loss function, e.g. entropy minimization used in TENT. However, as claimed by this paper, it is unclear what makes a good unsupervised loss in the setting of TTA. \n\nThis paper contributes to this important problem by following three means. (1) They show that the meta-learned unsupervised objective, which can be regarded as a best unsupervised loss, recover the well-known entropy minimization when the base classifier trained with cross entropy, but it recover different unsupervised loss function when using different supervised loss function. The results suggest that the best loss function differs among the choice of the supervised loss function. (2) They analyze the phenomenon from the lens of conjugated convex function, and show that it recovers the existing test-time adaptation strategies, which is similar to the objective found by meta-learning. (3) Based on the observation, they propose conjugate pseudo-labels as a method for test-time adaptation. It can be regarded as a general case of pseudo-labels, and can be used for loss function whose pseudo-labeling strategy is not trivial (e.g., recently proposed poly loss). They empirically show that the conjugate pseudo labels match the performance of entropy minimization when the source model is trained by cross-entropy loss, and outperforms existing methods when the source model is trained by Poly loss and squared loss. ", "questions": "---\n1. Regarding section 3\n- (a) What parameters of networks was updated via meta-objective function? Is it entire network, or the portion of it as with TENT?\n- (b) What is the exact optimization setup? What optimizer did you use? What is the learning rate, batch size, and training epoch?\n- (c) How did you repeat equation 1? Were both update computed via the same batch, or different batches?\n- (d) Did the difference of the above setup affect the learned objective function? In line 127, authors mentioned that the architecture change does not affect the learned objective function. I am appreciate if you can clarify what architecture did you use, and how does they consistently recovered the same objective function (e.g., does they recover the loss function with the similar scale and temperature?). \n- (e) What is the prediction score in Figure 1?\n\n---\n2. Regarding section 4\n- (a) How can we derive equation 2? May be it is a elemental question, but I think it makes the paper more self-contained. At least the pointer for the derivation is necessary (same for other important equations). \n- (b) What is N in algorithm 1 represents for? Algorithm 1 seems to be assume that model parameters are updated for each sample, is it actually correct, or did you use the batch of data for each update?\n\n--- \n3. Regarding section 4\n- (a) Again, what parameters of networks was updated during test-time? Is it same for all reported methods?\n- (b) Just for sure, did you treat the test-data as stream data available in online manner, or offline data? In other words, did you repeat test-time adaptation for randomly sampled batch, or did you adapt the model only once for each samples? \n\n---\n4. Question about the comments 2 and 3 in weaknesses\n- (a) Can conjugate convex function perspective can explain why meta-learning recovered scaled temperature entropy, not the simple entropy? \n- (b) Can you explain why some heuristics (i.e., diversity regularization in SHOT and feature alignment objective in [1,2]) works better?\n\n\n[1] Kojima, Takeshi et al. “Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignment.” IJCAI2022\n[2] Liu, Yuejiang et al. “TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive?” NeurIPS (2021). ", "limitations": "See questions. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As mentioned in the summary, this paper provides several interesting insights about how to design good unsupervised loss functions by the series of experiments and theoretical analysis. This paper is well motivated, each finding is well described and interesting. Not only does the paper explain why the existing method works well, but also provide more generic forms of the similar loss function for a broader supervised loss function, called conjugated pseudo-label. Empirical validation supports the merits of the proposed method, especially the source model is trained via unusual loss function, such as poly loss and squared loss. \nWhile I found the paper interesting, there are several unclear points from the current manuscripts. \n\n--- \n<Major comments>\n\n(1) The details of meta-learning experiments are not sufficiently presented. For example, what parameters of the networks were updated, and what optimizer, learning rate, batch size was used? It is not clear whether these factors affect the meta-learned objective or not. There is no clear explanation about what is the prediction score in Figure 1. \n\nRegarding the meta learning experiments, I am also concerned about whether the unsupervised loss function is learned via online adaptation setup, or offline adaptation setup. In standard meta-learning literature, in my understanding, they repeat the inner loop and outer loop many times, i.e. they assume offline adaptation setup. While the approach can reveal good objective function, it might not be effective in actual test-time (online) adaptation setup, as there is the discrepancy between meta-learning and actual test-phase. For example, in online-adaptation, adaptation speed and adaptation stability might be an issue given that there are less clues about data distribution. \n\n(2) While I like the general story of the paper and I am positive to accept the paper, I am still not convinced that the conjugated convex perspective could fully explain what makes a good unsupervised loss function in test-time adaptation setup. For example, why does the meta-learning objective recover the scaled and temperature loss function rather than the standard entropy suggested by the equation (9)? I am wondering if it is related to the assumption that the source models are sufficiently overparameterized, or the nature of test-time adaptation setup, which needs the online adaptation (and may be fast adaptation rather than other unsupervised adaptation setup). \n\n(3) Related to the above question, the results can not explain why several prior methods are often better than TENT in practice, even when the source model is trained via cross-entropy loss. For example, [1] shows that SHOT-IM (adding diversity regularization term) often has better results than TENT. They also show that the feature alignment approach often gives better results than the simple feature modulation approach. While I understand that it is out of scope of this paper, i.e., authors do not claim that the conjugated pseudo labeling is the best possible solution, I am curious whether conjugate optimization perspective could give further insights. \n\n\n(4) While it depends on readers, I found the section 4 should be more self-contained. For example, the logic behind some important equation (e.g., eq. 2, 3, 5) should be sufficiently described. \n\n--- \n<Minor comments>\n\n* In algorithm 1, no description about N. \n* The index n seems to be used with different meanings around equation 6 (n represents batch size?) and in algorithm 1 (n means the index of sample or batch). \n* In Table 1, Hard PL without temperature should be bold. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0SJ_hZwUh9J", "reviewer": "Reviewer_so7v", "summary": "The paper probes and evaluates the use of meta-learnt convex conjugate of the original training loss for test time adaptation to distribution shifts. \nIt has shown an intuition into the development of the method via a visual analysis of the numeric form of the learnt test time loss with an analytical approximation o known losses, discovering the similarity with the so-called source loss available during training. \nThe pseudo labels formulated  via the conjugate seem to provide a general method of computing a pseudo-label that works for losses besides the venerable CE. \n", "questions": "\nI did not quite follow why the method would always produce the optimal TTA loss. It is after all learnt, and operates with a non-zero risk. \n", "limitations": "No  comments on this. \n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The reformulation of the problem within the Legendre-Fenchel duality may indeed be the first such work in this small but growing area of work.  This area would arguably grow in significance as models in use encounter non-iid conditions.  The ideas are  clearly enunciated and are therefore easy to follow. The experiments define clear baselines and use hopefully comprehensive SOTA benchmarks. The choice of the source model, i.e. a shallow ResNet is not motivated, and several small details that aid reproducibility may be added, e.g. the number of runs for obtaining the standard deviation in Table 1. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept