Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an approach for soft-labeling test data for test-time adaptation (TTA) purposes based on "conjugate pseudolabels". Motivated by the result that metalearning the TTA objective given test time labels roughly recovers the (temperature-scaled) training objective, the authors show that if the training objective can be expressed as a conjugate function $L(x,y) = f(h(x)) - y^Th(x)$ ($x$ inputs, $y$ labels), then the optimial test time loss can be approximated by $L(y)_{conj} = -f^*(\nabla f(h(x))$, (f* the conjugate function of f), which is independent of the label, and implies the "conjugate pseudo-labels" (the usual softmax of h(x) for CE loss). TTA results on CIFAR-10-C,CIFAR-100-C, and ImageNet-C demonstrate that conjugate pseudo-labels consistently outperform other PL approaches.
strengths_and_weaknesses: Originality: 7
Quality: 7
Clarity: 7
Significance: 7

- The paper is well written and the approach and results are interesting.
- Using conjugate PLs for polyloss (novel) delivers on-par or better results than existing methods
- The conjugate PL approach is adequately developed in the paper, but could benefit significantly from a section discussing current limitations, the more general case of $L(y)_{conj}$ not independent of $y$, the dual view, and future research directions. 
questions: See S&W section.
limitations: See S&W section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Test-time training is an emerging and promising approach for building robust machine learning models to distribution shifts. As the model can not access ground-truth labels,  the most prior test-time adaptation literatures design the unsupervised loss function, e.g. entropy minimization used in TENT. However, as claimed by this paper, it is unclear what makes a good unsupervised loss in the setting of TTA. 

This paper contributes to this important problem by following three means. (1) They show that the meta-learned unsupervised objective, which can be regarded as a best unsupervised loss, recover the well-known entropy minimization when the base classifier trained with cross entropy, but it recover different unsupervised loss function when using different supervised loss function. The results suggest that the best loss function differs among the choice of the supervised loss function. (2) They analyze the phenomenon from the lens of conjugated convex function, and show that it recovers the existing test-time adaptation strategies, which is similar to the objective found by meta-learning. (3) Based on the observation, they propose conjugate pseudo-labels as a method for test-time adaptation. It can be regarded as a general case of pseudo-labels, and can be used for loss function whose pseudo-labeling strategy is not trivial (e.g., recently proposed poly loss). They empirically show that the conjugate pseudo labels match the performance of entropy minimization when the source model is trained by cross-entropy loss, and outperforms existing methods when the source model is trained by Poly loss and squared loss. 
strengths_and_weaknesses: As mentioned in the summary, this paper provides several interesting insights about how to design good unsupervised loss functions by the series of experiments and theoretical analysis. This paper is well motivated, each finding is well described and interesting. Not only does the paper explain why the existing method works well, but also provide more generic forms of the similar loss function for a broader supervised loss function, called conjugated pseudo-label. Empirical validation supports the merits of the proposed method, especially the source model is trained via unusual loss function, such as poly loss and squared loss. 
While I found the paper interesting, there are several unclear points from the current manuscripts. 

--- 
<Major comments>

(1) The details of meta-learning experiments are not sufficiently presented. For example, what parameters of the networks were updated, and what optimizer, learning rate, batch size was used? It is not clear whether these factors affect the meta-learned objective or not. There is no clear explanation about what is the prediction score in Figure 1. 

Regarding the meta learning experiments, I am also concerned about whether the unsupervised loss function is learned via online adaptation setup, or offline adaptation setup. In standard meta-learning literature, in my understanding, they repeat the inner loop and outer loop many times, i.e. they assume offline adaptation setup. While the approach can reveal good objective function, it might not be effective in actual test-time (online) adaptation setup, as there is the discrepancy between meta-learning and actual test-phase. For example, in online-adaptation, adaptation speed and adaptation stability might be an issue given that there are less clues about data distribution. 

(2) While I like the general story of the paper and I am positive to accept the paper, I am still not convinced that the conjugated convex perspective could fully explain what makes a good unsupervised loss function in test-time adaptation setup. For example, why does the meta-learning objective recover the scaled and temperature loss function rather than the standard entropy suggested by the equation (9)? I am wondering if it is related to the assumption that the source models are sufficiently overparameterized, or the nature of test-time adaptation setup, which needs the online adaptation (and may be fast adaptation rather than other unsupervised adaptation setup). 

(3) Related to the above question, the results can not explain why several prior methods are often better than TENT in practice, even when the source model is trained via cross-entropy loss. For example, [1] shows that SHOT-IM (adding diversity regularization term) often has better results than TENT. They also show that the feature alignment approach often gives better results than the simple feature modulation approach. While I understand that it is out of scope of this paper, i.e., authors do not claim that the conjugated pseudo labeling is the best possible solution, I am curious whether conjugate optimization perspective could give further insights. 


(4) While it depends on readers, I found the section 4 should be more self-contained. For example, the logic behind some important equation (e.g., eq. 2, 3, 5) should be sufficiently described. 

--- 
<Minor comments>

* In algorithm 1, no description about N. 
* The index n seems to be used with different meanings around equation 6 (n represents batch size?) and in algorithm 1 (n means the index of sample or batch). 
* In Table 1, Hard PL without temperature should be bold. 


questions: ---
1. Regarding section 3
- (a) What parameters of networks was updated via meta-objective function? Is it entire network, or the portion of it as with TENT?
- (b) What is the exact optimization setup? What optimizer did you use? What is the learning rate, batch size, and training epoch?
- (c) How did you repeat equation 1? Were both update computed via the same batch, or different batches?
- (d) Did the difference of the above setup affect the learned objective function? In line 127, authors mentioned that the architecture change does not affect the learned objective function. I am appreciate if you can clarify what architecture did you use, and how does they consistently recovered the same objective function (e.g., does they recover the loss function with the similar scale and temperature?). 
- (e) What is the prediction score in Figure 1?

---
2. Regarding section 4
- (a) How can we derive equation 2? May be it is a elemental question, but I think it makes the paper more self-contained. At least the pointer for the derivation is necessary (same for other important equations). 
- (b) What is N in algorithm 1 represents for? Algorithm 1 seems to be assume that model parameters are updated for each sample, is it actually correct, or did you use the batch of data for each update?

--- 
3. Regarding section 4
- (a) Again, what parameters of networks was updated during test-time? Is it same for all reported methods?
- (b) Just for sure, did you treat the test-data as stream data available in online manner, or offline data? In other words, did you repeat test-time adaptation for randomly sampled batch, or did you adapt the model only once for each samples? 

---
4. Question about the comments 2 and 3 in weaknesses
- (a) Can conjugate convex function perspective can explain why meta-learning recovered scaled temperature entropy, not the simple entropy? 
- (b) Can you explain why some heuristics (i.e., diversity regularization in SHOT and feature alignment objective in [1,2]) works better?


[1] Kojima, Takeshi et al. “Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignment.” IJCAI2022
[2] Liu, Yuejiang et al. “TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive?” NeurIPS (2021). 
limitations: See questions.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper probes and evaluates the use of meta-learnt convex conjugate of the original training loss for test time adaptation to distribution shifts. 
It has shown an intuition into the development of the method via a visual analysis of the numeric form of the learnt test time loss with an analytical approximation o known losses, discovering the similarity with the so-called source loss available during training. 
The pseudo labels formulated  via the conjugate seem to provide a general method of computing a pseudo-label that works for losses besides the venerable CE. 

strengths_and_weaknesses: The reformulation of the problem within the Legendre-Fenchel duality may indeed be the first such work in this small but growing area of work.  This area would arguably grow in significance as models in use encounter non-iid conditions.  The ideas are  clearly enunciated and are therefore easy to follow. The experiments define clear baselines and use hopefully comprehensive SOTA benchmarks. The choice of the source model, i.e. a shallow ResNet is not motivated, and several small details that aid reproducibility may be added, e.g. the number of runs for obtaining the standard deviation in Table 1. 


questions: 
I did not quite follow why the method would always produce the optimal TTA loss. It is after all learnt, and operates with a non-zero risk. 

limitations: No  comments on this.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper proposes a general way of obtaining test-time adaptation loss which is used for more robust predictions under distribution shifts. First, this loss is learnt using meta-learning and the insights obtained from the same is used to obtain the proposed TTA loss (related to the conjugate of the training loss). The proposed loss is then reinterpreted as self-training with pseudolabels. 
strengths_and_weaknesses: **Strengths:**

1. Paper is novel and useful as it provides a general framework to perform test-time adaptation for any training loss. Further, it provides an informal theory on why existing TTA loss (softmax-entropy) is a good TTA loss. 

2. Paper is clearly written and well-motivated through preliminary experiments. 

3. Experiments show advantage of the proposed training procedure, particularly in more recent non-standard loss functions.

**Summary of weaknesses (details provided later):**

1. The paper’s claims about the proposed loss being “best” TTA loss are not substantiated theoretically. 

2. The use of the temperature parameter is not motivated theoretically; it seems like an empirical fix. 

3. Tuning temperature parameter requires validation data from some target domains which is not the gold-standard TTA setting. Conjugate PL seems to be sensitive to temperature tuning (e.g., in Table 1 temperature scaling is required to outperform baselines). 


questions: 1. I do not agree with the paper’s claims about the “best” TTA loss as:
- “Best” loss is not defined theoretically; Equation (6) is an approximation and it is unclear how close an approximation it is. 
- The scale of the “best” TTA loss from meta-learning seems to vary with task loss (Figure 5 & 6) which suggests that Equation (6) that has no scale is not the “best”. (My concern with temperature scaled TTA loss is below; point 2).
- “Best” TTA loss likely depends on the type of distribution shifts. The meta-learning experiments are done on CIFAR10 with noises/corruptions. But the theory does not discuss what kind of distribution shifts are allowed. For example, is the only requirement that learnt $\theta^\star$ is (close to) optimal for target domain as well? Could we observe different meta-learnt loss if the distribution shifts were stronger, say shifts in illumination etc.?

2. My other main concern is regarding the temperature scaling. 

- The paper does not provide a reasonable theoretical justification of using a temperature parameter since the approximation in Equation (6) doesn’t seem to demand it. It seems to be needed empirically though, as seen from Figure 1a and results in Section 5, but it is not clear why. 

- The need for temperature scaling creates an issue that held-out labeled validation data from target domain is required to tune it. Further, the paper also uses validation target domain data to tune the learning rate as well. This setting is easier than the standard TTA setting and many other domain adaptation methods could be more applicable. For example, I believe the relevant prior work TENT [1] does not assume the presence of this validation data and does no tuning. 


- While the paper claims that conjugate PL is much less sensitive to tuning T, I am not convinced it is. Table 1 shows that proposed approach + cross-entropy training loss is never best without temperature scaling and becomes best with temperature scaling. It would be helpful to see if similar trends appear on other domain adaptation datasets like SVHN->MNIST, etc. with the cross-entropy loss. In Table 2, while conjugate PL outperforms other loss functions, the sensitivity to temperature scaling is still comparable with other methods (compare ENT or MEMO). 

3. Minor:
- More detail regarding Equation 1 will help the reader unfamiliar with meta learning loss functions. For example, the role of $\mathcal{L}$, clarifying its difference from the source training loss.

- Section 4.1 should include a reference for the optimization problem and its solution. 

- $h^\star$ is not defined; I assume it is the optimal function. It is better to use another symbol to avoid confusion with conjugate symbol. At several places (like equations 8 and 9), $f^\star$ is used instead of $f^*$.

**AFTER REBUTTAL**: The authors have adequately addressed my concerns and updated the paper. I have increased my score. 


References:

[1] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.


limitations: It would be helpful to clarify the following limitations better: What kind of distribution shifts are allowed? When is the proposed loss "best" with/without temperature scaling?
review_title: null

Meta-review: metareview: All reviewers agree this paper presents a novel and principled approach to test time adaptation losses. All reviewers find the paper clearly written and contributions meaningful. I suggest acceptance.
confidence: Certain
Decision: Accept