Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed tailoring, which is a general framework that could help to finetune the prediction on each test sample according to some specific inductive biases. Tailoring provides a different perspective that avoids involving extra loss function in the proxy fashion.  Besides, the authors proposed meta-tailoring, which integrates the unsupervised loss in a way similar to meta-learning. The theoretical discussion and empirical results demonstrate the effectiveness of the proposed tailoring. 
main_review: Strengths:
1. The idea of tailoring is interesting. The current neural networks mainly conduct amortized optimization, how to reduce such gap especially on the test data is an important research direction. I believe the proposed framework could provide a different perspective on understanding the "generalization" gap and different inductive biases. 

2. The limitation of tailoring lies in the increased computational cost. To reduce such extra cost, the authors thus introduce CNGRAD which could efficiently parallel the evaluation of the model over multiple samples.  And there is detailed and sound theoretical justification on the CNGRAD provided. 

3. The authors provided extensive examples of inductive biases. And correspondingly the experiment results on symmetry constraints, inductive biases, contrastive loss, and adversarial examples justify the effectiveness of both tailoring and meta-tailoring.  The ablation study is well designed and the results are promising.

Weakness:
1. One particular concern of mine is when conducting tailoring the parameters of the model change according to a single sample, this is similar to the continual learning setting. I wonder how the methods could avoid catastrophic forgetting. It seems that the authors constrain the steps of the tailoring while small changes in parameter space could result in the relatively large change of the model output. I suggest the authors add more discussions on this part. 

2. Though the method does no limit the application scenarios, I feel that the method is highly related to the field of semi-supervised learning. Therefore, I suggest more discussions on the related works. For example, the adversarial examples setting of tailoring is related the virtual adversarial learning in semi-supervised learning [1].

[1]. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Questions:
I am curious about the setting when evaluating the test sample, we first assign a pseudo label according to the initial output. And minimizing the loss towards this pseudo label, I wonder whether tailoring in this setting could work. 
limitations_and_societal_impact: Yes.
Refers to main review
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes tailoring - a general framework of algorithms that can combine ideas for test-time generalization, self supervision, meta learning and transductive learning. Although the experiments are quite limited overall, the paper is a good addition for the ML community. 
main_review: The paper introduces tailoring and meta tailoring as a means to add inductive biases at test-time using contrastive losses. The paper overcomes some of the shortcomings of TTT and meta-tailoring seems like an interesting improvement over TTT. The idea of encouraging soft inductive biases (5.2) is very interesting and practical. I would like the authors to include experiments such as domain generalization (like the sort done by TTT) since I believe that those set of experiments are very good tests of how such algorithms can adapt to novel data distributions (not just adversarial samples as they are a very specific type of generalization. Although the authors mention the practical problems with implementation in popular deep learning frameworks (pytorch and tensorflow), it would be good for the authors to provide means to overcome such problems so that CNGrad can become a staple in the deployment of ML models.

The authors tend to focus on the breadth of results to show the generality of the solution, rather than depth in one or two fields, it might be useful to show more difficult tasks in any of the tasks to show a strict improvement and the scalability of the solution.
limitations_and_societal_impact: The most practical problems with the frameworks (as the authors discuss) is with privacy as the models are "tailored" for the given example. This can have negative impact and which the authors correctly discuss.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new method to adapt neural networks at inference time to the given input such that the model minimizes some given unsupervised loss. This method is called 'tailoring'. Additionally, 'meta-tailoring' is proposed that trains the model using 'tailoring', too. This way the gap between training and inference distribution is removed that 'tailoring' introduces.
main_review: The paper describes a, to the best of my knowledge, novel method. I believe this method is interesting for niche applications of ML. The model is replaced by a short SGD loop of the model, such that given x a few iterations of SGD are performed, before making the prediction. I do not understand why there is no possibility to make the model behave straight away like the model after a few steps.

Main Points:

1. Method Description. The method is not described enough in the main paper. One has to read the appendix to really understand what the method is, especially for meta-tailoring. There should be pseudo-code or at least a more detailed text description in the main paper. For example, I do not understand what the difference of Meta-Tailoring (0 st.) is to the baseline, even after looking into the appendix. I am pretty confident that algorithm 1 would actually not train at all with steps=0. I also find the naming confusing with the introduction of CNGrad, but naming the application of CNGrad in the experiments Meta-tailoring again.

2. First-order and detached CNGrad. You only consider detached CNGrad in all experiments (line. This is not the algorithm you provide guarantees for in Section 3. I think this is a severe short-coming, as the detached variant, also does not agree with the intuition one develops around meta-tailoring. For 2/4 experiments you write that you use first-order CNGrad, for the others it is not known. First-order CNGrad, goes even further away from meta-tailoring and detaches \gamma and \beta even earlier. This, thus makes the above problem even more severe.

3. Focus. I think this paper could benefit a lot of a more focused structure. You try to keep everything as general as possible, but there you also have to make changes for each experiment. I believe the paper might benefit from more focus. Only considering one or two CNGrad variants with fixed hyper-parameters across tasks, a main evaluation on which you reach something close to state-of-the-art and more focus on the method presentation.

4. Experiments. This paper proposes a broad method, thus I understand the motivation of the authors to include a diverse set of benchmarks. The problem with this setup is that, the evaluations themselves suffer from this. None, of the evaluations is described to a sufficient level, such that it is really hard to understand the effect.

5. Baselines. The baselines are not easily understood. While, I very much appreciate that you add the tailoring loss to the inductive baseline for the first experiment, I am still a little critical regarding the baselines. You now put a lot of work in making CNGrad work including selecting where to apply CN layers, what inner-lr to use, whether to use first-order CNGrad or detached CNGrad, but for the baselines it seems you performed less of this tuning. For some of the baselines my feeling is that it is reasonably easy to come up with a way to make the inductive baseline perform well and consider the constraints at testing time. Like you wrote for Adverserial Examples.

6. Costs. As this method introduces a considerable amount of extra gradient steps to the training loop, I believe there should be some considerations of the costs of this method in the main paper.

Summary: The paper proposes an interesting new method. It does not show conclusive evidence that it improves state-of-the-art models in any domain, though, and it uses a different algorithm compared to the described algorithm. Additionally, parts of the paper seem to not be quite ready for a main conference and rather in a draft-stage. Nevertheless, I believe this paper might be a great contribution after some more work on the experiments and the presentation.

DISCLAIMER: I did not take the time to read section 3 and the corresponding proofs in detail, as I do not believe that this changes my view of this paper too much, it is hard to follow and does, to my understanding, not apply to the algorithm actually used practically in the experiments as pointed out in 2.

Details:

i) Line 720 'key' -> 'this is key'

ii) Line 105 '. Losses' -> '.\n\nLosses'

iii) Line 63 -> Notice that the outer process now only optimizes the objective we care about, ...

iv) Confusing notation in algorithms, with var assignments and 'for' construct in the same line.

v) Table 1 description 'over-performs' -> 'outperforms'

vi) You used /begin{figure} for many tables where /begin{table} should be used.

vii) Line 356 ' Improving' -> 'Improving'
limitations_and_societal_impact: The authors consider interesting societal impacts and the limitations of their algorithm.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose optimizing an unsupervised loss function at test time as an inductive prior on a neural network. They propose two schemes for achieving this: training a network in a regular fashion and then applying the unsupervised loss at inference time only, or applying the unsupervised loss during training as a meta-learning scheme. The authors claim that this method can improve robustness, as well as improve train-test generalization gap in cases where there is a known inductive prior on the expected output (e.g. when modelling physical systems)

main_review: The paper's writing could be improved, as the writing is complex and hard to follow. Furthermore the model details were mixed with the introduction, making it hard to understand what is the author's proposed work. It is also hard to understand the implementation details of this idea, as the authors share neither pseudo-code nor actual code to illustrate their implementation. With such an idea, it is often the case that the devil is in the details, and efforts to replicate it could show very different results to the ones shown on the paper.

The idea is very interesting and novel. It is well motivated in learning theory and seems almost obvious in hindsight (a good thing!). The authors provide a wide variety of experiments to show the effectiveness of the idea, not just for different datasets, but for different types of inductive bias. Especially in the case of physical systems modelling, this idea could be very useful in the applied setting and have a broad impact on further research.
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work and Societal impact.
review_title: null

Meta-review: This paper proposes tailoring, a technique for incorporating unsupervised objectives during test time. The main distinction and past work is that tailoring is applied to each individual test datapoint. The paper includes some theoretical justification of the approach as well as rather extensive experiments. Reviewers were generally positive on the paper. The main criticism of the paper came down to lack of clarity in writing, but the authors have addressed this concern through their rebuttal. Before the camera-ready version, please incorporate the proposed changes to writing.
Decision: Accept (Poster)