Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ==========
Summary 

This paper proposes a meta-tailoring method where the auxiliary tailor loss is used to adapt the model parameters at test time.  The paper also provides a theoretical analysis of the advantages of the proposed tailoring/meta-tailoring.  The proposed method is evaluated on various application tasks and obtain significant improvements over baselines. 

==========
Pros

1. Test-time training is getting increasing attention recently, however, the theoretical analysis of the advantages of test time training is still behind. This paper provides a reasonable theoretical analysis of tailoring and meta-tailoring. 
2. The proposed CNGrad approach is simple but effective, which provides strong empirical improvements on various tasks. 
3. The paper is well written and clearly organized. 

===========
Concerns/confusions

1.  what is the formulation of the affine parameters?  Is there any study about adapting the entire model versus the affine parameters only?

2. Missing contrastive learning experiments on images?  In the last sentence of the abstract, it says "..., and using contrastive losses on the query image to improve generalization". However, I didn't find a discussion on it in the main paper.  Given that previous work (e.g., Sun et al.) has done experiments on image classification tasks, it would be more compelling to provide a comparison in the same setting. 


======

In general, this paper proposes an interesting approach and insightful analysis of the proposed method.  I'm willing to upgrade my score with comparable experiments with prior work on the image classification task. 








review_title: Very good paper with strong theoretical analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents tailoring and meta-tailoring to eliminate the generalization gap by optimizing at test time. This paper combines the idea of test time optimization and meta-learning and provides some theoretical analysis of the proposed methods. Experiments show the proposed method is effective in solving the machine learning problem and improving model robustness.

Strengths:
- The idea of using meta-learning to improve tailoring is interesting. Some theoretical justification for the proposed method is provided.
- The explanation of different learning settings is insightful.

Weaknesses:
- The proposed tailoring method is very close to Test Time Training (Sun et al. , 2019). However, this paper fails to clearly show the differences between tailoring and TTT. There is also no theoretical or empirical evidence to show that tailoring/meta-tailoring is better than TTT.
- The experiments in this paper are quite weak. The descriptions of the experiment settings are unclear. There is no direct comparison with closely related methods like TTT. The results of widely used benchmarks are not provided. 

Overall, although I think the idea is interesting, the proposed method is not well verified, which makes the effectiveness of the method is still unclear. Some closely related work is not sufficiently discussed. Therefore, I lean to recommend rejection for this paper.

review_title: Interesting idea, but the experiments are weak

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a learning method called tailoring, inspired
by transductive learning, which works
by fine tuning a model on an unsupervised loss given a test time example
input. The benefit of this approach is that arbitrary constraints
on predictions can be imposed without a suffering from a generalization gap
if the constraints were used as an auxiliary objective during training.
The authors also propose meta-tailor, which includes the tailoring process
as an inner optimization loop during training. In a sense, meta-tailoring
is like meta-learning but with  each training example considered a separate
task.

The authors provided extensive theoretical justification for tailoring
and meta tailoring, as well as an efficient means to implement it through
conditional normalization parameters.
They then perform two experiments, one where (meta-) tailoring
is used to impose physical constraints on a neural network physics simulator
and another where meta-tailoring is used to improve the robustness of an
image classifier to adversarial attacks.


The idea of (meta-)tailoring is quite intriguing and I could see it
applied to scenarios in structured prediction or as an alternative to
posterior regularization. However, this paper was very theory heavy and I must admit,
I struggled to understand the import or necessity of the provided theorems.
One of the claims of this paper is that they demonstrate that improving
prediction quality with contrastive learning, but this is only done
theoretically. As this claim is included in the list of experimental results,
I find that somewhat disingenuous. I would have much preferred a 
contrastive learning experiment.

I lean to reject this paper.



Miscellaneous note:

In the last paragraph of page 4, it is mentioned that parts of 
definition 1 and theorem 1 are in bold green, but I see no bold green in
this copy of the paper.

review_title: A very interesting idea, but needs more empirical validation of its claims.

final_score: rating: 7: Good paper, accept
review_content: The paper proposes tailoring and meta-tailoring, learning processes that fine-tune the model parameters during test-time using unsupervised objectives. This allows for designing and integrating powerful inductive biases into the model, leading to an improved test-time performance in two example tasks.

Strengths:
* The proposed approaches are well-motivated, and the paper is written clearly.
* CNGrad provides an elegant solution to implement tailoring efficiently.
* The experimental results show the benefits of the proposed approach convincingly.

Weaknesses:
* While CNGrad provides an efficient implementation of tailoring, it would be interesting to see how it compares to its “inefficient counterpart”, in which no additional parameters are introduced, but the parameters w are optimized for each individual sample using the supervised and the tailoring objective simultaneously.

* It would be nice to include the inductively trained model as a baseline to the experiment in section 5.2. This could highlight more clearly the benefit of tailoring when applied to adversarial examples.

* I find the statement that the paper showed “the applicability of tailoring on three domains” slightly misleading. The paper shows its applicability experimentally in two domains, and shows theoretical results for the third. Additionally, building on these theoretical results, it would be interesting to see how the contrastive loss might be used for tailoring in an experimental setting. 

Questions:
* How is the element-wise normalization in CNGrad performed at test-time? Do you keep a running average of the batch-statistics as is done in batch normalization?

* Which loss was used for the results in Table 1? 

* What are the run-time implications of tailoring, both at train and test time?

Additional Comments:
* Fig 1: Try to avoid using red and green as distinguishing colors to improve the paper’s accessibility.

* It might be nice to add a paragraph on inductive learning to the intro.


Disclaimer: I did not check the provided proofs in detail.
review_title: Tailoring - An interesting approach for more powerful inductive biases

Meta-review: This work proposes new learning algorithms that fine-tune ("tailor") a model at test-time using unsupervised objectives. This formulation allows for introducing an inductive bias into the model that might improve generalization on unseen data. The proposed algorithm is demonstrated on two example tasks.

The reviewers like the topic and also find the proposed approach to be interesting. However, they are unconvinced by the current empirical evaluation of the method. Additional experimental evaluation could improve our understanding of the proposed method and help contrast it to previously proposed techniques. Given these reviews I recommend rejecting the paper at this time.
Decision: Reject