Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
Authors: Ferran Alet
Kenji Kawaguchi
Maria Bauza Villalonga
Nurullah Giray Kuru
Tomas Perez
Leslie Pack Kaelbling
Venue:  (2021)
Keywords: inductive biases
meta-learning
semi-supervised learning
adversarial learning
representation learning
transductive learning

Abstract: From CNNs to attention mechanisms, encoding inductive biases into neural networks has been a fruitful source of improvement in machine learning. Auxiliary losses are a general way of encoding biases in order to help networks learn better representations by adding extra terms to the loss function. However, since they are minimized on the training data, they suffer from the same generalization gap as regular task losses. Moreover, by changing the loss function, the network is optimizing a different objective than the one we care about. In this work we solve both problems: first, we take inspiration from transductive learning and note that, after receiving an input but before making a prediction, we can fine-tune our models on any unsupervised objective. We call this process tailoring, because we customize the model to each input. Second, we formulate a nested optimization (similar to those in meta-learning) and train our models to perform well on the task loss after adapting to the tailoring loss. The advantages of tailoring and meta-tailoring are discussed theoretically and demonstrated empirically on several diverse examples: encoding inductive conservation laws from physics to improve predictions, improving local smoothness to increase robustness to adversarial examples, and using contrastive losses on the query image to improve generalization.

Under review as a conference paper at ICLR 2021
Tailoring:
ENCODING INDUCTIVE BIASES BY OPTIMIZING UNSUPERVISED OBJECTIVES AT PREDICTION TIME
Anonymous authors Paper under double-blind review
ABSTRACT
From CNNs to attention mechanisms, encoding inductive biases into neural net- works has been fruitful source of improvement in machine learning. Auxiliary losses are a general way of encoding biases in order to help networks learn better representations by adding extra terms to the loss function. However, since they are minimized the training data, they suffer from the same generalization gap regular task losses. Moreover, by changing the loss function, the network is opti- mizing different objective than the one care about. In this work we solve both problems: first, we take inspiration from transductive learning and note that, after receiving an input but before making prediction, any unsupervised objective. We call this process tailoring, because we customize the model each input. Second, we formulate a nested optimization (similar those in meta-learning) and train our models to perform well on the task loss after adapting the tailoring loss. The advantages of tailoring and meta-tailoring are discussed theoretically and demonstrated empirically on several diverse examples: encoding inductive conservation laws from physics, increasing robustness to ad- versarial examples, meta-tailoring with contrastive losses to improve theoretical generalization guarantees, and increasing performance in model-based RL.
fine-tune models
INTRODUCTION
The key to successful generalization in machine learning is the encoding of useful inductive biases. A variety of mechanisms, from parameter tying to data augmentation, have proven useful but there is no systematic strategy for designing and implementing these biases.
Auxiliary losses are paradigm for encoding a wide variety of biases, constraints and objectives, helping networks learn better representations and generalize more broadly. They add an extra to the task loss and minimize it over the training data or, in semi-supervised learning, on an extra set of unlabeled data. However, they have two major difficulties:
1. Auxiliary losses are only minimized training time, but not for the query points. This causes a generalization gap between training and testing, in addition to that of the task loss.
2. By minimizing the sum of the task loss plus the auxiliary loss, we are optimizing a different objective than the one we care about (only the task loss).
In this work we propose a solution to each problem:
1. We ideas from transductive learning minimize the auxiliary loss the query by running an optimization at prediction time, eliminating the generalization gap for the auxiliary loss. We call this process tailoring, because we customize the model each query.
2. We ideas from meta-learning learn model that performs well on the task loss assuming that we will be optimizing the auxiliary loss. This meta-tailoring effectively trains the model to leverage the unsupervised tailoring loss to minimize the task loss.
classical inductive supervised learning, an algorithm consumes a training dataset of onput-output pairs, ((I;, )))).,, and produces set of parameters á by minimizing a super- vised loss Σi= _ (ff(2z)) y:) and, optionally, unsupervised auxiliary loss Σm= Cu x.). These parameters specify a hypothesis (-) that, given new input x, generates an output ğ = fz(x). This problem setting misses substantial opportunity: before the learning algorithm sees the query
Tailoring a predictor

---

Under review as a conference paper at ICLR 2021
Transductive Learning
(2(1) Σ L (So((())) Σ Cunsup (So (cali))
((0), V:))
org min
(3G)))
Inductive Learning
a
((ri, Vel)
arg min
Coupl Coup (fo(m)).
fa(3))
θ
Tailoring
at
x
((0;, Mi))
arg min
uup Coup (fo(z:). uk)
org min 0m20
Ciallor (z, (x)
fer
θ x
Meta-tailoring
((xi, y:)) arg
(f +10, ctatlor an (2a). Mid
org min Chailor (z, 0) 0z28
fe,
0
Oz
Figure 1: Comparison of several learning settings with offline computation in the orange boxes and online computation in the green boxes, with tailoring in blue. For meta-tailoring training, 7(0, Cuaior, x²) arg ming' Ciailo (z) represents the tailoring process resulting in Oz. Although tailoring and meta-tailoring are best understood in supervised learning, they can also be applied in reinforcement learning, as shown in section 5.3
point x, it has distilled the data down to set of parameters, which are frozen during inference, and so it cannot use new information about the particular x that it will be asked to make a prediction for.
Vapnik recognized opportunity make more accurate predictions when the query point is known, in a framework that is now known as transductive learning (Vapnik) 1995 Chapelle et al. 2000). In transductive learning, a single algorithm consumes both labeled data, ((z), (())::)))=1) and set of input desired, (2:(5) and produces predicted outputs (i for each of
points for which predictions the queries, as illustrated in the top row of figure [ In general, however, we do not know queries priori, and instead we want an inductive rule that makes predictions on-line, as queries arrive. To obtain prediction function from a transductive system, we would need to encapsulate the entire learning procedure inside the prediction function. This strategy would achieve our objective of taking into account at prediction time, but would be computationally much too slow.
We observe that this strategy for combining induction and transduction would perform very similar computations for each prediction, sharing the same training data and objective. We can use ideas from meta-learning to find shared "meta-hypothesis" that can then be efficiently adapted to each query (trcating each prediction regular supervised learning obtain parameters θ; then, given query input x, fine-tune â on an unsupervised loss (Cualler to obtain customized parameters O2 and use them to make the final prediction: Sa_ (xr). We call this process tailoring, because we adapt the model to each particular input for a customized fit. Notice that tailoring optimizes the loss at the query point, eliminating the generalization gap on the auxiliary loss.
task). As shown in the third row of figure D we first run
Since we will be applying tailoring at prediction time, it is natural to anticipate this adaptation during training, resulting in two-layer optimization similar those used for meta- learning. Because this similarity, we call this process, illustrated in the bottom row of figure []] meta-tailoring. Now, rather than letting be the direct minimizer of the supervised loss, we set to
Meta-tailoring
E arg min
L (f (2:)) v.).
Notice that by optimizing this nested objective, the outer process is now optimizing the only objective care about, L instead of proxy combination of _ and Lussup, At the same time, learning to leverage the unsupervised tailoring losses in the inner optimization to affect the model before making the final prediction, both during training and at prediction time.
2

---

Under review as a conference paper at ICLR 2021
In many settings, wish to make predictions for large number of query points in (mini-)batel) but because tailoring adapts to every point, we must take extra care to be sure can run efficiently in parallel. Inspired by conditional normalization (Dumoulin al., 2016) we propose adding element- wise affine transformations and only adapting these parameters in the inner optimization. This allows us totailor outputs for multiple inputs in parallel, without inputs affecting each other's computations. We prove theoretically, in section 4 and provide experimental evidence, in section [5.]] that optimizing these parameters alone has enough capacity minimize a large class of tailoring losses.
Losses for tailoring Tailoring can be used any time we are making a prediction and have an unsupervised loss that can minimized on the current input; although we mostly focus on supervised learning, tailoring can also be applied to reinforcement learning, as shown in section [5.3] There are many types of tailoring losses that may be useful; here we enumcrate three broad classes.
Losses imposing priors and constraints satisfied by the correct predictions, such as conservation of momentum and energy when learning physics model, symmetry under specific transformations, or sycle-consistnncy when learning to translate between two languages. We show that tailoring loss of this type improves predictive losses in section [.] where we model a planetary system. We can also leverage soft priors; for instance, in section 5.3 we show how encouraging cantiincondiiional predictions to be likely under an action-uncondittonal model can improve model RI..
Losses that help learn better representations. such the contrastive losses Hadsell al.] (2006) explored in semi-supervised learning, or learning to predict one part of the input from another part,such as depth from RGB (Mirowski et al., 2016). In section 3.1 we show how tailoring with contrastive loss improves supervised prediction performance.
Losses informed by theoretical guarantees. The guarantees of many theorems depend on unsuper- vised quantities, such as smoothness or distance to the prediction boundary. By optimizing such quanti- ties on the query, or on the surrounding area, we can get better guaranteed performance. In section [..2] we show this on adversarial examples, where smooth predictions around the test point are critical.
Contributions
In summary, our contributions are the following:
1. Introducing tailoring, new framework for encoding inductive biases by minimizing unsuper- vised losses at prediction time, with theoretical guarantees and broad potential applications.
2. Formulating meta-tailoring, which adjusts the outer objective to optimize only the task loss, and developing a new algorithm, CNGRAD, for efficient meta-tailoring.
3. Demonstrating tailoring in four domains: encoding conservation laws in a physics prediction problem, increasing resistance to adversarial examples by increasing local smoothness at prediction time, making model-based reinforcement learning more data-efficient, and improving theoretical guarantees of prediction quality by tailoring with contrastive loss.
2 RELATED WORK
There are other learning frameworks that perform optimization at prediction time, such as energy-based models (Ackley et al., 1985 Hinton| 2002 LeCun et al. 2006) or models that embed optimization layers in neural networks, whose outputs are the solution of an optimization problem defined by the previous layer (Amos & Kolter 2017 Tschiatschek et al. 2018). In contrast to these lines of work, we optimize the parameters of the model, not the hidden activations the output.
Meta-learning (Schmidhuber 1987 Bengio et al. 1995] Thrun & Pratt 1998) has the same two-level optimization structure our work, but focuses on multiple prediction tasks, each of which has its own separate training data. Most optimization-bant meta-tailoring setting. There a particularly clear connection MAML (Finn et al. 2017). when we let the tailoring method are other optimization-ban based approaches to meta-learning whose adaptations can be batched (Zintgraf cet al., 2018 Rakelly et al., 2019 Alet et al. 2019). In particular, FiLM networks (Perez et al. 20188). which predict customized conditional normalizatio layers, have been used in meta-learning (Zintgral et al. 2018 Requeima et al. 2019). By optimizing the conditional normalization layers themselves, method CNGRAD is simpler, while remaining provably sufficiently expressive. More importantly, we can add to a trained model CNGRAD layers with weights initialized to the identity and adapt them to perform tailoring or fine-tune the model with meta-tailoring objective.
meta-learning al algorithms can be converted to the
step of gradient descent: r(θ, guailor .r) = XV Laalor 0). There
3

---

Under review as a conference paper at ICLR 2021
Tailoring is inspired by transductive learning. However, transductive methods, because they operate on a batch of unlabeled points, are able to make use of the underlying distributional properties of those points. On the other hand, tailoring does not need to receive the queries before doing the bulk of the computation. Within transductive learning, local learning (Bottou & Vapnik] 1992) has input-dependent parameters, but it uses similarity in input space select a few data- point-p instead of reusing the prior learned across the whole data. Some methods (Garcia & Bruna (0177) 7 Liu et al., 2018) in meta-learning propagate predictions along the test samples in a classic transductive fashion.
Optimization processes similar to tailoring and meta-tailoring have been proposed before, to adapt to different types of variations between training and testing. [Sun et al. (2019) propose to adapt to change of distribution with few samples by unsupervised fine-tuning at test-time. applying it
with a loss of predicting whether the input has been rotated. Other methods in the meta-
setting exploit test samples of a new task by minimizing either entropy (Dhillon et al. [2020) or a learned loss (Antoniou & Storkey (019) in the inner optimization. Finally, |Wang et al. (2019) uses entropy in the inner optimization to adapt to large scale variations in image segmentation. In contrast, we propose ameta-)tailonin as general effective way to impose inductive biases in the classic machine learning setting. We also unify tailoring and meta- teit-raii with arbitrary losses under one paradigm, theoretically and empirically showing the ad advantage of each modification to classic inductive learning. Appendix showw experimental results analyzing why using prior work on adapting to the test distribution performs worse than tailoring (which, in turn, performs worse than meta-tailoring) in the classic ML setting where test and training come from the same distribution.
THEORETICAL MOTIVATIONS OF META-TAILORING
In this section, we study potential advantages of meta-tailoring from the theoretical viewpoint, formalizing the intuitions conveyed in the introduction. By acting symmetrically during training and prediction time, meta-tailoring allows us to closely relate its training and expected losses, whereas tailoring in general may make them less related.
3.1 META-TAIL.ORIG WITH A CONTRASTIVE TAILORING LOSS
Contrastive learning (Hadsell al. 2 has seen significant successes recently in problems of semi- supervised learning from images (Oord et al. 2018 He et al. 2019 Chen et al. (020). The main idea to create multiple versions of each training image, and learn a representation in which variations of the same image are very close and variations of diffcrent images are far apart. Typical variations
involve cropping, color distortions and rotation. We show theoretically that, under reasonable conditions, meta- meta-tailoring using particular contrastive loss Levmt as Ctailor Leom helps us improve generalization errors in expectation compared with performing classical inductive learning.
When using meta-tailoring, we define 0g.s to be the f₂ obtained with a training dataset S = B))) and tailoring with the contrastive loss at the prediction point x.. Theorem pro- in terms of the expected contrastive loss Eg[Ccom 0g.s)] (which is defined and analyzed in more detail in Appendix B). the empirical supervised loss 1/2 , (Ser( (x:): y1) of the meta-tailoring al- gorithm, and the uniform stability Ç of the meta-tailoring algorithm. As a complement, in-
vides an upper bound on the expected supervised loss [. (J)...(z;), (x),
stead of using the uniform stability Ç. Theorem [6] (detailed in appendix C provides a similar bound with the Rademacher complexity (Bartlett & Mendelson 2002) R,, [Ssup o F) of the set CSPP F to Appendix C
{x. v) + L (f), v) (x fa, (x)) E FJ. All proofs in this paper are deferred
Definition Let S ((1; w)) and S' ((1/, D/ be any two training datasets that differ by a single point. Then, a mata-iii algorithm S + fors (x) is uniformly Ç-stable if ((r,y) E X' xx y, |C=T (z), (fo (z). S.
Theorem 1. Let S + fe.. (x) be uniformly Ç Ç-stable meta-tailorine algorithm. Then, for any 8 O. with probability at least 1-6 over an i...d. draw of n i.i.d. samples S = ((x,,
the following holds: for any E 10, 1). Ez,g " (fons (z), v)l $ KE [ (x. 0(,s)] + (1-
where J = - C_ (a:), v1) + + (2( + c2 (lm (l1(1))))/(22), and is the upper bound on the par-sample loss as (fo(())) cc
In the case of regular inductive learning we get a bound of the exact same form, except that we have single 0 instead of 0g tailored each input x. These differences are marked in bold
4

---

Under review as a conference paper at ICLR 2021
green in Definition I and Theorem This theorem illustrates the effect of meta- meta-tailoring on contrastive learning, with its potential reduction of the expected contrastive loss E ExlLCoom(2 (z. fg,s)l In classic induction, we may aim to minimize the empirical contrastive loss #. Eim Lcom (z,, 0) with a potentially unlabeled training samples (in addition to the empirical supervised loss), which incurs the additional generalization error of E, (Cooot (2,, 0z,s)] - -i=1 Leom (2:2, 0). In contrast, the meta-tailoring can avoid this extra generalization error by directly minimizing E,g ( (2r, 0z.s)]-
In the case where B.g (oooo 0z, }] is left large (e.g., due large computational cost at prediction still illustrates competitive generalization bounds of meta-tailoring with small K, when compared to classical induction. For example, with K it provides standard types of generalization bounds with the uniform stability for meta-tailoiing algorithms. Even in this case, the bounds are not equivalent to those of classic induction, and there are potential benefits of meta-tailoring, which are discussed in the following section with more general setting.
time), Theorem D
3.2 META-TAILORING WITH GENERAL TAILORING LOSSES
The benefits of meta meta-tailoring go beyond contrastive learning: the following remark provides the gen- eralization bounds for meta-tailoring with any tailoring loss Ctailor (1:, 0) and an arbitrary supervised loss (fo(x). v).
Remark 1. For any function yp such that (f. v)1 S Ez E 0)), Theorems / and 6 hold with the map Loom being replaced by the function o o Cuallor
Remark [ shows the benefits of meta-tailoring through its effects on three factors: the expected unlabeled loss ( (x,, ()], uniform stability Ç. and Rademacher complexity ( oF..
It is important to note that meta-tailoring can directly minimize the expected unlabeled loss ( (z, (,. whereas classic induction can only minimize its empirical version, which results in the additional generalization error on the difference between the expected unlabeled loss and its empirical version. For example, if yo is monotonically increasing and Luallr (xr, 0) represents the physical constraints at each input x (as in the application in section 5.1). then classic induction requires the physical constraints of neural networks at the training points to generalize to the phys- ical constraints at unseen (e.g.. testing) points. Meta-tail tailoring avoids this requirement by directly minimizing violations of the physical constraints at each point at prediction time.
Another potential benefit of meta- teta--ai can be understood through the improvement in the parameter stability Çe defined such that v(z, y) E X' x )), ||0,.5 - 02.s < Se. for all S, differing by single point. In the case of the meta-tailoring method of fz.s 0g X Aaalloor 0s), we can obtain an improvement on the parameter stability Ço if C (s. 0s) can pull 0g and 0g: closer so that ||0..s 0g < |lās 0s which is ensured, for example, if = -1111. 112 and where cos("1, 02) is the cosine similarity of tq and U2, with en 0g -
cos(1 U1, 02) rell
U/2 A(V Cuallr (z, 0ss) - V (x, 0s:)) and 02 + O. Here, the uniform stability < and the parameter stability Ço are closely related as ( S CCo, where C' is the upper bound on the Lipschitz constants of the maps + (f)((f( (ff(z))))) y) over all (x. y) E X x y under the norm ||. since
I (Jo..s (x), - L (Jup[J,,, (x), v)1 I C'10,,s - 0g < CCa
4 CNGRAD: A SIMPLE ALGORITHM FOR EXPRESSIVE, EFFICIENT TAILORING
In the previous sections, we have discussed the principal motivation and theoretical advantages of (meta- Itailoring. However, there is a remaining issue for efficient GPU computations: whereas one can efficiently parallelize the evaluation of single model over a batch across inputs, it is challenging to efficientl parallelize the evaluation of multiple tailored models over a batch. To overcome this issue, by building CAVIA (Zintgraf et al. (0188) and WarpGrad (Flennerhag et al. 2019), we propose CNGRAD which adapts only conditional normalization parameters and enables efficient GPU computations for staio--nai..oing.. CNGRAD can also be used in regular meta-learning; details and pseudo-codes of both versions can be found in appendix [
As is done in batch-norm (loffe& & Szegedy 2015) after element-wise normalization,
imple- ment an element-wise affine transformation with parameters (7, .8), scaling and shifting the output
h (1) (1r) of each k-th node at 1-th hidden layer independently: (1)
% (x)) + s() In conditional
5

---

Under review as a conference paper at ICLR 2021
normalization, Dumoulin et al. (2016) train collection of (7. 3) in multi-task fashion to learn different tasks with a single network. We propose to bring this concept to the meta-learning and Ameet-)ttailnninn settings and adapt the affine parameters (7, 5) to each query For meta- meta-tailoring, the inner optimization minimizes the tailoring loss at an input x by adjusting the affine parameters and the outer optimization adapts the rest of the network. Similar MAMI. (Finn et al.] (017)) we implement first-order version, which does not backpropagate through the optimization and a second-order version, which does. We can efficiently parallelize computations of multiple tailored models over a Jhini- in GPU in the same way that in a classic induction model, because the adapted parameters only require element-wiss multiplications and additions.
CNGRAD is widely applicable, since we can add these adaptable affine parameters to any hidden While
layer. While we are only changing a tiny portion of the network, we prove below that, under realistic assumptions, we can minimize the inner tailoring loss using only the afîne parameters. However, it worth noting that we still need the entire network to minimize the outer meta-objective.
To analyze properties of the adaptable affine parameters, let us decompose 0 into 0 == (w, 7, 8), where the w contains all the weight parameters (including bias terms), and the (7, contains all the affine parameters. Given arbitrary coefficients "/1..... ,bby E R and an arbitrary function
(fo (z), 2) + (nilor(ff())) (f) let Ciailo 0) -ee are arbitrary input augmentation functions at prediction time. Note that ng is typically small (ng < n) in meta- tailoring; e.g., ng in the method in Section 5. regardless the size of the dataset n.
x- Ti (f (x)), x)), where g()) ................................ g(
Corollary D states that for any given u if we add any non-degenerate Gaussian noise 8 as t + 8 with zero mean and any variance on 8, the global minimum value with respect to all parameters (wr, 7, can be achieved by optimizing only the affine parameters (7, .)) with prohability one.
Assumption 1. (Common activation) The activation function (zz) is real analytic, monotonically
increasing, and the limits exist as: lim (x) = a_ > -00 and lim (z) = = S +oc.
Theorem 2. For any x E X that satisfies (lg (z)1l - (x)) g()))))())))))))))))))()) > 0 (for all i + j), and for any fully-coomeeee and activation functions that satisfy Assumption ] the following holds: inf wr Czailo(zz, w, Y, B) = infy,9 _ Cuilor (I, w. 7) for any w & W where Lebesgue measure of W C. Ed is zero.
neural network with a single output unit. at least ng neurons per hidden layer,
Corollary 1. Under the assumptions of Theorem 2 for any i E RRd, with probability one over randomly sampled 5 E TR. accordingly to any non-degenerate Gaussian distribution, the following holds: inf Liailar J,,, 7)) = inf Luslor (I, v + 8, 7/, 3) for any x E X.
The assumption and condition in theorem 2 are satisfied in practice(see Appendix A for more details). Therefore, CNGRAD is a practical and computationally efficient method to implement (meta- Staloriga
EXPERIMENTS
5.1 TAILORING TO IMPOSE SY SYMERIIIE AND CONSTRAINTS AT PREDICTION TIME
Constructing inductive biases that exploit invariances and symmetries is an established strategy for boosting performance in machine learning. During training, we often regularize our networks to satisfy certain criteria; however, this does not guarantee that these criteria will be satisfied outside the training dataset (Suh & Tedrake 2020p. Another option is to construct ad hoc neural network architectures to exploit constraints for important problems, such as using Hamiltonian neural net- works (Greydanus et al. (2019) to impose energy (but not momentum) conservation. Tailoring provides general solution to this problem by adapting the model at prediction time to satisfy the criteria. In meta-tailoring, we also train the system to make good predictions after satisfying the constraint.
We demonstrate this use of tailoring by enforcing physical conservation laws to more accurately predict the evolution of a 5-body planetary system governed by gravitational forces. This prediction m-body systems become chaotic for > 2. We generate supervised-
problem is challenging, learning dataset with positions, velocities and masses of all 5 bodies as inputs and the changes in position and velocity at the next time-step targets. Appendix (eeandxxEEdeeeeeet describes the dataset in greater detail.
To predict the data, we use a 3-layer feed-forward network and apply a tailoring loss based on the laws of conservation of energy and momentum. More concretely, we take the original predictions and

---

Under review as a conference paper at ICLR 2021
Method
loss
relative
Inductive learning
.041
Outer prediction loss
Opt. output(50 st.)
.041
(0.7 0.1)%
6400-spl. TTT(50st.)
.040
(3.6 + 0.2)%
0.03
Tailoring(I I step)
.040
0.26%
0.02
We minimize the talloring loss the inner loop
Tailoring(5 st.)
.039
(6.3 + 0.3)%
Tailoring(10 10 st.)
.038
Optimizing the output
Meta-tailoring(O st.)
.030
+ (26.3 3.3)%
6400-sample TTT
Meta-tailoring(1 st.)
.029
(29,9 3.0%%
Tailoring
Meta-tailoring(5 st.)
.027
(35.3 + 2.6)%
Meta- Mail--ti
0.00
Meta-tailoring(L1 st.)
.026
(36.0 = 2.6)%
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Physics tailoring loss
Table I: Test MSE loss for different methods; the second column shows the relative improvement over basic inductive supervised learning. The test- time training (TTT) baseline uses full batch of 6400 test samples to adapt, not allowed in regular SL. With few gradient steps, tailoring signifi- cantly over- wer--erforms all baselines. Meta-tailoring improves even further, with 35% improvement.
Figure 2: Optimization at prediction time on the planet data; each path going from right to left as we minimize the physics tailoring loss. We use small step size to illustrate the path. Tailoring and the two baselines share only differ in their test-time computations, thus sharing their starts. Meta-tailoring has a lower starting loss, faster op- timization, and no overfitting the tailoring loss.
adapt our model using the L1 loss between the initial and final energy and momentum of the whole system. Ensuring this conservation can improve predictions, but notice that minimizing the tailoring loss alone does not guarantee good predictions: predicting that the output equals the input conserves energy and momentum perfectly, but is not correct.
Tailoring adapts some parameters in the network in order to improve the tailoring loss. An alternative for enforcing conservation would be adapt the output y value directly. Table [ compares the predictive accuracy of inductive learning to direct output optimization as well as tailoring and meta- tailoring, using varying numbers of gradient steps. We observe that tailoring more effective than adapting the output, as the parameters provide a prior on what changes are more natural. For meta- tailoring, we try both first-order and second-order versions of CNGRAD: the first-order version gave slightly better results, possibly because it was trained with higher tailor learning rate (10-3) 3) with which the second-order version meta-training was unstable (we thus used 10-4). More details can be found in Appendix E. Interestingly, meta-tailoring without any query-tima tailoring steps already performs much better than the original model, even though both models have almost the same number of parameters and can overfit the dataset. We conjecture meta-tail tailooiii training is adding inductive bias that guides optimization towards learning a more generalizable model, even without tailoring at test time. Finally, plot shows the mediction-tina optimization paths for different methods.
5.2 TAILORING FOR ROBUSTNESS AGAINST ADVERSARIAL EXAMPLES
Despite their successes, neural networks remain susceptible to the problem of adversarial exam- ples (Biggio et al. 2013 Szegedy et al. 2013): targeted small perturbations of an input can cause the network to mis-classify it. One approach is to make the prediction function smooth via adversarial training (Madry et al. 2017); however, this only ensures smoothness in the training points and constraining our model to be smooth everywhere makes it lose capacity. This is perfect opportunity for applying (meta-)tate )talloring, since we can ask for smoothness posteriori. only on the specific query.
We apply meta-tailoring to robustly classifying CIFAR-10 (Krizhevsky et al. 2009) and Ima- geNet (Deng al. (2009) images, tailoring predictions so that they are locally smooth. We meta-tailor our classifier using (the first-order version of) CNGRAD AD and a tailoring loss that enforces smoothness on the entire vector of features of the penultimate layer in the neural network (denoted go (x:):
Ctailor 0) gelcos_dist(gg (z:), go + 8))], 8 (O,
where cos dist(v1; , U2) is the cosine distance between vectors U1 and v2. This loss is inspired by Ilyas et al. (2019), who argue that adversarial examples are caused by features which are predictive, but non-robust to perturbations. In that way, our loss adjusts the model to ensure features are locally
7

---

Under review as a conference paper at ICLR 2021
Method
0.0
0.5
1.0
1.5
2.0
2.5
3.0
ACR
(Inductive) RandSmooth Meta-tailored
0.67
0.49
0.00
0.00
0.00
0.00
0.00
0,470
0.25
0.72
0.55
0.00
0.00
0.00
0.00
0.00
0.494
(Inductive) RandSmooth
0.57
0,46
0.37
0.29
0.00
0.00
0.00
0.720
0.50
Meta-tailored
0.66
0.54
0.42
0.31
0.00
0.00
0.00
0.819
(Inductive) RandSmooth
0.44
0.38
0.33
0.26
0.19
0.15
0.12
0.863
1.00
Mcta-tailored
0.52
0.45
0.36
0.31
0.24
0.20
0.15
1.032
Figure 3:
Percentage of points with certificate above different radii, and average certified radius (ACR) for on the ImageNet dataset. Meta-tailoring improves the Average Certification Radius by 5.1%, 13.8%, 19.6% respectively. Results for Cohen al. (2019) are taken from Zhai et al. (2020).
robust at the input query before making a prediction. To keep inference fast, we approximate this loss with single sample, but it could be improved with more samples or more sophisticated methods.
We build on the work of Cohen et al. (2019), who developed a method for certifying the robustness of model via randomized smoothing. It samples several points from a Gaussian N (3r, o²) around the query and, if there is enough agreement classification, provides a certificate that the example cannot be adversarially modified by a small perturbation to have a different class. We show that meta- tailoring improves the original randomized smoothing method, testing for o = 0.25, 0.5, 1.0. For simplicity,
= 0.1 for all experiments. We initialized with the weights of Cohen et al. (2099) to speed up training in all ImageNet experiments and to avoid training divergence for CIFAR-10, ==1 (this divergence had already been noted by Zhai al. (2020)). Note that we could leverage these pre-trained weights because of CNGRA ACGan Dcan start from pre-trained model by initializing the extra affine layers to the identity. Finally, we use o' = vo² va 2s 0.23, 0.49, 0.995 so that the points used in our tailoring loss come from N(x,a²).
In table 3 we show results on Imagenet where we improve the average certification radius by 5.1%, 13.8%, 19.6% respectively. In table in the appendix, we show results on CIFAR-IO where we improve the average certification radius by 8.6%, 10.4%, 19.2%. We chose to meta-tailor this randomized smoothing method because represents a strong standard in certified adversarial defenses, but it is important to note that there have been advances on this method that sometimes achieve better results than those we present here (Zhai et al., 2020; (Salman et al. 2019). see appendix H. However, it is likely that these methods could also be improved through meta-tailoring.
These experiments only scratch the surface of what tailoring allows for adversarial defenses: usually, the adversary looks at the model and gets to pick particularly bad perturbation + 6. With tailoring, the model responds, by changing to weights 0z+8- This leads to game, where both weights and inputs perturbed, similar to: m) min A|<ro Luup (fe+a(1 6), y) However, since don't get to observe y; we optimize the weight perturbation by minimizing Ltailor instead.
5.3 TAILORING TO IMPROVE THE PERFORMANCE OF MODEL- BASED RI.
In model-based reinforcement learning (MBRL) we consider Markov decision process (MDP) with state-space S, action-space A and unknown transition distribution p(x') [s, a). Here, we assume access known reward function r(s, a), although this assumption could be relaxed. In MBRL we learn deterministic transition model [7(s, a) + s' from past experience and use it to either learn a policy or (as done here) to plan good actions by maximizing the reward over some planning horizon H:
H
Oi... H arg max Σ r (sith) arg max at...t
(To (St, Gr.. Qv....h-11))
AL... H Am
where To (St, at... (t...+h--1) To (... To (To (st, 0t) (,441) a1th-1), i.e. applying To h times. We also often apply receding-horte horizon control, where we plan with horizon H, but then only apply the first action a,, observe the next state a++1 and re-plan again. However, in this setting we are optimizing the output of the transition model with respect to (part of) the input: the actions. Similar to adversarial examples in the previous section, this often results in the action optimisation finding regions of the input-space where the model makes overiy-optimistic predictions. In order to improve performance it has been previously proposed to regularize the confidence of the models (Ha & Schmidhuber 2018), to make them uncertainty-aware by learning ensemble of models (Nagabandi
8

---

Under review as a conference paper at ICLR 2021
al. 2020),
make pessimistic predictions (Kidambi al. 2020). among many other approaches.
Similar to the adversarial examples experiments, meta-tailoring opens new simple ways making transition models more robust.
In this section, we explain a simple way of using meta-tailoring to improve transition models in MBRL. In particular, we can learn an action- independent model (x:) - p ($t+.) 1) that out- puts a probability distribution for the next state given the current state, independently of the ac- tion This model presents a trade-off with respect to the original model as) on the one hand, it is intrinsically uncertain since it
Dclaw Turn
200
Reward
-200
lacks part of the information (the actions); we
thus output a probability distribution in the form of a mixture of Gaussians to model this uncer- tainty. On the other hand, it has the advantage of not being "hackable" " by the selliiin pro- cess, making the planner unable to find overly confident regions.
(Inductive) PDDM
Small all meta-teiloring Large meta-tailoring
11000
50
100
150
200
250
350
Iteration
Figure 4: Meta-tailoring improves the performance of MBRL in delaw, a complex manipulation prob- lem, especially for low-data regimes where the transition model imperfect and the prior is thus useful. We plot one confidence interval for the me- dian reward bootstrapped from 16 different runs per configuration. Further increasing the tailoring learning rate results (meta- (eraannt instabilities because the inner loop is highly variable.
More concretely, we first train an action- independent prediction model to maximize the log-likelihood M (st) (8.+1). Then, when mak- ing predictions with To(s,, ae) we use the insieedendeeendentt likeeeildoood of the action- dependent predictions as the tailoring loss: -M()) (To(#t, at)). with a minus sign be- cause we want to maximize the log-likelihood. From this we obtain tailored parameters ((s,,a.) and use them to make the final prediction. In meta-tailoring fashion we train these final predictions to be close to the truth To (o (st, az) z 81+1- In practice, we use the first-order version of CNGRAD.
We use this idea and build on PDDM (Nagabandi al. 2020 which recently showed great results in model- model-based reinforcement learning for robotic manipulation problems. Since they use an ensemble of deep networks Ti( (e) (to make robust predictions). independently meta-tailor each network to leverage a single action-independent learned prior (xx). leaving the rest of PDDM the same. In figure 4 we observe that meta-tailoring improves the performance of the overall predictive model, especially during early parts of training when the action-conditioned model has not become accurate. As expected, once the aenion-conditioned model is good enough (to get positive reward) meta-tailoring is not as useful as we do not need the aetion-independent prior anymore. For more experimental details, see appendix[[
6 CONCLUSION
We have presented tailoring, a simple way of embedding a powerful class of inductive biases into models, by minimizing unsupervised objectives at prediction time. We have leveraged the generality of auxiliary losses and improved them in two key ways: first, we eliminate the generalization gap on the auxiliary loss by optimizing it on the query point instead of on a training set; second, we change the optimization to minimize only the task loss in the outer optimization, and the tailoring loss in the inner optimization. This results in the whole network optimizing the only objective we care about, instead of a proxy loss. Finally, we have formalized these intuitions by proving the benefits of meta-tailoring under mild assumptions.
The framework is broadly applicable, as one can vary the model, the unsupervised loss and the task loss. We have shown its applicability in three diverse domains: physics prediction time-series, contrastive learning, and adversarial robustness. We also provide simple algorithm. CNGRAD, to make meta-tailoring practical with little additional code. Currently, most unsupervised or self-supervised objectives optimized in task-agnostic ways; instead, meta-tailoring provides generic way to make them especially useful for particular applications. It does so by learning how to best leverage the unsupervised loss to perform well on the final task care about.

---

Under review as a conference paper at ICLR 2021
REFERENCES
David H Ackley, Geoffrey E. Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 91):1177-169, 1985.
Ferran Alet, Erica Weng, Tomas Lozano-Perez, and Leslie Kaelbling. Neural relational inference with fast modular meta-leannn neta-learning. In Advances in Neural Information Processing Systems (NeurIPS) 32, 2019.
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization layer in neural networks. In Proceedings of the 34th International Conference on Machine Learming, pp. 335-155.0 JMLR, 2017.
Antreas Antoniou and Amos Storkey. Learning to learn by self-critique. In Advances in Neural Information Processing Systems, pp. 9940-9950, 2019.
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov)4463-488 2002.
Samy Bengio, Yoshua Bengio, and Jocelyn Cloutier. On the search for new learning rules for anns. Neural Processing Letters, 2(4):26 30, 1995.
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šmdić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 466)888-9900 1992.
Olivier Chapelle, Vladimir Vapnik, and Jason Weston. Transductive inference for estimating values of functions. In Advances in Neural Information Processing Systems, pp. 421 427, 2000.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 20iv:20002005099 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference Machine Learning, pp. 1310-1320, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Icee, 2009.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. baseline for few-shot image classification. In International Conference on Learning Representations, 2020.
Jeff Donahue, Yangqing Jia, Oriol Vinyals. Judy Hoffmann, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655, 2014.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.034000 2017.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. arXiv preprint arXiv: 00025, 2019.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04443 2017.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta-learning. arXiv preprint arXiv: 19000779/ 2019.
10

---

Under review as a conference paper at ICLR 2021
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15353-15363, 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.031667 2015.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint (oXiv:2005.0551, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 100), 2006.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint arXiv:/805, 10002, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1eXXiv/:066000 170.006083 2017.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint 16 2016.
Boris Mityagin. The zero set of a real analytic function. arXiv preprint arXiv:1512.07277: 2015.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.0080, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pp. 8024-8035, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off- off-policy rentrrreinfrr learning via probabilistic context variables. arXiv preprint 12X3::1903.882.. 2019.
11

---

Under review as a conference paper at ICLR 2021
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible mulliitask classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, pp. 7957-7968. 2019.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances Neural Information Processing Systems. pp. 11289-11300, 2019.
Jürgen Schmidhuber. Evolutionary principles self-referential learning, on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
HJ Suh and Russ Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. arXiv preprint arXiv:2002.09000 00.02000...099,,.... 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. arXiv preprint arXiv:1909.132.. 2019.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6999 2013.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Sebastian Tschiatschek, Aytunc Sahin, and Andreas Krause. Differentiable submodular maximization. arXiv preprint arXiv:1803.01 2018.
Vladimir N Vapnik. The nature of statistical learning theory, 1995.
Dequan Wang, Evan Shelhamer, Bruno Olshausen, and Trevor Darrell. Dynamic scale inference by entropy minimization. arXiv preprint arXiv: 03000,3382, 2019.
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, and Liwei Wang. Macer: Attack-fres and scalable robust training via maximizing certified radius. In International Conference on Learning Representations, 2020. URL net forum Pin NF4F
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson, Fast context adaptation via meta-learning, arXiv preprint arXiv:1810.03642, 2018.
12

---

Under review as a conference paper at ICLR 2021
A ON THE CONDITIONS IN THEOREM 22 AND COROLLARY
Assumption | is satisfied by using common activation functions such as sigmoid and hyperbolic tangent, as well as softplus, which defined as (x) = In(1+ +ex((az)) //x and satisfies Assumption with any hyperparameter
I
R>o- The softplus activation function can approximate the ReLU function to any desired accuracy: i.., (x) + relu(z) as a + oc, where relu represents the ReLU function.
In Theorem 2 and Corollary the condition - g)) > (for all i * j) can be easily satisfied, for example, by choosing g ..... gl to produce normalized and distinguishable argumented inputs for each prediction point at prediction time. To see this, with normalization
................................
(x)]l ( (z) l1.. the condition is satisfied if (z) اااو (x)1 0 for i * since
e (z)
(z) g0و (z) (gg -ali-gg(ii (x)) (z)
In general, the normalization is not necessary for the condition to hold; e.g., orthogonality on (x) and g0و) (z) along with (x) j satisfies it without the normalization.
B UNDERSTANDING THE EXPECTED
CONTRASTIVE LOSS
To analyze meta-tailoring for contrastive learning, we focus on the binary classification loss of the form _ (fs(z), = (f) (x),, (x))))))))) where foont convex and (0) = 1. With this, the objective function 0 + Luup (ff(( (z)), y) is still non-convex in general. For example. the standard hinge loss (com(() = max(0, 0, - =} and the logistic loss boom = logg + exp(2)) satisfy this condition.
We first define the meta-tailoring contrastive loss Coon( 8) in detail. In meta-tailoring contrastive learning, choose the probability measure of positive example z p2t (I) and the probability measure of negative example x- y ~ Ha-ly y (c), both of which are tailored for each input point prediction time. These choices induce the marginal distributions for the negative examples defined by -p, (x) (1 = y}). Define the lower and upper bound on the probability of y y as p(x) S Py(
~ My- (x) and x ~ Py- (x), as well as the unknown probability of y Py ( (+)) Ey ~
p(x) [0,1).
Then, the first pre-meta-tailoring contrastive loss can be defined by
L . (xr, 0)
F -pat (z)) coom (he (I:) (ha(at)))) ho (ar }))}
MMp (x)
where he (x) E 1 represents the output of the last hidden layer, including a constant neuron
corresponding the bias term of the last output layer (if there is no bias term, ha(x) E R""),, For every E 2 define d'.v.u- (z) Loom ((zy Zy- (h)()) where zy E RRixw is the y-th row vector of z. We define the second pre-meter ceee-meta--tiiinn tailoring contrastive loss by
c=t "0 (I, max E, -p,- (.)l (0(H+1)) - the 22.12 " 1,,
where un Ez mp mop (z) the [] and Uú Eg (w) (z))]. Here, we decompose 0 as (0(1:H), 0(H++))), where ((H+1) = [H-), b(H++)] E Rm, x(mx represents the parameters at the last output layer, and ((1:Hf) represents all others.
Then, the meta-tailoring contrastive loss is defined by
Lcone (x,
(x, 0) p(x)
(Comml" () + L (x, (xx,0)p(zx) p(x))
Theorem 3 states that for any (11:H), the convex optimization of S (x, 0) + xxtx== x-== w (z, over H +1) can achieve the value of cat (a. f)) without the value of cet x 3 (x, 0), allowing
us focus the first term C xt" (xr, 0), for some choice of pt- -y- (xr) and p (x).
13

---

Under review as a conference paper at ICLR 2021
Theorem For any O(1:H), --)- and Pat (2), the function O(H+1) b 2 (x, 0) t "" (2.0) convex. Moreover, there exists Pa- or (2) and px (z) such that, for any ((!:H) and any @(H+1),
inf B EERRRRR Eny xtna (1)
Ccont (z, + L2t x- M (x, S 12t x0 (x, ((H+))))
C
PROOFS
In order to have concise proofs, we introduce additional notations while keeping track of dependent variables more explicitly. Since hg only depends on ((1:H), let us write hgcr-1; ha. Similarly, fromt (I, ((1:H)) co (z, Let ((x) 02 and 0(x, S) 0z.s. Define C = Luup
C.I PROOF OF THEOREM 2
Proof of Theorem 2 The output of fully-connected neural networks for an imput I with a parameter
vector 0 = (w,7.3) 5) can be represented by fe(x) = ( (z) + ((#+1) where ((++1)) e II 11xma and ((H+1) E R are the weight matrix and the bias term respectively at the last layer, and (x) E R represents the output the last hidden layer. Here, my represents the number of neurons at the (-th layer, and (x)) = (O(()) (x) 3(0) E m for = I...., H. with trainable parameters y(0), 30 E RT where (z))))) = x. Let (x) = (2)))
(()).
Then, by rearranging the definition of the output of the neural networks.
fo(x)
(6 + ((H+1)
mg
W W 7
(H+1) (H) (H) (2)( W
(H++)) g(H)
D8 + ((#+11
((00)
(W(H+1)) oz (x) w(
3(0)) +1(
Thus, we can write
(z))
[y(H]]
Mew
+b(H+1)
1ng R9
[g(m)]
(o(g(m)(z))
where
((#+1) o (1) (g ,
((H+1)
M
R X2mH
w(H+1) 21t) (g(m))) ((#+1)
and I,, = 1, [I,
1 E R
Using the above equality, we show an exitance of a (y,3) such that Lmillr 71 = inf Chuilor 0) for any X R m and any W where Lebesgue measure of W Fdd is
zero. To do so, we first fix T' for f H-
1 and 3 0 for / - 1,... H 1, with which (1)) = (1)
Define (ww) = det MT), which is analytic since is analytic. Furthermore, we have that {w Rd Mn,, has rank less than ng} = {w E Rdd (p(w) = 0}, since the rank M, and the rank of the Gram matrix are equal. Since yp is analytic, if sp not identically zero (y 3f 0), the Lebesgue measure its zero set {w E Id ((w) = 0} is zero (Mityagin 2015). Therefore, if (p(w) + 0 for some ddd the Lebesgue measure of the set {w E M has rank less than ng } is zero.
Accordingly, now constructs E d such that (lw) + Set ((++1) 1 1TT Then,
M [M,y 1 mng E Xmn
14

---

Under review as a conference paper at ICLR 2021
where
(z)))
Mw
(g (x)) (x))))
and 1nyma E xm with (Iny.mn )is = for all i.j. For 1..... H. define
(z
G
E R
: (g(h))
Then, for = =1,...... 1,..., H,
G = (W))) + 1,,,,,,)))) (b
where a is applied wlement-wise (by overloading of the notation σ), and
(Mw) (G(")) )
c for all * j. From Assumption I there exists d' such that 0+ a_ > c. Using these constants,
From the assumption g(ar,, there exists
such that l1g (x)
wi (()) (x) and bit) ca a (1 (x)1 for i 1, ................................ ng. where W( l:m...m, lm, = a In, and 0y = d'a /2 a a a+ for all & and / = 2, ...., H. where W 1:,,,I:m, is the fist ng ng block matrix of ((2) and Ln, is the ng ng identity matrix. Set all other weights and bias to be zero. Then, for any (l,...,ngh ng),
represents the i-th row of (()), Moreover, set w (1)
k == =
and for any 1: E (1,....,,y not with 1: +
( () (1g (z), Since o (1) (co /2) - 0+ and o (-oo
(1g c/2)) S ccc-oo())2)
as (1) + co, with (1) sufficiently large, we have
C2 > 0. Note that of and c depends only on (!) and does not depend on any of a (2) (H), Therefore, with (1) sufficiently large,
that or (ca (1/22)
0+ + 2 2 d and 12) -4 +c// -c6-
(2) for some ci (2)
(G
-0 + 2 (0)
and
(G jm S (o(cca) - +212)) S o (2)
Repeating this process with Assumption
(1-1) sufficiently large, | we have that with a 11 a (1-1)
(G )e 2 (()
and
(G ik S so(-a (m) (m)
(H)
Here, (G(H));s + 04 and (G is + a_ as (1)) + oc. Therefore, with a!))... (()) ciently large, for any i E (l......... nggh
(1) suffi-
((M)) -
Σ ligfi
(Mu)it -01-
(3)
The inequality equation means that the matrix Mr = [(M))) - --lisissn, E R X is strictly diagonally dominant and hence nonsingular with rank ng. This implies that the matrix [Mr ng E R", x (1)+1) has rank ng. This then then implies that the matrix M., RR x H1) has rank ny. since the elementary matrix operations preserve the matrix rank. Since the set of all columns M,, contains all columns M,,. this implics that Map has rank ng and (ww)) o for this constructed particular
In
15

---

Under review as a conference paper at ICLR 2021
Therefore, the Lebesgue measure of the set W = {w B (p(w) = 0} is zero. If w & W. 5.,5,533 (g (x)), ...................... f. (xg())) equation 2 for some 7/11, 3() for = l,.... : H as shown above. Thus, for any t & W and for any (w, 7, B). there exists (5, 8) such that
R 7((1, B(0) Rm = R",, since My has rank ny
(00 (f a olg (6 fw. a (22) (2)()))) (z))
which implies the desired statement.
C.2 PROOF OF COROLLARY 1
Proof of Corollary Since non-degenerate Gaussian measure with any mean and variance is ab- !
solutely continuous with respect to Lebesgue measure, Theorem implies the statement of this corollary.
C.3
PROOF OF THEOREM 1
The following lemma provides upper bound on the expected loss via expected meta-tailoring contrastive loss.
Lemma 4, For every 0,
Bz,y/ [C(fo(x) S Eg
(Comi" (x,, (1:11)
C (x, - p(z))
p(x)
Proof of Lemma f
Using the notation PylHt- (x)),
Ex, Bz.y (1)) v)l
-Ery
((1 (())))(((()))))))
Er,
((1 (f0(x))) - fo(x), *u) + (lcom((((())) fo(x), =n) -p)
1
1 (Bg-an, (a) loom (allloofffll (fo(z), fo(x)) p)
BB ~ny- (a) Wa)War (0(H+1))) o)
Ery
1
1 (a.1t )) + Lem = (z. 0) -p) p)
< Ery
< Eny
(Ceti"" (x,, ((::0)
x= x= (x. -0) .)
where the third line follows from the definition of L(fo(x), y) and (fo(() (o(x)) Lomml (0) 1, the forth line follows from the definition of p and the expectation Ey - (w)) the
y- hotnon (I)), the sixth line follows from the definition of L x " The last line follows from the convexity of foom and Jensen's inequality: i.e.,
fifth line follows from fx 8 1! higrees (x) and fo(x), -
(e,,1,(uu,,,,]] ClonieEx-aa (Ex+ teettrat (x) Ea- Ba--ne---a-p (4) (h) (x+ hgaren (x (T hg (x))) -coml
< E mpp Ez up.- (z) Root (hgar (x+)
hoa (2 )x hgguo (x)).
16

---

Under review as a conference paper at ICLR 2021
Therefore,
Bz,y (F(o((())))
- Pylly (2)) (Caime (x,, ((1:t))) + Lovai x=t (a, Pe Po (1))))
Ez,y
(Ccom"" x (I,, (0,001-H)) Feont" x N (x, - (x,0)-p(xz) e(x))
$ Ex
p(x))
where we used p(x) Pa PyllKy (1)) S (()) (0,
Lemma 5.
Let S + fala.s) (x) be an uniformlly Ç-stable tailoring algorithm. Then, for any >>
with probability least over an i..d draw of n ii.d samples ((xx,))))) (:)) the following holds:
+26 () + 5 + (2C + (25++)) +ol 2n
In(1/5)
Ea,yLC (
Ex (u (x)) and (2((S) 1 Di=1 ()) yc), and (PS) (1) (S)).. To apply McDiarmid's incquality to (S), we compute an upper bound on (5(( - ((S") where S and S' be two training datasets differing by exactly one point of an arbitrary index io: i.., S Sy for all + in and Si, + 5
Proof of Lemma Define (s((5)
where ((14, (()4,))) Let Ç
č= { Then,
(s(( s(S")) = (5) - (01) + (12((5) sp2(5))
For the first term, using the ç-stability,
1 sp (s)) S Fn [Fnnl (1)
<C
For the second term, using Ç-stability and the upper bound on per-sample loss,
(12(S) 1
Σ
vi) ((for...))((s))...) n
7 i
(n
1)5
c
n
n
n
Therefore, -p(0)) 2 By McDiarmid's inequality, for any 5 > 0, with probability at least 1 - 6,
((S) Es(p(S)) (2C + +cly In(1/8)
2n
The reset the proof bounds the first term Es]{ By the linearity of expectation,
Es Es]p(S)] (s]()((S) Es(pi(S)).
For the first term,
Es(pi (S)] = / [[)o),.)) For the second term, using the linearity of expectation,
Es(p2(S)] Es
C ((
1
Es (T))
n
Eg,., [Cfor 2)1.
17

---

Under review as a conference paper at ICLR 2021
where S" is sample of n points such that (S = Sy for j i and (St.y)e (x,,)) 1). By combining these, using the linearity of expectation and ç-stability,
Es Es(p(S)]
=
Es [C(f ((())),))()))))) (x)
i=1
Es [If((((..))) (t))
n
Therefore, Es(p(s)] - 5.
Proof of Theorem ! For any and [0, 1),
En, ([(o(x)) <B.., (C([o()))))))))))))))))))))))))) + (C
Applying Lemma for the first term and Lemma yields the desired statement.
C.4 STATEMENT AND PROOF OF THEOREM 6
Theorem 6. Let F be an arbitrary set of maps I + fe, (z). Then, for any 5 >> O, with probability at least 1 8 over an i.i.d. draw of i.i.d samples ((Iz. Yi 1i))), Di-1, the following holds: for all maps (a +
fo, (z)) F and any к. E [0, 11, we have that Ez.z [ (fo, (2),0) S kE [F (2. O (1 where J' Ei= L [o.. (e))..)) + 2R, (Luu F) ey (In(1)))/(2nn))
The following lemma is used along with Lemma prove the statement of this theorem.
Lemma 7. Let F be an arbitrary set of maps æ + fa, (x). For any 8 > O, with probability at
least - 8 over an i.i.d. draw of n i..dd samples ((zi, 9;)) the following holds: for all maps (x + (x)) E F.
In(1/8)
(f (x), S "E (2)..,)) 2R. (
2n
Proof of Lemma
7
Let S ((21, v:)) and S' ((x1, v()) Define
((S)
sup
Em [C(e, U)1
Σ C(fez, (2)).
(a+ (a))
To apply McDiarmid's inequality to (SS). we compute upper bound on (x(() cp(S") where S and S' be two training datasets differing by exactly one point of an arbitrary index io: i.e., St = S for all & is and Sip # 5%, Then,
((ferss)) Vio) - [(foret, (2
(0S') - (p()) <
(2+
Similarly, spS) ((S') S. Thus, by McDiarmid's inequality, for any 8 >> with probability least 1 -.
ln(1/))
p(S) Eglp(SS) +
2n
18

---

Under review as a conference paper at ICLR 2021
Moreover, with f(x) foz (z),
1 Σ n
Es(p(S)) = Es
sup
Eg
C(So (2))
EEP
n
i=1
==
< Es,s
sup /EJ
(C( (Coce))))))) L(Jo,, (13))
1=1
< S.s
sup
((C(o L(fes, (2ci), will
JEF
- 2Bgs
sup EiC(fez, (2i) (i))) [EF 1=1
where the fist line follows the definitions of each term, the second line uses the Jensen's in- equality and the convexity of the supremum, and the third line follows that for each & E {-1,+1}, the distribution of each term Ei (((C(fot))(=)) (E(fe(a)) (zt). v() - L(fo., (Ia), y1)) since and are drawn iid with the same distribution. The forth line uses the subadditivity of supremum.
[(fe., (.r). y:)) is the distribution of
Proof of Theorem
6 For any and k: E 1).
Ex. [(( kBz.y (C((f()))))))))))o))))))(()) v)1 (1 x))2x(((fo(()) v)l-
Applying Lemma for the first term and Lemma yields the desired statement.
C.5 PROOF OF THEOREM 3
Proof of Theorem Let ((1:H) be fixed. We first prove the first statement for the convexity. The func- tion ((H+1), + Vango (0(H+2) is convex, since it composition of convex function foom and affine function 2 + (2y - 2y- Hagcm (x)). The function g + Ey-~y, (zx (=)[va... (0( (, 1 is convex since the expectation and affine translation preserves the convexity. Finally, ((H+1), -at (ar, (1:HH, 0(H+11) is convex since it is the piecewise maximum of the convex functions
g(H+1) Ey-~ny (o)l (B(H+1)) (2 u,17))
for each y.
We now prove the second statement of the theorem for the inequality. Let us write Hg+ = Maly and Ma- Maly- Let U [11, [2] E "2"" x (mx+1) where My = E hhgaauo (x)) for E (1, Then,
" E ~ (x) Moqan (x+) Ex ~ur [hacee (x
My:
and
Up Ex-~ (1)) (haci (x] E [pac uyy
Therefore,
(2 un 1. V2 (U),
with which
C x 21 (x,0) max Ey- ~M (w) Gelly.y. (0(H+1)) VaNN (U)).
Since U and ((1:1) do not contain 0(H+1), for any U, (((HH there exists ((H+1) U for which Vaaw (0(H-11))- vaana (U) 0 and hence cati "n (2, Therefore,
inf g( xim
ceti" (x, 0(2:H1) + cuti"" = (x, (1:HH) ((H+1))
Lotti" (x, g(1:H))
inf 0(++1) ( (mn)+))
Ce"i" "t (x, ((:Ht) g(H+1))
cotti" (x, g(l.ti)..
19

---

Under review as a conference paper at ICLR 2021
C.6
PROOF OF REMARK 1
Proof of Remark
7 For any 0,
Egy(C(fo(z). y)] = inf KEn, ((fo(()()) + + - x))x,ylC(fe(e v)]. ke|o,1]
Applying Lemma 5 for Theorem [ (and Lemma 7 for Theorem 6 to the second term and the
assumption Ez,y (x)( v)] I E, Cnn (fo(z)) to the first term yields the desired statement.
20

---

Under review as a conference paper at ICLR 2021
D DETAILS AND DESCRIPTION OF CNGRAD
In this section we describe CNGRAD in greater detail: its implementation, different variants and run-time costs. Note that, although this section is written from the perspective of meta-tailoring, CN- GRAD is also applicable to meta-learning, we provide pseudo-code in algorithm 2 The main idea behind CNGRAD is to optimize only conditional normalization (CN) parameters , in the inner loop and optimize all the other weights w in the outer loop. To simplify notation for imple- mentation, in this subsection only, we overload notations to make them work over a mini-batch as follows. Let be a (mini-Jbatch size. Given
E R 7 E RxxE," Th and E R m" furm..s, (X;) where X,, "/6 and 3; are the transposes of the i-th row vectors of X, y and 8, respectively. Similarly, Lup and Luallr are used over a mini-batch. We also refer to 0 = tw, Y, 8).
let (fwrz,(X))
Initialization of y, B In the inner loop always initialize 1,,5, may 0b More complex methods where the initialization of these parameters is meta eeta-trained are also possible. However, note two things:
1. By initializing to the identity function, we can pick an architecture trained with regular inductive learning, add CN layers without changing predictions and perform tailoring. In this manner, the prediction algorithm is the same regardless of whether we trained with meta-tailoring or without the CN parameters.
2. We can add a previous normalization layer with weights ,,,,3 that are trained in the outer loop, having a similar effect than meta-learning initialization. However, we do not do it in our experiments.
First and second order versions of CNGRAD: affect Lu in two ways: first, they directly affect the evaluation Swn...8 (X) by being weights of the neural network; second, they affect V V.C"ailor which affects 7/s 3. which in turn affect _ Similar to MAML (Finn et al. implement two versions: in the first order version only take into account the first effect, while in the second order version we take into account both effects. The first order version has three advantages:
2017).
1. It is very easy to code: the optimization of the inner parameters and the outer parameters are detached and we simply need to back mack-propagate Giailer with respect to B, y and _ with respect to w. This version is easier to implement than most meta-learning algorithms, since the parameters in the inner and outer loop are different.
2. It is faster: because we do not back-propagate through the optimization, the overall compu- tation graph is smaller.
3. It is more stable to train: second-order gradients can be a unstable to train; this required to lower the inner tailoring learning rate in experiments of section 3. the second-order version.
The second-order version has big advantage: it optimizes the true objective, taking into account how Ciailor will affect the update of the network. This is critical to linking the unsupervised loss to best serve the supervised loss by performing informative updates to the CN parameters.
WarpGrad-innpired stopping of gradients and subsequent reduction in memory cost: Warp- Grad (Flennerhag al. 2019) was an inspiration CNGRAD suggesting to interleave layers that are adapted in the inner loop with layers only adapted in the outer loop. In contrast to WarpGrad, we can evaluate inputs (in meta-tailoring) or tasks (in meta-learning) in parallel, which speeds up training and inference. This also simplifies the code because we do not have to manually perform batches of tasks by iterating through them.
WarpGrad also proposes to stop the gradients between inner steps; we include this idea as optional operation in CNGRAD, as shown in line 12 T The advantage of adding it is that decreases the memory cost when performing multiple inner steps, computation graph of the last step instead of all the steps, key when the networks are very deep like in the ex exrimeents of section 5.2 Another advantage is that it makes training more stable, reducing
now only have to keep in memory the
21

---

Under review as a conference paper at ICLR 2021
variance, back- pooppprppggaiing through the optimization is often very noisy for many steps. At the same time it adds bias, because it makes the greedy assumption that locally minimizing the decrease in outer loss at every step will lead to low overall loss after multiple steps.
Computational cost: in CNGRAD we perform multiple forward and backward passes, compared single forward pass in the usual setting. particular, if we perform tailoring steps, we execute (s + 1) forward steps and backward steps, which usually take the same amount of time as the forward steps. Therefore, in naive implementation, this method takes about + 1 times more than executing the regular network without tailoring.
However, it is well-known that we can often only adapt the higher layers of a network, while keeping the lower layers constant. Moreover, our proof about the capacity of CNGRAD to optimize broad
range of inner losses only required us to adapt the very last CN layer , _ This implies we can put the CN layers only on the top layer(s). In the case of only having one CN layer at the last network layer, we only require one initial full forward pass (as we do without tailoring). Then, we have & nockward-forwart steps that affect only the last layer, thus costing И case of layers of equivalent cost. This leads to a factor deep networks), is very small overcost. Moreover, for tailoring and meta-tailoring, are likely to get the same performance with smaller networks, which may compensate the increase in cost.
+ H
2s in cost, which for s small and H large (typical for
Meta-learning version: CNGRAD can also be used in meta-learning, with the advantage of being provably expressive, very efficient in terms of parameters and compute, and being able to parallelize across tasks. We show the pseudo-code for few-shot supervised learning in algorithm 2 There are two changes to handle the meta-learning setting: first, in the inner loop, instead of the unsupervised tailoring loss we optimize supervised loss on the training (support) set. Second, we want share the same inner parameters 7, ß for different samples of the same task. To do "repeat interlave" (PyTorch notation), which makes it contiguous copics of each parameter y, B, before feeding them to the network evaluation. In doing so, gradients coming from different samples of the same task get pooled together. At test time we do the same for the k' queries (k' can be different than k). Note that, in practice, this pooling is also used in meta-tailoring when have more than one data augmentation within Cuallrr
add the operation
22

---

Under review as a conference paper at ICLR 2021
Algorithm 1: CNGRAD for meta-tailoring
Subroutine Training(f, L Asup- Chillr, Ataitor. steps, (( y1))
randomly initialize w // Ail parameters except 7,8; trained in outer loop
while not done do
for 0 < n/b do
// b batch size
X, Y Libri(b+1): Vib.((b+1)
70 = 1,E1 mi
30 = Ob.Ei mi
for 1 I S sleps do
is 7/-1 Atastor V fiailor (w;; 7, 1, X)
B. B,-1 Aeasior V V Ciailer (10, 7a- B)
w Asup V wu (S 3, (X),Y)
3. 7 = -e.detach (), dettcc))
// Optional operation:
WarpGrad detach
avoid back-proping through multiple steps; reducing memory, and increasing stability, but adding bias.
13 return
14 Subroutine Prediction( w. Cuillr, steps, X)
// For meta-tailoring
tailoring
// x contains multiple inputs, with independent tailoring
processes
15
6 X.shape(0]
// number of inputs
7 1,,5, mi
3o 0g,E,
18
for 1 < S steps do
7 96-1 AV
Ctailor (w, 7/- 8,-1, X) 8, Ba-1 AV Ltailor (w, 7s-1, Bg-1, X)
return f fui Bateps (X)
23

---

// Take samples from each task for Under review as a conference paper at ICLR 2021
Algorithm 2: CNGRAD for meta-learning
Subroutine Meta-training(,, f. L Xinners Aouter, sleps,T)
randomly initialize w /1 All parameters except 7, trained in outer loop
while not done do
for 0 0 /n///bboo do
// b batch size
Xtrain, Y Ytain Xtest; test [1.1]
11:11
for ib S i(b do
(inp, out) ~*
training
X.append (imp): .Y.Y aappend (out)
(query, larget) ~ Tj // Take k' samples from each task for testing
X X.append (query) Y.append (target)
/1 We can now batch evaluations of multiple tasks Xtrain Yirin concat (Xtrasn dim 0), concat (Ytrain, dim 0)
Xtest, Y Ytest concal (Xtest, dim = 0) concal (Yeest, dim 0)
18
70 1,, m
14
3n = Og. m
for 1 < 8 S sleps do
// We now repeat the CN parameters k times so that samples from the same task share the same CN parameters
1 - "s-1 "es- interteaveek repett ret.eett_inteereaee(kk. 1)
7s 8--1 Ainner V w Btr, (Xtrain). Virai))
3. = Ba-1 Vinnner D (f (fw " (Xtrain); Ytrasn)
18
1s test glest r'peett interleave (K' 1), Br Be.repat interleave(k,, 1)
w Aouter Cuu (Swrggen gteat (Xtest), Yzest)
Br 7, dedelach()), delecc))
// Wa arrrradd detach to not backprop through multiple
steps
return
Subroutine Melu-tesi(J, J, w, Lup, Seee Ytrains Xtest)
// Assuming a single task, although we could evaluate
multiple tasks in parallel as in meta-training.
24
7 = 1, mi
// single 7 because we only have one task
25
80 = 01,2i mi
26
for 1 < S steps do
27
1.-1:1% = reeea.eepett_ime
prpes- 1)
7. -o-1 >innner V (f mo oor (Xtrein); Virain)
28
B, = B--1 Ainnner V PP (fwn Str, (Xerain), Y Ytrain)
29
glest "/steps. interleave 1), Sateps repeat interleave(k!, 1)
30
Tsteps:
31
return fwrnten gest (Xtest)
24

---

Under review as a conference paper at ICLR 2021
E EXPERIMENTAL DETAILS OF PHYSICS EXPERIMENTS
Dataset generation As mentioned in the main text, 5-body systems are chaotic and most random configurations are unstable. To generate our dataset we used Finite Differences to optimize 5-body dynamical systems that were stable for 200 steps (no planet collisions and no planet outside predetermined grid) and then picked the first 100 steps of their trajectories, to ensure dynamical stability. To generate each trajectory, we randomly initialized 5 planets within 2D grid of size w = 600, h = 300, with a uniform probability of being anywhere in the central grid of size w/ 22 h/2. each with a mass sampled from a uniform between [0.15, 0.25] (arbitrary units) and with random starting velocity initialized with a Gaussian distribution. We then use 4th order Runge- Kutta integrator to accurately simulate the ODE of the dynamical system until we either reach 200 steps, two planets get within certain critical distance from each other or planet gets outside the pre- configured grid. If the trajectory reached 200 steps, we added it to the dataset; otherwise we made small random perturbation to the initial configuration of the planets and tried again. If the new perturbation did not reach 200 steps, but lasted longer we kept the perturbation as the new origin for future initialization perturbations, otherwise we kept our current initialization. Once all the datasets were generated we picked those below a threshold mean mass and partitioned them randomly into train and test. Finally, we normalize each of the 25 dimensions (5 planets and for each planet U.y.Ua to have mean zero and standard deviation one. For inputs, we use each state and as target we use the next state; therefore, each trajectory gives us 100 pairs.
For more details, we attach the code that generated the dataset.
Implementation of tailoring, meta-tailoring and CNGRAD All of our code is implemented in PyTorch (Paszke et al. 2211)) using the higher library (Grefenstette et al. com/ www.aaaooooooaaaacaa.aa / to implement the second-orde version of CNGRAD. We implemented a 3-layer feedforward neural network, with a conditional normalization layer after each layer except the final regression layer. The result of the network was added to the input, thus effectively predicting the delta between the current state and the next state. For both the first-order and second-order versions of CNGRAD, we used the detachment of WarpGrad (line 12 in algorithm []] For more details, we also attach the implementation of the method.
Compute and all the and simply copied these values for tailoring and meta-tailoring. For the latter two, we also had to search for Atailor.
search To keep the evaluation as strict as possible, we searched affecting the inductive baseline and our tailoring versions with the baseline
The number of epochs was 1000, selected with the inductive baseline, although more epochs did not substantially affect performance in either direction. We note that meta-tailoring performance plateaued earlier in terms of epochs, but we left it the same for consistency. Interestingly, we found that regularizing the physics loss (energy and momentum conservation) helped the inductive baseline, even though the training data already has O physics loss. We searched over [10-2,3 3, 10-4, 10-3, 3. 10-3 10-20 for the weight assigned to the physies loss and chose 2 10-3 for best performance in the inductive baseline. To balance between energy and momentum losses we multiplied the momentum loss by 10 to roughly balance their magnitudes before adding them into a single physics loss, this weighting was not searched. We copied these settings for meta-tailon meta-tailoring.
In terms of the neural network architecture, we chose simple model with 3 hidden layers of the same size and tried [128, 256, 512] on the inductive baseline, choosing 512 and deciding not to go higher for compute reasons and because we were already able to get much lower training loss than test loss. We copied these settings for the meta-tailoring setup. We note that since there are approximately O(m)) weight parameters, yet only O(m)) affine parameters used for tailoring, adding tailoring and meta-tailoring increase parameters roughly by fraction ((1/ma) inductive bascline, we tried adding Batch Normalization (loffe & Szegedy 2015), since it didn't 't affect performance decided not to add it.
about 0.2%. Also in the
We chose the tailoring step size parameter by trying [10-5, 1004, 10-3 10 10-2), 2], finding 10-3 worked well while requiring less steps than using a smaller step size. We therefore used this step for meta- tailoring as well, which worked well for first-order CNGRAD, but not for second-order CNGRAD, whose training diverged. We thus lowered the tailoring step size to 10-1 for the second-order version,
25

---

Under review as a conference paper at ICLR 2021
which worked well. We also tried clipping the inner gradients increase the stability of the second- order method; since gains on preliminary experiments were small, we decided to keep out for simplicity.
For meta-tailoring we only tried and tailoring steps (we wanted more than one step to show the algorithm capability, but few tailoring steps keep inference time competitive). Since they worked similarly well, we chose 2 steps to have a faster model. For the second-orde version we also used 2 steps, which performed much better than the inductive baseline and tailoring, but worse than the first-order version reported in the main text(about 20% improvement of the second-order version vs. 7% of tailoring and 35% improvement of the first-order version).
For the baseline of optimizing the output we tried step size of 10 4, 10-3, 10 1002, 1.101 Except for step size of 10 results optimized the physics loss and always achieved very small improvement, without overfitting to the physics loss. We thus chose big learning rate and high number of steps to report the biggest improvement, of 0.7%.
Runs, compute and statistical confidence: ran each method 2 times and averaged the results. Note that the baseline of optimizing the output and tailoring start from the inductive learning baseline, they meant to be methods executed after regular inductive training. This is why both curves start at the same point in figure 2 For those methods, we report the standard deviation the mean estimate of the improvement, since they executed the same run. Note that the standard deviation of the runs would be higher, but less appropriate. For meta-tailoring, we do use the standard deviation of the mean estimate of both runs, since they are independent from the inductive baseline.
All experiments were performed on a GTX 2080 Ti with 4 CPU cores.
26

---

Under review as a conference paper at ICLR 2021
F UNDERSTANDING THE DIFFERENCE BETWEEN TAILORING AND TEST-TIME TRAINING WITH EXPERIMENTS ON THE PLANETS DATASET
As discussed in section multiple methods have been proposed to optimize unsupervised objectives at test time to account for some deficiency in the training distribution. This makes a lot of sense: has been widely observed that training deep model in some task can learn useful features so that fine-tuning it on small amounts of data for a different task achieves high performance (Donahue et al. 2014). When having access to some unlabeled test data that differs from the training data in some way (distribution, task, image scale, simulation vs real, etc) we would want to fine-tune the model to account for this, but by definition, we do not have labels. Therefore, multiple methods have been proposed to optimize an imsupervised loss to adapt to the test.
Probably the closest tailoring (but not meta- meta-tailoring)) is "Test-time training with self-supervision for cut-of--disribution generalization" (Sun et al. 2019) which uses available test samples to approximate the test distribution when there is a distribution shift and optimizes an unsupervised loss to adapt to that distribution. There are three key differences with our work.
1. In contrast to test- test-time-tres time-trainnng for -iit----iittibttionn generalization and other prior work, we suggest that adaptation is useful even when there is no change between training and test distribution or task, solving different problems. We show test-time training is not good for same-distribution generalization and that meta-tailoring is not the right approach for out-of-distribution generalization.
2. Because adaptation makes sense even on the same distribution, it also makes sense to do tailoring at training time, not just at test time, resulting in meta-tailoring. Moreover, this work shows theoretically and empirically shown that meta-tailoring is a better paradigm than simply tailoring because, when training and test come from the same task and distribution, we want to train as we test
3. Test-time training and other prior work suggest to adapt via unsupervised samples on available data from the correct test distribution. This suggest two testable consequences that test-time training predicts but tailoring doesn't 't; we show neither is satisfied when the distributions are the same:
(a) Since samples are used to adapt to the test distribution, most of the benefit of adapting with sample x before evaluating on z should be captured by adapting on another test sample x' # x. We show this is not the case when training matches test.
(b) As we get more test samples performance should increase (and it indeed does in their OOD experiments). We show this is not the case when the distributions are the same: even with 6400 samples (a full batch, including the evaluated sample), performance is worse than just tailoring on the one trained sample. Meta-tailoring further increases this gap.
Related work on unsupervised adaptation are solving different problems "Test- -test-time training with self-supervised objectives for dutti--distiibution generalization" (Sun et al. 2019) focuses on problems where there is a distribution shift between training and testing. Other methods have used similar unsupervised fine-tuning methods to adapt from simulated to real data or to different tasks in the meta-learning context. The key insight of this work (which echoes that of Vapnik in the 80s) is that we can tailor the model to each query (both test and training queries) at prediction-time.
The single-sample case of test-time training is computationally equivalent to tailoring when the adaptation sample is the same as the evaluated sample (which is the most common case). However, this new application of fine-tuning motivates meta-tailoring which also applies it at training time in an inner loop. At the end of this section we show experimental evidence that test-time training is worse than tailoring in the classic ML setting. Conversely, meta-tailoring (which is well-motivated for same-distribution generalization) may not be well-suited for extrapolation: first, our theoretical guarantees no longer apply and second, we experimentally tested on an extrapolation version of the planets dataset (asking to generalize to bigger planets) and meta-tailoring performs worse than just
We call our paper tailoring instead of meta- tatt--ti because the key idea is that fine-tuning is useful to encode inductive biases in the classic learning paradigm where test matches training. Moreover, meta-tailoring could confuse readers into believing we are meta-learning paradigm.
27

---

Under review as a conference paper at ICLR 2021
tailoring. The intuition for why this may be the case is that meta-tailoring couples the unsupervised loss to the task loss; however, this coupling is learned for the training distribution, not the test distribution. When applied to a different distribution, this coupling may be incorrect.
Meta-tailoring the correct paradigm for classic generalization Our theoretical guarantees in section 3 apply to meta-tailoring, but not tailoring. This is because in machine learning to get guarantees we usually need to test as we train; which meta-tailoring does, but tailoring and test-time training do not. Note that the guarantees done by test-time training (Sun et al. 2019), which would also apply to tailoring, assume a relation between the two gradients when the model is already trained. This posteriori assumption is much more strict than the a priori assumptions need to make assumptions about the trained model and instead guarantee that (meta-)training will happen in such way that the conditions will hold for the trained model.
use, which do not
Finally, all our experiments use meta-tailoring to obtain results, since it also performs better in practice.
Experiments show that tailoring a better approach than test-time training in same- distribution generalization. Test-time training (and other fine-tuning for adaptation methods) suggest to fine-tune on the available samples. The most useful case when we use the single test sample for itself (tailoring). However, we can make the artificial experiment of using different sample for adaptation than the one we test. This separates the adaptation component from the tailoring component. Our experiments showed that optimization on a different sample is much slower (50 steps vs less than 10) and results in less than half the improvement than tailoring (2.9% vs 7.5%). Notably, meta-tailoring adapts even faster than tailoring, and much larger improvements (36%).
Possibly more surprising, we also experiment on allowing test-time training to adapt on the entire test batch, which by definition will now include each sample we evaluate on. Each batch contained 64 trajectories of 100 states, thus 6400 samples. 1 of which is the test query and 99 others are from the same trajectory, and thus specially related. Note that this is not allowed in the classic test evaluation where test samples do not see each other.
For test-time training and other munituning methods, having 6400 times more data would be extremely useful to have much better adaptation. From the tailoring point-of-view, however, the extra samples are a nuisance, since it now harder to tailor the model to each individual sample. We experimentally find that 6400-sample test-time training slightly improves over single-sample test-time training, but is still much worse than tailoring and meta-tailoring: 3.6% vs 7.5%, 36% improvement. Note that, since the evaluated query is part of the 6400 optimized samples, this provides strong evidence that the other 6399 samples are hurting the fine-tuning, aligning with the tailoring approach instead of the adaptation approach.
G TOY ADVERSARIAL EXAMPLES EXPERIMENT
We illustrate the tailoring process with a simple illuminating example using the data from llyas et al. (2019). They use discriminant analysis on the data in figure 5 and obtain the purple linear separator. It has the property that, under assumptions about Gaussian distribution of the data, points above the line are more likely to have come from the blue class, and those below, from the red class. This separator is not very adversarially robust, in the sense that, for many points, a perturbation with small 8 would change the assigned class. We improve the robustness of this classifier by tailoring using the loss Caillrr (a. 0) (L()(fo(z)) || ((fo(z + arg max E, efolm+)))) ))), where KL represents the KI. divergence, & is the logistic function, and & ((((())))) the probability of being in class i, so that (z)) represents the entire class distribution.
With this loss, we can adjust our parameters so that the KL divergence between our prediction at is closer to the prediction at perturbed point 5, over all perturbations in radius Each of the curves in figure 3 corresponds to a decision boundary induced by tailoring the original separator with different value for the maximum perturbation Note that the resulting separators are -ineer, even though we are tailoring a linear separator, because the tailoring is local to the prediction point. We also have the advantage of being able choose different values of at prediction time.
28

---

Under review as a conference paper at ICLR 2021
Hyper-parameter the model does not have any paeer-paramterss et al. (2019), which is based on the mean μl and standard deviation a of the Gaussians. For tailoring. used a x 5 grid to initialize the inner optimization find the point of highest probability within the e-ball. Using a single starting point did not work as gradient descent found a local optima. Using more (10 10) did not improve results further, while increasing compute. We also experimented between doing a weighted average of the predictions by their energy to compute the tailoring loss or picking the element with the biggest energy. Results did not seem to differ much (likely because likelihood distributions are very peaked), so we picked the simplest option of imitating the element of highest probability. Doing single tailoring step already worked well (we tried step sizes of 10 1, 10, 30 with 10 working best), so we kept that for simplicity and faster predictions.
the model from Ilaas
Regarding compute, this experiment can be generated few minutes using single GTX 2080
10
5
0
-5
robust classifier for ε 0,
1,2,
-10
-20
-15
-10
-5
5
10
15
20
Figure 5: Decision boundary our model at multiple levels robustness on an example from llyas ct al. (2019).
29

---

Under review as a conference paper at ICLR 2021
Method
0.0
0.25
0.5
0.75
1.0
1.25
1.5
1.75
2.00
2.25
ACR
RandSmooth
0.75
0.60
0,43
0.26
0.00
0,00
0.00
0,00
0.00
0.00
0,416
0.25
Meta-tailored
0.80
0.66
0.48
0.29
0.00
0.00
0.00
0.00
0.00
0.00
0.452
RandSmooth
0.65
0.54
0.41
0.32
0.23
0.15
0.09
0.04
0.00
0.00
0.491
0.50
Meta-tailored
0.68
0.57
0.45
0.33
0.23
0.15
0.08
0.04
0.00
0.00
0.542
RandSmooth
0.47
0.39
0.34
0.28
0.21
0.17
0.14
0.08
0.05
0.03
0.458
1.00
Meta-tailored 0.50
0.43
0.36
0.30
0.24
0.19
0.14
0.10
0.07
0.05
0.546
Figure 6: Percentage of points with certificate above different radii, and average certified radius (ACR) for on the CIFAR-10 10 dataset. Meta-tailoring improves the Average Certification Radius by 8.6%, 10.4%, 19.2% respectively. Results for al. (2019) are taken from Zhai et al. (2020)
because they add more measures than the original work, with similar results.
Method
0.0
0.25
0.5
0.75
1.0
1.25
1.5
1.75
2.00
2.25
ACR
RandSmooth
0.75
0.60
0.43
0.26
0.00
0.00
0.00
0.00
0.00
0.00
0.416
Salman
0.74
0.67
0.57
0.47
0.00
0,00
0.00
0.00
0.00
0.00
0,538
0.25
MACER
0.81
0.71
0.59
0.43
0.00
0.00
0.00
0.00
0.00
0.00
0.556
Meta-tailored
0.80
0.66
0.48
0.29
0.00
0.00
0.00
0.00
0.00
0.00
0.452
RandSmooth
0.65
0.54
0.41
0.32
0.23
0.15
0.09
0.04
0.00
0.00
0.491
Salman
0.50
0.46
0.44
0.40
0.38
0.33
0.29
0.23
0.00
0.00
0.709
0.50
MACER
0.66
0.60
0.53
0.46
0.38
0,29
0.19
0.12
0.00
0.00
0.726
Meta-tailored
0.68
0.57
0.45
0.33
0.23
0.15
0.08
0.04
0.00
0.00
0.542
RandSmooth
0.47
0.39
0.34
0.28
0.21
0.17
0.14
0.08
0.05
0.03
0.458
Salman
0.45
0.41
0.38
0.35
0.32
0.28
0.25
0.22
0.19
0.17
0.787
1.00
MACER
0.45
0.41
0.38
0.35
0.32
0.29
0.25
0.22
0.18
0.16
0.792
Meta-tailored
0.50
0.43
0.36
0.30
0.24
0.19
0.14
0.10
0.07
0.05
0.546
Percentage of points with certificate above different radii, and average certified radius (ACR) for on the CIFAR-10 100 dataset, comparing with SOA methods. In contrast to pretty competitive results in ImageNet, meta-tailoring improves randomized smoothing, but not enough to reach SOA. It is worth noting that the SOA algorithms could also likely be improved via meta-tailoring.
Figure 7:
H EXPERIMENTAL DETAILS ADVERSARIAL EXPERIMENTS
Results for CIFAR- experiments can be found table 6 state-of-the-arr methods can be found in
7 results for ImageNet comparing to
to try remain close possible to the experiments from Cohen et al. (209)) 9). In particular, we tried different added noises v E [0.05, 0.1, 0.2] and tailoring inner steps A [1 10 10-1 1100] for 0.5. To minimize compute, we tried these settings by tailoring (not meta-tailoring) the original model and seeing its effects on the smoothness and stability of optimization, choosing v 0.1, À 0.1 (the fact that they' re the same coincidence). We chose to only do single tailoring step to reduce the computational burden, since robustness certification is very expensive, each example requires 100k evaluations (see below). For simplicity and avoid excessive tuning, we chose the hyper-parameters for = 0.5 and copied them for 0.25 and 1. As mentioned in the main text, a 1 required initializing our model with that Cohen et al. (0119) (training wasn't 't stable otherwise), which is easy to do using CNGRAD.
Haper-parameters and other details of the experiments there are just three tweak for these experiments,
In terms of implementation, we use the codebase Cohen et al. httttt::/ttttttttttt /om/ locus lab/ smoothing) extensively, modifying it only in few places, most notably in the architecture to include tailoring in its forward method. is also worth noting that we had to deactivate their disabling of gradients during certification, because tailoring requires gradients. We chose to use the first-order version of CNGRAD which made it much easier to keep our implementation very close to the original. It is likely that doing more tailoring steps would result in better performance.
30

---

Under review as a conference paper at ICLR 2021
Method
2.5
3.0
ACR
RandSmooth
0.67
0.49
0.00
0.00
0.00
0.00
0.00
0,470
Salman
0.65
0.56
0.00
0.00
0.00
0.00
0.00
0.528
0.25
MACER
0.68
0.57
0.00
0.00
0.00
0.00
0.00
0.544
Meta-tailored RS
0.72
0.55
0.00
0.00
0.00
0.00
0.00
0.494
RandSmooth
0.57
0.46
0.37
0.29
0.00
0.00
0.00
0.720
Salman
0.54
0.49
0.43
0.37
0.00
0.00
0.00
0.815
0.50
MACER
0.64
0,53
0,43
0.31
0.00
0.00
0.00
0.831
Meta-tailored RS
0.66
0.54
0.42
0.31
0.00
0.00
0.00
0.819
RandSmooth
0.44
0.38
0.33
0.26
0.19
0.15
0.12
0.863
Salman
0,40
0.38
0.33
0.30
0.27
0.25
0.20
1.003
1.00
MACER
0.48
0.37
0.34
0.30
0.25
0.18
0.14
1.008
Meta-tailored RS
0.52
0.45
0.36
0.31
0.24
0.20
0.15
1.032
Figure 8: Percentage of points with certificate above different radii, and average certified radius (ACR) for the ImageNet dataset, including other SOA methods. Randomized smoothing with meta-tailoring are very competitive with other SOA methods, including having the biggest ACR for 1
We note that other works focused on adversarial examples, such as Zhai et al. (2020); Salman et al. (2019), improve on Cohen et al. (2019) by bigger margins. However, tailoring and meta-tailoring can also improve broad range of algorithms in applications outside of adversarial examples. Moreover, they could also improve these new algorithms further, as these algorithms can also tailored and meta-tailored.
Compute requirements For the CIFAR-10 experiments building on Cohen et al. (2019) each training the meta-tailored method was done in single GTX 2080 Ti for hours. Certification was much more expensive (10k examples with 100k predictions each for a total of 10" predictions), Since certifications of different images can be done parallel, we used cluster consisting of 8 GTX 2080 Ti, 16 Tesla V-100, and 40 K80s (which are about 5 times slower), during 36 hours.
For the ImageNet experiments, fine-tuned the original models for epochs; each took 18 hours on Tesla V-100. We then used 30 Tesla V-100 for 20 hours for certification.
31

---

Under review as a conference paper at ICLR 2021
I EXPERIMENTAL DETAILS OF TAILORING FOR MODEL-BASED REINFORCEMENT LEARNING
We re-implemented PDDM (Nggbaandi al. 2020 in Pytorch (Paszke al. 2019). We keep all of its hyper-parameters the same between the base inductive version and the meta-tailoring versions. We note that, the process, we found out that some of the nyper-parametee for the dclaw environemnt were wrongly set in their github and instead were correctly specified in their appendix. We could not reproduce the other complex environment, baoding, and we are eurrently in conversations with them trying to reproduce it. This (current) lack of reproducibility has been confirmed by the authors as well as Github issues raised by other researchers.
There are three main papee-pasameters we tried changing: the learning rate of the likelihood model (we tried 10-2, 10-3 10-1 and settled for 10-3), the number of Gaussians (we set it to without parameter search), and the tailoring learning rate: tried 10 2, 10-3, 10 1005 and show 10-3 as large tailoring and 14-4 as small. 10-2 was too unstable to obtain proper results.
A few implementation tricks were required to make training stable:
We clamped the affine parameters after tailoring between -10 and 10.
We set a minimum predicted standard deviation each Gaussian in the mixture of Gaussians to 10-2, to avoid degenerate Gaussians.
32


Reviews:
[{"review_id": "tzXCvboDF-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes tailoring and meta-tailoring, learning processes that fine-tune the model parameters during test-time using unsupervised objectives. This allows for designing and integrating powerful inductive biases into the model, leading to an improved test-time performance in two example tasks.\n\nStrengths:\n* The proposed approaches are well-motivated, and the paper is written clearly.\n* CNGrad provides an elegant solution to implement tailoring efficiently.\n* The experimental results show the benefits of the proposed approach convincingly.\n\nWeaknesses:\n* While CNGrad provides an efficient implementation of tailoring, it would be interesting to see how it compares to its “inefficient counterpart”, in which no additional parameters are introduced, but the parameters w are optimized for each individual sample using the supervised and the tailoring objective simultaneously.\n\n* It would be nice to include the inductively trained model as a baseline to the experiment in section 5.2. This could highlight more clearly the benefit of tailoring when applied to adversarial examples.\n\n* I find the statement that the paper showed “the applicability of tailoring on three domains” slightly misleading. The paper shows its applicability experimentally in two domains, and shows theoretical results for the third. Additionally, building on these theoretical results, it would be interesting to see how the contrastive loss might be used for tailoring in an experimental setting. \n\nQuestions:\n* How is the element-wise normalization in CNGrad performed at test-time? Do you keep a running average of the batch-statistics as is done in batch normalization?\n\n* Which loss was used for the results in Table 1? \n\n* What are the run-time implications of tailoring, both at train and test time?\n\nAdditional Comments:\n* Fig 1: Try to avoid using red and green as distinguishing colors to improve the paper’s accessibility.\n\n* It might be nice to add a paragraph on inductive learning to the intro.\n\n\nDisclaimer: I did not check the provided proofs in detail.", "extra_scores": {}, "extra_text": {}}, {"review_id": "epS5O7i3Zeq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==========\nSummary \n\nThis paper proposes a meta-tailoring method where the auxiliary tailor loss is used to adapt the model parameters at test time.  The paper also provides a theoretical analysis of the advantages of the proposed tailoring/meta-tailoring.  The proposed method is evaluated on various application tasks and obtain significant improvements over baselines. \n\n==========\nPros\n\n1. Test-time training is getting increasing attention recently, however, the theoretical analysis of the advantages of test time training is still behind. This paper provides a reasonable theoretical analysis of tailoring and meta-tailoring. \n2. The proposed CNGrad approach is simple but effective, which provides strong empirical improvements on various tasks. \n3. The paper is well written and clearly organized. \n\n===========\nConcerns/confusions\n\n1.  what is the formulation of the affine parameters?  Is there any study about adapting the entire model versus the affine parameters only?\n\n2. Missing contrastive learning experiments on images?  In the last sentence of the abstract, it says \"..., and using contrastive losses on the query image to improve generalization\". However, I didn't find a discussion on it in the main paper.  Given that previous work (e.g., Sun et al.) has done experiments on image classification tasks, it would be more compelling to provide a comparison in the same setting. \n\n\n======\n\nIn general, this paper proposes an interesting approach and insightful analysis of the proposed method.  I'm willing to upgrade my score with comparable experiments with prior work on the image classification task. \n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y921m80PMc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents tailoring and meta-tailoring to eliminate the generalization gap by optimizing at test time. This paper combines the idea of test time optimization and meta-learning and provides some theoretical analysis of the proposed methods. Experiments show the proposed method is effective in solving the machine learning problem and improving model robustness.\n\nStrengths:\n- The idea of using meta-learning to improve tailoring is interesting. Some theoretical justification for the proposed method is provided.\n- The explanation of different learning settings is insightful.\n\nWeaknesses:\n- The proposed tailoring method is very close to Test Time Training (Sun et al. , 2019). However, this paper fails to clearly show the differences between tailoring and TTT. There is also no theoretical or empirical evidence to show that tailoring/meta-tailoring is better than TTT.\n- The experiments in this paper are quite weak. The descriptions of the experiment settings are unclear. There is no direct comparison with closely related methods like TTT. The results of widely used benchmarks are not provided. \n\nOverall, although I think the idea is interesting, the proposed method is not well verified, which makes the effectiveness of the method is still unclear. Some closely related work is not sufficiently discussed. Therefore, I lean to recommend rejection for this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1AO51IDpSR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a learning method called tailoring, inspired\nby transductive learning, which works\nby fine tuning a model on an unsupervised loss given a test time example\ninput. The benefit of this approach is that arbitrary constraints\non predictions can be imposed without a suffering from a generalization gap\nif the constraints were used as an auxiliary objective during training.\nThe authors also propose meta-tailor, which includes the tailoring process\nas an inner optimization loop during training. In a sense, meta-tailoring\nis like meta-learning but with  each training example considered a separate\ntask.\n\nThe authors provided extensive theoretical justification for tailoring\nand meta tailoring, as well as an efficient means to implement it through\nconditional normalization parameters.\nThey then perform two experiments, one where (meta-) tailoring\nis used to impose physical constraints on a neural network physics simulator\nand another where meta-tailoring is used to improve the robustness of an\nimage classifier to adversarial attacks.\n\n\nThe idea of (meta-)tailoring is quite intriguing and I could see it\napplied to scenarios in structured prediction or as an alternative to\nposterior regularization. However, this paper was very theory heavy and I must admit,\nI struggled to understand the import or necessity of the provided theorems.\nOne of the claims of this paper is that they demonstrate that improving\nprediction quality with contrastive learning, but this is only done\ntheoretically. As this claim is included in the list of experimental results,\nI find that somewhat disingenuous. I would have much preferred a \ncontrastive learning experiment.\n\nI lean to reject this paper.\n\n\n\nMiscellaneous note:\n\nIn the last paragraph of page 4, it is mentioned that parts of \ndefinition 1 and theorem 1 are in bold green, but I see no bold green in\nthis copy of the paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes new learning algorithms that fine-tune ("tailor") a model at test-time using unsupervised objectives. This formulation allows for introducing an inductive bias into the model that might improve generalization on unseen data. The proposed algorithm is demonstrated on two example tasks.

The reviewers like the topic and also find the proposed approach to be interesting. However, they are unconvinced by the current empirical evaluation of the method. Additional experimental evaluation could improve our understanding of the proposed method and help contrast it to previously proposed techniques. Given these reviews I recommend rejecting the paper at this time.
Decision: Reject