Meta-learning to Improve Pre-training
Authors: Aniruddh Raghu
Jonathan Peter Lorraine
Simon Kornblith
Matthew B.A. McDermott
David Duvenaud
Venue: NeurIPS 2021 Poster (2021)
Keywords: metalearning
hyperparameter optimization
pre-training

Abstract: Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains.  PT can incorporate various design choices such as task and data reweighting strategies, augmentation policies, and noise models, all of which can significantly impact the quality of representations learned. The hyperparameters introduced by these strategies therefore must be tuned appropriately. However, setting the values of these hyperparameters is challenging. Most existing methods either struggle to scale to high dimensions, are too slow and memory-intensive, or cannot be directly applied to the two-stage PT and FT learning process. In this work, we propose an efficient, gradient-based algorithm to meta-learn PT hyperparameters. We formalize the PT hyperparameter optimization problem and propose a novel method to obtain PT hyperparameter gradients by combining implicit differentiation and backpropagation through unrolled optimization. We demonstrate that our method improves predictive performance on two real-world domains. First, we optimize high-dimensional task weighting hyperparameters for multitask pre-training on protein-protein interaction graphs and improve AUROC by up to 3.9%. Second, we optimize a data augmentation neural network for self-supervised PT with SimCLR on electrocardiography data and improve AUROC by up to 1.9%.

Meta-Learning to Improve Pre- Pre-Training
Aniruddh Raghu Massachusetts Institute Technology araghu@mit. edu
Jonathan Lorraine
Simon Kornblith Google Research
University of Toronto
Matthew McDermott Massachusetts Institute of Technology
David Duvenaud
Google Research & University of Toronto
Abstract
Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many do- mains. PT can incorporate various design choices such as task and data reweighting strategies, augmentation policies, and noise models, all of which significantly impact the quality of representations learned. The hyperparameters introduced by these strategies therefore must be tuned appropriately. However, setting the values of these hyperparameters is challenging. Most existing methods either struggle to scale high dimensions, are slow and memory-intensive, applied to the two-stage PT and FT learning process. In this work, we propose an efficient, gradient-based algorithm meta-learn PT hyperparameters. We for- malize the PT hyperparameter optimization problem and propose novel method obtain PT hyperparameter gradients by combining implicit differentiation and backpropagation through unrolled optimization. We demonstrate that our method improves predictive performance on two real-world domains. First, we optimize nigh-dimensional task weighting hyperparameters for multitask pre-training on protein-protein interaction graphs and improve AUROC by up to 3.9 3.9%. Second, we optimize data augmentation neural network for self-supervised PT with SimCLR on electrocardiography data and improve AUROC by up to 1.9%.
cannot be directly
Introduction
A popular and important learning paradigm for neural networks is pre-training (PT) followed by fine-
tuning (FT), an approach commonly used transfer learning [13
59 19
27
52
37
74 35
28
and semi-supervised learning 9 24 This paradigm has led to performance improvements in
many domains, including computer vision [13
59
19 37
35
natural language processing
27
(2 11 40 344, graph structured prediction
22 and clinical machine learning [45] 46 48
and is especially helpful in settings where downstream tasks have limited training data.
The PT & FT paradigm introduces high-dimensional, complex PT hyperparameters, such as pa- rameterized data augmentation policies used in contrastive representation learning [8] 2 or the use of task, class, or instance weighting variables in multi- task PT to avoid negative transfer [70]. These hyperparameters can significantly affect the quality of pre-trained models (8], and thus finding techniques to set their values optimally is an important area of research.
Choosing optimal PT hyperparameter values is challenging, and existing methods do work well. Simple approaches such as random or grid search are inefficient since evaluating hyperparameter setting requires performing the full, two-stage PT & FT optimization, which may be prohibitively computationally expensive. Gradient-free approaches, such as Bayesian optimization or evolutionary algorithms
33
also limited in how well they scale this setting. Gradient-based
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

can be used online to jointly learn hyperparameters and model parameters and can scale to millions of hyperparameters [42) but typically deal with a standard single-stage learning problem (e.g., normal supervised learning) and are therefore not directly applicable to the two-stage PT FT learning problem.
approaches 44 41 43 22
In this work, we address this gap and propose a method for high-dimensional PT hyperparameter op- timization. We first formalize variant of the PT & FT paradigm. which we call meta-parameterizee pre-training (Figure D. where pheaa-parsesters refer to arbitrary PT hyperparameters or parameteriz- able architectural choices that can be optimized improve the learned representations P We outline a meta-learning problem characterizing the optimal meta- peraa-ersssstess propose a gradient- based method learn meta-pant paramearssssrss. Our contributions are:
We formalize metta-paramtterize pre-training, a variant of the pre-training and fine-tuning (PT' & FT) paradigm where PT' is augmented to incorporate meta-parameters: arbitrary structures that can be optimized to improve learned representations.
We propose a scalable gradient-basee algorithm learn meta- pattappaameterss using a novel method to obtain meta-parameter gradients through the two-stage PT & FT process. Our gradient estimator composes constant-memory implicit differentiation approximation for the longer PT stage and exact backpropagation through training for the shorter FT stage.
We show that our algorithm recovers optimal meta-parametert in toy experiments on synthetic data. In two real-world experimental domains, we demonstrate our algorithm improves performance. Firstly, on multitask PT benchmark over biological graph-structured data (28). using our method to optimize peramparsmesees representing task weights improves performance by up to 3.9% AUROC. Secondly, for semi-supervised learning using SimCLR g over eleetrocardiogaaphy data, using our algorithm to optimize meta- parameters representing the weights of a data augmentation neural network improves performance by up to 1.9% AUROC.
2 Problem Setup and Preliminaries
In this section, we define the meta-parameteriieed pre-training meta-learning problem, and compare to traditional fine-tuning and pre-training. A full glossary of notation is in Appendix B Table 3
Notation. Let the subscript be placeholder for either PT (pre-training) or FT (fine-tuning), X' C I be our input domain, V. and )d. be the true and predicted output spaces for some model respectively, and θ, V., & be spaces of parameters for models. We will use f. : X: (θ, v.. - V. to refer to a parametric model, with the semicolon separating the input space from the parameter spaces. We then define f. ((head) (feat), such that (feat) (:: E +) is a feature extractor that is transferable across learning stages (e.g., pre-training to fine-tuning), and f. (head) (: E v..)) is a stage-specific head that is not transferable. Given data distribution X. D., parametric model f.. and loss function L. ).. V + R. will also define for convenience a corresponding expected loss L. : θ V. + IR via (O, D.) Ep, [S. 0,vb)...)..))) v.))., V=]]- We also adopt the convention that the output of the argmin operator is any arbitrary minimum, rather than the set of possible minima, to avoid complications in notation.
2.1 Problem Formulation
Supervised Learning (Fig. A). In a fully-supervise setting (our fine-tuning domain), we are given a data distribution Dyr, model f, and loss Ler. Using a learning algorithm Algrt (e.g., SGD) that takes as input initial parameters o w our goal to approximate the Lfr-optimal parameters: OFr: vir = Algrr (0 + Prr) ss argminggo. DeVT LIT (0, bb; Drr)
Pre-training (Fig. (B). For tasks where data is scarce, can additionally incorporate pre- training step and approximate the optimal initial parameters for FT (i.e., the final pre-trained weights are used as initialization weights of the FT stage), again via an optimization algorithm Alger: (0) Drr) R argmingce T.r((Agg ( (b Drr); Drr).
ppT = AlgpT ( +
Wee use the term meta- veta--parameter since these structures do not directly affect inference of the final model after FT, but instead inform the process of learning this model (by modulating the PT process).
Note that we discard the PT head uhin here only the PT feature extractor dög is transferred.

---

Meta-Paramettrized Pre-troining
Fine-tuning
Data Distribution
Fine-tuning (A)
Meta-Parameters dana/losk rewnighting augrennattion polties, nose modes,
Meta- Pre-trariig aigorithin
Meta Fine-tur Dataset
Somple.
Sample
Standard Pre-trolming (B)
Fine-tune
Datase:
Model
Pre-training alporithm
Fine-tuning algorithm
Pre traitriiedd Model
Fine-tuned Model
Pre-train
Dataset
Figure (I) Meta-Parameterized Pre- Training. A paradigm where meta- parameters rich, potentially high dimensional structures that generalize PT hyperparameters are incorporated in PT to improve the learned representations. Meta-paramn parameerssssrs optimized in meta- PT phase, using data from FT task(s) in meta-FT dataset. The FT and meta-FT datasets (potentially overlapping) samples from the FT data distribution.
Meta-Parameteriize PT (Fig. TC). In Meta- Parametrraeee PT, we recognize that, in addition to taking input the PT parameters 0, AlgeT is itself parameterized by set of meta-parameters Φ E Φ: arbitrary, potentially high dimensional quantities that inform the structure of the algorithm directly, These could represent weighting strategies, data augmentation policies, sampling processes. The optimal meta-parameten (op)) are the solution the following meta-P" optimization problem:
(opt) argmin LFT (AlgrT (AlgeT (off) d
(0) Dpr. 6) w/ DFI) :PF))
ΦE#
2.2 Example: Multitask Meta- Patametarmtter Pre- Pre-Training
To make our notation concrete, here instantiate our setup for a multitask pre-training problem.
Problem:
Suppose we have a multitask classification dataset, (X' y) such that y =1 .... YK consists of labels for K distinet tasks. Of this full set of tasks, we are in- terested only in a subset of M tasks, S = {41...., L.3 C fl,....,K? K).
Supervised FT: Under supervised FT alone, we can directly average a cross-entropy loss LCE over only the tasks in 5, Lrr(ū.
PT: we assume that (ū. y) = nt Σ (cn()(t)), ((ggg//),y(,, (y())) and then solve this problem via SGD. all tasks: CPT(1 3) = 1
a random subset of the full set of tasks, we can introduce PT stage over K Σ (CE(g(), ()), followed by FT on S alone. As S is a random subset, leveraging all tasks for PT is well motivated and may improve performance. Meta-Parameteer
PT: In the case where T is not a random subset, the PT strategy described above is longer well-motivated. However, using meta- notaa-arametterized PT, we can still effectively pre-train by introducing the meta parameterssss that weight the tasks & = (Ø]
Φx] and modulate the loss function LpT: LpT(g. y: Φ) = ΣX g,CoE(j y²). With optimal meta meeaa-arameterss & ("P)) the PT stage will leverage only that subset of tasks that best informs the final FT performance. This setting mirrors real-world experiment in Section 5
3 Methods: Optimizing Meta-Parameters for Two-Stage Training
We now introduce our gradient- baseed algorithm to optimize meta- peeta-arammters... We first describe how to efficiently approximate meta-pant perampar gradients through the two-stage PT and FT optimization. We then present our algorithm, and outline practical considerations when using it.
3.1 Efficient Computation Meta-Parameter Gradients
We begin by defining:
Parameter Gpr
Φ0 (0)
appp (1 (0)) = LpT
Alger A Alger (0 (0 up Drr. Φ), (0: Drr); DFT).
(1)
Parameters Opr.v/er
so that &f argmin g()).
3

---

We also define two best-response values:
(p) Alger (0p: (0) DpT. $), iT (T) = (Og FTF Drr).
We do not explicitly include the dependence of the best responses the initialization values for notational convenience.
With these defined, we now consider the desired gradient term,
3g Under our definitions, the direct
partial derivatives
ØLn and @Algm are zero, so T σΦ
8g
reduces to simple expression of the chain rule:
06
FT Best Response Jacobian
0g 0Φ
Ô [OPTs
AlgFT r0p (6)
b Alggt dΦ
dpr (d)
Loss Gradicat
PT Best Response Jacohian
The FT Loss Gradient term on the RHS of 2 is easily computed using backpropagation. Computing the other two terms is more involved, and we detail each below, beginning with the PT best response Jacobian. The full algorithm with both gradient estimation terms is provided in Algorithm [
PT Best Response Jacobian
aAlger Using recent work in hyperparameter optimization with implicit differentiation 42]. we re-express this term using the implicit function theorem (IFT). If we assume that OpT AlgeT 0/- Ppr, Φ is a good approximation of argmingge LpT (0; Dpr, Φ) (i.e., the PT model converges to Opr-optimal parameters), then under certain smoothness and regularity assumptions on the PT parameters and peta-paramettes, the IFT allows us to re-express
@Alger as: 0p
aAlgPT dΦ
8 Lore
(3)
36 88
00pr 0Φ T
(.)))
which is the product of the inverse Hessian and a matrix of mixed partial derivatives. Following [42 the inverse be efficiently approximated using a truncated Neumann series.
FT Best Response Jacobian aAlgrr First, note that without additional constraints Alger the 00p
FT best response Jacobian may be zero. This is because LeT has functional dependence the variable Opr and, if we assume the convergence point f stable (as we did for the PT best response Jacobian), this implies that the gradient of OFr with respect to OPr would be zero. To enable effective learning, we must therefore either (1) impose restrictions on AlgFT to ensure there is dependence between the initialization point and the final loss value (e.g.. proximal regularization (55)) (2)
leverage methods that do not differentiate through Alger through convergence, as at non-converged points we will still observe nonzero LFr-gradients (29 5.. Given that the FT phase often involves shorter optimization horizons than PT, we take approach 2 here, and iteratively update OFr for K
steps. We first initialize the FT head (0 and then compute:
= copy(0 (FTT)
(init with PT solution, implicitly performing stop gradient)
FT
0 o FT
FT
= 0 1)
wb
(k-1)] FT
TFT a [OFT
1Lr
Verll e(k-1) (1 (k--1)
K
(4)
0 pT. vi
e (K)
(K)
and compute the gradient
8Almm 30
by differentiating through this optimization
We can also choose to freeze the feature extractor parameters Orr and update only the head parameters 1'FT during truncated FT, and use this obtain meta- perameereeter gradients. This resembles linear evaluation, where a linear classifier is trained on top of fixed, pre-trained feature extractors [50]
Together, these two approximations allow for efficient computation of meta--arameter gradients.
While Equation 4 uses standard gradient descent, we could other differentiable optimizers (e.g., Adam).

---

Algorithm 1 Gradient-basse algorithm to learn meta- parameteas.ssss. Notation defined in Appendix B 3 Vector-Jacobial products (VJPs) can be efficiently computed by standard autodifferentiation.
1: Initialize PT parameters 0pT (imii) .FFT and meta-parameters o (0)
2: for n = 1,..... N iterations do
3: Initialize 0pT = 0 (iinit)
and u/P PP
finir)
4: for p ,....
: P PT iterations do
5: 0 (p) e (e(m-1) 1)
w1o-11 | TP prr PP
8L.
1) 111-111)
6: end for
7: Initialize FT encoder with PT solution: / copy((p7) (P)) (
8: Approximate frr: कir using Eq.
9: Compute 91 o(OTT:
FT
Compute VJP 92
91
8AlEE 00pt
using the unrolled learning step from line
O (
.
Approximate VJP
0g 0e
(in-1)
92
BAlEe
d
1(n-1)
using the IFT (Eq. 3).
12:
Φ
(m-1)
TN
0g
Update PT initialization by setting: ((iit) ofP²) and (b(iit)) u/pp
14: end for
3.2 Our Algorithm and Practical Considerations
By leveraging the above approximations, we obtain Algorithm I to optimize meta-parameters &i online during PT & FT of the base model. Note that Alger is explicitly written out as sequence of gradient updates (lines 4-6 in Algorithm T We now discuss practical considerations when using this algorithm, with further details given in Appendix C
(1) Access to DrT and generalizing to new FT tasks: Solving the meta-PT problem requires avail- ability of: the model fa. the PT data Dpr, and the FT' data Dpr- In this work, we assume availability of the model and PT dataset, but since assuming access to the complete FT dataset at meta-PT time is more restrictive, we study two scenarios: Full FT' Access, where all FT data that we expect to encounter is available meta-PT time, and Partial FT Access, where the FT data available at meta-PT time is only a sample from distribution of FT data that we may encounter later.
Full FT Access occurs in settings like semi-supervised learning, where we are given a large unlabelled PT dataset and a small labelled FT dataset and our goal is to achieve the best possible performance g 9..
by leveraging these two fixed datasets [68 73 25
24
Partial FT Access occurs when our goal is learn transferable representations: at meta-PT time, we might have limited knowledge of FT tasks or data. In evaluating this scenario, we examine generalizability to new FT tasks, given only small amounts of FT data/task availability at meta-PT time, demonstrating that even very limited FT access can be sufficient for effective meta-parameter optimization [ 56 28).
(2) Prr splits: In practice, we have access to finite datasets and use minibatches, rather than data- generating processes. Following standard convention, we split DFr into two subsets for meta-learning learning: D(0 and D(val) (independent of any held-out DFr testing split), and defne the FT data available at
meta-PT time as D = DMf U D
We use D for the computation of @Alser
and
aAlge de
and Dival) for the computation of
Blar 2
in Algorithm
1)
(3) Online updates: Given that PT phases often involve long optimization horizons, for computa- tional efficiency, we update Per and dhpr online rather than re-initializing them at every meta-iteration (see Algorithm I. FT phases are often shorter could in theory re-initialize 1/PT at each
5

---

meta-iteration as is presented in Algorithm|| [ However. it is more computationally efficient to also optimize this online, and we follow this approach in our experiments. A description of the algorithm with these details in Appendix C
Note that prior work [67 has suggested that online optimization of certain hyperparameters (e.g.. learning rates) using short horizons may yield suboptimal solutions, We comment on this in Ap- pendix study this effect for our algorithm in synthetic experiments in Appendix E and in real-world experiments on self-supervised learning in Appendix G revealing it is not a significant concern.
(4) Computational tractability: Our method can scale to large encoder models and high- dimensional meta- parametra,,sers, despite the complexity of the two-stage PT & FT process. This is because: (i) pertapassseserss are optimized jointly with the base model parameters; (ii) using the IFT to obtain gradients has similar time and memory complexity to one iteration of training 42] : (iii) the FT best response Jacobian can be approximated efficiently using small number of unrolled optimization steps K, and by only unrolling the FT head of the network. In our real-world experiments (Sections 5 and (6. meta-parameterrzed PT has less than twice the time cost of standard PT. Further details on time and memory cost are provided in Appendices Faadd
(5) Setting optimizer parameters: Learning rates and momentum values can impact the efficacy of the algorithm. A discussion on how to set them in practice is provided in Appendix D
4 Synthetic Experiments
We validate that our algorithm recovers optimal low and high dimensional mnta--arameteer in two synthetic MNIST experiments with Full FT Access. Further details and results are provided in Appendix E including a study of how our method performs comparably to differentiating exactly through the entire learning process of PT & FT, without approximations.
First, we optimize low dimensional meta- parameters characterizing a data augmentation scheme. We tune a 1-D meta-parameter representing the mean of a Normal distribution (V, 12) from which we sample rotation augmentations to apply to PT images. FT images undergo rotations from a Normal distribution N ((UFT, 12) with μFFT 90%: we therefore expect that should converge to near μUFT. Using Algorithm to optimize ø we find that the mean error in the optimized meta-parameter over 10 different initializations is small: 7.2 + 1.5%, indicating efficacy of the algorithm.
Next, we consider learning high dimensional meta- parameters that characterize a PT per-example weighting scheme. The PT dataset contains some examples that have noisy labels, and FT examples all have clean labels. The veta-paaraeetes are the parameters of a neural network that assigns importance weights each PT example, which is used weight the loss on that example during PT. We use Algorithm|] [ agaan to optimize o over 10 random initializations, finding the ratio of assigned importance weights between clean label PT examples and noisy label PT examples is greater than 102. This is expected since the noisy label classes may worsen the quality of the PT model and so should be down-weighted.
5 Meta-Parameterized Multitask Pre-Training for Graph Neural Networks
We consider optimizing PT task weights for a multitask PT FT problem of predicting the presence of protein functions (multitask binary classification) given graph-structured biological data as input. We have two experimental goals: first, in the Full FT Access setting, where methods are given access to all FT data at PT time, we evaluate whether optimizing task weighting meta-parameterss can improve predictive performance on the FT tasks. Second, motivated by how in typical transfer learning problems, new tasks labels not available at PT time may become available at FT time, study the Partial FT Access setting, investigating how our method performs when only sees limited FT tasks at PT time. In both settings, our method outperforms baselines.
5.1 Problem Setup
Dataset and Task, We consider the transfer learning benchmark introduced in 28 where the pre- diction problem both PT and FT multitask binary classification: predicting the presence/absence of specific protein functions (y) given a Protein Protein Interaction (PPI) network as input (rep-
Protein-Protein

---

resented as a graph æ).. The PT dataset has pairs Dpr = {( _))__r where E {0, 1 |55000 characterizes the presence/absence of 5000 particular protein functions. The FT dataset has pairs where E {0. 11:40 now characterizes the presence/absence of 40 different protein functions. Further dataset details Appendix
|Prll
DFr = v4) Perl Mi))
Meta--aarameterized Multitask PT. To define a meta-parameterized PT scheme, we let meta- parameters & 50000 be weights for the binary PT tasks. Then, we define a PT loss incorporating
the weights: Lpr = 5000 Ei=1 (é:) LCE(frr(2; Opr. ppr), ,eih indexing the tasks, (+) representing the sigmoid function (to ensure non-negativity and clamp the range of the weights), and LCE denoting the binary cross-entropy loss. With this loss defined, we use Algorithm [ (with P 10 PT steps and K = 1 truncated FT steps) to jointly learn d and the feature extractor parameters Opr- For computational efficiency, we only update the FT head when computing the FT best response Jacobian and keep the feature extractor of the model fixed. We use the training and validation splits of the FT dataset PFr proposed by the dataset creators (28] for computing the relevant gradient terms.
Baselines. Motivated by our goals, we compare with the following PT baselines:
No PT: Do not perform PT (i.e., feature extractor parameters are randomly initialized).
Graph Supervised PT: As explored in prior work on this domain [28]. perform multitask super- vised PT with Dpr- This corresponds setting all task weights to 1: Φ; 1, 1..... 5000.
CoTrain: A common baseline that makes use of the FT data available during PT [70] (like meta- parameterized PT). We PT model with 5000 + 40 outputs (covering the space of PT and FT labels) jointly on both Dpr and Dpr. We do so by alternating gradient updates on batches sampled from each dataset in turn. Further details are in Appendix [F]
CoTrain + PCGrad: An extension of CoTrain, where we leverage the method PCGrad 72] to perform gradient projection and prevent destructive gradient interference between updates from Dpr and Drr. Further details and variants we tried are in Appendix F
Experimental Details. We use standardized setup to facilitate comparisons. Following [28 all methods use the Graph Isomorphism Network architecture [69]. undergo PT for 100 epochs, and FT for 50 epochs, over 5 random seeds, using early stopping based on validation set performance. During FT, we initialize a new FT network head and either FT the whole network or freeze the PT feature extractor and learn the FT head alone (Linear Evaluation (50) We report results for the strategy that performed best (full results in the appendix). We consider two experimental scenarios: (1) Full FT Access: Provide methods full access to DpT and DFTT at PT time (D(Men)) = Dpr) and evaluate the full set of 40 FT tasks; (2) Partial FT Access: Limit the number of FT tasks seen at PT time, by letting D/Meta) include only 30 of the 40 FT tasks. At FT time, models are fine-tuned on the held-out 10 tasks not in D(Meta) We use a 4-fold approach where we leave out 10 of the 40 FT tasks in turn, and examine performance across these 10 held-out tasks, over the folds.
5.2 Results
Key Findings. By optimizing PT task weights, meta peeta-praaettrizeed multitask PT improves perfor- mance on the FT problem of predicting presencc/absence of protein functions given a protein- prottin interaction graph as input. Performance improvements are also seen when generalizing to new FT tasks (protein functions). unseen at meta-PT -T time.
Table presents quantitative results for the two experimental settings described. For the No PT and Graph Supervised PT baselines, we implement the methods from 228) obtaining improved results (full comparison in Appendix Table In both full and partial FT access settings, meta-parameteriied PT improves significantly on other methods, indicating that optimizing perammarsssserrs can improve predictive performance generally, and be effective even when new, related tasks are considered at evaluation time. Interestingly, we observe that CoTrain and CoTrain PCGrad obtain relatively poor performance compared to other baselines; this could be because the methods overfit to the FT data during PT. Further analysis of this is presented in Appendix F
Further experiments. In Appendix F we study another partial FT access scenario with smaller D(Mosa) setting (MMtta) 0.5 [Prr]. and find that meta- peetapaarameteiieedd PT again outperforms other methods. (Table 7. We also examine another meta-parameter learning baseline, namely a version of CoTrain where we optimize task weights using traditional hyperparameter optimization algorithm 42 jointly with the main model. We find that our method outperforms this baseline also (Table
7

---

Method
AUC (DFT (Meta)
=DF)
AUC (DFt (Meia)
excludes tasks)
No PT
66.6 - 0.7
65.8 + 2.5
Graph Supervised PT
74.7 - 0.1 70.2 03
74.8 + 1.8
CoTrain
69.3 = 1.8
CoTTaai + PCGrad
69,4 0.2
68.1 + 2.3
Meeameearaetteiize PT
78.6 0.1
77.0 + 1.3
Table (1) Parameeeemeer PT improves predictive performance over baselines. Table showing mean AUC and standard error for evaluation settings. When provided all FT data PT time (first results column). meta-parameterized PT significantly improves predictive performance. In more challenging setting when DM excludes FT tasks (10 of the 40 available tasks are held-out), evaluating mcan AUC/standard error across four folds with each set of 10 FT tasks held out in turn, meta parameterized PT again obtains the best performance: is effective even with partial information about the downstream FT tasks.
Analysis of learned structures. In Appendix F we conduct further analysis and study the effect of various PT strategies on the pre-trained representations (FigureB3, finding intuitive patterns of similarity between different methods. We also examine the learned task weights (Figure 4 and examine performance on per-FT task basis with/without meta-parameterizee PT (Figure 5 finding little evidence of negative transfer.
Meta-Parameterized SimCLR for Semi-Supervised Learning with ECGs
We now explore second real-world application of our method: optimizing data augmentation policy for self-supervised PT with SimCLR (8 on elcctrocardiograms (ECGs). SimCLR is a popular self-supervised PT method that leverages data augmentations define contrastive PT objective (details in Appendix G.I). The choice/strength of the augmentations used significantly impacts the effectiveness of the algorithm [8]. In settings where relevant augmentations known (e.g., natural images), SimCLR is readily applicable; however, for ECGs, effective augmentations are less clear, motivating the use of our algorithm to optimize the augmentation pipeline.
We have two experimental goals. Firstly, we examine the typical semi-supervised learning setting of Full FT' Access: we explore whether optimizing the augmentations in SimCLR PT can improve performance on the supervised FT task of detecting pathologies from ECGs, given access all FT data at meta-PT time. Secondly, to study the data efficiency of our method, we consider the Partial FT Access setting and explore performance given access to limited data at meta-PT time, We find that our method improves the performance of SimCLR, and that it is effective even with very limited amounts FT data provided at meta-PT time.
6.1 Problem Setup
Dataset and Task. We construct semi-supervised learning (SSL) problem using PTB-XL 64 20 an open-source dataset of electrocardiogram (ECG) data. Let the model input at both PT and FT time be denoted by æ. which represents 12-lead (or channel) ECG sampled 100 Hz for 10 seconds resulting in a 1000 x 12 signal. Our goal is to pre-train a model fPT on an unlabeled PT dataset
of ECGs Dpr
{x: )=1 [Perl using SimCLR PT [8), and then fine-tune it the labeled FT dataset
Drr = (m, 9:) | where the FT labels y E {0.1) 13 encode whether the signal contains certain features indicative of particular discases/pathologies. Further dataset details in Appendix G
ECG Data Augmentations. To augment each ECG for SimCLR (example in Appendix Figure we apply three transformations in turn (based on prior work in time series augmentation 30 66
Random cropping: randomly selected portion of the signal is zeroed out.
2. Random jittering: IID Gaussian noise is added to the signal.
Random temporal warping: The signal is warped with a random, diffeomorphic temporal to generate temporal displacement field, which is applied to the signal.
transformation. This is formed by sampling from zero mean, fixed variance Gaussian at each temporal location in the signal obtain a velocity field, and then integrating and smoothing (following 4 5)

---

Test AUC at different FT dataset sizes [Pr
FT' dataset size [Drrl
100
250
500
1000
2500
No PT
71.5 - 0.7
76.1 + 0.3
78.7 + 0.3
82.0 + 0.2
84.5 + 0.2
SimCLR
74.6 + 0.4
76.5 + 0.3
79.8 + 0.3
82.2 + 0.3
85.8 + 0.1
Meta-Parameterized SimCLR
76.1 + 0.5
77.8 + 0.4
81.7 - 0.2
84.0 + 0.3
86.7 = 0.1
Table (2) Meta-Parrmeterizee SimCL. obtains improved semi-supervised learning performance. Ta- ble showing mean AUC/standard error over seeds across FT binary classification tasks for baselines and meta-parameterizee SimCLR different sizes of DFr. with Dpr DFr. We observe improvements in performance with meta- paramettrizzed SimCLR, which optimizes the augmentation pipeline.
Meta- aitta-PPrametrizeed SimCLR. To construct meta- eettapaaameterized SimCLR PT scheme, instan- tiate meta- meta-paramen parameters Φ as the weights neural network w(x; Φ) that takes an input signal and outputs the warp strength: the variance of the Gaussian that used obtain the velocity field for temporal warping. This parameterization permits signals to be warped more/less aggressively depending on their individual structure. With this definition, the SimCLR PT loss is directly function of the meta parameerrs,,rss, and we can use Algorithm | (with / = 10 PT steps and K = 1 truncated FT steps) to jointly learn & and the feature extractor parameters PPr. For computational efficiency, we only update the FT head when computing the FT best response Jacobian and keep the feature extractor of the model fixed. We use the training and validation splits of the FT dataset DFr proposed by the dataset creators (64] for computing the relevant gradient terms.
Baselines. Our experimental goals suggest the following PT baselines:
No PT: Do not perform PT (i.e., feature extractor parameters are randomly initialized).
SimCLR: Pre-train a model using SimCLR with the above three augmentations without learning per-example temporal warping strengths.
Experimental Details. We standardize the experimental setup to facilitate comparisons. All methods use ID CNN based on a ResNet-18 [23] architecture. The temporal warping network w(æ; o) is four layer ID CNN. SimCLR PT takes place for 50 epochs for all methods, over three PT seeds. At evaluation time, for all methods, we initialize a new FT network head over the PT network feature extractor and FT the whole network for 200 epochs, over five FT seeds. Validation set AUC is used for early stopping. We consider two experimental settings: (1) Full FT Access, standard SSL: consider different sizes of the labelled FT dataset DFT and make all the FT data available at meta-PT time, our algorithm: SSL. when
D(Mett) DFr: and (2) Partial FT Access, examining data efficiency
only limited FT data is available at meta-PT time: Meta C Dpr. We evaluate performance across the 5 binary classification tasks in both settings. Further details provided in Appendix G
6.2 Results
Key Findings. By optimizing the data augmentation policy used in SimCLR PT, meta-parameterized SimCLR improves performance on the FT problem of detecting pathologies from ECG data. Even a small amount of FT data provided at me time can lead to improved FT performance.
Table shows results for the Full FT Access setting, D = Dpr mean AUC/standard error seeds across the 5 FT binary classification tasks different sizes of DFr. We observe that meta-parameterizee SimCLR improves on other baselines in all settings. Note that while these gains are modest, they are obtained with simple augmentation policies; our method may yield further improvements if applied to policies with more scope to specialize the augmentations.
(MMeta)) C Drr, which is relevant when we Next, we consider the Partial FT Access scenario where D
only have a small amount of FT data at meta-PT time. Fixing [Drrl = 500, we find that with [D as small as 50, we obtain test AUC of 81.3 + 0.5, compared 79.8 0.3 with no optimization of augmentations: this shows that even small [D (M(eea)) appear to sufficient for meta-parameter learning. Further results showing performance curves varying ((Meaa)) are in Appendix G
we study other aspects of our method on this domain, including: (1) Exploring different values of K, the number FT steps differentiated through when obtaining meta- perammar gradients; and (2) Examining petaa-arrmettr learning baseline where
Further experiments. In Appendix G

---

augmentations are optimized for supervised learning, using the method in [42], and then applied to semi semi-supervised learning (to compare how optimizing augmentations for supervised learning compares to optimizing them for semi-supervised learning). We find that our method is not very sensitive to the value of K (provided K > 0), and that it outperforms this additional baseline.
Related Work
Gradient-based hy hyperparameter optimization (HO); Gradient- based HO roughly falls into two camps. The simpler and less scalable approach differentiates through training [12 44 The other approach assumes that optimization reaches a fixed point, and approximates the best-response Neither of these approaches can be straightforwardly applied scalably differentiate through two stages of optimization (PT & FT). Direct differentiation through both stages would be too memory-intensive. Approximating the best-response Jacobian using the IFT in [42 twice is feasible, but requires changing the FT objective to include a proximal term [55]. and tuning two sets of interacting approximations. Instead, we compose constant-memon IFT approximation for the lengthy PT stage with an exact beokprop-through-raeee
Jacobian 7 41 43 42
for the shorter FT stage.
Applications of Nested Optimization: Many prior works frame learning nested optimization, including few-shot learning [6 17 55
21
58
53
75
31,
38 neural network teaching [1] [5]
62 learning data augmentation and reweighting strategies 32
60) 29], and auxiliary task learning [49 51 39. The majority of this work studies nested optimization in the standard one-stage supervised learning paradigm, unlike our setting: the two-stage PT & FT problem. The most closely related works to ours are (70), where PT task weights are learned for a multitask PT problem using electronic health record data, and (71) where masking policy is learned for masked language modelling PT. In contrast to our work, which introduces the more general framing of meta petaa-er optimization, [70] and 71 are focused only on specific instantiations of meta-parameters as task weights and masking policies. The learning algorithms in these works either: differentiate directly through truncated PT & FT [7]] (which may not be scalable to longer PT/llage encoder models), or leverage extensive first-order approximations 770, unlike our more generally applicable approach.
22
57
8 Scope and Limitations
Our gradient-bar based algorithm applies in situations where want to optimize (potentially high- dimensional) PT hyperparameters, or mneta-parameter and have access to a model, PT data, and FT data. We demonstrated that even limited FT data availability can be sufficient to guide meta- parameter learning; however, our method would not apply when no FT data at all is available at meta-PT time, or if the model or PT data were not available. Our algorithm requires meta- parameters to be differentiable, and cannot directly be used to optimize meta-parameters that do not affect the PT optimization landscape (e.g., PT learning rates).
9 Conclusion
In this work, we studied the problem of optimizing nigh-dimensional pre-training (PT) hyperparame- ters, or mela--araaeteers We formalized Meta-Parametericee Pre-Training, variant of standard PT incorporating these meta-parameters, and proposed a gradient-based algorithm to efficiently learn reta-pparametes by approximately differentiating through the two-stage PT & FT learning process. In experiments, we used our algorithm to improve predictive performance on two real-world PT tasks: multitask PT with graph structured data (28). and self-supervised contrastive PT on electrocardiogram signals using SimCLR [8]. Future work could apply our method to learn other potential instantiations of meta-parameters, such as learned auxiliary tasks and noise models.
Societal Impact. Our contribution in this work is methodological, namely new algorithm to optimize digh-dinensional pre-training hyperparameters. We do not expect there to be direct negative societal impacts of this contribution. However, to evaluate our method, we considered an experimental domain using healthcare data. Given the high risk nature of this domain, before use in real- worldw settings, the method should be validated in retrospective and prospective studies. This to detect any failure modes and identify potential harm that may come from deploying it.
10

---

Acknowledgements
This work was supported in part by funds from Quanta Computer, Inc. The authors thank the members of the Clinical and Applied Machine Learning group at MIT and Paul Vicol for helpful feedback.
References
[1] A. Antoniou, H. Edwards, and A. Storkey. How to train your MAML. arXiv preprint arXiv:1810.09500 2018.
[2] S. Azizi, B. Mustafa, F. Ryan, Z. Beaver, J. Freyberg, J. Deaton, A. Loh, A. Karthikesalingam, Kornblith, T. Chen, et al. Big self-supervised models advance medical image classification. arXiv preprint arXiv: arXiv:2101.0522 101.0222, 2021.
[3] Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv: 10, 2019.
[4] G. Balakrishnan, A. Zhao, M. Sabuncu, J. Guttag, and A. V. Dalca. An unsupervised learn- ing model for deformable medical image registration. CVPR: Computer Vision and Pattern Recognition, pages
2018.
[5] G. Balakrishnan, A. Zhao, M. Sabuncu, J. Guttag, and A. V. Dalca. Voxelmorph: A learning framework for deformable medical image registration. IEEE TMI: Transactions Medical Imaging, 38: 05:11888000 2019.
[6] A. Beatson and R. Adams. Efficient optimization of loops and limits with randomized telescoping sums. In International Conference on Machine Learning, pages 534-543. PMLR, 2019.
[7] Y. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8): 1889-1900, 2000.
[8] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.055: 05000, 2020.
[9] T. Chen, S. Kormblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[10] C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithmss for lcarning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-82) 728,898, 2012.
[11] Devlin, M.-W. Chang, K. Lee, and Toutanova. BERT: Pre- Pre-training of deep bidirectional
transformers for language understanding. arXivv preprint 111vv08800004803 2018.
[12] J. Domke. Generic methods for optimization- based modeling. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012. JMLR Pro- ceedings. JMLR.org, 2012. URL WTLLhttp:://pottt/ http:://proceeeiiggs mlr www.sr.resssvveee vvtess//v2/aa/ http/////////eeeeee//////// domke 12, html
[13] Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang. E. Tzeng, and Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647-655. PMLR, 2014.
[14] Y. Fan, F. Tian, T. Qin, X.-Y. Li, and T.-Y. Liu. Learning to teach. arXiv preprint (tto::805..3444 2018.
[15] Y. Fan, Y. Xia, L. Wu, Xie, W. Liu, J. Bian, T. Qin, X.-Y. Li, and T.-Y. Liu. Learning to teach with deep interactions. arXiv preprint 2rX1v:2007.04664, 2020.
[16] C. Finn, P. Abbeel, and S. Levine. aodes--agnostic meta-learning for fast adaptation of deep
networks. In International Conference Machine Learning, pages 1126-1135, 2017.
[17] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pages 9516-9527, 2018
[18] G. Gidel, R. A. Hemmat, M. Pezeshki, R. Le Priol, G. Huang, S. Lacoste-Julien, and
I. Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference Artificial Intelligence and Statistics, pages 1802-1811. PMLR, 2019.
[19] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.

---

[20] A. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. Ivanov, R. Mark, J. Mietus, G. Moody, C. Peng, and H. Stanley. PhysioBank, PoosioToolkii, and PhysioNet: components of new research resource for complex physiologic signals. Circulation, 101 23:E215-20, 2000.
[21] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths. Recasting gradient-bass based meta-learning hierarchical bayes. arXiv preprint arXiv: 1801 0089000 2018.
[22] R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama. Meta approach to data augmentation optimization. arXiv preprint arXiv:2006. 07960, 2020.
[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[24] K. He, H. Fan, Y. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
[25] O. Henaff. Data-efficient image recognition with contrastive predictive coding. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Reseurch, pages 4182-4192. PMLR, 13-18 Jul 2020. URL http::://proceeediiiiiee wwtppsswww........eeeeeeeeeeesee
mlr. wwwpppresssviiii http//1/11/haaaff22aamaa html.
[26] R. D. Hjelm, A. Fedorov, Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019. URL [RLLattps///ppereeviee net//torum'1ddBk1333300
[27] J. Howard and S. Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 100iv 06646,,66..6 2018.
[28] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovee. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, het/fforumiid-tHIWW..
2020. URL https: htttps:::ppenreriieeeee
[29] Z. Hu, B. Tan, R. R. Salakhutdinov, T. M. Mitchell, and E. P. Xing. Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems, volume 32, 2019.
[30] B. K. Iwana and S. Uchida. An empirical survey of data augmentation for time series classifica- tion with neural networks. arXiv preprint arXiv:2007.15951 2020.
[31] K. Javed and M. White. Meta-learning representations for continual learning. arXiv preprint arXiv:1905. 12588, 2019.
[32] L. Jiang, Z. Zhou, T. Leung. L.-J. Li, and L. Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304-2313, 2018.
[33] K. Kandasamy, K. R. Vysyaraju, W. Neiswanger, B. Paria, C. Collins, J. Schneider, B. Poczos, and F. P. Xing. Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly. arXiv preprint arXiv: /903. 006694 2019.
[34] M. Kang, M. Han, and S. J. Hwang. Neural mask generator: Learning to generate adaptive word maskings for language model adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lnguage Processing (EMNLP), pages 1002-110,, 2020.
[35] A. Kolesníkov, I.. Beyer, X. Zhai, J. Puigcerver, J. Yung. S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint arXiv: 11370, 6(2):8, 2019.
[36] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[37] S. Kornblith, J. Shlens, and Q V. Le. Do better imagenet models transfer better? In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 2661-2000 1671, 2019.
[38] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex opti- mization. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 10657-10665, 2019.
[39] S. Liu, A. Davison, and F. Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems, pages 1679-1689, 2019.
12

---

[40] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
[41] J. Lorraine and D. Duvenaud. Stochastic hyperparameter optimization through hypernetworks. arXiv preprint arXiv:1802.09412 2018.
[42] J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1540-1552. PMLR, 2020.
[43] M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. In International Conference on Learning Representations, 2019.
[44] D. Maclaurin, D. Duvenaud, and R. Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pages 2113- 2122, 2015.
[45] M. McDermott, B. Nestor, E. Kim, W. Zhang, A. Goldenberg, P. Szolovits, and M. Ghassemi. A comprchensive evaluation of multi-task learning and multi-task pre-training on chr time-series data. arXiv preprint arXiv:2007. (tt.:20077.10855.. 2020.
[46] M. McDermott, B. Nestor, E. Kim, W. Zhang, A. Goldenberg, P. Szolovits, and M. Ghassemi. A comprehensive ehr timeseries pre-training benchmark. In Proceedings of the Conference on Health, Inference, and Learning, pages 257-278, 2021.
[47] Močkus. On Bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, pages 400-404, 1975.
[48] B. Mustafa, A. Loh, J. Freyberg, P. MacWilliams, A. Karthikesalingam, N. Houlsby, and V. Natarajan. Supervised transfer learning at scale for medical imaging. arXiv preprint ar arXiv:2101.059/ 2021.
[49] A. Navon, I. Achituve, H. Maron, G. Chechik, and E. Fetaya. Auxiliary learning by implicit differentiation. In International Conference on Learning Representations, 2021. URL httpss wttp://formmmddnn/lliia //openreview.
[50] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03788 2018.
[51] H. Pham, Z. Dai, Q Xie, M.-T. Luong, and Q. V. Le. Meta pseudo labels. arXiv preprint XrXiv:2003. / 10580, 2020.
[52] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training. OpenAl blog, 2018.
[53] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards under- standing the effectiveness of maml. In International Conference on Learning Representations, 2019.
[54] A. Raghu, M. Raghu. S. Kornblith, D. Duvenaud, and G. Hinton. Teaching with commentaries. arXiv preprint
2020.
[55] A. Rajeswaran, C. Finn, S. Kakade, and S. Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 2019.
[56] R. Rao, N. Bhattacharya, N. Thomas, Y. Duan, X. Chen, J. Canny, P. Abbeel, and Y. S. Song. Evaluating protein transfer learning with tape. Advances in Neural Information Processing Systems, 32:9689, 2019.
[57] M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4331-4340, 2018.
[58] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960, 2018.
[59] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 306-813, 2014.
13

---

[60] J. Shu, Q. Xie, L. Yi, Q. Zhao, S. Zhou, Z. Xu, and D. Meng, Meta-weight weight-net, Learning an explicit mapping for sample weighting. Advances in Neural Information Processing Systems, pages 1919-1930, 2019.
[61] J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951-2959, 2012.
[62] F. P. Such, A. Rawal, J. Lehman, K. Stanley, and J. Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In International Conference Machine Learning, pages 9206-9216. PMLR, 2020.
[63] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. arXiv: (906,0 2019.
arXiv preprint
[64] Wagner, N. Strodthoff, R.-D. Bousseljot, D. Kreiseler, F. I. Lunze, W. Samek, and T. Schaeffter. PTB-XL, large publicly available elcctrocardiography dataset. Scientific data, 7(1):1-15, 2020.
[65] X. Wang and A. Gupta. Unsupervised learning of visual representations using vidcos. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2015.
[66] Q. Wen, L. Sun, X. Song, Gao, X. Wang, and H. Xu. Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002. 2020.
[67] Y. Wu, M. Ren, R. Liao, and R. Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint arXiv:1803.0202. 02021, 2018.
[68] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018
[69] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL /ttppptts///ppeereiiie net//orrumiidrrye66i1
[70] Y. Xue, N. Du, A. Mottram, M. Seneviratne, and A. M. Dai. Learning to select best forecast tasks for clinical outcome prediction. Advances Neural Information Processing Systems, 33, 2020.
[71] Q. Ye, B. Z. Li, S. Wang, B. Bolte, H. Ma, W. tau Yih, X. Ren, and M. Khabsa. On the influence of masking policies intermediate pre-training, arXiv preprint arXiv:1801.06144 2021.
[72] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.0677 2020.
[73] X. Zhai, A. Oliver, Kolesnikov, and Beyer. S41: Self-supervised sumi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1476- 1 1485, 2019.
[74] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, Djolonga, A. Pinto, M. Neumann, A. Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.040 1ett.09900.. 2019.
[75] L. M. Zintgraf, K. Shiarlis, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. arXiv preprint 10X0:I8100.336 2018.
14


Reviews:
[{"review_id": "unRJTQgb4iG", "reviewer": "Reviewer_UJ1Z", "summary": "This paper propose to solve a very important problem, meta-learning hyperparameters in pre-training stage, followed by fine-tuning a target task. Considering that the hyperparameters can be high-dimensional, the authors consider gradient-based hyperparameter optimization (HO) methods such as IFT based methods and unrolled differentiation. The main difficulty comes from the huge computational cost of dealing with long PT and FT trajectory and repeating such meta-optimization steps. Authors thus simply reduce the number of gradient steps used for each PT and FT stage. Yet, the experimental results demonstrate the effectiveness of the proposed methods over some of the simple baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "== Pros ==\n\n1. I think the paper tackles the very important and interesting problem, meta-learning of PT hyperparameters followed by FT. As far as I know, there are few literatures that explicitly tackle this problem. The problem is challenging because both PT and FT involve long optimization trajectory in practice, leading the optimization problem computationally very expensive. Therefore, I would say that the motivation of this paper is very clear and important, in terms of extending the current range of meta-learning and hyperparameter-optimization study.\n\n2. The paper is well written and easy to understand, combining the two most popular techniques for HO (IFT and unrolled diff.). \n\n3.  The experimental results show that although the proposed method is simple and straightfoward, it could gain some improvements over the baselines they considered.\n\n\n== Cons ==\n\n1. As far as I understand, the main difficulty should come from the long PT and FT optimization trajectory used in practice (e.g. each at least thousands of SGD steps). However, the authors simply set each of them to very short number of steps, such as $P=10$ and $K=1$ for both of the real-world experiments. It will lead to short horizon bias, with the solution being less appealing. For example, Shin et al. recently proposed a method that can increase the frequency of meta-updates even with long inner-optimization trajectories. It would be better if the authors could think of some ways to deal with the short horizon bias in any way, because exactly that is the main challenge of this problem.\n\n2. The proposed method is fairly straightforward to think of, combining the two existing techniques for HO and unrolled optimization (MAML). I agree that the method is reasonable, but not sure how much it is technically contributing. Also, as mentioned in section 8 Scopes and Limitations, IFT based methods cannot handle hyper-parameters that do not affect the PT optimization landscape. \n\n3. In the experiments, I think the baselines are too few. Specifically, there are no baselines that can learn the hyperparameter $\\phi$. What if we learn $\\phi$ only with the PT dataset based on conventional HO framework? We may split the $D_\\text{PT}$ into $D_\\text{PT}^\\text{train}$ and $D_\\text{PT}^\\text{val}$, and optimize $\\phi$ with Neumann IFT method as you did. I expect it will work well because the authors reported good performance on Partial FT-data setting, where we only use a fraction of FT tasks for meta-training and meta-test with the exclusive set of FT tasks no seen during the meta-training. It means that whatever tasks we use for learning $\\phi$, the learned $\\phi$ will generalize well to unseen tasks. If it works well, then the importance of meta-learning over PT-FT framework will become questionable.\n\n4. (minor comment) There is no qualitative analysis. It would be interesting to visualize the learned $\\phi$ and give the readers some intuition how it helped with the performance (or any other visualization).\n\n5. (minor comment) Missing reference for meta-learning of self-supervised learning : Kang et al.\n\n= References = \n- Shin et al., Large-Scale Meta-Learning with Continual Trajectory Shifting, ICML 2021\n- Kang et al., Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation, EMNLP 2020\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately adressed their limitations and social impract.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "pnR_r58f7yo", "reviewer": "Reviewer_q25x", "summary": "The paper describes a gradient-based hyperparameter optimization for differentiable pre-training hyperparameters in a pre-training and fine-tuning setup. The proposed approach approximately differentiates through the two-stage learning process. The authors show the benefit of their approach in two experiments, where they outperform multiple baselines.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: In my opinion, this kind of meta-learning on two-stage setups is not new (see ref. [67] and [68]) but the more general approach is novel. It feels a bit like MAML [15] for differentiable hyperparameters (MAML is mentioned in the related work). The difference to related work is described and related work is cited. \n\nClarity: I had issues following this paper. While in the abstract and introduction the authors talk about “hyperparameters” they mention only in the limitations section that the hyperparameters have to be differentiable. When I read hyperparameters of neural network training, I think mainly on parameters like the learning rate. The abstract promises an HPO method for hyperparameters in general without mentioning this limitation.\nIn the second section in the paragraph “Notation”, the authors introduce uppercase theta, uppercase psi, and uppercase phi as “spaces of parameters for models” without any further description on the difference between these “spaces”.  Two pages later in line 123, the authors name phi “encoder parameters” and psi “decoder parameters” but without mentioning that they assume an encoder-decoder setup. \nIn the abstract and introduction, the authors write about optimizing neural networks while in the problem setup they describe the more general case of “parametric model”. \nI wasn’t able to follow their algorithm due to the lack of knowledge about theta, psi, and phi. \nIn the experiment section, it is unclear where the definition of the loss comes from and how P and K are tuned. \nI would propose to clarify what kind of hyperparameters are addressed by this approach and rework the problem formulation and notation.\n\nQuality: I can’t assess the methods since I didn’t understand the algorithm in detail due to the lack of knowledge about theta, psi, and phi. The method is evaluated in two experiments with good results.\n\t \nSignificance: In my opinion, the experiments are very specialized for a general tone in the abstract and introduction. I couldn't follow the notation easily. Even if I am not a domain expert for gradient-based hyperparameter optimization this shouldn’t be the case in my opinion. In the current state of this paper, I find it hard to make use of the approach. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NMg5zIV_vT", "reviewer": "Reviewer_1DnG", "summary": "This paper introduces a gradient-based algorithm which improve the first stage of two-stage pre-trained and fine-tune process with meta-learning techniques. The major contribution is proposing to use additional parameters called meta-parameters to augment pre-trained stage and use a novel gradient-based algorithm to learn meta-parameters through the two-stage process. The experimental results demonstrate the significant improvement of the proposed approach over the standard two-stage PT and FT process.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "== Pros ==\n- Using meta-parameters to improve pre-trained model has enough novelty, and further research on this open problem is meaningful.\n- The authors propose a novel gradient-based algorithm to optimize meta-parameters.\n- The proposed approach achieves significant improvements on both tasks.\n\n== Cons ==\n- Meta-parameters as neural networks could have much more representative power than weight of task or temporal transformation, I'd like to see more complex scenarios where this approach can be applied.\n- Despite the experiment setting in this paper may be new, three still need some other meta-learning techniques as baseline if possible. Because as far as I know, there are many gradient-based meta-learning algorithms that are not used in PT and FT process for now.\n- Meta-learning techniques usually requires much more computing resources and time, an analysis on GPU memory and execution speed would be appreciated.\n- Some symbols may be confused for me, such as line 4 in Algorithm 1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Ioonax566-k", "reviewer": "Reviewer_BiAi", "summary": "This work seeks to extend gradient-based HPO to the two stage setting of pretraining + finetuning. While the method used isn't original it is used in an original problem set up.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-----------------------------------PROS\n- The PT+FT paradigm considered is original and relevant\n- the paper is mostly clear\n- Considering Full FT access and Partial FT access is a good thing\n\n-----------------------------------CONS\n\nMAJOR\n\n- You explain that you update psi_FT online, which may remove memory issues with BPTT, but has been shown to lead to biased (greedy) solutions (\"Short horizon bias\", Wu2018). These don't converge to hypergradients close to the actual (full horizon) hypergradients we care about. This is a major limitation to your model since the gradients that come from the finetuning stage are in my experience almost as good as random noise, and yet this isn't listed in your Limitations section. A toy model where you can use full-horizon BPTT for both PT and FT would allow you to measure how accurate your algorithm is at approximating hypergradients as the number of steps increase in the PT and FT stage.\n- While experiments seem sensible, I feel like the experiments I would need to see to make sure your method is competitive with sota methods aren't included. For instance, I would have liked to see comparisons with sota methods in multi-task learning and/or semi-supervised learning and/or domain adaptation for common image datasets like CIFAR-10 where lots of other methods have been applied. This is because the main difficulties of gradient-based HPO (e.g. gradient degradation) come in for a large number of steps in the PT/FT stages. In line 196 you seem to be using a very short horizon of P=10, which is nowhere near the 10^4 gradient steps you'd need for a CIFAR10-like dataset.  \n- There is also an issue of novelty, since the combination of implicit differentiation and BPTT is fairly straightforward (just multiplication as per chain rule) and one may argue doesn't really make up a \"new\" algorithm in itself. \n\nMINOR\n\n- Your toy experiments in section 4 are both experiments that don't require your PT + FT 2-stage method. Indeed, these have been done where the properties you apply to the FT data are simply applied to the validation data instead. It would be nice to have experiments where ONLY your 2-stage approach is sensible. \n- I found Figure 1 somewhat confusing. For instance I'm not sure what \"Finetuning Data\" + \"Sample\" labels achieve, why the \"meta parameters\" are connected to \"Pre-training\" on top of the \"Model\" and \"Pre-train Dataset\" labels since potentially the \"meta parameters\" could include \"Model\" and \"Pre-training dataset\". I understood your set up from the equations in 2.1, although a bit more details would have saved me time. For example in line 78 you could explicitely state the the final weights of the PT stage are used as the init weights of the FT stage, such that the final weights of the FT stage minimize the FT loss. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pretrain-finetuning is an important and popular learning framework for modern deep learning models, and has achieved significant success. How to learn some hyperparameter of the framework is important yet not well explored. This paper proposes a meta learning strategy for this via direct and implicit differentiation to perform gradient descent. The technique is not new but has not been applied to this problem.

All reviewers seem to consider the paper to be in the borderline. They raised a number of questions including the setting of the number of PT/FT steps, if it will cause short horizon bias, and also asked for some additional baselines to demonstrate the effectiveness of the proposed method. Most of the questions, in my opinion, have been well addressed in the rebuttal with further explanation and extra experiments. Thus I would recommend acceptance of the paper. One concern I have is that it seems the scale of the data in the experiments is not large enough (from what I understand in the paper, though the statistics of some data have not been provided). I guess investigating the large-scale setting is important because this is where the PT/FT framework really reveals its power. I would recommend the authors to revise the paper according to the comments from the reviewers, and consider extra large-scale experiments (such as the ImageNet scale data) to make the paper stronger.
Decision: Accept (Poster)