Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning
Authors: Zhuoyan Xu
Zhenmei Shi
Junyi Wei
Fangzhou Mu
Yin Li
Yingyu Liang
Venue: ICLR 2024 poster (2024)
Keywords: Foundation model
Multitask finetuning
Few-Shot learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We delve into theoretical framework of multitask finetuning for foundation models on new tasks with limited labels. Our introduced task selection algorithm notably boosts model performance, backed by extensive empirical evidence.

Abstract: Foundation models have emerged as a powerful tool for many AI problems. Despite the tremendous success of foundation models, effective adaptation to new tasks, particularly those with limited labels, remains an open question and lacks theoretical understanding. 
  An emerging solution with recent success in vision and NLP involves finetuning a foundation model on a selection of relevant tasks, before its adaptation to a target task with limited labeled samples. In this paper, we study the theoretical justification of this multitask finetuning approach. 
Our theoretical analysis reveals that with a diverse set of related tasks, this multitask finetuning leads to reduced error in the target task, in comparison to directly adapting the same pretrained model. We quantify the relationship between finetuning tasks and target tasks by diversity and consistency metrics, and further propose a practical task selection algorithm.
  We substantiate our theoretical claims with extensive empirical evidence.
Further, we present results affirming our task selection algorithm adeptly chooses related finetuning tasks, providing advantages to the model performance on target tasks.
  We believe our study shed new light on the effective adaptation of foundation models to new tasks that lack abundant labels.
  Our code is available at https://github.com/OliverXUZY/Foudation-Model_Multitask.

Published as a conference paper at ICLR 2024
TOWARDS FEW-SHOT ADAPTATION OF FOUNDATION MODELS VIA MULTITASK FINETUNING
Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, Yingyu Liang
University of Disconsin-Madise (zhuoyan. xu, jwei 53, fmu2, yi n. 1i (wwisc.edu (zhmeishi, y yliang) êcs. wi sc..cded
ABSTRACT
Foundation models have emerged as a powerful tool for many Al problems. De- spite the tremendous success of foundation models, effective adaptation to new tasks, particularly those with limited labels, remains an open question and lacks theoretical understanding. An emerging solution with recent success in vision and NLPP involves finetuning a foundation model on a selection of relevant tasks, before adaptation to a target task with limited labeled samples. In this paper, we study the theoretical justification of this multitask finetuning approach. Our theoretical analysis reveals that with a diverse set of related tasks, this multitask finetuning leads to reduced error in the target task, in comparison to directly adapting the same pretrained model. We quantify the relationship between finetuning tasks and target tasks by diversity and consistency metrics, and further propose practical task selection algorithm. We substantiate our theoretical claims with extensive empirical evidence. Further, present results affirming our task selection algorithm adeptly chooses related finetuning tasks, providing advantages to the model performance target tasks. We believe our study shed new light on the effective adaptation of foundation models to new tasks that lack abundant labels. Our code available at bttps://gitinn com/ wwwoooivvuuuo / Fouudtt Loooooooooooooooooo Muul bilutkttak..
INTRODUCTION
The advent of large-scale deep models trained on massive amounts of data has ushered in new era of foundation models (Bommasani et al., 2021). These models, exemplified by large language models (e.g., BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020)) and vision models (e.g.. CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023)), offer the promise for adapting to wide range of downstream tasks, and have led to some of the most exciting developments Al to date, including the latest conversational AI ChatGPT (OpenAI, 2022) and GPT4 (OpenAI, 2023). Despite encouraging empirical results (Zhang et al., 2020; Brown et al., 2020; Gao et al., 2021a), the effective adaptation of foundation models, especially to new tasks with limited labels, remains practical challenge and lacks theoretical understanding.
In this paper, focus the problem of adapting pretrained foundation model few labeled samples, where the target task can differ significantly from pretraining and the limited labeled data insufficient for finetuning. This few-shot learning problem has been long-standing challenge in machine learning (Wang et al., 2020). Prior approaches include learning from examples in the context prompt (in-context learning) (Brown et al., 2020), constructing simple classifiers based on the pretrained representation (Zhang et al., 2020), or finetuning the model using text prompts converted from labeled data (Gao et al., 2021a). ) An emerging solution involves finetuning a pretrained model on multiple auxiliary tasks pertaining to the target task. This multitask finetuning approach, related to meta learning (Hospedales et al., 202 1), has been recently explored in NLP and vision (Murty et al., 2021; I; Vu et al., 2021; Zhong et al., 2021; Hu et al., 2022b; Chen et al., 2022; Min et al., 2022a). For example, latest studies (Sanh et al., 2022; Muennighoff et al., 2023) show that finetuning language models on large set of tasks enables strong sero-shot generalization tasks. Nonetheless, the lack of sound theoretical explanations behind these previous approaches raises doubts about their ability to generalize real-world tasks (Perez al., 2021).
task with
unseen

---

Published as a conference paper at ICLR 2024
To bridge the gap, we study the theoretical justification of multitask finetuning. We consider an intermediate step that finetunes a pretrained model with a set of relevant tasks before adapting to target task. Each of these auxiliary tasks might have small number of labeled samples, and categories of these samples might not overlap with those on the target task. Our key intuition is that a sufficiently diverse set of relevant tasks can capture similar latent characteristics the target task, thereby producing meaningful representation and reducing errors in the target task. To this end, we present rigorous theoretical analyses, provide key insight into conditions necessary for successful multitask finetuning, and introduce a novel algorithm for selecting tasks suitable for finetuning.
Our key contributions are three folds. Theoretically, we present framework for analyzing pretraining followed by multitask finetuning. Our analysis (Section 3) reveals that with limited labeled data from diverse tasks, finetuning can improve the prediction performance on a downstream task. Empirically, we perform extensive experiments on both vision and language tasks (Section 4) to verify our theorem. Our results suggest that our theorem successfully predicts the behavior of multitask finetuning across datasets and models. Practically, inspired by our theorem, we design task selection algorithm for multitask finetuning. On the Meta-Dataset (Triantafillou et al., 2020), our algorithm shows significantly improved results in comparison to finetuning using all possible tasks.
1.1 RELATED WORK
We briefly summarize related work and refer our readers to Appendix B for a detailed discussion.
Foundation models (Bommasani et al., 2021) are typically pretrained over broad data using approaches include contrastive learning (Oord et al., 201 88; Chen et al., 2020; He et al., 2020; Tian et al., 2020a; Grill et al., 2020; Radford et al., 2021) vision and masked modeling in NI.P (Devlin al., 2019; Liu et al., 2019). Adapting foundation models to downstream target tasks has received significant attention, c.g., minorly finetuning (Vinyals et al., 2016; Chen et al., 2020; He et al., 2020; 2022), prompt-based finetuning (Gao et al., 2021a; Hu et al., 2022a), prompt tuning (Lester et al., 2021; Li & Liang, 2021), and in-context learning (Min et al., 2022b; Wei et al., 2022a). Our work studies an emerging solution of multitask finetuning (Zhong et al., 2021; Sanh al., 2022: Min et al., 2022a; Chen et al., 2022; Wang et al., 2023), which finetunes the pretrained foundation model using multiple relevant tasks before adapting the target task. Multitask finetuning has been shown to induce zero-shot generalization in large language models (Sanh et al., 2022; Muennighoff et al., 2023), and enable parameter efficient tuning by prompt tuning ( (Wang et al., 2023). Our work seeks provide theoretical justification to prior approaches. Part of our results also confirm previous findings.
line of theoretical work provides the error bound of the target task in terms of sample complexity (Du et al., 2021; Tripuraneni et al., 2021; Shi et al., 2023a). Their work mainly analyzed representations from supervised pretraining using multitasks. In contrast, our work considers representations from self-supervised pretraining, and focuses on multitask finetuning. Our approach and analysis guarantee that limited but diverse finetuning data can improve the prediction performance on a target task with novel classes. On the other hand, with only limited target labels, few-shot learning necessitates the generalization to new tasks (Wang et al., 2020; Yang et al., 2022). Direct training with limited data is prone to overfitting. Meta learning offers a promising solution that allows the model to adapt the few-shot setting (Snell et al., 2017; Finn et al., 2017; Raghu et al., 2020; Chen et al., 2021b; Hu et al., 2022b). Inspired by meta learning, our analysis extends the idea of multitask finetuning by providing sound theoretic justifications and demonstrating strong empirical results. We further introduce a task selection algorithm that bridges our theoretical findings with practical multitask finetuning.
BACKGROUND: MULTITASK FINETUNING FOR FEW-SHOT LEARNING
This section reviews the pretraining of foundation models and adaptation for few-shot learning, and then formalizes the multitask finetuning approach.
Pretraining Foundation Models. We consider three common pretraining methods: contrastive learning, masked language modeling, and supervised pretraining. Contrastive learning is widely considered in vision and multi-modal tasks. This approach pretrains a model & from a hypothesis class & of foundation models via loss contrastive pairs generated from data points First sample a point x and then apply some transformation to obtain x independently sample another point
x= The population contrastive loss is then Loon pre (Ø) E [e, (Ø(x) (0(x+) ) ((())))]
2

---

Published as a conference paper at ICLR 2024
where the loss function ly is non -egaative decreasing function. In particular, logistic loss (v) log (1 + exp (-+)) recovers the typical contrastive loss in most empirical work (Logeswaran & Lee, 2018; Oord et al., 2018; Chen et al., 2020). Masked language modeling is a popular self-supervised learning approach in NIP. It can be regarded as a kind of supervised pretraining: the masked word is viewed the class (see Appendix C for more details). In what follows we provide unified formulation. On top of the representation function 6, there is a linear function f E F FC { + R*} predicting the labels where K is the number of classes. The supervised loss is: Lsup-pre(C) (0) = minser E [e (f 6(z), y)], where E(., y) is the cross-entropy loss. To simplify the notation, we unify fere as the pretraining loss.
Adapting Models for Few-shot Learning. A pretrained foundation model & can be used for downstream target tasks T by learning linear classifiers on Φ. We focus on binary classification (the general multiclass setting is in Appendix D). A linear classifier on ø is given by w ((x) where w E gd The supervised loss of á w.r.t the task T is then:
C sup (T, 6) == min (I.D)~DT
E
[e (w" o2)),))]
(1)
where Dr y) is the distribution of data (x. y) in task T. In few-shot learning with novel classes, there are limited labeled data points for learning the linear classifier. Further, the target task To may contain classes different from those in pretraining. We are interested in obtaining a model & such that fou Φ) is small.
Multitask Finetuning, In the challenging setting of few-shot learning, the data in the target task is limited. On the other hand, we can have prior knowledge of the target task characteristics and its associated data patterns, and thus can collect additional data from relevant and accessible sources when available. Such data may cover the patterns in target task and thus can be used auxiliary tasks to finetune the pretrained model before adaptation to the target task. Here we formalize this idea in general form and provide analysis in later sections. Formally, suppose we have M auxiliary tasks (Ti, T TT...,,T,,,,, J. each with m labeled samples S, == {(r), vi) j E [m]]. The finetuning data are = UIE(M]S Given a pretrained model Φ. we further finetune it using the objective:
M Σ Loup (Te, Φ), where L (Ti, = min ERd m
min OE M
Σ Elw) ((a)). J=1
(2)
This can be done via gradient descent from the initialization & (see Algorithm 2 in the Appendix). Multitask finetuning is conceptually simple, and broadly applicable to different models and datasets. While its effectiveness has been previously demonstrated (Murty et al., 2021; Vu et al., 2021; Zhong et al., 2021; Hu et al., 2022b; Chen et al., 2022; Min et al., 2022a; Sanh et al., 2022; Muennighoff et al., 2023), the theoretical justification remains to be fully investigated and understood.
3 THEORETICAL ANALYSIS: BENEFIT OF MULTITASK FINETUNING
To understand the potential benefit of multitask finetuning, we will compare the performance of (from pretraining) and ø' (from pretraining and multitask finetuning) target task To. That is, will compare Leup (To, Φ) and Caup (To, d'), where (T Φ) is the population supervised loss of on the task T defined in Equation (1). For the analysis, we first formalize the data distributions and learning models, then introduce the key notions, and finally present the key theorems.
Data Distributions. Let X' be the input space and E C Rd be the output space of the foundation model. Following Arora et al. (2019), suppose there is set of latent classes with [C] = K, and a distribution 1 over the classes; each class E C has a distribution D(y) over inputs x. In pretraining using contrastive learning, the distribution Deon (1) of the contrastive data (ar, x+ _=) is given by: (v.yT) ~ 172 and x, - D(v), x- ~ D(y ). In masked self-supervised or fully supervised pretraining, (x, v) is generated by y ~ JJ, ~ D(y). In a task T with binary classes (M1, 3/2, the data distribution D2x2,0 y) is by first uniformly drawing y E {v1, 1/2) and then drawing x D(y). Finally, let C denote the conditional distribution of (y1, Y2) ~ 17² conditioned on ÿ1 * y2, and suppose the tasks in finetuning are from Ç. Note that in fhw-shot learning with novel classes, the target task's 's classes may not be the same as those in the pretraining. Let Co be the set of possible classes in the target task, which may may not overlap with C.
3

---

Published as a conference paper at ICLR 2024
Learning Models. Recall that & is the hypothesis class of foundation models Ø x Z. To gauge the generalization performance, let o* E denote the model with the lowest target task loss Lsup(To o²) and o E o denote the model with the lowest average supervised loss over the set of auxiliary tasks ( (
:= ET~ [Caup (T, x3)]. Note that if all ø E & have high supervised losses, we cannot expect the method to lead to good generalization performance, and thus we need to calibrate w.r.t. 6 and o We also need some typical regularity assumptions.
Assumption 1 (Regularity Assumptions). | S R and linear operator || B. The loss lu is bounded in [0, C) and T--lipschizz The supervised loss Ceup(T ,6) is L-Lippshitz with respect to é.
Diversity and Consistency. Central to our theoretical analysis lies in the definitions of diversity in auxiliary tasks used for finetuning and their consistency with the target task.
Definition 1 (Diversity). The averaged representation difference for two model o, á on a distribution C over tasks is do Φ) = E T~C
(Tsup(T (T, Φ) -uu (T, .) = Lsup(d) - Suu (Ø). The worst-case
representation difference between representations o, e on the family of classes Co is de, (ó, Ø) == supT.co. Caup (To, 6) - Ceup(To: o) We say the model class & has v-diversity (for Ç and Co) with respect to 0%. for any & E t de (0, +3) S de(0, $2)/v.
Such diversity notion has been proposed and used to derive statistical guarantees (e.g., Tripuraneni al. (2020); Zhao al. (2023)). Intuitively, diversity measures whether the data from ( covers the characteristics of the target data in Co. e.g., whether the span of the linear mapping solutions w for tasks from ( can properly cover the solutions for tasks from Co (Zhao et al., 2023). Existing work showed that diverse pretraining data will lead to large diversity parameter and can improve the generalization in the target task. Our analysis will show the diversity in finetuning tasks from Ç can benefit the performance of a target task from Co-
Definition (Consistency). We say the model class has k-consistency (for aand Co) with respect oi and oz where SUPT.co. (ooup(Ta - (To:o*) x*)
This consistency notion measures the similarity between the data in tasks from ( and the data in the target task from Cg. Intuitively, when the tasks from & are similar to the target task To. their solutions of and 6² will be similar to each other, resulting in a small s. Below we will derive guarantees based on the diversity v and consistency x to explain the gain from multitask finetuning.
Key Results. We now present the results for a uniform distribution y) and include the full proof and results for general distributions in Appendix C and Appendix D. Recall that will compare the performance of é (the model from pretraining) and ø' (the model from pretraining followed by multitask finetuning) on a target task To. For & without multitask finetuning, we have:
Theorem 3.1. (No Multitask Finetuning) Assume Assumption and that & has v-diversity and k-consistency with respect to oi and &z Suppose & satisfies Epre (x) S c0. Let Pr {u = =. Then for any target task To S Co. (Ma
1 200 (Tu Φ) Coup(TT, (T)
Coup(0t)
+ KG.
1/ 1 1-T
In Theorem 3.1, (p) is the empirical loss of Lpre(Ø) with pretraining sample size N. We now consider o' obtained by multitask finetuning. Define the subset of models with pretraining loss Recall the Rademacher complexity of t on n
smaller than as (Z)) = Φ E कb Epre( S {)..
points is R2 (o,))=n-<=;);
E
suppee
Theorem 3.2 below showing that the target prediction performance of the model ø' from multitask finetuning can be significantly better than that of Φ without multitask finetuning, In particular, achieves an error reduction 1 (1 a) |=+ The reduction achieved when multitask finetuning is solved to small loss e for small a sufficiently many finetuning data.
4

---

Published as a conference paper at ICLR 2024
Theorem 3.2. (With Multitask Finetuning) Assume Assumption and that & has v-diversity and k-consistency with respect to o" and 6.. Suppose for some constant & E (0, 1). we solve Equation (2) with empirical loss lower than e₁ o 2ea_ and obtain &'. For any 8 > 0, if for ē = =Cpreld').
1 (V2)) 40 M then with probability 1 8, for any larget task To - Co.
log( 1ogt3) Mm 2
402 2
(...RR + log
())
Gi
6i
C
2E0
S x¹) - (To x*) - 1/
a 1-
(4)
The requirement is that the number of tasks M and the total number of labeled samples M m across tasks are sufficiently large. This implies when M is above the threshold, the total size Mm determines the performance, and increasing either M or while freezing the other can improve the performance. We shall verify these findings in our experiments (Section 4.1).
Theorem 3.2 also shows the conditions for successful multitask finetuning, in particular, the impact of the diversity and consistency of the finetuning tasks. Besides small finctuning loss on sufficiently many data, large diversity parameter v and small consistency parameter x will result in small target error bound. Ideally, data from the finetuning tasks should be similar to those from the target task, but also sufficiently diverse cover wide range of patterns that may be encountered in the target task. This inspires us to perform finer-grained analysis of diversity and consistency using a simplified data model (Section 3.1), which sheds light on the design of an algorithm select a subset of finetuning tasks with better performance (Section 3.2).
3.1 CASE STUDY OF DIVERSITY AND CONSISTENCY
Our main results, rooted notions of diversity and consistency, state the general conclusion of multitask finetuning on downstream tasks. A key remaining question is how relevant tasks should be selected for multitask finetuning in practice. Our intuition is that this task selection should promote both diversity (encompassing the characteristics of the target task) and consistency (focusing on the relevant patterns in achieving the target task's 's objective). To illustrate such theoretical concepts and connect them practical algorithms, interpretation of diversity and consistency. In this section, we provide a toy linear case study and we put the proof and also the analysis of a more general setting in Appendix E, e.g., more general latent class C, Co. more general distribution Ç. input data with noise.
specialize the general conclusion to settings that allow easy
In what follows, we specify the data distributions and function classes under consideration, and present an analysis for this case study. Our goal is to explain the intuition behind diversity and consistency notions: diversity is about coverage, and consistency is about similarity in the latent feature space. This can facilitate the design of task selection algorithms.
Linear Data and Tasks. Inspired by classic dictionary learning and recent analysis on repre- sentation learning (Wen & Li, 2021; Shi et al., 2023a), we consider the latent class/representation setting where each latent class z E [1] is represented as feature vector. We focus on individual binary classification tasks, where D [-1,+1) is the label space. Thus, each task has two latent classes 2, (denote the task as Tz,²) and we randomly assign -1 and +1 to each latent class. Namely, Te. is defined
d-kc
Figure 1:
Illustration of features in linear data. Blue the features encoded in C while red not.
2, if -1
as: x = [2', if --y=1 +1'
We show a diagram in Figure 1, we denote each task containing
two latent classes, namely (z,z'). Each task in diagram can be represented as (Ti T T2 T2 T zg). We further assume balanced class setting in all tasks, i.e. ply -1)
5

---

Published as a conference paper at ICLR 2024
ply +1) 2. Now, define the latent classes seen in multitask finetuning tasks: C 1, - 1,0,
(1,1,...,, 1.0,
(-1,1,
d-ke
kC
Note that their feature vectors only encode the first ke features, and [C] kc. We let Co : ,1(0))(())) C +0,-1,+1] which used for the target task, and assume that 2(1) and 2(2) only differ in dimension, i.e. the target task can be done using this one particular dimension. Let be a distribution uniformly sampling two different latent classes from C. Then, our data generation pipeline for getting multitask finetuning task is (1) sample two latent classes (2, (2,2")= Ç; (2) assign label -1, +1 to two latent classes.
Linear Model and Loss Function. We consider linear model class with regularity Assumption 1, ie., +b = (Ø E Radxd || Σ 1} and linear head w E Rd where ||æ]|2 I 1. Thus, the final output of the model and linear head w or. We use the loss in Shi et al. (2023a), i.e.
e (w ex, y) = -yw ςxx.
Remark 3.1. Although we have linear data, linear model, and linear loss, Suu (() is a non-linear function on à as the linear heads are different across tasks, i.e., each task has own linear head.
Now we can link our diversity and consistency features encoded by training or target tasks.
Theorem 3.3 (Diversity and Consistency). If encodes the feature in Co. i.e., the different entry dimension of 2 and 22)) in Co is in the first ke dimension, then we have v is lower
bounded by constant 2 2v2-2 and S V ke Otherwise, we have v and kc-1
Theorem 3.3 establishes č-diversity and s-consistency Definition and Definition 2. The analysis shows that diversity can be intuitively understood as the coverage of the finetuning tasks on the target task in the latent feature space: the key feature dimension of the target task covered by the features encoded by finetuning tasks, then we have lower-bounder diversity μt if not covered, then the diversity tends to 0 (leading to vacuous error bound in Theorem 3.2). Also, consistency can be intuitively understood as similarity in the feature space: when he is small, large fraction of the finetuning tasks are related to the target task, leading to a good consistency (small k); when Acc is large, we have less relevant tasks, leading worse consistency. Such an intuitive understanding of diversity and consistency will be useful for designing practical task selection algorithms.
3.2
TASK SELECTION
Our analysis suggests that out of a pool of candidate tasks, a subset 5 with good consistency (i.e., small x) and large diversity (i.e., large v) will yield better gen- eralization to target task. To realize this insight, present greedy selection approach, which sequen- tially adds tasks with the best consistency, and stops when there no significant increase the diversity of the selected subset. In doing so, our approach avoids enumerating all possible subsets and thus is highly practical.
Coverage Boundaryy
Figure 2: Illustration of the similarity and cover- age. Target tasks (To) with the most similar tasks in yellow and the by yellow tasks the coverage Adding more tasks in blue to the ellipsoid does not increase the coverage boundary.
key challenge is to compute the consistency and diversity of the data. While the exact computation deems infeasible, we turn to approximations that capture the key notions of consistency and diver- sity. We show simplified diagram for task selec- tion in Figure model é. assume any task data T {æ,} follows Gaussian distribution in the repre- sentation space: let ((T) = { denote the representation vectors obtained by applying the data points in Т; compute the sample mean μт and covariance Cf for o(T), and view it as the Gaussian (Hr, Cy). Further, following the intuition shown the case study, simplify consistency to similarity: for the target task To and candidate task T, if the
blue. The ellipsoid spanned the target task.
Specifically, given foundation

---

Published as a conference paper at ICLR 2024
sine similarity CosSim(To,11 T:) PT, (lll plllllllalll is large, we view T: consistent with To. Next, we simplify diversity to coverage: if a dataset (as a collection of finetun- ing tasks) largely "covers" the target data To, we view D as diverse for To- Regarding the task data as Gaussians, we note that the covariance ellipsoid of D covers the target data P% iff (ND - -r,)" (pD - PT,) I 1. This inspires us to define the following coverage score as a heuristic for diversity: coverage( D; To) == 1/(up (T)) " (T)).
Using these heuristics, descending order of their cosine similarities to the target data; sequentially add tasks in the sorted order until coverage LL; To) has no significant increase. Algorithm illustrates this key idea.
arrive at the following selection algorithm: the candidate task in
Algorithm 1 Consistency--Di Diversstt Task Selection
Input: Target task To. candidate finetuning tasks: (T1, Te...., TM.. model o, threshold p. 1: Compute o(T)) and μt, for = 0, I....,,..
2: Sort T/'s s in descending order of similarity (To, T:). Denote the sorted list as (T', T2,.... Thr
3: L+ (TT)
4: for =.ri= M do
5: If U 2 p) -ooeerage((; To), then L + LUT; otherwise, break.
6: end for
Output: selected data L for multitask finetuning.
4 EXPERIMENTS
We now present our main results, organized in three parts. Section 4.1 explores how different numbers of finetuning tasks and samples influence the model's performance, offering empirical backing to theoretical claims. Section 4.2 investigates whether our task selection algorithm can select suitable tasks for multitask finetuning. Section 4.3 provides more extensive exploration of the effectiveness of multitask finetuning on various datasets and pretrained models. We defer other results to the appendix. Specifically, Appendix F.4.1 shows that better diversity and consistency of finetuning tasks yield improved performance on target tasks under same sample complexity. Appendix F.4.2 shows that finetuning tasks satisfying diversity yet without consistency lead to no performance gain even with increased sample complexity. Further, Appendix G and Appendix H present additional experiments using NLP and vision-language models, respectively.
Experimental Setup. We use four few-shot learning benchmarks: minilmageNNt (Vinyals et al., 2016), maeredI (Ren et al., 2018), DomainNet (Peng et al., 2019) and Meta-dataset (Triantafil- lou et al., 2020). We use foundation models with different pretraining schemes (MoCo-v3 (Chen et al., 202 DINO-v2 (Oquab et al., 2023), and supervised learning with ImageNet (Russakovsky et al., 2015)) and architectures (ResNet (He et al., 2016) and ViT (Dosovitskiy al., 2021)). We consider sho-shot tasks consisting of N classes with K support samples and Q query samples per class (known as N-way K-shot). The goal is to classify the query samples based on the support samples. Tasks used for finetuning are constructed by samples from the training split. Each task is formed by randomly sampling 15 classes, with every class drawing or 5 support samples and 10 query samples. Target tasks are similarly constructed from the test set. We follow (Chen et al., 2021b) for multitask finetuning and target task adaptation. During multitask finetuning, we update all parameters in the model using a nearest centroid classifier, in which all samples are encoded, class centroids are computed, and cosine similarity between a query sample and those centroids are treated as the class logits For adaptation to a target task, we only retain the model encoder and consider a similar nearest centroid classifier. This multitask finetuning protocol applies to all experiments (Sections 4.1 to 4.3). We provide full experimental set up in Appendix F.
4.1 VERIFICATION OF THEORETICAL ANALYSIS
We conduct experiments on the tieredImageNet dataset to confirm the key insight from our theorem the impact of the number of finetuning tasks (M) and the number of samples per task (m).
Results.
We first investigate the influence of the number of shots. We fix the target task as a --shot setting but vary the number of shots from to 4 in finetuning, and vary the total sample size
7

---

Published as a conference paper at ICLR 2024
81.0
sa mali Nhm
80.0
79.5
Accuracy (%)
Arturay
79.0
Accuracy
75
75
78.5
78.0
Semples per lask
Number of tadks [M]
77.5
150
201
320
403
77.0
420
.20
76.5
200
600
300
600
Number of shot samples task
Number of tasks
Samples
(a) # shots during finetuning.
(b) A tasks during finetuming.
(c) samples during finetuning.
Figure 3: Results on ViT-B backbone pretrained by MoCo v3. (a) Accuracy v.s. number of shots per finetuning
task. Different curves correspond to different total numbers of samples m. (b) Accuracy v.s. the number of tasks M. Different curves correspond to different numbers of samples per task (c) Accuracy number of samples per task m. Different
correspond different numbers of tasks M.
Pretrained
Selection
INet
Omglot
Acraft
CUB
QDraw
Fungi
Flower
Sign
COCO
CLIP
Random
56.29
65.45
31.31
59.22
36.74
31.03
75.17
33.21
30.16
No Con. 60.89
72.18
31.50
66.73
40.68
35.17
81.03
37.67
34.28
No Div. 56.85
73.02
32.53
65.33
40,99
33.10
80.54
34.76
31.24
Selected 60.89
74.33
33.12
69.07
41.44
36.71
80.28
38.08
34.52
DINOv2
Random 83.05
62.05
36.75
93.75
39.40
52.68
98.57
31.54
47.35
No Con. 83,21
76.05
36,32
93.96
50.76
53,01
98.58
34,22
47.11
No Div. 82.82
79.23
36.33
93.96
55.18
52.98
98.59
35.67
44.89
Selected
83.21
81.74
37.01
94.10
55.39
53.37
98.65
36.46
48.08
MoCo v3
Random 59.66
60.72
18.57
39.80
40,39
32.79
58.42
33.38
32.98
No Con. 59.80
60.79
18.75
40.41
40.98
32.80
59.55
34.01
33.41
No Div. 59.57
63,00
18.65
40.36
41.04
32.80
58.67
34.03
33.67
Selected 59.80
63.17
18.80
40.74
41.49
33.02
59.64
34.31
33.86
Table 1: Results evaluating our task selection algorithm Meta-dataset using VYT-B backbone. No Con.: Ignore consistency, No Div.: Ignore diversity. Random: Ignore both consistency and diversity.
Mm = [10k, 20k, 40k). The results in Figure 3a show no major change in accuracy with varying the number of shots in finetuning. It is against the common belief that meta-learning like Prototypical Networks (Snell et al., 2017) has to mimic the exact few-shot setting and that mismatch will hurt the performance. The results also show that rather than the number of shots, the total sample size Mm determines the performance, which is consistent with our theorem. We next investigate the influence of M and m. We vary the number tasks (M [200, 400, 600, 800) and samples per task (m [150, 300, 450, 600)) while keeping all tasks have one shot sample. Figure 3b shows increasing M with fixed m improves accuracy, and Figure 3c shows increasing m with fixed M has similar behavior. Furthermore, different configurations of M and m for the same total sample size Mm have similar performance (e.g., .. M 400, m 450 compared to 600, m 300 in Figure 3b). These again align with our theorem.
4.2 TASK SELECTION
(Triantafillou et al., Setup. To evaluate our task selection Algorithm 1, we use the domains, each partitioned into 2020). It contains extensive public image datasets from various
train/val/test splits. For each dataset except Describable Textures due to small size, we conduct an experiment, where the test-split of that dataset used as the target task while the train-split from all the other datasets are used as candidate finetuning tasks. Each experiment follows the experiment protocol in Section 4. We performed ablation studies on the task selection algorithm, concentrating either consistency diversity, while violating the other. See details in Appendix F.4.
Results. Table 1 compares the results from finetuning with tasks selected by our algorithm to those from finetuning with tasks selected by other methods. Our algorithm consistently attains performance gains. For instance, on Omniglot, our algorithm leads to significant accuracy gains random selection of 8.9%, 19.7%, and 2.4% with CLIP, DINO v2, and MoCo v3, respectively.
8

---

Published as a conference paper at ICLR 2024
minilmageNet
ImageeelageNet
DomainNet
pretrained
backhone
method
I-shot
5-shot
1-shot
5-shot
1-shot
5-shot
MoCo v3
B
Adaptation
75.33 (0.30)
92.78 (0.10) 62.17 (0.36)
83.42 (0.23)
24.84 (0.25)
44.32 (0.29)
Standard FT
75.38 (0.30)
92.80 (0.10)
62.28 (0.36)
83.49 (0.23)
(0.15
44.76 (0.27)
Ours
80.62 (0.26)
93.89 (0.09)
68.32 (0.35)
85.49 (0.22)
32,88 (0.29)
54.17 (0.30)
ResNet50
Adaptation
68.80 (0.30)
88.23 (0.13)
55.15 (0.34)
76.00 (0.26)
27.34 (0.27)
47.50 (0.28)
Standard FT
68.85 (0.30)
88.23 (0.13)
55.23 (0.34)
76.07 (0.26)
27.43 (0.27)
47.65 (0.28)
Ours
71.16 (0.29)
89,31 (0.12)
58,51 (0.35)
78.41 (0.25)
33,53 (0.20)
55.82 (0.29)
DINO v2
VIT.S
Adapiation
85.90 (0.22)
95.58 (0.08)
74.54 (0.32)
89.20 (0.19)
52.28 (0.39)
72.98 (0.28)
Standard FT
86.75 (0.22)
95.76 (0.08)
74.84 (0.32)
89.30 (0.19)
54.48 (0.39)
74.50 (0,28)
Ours
88.70 (0.22)
96.08 (0.08)
77.78 (0.32)
90.23 (0.18)
61.57 (0.40)
77.97 (0.27)
VIT-B
Adaptation Standard FT
90.61 (0.19)
97.20 (0.06)
82.33 (0.30)
92.90 (0. 16)
61.65 (0.41)
79. 34 (0.25)
91.07 (0.19)
97.32 (0.06)
82.40 (0.30)
93.07 (0.16)
61.84 (0.39)
79.63 (0.25)
Ours
92.77 (0.18)
97.68 (0.06)
84.74 (0.30)
93.65 (0.16)
68.22 (0.40)
82.62 (0.24)
Supervised pretraining
VIT-B
Adaptation Standard FT
94.06 (0.15)
97.88 (0.05)
83.82 (0.29)
93.65 (0.13)
28.20 (0.29)
49.70 (0.28)
95.28 (0.13)
98.33 (0.04)
86.44 (0.27)
94,91 (0.12)
30.93 (0.31)
$2.14 (0.29)
on ImageNet
Ours
96.91 (0.11)
98.76 (0.04)
89.97 (0.25)
95.84 (0.11)
48.02 (0.38)
67.25 (0.29)
ResNet50
Adaptation Standard FT
81.74 (0.24)
94.08 (0.09)
65. $5.98/7 (0.34)
84.14 (0.21)
27.32 (0.27)
46.67 (0.28)
84.10 (0.22)
94.81 (0.09)
74.48 (0.33)
88.35 (0.19)
(0.31)
55.08 (0.29)
Ours
87.61 (0.20)
95.92 (0.07)
77.74 (0.32)
89.77 (0.17)
39.09 (0.34)
60.60 (0.29)
Table 2: Results of few-shot image classification. We report average classification accuracy (%) with 95% confidence intervals on test splits. Adaptation: Direction adaptation without finctuning: Standard FT: Standard finetuning; Ours: Our multitask finetuning: 1-/5-shot: number of labeled images per class in the target task.
Violating consistency or diversity conditions generally result in reduced performance compared to our approach. These results are well aligned with our expectations and affirm our diversity and consistency conclusions. We provide more ablatioin study on task selection in Table 9 in Appendix F.4. We also apply task selection algorithm on DomainNet in Appendix F.5. Furthermore, in Appendix G, we employ our algorithm for NLP models on the GLUE dataset.
4.3 EFFECTIVENESS OF MULTITASK FINETUNING
Setup. We also conduct more extensive experiments on large-scale datasets across various settings to confirm the effectiveness of multitask finetuning. We compare to two baselines: direct adaptation where we directly adapt pretrained model encoder on target tasks without any finetuning, and standard finetuming where we append encoder with a linear head to map representations to class logits and finetune the whole model. During testing, removed the linear layer and used the same few-shot testing process with the finetuned encoders. Please refer Table 14 in Appendix F.8 for full results.
Results. Table 2 presents the results for various pretraining and finetuning methods. backbones, datasets, and few-shot learning settings. Multitask finetuning consistently outperforms the baselines in different settings. For example, in the most challenging setting of 1-shot on NemainNet, it attains major gain of 7.1% % and 9.3% in accuracy over standard finetuning and direct adaptation, respectively, when considering self-supervised pretraining with DINO v2 and using a Transformer model (ViT- S). Interestingly, multitask finetuning achieves more significant gains for models pretrained with supervised learning than those pretrained with contrastive learning. For example, on DomainNet. multitask finetuning on supervised pretrained ViT-B achieves a relative gain of 67% and 35% for 1- and 5-shot, respectively. In contrast, multitask finetuning on DINO v2 pretrained ViT-B only shows a relative gain of 10% and 4%. This suggests that models from supervised pretraining might face larger domain gap than models from DINO v2, and multitask finetuning helps to bridge this gap.
5 CONCLUSIONS
In this work, we studied the theoretical justification of multitask finetuning for adapting pretrained foundation models to downstream tasks with limited labels. Our analysis shows that, given sufficient sample complexity, finetuning using a diverse set of pertinent tasks can improve the performance on the target task. This claim was examined in our theoretical framework and substantiated by the empirical evidence accumulated throughout our study. Built on this theoretical insight, we further proposed a practical algorithm for selecting tasks for multitask finetuning, leading to significantly improved results when compared to using all possible tasks. We anticipate that our research will shed light on the adaptation of foundation models, and stimulate further exploration in this direction.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Our work aims to study the theoretical justification of this multitask finetuning approach. Our paper is purely theoretical and empirical in nature and thus we foresee no immediate negative ethical impact. We quantify the relationship between multitasks and target task by diversity and consistency metrics and propose a practical task selection algorithm, which may have positive impact on the machine learning community. We hope our work will inspire effective algorithm design and promote better understanding of effective adaptation foundation models.
REPRODUCIBILITY STATEMENT
For theoretical results in the Section 3, complete proof is provided in the Appendix C. The theoretical results and proofs for a multiclass setting that more general than that in the main text are provided in the Appendix D. The complete proof for linear case study on diversity and consistency is provided the Appendix E. For experiments in the Section 4, complete details and experimental results are provided in the Appendices F to H. The source code with explanations and comments is provided in https: gá thub. www www.c///oivvee////eeeeee -ondddeeein--eeeee Multit Muiliitss itssk.
ACKNOWLEDGMENTS
The work is partially supported by Air Force Grant 119550--1-1-0166 the National Science Founda- tion (NSF) Grants 2008559-IIS, CCF-2046710, and 2023239-DMS. The work also received partial support from grants by McPherson Eye Research Institute and VCGRE at UW Madison, and from the Army Research Lab under contract number I90222200
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoretical analysis of contrastive unsupervised representation learning. 36th International Conference Machine Learning, ICML 2019. International Machine Learning Society (IMLS), 2019.
David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of Machine Learning research, 2003.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv: (tt.::200000000 07258, 2021.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. PMLR, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021a.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
10

---

Published as a conference paper at ICLR 2024
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction cestruttion-finetueed language models. arXiv preprint arXiv:2210./ 1416, 2022.
Jacob Devlin, Ming-Woe Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguisties, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learming. PMLR, 2017.
Tomer Galanti, András György, and Marcus Hutter. Generalization bounds for transfer learning with pretrained classifiers. arXiv preprint wxXiv2221..125333 2022.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing, 2021a. a.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021b.
Siddhant Garg and Yingyu Liang. Functional regularization for representation learning: A unified theoretical perspective. Advances in Neural Information Processing Systems, 2020.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tune fnn-ttunng. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
11

---

Published as a conference paper at ICLR 2024
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2022.
Timothy Hospedales, Antreas Antoniou, Paul Micaclli, and Amos Storkey. Meta-learning in neural networks: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022a.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference Knowledge discovery and data mining, 2004.
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022b.
Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. In The Eleventh International Conference Learning Representations, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenRe- view.net, 2022.
In The Tenth International
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
Xiang Lisa Li and Perey Liang, Prefix- Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11 Ith International Joint Conference Natural Language Processing. Association for Computational Linguistics, 2021.
Chen Liu, Yanwei Fu, Chengming Xu, Siqian Yang, Jilin Li, Chengjie Wang, and Li Zhang. Learning few-shot embedding model with contrastive learning. In Proceedings of the AAAI conference on artificial intelligence, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In 6th International Conference on Learning Representutions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018.
Andreas Maurer. A vector-contraction inequality for rademacher complexities In International Conference Algorithmic Learning Theory. Springer, 2016.
12

---

Published as a conference paper at ICLR 2024
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022b.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji. Khalid Almubarak, Samuel Albanie, Zaid Alyafcai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting the Association for Computational Linguistics (ACL), 2023.
Shikhar Murty, Tatsunori B Hashimoto, and Christopher D Manning. Dreca: A general task augmen- tation strategy for few-shot natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
Jianmo Ni. Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguisties: ACL 2022, 2022.
B. Olshausen and D. Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision Research, 1997.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXi::180..03748, 2018. 2018.
OpenAl. Introducing ChatGPT. https:/ http:////peaaaa.ccooaaocaaaaaaaa 2022. Accessed: 2023-09-10.
OpenAI. GPT-4 technical report. arXiv preprint arxiv:2303.08771 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193, 2023.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'OS, '05), 2005.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 2021.
13

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference Learning Representations, 2018.
Nicholas Roberts, Xintong Li, Dyah Adila, Sonia Cromp. Tzu- Huang, Jitian Zhao, and Frederic Sala. Geometry-aware adaptation for pretrained models. arXiv preprini arXiv:2307. 12226,, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations, 2022.
Zhenmei Shi, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha.
The trade-off between universality and label efficiency of representations from contrastive learning. In International Conference on Learning Representations, 2023a.
Zhenmei Shi, Yifei Ming, Ying Fan, Frederic Sala, and Yingyu Liang. Domain generalization via nuclear norm regularization. In Conference on Parsimony and Learning (Proceedings Track), 2023b.
Zhenmei Shi, Junyi Wei, and Yingyu Liang. Provable guarantees for neural networks via gradient feature learning. In seirth- Conference Neural Information Processing Systems, 2023c.
Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? In RQ-FooM:Robuuttess of Few-shot and Zero-shot Learning in Large Foundation Models, 2023d.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013.
Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Yiyou Sun, Zhenmei Shi, and Yixuan Li. A graph-theoretic framework for understanding open- world semi-supervised learning. In Thirth-seventh Conference on Neural Information Processing Systems, 2023a.
Yiyou Sun, Zhenmei Shi, Yingyu Liang, and Yixuan Li. When and how does known class help discover unknown ones? Provable understanding through spectral analysis. In Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research.
PMLR, 2023b.
14

---

Published as a conference paper at ICLR 2024
Yonglong Tian, Dilip Krishnan, and Phillipf Isola. Contrastive multiview coding. In European Conference on Computer Vision. Springer, 2020a.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference Computer Vision. Springer, 2020b.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory. PMLR, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXin:2307.092288 2023.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Reprexentutions, 2020.
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory transfer learning: The importance of task diversity. Advances in Neural Information Processing Systems, 2020.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning. PMLR, 2021.
William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 2000.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 2016.
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 2000.
Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit lyyer. Strata: Self-training with task augmentation for better few-shot learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguisties, 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning. PMLR, 2020.
Yaqing Wang, Quanming Yao. James Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys, 2020.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2022.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multi-
task prompt tuning enables parfmeter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, 2023.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks unlabeled data. In International Conference on Learning Representations, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities large language models. arXiv preprint arX6v:2206.0766 2022a.
15

---

Published as a conference paper at ICLR 2024
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. th-tout prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022b.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning. PMLR, 2021.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 2005.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Perey Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169 2023.
Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Yin Li, and Yingyu Liang. Improving foundation models for few-shot learning via multitask finetuning. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. Do large language models have compositional ability? an investigation into limitations and scalability. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https: : / /pw.neveeeeeeeeeeeeooeeeee / forum? d=4XPe70SbJs
Zhanyuan Yang. Jinghua Wang, and Yingying Zhu. Fhw classification with contrastive learning. In European Conference on Computer Vision. Springer, 2022.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint 2rXiv:2303.16199 2023.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations, 2020.
Yulai Zhao, Jianshu Chen, and Simon Du Blessing of class diversity in pre-training. In International Conference Artificial Intelligence and Statistics. PMLR, 2023.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 2022b.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning. PMLR, 2021.
16

---

Published as a conference paper at ICLR 2024
CONTENTS
1 Introduction
1.1
Related Work
2 Background: Multitask Finetuning for Few-Shot Learning
3 Theoretical Analysis: Benefit of Multitask Finetuning
3.1 Case Study of Diversity and Consistency
3.2 Task Selection
Experiments
4.1
Verification of Theoretical Analysis
4.2
Task Selection
4.3
Effectiveness of Multitask Finetuning
5
Conclusions
A Limitation
19
B More Related Work
19
Deferred Proofs
20
C.I Contrastive Pretraining
22
C.1.1
Contrastive Pretraining and Direct Adaptation
22
C.1.2
Contrastive Pretraining and Multitask Finetuning
24
C.2 Supervised Pretraining
27
C.2.1 Supervised Pretraining and Direct Adaptation
27
C.2.2 Supervised Pretraining and Multitask Finetuning
................................
C.3 Masked Language Pretraining
30
C.4 Unified Main Theory
30
C.5 Bounded Task Loss by Task Diversity
31
D Multi-class Classification
31
D.I Contrastive Pretraining
32
D.2
Supervised Pretraining
33
E
Linear Case Study
33
E.1
Problem Setup...
34
E.2
Diversity and Consistency Analysis
35
E.2.1
Optimal Representation for Multitask Finetuning
35
E.2.2
Consistency
37
17

---

Published as a conference paper at ICLR 2024
E.2.3
Diversity
37
E.3
Proof of Main Results
39
F
Vision Experimental Results
39
F.1
Datasets
40
F22
Experiment Protocols
40
F3
Existence of Task Diversity
E33.1
minilmageNet and Omniglot
41
F33.2
tieredImageNett
F.4
Ablation Study
42
E4.1
Violate both consistency and diversity: Altering Finetuning Task Data with Invariant Sample Complexity
42
F.4.2
Violating consistency while retaining diversity: The Trade-Off between Task Consistency and Sample Complexity
42
E443
Diversity and Consistency of Task Data and Finetuning Methods
43
F.4.4
Ablation Study Task Selection Algorithm
44
F5
Task Selection Algorithm on DomainNet
F.6
More Results with CLIP Encoder
44
F7
Sample Complexity on Performance for tieredlmageNet
46
F88
Full results for Effectiveness of Multitask Finetuning
46
NLP Experimental Results
48
G.1
Summary
48
G.2 Datasets and Models
49
G.3 Experiment Protocols
50
G.4 Task Selection
50
G.4.1 Full Results with Task Selection
52
G.4.2
Additional Results on simCSE
H Vision Language Tasks
52
H.1
Improving Zero-shot Performance
53
H.2
Updating Text Encoder and Vision Encoder
54
H.3
CoCoOp
54
18

---

Published as a conference paper at ICLR 2024
Appendix
In this appendix, we first state our limitation in Appendix A. Then, we provide more related work in Appendix B. The proof of our theoretical results for the binary case presented in Appendix C, where we formalize the theoretical settings and assumptions and elaborate on the results to contrastive pretraining Appendix C. and supervised pretraining in Appendix C.2. We prove the main theory in Appendix C.4, which is direct derivative of C.1 and C.2. We generalize the setting to multiclass and provide proof in Appendix D. We include the full proof of the general linear case study in Appendix E. We provide additional experimental results of vision tasks in Appendix F, language tasks in Appendix G, and vision-language tasks in Appendix H.
A LIMITATION
We recognize an interesting phenomenon within multitask finetuning and dig into deeper exploration with theoretical analysis, while our experimental results may or may not beat sfate-of-the-ard (SOTA) performance, our focus is not presenting multitask finetuning as novel approach nor on achieving SOTA performance. On the other hand, the estimation of our diversity and consistency parameters accurately on real-world datasets is valuable but time-consuming. Whether there exists an efficient algorithm to estimate these parameters is unknown. We leave this challenging problem as our future work.
B
MORE RELATED WORK
Training Foundation Models. Foundation models (Bommasani et al., 2021) are typically trained using seff-superviee learning over broad data. The most commonly used training approaches include contrastive learning in vision and masked modeling in NLP. Our theoretical analysis considers both approaches under unified framework. Here we briefly review these approaches.
self-supervised setting, aims to group randomly augmented versions of the same data point while distinguishing samples from diverse groups. The success of this approach in vision and multi-modal training tasks (Oord et al., 2018; Chen et al., 2020; He et al., 2020; Tian et al., 2020a; Grill et al., 2020; Radford et al., 2021) has spurred consideaale interest. Several recent studies (Arora et al., 2019; 9;; HaoChen et al., 2021; Tosh et al., 20 Zimmermann et al., 2021; Wei et al., 2021; Wang & Isola, 2020; Wen & Li, 2021; Wang et al., 2022: Shi et al., 2023a; Huang et al., 2023; Sun et al., 2023b;a) seek to develop its theoretical understanding. Arora et al. (2019) established theoretical guarantees on downstream classification performance. HaoChen et al. (2021) provided analysis on spectral contrastive loss. Their analysis assumes the pretraining and target tasks share the same data distribution and focus on the effect of contrastive learning on direct adaptation. Our work focuses on the novel class setting and investigates further finetuning the pretrained model with multitask to improve performance.
Contrastive learning,
Masked modeling seeks to predict masked tokens in an input sequence. This sepf-sueeee approach the foundation of many large language models (Devlin et al., 2019; Liu al., 2019; Chowdhery et al., 2022; Ni et al., 2022; Touvron et al., 2023), and has been recently explored in vision (He et al., 2022). In the theoretical frontier, Zhao et al. (2023) formulated masked language modeling as standard supervised learning with labels from the input text. They further investigated the relationship between pretrained data and testing data by diversity statement. Our work subsumes their work as a special case, and can explain a broader family of pretraining methods.
Adapting Foundation Models. Adapting foundation models to downstream tasks has recently received significant attention. The conventional wisdom, mostly adopted in vision (Vinyals et al., 2016; Ge & Yu, 2017; Chen et al., 2020; He et al., 2020; 2022; Shi et al., 2023b), involves learning a simple function, such as linear probing, on the representation from a foundation model, while keeping the model frozen or minorly finetuning the whole model. In NLP, prompt-based finetuning (Gao et al., 1a; Hu et al., 2022a; Chung et al., 2022; Song et al., 2022; Zhou et al., 2022b; Xie et al., 2023; Zhang et al., 2023) was developed and widely used, in which prediction task is transformed into a masked language modeling problem during finetuning. With the advances in large language models, parfmeter-efffiient tuning has emerged attractive solution. Prompt tuning (Lester et al.,
19

---

Published as a conference paper at ICLR 2024
2021; Li & Liang, 2021; Roberts al., 2023) learns extra prompt token for new task, while updating minimal or no parameters in the model backbone. Another promising approach is in-context learning (Min et al., 2022b; Wei et al., 2022a;b; Shi et al., 2023d; Xu et al., 2024), where the model is tasked to make predictions based on contexts supplemented with a few examples, with no parameter updates. In this paper, we consider adapting foundation models to new tasks with limited labels. Parameter-efficient tuning, such as in-context learning, might face major challenges (Xie et al., 2022) when the distribution of the new task deviates from those considered in pretraining, Instead, our approach finetunes the model using multiple relevant tasks. We empirically verify that doing so leads to better adaptation.
Multitask Learning. Multitask supervised learning has been considered for transfer learning to target task (Zhong et al., 2021; Sanh et al., 2022: Chen et al., 2022; Min et al., 2022a; Wang et al., 2023). Multitask has been shown to induce zero-shot generalization in large language models (Sanh et al., 2022), and also enable parameter efficient tuning by prompt tuning (Wang et al., 2023). Our work leverages multitask learning unlock better zero-shot and few-shot performance of pretrained models. Min et al. (2022a); Chen et al. (2022) primarily focus on in-context learning, Zhong et al. (2021) focuses on the idea of task conversion where transfer classification task as question-answer format, our approach is based on utilizing original examples, in alignment with our theoretical framework. A line of theoretical work provides the error bound of the target task in terms of sample complexity (Du et al., 2021: Tripuraneni et al., 2021; Shi et al., 2023a; Xu et al., 2023). Tripuraneni et al. (2020) established a framework of multitask learning centered around the notion of task diversity for the training data. Their work mainly analyzed representations from supervised pretraining using multitasks. contrast, our work considers representations from self-supervised pretraining, and focuses on multitask finetuning. Our approach and analysis guarantee that limited but diverse and consistent finetuning task can improve the prediction performance on a target task with novel classes.
Few-shot Learning and Meta Learning. Fhw-shot learning necessitates the generalization to new tasks with only a few labeled samples (Wang et al., 2020; Vu et al., 2021; Murty et al., 2021; Liu al., 2021; Yang et al., 2022: Galanti al., 2022). Direct training with limited data is prone to overfitting. Meta learning offers promising solution that allows the model to adapt to the few-shot setting (Finn al., 2017: Raghu et al., 2020). This solution has been previously developed for vision tasks (Vinyals et al., 2016; Snell et al., 2017: Chen et al., 2021b; Hu et al., 2022b). Inspired by meta learning in the few-shot setting, our analysis extends the idea of multitask finetuning by providing sound theoretic justifications and demonstrating strong empirical results. We further introduce task selection algorithm that bridges our thcoretical findings with practical applications in multitask finetuning.
C DEFERRED PROOFS
In this section, we provide a formal setting and proof. We first formalize our setting in multiclass. Consider our task T contains r classes where r 2 2.
Contrastive Learning. In contrastive learning, we sampled one example from any latent class y. then apply the data augmentation module that randomly transforms such sample into another view of the original example denoted x We also sample other examples {- 1k=1 from other latent classes {yx |-=]- We treat (I,x as a positive pair and (z, x;)) as negative pairs. We define Dewa (n) over sample (x, IT. .:,1) by following sampling procedure
(u. "U.. vr-1) ~' D(y), x+ ~ D(y), zx ~ D(VE -1..,,.
(5)
(6)
We consider general contrastive loss lu (fo(=) (d(xt) -(x=t)-(<==)))
where loss func-
tion ly is non-negative decreasing function. Minimizing the loss is equivalent to maximiz- ing the similarity between positive pairs while minimizing it between negative pairs. In particular, logistic loss fu(v) = log (1 + Σ; exp (-u:)) for v E gr-1 recovers the one
exp (op To(=!))
used in most empirical works: - log expl + exp (d(x) (cx ) The popula-
20

---

Published as a conference paper at ICLR 2024
tion contrastive loss is defined as Cron-pre(0) = E Ca (fo(x) (6(xt) ò(ズ ()
denote our contrastive training set with Let Scon-pre = { TTs 2f Lii , 11r-1)) j=1
N
N samples, sampled from Doou(1), (n), we have empirical contrastive loss Lcon-pre(0) == N E Lu (fo(x)T (o(xt) ((x)))
Supervised Learning. In supervised learning have labeled dataset denoted as Scon pre {2g.0s1 Uo with N samples, by following sampling procedure:
y
(y))
There are in total K classes, denote C as the set consists of all classes. On top of the representation
function o, there is a linear function f E F c {1 {Rr + Rx} predicting the labels, denoted as g(x) = f (xx). We consider general supervised loss on data point (x, y) is
E(g(z),)) v) == lu (((())) (g(x)) (u.
(9)
where loss function ly is non-negative decreasing function. In particular, logistic loss (,) log (1 + >; exp (-v)) for E IR R recovers the one used in most empirical works:
((((x),)) = lu ((9(z)) (g(z))
(10)
K
= log
1
exp (- [(((x)) - (o(2))al) (
(11)
k
exp(g(x)),
log
(12)
K Ek=1 exp(g(x))
The population supervised loss is
Loup- pre (0) min E [ (f o(x),
(13)
fEF
For training set Ssup-pre pre (Ii, W with N samples, the empirical supervised pretraining loss is Leup-pre pre (Ø) == min
1 N En [f o o(z;), v.)]
Masked Language Modeling. Masked language modeling is self-supervised learning method. can be viewed as specific form of supervised pretraining above. The pretraining data is substantial dataset of sentences, often sourced from Wikipedia. the pretraining phase, a random selection of words is masked within each sentence, and the training objective is to predict these masked words using the context provided by the remaining words can be viewed as multi-class classification problem, where the number of classes (denoted as K) corresponds to the size of the vocabulary. Considering BERT and its variations, we have function Ø text encoder. This encoder outputs a learned representation, often known as [CLS] token. The size of such learned representation is d, which is 768 for BERT or 1024 for BERTLARGE.
sentence. This particular pretraining task
Supervised Tasks. Given representation function dh we apply a task-specific linear transformation W to the representation to obtain the final prediction. Consider r-way supervised task T consist a of distinct classes (y1...., yr-) S C. We define D (y) as the distribution of randomly drawing
E (BI,...,We Vr), we denote this process as ~ T. Let ST = {2,, 3/ y denote our labeled training set with samples, sampled i..dd from y, T and z₁ ~ D Define g(o(x))
W Wo(z) E 12" as prediction logits, where W E Rrxd The typical supervised logistic loss is ((goo(z),)) = tu (Lo()())) g(o(x)) ) Similar to Arora et al. (2019), define supervised loss w.r.t the task
(T,)) Coup(T, WeR yuT
Loup
min E
E x>D(w)
[e(W 6(z).
(14)
21

---

Published as a conference paper at ICLR 2024
Algorithm 2 Multitask Finetuning
Input: multitasks , TM. pretrained model & with parameter 0, step size Y
1: Initialize & with é
2: repeat
3: for all T. do
4: 0 - 0 Vo Csup (Ti,
( (Crup(Te b) is defined in (2))
5:
end for
6:
until converge
Output: The final model, denoted as of
Define supervised loss with mean classifier as C (T, ,0) == E [e(W 6(x), y)] where y z-D(u) ye == Pye = (o(z)), = ......r. r. In the z-
each row of WV is the mean each class in T, W
target task, suppose we have r distinct classes from C with equal weights, Consider T follows E (Cuup (T,o)] o)].
general distribution Ç. Define expected supervised loss as Loup(0) T~
Mutlitask Finetuning. Suppose we have M auxiliary tasks {Ti, T2, ... TM each with m labeled samples S; = ((r), ut) : j [m]]]. The finetuning data are S = Ui Given pretrained
model G we further finetune it using the objective:
M Ccup((T,,)) where Esup (su = min wn, ERd m
min ΦE
Σ M
Σ E(w! (())). j=l
(15)
2m
This can be done via gradient descent from the initialization & (see Algorithm 2).
Algorithm 2 has similar pipeline as Raghu et al. (2020) where in the inner loop only linear layer on top of the embeddings is learned. However, our algorithm is centered on multitask finetuning, where no inner loop executed.
Finally, we formalize assumption Assumption below.
Assumption 2 (Regularity Conditions). The following regularity conditions hold:
(A1) Representation functionn satisfies ||öl12 S R.
(A2) Linear operator W satisfies bounded spectral norm [[]]]2 B.
(A3) The loss function ly are bounded by [0, C) and {(-) T-lipschitz.
(A4) The supervised loss S is T--Lipschitz with respect to b for VT.
C.1 CONTRASTIVE PRETRAINING
In this section, we will show how multitask finetuning improves the model from contrastive pretraining. We present pretraining eπro in binary classification and Dy as uniform. See the result for the general condition with multi-class Appendix D.
C.1.1 CONTRASTIVE PRETRAINING AND DIRECT ADAPTATION
In this section, we show the error bound foundation model on target task, where the model is pretrained by contrastive loss followed directly by adaptation.
We first show how pretraining guarantees the expected supervised loss:
Lsup = E T~s [Lsup (T,up(TT,)) (T,+).
(16)
The error on the target task can be bounded by Lsup (6)) We use et denote Exmp(d2)
Lemma C.1 (Lemma 4.3 Arora et al. (2019)). For Vo E pretrained in contrastive loss, we have Les (0) - (Lron proennppee(( (Ø)
22

---

Published as a conference paper at ICLR 2024
We state the theorem below.
Theorem C.2. Assume Assumption and that + has v-diversity and k-consistency with respect
,0,0% oz Suppose & satisfies Ccon-pre S Eq. Let T
Pr {yı y23. Consider pretraining
set Spon-pre {Izz 25 3 1 j=1 For any 8 2 0. if
1 6o
80
log( 2)1
2
N
8IRRN (T)
cu
Then with probability for any target task To C Co.
1 Ssupl (To 3) Loup Φ²)
v 11 - (200 -)- Loup(d") +
(17)
The pretraining sample complexity is 0 Rx((2) + log(1/6)) 63
). The first term the Rademacher complexity of the entire representation space & with sample size N. The second term relates to the generalization bound. Pretraining typically involves a vast and varied dataset, sample complexity is usually not significant concern during this stage.
N Proof of Theorem C.2. Recall in binary classes, Scon = {z 2 ا= denote our contrastive training set, sampled from Deon (1). Then by Lemma A.2 in Arora al. (2019), with (A1) and (A3), we have for Vo E & with probability 5,
4LRRN 4
log 1
Ccon-pre (ro)- S
+
+C
(18)
N
N
To have above & 6p. we have sample complexity
8C2
2 ])
N z
8LRRN +
log(
In pretraining, we have à such that
Pcon pre pre S 60-
Then with the above sample complexity, we have pretraining
Lcom-pre (x) S 200-
Recall v-diversity and k-consistency, for target task Tos we have that for & and Φ²,
Loup 3) - Ceup((o x²) = Coup (To, Loup ot) + Ssup (To, o2)
(To Φ²)
(19)
S de (2, x2) + Lsup (To, ot) f. (To, (3¹)
(20)
< dØ 62)/v /VvV/ +
(21)
< V
(sup(t) Loup (o0) +
(22)
(Loon (Ccon-pre (oonn-pre(0)) (0) -T) - + k
(23)
<
(250 - 7) -6" + +
(24)
where the second to last inequality comes from Lemma C.I.
n
23

---

Published as a conference paper at ICLR 2024
C.1.2 CONTRASTIVE PRETRAINING AND MULTITASK FINETUNING
In this section, we show the error bound of foundation model on a target task can be further reduced by multitask finetuning, We achieve this by showing that expected supervised loss Ssup can be further reduced after multitask finetuning. The error on the target task can be bounded by Sup (o). We use e* denote (
Following the intuition in Garg Liang (2020), we first re-state the definition representation space.
Definition 3. The subset of representation space is
(()) o + Lyre se.
Recall S - ((x), uf) : [M] [m]] as finetuning dataset.
We define two function classes and associated Rademacher complexity. Definition 4. Consider function class
Ge(E) (1w v) gw,al., v) ( s B).
We define Rademacher complexity as
E R. (G.((E))) (0.)
sup (EGx(2)
o e(W ((T)).
Definition 5. Consider function class
G( {gs (T) = Ceup
(T, Φ(2))
We define Rademacher complexity as
(gr()()))
E 1o
sup (F)(F)
o, SuS (Tr, Φ)
The key idea is multitask finetuning further reduce the expected supervised loss of pretrained foundation model d:
sup (1) = E Tnp
(Ssup (T, d)]
(25)
We first introduce some key lemmas. These lemmas apply general classes in task T. Lemma C.3 (Bounded Rademacher complexity). By (A2) and (A3), have for Vn
RA, R9.(9( S i1r-11.
Proof of Lemma C.3. We first prove ((g(6(x)) y) is 1)..B with respect to ş for f(((((x)) = -(( By
all Vy E C. Consider
where fy RR + -R Note that
l(g ((z))())))) el (19(6(x)), o(((x))) ( by+v) =[(f((((((x)))
By (A3), we have & is Lipischhiz. We then prove fy /2(r - Li-iscchitz. Without loss generality,
consider = r. We have fy()) = [yr - w We have at. - liij = i), = 1,... 1,
æf: = 1. The Jacobian J satisfies [|]]2 < ||/|| (2) dr
Since is B-Lipschitz by (A2): 1|2 < B. Then ((g()(x)) y) is - 1) with
respect for all Vy C. The conclusion follows Corollary 4 in Maurer (2016).
24

---

Published as a conference paper at ICLR 2024
Lemma C.A (Bounded é). After finite steps in Multitask filttnaskknnttunngn in Algorithm 2, solve Equa- tion (2) with empirical loss lower than 61 = 3 I-r (2e0 - T) and obtain 6' Then there exists a bounded é such that &' E (()).
Proof of Lemma C.4. Given finite number of steps and finite step size ? in Algorithm 2, we have bounded || - 5lI. Then with (A2) and (A3), using Lemma C.3 we have ((g()(x)), y) is /2(r - L) Liipschizzz with respect to o for all Vy, using theorem A.2 in Arora et al. (2019) we have Lu is LC-Lipschitz with respect to 6, we have (pre(d) is M M-Lipschitz with respect to with bounded M. We have 3 e such that (0rr Epre(() S ellø' - ll|- We have (ora( S 60 + <llØ' Tl-
Take ē = 60 + elløo øl yields the result.
Lemma C.5. Assume Assumption and that t has u-diversity and R-consistency with respect to 6, 66. Suppose for some small constant E (0, 1) and we solve Equation (2) with empirical loss
lower than 6 - (2E0- - T) and obtain &'. For any 5 > if
1 AV2iR (0(E))
4C²
(o(())
1
8vr ILBR (B(E)) + 4C2
log( 2)
M 2 61
.M 2
61 then expected supervised loss lou S a 11-1(20)))
t1
61
T), with probability 1 1- 8. 1.
Proof of Lemma C.5. Recall S = {(x)) vz) : [M] [m]] as finetuning dataset. Consider in
Equation (2) we have W == (W,,.. W M) and d' such that tे E 1 m E3= C(W, o'(z)) vf) co-
We tried to bound
m
Ceup (0')
e(W o (z)),
m
Recall that
(Tup()
min
E
[e(W 6(x),))
WeR y-T,
x>D(2)
For Vé (()
[e (W ((x)) vi]
Loup(0) ETon [Csup (T, [0]]= ETas
min E W WE y
E z>D(y)
We have for Vó E (2),, by uniform convergence (see Mohri et al. (2018) Theorem 3.3), we have with probability 1 - 8/2
1 M
(9(E)
log(2/6)
ETn, (T
(T,
Σ
Coup (T,, o) S
(26)
M
M
M
i=1
2
log (2/()
(27)
M
M
where the last inequality comes from (A4) and Corollary 4 in Maurer (2016). To have above S / we have sample complexity
M 2 A22iR + 4C²
-ooo
G
Then we consider generalization bound for Vó and W = (W,.... WM)
M
1
Co W) M
E
E
l ((, d(x'), )'
(28)
z' DDy')
M
m
Loup(o, W)
(W; ((a)).1))
(29)
M
m
j=1
25

---

Published as a conference paper at ICLR 2024
where W = (W1,.... Was).
By uniform convergence (see Mohri et al. (2018) Theorem 3.3), we have with probability 1 - 5/2, Louple, W) Lsp w)
2R (Ge)
log(2/0)
8 8vr LLBRMm (*(EE) +C
log log(2/))
Mm
Mm
Mm
Mm
where the last inequality comes from Lemma C.3. To have above S f we have sample complexity
1 8Vr (m + log 8
40
Mm 2
satisfying Vé ((E)
M
M
Suu (Te, Ø)
Σ
min
E awD(ly)
[W 6(x),
M
M
WeR xd T
M
[e (W, (6x))()))))) v))
E z>D(y)
M
Lsup W)
Lo W) + c1/2.
Then combine above with Equation (26)
Ssup(0)
E Tnc
SSuup( (T.
< Lou W) +11.
We have
1 m
( m j=1
Σ
e(W; ('(x),, w/) Sa
(o') 2E1 S (2e0 T).
The boundedness of é follows Lemma C.4.
We state the theorem below.
Theorem C.6. Assume Assumption and that & has v-diersiity and k-consistencyy with respect to
6², 1b. Suppose for some small constant a E (0, 1), we solve Equation (2) with empirical loss lower
than 61 38 -2eo - 7) and obtain d'. For any 8 > 0, if
M 2 (
1
4C2
log(
2 8
.m 2
1
ML (T(2)) +
4C2 log( 2 58)
6
6
61
then with probability - for any target task To C
Co.
1
Lsuu Φ0) Saup (To, $²) S 0
(200 - T) -csup((t)
+ k.
(30)
Proof of Theorem C.6. Recall with v-diversity and k-consistency with respect to o², or. for target task To. we have that for ø' and o²
Csup(To. Φ6) Loup (To, Φ²) Cuup(Tor (To, Φ') - -Lsu (To: (2) + Loup x²) - Coup x*)
< do (())(())))))))))) + n
v
1
v [supp())) - [csup((b2)]] + n.
a 1 - (260 - r) -c- + Ki
where the last inequality comes from Lemma C.5, where taking r 2.
26

---

Published as a conference paper at ICLR 2024
C.2 SUPERVISED PRETRAINING
In this section, we will show how multitask finetuning improves the model from supervised pretraining. We present pretraining eπror in binary classification and DT as uniform. See the result for the general condition with multi-class Appendix
C.2.1 SUPERVISED PRETRAINING AND DIRECT ADAPTATION
In this section, show the error bound of foundation model target task, where the model is pretrained by supervised loss followed directly by adaptation. For general Let p; == Pr {v
you
J. where Σ =
Lemma C.7. Suppose y ~ 1 and S Pr (y = yi S u. Consider a task T containing r classes, y-u
which is subset of the total class set C. We have VΦ E o
Coup(d) & (7) Leup-pre (0),
where
Coup-pre(d)
min E [E (f v)z), FEF 2,21
(31)
Proof of Appendix C.2.1. We first prove where T = {11 82, Val. Then in supervised
pretraining, we have:
Laup- pre(()
(Ø) = min
E
(((
(32)
IEF yur zny
Let f = (fi, Ta) be the best linear classifier on top of o, the prediction logits are g(x) = Jo ((x) = (1))))) Then have:
exp(gi (x))
E zmyn
|e (g (()), log
E exp(Ox(x))
We let (x) = exp(gn(2)), 2,3. Then
Caup-pre(()
(x)
log (2(x
ya (a(x))
p1 E
log
+ P2 E
+ P3
E awaas
log
xm
Σi= B/ (x)
awwzy
Σx 1= Un (x))
Σ =k=n k= (1(())
3 Di=1 yk (x)
Σk=1 31
Σim (x(
=P E
log
+ P2 E
log
+
E 22000
log
zm-gi
81 (z)
(2
y2(xr)
Recall
(Tup(T,
min E y
E D>D())
[e (w <(xx.v)]
(33)
Consider
Cup (T, == E
E D~D(v)
[e(w (((x)))))1)) o(x).)))
(34)
yot
27

---

Published as a conference paper at ICLR 2024
where w is the corresponding sub-vector of f according task (for e.g., {11, 32 Then we have
(f1, f2) if if
Su (T,
PIP2
1
E prwy1
log
311 (1))
+ E
log
y2(2x)
P1P2 + PiPs + P2Ps
2
1/1 (2) + (2(())
y1 (I) + y22
PIP3
1
IE
log
(1)
+ E
log
Va(z)
P1P2 + P1P3 + P2P3
2
3/1 (2) + (1)
y1 (x) + Va
P2P3
1
(2(21)
ya(a)
IE
log
E
log
P1P2 + PiPs + P2P3
2
1/2 (2 + v)(
y2 + ys (3))
P1P2
1
31 (I) + (2(2)
(z) + (2
E
log
+ E
log
P1P2 + PiPs + P2Ps
2
y1 (x)
(2(x)
PIP3
1
31 (I) + (z)
U1 (2) + (a
E
log
+ E
log
P1P2 + P1P3 P2P3
2
1/ (x)
(2(2))
PePs
1
log (2(x)) + y3(2))
(2(2) + (z(z))
E
+
E
log
P1P2 + PiPs
P2Ps
2
(2)
xa
y2(2)
By observing the terms with y1 (I) as denominator (similar y2(x)) vz(x)), we want prove:
2
PiP2 + PIPs
P1
2
2
PIP2 + PiPs P2P3
This obtained by (4) 2 3
We have
2 Sap (T,0) S (7)² Leup Crup-pre(0).
For the general K-class setting, we follow similar steps, we have
Csup- pre (0) = Σ
y (x) Ex=1 ()((r)
E z0wgs
log
We denote J as all possible product of ps .ki....,,,pj..... J tpi P... J. Similarly, we have
1 LieT
(x(
(T, x) (T)
E
log
xm iT
Ejer (x(
where T are all tasks with classes. By observing, inside the summation there are in total (K-1) terms with yi (I)) the numerator, where corresponding probability is
Pi IliET, Pt
J
where each term can be upper bounded by - -()" P1 E z2yys
(log paiee) vi(a) (e(()).)) 5x ve(m)) (similar as y, e
T).
We state the theorem below.
Theorem C.8. Assume Assumption and that de has v-diversity and x-consistency with respect to
oz. Suppose à satisfies Csup-pre(d) I co. Let pi
Pr
yy vi where ΣK 2i= Pi = 1. Let
y-
p max, Ps, Consider pretraining set Saup-pre == (xi, yi) 1 for any 8 2 0, if
min, Ps
802 log 8
(2)
N 1 SLRVKRN (D) +
Then with probability 1 - 8, for any target task To C Co.
Lsup Φ) Loup (To, 0²) S 12p²sr cu c*l + k.
(35)
28

---

Published as a conference paper at ICLR 2024
Proof of Theorem C.8. The proof follows similar steps in Theorem C.2. For supervised pretraining, the sample complexity is similar to Theorem C.2, note that there is extra VK term. We show how we have this term below:
Consider function class
G. (aw, y) gwiola, y) = e(W T v5), (z)))
2 11 s B}.
The Rademacher complexity is
Rr (9t) 11.3611
sup Σ EE9, j=1
Then from Lemma C.3, the pretraining is a large task with classification among K classes.
(9e) S (V
Then by Theorem 3.3 in Mohri et al. (2018), with (A1) and (A3), we have for Vél E & with probability
1 6,
puup-pre (6) Crup-pre (0) (LRVKRN(O)
log
CV
(36)
N
N
To have above I co. we have sample complexity
802
2
N
(LRVKRN(0)+ + 6n
log(
en
With the above sample complexity of Saup-pre = fri, Vi Ve we have pretraining e
Cuup- (0) S 200
Recall r-diversity and k-consistency, with respect to o², 66. for target task To. we have that for and Ø",
Csup (T. - Loup (To, x²) _ do, + Loup (To, o7)
-Laup x²))
(37)
$ de (b, 2b2) u+v
(38)
1 Ceup (6) feou (15)) +
(39)
V
1
< Laup-pro E" +
(40)
$ [2p² eo e¹] +
(41)
where the second to last inequality comes from Lemma C.7.
C.2.2 SUPERVISED PRETRAINING AND MULTITASK FINETUNING
In this section, we show the error bound of a supervised pretrained foundation model on target task can be further reduced by multitask finetuning. We follow similar steps in Appendix C.1.2. Recall Definition 3, similar to Lemma C.5, we introduce the following lemma under supervised pretraining loss.
Lemma C.9. Assume Assumption and that &o has (v, c)-diversity for Ç and Co. Suppose for some small constant a (0, 1), we solve Equation (2) with empirical loss lower than E = o 2pp2e en and obtain ø' For any 8 > 0, if
402 108(2) Mm é1
2C
M > 61
(V(E)
16LBRMM (+(E)) + log {
61
then expected supervised loss Loup S 20 with probability 1-6. 6.
29

---

Published as a conference paper at ICLR 2024
Proof of Lemma C.9. The steps follow similar steps in Lemma C.5.
We state the main theorem below.
Theorem C.10. Assume Assumption and that t has (v, c) d-iiversyty for ( and Co. Suppose for some
small constant (0, 1), we solve Equation (2) with empirical loss lower than C 22p² co and obtain &'. For any 8 > 0, if
4 (D(E)) + 4C² M 2 él
1
log(
. 2 J
4C 1mLBRMM( + log 5)
E1
then with probability - for any target task To S Co.
Suu (Tu, x¹) CTup x²) - (2ap²co L +
(42)
Proof of Theorem 1. Recall u-diversity and x-consistency, with respect to o² .66 for target task To, we have that for & and o",
Ceu (T,, LTo (To s dc, (To Laup o*)
(43)
I d'(o', .6))/v + +
(44)
1
y [Luup() Ssup (dt) +
(45)
1
$ (2ap² co - -(") +
(46)
where the last inequality comes from Lemma C.9.
C.3 MASKED LANGUAGE PRETRAINING
The theoretical guarantee in masked language pretraining follows the same error bound in supervised pretraining, with K representing the size of the vocabulary.
C.4 UNIFIED MAIN THEORY
We now prove the main theory below. We first re-state the theorem.
Theorem 3.1. (No Multitask Finetuning) Assume Assumption and that T has v-diversity and consistency with respect to 6² and oz. Suppose & satisfies Eyre(Ø) S cv. Let ==
Pr
{31
(21.
v2). Then for any target task To C Co.
1
2ed
(To 8) Esup((Tr d0) 1
+ k.
Proof of Theorem 3.1. The result is a direct combination of Theorem C.2 and Theorem C.8.
Theorem 3.2. (With Multitask Finetuning) Assume Assumption 1 and that & has v-diversity and k-consistency with respect to o² and Φ². Suppose for some constant a (0, 1), we solve Equation (2)
with empirical loss lower than c 1- and obtain d' For any 5 > 0, for ē (pr)
M > 1 AV2IRA((((8) 402 logl [ Mm > 6
1 é1
16LBR (o()) + logl 2
2C
61
then with probability 1 - for any target task To C C
2eD
Leup (To, 0²) - Loup Φ²) <
a
(oup((tt tR.
v
1 1-7
30

---

Published as a conference paper at ICLR 2024
Proof of Theorem 3.2. Follow the similar steps in proof of Lemma C.5, we have
2p² co-
Laupld) - 2c, S a
Recall v-diversity and k-consistency, with respect to d*, oz for target task To, we have that for d' and 6²
Cou (') - Laup(To;
(To, x3) ) S dc (66,6)))) Csup(To: +32
Laup 6*)
(47)
< do( .6///v v +
(48)
1
1 V/
(uup - [(mp((())] + n
(49)
2p²
v "/
a1 1
c0 E"
+ k.
(50)
The sample complexity of finetuning depends on é Eyre (Ø'). Below we show that can be upper bounded in finite step finetuning.
Lemma C.11 (Bounded E), After finite steps in Multitask finetuning in Algorithm 2, we solve Equation (2) with empirical loss lower than 61 = 0 1 (260 - 7) and obtain Φ' Then there exists bounded E such that ø' E (()).
Proof of Lemma C.II. Given finite number of steps and finite step size y in Algorithm bounded 1 - -àll- Then with (A2) and (A3), using Lemma C.3 and lemma A.3 in Arora al. (2019), we have Cyre(() M-Lipschitz with respect to & with bounded M. We have 3 e such that (Ør) Eyre(d) (Ø) I ellø' Q11. We have (0r) < + ellø' - Ø11. Take = eo + ellø' - ~11
have
yields the result.
C.5 BOUNDED TASK LOSS BY TASK DIVERSITY
By the previous lemma and claim, we have the below corollary.
Corollary C.12. Suppose we have ó in pretraining: for Vo E t, Loup(0) S (4) Cpre(Ø),
where ( is (hom-pre(d)) if contrastive learning and Csap-pre(()) if supervised learning.
Consider 7 and Corollary C.12,
Recall r-diversity and s-consistency, with respect to 6², 6%. for target task To. we have that for à and o",
(Top Φ) - Coup Φ²) S de, (0, ot) + Coup 62) Esuup(o o²)
(51)
< de (b, x²)/ //
(52)
1
Eeup(0) Leup (0t)) +
(53)
1 p² Lmre(d) Loup(dd)
(54)
D MULTI-CLASS CLASSIFICATION
In this section, we provide general result for multi-classes.
31

---

Published as a conference paper at ICLR 2024
D.1 CONTRASTIVE PRETRAINING
Lemma D.1 (Theorem 6.1 in Arora et al. (2019)). For multi-classes, we have
Loup(0) S C sup (
(6) -
(55)
where T
1y does not appear in (2.91 Mo 1
E
Proof of Lemma D.I. The proof of Lemma D.1 follows the first two steps in the proof of Theorem B.1 of Arora al. (2019). denote distribution of T (r and it's uniform distribution.
We first provide contrastive pretraining error similar to Theorem C.2 in multiclass setting.
Theorem D.2. Assume Assumplion 1 and that & has u-diversity and k-consistency with re- spect to o², 9% Suppose & satisfies Lcon-prel S 60- Consider pretraining set Sun
xf 1) j=1
For target task To, with sample complexity
SLRVT (2) + log( (
8C²
N
ED
60
it's sufficient to learn an & with classification error Soup (ToppTo, x) Suu (omp(Ton (To, o²) S I-7,
-- 60 -
with probability - 5.
Proof of Theorem D.2. Following similar step of proof of Theorem C.2, we have with
8C2
2 0
N
SLRVr 1R,
log(
60
60
Then pretraining à
Loon Ccon-pre(@) pre < 200
Recall v-diversity and k-coonistencyy, for target task To, we have that for â and ø",
Csup 8) Lxup 6¹) S de(0, u
(56)
< Coup (0) Louu Loup((?))
(57)
(58)
Consider Lemma D.I, we have;
Coup (To, Φ) Leup (To, x²) 5
Coon-pre( -e +
(59)
1/
1 Tr
260
e²)
+ k.
(60)
1 1- Tr
Below, we provide our main result similar Theorem C.6 for multi-classes setting,
Theorem D.3. For target evaluation task To consider the error bound in pretraining is Loup ②)
Laup Φ²) <
2eg
+K. Consider as any small constant, aor < I-7, consider
25g
1-
32

---

Published as a conference paper at ICLR 2024
multitask finetuning set S of samples in each task. Then, with sample complexity
{()),y) : [M] [m]},
with M number of tasks, and number
4 + 4C2
M 2
log(
0
E1
40
log 2 2
Mm 8LBVT TR (o())
Solving Equation (2) with empirical risk lower than ei is sufficient to learn an ø' with classification
error Coup (To, Φ') Loup (To, < 2eg I-7, E*) + K, with probability 1 6.
Proof of Theorem D.3. Recalling Lemma C.3 and Lemma C.5, the proof follows the same steps in the proof of Theorem C.6, with different
D.2 SUPERVISED PRETRAINING
first provide contrastive pretraining error similar to Theorem C.8 the multiclass setting.
Theorem D.4. Assume Assumption and that t has v-diversity and k-consistency with respect to
6. or. Suppose è satisfies Csup-pre S ED. Let pi
Pr {y yr where E Let
yー p max, Pi, Consider pretraining set Ssup-pre == (Ti, Mi 1- for any 6 2 0, if
min,
8C2 log( 2)
N (LRVKRN(D)
0
60
60
Then with probability 1 - 8, for any target task To C Co.
1 Caup(To .) (To Φ³) < [2p"en (+7) +
(61)
Proof of Theorem D.4. The proof follows similar steps of Theorem C.8.
Below, we provide our main result similar to Theorem C.10 for multi-classes setting.
Theorem D.5. Assume Assumption and that & has v-diversity and k-consistency with respect to 6², ot. Suppose for some small constant a E (0, 1), we solve Equation (2) with empirical loss lower than E = 22p'ee eg and obtain d' For any > 0, if
1
4C²
2)
1
4C²
2 log
M N
AV2LRM (x(())
log(
Mm 2
8LBVT (0
0
ei
E1
Ei
e1
then with probability 1 for any target task To C Co.
Coup (To, Φ') Laup Φ³) < (2ap"e0 - Lomp( + sc
(62)
Proof of Theorem D.5. Recalling Lemma C.3 and Lemma C.5, the proof follows the same steps in the proof of Theorem C.10, with different r.
E LINEAR CASE STUDY
In this section, provide full analysis of the linear case study to provide intuition about consistency, diversity, and task selection algorithm. Intuitively, we have multiple classes, each centered around its mean vector. Target data has subset of classes, while training data has another subset of classes. Consistency and diversity are related to how these two subsets overlap, i.e., the number of shared features and the number of disjoint features. Then, can link it to the task selection algorithm.
In this section, z; means the i-th dimension of vector rather than the sample index.
33

---

Published as a conference paper at ICLR 2024
E. PROBLEM SETUP
Linear Data and Tasks. We consider dictionary learning or sparse coding settings, which is classic data model (e.g., Olshausen & Field (1997); Vinje Gallant (2000); Blei et al. (2003); Shi et al. (2022; 2023c)). Let X' C Rd be the input space and we have input data E X. Suppose QER4XD is an unknown dictionary with D columns that can be regarded as patterns or features. For simplicity, assume d D and Q is orthonormal. We have E 10, +13
latent class, where 2 is hidden vector that indicates the presence of each pattern. Each latent class 2 has a distribution Dx(x) over inputs z. We assume D2 (z be distribution with mean Qz, i.e., I Q2 Cz2 where e₂ E R is some noise vector drawing from a zero-mean distribution.
For simplicity, we consider each task to be binary classification task, where y (-1, +1} is the label space. In each task (in multitask finetuning or target task), we have two latent classes 2, z' (denote the task as T and randomly assign -1 and +1 each latent class. W.l.o.g., we have in Te,s!
Q2 1Q2+ez e2, if y = -1 10' tez, if y +1.
(63)
For simplicity, we consider a balanced class setting in all tasks, i.e., p(u = -1) p(u +1)
Now, we define multitask finetuning tasks and target tasks. Suppose there is set of latent classes C C {0,-10 -1, +13" used for multitask finetuning tasks, which has an index set Jc S [d], Ar Jel such that for any = E C. we have ZJc E [-1,+11 and dddide E (0)d-ke, Similarly, suppose there is a of latent classes Co S {0, +1) used for target tasks whose index set is Jn C [4]. An =
[Jo]. Note that Je may or may not overlap with Jo and denote the set of features encoded both by Co and C as Le == Jo n Jc, lc == |Lc]. Intuitively, Le represents the target features covered by training data. Let over C be the distribution of multitask finetuning tasks. Then, in short, our data generation pipeline for multitask finetuning tasks is (1) sample two latent classes (2, 2²) as a task Tz.z;; (2) assign label 11 +1 two latent classes; (3) sample input data from D2(2) and (xr) with balanced probabilities.
For simplicity, we have a symmetric assumption and a non-degenerate assumption for Ç. Symmetric assumption means each dimension is equal important and non-degenerate assumption means any two dimensions are not determined by each other in all tasks.
Assumption 3 (Symmetric). We assume for any multitask aultitaskfinetunnng tasks distribution for any k E Je, switching two dimensions 2j and z does not change the distribution Ç.
Assumption 4 (Non-degeneratee. We assume for any multitask finetuning tasks distribution for any j, k E Jc, over ( we have p(zz = 21y; & =4) > O.
Remark E.1. There exists many Ç satisfying above assumptions, e-g., (1) zJe and Z'1c uni- formly sampling from {-1,+1 or (2) let kc = 2, ZJe and z'sc uniformly sampling from ((+1,+1), (-I,++), (- (+1,-113 (note that uniformly sampling from not satisfy non-degenerate assumption). Also, we note that even when C ===0 Co. the larget latent class may not exist in the multitask finttnning tasks.
does
Linear Model and Loss Function. Let T be the hypothesis class of models & X + 2, where Z C Rdd is the output space of the model. We consider linear model class with regularity Assumption I, i.e. & = (Ø E dddxd || p <. R} and linear head w E IR where ||w||| < B. Thus, the final output of the model and linear head is w oz. We use linear loss in Shi et al. (2023a), i.c.. é (w" T oz, v) = yyw dix and we have
Ssup (T, =
min WERR.JWWlaSS SB TNT (>).(()
E
E
[e (w Φz. v)]
(64)
Csup(Ø) = E
[Coup (T 6)
(65)
T~e
05 = arg min Lxup ((); deD
(66)
where oz is the optimal representation for multitask finetuning.
34

---

Published as a conference paper at ICLR 2024
E.2
DIVERSITY AND CONSISTENCY ANALYSIS
E.2.1 OPTIMAL REPRESENTATION FOR MULTITASK FINETUNING
Lemma E.1. Assume Assumption 3 and Assumption 4. We have 0f = UAQQI, where U is any R and X; 0 otherwise.
orthonormal matrix, A diaag(>).. For any Jc. X3 vke
Proof of Lemma E.I. We have the singular value decomposition e UAVT, where diag(A), where À E IRd. Then, we have
(sup(1) = E Tne
SLsup( (T, [6]
(67)
[e (((w' (Tx, pox, w)]
E
min
(68)
T~(
[oERd||| 2 a>D..(>)
=
E
WER M 2 (>)) [e (ww" (wr, Φx,, -1]] +
min
E
[e [w( ox, +1]]
T (
(69)
E
min
E (a)
[w T dbo]] +
E xnDw(()) [-w' {ioe
(70)
2T, 2
[EER
E
min
w 6Q= - w =oQQ=
(71)
B
[10Q((= =2])
(72)
T
B
[|AV Q(= =1)1l2]
(73)
Te
W.l.o.g,, assume V =Q As ||
||Ale
|A/2 thus we have
B
min C (p)(()
max
E
[IA((z - 21)112l
oco
2
| T T.
B
Σ A7(2 2f)²
max
2
|| T
B
max
E
X2(2 2/)²
2 | T T.
- max
E I
Σ TeJc
X 12 # 21
(74)
where 1/2 # =[] is Boolean function, mapping True to 1 and False to 0.
35

---

Published as a conference paper at ICLR 2024
Let 07 UA with corresponding X" Now, we use contradiction prove for any j. Jc. we have X; = X W.l.o.g., suppose Xj < X/e-
(
- E Telervn
1721125 # 21 + 1/2 # #1 +
E
1321/2 # #
-BB
BB(b plzz # 2yy zk) E
122++ + X²2 +
Σ
1;21 # +1
3j + 25 =k + #
T
iEJoMAR)
+ +p(zz 2/ 2k # 2/6
E T
xx²++ +
E
1 + +
2
- 2 2/
iEJe)(in)
+2 = 2/) E
122 +
Σ
1 * 21
23 2/, 2/4
Te.e'mc
(EJc)AA)
By symmetric Assumption and non-degenerate Assumption 4, we have p(z} = Z 2x + z/2)
pl2z # z'y, 2k = =/{) 0, and
E Te.e!
X=2 + (EJe((iA)
Σ
x/11/2 * 21
23 2/. 2 + 2
E
212
E
1/2 # =1
3 + #/ 2
Te,! S
(EJO
= E
22 +
Σ
1x21(zz # x1
23 2/3 AM # 2/
Ta
(EJO
+ E
Σ
1/21/ 41
26 x 2/ 23= 2/3
<2 E T T
Σ
1 # +
= 24 #/
2
X12 $2
E T T
Σ iEJo
111l2 # +11
21= 25
X + 42 R
E T
Σ
1/21 # 21
2x 4
2/. 2g
2/1
where two equality follows Assumption and the inequality follows Jensen's 's inequality. Let o' UNQQ1 with corresponding X', where X/ =x== v 172 and for any i E Je ( k}. X A₁. We have ||o'|| F || rr and Cxup (d²) > (o') which is contradiction we have of is the optimal solution. Thus, for any k E Jc, we have X3 >j and we finish the proof under simple calculation.
Now, we ready analyze consistency and diversity under this linear case study.
36

---

Published as a conference paper at ICLR 2024
E.2.2 CONSISTENCY
The intuition that Ç not only covers Cu but contains too much unrelated information. Recall that the consistent term in Definition 2 is suPrice, [Ssup x2) Ssup (Top(TTo-1 66]
We first define some notation we will use later. Let Ço be a multitask finetuning tasks distribution over Co Co and denote the corresponding optimal representation model as o6. Suppose for any larget task To contains two latent classes z, from Co. W.l.o.g., denote 2, 2' differ in no entries S Rog I ku), whose ne entries fall in Le where 0 I ne ny- Then, we get the lemma below:
Lemma E.2. Assume Assumption 3 and Assumption 4. We have
K sup ToECo
[Esup (To, x5) - Laup (To, 06)
BR
no ko
RC kc
(75)
V
Proof of Lemma E.2. Recall 1 S no S hop and 0 S ne S no- By Lemma E. we have of UA* TA*Q"! Q-', where U is any orthonormal matrix, A diag(X*). For any E Jc, X vke and X; = 0 otherwise. We also have oi = UoA where Uo is any orthonormal matrix, R and A.. = 0 otherwise. Thus, we have vko
R
Ag = d. For any i E Jo. 10
K sup ToCCo
[Suu (To: x2) Ssup (Tu, (Tu.6))
(76)
no
nc
= BR
(77)
kn
kc
Let n²z = ke- - ng. Note this ke is an increasing factor C contains more features. Moreover, ne is the number of features encoded by both target and training data, representing the information of target data covered by training data, ne increases as more target information covered by training data, the loss will decrease. n'c the number of features encoded in training data but encoded by target data, representing the un-useful information, n'e increases as more un-related information is covered by training data, the loss will increase.
E.2.3 DIVERSITY
We first review some definitions in Definition 1. The averaged representation difference for two model o, on a distribution ( over tasks is
de(b, δ) = - [Coup () - ( (0)
Lo)
(78)
The worst-case representation difference between representations Φ, on the family classes Co is
de, (o. = sup ToCCo
(Tu Φ) - Lxup 2)
(79)
We say the model class + has v-diversity for and Co if for any o E ob and 07.
de. +z) < de( 02)/v.
(80)
To find the minimum value of in Definition 1, we need further information about Ç. For simplicity, we have a fixed distance assumption, e.g., uniformly sampling from +1,),
-1, (1, +1). (-1,+1, +1)}. Then, consider two different cases below. We consider that all To C Co such containing z, that differ in only entry.
Assumption 5 (Fixed Distance), We assume for any multitask finetuning tasks distribution for any two latent classes (z, z') ~ we have z, ' differ in ny entries.
37

---

Published as a conference paper at ICLR 2024
Case Le # Jo- In this case, Jo \ Le 0, we have the features learned in multitask finetuning that do not cover all features used in the target task. Then, we have the following lemma, which means Le # Jn
have infinitesimal y to satisfy the diversity definition.
Lemma E.3. Assume Assumption 3, Assumption 4 and Assumption 5. When Le # Jo. we have 0.
Proof of Lemma E.3. As features in Co not covered by C, can always find a To such containing z, that only differ in entries in Jg Le, say entry i.
By Lemma E. we have 0 = UA TTA"Q-, where U is any orthonormal matrix, A = diag()"). For n and X 0 otherwise. We have Liup (To; +3) 0 and by Equation (74),
any E Jc, X3- vie
Csup(12) = B E Te.
1,211 [2r + zf}
(81)
sEJc
nkk
-BRR
(82)
kc
On the other hand, for any & E & we have Caup(Te x) x)= --=-BBaa B|Agl. Thus, we have
(sup(() fisupp (155)
min
Soup (To: Φ) Lsuu (Tu, [t)
Csup (0) + +BRy
Rec
min dE
B|2El
B E Ta [V EiEse Ap1 # =3]
+ BR
nE kc
min
@
R²
V Eieso To
E
[ * 2/1
+ +
n2 Rcc
he
+Ry
ke
(83)
|Ag
where the first inequality is by constructing specific d. Note that Equation (83) + 0 when |Agl + 0. é is constructed as: for any i E Jc. d
and Adg + 0. Thus, we finish the proof.
ke
Case Le = Jo- In this case Jo Le = 0. we have all features in Co covered by C.
Lemma E.4. Assume Assumption 3, Assumption 4 and Assumption 5. When all To C Co such containing z, that differ in only I entry and Le = Jo. we have v is lower bounded by some constant ē TT (1- (I-VEethe ---kktkk-1) IT (Vng(Te 1) + kc n1
Proof of Lemma E.4. We say the differ entry in To as entry i. By Lemma E.I, have oz UAQQ11 where U is any orthonormal matrix, A = diag(X*). For any i E Jc. A3 = R
and X
Thc
otherwise. By Equation (74), we have Loup (To, x)) = - BR
and Coup( = - BR Re
uh
ne.
Re
38

---

Published as a conference paper at ICLR 2024
On the other hand, for any D, we have Loupl (To. B|A;]. Thus, by Assumption 3, we have
(oup(() ((x
V =
min ToCCD.EE
(oup (To: 6) Loup +22)
Csup(Ø) + BRR/RRR Rc
min
Bl1a + BRR Rtc ToCCO
B E Tene
IV EVEEde >Z1lZz + =1{
+ BR
Bk
Re
min ToCCo.0E4
B|2 + BR/AL. hcl
E Ta
22 # 2/1 EiEdeN
R²-23
125 # 2/
R
nk Rc
he-
min ToCCo
- 1 + RV
ke
2k RC
X2
+
-²
(1ig= 1) + ke-ng ke
Th 1112--22
R
a
kc 1
lie
Re
min To
1 111b + R
ke
= =v 1 11- kkkke - 1) ( (Vna(nu - 1) + kc Tok) )
(84)
where the last equality take \ = O.
E.3 PROOF OF MAIN RESULTS
Proof of Theorem 3.3. Note that = B no no= = ko 1, ni 2.
We see that Ç satisfies Assumption 3, Assumption 4 and Assumption 5. We finish the proof by Lemma E.2, Lemma E.3 and Lemma E,4 with some simple calculations.
Thus, we can link our diversity and consistency parameters to the number of features in = encoded by training tasks target tasks. Based on this intuition, we propose selection algorithm, where selection is based on we want to select data that encodes more relevant features of this can be achieved by comparing x from target data and training data either using cosine similarity or KDE,
F VISION EXPERIMENTAL RESULTS
We first provide a summary of dataset and protocal we use, we provide details following sections.
Datasets and Models. We use four widely used few-shot learning benchmarks: minilmageNNt (Vinyals et al., 2016), mageed (Ren et al., 2018), DomainNet (Peng et al., 2019) and Meta- dataset (Triantafillou et al., 2020), following the protocol in Chen et al. (2021b); Tian et al. (2020b). We use exemplary foundation models with different pretraining schemes (MoCo-v3 (Chen et al., 2021a), DINO-v2 (Oquab et al., 2023), and supervised learning with ImageNet (Russakovsky et al., 2015)) and architectures (ResNet (He et al., 2016) and ViT (Dosovitskiy al., 2021)).
Experiment Protocol. We consider few-shot tasks consisting of N classes with K support samples and Q query samples per class (known as N-way K-shot). The goal to classify the query samples into the N classes based on the support samples. Tasks used for finetuning are constructed by samples from the training split. Each task is formed randomly by sampling 15 classes, with every class drawing 1 or 5 support samples and 10 query samples. Target tasks are similarly constructed, yet from the test set. We follow (Chen et al., 2021b) for multitask finetuning and target task adaptation. During multitask finetuning, we update all parameters in the model using nearest centroid classifier,
39

---

Published as a conference paper at ICLR 2024
in which all samples are encoded, class centroids are computed, and cosine similarity between query sample and those centroids are treated as the class logits. For adaptation to a target task, we only retain the model encoder and consider similar nearest centroid classifier. This experiment protocol applies to all three major experiments (Sections 4.1 to 4.3).
F.1 DATASETS
The minilmageNeet dataset is common benchmark for few-shot learning. It contains 100 classes sampled from ImageNet, then is randomly split into 64, 16, and 20 classes as training, validation, and testing respectively.
The tiered dataset is another widely used benchmark for few-shot learning. contains 608 classes from 34 super-categories sampled from ImageNet. These categories are then subdivided into 20 training categories with 351 classes, 6 validation categories with 97 classes, and 8 testing categories with 160 classes
DomainNet is the largest domain adaptation benchmark with about 0.6 million images. It consists of around 0.6 million images of 345 categories from domains: clipart (clp), infograph (inf), quickdraw (qdr), real (rel) and sketch (skt). We split it into 185, 65, 100 classes as training, validation, and testing set respectively. We conduct experiments on Sketch (skt) subsets.
Meta-Dataset encompasses ten publicly available image datasets covering wide array of domains: ImageNet-IK, 1k, Omniglot, FGVC-Aircraft, CUB-200-2011, Deseribable Textures, QuickDraw, FGVCx Fungi, VGG Flower, Traffic Signs, and MSCOCO. Each of these datasets is split into training, validation, and testing subsets. For additional information on the Meta-Dataset can be found in Appendix 3 of Triantafilouu et al. (2020).
F.2 EXPERIMENT PROTOCOLS
Our evaluation and the finetuning process take the form few-shot tasks, where a target task consists of N classes with K support samples and Q query samples in each class. The objective is to classify the query samples into the classes based on the support samples. To accomplish this, we take the support samples in each class and feed them through an image encoder obtain representations for each sample. We then calculate the average of these representations within each class to obtain the centroid of each class. For given query sample xr, we compute the probability that x belongs to class y based on the cosine similarity between the representation of x and the centroid of class y.
In our testing stage, we constructed 1500 target tasks, each consisting of 15 classes randomly sampled from the test split of the dataset. Within each class, we randomly selected 1 or 5 the available images as shot images and 15 images query images. These tasks are commonly referred as 1-shot or 5-shot tasks. We evaluated the performance of our model on these tasks and reported the average accuracy along with 95% confidence interval.
During multitask finetuning, the image encoder is directly optimized on few-shot classification tasks. To achieve this, we construct multitasks in the same format as the target tasks and optimize from the same evaluation protocol. Specifically, we create a total of 200 finetuning tasks, each task consists of 15 classes sampled from the train split of data, where each class contains support image and query images, resulting in 150 images per task. The classes in finetuning task are sampled from the train split of the data.
To ensure fair comparison with the finetuning baseline, we used the same training and testing data, as well as batch size, and applied standard finetuning. During standard finetuning, we added linear layer after the encoder and trained the model. We also utilized the linear probing then finetuning (LP-FT) technique proposed by Kumna et al. (2022), which has been shown to outperform finetuning alone on both in-distribution and out-of -aitof-distribution data. In the testing stage, we removed the linear layer and applied the same few-shot testing pipeline to the finetuned encoders.
For task selection, we employ the CLIP VIT-B image encoder to obtain image embeddings. We assess consistency by measuring the cosine similarity of the mean embeddings and we evaluate diversity through a coverage score derived from the ellipsoid formula outlined in Section 3.2.
40

---

Published as a conference paper at ICLR 2024
For optimization, the SGD optimizer with momentum 0.9, the learning rate is le-5 for CLIP and moco v3 pretrained models, and 2e-6 for DINO v2 pretrained models. The models were finetuned over varying numbers of epochs in each scenario until they reached convergence.
F.3 EXISTENCE OF TASK DIVERSITY
Task diversity is crucial for the foundation model to perform well on novel classes in target tasks.
In this section, we prove for task satisfying consistency, greater diversity in the related data can help reduce the еπrоr on the target task. Specifically, for the target task, where the target tasks data originates from the test split of specific dataset, we utilized the train split of the same dataset as the finetuning tasks data. Then finetuning tasks satisfied consistency. In experiments, we varied the number of classes accessible to the model during the finetuning stage, while keeping the total sample number the same. This serves measure of the diversity of training tasks.
F.3.1 MINIIMAGENET AND OMNIGLOT
We show the results of CLIP encoder minilmageNet and Omniglot. We vary the number of classes model access to in finetuning stage. The number classes varies from all classes, i.e., 64 classes, to 8 classes. Each task contains 5 classes. For finetuning tasks, each class contains shot image and 10 query images. For target tasks, each class contains the 1-shot image and 15 query images.
# limited classes
64
32
16
8
0
Accuracy
90.02 + 0.15
88.54 + 1.11
87.94 + 0.22
87.07 + 0.20 83.03 + 0.24
Table Class diversity on VIT-B32 backbone on minilmageNet.
Table shows the accuracy of ViT-B32 across different numbers of classes during the finetuning stage. The "Class 0" represents direct evaluation without any finetuning. We observe that finetuning the model leads to an average accuracy improvement of 4%. Furthermore, the diversity of classes increases, we observe a corresponding increase performance. This indicates that incorporating wider range of classes during the finetuning process enhances the model's overall accuracy.
For task diversity, we also use dataset Omniglot (Lake al., 2015). The Omniglot dataset designed to develop more human-like learning algorithms. It contains 1623 different handwritten characters from 50 different alphabets. The 1623 classes are divided into 964, 301, and 358 classes as training, validation, and testing sets respectively. We sample multitask in finetuning stage from training data and the target task from testing data.
limited classes
482
50
Accuracy
95.35 0.14 95.08
0.14
94.29 + 0.15
88.48
0.20
80.26 + 0.24 74.69 0.26
Table 4: Class diversity ViT-B32 backbone on Omniglot.
Table 4 shows the accuracy of VIT-B32 different numbers of classes in finetuning stage, where class 0 indicates direct evaluation without finetuning. Finetuning improves the average accuracy by 5.5%. As class diversity increases, performance increases.
F.3.2 TIEREDIMAGENET
We then show results tieredImageNet across learning settings for the VIT-B backbone. We follow the same setting where we restrain each task that contains 15 classes.
We found that using more classes from related data sources during finetuning improves accuracy. This result indicates that upon maintaining consistency, a trend is observed where increased diversity leads to enhancement in performance.
41

---

Published as a conference paper at ICLR 2024
Pretrained
351
175
43
DINOv2
84.74
82.75
82.60
82.16
CLIP
68.57
67.70
67.06
63.52
Supervised
89.97
89.69
89.19
88.92
Table 5: The performance of the ViT-B backbone using different pretraining methods on tieredlmagenet, varying, the number of classes accessible to the model during the finetuning stage. Each column represents the number of classes within the training data.
F.4 ABLATION STUDY
In Section 4 and the result in Table 2, we utilize the train split from the same dataset to construct the finetuning data. It is expected that the finetuning data possess a diversity and consistency property, encompassing characteristics that align with the test data while also focusing on its specific aspects.
In the following ablation study, we explore the relationship between the diversity and consistency of data in finetuning tasks, sample complexity, and finetuning methods. We seek to answer the following questions: Does multitask finetuning benefit only from certain aspects? How do these elements interact with each other?
F.4.1
VIOLATE BOTH CONSISTENCY AND DIVERSITY: ALTERING FINETUNING TASK DATA WITH INVARIANT SAMPLE COMPLEXITY
In this portion, we examine the performance when the model is finetuned using data completely unrelated to the target task data. With the same finetuning sample complexity, the performance cannot be compared to the accuracy we have currently attained.
In this section, we present the performance of MoCo v3 with a ViT-B backbone on the DomainNet dataset. We finetuned the model using either ImageNet data DomainNet train-split data and evaluated its performance on the test-split of DomainNet. We observed that finetuning the model with data selected from the DomainNet train-split resulted in improved performance the target task. This finding aligns with our expectations and highlights the significance of proper finetuning data selection.
When considering the results presented in Table 6, we also noticed that for MoCo v3 with ResNet50 backbone and DINO with a ViT-S backbone, multitask finetuning on ImageNet led to a decrease in model performance compared to direct adaptation. This suggests that inappropriate data selection can have detrimental effect on the final model performance. This conclusion also supported by the findings of Kumar et al. (2022).
F.4.2 VIOLATING CONSISTENCY WHILE RETAINING DIVERSITY: THE TRADE- OFF BETWEEN TASK CONSISTENCY AND SAMPLE COMPLEXITY
Finetuning tasks with superior data are expected excel under identical complexity, natural question can be proposed: Does additional data enhance performance? Our results in this section negate this question. Testing the model on the DomainNet test-split, we employ two settings. In the first setting, we finetune the model on the DomainNet train-split. In the second, the model is finetuned with a combination of the same data from DomainNet as in the first setting, along with additional data from ImageNet.
Within our theoretical framework, mixing data satisfies diversity but fails consistency. The finetuning data, although containing related information, also encompasses excessive unrelated data. This influx of unrelated data results in a larger consistency parameter x in our theoretical framework, adversely impacting model performance the target task. We offer empirical evidence affirm theoretical conclusion.
Table 7 shows mixed data of domainNet and ImageNet will doesn't 't provide the same advantages as using only DomainNet data. In this case, an increasing in data does not necessarily mean better performance.
42

---

Published as a conference paper at ICLR 2024
pretrained
backhone
FT data
Accuracy
MoCo v3
ViT-B
ImageNet
24.88 (0.25)
DomainNet
32.88 (0.29)
ResNet50
ImageNet
27.22 (0.27)
DomainNet
33.53 (0.30)
DINO v2
ViT-S
ImageNet 51.69 (0.39)
DomainNet
61.57 (0.40)
ViT-B
ImageNet
62.32 (0.40)
DomainNet
68.22 (0.40)
Supervised
ViT-B
ImageNet
31.16 (0.31)
DomainNet
48.02 (0.38)
ResNer50
ImageNet
29.56 (0.28)
DomainNet
39.09 (0.34)
Table 6: Finetuning data selection model performance. FT data: dataset we select for multitask finctuning. Report the accuracy on the test-split of DomainNet.
Pretrained
DomainNet
DomainNet ImageNet
DINOv2
68.22
66.93
CLIP
64.97
63.48
Supervised
48.02
43.76
Table Results evaluating on DomainNet test-split using ViT-B backbone. First column shows performance where model finetune on data from DomainNet train-split alone, second column shows the performance of the model finetuned using blend of the same data from DomainNet, combined with additional data from ImageNet.
F.4.3 DIVERSITY AND CONSISTENCY TASK DATA AND FINETUNING METHODS
To provide a more comprehensive understanding of the impact of task data and finetuning methods model performance, conduct additional experiments, utilizing varying finetuning methods and data. The model tested on the DomainNet test split. We employ either multitask finetuning or standard finetuning, where linear layer is added after the pretrained model. This linear layer maps the representations learned by encoders to the logits. The data of finetuning tasks derive from either the DomainNet train-split ImageNet.
3
5
Pretrained
Adaptation
ImageNet (SFT)
ImageNet (Ours)
DomainNet (SFT) DomainNet (Ours)
DINOv2
61.65
59.80
62.32
61.84
68.22
CLIP
46.39
46.50
58.94
47.72
64.97
Supervised 28.70
28.52
31.16
30.93
48.02
Table 8: Results evaluating on DomainNet test-split using ViT-B backbone. Adaptation: Direction adaptation without finetuning; SFT: Standard finetuning; Ours: Our multitask finetuning. Col-1 shows performance without any finctuning, Co1-2,3,4,5 shows performance with different finetuning methods and data.
In Table 8, we detail how data quality and finetuning methods of tasks impact the ultimate performance. Standard finetuning (SFT) with unrelated data diminishes performance compared to direct adaptation (col-1 vs col-2). On the other hand, multitask finetuning using unrelated data (ImageNet), or SFT with related data (DomainNet), both outperform direct adaptation. However, multitask finetuning
43

---

Published as a conference paper at ICLR 2024
Pretrained
Selection
INet
Omglot
Acraft
CUB
QDraw
Fungi
Flower
Sign
COCO
CLIP
All Selected
60.87
70.53
31.67
66.98
40.28
34.88
80.80
37.82
33.71
60.87
77.93
32.02
69.15
42.36
36.66
80.92
38.46
37.21
DINOv2
All 83.04
72.92
36.52
94.01
49.65
52.72
98.54
34,59
47.05
Selected
83.04
80.29
36.91
94.12
52.21
53.31
98.65
36.62
50.09
MoCo v3
All Selected
59.62
60.85
18.72
40.49
40.96
32.65
59.60
33.94
33,42
59.62
63.08
19.03
40.74
41.16
32.89
59.64
35.25
33.51
Table Results evaluating our task selection algorithm on Meta-dataset using ViT-B backbone.
with unrelated data proves more beneficial than the latter (col-3 vs col-4). The peak performance is attained through multitask finetuning related data (col-5).
F.4.4 ABLATION STUDY ON TASK SELECTION ALGORITHM
We show simplified diagram for task selection in Figure 2.
We first provide some details of Table We first create an array finetuning tasks, and then apply our task selection algorithm to these tasks. Specifically, we design 100 finetuning tasks by randomly selecting 15 classes, each providing support sample and 10 query samples. The target tasks remain consistent with those discussed in Section 4. For more comprehensive analysis of our algorithm, we performed ablation studies on the task selection algorithm, concentrating solely on either consistency or diversity, while violating the other. Violate Diversity: If the algorithm terminates early without fulfilling the stopping criteria, the data utilized in finetuning tasks fails to encompass all the attributes present in the target data. This leads to breach of the diversity principle. Violate Consistency: Conversely, if the algorithm persists beyond the stopping criteria, the finetuning tasks become overly inclusive, incorporating an excessive amount of unrelated data, thus breaching the consistency.
This section details an ablation study on task selection for the dataset, we implement our task selection process meta-dataset, treating each dataset distinct task and choosing datasets to serve as data sources for the finetuming tasks. We show the result in Table 9.
Table indicates that maintaining both consistency and diversity in the task selection algorithm is essential for optimal performance. This is evident from the comparison between the Random selection and the our approach, where the latter often shows improved performance across multiple datasets. ImageNet as the target task is an exception where the two approaches give the best results. Due to its extensive diversity, all samples from all other datasets are beneficial for finetuning. Consequently, the task selection algorithm tends to select all the candidate tasks.
F.5 TASK SELECTION ALGORITIIM ON DOMAINNET
We verify our task selection algorithm by applying it on DomainNet. Here, the mini-ImageNet test-split is regarded as the target task source, and diverse domains (such clipart (clp), infograph (inf), quickdraw (qdr), real (rel), and sketch (skt)) are considered as sources for finetuning tasks. We view different domain datasets distinct finetuning tasks. With 6 domains in focus, our objective is to select a subset that optimizes model performance. We systematically apply Algorithm 1. Initially, we calculate the cosine similarity of mean embeddings between each domain and target tasks, ordering them from most to least similar: real, painting, sketch, clipart, infograph, and quickdraw. Sequentially adding datasets in this order, the process continues until the diversity score (I over Mahalanobis distance) stops exhibiting significant increase.
As we can see in Figure 4, the diversity does not increase when we just select real and painting as our finetuning task data. For a comprehensive analysis, each combination is finetuned and the model performance accuracy on the target task displayed.
As we can see in Figure 5,the accuracy aligns with the conclusions drawn based on consistency and diversity. Remarkably, only real and painting suffice for the model to excel on the target task.
F.6 MORE RESULTS WITH CLIP ENCODER
In this section, we show additional results CLIP (Radford al., 2021) model.
44

---

Published as a conference paper at ICLR 2024
Embedding similarity for Each Dotasett
3/Mahalanobis distance Each Dataset
0.014
0.20
0.00
6.12-
0.010-
0.00
0.20-
0.006
0.04-
0.05
0.022
real
painting
sleetch
dipar
Infograph
quickdraw
0.000
1031
rpa
rpsc
rpsc3
an
(a) Mean embeddiing similarity for each data sort from most similar to least similar.
(b) Diversity when adding lask
where rp: and
painting: rps: real and painting and sketch and so on.
Figure 4: Dataset selection based on consistency and diversity on domainNet. Figure 4a shows consistency.
Figure 4b shows the diversity.
clip Model
dinov2 Model
mocov3 Model
17.00
76.
16.35
1250
25.55
92.
023
91.
real
rp
rps
rpsc
rpsci
all
real
rp
rps
rpsc
rpsci
all
real
rp
rps
rpsc
rpsci
all
Figure 5: Finetuning with different selection of domain datasets, where rp: real and painting: rps: real and painting and sketch and so on.
We can observe from Table 10 standard finetuning improves performance compared to direct adapta- tion. However, proposed multitask finetuning approach consistently achieves even better results than the standard baseline.
Task (M) Sample (m). We vary the task size and sample size per task during finetuning. We verify the trend of different numbers of tasks and numbers of images per task. Each task contains classes. For finetuning tasks, 50 indicates each class contains the 1-shot image and 9-query 200 indicates
images. m = 100 indicates each class contains 2-shot and 18 18-query images. each class contains 4-shot and 36-query images. . m indicates direct evaluation without finetuning. For target tasks, each class contains the 1-shot image and 15 query images.
Table shows the results on the pretrained CLIP model using the ViT backbone. For direct adaptation without finetuning, the model achieves 83.03% accuracy. Multitask finetuning improves the average accuracy at least by 6%. For fixed number of tasks or samples per task, increasing samples tasks improves accuracy. These results suggest that the total number of samples (M m) will determine the overall performance, supporting our main theorem.
Few-shot Effect. We perform experiments the few-shot effects of finetuning tasks. We aim to evaluate whether increasing the number of few-shot images in the finetuning task leads to significant improvements. Each finetuning task consists of 5 classes, and maintain fixed number of 10 query images per class while gradually increasing the number of shot images, as illustrated in Table 12. As for the target tasks,
each class contains shot image and 15 query images for evaluation.
45

---

Published as a conference paper at ICLR 2024
minilmageNNet
migredImageNet
DomainNet
backbone
method
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
CLIP-ViTB32
Direct Adaptation
68.41 (0.54)
87.43 (0.15)
59.55 (0.21)
79.51 (0.27)
46.48 (0.37)
72.01 (0.29)
Standard FT
69.39 (0.30)
88.39 (0.15)
61.20 (0.37)
80.65 (0.27)
47.72 (0.37)
72.82 (0.29)
Multitusk FT (Ours)
78.62 (0.15)
93.22
68.57 (0.37)
84.79 (0:22)
64.97 (0.39)
80.49 (0.25)
CLIP- CLIP-ResNe150
Direct Adaptation
61.31 (0.31)
82.03 (0.18)
51.76 (0.36)
71.40 (0.30)
40.55 (0.36)
64.90 (0.31)
Standard FT
63.15 (0.31)
83.45 (0.17)
55.77 (0.35)
75.28 (0.29)
43.77 (0.38)
67.30 (0.31)
Multitask FT (Ours)
67.03 (0.30)
(0.17) 57.56 (0.36)
(0.28)
(0.39)
(0.30)
Table 10: Comparison 15-way classification. Average few-shot classification accuracies (%) with 95%
confidence intervals clip encoder.
Sample (m)
50
100
200
Task (M)
83.03 = 0.24
200
89.07 + 0.20
89.95 0.19
90.09
0.19
400
89.31
0.19
90.11H 0.19
90.70
0.18
800
89.71 + 0.19
90.27 0.19
90.80 0.18
Table 11: Accuracy with a varying number of tasks samples (ViT-B32 backbone).
Table 12 displays the accuracy results of ViT-B32 when varying the number few-shot images in the finetuning tasks. We observe that increasing the number of few-shot images, thereby augmenting the sample size within each task, leads to improved performance. This finding is quite surprising, considering that the finetuning tasks and target tasks have different numbers of shot images. However, this aligns with our understanding of sample complexity, indicating that having access to more training examples can enhance the model's ability to generalize and perform better on unseen data.
F.7 SAMPLE COMPLEXITY ON PERFORMANCE FOR TIEREDIMAGENHT
We provide table and visualization of the trend of the number of tasks and the number of samples per task for the MoCo v3 VIT model on magred Table 13 and Figure 6.
As demonstrated in the paper, we have observed that increasing the number of tasks generally leads performance improvements, while keeping the number of samples per task constant. Conversely, when the number of samples per task is increased while maintaining the same number of tasks, performance also tends improve. These findings emphasize the positive relationship between number of tasks and performance, well the influence of sample size within each task.
80
Samples
task (m)
78
- 150
Accuracy
76
300
450
74
600
72
70
68
200
300
400
500
600 700
800
Number of tasks (M)
Figure 6: Finetuning using tirredImageNet train-split, test on test-split.
F.8 FULL RESULTS FOR EFFECTIVENESS OF MULTITASK FINETUNING
In this section, we provide another baseline in complement the results in Section 4.3.
46

---

Published as a conference paper at ICLR 2024
# shot images
20
10
0
Accuracy
91.03 + 0.18
90.93
0.18
90.54
0.18
90.02 + 0.15
83.03 = 0.24
Table 12: Few-shot effect on ViT-B32 backbone minilmageNet.
Sample (m)
150
300
450
600
Task (M)
200
68.32 (0.35)
71.42 (0.35)
73.84 (0.35)
75.58 (0.35)
400
71.41 (0.35)
75,60 (0.35)
77.57 (0.34)
78.66 (0.34)
600
73.85 (0.35)
77.59 (0.34)
79.04 (0.33)
79.76 (0.33)
800
75.56 (0.35)
78.68 (0.34)
79.78 (0.33)
80.26 (0.33)
Table 13: Accuracy with a varying number of tasks and samples (ViT-B32 backbone).
We incorporated the Model-Agnostic Meta-Learning (MAML) algorithm, outlined by Finn al. (2017), as another baseline for our few-shot tasks. MAML operates in two-step process: it initially updates parameters based on wuthin-episode loss (the inner loop), then it evaluates and updates loss based on learned parameters (the outer loop). We follow the pipeline in Triantafillou et al. (2020) to implement MAML. for few-shot tasks. We show results Table 14.
Table 14 reveals that MAML exhibits variable performance across different settings. For instance, outperforms both Adaptation and Standard FT methods in scenarios like MoCo v3 ViT-B on minilmageNet, DomainNet, and ResNet 50 supervised training for tieredImageNet. However, its performance less impressive other contexts, such as DINOv2 ViT-B on minilmageNet and ViT-B on supervised training for minilmageNet. This variability in performance is attributed to the constraints of our few-shot tasks, where the limited number of support samples restricts the model's s capacity to adapt to tasks. Despite these fluctuations, multitask finetuning approach consistently surpasses the mentioned baselines, often by a significant margin, across evaluated scenarios.
47

---

Published as a conference paper at ICLR 2024
minilmageNet
IigredImageNet
DomainNet
pretrained
backbone
method
I-shot
5-shot
I-shot
5-shot
1-shot
5-shot
MoCo v3
VIT-B
Adaptation
75.33 (0.30) 92.78 (0.10) 62.17 (0.36)
83:42 (0.23) 24.84 (0.25)
44,32 (0.29)
Standard FT
75.38 (0.30)
92.80 (0.10)
62.28 (0.36)
83.49 (0.23)
25.10 (0.25)
44.76 (0.27)
MAML
79.26 (0.28)
93.02 (0.08)
67.96 (0.32)
84.66 (0.19)
28.91 (0.39)
(0.28)
Ours
80.62 (0.26)
93.89 (0.09)
68.32 (0.35)
N5.49 (0.22)
32.88 (0.29)
54,17 (0.30)
ResNe150
Adaptation
(0.30)
88.23 (0.13)
55.15 (0.34)
76.00 (0.26)
27.34 (0.27)
47.50 (0.28)
Standard FT
68.85 (0.30)
88.23 (0.13)
55.23 (0.34)
76.07 (0.26)
27.43 (0.27)
47.65 (0.28)
MAML.
69.28 (0.26)
88.78 (0.12)
55.31 (0.32)
75.51 (0.19)
27.53 (0.39)
47.73 (0.28)
Ours
71.16 (0.29)
89.31 (0.12)
58.51 (0.35)
78.41 (0.25)
33.53 (0.30)
55.82 (0.29)
D2
V
Adaptation
85.90 (0.22)
95.58 (0.08)
74.54 (0.32)
89.20 (0.19) 52.28 (0.39)
72.98 (0.28)
Standard FT
86.75 (0.22)
95.76 (0.08)
74 84 (0.32)
89.30 (0. 19)
$4.48 (0.39)
74,50 (0.28)
MAML.
86.67 (0.24)
95.54 (0.08)
(0.34)
89 (0.19)
52.72 (0.34)
73.35 (0.28)
Ours
88.70 (0.22)
96.08 (0.08)
77.78 (0.32)
90.23 (0.18)
61.57 (0.40)
77.97 (0.27)
VIT-B
Adaptation
90.61 (0.19)
97.20 (0.06) 82.33 (0.30)
92.90 (0.16) 61.65 (0:41)
79.34 (0.25)
Standard FT
91.07 (0.19)
97.32 (0.06)
82.40 (0.30)
93.07 (0.16)
61.84 (0.39)
79.63 (0.25)
MAML
90.77 (0.18)
97.20 (0.08)
82.54 (0.32)
92.88 (0.19)
(0.39)
79:01 (0.28)
Ours
92.77 (0.18)
97.68 (0.06)
84.74 (0.30)
93.65 (0.16)
68.22 (0:40)
82.62 (0.24)
Supervised
ViT-B
Adaptation Standard FT
94,06 (0.15)
97.88 (0.05)
83.82 (0.29)
93.65 (0.13) (0.29)
49.70 (0.28)
pretraining
95.28 (0.13)
98.33 (0.04)
86.44 (0.27)
94,91 (0.12)
30.93 (0.31)
52.14 (0.29)
on ImageNet
MAML
95.35 (0.12)
98.50 (0.08)
86.79 (0.32)
94,72 (0.19)
30.53 (0.39)
$2.21 (0.28)
Ours
(0.11)
98.76 (0.04)
89,97 (0.25)
95.84 (0.11)
48.02 (0.38)
67.25 (0.29)
ResNet50
Adaptation
(0.24)
94.08 (0.09)
65.98 (0.34)
84.14 (0.21)
(0.27)
46.67 (0.28)
Standard FT
84.10 (0.22)
94.81 (0.09)
74.48 (0.33)
88.35 (0.19)
34 (0.31)
55.08 (0.29)
MAML
82.07 (0.28)
94.12 (0.08)
75.69 (0.32)
89.30 (0.19)
35.10 (0.39)
56,51 (0.28)
Ours
87.61 (0.20)
95.92 (0.07)
77.74 (0.32)
89.77 (0.17) 39.09 (0.34)
60.60 (0.29)
Table Results few-shot image classification, We report average classification accuracy (%) with 95%
confidence intervals on test splits. Adaptation: Direction adaptation without finetuning; Standard FT: Standard finetuning; MAML: MAML algorithm in Finn et al. (2017); Ours: Our multitask finetuning; 1-/5-shot: number of labeled images per class in the target task.
G NLP EXPERIMENTAL RESULTS
We first provide a summary of the experimental setting and results in the below subsection. Then we provide details in the following subsections.
G.1
SUMMARY
To further validate our approach, we conducted prompt-based finetuning experiments on masked language models, following the procedure outlined in Gao al. (2021a).
Datasets and Models. We consider collection of 14 NLP datasets, covering 8 single-sentence
and 6 sentence-pair English tasks. This collection includes tasks from the GLUE benchmark (Wang et al., 2018), as well as 7 other popular sentence classification tasks. The objective is predict the label based on a single sentence or a sentence-pair Specifically, the goal is to predict sentiments for single sentences to estimate the relationship between sentence pairs. Each of the datasets is split into training and test set. See details in Appendix G.2. We experiment with a pretrained model RoBERTa (Liu et al., 2019).
Experiment Protocols. We consider prompt-based finetuning for language models (Gao et al., 2021a). This approach turns a prediction task into a masked language modeling problem, where the model generates text response to a given task-specific prompt as the label. Our experiment protocol follows Gao et al. (2021a). The experiments are divided into parallel experiments, each corresponding to dataset. For the few-shot experiment, use test split data as the target task data and sample 16 examples per class from the train split as finetuning data. The evaluation metric is measured by prompt-based prediction accuracy.
During the testing stage, we conduct experiments in zero-shot and few-shot settings for a given dataset. In the zero-shot setting, directly evaluate the model's pompt-bbsed prediction accuracy. In the few-shot setting, we finetune the model using support samples from the same dataset and assess its accuracy on the test split. For multitask finetuning, we select support samples from other datasets
48

---

Published as a conference paper at ICLR 2024
SST-2
SST-5
MR
CR
MPQA
Subj
TREC
CoLA
(acc)
(acc)
(a0c)
(acc)
(acc)
(ace)
(acc)
(Mait)
Prompt-based zero-shot
83.6
35.0
80.8
79.5
67.6
51.4
32.0
2.0
Multitask FT zero-shot
92.9
37.2
86.5
88.8
73,9
55.3
36.8
+0.065
Prompt-based FT'
92.7 (0.9)
47.4 (2.5)
87.0 (1.2)
90.3
84.7 (2.2)
91.2 (1.1)
84.8 (5.1)
(7.3)
Multitask Prompt-based FT
(1.2(1.2)
48.5 (1.2)
86.9 (2.2)
90.5 (1.3)
86.0 (1.6)
89.9 (2.9)
83.6 (4.4)
5.1 (3.8)
task selection
92.6 (0.5)
47.1 (2.3)
87.2 (1.6)
91.6 (0.9)
85.2 (1.0)
90.7 (1.6)
87.6 87.6
3.8 (3.2)
MNLI
MNLI-mm
SNLI
QNLI
RTE
MRPC
QQP
(acc)
(acc)
(acc)
(acc)
(acc)
(FI)
(F1)
Prompt-bused zero-shot
50.8
51.7
49.5
50.8
51.3
61.9
49.7
Multitask FT sero-shot
63.2
65.7
74.0
81.6
prompt-busee
68.3 (2.3)
70.5 (1.9)
77.2 (3.7)
64.5 (4.2)
69.1 (3.6)
74.5 (5.3)
(5.3)
Multitask Prompt-based FT
70.9 (1.5)
73.4 (1.4)
78.7 (2.0)
71.7 (2.2)
74,0 (2.5)
79.5 (4.8)
67.9 (1.6)
+ task selection
73.5 (1.6)
75.8 (1.5)
72.0 (1.6)
70.0 (1.6)
76.0 (6.8)
69.8 (1.7)
Table 15: Results of few-shot learning with NLP benchmarks. All results ohtained using RoBERTa-large. We report mean (and standard deviation) of metrics over different splits. f: Result in Gao et al. (2021a); FT: finetuning: task selection: select multitask data from customized datasets.
and construct tasks for prompt-based finetuning. We then evaluate the performance of the finetuned model on the target task. More details can be found in Appendix G.3.
Task Selection. We select datasets by using task selection algorithm of feature vectors, which are obtained by computing the representations of each dataset and analyzing their relationship. We first obtain text features for each data point the dataset. We select few-shot samples for generating text features. For each example, we replace the masked word with the true label in its manual template, then we forward them through the BERT backbone. Then, we compute the first principal component to obtain feature vector for each dataset. Dataset selection provides certain improvements on some datasets, as elaborated below. Further details can be found in Appendix G.4.
Results. Our results are presented in Table 15. Again, our method outperforms direct adaptation on target tasks across most datasets. For zero-shot prediction, our method provides improvements on all datasets except CoLA. Our multitask finetuning approach results in performance improvements on 12 out of 15 target tasks for few-shot prediction, with the exceptions being SST-2, Subj, and CoLA. CoLA is also reported by Gao al. (2021a) as an exception that contains non- non-grammatical sentences that are outside of the distribution of the pretrained language model. SST-2 already achieves high accuracy in zero-shot prediction, and our model performs best in such setting. Subj is unique that its task is to predict whether given sentence is subjective or objective, therefore multitasking with sew-shot samples from other datasets may not provide significant improvement for this task.
G.2 DATASETS AND MODELS
The text dataset consisted of 8 single-sentence and 6 sentence-pair English tasks, including tasks from the GLUE benchmark (Wang et al.. 2018), as well as 7 other popular sentence classification tasks (SNLI (Bowman et al., 2015). SST-5 (Socher et al., 2013), MR (Pang & Lee, 2005). CR (Hu & Liu, 2004), MPQA (Wiebe et al., 2005), Subj (Pang & Lee, 2004), TREC (Voorhees & Tice, 2000)). The objective was predict the label based on single sentence or sentence -air. Specifically, for single sentences, we aimed to predict their semantics as either positive or negative, while for sentence-pairs, we aimed to predict the relationship between them. We experiment with the pretrained model RoBERTa. We have 14 datasets in total. We split each dataset into train and test split, see details below. We experiment with the pretrained model RoBERTa.
We follow Gao et al. (2021a) in their train test split. We use the original development sets of SNLI and datasets from GLUE for testing. For datasets such as MR, CR, MPQA, and Subj that require a vols-validation evaluation, we randomly select 2,000 examples for testing and exclude them from training. For SST5 and TREC, we utilize their official test sets.
To construct multitask examples from support samples, we gather support samples from all datasets except the testing dataset. For each task, we randomly select ten support samples and prompt-based finetuning the model.
49

---

Published as a conference paper at ICLR 2024
Task
Template
Label words
SST-2
<8,> It was [MASK]
positive: great, negative: terrible
SST-5
<8,> It was [MASK]
v.positive: great, positive: good, neutral: okay, negative: had, v.negative: terrible
MR
<S,> It [MASK]
positive: great, negative: terrible
CR
<S,> It [MASK]
positive: great, negative: terrible
Subj
<S1> This is [MASK]
subjective: subjective, objective: objective
TREC
[MASK] : <S
ubbreviation: Expression, entity: Entity, description: Description
human: Human, location: Location, numeric: Number
COLA
<S - This is [MASK]
grammatical: correct, not grammmmttiia incorrect
MNLI
<S ? [MASK] <Sp>
entailment: Yes, netural: Maybe, contradiction: No
SNLI
<S ? [MASK] <S²>
entailment: Yes, netural: Maybe, contradiction: No
QNLI
<S ? [MASK] <Sp>
entailment: Yes, not_ _ol__ettilmmnt: No
RTF
<S,> ? [MASK] <S2>
entailment: Yes, not _t__etaalleet: No
MRPC
<S1> [MASK] <82>
equivalent: Yes, not equivalent: No
QQP
<S,> [MASK] <S²>
equivalent: Yes, not_equivalen:: No
Table 16: Manual templates and label words that we used in experiments, following Gao et al. (2021a).
G.3 EXPERIMENT PROTOCOLS
Gao et al. (2021a) proposed a prompt-based finetuning pipeline for moderately sized language models such as BERT, RoBERTa. Prompt-based prediction converts the downstream prediction task as a (masked) language modeling problem, where the model directly generates textual response also known as a label word, to given prompt defined by a task-specific template. As an illustration, consider the SST-2 dataset, which comprises sentences expressing positíve or negative sentiment. The binary classification task can be transformed into a masked prediction problem using the template <S>, it was <MASK>.. .. where <S> represents the input sentence and <MASK> is the label word (e.g.. "great" or "terrible") that the model is supposed to predict, see full templates in Table 16. Prompt-based finetuning updates the model with prompt-based prediction loss for given example, such as a sentence or sentence-pair.
To conduct the few-shot experiment, we use all data from the test split as the target task data for each dataset, and sample 16 examples per class from the train split as the support samples. The experiments are divided into 14 parallel experiments. with each corresponding to one dataset. The evaluation accuracy is measured as the prompt-based prediction accuracy. We subsampled 5 different sets of few-shot examples to run replicates experiments and report average performance.
During the testing stage, for given dataset (e.g. QNLI), we consider the entire test split the target task and divide the experiment into zero-shot and few-shot settings. In the zero-shot setting, we directly evaluate the model by measuring the accuracy of prompt- based predictions. In the few-shot setting, we first prompt-based finetune the model with support samples from the same dataset (QNLI) and then evaluate the accuracy on the test split. This experimental protocol follows the same pipeline as described in Gao et al. (2021a).
To perform multitask finetuning for target task on a particular dataset (e.g. QNLI). select support samples from other datasets (e.g. SST-2, Subj, QQP, etc.) as finetuning examples. We construct tasks using these examples and apply the same prompt-based finetuning protocol to multitask finetune the model on these tasks. Finally, we evaluate the performance of the finetuned model on the target task.
G.4 TASK SELECTION
The importance of the relationship between the data used in the training tasks and the target task cannot be overstated in multitask finetuning. Our theory measures this relationship through diversity and consistency statements, which require that our finetuning data are diverse enough to capture the characteristics of the test data, while still focusing on the specific regions where the test data aligns. We visualize this diversity and relationship through the feature maps of the datasets.
To visualize the relationship between feature vectors of different datasets, we first obtain text features for each data point in the dataset. We select few-shot samples for generating text features. For each example, we replace the masked word with the true label in its manual template, then we forward them through the BERT backbone. The reason for using BERT over RoBERTa is that the latter only
50

---

Published as a conference paper at ICLR 2024
0.9
trec
mpga
MRPC
QNLI
RTE
CoLA
sst-5
mr
SNLI
QQP
MNLI
trec
mpqa
MRPC
QNLI
RTE
CoLA
subj
d
sst
SST?
mW
SNL
00p NNN
Figure 7: Linear similarity among features vectors among 14 language datasets.
cola: mr, cr.sst-2.sst-5.sub) sst-2: cola.mr, cr,sst-5,subj.
mrpe: qnli, rte qqp: snli, mnli
mnli: snli, qqp
snli: qqp. mnli
qnli: mrpce rte
rte: mipc, qnli
mr. cola, cr.sst-
sst-5: cola,mr, cr,sst-2.subj
subj: cola,mr, cr,sst-2,sst-5
mpqa
cola,mr.stt22sstt-5sut
mpqa: trec
Table 17: Dataset selection.
has masked token prediction pretraining, the [CLS] pretrained RoBERTa model might not contain as much sentence information BERT. Then, we compute the first principal component to obtain a feature vector for each dataset. We illustrate the relationship between these feature vectors in Figure 7.
We further perform training data selection based on the task selection algorithm among the feature vectors, the selected dataset is shown in table Table 17.
By performing task selection, we observed further improvements in multitask prompt-based based finetuning on MR, CR. TREC, MNLI, QNLI, and QQP datasets. However, it's worth noting that the CoLA dataset is an exception, as involves predicting the grammaticality of sentences, and its inputs may include non-grammatical sentences that outside the distribution of masked language models. as
51

---

Published as a conference paper at ICLR 2024
SST-2
SST-5
MR
CR
MPQA
Subj
TREC
CoLA
(acc)
(acc)
(a0c)
(acc)
(acc)
(ace)
(acc)
(Mait)
Prompt-based zero-shot
83.6
35.0
80.8
67.6
51.4
32.0
2.0
Multitask FT zero-shot
92.9
37.2
86.5
88.8
73.9
55.3
36.8
40.065
+ task selection
92.5
34,2
87.1
88.7
71.8
72.0
36.8
0.001
Prompt-based FT"
92.7 (0.9)
(2.5)
87.0 (1.2)
90.3 (1.0)
84.7 (2.2)
91.2 (1.1)
84.8 (S.1)
9.3 (7.3)
Multitask Prompt-based FT
92.0 (1.2)
48.5 (1.2)
86.9 (2.2)
90.5 (1.3)
86.0 (1.6)
(2.9)
83.6 (4.4)
5.1 (3.8)
task selection
92.6 (0.5)
47.1 (2.3)
87.2
91.6 (0.9)
85.2 (1.0)
90.7 (1.6)
87.6 (3.5)
3.8 (.2)
MNLI
MNLI-mm
SNLI
QNLI
RTE
MRPC
QQP
(acc)
(acc)
(acc)
(acc)
(acc)
(FI)
(F1)
Prompt hased Multitask FT zero-shot
50.8
51.7
61.9
63.2
65.7
61.8
65.8
74.0
81,6
63.4
+ task selection
62.4
64.5
65.5
61.6
64.3
75.4
$7.6
Prompt-hused FT"
68.3 (2.3)
(1.9)
77.2 (3.7)
64.5 (4.2)
69.1 (3.6)
74.5 (5.3)
65.5 (5.3)
Multitusk Prompt-based FT
70.9 (1.5)
73.4 (1.4)
78.7
71.7 (2.2)
74.0 (2.5)
79.5 (4.8)
67.9 (1.6)
task selection
73.5 (1.6)
(1.5)
(1.6)
72.0 (1.6)
70.0 (1.6)
76.0 (6.8)
69.8 (1.7)
Table 18: Results of few-shot learning with NLP benchmarks. All results are obtained using RoBERTa-large. We report the mean (and standard deviation) of metrics over 5 different splits. f: Result in Gao et al. (2021a) in our paper, FT: finetuning: task selection: select multitask data from customized datasets.
noted in Gao et al. (2021a). Overall, our approach shows promising results for multitask learning in language tasks.
G.4.1 FULL RESULTS WITH TASK SELECTION
To complement task selection in Table 15, we provide full results here and explain each method thoroughly.
We first elaborate on what each method did each stage. During the testing stage, we conducted experiments in zero-shot and few-shot settings for a given dataset following Gao et al. (2021a), who applied prompt-based methods on moderately sized language models such as RoBERTa. Prompt- based finetuning method updates the model with prompt-based prediction loss for given example. The given example can either be from a testing dataset or other datasets.
Table 18 shows our multitask finetuning and task selection provide helps on target tasks, as detailed in Appendix G.I. We will elaborate on what each method did in the "Multitask fintuning phase" and "Downstream phase".
In the "Multitask fintuning phase": For prompt-based zero-shot (col-1) and prompt-based FT (col-4) do not finetune any model. For Multitask Prompt-based finetuning (c0,-2,3,5,6), we conduct prompt-based finetuning methods using finetuning (auxiliar))))) tasks. The data of tasks are from datasets other than testing datasets. For instance, consider model designated to adapt to dataset (say SST-2), we choose data from other datasets (mr, сr, etc. and combine these data together and form multiple auxiliary tasks, these tasks updated the model using prompt- based finetuning methods. In the "downstream phase" where we adapt the model: In the zero-shot setting (col-1,2.3), we directly evaluate the model's prompt-based prediction accuracy. In the few-shot setting (col-4.5,6), we finetune the model using shot samples from the same dataset (sst-2) and assess its accuracy on the test split.
G.4.2 ADDITIONAL RESULTS ON SIMCSE
We present our results using the same approach described in our paper. However, we used different pretrained loss, namely SimCSE, as proposed by Gao et al. (2021b). However, the results are promising, The reason SimCSE is trained with a contrastive loss instead of masked language prediction, making it less suitable for prompt-based finetuning.
H VISION LANGUAGE TASKS
Pretrained vision-languue as another type of foundation model has achieved tremendous success across various downstream tasks. These models, such CLIP (Radford et al., 2021) and ALIGN (Jia
52

---

Published as a conference paper at ICLR 2024
SST-2
SST-5
MR
CR
MPQA
Subj (acc)
TREC
CoLA
(acc)
(acc)
(acc)
(acc)
(acc)
(acc)
(Mat.)
Prompt-based zero-shot
50,9
19.3
50
50
50.4
27.2
Multitask FI zero-shot
51,3
13.8
50
50
50
50.6
18.8
0
Prompt-based FT?
51.8(2.6)
20.5 (6.1)
50.6 (0.8)
50.8 (1.1)
52.3 (1.9)
(3.7)
19.8 (7.3)
0.8 (0.9)
Multitask Prompt-based FT
(0.7
22.1 (6.2)
50.5 (1.0)
(1.5
53.4 (2.7)
51.0
26.4 (8.5)
0.9 (1.3)
task selection
51.7
19.7 (5.6)
50.6 (0.8)
51.6
52.3 (2.7)
54.7 (2.5)
23.2 (9.9)
(0.7)
MNLI
MNLI-mm
SNLI
QNLI
RTE
MRPC
QQP
(acc)
(acc)
(acc)
(ace)
(acc)
(FI)
(F1)
Prompt-based zero-shot
35,4
35.2
33.8
50,5
47.3
1,4
Multitask FT zcro-shot
35,4
35.2
33.6
49.5
47.3
53.8
53.8
Prompt based FT
32.9 (0.8)
33.0 (0.7)
33.7 (0.6)
50.6 (1.4)
48.7 (3.7)
79.2 (4.1)
53.5 (2.7)
Multitack Prompt-based FT
32.5 (0.6)
32.5 (0.7)
33.5 (0.4)
50.6 (2.4)
50.0 (2.0)
76.3 (6.5)
54.2 (0.8)
task selection
(1.2)
(1.1)
35.0 (0.8)
50.3 (0.4)
51.8 (2.0)
72.2 (10.8)
(3.0)
Table 19: Our main results using simCSE (Gao et al., 2021b), We report mean (and standard deviation) performance over 5 splits of few-shot examples. FT: fine-tuning; task selection: select multitask data from customized dataset.
et al., 2021), align images and text in a shared space, enabling zero-shot classification in target tasks. Finetuming such models has resulted in state-of-the-as accuracy in several benchmarks.
Vision-language model enables the classification of images through prompting, where classification weights are calculated by a text encoder. The text encoder inputs text prompts containing class information, and outputs features aligned with features from the vision encoder in the same space.
However, standard finetuning can be affected by minor variations underperforming direct adapta- tion (Kumar et al., 2022; Wortsman et al., 2022). Additionally, standard finetuning can be computa- tionally expensive, as it requires training the model on a large amount of target task data.
We perform our multitask finetuning pipeline on the vision langua- model and observe certain im- provements. It's worth mentioning although the vision-language model is pretrained using contrastive learning, the model does not align with our framework. Vision-language model computes contrastive loss between image and text encoder, whereas our pretraining pipeline formulates the contrastive loss between the same representation function for positive and negative sample pairs. Despite the discrepancy, we provide some results below.
H.1 IMPROVING ZERO-SHOT PERFORMANCE
We investigate the performance of CLIP models in zero-shot setting, following the established protocol for our vision tasks. Each task includes 50 classes, with one query image per class. We employ text features combined with class information as the centroid to categorize query images within the 50 classes. During adaptation, we classify among randomly selected classes in the test split, which consists of 50 classes.
We experimented with our methods on tieredImageNet and DomainNet. The text template utilized in tieredImageNee was adapted from the CLIP documentation. In adaptation, we classify among all classes in the test split (160 classes in tieredImageNet and 100 classes in DomainNet). For text features on tieredImageNet, we use templates adapted from CLIP photo of a O), itap of a (h, a bad photo of the (), origami i a photo of the large O, () in a video game, art the (,, a photo of the small (). For templates on DomainNet, we simply use a photo of (). In the DomainNet The text template used for this experiment "a photo of ()", We perform Locked- Tuning, where we fixed the text encoder and update the vision encoder alone.
Table 20 demonstrates that CLIP already exhibits a high level of zero-shot performance. This is due to the model classifying images based on text information rather than relying another image from the same class, which enables the model to utilize more accurate information to classify among query images. We show the effectiveness of zero-shot accuracy in tieredImageNet and DomainNet. is worth highlighting that our multitask finetuning approach enhances the model's zero-shot performance, particularly the more realistic DomainNet dataset. We have observed that
53

---

Published as a conference paper at ICLR 2024
Backbone
Method
tieredImageNet
DomainNet
ViT-B
Adaptation
84.43 (0.25)
70.93 (0.32)
Ours
84.50 (0.25)
73.31 (0.30)
ResNet50
Adaptation
81.01 (0.28)
63.61 (0.34)
Ours
81.02 (0.27)
63.55 (0.34)
Table 20: Multitask finetune on zero-shot performance with CLIP model.
multitask finetuning pipeline yields greater improvements for tasks on which the model has not been extensively trained.
H.2 UPDATING TEXT ENCODER AND VISION ENCODER
We also investigated whether updating the text encoder will provide better performance. On the tieredlmageNer dataset, We finetune the text encoder and vision encoder simultaneously using the contrastive loss, following the protocol in Goyal al. (2023).
Method
Zero-shot
Multitask finetune
Accuracy 84.43 (0.25)
85.01 (0.76)
Table 21: Multitask finetune zero-shot performance with ViT-B32 backhone tieredImageNet..
In Table 21, we observed slightly better improvements compared to updating the vision encoder alone. We anticipate similar performance trends across various datasets and backbone architectures. We plan to incorporate these findings into our future work.
H.3 CoCOOp
We also multitask finetune the vision language model following the protocol outlined in Zhou et al. (2022a). This approach involved prepending image-specific token before the prompt enhance prediction accuracy. To generate this token, we trained small model on the input image. We evaluate the performance of our model all classes the split, which corresponds a 160-way classification task. This allows us to comprehensively assess the model's 's ability to classify among large number of categories.
Method
Zero-shot
Multitask finetune
VIT-B32
69.9
71.4
Table 22:
Multitask finetune zero-shot performance with ViT-B32 backbone tieredImageNet.
Table 22 showed the result of the performance of the CoCoOp method. We observed an improvement of 1.5% in accuracy on direct adaptation.
54


Reviews:
[{"review_id": "N32RQmi8m7", "reviewer": "Reviewer_72L9", "summary": "This paper explores how to effectively adapt foundational models to new tasks, especially those with limited labeled data. The authors investigate a multi-task fine-tuning approach, wherein the foundational model is fine-tuned for a set of related tasks before fine-tuning it for the target task.", "questions": "1. In Section 1.1, it would be beneficial to provide a more detailed enumeration of recent advancements in the field of multitask fine-tuning.\n2. I suggest adding some diagrams in Section 3.2 to visually illustrate the entire process. Visual representation will help readers gain a clearer understanding and assess the effectiveness of task selection.\n3. I suggest conducting more detailed ablation experiments for TASK SELECTION to provide stronger evidence for its effectiveness.\n4. The experimental section lacks sufficient detail, such as the absence of descriptions for hyperparameter settings. Providing these details would make the findings more convincing.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper provides a theoretical analysis, revealing that fine-tuning with a diverse set of related tasks reduces errors in the target task compared to directly adapting a pre-trained model. The authors introduce diversity and consistency metrics to quantify the relationship between fine-tuning tasks and the target task and propose a practical task selection algorithm.", "weaknesses": "1. In Section 1.1, it would be beneficial to provide a more detailed enumeration of recent advancements in the field of multitask fine-tuning.\n2. I suggest adding some diagrams in Section 3.2 to visually illustrate the entire process. Visual representation will help readers gain a clearer understanding and assess the effectiveness of task selection.\n3. I suggest conducting more detailed ablation experiments for TASK SELECTION to provide stronger evidence for its effectiveness.\n4. The experimental section lacks sufficient detail, such as the absence of descriptions for hyperparameter settings. Providing these details would make the findings more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NJ4TVO3xMg", "reviewer": "Reviewer_1yNr", "summary": "Given a pretrained foundation model, a target task with limited labeled samples, and a bunch of auxiliary tasks possibly with many labeled data, this paper analyzes how to select good auxiliary tasks to finetune the foundation model first, then adapt the auxiliary-task-finetuned model to the target task (a.k.a multitask finetuning), such that the resultant model can outperform the model that directly adapted to the target task from pretraining, or the model that finetuned with all the auxiliary tasks before adaption. To this end, the authors derive several bounds which indicate that the model with multitask finetuning outperforms that without, and the selected auxiliary tasks should be sufficiently diverse and have good consistency w.r.t. the target task. According to this, the authors formulate an auxiliary-tasks-selection algorithm, whose outperformance over standard finetune is validated on various datasets.", "questions": "1. Why does the performance decrease with the increase of sample shots in Fig. 2a?\n2. Is Standard FT in Table 2 the same as All in Table 1?\n3. First Paragraph on Page 4, In contrastive pretraining -> In contrastive to pretraining; First Paragraph on Page 7, the cosine similarity similarity -> the cosine similarity\n4. It seems that Theorem 3.3 (or the index 3.3) is missing in the main text.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Several bounds about multitask finetuning are derived.\n2. The multitask finetuning with task selection outperforms that finetuned on all the tasks.\n3. The experiments have been extensively performed on a dozen of datasets.", "weaknesses": "1. While several bounds have been derived, their implications are quite straightforward. For instance, the model with multitask fine-tuning outperforms the one without it, and the selected auxiliary tasks should exhibit sufficient diversity and strong consistency w.r.t. the target task, are well accepted by the community, despite those being derived based on the somewhat strict assumptions. It would be more valuable if these bounds could lead to novel insights that were previously overlooked by the community.\n2. Such limitation also lies in the task-selection algorithm, where the authors simply assume a Gaussian distribution of features, then use the angle of mean and variance-scaled L2 distance indicating the consistency and the diversity.\n3. As \\phi evolves during the training, should Algorithm 1 perform repeatedly during the training?\n4. The paper in its current version is quite difficult to read, it is suggested to give more intuition before diving into the derivation, and also simplify/brief the symbols if applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YFJ8d43cjo", "reviewer": "Reviewer_mX1E", "summary": "This paper presents theoretical results that to adapt a pretrained model to a target task, it is beneficial to first finetune the model on a diverse set of related tasks and then tune the model on the target task. The authors prove the theorems by first defining the diversity and consistency between different tasks that the model is going to adapt to, where diversity refers to the coverage of the other tasks on the target task and consistency refers to the similarity between the other tasks and the target task. Then, under some Lipschitzness assumptions, sufficient consistency and diversity assumptions, and sufficient tasks and sufficient sample assumptions (on the finetuning tasks, not the target task), the model with multitask finetuning can achieve a reduced error on the target task compared to the setting without multitask finetuning. Moreover, the authors propose a multitask selection algorithm based on the consistency and diversity requirements. Experimental results indicate that with the proposed multitask selection, the ViT-B model achieves better results on multiple target datasets, and increasing the number of tasks and the number of samples per task are most effective in improving target task performance. Lastly, with multitask finetuning, various pretrained models achieve better results than direct adaptation / standard finetuning on multiple datasets.", "questions": "- I found the notations in Section 3.1 (Linear Data and Tasks) somewhat difficult to understand. $T_{z, z'}$ is discussed in the text but how is it related to the $T_i (i = 1, \\dots, M)$ in figure 1? It would ease the reader's understanding by revising the explanation in that part. \n- Does the Experiment Protocol apply to all three major experiments (4.1, 4.2, and 4.3)? It seems to me that the experiments have their own protocols, which causes confusion.\n- What are the multitasks for finetuning in Section 4.3?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presented theoretical results are quite intuitive -- if the tasks in multitask finetuning are diverse and consistent with the target task $T_0$, then the model with pretraining followed by multitask finetuning achieves better target prediction performance than the model only with pretaining. My intuition is that, through the training/finetuning on the proxy tasks, the model learns some knowledge related to the target tasks. It is nice that the authors present the theorems that support this intuition, although I didn't verify the proofs. \n- Although the theorems are not directly applicable to real problems, the authors propose a practical Consistency-Diversity Task Selection algorithm, which is effective in experiments. The algorithm is a nice method to try for target task adaption in the real world.\n- The overall experiments are quite extensive and echo the theoretical results, clearly demonstrating the advantages of multitask finetuning over standard finetuning and direct adaptation.", "weaknesses": "- The Gaussian distribution assumption in Section 3.2 may not be realistic, leading to biased task selection that may be suboptimal to the model performance on the target task. However, it is okay to derive further refined algorithms in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZSDid0UnX", "reviewer": "Reviewer_AgCa", "summary": "This paper studies multitask finetuning for adapting foundation models to new tasks in few-shot. It provides a theoretical justification to analyze the multitask finetuning and a practical task selection algorithm for choosing good finetuning tasks. Extensive experiments are conducted to validate the proposed algorithm.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a theoretical framework to analyze multitask finetuning for foundation models, providing definitions and analysis of task diversity and consistency. Based on the theoretical concepts, the authors introduce a practical task selection algorithm.\n- The authors conducted extensive experiments and showed the effectiveness of the proposed method compared to the direct adaptation and full fine-tuning baseline.\n- The overall writing is good, which clearly explains the problem, approach, analysis and results.", "weaknesses": "- The theoretical part discussed diversity and consistency, but the method part simplifies consistency to similarity and diversity to coverage. So the final algorithm is basically a similarity-based sorting. The simplicity is not a problem, but I wonder if it is really related to the theoretical concepts.\n- The authors evaluated the proposed algorithm on a wide range of datasets/models, but how does it compare with some stronger baselines besides direct adaptation and full fine-tuning, e.g. meta-learning algorithms?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)