Tackling Diverse Tasks via Cross-Modal Transfer Learning
Authors: Junhong Shen
Liam Li
Lucio M. Dery
Corey Staten
Mikhail Khodak
Graham Neubig
Ameet Talwalkar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Cross-modal transfer learning
pretrained models
fine-tuning

Abstract: Fine-tuning large-scale pretrained models has led to remarkable progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many other tasks due to an assumed lack of relevant pretrained models for these diverse modalities. In this work, we revisit this assumption by studying the cross-modal transfer ability of large-scale pretrained models. We introduce ORCA, a general cross-modal fine-tuning workflow that enables fast and automatic exploitation of existing pretrained models for diverse tasks. ORCA achieves task-specific adaptation by learning feature embeddings that minimize an optimal transport distance metric to map the data distribution in the end-task modality to the pretraining modality. We test ORCA on 13 tasks with varying modalities and input-output types. ORCA performs the best on 10 of them and is in the top three on the others. We further quantify the importance of embedding distance for downstream performance, highlight ORCA’s utility for data-limited tasks, and demonstrate its compatibility with same-modality transfer.

Under review as a conference paper at ICLR 2023
TACKLING DIVERSE TASKS VIA CROSS-MODAL TRANSFER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Fine-tuning large-scale pretrained models has led to remarkable progress in well- studied modalities such vision and NLP. However, similar gains have not been observed in many other tasks due to an assumed lack of relevant pretrained models for these diverse modalities. In this work, we revisit this assumption by studying the cross-modal transfer ability of large-scale pretrained models. We introduce ORCA, general cross-modal fine-tuning workflow that enables fast and automatic exploitation of existing pretrained models for diverse tasks. ORCA achieves task- specific adaptation by performing data alignment before fine-tuning: it learns an embedding network that minimizes the optimal transport dataset distance between the end-task data and the pretraining data to close the modality gap. Through ex- tensive experiments,
show that ORCA is the first viable approach that allows practitioners to use pretrained models to outperform hand-designed, AutoMI. searched, and general-purpose architectures- sults on 10 of 13 diverse tasks we evaluate and ranks among the top three on the others. We shed light on why cross-modal transfer works by quantifying the im- portance of data alignment and highlight ORCA's 's utility for data-limited domains.
obtains state-of-the-art re-
INTRODUCTION
The success of machine learning (ML) in vision and natural language processing (NLP) has spurred its application beyond these traditional ML domains to diverse tasks such as solving partial differ- ential equations (Li et al., 202 music modeling (Lewandowski et al. 2012). detecting cardiac disease (Hong et al. 2020). and many others. However, progress in these less-explored areas can be challenging due to (1) limited amounts of labeled data, (2) high computational cost and human effort for developing models from scratch, and (3) a lack of relevant large-scale pretrained models, which have in many cases obviated the first two issues in vision and NLP (e.g.. Devlin et al. 2019 Carion et al.) 2020 Dosovitskiy et al.) 2021 Liu et al. 2021b Radford et al. 2021).
There are two common approaches for practitioners to handle these issues: automated machine learning (AutoML) techniques (e.g., Roberts et al., 2021 Shen et al. 2022) that focus on designing task-specific networks in data-efficient manner; and multimodal general-purpon methods that either propose flexible architectures applicable to various tasks (Jacgle et al. 2022a) or expand the set of modalities for which pretrained models exist (c.g., Reed et al. 2022 Lu al. 2022a). However, both classes of approaches require training from scratch when applied to a new modality and proceed under the assumption of a lack of relevant pretrained models for these diverse problems.
In this work, we re-examine this assumption by considering the general problem of cross-modal transfer. Our goal is to exploit existing large-scale pretrained models in data-rich modalities for solving diverse downstream tasks. A few recent works have demonstrated the potential promise of cross-modal transfer by applying language transformers to vision (Kiela et al., 2019, Dinh et al.
2022 Lu et al. 2022b), referential games (Li al.) 2020c), and reinforcement learning (Reid al. 2022). ) However, many of these approaches are ad-hoc (e.g., rely on manual prompt engineering
or hand-craft new architecture components to solve specific tasks), and none of them yield models competitive with those trained from scratch. We tackle both shortcomings in our work.
We introduce
purposs, mrosl transfer workflow called ORca (Optimal tRansport fross-modal Adaptation) that yields state-of-the-an results on a wide range of non-text and non- vision problems using pretrained transformers (Figure T). Our key insight is align the feature

---

Under review as a conference paper at ICLR 2023
Pretrained Transformer
Inputs
Stage 1: Construct Task Embedder & Predictor
Stage Learn Task Embedder
Stage 3: Fine-Tune
Entire Architecture
Embedder
y
y
Body
Target Dataset (a's 3/)
A'
Embeddded Target 2
Embedded Source di
Source Dalaset (a', v/)
Minimize Optimal Transport Dataset Distance OTDD(( (a", wz))
Task Loss
Figure I: ORCA's 's three-stage cross-modal transfer workflow enables fast and automatic exploitation of large- scale pretrained models in data-rich modalities for solving diverse tasks, First, given target data (x", y²) and a pretrained transformer body 9" ORCA constructs embedder architecture f' to match the input dimension- ality of Ja. and predictor architecture h' convert the output g back the appropriate output space for the target task, e.g., classification logits or dense maps. Note that ORCA does not learn the weights for Sa or by during this stage. Next, ORCA learns the parameters for the embedder f by minimizing the OTDD between the target dataset and an in-modality source dataset. Finally, ORCA fine-tunes the entire architecture (ʃ" .h".
distribution of an unfamiliar, out-of modllity dataset with that of a familiar, in-modal dataset before fine-tuning. This data alignment process not only prevents distortion of pretrained weights but also enables cross-modal knowledge transfer, as we will show via extensive experiments in Section I Concretely, for any downstream task, we first generate an embedding network that maps the (po- tentially high-dimensional) inputs to sequence features. Then, we train it to minimize the optimal transport dataset distance (OTDD) KAlvarez-Meli & Fusi (020) between the feature-label distribu- tion of the target data and data from the pretraiming domai Finally, we fine-tune the pretrained model and the embedding network. Using OTDD allows us relax many distributional assump- tions required by traditional domain adaptation and perform data alignment using both the feature and label information of the target data. However, we show in an ablation study in Section 4.2. that substituting OTDD with other distance metrics, such as maximum mean discrepancy (MMD) (Gret- ton et al. 2012), can also aid tross-modal transfer, albeit to lesser extent. This implies that is the general idea of first-align- alint-thnnn----n---- then-fine-ture fine-tune that enables ORCA to obtain significantly better results than previous cross-modal learning methods that rely on vanilla fine-tuning (Lu et al.. 20226).
We evaluate ORCA on a diverse set of 13 tasks with different input dimensions (ID and 2D), pre- diction types (point and dense), and modalities (vision, audio, electrocardiogram, physics, protein, genomics, cosmic-ray, and music). ORCA outperforms various competitors, including task-specific hand-designed architectures, leading AutoML methods, and general-purpot on 10 tasks and in the top three on all tasks. We compare ORCA with existing fine-tuning techniques and confirm that effective cross-mode transfer is only enabled by ORCA's 's feature aligmment pro- cess. We further reveal empirical correlation between the alignment quality and the downstream performance. Finally, we demonstrate ORCA's 's efficacy for limited-data tasks. Overall, our work not only explores the cross-modal transfer ability of pretrained models, but also establishes practical workflow for solving diverse prediction problems efficiently and automatically.
models, ranking first
2 RELATED WORK
In this section, we review several groups of related work in the areas of AutoML, in-modal trans- fer learning (unimodal domain adaptation, unimodal/multimodal fine-tuning, and general purpose methods), and cross-modal transfer learning (heterogeneous domain adaptation, task-specific fine- tuning, and FPT). Table summarizes these groups along relevant axes, and contrasts them ORCA.
Auto.L for diverse tasks is a growing research area, as evidenced by the NAS- MAS--ench-366 bench- mark (Tu et al. 2022), along with several recent neural architecture search (NAS) methods that target this problem, e.g., AutoML-Zero (Real et al. 2020), XD (Roberts et al. 2021 and DASH (Shen) et al. 2.22)) ). In contrast to these NAS methods, ORCA takes transter learning approach in order to leverage existing pretrained models from data-rich modalities for more esoteric tasks, rather than re- peatedly incurring the overhead of designing new architectures and training them from scratch. That said, given the shared underlying motivation, our experimental evaluation makes use of the diverse
We do not assume access to the pretraining data due to practical concerns about data access and compu- tational efficiency. We instead work with publicly available proxy data from the pretraining modality, e.g., CIFAR-10 for models pretrained ImageNet and CoNI.I-2003 for models pretrained larger text corpora.
2

---

Under review as a conference paper at ICLR 2023
Table I: Summary of existing approaches aiming develop models for diverse tasks.
Task-specific aduptation?
General-purpose workflow?
Supports transfer different: input dim? output dim? modality?
Task-specific
Hand-designed models AutoML models
learning
Unimodal DA
In-modal
Uni/Multimodal fine-tuning
transfer
General-purn purpose models
Heterogencous DA Task-specific fine-tuning FPT ORCA
Cross-modal
transfer
tasks comprising NAS-Bench-360, and compares ORCA with its expert and AutoML baselines. We also compare against DASH, the state-of-the-arr method on this benchmark.
Unimodal domain adaptation (DA) is a form of transductive transfer learning where the source and target tasks are the same but the domains differ (Pan & Yang 2009; Wang & Deng 2018). Many DA methods assume that the target data has the same input space and support as the source data, and are concerned with problems where the output spaces and the joint/marginal distributions differ, such covariate and label shifts. Recent work considers more general settings such different feature spaces (heterogeneous DA) or label spaces (universal DA). Our focus on cross-modal transfer goes one step further to the case where neither the input-space nor the output-space support overlaps.
Unimodal fine-tuning is a more flexible transfer approach that can be applied to downstream tasks with different lahel spaces or input spaces. Pretrained models are used for in-modality fine-tuning in NLP (c.g.. (Aghajnyyn et al. 20211 Jiang et al. 2020), vision (e.g., WWi et al., [202]] Li et al.
2022). speech (c.g.. Chen et 2022 Jiang et al.] 2021 ), protein sequences (Jumper et al. 2021),
and robotics (Ahn et a 20022) Adapter networks (He et al., 2022) have been developed to improve the downstream performance of in- mnmodaaity transfer. Multimodal fine-tuning expands the applica-
ble modalities of a single pretrained model by learning embeddings of several data-rich modalities
together (e.g.. Lu et al., 2019; Radford et al. 2021 Hu & Singh 2021 Kim et al., 2021 Alayrac
et al. 2022). However, these approaches still focus solving in-modality downstream tasks.
General-purpose models propose flexible architectures applicable to various tasks such as optical flow, point clouds, and reinforcement learning (Jaegle et al. [021 2022a Reed et al., 2022). These approaches train multitask transformers from scratch using a large body of data from different tasks. Though more versatile than unimodal models, they still focus transferring to problems within the pretraining modalities considered. Nonetheless, the success of transformers for in in-modality fine- tuning motivates us focus on adapting transformer-type architectures for cross-modal transfer.
Heterogeneous DA (HDA) considers nonequivalent feature spaces between the source and target domains. While most HDA methods are developed for e.g., between images of different resolutions, there are indeed a few works studying cross-modal tasks such as text-to-image (Yao al. 2019 Li al. 2020b). However, a crucial assumption that HDA makes is that the target and source tasks are the same. Thus, we operate in much more flexible setting and consider knowledge transfer between drastically different domains with distinct tasks and label sets, such as applying Swin Transformers (Liu et al., (202cc) to solving partial differential equations or RoBERTa to classifying satellite images and electrocardiograms.
transfer,
Cross,modal, task-specific fine-tuning is a recent line of research, with most work focusing on transferring NLP models to other modalities like vision (Kiela et al.) 2019), referential games (Li al. 2020c). and reinforcement learning (Reid et al. (022). These works provide initial evidence of the cross-modal transfer capacity of pretrained models. However, they focus on hand-tailoring to
single modality, e.g., by adding ad-hoc encoders that transform agent messages (Li et al. 2020c)
or decision trajectories (Reid et al. 2022) into tokens. Even when not relying on fine-tuning, work
like LIFT (Dinh et al. 2022) that attempts cross-modas learning via prompting (Liu et al. 202 a))
still require ad-hoc conversion of tasks to natural text.
Frozen Pretrained Transformers (FPT) (Lu al. 2022b) is a general cross-modal fine-tuning workflow that transforms input features to be compatible with the pretrained models. Although FPT and ORCA are both target and pretraining modalities, which we show is necessary to achieve accurate predictive models and outperform existing baselines.
workflows, FPT does not account for differences between the
3

---

Under review as a conference paper at ICLR 2023
3 ORCA WORKFLOW
In this section, we first formalize the problem setup and then introduce the ORCA workflow for adapting pretrained transformers to diverse end tasks.
Problem Setup. A domain D consists of a feature space X, label space V. and a joint probability distribution P(X, y). In the cross-modal setting we study, the target (end-task) domain DD' and source (pretraining) domain De differ not only in the feature space but also the label space and by extension have differing probability distributions, i.e., X * X², ) + y", and (X", ) of (* ). This is in contrast to the transductive transfer learning setting addressed by domain adaptation, where source and target domains share the label space and end task (Pan & Yang 2009).
Given target data {at, RE(nn] sampled from a joint distribution p' in domain D', our goal is to learn a model m² that correctly maps each input x' to its label y². We are interested in achieving this using pretrained transformers. Thus, we assume access to a model m² that has been trained with data {ar y he( the source domain D", where (a:7. 3/¹) P Then, given a predefined loss function 1, we aim to develop m' based on m such that L(m") = E )r) (((m"(x"), y²)] is minimized. This problem formulation does not define modality explicitly and includes both in- modal and cross-modal transfer. Given the generality of the tasks we wish to explore, it is hard to provide precise mathematical definition, so we rely semanties to differentiate the two settings: intuitively, cross-modal domains (e.g., natural images vs. protein sequences) are more distinct to each other than in in-modal domains (e.g.. photos taken in two different geographical locations).
Having defined the learning problem, we now present our shree-stage cross- modal transfer workflow: (1) architecture design to support diverse input-output dimensions, (2) embedder pretraining to align the source and target feature distributions, and (3) fine-tuning to minimize the target task loss.
3.1 TASK-SPECIFIC ARCHITECTURE DESIGN
Applying pretrained models to another downstream problem usually requires addressing the prob- lem of mismatched dimensions. To make ORCA work for input and output tensors of different dimensions, we decompose a cunsformer-based learner m into three parts (Figure [ stage 1): an embedder f that transforms input x into a sequence of features, a model body g that applies pre- trained transformer (i.e., series of attention layers) to the embedded features, and a predictor h that generates predictions with the desired output shape. ORCA uses pretrained architecture and weights to initialize the model body g but replaces f and h with layers designed to match the target data with the pretrained model's embedding dimension. Next, we describe each module in detail.
Custom Embedding Network. Denote the feature space compatible with the pretrained model body as X. For transformer with maximum sequence length S and embedding dimension D, X RSXD The embedding network f X + E is designed to take in a tensor of arbitrary dimension from X' and transform it to the feature space X. In ORCA, f is composed of a convolutional layer with input channel Ca., output channel Cout, kernel size k, and stride k, generalizing the patching operations used in vision transformers to 1D and higheerdimeeniionll cases. We set Can to the input channel of x and Cout to the embedding dimension D. To take full advantage of the representation power of the pretrained model, we choose the smallest k for which the product of output shape excluding the channel dimension < S. That is, when we flatten the non-channel dimensions of the output tensors after the convolution, pad and then transpose it, we can obtain sequence features with shape S x D. Finally, we add layer norm and a positional embedding to obtain 3
Pretrained Transformer Body. The model body takes the embedding à E & as input and outputs features y E )) the dot is used to differentiate these intermediate representations from the raw inputs and labels. For mansformer-based g, both the input and output feature spaces X,, )) are SXD
Custom Prediction Head. Finally, the prediction head must take ÿ E ) as input and return a task-dependent output tensor. Different tasks often specify different types of outputs, e.g., classi- fication tasks require logits in TR K where K is the number of classes, and dense prediction tasks require dense maps with the same spatial dimension as the input and per index logits correspond-
2 As a concrete example, consider an image tensor with shape (Cin, Hin, Won). ). We first choose stride & for the convolution such that Hout x Wout ss S to get output tensor with shape (D) Hout, W Wout). Then, we flatten it to shape (D, Hout x Wout). pad along the last dimension to shape ( (D, S), and transpose.
4

---

Under review as a conference paper at ICLR 2023
ing to K classes. Thus, it is crucial to define task-specific output modules and fine-tune them when transferring to new tasks. In our workflow, we use the simplest possible instantiation of the predictor modules. For classification, the classification token of language models) to obtain 1D tensors with length /) and then use a linear layer that maps D K. For dense prediction, we apply linear layer to the sequence outputs so the resulting tensor has shape (S, {ddim())-11 K), where Adimm)))--1 is the downsampling factor of the embedder convolution kernel with stride k. This upsamples by the same factor that the embedder convolution downsampled. Then, we can mold the tensor to the desired output dimension f
apply average pooling along the sequence length dimension (or take
With an architecture based on the pretrained model but is also compatible with our target task, we now turn our attention pretraining the embedder via matching source and target distributions.
3.2 EMBEDDING LEARNING DOR DATA ALIGNMENT
Intuitively, transferring knowledge across similar modalities should be easier than across distant ones. Hence, given a target task in a new modality, we aim to manipulate the task data so that they become closer to the pretraining modality. We use the optimal transport dataset distance (OTDD) Allarez--eeiis & Fusi, 2020) to measure the closeness between datasets in different domains. Unlike the classic OT distance which operates only on the feature distributions, OTDD considers both the feature and the label information and can work even if the label sets are unrelated or disjoint. Thus, while OT is mostly used for unsupervised or semi-supervised domain adaptation Courty al. 2017 Yan et al. 2018). OTDD is particularly suitable for distance estimation between cross-modal labeled datasets. In the following, we will briefly explain how ORCA uses OTDD. We refer the readers Alvarez-Melis & Fusi 2020 for detailed exposition the metric itself.
Formally, let f X + X' denote the pretrained embedder (the part of m that transforms the source data to sequence features) and ft x² + X' be randomly initialized target embedder with architecture discussed in the previous section. We train f to minimize the expected OTDD) between the embedding-label distributions (f*(x"), v') and (3*)),, v²). That is, for both datasets, we first represent each class label as a distribution over the in-class features: y + P(X = y). This transforms the source and target label sets into the shared space of distributions over X'. Then, we can define the distance dy(y", :y") between different labels using the p- p-Wasserstein distance associated with a metric dø over the feature space, e.g., the l2 distance || - 1 This allows us to measure the difference between distributions in &' x y using the following p- Wasserrstiin metric:
dxxx ((+ v'), (#", v²)) (dx( D'P ,,()""")))"" 1/2
Plugging this into the OT formulation leads the OTDD over A x x), which we optimize learn f*.
Leveraging the clustering structure of the datasets, OTDD provides a better distance estimation be- tween the target and source data, demonstrating better alignment ability in practice. In Section 4.2. we examine several distance metrics for learning the emhedder. While data alignment generally im- proves downstream performance for all metrics, OTDD leads to the best empirical results.
As for the computational cost of embedding learning, we analyze the complexity of OTDD in Ap- pendix A.I and show that this stage takes much less time than the later fine-tuning stage in Ap- pendix A.5 Thus, ORCA achieves a significant performance gain at a low cost for feature alignment.
3.3 FINE-TUNING FOR DOWNSTREAM ADAPTATION
After training the embedder, we perform full fine-tuning by updating all model parameters to mini- mize the target loss. This step further aligns the embedder and predictor with the pretrained model to improve downstream performance. We perform ablation study comparing ORCA to standard fine-tuning without feature matching in Section 4.2.1 and show that our approach improves predic- tion accuracy and reduces performance variance. There are orthogonal lines of work that study how to best tne-tune& a pretrained model (e.g., Liu et al. 2022: He et al. 2022). We compare with one strategy used in FPT (Lu et al. 2022b) Section 4.2.2 but leave further exploration for future work.
3 As concrete example, for an image tensor with embedding convolution kernel size k. the linear layer will yield output of shape (S. k.2 K), which we transpose, pad, and reshape (I:² K, Hous, Wout). Finally, we apply pixelshuffle (Shi 2016) to get output of shape (K, Hen, W..).
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
Having introduced how ORCA addresses the dimension mismatch between the target and source datasets via architecture design and tackles the distribution mismatch via embedder learning, we now proceed with showing its empirical effectiveness. In the following, we will demonstrate that ORCA is the first approach that allows practitioners to obtain models better than hand-designed, AutoML- searched, and general purposs architectures on variety of diverse tasks. Then, we will analyze key components of ORCA to better understand the mechanism underlying cross moddll transfer.
Experiment Protocol. While our workflow accepts a wide range of pretrained transformers as model bodies, we use RoBERTa (Liu et al.) 2019b) and Swin Transformers (Liu et al.) 2021c). which are representatives of the most studied language and vision modalities, to exemplify ORCA's efficacy. We implement the base models, which have around 100 million parameters, and use the pretrained weights available in the Hugging Face transformers library (Wolf et al.) (2019). As stated in the introduction, because they are often not publicly available and can be too large to compute OTDD efficiently. We use the proxy datasets: CoNLL-2000 for RoBERTa and CIFAR-IO for Swin.
do not the exact pretraining data represent the source modalities
For each task, we first apply the hyperparameter tuning algorithm ASHA (Li et al. 2020a) to the standard fine-tuning baseline ("Fine-tuning" in Table 3) to identify suitable batch size, optimizer, learning rate, and weight decay. These hyperparameters are then applied to all fine-tuning baselines as well as ORCA. During embedder learning, while classification tasks naturally come with discrete labels required for computing OTDD, for dense prediction tasks where labels are high-dimensional maps, we perform clustering on the dense maps to generate pseudo labels, which not only preserves the intrinsic distribution of the target data but also speeds up OTDD computation. We manage our experiments using the Determined AI platform. All experiments are performed on NVIDIA V100 GPUs and results are averaged over 5 random seeds. For other experiment details, see Appendix A.3
4.1 CAN PRETRAINED MODELS TRANSFER ACROSS MODALITY TO SOLVE DIVERSE TASKS?
In this section, we highlight the most important observation of this work: cross-modal fine-tuning with ORCA can solve a variety of tasks effectively and efficiently. To demonstrate this, we evaluate ORCA on 13 tasks detailed below. We first include 10 tasks from NAS (AS-Bench-36CF) which covers problems such as PDE solving, protein folding, and cardiac disease detection. This benchmark contains tasks for 1D and 2D classification, 2D dense prediction, but not ID dense prediction, so we added JSB Chorales, a music modeling dataset widely used for evaluating recurrent networks (Chung et al., 2017 Bai et al. 2018). We also added ListOps (parsing math expressions) (Tay al., 2021) and Homology (classifying protein structure) (Rao et al. (019) for comparison with FPT. Together, these 13 tasks represent wide collection of modalities for comprehensive evaluation.
Following the taxonomy in Table 1 we consider three classes of baselines: (1) hand-designed expert architec-
Fraction of r-Suboptimal Tasks
1.0
tures for each task, as identified by |Tu et al.] (2022),
Rao et al. (211)). and TTa et al.] (2021): (2) general-
0.8
ORCA
purpose models, as represented by Perceiver I0 (Jacgle af., 2022b); and (3) AutoMI. baselines, as represented by those evaluated in
0.6
Hend-designed
Perceiver
0.4
NAS Bench 360
and DASH (Shen
0.2
DASH
FPT
(ct al. 2022). We will compare with FPT later, the only re-
0.0
10
12
maining approach with general workflow from Table
In Table we report the prediction error for each method on each task. ORCA achieves the lowest eиror rate on 10 of 13 tasks and is the most effective in terms of aggregated performance. This is also supported by the performance summary in Figure? 2 More specifically, we outperform all hand-designed architectures all tasks except ECG, where we rank second but do much better than the other
Figure 2: Aggregate performance of ORCA and baselines for Table 2 results measured by performance profiles (Dolan & More 22022.. Larger values (larger fractions of tasks on which a method is within factoor of the best) are better. ORCA being in the top left corner means it is often the best.
'CoNI.L-2003 is for named entity recognition. It is used to interpret language models Jawahar et al. 2019 NNAS-Bench-3G is designed for testing how well MI. algorithms can generalize and is core component of the 2022 AL. Decathlon competition. For summary of included tasks, Table the Appendix

---

Under review as a conference paper at ICLR 2023
Prediction errors (lower is better) for 13 tasks across diverse application domains. On 10/13 problems, ORCA ranks the first among all hand-designed expert models, AutoML. and general-purpose methods. NAS- Bench-360 refers to the task-wise best of all baselines evaluated the benchmark, including DARTS (Luu et al.. (2019)), DenseNAS (Fang et al. (0200. XGBoost (Chen & Guestrin (016). and others. FPT refers to
Table 2:
line-tuming the layer norms of RoBERIa la or Swin. For list of hand-designed models, see Appendix A.2
CIFAR- 100
Spherical
Dary Flow
PSICOV MAEs
Cosmic 1-AUROC
NinaPro
FSDS0K
0-1 error (%)
0-1 emror (%)
relative f2
0-1 error (%)
1- mAP
Hand designed
19.39=0.20
67.41:10.76
8E-31IE-3
3.35::0.14
0.24::00155
8.73:10.9
0.62:10.00
NAS- MAS-Bench-360
23.39-0.01
48.234+2.87
2.6B-241
2,94=0.13
0.22400035
7.34=0.76
0.60++0.00 0.1
DASH
24.37=0881
75.44:2.33
7.9E-3+2E-3
3.30=0.16
0,25=0002
6.60=0.33
0.60=20.008
Perceiver IO
70,04-0,44
82,57:-0.19
2.4E-2+1E-2
8.06=0.06
0,48+0001
22.22++:1.88 80
0,72++0.002
FPT-Swin
10.11=1.18
76,38:44.89
2.1E-2411.3E-
4.66:10.054
0.3240.0021
15.69:2..3
0.67=000068
ORCA-SWIN
6.53:10.079
29.85+40.72
7.38-3-6.88-5
1.91+0.038
0.21+-0.000
7.54=0.39
0.56=00013
ECG
Satellite
DeepSEA
JSB Chorales
ListOps
Homology
score
0-1 enor (%)
AUROC
NLL
0-1 enor (%)
0-1 error (%)
Hand-designed
0.28:10.00
19.8:0.00
0.30:0.024
8.43=0.00
62.73:0.00
88:10.00
NAS Rench-360
0.33++0.02
+-.21
0.32++0.01
DASH
0.32:40.007
12.28:0..5
0.28:10.013
6.13100006
57.33:10.14
89.71:10.54
Perceiver 10
0.66-+0.01
15.93-+0.0088
0.38++0.004
FPT-RoBERTa
0,50+00.0098
20.83++0.24
0.37=0.0002
2.72:40.026
64.35:00.79
91.48:+1114
ORCA-ROBERTA 0.29:-0.0052
11.59
0,29+40.006
2.44+-0.056
51,90++2.18
87.21 +0.50
methods. We also beat all AutoML baselines on all tasks except DeepSEA and NinaPro, where ORCA is second and third, respectively. The improvements from ORCA come at small computa- tional overhead associated with pretraining the embedder to match the source and target modalities. Table in the Appendix shows the time needed for embedder learning with OTDD, which is a small portion (10.2% on average) of the fine-tuning time. ORCA's 's efficiency and its state-of-the-an artt results on 10 tasks make practical tool for model development in diverse areas.
Our experiments further validate the findings in Lu et al. (202T) that pretrained transformers can learn knowledge transferable seemingly unrelated tasks. In the following, we delve into the mech- anism of ORCA to provide intuition for necessary components of successful cross-modal learning.
4.2 KEY FACTORS FOR SUCCESSFUL CROSS-MODAL TRANSFER
Here, we dissect the success of cross-modal transfer with ORCA through a series of ablation studies. As preview, we identify three aspects to be key to its success: data alignment through embedding learning, full fine-tuning of all model weights, and suitable pretrained model selection.
4.2.1
MATCHING FEATURE DISTRIBUTIONS HELPS ADAPTATION
Table 2 shows that ORCA leads to effective transfer using the proposed three-stage workflow. How- ever, as learning the embedder via OTDD is an instantiation of the general first-align-ther teen---the--in-tuuee paradigm, we ask the question: does cross-modal transfer work because of the specific application of OTDD or the core approuch of data alignment across modalities?
To answer this question, we perform an ablation study on the embedding learning metrics and compare their per- formance to fine-tuning without embedder pretraining. We experiment with OTDD, maximum mean discrepancy (MMD) (Gretton et al.] 2012), the mutual-informattee information-
Fraction Sefr--Subottiie Tasks
1.0-
0.8
0.6
OTDD
MMD
based RransRate (Huang et al. 2022). and Euclidean dis-
0.4-
TransRate
tance. The performance profile is in shown Figure 3 and the detailed results are shown in Appendix A.4 We
0.2
Eudidean
w/o Emb. Learning
0.0
1.0
1.2
1.4
1.6
1.8
2.0
highlight the following observations. First, bringing the target modality closer to the pretraining modality gener- ally aids cross-modal transfer, regardless of which metric we minimize. This is evident from the fact that pretrain- ing the embedder with any of the metrics can outperform vanilla fine-tuning without embedder learning on many tasks. Second, among the evaluated metrics, OTDD leads to the best overall performance. This is why we use it in workflow. The middle rows of Table 3 demonstrate that ORCA with OTDD consistently outperforms naive fine-tuning. This supports our argument that closing the gap between modality and the pretraining modality can facilitate model's adaptation new task.
Figure 3: Performance profiles for us- ing different embedder learning metries on NAS-Bench-360. Data alignment generally improves fine-tuning performance.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Performance of ORCA, vanilla fine-tuning, and training RoBERTa/Swin from scratch. We consider line-tuning all parameters (full setting) vs. only the layer norms (FPT setting). ORCA is better in both settings.
CIFAR-10 100
Spherical
Darcy Flow
PSICOV
Cosmic
NinaPro
FSDSOK
Train-from-seratch
50.87+0032
76.6740021
8.0E-2+1
5.09=00014
0.50=00.0
9,96+1.677
0.75=00.17
Fine- tuning
7.67+00.5
55,26++1.63
2.ME3+1111.4
1.92+0.039
0.24-000088
8.35++0.75
0.63+00.014
ORCA
6.53:00079
29.85-10.72
7.88-311..8--
1.91
0.21:000050
0.34
0.56.10.013
Fine-tuning (layemorm)
10.11+1.18
76.38444.89
2.1E-241.3..
4.66+00.04
0.32+000021
15.69-2.33
0.67+0.0068
ORCA (layernorm)
7,99=0.098
42.45+0.211
2.1E-2+7.4-4
4.97=0.14
0.25=00.020
15.99=1.92
0.64=0.0093
ECG
Satellite
DeepSEA
JSB Chorales
ListOps
Homology
Train-from from scratch
0.42-0.011
12.38440.1
0,39+0.01
2.71+0.35
63.50-+0.55
00.39-0.11
Fine tuning
0.44:10.0056
13.86.11.47
0.5100.0001
2.59000.98
1,979
89.31 0.60
ORCA
0,29::0,0052
11.59:10.18
0.29:10.006
2.44:0.056
51.90=2.18
87.21=0.50
Finc-tuning (layemorm)
0,50=0.0098
20.83:+0.24
0.37=0.0002
2.72=0.026
64.35=20.79
91.48=1.14
ORCA (layernorm)
0.47=00007
20.54+00.9
0.36=0.0070
2.68+0.133
63.62-00755
90. 20.65-1.66 1.66
Embadding OTDD (1o2)
0,160
NinaPro
82.5
Sm
0.20
Homornn
13 0.50
DaipSFA
Aocursay
92.0 1.
9.6
0.10-
10 0.40
0.146-
0
0.05
0.55
Embedder Learning Epochs
Figure 4: Final model accuracy and embedding OTDD vs. number of embedding training epochs when applying ORCA to four tasks. As the embedder lcarns to map the target data to the source modality better (smaller final OTDD), we generally obtain models with better downstream performance.
To further isolate the impact of data alignment, compare ORCA with line, which trains RoBERTa and Swin using only the target data for the same number of epochs. Table[]shows that train- from-scratch is better than fine-tuning but worse than ORCA on many tasks like Satellite and DeepSEA, indicating that when the target modality differs significantly from the pretraining modality, naive fine-tuning may harm transfer, but aligning the feature distribution ing ORCA can resolve this issue and benefit transfer. Indeed, recent work has shown that optimizing directly for the task loss may distort the pretrained weights and lead to suboptimal solutions (Kumar et al.] [022 Lee et al. (022). By manipulating the target data distribution to look like the source dis- tribution, we can lower the risk of catastrophic forgetting, which may explain the success of ORCA.
seiin--fom--creee base-
Lastly, we perform experiments to quantify the effect of data alignment from a task-wise perspective. We train the embedder for different number of epochs before fine-tuning to see how optimizing OTDD to various levels of convergence affects downstream performance. Figure 4 plots the fine- tuning accuracy along with the final OTDD objective for different levels of embedder pretraining- Evidently, the dataset distance decreases, the final fine-tuning accuracy increases. This correlation supports the effectiveness of embedder learning for cross-modal transfer. In addition, we observe that learning the embedder prior to fine-tuning can stabilize training, the performance variance of ORCA is consistently lower than that of standard mee-tuing.
4.2.2 FINE-TUNING ALL MODEL PARAMETERS FOR CROSS- CROSS-MODAL TASKS
As discussed in Section? 2 Frozen Pretrained Transformers (FPT) (u et al. 2022b) is a related work that showed pretrained language models contain knowledge relevant for oft-of-modality tasks. While FPT presented a general pipeline that transfers GPT-2 to tasks like CIFAR-10, 10, Homology, and ListOps, the resulting models were not as good as those directly trained on the target data. FPT differs from ORCA in that (1) it does not pretrain an embedder for task-specific adaptation and (2) it only fine-tunes the layer norms. We have already verified the importance of (1). Now, to isolate the impact of (2). we evaluate ORCA with fine-tuning the layer norms vs. FPT on our task set.
The bottom rows of Table 3 show ORCA with fine-tuning just the layer norms outperforms FPT, indicating pretraining the embedding layers boosts the modal performance of FPT. However, this performance gain is smaller than that seen in the full fine-tuning setting, which implies that full fine-tuning can take better advantage of the learned embeddings. Also, partial fine-tuning is less effective than full fine-tuning on all tasks except for DeepSEA. This exception might be due to the fact that full fine-tuning without learned embeddings is more prone to overfitting. In terms of run- time, FPT only results in less than 2x speedups compared with full fine-tuning (see Appendix A.5)) despite the fact that are updating significantly fewer parameters. This is unsurprising since gra- dients are still back- propagaae through the entire network. Therefore, when computation allows, recommend using ORCA with full fine-tuning for better downstream performance.
8

---

Under review as a conference paper at ICLR 2023
4.2.3 PRETRAINING MODALITY CAN AFFECT TRANSFER PERFORMANCE
Finally, we study how the pretraining modal- ity affects fine-tuning performance. For ex- periments in Table 2 we chose pretrained models for each task based on the input di- mension, i.e., we use RoBERTa for all 1D tasks and Swin for all 2D tasks. Now, can switch the model bodies and apply ORCA. This is easy to implement because ORCA is model-agnostic and the embedder architec- ture handles all necessary input transformation to obtain sequence features. As shown in Table fine-tuned RoBERTa outperforms fine-tuned Swin on the ID task, and the final OTDD objective for RoBERTa is also smaller than that of Swin. We hypothesize that this because the considered DeepSEA data (genomics sequences) are structured more like language than images with discrete units of information and general grammatical rules. The FPT paper observes a similar trend for Homology. As for the 2D tasks, we again notice that models with better fine-tuning accuracy have smaller OTDDs. This suggests a way of selecting pretrained models from a predefined model hub for each task, e.g., by comparing the optimized OTDDs and picking the one with the smallest value.
Table 4: Test errors of ORCA on DeepSEA and Spherical with language- and mmage-pretrainee model bodies. Num- bers the parenthesis represent the OTDD after embed- ding learning. Smaller OTDD leads to better performance.
Errrr (OTDD)
DeepSEA (ID)
Spherical (2D)
RoBERTa (ID)
0.2951100006 (37.40) 68.281000017 (19.54)
Swin (2D)
0.361+0.001 (64.83) 29.85=0.072(11.78)
Case Study: Low-Data Regime. Now that we have better understanding of ORCA, recall that one of our motivations for transferring pretrained models to various modalities is to help task-solving in data-limited regimes, where training models from scratch can be challenging. To this end, we investigate whether ORCA can facilitate fine-tuning large-scale models on small target datasets.
Indeed, for vanilla fine-tuning, a small amount of data may not give enough signal to update the pretrained weights. However, it is possible to obtain good feature embedder with the same amount of data using ORCA, which can then make fine-tuning easier. In Figure we vary the amount of target data and plot the performance of ORCA and vanilla fine-tuning. The performance gain of ORCA increases as the amount of data used decreases. This shows that fine-tuning does suffer from limited data, but ORCA can considerably alleviate the problem and improve downstream performance. Moreover, ORCA al- lows us to use third of the data to match the perfor- mance of standard fine-tuning. Thus, it can benefit model development in domains where data collection is costly.
Sateliile
80
Accuracy
60
40
20
ORCA
Fine-tuning
3.0
3.5
4.0
4.5
5.0
Num Target Data (log10)
Figure 5: Prediction accuracy of ORCA and the fine-tuning baseline trained with different amounts of target data. ORCA has a larger performance gain in low-data regime.
Discussion and Future Work. We identify several future directions based on our experiment re- sults. First, it is worth studying the effect of pretraining modality further and develop systematic way of selecting pretrained models. Then, we can incorporate model selection into ORCA for more automated transfer pipeline. Second, while ORCA leverages the simplest fine-tuning paradigm, we believe it is possible to combine it with more sophisticated transfer techniques such as adapters (He) al. 2022). We briefly study how prompting (Bahng al. 2022 Jia al. 2022) can be applied to diverse tasks in Appendix [A.6] and find that it is in general less effective for out-of -fo-of-modality prob- lems, so we can possibly boost its performance using ORCA. Lastly, we currently evaluate ORCA on diverse 1D/2D tasks and in in-modality vision tasks (Appendix A.7]. It is also important to validate it on more settings, such as high-dimensional problems and reinforcement learning (Reid et al. 2022).
5 CONCLUSION
In this paper, we argue that an important step towards developing more general ML. methods is to study how we can reuse existing models effectively for new and less-explored tasks. To this end, we propose novel framework that allows transferring pretrained transformers to distinct downstream modalities. Our method, ORCA, can map target data from an arbitrary end task's modality to a model's 's pretraining modality to improve fine-tuning performance. We believe that this work not only signals the potential of large-scale pretraining for diverse tasks but also lays out a path for a largely uncharted data-centric paradigm in machine learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Badri Adhikari. DEEPCON: protein contact prediction using dilated convolutional neural networks with dropout. Bioinformatics, 36(2):470-477 07 2019.
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and S. Gupta. Better fine-tuning by reducing representational collapse. ArXiv, 0bss/2008.03156, 2021.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho. Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Lee. Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kan- ishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as say: Grounding language in robotic affordances. ArXiv, abs/2204.01691, 2022.
Jean-Bac Sann-Baptstte Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, 14199, 2022.
David Alvarez-Melis and Nicoló Fusi. Geometric dataset distances via optimal transport. ArXiv, 0oss/2002.02923, 2020.
Hyojin Bahng, Ali Jahanian, Swami Sankaramarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. 2022.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. ArXiv, abs/1803.01271, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ArXiv, abs/2005. 12872, 2020.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 2022.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural net- works. ArXiv, hbbs/160.01100 2017.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical cnns. In International Conference on Machine Learning, 2018
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamoniy. Optimal transport for do- main adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39: 39:1853- 1865, 2017.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, 2013.
Angus Dempster, Franccois Petitjean, and Geoffrey I. Webb. Rocket: exceptionally fast and accu- rate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34:1454-1499 1495, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04405, 2019.
10

---

Under review as a conference paper at ICLR 2023
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Shashank Rajput, Michael Gira, Jy yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaeed interfacced fine-tuning for non- language machine learning tasks. ArXiv, 00552206.006655, 2022.
Elizabeth D. Dolan and Jorge J Moré. Benchmarking optimization software with performance pro- files. Mathematical Programming, 91:201-213, 2002.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. ArXiv, abs/2010. 11929, 2021.
Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Densely con- nected search space for more flexible neural architecture search. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10625-10634, 2020.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. ArXiv, abs/2010.00475, 2021.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alex Smola. A kernel two-sample test. JMLR, 13:723-773, 2012.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Korg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. ArXiv, 0oos2210.043366 2022.
Shenda Hong, Yanbo Xu, Alind Khare, Satria Priambada, Kevin O. Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov. Holmes: Health online model ensemble serving for deep learning models in intensive care units. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified trans- former. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1419-1429, 2021.
Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2261-2269, 2017.
Long-Kai Huang, Ying Wei, Yu Rong, Qiang Yang, and Junzhou Huang. Frustratingly easy trans- ferability estimation. In ICML, 2022.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding. Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier J. H' H'enaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver io: A general ar- chitecture for structured inputs & outputs. ArXiv, abs/2107.14795. 2022a.
Andrew Jaegle, Sebastian Borgeaud, Jean- -aptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general archi- tecture for structured inputs & outputs. International Conference on Learning Representations, 2022b. htth:tt::://
net /ttpttss///peeeeeeee wpw.revieeeeeeeeeeeeeeeeee htt/://ruuumidiiiee Lttp:///www...ettooooooooooooa1. 17pp 17mppl---iipii-i-----
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser Nam Lim. Visual prompt tuning. In ECCV, 2022.
Dongwei Jiang, Wubo Li, Ruixiong Zhang, Miao Cao, Ne Luo, Yang Han, Wei Zou, Kun Han, and Xiangang Li. A further study of unsupervised pretraining for transformer based speech recog- nition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6538 66522 IEEE, 2021.

---

Under review as a conference paper at ICLR 2023
Haoming Jiang, Pengcheng He. Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao, Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. ArXiv, abs/1911.03437, 2020.
David Josephs, Carson Drake, Andrew M Heroy, and John Santerre. semg gesture recognition with simple model of attention. Machine Learning for Health, pp. 126-138 2020.
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- tion with alphafold. Nature, 596:583 589, 2021.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. ArXiv, abs//9099 0909000 2019.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: lain--and-language transformer without convo- lution or region supervision. ICML, 2021.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ArXiv, 155/2001.04451, 2020.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. ArXiv, fbs/2202..10044, 2022.
Yoonho Lee, Annic S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Perey Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. ArXiv, abs/2210.1 1466, 2022.
Nicolas Boulanger Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. arXiv: Learning, 2012.
Feng Li, Hao Zhang. Hu-Sheng Xu, Siyi Liu, Lei Zhang, Lionel M. Ni, and Heung yeung Shum. Mask dino: Towards unified transformer-baas baseed framework for object detection and segmenta- tion. ArXiv, abs/2206.002777, 2022.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben- Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230-246, 2020a.
Shuang Li, Binhui Xie, Jiashu Wu, Ying Zhao, Chi Harold Liu, and Zhengming Ding. Simultaneous semantic alignment network for heterogeneous domain adaptation. In Proceedings of the 28th ACM international conference on multimedia, pp. 3866-3874, 2020b.
Yaoyiran Li, E. Ponti, Ivan Vulic, and Anna Korhonen. Emergent communication pretraining for few-shot machine translation. In COLING, 2020c.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Proceedings of the 9th International Conference Learning Represen- tations, 2021a
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference Learning Representations, 2021b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference Learning Representations, 2019a.
12

---

Under review as a conference paper at ICLR 2023
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586 2021a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys (CSUR), 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692. 2019b.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. ArXiv, abs/2111.09883, 2021b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF In- lernational Conference on Computer Vision (ICCV), pp. 9992-10002, 2021c.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In NeurIPS, 2019.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. ArXiv, abs/2206.08916, 2022a.
Kevin Lu, Aditya Grover, P. Abbeel, and Igor Mordatch. Pretrained transformers as universal com- putation engines. ArXiv,
2021.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference Artificial Intelligence, 36 (7):7628-7636, Jun. 2022b.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22(10)):345-133 2009.
Ofir Pele and Michael Werman. Fast and robust earth mover's distances. 2009 IEEE 12th Interna- tional Conference on Computer Vision, pp. 460-467, 2009.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1406-1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML., 2021.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John F. Canny, P. Abbeel, and Yun S. Song. Evaluating protein transfer learning with tape. bioRxiv, 2019.
Esteban Real, Chen Liang, David R. So, and Quoe Le. Automl-zero: Evolving machine learning algorithms from scratch. In ICML, 2020.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv, Abs/2205.06172. 2022.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? ArXiv, abs/2201.12123 2022.
13

---

Under review as a conference paper at ICLR 2023
Nicholas Carl Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Re, and Ameet Talwalkar. Rethinking neural operations for diverse tasks. In A. Beygelzimer, Y. Dauphin, P. Liang, and in Neural Information Processing Systems, 2021.
J. Wortman Vaughan
Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang- Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1874-1883, 2016.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey. In ECCV Workshops, 2020.
Yi Tay, Mostafa Dehghani. Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. ArXiv, abs/201 1. .04000,, 2021.
Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-bench- 360: Benchmarking neural architecture search diverse tasks. In Advances in Neu- ral Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312: 135-153, 2018
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. ArXiv, abs/2205. 14141, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: State-of-the-arr natural language processing. ArXiv, abs/1910.03771, 2019.
Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. Semi-supervised optimal transport for heterogeneous domain adaptation. In IJCAI, 2018
Yuan Yao, Yu Zhang, Xutao Li, and Yunming Ye. Heterogeneous domain adaptation via soft transfer network. In Proceedings of the 27th ACM international conference on multimedia, pp. 1578- 1586, 2019.
Keming Zhang and Joshua S. Bloom. deeper: Cosmic ray rejection with deep learning. The Astro- physical Journal, 889(1):24, 2020.
Jian Zhou and Olga G. Troyanskaya. Predicting effects of noncoding variants with deep learn- ing- insebased sequence model. Nature Methods, 12:931-934, 2015.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 EMBEDDING LEARNING WITH OPTIMAL TRANSPORT DATASET DISTANCE
A.1.1 LITERATURE REVIEW
Due to the limited space, we do not give full review of the optimal transport dataset distance (OTDD) (Alvarez-Mehi Melis & Fusi 2020) in the main text. Here, we briefly recall the optimal transport (OT) distance and explain OTDD in detail.
Consider complete and separable metric space X' and let P(t) be the set of probability measures on X. For a 3 E P)))), let [[ 3) be the set of joint probability distributions on X' x X' with marginals and 3 in the first and second dimensions respectively. Then given cost function d(.-) X x X' - R+, the classic OT distance with cost c is defined by:
(T, min Sx c(z, (ddT)d v).
xEll(a, xxx
When X' is equipped with metric dx, we can use c(x,)) = dx (z, y) for some p 2 and obtain the p- Wasserstein distance, Wp(a, 3) = (OTay, ())
Now consider the case of finite datasets with features in X' and labels in a finite set y Each dataset can be considered a discrete distribution in P(X x J)). To define distance between datasets, natural approach is to define an appropriate cost function on Z == X' x y and consider the optimal transport distance. Indeed, for any metric dy on y and any p 1, 2 can be made complete and separable metric space with metric
d2 (1,, v), (z', ,')) (2x + (..."'')'
It is usually not clear how to define a natural distance metric in y so instead we proceed by repre- senting each class y E y by P(X = y), the conditional distribution of features A' given ) y- More specifically, for a dataset D E (P('' x Y)), denote this map from classes to conditional dis- tributions by F(D,-) -) y + P(x)). Then we can transform any dataset over x y into one over X' x P(X) via G(D) (projx, F(D, pr.
As discussed above, W, a natural notion of distance in P(X)), so by substituting y + P(X) and dy + W, in Equation we can define the (p-)optimal transport dataset distance between datasets DA and DB by
OTDD(DA, DB) OT (dfx ( (((D))
A.1.2 COMPUTATIONAL. CONSIDERATIONS
As we aim for a practical fine-tuning workflow, computational cost is crucial concern. While Alvarez-Melis & Fusi 2020) proposed two variants of OTDD-the exact one and Gaussian ap- proximation, we observe from our experiments that optimizing the exact OTDD leads to better per- formance. In the following, we will focus on analyzing the computational cost of the exact OTDD.
Given datasets with D-dimensional feature vectors, estimating vanilla OT distances can be com- putationally expensive and has worst-case complexity of O(D* log D) (Pele & Werman 2009). However, adding an entropy regularization term cH (TT a @ B) to Equation where H is the relative entropy and controls the time-accuracy trade-off, can be solved efficiently with the Sinkhorn al- gorithm (Cuturi 2013). This reduces OT's empirical complexity to ( and makes the time cost for computing OTDD manageable for ORCA's workflow.
During implementation of ORCA, we also observed memory issues for computing OTDD using the entire target and source datasets on GPUs To alleviate this, we propose a class-wise subsampling strategy for approximating OTDD on GPUs (Algorithm []. In short, we split the K-class target dataset into datasets based on the labels and compute the class-wiss OTDD between each single- class target dataset and the entire source dataset. Each class wise OTDD can be approximated with the average of batch samples similar to how stochastic gradient descent approximates gradient descent. After that, approximate the OTDD between the target and source datasets using the
15

---

Under review as a conference paper at ICLR 2023
Algorithm Efficient approximation of OTDD using class-wise subsampling.
Input:
target dataset {x²; y²), number of target classes K', source dataset S f:r ,0", sub-
sample size b, subsample round R
for each class i [K*] in the target dataset do
Compute class weight wi
number target in
total number of target data
Generate data loader D; consisting of data in class end for for i E [K*) do for E [R]] do Subsample b target data points D.. uniformly at random from D; Compute class-wise distance dir OTD
end for
Approximate class-wiss OTDD by d; R Σi= dr
end for Approximate OTDD by d = Kt
Wi di
0.58
61
+206
40130
10:00 120000 19A000
Dester
Figure 6: Screenshot of OTDD curves during embedding learning for the task ListOps. x-axis the number of optimization steps, y-axis represents OTDD (1F-2). We use Algorithm To approximate the exact OTDD the loss function for optimization GPU (purple curve). We also track the actual OTDD CPU (blue curve). can see that the proposed algorithm works well, which allows us to perform emhedding learning efficiently.
weighted sum of the K class-wise OTDDs. track the approximated OTDD (computed on GPUs) and the actual OTDD (computed on CPUs) and visualize the loss curves during ORCA's embedder learning process (Figure 6 We see that the estimated value adheres to the actual value.
To verifyy that the approximation works empirically,
Leveraging both the Sinkhorn algorithm and class-wise approximation, the embedder learning pro- cess only takes up small fraction of the total fine-tuning time practice, as shown Table 5 Hence, invest reasonable time budget but achieve significantly improved cross-domain transfer performance using ORCA.
Table 5: We record the runtime (in hours) of ORCA's 's embedding lcarning stage and the fine-tuning stage for each task. Then, compute the ratio between the two. Averaged across tasks, embedding learning with OTDD only takes about 10% of the time needed for fine-tuning. All experiments performed NVIDIA V100 GPUs.
100
Spherical
Darcy Flow
PSICOV
Cosmic
NinaPro
FSD50K
Embedding
1.6
1.8
0.18
0.35
0.2
0.3
0.21
Finc-tuning
9,2
9.3
0.86
1.1
3.5
1.1
12.5
Embeediig//line tuning
17%
19%
20%
31%
5%
27%
2%
ECG
Satellite
DecpSEA
JSB Chorales
ListOps
Homology
Libeedding
0.6
0.26
0.14
0.1
0.2
0.16
Fine-tuning
23.1
37.5
14.8
0.13
17.8
6.0
Fmbbedding/Fin--uninng
1%
1%
76%
1%
2%
16

---

Under review as a conference paper at ICLR 2023
A.2 INFORMATION ABOUT EVALUATION TASKS
Table 6: Summary about each evaluation task and the hand- denddddd expert models used. The top 10 rows are for the 10 datasets in
(Tu al. 2022).
Task name
Dosa
Duta dim.
Type
License
Learning obyective
Expert arch.
CIBAR-100
60K
21
Point
CC BY 4.0
Classity nntaral images into 100 classes
DemseNe: BC
Huang et al. 2013
Spherical
60K
2D
Point
CC
Clussfy spherically projected images 100 classe
S2CN
Cchen al. 2018
NinaPro
2956
Point
BY-ND
Classit sEMG into 18 classes concsponding hand gestares
Auention Model Josqph
FSD50X
Print
Clussity sound lop-ttel spectrograms
Ponseca
tinal
Duncy
1100
inttall couditions
Li
PSICOV
3506
Dense
GPL
duals froes 2D prosein sequence features
Predict pairwise discances between res-
DEGICON
Addlkiaii 2019
Coomii
$2.00
21>
Dene
Open Ticess
mic rays telessopee images
Predict propubiittc maps to slentity cos-
Zhing & Bloom 2000
DOG
330K
1D
Point
ODC BY 1.0
Desect utrial curdiac disease from
BCG recording 14 classes)
ResNet-ID
Hang al. 2020
Sateltiee
IM
ID
Pvint
GPI 3.0
Classity sanetliee image picets' time
series into 24 lund cover types
ROCKET
Dempster ml 2020
DeepSEA
250K
Point
CC BY
Predict chromsttin siates and binding
DeepSGA
Zhow Troyanskays 2015
JSR Chanles
BY-SA
Prodict
sheet music
Dilated TCN
Rai 2018
LisoOps
Model hienrchically structured lungonnees
Homology
fold for a prosein
A.3 EXPERIMENT DETAILS
Below, we summarize details for implementing ORCA and evaluating it on the selected 13 tasks. The code and configuration file for reproducing each experiment can be found the supplementary material. We will also release ORCA's 's best checkpoint for each task later.
A.3.1 PRETRAINED MODELS
We evaluated ORCA with two pretrained models in our experiments. In Table 2 for all 2D tasks including CIFAR-100 100, Spherical, Darcy Flow, PSICOV, Cosmic, NinaPro, and FSD50K, we use the following model. As Swin has a pretrained resolution, we reshape the inputs for our tasks to the resolution before feeding them into the model.
Name
Pretrain
Resolution
Num Params
FLOPS
FPS
Swin-base Liu et al. 2021c
ImageNet-22K
224x224
88M
15.4G
278
For all ID tasks including ECG, Satellite, DeepSEA, JSB Chorales, ListOps,and Homology, we use
the following model:
Name
Pretrain
Num Params
FLOPS
RoBERTa-base (Liu et al. 2019b) Five English-language corpora
125M
1.64E20
We use the Hugging Face transformers library Wolf 2019 to implement the pretrained models.
17

---

Under review as a conference paper at ICLR 2023
A.3.2 TASK DATA PREPARATION
For all the NAS-Bench-360 tasks, each dataset is preprocessed and split using the script avail- able on /ttps:://qtta: www./rrul1//MA--Beecc----------.
with the training set being used for hyperparameter tuning, embedding learning, and fine-tuning. We obtain the data processing script for JSB data from https:://tttthtie
com/
wwtp:/ccc.o/ooo/oooooccaa.aac for ListOps and for Homology from
from http:////....b...ooooooooo.. com/ www.ssooglllbboooo aab-galllaaalaaa tape
https://g
A.3.3 HYPERPARAMETER TUNING
As ORCA is both task-agnostic and model-agnostic, it can be applied to fine-tuning variety of pretrained transformers on drastically different end tasks with distinct datasets. Hence, is hard to define set of fine-tuning hyperparameters for all (model, task) pairs. At the same time, opti- mizing large-scale pretrained transformers can be challenging due to their large model sizes, as the downstream performance depends largely the hyperparameters used. For instance, using large learning rate can distort pretrained weights and lead to catastrophic forgetting. Therefore, in our ex- periments, given (model, task) pair, we first apply hyperparameter tuning using the Asynchronous Successive Halving Algorithm (ASHA) Li et al. (020a) to the standard fine-tuning setting (i.e., after initializing the embedder and predictor architectures, directly updating all model weights to minimize the task loss) to identify a proper training configuration. Then, we use the same set of hy- perparameters found for all our experiments for the particular (model, task) combination. Note that even though we did not explicitly state this in the main text, the hyperparameter tuning stage can be directly integrated into the ORCA workflow between stage and stage 2. this sense, ORCA still an automated cross- modal transfer workflow that works for diverse tasks and different pretrained models.
The configuration space for ASHA is follows:
Batch size: 32, 128, 512, 1024 for Swin; 16, 56, 256, 512 for RoBERTa
Optimizer: SGD, Adam, AdamW
Learning rate: IE-2, IE-3, IE-4, IE-5, 1E-6
Weight decay: 1E-2, 1E-3, 1E4, 1E-5, 1E-6
Note that to fit each experiment on a single GPU, we set a fixed batch size (32 for Swin and 16 for Roberta) and vary the gradient accumulation step instead of actually varying the batch size, but the effect the same.
A.3.4 ORCA ONLY: EMBEDDING LEARNING WITH OTDD
After initializing the embedder architecture for each task, we train it to minimize the OTDD between the embedded target features and embedded source features.
For source datasets, we use CIFAR-10 for Swin and CONLL-2003 for RoBERTa. We sample 5000 data points compute OTDD. In practice, embedder once and save all the embedded features, so we don't have to pay the cost of obtaining the source features each time we fine-tune new model.
pass the source data through the pretrained
For classification tasks, we directly use the labels provided by the end task to compute OTDD, For dense tasks, we perform K-Means clustering on the target data obtain pseudolabels for OTDD computation. The number of clusters is set to the number of classes of the source dataset, e.g., 10 for 2D tasks that use CIFAR-J0 the source dataset.
To compute the embedding learning objective, we use the OTDD implementation of the original paper provided here: https: /tll:///qitanne qithub. com mi http///ccooooffffftoooo As for the hyperparameters, we use the batch size, learning rate, optimizer, and weight decay obtained from A.3.3 The others are fixed across different tasks:
Embedding learning epochs: 60
Learning rate scheduler: decay by 0.2 every 20 epochs
18

---

Under review as a conference paper at ICLR 2023
A.3.5 FINE-TUNING
Besides the searched hyperparameters, we also fix the following hyperparameters for fine-tuning.
Fine-tuning epochs: 100 for Swin tasks, 60 for RoBERTa tasks
Learning rate scheduler: we use the linear decay with min Ir = 0 and 5 warmup epochs
A.3.6 TRAIN-FROM-SCRATCHH
This baseline is trained using the same hyperparameter configuration (number of epochs, batch size, learning rate, etc) as the fine-tuning baseline.
A.3.7 EVALUATION
When training/fin-ttningg is finished, evaluate the performance of all models following the NAS- Bench-360 protocol. We first report results of the target metric for each task by running the model of the last epoch the test data. Then, report aggregate results via performance profiles (Dolan & Moré 2002). a technique that considers both outliers and small performance differences to compare methods across multiple tasks robustly. In such plots, each curve represents one method. The on the x-axis denotes the fraction of tasks on which a method is no worse than a r-factor from the best. The performance profile for our experiments is shown in Figure 2
A.4 ABLATION STUDY ON EMBEDDING LEARNING METRICS
As motivated in Section 4.2.1 we present here ablation study the embedding learning metrics that we have considered for minimizing distribution dissimilarity. The results show that (1) per- forming feature aligmment generally helps downstream adaptation, regardless of which metric minimize; (2) OTDD leads to the best overall performance, so we chose it for our workflow. Our findings confirm that it is the general idea of data alignment, rather than specific metric, that makes cross-modal transfer work.
Specifically, experiment with OTDD, maximum mean discrepancy (MMD) Gretton al. 2012) the nutual-informationn based TransRate (Huang et al. 2022). and (pairwise) Euclidean distance. We learn the embedders minimize these metrics and then fine-tune the pretrained models. The test eтrors are as follows.
Table 7:
Prediction errors of different distance metries. OTDD achieves the best overall performance. "w/o Emb. Learning" represents fine-tuning without embedder learning.
CIFAR-100
Spherical
Darcy Flow
PSICOV
Cosmic
NinaPro
FSD50K
OTDD
6.53:10.079
29.85:0.72
7.3E-31..8E-5
1.911000338
0.2110.0050
7.54=0.39
0.56:10.013
MMD
6.62:1:0.022
33.64:2.557
7:443343.4444
1.9:100016
0.32100054
7.48:10.23
0.58=00.004
TransRate
6.66-+00.08
30.32+0.53
8.05-3+6.5E-5
1.92++0.03
0.31+0.0037
8.64=0.26
0.57++0.01
Euclidean
7.09=1048
32.3342.03
7.39-311.9E-
1.1110.019
0.27100021
7.51=0.11
0.5910.02
w/o Emb. Learinng
7.67=0.55
1..2
7.3E-3:L.EE-
1.92:000399
0.2420.0080
8.35=0.75
0.63#0.014
ECG
Satellite
DeepSEA
JSB Chorales
ListOps
Homology
OTDD
0.29 0.0052
11.59+0.18 18
0.29+
0.441
51.90+2.118
87.21+0.500
0.40:40.018
11.29-10.887
0.38-10.077
2.57:10.045
63.9441.3
89.3=0.57
TransRate
0.44:10.021
11.35:10.2
0.38:10077
2.56:00044
63.0243.37
97.21+0000
Euclideun
0.41:10.009
1144=0.77
0.34:10.02
2.56:10.043
60.2=0.41
89.97=.1.18
w/o Emb. Learning
0.44:
13.8
0.0000
2.59:10.098
72.79::
89.31-10.60
19

---

Under review as a conference paper at ICLR 2023
A.5 RUNTIME OF ORCA VS. FPT
In 3 we compare with the FPT setting, which only fine-tunes the layer norms of the pretrained transformer models. As have shown already, the downstream performance of fine-tuning only a subset of the parameters is less competitive than fine-tuning all parameters. Below, we show that the time saved for updating only layer norms is also not that significant. Therefore, we suggest performing full fine-tuning when time and computational resources allow.
Table 8: We record the total runtime (in hours) for four settings: ORCA with full fine-tuning, ORCA with tuning layer norms, full fine-tume ane-ttning (without embedding learning), and fine-tuning layer norms (FPT). We can see that tuning the layer norms does not bring significant benefit in terms of reducing model development time, but sacrifices the downstream performance the resulting models.
CIFAR-100
Spherical
Darcy Flow
PSICOV
Cosmic
NinaPro
FSD50K
ORCA
10.8
11.1
1.04
1.45
3.7
1.4
12.71
ORCA (layernorm)
8.7
8.9
0.76
1.15
3,5
1.0
8.96
Fine-tuning
9.2
9.3
0.86
1.1
3.2
1.1
12.5
Fine-tuning (layernorm)
7.1
7.1
0.58
0.8
3,0
0.7
8.75
ECG
Satellite
DeepSEA
Chorales
ListOps
Homology
ORCA
23.7
37.76
14.94
0.23
18.0
6.16
ORCA (layemorm)
18.0
25.56
11.24
0.2
13.2
Fine-tuning
23.1
37.5
14.8
0.13
17.8
6.0
Fine-tuning (layernorm)
25.3
11.1
0.1
13.0
4.4
A.6 PROMPTING
Apart from fine-tuning, new paradigm of working with large-scale pretrained models is prompting, ie., we do not update the pretrained weights but only modify the input and query the model for the desired output. Existing language prompting methods (e.g., Liu et al.) 2022) are generally not suitable for cross-modal learning due to the difficulty of designing natural prompts for diverse data types. For the ID tasks we study, there is even notion of "discrete tokens." Another line of work studies visual prompting by modifying 2D inputs for querying vision transformers. We test two such algorithms, VP (Bahng et al. 2022 and VPT (Jia al. 2022), on three classification tasks in our task suite. They are not applicable to the remaining tasks because either the inputs cannot be reshaped to look like images the outputs
classification logits.
We test VPT with the pretrained Twin-Bass Transformer (the same model we used for ORCA) and VP with the pretrained ResNet-50 (as the official implementation does not support vision transform- ers). The results are shown in Table 9 In general, prompt tuning is less effective than fine-tuning, and the two baselines perform significantly worse than ORCA. This is not surprising given that prompting methods are more intuitively suited to in in-modality transfer, where the target and the source data have similar structure or semantic meaning. However, when the target data (e.g., elec- tromyography signals, as in the NinaPro dataset) is drastically different from image data, it is diffi- cult to design prompts or expect good performance by only modifying the inputs without fine-tuning the pretrained models.
Table 9: Prediction errors of ORCA visual prompting methods.
Spherical
NinaPro
ECG
ORCA
29.85 0..85-0.72 98.05-0.11
0.39 33.13:0.23 31.46:10.83
0.29+0.0052
VP
0.57+0.0044
VPT
49.53=1.45
0.40+40.016
20

---

Under review as a conference paper at ICLR 2023
A.7 COMPATIBILITY WITH IN-MODALITY TRANSFER
natural question to ask is whether ORCA also tackle in-modality tasks. While we design ORCA to enable cross-modal transfer, we hypothesize that it should fa- cilitate same-modality transfer if two do- mains have large dataset distance. To val- idate this, we test ORCA on DomainNet datasets, which are commonly used to evaluate homogeneous DA methods (Peng et al. 2019). From
Table 10:
use dataset splits Tan al. (2020)
which removed some mislabeled outliers, and report the pre-
diction enrors for ORCA and fine-tuning (using Swin-base)
Real
Painting
Sketch
Clipart
ORCA 96.71 0.02
94.71 0.11
94,93 0.24
93.61 1-.54
Fine-tuning
93.33=1.33
75.794
33.00-0.13
86.01+2.62
Table IO we can see that ORCA achieves significantly better performance than the fine-tuning base- line, which shows that the feature matching ORCA can also help in-domain generalization.
21


Reviews:
[{"review_id": "t5f5aWZQhDL", "reviewer": "Reviewer_uNsA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows an universal approach of handling transfer learning in terms of cross-modal tasks, by using pre-training the embedding layers using OTDD loss metric. This paper although conduct empirical study on 13 tasks, demonstrating the effectiveness of embedding layer pretraining in cross modality learning. This approach is universal, effective and beats counterparts like NAS solutions, handcraft models, and other cross modal transfer tasks by quite some margin.", "summary_of_the_review": "This is a good empirical paper that the idea makes quite sense, and the authors conduct thorough experiments to support the claim such that pretraining embedding layer served as cross modal transfer is very effective. ", "strength_and_weaknesses": "Strengthes\nThe paper proposes an general/universal approach to do cross modal transfer learning, by adding a dimension transformation embedding layer and pretrain it using OTDD\nThe paper describe the motivation, challenges and solution clearly, and there are enough empirical results supporting the claim.\nThe comparisons against other cross modal transfer solution is convincing, e.g from NAS based class, hand-crafted expertise class.\nThe paper did some deep dives into the effectiveness of transformation embedding pretaining.\n\nWeaknesses\nSome of the empirical results and statements requires better reasoning, e.g:\nWhy just pretraining the embedding transformation layers will make the final result even better than hand-crafted expertise modes by a big margin in some downstream tasks? like Spherical and JSB Chorales.\nWhy using OTDD when pretraining the transformation embedding layers, the metric and semantic goal is not connected explicitly. \n\nThere are some places requires more deep diving, e.g:\nUsing convolutional network to do dimension transferring from target X -> source X, using it for Orca-swin makes sense because they both belong to vision modeling, but why still use it for Orca-roberta? Why convolutional network is selected, and how does it compare to simple feed-forward network for dimension transformation, their comparison in down stream tasks, etc, to demonstrate the effectiveness of this choice\n\nIt is good to show both Orca-swin/roberta results in Table 2, let people understand the impact of pretrained body as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nNPQ4odIxhg", "reviewer": "Reviewer_J4V2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies an pratical problem that how to effectively transfer pretrained models for diverse tasks. To this end, the authors proposed ORCA, which conducts task-specific adaptation by minimizing the OTDD between the proxy source embeddings and target embeddings. Extenvise transferring experiments on different tasks have been conducted to verify the effectiveness of the proposed ORCA.", "summary_of_the_review": "Please see weaknesses", "strength_and_weaknesses": "Strengths:\nIn the big model era, how to effectively adapt pre-trained models to different downstream tasks is an interesting and practical problem. Therefore, studying the cross-modal transfer ability of large-scale pre-trained models is significant.\n\nWeaknesses:\n1. The key idea of the proposed method is resorting to the OTDD between the proxy source data and target data to fine-tune the given pre-trained model. However, the definition and usage of OTDD are not clear enough. More importantly, according to the unclear statement, OTDD seems to be a modified version of OT which introduces the label information for complimentary. From this perspective, the reviewer wonders how the OTDD works under the scenario where the source and target data have no share labels. \n2. The authors argue that the proposed ORCA is an effective transfer learning method for pre-trained models compared to vanilla fine-tuning and hand-craft prompt tuning. Although the empirical results on two popular vision and language models (Swin and Roberta) show the superiority of the proposed method compared to the vanilla baselines. The effectiveness of ORCA should be further defended by comparing it to the recently-proposed vision and language prompt tuning methods such as [1].\n[1] Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i9ZpWm6nY1", "reviewer": "Reviewer_AGVC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new approach, ORCA, to tackle the cross-model transfer problem. ORCA equips a pre-trained transformer with a task-specific embedder and a task-specific predictor. When facing a downstream task, the embedder is learned to map data of any dimension to a standard sequence of tokens that is then fed into the pre-trained transformer. The predictor is learned to map the output to the specific label space. The learning objective is an optimal transport dataset distance that is used to match the source and target data distribution. After matching, the whole network is finetuned using standard loss.", "summary_of_the_review": "Overall, this paper tackles the important and challenging cross-modal transfer problem and gives a feasible solution, despite lack of novelty and potentially unfair comparisons with other methods. Given that the problem and the proposed approach would be interesting to ICLR audience, I would tend to accept this paper.", "strength_and_weaknesses": "Strength:\nAs pre-trained models become increasingly powerful, there exists a desire to apply these models to aid the learning of downstream tasks. But the mismatch of data distribution from different modalities hinders the application. This paper tackles this challenging and important problem and gives a simple but effective solution. The solution of this paper leverages the property of transformer that it has a standard input structure without any inductive bias of data, and proposes to learn an embedder that can map any input to match such structure. To better align different modalities, the authors further use optimal transport to match the source and target datasets. Although being simple, this adaptation scheme is reasonable and the authors give enough empirical evidence of that this approach is effective.\nThe experiments show that ORCA performs generally better than SOTA methods.\n\nWeakness:\nOne concern is the use of pre-trained model, which serve as a potential unfair comparison with other methods. In particular, the pretrained model the author chooses is RoBERTa and SwinTransformer trained on Five English-language corpora and ImageNet-22K, respectively. This may differ from other methods which potentially use different datasets for pre-training. So can authors provide some comparisons with state-of-the-art methods trained on the same dataset using identical architecture? I understand that such large-scale experiments are difficult to reimplement, but such comparisons would be greatly appreciated and would bolster the work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GG8KnFl4Y0", "reviewer": "Reviewer_WPBn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on cross-modal transfer learning that aims to utilize the pre-trained models for downstream tasks with diverse modalities. It devises a task specific embedding network to map data distribution to the pertained modality using the optimal transport dataset distance metric. Extensive experiments verify the efficacy of the proposed ORCA.", "summary_of_the_review": "Though the proposed method achieves good performance on a range of tasks, it is an integration of existing techniques, which weakens the contribution of this paper.", "strength_and_weaknesses": "Strength\n1. ORCA achieves good performance compared to other approaches on 13 tasks with different modalities, input dimensions, and prediction types.\n2. The importance of matching feature distribution is well supported and it is compatible with the in-modality transfer.\n\nWeaknesses\n1. The technical contribution is not significant. Though the performance is good and the effect of OTDD is verified, the main idea of using OTDD as a target for data distribution mapping is somewhat trivial.\n2. The custom embedding network uses a convolution layer for patch embedding like Vit. Does it only process image-like data? If so, it is inconsistent with the state as claimed before.\n3. What is the effect of other metrics in Stage I?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D5WchMwo4YJ", "reviewer": "Reviewer_tx7E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the cross-modal transfer ability of large-scale pre-trained models. The authors present a general workflow to fast and automatically exploit existing pre-trained models for different tasks. The authors provide some experiments to evaluate the proposed method.", "summary_of_the_review": "The paper is not clear and hard to follow. This work is incremental and lacks novelty. The experiments are not solid for comprehensively evaluating the proposed method.", "strength_and_weaknesses": "Strengths: \n- The authors have evaluated the proposed method on many different datasets.\n- The motivation of this work is interesting.\n\nWeaknesses:\n- The presentation of the paper is not clear. For example, Figure 1 cannot give us clear information (e.g., how does ORCA work? What represents cross-modal transfer?)\n- This work lacks novelty. It utilizes the existing techniques to fine-tune models on different datasets, which is incremental.\n- Although this paper evaluates the proposed method on many different datasets, it lacks sufficient comparison experiments with state-of-the-art baselines.\n- The authors claim their method could achieve task-specific adaption. Why does it need fine-tune? If it could achieve task-specific adaption on the upstream pre-training, it could achieve good performance on the downstream tasks without finetuning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject