Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper shows an universal approach of handling transfer learning in terms of cross-modal tasks, by using pre-training the embedding layers using OTDD loss metric. This paper although conduct empirical study on 13 tasks, demonstrating the effectiveness of embedding layer pretraining in cross modality learning. This approach is universal, effective and beats counterparts like NAS solutions, handcraft models, and other cross modal transfer tasks by quite some margin.
strength_and_weaknesses: Strengthes
The paper proposes an general/universal approach to do cross modal transfer learning, by adding a dimension transformation embedding layer and pretrain it using OTDD
The paper describe the motivation, challenges and solution clearly, and there are enough empirical results supporting the claim.
The comparisons against other cross modal transfer solution is convincing, e.g from NAS based class, hand-crafted expertise class.
The paper did some deep dives into the effectiveness of transformation embedding pretaining.

Weaknesses
Some of the empirical results and statements requires better reasoning, e.g:
Why just pretraining the embedding transformation layers will make the final result even better than hand-crafted expertise modes by a big margin in some downstream tasks? like Spherical and JSB Chorales.
Why using OTDD when pretraining the transformation embedding layers, the metric and semantic goal is not connected explicitly. 

There are some places requires more deep diving, e.g:
Using convolutional network to do dimension transferring from target X -> source X, using it for Orca-swin makes sense because they both belong to vision modeling, but why still use it for Orca-roberta? Why convolutional network is selected, and how does it compare to simple feed-forward network for dimension transformation, their comparison in down stream tasks, etc, to demonstrate the effectiveness of this choice

It is good to show both Orca-swin/roberta results in Table 2, let people understand the impact of pretrained body as well.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written 3/4,
clarity/easy to understand 2/4, some ideas needs better reasoning 
and novelty 3/4.
summary_of_the_review: This is a good empirical paper that the idea makes quite sense, and the authors conduct thorough experiments to support the claim such that pretraining embedding layer served as cross modal transfer is very effective.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new approach, ORCA, to tackle the cross-model transfer problem. ORCA equips a pre-trained transformer with a task-specific embedder and a task-specific predictor. When facing a downstream task, the embedder is learned to map data of any dimension to a standard sequence of tokens that is then fed into the pre-trained transformer. The predictor is learned to map the output to the specific label space. The learning objective is an optimal transport dataset distance that is used to match the source and target data distribution. After matching, the whole network is finetuned using standard loss.
strength_and_weaknesses: Strength:
As pre-trained models become increasingly powerful, there exists a desire to apply these models to aid the learning of downstream tasks. But the mismatch of data distribution from different modalities hinders the application. This paper tackles this challenging and important problem and gives a simple but effective solution. The solution of this paper leverages the property of transformer that it has a standard input structure without any inductive bias of data, and proposes to learn an embedder that can map any input to match such structure. To better align different modalities, the authors further use optimal transport to match the source and target datasets. Although being simple, this adaptation scheme is reasonable and the authors give enough empirical evidence of that this approach is effective.
The experiments show that ORCA performs generally better than SOTA methods.

Weakness:
One concern is the use of pre-trained model, which serve as a potential unfair comparison with other methods. In particular, the pretrained model the author chooses is RoBERTa and SwinTransformer trained on Five English-language corpora and ImageNet-22K, respectively. This may differ from other methods which potentially use different datasets for pre-training. So can authors provide some comparisons with state-of-the-art methods trained on the same dataset using identical architecture? I understand that such large-scale experiments are difficult to reimplement, but such comparisons would be greatly appreciated and would bolster the work.
clarity,_quality,_novelty_and_reproducibility: Clarity, quality: The paper is well organized and well written. 
Novelty: As pre-trained models become increasingly powerful, there exists a desire to apply these models to aid the learning of downstream tasks. But the mismatch of data distribution from different modalities hinders the application. This paper tackles this challenging and important problem and gives a simple but effective solution. 
Reproducibility: The code and configuration file for reproducing each experiment can be found in the supplementary material.
summary_of_the_review: Overall, this paper tackles the important and challenging cross-modal transfer problem and gives a feasible solution, despite lack of novelty and potentially unfair comparisons with other methods. Given that the problem and the proposed approach would be interesting to ICLR audience, I would tend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on cross-modal transfer learning that aims to utilize the pre-trained models for downstream tasks with diverse modalities. It devises a task specific embedding network to map data distribution to the pertained modality using the optimal transport dataset distance metric. Extensive experiments verify the efficacy of the proposed ORCA.
strength_and_weaknesses: Strength
1. ORCA achieves good performance compared to other approaches on 13 tasks with different modalities, input dimensions, and prediction types.
2. The importance of matching feature distribution is well supported and it is compatible with the in-modality transfer.

Weaknesses
1. The technical contribution is not significant. Though the performance is good and the effect of OTDD is verified, the main idea of using OTDD as a target for data distribution mapping is somewhat trivial.
2. The custom embedding network uses a convolution layer for patch embedding like Vit. Does it only process image-like data? If so, it is inconsistent with the state as claimed before.
3. What is the effect of other metrics in Stage I?
clarity,_quality,_novelty_and_reproducibility: The key idea and the pipeline are described clearly. According to the detailed instructions, I believe that this idea is easy to reproduce. However, it lacks enough novelty since the whole pipeline is build on previous techniques.
summary_of_the_review: Though the proposed method achieves good performance on a range of tasks, it is an integration of existing techniques, which weakens the contribution of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies an pratical problem that how to effectively transfer pretrained models for diverse tasks. To this end, the authors proposed ORCA, which conducts task-specific adaptation by minimizing the OTDD between the proxy source embeddings and target embeddings. Extenvise transferring experiments on different tasks have been conducted to verify the effectiveness of the proposed ORCA.
strength_and_weaknesses: Strengths:
In the big model era, how to effectively adapt pre-trained models to different downstream tasks is an interesting and practical problem. Therefore, studying the cross-modal transfer ability of large-scale pre-trained models is significant.

Weaknesses:
1. The key idea of the proposed method is resorting to the OTDD between the proxy source data and target data to fine-tune the given pre-trained model. However, the definition and usage of OTDD are not clear enough. More importantly, according to the unclear statement, OTDD seems to be a modified version of OT which introduces the label information for complimentary. From this perspective, the reviewer wonders how the OTDD works under the scenario where the source and target data have no share labels. 
2. The authors argue that the proposed ORCA is an effective transfer learning method for pre-trained models compared to vanilla fine-tuning and hand-craft prompt tuning. Although the empirical results on two popular vision and language models (Swin and Roberta) show the superiority of the proposed method compared to the vanilla baselines. The effectiveness of ORCA should be further defended by comparing it to the recently-proposed vision and language prompt tuning methods such as [1].
[1] Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models
clarity,_quality,_novelty_and_reproducibility: Quality: The paper is likely to have a modest impact on the community. Clarity: The paper is well organized but the presentation has minor details that could be improved. Originality: The main ideas of the paper are not novel or have limited novelty. Please see the weaknesses. 
summary_of_the_review: Please see weaknesses
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study the cross-modal transfer ability of large-scale pre-trained models. The authors present a general workflow to fast and automatically exploit existing pre-trained models for different tasks. The authors provide some experiments to evaluate the proposed method.
strength_and_weaknesses: Strengths: 
- The authors have evaluated the proposed method on many different datasets.
- The motivation of this work is interesting.

Weaknesses:
- The presentation of the paper is not clear. For example, Figure 1 cannot give us clear information (e.g., how does ORCA work? What represents cross-modal transfer?)
- This work lacks novelty. It utilizes the existing techniques to fine-tune models on different datasets, which is incremental.
- Although this paper evaluates the proposed method on many different datasets, it lacks sufficient comparison experiments with state-of-the-art baselines.
- The authors claim their method could achieve task-specific adaption. Why does it need fine-tune? If it could achieve task-specific adaption on the upstream pre-training, it could achieve good performance on the downstream tasks without finetuning.
clarity,_quality,_novelty_and_reproducibility: The paper is not clear and hard to follow. This work is incremental and lacks novelty.
summary_of_the_review: The paper is not clear and hard to follow. This work is incremental and lacks novelty. The experiments are not solid for comprehensively evaluating the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper dives into the study of the pre-trained model (PM) finetuning and designs a cross-modal transfer method via minimizing OTDD between the source and target domain.

Strengths: 
It should be acknowledged that how to adapt PM to diverse downstream tasks is a practical problem. The performance is promising. 

Weakness: This paper still suffers from some deficiencies, as the reviewers pointed out. First, the methodology is mainly based on the existing OTDD, and no specific strategies are proposed (that is the biggest concern on its novelty. Simply applying the published method cannot have a great impact on the ICLR community.). Besides, the SOTA prompt-based baselines on PM finetuning are not included and compared, which cannot fully convince the effectiveness of the proposed method.

In summary, this paper is under the bar of ICLR and thus cannot be accepted in its current form.
justification_for_why_not_higher_score: The novelty of proposed method cannot reach the bar of ICLR
justification_for_why_not_lower_score: Good applications with extensive experiments.
Decision: Reject