In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

## Abstract

Abstract In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task.
Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs.
However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs.
To address this, inspired by transfer learning, we propose In-Context Transfer Learning ( ICTL ), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer.
First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task.
Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task.
Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0 % percent 2.0 2.0\% 2.0 % on average, demonstrating the effectiveness of our method 1 1 1 Our code and data are released in https://github.com/zirui-HIT/ICTL . .

## 1 Introduction

In-context learning (ICL) is an effective approach for large language models (LLMs) to adapt to various tasks based on the brilliant generalize ability of LLMs (Xun et al., ; Song et al., ; Luo et al., ).
During the inference with ICL, input not only includes user questions but also several demonstrations to guide LLMs in generating answers correctly.
Considering the high cost of demonstration labeling, many methods utilize LLMs to synthesize demonstrations from scratch without human involvement (Kim et al., ; Jin & Lu, ).
For instance, Self-ICL (Chen et al., ) employs LLMs to synthesize demonstration based on the task definition, while Su et al. () improves the synthesis through iterations, where each iteration uses the previous results.

However, the synthesis using LLMs from scratch is constrained by the capabilities and knowledge of LLMs, limiting the quality of the synthesized demonstrations (Yu et al., ).
For example, a model trained pre-2023 can not use knowledge after 2023, while a model not trained on code tasks can not understand codes well (Rozière et al., ; Luo et al., ).
To solve this issue, thereby improving ICL performance while reducing human involvement, motivated by transfer learning (Pan & Yang, ; Iman et al., ), we propose to synthesize demonstrations for the target task by transferring the labeled demonstrations of similar tasks.
We use the idea of transfer learning since the previous works show that given similar source tasks, the performance of the target task can be enhanced according to the source task learning (Sun et al., ; Wang et al., ).
For example, as shown in Figure 1, the model can combine the context and the answer in the input of the sampled source demonstration, which is then used as the demonstration of the target task.

Based on the above discussion, we present In-Context Transfer Learning (ICTL), which obtains the demonstrations of the target task by transferring the demonstrations of the source tasks.
ICTL consists of two steps: sample the demonstrations similar to the target task, and transfer the sampled demonstrations to the target task, as shown in Figure 1.
First, we present an optimization objective to measure the transfer error, where we minimize the transfer error to sample the demonstrations highly similar to the target task.
Then, we transfer the sampled demonstrations to the target task with LLMs, taking the sampled results and the target task definition as the input.

To validate ICTL, we conduct experiments on Super-NaturalInstructions (Super-NI) (Wang et al., ), which can fully evaluate the multi-task capability of models with more than $1,600$ different tasks.
Compared to the demonstration synthesis by LLMs from scratch, our method achieves an average $2.0\%$ performance improvement, demonstrating its effectiveness.
Further analysis shows that our method can effectively sample demonstrations that are highly similar to the target task from source tasks, showing the effectiveness of our optimization objective.

Our contributions are as follows:

- •
We present to synthesize demonstrations by transferring labeled demonstrations of similar tasks, addressing that synthesis from scratch is constrained by the capabilities and knowledge of LLMs;
- •
We introduce an optimization objective to guide the source sampling, ensuring the similarity between the sampled results and the target task;
- •
Experiments on Super-NI show that, compared with the synthesis from scratch, ICTL delivers a $2.0\%$ performance improvement on Super-NI, proving the effectiveness of ICTL.

Figure: Figure 1: Comparison between previous demonstration synthesis methods (top) and our method (bottom). The blue part denotes the definition of the target task. The previous method synthesizes demonstration from scratch, while the model misinterprets the definition and generates a demonstration with the wrong answer, where the answer is not explicit mentioned by the sentence. In contrast, our method synthesizes demonstrations by transferring the sampled demonstrations, reducing the reliance on the capabilities of LLMs. The corresponding parts between the source and the target demonstrations of our method are marked in bold.
Refer to caption: x1.png

## 2 Related Works

### 2.1 Demonstration Synthesis

Demonstrations are of great importance in ICL, which can effectively help LLMs adapt various target tasks (Dong et al., ).
Considering the high cost of human labeling, many methods present to synthesize demonstrations using LLMs from scratch, lowering the human involvement (Kim et al., ; Chang & Fosler-Lussier, ; Jin & Lu, ).
Some methods focus on ensuring the correctness of the synthesized demonstrations, meeting the task definitions by filtering out low-quality synthesized results (Chen et al., ; Su et al., ; Yang et al., ).
Another type of method aims to increase the diversity of the synthesized demonstrations, creating ones dissimilar to synthesized results (Zhang et al., ; Shum et al., ; Wang et al., ).

However, the demonstrations synthesized by the current methods are constrained by the knowledge and capabilities of LLMs themselves, limiting their performance on the tasks unseen in their pre-training (Yu et al., ).
Although human-labeled demonstrations for new task scenarios can help LLMs generalize to these new tasks, labeling demonstrations for any new task or domain is costly (Wang et al., ).
To address these issues, we present ICTL, which synthesizes demonstrations for new target scenarios by transferring labeled source demonstrations similar to the target task, addressing the limitation of the knowledge and capabilities of LLMs.

### 2.2 Deep Transfer Learning

Transfer learning is a widely researched direction aimed at helping models acquire the ability to solve target tasks based on their existing capabilities from the source tasks (Pan & Yang, ; Zhuang et al., ).
With the impressive performance demonstrated by deep learning methods, deep transfer learning has become an important approach within the field of transfer learning (Iman et al., ).
Some methods focus on transferring and freezing model parameters to retain and learn features of different tasks (Scialom et al., ; Song et al., ; Wang et al., ; Rostami et al., ; Du et al., ).
Other transfer learning methods enhance the performance from the data perspective, studying how to adjust the training sequence of tasks, mix source task data with target task data, or modify the source task format to improve transfer learning performance (Xu et al., ; Wang et al., ; Madine, ).

However, current transfer learning methods rely on the labeled data of the target task and the model training, leading to the high cost of the adaption considering the high cost of labeling and LLM training.
Therefore, in this paper, we present to employ transfer learning to enhance ICL by synthesizing demonstrations using the labeled source demonstrations, lowering the human involvement and training cost, meanwhile helping LLMs adapt to various target tasks.

## 3 Methodology

Figure: Figure 2: The illustration of ICTL, taking the target task definition “If the provided sentence contains an explicit mention that answers the given question” as an example. ICTL consists of two steps: (i) Source Sampling: sample demonstrations that are similar to the target task from the source tasks; (ii) Target Transfer: transfer the sampled demonstrations to the target task. The blue part indicates the task definitions and demonstrations similar to the target task, and the gray part indicates that it is dissimilar. The green part denotes the transferred demonstrations.
Refer to caption: x2.png

In this section, we present ICTL, which synthesizes the demonstrations of the target task by transferring the labeled source demonstrations.
The illustration of ICTL is shown in Figure 2, which consists of two steps: source sampling (§3.1) and target transfer (§3.2).
Following the previous methods (Wang et al., ; Yang et al., ), we synthesize demonstrations for each target task offline, where we do not synthesize for each target question since we want to ensure high efficiency of the inference.
The prompts we used can be seen in Appendix B.
The computational efficiency analysis of ICTL is shown in Appendix E.

### 3.1 Source Sampling

The source sampling step is designed to sample demonstrations that are highly similar to the target task from the labeled source demonstrations.
In this paper, we define similarity as, if the sampling scale is $N$, the $N$ demonstrations most similar to the target task are called similar, while the others are called dissimilar.
We first present an optimization objective to guide the source demonstration sampling by minimizing the transfer error.
Then, we discuss how to sample the source demonstrations similar to the target task using our objective specifically.

#### 3.1.1 Optimization Objective for Source Sample

Supposing $S$ and $T$ represent the source and target tasks, respectively.
$\epsilon(h)$ denotes the task error of the hypothesis $h$, $\hat{\mu}$ represents the empirical distribution for each task, $W$ is the Wasserstein distance (Rabin et al., ) measuring the divergence between two distributions, $N$ denotes the sample scale for each task, and $\varphi$ is a negligible function.
The previous work (Redko et al., ) proves that the error of the transfer learning satisfies:

$$ $\epsilon_{T}(h)\leq\epsilon_{S}(h)+W(\hat{\mu}_{S},\hat{\mu}_{T})+\varphi(N_{S },N_{T})$ (1) $$

Further details of Equation 1 are discussed in Appendix A.
From Equation 1, we can see that the upper bound of the error for the target task is mainly determined by the error of the source task and the divergence between the source and target tasks.
It is hard to reduce the source task error since the source demonstrations can not be modified.
So we aim to minimize the target error by minimizing the divergence between the source and target tasks $W(\hat{\mu}_{S},\hat{\mu}_{T})$.

However, directly minimizing the upper bound results in $\hat{\mu}_{T}=\hat{\mu}_{S}$, which makes the transferred demonstrations irrelevant to the target task.
Therefore, giving $x$ as the representation vector of the task definition, we ask $\hat{\mu}_{T}$ to satisfy that:

$$ $\hat{\mu}_{T}=\operatorname*{arg\,min}_{\hat{\mu}}W(\hat{\mu},\hat{\mu}_{S})+W (\hat{\mu},x_{T})\vspace{-0.4em}$ (2) $$

In Equation 2, the first term minimizes the divergence between the target and source demonstrations, and the second term ensures that the target demonstrations are consistent with the target task definition.
We discuss the effectiveness of Equation 2 with experiments in Appendix F.1.

Given a series of source tasks $\{S_{i}\}$, suppose $N$ is the sampling scale of demonstrations from multiple source tasks $\{\hat{\mu}_{S_{i}}\}$, $N_{S_{i}}$ is the sampled number of $S_{i}$ and $\hat{\mu}$ is the empirical distribution of all possible sampled source demonstrations.
Based on Equation 1 and Equation 2, we can derive the optimization objective to sample the source demonstrations:

$$ $\hat{\mu}_{S}=\operatorname*{arg\,min}_{\hat{\mu}}\sum_{S_{i}}\frac{N_{S_{i}}} {N}(6W(\hat{\mu}_{S_{i}},x_{T})+W(x_{S_{i}},x_{T}))$ (3) $$

The proof of Equation 3 is provided in Appendix A.
It can be observed that the first term in the summation ensures that the sampled source task demonstrations are similar to the target task definition, and the second term ensures that the source task definitions are similar to the target task definition.
Using Equation 3, we can sample source demonstrations highly similar to the target task, thereby lowering the transfer error, and ensuring the quality of the transferred demonstrations.

#### 3.1.2 Sampling with Equation 3

Based on the above discussion, we then discuss how to sample source demonstrations specifically.
First, we embed the definitions and demonstrations of all source tasks, as well as the definition of the target task, into vectors using an embedding model.
Following previous work (Wang et al., ), we then filter the source tasks to select those most similar to the target task, reducing the overhead of subsequent calculations while ensuring performance.
The filtering is done by ranking the Wasserstein distance between the embedding vectors of the source and target task definitions.
From the filtered source tasks, we sample a fixed number of demonstrations using Equation 3.
We employ a randomized algorithm for the sampling, with details provided in the Appendix C.

### 3.2 Target Transfer

The target transfer step focuses on transferring the sampled demonstrations to the target task while ensuring that the transferred demonstrations are consistent with both the target task and the sampled demonstrations, transcending the limitations of the inherent capabilities and knowledge of LLMs.
The target transfer step consists of: Transfer, Verify, and Sample.

##### Transfer

is to transfer the sampled demonstrations to match the target task definition and format.
We employ LLMs for the transfer, where the input includes the definitions of both the source and target tasks, the source demonstration to be transferred, and a human-labeled example of the target task to specify the input and output formats.

##### Verify

is designed to check whether the transferred demonstration is consistent with the definition of the target task, improving the quality of the transferred demonstrations.
We employ LLMs to verify the transferred results.
The target task definition, one example, and the transferred demonstration are provided as input to check whether the transferred demonstration consistent with the task definition, with the correct input and output formats.
Any demonstration verified by the LLM as inconsistent is discarded to ensure the quality of the transferred results.

##### Sample

is to sample the verified target demonstrations with Equation 2, ensuring that the sampled demonstration is consistent with the target task while staying similar to the sampled source demonstrations, thereby transcending the limitations of the capabilities and knowledge of LLMs.
The sampling algorithm used for the transferred demonstration sampling is the same as the source sampling, with the optimization objective defined by Equation 2.
The sampled demonstrations are considered as the final output of our transfer method.

## 4 Experiments

### 4.1 Experiment Setup

#### 4.1.1 Dataset

We use the Super-NaturalInstructions dataset (Super-NI) (Wang et al., ) to validate our method, which contains over $1,600$ tasks, allowing for a comprehensive evaluation of the model cross-task generalization ability.
Following previous work (Wang et al., ), we conduct experiments on all English tasks in Super-NI, including $756$ tasks in the training set and $116$ tasks in the test set.
Based on prior research (Wang et al., ), we categorize all tasks in the test set into six categories to better analyze the performance of our method across different tasks, as shown in Appendix D.

#### 4.1.2 Metric

Following the Super-NI setup, we use Rouge-L (Rouge) and Exact Match (EM) as the evaluation metrics.
Rouge measures the overlap between the predicted output and the reference answer, while EM assesses whether the predicted output exactly matches the reference.
Following Wang et al. (), we mainly use Rouge as the evaluation metric, since EM is not suitable for tasks that can be answered in multiple ways (e.g., summarization, title generation).

#### 4.1.3 Model

We use BGE-EN-ICL (Chen et al., ) to embed task definition and demonstrations for the sampling, which is the state-of-the-art (SOTA) embedding model during our experiments.
For the transfer and inference, we use Llama3.1-8b-Instruct (Llama3.1-8b) (Dubey et al., ) and GPT-4o (OpenAI et al., ) as the experimental models.
Llama3.1-8b is one of the current best-performing open-source LLMs.
GPT-4o is one of the most powerful LLMs at present, which achieves SOTA performance on multiple mainstream benchmarks.
We mainly use Llama3.1-8b as the model of our analysis experiments due to the high cost of GPT-4o.

#### 4.1.4 Baseline

To thoroughly evaluate the effectiveness, we compare ICTL with the following baselines:

- •
Zero: No demonstrations are provided during inference, using a zero-shot setting;
- •
Direct: Directly use the sampled source demonstrations without transferring;
- •
Single: Only use the single human-labeled example as the demonstration;
- •
Synthesis: Synthesize demonstrations from scratch based on the one example provided.

#### 4.1.5 Implementation Detail

During sampling, we first select $16$ source tasks that are most similar to each target task.
For each target task, we sample $128$ demonstrations from the source tasks to be transferred.
Since Super-NI labels more than one answer for some questions, we transfer each answer with the question separately.
For the transferred results, we sample $512$ demonstrations for the inference.
We employ the 3-shot inference, selecting demonstrations for each test question based on the BM-25 similarity.
The reason for the parameter selection in this part is discussed in §4.4.

### 4.2 Main Experiment

**Table 1: The main experiment results on Super-NI. For each category, we use Rouge for evaluation. The best result for each category is highlighted in bold. Considering the high cost of GPT-4o, we only adapt experiments on $12$ tasks of the Super-NI test set for GPT-4o, where we randomly select $2$ tasks for each category, as shown in Appendix D.**
| Model | Category | Zero | Direct | Single | Synthesis | Ours |
| --- | --- | --- | --- | --- | --- | --- |
| Llama3.1-8b | Classification | $62.5$ | $60.3$ | $61.9$ | $65.4$ | $68.0$ |
| Comprehension | $56.1$ | $55.3$ | $60.0$ | $62.8$ | $67.8$ |  |
| Dialogue | $57.2$ | $62.7$ | $65.2$ | $73.1$ | $72.3$ |  |
| Extraction | $43.4$ | $38.7$ | $48.3$ | $53.2$ | $51.2$ |  |
| Generation | $38.4$ | $34.6$ | $41.1$ | $42.3$ | $45.8$ |  |
| Rewriting | $46.6$ | $32.6$ | $58.1$ | $60.5$ | $61.0$ |  |
| Overall (EM) | $36.9$ | $35.6$ | $39.7$ | $41.9$ | $44.0$ |  |
| Overall (Rouge) | $52.0$ | $48.8$ | $54.7$ | $57.8$ | $60.3$ |  |
| GPT-4o | Classification | $76.0$ | $72.2$ | $78.0$ | $79.0$ | $81.0$ |
| Comprehension | $78.4$ | $76.4$ | $74.9$ | $72.2$ | $78.4$ |  |
| Dialogue | $80.5$ | $78.5$ | $80.5$ | $83.5$ | $82.0$ |  |
| Extraction | $72.7$ | $65.2$ | $73.0$ | $71.0$ | $70.9$ |  |
| Generation | $39.1$ | $38.4$ | $42.6$ | $44.5$ | $45.4$ |  |
| Rewriting | $65.3$ | $59.3$ | $79.6$ | $80.2$ | $80.7$ |  |
| Overall (EM) | $49.2$ | $44.6$ | $49.4$ | $49.7$ | $51.8$ |  |
| Overall (Rouge) | $68.7$ | $65.0$ | $71.4$ | $71.8$ | $73.1$ |  |

As shown in Table 1, ICTL outperforms all baselines without transfer across different metrics and models on most categories, showing the effectiveness of our method.
Additionally, the results in the table also reveal the following insights:

##### Baseline

Compared to all baselines without transfer, our method achieves better performance, demonstrating the effectiveness of transferring.
Notably, ICTL brings $2.0\%$ improvement on average compared to the Synthesis setting.
This shows that the demonstrations synthesized by LLMs from scratch are constrained by the capabilities and knowledge of LLMs themselves.
In contrast, ICTL overcomes this constraint by providing the labeled demonstrations of other similar tasks, lowering the capability and knowledge requirement.
Additionally, the Direct setting directly using the sampled results as demonstrations leads to worse performance compared to the Zero setting.
This indicates that transfer is necessary when using demonstrations from other tasks to enhance performance, even if the sampled source demonstrations are highly similar to the target task.

##### Task

ICTL improves performance across most task categories, proving its effectiveness.
Specifically, the performance improvement is more significant for tasks with a higher rate in all test data, as there are sufficient similar source demonstrations for transfer, where the rates of different tasks are shown in Appendix D.
However, our method slightly underperforms compared to other settings in the Dialogue and Extraction tasks.
This is because these two tasks comprise only about $5\%$ of the total data, leading to lower-quality transfer results due to a lack of similar source demonstrations.
These findings suggest that it is important to use source demonstrations that are highly similar to the target task, as discussed in detail in §4.4.3.
To better observe the relationship between the source and target tasks of various categories, we static the transfer status of ICTL in Appendix F.2.

##### Metric

On both the EM and Rouge metrics, ICTL results in performance improvements, demonstrating its effectiveness.
Compared to EM, the performance improvement on Rouge is more significant.
That is because EM is harder to improve since it requires the generated answer to be completely identical to the reference answer, while Rouge allows for partial matches and flexibility in answer formats, providing credit for partially correct outputs, making it relatively easier to improve.

##### Model

With both Llama3.1-8b and GPT-4o, ICTL demonstrates performance improvements, confirming its effectiveness on LLMs with different levels.
Besides, compared to Llama3.1-8b, the performance enhancement of GPT-4o is somewhat weaker.
That is because, it can be observed that even under the Zero setting without demonstrations, GPT-4o is already capable of effectively addressing the tasks within Super-NI.
Therefore, when the model struggles to adequately tackle the target task on itself, ICTL can yield more significant performance gains.

### 4.3 Ablation Study

**Table 2: The ablation experiment results using Llama3.1-8b for the following components: (i) Transfer Verify: remove target verification; (ii) Source Sample: sample source demonstrations randomly; (iii) Target Sample: directly use the verified target demonstrations without sampling.**
| Method | EM | Rouge |
| --- | --- | --- |
| ICTL | $44.0$ | $60.3$ |
| - Target Verify | $41.4(-2.6)$ | $56.3(-4.0)$ |
| - Source Sample | $41.7(-2.3)$ | $43.7(-3.5)$ |
| - Target Sample | $43.7(-0.3)$ | $60.0(-0.3)$ |

To verify the effectiveness of each component in ICTL, we conduct ablation studies, where the experimental results are shown in Table 2.
Based on the table, we analyze each ablation study in order of its impact on performance, from most to least significant.

##### Target Verify

Removing transfer verification results in the most significant performance drop of $3.3\%$ on average across two metrics.
This indicates that the quality of demonstrations transferred directly is relatively low, showing the necessity of the verification.
There are two main reasons for the low quality of demonstrations transferred directly:
(i) For many test tasks, especially those that can be answered in multiple ways, it is difficult for LLMs to determine the format of the task, resulting in poor transfer results;
(ii) Previous research (Min et al., ) shows that LLMs could generate responses according to their prior experience during the pre-training while ignoring instructions, resulting in some generated results not meeting the definition and format of the target task.

##### Source Sample

Removing source sampling also causes a sharp performance drop of $2.9\%$ on average.
This is because, without source sampling, our method uses random sampling of source demonstrations, which leads to many dissimilar source demonstrations being sampled, decreasing the performance.
This result proves the necessity of sampling the source demonstrations according to the similarity to the target task before the transfer.
Besides, after removing source sampling, the performance of ICTL is near the Synthesis setting.
This shows that when the source demonstrations provided are significantly different from the target task, LLMs are more inclined to synthesize results by themselves without referring to the demonstrations provided.

##### Target Sample

Removing target sampling has the least impact on performance, causing only a $0.3\%$ decrease.
This is because, considering that ensuring the similarity between the demonstration and the question can effectively ensure the performance of ICL (Shum et al., ; Yang et al., ), during the evaluation, we also select the demonstration corresponding to each question based on BM-25, which overlaps with transfer sampling to a certain extent.

### 4.4 Analysis

Figure: (a) The performance with different scales of the sampled source demonstrations.

In this part, we analyze how different parameters affect the performance of ICTL to guide the selection of parameters in practical applications, as shown in Figure 3.
To better observe the performance changes brought about by ICTL with the change of different parameters, we use the Single setting as our baseline.
We also present the case study in Appendix G to present how ICTL transfer demonstrations, and evaluate the performance of ICTL under human-labeled target task demonstrations and cross-domain settings in Appendix F.4 and Appendix F.5.

#### 4.4.1 Source Demonstration Scale

The scale of source demonstrations available for different practical applications varies, so we analyze the impact of different scales of source demonstrations on the performance of our method, as shown in Figure 3(a).
From the figure, we can see that:
(i) When the scale of the source demonstration sampling is smaller than $128$, the overall experimental results exhibit an upward trend, demonstrating that increasing the amount of source demonstrations can effectively enhance the performance of our method;
(ii) When the sampling scale exceeds $128$, there is a slight decrease in performance, indicating that further addition of new source demonstrations does not continue to improve performance, as the number of demonstrations similar to the target task is limited.
Therefore, when obtaining demonstrations of source tasks, it is necessary to obtain as many demonstrations as possible to ensure that there are enough different abilities or knowledge for the target task.

Notably, compared to not using transfer learning, even transferring using one source demonstration can also effectively improve the performance of the target task.
This is because:
(i) Even using one single source demonstration, we can also synthesize a large amount demonstrations of the target task, resulting in a high-quality demonstration pool and thus better performance than without transfer learning;
(ii) Previous research (Kim et al., ; Wang et al., ) and the Synthesis setting of Table 1 show that even without source demonstrations, LLMs can still synthesize demonstrations based on the inherent knowledge of themselves, thereby enhancing inference performance.

#### 4.4.2 Source Task Scale

The scale of source tasks that can be obtained varies in practical applications, so we analyze the impact of different task scales on the performance of ICTL.
The experimental results are shown in Figure 3(b), from which we can see that:
(i) When the scale of source tasks is less than $16$, the overall performance exhibits an upward trend, while when the scale exceeds $16$, the performance starts to decline sharply, showing that blindly increasing the scale of the source task cannot bring about continuous improvement and the importance of ensuring the similarity between the source and the target tasks;
(ii) Compared to the source demonstration scale, the performance degradation is more pronounced with the increase in source task scale, since the scale of source tasks similar to the target task is limited, whereas simply increasing the scale of tasks, rather than the demonstrations, introduces more irrelevant information, leading to a more significant decrease in the quality of the transferred demonstrations and the inference performance.

#### 4.4.3 Task Similarity Rank

Considering there could be many new tasks emerging in future research and applications, to explore the adaptability of ICTL to new tasks, we conduct experiments to examine the impact of the similarity between the source and target tasks on performance.
We rank the Wasserstein distance of the embedding vectors of the source and target task definition in descending order, selecting the 1st, 10th, 100th, and last-ranked (756th in the Super-NI train set) source tasks to be transferred.
The experimental results are shown in Figure 3(c), from which we can observe the following:
(i) When the similarity ranking of the source tasks is within the top $10$, the performance of our method does not fluctuate significantly, since there exists multiple source tasks similar to those in the Super-NI test set, resulting in transferred demonstrations of comparable quality;
(ii) After the similarity ranking exceeds $10$, the performance of our method begins to decline sharply, indicating that demonstrations of tasks with large gaps can not help the target task, showing the importance of ensuring the similarity between the source tasks and target tasks.

#### 4.4.4 Target Transfer Scale

Due to the computational resource limitation in practical applications, the scale of the transferred demonstrations could be limited.
Therefore, we evaluate the performance of ICTL under different scales of transferred demonstrations, as shown in Figure 3(d).
From the figure, we can observe the following:
(i) In cases where only one single demonstration is transferred, the model performance decreases compared to without transfer, since the quality of the single transferred demonstration is lower than the provided example labeled by humans, leading to a performance decline;
(ii) Even only obtains $10$ demonstrations by transferring, our method achieves better performance than no transfer, whereas the scale of transferred demonstrations increases, the performance improves accordingly, demonstrating the necessity of sufficient transferring;
(iii) However, after the transferred demonstrations reach a certain scale, the model performance plateaus, since the information contained in the sampled source demonstrations is fully represented with $512$ transferred demonstrations, and further increasing the scale does not yield new high-quality demonstrations, while the performance is reduced since mixing more low-quality demonstrations.

## 5 Conclusion

In this paper, motivated by transfer learning, we propose ICTL, which synthesizes the demonstrations of the target task by transferring the similar labeled demonstrations, addressing the constraint that synthesizing from scratch with LLMs is limited by the capabilities and knowledge of LLMs.
We first present an optimization objective for sampling source demonstrations, aiming to minimize transfer errors by ensuring that sampled demonstrations are highly similar to the target task.
Subsequently, we transfer the sampled demonstrations to the target task using LLMs without human involvement, taking the sampled results and the target task definition as the input.
Experiments on Super-NI demonstrate that our method achieves an average improvement of $2.0\%$ over demonstrations synthesized without transfer, validating its effectiveness.
Additionally, analysis confirms that our method ensures a high similarity between sampled source demonstrations and the target task, proving the effectiveness of our proposed optimization objective.

## Appendix A Prove of Equation 3

In this section, we present the proof of Equation 3.
The proof includes three parts.
First, we discuss how to measure the transfer error when transferring across multiple source tasks.
Next, we address how to measure the discrepancy between the source tasks and the target task, denoted as $W(\hat{\mu}_{S},\hat{\mu}_{T})$.
Finally, we combine the existing results to derive Equation 3.

$$ $\epsilon_{T}(\hat{h}_{\bm{\alpha}})\leq\min_{h}\epsilon_{T}(h)+c_{1}+2\sum_{i= 1}^{N}\alpha_{i}\left(W(\hat{\mu}_{S_{i}},\hat{\mu}_{T})+\lambda_{i}+c_{2}\right)$ (4) $$

Suppose $\bm{\alpha}=\{\alpha_{i}\}$ represents the proportion of each source task, $c_{1},c_{2}$ are dependent on $n,N_{S_{i}},N_{T}$, and $\lambda_{i}=\min_{h}(\epsilon_{S_{i}}(h)+\epsilon_{T}(h))$ denotes the joint error of each source task $S_{i}$.
Based on Equation 1, the previous work (Redko et al., ) has proved that, for the transfer learning across multiple source tasks, the error satisfies Equation 4.

$$ $\hat{\mu}_{T}=\operatorname*{arg\,min}_{\hat{\mu}}\sum_{i=1}^{N}\alpha_{i}W( \hat{\mu}_{S_{i}},\hat{\mu})$ (5) $$

To minimize the error, we aim to minimize the upper bound of the error.
Since $\min_{h}\epsilon_{T}(h)\leq\sum_{i=1}^{N}\alpha_{i}\epsilon_{T}(h_{S_{i}})$, and $\sum_{i=1}^{N}\alpha_{i}\lambda_{i}\leq\sum_{i=1}^{N}\alpha_{i}\epsilon_{T}(h_
{S_{i}})+\alpha_{i}\epsilon_{S_{i}}(h)$, by replace $\epsilon_{T}(h_{S_{i}})$ with Equation 1, and ignoring the terms related to the error of source tasks and constants unrelated to $\mu$, we can obtain Equation 5.
Equation 1 defines how to sample the target demonstrations given the source demonstrations.
Then, we discuss the upper bound of the value of Equation 1, where we can adjust the source demonstrations to minimize the upper bound, thereby lowering the transfer error.

###### Theorem 1

Let $x_{S},x_{T}$ represent the representation vectors of the task definition of $S$ and $T$.
If

$$ $\hat{\mu}_{T}=\operatorname*{arg\,min}_{\hat{\mu}}W(\hat{\mu},\hat{\mu}_{S})+W (\hat{\mu},x_{T}),$ $$

then

$$ $W(\hat{\mu}_{S},\hat{\mu}_{T})\leq 6W(\hat{\mu}_{S},x_{T})+W(x_{S},x_{T}).$ $$

###### Proof 1

Let $\hat{\mu}_{S,T}$ represent the empirical distribution of the subset sampled from $X_{S}$, which has the data most close to $x_{T}$.
It is obvious that $W(\hat{\mu}_{S,T},x_{T})\leq W(\hat{\mu}_{S},x_{T})$.

Because $\hat{\mu}_{T}=\operatorname*{arg\,min}_{\hat{\mu}}W(\hat{\mu},\hat{\mu}_{S})+W
(\hat{\mu},x_{T})$, we can get:

$$ $\displaystyle W(\hat{\mu}_{T},\hat{\mu}_{S})+W(\hat{\mu}_{T},x_{T})$ $\displaystyle\leq W(\hat{\mu}_{S,T},\hat{\mu}_{S})+W(\hat{\mu}_{S,T},x_{T})$ $\displaystyle\leq W(\hat{\mu}_{S,T},\hat{\mu}_{S})+W(\hat{\mu}_{S},x_{T})$ $\displaystyle\leq W(\hat{\mu}_{S,T},x_{T})+2W(\hat{\mu}_{S},x_{T})$ $\displaystyle\leq W(\hat{\mu}_{S,T},\hat{\mu}_{T})+W(\hat{\mu}_{T},x_{T})+2W( \hat{\mu}_{S},x_{T})$ $$

Erase $W(\hat{\mu}_{T},x_{T})$ on both sides of the unequal sign, we can get:

$$ $\displaystyle W(\hat{\mu}_{T},\hat{\mu}_{S})$ $\displaystyle\leq W(\hat{\mu}_{S,T},\hat{\mu}_{T})+2W(\hat{\mu}_{S},x_{T})$ $\displaystyle\leq W(\hat{\mu}_{S,T},x_{T})+W(\hat{\mu}_{T},x_{T})+2W(\hat{\mu} _{S},x_{T})$ $\displaystyle\leq 3W(\hat{\mu}_{S},x_{T})+W(\hat{\mu}_{T},x_{T})+W(\hat{\mu}_{ T},\hat{\mu}_{S})$ $\displaystyle\leq 3W(\hat{\mu}_{S},x_{T})+W(\hat{\mu}_{S,T},\hat{\mu}_{S})+W( \hat{\mu}_{S,T},x_{T})$ $\displaystyle\leq 5W(\hat{\mu}_{S},x_{T})+W(\hat{\mu}_{S,T},x_{T})+W(x_{T},x_{ S})$ $\displaystyle\leq 6W(\hat{\mu}_{S},x_{T})+W(x_{T},x_{S})$ $$

Thus, we conclude:

$$ $W(\hat{\mu}_{T},\hat{\mu}_{S})\leq 6W(\hat{\mu}_{S},x_{T})+W(x_{T},x_{S}).$ $$

Theorem 1 provides an upper bound for measuring the difference between the demonstrations of the target task and the source task in task transfer, based on the discrepancy between the task definitions of the source and target tasks.
The reason this measurement holds is that the demonstrations for the target task are entirely transferred from the source demonstrations and the target task definition, meaning its characteristics can be described by them.
By substituting Theorem 1 into Equation 5, we can derive Equation 3.

###### Theorem 1

###### Proof 1

## Appendix B Prompts of ICTL

**Table 3: The prompt of transfer.**
| The Prompt of Transfer of ICTL |
| --- |
| Convert an example from Task A into an example for Task B, ensuring that both examples are consistent in terms of domain and knowledge. A sample for Task A is provided below. Please create a corresponding example for Task B, while maintaining the same domain and knowledge context. |
| The definition of Task A: {task_a_definition} |
| The definition of Task B: {task_b_definition} |
| — |
| For example, given the following example for Task A: |
| Input: |
| {task_A_question_demo} |
| Reason: |
| {task_A_rationale_demo} |
| Answer: |
| {task_A_answer_demo} |
| The corresponding example for Task B could be: |
| Input: |
| {task_B_question_demo} |
| Reason: |
| {task_B_rationale_demo} |
| Answer: |
| {task_B_answer_demo} |
| — |
| Based on the above example, please transfer the following example from Task A to Task B: |
| Input: |
| {task_A_question} |
| Answer: |
| {task_A_answer} |
| Your output format should be as follows: |
| Input: |
| <Converted input of Task B > |
| Reason: |
| <Explanation of the converted > |
| Answer: |
| <Converted answer of Task B > |

**Table 4: The prompt of verification.**
| The Prompt of Verification of ICTL |
| --- |
| Given a task description, several examples, and a pre-synthesized example, evaluate whether the pre-synthesized example matches the format and functionality of the provided examples and aligns with the task description. Based on the evaluation, determine whether the pre-synthesized example is ”Qualified” |
| You should check the pre-synthesized example based on the following criteria: |
| 1. Format Consistency: Does the pre-synthesized example follow the format of the provided examples? |
| 2. Task Fulfillment: Does the pre-synthesized example fulfill the requirements of the task description? |
| 3. Functional Accuracy: Are the input and output in the pre-synthesized example consistent with those in the provided examples? |
| If the pre-synthesized example meets all the criteria above, return: ”Qualified.” |
| If the pre-synthesized example fails to meet any of the criteria, return: ”Unqualified.” |
| Think it step by step. |
| Task Description: |
| {definition} |
| Examples: |
| Input: |
| {input_demo} |
| Reason: |
| {reason_demo} |
| Answer: |
| {answer_demo} |
| — |
| … |
| — |
| Pre-synthesized Example: |
| Input: |
| {input_transferred} |
| Reason: |
| {reason_transferred} |
| Answer: |
| {answer_transferred} |

**Table 5: The prompt of inference.**
| The Prompt of Inference of ICTL |
| --- |
| {task_definition} |
| Here are some demonstrations of the task: |
| — |
| Input: |
| {input_demo} |
| Reason: |
| {reason_demo} |
| Answer: |
| {answer_demo} |
| — |
| … |
| — |
| Based on the above demonstrations, please generate a response to the following question. |
| Your output format should be as follows: |
| Reason: |
| <Explanation of the answer > |
| Answer: |
| <Your answer > |
| Think it step by step. |
| Input: |
| {input_user} |

The prompts we used in ICTL are shown in Table 3, Table 4 and Table 5.

## Appendix C Algorithm for Dataset Sampling

In this section, we introduce the specific design of the randomized algorithm for sampling.
The algorithm utilizes simulated annealing (Bertsimas & Tsitsiklis, ) to optimize the sampling of demonstrations most similar to the target task with low computational costs.

Simulated annealing is a probabilistic global optimization algorithm that initially accepts suboptimal solutions at high temperatures to avoid local optima.
As the temperature gradually decreases, the algorithm converges.
The initial solution is generated through random sampling, where samples from the given demonstrations are randomly selected as the starting candidate solution.
We use Equation 3 and Equation 2 to evaluate the quality of random sampling from the given demonstrations, where we calculate the Wasserstein distance following Rostami et al. ().

During each iteration, the algorithm perturbs the current candidate solution to generate a new one.
Perturbations are categorized as ”small-step perturbations” and ”large-step perturbations.”
If the algorithm fails to find a better solution after several attempts, large-step perturbations are triggered to escape local optima.
Whether the perturbed candidate is accepted depends on the difference in scores between the new and current solutions.
Even if the new candidate is worse, there is a certain probability it is accepted.
This probability decreases as the temperature drops, promoting sufficient exploration of the search space.

The annealing process starts with an initial temperature of $1.0$, with a cooling rate of $0.99$.
The temperature decays after each iteration until it reaches the minimum value of $10^{-4}$, at which point the algorithm stops.
Additionally, we set a threshold: if no better solution is found after $100$ iterations, large-step perturbations are applied.

## Appendix D Category of Super-NI Test Tasks

**Table 6: Category of the Super-NI test set. The tasks used for GPT-4o experiments are marked in bold.**
| Category | Task ID |
| --- | --- |
| Classification | 20, 50, 190, 199, 200, 201, 202, 226, 232, 233, 242, 290, 349, 391, 392, 393, 520, 614, 623, 640, 641, 642, 738, 827, 828, 890, 935, 936, 937, 970, 1344, 1385, 1386, 1387, 1388, 1393, 1439, 1442, 1516, 1529, 1554, 1612, 1615, 1624, 1640 |
| Comprehension | 33, 133, 249, 304, 329, 330, 401, 648, 891, 892, 893, 1390, 1391, 1664 |
| Dialogue | 362, 879, 880, 1394, 1531, 1533, 1534 |
| Extraction | 36, 39, 281, 613, 620, 645 |
| Generation | 102, 219, 220, 288, 418, 500, 510, 569, 602, 619, 677, 743, 760, 769, 957, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1161, 1342, 1356, 1358, 1407, 1409, 1540, 1586, 1598, 1631, 1659, 1728 |
| Rewriting | 34, 35, 121, 402, 442, 670, 671, 1195, 1345, 1557, 1562, 1622 |

Figure: Figure 4: Category distribution of the Super-NI test set.

The category of the Super-NI test set is shown in Table 6, where we follow the category of Wang et al. ().
To better observe the impact of demonstration volume on transfer performance, we also count the distribution of demonstrations corresponding to different categories of tasks in the Super-NI test set, as shown in Figure 4.

## Appendix E Efficiency Analysis of ICTL

In this section, we provide a detailed analysis of the computational efficiency of ICTL.
Our goal is to analyze how the efficiency of source sampling and target transfer impacts the overall runtime and resource utilization, particularly in terms of the source demonstration scale and model inference time.

Let $N_{s}$ represent the total scale of the source demonstrations, $N_{s}^{S}$ the scale of the sampled source demonstrations, and $N_{t}^{S}$ the scale of the sampled target demonstrations.
The symbol $c_{\theta}$ denotes the time taken by the sampling algorithm to process one single data with parameter $\theta$.
Similarly, $c_{\mathcal{M}}$ represents the time for the model $\mathcal{M}$ to process a single data.

$$ $c_{\theta}N_{s}N_{s}^{S}+c_{\mathcal{M}}N_{s}^{S}+c_{\mathcal{M}}N_{s}^{S}+c_{ \theta}N_{s}^{S}N_{t}^{S}$ (6) $$

Then, we can represent the total computational cost with Equation 6.
In Equation 6, the first term represents the efficiency of source sampling, the second term corresponds to the target transfer, the third term describes the transfer verification, and the fourth term reflects the efficiency of the sampling of the synthesized demonstrations.

$$ $(c_{\theta}N_{s}+2c_{\mathcal{M}})N_{s}^{S}+c_{\theta}N_{s}^{S}N_{t}^{S}$ (7) $$

Based on Equation 6, we can derive Equation 7.
From the equation, it can be observed that the total runtime is primarily dependent on $N_{s}^{S}$, which is the scale of the sampled demonstrations.
Therefore, when computational resources are limited and the overall scale of the source demonstrations $N_{s}$ is large or the model inference time $c_{\mathcal{M}}$ is high, we can reduce $N_{s}^{S}$ to improve efficiency.

## Appendix F Further Analysis Experiment

### F.1 Target Sampling Divergence

Figure: Figure 5: Rouge on the Super-NI test set using the 32 different sets of randomly sampled transferred demonstrations with different values of Equation 2 using Llama3.1-8b. To better observe the changes, we normalize the values of the X-axis.

To validate the effectiveness of Equation 2 as a sampling metric, we randomly sample $32$ different sets of synthesized demonstrations.
For each set, $128$ demonstrations are randomly selected for each task, where the corresponding Equation 2 values and performance are shown in Figure 5.
From the figure, we can observe the following:
(i) As the Equation 2 value increases, the model performance shows a declining trend, indicating that the equation we proposed can effectively evaluate the divergence between the source demonstrations, the target task definition, and the synthesized demonstrations, which in turn helps assess model performance;
(ii) The variation in all experimental results is less than two points, suggesting that sampling synthesized demonstrations has a relatively small impact on performance, matching the results in Table 2, since we also select the question-related demonstrations during subsequent inference, which overlaps the effectiveness of the target sampling.

### F.2 The Transfer between Source Demonstrations and Target Demonstrations

Figure: Figure 6: The Sankey figure of the transfer between different source categories and target categories. The category follows Wang et al. ().
Refer to caption: x3.png

To investigate the relationships between different source tasks and target tasks, we static the number of target demonstrations corresponding to various source tasks after source sampling.
The statistical results are shown in Figure 6, from which we can see that:
(i) Almost all source tasks contribute to various target tasks, showing the necessity of sampling source demonstrations from different source tasks;
(ii) Among all source tasks, Generation provides significant assistance to all categories of target tasks, indicating that the generalization capability of different source tasks to various target tasks differs;
(iii) All categories of source tasks contribute demonstrations to target tasks for transfer learning, suggesting that when choosing source tasks to be sampled, it is essential to cover different categories of tasks to help target tasks acquire diverse capabilities and domain knowledge.

### F.3 Pass Rate of Transfer Verification

**Table 7: The pass rate of the transfer verification of ICTL on the Super-NI test set using Llama3.1-8b.**
| Category | Pass Rate (%) |
| --- | --- |
| Classification | $85.6$ |
| Comprehension | $68.4$ |
| Dialogue | $76.5$ |
| Extraction | $80.8$ |
| Generation | $61.3$ |
| Rewriting | $66.6$ |
| Overall | $74.1$ |

To verify the quality of the transfer results across different target tasks, we report the pass rates of transfer verification across various task categories, as shown in Table 7.
From the table, we can observe that:
(i) For all task categories, the synthesized demonstrations of ICTL achieve a pass rate of over $60\%$, indicating that the synthesized results generally satisfy the requirements of the target tasks;
(ii) Compared to tasks with more definite answers (e.g., Classification, Extraction), tasks with more open-ended answers (e.g., Generation, Rewriting) exhibit lower pass rates, since during transfer for these tasks, the model struggles to determine the appropriate answer format based on the task definition, leading to poorer transfer results.

### F.4 Combine ICTL with Human-Labeling Demonstrations

**Table 8: The performance of ICTL with and without additional human labeling using Llama3.1-8b. Single denotes only using the example of each target task. Multiple denotes using additional human-labeled demonstrations provided by Super-NI.**
| Metric | Single | + ICTL | Multiple | + ICTL |
| --- | --- | --- | --- | --- |
| EM | $39.7$ | $44.0$ | $41.5$ | $45.6$ |
| Rouge | $54.7$ | $60.3$ | $57.6$ | $60.4$ |

To verify the performance of our method in the presence of human-labeled demonstrations, we conduct experiments using additional demonstrations labeled by humans.
For each test task, we utilize the dataset excluding the $100$ test instances as the demonstration pool for the experiments.
We perform two sets of experiments: one using only human-labeled demonstrations and the other combined with the demonstrations transferred by ICTL.
The experimental results are shown in Table 8.
From the table, we can see that compared to the results using only human-labeled demonstrations, our method achieves further performance improvements, demonstrating the effectiveness in augmenting demonstrations labeled by humans.

### F.5 Performance of ICTL cross Different Domain

**Table 9: The cross-domain performance of ICTL on BOSS (Yuan et al., ) under different settings present in §4.1.4 using Llama3.1-8b. The performance of each category is evaluated with Rouge. We delete all toxic detection questions because the security restrictions of the model we use lead to refusal to answer questions with sensitive words. The best performance of each category is marked in bold.**
| Category | Zero | Direct | Single | Synthesis | Ours |
| --- | --- | --- | --- | --- | --- |
| Name Entity Recognition | 28.2 | 84.4 | 85.0 | 84.6 | 85.4 |
| Natural Language Inference | 21.1 | 21.7 | 21.0 | 22.5 | 24.8 |
| Question Answering | 60.6 | 62.5 | 64.2 | 62.3 | 64.8 |
| Sentiment Analysis | 71.5 | 73.8 | 70.0 | 70.8 | 74.0 |
| Overall (EM) | 33.2 | 36.8 | 34.8 | 35.3 | 39.9 |
| Overall (Rouge) | 45.4 | 60.6 | 60.0 | 60.0 | 62.2 |

To evaluate the performance of ICTL across different domains for the same task, we conduct cross-domain experiments.
Since all different tasks of Super-NI exhibit some variation, we opt to use BOSS (Yuan et al., ) for the experiments, which standardizes the input-output format for data across different domains within the same task, allowing for a more accurate evaluation of cross-domain performance.
The experimental results are shown in Table 9, from which we can observe the following:
(i) Under the setting of the same task across different domains, our method still yields performance improvements, demonstrating its effectiveness in cross-domain scenarios.
(ii) Apart from our method, Direct achieves the best performance, since despite being in different domains, the task and input-output format are identical, allowing the model to learn how to perform accurate reasoning from demonstrations in other domains of the same task.

## Appendix G Case Study

**Table 10: The case study of the capability transfer for the classification task.**
| Source Data | Definition | QA ZRE Question Generation on Subject Relation: |
| --- | --- | --- |
| You will be given a context, a subject and a relation. Your task is to generate a question based on the subject and relation. The generated question should include the given subject. Try to use a minimum number of words that are not present in either context, subject or relation while generating question. |  |  |
| Input | Context : Blind Company was shot in Bicheno, Tasmania in September 2008. |  |
| Subject : Blind Company |  |  |
| Relation : narrative location |  |  |
| Output | Which place is Blind Company in? |  |
| Transferred Data | Definition | Scitail1.1 Classification: |
| You are given two sentences. You have to find if there is entailment or agreement of the Hypothesis by the Premise. From the given pair of sentences, you should identify if there is enough information in the Premise to support the claim made in the Hypothesis. The Premise may not exactly be the same as Hypothesis. Your task is to return ’entails’ if the premise supports hypothesis else return ’neutral’. |  |  |
| Input | Premise: Blind Company was shot in Bicheno, Tasmania in September 2008. |  |
| Hypothesis: Blind Company is in Bicheno. |  |  |
| Output | entails |  |

**Table 11: The case study of the domain transfer for the classification task.**
| Source Data | Definition | XLWIC True or False Answer Generation: |
| --- | --- | --- |
| In this task, you are given a word, followed by two sentences. Your task is to figure out whether both the sentences use the aforementioned word with the same meaning. You should respond with ’True’ if the words in both sentences share the same meaning, and ’False’ otherwise. |  |  |
| Input | spring |  |
| Sentence1: I spent my spring holidays in Morocco. |  |  |
| Sentence2: He will hold office until the spring of next year. |  |  |
| Output | False |  |
| Transferred Data | Definition | ANLI R2 Entailment: |
| In this task, you will be presented with a premise and a hypothesis sentence. Determine whether the hypothesis sentence entails (implies), contradicts (opposes), or is neutral with respect to the given premise. Please answer with ”Contradiction”, ”Neutral”, or ”Entailment”. |  |  |
| Input | Premise: The spring season is a time of renewal and growth, often associated with warmer weather and longer days. |  |
| Hypothesis: He will hold office until the spring of next year. |  |  |
| Output | Neutral |  |

**Table 12: The case study of the capability transfer for the generation task.**
| Source Data | Definition | Para-NMT Paraphrasing: |
| --- | --- | --- |
| This is a paraphrasing task. In this task, you’re given a sentence and your task is to generate another sentence which express same meaning as the input using different words. |  |  |
| Input | someone other than the owner must have known it . |  |
| Output | someone , outside the owner , must have known about that . |  |
| Transferred Data | Definition | Ollie Sentence Answer Generation: |
| Given two noun phrases (arguments) and relationship between them, form a sentence that expresses these arguments with the given relationship. |  |  |
| Input | Relationship: ’known’ |  |
| Argument/Subject 1: ’someone other than the owner’ |  |  |
| Argument/Subject 2: ’it’ |  |  |
| Output | someone other than the owner must have known it. |  |

**Table 13: The case study of the domain transfer for the generation task.**
| Source Data | Definition | Peixian Rtgender Sentiment Analysis: |
| --- | --- | --- |
| Given a ’poster’ sentence and a corresponding ’response’ (often, from Facebook or Reddit)classify the sentiment of the given response into four categories: 1) Positive, 2) Negative, 3) Neutral, and 4) Mixed if it contains both positive and negative. |  |  |
| Input | Poster: La edad hace de las suyas con mis ojitos. Aging is getting to my eyes. OMG!!!! |  |
| Responser: sorryy jeje eso dije |  |  |
| Output | Neutral |  |
| Transferred Data | Definition | Reddit Tifu Title Summarization: |
| In this task, you are given a Reddit post as a text. Your task is to generate a title for this text. The title should start with T̈IFU by,̈ followed by a situation that caused humor. The title should contain 7-12 words, ideally. |  |  |
| Input | Text: La edad hace de las suyas con mis ojitos. Aging is getting to my eyes. OMG!!!! |  |
| Output | TIFU by letting aging ruin my eyes in seconds |  |

In this section, we conduct a case study on the data transferred by ICTL to gain a deeper understanding of how task transfer is performed.
We investigate from two perspectives: capability transfer (Table 10, Table 12) and domain transfer (Table 11, Table 13).
From these cases, we can observe that:
(i) Capability transfer generally occurs when the source and target tasks are highly similar, where when the definition or format of the source and target tasks are similar, our method can effectively understand the meaning of the source task and apply it to the target task;
(ii) Domain transfer occurs when there is a significant difference between the source and target tasks, where the model leverages the original input information from the source task, which includes domain knowledge, while the answers or other information for the target task are generated independently by the model.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In Equation 2, $\mu$ represents a data distribution of a task, while $x$ is a representation vector of a task definition. Since these two elements are in different spaces, how is the Wasserstein distance between them computed?

1. Lines 206-207 mention that task selection is ranked by the Wasserstein distance between the embedding vectors of task definitions. How is the Wasserstein distance calculated between two vectors? Typically, the Wasserstein distance is used to compare distributions rather than individual points.

1. How does the proposed source task filtering compare to retrieval-based approaches, such as those using Contriever or BERT?

1. What are the "perturbations" used in simulated annealing during the sampling of source demonstrations?

1. What kind of score function is employed in the simulated annealing process?
soundness: 2
strengths: 1. The synthesis of demonstrations from similar tasks is an interesting and novel approach to enhancing the quality of in-context learning.
1. The paper provides an optimization objective for sampling demonstrations, aimed at minimizing the task error bound.
1. The experimental results demonstrate improvements over traditional direct demonstration synthesis methods, showing the effectiveness of the proposed approach.
summary: The paper introduces In-Context Transfer Learning (ICTL), a method designed to improve demonstration synthesis for in-context learning (ICL) by leveraging labeled examples from similar tasks. 
Unlike conventional approaches that generate task demonstrations from scratch using LLMs, ICTL employs a transfer learning-inspired process consisting of two steps: Source sampling and target transfer. 
In the source sampling phase, relevant source tasks are filtered based on task definitions, and demonstrations are subsequently selected using a simulated annealing approach that minimizes transfer error.
During the target transfer phase, LLMs are used to adapt these selected demonstrations to synthesize examples suitable for the target task.
Experimental results on the Super-NI dataset indicate that ICTL outperforms traditional demonstration synthesis methods, effectively addressing the limitations of LLMs in producing high-quality demonstrations from scratch.
weaknesses: 1. The proof of Equation 3 (Appendix A) is not clearly explained. At the end of the proof (lines 868-869), the statement "by substituting Theorem 1 into Equation 5, we can derive Equation 3" skips too many intermediate steps, which makes the derivation ambiguous. While Theorem 1 and Equation 5 are related to finding the target task $\hat{\mu}_T$, Equation 3 is focused on determining the source task $\hat{\mu}_S$. This discrepancy is confusing and should be clarified further to convincingly demonstrate that Equation 3 is a reasonable objective.

1. The proposed approach is primarily applicable to benchmarks that provide explicit task definitions, which restricts its applicability to other benchmarks. This limitation confines the evaluation to the Super-NI dataset, and it would be beneficial to explore how this method could generalize to other settings.

1. The sampling process involving simulated annealing lacks sufficient clarity. Given that simulated annealing typically involves multiple iterations, it raises concerns about efficiency. Further details on how the sampling algorithm is implemented, and any measures taken to ensure efficiency, would be helpful.

Other Issues:
1. Rouge -> RougeL

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: While the paper provides an interesting application of transfer learning to in-context learning, I have the concern of the incremental improvement and relatively complex framework. The contribution is not sufficiently novel, and the results are a bit underwhelming. I would be happy if the authors can provide more compelling evidence of ICTL’s practical benefits (e.g., compare the computational cost in Table 1, provide additional evaluation on other tasks).
soundness: 3
strengths: 1. The paper is well written and easy to understand. It tackles an important problem of enhancing the generalibility of LLMs.
2. The proposed in-context transfer learning is simple yet produce reasonable improvements over generating demonstrations from scratch. The ablation studies and sensitivity analysis of paramters are beneficial. Evaluations on the Super-NI demonstrate the method’s cross-task performance. The paper also provides a code release for reproducibility, which is beneficial to the community.
summary: This paper proposes In-Context Transfer Learning (ICTL), a framework to enhance demonstration synthesis for in-context learning (ICL) by transferring labeled demonstrations from similar tasks. The motivation is to overcome the limitations of synthesizing demonstrations from scratch using large language models (LLMs). The framework aims to address cost and efficiency issues in generating high-quality task demonstrations by leveraging transfer learning. ICTL has two steps: (1) source sampling: select source tasks similar to the target task by minimizing transfer error. (2) target transfer: transfer these selected demonstrations to the target task’s format using LLMs (transferring, verify, sample). Experiments on the Super-NI dataset show that ICTL achieves 2.0% improvements on average compared to synthesizing demonstrations from scratch.
weaknesses: I have two major concerns:

1. The performance improvements seem marginal. The 2% performance improvement over baseline method (generating target demonstartions from scratch) is marginal and may not justify the added complexity of ICTL. The procedure of sampling, transferring, verifying, and re-sampling introduces computational overhead. It would be great to include the computation cost comparison between different approaches in Table 1. Moreover, from the analysis of different parameters in Figure 3, seems the performance can drop signficiantly (more than 2%) if the parameters are not selected properly. This let me suspect the effectiveness of the proposed framework in more general real-world scenarios. It would be interesting to see more results on other tasks. 

2. The proposed framework follows the standard transfer learning pipeline. The theoretical analysis is also straight-forward extension. In general, I feel the technical contribution is limited while there are some specific designs for LLM domain.

---

confidence: 4
contribution: 2
initial_rating: 1
presentation: 1
questions: 1. What are some examples of source tasks? Where are the 128 or 512 demonstrations (mentioned in section 4.1.5) from? What are source-target task pair examples that were ranked most similar to each other?

2. How many tokens is the average Super-NI answer in your evaluation dataset?
soundness: 1
strengths: 1. The proposed idea seems reasonable at a high level.

2. Results show some promise of proposed method.
summary: This paper proposes a method to improve ICL (in-context learning) by integrating principles from transfer learning. Specifically, it finds similar demonstrations from another task, using distributional similarity between tasks, to copy into the ICL prompt. It finds very modest increases in ROUGE score on the Super-NI benchmark.
weaknesses: I strongly believe this paper, in its current form, does not meet the threshold for ICLR acceptance.

1. It is a bit hard to determine novelty of this work. Authors cite the previous main ICL paradigm of generating synthetic examples from a single human-provided example, but do not compare to prior methods that combine transfer learning, or similarity of examples, to do in-context learning. For instance, Dr.ICL (Demonstration-Retrieved ICL; Luo et al 2023), seems to be retrieving demonstrations that are similar to the task at hand. Thus, Luo et al seems relevant but was not mentioned or compared against.

2. Much of the paper language needs editing. There are issues with writing in most paragraphs.

3. Figure 2 is not particularly informative after Figure 1. Additionally, the examples in Figures 1 and 2 do not make much sense. For instance, in Figure 1, most of the “Definitions” boxes do not make sense (they are not well defined sentences), and the “Question” and “Output” pairs do not make sense either--the output is “Yes” when the question is not a yes/no question. Therefore, it confuses the reader about how the proposed transfer step would help ICL.

4. There are serious coherence issues in Section 3: Method. In Section 3.1: “...We define similarity as, if the sampling scale is N, the N demonstrations most similar to the target task are called similar...” Isn’t this a recursive definition? How are the most similar N demonstrations computed? Variables in 3.1.1 are not well defined, which is a serious issue as this is the only section in the paper with math. What is a hypothesis? What is a task error? What does the “distribution for each task” mean? How are tasks defined statistically/numerically? Where is this task error term used in the method?

5. Evaluations: Table 1 does not seem to show Super-NI to be a challenging task. There is not much of a difference, especially with GPT-4o, on zero-shot and proposed method (68.7 vs 73.1 ROUGE score), as acknowledged by authors. The gap is 52.0 vs 60.3 with Llama3.1-8b. Given the small gap, I would recommend authors to choose a more challenging task where ICL is much more crucial to the model’s success, so that any differences are less likely to be due to noise.

6. Table 2 has a serious typo/weird result. Why is “Source Sample” row, “Rouge” column listed as 43.7 (-3.5), even though 43.7 should be a decrease of -16.6, instead of -3.5? The fact that this number is so low compared to the other entries in the column makes it seem like it is a typo.

7. ROUGE is not necessarily a good metric, depending on the task and dataset answer length. A better one would be expert human evaluations (preferably pairwise evaluations between the methods). If authors do not have the resources for human eval, there are better automated metrics available.

8. All of the results discussion about Figure 3, from Sections 4.4.1 through 4.4.4, lack insight, provide no analysis, and are just restating the graph trends in many words. The redundancy of information throughout this paper is prevalent in other sections as well, not just these sections.

9. Section 4.4.1: “Even transferring using one source demonstration can also effectively improve the performance of the target task. This is because: (i) Even using one single source demonstration, we can also synthesize a large amount demonstrations [sic]  ... (ii) ... even without source demonstrations, LLMs can still synthesize demonstrations...” This is a tangled bunch of sentences that has many issues. First, (ii) seems to contradict the premise of using one source demonstration, as written at the beginning of the section and in (i). Second, (i) and (ii) seem to both talk about the same reason--demonstrations can be synthesized.

10. I encourage the authors to improve their paper by cutting down on a lot of redundancy. A 5-page paper undershooting the page limit is better than a 9-page paper that repeats much of the information multiple times.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - The experiments in the paper focus solely on final task performance, with no analysis of generated target demonstration quality. It would strengthen the paper’s contributions if more analysis was presented. Could the authors provide example target demonstrations to illustrate the model’s limitations and how they were improved? 

- How does ICTL perform when there were no closely related source tasks available? Experimental results on how ICTL performs when there are limited or no closely related source tasks available would be beneficial.

- How does the computational cost of ICTL compare to traditional ICL?

- Could the authors compare ICTL with recent works on learning-based retrievers in LLM RAG approaches instead of Direct?
soundness: 3
strengths: - ICTL combines ICL and Transfer Learning, presenting a simple yet effective method for ICL across different tasks, where  target demonstrations are strictly limited.

- ICTL achieves consistent improvement on the Super-NI dataset, with a 2% average performance gain over other baselines.

- The paper is written in a clear manner, making it easy to understand the proposed method and its applications.
summary: This paper introduces In-Context Transfer Learning (ICTL), a method to enhance in-context learning by synthesizing examples for new target tasks through the transfer of demonstrations from similar tasks. Unlike conventional synthesis methods constrained by the limitations of LLMs, ICTL leverages labeled demonstrations from related source tasks. The approach involves selecting source tasks closely aligned with the target task and then refining these examples through LLMs to match the target. Experimental results on the Super-NI dataset show that ICTL achieves a 2% average performance improvement over other baselines.
weaknesses: - While the proposed ICTL combines the concepts of Transfer Learning and ICL, its contribution is not clearly specified compared to prior research involving advanced demonstration generation in zero-shot ICL settings and advanced retrieval methodologies for ICL with RAG. 

[1] SELF-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations (EMNLP 2023)

[2] Demonstration Augmentation for Zero-shot In-context Learning (ACL 2024)

[3] Bridging Distribution Gap via Semantic Rewriting with LLMs to Enhance OOD Robustness (ACL 2024)
   
- The selected baselines used in the experiments mostly correspond to ablations of the proposed method. For a fairer evaluation, it is necessary to directly compare ICTL with similar approaches discussed in the related work. Specifically, the authors could compare with methodologies that rely on LLMs to generate demonstrations for ICL; e.g., [1, 2] or those that modify source task formats to enhance transfer learning; e.g., [3]. These comparisons could state the contributions and novelty of the proposed ICTL clearly. 

- While restructuring the retrieval process based on a Transfer Learning objective is a positive step, the generation of new target demonstrations relies only on LLM prompting and self-verification, which can be insufficient to fundamentally improve task performance. Recent studies such as [4] claim that LLMs are limited in their ability to self-correct reasoning errors. To position target demonstration generation as a significant contribution, the paper needs to further enhance this aspect beyond sampling by proposing novel approaches to optimize the generation process.

[4] LARGE LANGUAGE MODELS CANNOT SELF-CORRECT REASONING YET (ICLR 2024)

- ICTL involves multiple steps, including sampling, transferring, and verifying demonstrations, which may increase computational demands compared to simpler in-context learning approaches. Considering the use of high-cost and high-quality models like GPT-4o and Llama3.1-8b, along with the extra inference required for demonstration generation and self-verification within ICTL, the 2%
average performance improvement seems less significant to justify the increased cost.

- If there are few or no closely matching source tasks, the quality of transferred demonstrations may suffer, impacting the model’s ability to generalize to the target task.

- The Super-NI dataset used to demonstrate performance improvement is somewhat limited. In Section 1, the authors claim that their approach aims to enhance performance on tasks where the LLM lacks prior knowledge. However, it is hard to argue that the tasks or knowledge in the Super-NI dataset are entirely absent from GPT-4o or Llama3.1-8b-Instruct models. To provide more convincing validation, the authors should either use a more up-to-date benchmark or design tasks in environments like Alfworld [5] for additional testing.

[5] ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (ICLR 2021)

Meta-review: The paper presents a transfer learning method which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. The reviewers acknowledged the importance and potential of the problem addressed. However, they identified several limitations that hinder the paper's readiness for acceptance:

- The novelty of the proposed is not sufficiently clear, especially in differentiating it from existing methods, and the contributions do not constitute a substantial advancement.

- The evaluations are limited to the Super-NI dataset, and the performance improvements are modest relative to the computational overhead. Reviewers suggested broader evaluations and deeper insights into the results.

- Some reviewers highlighted issues in writing clarity, methodology explanation, and figure presentation. 

- The simulated annealing step and computational costs raised concerns about the method's practicality in real-world scenarios.

Despite the strengths of the paper, such as the reasonably novel combination of ICL and transfer learning, and empirical results demonstrating improvements, the reviewers were not uniformly convinced of its readiness for publication. Therefore, I recommend rejecting this paper in its current form.

Decision: Reject