Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance - A Case Study in Finance

## Abstract

Abstract The application of large language models (LLMs) in domain-specific contexts, including finance, has expanded rapidly. Domain-specific LLMs are typically evaluated based on their performance in various downstream tasks relevant to the domain. In this work, we present a detailed analysis of fine-tuning LLMs for such tasks. Somewhat counterintuitively, we find that in domain-specific cases, fine-tuning exclusively on the target task is not always the most effective strategy. Instead, multi-task fine-tuning - where models are trained on a cocktail of related tasks - can significantly enhance performance. We demonstrate how this approach enables a small model, such as Phi-3-Mini, to achieve state-of-the-art results, even surpassing the much larger GPT-4-o model on financial benchmarks. Our study involves a large-scale experiment, training over 200 models using several widely adopted LLMs as baselines, and empirically confirms the benefits of multi-task fine-tuning. Additionally, we explore the use of general instruction data as a form of regularization, suggesting that it helps minimize performance degradation. We also investigate the inclusion of mathematical data, finding improvements in numerical reasoning that transfer effectively to financial tasks. Finally, we note that while fine-tuning for downstream tasks leads to targeted improvements in task performance, it does not necessarily result in broader gains in domain knowledge or complex domain reasoning abilities.

## 1 Introduction

Recently, the application of large language models (LLMs) in domain-specific contexts has seen rapid growth, particularly in fields such as medicine (Singhal et al., ; Wu et al., ), law (Huang et al., ), and finance (Cheng et al., ; Wu et al., ). As LLMs are increasingly adopted across various domains, accurate evaluation of their domain-specific capabilities has become more necessary. While many benchmarks exist to evaluate LLM performance, they are typically designed for general purposes and not specifically for domain-specific evaluations.

A common method for assessing LLM performance within a domain is through downstream tasks (Yang et al., ; Gu et al., ; Xie et al., ). Such benchmarks emphasize well-defined, highly specific tasks that seek to reflect real-world applications within the target domain. These tasks are frequently framed as standard natural language processing (NLP) problems, such as text classification, summarization, causal reasoning, arithmetic reasoning, and more. While each test individually provides limited insight into domain-specific capabilities, when combined, they offer a broader representation, facilitating a more comprehensive evaluation.

LLMs possess zero-shot capabilities (Kojima et al., ), allowing them to perform downstream tasks without prior task-specific training. However, they sometimes struggle with these tasks due to issues such as formatting, problem understanding, or reasoning failures. A common approach to improve their performance is to fine-tune the models directly on the downstream task, improving performance on it directly (Zhou et al., ). Consequently, many benchmarks provide both training and test splits to facilitate fine-tuning and evaluation. Still, fine-tuning on a single task may not fully optimize the model’s performance.

In this work, we investigated the impact of multi-task fine-tuning. Instead of fine-tuning the model solely on the target downstream task, we fine-tune it on multiple related downstream tasks simultaneously. We conduct a massive ablation study to explore the interactions between various financial tasks and datasets. In total, we train 220 models to provide an in-depth evaluation of different financial benchmarks and LLMs. Our empirical findings demonstrate that incorporating training data from multiple downstream tasks creates a cocktail effect, where the integration of multiple datasets creates a synergistic improvement in model performance, even for a single task.

Beyond task-specific data, we explore the use of general instruction-following data during the fine-tuning process and assess its impact, suggesting that it may play a regularization role. Since financial tasks often involve numerical reasoning, we also investigate the effect of incorporating general mathematical data, particularly word problems, into the training mix.

We showcase the power of the multi-task fine-tuning approach by achieving state-of-the-art results on well-established financial benchmarks. Notably, we improve the performance of the 3.8B model Phi-3-Mini (Abdin et al., ), enabling it to surpass the much larger and more powerful GPT-4-o model (OpenAI, ) in terms of benchmark accuracy, as can be seen in Fig. 1. More details are provided in Section 4.3.

Finally, after thoroughly examining how different tasks interact, we evaluate the effect of multi-task fine-tuning on extrapolation capabilities. To assess this, we test the models on domain-specific benchmarks that were not included in the training process and analyze how fine-tuning impacts performance. Our results suggest that training on downstream tasks alone may not lead to significant improvements in domain knowledge or complex reasoning abilities.

## 2 Multi-task Fine-Tuning

Given a set of downstream tasks that have been selected to assess a model’s capabilities in a target domain, the challenge becomes finding the optimal way to fine-tune the model across these tasks to maximize performance. In multi-task learning, the goal is to assess whether there exist synergies among the tasks, allowing for leveraging shared information to enhance individual task performance.

### 2.1 Background

Multi-task training is not a new concept. The efficiency of this approach has been demonstrated for general domains in the past (Aribandi et al., ; Aghajanyan et al., ). More recent work has shown success with instruction tuning specifically (Wang et al., ; Yue et al., ), as well as showing the impact of additional datasets. On the other hand, the exact interactions between tasks are still understudied, especially in the domain-specific case, and more specifically for finance. Past approaches to domain-specific adaptation, such as Cheng et al. (), used broader domain data, removing the ability to observe the interactions between the tasks themselves. While Wang et al. () use a task oriented approach in finance, there is no measurement on the task level, or experimentation around adding general data.

### 2.2 Problem Formulation

Let $\mathcal{M}$ be a pre-trained language model, and let $\mathcal{D}=\{D_{1},D_{2},\dots,D_{n}\}$ represent a set of $n$ datasets used for fine-tuning. The set $\mathcal{D}$ is partitioned into two subsets: domain-specific datasets $\mathcal{D}_{\text{domain}}=\{D_{1},\dots,D_{k}\}$, which correspond to tasks $T_{1},\dots,T_{k}$, and general datasets $\mathcal{D}_{\text{gen}}=\{D_{k+1},\dots,D_{n}\}$, which are not directly evaluated in the test tasks. Our goal is to determine what is the optimal combination of datasets for fine-tuning $\mathcal{M}$ to maximize performance on a domain-specific task.

The task-level objective for multi-task fine-tuning can be formalized as:

$$ $\mathcal{D}^{*}_{i}=\arg\max_{\mathcal{D}_{i}}\left(\mathcal{E}_{T_{i}}( \mathcal{M}_{\mathcal{D}_{i}})\right)$ (1) $$

where $\mathcal{M}_{\mathcal{D}_{i}}$ represents the model trained on $\mathcal{D}_{i}\subseteq\mathcal{D}$, and $\mathcal{E}_{T_{i}}$ represents the specific evaluation metric for $T_{i}$.

The key questions we aim to address are:

- 1.
Given $\mathcal{D}$, is fine-tuning on the domain-specific dataset $D_{i}$ alone the most effective way to improve performance on task $T_{i}$ (i.e., does $\mathcal{D}_{i}^{*}=\{D_{i}$})?
- 2.
Can fine-tuning on general datasets $D_{j}\in\mathcal{D}_{\text{gen}}$ improve performance on the domain-specific tasks $T_{1},\dots,T_{k}$?

### 2.3 Methodology

Figure: Figure 2: Overview of the methodology. The steps are: $\binom{n}{0}\rightarrow\binom{n}{1}\rightarrow\binom{n}{2}\rightarrow\binom{n} {n-1}\rightarrow\binom{n}{n}$.
Refer to caption: extracted/5894109/figs/fine-tuning_overview.png

To investigate these questions, we employ a systematic empirical approach by fine-tuning the model on different combinations of datasets.
We use an incremental approach for fine-tuning the model, starting from single-dataset fine-tuning to more complex mixtures. This methodology allows us to isolate the impact of individual datasets as well as explore the interactions between datasets when fine-tuned together. All fine-tuning steps use the base model $\mathcal{M}$, and a standard uniform shuffling of $\mathcal{D}_{i}$. An overview of our approach for $n$ training datasets is shown in Fig. 2.

Before fine-tuning, we evaluate the ’vanilla’ model in its pre-trained state. This step establishes the baseline for all further comparisons, allowing us to quantify the relative changes in performance when fine-tuning.

After the initial fine-tuning stage, we use a single dataset at a time. We use this step primarily to understand the performance of standard single task finetuning. Additionally, this step enables us to identify the number of samples required from each dataset for stable convergence of the training loss (in less than three epochs).

To explore the interactions between datasets, we fine-tune the model on pairs of datasets. By training on two datasets simultaneously, we aim to investigate the degree of influence each dataset has on improving or impairing the model’s performance on another.

Next, to fully understand the impact each dataset has, we remove a single dataset at a time, and use all other datasets for training. This step is crucial for understanding exactly how much a specific dataset influences the overall results when added to a cocktail.

Finally, we fine-tune the model on the entire set of datasets simultaneously, completing the study.

## 3 Datasets

As part of our study we selected a variety of datasets for training and evaluation. These datasets represent central downstream NLP tasks from the financial domain, covering central benchmarks from previous works (Wu et al., ; Cheng et al., ; Wang et al., ). These tasks include named entity recognition (NER), sentiment analysis, numerical reasoning, and other domain-specific challenges.
The datasets are categorized into two: training and evaluation datasets. The training set includes two general datasets, as well as the training split of seven financial tasks. The evaluation set includes the test split of the seven tasks and additional datasets aimed at testing broader financial reasoning abilities. Descriptions of the datasets are below, a summary of their key properties can be found in Table 1, and an example from each dataset can be found in Appendix E.

Table: Table 1: Summary statistics of the datasets used for training.

### 3.1 Core Financial Datasets

The following datasets are used both for fine-tuning and for evaluation:

- •
Headline: This dataset consists of financial news headlines, accompanied by binary questions. The dataset aims to represent how financial information is presented in news media, and the primary purpose of the dataset is event detection in finance. This dataset is an adaptation of the original headline dataset (Sinha & Khandait, ) by FinGPT (Wang et al., ).
- •
FPB: The Financial PhraseBank (FPB) (Malo et al., ) dataset is widely used for sentiment analysis in the financial domain. It contains annotated financial phrases and sentences, allowing the model to learn financial sentiment nuances.
- •
FinNerCLS: This dataset, created by FinGPT (Wang et al., ), frames named entity recognition (NER) in finance as a classification task. This allows for more straightforward evaluation, and greater similarity to other tasks. The dataset includes sentences, entities from the sentence, and entity type labels.
- •
FinQA: FinQA (Chen et al., ) is a question-answering dataset that contains real-world financial documents and requires models to extract and reason over financial data to provide accurate answers. It focuses on reading comprehension tasks in finance involving numerical reasoning.
- •
ConvFinQA: The ConvFinQA dataset (Chen et al., ) extends FinQA by including conversational aspects, making the question-answering process more complex. It tests the model’s ability to handle multi-turn financial dialogues when extracting relevant information from financial documents. For simplicity we use the BloombergGPT (Wu et al., ) adaptation of the dataset.
- •
Twitter-Topics: This dataset consists of finance-related topics discussed on Twitter. Each tweet needs to be classified in to one of 20 optional labels(^1^11https://huggingface.co/datasets/zeroshot/twitter-financial-news-topic).
- •
Twitter-SA: A dataset of financial-sentiment annotated tweets. Each tweet needs to be classified as one of [’Bearish’, ’Bullish’, ’Neutral’](^2^22https://huggingface.co/datasets/zeroshot/twitter-financial-news-sentiment).

### 3.2 General Training Datasets

Besides the financial datasets discussed earlier, we also use two non-financial training datasets. The rationale for incorporating the first dataset is the proven benefit of instruction tuning in general (Longpre et al., ). Additionally, since finance-related tasks often involve mathematical reasoning, we include mathematical training data to improve the model’s performance in this area. Neither of these datasets are incorporated during evaluation. The datasets are as follows:

- •
Open-Orca: Open-Orca (Lian et al., ) is an open source recreation of the Orca (Mukherjee et al., ) dataset, containing diverse instructions spanning multiple keys LLM ’skills’. The dataset was created by using GPT4 and GPT3.5 to augment the FLAN collection (Longpre et al., ).
- •
Orca-Math: Orca-Math (Mitra et al., ) is a mathematical reasoning dataset that includes synthetic mathematical word problems. This dataset does not involve any domain-specific financial knowledge, but rather is used to enhance mathematical reasoning abilities.

### 3.3 Additional Evaluation Datasets

In addition to the core datasets outlined in Section 3.1, we also use FinanceBench (Islam et al., ) and MMLU-Pro (Wang et al., ) for evaluation.
The FinanceBench dataset includes pairs of real-world questions about publicly traded companies, and information extracted from financial documents for answering the questions. This dataset aims to represent real-world professional use cases.
MMLU-Pro contains multiple choice questions about various domains, requiring reasoning and knowledge for answering. Each question includes 10 options, reducing the probability of guessing correctly. We use only the business and economics subsets, as they are most applicable for finance.

## 4 Evaluation and Results

### 4.1 Experiment Setup

To verify that there were no biases in the results towards a particular model, we selected three of the currently top performing small models, namely Phi-3-Small(^3^33https://huggingface.co/microsoft/Phi-3-small-128k-instruct) (Abdin et al., ), Llama-3.1-8B-Instruct(^4^44https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) (Dubey et al., ), and Mistral-7B-Instruct-v0.3(^5^55https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) (Jiang et al., ). Additionally, to further demonstrate the effectiveness of multi-task fine-tuning, we chose a top performing miniature model, Phi-3-Mini(^6^66https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (Abdin et al., ). We opted for the instruct versions of each model.

All experiments were conducted using a single machine with 2 Nvidia H100 GPUs. All experiments were done using full fine-tuning of all weights in the model. We experimented with various learning rates, ranging from $3\text{e}^{-6}$ to $3\text{e}^{-5}$. We used three epochs for the smaller runs ($\binom{n}{1},\ \binom{n}{2}$), and two epochs for the rest. The longest single fine-tuning experiment took under three hours to run.

### 4.2 Metrics

To properly interpret our results, we aggregate the experiments and present three main metrics for each model and downstream task: single-task fine-tuning (FT), multi-task fine-tuning, and baseline scores.

For single-task fine-tuning, we evaluate the model on the test split of a specific task after being trained exclusively on the training split of that task. Using the notation from Section 2.2, the single-task score for the $i$-th dataset is defined as:

$$ $\text{Single-task Score}:=\mathcal{E}_{T_{i}}(D_{i})$ (2) $$

For multi-task fine-tuning, we consider all multi-task experiments where one of the training datasets is the relevant dataset for the target task, combined with other datasets. The multi-task score is computed as:

$$ $\text{Multi-task Score}:=\max_{\mathcal{D}_{i}}\left(\mathcal{E}_{T_{i}}\left( \mathcal{M}_{\mathcal{D}_{i}}\right)\right)=\mathcal{E}_{T_{i}}\left(\mathcal{ M}_{\mathcal{D}_{i}^{*}}\right)$ (3) $$

The baseline score represents the performance of the pre-trained model on the test split of the downstream task, without any fine-tuning. It is defined as:

$$ $\text{Baseline Score}:=\mathcal{E}_{T_{i}}(\mathcal{M})$ (4) $$

Numerical Evaluations: FinQA and ConvFinQA require evaluating numerical exact match (EM) for scoring. To prevent issues stemming from rounding errors, or scale representations, we used a heuristic relaxation of exact match. We say that $x$ is numerically same to $y$ if for some small $\epsilon$, $y\pm\epsilon=x^{n},n\in\{10^{-6},10^{-3},10^{-2},10^{0},10^{2},10^{3},10^{6}\}$. While not exhaustive, these are very common scales in finance (millions vs thousands vs billions, dollars vs cents, basis points, etc.).

Classification: To evaluate classification tasks we use standard accuracy scores.

Open-End Evaluation: Unlike the other datasets, FinanceBench contains open-end question. To properly score model responses, we used LLM-as-a-Judge (Zheng et al., ) for evaluation. Specifically, we used GPT-4-o as the LLM, and use the prompt in Appendix A. We consider only a strict match as correct (i.e. a score of 2), and normalize by dividing by two.

Table: Table 2: Full experiment results for single-task and multi-task fine-tuning, aggregated across all experiments for three LLMs. Baseline results from the original models are provided for reference. The multi-task fine-tuning result represents the best performance across multi-task combinations.

Figure: Figure 3: A visualization of Table 2. The experiment results for single-task and multi-task fine-tuning, aggregated across all experiments.
Refer to caption: extracted/5894109/figs/radar_plot.png

### 4.3 Main Results

The Cocktail Effect: In Table 2, we present a comparison for the three LLMs using the metrics discussed above. A visualization of these results is provided in Fig. 3. It is clear that fine-tuning, whether single-task or multi-task, significantly improves performance compared to the baseline. Both fine-tuning approaches outperform the baseline across the vast majority of benchmarks, with this trend holding consistently across all three models.

When comparing multi-task and single-task performance, we observe a distinct advantage in favor of multi-task fine-tuning. Notably, there is a performance boost on the Headline and Twitter Sentiment Analysis tasks, which rely heavily on the model’s ability to interpret and generate stylistically appropriate responses. The clear improvements on all tasks demonstrate the cocktail effect of multi-task fine-tuning and show the robustness of this method. Appendix D contains more in depth results regarding optimal dataset interactions, showing the top combinations per task.

Phi-3-Mini: To further stress-test this concept, we shifted our focus to the smaller Phi-3-Mini model, with 3.8 billion parameters, approximately 50% smaller than the primary LLMs used in our previous experiments. We replicated the same experiments but this time compared the results with the significantly larger and state-of-the-art GPT-4-o model. The results, summarized in Fig. 1, highlight a substantial performance gap between the baseline Phi-3-Mini and GPT-4-o (with the exception of the FinNerCLS task).

However, by fine-tuning the model on the datasets mentioned above, we significantly outperformed GPT-4-o on most tasks. All classification tasks showed substantial improvements over GPT-4-o, emphasizing the effectiveness of targeted fine-tuning. Notably, a fine-tuned Phi-3-Mini model even slightly outperformed GPT-4-o on the challenging ConvFinQA benchmark. ConvFinQA involves conversations, which likely provide implicit few-shot learning opportunities, enabling the model to better understand and anticipate the structure of the questions. This contrasts with the FinQA dataset, which lacks conversational context, resulting in only modest gains for the fine-tuned model.

This experiment demonstrates that by using multi-task fine-tuning, and by specifically targeting downstream tasks, it is possible to outperform much larger and more powerful models in these tasks. The full results are presented in Appendix B.

Table: Table 3: Performance comparison for MMLU-Pro Business, MMLU-Pro Economics, and FinanceBench. For each model the best multi-task fine-tuning score is compared with the baseline.

Figure: Figure 4: Normalized averaged scores for all seven core tasks described in Section 3.1 across all experiments. Each point represents the average score for a single fine-tuned model. The colors represent the type of datasets used in the experiment.
Refer to caption: extracted/5894109/figs/violin_plot.png

Domain Generalization With the exception of Llama on FinQA, all the downstream tasks improve significantly with multi-task finetuning, across all models. Table 3 shows that this trend does not necessarily implicate that the models have improved in the general finance domain. While there may be some improvement in FinanceBench, there is no clear improvement in the other two tasks, and possibly even a regression. This finding raises a strong concern regarding the use of these downstream tasks, or many of the other commonly used benchmarks, as proxies for successful domain adaptation.

Data Regularization Hypothesis We provide a further analysis of the data by examining the effect of the two non-financial datasets: Open-Orca and Orca-Math. In Fig. 4 we present a summary of all fine-tuning experiments. We compute the average score of each fine-tuned model across the seven core tasks described in Section 3.1. For visualization purposes, we normalize the results for each model separately to be between $0.15$ and $0.85$. There is a clear distinction between models that used the non-financial datasets, and models that relied purely on the downstream tasks.

Open-Orca performs well across tasks and models. Unlike Orca-math, where strengthening mathematical reasoning abilities is directly related to model performance on tasks, it is nontrivial to interpret why adding general data would help with domain-specific downstream tasks. Moreover, it is very likely that the models were exposed to this data during pre-training, i.e., no new reasoning abilities were added.

When aligning LLMs, Ouyang et al. () adapt the loss use by Stiennon et al. (), including a regularization term: $\beta\log\left[\mathcal{M}_{\text{RL},\phi}(y|x)/\mathcal{M}_{\text{SFT}}(y|x)\right]$. This component is used to ensure the new model does not stray ’too far’ from the original model, and is missing in the standard domain adaptation regime. We hypothesize that since the pretrained model $\mathcal{M}$ has already been exposed to Open-Orca, incorporating it in finetuning serves a similar purpose. In other words, we assume:

$$ $\log\left[\mathcal{M}_{\mathcal{D}_{\text{domain}}}(y|x)/{\mathcal{M}(y|x)} \right]\geq\log\left[\mathcal{M}_{(\mathcal{D}_{\text{domain}}\cup\mathcal{D}_ {\text{gen}})}(y|x)/\mathcal{M}(y|x)\right].$ $$

We leave the exploration and research of this hypothesis to future work.

## 5 Related Work

Domain-specific LLMs: Recent advances in LLMs have led to many attempts at creating models tailored to specific domains. These models aim to outperform general-purpose ones by having deeper knowledge of the domain, being more effective at solving tasks relevant to that domain, or adopting a more appropriate style. Several methods have been suggested for training these models. One approach is to pre-train a language model entirely on domain-specific data, as seen in (Wu et al., ; Singhal et al., ). Another common approach is to take pre-trained LLMs and fine-tune them for specific downstream tasks (Xie et al., ; Wang et al., ; Cheng et al., ; Jiang et al., ; Cheng et al., ) in a domain adaptation process.

Domain Adaptation of LLMs: Various techniques have been developed to transform a general language model into a domain-specific one. One option is continual pre-training (CPT) (Gururangan et al., ), where a pre-trained LLM undergoes further training on raw data that contains relevant domain-specific knowledge, enhancing the model’s understanding of that domain. Another method involves supervised fine-tuning (SFT), where the model is trained on a large set of domain-specific instructions (Wei et al., ). Some approaches focus on specific tasks within the domain, fine-tuning the model with instruction datasets tailored to those particular tasks (Wang et al., ). Additionally, a hybrid approach has been proposed, where CPT is performed first, followed by domain-specific instruction tuning to refine the model’s capabilities (Bhatia et al., ; Wu et al., ; Xie et al., ).

Finance Benchmarks: With the increasing adoption of LLMs, several benchmarks have been proposed to evaluate model performance in the financial domain. Recently, efforts have been made to combine existing tests and datasets into more comprehensive evaluation frameworks. For instance, FinBen (Xie et al., ), PIXIU (Xie et al., ), and BBT-Fin (Lu et al., ) aggregate a variety of common tasks to provide a broad analysis of general financial skills. Other benchmarks focus on more specialized scenarios. For example, FinEval (Zhang et al., ) was developed to assess LLM financial knowledge based on academic textbooks, while SuperCLUE-Fin (Xu et al., ) aims to replicate real-world financial tasks through a detailed breakdown of subtasks. Another example is FinDABench (Liu et al., ), which places a strong emphasis on financial analysis and reasoning rather than pure knowledge evaluation.

## 6 Conclusions

In this work, we demonstrated the potential of multi-task fine-tuning as a robust approach to optimizing the performance of LLMs on downstream tasks. Through extensive experimentation involving over 200 models, we showed that combining training data from multiple related financial tasks creates a ”cocktail effect”, yielding significant performance gains, and even allowing smaller models such as Phi-3-Mini to surpass larger counterparts like GPT-4-o on targeted benchmarks. Our findings highlight the advantages of a training approach that leverages synergies between tasks.

Furthermore, our exploration of integrating general instruction-following and mathematical datasets demonstrated promising results, combining what may be a regularization effect, with an enhancement of numerical reasoning abilities. Nevertheless, we observed that while multi-task fine-tuning significantly boosts specific task performance, it does not necessarily translate into improved overall domain knowledge. This suggests that while multi-task fine-tuning is effective for task-specific improvements, broader gains in domain competency may require more sophisticated strategies.

Overall, our results provide strong empirical evidence for the benefits of multi-task fine-tuning in domain-specific model adaptation. This approach not only optimizes task performance but also underscores the importance of thoughtful dataset selection and the value of leveraging cross-task learning. Future work may benefit from exploring hybrid approaches that combine multi-task learning with targeted domain adaptation, aiming to bridge the gap between task-specific proficiency and more comprehensive domain understanding.

## Appendix A LLM as a Judge Prompt

We used the following prompt:

> <Instruction > Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. You will be given a reference answer and the assistant’s answer. Begin your evaluation by comparing the assistant’s answer with the reference answer. Identify and correct any mistakes. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 0 to 2 by strictly following this format: [[rating]], for example: The rating is: [[1]], or: My rating is [[0]]. Note! The answers have to answer the question correctly, but they do not have to be identical, or equally detailed, or equally helpful! You are only measuring equality of correctness, not completeness. Be forgiving of rounding errors, as long as they are not essential, as well as over/under explaining. You should provide a 0 rating when the answers does not match the reference. You should provide a 1 rating when the answer is partially correct. You should provide a 2 rating when the answer is correct. For example, if the reference answer is ”It cost $5B annually” and the assistant answer is ”It cost $5 billion per year”, the rating should be 2. If the assistant answer is ”It cost $5”, the rating should be 1. If the assistant answer is ”It cost $4 million per month”, the rating should be 0. For example, if the reference answer is a list of most major locations on Earth and the assistant replies concisely ’Globally’, the rating should be 2. If the assistant replies ’A variety of places worldwide’, the rating should be 1. If the assistant replies ’In Europe’, the rating should be 0. For example, if the question is ”What was his salary?” and the reference answer is ”We can see that by adding the various components in table 3, we get that 3K + 7.5K equals a total salary of 10.5K annually”, and the assistant’s answer is ”10,500”, the rating should be 2. If the assistant’s answer is ”10.5K. This salary reflects and excellent compensation given the low cost of living in the area”, the rating should still be 2. If the assistant’s answer is ”the answer can be found in table 3 by adding 3K + 7.5K”, the rating should be 1. If the assistant’s answer is ”7.5K”, the rating should be 0. </Instruction > <Question > {question} </Question > <Reference Answer > {ref_answer} </Reference Answer > <Assistant’s Answer > {answer} </Assistant’s Answer >

## Appendix B Phi-3-Mini Full Results

**Table 4: Comparison of GPT-4-o to Phi-3-Mini, including its baseline, single-task fine-tuning, and multi-task fine-tuning variants.**
|  | Phi-3-Mini | GPT-4-o |  |  |
| --- | --- | --- | --- | --- |
| Baseline | Single-task FT | Multi-task FT |  |  |
| Twitter SA | 0.65 | 0.66 | 0.91 | 0.75 |
| Twitter Topics | 0.41 | 0.87 | 0.88 | 0.65 |
| FinNerCLS | 0.71 | 0.97 | 0.98 | 0.66 |
| FPB | 0.48 | 0.13 | 0.89 | 0.80 |
| FinQA | 0.47 | 0.31 | 0.54 | 0.72 |
| ConvFinQA | 0.65 | 0.66 | 0.76 | 0.75 |
| Headline | 0.67 | 0.67 | 0.96 | 0.80 |

## Appendix C Full Results

Figure: Figure 5: Evaluation scores of all four models on all seven core tasks described in Section 3.1. The relative gain (in percentage) is reported of each fine-tuning experiment.
Refer to caption: extracted/5894109/figs/bar_chart.png

## Appendix D Ablation Study Results

Table 5, Table 6, and Table 7 present the top 3 most helpful dataset combination for Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Phi-3-Small, respectively, across each task used in our ablation study. The tables provide detailed results for each task, showing the score achieved, the difference from the maximum score, and the percentage of the maximum score. Note that since using the dataset itself trivially enhances abilities, we only include $\mathcal{D}_{i}\ \text{such that}\ D_{i}\notin\mathcal{D}_{i}$.

**Table 5: Top 3 most helpful datasets for Llama-3.1-8B-Instruct**
| Task | Datasets | Score | Diff from Max | % of Max |
| --- | --- | --- | --- | --- |
| Twitter_SA | Orca-Math, Headline, FPB, FinNerCLS, ConvFinQA, FinQA, TwitterTopics, Open-Orca | 0.8652 | 0.0419 | 95.38 |
| Twitter_SA | Headline, FPB | 0.8635 | 0.0436 | 95.20 |
| Twitter_SA | FPB, Open-Orca | 0.8425 | 0.0645 | 92.89 |
| TwitterTopics | FPB, Twitter_SA | 0.5903 | 0.2812 | 67.73 |
| TwitterTopics | FinNerCLS, Twitter_SA | 0.5834 | 0.2880 | 66.95 |
| TwitterTopics | FinQA, Twitter_SA | 0.5799 | 0.2915 | 66.54 |
| FinNerCLS | Headline, Open-Orca | 0.6912 | 0.2972 | 69.93 |
| FinNerCLS | Orca-Math, Open-Orca | 0.6851 | 0.3032 | 69.32 |
| FinNerCLS | ConvFinQA | 0.6805 | 0.3079 | 68.85 |
| FPB | FinQA, TwitterTopics | 0.8121 | 0.0775 | 91.29 |
| FPB | Headline, TwitterTopics | 0.8106 | 0.0791 | 91.11 |
| FPB | Twitter_SA, Open-Orca | 0.8079 | 0.0817 | 90.81 |
| ConvFinQA | FPB | 0.7927 | 0.0592 | 93.05 |
| ConvFinQA | TwitterTopics, Twitter_SA | 0.7672 | 0.0848 | 90.05 |
| ConvFinQA | FPB, FinQA | 0.7618 | 0.0902 | 89.42 |
| Headline | FPB, FinQA | 0.7235 | 0.2256 | 76.23 |
| Headline | FPB | 0.6917 | 0.2574 | 72.88 |
| Headline | FPB, FinNerCLS | 0.6899 | 0.2592 | 72.69 |
| FinQA | Orca-Math, FPB | 0.6507 | 0.0000 | 100.00 |
| FinQA | Orca-Math, TwitterTopics | 0.6480 | 0.0027 | 99.59 |
| FinQA | Orca-Math | 0.6418 | 0.0089 | 98.63 |

**Table 6: Top 3 most helpful datasets for Mistral-7B-Instruct-v0.3**
| Task | Datasets | Score | Diff from Max | % of Max |
| --- | --- | --- | --- | --- |
| Twitter_SA | Orca-Math, Headline, FPB, FinNerCLS, ConvFinQA, FinQA, TwitterTopics, Open-Orca | 0.8643 | 0.0486 | 94.68 |
| Twitter_SA | FPB, Open-Orca | 0.8555 | 0.0574 | 93.72 |
| Twitter_SA | TwitterTopics, Open-Orca | 0.8513 | 0.0616 | 93.26 |
| TwitterTopics | Orca-Math, Headline, FPB, FinNerCLS, ConvFinQA, FinQA, Twitter_SA, Open-Orca | 0.4873 | 0.3964 | 55.14 |
| TwitterTopics | Headline, FinQA | 0.4800 | 0.4038 | 54.31 |
| TwitterTopics | FPB, Open-Orca | 0.4753 | 0.4084 | 53.78 |
| FinNerCLS | Headline, ConvFinQA | 0.7581 | 0.2226 | 77.30 |
| FinNerCLS | Headline, FinQA | 0.7353 | 0.2454 | 74.98 |
| FinNerCLS | ConvFinQA, FinQA | 0.7327 | 0.2480 | 74.72 |
| FPB | Orca-Math, Headline, FinNerCLS, ConvFinQA, FinQA, TwitterTopics, Twitter_SA, Open-Orca | 0.8193 | 0.0660 | 92.54 |
| FPB | Orca-Math, FinQA | 0.8098 | 0.0756 | 91.46 |
| FPB | Twitter_SA, Open-Orca | 0.8092 | 0.0761 | 91.40 |
| ConvFinQA | Orca-Math, FPB | 0.6891 | 0.1258 | 84.56 |
| ConvFinQA | Orca-Math | 0.6884 | 0.1265 | 84.48 |
| ConvFinQA | Orca-Math, Headline | 0.6824 | 0.1326 | 83.73 |
| Headline | TwitterTopics, Open-Orca | 0.7377 | 0.2145 | 77.48 |
| Headline | Open-Orca | 0.7299 | 0.2223 | 76.65 |
| Headline | ConvFinQA, Open-Orca | 0.7275 | 0.2247 | 76.40 |
| FinQA | Orca-Math, FPB | 0.5609 | 0.0000 | 100.00 |
| FinQA | Orca-Math, TwitterTopics | 0.5564 | 0.0044 | 99.21 |
| FinQA | Orca-Math | 0.5538 | 0.0071 | 98.73 |

**Table 7: Top 3 most helpful datasets for Phi-3-Small**
| Task | Datasets | Score | Diff from Max | % of Max |
| --- | --- | --- | --- | --- |
| Twitter_SA | Headline, Open-Orca | 0.8677 | 0.0461 | 94.96 |
| Twitter_SA | Orca-Math, TwitterTopics | 0.8597 | 0.0540 | 94.09 |
| Twitter_SA | TwitterTopics, Open-Orca | 0.8526 | 0.0611 | 93.31 |
| TwitterTopics | Orca-Math, Headline, FPB, FinNerCLS, ConvFinQA, FinQA, Twitter_SA, Open-Orca | 0.5629 | 0.3203 | 63.74 |
| TwitterTopics | Headline, Open-Orca | 0.5449 | 0.3383 | 61.70 |
| TwitterTopics | ConvFinQA, Open-Orca | 0.5418 | 0.3414 | 61.34 |
| FinNerCLS | Orca-Math, ConvFinQA | 0.7912 | 0.1872 | 80.87 |
| FinNerCLS | ConvFinQA, Open-Orca | 0.7866 | 0.1919 | 80.39 |
| FinNerCLS | Orca-Math, FinQA | 0.7702 | 0.2082 | 78.72 |
| FPB | Orca-Math, Headline, FinNerCLS, ConvFinQA, FinQA, TwitterTopics, Twitter_SA, Open-Orca | 0.8365 | 0.0583 | 93.48 |
| FPB | Twitter_SA, Open-Orca | 0.8333 | 0.0616 | 93.12 |
| FPB | Headline, Open-Orca | 0.8189 | 0.0760 | 91.51 |
| ConvFinQA | Orca-Math, FinNerCLS | 0.7416 | 0.0680 | 91.60 |
| ConvFinQA | Orca-Math, TwitterTopics | 0.7409 | 0.0686 | 91.52 |
| ConvFinQA | Orca-Math, FPB | 0.7396 | 0.0700 | 91.35 |
| Headline | ConvFinQA, Open-Orca | 0.6956 | 0.2644 | 72.46 |
| Headline | Open-Orca | 0.6846 | 0.2754 | 71.32 |
| Headline | Orca-Math, Open-Orca | 0.6794 | 0.2806 | 70.77 |
| FinQA | Orca-Math, FinNerCLS | 0.6364 | 0.0000 | 100.00 |
| FinQA | Orca-Math, TwitterTopics | 0.6329 | 0.0036 | 99.44 |
| FinQA | Orca-Math, FPB | 0.6178 | 0.0187 | 97.07 |

## Appendix E Dataset Examples

### Dataset: Headline

Instruction:
Assess if the news headline touches on price in the past. Options: Yes, No
Input:
april gold down 20 cents to settle at $1,116.10/oz
Output:
No

### Dataset: FPB

Instruction:
You are given a financial document. Your task is to infer its sentiment. Answer using one of the following labels: [’Negative’, ’Neutral’, ’Positive’], and include nothing else. You must answer with a single word, and no additional context.
Input:
Under the terms of the agreement, Bunge will acquire Raisio’s Keiju, Makuisa and Pyszny Duet brands and manufacturing plants in Finland and Poland.
Output:
neutral

### Dataset: FinNerCLS

Instruction:
What is the entity type of ’40 William St’ in the input sentence. Options: person, location, organization
Input:
This LOAN AND SECURITY AGREEMENT dated January 27, 1999, between SILICON VALLEY BANK (”Bank”), a California-chartered bank with its principal place of business at 3003 Tasman Drive, Santa Clara, California 95054 with a loan production office located at 40 William St., Ste.
Output:
location

### Dataset: FinQA

Instruction:
Please answer the given financial question based on the context.
Input:
Interest rate to a variable interest rate based on the three-month LIBOR plus 2.05% (2.34% as of October 31, 2009). If LIBOR changes by 100 basis points, our annual interest expense would change by $3.8 million…
Question:
What is the interest expense in 2009?
Output:
3.8

### Dataset: ConvFinQA

Instruction:
Read the following texts and table with financial data from an S&P 500 earnings report carefully. Based on the question-answer history (if provided), answer the last question. The answer may require mathematical calculation based on the data provided.

Input:
Charges during the years then ended are presented below:

The fair value of restricted stock that vested during the years ended December 31, 2013, 2012, and 2011 was $1.6 million, $22.4 million, and $4.3 million, respectively.

Substantially in accordance with the original terms of the program, 50% of these LTIP units vested on December 17, 2012 (accelerated from the original January 1, 2013 vesting date), 25% vested on December 11, 2013 (accelerated from the original January 1, 2014 vesting date), and the remainder is scheduled to vest on January 1, 2015.

Question:
What was the total, in millions, capitalized to assets associated with compensation expense related to long-term compensation plans, restricted stock, and stock options in the year of 2013?
Output:
4.5

Question:
And what was it in 2012, also in millions?
Output:
4.1

### Dataset: Twitter-Topics

Instruction:
You are given a financial sentence taken from Twitter. Your task is to infer its topic. Answer using one of the following labels: [’Analyst Update’, ’Fed — Central Banks’, ’Company — Product News’, ’Treasuries — Corporate Debt’, ’Dividend’, ’Stock Commentary’, ’Stock Movement’], and include nothing else.
Input:
Here are Thursday’s biggest analyst calls: Apple, Amazon, Tesla, Palantir, DocuSign, Exxon & more…
Output:
Analyst Update

### Dataset: Twitter-SA

Instruction:
You are given a financial sentence taken from Twitter. Your task is to infer its sentiment. Answer using one of the following labels: [’Bearish’, ’Bullish’, ’Neutral’], and include nothing else.
Input:
$BYND - JPMorgan reels in expectations on Beyond Meat…
Output:
Bearish

### Dataset: Orca-Math

Instruction:
You are given a mathematical word problem. Solve it step by step and provide the answer.
Input:
You wanted to subtract 46 from a number, but you accidentally subtracted 59 and got 43. How much do you get from the correct calculation?
Output:
If you subtract 46 from the original number, you would get 56.

### Dataset: Open-Orca

Instruction:
You are an AI assistant. You will be given a task. You must generate a detailed and long answer.
Input:
Generate a fifteen-word sentence that describes all this data: Midsummer House eatType restaurant; Midsummer House food Chinese; Midsummer House priceRange moderate; Midsummer House customer rating 3 out of 5; Midsummer House near All Bar One.
Output:
Midsummer House is a moderately priced Chinese restaurant with a 3/5 customer rating, located near All Bar One.

### Dataset: FinanceBench

Instruction:
You are given a financial question and a financial document. Your task is to answer the question based on the document.
Input:
Amcor plc and Subsidiaries Consolidated Statements of Income ($ in millions, except per share data)…
Question:
Does AMCOR have an improving gross margin profile as of FY2023? If not, explain why.
Output:
No. For AMCOR, there has been a slight decline in gross margins by 0.8%.

### Dataset: MMLU-Pro Economics

Instruction:
The following are multiple choice questions (with answers) about economics. Think step by step and then finish your answer with ”the answer is (X)” where X is the correct letter choice.
Input:
Mr. Jones is president of the First National Bank of St. Louis and wishes to determine if his bank is holding too much of its demand deposits as reserves. The bank’s total deposits = $1,700,000 and the reserve ratio is 20%. If Mr. Jones finds that reserves = $850,000 what might he conclude about excess reserves?
Options:
A: ”$340,000”, B: ”$600,000”, C: ”$425,000”, D: ”25%”, E: ”10%”, F: ”$510,000”, G: ”$1,700,000”, H: ”30%”, I: ”$255,000”, J: ”15%”
Output:
F

### Dataset: MMLU-Pro Business

Instruction:
The following are multiple choice questions (with answers) about business. Think step by step and then finish your answer with ”the answer is (X)” where X is the correct letter choice.
Input:
Mr. Frankel wants to borrow $2,000 from November 16 for 143 days. The interest rate is 6%. What would the difference in the interest charge amount to if the bank used exact interest instead of bankers’ interest?
Options:
A: ”$2.00”, B: ”$0.25”, C: ”$1.50”, D: ”$1.32”, E: ”$3.30”, F: ”$0.50”, G: ”$0.99”, H: ”$0.66”, I: ”$1.98”, J: ”$2.64”
Output:
H

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: When comparing to GPT-4o, did you verify that the gains of the smaller fine-tuned models are due to improved performance or because they follow the format better? GPT-4o may provide correct responses in the wrong format, leading to lower performance when using rule-based evaluation. Using LLM judge as you did for FinanceBench, could help to alleviate potential biases of rule-based evaluation (i.e. the judge compares LLM response to ground truth).
soundness: 2
strengths: If code and all models are released, it could be a valuable resource for researchers studying topics such as data selection and model merging.
summary: This paper presents extensive empirical results comparing the performance of various LLMs fine-tuned for a specific task to the performance of these LLMs fine-tuned on various combinations of tasks. The paper suggests that there often exists a combination of tasks that can improve performance on a downstream task of interest while also outperforming frontier models such as GPT-4o.
weaknesses: - It seems unfair to compare best-of-200 models evaluated on the test set vs a single model when arguing that multi-task training is beneficial. A comparable budget should be given to the single-task fine-tuning. You could, for example, perform fine-tuning for a single task with varying number of epochs (or using early stopping), learning rate, data shuffling, etc., to have an equivalent of 200 models and then report the best test performance.
- There is very little insight into how one should select the tasks for fine-tuning (besides that general instruction tuning with Orca datasets seems to help). There are also prior works that study similar problems which would be interesting to consider:

[1] Data Selection for Language Models via Importance Resampling. Xie et. al, 2023 

[2] LESS: Selecting Influential Data for Targeted Instruction Tuning. Xia et. al, 2024

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the question proposed in the weaknesses section.
soundness: 2
strengths: The paper is well-written, featuring a clear structure and logical flow, although it potentially omits some important details, as noted in the Weaknesses section.

The evaluation metrics established by the authors for different types are reasonable and appropriate but could be more comprehensive.

The presentation of visualizations is informative and insightful.
summary: In this paper, the authors investigate the effectiveness of multi-task fine-tuning as a method to enhance the performance of large language models (LLMs) on downstream tasks. 

They claimed that they conduct extensive experiments with over 200 models and demonstrate that combining training data from multiple related financial tasks results in a "cocktail effect," which significantly boosts performance. They argue that their approach even enables smaller models, such as Phi-3-Mini, to outperform larger counterparts like GPT-4-o in specific benchmarks. The authors also state the benefits of this training strategy, emphasizing the synergistic advantages gained from leveraging inter-task relationships.
weaknesses: 1. Given the author mention multiple times in abstract, introduction and conclusion sections that they have trained 200+ models, however, there is no relevant details about it in the main body of the paper. It would be necessary to include critical information such that what is the full implemented model list and how the model selection is conducted as the author seem to eventually choose Phi-3 Mini to present their research outcomes.

2. Although the authors fine-tuned their models on multi-purpose datasets, they selected a dataset with limited variety and evaluated their models on a relatively narrow range of financial tasks, which is even fewer than some previous studies [1]. This raises concerns about the novelty and comprehensiveness of their work. It could further undermine the soundness of their training approach. Could you please elaborate on the advancements made by your research?

[1] Xie, Q., Li, D., Xiao, M., Jiang, Z., Xiang, R., Zhang, X., ... & Ananiadou, S. (2024). Open-finllms: Open multimodal large language models for financial applications. arXiv preprint arXiv:2408.11878.

3. The model comparison is based solely on various types of accuracy metrics. The paper lacks detailed statements regarding repeated experiments and does not provide error bars or significance levels for most of the experiments. Given that some accuracy results, such as those in Table 2 comparing their fine-tuned Phi-3 small with other models, are closely matched, it would be beneficial to include such evaluation metrics to ensure a more robust analysis.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. The paper presents a thorough analysis of multi-task fine-tuning for domain-specific LLMs, including an extensive experimental setup involving over 200 models.
2.  The authors provide strong empirical evidence for the benefits of multi-task fine-tuning, demonstrating significant performance gains on financial benchmarks.
3. The paper explores the use of general instruction-following and mathematical data during fine-tuning, which adds valuable insights into the regularization and numerical reasoning abilities of the models.
4. The authors use an incremental approach for fine-tuning the model, which allows them to isolate the impact of individual datasets and explore the interactions between datasets when fine-tuned together.
summary: This paper presents a detailed analysis of fine-tuning large language models (LLMs) for domain-specific tasks, with a focus on finance. The authors propose a multi-task fine-tuning approach, where models are trained on a combination of related tasks, and demonstrate its effectiveness by achieving state-of-the-art results on financial benchmarks. They compare the performance of a smaller model, Phi-3-Mini, with a larger model, GPT-4-o, and show that multi-task fine-tuning allows Phi-3-Mini to surpass GPT-4-o. The authors also explore the use of general instruction-following and mathematical data during fine-tuning and analyze the impact of different datasets on model performance.
weaknesses: 1.While the paper provides strong empirical evidence, it lacks theoretical insights into why multi-task fine-tuning is effective.
2.The paper only compares multi-task fine-tuning with single-task fine-tuning and does not consider other fine-tuning strategies such as adapter-based fine-tuning or prompt-tuning. The proposed finetuning method is similar to multi-task deep neural networks (MT-DNN), see Liu et al. 2019, ACL.