MULTI-TASK INFERENCE: Can Large Language Models Follow Multiple Instructions at Once? (2024)

Abstract: Large language models (LLMs) are typically prompted to follow a single instruction per inference call. In this work, we analyze whether LLMs also hold the capability to handle multiple instructions simultaneously, denoted as MULTI-TASK INFERENCE. For this purpose, we introduce the MTI BENCH (Multi-Task Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000 instances across 25 tasks. Each task in the MTI BENCH involves 2 to 3 sub-tasks. As expected, we first demonstrate that MULTI-TASK INFERENCE reduces the total inference time by ×1.46 times in average since it does not require multiple inference calls. Interestingly, contrary to the expectation that LLMs would perform better when tasks are divided, we find that state-ofthe-art LLMs, such as LLAMA-2-CHAT-70B and GPT-4, show up to 7.3% and 12.4% improved performance with MULTI-TASK INFERENCE compared to SINGLE-TASK INFERENCE on the MTI BENCH. We release the MTI BENCH dataset and our code at this link1.

heading: 1 Introduction
text: Large language models (LLMs) capable of following instructions have demonstrated impressive performance across a wide range of tasks (Xu et al., 2023; OpenAI, 2023; Anil et al., 2023; Tunstall et al., 2023; Wang et al., 2023a). However, since LLMs are trained to follow a single instruction per inference call, it is questionable whether they also hold the ability to follow complex instructions that necessitate handling multiple sub-tasks (Yang et al., 2018; Geva et al., 2021; Cheng et al., 2023). Moreover, current evaluation resources are either confined to measuring the LLM’s capability in following one-step instructions (Li et al., 2023; Chiang
*Corresponding Author 1https://github.com/guijinSON/
MTI-Bench
et al., 2023; Zheng et al., 2023) or only diagnose the capability to process multi-step instructions in a particular domain such as commonsense reasoning and arithmetic (Geva et al., 2021; Cobbe et al., 2021; Lightman et al., 2023).
In this paper, we analyze whether LLMs hold the capability to handle tasks composed of multiple instructions at one inference call, which we denote as MULTI-TASK INFERENCE. As shown in Figure 1, we compare the performance and speed with two baselines: (1) SINGLE-TASK INFERENCE: addressing sub-tasks sequentially and (2) BATCH PROMPTING: simultaneously processing multiple instances from the same task (Cheng et al., 2023).
For this purpose, we construct the MTI BENCH (Multi-Task Inference Benchmark), an evaluation dataset featuring 25 tasks, each consisting of 2 to 3 sub-tasks. As shown in Figure 2, the MTI BENCH is divided into two distinct subsets: (a) the MULTISTEP subset, which evaluates the models’ ability follow multiple instructions sequentially and (b) the MULTI-PART subset, focusing on the models’
5606
capability to handle multiple sub-tasks that do not have a sequential dependency. Notably, the MTI BENCH sets itself apart from previous multi-hop reasoning (Yang et al., 2018; Geva et al., 2021) and multi-turn conversation (Zheng et al., 2023) evaluation suites by providing annotations to assess the intermediate performance of LLMs while solving multi-task instructions. This enables researchers to check if LLMs reach the correct answers and evaluate whether their reasoning process is consistent and logical throughout the process.
We evaluate 11 LLMs capable of following instructions, varying in parameter size. Surprisingly, on the MTI BENCH, state-of-the-art LLMs such as LLAMA-2-CHAT-70B and GPT-4 show up to 7.3% and 12.4% better performance with MULTITASK INFERENCE compared to SINGLE-TASK INFERENCE. Moreover, MULTI-TASK INFERENCE requires x1.46 times less average inference time than SINGLE-TASK INFERENCE. These results indicate that users could obtain similar performance with substantially less time when querying instructions that necessitate solving multiple sub-tasks. Through ablation experiments, we suggest that looking at the next sub-task provides critical clues on the answer format for solving the previous subtask.
Our contributions are as follows:
• We are, to the best of our knowledge, the first to develop an evaluation benchmark, the MTI BENCH, tailored to analyze the MULTI-TASK INFERENCE capabilities of LLMs. We fully open-source our code and data.
• Our findings demonstrate that MULTI-TASK INFERENCE surprisingly works well compared to SINGLE-TASK INFERENCE only for stronger models.
• We show that MULTI-TASK INFERENCE offers x1.46 times speed-up compared to SINGLE-TASK INFERENCE. This suggests that practitioners can fully leverage the capability of LLMs to solve multiple tasks at one inference call.

heading: 2 Related Works

heading: 2.1 Language Model Evaluation
text: While Large Language Models (LLMs) demonstrate impressive performance across a wide range of tasks, it remains essential to assess their
properties and behaviors from various perspectives (Chang et al., 2023; Wang et al., 2023b; Chia et al., 2023). Traditionally, evaluations of LLMs primarily focused on performance in specific domains or tasks (Hendrycks et al., 2020; Srivastava et al., 2023). However, there is a growing interest in holistically evaluating LLMs’ properties and high-level capabilities across multiple facets (Liang et al., 2022; Holtzman et al., 2023; Kim et al., 2023b). Prior research in this area includes measuring overall helpfulness and harmlessness in user interactions (Dubois et al., 2023; Li et al., 2023; Zheng et al., 2023), assessing the ability to generate coherent thought chains in reasoning tasks (Fu et al., 2023; Ott et al., 2023), examining the presence of a theory of mind (Zhou et al., 2023; Kim et al., 2023a; Mireshghallah et al., 2023), and evaluating the capacity to avoid producing toxic content (Gehman et al., 2020). In our work, we focus on multi-processing capabilities, specifically the ability of LLMs to process multiple instructions simultaneously, as a novel and significant area to explore and evaluate across various LLMs.

heading: 2.2 Multiprocessing Capabilities of LLMs
text: The ability to concurrently process multiple pieces of information is a key indicator of intelligence (Meyer and Kieras, 1997). Previous studies have introduced datasets like HotpotQA (Yang et al., 2018) and StrategyQA (Geva et al., 2021), which require multi-hop reasoning. These are designed to train and test the LLM’s capability to follow the internal reasoning processes needed for a valid final prediction. However, these datasets do not offer a comprehensive method to assess the accuracy of intermediate steps or to compare concurrent versus sequential processing. Recently, Cheng et al. (2023) introduced BATCH PROMPTING, aligning with the research direction of our study. However, this approach is limited to examining if LLMs can process multiple instances within the same task. In contrast, our MTI BENCH encompasses a broader range of scenarios, including instructions comprising multiple sub-tasks that either follow a sequential order (MULTI-STEP subset) or solve different tasks (MULTI-PART).

heading: 3 The MTI BENCH Dataset
text: In this section, we explain how the MTI BENCH is formulated (Section 3.1), how we constructed it (Section 3.2), and provide an analysis of the diver-
sity, compositionality, and quality. (Section 3.3).

heading: 3.1 Task Formulation
text: The MTI BENCH (Multi-Task Inference Benchmark) is a comprehensive benchmark to evaluate the MULTI-TASK INFERENCE capabilities of LLMs. The benchmark comprises 25 tasks, each with 200 instances, summing up to 5,000 instances in total. Each task within the benchmark comprises 2 to 3 sub-tasks, selected from a diverse pool of 28 NLP tasks, including Classification, Multiple-Choice Question Answering (MCQA), Arithmetic, and Natural Language Inference.
These tasks are divided into two subsets: MULTISTEP and MULTI-PART containing 13 and 12 tasks respectively. Five of the 25 tasks consist of 3 sub-tasks. Table 1 presents detailed statistics for each subset.
Tasks in the MULTI-STEP subset demand a sequential approach, with the accuracy of each step being vital for the following ones. This subset assesses LMs’ proficiency in managing interdependent tasks. Conversely, the MULTI-PART subset consists of contextually related but independent sub-tasks, evaluating LLMs’ capacity to process multiple, disparate tasks simultaneously. Both subsets employ exact string matching as the evaluation method, focusing on both intermediate and final accuracy. An example instance for each subset is illustrated in Figure 2.

heading: 3.2 Dataset Construction
text: To construct the MTI BENCH, we select a wide range of tasks from existing NLP benchmarks. Our primary sources include Quoref (Dasigi et al., 2019), SNLI (Bowman et al., 2015a), MMLU (Hendrycks et al., 2020), and MATH (Hendrycks et al., 2021). Tables 13-37 provides a comprehensive list of datasets used to construct the benchmark. The key criteria for source dataset selection are (1) the presence of a rigorous quality control process in the datasets and (2) the potential to integrate the datasets into more complex tasks. The co-authors split into two groups for efficiency: one focused on combining different tasks into composite tasks, while the other screened for and eliminated any combinations that were uninformative or of low quality, subsequently categorizing the tasks into either MULTI-STEP, or MULTI-PART subsets. During the process, 7 out of the initial 32 multi-tasks were deemed unsuitable and removed, resulting in a refined final version of 25 high-quality tasks. Additionally, we crafted a one-shot demonstration for each task, which sequentially resolves the sub-tasks by generating a Chain-of-Thought (Wei et al., 2022b).

heading: 3.3 Dataset Analysis
text: Diversity The distribution of NLP tasks in their respective order within the sub-tasks is detailed in Table 2. No single task type dominates, ensuring a wide-ranging evaluation of model capabilities. There are only five multi-tasks comprised of three sub-tasks, resulting in a relatively constrained diversity for 3RD sub-task.
Compositionality To statistically verify the authors’ manual classification of multi-tasks into MULTI-STEP and MULTI-PART subset, we conduct a chi-squared test to study the interdependency within each subset. Initially, a GPT-3.5TURBO model was used to solve 200 instances of each multi-task combination. Subsequently, a chi-squared test was applied to the outcomes to assess the dependency between the accuracy of each sub-task. In Table 3, both subsets demonstrated p-values below the 0.01 threshold, refuting the null hypothesis that the sub-tasks are independent. Furthermore, the MULTI-STEP subset features chisquare statistic and odds ratio substantially higher than the MULTI-PART subset, indicating a more pronounced linear association among its tasks.
Quality To ensure the quality of the MTI BENCH, we conduct a two-step quality check. Initially, we selected a random sample of eight instructions from each task, making a total of 200 instructions for evaluation. Two of our authors labeled whether each instance showed valid dependencies between sub-tasks and were properly categorized. Tasks were recategorized and rephrased according to the results. After these adjustments, a final round of quality assessment was conducted. This phase involved ten professional annotators, including authors from our team and five externally recruited experts. The hired experts, all master’s graduates in finance, business, and computer science, were paid at the rate of $0.11 per question.
The evaluation results, presented in Table 4, indicate that after the modification process, majority of the multi-tasks in the benchmark demonstrate valid sub-task dependencies and are correctly categorized. Two annotators reviewed each question, and the Cohen’s kappa statistic (McHugh, 2012) for inter-annotator agreement on these questions scored 0.82, 0.68, and 0.89, indicating a substantial level of consensus. It was also noted that the remaining misclassifications did not reflect the overall task labeling but were somewhat isolated incidents, likely due to the specific contexts of individual samples. Importantly, even in cases with errors, no instances fail the quality assessment criteria completely, suggesting that the errors were not severe enough to affect the dataset’s reliability as a benchmarking tool.

heading: 4 Experimental Setup
text: In this section, we explain our experimental setup for investigating the MULTI-TASK INFERENCE capabilities of LLMs.
Baseline Inference Methods In addition to MULTI-TASK INFERENCE, the method in our main consideration, we compare with SINGLE-TASK INFERENCE and BATCH PROMPTING (Cheng et al., 2023). Figure 1 illustrates a scenario that compares the three inference methods. Assuming that we are testing an LLM with two instances that consist of 3 sub-tasks, the most naive approach, SINGLETASK INFERENCE prompts an LLM 6 times, where each inference call corresponds to solve a single sub-task. On the other hand, BATCH PROMPTING groups the same sub-tasks and prompts an LLM to solve multiple instances at once. Lastly, MULTITASK INFERENCE prompts the LLM to solve all
the multiple sub-tasks within a single inference call. In general, if N instances consisting of M sub-tasks are given, SINGLE-TASK INFERENCE requires N times more inference calls compared to BATCH PROMPTING and M times more inference calls compared to MULTI-TASK INFERENCE.
Test Models We evaluate eleven LLMs capable of following instructions including: (1) GPT4 (OpenAI, 2023), (2) GPT-3.5 (OpenAI, 2022), (3) TULU (7b, 13b, 30b, 65b) (Wang et al., 2023c), (4) VICUNA (7b, 13b) (Chiang et al., 2023), and (5) LLAMA-2-CHAT (7b, 13b, 70b) (Touvron et al., 2023). For GPT-4 and GPT-3.5, we utilize the 0613 version. Reported results represent the average of three runs, except for GPT-4, which were evaluated in a single run to minimize costs. Opensource models were run using fp16 precision. All evaluations were conducted in a single-shot setting, incorporating Chain-of-Thought reasoning. The hyperparameters used for evaluation are detailed in Appendix B.
Evaluation Methodology The MTI BENCH comprises 28 types of NLP tasks, yielding diverse outputs such as multiple-choice answers, numerical answers(fractional form), and extensive generative responses. Given this variety, directly applying verbalizers like LM-Eval-Harness (Gao et al., 2021) is impractical. Therefore, we prompted LLMs to return their outputs within an HTML tag (e.g., <task1>output<task1/>), which is then assessed via exact match (EM).
Hardware Specifications In Section 5.2, we examine the inference speed of four models: TULU (7b, 13b, 30b, 65b) (Wang et al., 2023c). For observation, the hardware configuration for each model size is fixed. Specifically, the TULU models with 7B and 13B parameters were tested using a single NVIDIA SXM4 with 80GB RAM. The 30B model utilized two of these NVIDIA SXM4 80GB GPU, while the largest, the 65B model, was evaluated using eight RTX A6000 with 48GB RAM each.

heading: 5 Experimental Results
text: In this section, we compare SINGLE-TASK INFERENCE, BATCHING PROMPTING and MULTI-TASK INFERENCE on the MTI BENCH (Section 5.1), study the inference latency of each method (Section 5.2) and study the efficicacy of MULTI-TASK INFERENCE on free-form generation (Section 5.3).

heading: 5.1 Main Results
text: We first evaluate SINGLE-TASK INFERENCE, BATCHING PROMPTING, and MULTI-TASK INFERENCE using the MTI BENCH. In Table 5 we focus on the final accuracy of each model, only considering the cases where it correctly solves the entire combination of sub-tasks. Surprisingly, MULTI-TASK INFERENCE consistently outperforms the other methods across various models. Notably, the performance gap between the inference strategies is larger in more powerful models. For instance, with the LLAMA-2-CHAT-70B model, accuracy under SINGLE-TASK INFERENCE and BATCHING PROMPTING is 8.7% and 7.9%, respectively, but it leaps to 16.0% using MULTI-
TASK INFERENCE. A similar trend is observed in GPT-4, where accuracy escalates from 30.8% and 32.2% to 43.2%. In Figure 3, we observe a clear upward scaling trend, which demonstrates that more advanced models exhibit enhanced performance on the MTI BENCH, irrespective of the prompting methods employed. This trend suggests that the capability to concurrently handle multi-task instructions could be an emergent property (Wei et al., 2022a), associated with the increased scale of models.
The intermediate accuracy for each prompting method is illustrated in Figure 3. Notably, MULTITASK INFERENCE, depicted in blue, consistently surpasses alternative prompting methods in both initial and final performances. Furthermore, the efficacy of BATCH PROMPTING, depicted in green, improves as the model size increases, reaching its peak with GPT-4. Despite the improvement, however, a performance gap exists with the remaining inference methods. We conjecture that the performance margin may be tied to the operational nature of BATCH PROMPTING. It combines multiple tasks without regard to their inter-dependencies, potentially introducing unrelated contexts into a single prompt. This mixing of tasks can confuse the model, as it needs to navigate through irrelevant information multiple times to address the prompt accurately. This observation aligns with existing research that the performance of batching inference improves with model scale (Cheng et al., 2023) and that the presence of non-relevant context can ad-
versely affect model performance (Shi et al., 2023). Finally, the MTI BENCH is divided into two
subsets: MULTI-STEP and MULTI-PART. As seen in Table 5, models generally perform better in the MULTI-PART subset. This suggests that inter-task dependency in multi-task instructions is a significant factor that hinders LLM performance, and the ability to manage sequential task dependencies effectively is not uniformly developed across different models.

heading: 5.2 Inference Latency
text: Considering KV caching, intuitively, a model requiring fewer inference calls would be faster in terms of inference speed, assuming it generates an equal number of tokens. Empirically, in Tables 6 and 7, we observe a 1.46× increase in speed using MULTI-TASK INFERENCE compared to SINGLETASK INFERENCE. This acceleration remains consistent as the batch size increases from 1 to 4.
Additionally, BATCH PROMPTING demonstrates a 2.1× increase in speed compared to SINGLETASK INFERENCE, aligning with the findings in (Cheng et al., 2023). However, as highlighted in Section 5.1, employing BATCH PROMPTING for the MTI BENCH results in a marked decrease in performance, making MULTI-TASK INFERENCE the most viable option.

heading: 5.3 FREE-FORM GENERATION Subset
text: As mentioned in Section 4, LLMs are prompted to return their outputs within an HTML tag, which
are parsed using regular expressions. During our evaluation, we notice that models often struggle to produce outputs in the correct format, potentially skewing their perceived performance. To address this issue, we introduce a new ablation subset called FREE-FORM GENERATION. This subset comprises 11 tasks, each divided into two sub-tasks, primarily focused on translation and summarization. Performance evaluation is conducted using the Rouge-L metric. Due to constraints in budget and time, this ablation is narrowed down to assess performance in the following methods: SINGLE-TASK INFERENCE and MULTI-TASK INFERENCE. Further details on the subset are provided in Appendix C.
Table 8 shows the result of our evaluation on the FREE-FORM GENERATION subset. We observe that smaller open-source models tend to perform better with SINGLE-TASK INFERENCE outperforming MULTI-TASK INFERENCE, with margins ranging from 0.02 to 0.15. However, this performance gap narrows for larger open-source models and proprietary models. Notably, for GPT-4, the difference in performance between the two methods is a mere 0.01, indicating that there is no significant difference in their effectiveness regardless of their output formatting.
We conjecture that the slight decrease in the performance of MULTI-TASK INFERENCE within the FREE-FORM GENERATION subset can be attributed to the weaker interdependence of the subtasks involved. For example, in task combinations such as translation and summarization, the information provided by the second instruction offers limited insights into solving the first task. This lack of inter-task informational clues may lead to a reduced level of synergy between the tasks, diminishing the benefit of MULTI-TASK INFERENCE in such scenarios.
In an effort to conduct a more comprehensive comparison between MULTI-TASK INFERENCE and SINGLE-TASK INFERENCE within free-form generation, we conducted further evaluations using the MT-BENCH (Zheng et al., 2023). A GPT4 model, with the default pairwise comparison prompt from the original paper, was leveraged to judge and select the better response. The results, depicted in Figure 4, reveal that LLMs show a slightly improved performance under MULTI-TASK INFERENCE, with an average win rate of 58% across the prompts. Remarkably, LLAMA-2-CHAT-70B and
2https://pypi.org/project/rouge-score/
GPT-4 under MULTI-TASK INFERENCE outperformed at 65.2% and 71.9% on the prompts, respectively. This shows that the benefit of MULTI-TASK INFERENCE persists beyond MTI BENCH and can be generalized to diverse use cases.

heading: 6 Analysis of MULTI-TASK INFERENCE
text: In our previous section, although it is clear that MULTI-TASK INFERENCE guarantees speed-up (as explained in Section 5.2), it is rather unexpected and surprising that larger models show improved performance on the MTI BENCH (Table 5 and Figure 3) and the MT BENCH (Figure 4) compared to SINGLE-TASK INFERENCE. For a better understanding, we conduct an ablation experiment by inserting additional input components (Section 6.1) and conduct a human evaluation to categorize what would be the reason behind the performance improvement (Section 6.2).

heading: 6.1 Ablation Experiment
text: A two-step instance within the MTI BENCH would consist of four input components when inferenced via MUTLI-TASK INFERENCE: (1) 1st Instruction, (2) 1st Context, (3) 2nd Instruction, (4) 2nd Context. On the other hand, when inferenced via SINGLETASK INFERENCE, only (1) 1st Instruction and (2) 1st Context would be provided as the input during the first inference call. Then on the second inference call, (3) the output of the first inference call, (4) 2nd Instruction, and (5) 2nd Context would be additionally provided.
In Table 9, we check the effect when adding the 2st Instruction and 2nd Context during SINGLETASK INFERENCE. Note that SINGLE-TASK INFERENCE is the same as excluding both the 2ND CONTEXT and 2ND INSTRUCTION. Interestingly, across different models and data subsets, we observe a consistent performance improvement when either the 2nd Instruction or the 2nd Context is provided as additional input when solving the 1st Instruction, indicating a sign of a look-ahead effect.

heading: 6.2 Qualitative Analysis
text: To analyze what kind of look-ahead effect might enable language models to show improved performance on the first instruction, we conduct a qualitative analysis by checking the 107 instances where GPT-4 correctly solves using MULTI-TASK INFERENCE but not with SINGLE-TASK INFERENCE. Interestingly, we discover the following four patterns that supplement the look-aheading behavior of LMs: (1) No Outputs: SINGLE-TASK INFERENCE provided no output, suggesting there were no viable answers. Conversely, MULTI-TASK
INFERENCE, while acknowledging the implausibility of all answers, still opts to select one. (2) Multiple Outputs: SINGLE-TASK INFERENCE offered multiple answers, whereas the MULTI-TASK INFERENCE approach selected the most relevant one. (3) Referencing: MULTI-TASK INFERENCE leveraged information from a subsequent task to enhance its response to the initial task. (4) Planning: MULTI-TASK INFERENCE appeared to plan its solution before addressing the task.
Patterns 1 and 2 highlight the role of MULTITASK INFERENCE in providing a form of external feedback. The existence of subsequent tasks indicates whether an answer exists, thereby eliciting a response from the model. Conversely, Patterns 3 and 4 demonstrate that MULTI-TASK INFERENCE enables LLMs to utilize their full context window. This broader context usage, which extends beyond the immediate task, allows for more comprehensive problem-solving. The frequency of each pattern from our qualitative assessment is provided in Table 10. Sample instances of the observed patterns are provided as Figure 5.

heading: 7 Conclusion
text: In this work, we present the MTI BENCH, a comprehensive benchmark consisting of 5,000 in-
stances spanning 25 diverse tasks, designed to assess the capability of LLMs in simultaneous multitasking. Our analysis within the benchmark compares MULTI-TASK INFERENCE, SINGLE-TASK INFERENCE and BATCH PROMPTING. The results indicate a superior performance by MULTI-TASK INFERENCE, despite reduced inference steps and a 1.46-fold increase in speed, demonstrating its efficiency in handling concurrent tasks.

heading: 8 Limitations
text: In this work, we try our best to offer a broad range of analyses, yet there are limitations that future studies should consider.
First, the MTI BENCH predominantly focuses on English, with the FREE FORM GENERATION ablation subset, adding French, and German. This linguistic range falls short of encompassing the wide diversity of different dialects and languages.
Second, the source dataset for MTI BENCH is largely oriented towards academic benchmarks. This focus might restrict its applicability in more general, user-oriented contexts. Future iterations should consider integrating more varied datasets to better mirror the multifaceted nature of everyday language use.
Third, another significant area concerns the automatic evaluation of model performance. Although our work employs a variety of methods such as model-based evaluation, exact matching, and Rouge-L, there is a need for additional studies on alignment with human preferences.
Fourth, the MTI BENCH only has a test set since the motivation was to test the MULTI-TASK INFERENCE capabilities of language models. Yet, it would be an interesting direction to see if the smaller models that underperformed in this work could improve their multi-processing capabilities by training on data instances with a similar format as the instances in the MTI BENCH.
Lastly, we only conducted our experiments in a one-shot setting. This was primarily because we observed that smaller models exhibit near zero accuracy when tested in a zero-shot setting and including more than two demonstrations resulted in too long input length. Yet, we acknowledge the importance of examining the impact of including additional demonstrations since models that support longer input lengths are gradually being introduced. We view this as a promising future research direction.

heading: A MTI Bench
text: Tables 13-37 provide a comprehensive overview of the 25 multi-tasks featured in the MTI BENCH. Each table includes the category, sub-tasks, and the original dataset of each multi-task. Furthermore, an example is provided to help a better understanding of the benchmark. Please note that for some examples, the context has been abbreviated for better readability.
B Inference Details
During our experiments, we use the hyperparameters as shown in Table 11.

heading: C FREE FORM GENERATION Subset
text: Apart from MULTI-STEP and MULTI-PART subsets discussed in Section 3.1, we introduce a FREEFORM GENERATION subset in our ablation studies detailed in Section 5.3. This subset follows the same creation process as the original MTI Bench, except for the hired annotators for quality assessment.

heading: D Examples for Section 6.2
text: In this section we provide sample instances for the following patterns discussed at Section 6.2: No Outputs, Multiple Outputs, Referencing, and Planning. See Figure 5.
Task ID 001
Category MULTI-STEP
Sub-Tasks Answerability Classification - Extractive Question Answering
Source Dataset QUOREF (Dasigi et al., 2019)
Example
Read the following passage, and follow the given steps, #1 Go through the provided list of questions and choose the one that is answerable given the context. Return the answer in <task1>N<task1/>format. #2 Answer the question you have chosen in step #1. Return the answer in <task2>N<task2/>format.
### Context: Passage: Big Butte Creek drains approximately 245 square miles (635 km2) of southern Oregon. [. . .] ### List of Questions: (1) What watershed is split into two geographic regions? (2) What two entities was the foundation split into in october 2016? (3) What century was Europe split into two city states and kingdoms? (4) How many years was Nashua split into two cities? (5) Who likes to divide their projects into relevent time periods and geographic regions?
Example
Read the following passage, and follow the given steps. #1 The list of sentences come after the context. Reorder them to its original order. Return the answer in <task1>(1)-(2)-(3)-(4)<task1/>format. #2 Use your answer for step#1 as a source sequence. Calculate the minimum number of swaps required to change the source sequence to (1)-(2)-(3)-(4). Swapping is only allowed between neighboring sequences. Return the answer in <task2>N<task2/>format. #3 Read the reordered text and solve the question. If the answer is "yes" multiply 2 to the answer for step#2. If the answer is "no" multiply 0.5. Return the answer in <task3>N<task3/>format.
### Context: **TL;DR My mother died this week, my girlfriend started a fight days later over my inability to talk about it, then she [. . .] ### List of sentences: (1) Or else am I justified in feeling this way? (2) I do try very hard to be self-critical, so if you all think I’m in the wrong here, need to just let this go or even apologize to her to prevent things from getting worse, then please tell me. (3) To me, she’s being unnecessarily heated and petty on an awful week for me. (4) If she decides not to come, I feel like that might be the end of our relationship. ### Question: Is my girlfriend wrong in the post ?
Example
Read the following passage, and follow the given steps, #1 Go through the provided list of questions and choose the one that is answerable given the context. Return the answer in <task1>N<task1/>format. #2 Choose the correct answer for the question you have chosen in step #1. Return the answer in <task2>N<task2/>format.
### Context: Two cats ( one is an itty bitty kitty that they bought a couple of days ago ) , [. . .] ### List of Questions: (1) What makes it possible for humans to live on other planets? (2) What may be the reason there are so many people living in a small apartment? [. . .] ### Answer Choice: (1) The individual makes pictures all the time . (2) We all split the bills and makes it easier to live . [. . .]
Table 15: Multi-Task 003 from the MTI BENCH.
Task ID 004
Category MULTI-PART
Sub-Tasks Answerability Classification - Answer & Question Matching
Source Dataset DROP (Dua et al., 2019)
Example
Read the following passage, and follow the given steps, #1 Go through the provided list of questions and choose all that is answerable given the context. Return the answer in <task1>[N, N, ..]<task1/>format. #2 From the questions selected at task#1 choose the one that best suits the given answer. Return the answer in <task2>N<task2/>format.
### Context: Passage: Until 1998, Shearer was paid $30,000 per episode. During a pay dispute in 1998, [. . .] ### List of Questions: (1) Which year was the 400,000 salary per episode cut down by 100,000? (2) How many more dollars did voice actors receive in 2008 than they negotiated for in 2004? (3) How many years after taking the throne for himself and refusing to pay tribute did a military response begin? (4) How many years after receiving a raise did Shearer take a pay cut? (5) How many students does $16,000 a year pay for? ### Answer: 3.
Example
Read the following passage, and follow the given steps, #1 Read the following list of text and determine which one contains the answer to the question. Return the answer in <task1>N<task1/>format. #2 Read the list of wrong candidates provided determine which one serves as the best wrong answer for the question. Return the answer in <task2>N<task2/>format.
### Question: What does the narrator think about the video game they were playing ? ### List of Text: (1) The walk in was quite tiring actually plus the hot scorching sun. [. . .] (2) So basically the lecture was on when to know if the guy is a nutcase or not. [. . .] (3) I almost cried when I saw the mud in the arena , it was fucking insane! [. . .]
### Wrong Candidates: (1) She wants a PC. (2) Because it stopped running Firefox . (3) They lost it at school . (4) It could be a lot better . (5) They were taking a fitness test at the gym .
Example
Read the following passage, and follow the given steps, #1 Go through the provided list of questions and choose the one that is answerable given the context. Return the answer in <task1>N<task1/>format. #2 Choose sentences from the context that is necessary to answer the question you have chosen in step #1. Return the answer in <task2>[N, N, ..]<task2/>format.
### Context: Sent 1: The film opens with Sunita , a medical student , and her friends working on a project about the human brain. Sent 2: She wants to investigate the curious case of Sanjay Singhania , a notable city businessman , who is reported to have anterograde amnesia. Sent 3: Her professor denies access to Sanjay ’s records as it is currently under criminal investigation. [. . .]
### List of Questions: (1) can a person function with half a brain (2) Sunita is working on a project about the human brain and wants to interview which person with anterograde amnesia? (3) Beyonce did an interview with which magazine and was asked about feminism? (4) What is anterograde amnesia? (5) Why is the writer working on a project?
Table 18: Multi-Task 006 from the MTI BENCH.
Task ID 007
Category MULTI-PART
Sub-Tasks Sentence Sorting - Inappropriate Question Identification
Source Dataset MULTI RC (Khashabi et al., 2018)
Example
Read the following passage, and follow the given steps, #1 The provided list of sentences come after the provided context, order the properly. Return the answer in <task1>(1)-(2)-(3)(4)<task1/>format. #2 Choose one question that cannot be answered with the context. Return the answer in <task2>N<task2/>format.
### Context: Preservation and Conservation: In 1857 the Great Western Railway Company built a main line to Scotland, [. . .] ### List of Sentences: 1: In 1974 a total reorganization of local government throughout the UK did away with the old counties of Cumberland and Westmoreland and created the larger county of Cumbria. 2: While the Lake District encourages and welcomes visitors, its popularity can damage the landscape and tax local transportation services. [. . .]
### List of Questions: 1: What 1879 event caused a group of concerned individuals to form the Lake District Defense Association? 2: What organization was a precursor to the National Trust? [. . .]
Example
Read the following passage, and follow the given steps, #1 The provided list of sentences come after the provided context, order them properly. Return the answer in <task1>(1)-(2)-(3)(4)<task1/>format. #2 Choose one question that best suits the given passage and answer. Return the answer in <task2>N<task2/>format.
### Context: New species develop naturally through the process of natural selection. [. . .] ### List of Sentences: (a): Mike lives in a cold mid-western city, where there is not much predator prey interaction. (b): He also knew that darker coats are more suitable in cold environment with less predator prey interaction. [. . .]
### List of Questions: 1. Which squirrels would most likely reproduce in greater numbers, lighter or darker? 2. Would the color be darker or lighter at point B than at point A? [. . .]
### Answer: greater.
Example
Read the following passage, and follow the given steps, #1 Choose one sentence that does not originally belong to the passage. Return the answer in <task1>N<task1/>format. #2 Reorganize the remaining sentences into its original order. Return the answer in <task2>(1)-(2)-(3)-(4)<task2/>format.
### List of Sentences: (1) My daughter jumped up and grabbed the blue one out of her hand (2) Nana chased her down, caught her, and tickled her until she laughed (3) She took off running down the hall while waving the sock in the air (4) She held up an orange sock and a blue one. (5) Nana came into the room with a puzzled look on her face. (6) She held up an orange shirt and a blue one.
Table 21: Multi-Task 009 from the MTI BENCH.
Task ID 010
Category MULTI-STEP
Sub-Tasks Coherent Passage Detection - Sentence Sorting
Source Dataset ABDUCTIVENLI (Bhagavatula et al., 2019)
Example
Read the following passage, and follow the given steps, #1 You will be given five group of sentences. Only one of them is a group of coherent sentences. The others include an injected sentence. Find the coherent passage. Return the answer in <task1>N<task1/>format. #2 Reorganize the passage you chose in step 1 into its original order. Return the answer in <task2>(1)-(2)-(3)-(4)<task2/>format.
### List of Sentences: 1. (1) Jackson now lives with the guilt of being a thief. (2) Mark kept the wallet. (3) Jackson stole a wallet at a party on Friday. 2. (1) The teacher also gave the lab partner detention for not doing anything. (2) The lab partner sat there like they knew everything. (3) The instructor announced the lab that we’re going to perform.[. . .]
Example
Read the following questions, and follow the given steps, #1 Choose one question that best suits a "CommonsenseQA" dataset. Return the answer in <task1>N<task1/>format. #2 Read the options and solve the question you chose at step#1. Return the answer in <task1>N<task1/>format.
### List of Questions: (1) What does the client think about the house? (2) Where would you put uncooked crab meat? (3) Why did the man buy dog food at the supermarket? (4) _, 52, earned about $94million in salary during his 16 seasons in the National Basketball Association. (5) Question: What is Hector Hammond’s job?
Example
Read the following text, and follow the given steps, #1 Reorder the given sentences to its original order. Return the answer in <task1>(1)-(2)-(3)-(4)<task1/>format. #2 Go through the provided list of questions and choose the one that is answerable given the context. Return the answer in <task2>N<task2/>format. #3 Solve the question you have chose from step#2. Extract the answer from the passage of step#1. Return the answer in <task3>N<task3/>format.
### List of Sentences: (1) The flowers tended to grow in a spiral pattern, to be bisexual (in plants, this means both male and female parts on the same flower), and to be dominated by the ovary (female part). (2) The most primitive flowers probably had a variable number of flower parts, often separate from (but in contact with) each other. [. . .]
### List of Questions: (1) Who’d tactic evolved? (2) When do they plant yams and millet? (3) What did some plant parts do as they evolved? (4) what became more mammal-like as they evolved? (5) What did some plant parts do when the flower had only male parts?
Table 24: Multi-Task 012 from the MTI BENCH.
Task ID 013
Category MULTI-PART
Sub-Tasks Answer & Question Matching - Wrong Candidate Ranking
Source Dataset PIQA (Bisk et al., 2020)
Example
Read the following text, and follow the given steps, #1 Choose the correct answer for the given question. Return the answer in <task1>N<task1/>format. #2 Choose the best incorrect answer for the given question. Return the answer in <task2>N<task2/>format.
### List of Answers: (1) Using a fork, stir the pecan mixture with the butter until evenly coated. Press pecan butter mixture into the bottom of your springform pan. (2) If the semolina mixture is too dry, you can add a few teaspoons of milk until it reaches the right consistency (3) Using a pie plate, stir the pecan mixture with the butter until evenly coated. Press pecan butter mixture into the bottom of your springform pan. (4) Heat up milk in the colander until it is 105 degrees, then add yeast and a pinch of sugar to the bowl of milk (5) Take some boiled milk in a small bowl and add the saffron strands to it and watch the saffron turn the milk yellow.
### Question: How do I add the pecan mixture in the pan when making creamy chocolate toffee torte?
Example
Read the following text, and follow the given steps, #1 Read through the following list of sentences and choose all sentences that are plausible and matches commonsense. Return the answer in <task1>[N, N,...]<task1/>format. #2 Count the number of inplausible sentences and express its ratio in fraction form. Return the answer in <task2>n/N<task2/>format.
### List of Sentences: (1) Natalie was embarrassed when her husband yelled at her in the store, so she told all her classmates about the experience. (2) It is better to have white wine with fish than red wine (3) Ricki was delighted to see that 2 customers came to her opening night. [. . .]
Example
Read the following text, and follow the given steps, #1 Read through the following list of sentences and choose all sentences that are incorrect reasons for the given question. Return the answer in <task1>[N, N, ..]<task1/>format. #2 Count the number correct reasons and express its ratio in fraction form. Return the answer in <task2>n/N<task2/>format. #3 Solve the following equation: (ratio_of_correct_reason) add (ratio_of_wrong_reason) Write in decimal form. Return the answer in <task3>N<task3/>format.
### Question: Sentence: Carol believed that Rebecca suspected that she had stolen the watch. Question: Why does the ’she’ refer to carol? ### List of Sentences: (1) Because If Rebecca regrets something of course she must of been the one that stole the watch. (2) Because Because rebecca wouldn’t suspect herself in a crime, she would know. (3) Because Rebecca was known to have been in an abusive relationship with Carol. [. . .]
Table 27: Multi-Task 015 from the MTI BENCH.
Task ID 016
Category MULTI-PART
Sub-Tasks Classification - Classification - Multiple Choice Question Answering
Source Dataset ARGUMENT FACET SIMILARITY CORPUS (Misra et al., 2016)
Example
Read the following text, and follow the given steps, #1 Read through the following list of texts. The topic of each text is one of the following: (1) death_penalty (2) gun_control (3) gay_marriage. Choose all text that suits the death_penalty topic. Return the answer in <task1>[N, N, ..]<task1/>format. #2 The type of each text is one of the following: (1) argument_similarity (2) argument_clarity. Out of the text you have chose in step#1 choose argument_clarity text. Return the answer in <task2>N<task2/>format. #3 Solve the question you have chose in step#2. Choose from: (1) Similar (2) Not Similar (3) Valid (4) Ivalid. Return the answer in <task3>N<task3/>format.
List of Texts: (1) Sent1: Since heterosexuals are provided the means to have a happy marriage and homosexuals are not, homosexuals are not equal to heterosexuals. Sent2: Allowing straight marriage to provide for U.S. citizenship, while gays have no option (marriage or civil union). (2) Well, if that’s a reason to ban homosexuals from marriage, then along the same line of thought, then any couple that is infertile or chooses not to have children should not be permitted to get married. (3) Sent1: The judge may or may not feel the death penaly is warranted. Sent2: Many people find some crimes heinous enough to warrent the death penalty.[. . .]
Example
Read the following text, and follow the given steps, #1 The list of sentences come after the context. Reorder them to its original order. Return the answer in <task1>(1)-(2)-(3)-(4)<task1/>format. #2 Choose the best answer for the given question. Return the answer in <task2>N<task2/>format.
### Context: I find that cats are very good about reminding you when it is time for them to eat. They will meow and often stand by their bowl. [. . .] List of Sentences: (1) So the first thing I do is head to the kitchen to see if there is an open can of her food in the refrigerator. (2) I am careful to measure her food so that she gets just a quarter cup of wet and a quarter cup of dry because I don’t want her to be overweight. (3) Next I ’ll go to my pantry and pull out a bag of her favorite dry food and mix a little of each into her food bowl. (4) Then I ’ll take the time to make sure she has plenty of water before I set her dish on the floor for her to begin eating.
### Question: What is taken from the kitchen cupboard? Options: 1: measuring cup 2: Bag of cat food.
Example
Read the following passage, and follow the given steps. #1: The list of sentences come after the context. Choose one that does not original belong to the context. Return the answer in <task1>N<task1/>format. #2: Reorder the remaining into its original order. Return the answer in <task2>(1)-(2)-(3)-(4)<task2/>format. #3: Answer the given question. Return the answer in <task3>N<task3/>format.
### Context Deepak (Shashi Kapoor) is on trial for the murder of his wealthy wife Vimla, but is acquitted and set free. [. . .]
### List of Sentences (1) Soon, Sapna learns that Gopal’s real name is Deepak, who was previously accused of murdering his first wife. (2) Gopal’s ever changing behavior throws everyone into suspicion and Sapna fears she will be his next victim. (3) Is Gopal innocent or Guilty? [. . .] ### Question Who believes that Gopal is annoying and is stalking her?
Table 30: Multi-Task 018 from the MTI BENCH.
Task ID 019
Category MULTI-PART
Sub-Tasks Natural Language Inference - Natural Language Inference
Source Dataset SNLI (Bowman et al., 2015b)
Example
Read the following text, and follow the given steps, #1 Determine the relationship. between sentences 1&2. Choose from: (1) Entailment (2) Contradiction (3) Neutral. Return the answer in <task1>N<task1/>format. #2 Choose between the given list of sentences that replaces sentence 2 and make a entailment relationship with sentence 1. Return the answer in <task2>N<task2/>format.
### Sentence 1: An older man, dressed in red, yellow, and black, is standing outside waving a large flag and a long horn. ### Sentence 2: An older man is standing outside waving to a car driving past.
### List of Sentences: (A) An older man is proudly waving a large American flag. (B) There is a man outdoors waving a flag.
Example
Read the following text, and follow the given steps, #1 Classify the given statements to one of the following categories : 1. FACE-TO-FACE, 2. GOVERNMENT, 3. LETTERS, 4. 9/11, 5. SLATE, 6. TELEPHONE, 7. TRAVEL, 8. VERBATIM, 9. OUP, 10. FICTION. Choose all that fits in category 5. Return the answer in <task1>[N, N, .. ]<task1/>format. #2 Choose a sentence that is in an entailment relationship with the statement you chose in step#1. If their are two or more answer for step#1 use the first one. Return the answer in <task2>N<task2/>format.
List of Statements: (1) yes but yes and i kind of have always pooh-poohed military educations but i think that for this kid [. . .] (2) He was pro-German, as he would have been pro-Boer. (3) Historian Thomas Reeves believes that, despite the media’s reluctance to look into Kennedy’s private life, if he had lived to have a second [. . .]
List of Sentences: 1. This kid is not very well behaved or smart. 2. I generally don’t like the idea of military educations. 3. I fully support military educations for kids.
Task ID 023
Category MULTI-STEP
Sub-Tasks Classification - Arithmetic
Source Dataset MATH (Hendrycks et al., 2021)
Example
Read the following passage, and follow the given steps. #1 Read through the given questions. Each question fall into one of the following categories. Choose a question of measurement category. Return the answer in <task1>N<task1/>format. #2 Solve the question you have chose at step#1. Return the answer in <task2>N<task2/>format.
### List of Questions: (1) What is prob of picking 1 h and 2 p when three letters picked without replacement from {h: 1, e: 3, p: 2, n: 6, q: 1}? (2) Let p = 182843/22 + -8316. Calculate the common denominator of 70/32 - (1 + -1) and p. (3) How many milliseconds are there in 38.5396 microseconds? (4) Let y(a) = -a + 5. Let m be y(3). Solve f + 16 = -0*f - 4*c, -3*c - 12 = -m*f for f. (5) Calculate (3/(-6))/(33/(-44)).
Example
Read the following passage, and follow the given steps. #1 Read through the given questions. Choose one question that is high school level. Return the answer in <task1>N<task1/>format. #2 Solve the question you have chose at step#1. Return the answer in <task2>N<task2/>format.
List of Questions: (1) A discrete graph is complete if there is an edge connecting any pair of vertices. How many edges does a complete graph with 10 vertices have? (A)10 (B)20 (C)25 (D)45 (2) When n = 11, what is the value of 10 – (n + 6)? (A)–7 (B)5 (C)7 (D)27 (3) Find the area of the first quadrant region bounded by y = x^2, y = cos(x), and the y-axis. (A)0.292 (B)0.508 (C)0.547 (D)0.667
Example
Read the following passage, and follow the given steps.
#1 Read through the given text. Choose one phrase that best suits the blank([MASK]) in the text. Return the answer in <task1>N<task1/>format. #2 Given the text choose the correct judicial decision. Return the answer in <task2>N<task2/>format.
### Context: Denial of his motions in Leo, Mr. Leo filed his second suit in the Kansas federal district court (No. 10-2495-JTM), [. . .] In a memorandum and order dated March 22, 2011, 2011 WL [MASK] [MASK] [MASK] [MASK] concluded that [. . .]
### List of Phrase: (1) contemporaneous objections. See, e.g., (2) the reality and the (3) substantially certain to result (4) 1097759, the district court (5) the City bears the
### List of Judicial Decisions (1) holding that although pro se briefs are liberally construed even pro se litigants must brief arguments in order to preserve them (2) holding that pro se pleadings are to be liberally construed [. . .]
Table 37: Multi-Task 025 from the MTI BENCH.

heading: Future Work
text: In this work, we try our best to offer a broad range of analyses, yet there are limitations that future studies should consider. First, the MTI BENCH predominantly focuses on English, with the FREE FORM GENERATION ablation subset, adding French, and German. This linguistic range falls short of encompassing the wide diversity of different dialects and languages. Second, the source dataset for MTI BENCH is largely oriented towards academic benchmarks. This focus might restrict its applicability in more general, user-oriented contexts. Future iterations should consider integrating more varied datasets to better mirror the multifaceted nature of everyday language use. Third, another significant area concerns the automatic evaluation of model performance. Although our work employs a variety of methods such as model-based evaluation, exact matching, and Rouge-L, there is a need for additional studies on alignment with human preferences. Fourth, the MTI BENCH only has a test set since the motivation was to test the MULTI-TASK INFERENCE capabilities of language models. Yet, it would be an interesting direction to see if the smaller models that underperformed in this work could improve their multi-processing capabilities by training on data instances with a similar format as the instances in the MTI BENCH. Lastly, we only conducted our experiments in a one-shot setting. This was primarily because we observed that smaller models exhibit near zero accuracy when tested in a zero-shot setting and including more than two demonstrations resulted in too long input length. Yet, we acknowledge the importance of examining the impact of including additional demonstrations since models that support longer input lengths are gradually being introduced. We view this as a promising future research direction