RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Augmenting Large Language Models (LLMs) with tools has emerged as a promising approach to further complement LLMs’ capabilities with specialised mechanisms, leading to improved accuracy and reliability Schick et al. ( 2023 ); Yao et al. ( 2023 ) .
This approach offloads tasks, fully or partially, to a deterministic offline tool such as a python interpreter (Gao et al., 2023 ) , calculator (Cobbe et al., 2021 ) , knowledge base (Borgeaud et al., 2022 ) , or online APIs of models and services Yao et al. ( 2023 ); Qin et al. ( 2023 ); Shen et al. ( 2023 ) .

This paradigm holds particular appeal in fields demanding precision, such as finance (Yang et al., 2023 ) and healthcare (Luo et al., 2022 ; Singhal et al., 2022 ) .
Specifically, the specialised terminology within the finance domain and the diverse range of data sources, encompassing both structured and unstructured data, along with the complex numerical reasoning requirements across such heterogeneous sources, render it an ideal candidate for potential improvements through tool augmentation. Nevertheless, there has been limited research dedicated to this specialised domain.

A satisfying review of existing works on tool augmentation of LLMs is beyond the scope of this work; however, this space can be divided into two primary directions: (1) approaches that require an LLM at the center and uses few-shot in-context learning to either provide tool and API documentations, or demonstrations that involve tool use Hsieh et al. ( 2023 ); Qin et al. ( 2023 ); Shen et al. ( 2023 ); Hsieh et al. ( 2023 ) , and (2) approaches that build fine-tuned smaller LMs under a static tool use protocol Schick et al. ( 2023 ) , or through expensive annotations collected from commercial LLMs Chen et al. ( 2023 ); Yao et al. ( 2023 ) .

In this work, our primary focus lies in demonstrating the potential of tool augmentation within the finance domain. Acknowledging the utmost significance of privacy concerns within the financial sector, we have chosen to adopt a fully offline approach, equipping a language model with diverse tool utilisation mechanisms. More concretely, we employ Parameter Efficient Fine-Tuning (PEFT) (Hu et al., 2022 ; Houlsby et al., 2019 ) to equip a Llama 2 13B chat (Touvron et al., 2023 ) with tool use capabilities.
Our approach differs from previous research in two significant ways. First, we do not rely on costly annotations of training examples produced by commercial language models. Second, we enhance existing question-answering training datasets by incorporating instructions and merge data representing various tasks. This approach instructs the model to adapt dynamically and determine the most appropriate mechanism (either internal or tool-based) to address each specific query.

Our model, Raven , achieves significant improvements in reasoning over structured data. For example, compared to the base model we demonstrate a lift in exact match accuracy of 63.8\% ( 21.68\%\rightarrow 85.52\% ) on the Wiki-SQL (Xu et al., 2018 ) . Despite being much smaller in size, Raven also outperforms GPT-3.5 on all benchmarks with an absolute average accuracy lift of 9.2\% .
Raven
We use the Llama 2 13B chat (Touvron et al., 2023 ) model as the backbone and fine-tune it using Low Rank Adaptation ( LoRA ) (Hu et al., 2022 ) . In this section we provide training details of Raven . The overall architecture of Raven is shown in Figure 1 .
Fine-tuning Data
We use a mixture of four financial and generic structured and unstructured question-answering datasets.
We provide a brief summary in below.

TAT-QA . Consists of questions generated by financial experts associated with
hybrid contexts drawn from real-world financial reports (Zhu et al., 2021 ) . The questions typically require a range of data extraction and numerical reasoning skills, including multiplication, comparison, sorting, and their various combinations. Apart from the answer, TAT-QA also provides the derivation, where applicable, which proves beneficial for offloading the calculation to an external tool, as will be explained in § 2.2 .

Financial PhraseBank . Consists of phrases derived from English news on listed companies in OMX Helsinki (Malo et al., 2014 ) . The dataset contains phrase-level annotation by financial markets experts, that categorise each sample sentence as either positive, negative, or neutral, from an investor’s standpoint. This dataset is relevant because sentiment analysis models trained on general datasets do not perform well in specialised domains due to the unique vocabulary found in financial texts, which often do not rely on easily identifiable positive or negative words (Araci, 2019 ) .

Wiki-SQL . Consists of manually annotated crowd sourced examples of natural language questions and SQL queries over tables found on Wikipedia (Zhong et al., 2017 ) . Whilst this is not specifically a financial domain dataset its relevancy is in the availability of the script that produces the answer. Similar to the derivation in the TAT-QA dataset this script is crucial to steer our model to use a tool instead of producing the answer directly.

OTT-QA . Similar to TAT-QA , this dataset consists of questions over tabular data and unstructured text across diverse domains (Chen et al., 2021 ) . The majority of questions necessitate multi-hop inference involving both forms of data. The dataset’s relevance lies in its omission of derivation or intermediate steps, which poses a challenge for the model to infer the correct answer.

Data splits. Among the four datasets, FPB 2 2 2 https://github.com/vrunm/Text-Classification-Financial-Phrase-Bank and OTT-QA 3 3 3 https://github.com/wenhuchen/OTT-QA lack a published test split. TAT-QA 4 4 4 https://nextplusplus.github.io/TAT-QA/ has a test split without gold labels. WikiSQL 5 5 5 https://github.com/salesforce/WikiSQL provides a public test set. We used the WikiSQL test split, and for the other 3 datasets generated random 80-10-10 splits (available here ). Table 1 summarises the statistics of the datasets.
Tools
Raven is equipped with two external offline tools: a calculator and a SQL engine. The Calculator is instantiated in a python interpreter and is used to evaluate well-formed arithmetic expressions. The API expects one input representing the arithmetic expression and returns the evaluated result. The Lightweight SQL engine is an API capable of executing SQL scripts on relational data. The API expects two inputs, (1) a string representation of the structured data and (2) a SQL script. The API’s lightweight database engine converts structured data from its textual form to the engine’s relational representation and converts data types where applicable. The SQL script is executed on this representation and the API returns the result.
Instruction Tuning
Inspired by Wang et al. ( 2023 ) and Taori et al. ( 2023 ) we engineer various templates for SFT instruction tuning.
In general, we require to extract up to four key attributes from the original datasets. These are (1) instruction that describes the task to perform, for example, " Determine the sentiment of the following phrase ", or the question " What is the percentage change in revenue after the adoption of ASC 606? " (2) input that provides more context such as the phrase to classify or a passage, (3) data that accompanies the context in tabular format, (4) derivation that produces the answer or expected response . The instruction and one of derivation or response are mandatory, whilst the other attributes are included if applicable.

To ensure training diversity, our model is trained on a combination of all available training data. Based on the data, we craft different templates depending on which tool the model should choose or if the model should directly answer the question on its own (i.e., to train the Task Solver in Figure 1 ). We also automatically generate another dataset, that supplements the above question-answer dataset for training our model to select the appropriate template based on the context (i.e., to train the Task Router in Figure 1 ). Refer to appendix C for template examples.
Inference
During inference, we follow a two-step process with Raven . First, we employ a specialised template choice prompt to determine the most suitable prompt template (from "arithmetic," "classification," "script," or "information extraction") based on the input. Next, we wrap the instruction, including the input and relevant data, in the inferred prompt template and send it to Raven for generating the subsequent output. Depending on the selected template, the Task Solver either activates a tool to fulfil the request or directly produces the response.

We discuss the inference behaviour when each of these templates are used. For Script the model is expected to produce a well-structured SQL script. In this scenario, the structured data table provided in the prompt is temporarily loaded in memory using a lightweight database engine, and the script execution on the table produces the output. For Arithmetic the model is expected to predict a well formed arithmetic expression. This expression is evaluated by a calculator and the resulting value passed as output. The Information Extraction template instructs the model that there is information included in structured form that needs to be considered before producing the answer. In this case no tool is used and the model is expected to infer the correct output based solely on the information in the prompt. The Classification template is used when the prediction of the model should be taken as-is.
Experiments
We compare with the base Llama 2 13B Chat with and without SFT 6 6 6 To steer the base model into producing a short answer we add ”Output the answer only with no explanation.” to the prompt. . We also report GPT-3.5 7 7 7 gpt-3.5-turbo (5-shot), GPT-3.5 (Chain-of-Thought Wei et al. ( 2022 ) ) and GPT-3.5 (5-shot + Tools).
The SFT model trained with tool use is denoted as Raven . When tool use fails due to ill-formed arguments we have a fallback mechanism to produce the answer by the SFT model, denoted as Backoff . For training details and hardware, see Appendix B . We evaluate the models using exact match . The task router has determined the correct type 100% of the time, except for TAT-QA where the accuracy was 90.62%.
Main Results
The results are summarised in Table 1 .
Compared to the base model, Raven significantly improves the results on the PhraseBank dataset by an absolute 25.9%.
On the Wiki-SQL dataset the base model is unable to infer the correct answer almost 80% of the time. This figure is inverted
for Raven which obtains a 4-fold improvement over the base model inferring the correct answer more than 85% of the time. Our model improves on the best GPT-3.5 performance by close to 10% (absolute). All the questions in this dataset can be addressed using the lightweight database engine and involve a combination of data selection, ranking and arithmetic operations on structured data. This result underscores the distinct advantage of delegating this task to a tool rather than relying on the language model to infer the results in a zero-shot manner. Despite the results not being as strong as Raven we observe a similar pattern on the GPT-3.5 evaluation in which better results are incrementally obtained when including examples in the context and using tools compared to CoT .

We see a similar pattern on the TAT-QA benchmark with the tool augmented model achieving a 5-fold improvement on the base model. Approximately 46% of the observations of the TAT-QA dataset are annotated with an intermediate arithmetic derivation that Raven evaluates using a calculator at inference time. We perform a comparative analysis to explore whether our model performs better on this portion of the data in the analysis section (§ 3.2 ).

In OTT-QA , the majority of questions require multi-hop inference involving both tabular data and unstructured text, with the information needed to answer the questions dispersed across these two input types.
This dataset does not have annotated intermediate steps to get to the answer and therefore all models are expected to infer the answer without relying on tools. Despite SFT achieving an increase in accuracy compared to the base model, the relatively low score underscores the importance of intermediate reasoning steps and tools Chen et al. ( 2023 ) .

We observed the Backoff mechanism to bring slight improvement on TAT-QA ( 51.35\%\rightarrow 52.27\% ) and Wiki-SQL ( 84.25\%\rightarrow 85.52\% ).
Analysis
Is it better to have a separate model for each task? We developed a model specifically using the TAT-QA dataset, achieving an evaluation score of 54.70%. This dedicated model outperforms Raven by 2.4%. We contend that this modest performance gain does not warrant the added complexity of maintaining separate models and switching between them during inference.

Why tool augmentation is necessary? Approximately half of the questions within the TAT-QA dataset are annotated with an arithmetic equation. The presence of the equation implies that the language model needs to perform multiple actions to output the correct answer. This process involves the correct extraction of, at a minimum, two numerical values from the context, followed by the execution of an arithmetic operation, such as addition or division. This particular scenario is ideal to understand the effect of SFT and tool augmentation by comparing the performance of different models on the two categories of data from the same dataset. As shown in Figure 2 the base model without any fine-tuning is ill-equipped to perform multi-hop reasoning achieving close to 2% accuracy equating to ten correct answers of approximately 620. Although we observe an improvement in the SFT model, the impact of using tools is evident in the substantial jump to 56.7% accuracy achieved by Raven . These findings further confirm that SFT models are able to accurately extract multiple data points from the context but require external tools to correctly compose the final answer from the gathered data. This is also evidenced by the consistent performance of the Information Extraction type questions between SFT and Raven , which only requires data extraction to answer the question.

The utility of augmenting language models with external tools is substantiated further through a comparative analysis of experimental outcomes on two similar datasets. Addressing questions on Wiki-SQL and OTT-QA requires multi-hop reasoning across diverse forms of data, spanning both structured and unstructured formats. The primary difference lies in the annotation method: the Wiki-SQL dataset is annotated with a data extraction script which, when executed on the structured data, yields the answer. In contrast, the OTT-QA dataset lacks this intermediate derivation step. By delegating the script execution to an external tool, Raven achieves an exact match accuracy of 85.52% on Wiki-SQL and 16.03% on OTT-QA , underscoring the effectiveness of fit-for-purpose external tools in this scenario.

What is the impact of question complexity? On the TAT-QA dataset we can use the number of arithmetic operators in the gold arithmetic equation as a proxy for question complexity. One arithmetic operator implies the extraction of two numerical values from the context, two operators, three numerical values, and so on. As shown in Figure 2 , Raven ’s performance degrades with the number of numerical values to extract from the context.
Conclusion
In this paper we have demonstrated the feasibility of equipping a Llama 2 13B chat model with tool use capabilities via fine-tuning a mere 0.2% of its parameters on a relatively small and diverse dataset. The augmentation with tools remarkably elevated the performance of the base model by an average of 35.2% across 4 datasets, surpassing even a significantly larger GPT-3.5 model by 9.2%. Additionally, through a comparative analysis
of question answering datasets we demonstrate the effectiveness of augmenting language models with external tools, showing significant improvements in accuracy when addressing multi-hop questions with tools.
Limitations
Infrastructure Bottleneck . Our experiments were constrained with fitting our model on available commodity hardware. We hypothesise that it would be possible to obtain better performance using the larger Llama 2 70 billion-parameter model and a longer context length. Experiments by Touvron et al. ( 2023 ) demonstrated that the 70-billion-parameter model consistently achieves the highest performance across various prominent natural language understanding benchmarks. Additionally, a longer context length enables experimentation with diverse prompts as well as alternative representations of structured data.

Language model evaluation .
Free-form natural language generation (NLG) poses significant evaluation challenges that remain under-studied to this date (Liu et al., 2023 ) . Zheng et al. ( 2023 ) argue that while users prefer the responses of an instruction-tuned model over the base model, traditional LLM benchmarks (Liang et al., 2022 ; Hendrycks et al., 2021 ) cannot tell the difference. This challenge is heightened in specialised domains such as finance. Common similarity scores such as BLEU (Papineni et al., 2002 ) which measures n-gram overlap between candidate and reference sentences are unsuitable due to misleading accuracy or penalised semantic correctness (Freitag et al., 2022 ) . Although BERTScore (Zhang et al., 2020 ) addresses some of these pitfalls by measuring the similarity of candidate and reference sentences using pre-trained contextualised embeddings it can still produce high scores for inaccurate results. For example the candidate and reference sentences "The amount of goodwill reallocated to the IOTG operating segment in 2018 was $480 million " , and "The amount of goodwill reallocated to the IOTG operating segment in 2018 was $480" have a BERTScore (f1) of 99.17%! These measures are not suitable for comparing numerical content.

Conversely, using exact match criteria might unjustly penalise NLG models, given that identical numerical values can be expressed in varying forms - such as "$4 million" and "$4,000,000," or "0.24" and "24%,". In some cases, numerical values can be integrated within a passage of text, rendering the evaluation of such content very challenging. In our evaluation we have normalised different formatting (such as converting values to percentages where appropriate), however a universal normalising algorithm in this space is outside the scope of our research.

GPT-3.5 evaluation . Evaluating our benchmark with GPT-3.5 poses significant challenges, especially when using Zero-Shot (CoT) (Kojima et al., 2022 ) . GPT-3.5 does not consistently adhere to instructions for providing a concise response, such as a single word or number, which makes exact match comparisons challenging. Additionally, we have noticed that GPT-3.5 does not generate a response when uncertain. This is particularly evident when evaluating the FPB, which does not exhibit common sentiment negative or positive words.
Ethics Statement
Our work is built on top of existing pre-trained language models. Our goal was not to attend to alleviate the well-documented is- sues (e.g., privacy, undesired biases, etc) that such models embody. For this reason, we share the similar potential risks and concerns posed by these models. Additionally, our SFT was conducted on publicly available research benchmarks, and as such the additional SFT step used in Raven is unlikely to introduce any new area of risk.
C.2 PhraseBank
Example 1 Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Determine the sentiment of the following. ### Input: The plant will be fired with a combination of spruce bark, chipped logging residues or milled peat. ### Response: neutral Example 2 Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Determine the sentiment of the following. ### Input: Operating profit improved by 27% to EUR 579.8mn from EUR 457.2mn in 2006. ### Response: positive
C.3 Wiki-SQL
Example 1 Below is an instruction that describes a task, coupled with contextual data. Compose an SQL script capable of being run on the data to generate the solution. ### Instruction: How many people watched at Glenferrie Oval? ### Data: {"header": ["Home team", "Home team score", "Away team", "Away team score", "Venue", "Crowd", "Date"], "rows": [["North Melbourne", "12.10 (82)", "South Melbourne", "11.14 (80)", "Arden Street Oval", "6,000", "4 August 1928"], ["Fitzroy", "13.12 (90)", "Footscray", "12.17 (89)", "Brunswick Street Oval", "12,000", "4 August 1928"], ["Richmond", "11.13 (79)", "Melbourne", "7.8 (50)", "Punt Road Oval", "26,000", "4 August 1928"], ["Geelong", "4.14 (38)", "Essendon", "12.10 (82)", "Corio Oval", "10,000", "4 August 1928"], ["Hawthorn", "9.9 (63)", "Collingwood", "17.18 (120)", "Glenferrie Oval", "5,000", "4 August 1928"], ["St Kilda", "13.15 (93)", "Carlton", "10.9 (69)", "Junction Oval", "31,000", "4 August 1928"]], "types": ["text", "text", "text", "text", "text", "real", "text"], "caption": "Round 15"} ### SQL: SELECT SUM([Crowd]) FROM data_table WHERE LOWER([Venue]) = LOWER(’glenferrie oval’)’
C.4 OTT-QA
Example 1 Here is a instruction detailing a task, accompanied by data providing additional context. Provide a suitable reply that effectively fulfills the inquiry. ### Instruction: How many kilometers is the airport from the Australian city known for housing the Towsers Huts? ### Data: {"header": ["Community", "Airport name", "Type", "ICAO", "IATA"], "rows": [["Albury", "Albury Airport", "Public", "YMAY", "ABX"], ["Armidale", "Armidale Airport", "Public", "YARM", "ARM"], ["Ballina", "Ballina Byron Gateway Airport", "Public", "YBNA", "BNK"], ["Balranald", "Balranald Airport", "Public", "YBRN", "BZD"], ["Bankstown , Sydney", "Bankstown Airport", "Airschool", "YSBK", "BWU"], ["Bathurst", "Bathurst Airport", "Public", "YBTH", "BHS"], ["Bourke", "Bourke Airport", "Public", "YBKE", "BRK"], ["Brewarrina", "Brewarrina Airport", "Public", "YBRW", "BWQ"], ["Broken Hill", "Broken Hill Airport", "Public", "YBHI", "BHQ"], ["Camden", "Camden Airport", "Public", "YSCN", "CDU"], ["Cessnock", "Cessnock Airport", "Public", "YCNK", "CES"], ["Cobar", "Cobar Airport", "Public", "YCBA", "CAZ"], ["Coffs Harbour", "Coffs Harbour Airport", "Public", "YCFS", "CFS"], ["Collarenebri", "Collarenebri Airport", "Public", "YCBR", "CRB"], ["Condobolin", "Condobolin Airport", "Public", "YCDO", "CBX"], ["Coolah", "Coolah Airport", "Public", "YCAH", ""], ["Cooma", "Cooma - Polo Flat Airport", "Public", "YPFT", ""], ["Cooma", "Cooma - Snowy Mountains Airport", "Public", "YCOM", "OOM"], ["Coonabarabran", "Coonabarabran Airport", "Public", "YCBB", "COJ"], ["Coonamble", "Coonamble Airport", "Public", "YCNM", "CNB"]], "caption": "List of airports in New South Wales"} ### Response: 3