RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
C.5 Template choice
Example 1 - Arithmetic Template Here is a instruction, input and data detailing a task. Which template is best suited to fulfil this inquiry. ### Instruction: What was the % change in gains recognized in other comprehensive income (loss), net of tax of $1, $11, and $4 from 2018 to 2019? ### Input: Cash Flow Hedge Gains (Losses) We recognized the following gains (losses) on foreign exchange contracts designated as cash flow hedges: We do not have any net derivative gains included in AOCI as of June 30, 2019 that will be reclassified into earnings within the following 12 months. No significant amounts of gains (losses) were reclassified from AOCI into earnings as a result of forecasted transactions that failed to occur during fiscal year 2019. ### Data: {"header": ["(In millions)", "", "", ""], "rows": [["Year Ended June 30,", "2019", "2018", "2017"], ["Effective Portion", "", "", ""], ["Gains recognized in other comprehensive income (loss), net of tax of $1, $11, and $4", "$ 159", "$ 219", "$ 328"], ["Gains reclassified from accumulated other comprehensive income (loss) into revenue", "341", "185", "555"], ["Amount Excluded from Effectiveness Assessment and Ineffective Portion", "", "", ""], ["Losses recognized in other income (expense), net", "(64)", "(255)", "(389)"]]} ### Template: arithmetic Example 2 - Script Template Here is a instruction and data detailing a task. Which template is best suited to fulfil this inquiry. ### Instruction: In what division was there a population density in km2 of 4,491.8 in 2011? ### Data: {"header": ["Administrative division", "Area (km) 2011**", "Population 2001 Census (Adjusted)", "Population 2011 Census (Adjusted)", "Population density (/km 2011)"], "rows": [["Dhaka District", "1,463.6", 9036647, 12517361, "8,552.4"], ["=> Savar Upazila", "282.11", 629695, 1442885, "5,114.6"], ["=> Keraniganj Upazila", "166.82", 649373, 824538, "4,942.68"], ["Narayanganj District", "684.37", 2300514, 3074078, "4,491.8"], ["=> Narayanganj Sadar Upazila", "100.74", 946205, 1381796, "13,716.5"], ["=> Bandar Upazila", "54.39", 267021, 327149, "6,014.8"], ["=> Rupganj Upazila", "176.48", 423135, 558192, "3,162.9"], ["Gazipur District", "1,806.36", 2143200, 3548115, "1,964.2"], ["=> Gazipur Sadar Upazila", "457.67", 925454, 1899575, "4,150.5"], ["=> Kaliakair Upazila", "314.13", 278967, 503976, "1,604.3"], ["Narsingdi District", "1,150.14", 1983499, 2314899, "2,012.7"], ["=> Narsingdi Sadar Upazila", "213.43", 606474, 737362, "3,454.8"], ["=> Palash Upazila", "94.43", 198106, 221979, "2,350.7"]], "types": ["text", "text", "real", "real", "text"]} ### Template: script
Figure 1: Raven Inference Flow . Using the language model the Task Router infers the optimal task format to use conditioned on the given prompt. The Task Solver re-formats the instruction according to the selected template by the task router and sends it to the language model again. The pipeline will branch between serving the response directly or calling a tool API to perform an intermediate evaluation before serving the final output.; Raven Inference Flow . Using the language model the Task Router infers the optimal task format to use conditioned on the given prompt. The Task Solver re-formats the instruction according to the selected template by the task router and sends it to the language model again. The pipeline will branch between serving the response directly or calling a tool API to perform an intermediate evaluation before serving the final output.
Figure 2: Comparison of model performance on the TAT-QA dataset specifically highlighting the effect of a tools-augmented model on questions that require multi-hop reasoning.; Comparison of model performance on the TAT-QA dataset specifically highlighting the effect of a tools-augmented model on questions that require multi-hop reasoning.
Figure 3: Comparison of model performance on the TAT-QA dataset highlighting the effect of complexity on model performance.; Comparison of model performance on the TAT-QA dataset highlighting the effect of complexity on model performance.
The data statistics and experimental results (Exact Match) of various benchmarks and models. The best results are in bold . GPT-3.5 results are based on 5-shots. Sota is based on previously published results.