Benchmarks and Evaluation
Part 2 of the Tutorial on Complex Reasoning in Natural Language
https://wenting-zhao.github.io/complex-reasoning-tutorial/

Live Q&AEvaluation of complex reasoning is hard!
●

Many types of reasoning skills●

Many ways in which skills can be combined●

Many possible settings and tasks in which skills can be applied●

Hard to deﬁne settings that evaluate complex reasoning processes
in a reliable and informative way.

But there are exciting tasks that even the best models today often fail on!
2Plan1.

Core reasoning skills

2.

Multimodal reasoning

3.

Reasoning with background knowledge

4.

Designing evaluations for complex reasoning

3Core reasoning skills

4Compositional (multi-hop) reasoning

Who succeeded the first
President of Namibia?

Who was the first
President of Namibia?

Hifikepunye
Pohamba

Who succeeded SamNujoma?

⚠ GPT3 davinci-002 with self-ask prompting

and an external search tool obtains 15.2 EMon MuSiQue 2-hop questions.
5MuSiQue dataset, Trivedi et al., TACL 2022.

Arithmetic reasoning
A carnival snack booth made $50 selling popcorn eachday. It made three times as much selling cotton candy.
For a 5-day activity, the booth has to pay $30 rent and$75 for the cost of the ingredients.
How much did the booth earn for 5 days after
paying the rent and the cost of ingredients?

$895

How much did the booth make selling cotton candy each day? ($50 x 3 = $150)
How much did the booth make in a day? ($150 + $50 = $200)
How much did the booth make in 5 days? ($200 x 5 = $1000)
How much did the booth have to pay? ($30 + $75 = $105)
How much did the booth earn after paying the rent and the cost of ingredients? ($1000 - $105 = $895)

6

GSM8K dataset, Cobbe et al., 2021.

Social reasoning
Alex spilled the food she just prepared all overthe floor and it made a huge mess.
What will Alex want to do next?
(a) taste the food(b) mop up(c) run around in the mess(b) mop up

Social IQA dataset, Sap et al., EMNLP 2019.

7

Temporal reasoning⚠ Gopher in a 5-shot setting obtains

50.9 accuracy on TIMEDIAL
compared to 97.8 human performance.

A: Yes, sir. May I help you?
B: Please I’d like a ticket to New York.
A: For today?
B: No, early Saturday morning .
A: We have a flight that we’ll put you there at ________ . Is that ok?
B: Nothing earlier? I prefer flight at 9 thirty.
A: I’m afraid not , unless you want a night flight.
(a) ten AM
B: No, exactly not.
(b) 9:30 PM(c) eleven AM(d) four AM

(a)
(b)
(c)
(d)ten AM
9:30 PMeleven AMfour AM

8TIMEDIAL dataset, Qin et al., ACL 2021.

Multimodal reasoning

9Who is taller, LeBron James or Stephen Curry?
In his sophomore season in 2007–08, Curry had grown to his
adult height of 6 ft 3 in (1.91 m) and again led the Southern
Conference in scoring, averaging 25.5 points per game while
adding 4.7 rebounds per game and 2.8 assists per game.
Lebron James is considered one of the greatest basketball playersof all time, and his height has been a huge part of his success. At
16 years old, he was 6’7″ and weighed an impressive 240 pounds.
As he grew older and entered the NBA, his height increased to 6’8″and his weight increased to 250 pounds. Currently, at the age of35, Lebron stands at 6’9″ and weighs 250 pounds. His height,
combined with his athleticism and skill, has allowed him todominate the sport of basketball for manyyears.
Photo from Sports Illustrated
From: craftednba.com
10⚠ Best methods reach~66 EM on HybridQA.

11HybridQA dataset, Chen et al., Findings of EMNLP, 2020.

12

TabFact dataset, Chen et al., ICLR, 2020.

13

MultiModalQA dataset, Talmor et al., ICLR, 2021.

Reasoning with background knowledge

14Wilhelm Müller was born on 7 October 1794 in Dessau, the son of atailor. In 1813-1814 he took part, as a volunteer in the Prussianarmy, in the national rising against Napoleon. He participated in thebattles of Lützen, Bautzen, Hanau and Kulm. In 1814 he returned
to his studies at Berlin. Müller's son, Friedrich Max Müller, was an
English orientalist who founded the comparative study of religions.
Battle of Kulm
The French lost more than halfof the pursuing force of 34,000;
The allies lost approximately
13,000 soldiers.

Which battle Wilhelm Müller fought in while inthe Prussian army had the most casualties?
Kulm
Battle of Lützen (1813)
Napoleon lost 19,655
men, while the Prussians
lost 8,500 men and theRussians lost 3,500 men

Battle of Bautzen
Losses on bothsides totaled
around 20,000.

Battle of Hanau
Overall, 4,500 Frenchsoldiers and 9,000
allied soldiers werelost in the battle.
15IIRC. Ferguson et al., EMNLP, 2020.

Implicit reasoning
Was Aristotle alive when the laptop was invented?

Did Aristotle use a laptop?

explicit
1.
2.
3.

implicit
When did Aristotle live?
When was the laptop invented?
Is #2 before #1?

No16

StrategyQA dataset, Geva et al., TACL, 2021.

More implicit reasoning examples

Would a pear sink in water?

The density of a raw pear is about 0.59 g/cm^3.
The density of water is about 1 g/cm^3.
Objects only sink if they are denser than the surrounding fluid.

No

Would a monocle beappropriate for a cyclop?

Cyclops have one eye.
A monocle helps one eye at a time.

Yes⚠ GPT3 and PaLM 540B obtain~77 accuracy on the StrategyQA dataset.
17

StrategyQA dataset, Geva et al., TACL, 2021.

Retrieval + commonsense reasoning over entity knowledge

18CREAK dataset, Onoe et al., NeurIPS, 2021.

⚠ Best models on the CSQA2leaderboard obtain ~78 accuracy.

19

CommonsenseQA2 dataset, Talmor et al., NeurIPS, 2021.

Designing evaluations forcomplex reasoning

20

Reasoning “shortcuts”
The Oberoi family is an Indian family that is famous for its
involvement in hotels, namely through The Oberoi Group.

The Oberoi Group is a hotel company with its head office inDelhi. Founded in 1934, the company owns and/or operates
30+ luxury hotels and two river …

The Oberoi family is part of a hotel company
that has a head office in what city?

Delhi
21

Chen and Durrett, NAACL, 2019.

Various forms of bias in ourcrowdsourcing practices…

22

Instructions given to crowdworkers often include biases
Generate 5 data instances for “Temporal reasoning”.
The question has to require “temporal” understanding regarding some events inthe sentence. In this task, we ask you to create ﬁve different questions for ﬁndingout Event duration. Event Duration for events is the understanding of duration ofevent such as “brushing teeth”, usually takes few minutes.
Examples:- How long did Jack play basketball?
- How long has the software been having this issue?
- How long is a drive from Toronto to New York City?

23

Parmar et al. EACL, 2023.

Instruction bias limits the coverage of the “real” task
Dominant pattern in the Quoref dataset:

Instruction Examples

What is the first name …

Training Set

Evaluation Set

24

Parmar et al. EACL, 2023.

What can we do about it?
●Improve our data collection protocols – instructions should be diverse,
measure correlation between model performance and input patterns,model-in-the-loop, etc.

●Build and evaluate on contrast sets / counterfactuals●Evaluating the reasoning process itself●More “realistic” setups – combining reasoning skills, testing reasoning
as part of “real” tasks.

25

Next: Knowledge-Augmented Approaches after Pretraining

Live Q&A26

References

27

Core reasoning skills[1]

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio,William Cohen, Ruslan Salakhutdinov, Christopher D. Manning. EMNLP, 2018.

[2]

MuSiQue: Multihop Questions via Single-hop Question Composition. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot,
Ashish Sabharwal. TACL, 2022.

[3]Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. Xanh Ho, Anh-Khoa Duong Nguyen, Saku
Sugawara, Akiko Aizawa. COLIN, 2020.

[4]

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. Tao Yu,Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang,Dragomir Radev. EMNLP, 2018.

[5]

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. Dheeru Dua, Yizhong Wang,
Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner. NAACL, 2019.

[6]

Training Veriﬁers to Solve Math Word Problems. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun,Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman. 2021.

[7]

Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning. Pradeep Dasigi, Nelson F. Liu, Ana
Marasović, Noah A. Smith, Matt Gardner. EMNLP, 2019.

28

Core reasoning skills[8]

How much coffee was consumed during EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI. Ashwin Kalyan,Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, Peter Clark. EMNLP, 2019.

[9]

TIMEDIAL: Temporal Commonsense Reasoning in Dialog. Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi,Manaal Faruqui. ACL, 2021.

[10]Social IQa: Commonsense Reasoning about Social Interactions. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, Yejin
Choi. EMNLP, 2019.

[11]

QASC: A Dataset for Question Answering via Sentence Composition. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen,Ashish Sabharwal. AAAI, 2020.

[12]

TempQuestions: A Benchmark for Temporal Question Answering. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik
Strötgen, Gerhard Weikum. The Web Conference, 2018.

29Multimodal reasoning[13]

HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data. Wenhu Chen, Hanwen Zha, Zhiyu Chen,Wenhan Xiong, Hong Wang, William Yang Wang. Findings of EMNLP, 2020.

[14]

MultiModalQA: complex question answering over text, tables and images. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav,Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, Jonathan Berant. ICLR, 2021.

[15]

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. Fengbin Zhu, Wenqiang Lei,Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chua. ACL, 2021.

[16]

ManyModalQA: Modality Disambiguation and QA over Diverse Inputs. Darryl Hannan, Akshay Jain, Mohit Bansal. AAAI, 2020.

[17]

TabFact: A Large-scale Dataset for Table-based Fact Veriﬁcation. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang,
Hong Wang, Shiyang Li, Xiyou Zhou, William Yang Wang. ICLR, 2020.

[18]

Compositional Semantic Parsing on Semi-Structured Tables. Panupong Pasupat, Percy Liang. ACL, 2015.

30

Reasoning with background knowledge[19]

IIRC: A Dataset of Incomplete Information Reading Comprehension Questions. James Ferguson, Matt Gardner, Hannaneh
Hajishirzi, Tushar Khot, Pradeep Dasigi. EMNLP, 2020.

[20]

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Mor Geva, Daniel Khashabi, EladSegal, Tushar Khot, Dan Roth, Jonathan Berant. TACL, 2021.

[21]

CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge. Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, Greg
Durrett. NeurIPS, 2021.

[22]

CommonsenseQA 2.0: Exposing the Limits of AI through Gamiﬁcation. Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra
Bhagavatula, Yoav Goldberg, Yejin Choi, Jonathan Berant. NeurIPS, 2021.

[23]

Thinking Like a Skeptic: Defeasible Inference in Natural Language. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra
Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, Yejin Choi. Findings of EMNLP, 2021.

[24]

The Web as a Knowledge-Base for Answering Complex Questions. Alon Talmor, Jonathan Berant. NAACL, 2018.

[25]

TheoremQA: A Theorem-driven Question Answering dataset. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma,Jianyu Xu, Xinyi Wang, Tony Xia. 2023.

31

Designing evaluations for complex reasoning[26]

Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning. Harsh Trivedi, Niranjan Balasubramanian,Tushar Khot, Ashish Sabharwal. EMNLP, 2020.

[27]

Understanding Dataset Design Choices for Multi-hop Reasoning. Jifan Chen, Greg Durrett. NAACL, 2019.

[28]

Compositional Questions Do Not Necessitate Multi-hop Reasoning. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner,Hannaneh Hajishirzi, Luke Zettlemoyer. ACL, 2019.

[29]

Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. MorGeva, Yoav Goldberg, Jonathan Berant. EMNLP, 2019.

[30]

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension. Max Bartolo, Alastair Roberts, Johannes
Welbl, Sebastian Riedel, Pontus Stenetorp. TACL, 2020.

[31]

Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions. Mihir Parmar, Swaroop Mishra, Mor Geva, Chitta
Baral. EACL, 2023.

[32]

On the Eﬃcacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study. DivyanshKaushik, Douwe Kiela, Zachary C. Lipton, Wen-tau Yih. ACL, 2021.

32

Designing evaluations for complex reasoning[33]

Evaluating Models’ Local Decision Boundaries via Contrast Sets. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, BenBogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, GabrielIlharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith,Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou. Findings of EMNLP, 2020.

[34]

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition. Mor Geva, Tomer Wolfson,Jonathan Berant. TACL, 2022.

[35]

What Ingredients Make for an Effective Crowdsourcing Protocol for Diﬃcult NLU Data Collection Tasks? Nikita Nangia, Saku
Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman. ACL, 2021.

[36]

What Will it Take to Fix Benchmarking in Natural Language Understanding? Samuel R. Bowman, George Dahl. NAACL, 2021.

[37]

REV: Information-Theoretic Evaluation of Free-Text Rationales. Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi,
Swabha Swayamdipta. ACL, 2023.

[38]

ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke
Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz. ICLR, 2023.

33