Title: MULTI-HOP QUESTION ANSWERING VIA REASONING CHAINS

Abstract: Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model (Devlin et al., 2018) to do final answer prediction. Critically, we do not rely on gold annotated chains or "supporting facts:" at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution. Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone. We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop (Welbl et al., 2018) and HotpotQA (Yang et al., 2018), and achieve state-of-art performance on WikiHop and strong performance on HotpotQA. Our analysis shows properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way. Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.

1. INTRODUCTION:

As high performance has been achieved in simple question answering settings (Rajpurkar et al., 2016) , work on question answering has increasingly gravitated towards questions that require more complex reasoning to solve. Multi-hop question answering datasets explicitly require aggregating clues from different parts of some given documents (Dua et al., 2019; Welbl et al., 2018; Yang et al., 2018; Jansen et al., 2018; Khashabi et al., 2018) . Earlier question answering datasets contain some questions of this form (Richardson et al., 2013; Lai et al., 2017) , but typically exhibit a limited range of multi-hop phenomena. Designers of multi-hop datasets aim to test a range of reasoning types (Yang et al., 2018) and, ideally, systems should have to behave in a very specific way in order to do well. However, Chen & Durrett (2019) and Min et al. (2019a) show that models achieving high performance may not actually be performing the expected kinds of reasoning. Partially this is due to the difficulty of evaluating intermediate model components such as attention (Jain & Wallace, 2019) , but it also suggests that models may need inductive bias if they are to solve this problem "correctly."

In this work, we propose a step in this direction, with a two-stage model that identifies intermediate reasoning chains and then separately determines the answer. A reasoning chain is a sequence of sentences that logically connect the question to a fact relevant (or partially relevant) to giving a reasonably supported answer. Figure 1 shows an example of what such chains look like. Extracting chains gives us a discrete intermediate output of the reasoning process, which can help us gauge our model's behavior beyond just final task accuracy. Formally, our extractor model scores sequences of sentences and produces an n-best list of chains via beam search.

To find the right answer, we need to maintain uncertainty over this chain set, since the correct one may not immediately be evident, and for certain types of questions, information across multiple chains may even be relevant. Sifting through the retrieved information to actually identify the answer requires deeper, more expensive computation. We employ a second-stage answer module, a BERT- 

Shirley Temple Shirley Temple:

Reasoning Chain 1 Reasoning Chain 2 q < l a t e x i t s h a 1 _ b a s e 6 4 = " b + n 3 e 1 Y Q E a 9 M n e U T a n z P e 5 6 I i C k = " > A A A B 6 H i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H b R R I 9 E L x 4 h k U c C G z I 7 9 M L I 7 O w 6 M 2 t C C F / g x Y P G e P W T v P k 3 D r A H B S v p p F L V n e 6 u I B F c G 9 f 9 d n J r 6 x u b W / n t w s 7 u 3 v 5 B 8 f C o q e N U M W y w W M S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a H S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t g r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x M a a T 2 O A t s Z U T P U y 9 5 M / M / r p C a 8 9 i d c J q l B y R a L w l Q Q E 5 P Z 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 B h u A t v 7 x K m p W y d 1 G u 1 C 9 L 1 Z s s j j y c w C m c g w d X U I U 7 q E E D G C A 8 w y u 8 O Q / O i / P u f C x a c 0 4 2 c w x / 4 H z + A N z r j P k = < / l a t e x i t > q < l a t e x i t s h a 

v 5 B 8 f C o q e N U M W y w W M S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a H S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t g r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x M a a T 2 O A t s Z U T P U y 9 5 M / M / r p C a 8 9 i d c J q l B y R a L w l Q Q E 5 P Z 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 B h u A t v 7 x K m p W y d 1 G u 1 C 9 L 1 Z s s j j y c w C m c g w d X U I U 7 q E E D G C A 8 w y u 8 O Q / O i / P u f C x a c 0 4 2 c w x / 4 H z + A N z r j P k = < / l a t e x i t > s 2

< l a t e x i t s h a 1 _ b a s e 6 4 = " 2 c g p s r i T 0 F Z 8 6 j L i 3

I G R 6 s e f + N g = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l q Q Y 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 2 Z Q G 5 Q r b t V d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r D I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 Y 2 f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O y Y b g r b 6 8 T t q 1 q n d V r d 3 X K 4 1 G H k c R z u A c L s G D a 2 j A H T S h B Q x G 8 A y v 8 O Z I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w A G S o 2 e < / l a t e x i t > s 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " R G X p O 6 T x 8 L + 4 5 H j a O / V M P k v K d A A = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l a Q Y 8 F L x 4 r 2 g 9 o Q 9 l s J + 3 S z S b s b o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K H S P J a P Z p q g H 9 G R 5 C F n 1 F j p Q Q / q g 3 L F r b o L k H X i 5 a Q C O Z q D 8 l d / G L M 0 Q m m Y o F r 3 P D c x f k a V 4 U z g r N R P N S a U T e g I e 5 Z K G q H 2 s 8 W p M 3 J h l S E J Y 2 V L G r J Q f 0 9 k N N J 6 G g W 2 M 6 J m r F e 9 u f i f 1 0 t N e O N n X C a p Q c m W i 8 J U E B O T + d 9 k y B U y I 6 a W U K a 4 v Z W w M V W U G Z t O y Y b g r b 6 8 T t q 1 q l e v 1 u 6 v K o 1 G H k c R z u A c L s G D a 2 j A H T S h B Q x G 8 A y v 8 O Y I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w A H z o 2 f < / l a t e x i t > s 4

< l a t e x i t s h a 1 _ b a s e 6 4 = " g A x H S T 9 s y 4 3

J i O 1 3 q 0 F 5 J V d s m P k = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l q Q Y 8 F L x 4 r 2 g 9 o Q 9 l s J + 3 S z S b s b o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K H S P J a P Z p q g H 9 G R 5 C F n 1 F j p Q Q / q g 3 L F r b o L k H X i 5 a Q C O Z q D 8 l d / G L M 0 Q m m Y o F r 3 P D c x f k a V 4 U z g r N R P N S a U T e g I e 5 Z K G q H 2 s 8 W p M 3 J h l S E J Y 2 V L G r J Q f 0 9 k N N J 6 G g W 2 M 6 J m r F e 9 u f i f 1 0 t N e O N n X C a p Q c m W i 8 J U E B O T + d 9 k y B U y I 6 a W U K a 4 v Z W w M V W U G Z t O y Y b g r b 6 8 T t q 1 q n d V r d 3 X K 4 1 G H k c R z u A c L s G D a 2 j A H T S h B Q x G 8 A y v 8 O Y I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w A J U o 2 g < / l a t e x i t > s 5 < l a t e x i t s h a 1 _ b a s e 6 4 = " D C r H V 9 b O R d d G G w h k g 5 e m Z B n z A K A = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q o s e C F 4 8 V 7 Q e 0 o W y 2 k 3 b p Z h N 2 N 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 n 1 B p H s t H M 0 n Q j + h Q 8 p A z a q z 0 o P t X / X L F r b p z k F X i 5 a Q C O R r 9 8 l d v E L M 0 Q m m Y o F p 3 P T c x f k a V 4 U z g t N R L N S a U j e k Q u 5 Z K G q H 2 s / m p U 3 J m l Q E J Y 2 V L G j J X f 0 9 k N N J 6 E g W 2 M 6 J m p J e 9 m f i f 1 0 1 N e O N n X C a p Q c k W i 8 J U E B O T 2 d 9 k w B U y I y a W U K a 4 v Z W w E V W U G Z t O y Y b g L b + 8 S l q 1 q n d R r d 1 f V u r 1 P I 4 i n M A p n I M H 1 1 C H O 2 h A E x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g A K 1 o 2 h < / l a t e x i t > s 6 < l a t e x i t s h a 1 _ b a s e 6 4 = " Z a C 3 x E 1 R K N M a j j Q h / K 6 J b N H D 3 T c = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q q M e C F 4 8 V 7 Q e 0 o W y 2 k 3 b p Z h N 2 N 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 n 1 B p H s t H M 0 n Q j + h Q 8 p A z a q z 0 o P t X / X L F r b p z k F X i 5 a Q C O R r 9 8 l d v E L M 0 Q m m Y o F p 3 P T c x f k a V 4 U z g t N R L N S a U j e k Q u 5 Z K G q H 2 s / m p U 3 J m l Q E J Y 2 V L G j J X f 0 9 k N N J 6 E g W 2 M 6 J m p J e 9 m f i f 1 0 1 N e O N n X C a p Q c k W i 8 J U E B O T 2 d 9 k w B U y I y a W U K a 4 v Z W w E V W U G Z t O y Y b g L b + 8 S l q 1 q n d R r d 1 f V u r 1 P I 4 i n M A p n I M H 1 1 C H O 2 h A E x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g A M W o 2 i < / l a t e x i t > s 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " W 7 I d 2 l 0 M H 4 B c M R V B a W e D v C H a X r 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l q Q Y 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 2 b g D c o V t + o u Q N a J l 5 M K 5 G g O y l / 9 Y c z S i C t k k h r T 8 9 w E / Y x q F E z y W a m f G p 5 Q N q E j 3 r N U 0 Y g b P 1 u c O i M X V h m S M N a 2 F J K F + n s i o 5 E x 0 y i w n R H F s V n 1 5 u J / X i / F 8 M b P h E p S 5 I o t F 4 W p J B i T + d 9 k K D R n K K e W U K a F v Z W w M d W U o U 2 n Z E P w V l 9 e J + 1 a 1 b u q 1 u 7 r l U Y j j 6 M I Z 3 A O l + D B N T T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 o K T z 5 z C H z i f P w T G j Z 0 = < / l a t e x i t >

s < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 c g p s r i T 0 F Z 8 6 j L i 3 I G R 6 s e f + N g = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l q Q Y 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 as context to answer a question. We show two possible "reasoning chains" that leverage connections (shared entities or coreference relations) between sentences to arrive at the answer. The first chain is most appropriate, while the second requires a less well-supported inferential leap.

I J H C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 2 Z Q G 5 Q r b t V d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r D I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 Y 2 f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O y Y b g r b 6 8 T t q 1 q n d V r d 3 X K 4 1 G H k c R z u A c L s G D a 2 j A H T S h B Q x G 8 A y v 8 O Z I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w A G S o 2 e < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " R G X p O 6 T x 8 L + 4 5 H j a O / V M P k v K d A A = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l a Q Y 8 F L x 4 r 2 g 9 o Q 9 l s J + 3 S z S b s b o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K H S P J a P Z p q g H 9 G R 5 C F n 1 F j p Q Q / q g 3 L F r b o L k H X i 5 a Q C O Z q D 8 l d / G L M 0 Q m m Y o F r 3 P D c x f k a V 4 U z g r N R P N S a U T e g I e 5 Z K G q H 2 s 8 W p M 3 J h l S E J Y 2 V L G r J Q f 0 9 k N N J 6 G g W 2 M 6 J m r F e 9 u f i f 1 0 t N e O N n X C a p Q c m W i 8 J U E B O T + d 9 k y B U y I 6 a W U K a 4 v Z W w M V W U G Z t O y Y b g r b 6 8 T t q 1 q l e v 1 u 6 v K o 1 G H k c R z u A c L s G D a 2 j A H T S h B Q x G 8 A y v 8 O Y I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w A H z o 2 f < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " g A x H S T 9 s y 4 3 J i O 1 3 q 0 F 5 J V d s m P k = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l q Q Y 8 F L x 4 r 2 g 9 o Q 9 l s J + 3 S z S b s b o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K H S P J a P Z p q g H 9 G R 5 C F n 1 F j p Q Q / q g 3 L F r b o L k H X i 5 a Q C O Z q D 8 l d / G L M 0 Q m m Y o F r 3 P D c x f k a V 4 U z g r N R P N S a U T e g I e 5 Z K G q H 2 s 8 W p M 3 J h l S E J Y 2 V L G r J Q f 0 9 k N N J 6 G g W 2 M 6 J m r F e 9 u f i f 1 0 t N e O N n X C a p Q c m W i 8 J U E B O T + d 9 k y B U y I 6 a W U K a 4 v Z W w M V W U G Z t O y Y b g r b 6 8 T t q 1 q n d V r d 3 X K 4 1 G H k c R z u A c L s G D a 2 j A H T S h B Q x G 8 A y v 8 O Y I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w A J U o 2 g < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " D C r H V 9 b O R d d G G w h k g 5 e m Z B n z A K A = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q o s e C F 4 8 V 7 Q e 0 o W y 2 k 3 b p Z h N 2 N 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 n 1 B p H s t H M 0 n Q j + h Q 8 p A z a q z 0 o P t X / X L F r b p z k F X i 5 a Q C O R r 9 8 l d v E L M 0 Q m m Y o F p 3 P T c x f k a V 4 U z g t N R L N S a U j e k Q u 5 Z K G q H 2 s / m p U 3 J m l Q E J Y 2 V L G j J X f 0 9 k N N J 6 E g W 2 M 6 J m p J e 9 m f i f 1 0 1 N e O N n X C a p Q c k W i 8 J U E B O T 2 d 9 k w B U y I y a W U K a 4 v Z W w E V W U G Z t O y Y b g L b + 8 S l q 1 q n d R r d 1 f V u r 1 P I 4 i n M A p n I M H 1 1 C H O 2 h A E x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g A K 1 o 2 h < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " Z a C 3 x E 1 R K N M a j j Q h / K 6 J b N H D 3 T c = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q q M e C F 4 8 V 7 Q e 0 o W y 2 k 3 b p Z h N 2 N 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 n 1 B p H s t H M 0 n Q j + h Q 8 p A z a q z 0 o P t X / X L F r b p z k F X i 5 a Q C O R r 9 8 l d v E L M 0 Q m m Y o F p 3 P T c x f k a V 4 U z g t N R L N S a U j e k Q u 5 Z K G q H 2 s / m p U 3 J m l Q E J Y 2 V L G j J X f 0 9 k N N J 6 E g W 2 M 6 J m p J e 9 m f i f 1 0 1 N e O N n X C a p Q c k W i 8 J U E B O T 2 d 9 k w B U y I y a W U K a 4 v Z W w E V W U G Z t O y Y b g L b + 8 S l q 1 q n d R r d 1 f V u r 1 P I 4 i n M A p n I M H 1 1 C H O 2 h A E x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g A M W o 2 i < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " W 7 I d 2 l 0 M H 4 B c M R V B a W e D v C H a X r 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l q Q Y 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 2 b g D c o V t + o u Q N a J l 5 M K 5 G g O y l / 9 Y c z S i C t k k h r T 8 9 w E / Y x q F E z y W a m f G p 5 Q N q E j 3 r N U 0 Y g b P 1 u c O i M X V h m S M N a 2 F J K F + n s i o 5 E x 0 y i w n R H F s V n 1 5 u J / X i / F 8 M b P h E p S 5 I o t F 4 W p J B i T + d 9 k K D R n K K e W U K a F v Z W w M d W U o U 2 n Z E P w V l 9 e J + 1 a 1 b u q 1 u 7 r l U Y j j 6 M I Z 3 A O l + D B N T T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 o K T z 5 z C H z i f P w T G j Z 0 = < / l a t e x i t > K is s a n d T e ll C o

based QA system (Devlin et al., 2018) , which can be run relatively cheaply given the pruned context. Our approach resembles past models for coarse-to-fine question answering (Choi et al., 2017; Min et al., 2018; Wang et al., 2019) , but explores the context in a sequential fashion and is trained to produce principled chains.

To train our model, we heuristically label examples with reasoning chains. We use a search procedure leveraging coreference and named entity recognition (NER) to find a path from the start sentence to an end sentence through a graph of related sentences. Constructing this graph requires running an NER system at train time, but does not rely on the answer or answer candidates (Kundu et al., 2018) . Our system also does not require these annotations at test time, operating instead from raw text.

Our chain identification is effective and flexible: we can use it to derive supervision on two existing datasets, and on HotpotQA (Yang et al., 2018) , we found that these derived chains are essentially as effective as the ground-truth supporting fact sets labeled in the dataset. In terms of final question answering accuracy, on the WikiHop dataset (Welbl et al., 2018) , our approach achieves state-of-theart performance by a substantial margin, and on HotpotQA, we achieve strong results and outperform several recent published systems.

Our contributions are as follows: (1) We present a method for extracting oracle reasoning chains for multi-hop reasoning tasks. These chains are grounded in simple types of reasoning and generalize across multiple datasets. (2) We present a model that learns from these chains at train time and at test time can produce a list of chains. These are fed into a simple downstream model (BERT) to extract a final answer. (3) Results on two large datasets show strong performance by our chain extraction and show that our chains are intrinsically a good representation of evidence for question answering.

2. QUESTION ANSWERING VIA CHAIN EXTRACTION:

We describe our notion of chain extraction in more detail. A reasoning chain is a sequence of sentences that logically connect the question to a fact relevant to determining the answer. Two adjacent sentences in a reasoning chain should be intuitively related: they should exhibit a shared entity or event, temporal structure, or some other kind of textual relation that would allow a human reader to connect the information they contain. Figure 1 shows an example of possible reasoning chains of an real example. In this case, we need to find information about the actor who played Corliss Archer in Kiss and Tell. These question entities may appear in multiple places in the text, and it is generally difficult to know which entity mentions might eventually lead to text containing the answer. If we arrive at s 4 and find the new entity Shirley Temple, we then need to determine what government position she held, which in this case can be found by two additional steps. Other reasoning chains could theoretically lead to this answer, such as the second chain: Shirley Temple starred in the sequel to Kiss and Tell, which might lead us to infer that Shirley Temple also plays Corliss Archer in Kiss and Tell. Although less justified, we also view this as a valid reasoning chain. However, in general, there are also "connected" sequences of sentences that don't imply the answer; for example, they are connected by an entity which is not related to the question.

In determining this chain, we largely used information about entity coreference to connect the relevant pieces: either cross-document coreference about Shirley Temple or resolution of various pronouns. Another relevant cue is that subsequent information about Shirley Temple in Document 1 occurs later in the discourse, which in this case reflects temporal structure. However, solving coreference or temporal relation extraction in general is neither necessary nor sufficient to do chain extraction. Therefore, we design our system so that it does not rely on coreference at test time, but can instead directly extract reasoning chains based on what it has learned at training time.

Having established this notion of a reasoning chain, we have three questions to answer. First, how can we automatically select pseudo-ground-truth reasoning chains? Second, how do we model the chain extraction process? Third, how do we take one or more extracted chains and turn them into a final answer? We answer these three questions in the next section.

3. LEARNING TO EXTRACT CHAINS:



3.1. HEURISTIC ORACLE CHAIN CONSTRUCTION:

Following the intuition in Figure 1 , we assume that there are two relevant connections between sentences that can form reasoning chains. First, the presence of a shared entity often implies some kind of connection. This is not always a sufficient clue, since common entities like United States may occur in otherwise unrelated sentences; however, because this oracle is only used at train time, it does not need to be 100% reliable for the model to learn a chain extraction procedure. Second, we assume that any two sentences in the same paragraph are connected; this is often true on the basis of coreference or other kinds of bridging anaphora, but these may be hard to identify automatically.

We derive heuristic reasoning chains by searching over a graph which is constructed based on these factors. Each sentence s i is represented as a node i in the graph. We run an off-the-shelf named entity recognition system to extract all entities for each sentence. If sentence i and sentence j contain a shared entity based on string match, we add an edge between node i and j. We then also add an edge between all pairs of sentence within the same paragraph. 1Starting from the question node, we do an exhaustive search to find all possible chains that could lead to the answer. This process yields a set of possible chains with different lengths; two examples are shown in Figure 1 . We use two different criteria to select heuristic oracles:

• Shortest Path: We simply take the shortest chain from the chain set as our oracle.

• Question Overlap: We compute the Rouge-F1 score for each chain's sentences with respect to the question and take the chain with the highest score. This encourages selection of more complete answer chains which address all of the question's parts without finding shortcuts.

3.2. CHAIN EXTRACTION MODEL:

Our chain extractor is a neural network that takes the input documents and questions as input and returns a variable-length sequence of sentence pointers as output.

The processing flow of our chain extractor can be divided into two main parts: sentence encoding and chain prediction as shown in Figure 2 .

Sentence Encoding Given a document containing n paragraphs and a question, we first concatenate the question with each paragraph and then encode them using the pre-trained BERT encoder (Devlin et al., 2018) . We denote the encoded ith paragraph as p i . We also encode the question by itself with BERT, denoting as q. To compute the representation of a sentence, we extract it from ). For simplicity, we choose max-pooling as our span extractor, though other choices are possible. We name this scheme of sentence representation as BERT-Para. This paragraph-factored model is much more efficient and scalable than attempting to run BERT on the full context, as full contexts can be thousands of words long. We also explore an even more factored version where each sentence is concatenated with the question and encoded independently, which we denote as BERT-Sent. Finally, instead of using BERT as the sentence encoder, we use a bidirectional attention layer between the passage and question (Seo et al., 2017) as a baseline; we call this model BiDAF-Para.

Chain Prediction:

We treat all the encoded sentence representations as a bag of sentences and adopt an LSTM-based pointer network (Vinyals et al., 2015) to extract the reasoning chain, shown on the right side of Figure 2 . At the first time step, we initialize the hidden state h 0 in the pointer network using the max-pooled representation of the question q, and feed a special token SOS as the first input.

Let c 1 , . . . , c l denote the indices of sentences to include in the reasoning chain. At time step t, we compute the probability of sentence i being chosen as P (c t = i|c 1 , . . . , c t-1 , s) = softmax(α) [i] , where α i = W[h t-1 ; s ct-1 ; h t-1 s ct-1 ], and W is a weight matrix to be learned.

Training the Chain Extractor During training, the loss for time step t is the negative log likelihood of the target sentence c * t for that time step: loss t = -log(P (c * t )|c * 1 , . . . , c * t-1 s). We also explored training with reinforcement learning. For the two datasets we considered, pre-training with our oracle and fine-tuning with policy gradient this did not lead to an improvement. Pure oracle chain extraction appears strong enough for the model to learn the needed associations across chain timesteps, but this may not be true on other datasets.

At evaluation time, we use beam search to explore a set of possible chains, which results in a set of chains c 1 , c 2 , ..., c k , with each chain containing different number of sentences.

3.3. ANSWER PREDICTION:

Since different beams may contain different plausible reasoning chains as shown in Figure 1 , we consider the sentences in the top k beams predicted by our chain extractor as input to our answer prediction model. Different datasets may require different modifications of the basic BERT model as well as different types of reasoning, so we present the answer prediction module in the following section.

4. EXPERIMENTAL SETUP:



4.1. DATASETS:

WikiHop Welbl et al. (2018) introduced this English dataset specially designed for text understanding across multiple documents. The dataset consists of around 40k questions, answers, and passages. Questions in this dataset are multiple-choice with around 10 choices on average.

HotpotQA Yang et al. (2018) proposed a new dataset with 113k English Wikipedia-based question-answer pairs. Similar to WikiHop, questions require finding and reasoning over multiple supporting documents to answer. Different from WikiHop, models should choose answers by selecting variable-length spans from these documents. Sentences relevant to finding the answer are annotated as "supporting facts" in the dataset.

4.2. IMPLEMENTATION DETAILS:

Oracle chain extraction We use the off-the-shelf NER system from AllenNLP (Gardner et al., 2017) . We treat any entity that appears explicitly more than 5 times across sentences as a common entity,2 and ignore it when we build the graph. Because these documents are only short snippets from Wikipedia, this criterion is loose enough to keep most useful mentions.

Chain extractor We use the uncased BERT tokenizer to tokenize both question and paragraphs. We use the pretrained bert-base-uncased model and fine-tune it using Adam with a fixed learning rate of 5e-6. At test time, we produce our chains using beam search with beam size 5.

Answer prediction We concatenate the question and the combined chains from previous step in the top k beams in the standard way as described in the original BERT paper Devlin et al. (2018) and encode it using the pre-trained BERT model. We denote its [CLS] token as [CLS] p .

WikiHop is a multiple-choice dataset. Since we need to choose an answer from a candidate list, we encode each candidate with BERT. The [CLS] token for candidate i is denoted as [CLS] Ci . We then compute the score of a candidate C i being choose as the dot product between [CLS] p and [CLS] Ci .

HotpotQA is a span-based question answering task, where finding the answer requires predicting the start and end of a span in the context. We compute distributions over these positions via two learned weight matrices W start and W end . Each position in the concatenated sequence except the [CLS] token is multiplied by the corresponding weight matrix and softmaxed. Since we also need to predict the question type on HotpotQA (to handle yes/no questions vs. span extraction ones), we predict the type by taking the dot product of [CLS] p with a trainable matrix W type . We use bert-large-uncased instead of bert-base-uncased in the answer prediction module. We use the same optimizer and learning rate as chain extractor.

5. RESULTS:

In this section we aim to answer two main questions. First, which of our proposed chain extraction techniques is most effective, and how do they compare? Second, how does our approach compare to state-of-the-art models on these datasets? Finally, can we evaluate our extracted chains intrinsically: how important is ordering and how well do they align with human intuition about question answering?

5.1. COMPARISON OF CHAIN EXTRACTION METHODS:

In this section, we study the characteristics of our extracted chains with several experiments focused on HotpotQA. We choose this dataset since it provides human-annotated supporting facts so we can directly compare these against our model. Several statistics are shown in Table 1 . For different combinations of our model and which choice of chain oracle we use, we calculate several statistics, as described in the caption. We have the following observations:

Using more context helps chain extractors to find relevant sentences. Comparing BERT-Para and BERT-Sent, we find that with all other parts fixed and only by encoding more context, we improve the answer prediction performance by around 5%. This may indicate that BERT can capture cross sentence relations such as coreference and find more supporting evidence as a result. The Table 1 : The characteristics of different chains generated by different models under different supervision on the HotpotQA dev set: for different models and chain oracles, we report the average chain length, fraction of chains containing the answer, F1 with respect to the annotated supporting facts, and F1 on the final QA task. Here we only pick the chain in the first beam.

comparison with BiDAF-Para vs. BERT-Sent also indicates this. Despite finding many fewer answer candidates (62% instead of 72%), BiDAF-Para only achieves around 2% lower performance. One possible explanation to this is that without context, the BERT extraction model may pick up "distractor" sentences related to the question but which do not actually lead to the answer, potentially confusing the answer prediction module and cause a drop on the performance.

The one-best chain often contains the answer. This demonstrates the effectiveness of our chain extractor: the BERT-Para model with just 2 extracted sentences can locate the answer 76% of the time. We further analyze the quality of these chains in the following sections. Note that this is nearly the same amount of evidence as in the human-labeled supporting facts (2.4 sentences on average); the difference can be explained by cases where the model can jump directly to the answer (Chen & Durrett, 2019) .

Q-Overlap helps recover more supporting evidence. The main difference between our Shortest oracle and the Q-Overlap oracle is that Q-Overlap contains additional relevant sentences besides the one containing the answer. As a result, models trained with Q-Overlap should also yield a higher F1 score with respect to the supporting facts, which is supported by the results (64 vs. 56).

Performance can be improved by taking a union across multiple chains In the last row, we show a version of BERT-Para where the top 5 chains in the beam have been unioned together and truncated to 5 sentences. These top 5 chains contain permutations of roughly the same sentences, so this does not greatly increase the average length. However, this greatly increases answer recall and downstream F1. One reason is that this maintains uncertainty over the correct reasoning chain and can seamlessly handle question types involving comparison of multiple entities, which are difficult to address with a single reasoning chain of the sort presented in Figure 1 .

5.2. RESULTS COMPARED TO OTHER SYSTEMS:

We evaluate our best system from the prior section (BERT-Para with top-5 chains) on the blind test sets of our two datasets. Performance is shown in Table 2 . On WikiHop, our model significantly outperforms past models, although these models do not use BERT. For HotpotQA, we use RoBERTa (Liu et al., 2019) weights as the pretrained model instead of BERT, which gives a performance gain.

Our model also achieves strong performance compared to past models, including outperforming those which use labeled supporting facts3 . This indicates that our heuristically-extracted chains can stand in effectively for this supervision, which suggests that our approach can generalize to settings where this annotation is not available.

5.3. EVALUATION OF CHAINS:

Ordered extraction outperforms unordered extraction One question we can ask is how important ordered chain extraction is versus just selecting "chain-like" sentences in an unordered fashion.

dev test GCN (De Cao et al., 2018) 64.8 67.6 BAG (Cao et al., 2019) 66.5 69.0 CFC (Zhong et al., 2019) 66.4 70.6 JDReader (Tu et al., 2019) 68.1 70.9 DynSAN (Zhuang & Wang, 2019) 70.1 71.4 BERT-Para (top 5) 72.2 76.5 EM F1 Supp?

BiDAF++ (Yang et al., 2018) 45.60 59.02 Y DecompRC (Min et al., 2019b) 55.20 69.63 N QFE (Nishida et al., 2019) 53.86 68.06 Y DFGN (Qiu et al., 2019) 56 We compare our BERT-Para model with a variant of our model where, instead of using a pointer network to predict a sequence, we make an independent classification decision for each sentence to determine whether it is relevant to the question or not. We then pick top k sentences, for a specified value of k, with the highest relevance score and feed these to our BERT model. We name this model as unordered extraction. Both are trained with the shortest-path oracle4 . To make a fair comparison to the unordered model, we pick the same number of sentences ranked by prediction probability as the (top-5) chain extractor.

QA performance on those datasets is shown in Table 3 . Besides WikiHop and HotpotQA, we also train and test our model on a hard subset of HotpotQA pointed out by Chen & Durrett (2019) . We see that the sequential model is more powerful than the unordered model. On all datasets, our chain extractor leads to higher QA performance than the unordered extractor, especially on HotpotQA-Hard, where multi-hop reasoning is more strongly required. This is in spite of the fact that the fraction of answers recovered is similar. This implies that even for a very powerful pre-trained model like BERT, an explicitly sequential interaction between sentences is still useful for recovering related evidences. On WikiHop, the improvement yield by our chain extractor is more marginal. One reason is that correlations have been noted between the question and answer options (Chen & Durrett, 2019) , so that the quality of the extracted evidence contributes less to the models' downstream performance.

Chain extraction is near the performance limit on HotpotQA Given our two-stage procedure, one thing we can ask is: with a "perfect" chain extractor, how well would our question answering model do? We compare the performance of the answer prediction trained with our extracted chains against that trained with the human-annotated supporting facts. As we can see in Table 1 , BERT achieves a 75.4% F1 on the annotated supporting facts, which is only 5% higher than the result achieved by our BERT-Para (top 5) extractor. A better oracle or stronger chain extractor could help close this gap, but it is already fairly small considering the headroom on the task overall. It also shows there exist other challenges to address in the question answering piece, complementary to the proposed model in this work, like decomposing the question into different pieces (Min et al., 2019b) to further improve the multi-hop QA performance. Human evaluation We perform a human evaluation to compare the quality of our extracted chains with our oracle as well as the annotated supporting facts. We randomly pick 50 questions in Hot-potQA and ask three Turkers to answer each question based on those different evidences and rate their confidence in their answer. We pick the Turkers' answer which has the highest word overlap with the actual answer (to control for Turkers who have simply misunderstood the question) and assess their confidence in it. The statistics are shown in Table 4 . Human performance on the three sets is quite similar -they have similar confidence in their answers, and their answers achieve similar F1 score. Although sometimes the shortest oracle may directly jump to the answer and the extracted chains may contain distractors, humans still seem to be able to reason effectively and give confidence in their answers on these short chains. Since the difference between supporting facts and our oracle on overall question answering performance is marginal, this is further evidence that the human-annotated supporting facts may not be needed.

For examples of the chains themselves, please see Appendix A.

6. RELATED WORK:

Text-based multi-hop reasoning Memory Network based models (Weston et al., 2015; Sukhbaatar et al., 2015; Kumar et al., 2016; Dhingra et al., 2016; Shen et al., 2017) try to solve multi-hop questions sequentially by using a memory cell which is designed to gather information iteratively from different parts of the passage. These models are trained in an end-to-end fashion and the reasoning is conducted implicitly. More recent work including Entity-GCN (De Cao et al., 2018) , MHQA-GRN (Song et al., 2018) , and BAG (Cao et al., 2019) , form this problem as a search over entity graph, and adapt graph convolution network Kipf & Welling (2017) to do reasoning. Such kind of models need to construct an entity graph both at training and test time, while we only need such entities during training.

Coarse-to-fine question answering Selecting the most related content regarding the question is helpful to improve the performance of a QA model. Choi et al. (2017) combine a coarse, fast model for selecting relevant sentences and a more expensive RNN for producing the answer from those sentences. Wang et al. (2019) apply distant supervision to generate labels and uses them to train a neural sentence extractor. Another line of work proposes to use the answer prediction score as supervision to the sentence extractor (Wang et al., 2018; Indurthi et al., 2018; Min et al., 2018) . Instead of treating the sentence selection as a latent variable or learning the sentence extractor using policy gradient, we treat the sentence extractor as a sequence predictor and use step by step supervision generated by heuristics to train. This represents a step towards building models that represent the reasoning process more explicitly (Trivedi et al., 2019; Jiang et al., 2019) .

7. DISCUSSION AND CONCLUSION:

In this work, we learn to extract reasoning chains to answer multi-hop reasoning questions. Experimental results show that the chains are as effective as human annotations, and achieve strong performance on two large datasets. However, as remarked in past work (Chen & Durrett, 2019; Min et al., 2019a) , there are several aspects of HotpotQA and WikiHop which make them require multi-hop reasoning less strongly than they otherwise might. As more challenging QA datasets are built based on lessons learned from these, we feel that reasoning about more explicit reasoning and properties of chain-like representations will be critical. This work represents a first step towards this goal of improving QA systems in such settings.

A APPENDIX: CASE STUDY OF EXTRACTED CHAINS:

We dig into the chains picked up by our chain extractor to better understand what is captured by our model. Those examples are shown in Figure 3 . Seen from example (a), the model picks a perfect chain by first picking the sentence containing "Kiss and Tell" and "Corliss Archer", then finds the next sentence through "Shirley Temple". At the last step, to our surprise, it even finds a sentence via coreference. This demonstrates that although we do not explicitly model the entity links, the model still manages to learn to jump through entities in each hop.

Example (b) shows how our system can deal with comparison-style yes/no questions. There are two entities, namely, "Laleli Mosque" and "Esma Sultan Mansion" in the question, each of which must be pursued. The chain extractor proposes first a single-sentence chain about the first entity, then a single-sentence chain about the second entity. When unioned together, our answer predictor can leverage both of these together.

Example (c) shows that the extraction model picks a sentence containing the answer but without justification, it directly jumps to the answer by the lexical overlap of the two sentences and the shared entity "South Korean". The chain picked in the second beam is a distractor. There are also different distractors that contains in other hypotheses, of which we do not put in the example. The fifth hypothesis contains the correct chain. This example shows that if the same entity appears multiple time in the document, the chain extractor may be distracted and pick unrelated distractors.

:

We do not explicitly run a coreference system here since current coreference systems often introduce spurious arcs. Moreover, cross-document links can nearly always be found by exact string match, and since we add all within-paragraph links, exactly determining the coreference status of every mention is not needed.

These mentions are often extremely common entities like U.S., which are likely to introduce spurious edges rather than good ones.

Performance for other unpublished works can be find on the leader board: https://hotpotqa.github.io

We do not use the question overlap oracle since the questions in WikiHop are synthetic like "place of birth gregorio di cecco", which is uninformative for the Q-overlap method.

Figures and Tables:

What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell ?

1 _ b a s e 6 4 = " b + n 3 e 1 Y Q E a 9 M n e U T a n z P e 5 6 I i C k = " > A A A B 6 H i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H b R R I 9 E L x 4 h k U c C G z I 7 9 M L I 7 O w 6 M 2 t C C F / g x Y P G e P W T v P k 3 D r A H B S v p p F L V n e 6 u I B F c G 9 f 9 d n J r 6 x u b W / n t w s 7 u 3

Figure1: A multi-hop example chosen from the HotpotQA development set. Several documents are given as context to answer a question. We show two possible "reasoning chains" that leverage connections (shared entities or coreference relations) between sentences to arrive at the answer. The first chain is most appropriate, while the second requires a less well-supported inferential leap.

Figure 2: The BERT-Para variant of our proposed chain extractor. Left side: we encode each document paragraph jointly with the question and use pooling to form sentence representations. Right side: a pointer network extracts a sequence of sentences. the encoded paragraph. Suppose sentence j in the document is the jth sentence of paragraph i. Then s j = Span Extractor(p i , s START j

The blind test set performance achieved by our model on WikiHop and HotpotQA. On HotpotQA, all published works except DecompRC use the annotated supporting facts as extra supervision, which makes them not directly comparable to our model. However, our model still achieves strong performance on this dataset despite not using this annotation.

Table content: <table><tr><td/><td>.31 69.69</td><td>Y</td></tr><tr><td>Roberta-Para (top 5)</td><td>61.20 74.11</td><td>N</td></tr></table>

The downstream QA performance of the chains generated by different models on different datasets. The performance is evaluated by accuracy and F1 score respectively in WikiHop and HotpotQA dataset.

Table content: <table/>

The human evaluation on different evidence sets. For each row, 50 responses are bucketed based on the Turkers' confidence ratings, and numbers denote the answer F1 within that bucket.

Table content: <table><tr><td/><td colspan="3">quite confident somewhat confident not confident</td></tr><tr><td>shortest oracle</td><td>34 / 77.7</td><td>7 / 68.6</td><td>9 / 70.6</td></tr><tr><td>extracted chain</td><td>37 / 81.1</td><td>7 / 64.2</td><td>6 / 50.0</td></tr><tr><td>annotated supporting facts</td><td>33 / 78.8</td><td>12 / 60.0</td><td>5 / 88.0</td></tr></table>

Peer review (iclr2020):
The paper is tacking the problem of multi-hop machine reading. The method is evaluated on hotpotqa and wikihop datasets. The model consists of two steps: First, the model learns to retrieve supporting facts. To do so, the authors propose a heuristic method to retrieval *reasoning graph* based on entities present in the question and the text. In the method, The nodes are basically the sentences, the edges are the adjacency of sentences and the presence of shared entities. Each reasoning path starts from the question and ends at the reach of the answer. Finally, 2 strategies are used to select the path *shortest* and *ROUGE*. From this dataset, an RNN based extractor is trained. A first question is why not using the supporting facts training signal which is present in both datasets. It would have been useful to compare this heuristic extraction method for the training dataset extraction to the sole use of the supporting facts as this first step of the method is considered as the originality. Then, based on the supporting fact extraction, an SoA BertQA model is used to perform the extraction. As a second step of the method, the method seems to produce a quite marginal improvement compared to the current state of the art.