SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2: Class-wise decision prediction accuracy on the development set (*: reported in the paper).
Models	Oracle Question Generation Task
Development Set	Cross Validation
BLEU1	BLEU4	BLEU1	BLEU4
E3	52.79±2.87	37.31±2.35	51.75	35.94
E3+UniLM	57.09±1.70	41.05±1.80	56.94	42.87
EMT	62.32±1.62	47.89±1.58	64.48	52.40
Table 3: Performance on Oracle Question Generation Task. We show both results on the development set and 10-fold cross validation.  $\mathrm{E}^3 +$  UniLM replaces the editor of  $\mathrm{E}^3$  to our finetuned UniLM.
size of 16 and a learning rate of 2e-5, and we use a beam size of 10 for inference.
To reduce the variance of our experimental results, all experiments reported on the development set are repeated 5 times with different random seeds. We report the average results along with their standard deviations.
End-to-End Task. The end-to-end performance on the held-out test set is shown in Table 1. EMT outperforms the existing state-of-the-art model  $\mathrm{E}^3$  on decision classification in both micro- and macroaccuracy. Although the BLEU scores are not directly comparable among different models, EMT achieves competitive BLEU1 and BLEU4 scores on the examples it makes an Inquire decision. The results show that EMT has strong capability in both decision making and follow-up question generation tasks. Table 2 presents the class-wise accuracy on the four decision types. EMT improves on the Inquire decision significantly. It is because EMT can explicitly track the states of all rule sentences; it has a macro accuracy of  $80\%$  on the
Models	End-to-End Task	Oracle Question Generation Task
Micro Acc.	Macro Acc.	BLEU1	BLEU4	BLEU1	BLEU4
EMT	71.36±0.69	76.70±0.54	67.04±1.59	52.37±1.92	63.53±1.03	48.69±0.80
EMT (w/o data aug.)	70.67±0.52	76.33±0.69	65.86±2.25	51.02±2.52	62.38±1.34	47.58±1.30
EMT (w/o c2f)	70.41±0.94	75.96±0.91	65.73±1.76	50.84±2.31	61.98±1.26	47.66±1.33
EMT (w/o Lentail)	67.81±1.20	73.50±0.83	63.84±1.80	49.35±2.10	60.50±1.16	45.34±1.73
EMT (w/o tracker)	67.42±1.15	72.73±0.74	63.26±0.64	47.97±0.40	61.87±1.46	47.13±1.35
Table 4: Ablation Study of EMT on the development set of ShARC.
entailment state prediction task.
Oracle Question Generation Task. To establish a concrete question generation evaluation, we conduct experiments on our proposed oracle question generation task. We compare our model EMT with  $\mathrm{E}^3$  and an extension  $\mathrm{E}^3 +\mathrm{UniLM}$ ; implementations for other methods are not publicly available.  $\mathrm{E}^3 +\mathrm{UniLM}$  replaces the editor of  $\mathrm{E}^3$  with our fine-tuned UniLM. The results on the development set and 10-fold cross validation are shown in Table 3.
Firstly,  $\mathrm{E}^3 + \mathrm{UniLM}$  performs better than  $\mathrm{E}^3$ , validating the effectiveness of our follow-up question rephrasing module: finetuned UniLM. More importantly, EMT consistently outperforms  $\mathrm{E}^3$  and  $\mathrm{E}^3 + \mathrm{UniLM}$  on both the development set and the cross validation by a large margin. Although there is no ground truth label for span extraction, we can infer from the question generation results that our coarse-to-fine reasoning approach extracts better spans than the extraction and retrieval modules of  $\mathrm{E}^3$ . This is because  $\mathrm{E}^3$  propagates error from the span extraction module to the span retrieval module while our coarse-to-fine approach avoids this problem through weighting token-level span distributions with its sentence-level entailment scores.
We conduct an ablation study on the development set for both the end-to-end evaluation task and oracle question generation evaluation task. We consider four ablations of our EMT model:
(1) EMT (w/o data aug.) trains the model on the original ShARC training set and do not use any augmented data using the evidence.
(2) EMT (w/o c2f) extracts the rule span without weighted by the entailment score  $\zeta$  in Eqn. 12.
(3) EMT (w/o  $\mathcal{L}_{\mathrm{entail}}$ ) removes the entailment state prediction subtask in decision making, and thus there is no entailment score  $\zeta$  for underspecified span extraction in Eqn. 12.
(4) EMT (w/o tracker) that removes the explicit memory tracking module. Instead, it treats
the [CLS] token for each rule sentence as the state for decision making and span extraction.
Results of the ablations are shown in Table 4, and we have the following observations:
- With the help of data augmentation, EMT boosts the performance slightly on the end-to-end task, especially for the question generation task which originally has only 6804 training examples. The augmented training instances boosts the performance even though the augmentation method does not produce any new question. This implies that the size of the ShARC dataset is a bottleneck for an effective end-to-end neural models.
- Without the coarse-to-fine reasoning for span extraction, EMT (w/o c2f) drops by 1.53 on BLEU4, which implies that it is necessary for the question generation task. The reason is that, as a classification task, entailment state prediction can be trained reasonably well (80% macro accuracy) with a limited amount of data (6804 training examples). Therefore, the Unknown scores in the entailment state prediction can guide the span extraction via a soft modulation (Equation 12). On the other hand, one-step span extraction method does not utilize the entailment states of the rule sentences from EMT, meaning it does not learn to extract the underspecified part of the rule text.
- With the guidance of explicit entailment supervision, EMT outperforms EMT (w/o  $\mathcal{L}_{\text {entail }}$ ) by a large margin. Intuitively, knowing the entailment states of the rule sentences makes the decision making process easier for complex tasks that require logical reasoning on conjunctions of conditions or disjunctions of conditions. It also helps span extraction through the coarse-to-fine approach.
- Without the explicit memory tracker described in Section 2.2, EMT (w/o tracker) performs poorly on the decision making task. Although there exist interactions between rule sentences and user information in BERT-encoded representations through
Regulation Text A
(parsed into six rule sentences: S1 ~ S6)	Entailment States
Turn 1	Turn 2	Turn 3
S1: Statutory Maternity Pay	U (99.99)	U (99.99)	U (99.99)
S2: To qualify for cmp you must:	U (99.99)	U (99.99)	U (99.99)
S3: * earn on average at least £113 a week	U (99.93)	E (99.91)	E (99.67)
S4: * give the correct notice	U (99.97)	U (99.61)	C (99.81)
S5: * give proof you're pregnant	U (99.98)	U (99.75)	U (99.94)
S6: * have worked for your employer...	U (99.98)	U (99.70)	U (99.96)
Scenario: I've been old enough to get my pension. My wife just reached pension age last year. Neither of us have applied for it yet.
Initial Question: Do I qualify for SMP?
Decision: Generated Question Answer
Turn 1: Inquire Do you earn on average at least £113 a week? Yes
Turn 2: Inquire Did you give the correct notice? No
Turn 3: No
Regulation Text B
(parsed into four rule sentences: S1 ~ S4)	Entailment States
Turn 1	Turn 2
S1: Age Restrictions:	U (99.99)	U (99.99)
S2: Patients under the age of 18 must have a designated provider.	U (99.93)	C (99.86)
S3: Patients under the age of 21 cannot participate in a cooperative garden or ...	U (99.99)	U (96.35)
S4: The patient's designated provider may participate in a cooperative garden on behalf of the patient.	U (99.99)	U (99.99)
Scenario: The contract is for payments below the minimum wage.
Initial Question: Do I have to have a designated provider?
Decision: Generated Question Answer
Turn 1: Inquire Are you under the age of 18? No
Turn 2: No
Figure 3: Predicted decisions and generated questions by our EMT model. Extracted spans and their corresponding questions are marked in the same colors. We also visualize the transitions of predicted entailment states (Entailment, Contradiction, Unknown) over rule sentences (S1, S2, S3 ... as the conversation flows, with associated entailment scores  $[\beta_{\text{entailment}}, \beta_{\text{contradiction}}, \beta_{\text{unknown}}]$ .
multi-head self-attention, it is not adequate to learn whether conditions listed in the rule text have already been satisfied or not.
3.4 Interpretability
To get better insights into the underlying entailment-oriented reasoning process of EMT, we examine the entailment states of the rule sentences as the conversation flows. Two example cases are provided in Figure 3. Given a rule text containing several rule sentences (S1, S2, S3, ...), we show the transition of predicted entailment states  $[\beta_{\text {entailment}}, \beta_{\text {contradiction}}, \beta_{\text {unknown}}]$  over multiple turns in the dialogue.
Rules in Bullet Points. Figure 3 (a) shows an example in which the rule text is expressed in the conjunction of four bullet-point conditions. On the first turn, EMT reads "Scenario" and "Initial Question" and they only imply that the question from the user is relevant to the rule text. Thus the entailment states for all the rule sentences are Unknown, and EMT makes an Inquire decision, and asks a question. Once a positive answer is received from the user part for the first turn, EMT transits the entailment state for rule sentence S3 from Unknown to Entailment, but it still cannot conclude the dialogue, so it asks a second follow-up question. Then we see that the user response for the second question is negative, which makes EMT conclude a final decision No in the third turn.
Rules in Plain Text. Figure 3 (b) presents a more challenging case where the rules are in plain text. Therefore, it is not possible to put the whole sentence into a clarification question as EMT in Figure 3(a) does. In this case, both the decision making module and span extraction module contribute to helping the user. The span extraction module locates the correct spans inside S2, and EMT concludes a correct answer "No" after knowing the user does not fulfill the condition listed in S2.
3.5 Error Analysis
We analyze some errors of EMT predictions on the ShARC development set, as described below.
Decision Making Error. Out of 2270 examples in the development set, our EMT produces incorrect decisions on 608 cases. We manually analyze 104 error cases. In 40 of these cases, EMT fails to derive the correct entailment states for each rule sentence, while in 23 cases, the model predicts the correct entailment states but cannot predict correct decisions based on that. These errors suggest that explicitly modeling the logic reasoning process is a promising direction. Another challenge comes from extracting useful information from the user scenarios. In 24 cases, the model fails to make the correct decision because it could not infer necessary user information from the scenarios. Last but not least, parsing the rule text into rule sentences is also a challenge. As shown in Figure 3(b), the plain
text usually contains complicated clauses for rule conditions, which is difficult to disentangle them into separate conditions. In 17 cases, one single rule sentence contains multiple conditions, which makes the model fail to conduct the entailment reasoning correctly.
Question Generation Error. Out of 562 question generation examples in the development set, our EMT locates the underspecified span poorly in 115 cases (span extraction F1 score  $\leq 0.5$ ). We manually analyze 52 wrong question generation cases. Out of 29 cases of them, EMT fails to predict correct entailment states for rule sentences, and thus does not locate the span within the ground truth rule sentence, while in 9 cases, it finds the correct rule sentence but extracts a different span. Another challenge comes from the one-to-many problem in sequence generation. When there are multiple underspecified rule sentences, the model asks about one of these underspecified rule sentences which is different from the ground truth one. This suggests that new evaluation metrics could be proposed by taking this into consideration.
ShARC Conversational Machine Reading (Saeidi et al., 2018) differs from conversational question answering (Choi et al., 2018; Reddy et al., 2019) and conversational question generation (Gao et al., 2019) in that 1) machines are required to formulate follow-up questions to fill the information gap, and 2) machines have to interpret a set of complex decision rules and make a question-related conclusion, instead of extracting the answer from the text. CMR can be viewed as a special type of task-oriented dialog systems (Wen et al., 2017; Zhong et al., 2018; Wu et al., 2019) to help users achieve their goals. However, it does not rely on predefined slot and ontology information but natural language rules.
On the ShARC CMR challenge (Saeidi et al., 2018), Lawrence et al. (2019) propose an end-to-end bidirectional sequence generation approach with mixed decision making and question generation stages. Saeidi et al. (2018) split it into sub-tasks and combines hand-designed sub-models for decision classification, entailment and question generation. Zhong and Zettlemoyer (2019) propose to extract all possible rule text spans, assign each of them an entailment score, and edit the span with the highest score into a follow-up question.
However, they do not use these entailment scores for decision making. Sharma et al. (2019) study patterns of the dataset and include additional embeddings from dialog history and user scenario as rule markers to help decision making. Compared to these methods, our EMT has two key differences: (1) EMT makes decision via explicitly entailment-oriented reasoning, which, to our knowledge, is the first such approach; (2) Instead of treating decision making and follow-up question generation (or span extraction) separately, EMT is a unified approach that exploits its memory states for both decision making and question generation.
Memory-Augmented Neural Networks. Our work is also related to memory-augmented neural networks (Graves et al., 2014, 2016), which have been applied in some NLP tasks such as question answering (Henaff et al., 2017) and machine translation (Wang et al., 2016). For dialog applications, Zhang et al. (2019) propose a dialogue management model that employs a memory controller and a slot-value memory, Bordes et al. (2016) learn a restaurant bot by end-to-end memory networks, Madotto et al. (2018) incorporate external memory modules into dialog generation.
In this paper, we have proposed a new framework for conversational machine reading (CMR) that comprises a novel explicit memory tracker (EMT) to track entailment states of the rule sentences explicitly within its memory module. The updated states are utilized for decision making and coarse-to-fine follow-up question generation in a unified manner. EMT achieved a new state-of-the-art result on the ShARC CMR challenge. EMT also gives interpretability by showing the entailment-oriented reasoning process as the conversation flows. While we conducted experiments on the ShARC dataset, we believe the proposed methodology could be extended to other kinds of CMR tasks.
We thank Max Bartolo and Patrick Lewis for evaluating our submitted models on the hidden test set. The work described in this paper was partially supported by following projects from the Research Grants Council of the Hong Kong Special Administrative Region, China: CUHK 2300174 (Collaborative Research Fund, No. C5026-18GF); CUHK 14210717 (RGC General Research Fund).
Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv:1605.07683.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from Wikipedia. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1907-1917, Melbourne, Australia. Association for Computational Linguistics.
Yifan Gao, Piji Li, Irwin King, and Michael R. Lyu. 2019. Interconnected question generation with coreference alignment and conversation flow modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4853-4862, Florence, Italy. Association for Computational Linguistics.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural tuning machines. arXiv preprint arXiv:1410.5401.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the world state with recurrent entity networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. 2019. Attending to future tokens for bidirectional sequence generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1-10, Hong Kong, China. Association for Computational Linguistics.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1468-1478, Melbourne, Australia. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Roktaschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
Abhishek Sharma, Danish Contractor, Harshit Kumar, and Sachindra Joshi. 2019. Neural conversational qa: Learning to reason v.s. exploiting patterns. ArXiv, abs/1909.03759.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu. 2016. Memory-enhanced decoder for neural
machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 278-286, Austin, Texas. Association for Computational Linguistics.
Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808-819, Florence, Italy. Association for Computational Linguistics.
Zheng Zhang, Minlie Huang, Zhongzhou Zhao, Feng Ji, Haiqing Chen, and Xiaoyan Zhu. 2019. Memory-augmented dialogue management for task-oriented dialogue systems. ACM Transactions on Information Systems (TOIS), 37(3):34.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458-1467, Melbourne, Australia. Association for Computational Linguistics.
Victor Zhong and Luke Zettlemoyer. 2019. E3: Entailment-driven extracting and editing for conversational machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2310-2320, Florence, Italy. Association for Computational Linguistics.
A.1 Model Variant
In our preliminary investigation, we did not add the entailment state prediction subtask (Eqn.9 & 10) in Section 2.3. Consequently, there is no sentence-level entailment score in Eqn.11 for coarse-to-fine reasoning. Instead, we tried to predict the underspecified rule sentence separately and treat it as the sentence-level score:
$$
\zeta_ {i} = \mathbf {w} _ {\zeta} \left[ \mathbf {k} _ {i}; \mathbf {v} _ {i} \right] + b _ {\zeta} \in \mathbb {R} \tag {16}
$$
where  $\zeta_{i}$  is the score that determines how likely the  $i$ -th rule sentence contains the underspecified span. Let  $l$  indicate the correct decision and  $j$  be the underspecified rule sentence, the identification loss of the underspecified rule sentence is
$$
\mathcal {L} _ {\text {s e n t}} = - \mathbb {1} _ {l = \text {i n q u i r e}} \log \operatorname {s o f t m a x} (\zeta) _ {j} \tag {17}
$$
On the ShARC hidden test set, it turns out that this model EMT (prev.) performs slightly better on the decision making part but worse than EMT for the question generation task. However, it is hard to balance the decision making performance and the question generation (span extraction) performance for this model EMT (prev.), and thus we develop the current version described in the main paper which introduces an entailment state prediction subtask. Table 5 shows the results of these two models on the ShARC test set.
Models	End-to-End Task
Micro Acc.	Macro Acc.	BLEU1	BLEU4
EMT (prev.)	69.4	74.8	60.9	46.0
EMT	69.1	74.6	63.9	49.5
Table 5: Results of EMT and its previous version on the hidden test set of ShARC.