SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The ability to apply pragmatic reasoning is crucial for effective communication and achieving a thorough grasp of the natural language system of human beings. To address this, we propose conversational question-answering, wherein multiple questions are formulated for each dialogue, and an example is shown in Fig. 5. The questions focus on delving deeper into dialogues, often needs insights into intended meanings to answer. ChatGPT plays a pivotal role in question generation and thanks to AMT, we can ensure the collection of high-quality answers. Ultimately, 19, 482 question-answer pairs are assembled.
Question Generation ChatGPT is employed to generate questions with prompts consisting of dialogues and human
A: Finally, Ron, lots of talk about Congress releasing the second half of this \(700 billion bailout this week. Where do we stand with that?
B: Quite possible that Congress will get that done this week now that Barack Obama has asked George Bush, has asked the current president, to formally put in a request for that money. Congress has got a lot of questions about how this money is going to be spent, as it has questions about how the first half of the money was spent.
Rationale: Barack Obama's request for the $700 billion bailout may expedite the process.
$\mathbf{Q}_1$  : What may expedite the process?
$\mathbf{A}_1$  : Request
\(Q_{2}\): Which president-elect requested the \\(700 billion bailout to be released?
$\mathbf{A}_2$  : Obama
Figure 5: Conversational Question Answering example.
annotated reasons. We task it to generate questions challenging for individuals who are unaware of the dialogues' intended meanings. More than that, for the convenience of evaluation, the question is also asked to be able to answer within one or two words. Furthermore, to ensure diversity, we instruct ChatGPT to start the questions with "What", "Which" or "How". A preliminary assessment is carried out by sampling a few examples out of the question pool to guarantee quality.
Answer Collection The answers to the questions are obtained through the utilization of AMT. Each worker is provided with a dialogue along with several associated questions and it is requested to answer the questions in one single word. To minimize the potential for misinterpretation, we offer an example coming from our dataset, which is annotated by the author itself. Through the process of human annotation, we consistently evaluate the collected data and reject unqualified answers as well as block workers who fail to meet our standards.
Statistical features of Questions Fig. 3b showcases the diverse range of our questions. These questions encompass a variety of sentence structures, starting with interrogative words: What, Which, and How, and possessing a large diversity of the words that follow the interrogative ones. Apart from the questions, the answer set holds a vocabulary size of 8,179, which is also of great diversity and raises a challenge for models.
5.1 Pragmatics Identification and Reasoning
For  $\mathbf{C} \rightarrow \mathbf{P}$ , we partitioned the dataset into distinct subsets for training, validation, and testing. The training set consists of 13,708 examples, surpassing the 1,361 instances in the validation set and the 2,060 instances in the test set in terms of size. Models are trained on the training set and their
performance is evaluated on the validation set after each epoch to determine the optimal checkpoint. The best checkpoint is subsequently loaded for the final evaluation on the test dataset. The evaluation metric employed is the accuracy score, calculated as the ratio of correct predictions to the total number of instances. Similarly, for the task of  $\mathbf{CP} \rightarrow \mathbf{R}$ , the dataset is also partitioned into training, validation, and test subsets. The respective sizes of these subsets are 5,188, 244, and 1,062 examples. The training and evaluation procedures are identical to those of the previous subtask. For  $\mathbf{C} \rightarrow$
Table 3: Pragmatics Identification and Reasoning Results. The numerical results are accuracy scores in their percentage.
C → P	CP → R	C → PR
Random	50	20	10
BERTbase	63.2 ± 1.1	91.3 ± 0.7	50.2 ± 6.8
RoBERTa base	64.4 ± 1.3	92.0 ± 0.4	50.0 ± 11.28
RoBERTa large	63.8 ± 0.0	60.8 ± 0.5	0.0 ± 0.0
GPT-2 base	64.4 ± 0.7	90.9 ± 0.9	13.06 ± 1.1
DialoGPT medium	65.0 ± 0.6	24.5 ± 1.9	3.8 ± 1.5
DeBERTa base	64.9 ± 0.2	92.6 ± 0.6	43.9 ± 1.2
ALBERTbase	65.1 ± 0.4	90.6 ± 0.2	34.9 ± 1.8
PR, the test sets in previous subtasks are taken for evaluation, and it's worth noting that their test sets consist of exactly the same instances. This design ensures the prevention of any leakage of the test set into the training set, thereby maintaining the integrity of the evaluation process.
Results and Analysis We present four key observations for this task:
The primary performance bottleneck lies in the subtask  $\mathbf{C} \rightarrow \mathbf{P}$ . As shown in Tab. 3, the best model achieves an accuracy score of  $92.0\%$  in the  $\mathbf{CP} \rightarrow \mathbf{R}$  subtask, indicating that models possess the capability to reason to some extent. However, when it comes to the  $\mathbf{C} \rightarrow \mathbf{PR}$  task, the best-performing model achieves only  $50.2\%$  accuracy, while the highest accuracy achieved in the  $\mathbf{C} \rightarrow \mathbf{P}$  task is  $65.0\%$ . The substantial difference between the score of  $92.0\%$  and the score of  $65.0\%$  suggests that the difficulty in accomplishing the task primarily stems from the  $\mathbf{C} \rightarrow \mathbf{P}$  subtask.
$\diamond$  Accumulated variance in the  $\mathbf{C}\rightarrow \mathbf{PR}$  subtask. The models exhibit significant variance in the results of the third subtask, which can be attributed to the variance introduced by its constituent tasks.
Figure 6: Models performances of different reasoning types. The taxonomy of reasoning types comes from Fig. 2.
$\diamond$  The significance of pragmatic awareness in language models. Both the  $\mathbf{C} \rightarrow \mathbf{P}$  and  $\mathbf{C} \rightarrow \mathbf{PR}$  subtasks require pragmatic awareness, and the poor performances of the models on these subtasks highlight their limitations in accurately determining the optimal timing for deploying reasoning abilities.
As depicted in Fig. 6, we assess the performance of existing models across five dimensions by leveraging the taxonomy outlined in Fig. 2. It becomes evident that models exhibit a nearly uniform performance across these dimensions, leading us to the inference that pragmatic reasoning constitutes a cohesive task, and a segregated approach is ill-suited for its treatment.
5.2 Conversational Question Answering
Similarly to the previous task, the question-answering dataset is divided into training, validation, and test sets, comprising 15,585, 1,559, and 2,338 instances, respectively. Experimental subjects include BART [54], T5 [55], UnifiedQA [56], and mT5 [57]. The metric adopted is also accuracy score. Given ChatGPT's impressive performance in MMLU and AI2 Reasoning Challenge [13], we further examine ChatGPT's capability in the context of CQA by prompting it to provide one-word answers to questions. However, due to its uncontrollable nature, the generated answers may not always align with our desired settings. Hence, we introduce two evaluation metrics for ChatGPT: (1) em (exact match), which requires ChatGPT to produce the exact same word as our answer, and (2) pm (partially match), where we consider ChatGPT to be correct as long as our answer appears in its generated output. Two configurations are employed for evaluation. In the first configuration, models receive dialogues and questions while remaining blind to human-annotated rationales. The
Table 4: CQA task results with/without human annotated rationales. The numerical results are accuracy scores in their percentage. em: exact match, pm: partially match.
BART-base	T5-small	mT5-small	UnifiedQA-base	UnifiedQA-large	ChatGPT (em)	ChatGPT (pm)
w/o rationale	20.2 ± 1.1	24.9 ± 0.7	19.7 ± 0.4	28.9 ± 2.0	32.8 ± 0	1.0	40.6
w/ rationale	29.6 ± 0.6	34.1 ± 1.4	29.8 ± 0.8	38.8 ± 0.2	42.4 ± 0.6	1.5	45.1
Δ	+9.4 (46.5%)	+9.0 (35.8%)	+10.1 (51.3%)	+9.9 (34.1%)	+9.6 (29.3%)	+0.5 (60%)	+5 (12.3%)
second configuration is a contrasting experiment with human-annotated rationales provided. For each configuration, we run each model three times using different random seeds and report the mean and variance of their results as the final outcomes.
Results and Analysis The experimental results are presented in Tab. 4. Our observations can be categorized into three main aspects. Firstly, the importance of pragmatic meaning is proved. As shown in Tab. 4, there exists a notable disparity between the results of models that have access to human-annotated answers and those that do not. On average, the model performances improve by  $38.47\%$  after the introduction of human-annotated rationales. Even the lowest-performing model in the initial experiment, mT5-small, demonstrates a  $9.4\%$  increase in accuracy. The substantial discrepancy in results between the two configurations underscores the significance of elucidating intended meanings in the development of effective communicators. Second, the models display deficiencies in applying pragmatic reasoning. Our questions are designed to demand a deeper understanding of conversations, however, the models struggle to perform well on our task. The best-performing model, ChatGPT, achieves an accuracy score of  $40.6\%$ . It is worth noting that our questions are generated by ChatGPT itself, and our source dataset, Interview, was proposed prior to the emergence of ChatGPT, which means that ChatGPT may have encountered our text during training. These characteristics render its result unsatisfactory. Third, generalization across different types of pragmatic reasoning poses challenges. In this analysis, we focus exclusively on models other than ChatGPT due to the lack of clarity regarding its training process. As demonstrated in Tab. 4, these models showcase a substantial improvement in performance following the inclusion of human-annotated rationale. The extent of this improvement exhibits slight fluctuations among the various models, suggesting a shared obstacle that hinders their overall performance. Noticed that the models are fine-tuned on a training set that is 5.2 times larger than the test set, we can conclude that achieving effective generalization from one pragmatic reasoning process to another remains a formidable and challenging task.
5.3 Zero-Shot Natural Language Inference
We conduct the natural language inference (NLI) task [60] to evaluate the model's comprehension of language and to emphasize the importance of context in pragmatic reasoning. Different from previous tasks, zero-shot NLI task sheds a light on models' initial ability as they are tested without finetuning. The task involves providing two sentences: a premise and a hypothesis, and models are required to determine the relationship between the two sentences, which can be entailment, contradiction, or neutral. As there are no negative samples in our dataset, we simplify the task by asking the model to judge whether there is entailment. In this task, models are presented with a dialogue, a turn of the dialogue, and an intended meaning, they need to judge whether the turn entails the intended
meaning. Noticed that collected data as described in Sec. 3.1 consists of reasons and implied meanings, to better fit our task, we abandon the reasons and preserve the implied meanings. Models are tested under zero-shot setting, which means that they are not allowed to train on any data before testing. Thus the innate abilities of models play a decisive role. Baseline models include T5 [55], DeBERTa [61], and ChatGPT. It's important to note that ChatGPT and the other two models are tested on different settings. ChatGPT is tested with the whole dialogue and the implied meaning as a prompt. However, to inspect the significance of context, the other two models are only provided with the pragmatic turn and the corresponding pragmatic meaning. ChatGPT is evaluated using two types
Table 5: Results of Natural Language Inference Task.  $^\circ$  T5-XXL fine-tuned on true NLI mixture [58].  $^\dagger$  DeBERTaV3 trained on MNLI [25] and SNLI [59].
Method	Acc.
Random	50.0
DeBERTa-Large†	44.3
T5-XXL°	45.3
ChatGPT	85.7
ChatGPT w/ CoT	63.8
of prompts: with and without step-by-step instructions. "Let's think step by step" is a prompt discovered [62] to improve the model's reasoning ability; refer to Appendix C for more details.
Results and Analysis Results are listed in Tab. 5. As this task shares similar settings as binary classification, randomized answer accuracy is expected to be  $50\%$ . We observe below randomized performance on some previous SOTA models. Note that, each of the data is annotated by two humans, thus it's reasonable to view human performance as  $100\%$ . ChatGPT achieves the highest result but still shows a huge gap compared with human annotations. The outcomes highlight the imperfectness of the models' reasoning abilities. For T5-large and DeBERTa, context is blinded, but for ChatGPT, it is reachable. Hence, the performance gap among T5-large, DeBERTa, and ChatGPT shows the importance of context in our task. Interestingly, CoT doesn't offer help to ChatGPT but is harmful to ChatGPT's performance.
6 Discussions and Future Work
In this paper, we propose DiPlomat, a high-quality manually annotated multi-turn dataset of pragmatic reasoning in conversations. Along with the dataset, we propose two tasks and baselines. Comparing experimental results, we emphasize the nonnegligible impact of contexts and reasoning on building perfect communicators. We also highlight the importance of pragmatic awareness and its bottleneck effect on our tasks. There is still a significant disparity between current performances and established standards.
**Memorization vs. Reasoning** Noticed that models exhibit outstanding performance on  $\mathbf{CP} \rightarrow \mathbf{R}$  of Sec. 5.1. On the contrary, for  $\mathbf{C} \rightarrow \mathbf{PR}$ , models achieve poor results. Since the underlying knowledge is consistent for both tasks, the disparity in performance is hypothesized to be attributed to **memorization**. Instead of truly understanding the knowledge, the models tend to memorize patterns.
Subjectiveness vs. Objectiveness Ji et al. [63] emphasize the importance of modeling a distribution that encompasses a diverse range of possibilities, rather than solely relying on a single "best" prediction. During the annotation process, we observe a phenomenon that different workers hold diverse opinions regarding pragmatic turns and their intended meanings. Their annotations often exhibit significant variations, sometimes even presenting completely opposing interpretations. We maintain the possibility of subjectiveness with careful task metric design (Sec. 4.1).
Larger Models vs. Smaller Models Conventional wisdom suggests that larger models, endowed with a greater number of parameters, generally exhibit superior performance in comparison to their smaller counterparts. However, an intriguing observation comes to the forefront within the context of the Pragmatic Identification and Reasoning (PIR) experiment, as elucidated in Table 3: RoBERTa<sub>base</sub> surpasses RoBERTa<sub>large</sub> in performance. We posit that this phenomenon can be attributed to a dual-fold rationale. First and foremost, a divergence emerges between the domain of pragmatic data and the domain of pretraining data for language models. Consequently, larger models exhibit a more substantial and consistent influence of their pretraining data, rendering them less adaptable to domain shifts towards pragmatic data. Secondly, the current landscape of pragmatic data is characterized by its inherent diversity and relative scarcity. This inherent diversity poses a particular challenge for larger models in adapting to such heterogeneity; refer to Appendix A.
Future Work Achieving generalization across multiple types of pragmatic reasoning processes poses significant challenges. Consequently, we propose that the construction of a proficient communicator necessitates the incorporation of methods beyond purely data-driven approaches. Furthermore, the availability of comprehensive evaluation data is of utmost importance. As a result, we target more high-quality datasets and new methods other than data-driven for the problem.
The authors thank Jiaming Yu at Oxford for the preliminary survey, Dr. Xue Feng and Dr. Lifeng Fan at BIGAI for helpful discussions on data collection, Ms. Zhen Chen at BIGAI for designing the teaser figure, Junpeng Li and Dr. Zixia Jia at BIGAI for application experiments. This work presented herein is supported by the National Key R&D Program of China (2022ZD0114900) and the National Natural Science Foundation of China (62376031).
[64] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In Proceedings of Machine Learning Research, volume 202, pages 10421-10430, 23-29 Jul 2023. 17
[65] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2339-2352, Online, June 2021. Association for Computational Linguistics. 17
[66] Rishabh Misra. News headlines dataset for sarcasm detection, 2022. 17
[67] Lennart Wachowiak, Dagmar Gromann, and Chao Xu. Drum up SUPPORT: Systematic analysis of image-schematic conceptual metaphors. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 44-53, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. 17
[68] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. 18
[69] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR), 2020. 18
[70] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. 18
A Hypothetic Explanation for Experimental Results of RoBERTa<sub>base</sub> outperforms RoBERTa<sub>large</sub> 17
B Annotation Details 17
B.1 Details For Automatic Selection 17
B.2 Details For Fine-grained Annotation 17
B.3 Details on Human Refinements 18
C Experimental Detail 18
C.1 Pragmatic Identification and Reasoning (PIR) 18
C.2 Conversational Question Answering (CQA) 19
C.3 Zero-Shot Natural Language Inference 19
D More Detail on DiPlomat 20
E Grice Maxims and Pragmatic Reasoning 24
F Computational Resources 24
G Limitations & Negative Societal Impacts 24
H Ethics Concern 24
I Responsibility & Dataset Liscence 25
J Datasheets for Our Dataset 25
J.1 Motivation 25
J.2 Composition 25
J.3 Collection Process 26
J.4 Preprocessing/cleaning/labeling 27
J.5 Uses 28
J.6 Distribution 28
J.7 Maintenance 29
A Hypothetic Explanation for Experimental Results of RoBERTa<sub>base</sub> outperforms RoBERTa<sub>large</sub>
Conventional wisdom suggests that larger models, with more parameters, tend to outperform smaller models. However, as indicated in Table 3, a notable phenomenon emerges in the context of the PIR experiment: RoBERTa<sub>base</sub> outperforms RoBERTa<sub>large</sub>. This unexpected observation leads us to formulate a two-fold hypothesis:
1. domain discrepancy between pragmatic data and LM's pretraining data: Larger models exhibit a consistent influence of their pretraining phase, whereas smaller models are more susceptible to adaptation during the fine-tuning process. Importantly, beyond the application of semantic knowledge, our pragmatic data introduces deeper challenges for language models, leading smaller models to potentially "superficially" memorize patterns from the fine-tuning data. In contrast, larger LMs tend to adhere more closely to their original pretraining understanding. Similar trends have been observed in other domains, such as multi-step math reasoning [64] and few-shot learning [65].
2. Low-resource and diversity of pragmatic data: Pragmatic data is both scarce and diverse. Effectively tackling this data needs language models to move beyond mere semantic pattern memorization and engage in more nuanced reasoning over contextual information, social common sense, and even theory-of-mind reasoning. Such low-resource conditions challenge the conventional fine-tuning requirements for large language models, which typically benefit from substantial data for achieving superior performance.
To empirically substantiate these hypotheses, we undertake a zero-shot version of Task 1 (PIR). The results of this zero-shot experiment are reported in Tab. 6, and when compared to the results presented in Table 3, it becomes evident that the improvement in performance of  $\mathrm{RoBERTa_{base}}$  following fine-tuning is considerably more pronounced than that of  $\mathrm{RoBERTa_{large}}$ .
Table 6: Zero-Shot PIR
C → P	CP → R	C → PR
RoBERTalarge	61.1	21.8	1.7
RoBERTabase	57.2	21.5	5.1
B Annotation Details
B.1 Details For Automatic Selection
Different methodologies are employed to address various pragmatic phenomena. To leverage prior advancements in the field, we begin by segmenting each dialogue into individual utterances. Subsequently, we employ two distinct approaches, namely string matching and pretrained model classification, to identify these phenomena within our source data. In the case of scalar implicature, which exhibits a noticeable pattern characterized by word pairs such as (some, all) appearing in adjacent turns of dialogues, we employ string matching to annotate instances of scalar implicature in conversations. Similarly, for popeq implicature, which often features a continuous question mark, we utilize this characteristic as a means of detection. With regards to idioms, which exhibit more evident patterns, we employ the idiom set proposed by Saxena and Paul [18] to conduct searches. For other types of phenomena that lack obvious patterns, we leverage a pretrained RoBERTa base model [49], and fine-tune it for our specific task. The sarcasm dataset by Misra [66] is used for finetuning the sarcasm model, the MOVER dataset by Zhang and Wan [42] for hyperbole and the ColBERT dataset by Annamoradnejad and Zoghi [20] for patronomasia. Several models have been proposed for metaphor detection, thus we utilize an existing model [67] specifically designed for metaphor identification.
B.2 Details For Fine-grained Annotation
AMT is integral to our process. To ensure clarity and consistency, we provide explicit instructions to the workers. Additionally, to further elucidate the objectives of our study, we offer illustrative
examples. The task itself is presented below the instructions and examples, with the dialogue and corresponding turn numbers provided for workers to select. Furthermore, as workers check a checkbox, we prompt them to select a confidence score and provide a rationale. In order to strike a balance between our budget, the quality of annotations, and the speed of annotation, we have determined the compensation of \(0.1 per completed task. After the annotation process, we collect responses that are assigned with a confidence score of 4 or higher.
Specifically, we surveyed 10 users to accomplish our task. All users can complete a single task within 45 seconds, leading to a wage pay of around 8 dollars per hour, which is about a dollar higher than the federal minimum hourly wage of the United States.
B.3 Details on Human Refinements