SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Disturbing choices are chosen based on the BERTScore metric [51]. The rationale with the highest similarity, as determined by other dialogues, is selected and included in the pool of candidate options. The instructions provided to the workers align with those used for Fine-grained Annotation, wherein they are also instructed to assign a confidence score to their responses. The remuneration for workers is set at $0.05 per task.
AMT Workers Requirements In order to guarantee the quality of annotated data, the qualification rules for workers are strict and can be found in Tab. 7.
Table 7: AMT workers requirements
CountryIn	United States, Canada, Great Britain, Australia, Singapore, Ireland, New Zealand
# Tasks approvedGreater Than Or EqualTo	1300
Tasks approved RateGreater Than Or EqualTo	95%
C.1 Pragmatic Identification and Reasoning (PIR)
BERT<sup>base</sup> [68] BERT (Bidirectional Encoder Representations from Transformers) is a revolutionary language representation model that has had a significant impact on natural language processing (NLP) tasks. It has achieved remarkable performance across various NLP benchmarks, including question answering, sentiment analysis, named entity recognition, and many others. Its birth brings profound influence on pretrained language models.
RoBERTa<sub>base</sub> & RoBERTa<sub>large</sub> [49] RoBERTa improves upon BERT by incorporating enhancements such as larger and more diverse training data, longer pretraining duration, dynamic masking, and advanced training strategies. These improvements enable RoBERTa to achieve even better performance on a wide range of NLP benchmarks. While BERT paved the way for contextualized representations in NLP, RoBERTa further refines and pushes the boundaries of language understanding, making it a powerful and preferred choice for many researchers and practitioners in the field.
ALBERT<sup>base</sup> & ALBERT<sup>large</sup> [69] ALBERT (A Lite BERT) is a highly efficient and compact variant of the BERT model that addresses the computational limitations of the original architecture. It incorporates parameter-reduction techniques to alleviate training time constraints and achieve improved performance compared to BERT.
DeBERTa<sup>base</sup> [61] DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a state-of-the-art language representation model that builds upon the BERT architecture and introduces several key innovations, including disentangled attention mechanism. The performance of DeBERTa has been demonstrated to surpass that of BERT on a wide range of NLP tasks.
GPT2<sub>base</sub> [70] Leveraging transformers decoder, Radford et al. [70] proposed GPT2. It represents a significant breakthrough in natural language processing and generation. One of the most notable features of GPT-2 is its ability to generate coherent and contextually relevant text. Through unsupervised pretraining on a large corpus of internet text, GPT-2 learns to predict the next word in a sequence of text, enabling it to generate human-like responses.
Table 8: Hyperparameters for models on  $\mathbf{{CP}} \rightarrow  \mathbf{R}$
Model	learning rate	batch size	weight decay	epochs
BERTbase	5e-5	12	0.001	50
BERTlarge	5e-5	12	0.001	50
ALBERTbase	5e-5	12	0.001	50
ALBERTlarge	5e-5	12	0.001	50
DeBERTabase	5e-5	12	0.001	50
RoBERTabase	5e-5	12	0.001	50
RoBERTalarge	5e-5	12	0.001	50
GPT2base	0.001	8	0.01	50
DialoGPTmedium	0.001	2	0.01	50
Table 9: Batch size for models on  $\mathbf{C} \rightarrow  \mathbf{P}$
Model	Batch Size
BERTbase	80
ALBERTbase	24
ALBERTlarge	24
DeBERTabase	24
RoBERTabase	80
RoBERTalarge	24
GPT2base	24
DialoGPTmedium	8
DialogGPT<sub>medium</sub> [35] DialogGPT is dialogue-oriented GPT. It builds upon the GPT architecture and extends it to support interactive conversations. DialogGPT is trained in a supervised manner using a dialogue dataset, which allows it to understand and generate responses in a conversational context.
The PIR task encompasses three distinct settings:  $\mathbf{C} \to \mathbf{P}$ ,  $\mathbf{CP} \to \mathbf{R}$ , and  $\mathbf{C} \to \mathbf{PR}$ . In the  $\mathbf{C} \to \mathbf{P}$  setting, models are trained for 20 epochs, employing a batch size as indicated in Tab. 9, a learning rate of  $2e - 5$ , and weight decay of 0.01. As for  $\mathbf{CP} \to \mathbf{R}$ , the hyperparameters adopted are listed in Tab. 8. For the  $\mathbf{C} \to \mathbf{PR}$  setting, there is no training required; instead, we simply load the best checkpoint obtained from the previous training for this task. The concrete implementation is as follows: we initially flatten the test dataset of  $\mathbf{C} \to \mathbf{P}$ , ensuring that each instance contains both a dialogue and a pragmatic turn extracted from the same dialogue. As for the test dataset of  $\mathbf{CP} \to \mathbf{R}$ , no modifications are made. It should be noted that, following the processing steps, both datasets own the same dialogues and corresponding pragmatic turns, resulting in identical instance numbers. For an instance to be deemed correct, the models must successfully accomplish both component tasks i.e. succeed in Identification and Reasoning.
C.2 Conversational Question Answering (CQA)
CQA ChatGPT was instructed to generate questions for our tasks. The prompt template that starts the questions with "Which" is depicted in Tab. 10. Through this methodology, we collected a total of 19,482 questions. To ensure the reliability of the answers provided to these questions, AMT is utilized. In our experiment, the hyperparameters adopted are illustrated in Tab. 11. To assess the performance of ChatGPT, we conducted testing using the template outlined in Tab. 13.
C.3 Zero-Shot Natural Language Inference
Details are provided as follows. T5-XXL, and DeBERTa-v3 are tested with the pragmatic turn as premise and implied meaning as a hypothesis. The context is out of reach for these models. In contrast, as shown in Tab. 10, ChatGPT is given the context, and the red line labeled "Think step by step" represents two distinct configurations: one with step-by-step and one without it.
Table 10: ChatGPT question generation template: using "Which" to start the question.
You are sensitive and always view others' words as having some implied meanings. For the dialogue between "A" and "B" in this task, we have offered a statement that is the implied meaning of a turn, please only offer one reading comprehension question that can be answered with only one word based on the dialogue and mostly focuses on the turn the statement mentions. The question will be tested by only by viewing the dialogue, so please make the question hard enough that it's impossible to answer without viewing the statement. Use "Which" to ask the question! Following is the dialogue: {dialogue} Following is the statement: {statement} Use "Which" to ask the question! And please make the question hard enough that it's impossible to answer without viewing
Table 11: Hyperparameters for models on CQA.
Training Epoch 50  
Learning Rate  $5.6e - 5$   
Batch Size 24  
Weight Decay 0.001
Table 12: Test ChatGPT: answer questions with only one word.
For the dialogue between "A" and "B" in this task, please answer a question according to the dialogue with only one word Following is the dialogue: {dialogue} Following is the question: {question}
Table 13: ChatGPT test template of Zero-Shot CoT
This is a natural language inference task. Given the dialogue context: {context} Does {pragmatic turn} entails {implied meaning}? Reply 'entails' or 'not entails'. Think step by step.
D More Detail on DiPlomat
In this section, we will propose more examples of our dataset in Tab. 14, Tab. 15, Tab. 16, Tab. 17, and Tab. 18.
Table 14: Contextual reasoning examples of DiPlomat
A: Yeah. They say that he's the fastest pitcher there ever was. It's just he really couldn't find home plate. I mean, some of the stories you learn about this guy, it reads like fiction. When he was - I think this is around 1960. He's pitching in the minor leagues, and he pitched so fast he ripped the man's ear off.
B: Oh.
A: Yeah.	Rationale: The literal meaning is a simple expression of agreement, while the implied meaning is that the speaker is amazed by the story of Steve Dalkowski's feats.
B: We're talking about 2.8 million people. Has the rise of temporary workers figured into, at least, the statistical improvement of the U. S. economy for some people?
A: It has. Overall, about one seventh of the total job growth has been in the temp sector. The temp sector is growing nine times faster than the overall private sector as a whole. And the 2.9 million workers represents a record number, both in the number of temp workers and in the percentage of the economy that they make up.
B: You know in "Harvest Of Shame," Edward R. Murrow very famously said, the people we're showing you in this documentary have picked your Thanksgiving bounty with their bare hands, and this is how they live.	Rationale: The implied meaning of this turn is to reflect on our reliance on temporary workers in our day-to-day lives.
A: And so I got up and ran. And it wasn't too far. But I just - at that moment, I thought, I don't want to be shot in the back, and I need to find some cover. And there's really no place to hide. But there are these
B: You found a little, like, alcove that you could duck into.
A: There was a little alcove, yeah. And I just made myself as small as I could in that little corner.	Rationale: The speaker tried to protect itself from danger.
A: Well, there's a big argument in the United States about this. There's one group of folks who think that engagement policy failed. We engaged with China from 1979 until about 2013 when Xi Jinping came into power. And the idea of engagement was that coevolution was in the American interest as well as in China's interest. And you could bring China along to be a responsible player to some degree.
A: Many hardliners in the United States government - and outside and including in the expert community - now claim that engagement was a sucker's game and that we have raised up a tiger which could now devour us. But there are different schools of thought about this, and many of us think that we still need to engage with China, albeit more strategically.
B: That image of raising a tiger that will devour us is very dramatic.	Rationale: The situation is not necessarily an 'either/or' between China and the United States.
Table 15: Figurative language reasoning examples of DiPlomat
A: Thank you. How are you?
B: I'm pretty good. Thank you. You must be stuck like glue on this, but, you know, you've played in three World Cups, including one of the wins for the U. S. team in 1999. How would you describe what it's like to be out there on that field in that final game?	Rationale: Stuck like glue means to be attached to something, which is a particular issue or a person.
B: So in terms of what to do about it, we've said Twitter and Facebook have shut down these accounts, which prompts me to wonder - does shutting down a fake account do that much? Can't the Chinese government, if it's determined to go down this path, just open up two new ones in place of the one that was closed?
A: It is a cat-and-mouse game, and the companies are constantly trying to get ahead of it. [···] As you said, they can always set up new accounts.	Rationale: Mice are constantly trying to get away from cats and cats are constantly trying to catch mice. In the same way, the Chinese government will always be trying to escape restrictions on social media accounts and media companies will always be trying to find fake accounts.
A: I really didn't feel safe because the Turkish government is very famous for hunting down those who oppose Erdogan. So, I mean, I just didn't want to really risk my life by going to Europe. But, you know, I talked to my team. I told them all, like, how many times I want to come because I want to be with you guys there, and I want to get a win with you guys. And then, later on, they came back with the news and said, you know what?I think the best decision is if you don't come. Let's just not risk it for one game.
B: Do you feel safe in New York and elsewhere in the U. S.?
A: I have been getting last two, three days hundreds death threats, but I think I feel safe in America. But anywhere else in the world, I wouldn't really feel safe.	Rationale: He is implying that he is still not safe.
Table 16:Commonsense reasoning examples of DiPlomat
B: Yeah - African-American mayor from Tallahassee.
A: Yes. So this is sort of a test of whether real progressive candidates can win in these sort of purplish states. [...]	Rationale: "Purplish" states are not really colored. They refer to US states that are neither clearly Republican (red) nor Democrat (blue) in their voting.
B: He wrote a lot of letters by hand, didn't he?
A: He wrote tons of letters. I bet there are a hundred thousand - hundreds out there[...]	Rationale: tons of letters implies a very large number and not to full a ton.
B: Well, Pluto's official designation is a dwarf planet. And I have to tell you the people who sent this probe all the way out to Pluto are a little angry about that because when they launched it a decade ago, Pluto was still a planet.
A: (Laughter)
B: It got downgraded in the intervening years.
A: That seems so unfair.	Rationale: A is expressing sympathy for the people who sent the probe, showing that they understand why they feel so disappointed.
Table 17: External knowledge reasoning examples of DiPlomat
B: Inside of his house, family pictures decorate the walls and the fridge. Les has 15 great grandchildren. He grew up in an orphanage, and he couldn't wait to leave to join the military. And so in early 1944, he boarded a ship and crossed the Atlantic Ocean to go to the frontline.
A: I loved that sailing on, of course. It was so dramatic. You could see all these ships bobbing up and down on the ocean. And destroyers were weaving in and out of them to make sure they uncovered any mines or anything.	Rationale: Sailing across the ocean during wartime was a perilous experience.
A: . . . equivalent to a nuclear bomb?
B: Well, it's about - its equivalent - the energy in that explosion is about 10 times the energy in the first atomic bomb. . .	Rationale: The energy released in the explosion is incredibly pow-erful.
B: So in your polling, in your research, do you find that it's going to come down to maybe a couple thousand votes from these unaffiliated voters and on what issues?Or will they vote?
A: It is likely at the moment to be a very narrow victory. President Bush won in 2004 with five percent. That was 100,000 votes. In other words, if it is one percent, that would be 20,000 votes, and right now, the polls are moving around in just single percentage points. So it could be that narrow.
B: Now, I have read that Colorado is going to be this year's Florida and Ohio, that this is going to be the state that decides the election.
A: I think it could be, and the interesting thing is that Obama and Palin were both in Jefferson County a couple of days ago, indicating that there may be actually even a county that could be looked at to be beyond an entire state.	Rationale: The turn is suggesting that the county of Jefferson in Colorado could be a key factor in deciding the election, despite the fact that it is only one of many counties in the state and there are other swing states in the elec-tion.
Table 18: Others examples of DiPlomat
A: There's that feeling - I mean, so many of us have parents in the industry. I mean, that's what this region is about, especially around Detroit, and Wayne State's in Detroit, the heart of Detroit. So, it's nerve-racking. Everyone is nervous. Everyone doesn't know what's going to happen next. We're all watching the news very closely. But at the same time, it's interesting, because with my generation, we almost seem to, kind of, not be as directly impacted. I mean, our family is, it puts stress on us, but the day to day of the university and the day to day at school doesn't seem to have changed that much.
B: I understand you have friends there who are engineering majors. Do they have any sense of what their future looks like, and will be it there in Michigan?
A: Everybody is secure in their choices and secure in their decision. Everybody thinks that the industry will come around, especially now with the news that GM is getting money from the government. And everybody is more hopeful, and I mean, the auto industry has always been one of the largest industries and a staple in America, and to think that that industry is just going to vanish, nobody is willing to concede that.	Rationale: A believes that the auto industry will not vanish despite the current situation
B: In the meantime, what more have you learned in your reporting about the death of Carlos Hernandez Vasquez?
A: Well, a couple of things. One thing that really stands out is that Carlos Hernandez Vasquez died in a Border Patrol station. The previous migrant children who died were taken to the hospital first; Hernandez Vasquez was not even though immigration authorities clearly knew that he was sick. He was diagnosed with the flu by a nurse practitioner.	Rationale: The death of Carlos Hernandez Vasquez could have been prevented if he had been taken to the hospital.
B: So, how do you and the retired general, James Jones, know each other?
A: My gosh, I think - I can't even remember when I first met him. It's been so long ago. I'm sure I met him when he was head of the legislative liaison over the Senate. But I really became acquainted with him when he became a brigadier general, and, of course, I followed his career. Of course, he served very ably as a commandant in the marine corps and then as the European commander, just been with him from time to time. And I just consider him a very good friend.	Rationale: A has a high opinion of James Jones' character and career.
E Grice Maxims and Pragmatic Reasoning
The Gricean maxims have garnered substantial attention as a foundational theory within the domain of pragmatics. This theoretical framework comprises four distinct maxims: (1) The Maxim of Quality, (2) The Maxim of Quantity, (3) The Maxim of Relevance, and (4) The Maxim of Manner [16, 15]. In contrast to rigid rules or theorems, the Gricean maxims, which capture the prevalent dynamics of conversations, are susceptible to frequent breaches in the context of human communication. These breaches, stemming from the intricacies of real-world interaction, notably manifest in the violation of one or more of these maxims. Such breaches, aligned with the cooperative principle, give rise to pragmatic phenomena that necessitate the engagement of pragmatic reasoning by recipients of the communication [15].
For our experiment, we utilized two A100s and one 3090. The majority of our experiments were conducted on the A100s, while for practical reasons, only Unified-QA-base, BART-base, and T5-small were tested on the 3090. It is important to mention that each experiment was run on a single GPU. We record the training time of models in Appendix F.
Model	C → P	CP → R	Device
BERTbase	0.8min/epoch	0.9min/epoch	A100
RoBERTa base	0.8min/epoch	0.9min/epoch	A100
RoBERTa large	2.5min/epoch	2.8min/epoch	A100
GPT-2 base	5.8min/epoch	6.2min/epoch	A100
DialoGPTmedium	2.4min/epoch	4.2min/epoch	A100
DeBERTa base	0.9min/epoch	0.9min/epoch	A100
ALBERTbase	0.5min/epoch	0.8min/epoch	A100
Table 19: Training Time of Models
G Limitations & Negative Societal Impacts
We acknowledge two limitations in our study: bias and subjectivity. Since our dialogues primarily stem from an interview dataset, a considerable focus is placed on political topics. This is reasonable, as pragmatic phenomena frequently emerge in the statements of politicians to advance their specific goals. However, this focus introduces a certain degree of bias into our dataset. The second limitation relates to the absence of subjectivity. In our methodology, the data undergoes two stages of human annotation, ensuring higher quality and objectivity. However, pragmatic reasoning is inherently subjective, and prioritizing objectivity compromises the preservation of subjectivity, resulting in a limitation in terms of subjectivity coverage. Our dataset exhibits minimal negative societal impacts. This is primarily due to the fact that our dialogues are transcriptions of publicly available TV shows, which inherently limits the potential for negative effects.
H Ethics Concern
Were any ethical review processes conducted (e.g., by an institutional review board)? Our dataset does not involve human subjects. The dataset has undergone the institute's internal ethical review process.
Does the dataset contain data that might be considered confidential? No, the data source derives from an existing public interview dataset.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. Few of the dialogues may talk about offensive topics.
Does the dataset identify subpopulations (e.g., by age or gender)? Not explicitly.
Is it possible to identify individuals (i.e., one or more natural persons) directly or indirectly (i.e., in combination with other data) from the dataset? Yes, our data contains the names of celebrities.
I Responsibility & Dataset Liscence
We bear all responsibility in case of violation of rights and our dataset is under the license of CC BY-NC-SA (Attribution-NonCommercial-ShareAlike).
J Datasheets for Our Dataset
J.1 Motivation
1. For what purpose was the dataset created? (Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.)
This dataset was created to study pragmatic reasoning in dialogues, a specific gap is mentioned above in Appendix G.
2. Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
This dataset was created by the authors of this paper.
3. Who funded the creation of the dataset? (If there is an associated grant, please provide the name of the grantor and the grant name and number.)
The institute of the authors funded the creation of the dataset.
J.2 Composition
1. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? (Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.)
An instance of our dataset represents a piece of dialogue. Description is provided in our paper.
Our dataset owns 4,177 dialogues.
3. Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? (If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).)
It is a sample of all possible cases. As pragmatic phenomena aren't proved to be limited, we can't guarantee a full sampling of them.
4. What data does each instance consist of?
We mention it in our paper.
Yes. The description is in our paper.
6. Is any information missing from individual instances? (If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.)
No. We leverage the original dialogues.
7. Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? (If so, please describe how these relationships are made explicit.)
No. Instances are weakly related, but focus on the same phenomenon.
8. Are there recommended data splits (e.g., training, development/validation, testing)? (If so, please provide a description of these splits, explaining the rationale behind them.) Yes. We provide it.
9. Are there any errors, sources of noise, or redundancies in the dataset? (If so, please provide a description.)