SlowGuess/NAACL_2021_OCR huggingface dataset:
The annotation guidelines for collecting query rewrites on the MuDoCo dataset are provided in the following pages. Note that we annotate the rewrite label for every utterance, including the system response, even though they are not used in our experiments.
Dialogue	usr: Answer the call.
Context	sys: Its Sana and Erica, however, Deirdre on the other line and said its an emergency.
User Query	Very well, cancel the first one and put her through.
Rewrite Label	Very well, cancel the call with Sana and Erica and put Deirdre through. (one -> call, her -> Deirdre)
seq2seq+pg	Very well, cancel the first message and put her through
QR-only	Very well, cancel Deirdre and put her through.
Joint	Very well, cancel the call from Sana and Erica through. (one -> call, her -> Sana)
Table 8: A complex dialogue example where all systems fail to rewrite correctly. Ground-truth and prediction of coreference are appended correspondingly. Rewritten parts are highlighted in bold.
MuDoCo Grading Guideline
In this project, you will be given a conversation between a user and a virtual assistant. Certain parts of the utterances (both user and assistant) might require previous context to be fully understood. The goal of the project is to make minimal changes to the current turn so that it can be understood independently, without needing access to the prior context. This may mean one (or more) of several things:
- Replacing a pronoun with its full referent
- Spelling out the content of an elided phrase
- Adding elements mentioned in the prior context that are now a part of the understood Common Ground of the conversation.
Here is a simple example to get us started. This is a multi-turn conversation that contains a referring expression "that week". If we only had access to the "current" turn, we wouldn't be able to resolve the meaning of "that week"; also, we would incorrectly resolve the intended location of the question to the speaker's location instead of to Sao Paulo, as the previous context makes clear is actually intended.
User: Will next week be a good time to vacation in Sao Paulo?
Assistant: There will be thunderstorms and a lot of rain.
User: What will the temperature be like that week? -- current
Our goal in this project is to rewrite those parts of the current turn that require context. Here, we can replace "that week" with "next week", and insert "in Sao Paulo" to allow the correct location to be inferred from the request.
User: Will next week be a good time to vacation in Sao Paulo ?
Assistant: There will be thunderstorms and a lot of rain .
User: What will the temperature be like next week in Sao Paulo? --- current
Notice that we do not otherwise alter the rest of the utterance, beyond these minimal changes.
Case by Case Guidelines
We introduce our reference / ellipsis resolution guideline in the following categories:
1. Do rewrite ellipsis as well as references
2. Do not paraphrase or summarize but do use the phrasing that appeared in the context
3. A special case exception: calls, messages, reminders
4. Multiple references
5. Data errors
To formally define what an appropriate "resolution" of a reference / ellipsis is, we introduce the concept of minimal changes, which has the following properties we will illustrate with examples below:
1. You may add information that was explicitly uttered in one of the previous turns of the conversation.
2. You should not add more information than necessary to make the utterance understandable without additional context, even if such information is available.
3. Special case exception: references to calls, messages, and reminders don't need additional resolution beyond reference to "the call", "the message", or "the reminder".
Do rewrite ellipsis as well as references
As we can see, the intended time of the current utterance is understood to be "early next week", since that is the time that is added to the common ground in a previous turn in the conversation as relevant to this weather inquiry. However, if we don't have this information, we would naturally interpret the time of the current turn as "now". Consequently, we make a minimal change to the current turn by inserting this time information:
In this case, the user elided the entire clause "is it going to snow", which we should recover:
We may also see cases of noun phrases with some form of ellipsis. These are not implicit arguments but rather explicitly mentioned, but have some partially missing information. In such a case we want to fill those out according to our rules to ensure that they are recoverable based on the current turn alone:
The relevant NP here is "the version", which we want to spell out explicitly as "the version of Somewhere Over the Rainbow":
Do not paraphrase or summarize but do use the phrasing that appeared in the context
Do not rephrase / summarize. Instead, stick to the phrasing that appears in the context as much as possible. We give several examples of this below.
User: What is the latest news on the Brewers ?
Assistant: There are two recent stories. One is about their lose last night, and the User: What happened to him?
Assistant: He stepped on his Jesus Aguilar foot .
User: Then what happened?
Assistant: The Brewers bench cleared because they were mad at him .
User: Can you send more about this to my screen? -- current
In this case, the desired behavior is to simply resolve "this" into "this story", rather than providing a paraphrase that collects information from all of the previous turns to create a phrase that wasn't actually uttered in the conversation at all. There is a single story ongoing in this context, so there's not need for a fuller description of this entity:
Desired: Can you send more about this story to my screen?
Undersired: Can you send more about The Brewers bench cleared because they were mad at
User: Can you play Moonlight Sonata by Beethoven?
Assistant: Yes, you have the live recording of Moonlight Sonata.
User: I dont want that version. Can you search for the best version on the recor
Assistant: Yes, the best version is from 1914 and can stream from YouTube to your play
User: Okay, stream and save to my playlist under Relaxing.
There is an implied argument here (roughly, stream  $\underline{it}$  and save  $\underline{it}\ldots$ ), which we can recover from the context. Here, we choose the simplest form we can use to complete the utterance, and do not add information across turns.
Desired: Okay, stream the Moonlight Sonata by Beethoven and save it to my playlist
Undersired: Okay, stream the 1914 version of Moonlight Sonata by Beethoven from YouTuk
A special case exception: Calls, messages, reminders
For these entities, you don't need to resolve nominal ellipsis or write out its content in full, just call these entities "the call", "the message", "the reminder". This is an exception to the rules above - there is no need to provide additional information even if it present in the exchange.
User: Who is calling me right now?
Assistant: Marty and Brad are calling you via conference call.
User: Answer the call but change it to voice only. --- current
Reference is already made to "the call", and there is only one call referenced in this exchange, so nothing needs to be done here:
Desired: Skip
Undersired: Answer the conference call from Marty and Brad but change it to voice only
Likewise, there is just one message being discussed here, so there is no need for resolution or added information.
Here we have a pronoun that needs to be resolved to a noun phrase. The simple noun phrase "the reminder" is all that's needed, please don't add information about the content of the reminder, since this reminder isn't ambiguous with any other reminder in the context.
Multiple references
If there are multiple references towards the same entity, you only need to resolve it in the first one. If there is a reference towards an entity in the utterance itself, you don't need to resolve it at all since it's already fully interpretable from that turn alone.
The current turn contains two pronouns referring to the same individuals. Once the first pronoun is resolved, it provides a reference for the second pronoun within the same sentence. Therefore, we only need to resolve that first
instance (in fact, resolving the second pronoun leads to an unnatural paraphrase that should always be avoided):
Desired: Ask Stephen Bradley if he wants in on the office lottery pool.
Undesired: Ask Stephen Bradley if Stephen Bradley wants in on the office lottery pool
User: Set a reminder.
Assistant: Okay, what for?
User: Remind me to call my daughter for her wedding. --- current
The pronoun "her" receives a referent within the current turn (namely, "my daughter"), so there is no need to do anything here. You should skip, and not spell out the referent of the pronoun.
Undesired: Remind me to call my daughter for my daughters wedding.
Data errors
Some times the request itself could be faulty, here's a couple of cases:
When there is ambiguity in the reference / nominal ellipsis, you should skip
User: Who called me while I was in class?
Assistant: You have a missed call from James and also Sam
User: Okay, call him back and set my status to online. -- current
Here since there are two entities, "James" and "Sam", both common male names, it is impossible to determine its referent. You should skip in this case
Undesired: Okay, call James back and set my status to online.