SlowGuess/ACL_2020_OCR huggingface dataset:
The first extraction proportion for SubTle is low because of the prevalence of self-yes-ands in this corpus. Self-yes-ands are prompt and response pairs that evidently originate from the same speaker but otherwise meet the criteria of a yes-and. There are many incorrectly combined dialogue turns that actually come from the same speaker because of the heuristics employed for building SubTle. By providing labeled self-yes-and as negative samples, the classifier quickly learns to remove these self-yes-ands, leading to a significantly higher proportion of yes-ands in subsequent iterations. This is demonstrated in the specifics of the additional iterations, which are shown in Table 6.
The objective of this task is to collect a corpus consisting of "Yes, and..." type dialogue pairs (henceforth simply "Yes, and"), which will be used to train neural network models for creative language generation. For each of these HITs, you are expected to identify all the "Yes, and"s you hear from an improv session within a podcast episode.
What is a "Yes, and"?
"Yes, and..." is a rule-of-thumb in improvisational comedy that suggests that a participant in a dialogue should accept what another participant has stated ("Yes") and then expand on that line of thought or context ("and..."). In short, a "Yes, and" is a dialogue exchange in which a speaker responds by adding new information on top of the information/setting that was constructed by another speaker.
Note that a "Yes, and" does not require someone explicitly saying 'yes, and...' as part of a dialogue exchange, although it could be the case if it agrees with the description above. There are many ways in which a response could implicitly/explicitly agree to the prompt without specifically saying 'yes, and...'
$^{1}$ Reference: https://en.wikipedia.org/wiki/Yes, and...
Basic Structure of a "Yes, and"
Prompt	"Yes, and" Response
Provides some information.
(Either as a statement or a question)	Implicitly / explicitly accepts what has been said by the prompt and adds new relevant information.
If this is your first time doing this HIT or if you are unsure as to what qualifies as a "Yes, and", read through the "Yes, and" Annotation Examples section to develop a better understanding for this task.
Given the following dialogue sequence, those selectively shown in the table below are examples of "Yes, ands".
1. [A] Well, which one speaks to you?
2. [B] Well, the one over here in the northwest corner. The crow with the beady eyes that seem to follow you. Something about it. It reminds me of something.
3. [A] Yeah! I know what it reminds me of. It reminds me of that look that comes over Dean's face when he's done something terrible, when he says...
4. [B] When he says, "No, Mommy, I won't clean my room."
5. [A] That's right!
6. [B] It's a crow body, but with the face of our child. We'll give you fifty dollars for it.
7. [C] That's exactly what I am asking. There's a sign that says fifty.
8. [B] I didn't see the sign. So that was just a lucky guess. I'm a little bit psychic.
9. [A] My wife is being modest. She's very psychic.
10. [B] It is true. In fact, all the women in my family, we have abilities.
11. [C] Please tell me a psychic vision you have had that has come true.
12. [A] Oh, tell about, remember that time we were out with the kids and we were going to go to Six Flags, and then you said we shouldn't get on the roller coaster.
"Yes, and" Dialogue pair	Explanation
[B] Well, the one over here in the northwest corner. The crow with the beady eyes that seem to follow you. Something about it. It reminds me of something.
[A] Yeah! I know what it reminds me of. It reminds me of that look that comes over Dean's face when he's done something terrible, when he says...	Speaker B establishes that she is interested in a painting with a crow in it and that it reminds her of something. Speaker A adds on to her remark and suggests details of what it reminds him of.
[A] Yeah! I know what it reminds me of. It reminds me of that look that comes over Dean's face when he's done something terrible, when he says...
[B] When he says, "No, Mommy, I won't clean my room."	Speaker B establishes that the painting reminds him of Dean's face when he's done something terrible, and Speaker A provides a detailed situation in which Dean makes the face that Speaker B was mentioning.
[B] It's a crow body, but with the face of our child. We'll give you fifty dollars for it.
[C] That's exactly what I am asking. There's a sign that says fifty.	Speaker B suggests that she will pay $50 for the painting. Speaker C agrees with her price and adds on the fact that there is a sign with the price on it. Agreeing with the price is not necessary but rejecting a setting Speaker B has established should be refrained. For example, "I want $100 for it. There's a sign with the price." is okay but "I know you don't have $50" does not qualify as a "Yes, and".
[C] That's exactly what I am asking. There's a sign that says fifty.
[B] I didn't see the sign. So that was just a lucky guess. I'm a little bit psychic.	Speaker B understands that there is a sign that she didn't see, and suggests that she has psychic abilities that allowed her to guess the price correctly.
[C] Please tell me a psychic vision you have had that has come true.
[A] Oh, tell about, remember that time we were out with the kids and we were going to go to Six Flags, and then you said we shouldn't get on the roller coaster.	Speaker C suggests that Speaker B has psychic abilities. Speaker A, knowing of Speaker B's psychic powers, interrupts and suggests a specific instance in which she demonstrated her powers.
The other possible sequential pairs are not "Yes, and" dialogue pairs. The response ignores or contradicts the information present by the prompt, or it simply acknowledges the information without refining it or adding new information. If you had listened to an audio block containing any subset of this dialogue sequence, an ideal submission would have identified all the "Yes, and" dialogue pairs here and have them accurately transcribed.
Figure 6: Explanation of the objective and yes-and in the yes-and guideline provided to Turkers.
Label space
Here is a list of subcategories for a "Yes, and" and for those that are not. You do not need to specify subcategories for this task and the following information should only be used as an extra guideline to help you better identify "Yes, and". These labels are also used in the "Yes, and" Annotated Example section for succinct explanations.
"Yes, and" subcategory
- Yes-strong: specifics provided by the prompt are refined
- Yes-but: acknowledges and adds, but rejects a specific suggestion offered by the prompt, often with 'but' or 'however'; it's different from contradiction in that the reality itself isn't rejected but an offer within it isn't taken.
Non-"Yes, and" subcategory
- No (no-info): Not contradictory, but does not add more information to that added by the prompt
- No (ignore): The response ignores statements in the prompt; there is no plausible way the response and prompt could be connected
- No (contra): The response may add some information but contradicts the reality established by the prompt
1. The task of identifying individual "Yes, and" is the same as that of the qualification HITs. If a dialogue pair can be made into a "Yes, and" with easily identifiable corrections or by ignoring cross-talk, write the dialogue pair as a "Yes, and" in your answers as well. The main difference is that you are asked to identify all "Yes, and" in a longer audio file.
2. Whenever you identify a "Yes, and" while listening to the improv session, pause and transcribe the "Yes, and" in the answer section to the right side of the window. Make sure to insert the prompt and response in separate boxes.
3. A rough transcript of the audio is shown on the left of the window, which is divided into different lines that are estimations of speaker turns. The transcript and speaker turns are not always accurate. They are for your guidance only and you may ignore them.
4. To add a new answer, click the "Add +" button to add a new prompt and response slot. To delete any answer you've written, click the trash icon on the right side of the answer.
5. The sequence of the "Yes, and"s you identify does not matter! Don't worry about placing your answers in the order they appeared in the audio.
6. Here are some keyboard shortcuts and tips you can use to complete this task:
Esc:Play/Pause
F2: Skip backward 1.5 seconds
F4:Skip forward 1.5 seconds
F5: Speed up audio by 0.25 (max: 2x)
F6: Speed down audio by 0.25 (min: 0.5)
You can click on the buttons on the left side of the audio control panel for the same functionalities, except for controlling the audio speed.
- Click on the timestamp under the 'Start' column to jump to that time on the audio.
- Click on the transcript to copy the transcript to your clipboard. You can paste the transcript to your answer with the regular paste command on your device.
- You can also click on or slide along the audio bar (the middle section in dark grey of the control panel) to navigate yourself through different parts of the audio. The current time and total time of the audio is shown on the right side of the audio control panel.
7. Once you have reached the end of the audio and you are confident that you have identified most, if not all, the "Yes, and"s in the improv session, click the "Submit" button at the bottom. Before submission, please make sure that both prompts and responses are filled in for all answers and delete any answers that are empty.
Figure 7: Explanation of the label space for yes-ands and non-yes-ands and the detailed instructions for the transcription task.
Common Mistakes
Here are some common mistakes that we have seen in previous HIT submissions. Make sure to understand why
Common mistake 1: Write prompts as statements or questions that do not introduce information
Undesirable answer:
P: What do you think?
R: It looks nice, but it's like the eyes are following me.
Desirable answer:
P: Why do you want to do that? If they are alive and they have been caged their entire lives, why do you want to have them suffer?
R: Caged? They've been spoon-fed all kinds of luxuries in that castle. It's time for them to learn the tough lessons of life.
$\rightarrow$  Although the prompt is a question, it establishes that 'they' were alive and caged their entire lives and that the following speaker wants them to suffer.
Common mistake 2: Response that simply agrees to the prompt but does not significantly add/refine information provided by the prompt
P: They left me a note to read to you.
R: Oh did they? Let me see what that says.
P: They left me a note to read to you.
R: Oh did they? Let me see what that says. Is it that red piece of paper in your hands?
$\rightarrow$  The added question in the response establishes additional information that the previous speaker is holding onto a red piece of paper in her hands. (may or may not be the note)
Common mistake 3: Response logically follows the prompt but does not add/refine information provided by the prompt
P: God, I'm gonna punch you so hard in the stomach!
R: You've got to warn me first.
P: God, I'm gonna punch you so hard in the stomach!
R: You've got to warn me first. I have a bad tummy right now and I don't want there to be an accident in your carpeted living room.
→ The original answer could also be argued as a "Yes, and", but it is weak in saying that a warning is necessary is more an approval of the punch he's about to take than an addition of information. A better answer would provide a reason why a warning is necessary.
Figure 8: Common mistakes that Turkers made in the early stages of data collection were corrected and added to the guidelines to aid new Turkers.
"Yes, and" Annotated Examples
(Click to expand/hide)
X: I hate to quibble with you Rob, but she married her first boyfriend. Kevin Bannister.
Y: You gotta be kidding me.
Annotation: No (no-info). No contradiction, simply a mild expression of incredulity. A good answer would have included something like "Kevin Bannister the circus clown?" or "I can't believe Jill would do that".
X: Well, I wish you'd stop criticizing and picking on her.
Y: Forgive my crudity, darlin!. All I'm saying is that a girl who would wear clothes like that is going to get in trouble sooner or later.
Annotation: Yes (strong). The "forgive my crudity" acknowledges that the respondent is indeed 'criticizing and picking on her' and the "all I'm saying" part explains the behavior further.
X: With all she was doin! With all the shit she kept doing! You stayed stuck to that bitch's ass and you wouldn't let go.
Y: I know about how she was like. But we was different. I's the only person she talked to about it. How she's abused. Terrible things, Gill, just terrible...
Annotation: Yes (strong) [not particularly funny, but that's not relevant]. Explains why Y "stayed stuck". Adds more to the character.
X: Yeah, man. Let's throw a bachelor party with drugs, booze and broads.
Y: Yeah. Right. All the things that make life worth living.
Annotation: No(no-info). Simply acknowledges X. A good answer could have included "Oh, I know where we can get some ketamine!" or "Let's have it in Vegas!"
X: Instead of busting my chops you should do something about that girl. Fire her. Or something.
Y: Lisa's an extremely valuable member of the Skywire team. We've got our eyes on her. You keep yours on Milo.
Annotation: No (contra) Definitely adds info but is combative and contradictory; rejects the assertions offered by X. A better answer (a yes-but) would have preceded what is provided with, e.g. "I agree that Lisa has messed up, but". A true "yes, and" would have been "She's a valuable member of the team so we've been resisting, but you're right, she's gone too far. Milo is too, so you should also fire him"
X: Aw, come on Jerry. We've gotten all we're gonna get out of this place and its starting to rain.
Y: Shit, it is only sprinkling and it's worth the trouble. Hold on for two seconds.
Annotation: No (contra). Denies that it's not useful to stay around. Note that this does acknowledge the rain ("it is only sprinkling" supports "its starting to rain") but per the guidelines, there must be no contradictions for a 'yes'. Additionally, this doesn't add anything concrete. A good answer would be e.g. "yeah, well it is only sprinkling but it's true, staying here is too much trouble".
X: ...don't do it to me...
Y: She came to talk to me...
Annotation: No (ignore). Y is not responsive to X. A good answer might be "I'm just going to give you a little novocaine so you won't feel it when we remove your tooth"
Figure 9: Annotated examples provided to Turkers for understanding the label space of the yes-and transcription task.