SlowGuess/NAACL_2022_OCR huggingface dataset:
Rationale: (Sheldon is Amy's boyfriend. After identify P0 is Amy, based on the relationship between Amy and Sheldon, P1 can be identified as Sheldon.)
- Relationship
Candidates: [Amy, Penny, Sheldon]
P0: Hey, boyfriend.
P1: Can't talk. Spitball. Probably gonna die.
- Status
Background: (from TBBT) [The pub]
P0: So when do you guys plan on getting married?
P1: Uh, we're not sure. But I want to wait long enough to prove to my mother I'm not pregnant.
P2: May I have one of your fries?
P1: Of course. Can I have a bite of your burger?
P2: Absolutely not.
P3: Some perfect couple. He won't even share his food with her.
Answer:  $P3 \rightarrow$  Leonard
Rationale: (The aforementioned failure to determine Leonard's marriage led him to ridicule couples in harmonious relationships.)
Background: (from TBBT) [Penny's apartment]
Bernadette: Nah, you got this. Let's go for a drink.
I'll call Amy.
P0: Okay, good. She seemed like she really wanted to go out tonight.
P1 (phone ringing, running down stairs from outside penny's door): Hey, girl.
Answer: P1  $\rightarrow$  Amy
Rationale: (Bernadette said she will call Amy and P1 is the person who answers the phone.)
- Mention
Background: (from TBBT) [The apartment]
Candidates: {Raj, Leonard, Sheldon, Amy}
P0: Mmm, I love how they put a waterfall at centre field. It really ties the whole stadium together.
P1: This is fun, huh? We get to see our friend throw out the first pitch, have a hot dog, watch the game.
P2: Whoa. Nobody said anything about watching the game.
P3: Sheldon, what did you expect?
Answer: P2  $\rightarrow$  Sheldon
Rationale: (P3 mentioned the name of the person being questioned which is "Sheldon")
Inside-Scene
Background: (from Friends) [Scene: Outside the Janitor's Closet, there are people having  $s^*x$  and Mr. Geller is trying to give them some pamphlets.]
Candidates: {Monica, Chandler}
Mr. Geller: Kids, I spoke to a doctor and picked up this pamphlets on how to get pregnant. (He slides them under the door.
P0: (walking by with Chandler.) Hey dad!
P1: Hey.
Mr. Geller: (pause) Sorry to bother you again, but could you pass my pamphlets back? (They do so.) Thank you.
Answer: P1  $\rightarrow$  Chandler
Rationale: (Monica is Mr. Geller's daughter. P0 called Mr. Geller dad so she is Monica. There are only two candidate so the other one is Chandler)
C Extended Study of Required Reasoning Types on our TVSHOWGUESTS
This section provides an in-depth analysis of the types of reasoning used to infer evidence when guessing characters.
C.1 Our Annotation Schema of Reasoning Types
We define the following reasoning types with examples provided. A summary of our annotation schema of reasoning types can be found in Figure 7.
Multi-hop on Characters Reasoning on the basis of other characters that have already been guessed. Using the already guessed character as a bridge, users can employ history event or the relationship between characters to make guesses about the target character. The difference between multi-hop character and exclusion is that after identifying the other characters, the exclusion technique relies only on the list of characters provided for guessing, however, multi-hop character reasoning requires additional evidence such as relationship to infer the target character.
Background: (from TBBT) [Angels Stadium]
P5: Hey, I hear you're a dermatologist.
Emily: Uh, yeah, I'm a resident at Huntington Hospital.
P5: I have some odd freckles on my buttocks. Can I make an appointment for you to look at them?
Emily: Um, okay, I guess.
P0: I'm with him three years, nothing. She's with him two minutes, and he's taking his pants off.
Answer:  $\mathrm{P0} \rightarrow \mathrm{Amy}$
Rationale: (Using P5 (Sheldon) as a bridge and the couple relationship between Amy and him, we can identify P0 is Amy.)
Multi-hop on Textual Evidence Some evidences are not directly presented in the scene but can be inferred from the descriptions of context and
dialogues. Using the inferred evidences as bridges people can multihop over personality, or fact, or event inferred from the text to guess the characters.
Background: (from TBBT) [The apartment ]
Candidates: {Amy, Leonard, Raj, Howard', Penny, Sheldon}
Bernadette: I like your suit.
P0: Oh, thanks. Got a couple new outfits for work.
P1: How does it feel knowing your fiancée's job is to go out and flirt with doctors, looking like that, while you sit here, you know, looking like this?
Answer: P0  $\rightarrow$  Penny
Rationale: (P0 has a new job can be inferred from the textual evidence "Got a couple new outfits for work". Plus we know that Penny has a new job, we can determine that P0 is Penny)
Commonsense of Concepts/Events Task requires additional commonsense knowledge of attributes of daily concepts or social events, or their relations including causal/effect relations between an event and a social state or social relation. We restrict this category to be the aforementioned commonsense knowledge types, to distinguish from other relatively under-studied commonsense knowledge, such as the commonsense of dialogue flow required to work with our inside-scene evidence defined in Figure 6.
Background: (from TBBT) [Capital Comics]
Candidates: {Howard, Sheldon}
P0: I know that if I had a wife or a fiancée, I'd ask her first before I invested money in a comic book store.
P1: He's right.
Answer: P1  $\rightarrow$  Howard
Rationale: (A married or engaged person will answer "He's right". Howard is married.)
Default Conjunction A single piece of evidence will not solve this task; a combination between multiple pieces of evidence is needed to identify the person.
C.2 Analysis of the Human Annotation
Correlation between the Human Annotated Schema Categories Figure 2 visualizes the flow between (a) evidence types and the dependency of history and (b) evidence types and the reasoning types. Most evidence types correlate with history dependency. Personality and history dependency are most closely related. Default conjunction is the reasoning type that accounts for the largest percentage.
C.3 Experiments: Performance
Decomposition on the Reasoning Types
We further studied the impact of the required reasoning types on the performance (the right column
Reasoning Type	Description
Default Conjunction	No single piece of evidence can solve the task, hence the conjunction among multiple pieces of evidence is required. This is the default reasoning type if there are multiple evidence types labeled but no other reasoning types are labeled.
Multihop-Character	Task needs to be solved with the guessing results of other characters, then using the target person relation to or memory about the guessed ones to make the answer, i.e., multihop with guessed characters as bridges.
Multihop-Textual	Task needs to be solved with the persona/fact/event not directly described in the scene but can be inferred from the context, i.e., multihop over persona/fact/event inferred from dialog and scene context.
Commonsense attributes/relations of concepts/events	Task requires additional commonsense knowledge of attributes of daily concepts or social events, or their relations like causal relations between events. Those refer to the specific types of commonsense covered in ConceptNet- or Atomic-style KBs.
Figure 7: The definitions of reasoning types.
Reasoning Type	Friends(%)	TBBT(%)
Default	16.56	28.48
Multihop(CharACTER)	3.97	13.91
Multihop(Textual)	5.30	5.30
Commonsense	4.64	0.66
No Complex Reasoning	69.54	51.66
Figure 8: Visualization of the flow from the required evidence types to their required reasoning types.
in Figure 9). In general there is a clear gap (on average  $\sim 10\%$ ) between cases that require complex reasoning and those that do not. The Multihop-Textual type is most challenging, because it requires both deep understanding of what the texts implies and multihop reasoning. There is not a clear performance difference between Multihop-Character and Default Conjunction, though the former is conceptually harder. We surmise this is because both types are beyond the reasoning ability of the model so the predictions largely rely on fuzzy matching of evidence – recall that we predict identities of main characters, so there can be a statistical bias of their context co-occurrence. The results on the Commonsense type fluctuate due to the relatively smaller ratio.
Vanilla Longformer
Longformer-P
Human
Figure 9: Performance breakdown according to our reasoning schema (left: Friends, right: The Big Bang Theory).
Table 9: Percentage of the required reasoning types in the two TV shows, Friends and The Big Bang Theory.
#Unsolvable	#Human Mistakes
TBBT	Friends	TBBT	Friends
4882	2500	4921
4895	4894
4907	4910
4908
Table 10: Human Errors.
D Interface for the Human Study
Figure 10 shows the interfaces of the human study.
E Examples of Human Errors
Table 11 provides an example of unsolvable cases and Table 12 provides an example of human mistakes. The human mislabeled characters are marked as red.
We further provide all the scene IDs on which our human tester makes incorrect predictions in Table 10.
F Details of Human Study and Discussions on the Challenges of History Retrieval
Our experiments show that the history dependency challenges existing models. Finding the evidence
Unsolvable Case
08x02 4882
Background: (from TBBT) [the Apartment]
Candidates: {Howard, Sheldon, Raj, Amy, Leonard, Penny}
P0 : I recently read that during World War Two, Joseph Stalin had a research program to create supersoldiers by having women impregnated by gorillas.
P1: What a sick use of science.
P2: Hey, as long as the baby's healthy.
P3: I wonder if Stalin considered any other animals.
P4 : Hippos are the deadliest creature. A half-human, half-hippo soldier would be pretty badass.
P1: Yes, but when they're hungry-hungry, you can stop them with marbles.
P0: Yeah, the correct animal for interspecies supersolder is koala. You would wind up with an army so cute it couldn't be attacked.
P2: But half-man, half-owl could fly...
P0: The answer is cuddly soldiers with big flat noses. Moving on.
P1: So, Penny, when's the new job start?
P5: Next Monday.
Bernadette : Did you get a chance to look over the materials I gave you?
P5:Uh,not yet,but I will.
Bernadette : Great. When?
P5: I said I'll get to it.
P0: I'm sensing awkwardness, am I right?
P3:Yes.
P0:Swish.
Bernadette : I don't want to be pushy, but you've never done pharmaceutical sales before. It seems like you could use this time to get a head start.
P5: Well, the first few weeks will be all training. They'll tell me everything I need to know.
Bernadette : But imagine how impressed they’d be if you showed up already familiar with the material.
P5: Okay, so what, you want me to be like a teacher's pet?
Bernadette : Couldn't hurt.
P4: Mm, I don't know. Who here has ever been hurt because they were the teacher's pet?
P0: It was like the rest of the class wanted Ms. McDonald to forget the quiz.
Answer: P0: Sheldon, P1: Howard, P2: Raj, P3: Amy, P4: Leonard, P5: Penny
Table 11: Example of unsolvable case.
08x04 4921
Background: (from TBBT) [Penny's partment]
Candidates: {Raj, Penny}
P0: I'm so glad we could work this all out.
P1: Yeah, me, too.
Emily : You know, we should have dinner one night with you and Leonard.
P1: Oh, we would love that.
P0: Great.
background : (both chuckle)
P1: Okay, good night, guys.
Emily : All right, night.
P1:Bye.
Emily and Penny (simultaneously): I hate her.
Answer: P0: Raj, P1: Penny
Table 12: Example of mistake.
(a) Introduction page of human study.
The stairwell
P0: So, after drinks with Bernadette, I get home, and Penny calls to complain about her. And then while I'm talking to Penny, I get a text from Bernadette.
P1: I am trying to prepare my lesson plan for Howard. Why are you telling me this?
P0: Because it's taken 15 years, but high school is finally awesome. I love them both, but I'm in the centre now, and I love that even more.
P1: Amy, please. I am trying to figure out a way to intellectually emasculate a dear friend of mine. P0: But I'm just...
P1: Not now.
P0: You better watch that attitude, buddy. You're dating the popular girl now.
(b) Task 1: character guessing task
(c) Task 2:identifying used evidence types.
(d) Task 3: identifying used reasoning types.
Figure 10: interfaces of human studies.
history scenes for such cases is essentially a retrieval task (but without groundtruth). To see how it brings new challenges to existing semantic search, we applied a state-of-the-art model to retrieve the history scenes and conducted an additional human study to evaluate the results.
Task We conduct the study on scenes in our human
annotation sets that have the Memory type labeled. With each scene as a query, we retrieve from a window of 20 previous scenes with a state-of-the-art model The window size is decided so as to guarantee that at least one required memory appears in the window, according to our human annotation process. The task of human study is to recognize whether the top-3 returned scenes contain at least one related history scene.
Results The same annotators working on the study in Section 4 evaluated the retrieved scenes. Results show that the recall of the top-3 results from this state-of-the-art model is low (35.5%). This difficulty in scene retrieval may arise from: (1) the queries are scenes with structures, which leads to different query formats from standard IR tasks; (2) many relevant scenes are dissimilar to the query scenes in the semantic space, but associated with the query in specific aspects or even analogous to the query scene; (3) some scenes require multi-hop retrieval, especially when combined with ToM modeling (reasoning about what others know).
All these challenges are non-trivial, and calls for further studies on semantic search to address.
G Model Checklist
We implement our baselines based on HuggingFace Transformers. $^{10}$  We use the pretrained allenai/longformer-base-4096 and bert-base-uncased models. We train all the models with the Adam optimizer.
We train our model on a single V100 GPU. It takes around 1 hour and 40 minutes to train a Longformer-based model. It takes around 2 hour and 10 minutes to train a multi-row BERT model. For all the models, we train in total 40 epochs. But the models usually converge in less than 20 epochs.
Hyperparameters We set the number of rows in MR. BERT to 12, to maximize the usage of GPU memory. We set the maximum length of Longformer to 2000, which can handle the lengths of most of the input scenes. The window size is set to 256. We set the learning rate to 2e-5.
We report our result with a single run. However, for each model we run twice; and we found the average development accuracy varies less than  $0.5\%$ .
<sup>9</sup>We use the all-mpnet-base-v2 model from https://sbert.net/ that reports the top-1 performance on 14 sentence embedding tasks and 6 semantic search tasks.
<sup>10</sup>https://github.com/huggingface/transformers