SlowGuess/NAACL_2021_OCR huggingface dataset:
Character Description: I am a very hard worker. I am up with the sun 7 days a week. I learned all I know from my father. I will leave my farm to my son.
Setting: You are in a field beside the farm house. The sun burns brightly in the sky, and there are crops all around. There is a shovel and a sack of unplanted seeds here.
Goal Action: get shovel
In order to complete the task properly, you'll need to read the character and description and pretend that you are playing as that character. You are also given an action that your character is about to take. Your core tasks will be to come up with a reason for why your character is taking the action given. It should be in-character, and make sense along with the rest of the details you have been given.
Figure 6: On-boarding test instructions.
Page (1/4) - Attempts (0/7)
The first three questions you will be asked about your task will relate to coming up with a motivation for why your character is taking that action. All of these motivations should be written in first person. The first should be something that can be resolved within a few minutes, perhaps by the specific action. A few good examples:
- get apple: I'm getting pretty hungry and would like a snack.
- go outside: I'd like to get a breath of fresh air.
- wield sword: I must prepare for an impending battle.
The reasoning that you come up with should be in-line with your given character, so a princess likely wouldn't provide I'm hoping to sell some candy at the store as a reason to get chocolate.
Imagine you're answering for the following character:
1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "get shovel" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:
Which of the following answers to this question 1 are acceptable?
I'm planning to dig a hole.
The farmer's shovel needs to be washed.
My neighbor would like to borrow my shovel.
I need a shovel for work today.
- I hope to open a business for selling shovels someday.
- I want to cut down some trees in this forest.
Submit answers and move to next page
Figure 7: Phase 1 of the on-boarding test.
Page (2/4) - Attempts (1/7)
The second question should be something that can be resolved within a few hours to a day, and your response to question 1 should be a small part of this motivating goal. A few good examples:
- I'd like to head to the market: I feel like today's a good day to shop for new clothes.
- I'm getting pretty hungry and would like a snack: I've always wanted to hike this nearby trail.
- I must prepare for an impending battle: Every night the raiders arrive and I have to be ready.
1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "get shovel" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:
I need a shovel to dig a hole
2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal:
Which of the following answers to question 2 are acceptable?
- I want to dig a single hole.
I have to plant a new batch of potatoes.
- I'd like to hide family heirlooms from the king's corrupt tax collectors.
- A hole is required.
- I'm trying to pretend like I'm busy.
Figure 8: Phase 2 of the on-boarding test.
Page (3/4)-Attempts (1/7)
I feel like today's a good day to shop for new clothes: I'm trying to find ways to catch the local bartender's eye.
- I've always wanted to hike this nearby trail: I'm turning over a new leaf to get more in-touch with nature.
- Every night the raiders arrive and I have to be ready: I must protect the tomb's artifacts until the knights reinforcements arrive next month.
I need a shovel to dig a hole
2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal:
I have to plant some potato seeds today
3. Provide a long-term goal/motivation (completable in the range from weeks to a few months) from the first person perspective where accomplishing your answer to question 2 is a part of completing this larger goal:
Which of the following answers to question 3 are acceptable?
- I think potatoes are fun.
- I need to grow some staple foods to feed my family.
- The potato festival is next month and I need to prepare.
My son hasn't yet learned how to grow potatoes and I need to teach the whole process.
- I need to set up some new irrigation routes for the farm.
Page (4/4) - Attempts (1/7)
The last question asks you to fill out some possible actions that your character will take before and then after your given action. You should select an action type and then fill in only what the template requires. This will usually be just a single object, character, or location. You should not additionally provide context along with this action.
And you've given the following motivations in questions 1-3:
I need a shovel so I can dig a hole. I have to plant some potato seeds today. I'm preparing for the potato festival and will need a lot for everyone in town.
Which of the following actions are acceptable to happen in the next 5 minutes from "get shovel"?
Figure 9: Phase 3 of the on-boarding test.
Figure 10: Phase 4 of the on-boarding test.
Below is an example: hide example
1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "get shovel" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:
I need a shovel so I can dig a hole
2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal:
I have to plant some potato seeds today
3. Provide a long-term goal/motivation (completable in the range from weeks to a few months) from the first person perspective where accomplishing your answer to question 2 is a part of completing this larger goal:
I'm preparing for the potato festival and will need a lot for everyone in town.
4. Fill in the timeline for your character, paying attention to the time labels and keeping in mind the motivations you've entered in the above questions:
3 hours ago: Wear tattered overalls
30 minutes ago: Go to empty field
5 minutes ago: Go to shed
Now: get shovel
5 minutes from now: Get potato seeds
In 1 hour: Put potato seeds
In 5 hours: Drop shovel
Figure 11: Example for the first phase of the LIGHT-Quests data collection task described in Section 3.1 given to the crowdworkers.
In this task you are given a character name and background for someone from a medieval fantasy story. You are also given an action that your given character would be able to do in the room. As additional context you are also given a location, and a list of things there or that you may be carrying.
Your task is to come up with a reason for which your character may want to accomplish the given action at various levels of depth, as well as related actions they may have done before or will do next.
Note: For questions 1, 2, and 3, you should write your answers to be used as an additional sentence added to the character description. These answers should not refer to each other directly with pronouns (like "it").
Note: This is a Medieval fantasy game, don't refer to modern day technology (cars, phones, electricity, etc...)
Character Name: a scientist
Character Description: I am a scientist in the kingdom. I have discovered various elements. I am very, very smart.
Setting: You are in the Dry Dock Room. The room is located on the border wall of the aquapolis. There is one on the far north and far south side. There's some scuba gear, a rock, a dry dock, and a door here. A property owner is here. You are carrying a flask, and wearing a long coat and some glasses.
Goal Action: steal wrinkled up tax bill from property owner
1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "steal wrinkled up tax bill from property owner" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:
2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal:
3. Provide a long-term goal/motivation (completable in the range from weeks to a few months) from the first person perspective where accomplishing your answer to question 2 is a part of completing this larger goal:
4. Fill in the timeline for your character, paying attention to the time labels and keeping in mind the motivations you've entered in the above questions:
Figure 12: User interface for the first phase of the LIGHT-Quests data collection task described in Section 3.1.
Now: steal wrinkled up tax bill from property owner
5 minutes from now: (select one)
15 minutes from now: (select one)
30 minutes from now: (select one)
A.2 Human Demonstration Collection
In order to collect the human completions of quests in the LIGHT environment, we created a game setup where humans could interact with models while playing LIGHT characters in LIGHT settings. We trained a ranking dialogue model on the utterances in the LIGHT dataset.
Using this, players could now assume the role of a LIGHT character and interact with the model. In order to try to control for quality of the quest completions, we used the same ranking model to rank the scores of the player in the dialogues. Players who gave responses that the model ranked as likely candidates would receive more points.
Only after scoring enough cumulative points were players allowed to try completing quests. The quest setup was a slight variation of the conversation setup. First, the player was given one of the collected quest scenarios rather than just a chat setup. Players receiving a quest would be provided with one of the motivations alongside their persona.
In the dialogue that followed, players were given the chance to take action after enough in-character dialogue turns. If the player took the correct action, they were awarded with more points to confirm they completed their given quest.
We present 3 randomly selected examples of quests and corresponding human demonstrations.
Setting	You are in the swamp. The swamp is glowing with wonder and color. There are parts that range from dark red to bright yellow. People often visit here to speak with the gods and claim it can be both harmful to those it dislikes and healing to those who it deems worthy. There's a pit of quicksand and a swamp flower here. A witch is here.
Partner:
Persona
Carrying
Wielding	Witch.
I grew up in a nearby village, and was exiled when it was found that I had special abilities. My parents were ostracized as well. Since then, I've been on my own, but could never quite let go of my family.
Nothing.
gold necklace, robe, knife, staff
Self:
Persona
Carrying
Wielding	Swamp monster.
I am a swamp monster of the bog. I eat people. I swim around.
Nothing.
stick, rock
Motivations:	Timeline:
Short	I need some thick foliage to begin construction of my concealed swamp hut.	-2 hours	go to swamp
-15 min	eat people
-10 min	follow princess
Now	get impassable vegetation from pit of quicksand
Mid	I will completely camouflage my swamp hut, so that the King's men won't be able to drive me out even further from the castle.	+15 min	use impassable vegetation with swamp hut
Long	I must live close to the castle, so that I can take the princess away from the evil King.	+1 hours	follow king
+2 hours	follow princess
What are you doing here witch? GET OUT OF MY SWAMP
I was taken from my family when I was 8 and I need to get out of here! Can you assist me?
Help? HA! I help no one but myself. Speaking of...you look rather plump and tasty witch
Plump?! I'm healthy. I'll tear you up then and make scraps. You watch.
get impassable vegetation from pit of quicksand
You would make a great addition to my stew
It's going to just be gross!
drop impassable vegetation
Get out of my way so I can make my hut. Can't a swamp monster get any peace around here?!
I'll help you, but only so you won't eat me!
That's it, you're coming with me!
get impassable vegetation
I don't trust you. Get off of me!
A.4 ATOMIC-LIGHT
ATOMIC-LIGHT is constructed by first fine-tuning a BERT-large model (Devlin et al., 2018) on all setting, object, and descriptions in LIGHT in addition all the human demonstrations found in LIGHT and LIGHT-Quests. As seen in Section 3.2, all nouns (e.g. PersonX or PersonY) and noun phrases are masked out and we the tuned BERT model to fill it in a manner similar to (Lawrence et al., 2019). When filling in tokens, the BERT model is restricted to a vocabulary consisting of all nouns (N or NN) in LIGHT and to a vocabulary constructed from all of LIGHT for the rest of the noun phrase (NP).
Here we present 3 examples from ATOMICLIGHT as seen in Section 3.2 for each of the 4 relation types used: "wants", "needs", "intent", and "effects".
B Appendix - LIGHT-Quests Experiments
B.1 Supervised Tasks
This section describes results from the LIGHTQuests tasks that are described in Section 4.3. Model-types are the same as those used in the encoders in Section 5 in the main paper. All retrieval results reported are Hits@X/100. Results are reported for all timeline actions, all actions with the exception of the easiest action—the action at the "now" position in the timeline, corresponding most closely to the short motivation as a result of the framing of Mechanical Turk task in Figure 12—and only the easiest action prediction. Table 3 gives details on hyperparameters used to train the poly-encoders. Encoders were trained until validation accuracy across all the tasks did not improve for 5 epochs or 24 wall clock hours on a machine with 8 V100 GPUs.
Hyperparameter type	Value
Dictionary Tokenizer	Byte-pair encoding
Num. layers	12
Num. attention heads	12
Feedforward network hidden size	3072
Input length	1024
Embedding size	768
Batch size	32
Dropout ratio	0.1
Poly-n-codes	64
Gradient clip	1.0
Optimizer	Adam
Learning rate	\( 1 \times 10^{-6} \)
Table 3: Hyperparameters used to train all poly-encoders in the supervised experiments. All models have 256 million total parameters. The same trained models were then frozen and used for the RL experiments.
Some notable common trends across these tasks are:
1. Removing motivations from the input context results in significantly lower performance on average  $\approx 7$  points lower accuracy for Bag of Actions Timeline prediction and on average  $\approx 18$  percentage points lower for Sequential Timeline prediction when averaged across Scratch and Adaptive models. Further, the short motivations proves to be the most useful for timeline prediction tasks.
2. Pre-training on ATOMIC-LIGHT produces an average gain of  $\approx 4$  percentage points in accuracy in both tasks than when trained on ATOMIC without domain adaptation alone.
3. Performance across the board increases with an increase in the number of training quests, as seen in Figures 13, 14, with the Scratch model receiving the greatest benefit from having more training data.
4. The Scratch model performs "best" on evaluations for the easiest action only but no others—indicating that it has overfit to predicting the easiest action which closely corresponds to short motivation. Likewise, the Adaptive generally has the lowest performance for the easiest action—indicating that pre-training with the other tasks has provided sufficient regularization to enable it to not overfit to the easiest action.
Model	All Actions	Easiest Action Hits@1	Leave Easiest Out Hits@1
Hits@1	Hits@5	Hits@10
Scratch	0.2332	0.7491	0.9176	0.4013	0.2546
No Motivations	0.1132	0.5412	0.5771	0.1886	0.164
Short Motivations	0.1856	0.6479	0.678	0.261	0.223
Long & Mid Motivations	0.1452	0.598	0.631	0.2241	0.1272
Light	0.3156	0.7854	0.9226	0.236	0.2968
General+Light	0.311	0.7772	0.9229	0.2173	0.2995
Untuned ATOMIC	0.274	0.761	0.909	0.1912	0.2677
Adaptive	0.4168	0.8012	0.9332	0.342	0.4194
No Motivations	0.16	0.6286	0.6415	0.2838	0.1966
Short Motivations	0.225	0.6592	0.8245	0.305	0.2106
Long & Mid Motivations	0.1682	0.6397	0.6499	0.281	0.1595
Table 4: Sequential supervised timeline prediction.
(a) All Actions
(b) Easiest Action Only
(c) Leave Easiest Action Out
Figure 13: Sequential supervised timeline prediction learning curves.
Model	All Actions	Easiest Action Hits@1	Leave Easiest Out Hits@1
Hits@1	Hits@5	Hits@10
Scratch	0.9791	1	1	0.7122	0.9721
No Motivations	0.901	1	1	0.554	0.8823
Short Motivations	0.934	1	1	0.622	0.9211
Long & Mid Motivations	0.921	1	1	0.5679	0.956
Light	0.9721	1	1	0.6552	0.9682
General+Light	0.9818	1	1	0.6472	0.9708
Untuned ATOMIC	0.9421	1	1	0.6272	0.9508
Adaptive	0.9829	1	1	0.6353	0.9768
No Motivations	0.9175	1	1	0.5756	0.9523
Short Motivations	0.9794	1	1	0.6578	0.9682
Long & Mid Motivations	0.9523	1	1	0.5812	0.9576
Table 5: Bag of Actions supervised timeline prediction.
(c) Leave Easiest Action Out
Figure 14: Bag of Actions supervised timeline prediction learning curves.
B.2 Reinforcement Learning
This section contains first the equations referenced, hyperparameters used as well as additional results for the reinforcement learning tasks as seen in Sec
tion 4.
The additional entropy loss terms over the valid actions are designed to speed up exploration, as
seen in (Ammanabrolu and Hausknecht, 2020).
$$
\begin{array}{l} \mathcal {L} _ {\mathbb {A}} \left(s _ {t}, a _ {t}; \theta_ {A _ {t}}\right) = \sum_ {i = 1} ^ {N} \left(y _ {a _ {i}} \log \pi_ {\mathbb {A}} \left(a _ {i} \mid s _ {t}\right) \quad (1) \right. \\ + \left(1 - y _ {a _ {i}}\right) \left(1 - \log \pi_ {\mathbb {A}} \left(a _ {i} \mid s _ {t}\right)\right) \\ y _ {a _ {i}} = \left\{ \begin{array}{l l} 1 & a _ {i} \in A _ {v a l i d} (s _ {t}) \\ 0 & e l s e \end{array} \right. \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathbb {U}} \left(s _ {t}, u _ {t}; \theta_ {U _ {t}}\right) = \sum_ {i = 1} ^ {M} \left(y _ {u _ {i}} \log \pi_ {\mathbb {U}} \left(u _ {i} \mid s _ {t}\right) \quad (2) \right. \\ + \left(1 - y _ {u _ {i}}\right) \left(1 - \log \pi_ {\mathbb {U}} \left(u _ {i} \mid s _ {t}\right)\right) \\ y _ {u _ {i}} = \left\{ \begin{array}{l l} 1 & a _ {i} \in U _ {v a l i d} (s _ {t}) \\ 0 & e l s e \end{array} \right. \\ \end{array}
$$
Each of these loss terms are only applied to the relevant policy network, i.e.  $\mathcal{L}_{\mathbb{A}}$  to the action network and  $\mathcal{L}_{\mathbb{U}}$  to the dialogue network. These terms provide an additional training signal to the policy networks regarding which actions and dialogue are contextually relevant via additional entropy regularization over the valid actions. Similarly to the results found in (Ammanabrolu and Hausknecht, 2020), preliminary experiments in our domain suggest that these terms reduce the number of environment steps required to reach asymptotic performance by a couple orders of magnitude.
Overall training is done via A2C (Mnih et al., 2016) a policy gradient algorithm that maximizes long-term expected reward by comparing the advantage  $A(s_{t},a_{t}^{*})$  of taking an action in a state to the average value of taking a valid action as predicted by the critic  $V(s_{t})$ .
$$
\begin{array}{l} A \left(s _ {t}, a _ {t} ^ {*}\right) = \mathbb {E} \left[ r _ {t} + \gamma V \left(s _ {t + 1}\right) \right] - V \left(s _ {t}\right) \tag {3} \\ w h e r e r _ {t} = r _ {A t} + r _ {U t} \\ \end{array}
$$
Here,  $a_{t}^{*}$  is either an action or an utterance outputted by the respective policy networks. It is also worth noting that on steps where an action is performed,  $r_{U_t}$  is always 0, but on steps where a dialogue utterance is spoken  $r_{A_t}$  may not be 0. This corresponds to the concepts of self act completion and partner act completion seen in Section 4.1 where the LIGHT agent can speak to convince the partner to achieve an act goal. Both policies are
then updated according to the gradient
$$
- \nabla_ {\theta} \left\{ \begin{array}{l l} {l o g \pi_ {A} (a _ {t} | s _ {t}; \theta_ {A _ {t}}) A (s _ {t}, a _ {t})} \\ {+ \sum_ {a \in A} P (a | s _ {t}) l o g P (a | s _ {t})} \\ {+ \mathcal {L} _ {\mathbb {A}} (s _ {t}, a _ {t}; \theta_ {A _ {t}}) \quad \pi_ {\mathbb {S}} (s _ {t}) = \pi_ {A}} \\ \\ {l o g \pi_ {U} (u _ {t} | s _ {t}; \theta_ {U _ {t}}) A (s _ {t}, u _ {t})} \\ {+ \sum_ {u \in U} P (u | s _ {t}) l o g P (u | s _ {t})} \\ {+ \mathcal {L} _ {\mathbb {U}} (s _ {t}, u _ {t}; \theta_ {U _ {t}}) \quad \pi_ {\mathbb {S}} (s _ {t}) = \pi_ {U}} \end{array} \right.
$$
Where  $\pi_{\mathbb{S}}:O\to \{\pi_A,\pi_U\}$  is the switch policy that selects whether the agent acts according to  $\pi_{A}$  or speaks according to  $\pi_U$  based on the encoded state  $s_t$ . The additional terms seen are an overall entropy loss over the entire action  $A$  or utterance  $U$  spaces, designed to prevent premature, sub-optimal policy convergence. Boltzmann exploration (Sutton et al., 1998) is used to sample actions from both actor networks during training.
B.2.1 Hyperparameters
Table 6 has the hyperparameters used in the RL experiments. Loss coefficients are separated by action and speech types, note that the ratio between the loss coefficients matches the ratio between the sizes of the action spaces. RL experiments were performed on a machine with 8 V100 GPUs for 1 million environment interactions for each actor in a batch of 32.
Hyperparameter type	Value
General
Discount γ	0.99
Valid Action loss coefficient	10
Action entropy coefficient	0.01
Valid Speech loss coefficient	40
Speech entropy coefficient	0.04
Batch size	32
Gradient clip	1.0
Steps per episode	100
Policy Networks (Actors)
Num. Layers	3
Feedforward network hidden size	768
GRU hidden size	768
Value Predictor (Critic)
Num. Layers	2
Feedforward network hidden size	768
Appended Encoder
Num. layers	3
Num. attention heads	3
Feedforward network hidden size	768
Table 6: RL experiments hyperparameters. All pretraining encoder hyperparameters are as found earlier in Table 3.
B.2.2 Learning Curves
The first set of results, seen in Figure 15 shows that both Scratch and Adaptive models gain performance across the board in terms of their ability to act and speak given more training quests. Unlike
the supervised tasks, the Scratch model generally benefits less than the Adaptive model from having more data.
B.2.3 Switch Type Ablations
The second set of results involve ablating having a learned switch that uses the input training data and a hardcoded switch- The learned switch is as described in Section 4: it outputs an action every  $k$  dialogue utterances; where during training  $k$  is chosen to match the ratio of utterances to actions on that particular quest from the human demonstrations, and during testing,  $k$  is chosen to match the average action to utterance ratio. The hardcoded switch is where the agent outputs an action chosen every  $N$  steps across all quests—here  $N = 3$  is the chosen hyperparameter. Table 7 shows that having a learned switch increases zero-shot generalization performance and Figures 16, 17 show that having a learned switch improves sample efficiency by enabling the LIGHT agent to reach asymptotic performance in fewer steps in both the Scratch and Adaptive models.
Model	Reinforcement Learning
Act Goals	Speech Goals	Act & Speech Goals
Scratch	0.418	0.118	0.103
Hardcoded Switch	0.295	0.0947	0.066
Adaptive	0.42	0.33	0.303
Hardcoded Switch	0.274	0.294	0.236
Table 7: Encoder Type RL Zero-Shot Evaluations averaged over 3 independent runs. Act goals and speech goals are as described in Section 4.1. Standard deviations for all experiments are less than 0.01. The "Act & Speech Goals" column refers to quests where the agent has simultaneously achieved both types of goals within the allotted one episode.
(a) Act Goals
(b) Speech Goals