SlowGuess/ACL_2020_OCR huggingface dataset:
In this task we had crowd-sourced workers play with our bot and interact with it using in-game chat. The instructions shown to workers are shown in 8.
B.3 Annotation tool
The web based annotation tool has two subparts: Tool a and Tool b.
B.3.1 Tool a
This tool is the first tool in the process of annotation and asks crowd-sourced workers to help determine the intent (dialogue_type or action_type) of the sentence and highlight other pieces of the text based on the choices they made for the intent. (For example: if the intent was "Build" they are asked to select words for the thing to be built and the location respectively.) We also provided helpful tooltips with examples at every step of the process.
The instructions shown to workers for Tool a are shown in figure 9 and step by step annotation process is shown in figure 10
Figure 9: The task instructions shown to crowdsourced workers for the annotation Tool a
B.3.2 Tool b
After we determine the intent from Tool a and get highlighted span of words for respective children of the intent, we use this tool. This is the second tool in the annotation process and asks crowdsourced workers to help determine the fin-grained properties of specific entities of the action or dialogue. Note that we already got the words representing these, highlighted in B.3.1. For example: the words "big bright house" are highlighted in the sentence "destroy the big bright house by the tree" as an outcome of Tool a. The questionnaire changes dynamically based on the choices the workers make at every step of the tool. We provided helpful tooltips with examples at every step of the annotation process. Using the output of Tool a and Tool b, we can successfully construct the entire logical form for a given sentence.
The instructions shown to workers for Tool b are shown in Figure 11 and step by step annotation process for annotating properties of "location" in a "Move" action is shown in Figure 12 and annotating "reference_object" in "Destroy" action is shown in Figure 13
B.4 Tool for composite commands
This tool is meant for "composite" commands (commands that include multiple actions) and asks the users to split a command into multiple individual commands. The instruction for this are shown in figure 14. Once we get the split, we send out each command to annotation tool described in Section B.3
Figure 10: The step by step screenshot of annotations process for the command: "build three sets of book-shelves in front of me." in Tool a
Figure 11: The task instructions shown to crowdsourced workers for the annotation Tool b
Figure 13: The step by step screenshot of annotating properties of highlighted words for "reference_object" in a "Destroy" action.
Figure 12: The step by step screenshot of annotating properties of highlighted words for "location" in a "Move" action.
Figure 14: The task instructions shown to crowd-sourced workers for splitting composite commands
C Action Tree structure
This section describes the details of logical form of each action. We support three dialogue types: HUMAN_GIVE_COMMAND, GET MEMORY and PUT MEMORY. The logical form for actions has been pictorially represented in Figures: 1 and 2
We support the following actions in our dataset: Build, Copy, Dance, Spawn, Resume, Fill, Destroy, Move, Undo, Stop, Dig and FreeBuild. A lot of the actions use "location" and "reference_object" as children in their logical forms. To make the logical forms more presentable, we have shown the detailed representation of a "reference_object" (reused in action trees using the variable: "REF_OBJECT") in Figure 15 and the representation of "location" (reused in action trees using the variable: "LOCATION") in figure 16. The representations of actions refer to these variable names in their trees.
Figure 15: Logical form of a reference_object child
Figure 16: Logical form of a location child
The detailed action tree for each action and dialogue type has been presented in the following subsections. Figure 17 shows an example for a BUILD action.
Figure 17: An example logical form. The spans are indexed as : [sentence_number, [starting_word_index, ending_word_index]]. sentence_number is 0 for the most recent sentence spoken in a dialogue and is 0 in our dataset since we support one-turn dialogues as of now.
C.1 Build Action
This is the action to Build a schematic at an optional location. The Build logical form is shown in 18.
C.2 Copy Action
This is the action to copy a block object to an optional location. The copy action is represented as a "Build" with an optional "reference object". The logical form is shown in 19.
C.3 Spawn Action
This action indicates that the specified object should be spawned in the environment. The logical form is shown in: 20
C.4 Fill Action
This action states that a hole / negative shape at an optional location needs to be filled up. The logical form is explained in : 21
C.5 Destroy Action
This action indicates the intent to destroy a block object at an optional location. The logical form is shown in: 22
Destroy action can have one of the following as the child:
- reference object
C.6 Move Action
This action states that the agent should move to the specified location, the corresponding logical form is in: 23
Move action can have one of the following as its child:
- location
- stop condition (stop moving when a condition is met)
- location and stop condition
- neither
C.7 Dig Action
This action represents the intent to dig a hole / negative shape of optional dimensions at an optional location. The logical form is in 24
Figure 21: Details of logical form for Fill
C.8 Dance Action
This action represents that the agent performs a movement of a certain kind. Note that this action is different than a Move action in that the path or step-sequence here is more important than the destination. The logical form is shown in 25
C.9 FreeBuild Action
This action represents that the agent should complete an already existing half-finished block object, using its mental model. The logical form is explained in: 26
FreeBuild action can have one of the following as its child:
- reference object only
- reference object and location
C.10 Undo Action
This action states the intent to revert the specified action, if any. The logical form is in 27. Undo action can have on of the following as its child:
- target_action_type
- nothing (meaning : undo the last action)
C.11 Stop Action
This action indicates stop and the logical form is shown in 28
C.12 Resume Action
This action indicates that the previous action should be resumed, the logical form is shown in: 29
C.13 Get Memory Dialogue type
This dialogue type represents the agent answering a question about the environment. This is similar to the setup in Visual Question Answering. The logical form is represented in: 30
Figure 23: Details of logical form for Move action
Get Memory dialogue has the following as its children: filters, answer type and tag name. This dialogue type represents the type of expected answer : counting, querying a specific attribute or querying everything ("what is the size of X" vs "what is X")
C.14 Put Memory Dialogue
This dialogue type represents that a reference object should be tagged with the given tag and the logical form is shown in: 31
C.15 Noop Dialogue
This dialogue type indicates no operation should be performed, the logical form is shown in : 32
Figure 31: Details of logical form for Put Memory Dialogue
{ "dialogue_type": "NOOP" } Figure 32: Details of logical form for Noop Dialogue
D Crowd-sourced task and tools instructions