A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting
Authors: Pradyumna Tambwekar
Lakshita Dodeja
Nathan Vaska
Wei Xu
Matthew Gombolay
Venue: EMNLP 2023 Findings (2023)
Keywords: Text Classification
ChatGPT
Human-Evaluation
Low-Data
TL;DR: We define a novel task called Automated Strategy Translation, and build a langauge model which can outperform ChatGPT on this task.

Abstract: Many real-world tasks involve a mixed-initiative setup, wherein humans and AI systems collaboratively perform a task. While significant work has been conducted towards enabling humans to specify, through language, exactly how an agent should complete a task (i.e., low-level specification), prior work lacks on interpreting the high-level strategic intent of the human commanders. Parsing strategic intent from language will allow autonomous systems to independently operate according to the user's plan without frequent guidance or instruction. In this paper, we build a computational interface capable of translating unstructured language strategies into actionable intent in the form of goals and constraints. Leveraging a game environment, we collect a dataset of over 1000 examples, mapping language strategies to the corresponding goals and constraints, and show that our model, trained on this dataset, significantly outperforms human interpreters in inferring strategic intent (i.e., goals and constraints) from language (p < 0.05). Furthermore, we show that our model (125M parameters) significantly outperforms ChatGPT for this task (p < 0.05) in a low-data setting.

A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting
Pradyumna Tambwekar', Lakshita ; Nathan Vaska²; Wei Xu', and Matthew Gombolay'
'Scoool of Interactive Computing, Georgia Institute of Technology
CCmputer Science Department, Brown University
3 Massachusetts Institute of Technology, Lincoln Laboratory
pradyumna. tambwekare. waaeeeaaarrrtttt.eeee.m. edu, nathan. vaska@l1. edu, {wei xu, matthew.. wwwbolayyocc..c..o.oooooo wwteeecc...........oc.. edu
jtsettaddddejeeeboeeeeeeeeeeee edu,,
Abstract
Stratagy Description (W)
needed avsilable troops to defend Purple. sonted protact Purple. kith Purple
Many real-world tasks involve a mixed- initiative setup, wherein humans and Al sys- tems collaboratively perform a task. While significant work has been conducted towards enabling humans to specify, through language, exactly how an agent should complete a task (i..., low-level specification), prior work lacks on interpreting the high-level strategic intent the human commanders. Parsing strategic intent from language will allow autonomous systems to independently operate according to the user's plan without frequent guidance or instruction. In this paper, build computa- tional interface capable of translating unstruc- tured language strategies into actionable intent in the form of goals and constraints. Leverag- ing a game environment, we collect a dataset of over 1000 examples, mapping language strate- gies to the corresponding goals and constraints, and show that our model, trained on this dataset, significantly outperforms human interpreters in inferring strategic intent (i.c., goals and con- straints) from language (p 0.05). Further- more, we show that our model (125M param- eters) significantly outperforms ChatGPT for this task (p= < 0.05) in low-data setting.
like cannot
Constraiitss (C)
be purple. involved battles taking overly agaressive senatagy. black player first, people think
too
Chizi have troopu Purals
ca:si reed Lotal
defund a contirent
night constraints sotail
I's going
Constraint Decoder
Goal Decoder
Selections/Mup
Boals (6)
G2::Battle players time 100 G3::Keep your troops close together 85
cToest of i precettiee goslino
Figure I: Our work aims to facilitate humans to specify their strategy an Al system via language. Using the board game Risk as a simulated environment, we collect language descriptions of strategy (top-left) correspond- ing player's troop deployments (bottom- left). The player's is selections are shown by the white icons, and the grey and black icons denote the troops of the two opposing players. Each strategy corresponds to a set of goals (bottom-rrght) and constraints (top-right) The green and orange text corresponds the language relat- ing to constraints and goals respectively.
1 Introduction
Effective communication is essential for the proper functioning of organizational teams. "Comman- der's Intent" is a method for developing a theory of mind utilized in many domains such as the search and rescue, pandemic response, military, etc (Mer- cado et al., 2016; Rosen et al., 2002; Kruijff et al., 2014). Commanders and leaders often utilize the formulation of "Commander's Intent" convey the tasks that need to be accomplished and engen- der an understanding of the criteria for success to their subordinates (Dempsey and Chavous, 2013). Commander's s Intent could similarly function as
effective scaffold represent human's strate- gic intent in a mixed-initiative interaction (Novick and Sutton, 1997). Commander's Intent provides a functionality for expert-specifiers to engender a degree of "shared-cognitionn" between an AI- collaborator and a human-specifier, by aligning the actions of the AI system to the human-specifiers values or reward function.
Commander's intent is formally represented by set of goals and constraints. Goals (or prefer- ences) are categorized
desirable set of states or affairs that the agent intends to obtain (Moskowitz and Grant, 2009; Kruglanski, 1996) and constraints refer to conditions that are imposed on solutions
*These authors contributed to this paper while they were at Georgia Institute of Technology.

---

formulated by agent (Nickles, 1978). Translat- ing unstructured language-based strategy into this machine-reddable specification is a non-trivial chal- lenge. This translation could be conducted via a human interpreter, however, interpreters with the requisite expertise will not always be available. Al- ternatively, humans could utilize structured inter- face to specify their intent. However, interfaces can become overly complicated, and humans become demotivated work with an AI system when they cannot easily navigate the interface (Hayes, 1985). Enabling humans to express their strategic intent in everyday language provides an effective solution to these issues.
model and these baselines, providing concrete measure of the relative efficacy of our approach. Our contributions are as follows:
We propose one of the first complete machine learning pipelines including data collection, aug- mentation and model training for inferring struc- tured strategic intent from human language.
Through human study, we show that our pro- posed approach can interpret goals and con- straints from language descriptions better than the average human (p < 0.001).
Through in-context learning, we evaluate Chat- GPT's performance to gauge the relative efficacy our approach, and show that our approach sig- nificantly outperforms ChatGPT (p 0.05).
In this paper, we develop an approach to solve a task we call automatic strategy translation. wherein we learn to infer strategic intent, in the form of goals and constraints, from language. Prior work has developed methods to utilize language to spec- ify policies of Al agent (Tambwekar al., 2021; Gopalan et al., 2018; Thomason al., 2019; Blukis et al., 2019) or specify reward functions or tasks which can be optimized for, via reinforcement learning (RL) or a planner (Gopalan et al., 2018; Padmakumar et al., 2021: Silva et al., 2021a). How- ever, our work is the first to translate language into goals and constraints, which can be applied towards constrained optimization approaches for directing agent behavior independent of the original human specifier. Unlike prior work, we focus on inter- preting language description of complex gameplay strategies, rather than simple individual commands (e.g., "move from to B: open the door").
Related Work
This section covers prior work on learning strate- gies from language, as well methods and datasets to enable humans to specify Al-behavior in mixed-initiativy setting.
2.1 Learning strategies from Language
A common approach for specifying strategies through language has been through encoding lan- guage instructions, via planning- based representa- tion languages, such PDDL or LTL (Williams et al., 2018; Bahdanau et al., 2018; Thomason al., 2019; Tellex al., 2020), deep learning (Fu et al., 2019; Blukis et al., 2019; Gopalan al., 2018). Such formulations facilitate the ability constrain actions taken by the agent to the instruc- tion specified, e.g. "Go around the tree to your left and place the ball." Another popular alternative is language-conditioned learning, where language is employed to specify reward function, or a task (Silva et al., 202 Goyal et al., 2019; Andreas et al., 2017; Shridhar al., 2022). Such approaches seek improve the ability an agent to complete a task(s) through intermediate language inputs, such as "take the ladder to your left". However, these ap- proaches do not allow a supervisor to specify their strategic intent, such that the agent can complete it's primary task while still adhering to the speci- fier's plan. Recent work proposed novel approach to mapping language constraints and rewards via dependency tree (Rankin et al., 2021), however their approach relies on a pre-trained grammar extract dependency tree, thus may not scale to human-like language.
First, we collect a dataset of over 1000 exam- ples mapping language to goals and constraints, leveraging game environment of Risk. Next, we fine-tuned a pretrained RoBERTa model (Liu et al.. 2019), equipped with model augmentations and customized Joss functions such as Order-Agnostic Cross Entropy (Du et al., 2021), infer goals and constraints from language strategy specifications. Finally, we employ a human evaluation to test our approach. Recent work has shown that automated evaluation metrics for language models may pro- vide misleading measure of performance (Liang et al., 2022). Therefore, design head-to-heat evaluation, whereby, can directly compare our addi-
model to the average human intepreter. tion to humans, we prompted ChatGPT to perform the same task on a held-out set of 30 examples. We computed the statistical difference between our

---

Formally, the process of optimizing Al systems given goals and constraints has been broadly cate- gorized as Seldonian Optimization (Thomas et al., 2019, 2017). In this framework, the goal is to opti- mize the priorities of an objective function while adhering to given set of constraints as opposed to simply optimizing based on the reward or loss function. (Yang et al., 2020) proposed a Seldonian optimization approach to translate constraints into a feature representation, encoding invalid regions in the state space, which is then applied towards safe RL. However their application is restricted to learning to parse individual constraint statements such as "Don't I get too close to the water," rather than facilitating constraint extraction from more re- alistic descriptions pertaining to an entire strategy. In our work, provide a fost-of-its-kine dataset, and correspondent model, to capacitate seldonian optimization through unstructured language.
3 Natural Language Strategies in RISK
Our work aims to facilitate humans to specify their strategy or commander's 's intent to an Al system via language. In this section, we utilize the board game Risk to create dataset that maps unstructured natu- ral language descriptions of strategies to actionable intent in the form of goals and constraints.
3.1 Board Game RISK
Risk (Gibson et al., 2010) is a multiplayer strategy board game of diplomacy, conflict, and conquest, which was first invented in 1957. The gameplay of Risk consists of four phases: Draft, Recruit, Attack, and Move. The draft phase is conducted at the start of the game wherein each player drafts an initial set of continents and deploys fixed number of troops onto those continents. This allocation of troops is crucial participatory task (Muller and Kuhn, 1993) which involves humans reasoning about their strategy and setting up for the rest of the game. Par- ticipants may choose any of the empty territories on the map to deploy their troops, with a wide range of strategies that may depend on their opponent's s troop allocation. For example, more conservative player may draft troops to only one continent for better defense, whereas a player with more aggres- sive strategy may choose to spread out their troops. After the draft phase, each subsequent turn for a player involves iteratively conducting the recruit, attack, and move phases. Further details about Risk can be found in Appendix-I.
2.2 Language and Strategy Datasets
Prior datasets for instruction following and policy specifications are often comprised of shorter in- structions describing individual tasks. In contrast, our dataset consists of larger, unstructured descrip- tions of strategies which may be more reflective of potential strategy descriptions from in-the-wil wild users. Recent work has published dataset of pol- icy descriptions which are similar to the language descriptions we collect (Tambwekar al., 2021)- however, they describe specific policies, rather than broad strategies for a task. Other datasets look to map language to trajectories or goals states within the trajectory (Padmakumar et al., 2021; Misra et al., 2018; Suhr et al.. 2019). These datasets typically serve as a means of replacing physical demonstrations with language. These datasets lack explicit goals and constraints corresponding to the language collected, that can be applied towards seldonian optimization. Recent work provided a dataset with constraint statements (Yang et al., 2020) which are designer-specific; however, each constraint is associated with an isolated statement, making it unclear whether this approach will gener- alize to unprompted language describing multiple constraints. Unlike prior work, our dataset pro- vides the ability to apply Seldonian optimization approaches from unstructured language. Further- more, we conduct study wherein we provide a human and ChatGPT baseline for our dataset to highlight the challenging nature of this task.
In our setting, we use a map layout that has 5 continents with a total of 21 territories/coumries,, as illustrated in Figure 1. Instead of real country names used in the Risk game, we use ad-hoc names for each continent (e.g., Red, Green, Blue, etc.) to mitigate participant bias. In the draft phase, each player takes turns deploy 14 troops. The specific set of tasks that humans need to complete for our study include: (i) develop a strategy for Risk and deploy 14 troops after the two opposing players have completed their draft; (ii) provide six goals (on 200-point scale) and up to eight constraints that were relevant to their allocation of troops and broader intents; (iii) use natural language to de- scribe their overall strategy and the goals and con- straints they considered. The troops of the opposing player are shown to the participants prior to com- pleting these tasks. More details about this data collection process are discussed in Section 3.3.

---

3.2 Task Definition
each time with different map. The maps seen by participants were selected from a set of 15 unique initial troop settings.
Our goal is to develop computational interface capable of inferring strategic intent from unstruc- tured language descriptions of strategies. Formally, we define the task of Automatic Strategy Transla- tion as follows: Given the troop deployments S, a map M, and the strategy W, which is paragraph written in natural language, our task is to automati- cally derive set of goals G' and constraints C. The troop selections S include the name and number of troops for each territory drafted by the player. We have a total of predefined goals, each of which takes a numeric value between [-100, 100]. This numeric value corresponds to whether the goal pos- itively or negatively aligns with the strategy. For example, for the goal "maximize battles", 100 im- plies that the player intends to battle much as possible, and -10 implies that the player intends to battle as infrequently possible. Each constraint is comprised of a class and value. We restrict the number of possible constraints to 8 as a reasonable upper bound per strategy. To summarize, each ex- strategy
Participants needed approximately 10 minutes per data point. Figure depicts the format of our dataset. Our dataset included data from 230 partici- pants. The average length of language descriptions in our dataset was 99.21 words, and the overall vo- cabulary size was 2,356 words. Additional details regarding our data collection protocol are available in Appendix A.
Automatic Strategy Translation
Following the data collection in Section 3, our goal to leverage this dataset to develop a model that can perform the task of automatic strategy transla- tion. Inferring strategic intent from language is a non-trivial endeavor as unstructured language can be vague thus leading to ambiguous interpretations. We seek to develop an approach capable of per- forming this task better than the average human, so as enable strategy specification via language to reduce the potential risk of human errors or the need of third-party expert interpreters. In this sec- tion, we cover the technical details which make this task possible in a low-data setting.
ample (M, W, S, C, G) C 'D consists W described in natural language, for a player's troop selections, S, on map, M, from which C and G are the gold standard constraints and goals.
3.3 Data Collection
4.1 Text Encoder
We collected dataset D of 1053 unique examples by recruiting participants on Amazon Mechanical Turk and Profilic (pro, 2014). Firstly, to famil- iarize participants with the game, we designed a tutorial that provided a description and annotated examples to explain the rules of the game and the tasks that participants needed to perform. As a further measure of improving data quality, partici- pants were quizzed on the rules of Risk to reinforce their understanding (full quiz has been provided in SA.2). They were given three attempts answer correctly, after which they were shown the answers. Upon completing the quiz. participants began the task. We showed participants a map, which shows the drafted troops of the two opposing players, and asked them to provide their own troop deployments. Following their draft, participants are asked to pro- vide the goals and constraints they considered for their gameplay strategy/deployments and finally provide language description of their strategy. The language strategy they provided needed have at least 200 characters. Each participant was asked to repeat this task 5 times to create data points,
We adopted the pretrained RoBERTa model (Liu et al., 2019) as our encoder which is parameter- ized by 8. The input sequence comprised of the language description of the strat- egy, W [wn, u22 ... w/]. (/j]] and troop selections S = $2...2.... $1 where each troop selection is comprised of the country name along with the number of troops placed on that country (e.g.. S = [Red_A - 2, Red C 8, Purple_ 4)). The encoder learns the embedding function, which maps the text input, comprised of the strategy W and selections S, d-dimensional real-valued vector which then be used towards predicting goals ($4.2) and constraints ($4.3).
model is
Ordinarily, the final embedding for the single [CLS] token learned by RoBERTa, i.e., Ep - B BF. (WW is used for classification. In this work, incorporate multiple classification tokens (Chang al., 2023), each of which corre- sponds to an individual goal constraint. For ith goal or constraint, we learn a separate classification embedding, Eg = BE BERTCCL.. S). Using in- dividual class-specific tokens improves the model

---

Disribution over possibre constraints Constrains A1
Embeddings
6/6
[OR]
Dierisuttio overvolues for Goal IFl
Cru
Output Distibutiooe
Output
Distribution possible constriints Constraint
Roloerta Transformer Blocks
Embed
Ricc
Azz
Ery
Re
G/CC
[OR]
Embedding Layer
Distribution over values for Goal
Ocosfication Tokers
Language Description
Trengy Seleectionss
Multi heod Classifiction
Figure 2: Illustration of our Automatic Strategy Translation model. The input to the model includes the classification tokens, language description, and troop selections (Section 4.1). The encoder then generates embeddings for each classification token, and passes them onto individual classification head. Each classification head is fully-connected layer that predicts a probability distribution for the respective goal ($4.2) or constraint ($4.3).
the capability to learn different attention weights corresponding to the classification embeddings for each goal or constraint. We utilize different en- coders for predicting goals and constraints, which are parameterized by 0g and fg. respectively.
However, unlike for the goal extraction model, each of the eight constraint classification heads learns predict the constraint itself rather than a value for a fixed goal. Therefore, the model needs to pre- dict the set unordered constraints (C1, ...... (g). wherein each Cj is predicted from the set of all pos- sible constraints C' (190 total possible constraints). Each strategy can have maximum of cight con- straints, i.e., the set C' includes a null value.
4.2 Goal Extraction Model
We treat the subtask of deriving goals from lan- guage as an ordinal classification task. Originally, in our dataset goals are specified as continuous val-
While providing constraints during data- collection, participants merely assigned constraints to their strategy, but did not rank the ordering of constraints. As such, the order of constraints our dataset does not necessarily correspond the order in which each classification head needs predict the constraints. Therefore, each classification head does not have a strict label can utilize to com- pute a classification loss, making this task distinct from conventional sequence prediction multi- class classification tasks. For instance, if the con- straints predicted by the model {C, 0, B, D) and the labels for this strategy are {A, B, C, 0}, uti- lizing a standard classification loss function, such as cross-entropy, would result in higher loss than what is representative of the prediction, as three out of four constraints have been predicted cor- rectly. As such, this task requires a loss function that allows us to train our model to predict the cor- rect constraints for language strategy agnostic of the ordering of the labels. We chose to adopt recently proposed loss function called Order- Agnostic Cross Entropy (OaXE) (Du et al., 2021). Intuitively, OaXE is defined as the cross entropy for the best possible alignment of output tokens.
ues ranging from [- 100, 100), which we discretize by creating 5 uniform buckets, i.e., [-100, -60), [-60,- 20)) etc. That is, for each goal, predict ussignment 5-class classification
P3 = Lo, (Eg,).
(1)
where Pj represents the probability distribution across assignments for jth goal and Fij, corre- sponds to the embedding from the encoder. Each goal uses a separate classification layer L parame- terized by Φj. The goal extraction model is trained on dual-criteria loss function that combines cross- entropy (CE) and mean-square-error (MSE) loss:
Lgoat aLCE + (1 LsSE
(2)
where a a simple weighting hyperparameter. The addition of MSE loss helps to account for the ordi- nal nature of goal value predictions.
4.3 Constraint Extraction Model
Similar to the goal extraction model, the input to each classification head for constraint prediction is _ which corresponds to the classification embed- ding learned by the encoder for the tthh constraint.

---

SenteTee Condicate
Content Consistency Filters
Human Data
1010
Orighna
Pegasus
Minimum Edit Distance Filter
So Sentonce Candifote M
Synthetic Data
Paraphrasing
Sentence Candidate
Random Selecttion
Pipeline for augmenting synthetic human- ceeette data ($4.4), strategy description is first split into sentences, then passed into the PEGASUS (Zhang et al., 2020) paraphrasing model and data quality filter.
Figure 3:
Let O {01, 02. O be the ordering space all possible orderings of the target sequence of constraints, where each Oj one possible order- ing of the target tokens. The final loss function is computed as:
sentence from the remaining candidates as a re- placement sentence, and combined the replacement sentences to form an augmented data point (sce Fig- ure 3). The two Synthetic datasets (S, AS) were used to pretrain our model prior to training on hu- man data. The same techniques were also applied to our human dataset to form Augmented Humann dataset (AH). Our final Augmented Human data set is version of our original crowdsourced dataset where each example rephrased using our augmen- tation pipeline and is twice the size of our original human dataset. We experiment with utilizing the AH dataset in place of the original human dataset to see if the added diversity in our corpus through paraphrasing improves downstream performance. Examples of Synthetic (S), (AS), and Augmented-Humne (AH) data are pro- vided in Figure in the Appendix.
LoaxE -logP(O'
(3)
where O" represents the best possible alignment from O. This alignment is computed by applying the Hungarian algorithm, after casting this problem maximum bipartite matching (Du et al., 2021). As our final loss function, we follow Du et al. (2021) in combining OaXE with cross-entropy loss:
Coonstraint Tm CCE + T (4)
where Tm is a temperature parameter that is logis- tically annealed from to O. In our case, cross entropy (LCE) is computed using the default order- ing of labels in our dataset.
Experiments
This section will present the empirical evaluations of our approach. We design two evaluation exper- iments to contrast our model's 's performance with humans, as well as ChatGPT trained to perform our task through in-context learning. Both human and ChatGPT performance was computed using the 30 held-out examples in our test set. We statistically measure the difference in the average number of goals/constraints predicted correctly per data point between our model and the two baselines (Human + ChatGPT). We conclude with an ablation analysis across the model and data augmentations utilized in this approach.
4.4 Data Augmentation Methods
Finally, we applicd data augmentation procedures to improve our model's performance. First, we randomly generated 4000 unique sets of goals and constraints, and applied a text template to produce descriptions to develop Synthetic (S) training cor- pus. For example, the constraint, "I must have troops on Red" could be represented as "My strat- egy is take over Red," or "I need large army on Red," or "I need to place troops on Red." We further augmented this synthetic corpus with a pretrained PEGASUS (Zhang et al., 2020) para- phrasing model to create an Augmented -eemented--Snthetic (AS) dataset. We split each language description from the synthetic corpus into individual sentences and employed the paraphrasing model to gener- ate candidate paraphrases. Sentences that replaced important keywords, such as continent names, or were too similar to the original sentence in terms edit distance were removed. We randomly chose
5.1 Human Performance
In our first study, we ask how well the average hu- can perform the task of parsing strategic intent from language (see Table 1). We recruited 114 participants for our study from Prolific. Par- ticipants begin with a tutorial of the task and are provided an annotated example explaining how to

---

Baseline
Goals (Total
Constraints (Total 8)
Model Type
Data
Pretraining
Accuracy (Std)
Model (Ours)
2,76 + 1.0
5,33 + 1.26
RoBERTabase
44.37 (1.33)
Human
1.87 1.12
4,28 1.83
troop
AH
AS
46.04 (1.85)
ChaiGPT
2.10 1.27
3,80 1,51
troop + [CLS,]]
AH
AS
45.52 (1.48)
w/ troup [CLS.]
AH
S
45,32 (1,01)
Table I: Mean and standard deviations for the number correct predictions of each approach.
troop + [CLS.]
AH
45.89 (1.26)
[CLSS]]
AH
AS
44.29 (1.14)
w/ troop [CLSS]]
H
45.07 (1,33)
assign goals and constraints given language de- scription and map. Following this tutorial, each par- ticipant is provided three randomly selected maps and language descriptions from our test set of 30 unique data points and is asked to annotate the goals and constraints for each given strategy. Our study included attention checks to ensure partici- pants who were submitting random responses could be excluded. The average time taken for our study 21 minutes, and participants were paid $3.6 for completing our task. We utilized data filtering rubric to identify and remove individual data points which were inadequate or were from participants who appeared to blatantly ignore or misunderstand the instructions. The rubric is included in Appendix F. After filtering, total of 270 responses remained.
Table 2: Ablation study (10-fold cross-validation) with respect to model and data augmentations for goal ex- traction. the synthetic dataset created from templates; AH/AS: the augmented version of H/S via paraphrasing ($4,4), [CLS,]] represents use of individual classification to- kens for each goal/constraint ($4.1); 1); "troop" represents the inclusion of troop selections as a part of the input.
human-created dataset ($3.3); S:
Model
Data
Pretraining
Accuracy (Sid)
RoBERTa
62.60 (1.60)
w/ troop + [CLSS]
H
S
68.21 (1.08)
w/ trropp [CLS,]
AH
S
67.79 (1.58)
troop [CLS,]
AS
67.09 (1.28)
troop
S
65.96 (1.12)
w/ troop [CLSS]]
H
65.76 (1.13)
w/ troop [CLS,]
AH
65.52 (1.42)
[CLSS]]
65.31 (1.12)
5.2 ChatGPT Performance
Table 3: Ablation study (10-fold cross-validation) for constraint extraction.
We also evaluate ChatGPT (GPT-3.5 Default) as a baseline for our task (see Table 1). We design a 1000-word language prompt to train ChatGPT to perform the same task (see full prompt in Ap- pendix G.1). This prompt includes description of the environment and task, as well as an annotated example translating goals and constraints from lan- guage. Crucially, we design our prompt such that ChatGPT receives the same information that hu- mans receive in our study in $5.1. Following this prompt, we iteratively input each strategy and troop deployment in our test set and store the constraints selected by ChatGPT. The additional prompt engi- neering we conduct is to notify ChatGPT when it makes formational mistakes while predicting con- straints, such as predicting more than the maximum number of constraints or creating new constraint classes.
method for determining a model-fit so to choose the regression model which best fits our data. For the goals model, modeled each baseline (hu- man vs. model vs. CharGPT) as fixed effects cariate, and the datapoint number as a mixed effects variable. The datapoint corresponded to the numerical index (between 300) of the data- point from the test set. We performed the Lev- ene's 's test (Glass, 1966) to show homoscedasticity (F (2, 327) 0.5435, p 0.581). The residuals for model were not normally distributed; how- ever, prior work has shown that F-test is robust non-normality (Blanca Mena al., 2017; Cochran, 1947). Therefore, we proceeded with our linear regression analysis. The dependent variable within our analysis was the number of goals predicted cor- rectly. An ANOVA with respect to our dependent variable yielded a significant difference across con- ditions (F(2, 299.95) 10.605 P < 0.001). A Tukey post-hoc test (Abdi and Williams, 2010) for pairwise significance further revealed significant difference between the performance of our model vs humans (p < 0.001) and vs ChatGPT (p < 0.05). i.e., our approach was able to significantly predict
5.3 Results for Goal Extraction
The average number of goals predicted correctly per map can be seen in the first column of Table 1, We applied multivariate linear regression to com- pare the resuls of our model with our ChatGPT and human baselines, with Akaike information criterion (AIC) as our Occam's 's razor, AIC is mathematical

---

goals better than humans and ChatGPT.
Baseline
Constraints
Goals
Roberta-base (Best)
68.21 (1.08)
46.04 (1.85)
5.4 Results for Constraint Extraction
GPT-Nco 125M (Best)
65.22 (1.21)
46.08 (0.73)
The average number of constraints predicted cor- rectly per map be seen in column 2 of Ta- ble 1. To compare our constraint prediction model, to our human and ChatGPT baselines, we con- ducted -on-aaaaecteric Friedman's test (Pereira et al., 2015). We could not employ a multivari- ate regression analysis, as the regression model for constraints did not satisfy the assumption of ho- moscedasticity per Levene's test (2, 327) 5.4294, p < 0.01). The Friedman's 's test yielded significant difference across conditions for the task of predicting constraints (2, 90) 16.768, p < 0.001). A further pairwise Wilcoxon signed rank test (Woolson, 2007) revealed a significant differ- ence between humans and our model (p < 0.001) as well as ChatGPT and our model (p < 0.001), indicating that our approach is not just able to sig- nificantly outperform humans, but also ChatGPT for inferring constraints from language.
Table 4: This table depicts the performance when the roberta-bas base encoder is substituted with SOTA autore- gressive model, i.e. GPT-Neo (125 million parameters).
5.6 Ablation Study
Tables 3 and 2 provide the results from ablating each model augmentation discussed in Section 4. The effects of these augmentations are more promi- nent in the model for predicting constraints (~ 6% performance boost) than predicting goals (~ 1.5% performance boost). For the constraints model, when any parameter, i.e. troop selections, pretrain- ing, or CLS-Tokens, were removed, the accuracy dropped by ~ 3% individually. For predicting goals, the inclusion of troop selections was the only model-augmeettation which seemed to have decisive impact performance, as all models with selections had an accuracy of ~ 1% more than those without. We attribute the difficulty in im- proving the performance of the goals model to the contextual ambiguity for values assigned to each goal. Participants may not always follow the same metric while specifying goal values. Each partic- ipant could have a unique interpretation, for what any rating between -100 to 100 means for par- ticular goal, and description of that value through language(see Appendix for the data distribution corresponding to each goal). This disparity in in- terpreting values could be affecting the consistency of language descriptions for goals in our dataset. Finally, the last ablation conducted studied the effect of the type of encoder utilized in our ap- proach. Therefore, we performed a comparison with model which replaced the encoder with a SOTA pretrained autoregressive model. We uti- lized GPT-Neo (Black al., 2021) for our experi- ments, it has the same number of parameters as Roberta-base (125 million). Our findings (see Ta- ble 4) show that utilizing an autoregressive model as our encoder offers no benefits to a roberta-base model, the GPT-Neo model performed equivalently for predicting goals and about 3% worse for the constraints model.
5.5 Discussion
Our results emphasize that inferring strategic intent from language is non-trivial task, language in- terpretation can be subjective and malleable. Chat- GPT is capable of performing novel tasks such as text classification (Li et al., 2023), mathemat- ical problem solving (Frieder et al., 2023), and information extraction (He et al., 2023), through in-context learning. However, despite these capa- bilities, our model was found to significantly out- perform chatGPT for inferring strategic intent from language. Success in highly specific and complex language interpretation tasks, such as ours, requires the model to build an understanding of the domain and the task itself as generic language interpreta- tion learned by the majority of pretrained language models may not be applicable.
Recent work on evaluating open question- answering on a challenge-dataset has shown that even for large-scale language models with between 6B-100B parameters, none of these models out- performed humans (Peinl and Wirth, 2023). By developing a computational interface which can infer strategic intent from language significantly better than humans, we show the usefulness of our pipeline towards solving complex domain-specific task a low-data, -resource setting.
Conclusion
In this paper, we develop a novel computational in- terface to automate inferring strategic intent, in the

---

form of goals and constraints, from unstructured language descriptions of strategies. We develop a new benchmark for our dataset and broader task, and further conduct a novel head-to-head evalua- tion to determine the relative efficacy of our ap- proach. We show that in low-data setting, our ap- proach towards inferring goals and constraints from language strategy descriptions can significantly outperform humans for the same tasks. Further- more, we also found that our approach, with only 125 million parameters, was able significantly outperform ChatGPT for inferring strategic intent from language. Our work endows researchers with valuable tools to further seldonian optimization ap- proaches for
referencing the terminology utilized in the goals and constraints. While our dataset provides expla- nations that are the closest to natural, human-like descriptions of strategies, an important next step would entail comparing how our model performs on strategies collected "in-the-wild." Secondly, in this paper we assume that utilizing language is more accessible than learning to use mathematical specifications directly to specify their intent intelligent agent. However, we do not test whether this assumption bears out practice. In future work, we hope to develop human-subjects study to confirm this hypothesis. Finaly, despite convert- ing language to goals and constraints, in this work we do not directly train seldonian optimization approach. In this work, we focus on showing the capability of our machine learning pipeline in low-data setting. However,
interaction.
Future Work
have provided all the components needed train a reinforcement learning approach for an RL-agents constraining behavior through unstructured language (including a novel open-Al RL domain for the game Risk, see Appendix). Developing this approach is currently outside the scope of this work, and we thereby leave this exploration for future work.
To measure ChatGPT performance, we employ one-shot chain-of-thought prompt method with detailed instructions of the task. We chose this method to maintain consistency between the in- formation shown to humans and ChatGPT. Future work may explore ablations on the size of the initial prompt or the number of annotated examples in the prompt to tune the performance of ChatGPT on our strategy translation task. Secondly, an impor- tant next step that stems from this research pertains to multi-round inference and updating the initially learned strategy. In future work, it would be help- ful to develop methods to allow users to modify their initial strategy throughout the game or task as their goals or values change. These methods could utilize approaches proposed in prior work wherein language inputs were leveraged to change the sub-goals that agent is considering (Fu et al., 2019; Goyal et al., 2019). Furthermore, recent work has shown promise for the capabilities of ChatGPT/GPT-3.5 towards dialog-state tracking and task-oriented dialog (Labruna et al., 2023; Heck et al., 2023). Future work could also for- mulate this task of updating the initial strategy over the course of the game as a goal-oriented dialog. and tune GPT-3.5 or GPT-4 to update a user's 's ini- tially translated strategy after multiple rounds of the game through language feedback.
Ethics Statement
As pretrained large-language models are utilized in our approach for automated strategy translation, we need to cognizant of the prevalence of bias within these models. these systems are translat- ing strategies in safety-critical settings, it is impor- tant to make sure that the language models make the decisions solely based on the provided context rather than any inherent bias. Many sets prior work have studied approaches to identify and mitigate bias (Abid al.. 2021; Silva et al., 2021b; Guo et al., 2022; Viswanath and Zhang, 2023). We en- courage authors to seek out such works prior to deploying any strategy translation module, towards a real-world task.
Acknowledgements
This work was supported by the Office of Naval Re- search under awards, 100014-19 N00014- 22-1-2834, N00014-23-1-2887, and the National Science Foundation under award, FMRG-2229260.. We also thank Konica Minolta for their contribution to this work via a gift to the Georgia Tech Research Foundation.
Limitations
Firstly, we asked participants to provide natural lan- guage deseriptions after providing their structured intent the form of goals and constraints. This po- tentially biased the participant towards specifically

---

References
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. 2019. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint SeKd::9902.07744
2014. Online participant recruitment for surveys and market research.
Hervé Abdi and Lynne J Williams. 2010. Tukey's s hon- estly significant difference (hsd) test. Encyclopedia of research design, 3-1):1-5.
Richard Gibson, Neesha Desai, and Richard Zhao. 2010. An automated technique for drafting territories in the board game Risk. Proceedings of the AAAI Confer- ence Artificial Intelligence and Interactive Digital Entertainment, 6 6(1):15-20.
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent muti-muslim bias large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298-306.
Gene Glass. 1966. Testing homogeneity variances. American Educational Research Journal, 3(3):187- 190.
Léo Andeol. 2018. Leoandeol/gym-risk: Gym environ- ment for the risk game by hasbro.
Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask reinforcement learning with pol- icy sketches. International Conference on Ma- chine Learning, pages 166-175. PMLR.
Nakul Gopalan, Dilip Arumugam, Lawson Wong, and Stefanie Tellex. 2018. Segueece-to-Sequeee Lan- guage Grounding of Non-Markovian Task Specifi- cations. In Proceedings Robotics: Science and Systems, Pittsburgh, Pennsylvania.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Ed- ward Grefenstette. 2018. Learning to understand goal specifications by modelling reward, arXiv preprint arXiv:1806.01944
Prasoon Goyal, Scott Niekum, and Raymond Mooney. 2019. Using natural language for reward shap- ing in reinforcement learning. arXiv preprint 103.:009..
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. you use this software, please cite using these metadata.
Yue Guo, Yi Yang. and Ahmed Abbasi. 2022. Auto- debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1012-1023.
María José Blanca Mena, Rafael Alarcón Postigo, Jaume Arnau Gras, Roser Bono Cabré, Rebecca Ben- dayan, al. 2017. Non-normal data: Is anova still valid option? Psicothema.
Philip Hayes. 1985. The utility of natural language interfaces (panel session). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, page 19.
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross Knepper, and Yoav Arizi. 2019. Learning to map natural language instructions to physical quad- copter control using simulated flight. arXiv preprint arXiv:1910.0966
Jiabang He, Lei Wang. Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. 2023. Iel-d3ie: In-context learning with diverse demonstrations updating for document information extraction. arXiv preprint arXiv:2303.050633
Haw-Shiuan Chang, Ruei-Yao Sun, Kathryn Ricci, and Andrew McCallum. 2023. Multi-CLS BERT: An efficient alternative to traditional ensembling. In Proceedings of the 60th Annual Meeting of the Asxo- ciation for Computational Linguistics.
Michael Heck, Nurul Lubis, Benjamin Ruppik, Re- nato Vukovie, Shutong Feng, Christian Geishauser, Hsien-Chin Lin, Carel van Niekerk, and Milica Gašié. 2023. Chatgpt for zero-shot dialogue state track- ing: A solution or an opportunity? arXiv preprint arXiv:2306.01338
William G Cochran. 1947. Some consequences when the assumptions for the analysis of variance are not satisfied. Biometrics, 3(1):22-38.
Richard Dempsey and Jonathan M Chavous. 2013. Commander's 's intent and concept of operations. Mili- tary Review, (3):5666.
Arie W Kruglanski. 1996. Goals as knowledge strue- tures. P. M. Gollwitzer & J. A. Bargh (Eds.), The psychology of action: Linking cognition and motiva- tion to behavior, pages 599-618.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order- agnostic cross entropy for con-autoreeressive ma- chine translation. arXiv preprint 1rXiv:2106.05093.
Geert-Jan M Kruijff, M Janiček, Shanker Keshav- das, Benoit Larochelle, Hendrik Zender, Nanja JJM Smets, Tina Mioch, Mark A Neerinex, Jurriaan Van Diggelen, Francis Colas, et al. 2014. Experience in system design for human-robot teaming in urban search and rescue. In Field and Service Robotics. pages 1-1-2 Springer.
Simon Frieder. Luca Pinchetti, Ryan-Rhn Grif- fiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.1388

---

Tiziano Labruna, Sofia Brenna, Andrea Zaninello, and Bernardo Magnini. 2023. Unraveling chatgpt: A crit- ical analysis of ai-generated goal-oriented dialogues and annotations. arXiv preprint arXiv:2305. 14556.
Ian C. Rankin, Seth McCammon, and Geoffrey A Hollinger. 2021. Robotic information gathering us- ing semantic language instructions. In 2021 IEFF International Conference on Robotics and Automa- tion (ICRA), pages 4882-4888. IEEE.
Jiazheng Li, Runcong Zhao, Yulan He, and Lin Gui. 2023. Overprompt: Enhancing chatgpt capabilities through an efficient in-context learning approach. arXiv preprint arxiv:2305.
Joseph Rosen, Eliot Grigg, Jaron Lanier, Susan Mc- Grath, Scott Lillibridge, David Sargent, and C Everett Koop. 2002. The future of command and control for disaster response. IEEE engineering in medicine and biology magazine. 21(5):56-68.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprini arXi::2211.09112
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2022. Cliport: What and where pathways for robotic ma- nipulation. In Conference on Robot Learning, pages 894 906. PMLR.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint /90i:.v907.119999
Andrew Silva, Nina Moorman, William Silva, Zul- figar Zaidi, Nakul Gopalan, and Matthew Gombolay. 2021a. Lancon-learn: Learning with language to en- able generalization in multi-task manipulation. IEEE Robotics and Automation Letters.
Joseph E Mercado, Michael A Rupp. Jessie YC Chen, Michael Barnes, Daniel Barber, and Katelyn Procci. 2016. Intelligent agent transparency in human-agent teaming for multi-uxv management. Human factors, 58(3):401-415.
Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay. 2021b. Towards a comprehensive under- standing and accurate evaluation of societal biases in pre-trained transformets. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2383-2389.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environ- ments with visual goal prediction. arXiv preprint arXiv: 889.00788.
Alane Suhr, Claudia Yan, Jacob Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem. Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. arXiv preprint arXiv:1910.0365
Gordon B Moskowitz and Heidi Grant. 2009. The psy- chology of goals. Guilford press.
Michael Muller and Sarah Kuhn. 1993. Participatory design. Communications of the ACM, 36(6):24-28.
Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, and Matthew Gombolay. 2021. Interpretable policy specification and synthesis through natural language and RL.
Thomas Nickles, 1978. Scientific problems and con- straints. In PSA: Proceedings of the biennial meeting of the Philosophy of Science Association, volume 1978. pages 148. Philosophy of Science Associ- ation.
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. 2020. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55 55.
David G Novick and Stephen Sutton. 1997. What is mixed-initiatee interaction. In Proceedings of the
AAAI spring symposium on computational models for mixed initiative interaction, volume 2. page 12.
Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, and Emma Brunskill. 2017. On ensuring that intelligent machines are well-behaved. arXiv preprint arxiv:1708.05448.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Varayan-Cher Span- dana Gella, Robinson Piramuthu. Gokhan Tur, and Dilek Hakkani-Tur. 2021. Teach: Task- driven embodied agents that chat. arXiv preprint 1
Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. 2019. Preventing undesirable behavior of intelligent machines. Science, 366(6468):999-1004
René Peinl and Johannes Wirth. 2023. Evaluation of medium-large language models zero-shot closed book generative question answering, arXiv preprint 1.Xiv:3355.11911.
Jesse Thomason. Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J Mooney. 2019. Improving grounded natural language under- standing through human-robot dialog. In 2019 In- ternational Conference on Robotics and Automation (ICRA), pages 6934-6941. IEEE.
Dulce G Pereira, Anabela Afonso, and Fátima Melo Medeiros. 2015. Overview of friedman's 's test and post-hoe analysis. Communications Statistics- Simulation and Computation, 44( 44(10):2636-265)

---

Hrishikesh Viswanath and Tianyi Zhang. 2023, Fairpy: A toolkit for evaluation of social biases and their mitigation in large language models. arXiv preprint arXiv:2302.05508
bias, due to the specific wording of the goals and constraint templates, which could have impacted the strategies submitted by the participants. For goals, participants were asked to rate how impor- tant each goal was to their strategy on a scale of -10 to 100. A score of -00 indicated that pursu- ing the goal was completely detrimental to their strategy, while 100 indicated that pursuing the goal was essential to their strategy. For constraints, par- ticipants were provided 9 constraint templates, and were asked to select and fill in the appropriate con- straint that was represented in their strategy. Partic- ipants were required to provide at least three con- straints to ensure that they did not skip this question. The specific goals and constraints in our dataset can be depicted in Table 5. Finally, participants were asked to summarize their strategy for the given map as a language description. Participants were encouraged include references their goals and constraints, but these descriptions were otherwise unprompted. Participants were paid up to $8.5 based on the number of adequate responses submit- ted. The payment scale was updated if the average time taken significantly changed.
Edward Williams, Nakul Gopalan, Mine Rhee, and Stefanie Tellex. 2018. Learning to parse natural lan- guage to grounded reward functions with weak super- vision. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4430-4436 IEEE.
Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, pages 1-3.
Tsung-Yen Yen Yang. Michael Hu, Yinlam Chow, Peter J Ramadge, and Karthik Narasimhan. 2020. Safe rein- forcement learning with natural language constraints. arXiv preprint arXiv:2010.05150.
Jingqing Zhang. Yao Zhao, Mohammad Saleh, and Pe- ter Liu, 2020. Pegasus: Pre-traine Pre-training with extracted gap-sentences for abstractive summarization. In In- ternatiional Conference on Machine Learning, pages 11328-11339. PMLR.
A Additonal Data Collection Details
Our study applied participatory design princi- ples (Muller and Kuhn, 1993), to ensure that par- ticipants were engaged in the task and provided meaningful strategy descriptions. Each participant was initially given a partially setup map, where two other ""opponents" had placed their troops. The participant was then asked to provide their troop placements, based on these initial placements. In Risk, the initial troop placements have a substantial impact on the strategies that player can pursue for the rest of the game. As such, troop initial- ization provides a stand-in for player's 's overall strategy in a game. By asking participants to par- ticipate in an actual aspect of the gameplay, e.g. deploying troops, participants were encouraged en- vision future situations and think about how their decisions could affect future gameplay and develop grounded strategies that could actually function as viable Risk gameplay strategies.
As mentioned in the paper, we created three addi- tional augmented datasets from our original corpus. Figure 6 provides some examples of the effect of the various augmentations we employed in each augmented dataset. Our full dataset can be found at the following anonymized Github repository Anonymized Data Repository.
A.1 Data Cleaning/Filtering
We performed the least possible modifications to participant's responses to ensure responses were self-consistent while preserving the integrity of the organic data collection task. If participant specifi- cally referenced goal constraint in their lan- guage, and did not include in their response, then their response was modified include it, and vice versa. We also corrected typos within a participants specifications, such as if they meant to reference the "Blue" continent instead of the "Red" continent. If a response was not salvageable without mini- mum modifications, the response was thrown out. Discarded responses included responses where par- ticipants simply did not understand the task sub- mitted blatantly insincere responses such as copy- ing text from the study multiple times to reach the character limit. These decisions were made upon agreement of multiple reviewers.
Next, participants were asked to provide the goals and constraints which they considered af- ter selecting their troop placements. These specific goals and constraints were selected as they cater to potential strategies that could be employed while playing Risk. The presence of these templates pro- vided scaffold within which participants, who may or may not have any experience with Risk, could ground their strategics. However, it is impor- tant to acknowledge the presence of an inductive

---

Goal Surround enemy territories
700
Goal Maximize countries occupled
700
Goel 3:
close together
Data Points
Number of Deta Polnts
602
000
502
500
100
600
400
Number
200
2022
200
150 30
61:20
20.22
20.00
20-100
300:44
40:26
-30-0
20:60
03:100
0-
3000-00
40:35
30.23
22:00
50.105
Ranges
Ranges
Ranges
Goal Maximize battles
Goal 5: Fortify borders
.Spal Battles opposing players Individually
Datt Points
Nember of Data Pointts
600
600
500
500
500
400
100
-02
300
Namber
100
102
100
Goal Ranges
Goal Rangos
Goal Ranges
Figure 4: Distribution of assigned values for each goal. The titles for each goal have been shortened for readability.
Distribution of Constraints in the Dataset
Number of Exemples Deleret
1500
1000
150
500
230
Imuus laye troopa of
need to protect the harders of
Emmstt et koautt
Imus hovel at losut
masi havie at lcavt
Freeed least
must treeps mout
Imsss be able sccomt
must haver tntoper
Eorstraine cless
Figure 5: Distribution of assigned values for each constraint type
A.2 Data Collection Quiz
2. What is the objective of the game?
In order to ensure that participants understood the rules of Risk prior to providing strategies for our dataset, each participant was asked answer five question quiz. Participants needed to answer all questions correctly to proceed. Participants were given three tries answer the questions after which they were shown the correct answers. The five ques- tions in our quiz were as follows (correct answers to each question are bold):
(a) Control the rightmost continent
(b) Have the maximum number of island ter- ritories
(c) Have the most territories after 10 turns
(d) Occupy all territories on the board
Which of these decides how many troops you receive at the start of each turn? (TWO COR- RECT ANSWERS)
(a) The number of territories you control
1. Which of these are NOT phase in the game?
(b) The number of coastal territories on the map
(a) Attack
(b) Recruit
(c) They physical size of the board game
(c) Control opponent's troops
(d) The number of continents you fully oc- cupy
(d) Maneuver

---

4. Which of the following statements correct about attacking enemy territories in the game? (TWO CORRECT ANSWERS)
or Q&A comprchension--acc methods, train AI agents to answer questions regarding user's preferences by reading their strategy description.
(a) When you attack a territory you've 've al- ready attacked, your attack points are doubled
C Dataset Distributions
The data distribution for goals and constraints se- lected by participants are shown in Figure 4 and Figure respectively. For Goals 3 (Keep your troops close together) and (Maximize Battles) participants tended skew towards answers in the 60-100 range. For the other goals, the responses were relatively uniform. On average, participants submitted 5.62 unique constraints per response.
(b) You CANNOT attack in the opposite direction of the arrows
(c) You can only attack territories you have access
(d) You can never attack territory in the same continent
5. Which of the following statements are true regarding how attacks are conducted? (TWO CORRECT ANSWERS)
D Implementation Details
Hyperparameters for both models were computed through a grid search over parameters. The con- straints model was trained for 10 epochs with a batch size of 16 using a learning rate of 0.0005. The goals model was trained for 25 epochs with batch size of 8 using learning rate of 0.00001. The constraints model was Both models utilized Adam W optimizer. The constraints model em- ployed cosine learning rate scheduler, and the goals model employed a linear learning rate sched- uler. We hold-out 30 randomly selected examples for our human/ChatGPP evaluation (Section 5). We split the remaining 1023 examples into a 85/15 train/validattin split to perform our grid search over hyperparameters. Finally, to report the accu- racy of our model we computed the 10-fold cross- validation accuracy on the best performing hyper- parameter setting. The best performing model for predicting constraints was pretrained on the syn- thetic corpus and trained on the un-augmented human corpus. The best goals model was pre- trained on the synthetic-augmented dataset and trained on the auman--uggmntedd dataset. All ex- periments were conducted on 48GB NVIDIA Quadro RTX GPU. Our code can be found at the following anonymized repository for further refer- ence Anonymized Code Repository.
(a) player with scattered troops always wins
(b) player attacking from the left side al- ways wins
(c) Both players roll number of dice de- pendent on the number of their troops involved in the battle to decide the out- come
(d) player can attack with up 3 troops and defend with up to troops in one battle
B Dataset Utility
This section provides a brief discussion on the po- tential future utility of our collated dataset. Firstly, this dataset provides strategy specifications Risk that can be used to test seldonian optimization ap- proaches in future work. Our dataset provides the first such instance language descriptions of strate- gic intent. Future work can analyze the flaws and strengths of our data modify our data collection protocol and generate the specific examples they may need for their individual applications. How- ever, there are many tangential applications for this data that are unrelated to the use-case specified in this paper. There is a dearth of natural language datasets which contain language with human-like speech patterns that is not scraped from internet- corpora. Many NLP techniques can be applied to further study this language data such summa- rization, to figure whether these policies can be summarized into more easily digestible format, sentiment analysis, for broadly categorizing the lan- guage description into aggressive, defensive, etc.
E Human Evaluation Study Additional Details
In this section, report some additional details garding our human-evaluation experiment. Firstly, we report that on average, the difference between scores for a participant's first and last response was -0.2 143 for goals and -0.0102 for constraints, indi- cating that there is a negligible impact of factors

---

Synthetic Data
Synthetiz-Augmeene Data
Why would care about battling. plan to attack players garne one at time. don't think can handlo having troops more than 2 continents. need to spread my troops out as far as possible. cantt win put any troops on Blue. need to place troops least 5 countries. This time will use different
Idon't know why I care about fighting. I plan to attack players in the game one time, don't think can handle having troops on more than continents. My troops need be spread out as much as possible. If Blue, will not win. need to place troops on at least 5 countries. will be using different strategy this time. need have troops continents. I don't intend to control continents.
put any troops
strategy
need to have troops at least continents. don't intend to control continents.
least
Human Data
HumaanAAggented Data
am going to attack and take over green c. That country ripe the taking since have off from other grey troops, I also want 4 troops to present strong green
am going to attack and take over green c. Since cut off from other grey troops, country ripe taking, also want 4 troops to present strong force in green in case grey attack from yellow d. will move my armies the red continent fight black once the green continent is secure, Hopefully, while this is going on grey and black will be fighting over yellow and blue, but in case they don't I'm keeping all of my troops together Green.
case grey attack from yellow d. Once the green continent secure I will look to move my armies out to the red continent to battle black there, Hopefully, while this going grey and black will fighting over yellow and blue, but in case they don't I' keeping of my troops together Green
Figure Examples of data from Synthetic (top-left), Senthetic-Augmentt (top-right), Human (bottom-left) and Sluman-Augmente (bottom-right). Highlighted sections represent the specific sentences changed by our augmentation procedure.
Goals
Constraints
GI: Survoun enemy lerritoris
C1 : must have troups (continent)
G2: Maximize number of countries occupied
C2: must not have troops on (continent)
G3: Keep our troops close together
C3: I must be able to access (continent) in one move
G4: Maximize hatiles throughout the game
C4: I need to protect the borders of (continent)
G5: Fortify borders for the continents you control
C5 3: need total of at least (number) troops to defend continent
G6: Battle opposing players one at time
C6: must have at lenst (number) countries
C7: must have troops on al least (number) continents
C8: must place at least (number) troops to effectively defend a country
C9: must have troops on at most (uumber) continents
Table 5: Goals and Constraints Selected for our Dataset
such as cognitive load or a learning curve. Sec- ondly, it is important to note that we did not have the same number of responses per map from hu- mans, as the map condition was randomly assigned each participant. While this may slightly impact the results of the constraints model, as we aggre- gated performance across maps, due to the strong significant difference across baselines, it is unlikely to change our result.
2. If someone has submitted invalid constraints
e.g. If someone selects both "I need troops at least continents" "I need troops at most continent"
If someone mistakes "country" for "conti- nent"
3. If someone has selected the same value for all goals (or values within small range, say +-10), when this clearly does not align with the strategy
F Human Evaluation Study - Data Filtering Rubric
e.g. someone selects -100 for all goals when the strategy involves protecting a con- tinent
Next, we cover the rubric we applied to filter data for the human-subjects study. Each response was independently evaluated by two graders and was included if both graders deemed it acceptable as per the predefined rubric. The rubric was follows:
G ChatGPT Prompt
We utilized the following prompt for ChatGPT which included description of the domain and task, as well as an annotated example.
1. If constraints clearly don't match the selec- tions for locations or access
G.1 Full Prompt
e.g. if someone has selected, "I must have troops on Blue" when there are no troops on Blue
Reading the following section carefully will pro- vide you with the information needed to complete

---

this task.
must have troops on [Continent]
Risk is a board game which an army com- mander tries to take over the world by defcating all enemy troops and controlling all countries. Risk is a simplified version of real conflict, and has rules designed to reflect this. These include the follow- ing:
must not have troops on [Continent]
must be able to access [Continent] with one move
need to protect the borders of [Continent] Players control countries by having troops in them
need a total of at least [Number] troops to defend continent
The more countries and continents player con- trols, the more resources they get
must have least at least [Number] countries must have troops on at least [Number] conti- nents
Players win countries from other players by bat- tling with their troops
must place at least [Number] troops to effec- tively defend a country
The more troops a player has when battling, the more likely they are to win
I must have troops on at most [Number] conti- nents
Players can only attack or be attacked by coun- tries that are next to them
The possible constraint values you choose from are
In this task, you will be asked to provide a set of constraints corresponding to the human player's strategy for the board game Risk. This includes their troop placements and text description, which explains why the player decided to place their troops and how they plan win this game of Risk given their opponents' choices.
Continent Blue, Green, Yellow, Red, Purple
Number 1,23445.6..9.10011,1,
Our modified RISK Map contains 5 continents Red, Green, Purple, Yellow and Blue. Each con- tinent is made up of countries. Red continent has 3 countries, Green has 5 countries, Purple has 5 countries, Yellow has countries and Blue has 4 countries. Green A, Yellow _,, Blue C, etc. are referred countries or territories Green, Yel- low, Blue, Red, Purple are referred to as continents. Continents also have different connections between them through which the troops can move. These connections are one way i.e troops from the source country can only move to the destination country and not the other way round.
Your task will be to think about the player's strat- egy (selections and description) and predict what their constraints are with respect to the strategy. Constraints are rules that you think need to be fol- lowed to successfully execute a strategy. CONSTRAINTS: Note: For predicting goals, this section would be replaced with description of what goals are
Constraints are comprised of constraint classes and constraint values. Your job is to assign con- straints to the human's strategy. Each constraint is comprised of constraint class and constraint value. You will be provided a list of possible con- straint classes and values to choose from. You may choose the same class constraint more than once, but you may not submit duplicate constraints. For example, you may submit "I must have troops on Green" and "I must have troops on Blue" but you may not submit "I must have troops Green" twice. Choose all constraints relevant to the strat- egy. You may choose up to 8 constraints per strat- egy.
The map has the following connections Yel- low
connected to Green A, Greed is connected to Red A, Red A connected to Green D, Red B is connected to Purple Red_C is connected to Yellow_BB Red_C C is connected to Blue_ B, Blue_A is connected to Yellow_C, Yel- low_ C is connected to Blue_D, Blue_C is con- nected to Purple_A, Purple_A is connected Green E and Green E connected to Purple A We will now give you a tutorial on how ascer- tain the goals from human player's strategy and placements on the RISK board.
The constraints you can choose from are

---

The two opposing players are denoted by the "grey" and "black" player. In this scenario, the grey player has placed its troops on the following territo- ries 5 troops on Yellow_C, 4 troops on Yellow_D, troop on Red_A, A, 2 troops on Red_ B, 2 troops on Red_C. The black player has placed its troops on the following territories 4 troops Blue_A, 2 troops on Blue_ mive__ 2 troops on Green E, 5 troops on Purple_A A and troop on Purple_B. _B
Reason: The player mentioned that "with 7 troops on Purple_Es E, cannot be beaten on Purple"
H
Risk Reinforcement Learning Simulator
We have shown that our proposed computational in- terface can remove the need for -uman-interpreters for the task of parsing intent from unstructured lan- guage. However, to test how well commander's intent interpreted from language can be applied to- wards optimizing an agent's behavior, we require a reinforcement learning domain to train our agent. As such, to enable seldonian optimization, via un- structured language descriptions, we developed a novel open-ai gym enviromment for simulating Risk gameplay. This environment closes the loop on the methods presented in this paper by providing all the necessary components for humans to specify their intent to an Al agent and evaluate whether their specifications have been satisfied by the learnt agent. Our environment also provides an additional means of collecting data and conducting studies for human-specification within multi-player team scenarios.
Now that you have seen where the opposition troops are, you will now be shown how the human player has decided to deploy their troops and the strategy they used.
The human player (white) has placed 14 troops to battle the opponents. They have placed the troops on the following territories 7 troops on Purple_ E, 5 troops Purple_ and 2 troops Purple_ D. You will now be guessing the constraints the human player (white) focused on while coming up with their strategy. The following text contains the human player's s description of the strategy they used to place their troops. It is critical that you read this description, as it contains information about the constraints considered by the human player.
"I put all my troops in Purple, because I felt as though I needed all my available troops to defend Purple wanted to protect Purple With 7 troops on Purple_ I feel like cannot be beat on purple. wasn't t too keen on getting involved in battles, or taking an overly aggressive strategy. would like to focus on beating the black player first, I don't 't think can battle two people at the same time. I'm going to avoid Red for now since it seems to be the hardest continent to control."
For this task, adapted an existing open-ai gym environment for Risk (Andeol, 2018). We modified the codebase allow for RL agents to be trained to play all phases of Risk, according to the setup utilized in our approach. We also developed a pygame-UP UI for our simulator (see Figure 7). A de- tailed description of the functionality of the domain and the state space is provided in the appendix. In future work, we aim to leverage our domain to de- velop approaches which allow humans to constrain an agent's optimization methods through human- like language specifications of intent, which has not been accomplished in any prior work. We also provide a link to an anonymized github repository with the risk environment for further reference Anonymized Gym-Risk Environment
We will now show you how to determine con- straints from strategy and via an example. Please carefully review the example and use the given in- formation about both selections and text to fill out constraints for this strategy.
An appropriate set of constraints for the strategy shown above would be
must have troops on Purple
I
Risk Domain Additional Domain Information
Reason: The player mentioned that "they put all their troops on Purple"
This section provides additional information about setup for Risk Domain. In our version of Risk. the ego player (Alpha), plays against two oppo- nents (Charlic and Bravo) whose actions are con- trolled by pre-determined heuristic. The game- play within our Risk simulator is comprised four phases
I must not have troops on Red
Reason: The player mentioned that "they would like to avoid Red for now"
must place at least 7 troops to effectively de- fend a country

---

VElTA
Grea1
A maximum of three troops are chosen from the attacking territory, and a maximum of two troops are chosen from the defending territory
2. For both the attacker and defender, a number of die are rolled based on the number of troops involved in each attack.
Had
3. The rolls are sorted in descending order, and each roll is compared between the attacking and defending country.
Bae
Purpla
SHATLE Parpec soc
Figure 7: This figure shows our Risk simulator with the playable (teal) and two other (orange and pink) agents.
4. For each comparison, the country with the lower roll loses one troop. The defending territory wins all ties.
1. Drafting Players draft their initial troops empty territories.
5. The above steps repeated until either the at- tacking or defending player has been defeated.
2. Reinforce Players assign reinforcements to their existing territories.
Following combat, the agent can move but one troop into the conquered territory. Once the agent has finished attacking, they move on to the final phase in their turn, "Freemove." In the "Freemove" phase, the player can move troops from one ter- ritory they control to another, as long as the ter- ritories connected. Once the agent executes all their actions, the actions of the two agents are simulated and the player reset to the "Reinforce" phase to start their next turn. The game is complete when either the agent is out of troops or controls all territories.
3. Attack Players can choose to attack a neigh- boring territory with their troops.
4. Freemove Players can move their troops be- tween their territories.
The game begins with a drafting phase. During this phase, the agent decides where to place their initial 14 troops amongst the available territories. The two opposing players draft their troops before the agent is allowed to draft any troops. The oppos- ing players drafts are either hard-coded to match one of the maps utilized in our study, or they are drafted based on a drafting heuristic. The drafting phase occurs only once in the game. Following drafting, the agent executes the next three phases in sequence. First, in the "Reinforce" phase, the agent receives specific number of reinforcements based on the number of territories and continents they control. The agent needs assign the given reinforcements to the territories they control. Each country reinforced an individual action. Next, the agent moves on to the "Attack" phase. In this phase, the agent can attack adjacent territories with their troops. Within each attack action, the agent specifies which opposing territory they would like to attack, along with the territory they would like to attack from. The agent must also specify the number of troops they would like to move into the opposing territory should the win the conflict, Each combat sequence between two territories is executed in a similar manner to the physical board game,
An action is specified by four-item tuple, i.e. p, 8, tr >. The first item, p, specifies which type of action is being conducted, among the four possible phases in the game. Item two, s, denotes the source country for the action. For reinforce and drafting actions this is the country that the agent wants to add troops to, whereas for the attack and freemove actions, s denotes the country you will be attacking or moving from. The, final two items, t and (r, are specifically for attack and move actions. specifies the country that you would like to attack move to. For the attack action, fr specifies the number of troops you would like move from the attacking country if you win the combat. When the agent specifies a move action, tr denotes the number of troops to be moved from to t.
1.1 State Space
The state of the game stored dictionary. The state dictionary records information such as country ownership. number of troops on each country, con- tinent ownership, etc. We also record information

---

about players such as number of reinforcements available to a player, number of players alive, cur- rent turn number, etc. We have provided six func- tions to encode the state space which can passed as an input to Reinforcement Learning model.
others which weighted by factor of 1. The most simple reward function available is a sparse reward function which provides negative re- wards for losing the game and positive rewards for winning the game. In order to increase the number of rewards given throughout the game, we created the turn count reward function which rewards the agent for every turn it plays. Survival reward func- tion was built top of this provide additional negative reward for losing apart from the reward for surviving.
The first function encodes the state using 54 fea- tures. The initial 42 features contain country re- lated information for each opponent (21 features each) and the next 5 features contain continent own- ership data. The remaining features are used for other information related to the game like number of areas controlled by the player, troops left to be drafted by the player, troops left for reinforcement, number of players alive, current turn number and if the current turn belongs the player.
1.3 Human Drafting
Finally, we have also setup a functionality in our simulator that allows player or the opponents to skip the drafting phase and follow fixed draft based on a predefined map. In such cases, we have predefined fifteen types of map initialisation containing troops for both opponents, which corre- spond the exact maps utilized in our data collec- tion procedure. Our setup chooses one of the map initializations and corresponding selections made by a participant in the user study to simulate the game.
The second function encodes the information in the form of one hots. It has total of 132 fea- tures, the first 84 features contain information garding country ownership
hots, 21 each for the player, opponents and countries with no owner. The next 21 features denote the number troops on each country. The next 20 features contain information regarding continent ownership, 5 each for the player, opponents and no owner. The remaining features contain other relevant informa- tion described for the first function. For both of the first functions described, we also provide normalized versions of these functions where all the real valued spaces are divided by normalising constant.
The fifth encoding function contains all the 132 features of the third function and additional infor- mation for the current phase. contains 134 fea- tures in total. This function returns normalised values. The last encoding function contains 298 features. The initial features are similar to the ones present in the third encoding function. Apart of that it explicitly contains information about where an agent player can attack and execute freemove. This information help the reinforcement learn- ing model more casily. This function also returns normalised values.
1.2
Reward Functions
We have setup four different types of reward func- tions ranging from sparse to dense. The recom- mended reward function the rules-based reward which provides rewards for successful actions, fin- ishing a phase, successful action a phase and winning the game. The rewards for winning the game are weighted by factor of 10 compared to


Reviews:
[{"review_id": "ueEdx70Pxx", "reviewer": "Reviewer_ASwe", "summary": "", "questions": "Could the proposed model achieve multi-round inference? In practice, many of the strategic games are sequential decisions. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their measured review and for raising areas wherein we may clarify our contribution. Through the following response, we hope to clarify each question and critique raised in the review. \n - **“It will be beneficial if the model could handle a mixture of different modalities”** - We agree that multimodality will be beneficial. In this paper, we develop a solution to a novel task: our goal was to first establish a language approach that can help end-users interface with planning tools. Furthermore, our approach included a second modality: the user’s suggested troop selections. While multimodality is a highly valuable contribution towards our “Automated Strategy Inference task, a multimodal strategy translation module would constitute a novel contribution in its own paper. In our camera-ready version, we will add a discussion of the potential to incorporate additional modalities and the need for modalities such as symbolic information and mathematical specifications. \n- **“Could the proposed model achieve multi-round inference?”** - We thank the reviewer for this question. In our paper, we developed an approach to distill assignments for goals and constraints from users’ unstructured language descriptions. We specifically chose the draft phase as the draft phase is the primary phase which requires the users to develop and deploy their strategy and was thus the most suitable phase for our intervention. It is true that as the game proceeds, the user may want to provide modifications to their strategy. Instead of allowing the user to specify a new strategy to account for these modifications, a more appropriate approach could be to employ complementary methodologies similar to prior work wherein, language inputs/commands are utilized to update or change the sub-goals the agent is considering [Fu et al. 2019,  Goyal et al. 2019]. Such approaches could be utilized to individually update specific goals or constraints over the course of multiple rounds of the game to adapt an initially specified strategy, for example, “I am no longer interested in keeping my troops close together” would reduce the value for the specified goal. Recent work has also shown promising results with regards to ChatGPT’s zero-shot performance for dialog state tracking (Heck et al 2023). Therefore, we hope to experiment with deploying ChatGPT, or other SOTA LLMs, in a task-oriented dialog setup, to converse with a user to update the initially specified strategy after multiple rounds of the game, however, this is currently outside the scope of this work and remains a contribution for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well presented.\n2. The authors collected 1053 samples for RISK game on Amazon Mechanical Turk, which is extensive. \n3. The research showcases empirical results, proving the efficacy of the proposed approach against human interpreters and ChatGPT, providing a clearer understanding of the model's capabilities.", "reasons_to_reject": "1. The participants are asked to provide natural language descriptions. In practice, people may want to use language together with some symbolic information and mathematical specifications. It will be beneficial if he model could handle a mixture of different modalities.\n2. The collected data are only one step with randomized configurations. RISK is a multi-round game. The player may adapt their strategies and inference methods according to other players. Making the original multi-round game as a single step one simplifies it too much. This is my main concern.", "questions_to_authors": "Could the proposed model achieve multi-round inference? In practice, many of the strategic games are sequential decisions. ", "paper_topic_and_main_contributions": "The paper presents a computational interface that automatically infers strategic intent from unstructured language. In many real-world scenarios, humans and AI collaborate, requiring the AI to understand and act upon the strategic intent specified by humans. The paper introduces a model that translates unstructured language into machine-readable goals and constraints. This approach is demonstrated in a game environment, where the model learns to derive intent from language-based game strategies. The results show that the proposed model outperforms human interpreters and ChatGPT in interpreting strategic intent from language, especially in low-data scenarios.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Zz119JrvN", "reviewer": "Reviewer_XhaB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in reading and reviewing our paper. We hope to individually address each point through the following:\n- **“A new dataset would, clearly, need to be collected for any such effort”** - Yes, as stated by this reviewer, any new domain would require a new dataset. However, our task of automated strategy inference is inherently domain specific, in that there are many implicit details of the domain that need to be understood by the model in order to adequately perform the task. In many other domain specific tasks, the ML-practitioner often requires a small dataset to personalize or adapt to a new domain or user (for example, Li et al 2022, Morishita et al 2022, from the EMNLP 2022 proceedings). By providing an end-to-end pipeline for automated strategy specification, we have provided all the tools necessary (i.e, data collection/augmentation methodology, ML architecture, evaluation)  for future work to build on this work in other domains in a low-data setting (1000 data points).\n- **“it may not be easy to find games, or other applications, where the proposed approach would work seamlessly”** - We thank this reviewer for highlighting the importance of domain transfer. Firstly, there are many games/domains which require similar goals and constraints to our chosen domain, Risk. These games include Starcraft, Age-of-Empires, Dota 2, etc, which all have similar gameplay properties relating to troop allocation and resource scheduling. Furthermore, there are many applications in the domains of robotics and embodied agents (i.e. domains relating to navigation and teaming) which utilize formal planning specifications, such as goals and constraints. Some examples include search and rescue drones, which need to deploy policies specific to any given situation, or robot scheduling, which require humans to deploy robots with a plan to schedule activities on a factory floor. The scaffold of goals and constraints would be highly suitable in such instances.\n- **“The ChatGPT comparison may be unfair”** - Firstly, as stated in lines 512 - 515, our ChatGPT prompt included one annotated example as we wanted to provide the same information to ChatGPT as we provided to humans during the human-evaluation. Therefore, it enabled us to compute a fair comparison between humans and ChatGPT. Furthermore, our prompt was designed to follow the widely used chain-of-thought prompting approach [Wei et al. 2023], wherein, in our prompt we include the chain-of-thought reasoning for the annotation of each goal and constraint within the example (see Appendix G). Unlike our model, ChatGPT also had an inherent understanding of Risk gameplay and strategy games, whereas our Roberta model needed to learn to infer this information from scratch. For these reasons, we believe that we have provided ChatGPT with enough information to perform the task and facilitate a fair comparison with our other baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed task is interesting, and the results are fairly insightful in showing that models can do better than humans on this reasonably complex task. ", "reasons_to_reject": "It is hard to gauge the applicability of the proposed framework to tasks not following the exact paradigm that Risk falls under. In other words, it may not be easy to find games, or other applications, where the proposed approach would work seamlessly. A new dataset would, clearly, need to be collected for any such effort.\n\nThe ChatGPT comparison may be unfair, since (to the best of my understanding) only one input example is provided to ChatGPT, which differs significantly from the training data the RoBERTa model gets.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new task/paradigm - that of identifying strategic intent from highly unstructured language - in the setting of a war-themed board game. For the game in question (Risk), a number of goals and constraints are defined, and a dataset of unstructured plans mapping to goals and constraints is collected using crowdworkers. Then, a RoBERTa model is finetuned to identify these goals and constraints. Results show improvement over prompting ChatGPT.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vN1wAruo3x", "reviewer": "Reviewer_HrLg", "summary": "", "questions": "* How familiar are the annotators with RISK (S 3.3)?\n* How familiar are the human participants with RISK (S 5.1)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their time and thorough review of our submission. We hope to address each individual point from the reviewer through the following:\n- **“The quality of the annotation is somewhat questionable”** - Yes, as per any crowdsourced dataset, our annotations will be noisy. However, we have employed various measures to ensure data integrity. Firstly, we utilized the master's qualification filter on Mechanical Turk. Secondly, each datapoint was individually reviewed and filtered by two researchers to ensure that the goals and constraints were representative of the language descriptions. Third, we bucketted goal values into 5 uniform buckets (from highly irrelevant to highly relevant) instead of using the original, raw values in [-100,100]  (lines 356 - 360), which we found important to address user-specific variation in ratings. \n- **“the annotators seem to be hired randomly”** - Yes, annotators were hired randomly as is standard practice in online human-subjects research. We did not want to restrict our data to any specific population of users in order to bias our data. However, while the users were not guaranteed to be Risk experts, we provided an extensive tutorial which explained the rules of the game. Crucially, participants also needed to pass a 5-question quiz in order to proceed to the study (lines 285 - 290), thereby confirming that they have gained the necessary expertise in order to complete the task. We will include the full quiz in the appendix in the camera-ready version.\n - **“The goals and constraints ultimately serve as the intermediate stages of the game playing”** - Goals and constraints serve as a strategy to guide the policy of an agent in order to adhere to the specifications of a human. We are not specifically interested in getting players to win the game of Risk or understanding how much more likely you are to win the game of risk through the use of strategy specifications. Our goal is to facilitate strategy inference with a view towards real-world human-AI teaming applications wherein the human requires their AI teammate to perform the task in a specific way which conforms with the goals of the team/commander. Risk was an adequate proof-of-concept domain as it shares many similar properties (resource scheduling, troop allocation, multi-step planning, etc.) to real-world human-AI teaming problems such as search and rescue or drone scheduling. In future work, we hope to expand to these domains and also expand our approach by developing a Seldonian optimization method to allow an RL-agent to adapt its learned policy by utilizing the goals and constraints inferred through a user's language strategy.\n\nHere are our responses to the specific questions posed by the reviewer: \n1. **“How familiar are the annotators with RISK (S 3.3)?”** - We did not specifically recruit risk experts. We taught users the rules of the game through a tutorial and assessed minimum performance via a quiz. More details have been provided in our earlier answer under the “annotators seemed to be hired randomly” section.\n2. **“How familiar are the human participants with RISK (S 5.1)?”** - We did not specifically recruit participants with experience in playing Risk. We did however provide them with a tutorial as well as visually annotated examples to familiarize them with the domain and task. Our human-evaluation task also included attention checks in each annotation to ensure that participants were providing authentic annotations.\n\nFinally, here are some addition clarifications to the minor comments: \n- **“The color text in the upper-left box of Figure 1 is confusing”** - We regret the confusing color scheme here. The green text in the language description corresponds to the text relating to the constraints, and the orange text corresponds to text related to goals. The blue text is generic text in the description that does not directly relate to any goal or constraint. We will change that to black to avoid any confusion as well as add a description of the color scheme to the caption.\n- **“The introduction of \"Commander's intent\" comes too suddenly”** - Commander's Intent has been a widely used formulation for engendering an understanding of a mission or a plan for a team. The point of specifying Commander's Intent is for each team member to know what individual actions should be taken to accomplish a mission while still adhering to the goals and constraints of the commander. Commander’s Intent allows a commander to state what they value and design a reward structure around those values. Commander's Intent could similarly function as a helpful scaffold for a human commander or teammate to provide their intent to an autonomous agent. In instances where an agent needs to execute a plan or a mission independent of the human user, the human needs a mechanism for providing their intent such that the autonomous agent can identify the specific set of low-level actions that adhere to the goals and constraints specified. In the camera-ready version of this paper, we will expand on this description of Commander's Intent and why it is relevant", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work is comprehensive, it covers the full pipeline from data collection to the evaluation of different approaches \n* The experiment design is solid. It compares the proposed method with both human participants and strong LLMs such as ChatGPT\n* Converting the language descriptions to a set of goals and constraints is well-grounded in the literature, and the curated dataset in this work covers many aspects which are missing in the previous works.", "reasons_to_reject": "* The quality of the annotation is somewhat questionable. It is unclear what level of expertise the annotators have on RISK. According to Section 3.3, the annotators seem to be hired randomly. There is only one annotation per example. By looking at the annotated example shown in Figure 1, I also found the example is somewhat **subjective** (This is perhaps due to I don't have access to the full annotation instruction.) However, the above-mentioned issues worry me about the annotation quality -- can we really use the annotation as the ground truth for experiments? The proposed model trained with the augmented data seems to perform the best, however, if the annotation quality is questionable, the better performance can be due to overfitting to annotation artifacts (e.g., one annotator's personal preference). The lower human performance in S5.1 somewhat reflects this concern. It will be helpful to discuss the annotation quality. \n* How useful translating language description to goals and constraints is unclear. The goals and constraints ultimately serve as the **intermediate** stages of the game playing. It is unclear how much improvement a system's winning rate will have with `X%` gain from better goal and constraint extraction. Such experiments often require an end-to-end system and I understand it may go beyond the scope of this work. It will be helpful to have a quantitative study. For instance, the work may design experiments with human players by asking about their preferences on different models' output during *real* games.", "questions_to_authors": "* How familiar are the annotators with RISK (S 3.3)?\n* How familiar are the human participants with RISK (S 5.1)?", "paper_topic_and_main_contributions": "This works study translating natural language descriptions to goals and constraints in a strategic game called RISK. This work starts from data curation to form the first benchmark on mapping language description + state (troop selection) to a set of constraints and goals. Then, they propose augmenting data through paraphrasing. This work also designs a goal extraction and constraint extraction model with Roberta. The experiment demonstrates that the small model proposed in this work achieves more accurate goal and constraint prediction compared to humans and ChatGPT.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The color text in the upper-left box of Figure 1 is confusing. It took me a while to understand it aligns with contents in constraints and goals boxes other than troops. It may be helpful to have an explanation\n* The introduction of \"Commander's intent\" comes too suddenly. Maybe also explain why studying \"Commander's intent\" is interesting/beneficial"}}, {"review_id": "mthqkbm2Vx", "reviewer": "Reviewer_Y69H", "summary": "", "questions": "- Table 5 lists 9 constraint types, but main paper says 8?\n\n- What's the ChatGPT version (gpt-3.5-turbo?), and G.1 seems to use a very long zero-shot prompt, have you considered a shorter few-shot prompt that might work better?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and for raising areas where we may clarify our contributions. \n- **Theory of Mind discussion** - We thank the reviewer for pointing out this concern regarding our usage of \"theory of mind.\" The definition of theory of mind that we had in mind is the ability of the AI-agent to understand the mental state or desired behavior of a human specifier through commander's intent [Schaafsma et al 2015]. However, we do acknowledge that the broader concept of \"theory-of-mind\" is often bidirectional with regards to imparting mental states and will therefore remove it from the introduction. Our goal was to facilitate shared cognition between a human-AI team, specifically through aligning the values of the agent with the human’s needs. We will update the first paragraph of the introduction to more directly state this. We would also be happy to change the title of our task to “Automated Strategy Translation or Automated Strategy Parsing” rather than “Automated Strategy Inference”, if that is deemed by the reviewer to be a better fit. \n- **“my suspicion is just that Roberta model is specifically designed for the output format”** - Yes, one of the inherent advantages of the custom Roberta model is that it is designed for the specific output format, which is inherent in all custom models build to perform a task. ChatGPT also has inherent advantages which our Roberta model did not. For example, unlike our model, ChatGPT already had an abundance of knowledge about Risk Risk. We prompted ChatGPT to explain the rules of Risk and it provided us a very detailed response about the rules of risk, i.e. \n  - \"Risk is a classic strategy board game that involves players trying to conquer territories and continents in order to achieve global domination. Here's how you play the game:\nObjective: The ultimate goal of Risk is to eliminate your opponents and achieve global domination by controlling all the territories on the game board. \nSetup:\nGame Board: Place the game board on a flat surface. The game board represents a world map divided into territories and continents.\nArmies: Each player selects a color and takes their corresponding army pieces (infantry, cavalry, and artillery) and a set of territory cards. The number of armies each player starts with depends on the number of players:\n2 Players: 40 armies each\n3 Players: 35 armies each\n4 Players: 30 armies each\n5 Players: 25 armies each\n6 Players: 20 armies each\nTerritories: Players place one army on each territory they control. Players take turns placing one army at a time until all territories are occupied........... (continued)\" \n\n  Our model did not have this advantage and needed to learn about Risk from scratch. Each model comes with its inherent advantages and disadvantages which make it more or less suitable for a given task. \n- **“why Table 1 chooses a different metric than other tables”** - Table 1 depicts the raw scores, in terms of the average number of goals and constraints predicted correctly per example. Tables 2 and 3 report 10-fold cross-validation accuracies. Table 1 is depicted as such to more easily quantify the significant differences as per our statistical tests.\n\nAnswering other specific questions posed by the reviewer: \n1. **“Table 5 lists 9 constraint types, but main paper says 8?”** - We thank the reviewer for highlighting this typo. Table 5 is correct, there are 9 constraint classes. We will update this in the main paper.\n2. **“What's the ChatGPT version”** - We used the default 3.5 version of ChatGPT. We will add this information to Section 5.2. \n3. **“G.1 seems to use a very long zero-shot prompt, have you considered a shorter few-shot prompt that might work better”** - We used a one-shot chain-of-thought prompt which included an example with reasonings for each annotated goal and constraint. For this work, we wanted to provide ChatGPT the same information that we provided humans to ensure a fair comparison between both modalities. In future work, we plan to explore further prompt engineering approaches as well as the vertical of how many prompts are required for ChatGPT/other comparable LLMs to beat the performance of our custom Roberta model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem of inferring (latent) human intentions is very important and interesting. The paper provides a good instantiation in a grounded game setup, and contributes a dataset that could be of interest to the NLP community.\n\n- The design of multiple classification heads and the use of CE, MSE, OaXE losses are reasonable, \n\n- The performance of the proposed model looks good, outperforming a human baseline and a ChatGPT prompting baseline.", "reasons_to_reject": "- I'm not sure if the task is more like inferring or translation/semantic parsing, as the human input explicitly talks about these constraints and goals, and the main challenge is just to turn them into the structured format defined by the dataset, instead of any theory-of-mind or rational-speech-act kind of inferring. Also, in the intro, I'm not sure if \"theory of mind\" is just \"shared understanding of each. team member of their role\".\n\n- Given the task seems easy, I don't get why Roberta model can beat human and ChatGPT. Some analysis or error breakdown will be very helpful, and my suspicion is just that Roberta model is specifically designed for the output format, which is not surprising when it outperforms other models. Also, why Table 1 chooses a different metric than other tables? ", "questions_to_authors": "- Table 5 lists 9 constraint types, but main paper says 8?\n\n- What's the ChatGPT version (gpt-3.5-turbo?), and G.1 seems to use a very long zero-shot prompt, have you considered a shorter few-shot prompt that might work better?", "paper_topic_and_main_contributions": "The paper proposes a dataset of inferring constraints and goals from human descriptions, grounded in a broad game of Risk. The dataset has 1053 examples, and covers 6 predefined goals (each maps to a [-100, 100] human preference value) and 8 (or 9?) constraint types (\"I must xxx (number) xxx\").\n\nAuthors define a Roberta-based model with multiple classification heads, and fine it outperform a human baseline and a ChatGPT baseline. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a model that translates unstructured natural language into structured goals/constraints allowing the “intent” or meaning of natural language to be converted into logical representations tied to the underlying world state. Their pipeline shows how such data can be collected and then augmented allowing better training of a model on such data. They demonstrate improved performance of this explicit goal/constraint decoder over a baseline that predicts outputs sequentially with in-context learning as well as a human baseline and show improved performance over both. Most reviewers agree that this is an interesting paper and the dataset/pipeline creation and improved performance of the model are good contributions to the language/RL community. I urge the authors to consider the suggestions of better visualisation/examples in the paper and quality checks of the datasets, but even it's current state the methodology presented here is a valuable contribution to future work in language/RL/game environments.
Decision: Accept-Findings