A Textual Dataset for Situated Proactive Response Selection
Year: 2023

Abstract:
Recent data-driven conversational models are able to return fluent, consistent, and informative responses to many kinds of requests and utterances in task-oriented scenarios. However, these responses are typically limited to just the immediate local topic instead of being widerranging and proactively taking the conversation further, for example making suggestions to help customers achieve their goals. This inadequacy reflects a lack of understanding of the interlocutor’s situation and implicit goal. To address the problem, we introduce a task of proactive response selection based on situational information. We present a manuallycurated dataset of 1.7k English conversation examples that include situational background information plus for each conversation a set of responses, only some of which are acceptable in the situation. A responsive and informed conversation system should select the appropriate responses and avoid inappropriate ones; doing so demonstrates the ability to adequately understand the initiating request and situation. Our benchmark experiments show that this is not an easy task even for strong neural models, offering opportunities for future research.

heading: 1 Introduction
text: Conversational assistant systems have recently shown significant improvements for understanding users’ inquiries along with background knowledge, conducting requested operations, and returning natural language responses. Yet, typical systems are likely to be passive and only process user-initiated requests or merely ask values for domain-specific slots (Williams et al., 2013; Ammari et al., 2019). In contrast, human assistants like hotel concierges are more proactive, acting to address unmentioned needs and expected future events (Cho et al., 1996; Bellini and Convert, 2016). They do not only make
∗This work was done while the first author was at Carnegie Mellon University.
a direct response or a clarification question to their interlocutors but also provide personalized information/assistance based on context and knowledge.
To push the frontier of task-oriented conversation technologies, we propose a task of proactive response selection for single-turn help-seeking conversations in English. We mean by proactive that a system engages in an interaction in a cooperative manner (Grice, 1975) and suggests something helpful to a user. The proposed task touches upon two crucial aspects of help-seeking conversations: situation-awareness and goal-awareness.
Situation: Situational information plays an important role in conversations as we illustrate in Figure 1. The example shows a user utterance “Can you open the window for me?” (top) and two response candidates (bottom), “Sure. Shall I bring you cold water, too?” (left) and “Sure. Shall I run the air purifier, too?” (right). Although both candidates here sound helpful, their appropriateness varies depending on context: When the room is hot, suggesting a cold drink is appropriate assistance (left), but on the other hand, if the room is smoky, then running an air purifier is more helpful (right). Likewise, different situations make different responses more appropriate. A fair amount of situational information can be perceived as visual image, sound, and other kinds of sensory signals, and some of those are effectively incorporated into
3856
multi-modal conversational systems (Crook et al., 2019; Kottur et al., 2019). Yet, there are many other types of information that modern conversation assistance systems have access to, for example, via external APIs such as calendars and maps. In this study, we represent situational statements of six semantic categories (location, possession, etc.) in free English texts, which are more explicit as a semantic representation than just maintaining conversation histories (Lowe et al., 2015; Li et al., 2017; Henderson et al., 2019) and more flexible than structured representations of limited vocabulary (Williams et al., 2013; Budzianowski et al., 2018).
Goal: In the aforementioned example, the two actions address two different goals associated with opening a window, namely, to cool off and to air the room. While often being unspoken, underlying goals provide important semantic connections among context and utterances on many occasions (Allen and Perrault, 1980) particularly when language is indirect (Perrault, 1980; Walker et al., 2011; Stevens et al., 2015). We use goal information as a stimulus for soliciting naturalistic and proactive responses from human annotators in data collection.
We introduce a new dataset of SitUatated, GoalAware, and proactive Responses (SUGAR; §3), which contains 1,760 examples of single-turn English conversations.1 Each conversation includes a user request anchored by an implicit goal, a reference response, and 12 sentences of situational information. As a proof of concept, we perform the task of situated response selection on SUGAR by adding two extra response candidates to each example. All responses are annotated with three-point appropriateness ratings.
To create SUGAR, we extracted user utterances and goals from common-sense knowledge bases, ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), and collected proactive responses with supporting situational information by crowdsourcing. We then used a language generation model, COMET (Bosselut et al., 2019; Hwang et al., 2021), to generate additional situational statements. Finally, we selected two more response options for each reference response using an adversarial method to form examples of three-choice response selection. To ensure data
1https://github.com/notani/ sugar-conversational-dataset
quality, we performed multiple manual validation steps during data collection. In our experiments on SUGAR (§4), Transformer-based rankers achieved over 80% precision@1 when when only the relevant situational statements were presented. However, precision decreased when distractors were included in the input, and this trend further continued as more distractors were added in our controlled experiments. These results suggest potential opportunities for future research.

heading: 2 Related Work

heading: 2.1 Conversational Dataset
text: Acquisition of real or realistic conversational data has been an essential step for developing conversation engines that imitate human communication (Serban et al., 2018). Various datasets have been constructed with a focus on different aspects of communication.
With regard to target communicative aspects, the most relevant to our work is SIMMC (Moon et al., 2020). SIMMC encompasses surrounding situational information that gives a basis for verbal interactions in task-oriented scenarios in the shopping domain. Moon et al. collected visually-grounded conversation examples from pairs of human annotators interacting with each other in a virtual environment (Crook et al., 2019), where one annotator seeks help for shopping, and the other provides assistance. SUGAR is also concerned with how human interlocutors perform situated conversations in a help-seeking setting. Our work extends this direction to scenarios other than shopping and includes more diverse types of information that modern conversational assistants could access via sensors or external APIs (e.g., temperature and schedule) by representing situational information in a textual form as opposed to visual images.
The choice of modality is motivated by existing conversational datasets that express various kinds of background information in plain text: the persona of an interlocutor (Zhang et al., 2018; Dinan et al., 2020), emotional states (Rashkin et al., 2019; Ghosal et al., 2022), and related documents (Zhou et al., 2018; Dinan et al., 2019). These examples demonstrate the utility of textual forms for representing both explicit and implicit information of various kinds.
Some existing datasets are concerned with information-seeking conversations like restaurant recommendation where suggestions by assistants
naturally occur (e.g., “If you like French cuisine, how about RestaurantX?”, “I can find transportation for you.”). However, it is not trivial to solicit such naturalistic proactive utterances in more diverse help-seeking scenarios. In many cases, the minimum objective of a conversation can be achieved by responding to user-initiated inquiries, and such kinds of responses are relatively easy to collect from non-expert annotators (Budzianowski et al., 2018; Byrne et al., 2019; Eric et al., 2020). We address this problem by leveraging implicit goals behind user requests. The comprehension of goals in conversations has been recognized to be important not only in task-oriented dialog research but also in a broad range of research areas such as linguistics, psychology, and artificial intelligence. (Schank and Abelson, 1977; Clark and Schaefer, 1989; Gordon and Hobbs, 2004; Rahimtoroghi et al., 2017). Human interactions often involve indirect speech acts (Perrault, 1980; Gibbs and Bryant, 2008) and indirect responses like nonyes/no answers to polar questions (Hockey et al., 1997; de Marneffe et al., 2009; Stevens et al., 2015; Louis et al., 2020). These studies motivate our strategy for soliciting natural-sounding proactive responses from crowd workers.
In contrast to most datasets we introduced here, SUGAR only contains single-turn conversation examples due to the ease of data collection and quality control. Our primary focus is on conversational assistance, where short-turn conversations are common (Völkel et al., 2021). Thus, we believe that single-turn examples are still useful for system development. It is possible to extend our problem setting and data collection approach to a multi-tern setting, which we leave as future work.

heading: 2.2 Response Selection
text: Automatic response models can be divided into two approaches: response generation and response
selection. Response generation directly generates natural language response text from scratch, and response selection selects a response from a candidate pool built by humans, templates, or language generation systems. The latter approach is widely used in many real-world applications cases because of the controllability of responses and the easiness of evaluation (Deriu et al., 2020). In this study, we focus on the task of response selection as a proof of concept. We assume that an external response generation system generates candidates based on the system’s functionality and focus on picking the appropriate ones. SUGAR can also serve as a valuable resource for the development and evaluation of response generation systems, which is an interesting avenue for future research.
To train and evaluate a response selection system, each example must have distractors (negative responses), but typically, conversational datasets only contain ground truth responses. Thus, it has been commonly practiced to pick negative responses by random sampling (Lowe et al., 2015; Henderson et al., 2019). This approach comes in handy but may introduce negative responses that are clearly off-topic or false negatives (Akama et al., 2020; Hedayatnia et al., 2022). To alleviate this problem, we use an adversarial filtering algorithm (Zellers et al., 2018; Sakaguchi et al., 2019; Bhagavatula et al., 2020) to select competitive distractors and recruit crowd workers to rate candidates, allowing each example to have multiple acceptable responses.

heading: 3 Task and Data
text: The goal of this study is to provide a resource for developing a system that can observe situational information and return a proactive response to a user. We consider six categories of observable situational statements (Table 1): location (where the user is), possession (what the user has), time, date, behavior (what the user is/was doing), and envi-
ronment (temperature, etc.) We define a proactive response to be a response that provides suggestions to help users achieve their goals.

heading: 3.1 Problem Formulation
text: Our task has five components: (1) a user utterance u, (2) situational statements S = {si}i=1,··· ,l, where l is the number of statements, (3) responses R = {ri}i=1,··· ,m, where m is the number of response candidates2, (4) their appropriateness ratings Y = {yi}i=1,··· ,m, where yi is a three-point Likert scale, and (5) an implicit goal g. S can include distractors that are not directly relevant to the conversation. u, S, and R are given as input, and the task is to re-rank R. Response selection systems are trained and evaluated by Y . In this study, we set l = 12 and m = 3.

heading: 3.2 Data
text: SUGAR contains 1,760 high-quality examples, each of which has three response candidates and 12 sentences of situational information (situational statements). Table 2 shows the dataset statistics.
2We pick m − 1 responses automatically such that they are less appropriate than the reference response in a given context (See Step 7). Nevertheless, there usually exist one or more acceptable responses to a given user utterance. We thus annotate all acceptable responses manually (Step 8).
We constructed the dataset with the eight steps shown in Figure 2. We describe them below.3
(1) Seed Utterance & Goal Selection: We harvested action and goal events from two commonsense knowledge bases, ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), where knowledge is represented as nodes representing events or concepts and edges connecting them with semantic relations. The collected action-goal node pairs served as the seed utterance-goal for soliciting responses and situational statements in the following data collection steps. First, we extracted nodes consisting of verb phrases (VPs) that appear at least five times within English request phrases (e.g., Please VP, Could you VP?, etc.) in the OpenSubtitles corpus (Henderson et al., 2019). These request expressions were also used as the surface form of u. Two of the authors then selected 563 events that can be achieved within a reasonable time span, can be assisted by someone else, and can be triggered by a goal. We retrieved their implicit goals g by goal-related edges in ATOMIC and ConceptNet. Specifically, we used xNeed in the reverse direction and xIntent in ATOMIC and HasPrerequisite in the reverse direction and MotivatedByGoal in ConceptNet. Finally, two of the authors evaluated the node pairs and picked 501 (u, g) pairs for which we can naturally say “I do u to achieve g.” (e.g., open a window to cool off.) We also merged synonymous expressions (e.g., go to a market and go to a supermarket) into a single entry and corrected grammatical errors and unnatural phrases.
3See also Appendix A for technical details.
(2) Situation Collection I: We collected situational statements in two phases to simplify annotation work. The first phase focuses on u and g, and the second phase considers r in addition to u and g. In this step, we presented a pair of u and g texts to crowd workers and instructed them to specify situational information that is required to guess the goal based on the utterance. For example, an implicit goal “to cool off” can be naturally inferred by situations like “The user is home. The room temperature is hot.” We asked workers to write observable facts in the six semantic categories (Table 1). For example, “The room temperature is hot.” is valid, but “The user feels hot.” is invalid as assistance systems cannot observe the user’s feeling. We recruited one worker for each (u, g) pair and paid $0.12 per HIT4 (one (u, g) pair/HIT).
(3) Response Collection: In parallel to Step (2), we recruited two crowd workers for each (u, g) pair to collect responses. The workers created at least two responses: one of the responses accepts and the other rejects the request. We asked the workers to write a proactive response, a response providing suggestions for goal fulfilment.5 To solicit responses closely connected to implicit goals rather than to domain knowledge, we instructed the workers to avoid posing a clarification question like “Sure, I’ll turn on the air conditioner for you. Would you like it on a high or low setting? (= clarification)” The workers were presented one u-g pair in each HIT and were paid $0.30/HIT.
(4) Response Validation: We present the utterances, goals, and collected responses to crowd workers and evaluated the helpfulness of the response. A response is considered to be valid if it satisfies the following criteria: (1) the response suggests or requests something new, and (2) the suggestion or request is helpful for achieving the goal. Each response was evaluated by three workers. We then picked the responses that were approved by two or three workers. We call a verified response a reference response r1 hereafter. Each HIT contains up to seven responses, and one of them is a dummy question for evaluating crowd workers. For quality control, we filtered out crowd workers who participated in the task twice or more
4Human Intelligence Task, a unit of task in MTurk. 5For a response that rejects a user’s request, we instructed the workers to provide a reason for rejection (e.g., we cannot brew coffee because we are out of coffee filters) in addition to a suggestion.

heading: 1990 3546 1083 152 1699 2793
text: and did not reach 0.75% accuracy for the dummy questions. The workers were paid $0.18 for this task. Krippendorff’s α was 0.547.
(5) Situation Collection II: We collected situational statements from crowd workers with the following two goals: (1) to collect situational statements that cover the reference response r1 and (2) to verify the situational statements collected in Step (2). We presented (u, g, r1) with the statements obtained in Step (2) and again instructed crowd workers to write observable facts. The results of Step (2) were provided as editable initial values, and we encouraged workers to update the texts when it is necessary. We recruited one crowd worker for each (u, g, r1) with the reward of $0.42/HIT.
(6) Semi-automatic Situation Collection: We found that the collected situational statements were often under- or over-specified. We addressed this by automatic situation generation and manual verification.
The first author examined all the situational statements, discarded/modified inappropriate situations, and categorized them into six categories. We then used the cleaned and labeled texts to fine-tune a neural sequence-to-sequence to generate more situations. Specifically, we fine-tuned BART (Lewis et al., 2020) trained on ATOMIC2020 (Hwang et al., 2021)6 to take a concatenation of u, g, and r1 as input and generate a text for a given situation cat-
6Note that the framework of pre-training Transformer models on common-sense knowledge bases was originally proposed by Bosselut et al. (2019).
egory as illustrated in Figure 3. We performed a beam search of width 3 and took top-3 generation results for each input and relation. Finally, we manually verified the generated situations, resulting in 4,375 unique situations (6.4 ± 1.3 statements per example). We denote the situational statements attached to (u, g, r1) by S1. Table 3 shows the distribution of situation categories in SUGAR. Statements about possession and environment appear most frequently, which is reasonable because such situational information often decides actions that can be carried out (e.g., to drink coffee, coffee must be available). The other categories are less frequent, but 64% of examples have at least one time or date information, and 69% have a statement about behavior.
(7) Distractor Selection: The examples collected in the previous steps only contain reference responses r1 and supporting situational statements S1. We added m − 1 response candidates along with their relevant situational information as distractors so that all examples have m response candidates and l situational statements. We set m = 3 and l = 12. In this section, we describe the highlevel idea of our algorithm. Appendix B presents technical details.
Distractors can be obtained by random sampling as practiced in many studies (Henderson et al., 2019) or by advanced methods such as adversarial filtering (Li et al., 2019; Gupta et al., 2021). However, such approaches may introduce off-topic responses that are easy to rule out and false negatives — acceptable responses treated as negative examples, degrading system performance as well as reliability of evaluation (Akama et al., 2020; Hedayatnia et al., 2022).
To alleviate this problem, we combine lexical matching and adversarial filtering (Zellers et al., 2018; Sakaguchi et al., 2019; Bhagavatula et al., 2020) to construct distractors and validate them manually (see Step 8). We first created an initial dataset by a lightweight method based on sentence embeddings and lexical matching. We then performed J = 3 rounds of adversarial filtering. In each round, we split the dataset into K = 10 folds, and for each split, we trained a binary logistic regression classifier that takes sentence embeddings of u, S1, and a response candidate. We computed sentence embeddings by SentenceTransformers (Reimers and Gurevych, 2019) with MPNet (Song et al., 2020). We used the trained clas-
sifier to identify easy distractors and replace them with more confusing ones with respect to the score function. We sampled two responses for each example. All response candidates in the same example have the same polarity. Finally, we expanded S1, which only contains relevant information to u and r1, to obtain a set of l = 12 situations S such that some of them are related to distractors but do not disqualify r1, and statements do not contradict with each other. We again used sentence embeddings to find topically related situational information and avoid contradiction with keyword-based heuristics.
(8) Validation: There are usually multiple appropriate responses in one conversational context, and therefore, some of the challenging “distractors” picked in the previous step can be acceptable or even more appropriate than the reference r1. To avoid introducing false negatives, we rated all response candidates on a three-point Likert scale (Bad, Acceptable, or Best) by crowdsourcing. We recruited three crowd workers per example with the reward of $0.25/each and asked them to pick an appropriate response candidate (Krippendorff’s α (Krippendorff, 2006) of 0.484). We then aggregated ratings by the statistical model proposed by Zhou et al. (2014) to obtain the final rating Y .7 We discarded one example in this validation step and obtained 1760 examples with all responses rated. Figure 4 shows the annotation result. As we expected, a fair number of examples (56%) have
7In the first run, all candidates were rated as equally good or bad in 18 examples. We updated and re-annotated 17 examples.
more than one Best or Acceptable responses. The first author reviewed 61 examples (3.5%) where r1 was rated as Bad and fixed contradicting situational statements. Examples without Best responses were also reviewed and revised if necessary.

heading: 4 Experiments
text: We evaluate several baseline models on SUGAR to explore two questions concerned with the nature of the proposed task and dataset: (1) Is understanding of situational information required to identify proactive responses in SUGAR? (2) Can standard matching-based systems capture relevant situational information and solve the task?

heading: 4.1 Baselines
text: We evaluate a lexical-matching approach and several Transformer-based response selection systems. A variety of neural networks have been proposed for the task of response selection Tao et al. (2021), but we opted to focus on the direct application of pre-trained Transformers rather than equipping them with extra modules/resources. Pre-trained models have proven effective in conversation tasks with minimal adaptation (Budzianowski and Vulić, 2019) and even achieves the best performance in a response selection task (Han et al., 2021).
TF-IDF ranker: We used a lexical-matching baseline system that ranks response candidates by cosine similarity of TF-IDF vectors of context and a response candidate (Lowe et al., 2015). While this ranker is quite simple, it can outperform or perform on par with more complex supervised models in certain tasks (Thakur et al., 2021). We calculated TF-IDF weights on a training split with scikit-learn library.
Transformer ranker: We fine-tuned and evaluated four variants of Transformer-based rankers:
1. BERT-FP (Han et al., 2021): This model is an uncased BERTbasethat underwent additional training on the Ubuntu Dialogue Corpus (Lowe et al., 2015). The training process includes unsupervised post-training and supervised fine-tuning. As of 2023, this model is one of the leading systems on the Ubuntu dataset.
2. BERT (Devlin et al., 2019): We also tested an uncased BERTbasewithout the additional training of Han et al. to analyze its benefits in our task. In the experiments of Hedayatnia et al.
(2022), the BERT ranker performed similarly to BERT-FP.
3. RoBERTa (Liu et al., 2019): RoBERTa has the same architecture as BERT as a backbone but was trained using improved training configurations, resulting in better performance across multiple tasks and datasets. We used the pre-trained base model (12 layers≈ 125M parameters)
4. DeBERTa (He et al., 2021b,a): DeBERTa is a model that improves upon BERT and RoBERTa by using disentangled attention mechanisms. In our experiments, we used the base DeBERTa v3 model (12 layers ≈ 86M parameters).
Following Han et al., we encoded a concatenation of input tokens, which will be explained in the next section, and a response option using these Transformer encoders. We then roduced a score of the option by a logistic regression classifier that takes the last hidden state of a special token, [CLS], at the first position in the input. Model parameters were optimized using Adam (Kingma and Ba, 2015) to minimize the max-margin loss.

heading: 4.2 Experimental Setup
text: Input format: We concatenated context and a response candidate for the Transformer rankers. To address our questions, we experimented with three variants of context:
1. u: Utterance (u)-only 2. u+ S1: Utterance (u) plus relevant situation
(S1) 3. u+ S: Utterance (u) plus relevant and irrele-
vant situation (S)
Training and Test: We performed five-fold cross-validation (training:validation:test=6:2:2).8 For each round, we trained a Transformer ranker for 10 epochs with a batch size of 32 and evaluated the model by nDCG@3 on the validation split every epoch. We then selected the best checkpoint for evaluation. To stabilize training, we applied weight decay of 0.05, set the maximum gradient norm to 5.0, and used a linear learning rate scheduler with 5% (≈ 20) warm-up steps. We further performed light-weight grid-search for hyperparameter tuning based on an average nDCG@3 score on validation
8We removed examples without Bad response options from the validation and test splits
splits, with learning rate ∈ {5e − 5, 1e − 5}, and margin for the max-margin loss ∈ {1.0, 0.5, 0.1}. One epoch of training took 1-2m on GeForce GTX TITAN X. We report the average Precision@1 and nDCG@3 on the test splits.

heading: 4.3 Results
text: Table 4 shows the average test scores over a fivefold cross-validation. Two general patterns can be observed: (1) the Transformer-based models, except for BERT-FP, outperformed the TF-IDF baseline, and (2) the systems that were provided with the request utterance u and relevant statements S1 outperformed their counterparts with different input settings. In regard to the key questions, the results reveal several interesting findings:
1. Comparison of two input settings u and u+S1 demonstrates that relevant situational information leads to a clear performance boost as expected (e.g., +0.13 in Precision@1 and +0.05 in nDCG@3 with BERT).
2. The performance gain in u + S1 can be attributed to the increased word overlaps between the context and the correct responses, as indicated by the performance of the TF-IDF baseline. However, with the addition of distractors in the u+ S setting, the performance of the TF-IDF baseline dropped substantially (-0.20 in Precision@1 and -0.09 in nDCG@3). This result suggests that our dataset effectively avoids superficial clues, highlighting the importance of a higher-level understanding of situational statements.
3. Interestingly, in the u+ S setting, the performance of Transformer rankers also decreased significantly to the same level as their corresponding systems without situational statements in the input (the u setting).
4. Additional pre-training of BERT-FP was not effective in our task, which is consistent with the observation of Hedayatnia et al. (2022). We speculate that this is due to a domain mismatch of training corpora. BERT-FP is pretrained on technical topics related to Ubuntu, whereas SUGAR concerns a wider range of topics in daily life.
These findings provide valuable insights into our research questions. First, the understanding of relevant situational statements helps systems select proactive responses accurately, indicating that SUGAR is an effective resource for the development and evaluation of situated conversation systems. Secondly, it is challenging for Transformer rankers to identify useful clues from a mixture of relevant and irrelevant situational statements.

heading: 4.4 Robustness to Distractors
text: The results presented in the previous section indicate that Transformer rankers can be misled by irrelevant information. To explore this further, we evaluated these rankers with varying numbers of irrelevant situational statements (distractors).
In this experiment, we controlled the number of distractors by creating instances with 5, 10, and 15 distractors. Situational statements were randomly added as necessary. We trained and tested the same response rankers following the same setup, with the exception that we fixed the learning rate to 5e-5, which generally produced better results than 1e-5 in the main experiments. It is important to note that the first 1-7 distractors were adversarially selected (§3), while the remaining distractors were added at random.
Figure 5 displays the precision@1 and nDCG@3 scores of the response rankers. The performance of TF-IDF indicates that the addition of random distractors slightly increased the word overlap rates between input and distractor responses, but not substentially. However, as hypothesized, all systems demonstrated decreasing scores as more distractors were included. Interestingly, the performance of the advanced models, RoBERTa and DeBERTa, decreased drastically as more distractors were added (0.87 → 0.67 for RoBERTa and 0.90 → 0.61 for
DeBERTa in Precision@1). We speculate that these models are powerful but also susceptible to overfitting spurious patterns between situational statements and response options, resulting in low test scores. In contrast, the BERT-based rankers were more robust to distractors, but their absolute performance remained low (Precision@1 of 0.73 and nDCG@3 of 0.91 for BERT). This finding highlights the need for future work to develop models that are more robust to the inclusion of irrelevant situational context.

heading: 5 Conclusion and Future Work
text: We proposed a task of situated proactive response selection for developing and evaluating conversational assistants that can help users proactively in various help-seeking scenarios. We constructed a dataset of 1.7k examples by crowdsourcing and semi-automatic generation.
There are several interesting directions for future research. First, as shown in our experiments, it is challenging to pick up relevant situational information and use it to reason about user requests and potential assistance. To achieve this, conversational systems will need to be equipped with world knowledge to effectively align situation information with an interaction. One promising approach is knowledge-based response models such as graph neural networks, which recently has shown to be effective in various NLP tasks (Zhang et al., 2020; Zhou et al., 2022; inter alia). Second, although we leveraged implicit goals only for soliciting proactive responses in data collection in this study, understanding of goals should be necessary for building better conversation engines as claimed in early studies (Allen and Perrault, 1980; inter alia). We
believe SUGAR can facilitate future research in this direction.
Limitations
Data size: SUGAR is relatively small compared to recently published datasets. This is due to the complexity of our problem setting and annotation pipeline. We prioritized quality over quantity and performed multiple steps of manual intervention to reduce errors, false negatives, and annotation artifacts. These problems have been reported in various NLP tasks not limited to conversational tasks (Gururangan et al., 2018; Akama et al., 2020; Elazar et al., 2020). Nonetheless, our experiment has shown that pre-trained Transformer models can be trained to outperform a TF-IDF ranker by a clear margin, which is encouraging. In addition, we could automatically induce noisy but large-scale training instances from existing resources, for example, by harvesting event pairs that can be used as u and r from event knowledge bases such as ATOMIC2020and generating situation statements using our generator (§3).
Representation of situation information: In SUGAR, situation information is represented in textual expressions. In real-world applications, such information could be collected via external APIs (e.g., calendar and map) and sensors (e.g., camera) and stored in non-textual forms. Our study is a proof-of-concept that shows the understanding of situational information is very important for response selection. Future research should explore ways to process situation information that is expressed in other forms of data (e.g., structured texts, numbers, images). Even if the value is structured or images, we could transform them into textual forms
as done in data-to-text research (Shen et al., 2020; Miura et al., 2021). Besides, we acknowledge that situational information is often under-specified in SUGAR because some information is considered to be common-sense (e.g., a room has a door) or presupposed (e.g., “Please open the door” presupposes that the door is closed.), and such information was not explicitly stated by human annotators during data collection. Therefore, response selection systems should be equipped with a mechanism to handle implicit knowledge to solve the task.
Ethical Considerations
Undesired bias and abusive content: A multitude of sources have reported that data-driven conversational systems can (re)produce undesired bias or abusive language existing in language resources used for development. To minimize such a risk, we carefully curated conversation examples in SUGAR. Our target task is response selection, where systems only produce language in a pre-compiled response list, and therefore, it is not likely that resulting systems yield harmful content. However, users of SUGAR should be cautious when it is used for developing generation systems in future work.
Human subjects: Crowd workers in Amazon Mechanical Turk (MTurk) participated in our data collection pipeline. Our annotation tasks were reviewed by the institutional review process before being published in MTurk to avoid ethical issues. We did not collect any personally identifiable information of workers other than (anonymized) Turker IDs. Task rewards were decided by several rounds of trials so that workers can receive at least $6.50 hourly.
Use of external data and tools: We used external datasets such as ATOMIC2020and ConceptNet and tools such as spaCy and Transformers library. We have confirmed that the use of these resources for our research does not violate usage restrictions.

heading: Acknowledgments
text: We thank Yonatan Bisk, Benjamin Van Durme, Lori Levin, and the anonymous reviewers for their feedback.

heading: A Manual Annotation
text: We recruited non-expert crowd workers in Amazon Mechanical Turk in annotation steps (2-5). In all steps, crowd workers were required to meet the following qualification requirements: (i) Their number of tasks approved ≥ 5k, (ii) the task approval rate ≥ 99%, (iii) their location is the US, and (iv) they answer an exercise question correctly. Figure 6 shows the annotation interface.
Two of the authors were involved in the annotation steps (1), (4), (5), and (8). They are ESL with a degree in computer science from a school in the US (one holds a master’s degree, and the other holds a Ph.D.). They all have backgrounds in NLP/CL research.

heading: B Distractor Selection
text: This section presents the technical details of the distractor selection method (Step 7). Below, tunable parameters such as thresholds on scores and the number of iterations were empirically selected based on several pilot runs.
B.1 Response Selection Our method selects distractor responses from all the responses in the dataset in two steps: We first create an initial dataset by a light-weight method (Algorithm 1) and then perform adversarial filtering (Algorithm 2).
First Step (Algorithm 1) The objective of the first step is to avoid including false-negative responses (Lines 3-6). We discard
responses that are too similar to r1 in terms of the overlap coefficient of content words (noun, verb, adjective, and adverb).
Overlap(x, y) = |CW(x) ∩ CW(y)|
min (|CW(x)|, |CW(y)|) , (1)
where CW(x) is a set of content words in x. We set the threshold of overlap coefficient to 0.75. We use the same constraint on their goal texts. We also measure their closeness by the cosine similarity of their sentence embeddings (denoted as EmbSim) and discard candidates whose similarity is 0.5 or higher. We then sample m− 1 responses from this filtered response pool one by one (Lines 11-15). To diversify response options, we remove similar responses to the picked one from the pool based on overlap coefficient (Line 16-19).
Second Step (Algorithm 2) We then perform J = 3 rounds of adversarial filtering. Our method is a slightly modified version of the algorithm used by Bhagavatula et al. (2020). In each round, we split the dataset into K = 10 folds (Line 6), and for each split, we train a binary logistic regression classifier that takes sentence embeddings of u, S1, and a response candidate r ∈ R (Line 8). We pre-compute their sentence embeddings with the pre-trained SentenceTransformers (Reimers and Gurevych, 2019) with MPNet (Song et al., 2020). Once the classifier is trained, we score response candidates in each example and identify distractors whose scores are lower than that of the reference response r1 plus a margin γ = 0.05. We replace these easy distractors with more confusing ones (Line 14-16). In this way, we repeatedly update the dataset (Line 17) and output the final result (Line 18).
B.2 Situation Selection Next, we update S1, which only contains relevant information to u and r1, to include l statements in total such that some of them are associated with distractors or not directly related to the conversation. Otherwise, reference responses can be easily identified by superficial clues. Having irrelevant situational statements is also for simulating real use cases, where a conversational system has access to a wide range of sensory information or external APIs, but most of them are unimportant for addressing a user’s request.
It is required that (a) additional situational statements do not disqualify the reference response,
Algorithm 1 Create an initial dataset by light-weight filtering
Input: m, Dataset D = {(u(i), g(i), r(i)1 , S (i) 1 )}i=1,··· ,N , ▷ N := num. of examples in the dataset. Output: D′ = {(u(i), g(i), R(i), S(i)1 )}i=1,··· ,N ▷ R(i) := {r (i) 1 , · · · , r (i) m } ▷ Initial dataset
1: function INITDATASET(m,D) 2: D′ ← ∅ 3: for i : 1..N do 4: P ← {r(j)1 }j=i,··· ,i−1,i+1,··· ,N ▷ All the responses in D but r (i) 1

heading: 5: # (1) Remove too similar responses
text: 6: for j : 1..N do 7: if i=j then 8: continue 9: if Overlap(u(i), u(j)) ≥ 0.75 or Overlap(g(i), g(j)) ≥ 0.75
or EmbSim(u(i)1 , r (j) 1 ) ≥ 0.5) then
10: Remove r(j)1 from P 11: # (2) Pick m− 1 similar responses 12: R(i) ← {r(i)1 } 13: for j : 1..m− 1 do 14: Sample r ∈ P 15: Add r to R(i)
16: # (3) Remove similar responses from the pool 17: for all r′ ∈ P do 18: if Overlap(r, r′) ≥ 0.75 then 19: Remove r′ from P 20: Add (u(i), g(i), R(i), S(i)1 ) to D′ 21: return D′
and (b) they do not contradict others. To this end, we again use sentence embeddings with keywordbased heuristics. We first combine the statements associated with distractor responses and create a pool of candidates. Here, we drop statements that are similar to the response candidates in terms of the overlap coefficient of content words with a threshold of 0.75. We also used manually defined keywords to discard situational statements that tend to contradict others (e.g., the time is midnight, the user is injured, etc.). We then iterate over six categories and pick situational statements from the pool one by one. We score statement s of category c using the function below:
f(s;R,S′) = max r∈R EmbSim(s, r)
− max s′∈S′c
EmbSim(s, s′)
− 1 2 max s′∈S′C\{c} EmbSim(s, s′), (2)
where S′ is the current situational statements, S′c ⊂ S′ represents the statements in S of category c, and C denotes a set of situation categories. We
pick distractor statements until we exhaust all the candidates in the pool or the maximum score does not reach 0. We then draw statements from the entire dataset in the same way until |S| reaches l = 12. For time, date, behavior, and location categories, we pick zero or one statement as those categories are not likely to have more than one value.

heading: C Response Selection Example
text: Table 5 shows a conversation example included in SUGAR.
Algorithm 2 Adversarial filtering (AF) for R
Input: m, Dataset D = {(u(i), g(i), r(i)1 , S (i) 1 )}i=1,··· ,N , ▷ N := number of examples in the dataset. Output: D′ = {(u(i), g(i), R(i), S(i)1 )}i=1,··· ,N ▷ R(i) := {r (i) 1 , · · · , r (i) m }
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly (https://www.grammarly.com/) and ChatGPT (https://chat.openai.com/) for proofreading and improving clarity (the whole paper).
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 3 and 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Limitations

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Limitations

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethical Considerations and Appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 1 and 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Sections 3 and 4, Appendix.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix presents a few screenshots of the annotation interface. We will release more details in our GitHub repository upon internal approval.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Ethical Considerations
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? As our annotation does not collect PI, our annotation study just underwent an internal review process (not IRB).

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix A

heading: Future Work
text: There are several interesting directions for future research. First, as shown in our experiments, it is challenging to pick up relevant situational information and use it to reason about user requests and potential assistance. To achieve this, conversational systems will need to be equipped with world knowledge to effectively align situation information with an interaction. One promising approach is knowledge-based response models such as graph neural networks, which recently has shown to be effective in various NLP tasks (Zhang et al. , 2020; Zhou et al. , 2022; inter alia). Second, although we leveraged implicit goals only for soliciting proactive responses in data collection in this study, understanding of goals should be necessary for building better conversation engines as claimed in early studies (Allen and Perrault, 1980; inter alia). We
believe SUGAR can facilitate future research in this direction. Limitations
Data size: SUGAR is relatively small compared to recently published datasets. This is due to the complexity of our problem setting and annotation pipeline. We prioritized quality over quantity and performed multiple steps of manual intervention to reduce errors, false negatives, and annotation artifacts. These problems have been reported in various NLP tasks not limited to conversational tasks (Gururangan et al. , 2018; Akama et al. , 2020; Elazar et al. , 2020). Nonetheless, our experiment has shown that pre-trained Transformer models can be trained to outperform a TF-IDF ranker by a clear margin, which is encouraging. In addition, we could automatically induce noisy but large-scale training instances from existing resources, for example, by harvesting event pairs that can be used as u and r from event knowledge bases such as ATOMIC2020and generating situation statements using our generator (§3). Representation of situation information: In SUGAR, situation information is represented in textual expressions. In real-world applications, such information could be collected via external APIs (e. g. , calendar and map) and sensors (e. g. , camera) and stored in non-textual forms. Our study is a proof-of-concept that shows the understanding of situational information is very important for response selection. Future research should explore ways to process situation information that is expressed in other forms of data (e. g. , structured texts, numbers, images). Even if the value is structured or images, we could transform them into textual forms
as done in data-to-text research (Shen et al. , 2020; Miura et al. , 2021). Besides, we acknowledge that situational information is often under-specified in SUGAR because some information is considered to be common-sense (e. g. , a room has a door) or presupposed (e. g. , “Please open the door” presupposes that the door is closed. ), and such information was not explicitly stated by human annotators during data collection. Therefore, response selection systems should be equipped with a mechanism to handle implicit knowledge to solve the task. Ethical Considerations
Undesired bias and abusive content: A multitude of sources have reported that data-driven conversational systems can (re)produce undesired bias or abusive language existing in language resources used for development. To minimize such a risk, we carefully curated conversation examples in SUGAR. Our target task is response selection, where systems only produce language in a pre-compiled response list, and therefore, it is not likely that resulting systems yield harmful content. However, users of SUGAR should be cautious when it is used for developing generation systems in future work. Human subjects: Crowd workers in Amazon Mechanical Turk (MTurk) participated in our data collection pipeline. Our annotation tasks were reviewed by the institutional review process before being published in MTurk to avoid ethical issues. We did not collect any personally identifiable information of workers other than (anonymized) Turker IDs. Task rewards were decided by several rounds of trials so that workers can receive at least $6. 50 hourly. Use of external data and tools: We used external datasets such as ATOMIC2020and ConceptNet and tools such as spaCy and Transformers library. We have confirmed that the use of these resources for our research does not violate usage restrictions