End-to-End Task-Oriented Dialogue Systems Based on Schema
Year: 2023

Abstract:
This paper presents a schema-aware end-toend neural network model for handling taskoriented dialogues based on a dynamic set of slots within a schema. Contrary to existing studies that proposed end-to-end approaches for task-oriented dialogue systems by relying on a unified schema across domains, we design our approach to support a domain covering multiple services where diverse schemas are available. To enable better generalizability among services and domains with different schemas, we supply the schema’s context information including slot descriptions and value constraints to the model. The experimental results on a well-known Schema-Guided Dialogue (SGD) dataset demonstrated the performance improvement by the proposed model compared to state-of-the-art baselines in terms of end-to-end modeling, dialogue state tracking task, and generalization on new services and domains using a limited number of dialogues.1

heading: 1 Introduction
text: Task-oriented dialogue (TOD) systems enable a flexible alternative for users to fulfill tasks, such as flight booking, arranging transportation, and checking the weather, through a natural language conversation with a virtual assistant. Due to the potential benefits of such a system for both users and industries, an increasing number of application domains and individual services within the domain are available for the system to support. Typically, each service can be distinguished by its schema which determines different functionalities and criteria for the task. To maintain user engagement and satisfaction, it is important for the system to be adjustable to a variety of schemas to effectively interpret the user’s goal as well as deliver appropriate responses.
To promote the system’s scalability and transfer ability, recent studies proposed approaches that uti-
1The code is available at https://github.com/ aistairc/SchemaTOD.
lize a pre-trained language model (Peng et al., 2021; Yang et al., 2021; Lee, 2021) such as GPT-2 (Radford et al., 2018) and T5 (Raffel et al., 2020). These approaches identify task-specific slots and values within a user input utterance, track belief states (i.e. user’s goals), retrieve database records using current belief states, estimate an appropriate system action, and translate the system action to a natural language response in an end-to-end manner. These approaches allow the system to be generalized to a new schema of a new domain, however, they are based on the assumption that each domain is associated with a unified schema where its schema elements (i.e. slots) are commonly shared across domains. For example, address and postcode are used as shared slots for both Hotels and Restaurants domains. This assumption does not necessarily reflect a real-world scenario where multiple services with different schemas exist for a single domain. Each service schema can consist of slots that are semantically similar to slots of other services, and slots that are specific to its own service. As illustrated in Figure 1, while Restaurants_1 and Restaurants_2 services of Restaurants domain share the same
10148
slots such as restaurant_name and date, there are also slots such as cuisine of Restaurants_1 and category of Restaurants_2 which correspond to a similar interpretation and usage, but their surfaces are different. In addition, Restaurants_1 has a specific slot, serves_alcohol, whereas Restaurants_2 has has_seating_outdoors specified in its schema.
Without addressing the differences in schemas of multiple services of the same domain, the system would unsuccessfully associate values within the user utterance with relevant slots that are applicable for each service. This could lead to inaccurate tracking of the user’s goal which in turn, causes the system to generate an irrelevant response. Therefore, in this work, we proposed a schema-aware end-to-end neural network model, initialized with a pre-trained language model T5 (Raffel et al., 2020), to handle task-oriented dialogues of multiple services and domains with diverse schemas. Our proposed model is adaptable to a dynamic set of slots within a schema as we feed slot descriptions and value constraints as the additional context information, along with the dialogue history to the model. For the experiments, the variations of the proposed model using different schema information were evaluated and compared to recent approaches for end-to-end TOD systems using a publicly available dataset, Schema-Guided Dialogue (SGD) dataset (Rastogi et al., 2020). According to the experimental results, the variations of the proposed model outperformed the baselines in all evaluation metrics in terms of the end-to-end modeling and dialogue state tracking (DST) task. In addition, our proposed model demonstrated improved generalizability on new services and domains under zero-shot and few-shot settings.

heading: 2 Related Works

heading: 2.1 End-to-end approach
text: Traditionally, the TOD system relies on a pipeline module comprising four subtasks including natural language understanding (NLU) (Yao et al., 2013; Hashemi et al., 2016; Wu et al., 2020), dialogue state tracking (DST) (Mrkšić et al., 2015, 2017; Chen et al., 2020), dialogue policy (POL) (Peng et al., 2017, 2018), and natural language generation (NLG) (Mi et al., 2019; Peng et al., 2020). However, due to an advanced neural approach, it has been proven that building TOD systems based on a single end-to-end neural model achieved promising results and potentially enables efficient optimiza-
tion (Wen et al., 2017; Lei et al., 2018; Zhang et al., 2020; Ham et al., 2020; Lin et al., 2020; HosseiniAsl et al., 2020; Wang et al., 2022; Peng et al., 2021; Yang et al., 2021; Lee, 2021). Among the end-toend approaches, Wen et al. (2017) first proposes a framework using a sequence-to-sequence architecture (Sutskever et al., 2014) without explicitly modeling the user goal. Similarly, Lei et al. (2018) relies on a sequence-to-sequence architecture with an extendable feature for modeling belief spans. Zhang et al. (2020) considers one-to-many conversational property to generate multiple responses. Another line of work leverages transfer learning using a large-scale pre-trained language model (Peng et al., 2021; Yang et al., 2021; Lee, 2021). However, none of these works consider the presence of multiple services within a single domain and only assume that most schema elements are shared among domains. Thus, their approaches cannot recognize the difference in schemas among services and could not be effectively applied when a new schema of a new service does not correlate with known schemas.

heading: 2.2 Dialogue state tracking using context
text: DST is an essential and challenging task for TOD systems, especially when considering tracking slots on different schemas. One effective approach for this challenge is to utilize cross-domain transfer learning which allows the neural models to be applied to new domains (Wu et al., 2019; Kumar et al., 2020; Lin et al., 2021a).
In addition, some recent works propose to incorporate encoding of slot names or descriptions based on schemas to the model and it demonstrated a greater performance in cross-domain DST (Rastogi et al., 2020; Chen et al., 2020; Zhu et al., 2020; Lee et al., 2021; Lin et al., 2021b). Accordingly, we urge that leveraging context information of slots such as descriptions and value constraints in endto-end modeling could improve the generalizability of the system on new services and domains.

heading: 3 Schema-aware End-to-End Model for Task-Oriented Dialogue Systems
text: In this section, we describe our proposed schemaaware end-to-end model which is composed of five main components as illustrated in Figure 2. Each component is initialized with either a pretrained T5 encoder or decoder (Raffel et al., 2020). Our model utilizes slot descriptions and value con-
straints to handle a dynamic set of slots within a schema of individual services. The flows to achieve end-to-end modeling based on the five main components are explained in the following subsections:
Dialogue Encoder Initially, the model receives a given dialogue history Ht which includes all dialogue components from turn 1 to t− 1 and a user utterance Ut at each dialogue turn t to generate an encoded dialogue state through a dialogue encoder.
Different from the previous works which utilize a user utterance U , a belief state B, a database (DB) state DB, a system action A, and a system response R within a dialogue history H , we also include a domain state D as it signals the model with a different focused domain which later used to determine a schema at each turn t. Thus, Ht is constructed by a sequence of U1, D1, B1, DB1, A1, R1, . . . , Ut−1, Dt−1, Bt−1, DBt−1, At−1, Rt−1.
Domain Decoder By using the encoded dialogue state, a domain decoder predicts a domain state Dt where it represents a domain dt relevant to the given dialogue state. A domain dt is used as an indicator for retrieving a set of slots St based on a service schema.
We opted to predict a domain related to each dialogue turn instead of directly predicting a ser-
vice because services of the same domain are too similar. This makes it difficult for the model to differentiate. Thus, during inference, we retrieve a set of slots St by comparing dt to the domain of the true service at each turn t. The model will be given a relevant set of slots for the dialogue if a predicted dt indicates the domain of the true service, otherwise, the model will be given a set of slots belonging to any service of the predicted dt.
Schema Encoder After obtaining a set of slots St within a service schema of a domain, we represent each slot Sit ∈ St as a text sequence using its description, value constraint which indicates possible slot values, and slot token. Each sequence is fed to a schema encoder, where its parameters are shared with a dialogue encoder, to generate vector representations. Then, we regard the encoded representation of the last token within a given sequence as an embedding of each slot Sit in a schema.
Schema-aware Belief Decoder With the encoded dialogue state and a collection of slot embeddings, the model generates a belief state Bdt that tracks a user’s active goal according to a domain dt via a schema-aware belief decoder.
As opposed to the existing works where their approaches attempt to generate a belief state rep-
resenting the user’s goals on several domains at once, we focus on tracking changes in the user’s goal for a single domain at a time. This is because we observed that the user’s goal of a single domain would be updated per each dialogue turn. Therefore, it is nonessential for the decoder to repeat the same user’s goal for other inactive domains that do not change. In addition, updating a belief state per domain allows the decoder to pay attention to one set of slots and not mix up with unrelated slots of other services and domains. Bdt is then used for querying DB records matching its constraints for constructing a DB state, DBt, representing a number of matching entities.
At the end of each turn, we update Bt which indicates the user’s active goals for multiple related domains for the dialogue by accumulating every Bdt of every turn up to t.
Schema-aware Response Decoder Lastly, a schema-aware response decoder utilizes the encoded dialogue state by Ht and Ut, a collection of slot embeddings by St, as well as a DB state DBt, to estimate a proper system action At and generate a system response Rt in reply to the user utterance Ut.

heading: 3.1 Schema-Aware Decoder
text: In this section, we describe the schema-aware decoder and its importance as the bases for our belief and response decoders in the proposed model.
In general, the decoder would produce a target representation for each target token in a sequence based on attention scores using representations of the input sequence from the encoder, and former target representations by the decoder. Each attention score captures the influence of each token within the input sequence on a target token. Thus, in addition to dialogue token representations Vdial from a dialogue encoder, we infuse slot embeddings Vslot based on a service schema as supplementary information to the decoder by [Vslot,Vdial], where [·, ·] is the concatenation of two matrices. In this way, we expect that the decoder could make use of slot embeddings to estimate the correlation between slots and a target token such that the decoder could infer relevant slots in the generated sequence.
To further encourage predictions among slots of a given service schema, after projecting a final target representation to a fully-connected linear layer, we suppress scores x of target tokens where they indicate the irrelevant slots that do
not belong to a given schema. To achieve this, we add weights α = (α1, α2, . . . , αn) to scores x = (x1, x2, . . . , xn), where αi marks a value zero if i-th presents a relevant slot to a schema, otherwise ∞, and n denotes the number of target tokens.

heading: 3.2 Training Objective
text: The optimization functions for the domain, belief, and response (system action and response) generation are defined as,
Ldomain = − log p(Dt | Ht, Ut) (1) Lbelief = − log p(Bt | Ht, Ut, St) (2) Lresp = − log p(At, Rt | Ht, Ut, St, DBt) (3)
Our model is trained to jointly minimize the weighted sum of the three losses, L = Ldomain + Lbelief + Lresp.

heading: 4 Experiments

heading: 4.1 Dataset
text: We conducted the experiments using a well-known dataset, the Schema-Guided Dialogue (SGD) dataset (Rastogi et al., 2020). As opposed to other datasets such as MultiWOZ (Budzianowski et al., 2018) which only defines a single service per domain, the SGD dataset was specifically designed to cover multi-turn dialogue conversations concerning the presence of multiple services per domain. Moreover, as shown in Table 1, the SGD dataset includes a high number of slots per service and domain which is very suitable to our setting as we consider multiple related services per domain and expect the model to be adaptable to various sets of slots within a service schema of a domain.

heading: 4.2 Preprocessing
text: Dialogue components are represented similarly as in the existing works (Yang et al., 2021; Lee, 2021) in which they converted each dialogue information into a span and enclose it with a start tag <sos_∗> and an end tag <eos_∗> where ∗ denotes a span’s
abbreviation. To represent a slot in belief and system action spans, we delexicalized a slot with [slot] placeholder where slot denotes a slot name with an underscore replacing a space since there are slots with long names in the SGD dataset. Additionally, we also replaced a slot value in a response span with [slot] instead of [value_slot] as in the previous works. The reason is that it is not necessary for the model to learn a separate correlation between a slot and a target token and between a slot value and a target token. This helps the model to conveniently determine a slot placeholder for every type of span. For slot context span, we form a text sequence following a template, "<sos_c> ts [desc] ds [cons] vs <eos_c>", where ts, ds and vs denote a slot token, description and value constraint, respectively.

heading: 4.3 Experimental Setting

heading: 4.3.1 Baselines and Comparisons
text: We compared our proposed model with the three most recent end-to-end modeling approaches for TOD systems, SOLOIST (Peng et al., 2021), UBAR (Yang et al., 2021) and MTTOD (Lee, 2021). SOLOIST is the model implemented based on GPT-2 with a contrastive objective using negative examples. UBAR is built on a session-level of dialogues based on DistilGPT2 (Sanh et al., 2019), a distilled version of GPT-2. MTTOD relies on a base version of T5 and is separated into three components (i.e. a dialogue encoder, belief encoder, and response decoder). We followed the implementation that was released by each baseline to obtain results on the SGD dataset.
For our proposed model, we examine the effectiveness of using different slot context information. The variations include (1) SchemaTOD (w/o context) which is the proposed model without exploiting any slot context information, (2) SchemaTOD (desc) which is the model using only slot descriptions, (3) SchemaTOD (cons) which is the model using only slot value constraints, and (4) SchemaTOD (desc, cons) which is the model where we incorporate both slot descriptions and slot value constraints.

heading: 4.3.2 Evaluation Metrics
text: To evaluate task completion and the quality of generated responses, we follow the automatic evaluation metrics by Budzianowski et al. (2018) including (1) BLEU which measures the fluency of the generated responses (Papineni et al., 2002), (2)
Inform which measures whether a system has provided accurate entities according to a belief state, (3) Success which verifies whether all requested information has been given to the user, and (4) Combined as a combined score of the three aforementioned scores to measure an overall quality suggested in Mehri et al. (2019), which is computed as (Inform + Success) × 0.5 + BLEU. In addition, we use Joint Goal Accuracy (JGA) to evaluate the performance of the model for the DST task.

heading: 4.3.3 Implementation Details
text: We implemented the proposed model based on the base variant of T5. All baselines and model variations were trained using the AdamW (Loshchilov and Hutter, 2018) optimizer with a constant learning rate of 0.0003 for a total of 15 epochs for models operating on a turn-level of dialogues (i.e. SOLOIST, MTTOD, and SchemaTOD) and 20 epochs for models operating on a session-level (i.e. UBAR). For SOLOIST and UBAR, we trained the models using a batch size of 4 and a gradient accumulation step of 4 on a four-GPUs NVIDIA V100, while we trained MTTOD and variations of SchemaTOD using a batch size of 16 with no gradient accumulation on an eight-GPUs NVIDIA A100. We adopted a simple greedy decoding technique with a temperature of 0.7. For evaluating results on a test set, we choose the best model checkpoint with the best performance based on the Combined score on a validation set.

heading: 4.4 Experimental Results

heading: 4.4.1 End-to-End
text: Table 2 shows the performance of the baseline approaches and the proposed model using different slot context information for the evaluation of endto-end modeling. All variations of our Schema-
TOD outperformed all of the baselines regarding Inform, Success, and combined scores. For BLEU, SchemaTOD (desc) and SchemaTOD (desc, cons) achieved higher BLEU scores compared to BLEU scores achieved by all baselines. This suggests that our proposed SchemaTOD could retrieve more correct entities based on the tracked belief states as well as generate system responses with the requested information more precisely compared to the baselines. As the baselines do not concern with diverse schemas of different services in a domain, they were unable to pick the correct slots that be-
long to a target service at each dialogue turn. Therefore, the baseline could not successfully retrieve a set of entities and failed to provide the requested information.
Among the variations of SchemaTOD, SchemaTOD (desc, cons) achieved the highest BLEU score at 15.03, while SchemaTOD (desc) achieved the highest Inform, Success, and combined score at 26.65, 20.44, and 38.27, respectively. SchemaTOD (w/o context) without using any slot context information achieved the lowest scores for all evaluation metrics compared to SchemaTOD (desc) and SchemaTOD (desc, cons). Thus, incorporating slot context information allows the proposed model to be more flexible to diverse service schemas which contain different sets of slots for each domain.
To examine more closely how different slot context information contributes to the performance of the TOD system regarding different service schema of the domain, Table 3 presents combined scores achieved by three of our SchemaTOD variations including SchemaTOD (desc), SchemaTOD (cons) and SchemaTOD (desc, cons) for each domain. The results suggest that the highest combined score could be achieved by different variations depending on the domain. While SchemaTOD (desc) which uses only slot descriptions achieved better performance for most domains with seen services during training (four out of six domains with seen services including Services, Movies, Ridesharing and Travel), SchemaTOD (desc, cons) using both slot descriptions and value constraints achieved better performance for most domains with only unseen services (five out of eight domains including Events, Rental Cars, Buses, Flights and Music). For Homes domain, the highest combined score could be achieved by SchemaTOD (cons) which uses only slot value constraints. According to Figure 3 which illustrates the portion of categorical
and non-categorical slots of each domain in the SGD test set, Homes domain has the highest percentage of categorical slots (45.45%) when compared to other domains where only unseen services are available in the test set.
In addition, although SchemaTOD (desc) outperformed other variations of SchemaTOD for most unseen domains (three out of four unseen domains including Trains, Messaging and Alarm), most slots of these domains are non-categorical. For the unseen domain with more categorical slots like Payment, SchemaTOD (desc, cons) achieved better performance with the highest combined score. This suggests that using both slot description and value constraints works well for domains with unseen services and unseen domains where the number of categorical slots is significantly high, otherwise using only slot descriptions would suffice. Using slot value constraints would enhance the performance in the case that the schema contains enough useful clues for the possible values of the categorical slots.

heading: 4.4.2 Dialogue State Tracking
text: Apart from the evaluation of the proposed model for the end-to-end modeling, we also evaluate the performance of variations of our SchemaTOD for the DST task which estimates the user’s goals based on the dialogues. Table 4 shows the overall performance of baselines and variations of our SchemaTOD in terms of JGA on all test dialogues. SchemaTOD (desc, cons) achieved the highest JGA at 31.14% which is approximately 6.21% higher than the best JGA achieved by the baseline, MTTOD, at 24.93%. The results achieved by the variations of SchemaTOD are comparable to each other with a small difference in JGA.

heading: 4.4.3 Generalization on New Services and Domains
text: To evaluate the model’s generalizability on new services and domains with zero or few training examples, we tested the baselines and our proposed SchemaTOD on zero-shot and few-shot settings. Since there are many domains available in the SGD test set, we chose to focus on the performance of the baselines and the best variation of SchemaTOD based on the performance achieved in Table 3 on the top four domains with the highest number of dialogues with unseen services. The top four domains include Events, Hotels, Rental Cars, and Buses. The best variation of SchemaTOD is SchemaTOD (desc, cons) for all four domains.
For a zero-shot setting, we held out all dialogues in the training and development sets that involve each target domain and trained all models using the remaining dialogues by following the same procedure as described in Section 4.3.3. Then, we tested the best-trained models on the dialogues involving the target domain within the test set.
For a few-shot setting, we finetuned each model that was trained without exposing it to each target domain by using a random sample of 64 and 32 dialogues of each target domain from the training and development sets, respectively. We finetuned the models for a total of 5 epochs and chose the best model according to the combined score achieved on the random sample dialogues from the development
set. Like the zero-shot setting, we tested the bestfine-tuned models on all dialogues involving the target domain within the test set.
According to Table 5 which demonstrates the performance of the models on different training settings for four target domains, the proposed model, SchemaTOD, achieved the highest combined scores in every domain when the model was trained using the original training set. The highest combined score when using the original test dialogues could be achieved for Hotels domain as the test dialogues for this domain involve seen services during training. While under zeroshot and few-shot settings, the combined scores achieved by every model were reduced compared to those achieved by the models trained on the original training set, and the highest scores were achieved mostly by the proposed model. For the zero-shot setting, SchemaTOD performed the best on Events and Rental Cars, while achieving the highest combined scores on the three domains including Events, Hotels, and Rental Cars for the few-shot setting with the improvement of 4% from the best baseline.
Since our proposed model relies considerably on a retrieved set of slots based on a predicted domain, this makes it challenging for our model to interpret the context of the dialogues when the predicted domain is incorrect and causes the retrieved set of slots to be irrelevant to the dialogues. As suggested by the results for the zero-shot setting, SchemaTOD could perform better for Events and Rental Cars since these two domains have more slots that are shared with other domains such as date, time, and city. Thus, even though the proposed model failed to utilize the relevant slots, it could still make use of most slots available for other domains interchangeably. On the other hand, the Hotels and Buses domains have more domain-specific slots such as check_in_date, has_laundry_service,
has_wifi, transfers, and additional_luggage. In this case, the model could not leverage slots of any other domains when the model fails to predict a correct domain for the dialogues.
However, once the model has been exposed to a few dialogue examples of a target domain, SchemaTOD could achieve better performance for most domains in the few-shot setting due to its ability to utilize a relevant set of slots based on the correct predicted domain. Figure 4 presents the actual responses generated by our model under zeroshot and few-shot settings in comparison with the ground truth responses. The second response by the proposed model under the zero-shot setting contains slots that do not belong to the Hotels domain since the model predicted Homes for the scope of this dialogue. Therefore, this response implies more detail on the available home. In contrast, the second response by the model under the few-shot setting suggests the hotel recommendation which corresponds to the ground truth as the model determines the dialogue as for Hotels domain.

heading: 5 Conclusion
text: In this paper, we focus on improving the end-to-end TOD systems to support a domain covering multiple services as well as to achieve better generalizability. We propose SchemaTOD, a schema-aware end-to-end neural network model, which incorporates context information of slots within a service schema of a domain. Experiments conducted on the SGD dataset suggest that our proposed model when using both slot descriptions and value constraints achieved the best performance when the domains consist of only unseen services and when the domains are unseen with more categorical slots. Otherwise, the proposed model using only slot descriptions outperformed other models for the domains with seen services and the unseen domains with a high number of non-categorical slots. For
generalizability, the performance achieved by the proposed model demonstrated improvement over the baselines when using a few dialogue examples of the new services and domains.
Limitations
As described in Section 4.4.3, the proposed model requires precise prediction on the target domain at each dialogue turn to utilize the relevant slot information within the service schema. This possesses a difficulty to apply the proposed model to some new domains, especially when the domains share high similarity with seen domains. To reduce this difficulty, as a future improvement, one potential approach is to process on a hierarchical structure of slots within schemas, instead of individual slots. In this way, the model does not need to rely on the domain, but only on a group of similar slots. The model can then perform matching of a chosen group of slots with available slots within a schema and composes responses based on those slots.

heading: Acknowledgements
text: We thank all anonymous reviewers for their insightful feedback. This work is based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).

heading: A Example Outputs

heading: B Domain Accuracy
text: Table 7 shows accuracy by the four variations of SchemaTOD in predicting domains for the dialogue turns, while Table 8 presents the accuracy by SchemaTOD (desc, cons) trained on the original training set, and under zero-shot and few-shot settings on the four target domains.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations".
7 A2. Did you discuss any potential risks of your work? We did not collect any data to construct a new dataset. We only use a well-known dataset for task-oriented dialogues which was well-designed and reliable. Thus, we do not involve any human subjects and annotations on our own. All the experiments are done using automatic evaluations. It is unlikely that our work would risk personal information exposure and misconduct.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section "Abstract" and 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.
3 B1. Did you cite the creators of artifacts you used? Section 4.1 and 4.3.1.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The license and terms of use for our code can be found in the supplementary material. Also, for other existing artifacts that we used, the license and terms of use can be found in the original repository addressed in each original paper.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We include an Acknowledgement section in the supplementary material that acknowledges the released code and data of each existing artifact.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not discuss the steps of the data collection process as we utilize a publicly available dataset. A detailed explanation of the dataset can be found in the original paper.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4.1.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.3.3.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.3.3.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.3.1, 4.3.2 and 4.3.3.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To reduce this difficulty, as a future improvement, one potential approach is to process on a hierarchical structure of slots within schemas, instead of individual slots. In this way, the model does not need to rely on the domain, but only on a group of similar slots. The model can then perform matching of a chosen group of slots with available slots within a schema and composes responses based on those slots