Workflow Discovery from Dialogues in the Low Data Regime
Authors: Amine El hattami
Issam H. Laradji
Stefania Raimondo
David Vazquez
Pau Rodriguez
Christopher Pal
Venue: Accepted by TMLR ()

Abstract: Text-based dialogues are now widely used to solve real-world problems. In cases where solution strategies are already known, they can sometimes be codified into workflows and used to guide humans or artificial agents through the task of helping clients. We introduce a new problem formulation that we call Workflow Discovery (WD) in which we are interested in the situation where a formal workflow may not yet exist. Still, we wish to discover the set of actions that have been taken to resolve a particular problem. We also examine a sequence-to-sequence (Seq2Seq) approach for this novel task. We present experiments where we extract workflows from dialogues in the Action-Based Conversations Dataset (ABCD). Since the ABCD dialogues follow known workflows to guide agents, we can evaluate our ability to extract such workflows using ground truth sequences of actions. We propose and evaluate an approach that conditions models on the set of possible actions, and we show that using this strategy, we can improve WD performance.  Our conditioning approach also improves zero-shot and few-shot WD performance when transferring learned models to unseen domains within and across datasets. Further, on ABCD a modified variant of our Seq2Seq method achieves state-of-the-art performance on related but different problems of Action State Tracking (AST) and Cascading Dialogue Success (CDS) across many evaluation metrics.

Published in Transactions on Machine Learning Research (02/2023)
Workflow Discovery from Dialogues in the Low Data Regime
Amine El Hattami
ServiceNow Research
Polytechnique Montréal, Montréal, Canada
Issam Laradji
ServiceNow Research
Stefania Raimondo
ServiceNow Research
David Vázquez ServiceNow Research
Pau Rodriguez ServiceNow Research
Christopher Pal
ServiceNow Research
Polytechnique Montréal, Montréal, Conada
Reviewed on OpenReview: httep///ppaaaerrssseee////rrr/ir
Abstract
Text-based dialogues are now widely used to solve real-world problems. In cases where solution strategies are already known, they can sometimes be codified into workflows and used to guide humans or artificial agents through the task of helping clients. We introduce problem formulation that call Workflow Discovery (WD) in which in the situation where formal workflow may not yet exist. Still, wish to discover the set of actions that have been taken to resolve particular problem. We also examine (Seq2Seq) approach for this novel task. We present experiments where we extract workflows from dialogues in the Action-Based Convorsations Dataset (ABCD). Since the ABCD dialogues follow known workflows to guide agents, we can evaluate our ability to extract such workflows using ground truth sequences of actions. We propose and evaluate an approach that conditions models on the of possible actions, and we show that using this strategy, we improve WD performance. Our conditioning approach also improves zero-shot and few-shot WD performance when transferring learned models to unseen domains within and across datasets. Further, on ABCD modified variant of our Seq2Seq method achieves state-of-the-art performance on related but different problems of Action State Tracking (AST) and Cascading Dialogue Success (CDS) across many evaluation metrics.
interested
Code available http://ghhhi..anneeeeeee/////oor

---

Published in Transactions on Machine Learning Research (02/2023)
1 Introduction
Task-oriented dialogues ubiquitous in everyday life and customer service in particular. Customer support agents use dialogue to help customers shop online, make travel plans, and receive assis- lance for complex problems. Behind these dialogues, there could be either implicit or explicit work- flows actions that the agent has followed to ensure the customer request is adequately addressed.
For example, booking an airline ticket. might comply with the following work- flow: pull up account, register seat, and request payment. Services with formal workflows struggle to handle vari- ations in how a particular issue is re- solved, especially for cases where cus- tomer support agents tend follow "un- written rules" that differ from one agent to another, significantly affecting cus- tomer satisfaction and making training new agents more difficult. However, cor- rectly identifying each action constitut- ing a workflow can require significant do- main expertise, especially when the set of possible actions and procedures may change over time. For instance, newly update to the returns
Task-Oriented Dialogue
Predicted Workflow
need to check on the status of my subscription.
Pull up account [Janne55]
Can you give your usemame. please?
it Janne55
WD
Verify identity
Your subscríption is valid unfil next month.
Get subscription status
Possible Actions
Offer refund, Pull up account. Send link. Verify identity, Get subscription status,
Our method for extracting workflows from dialogues. The input consists of the dialogue utterances and the list of pos- sible actions if available. The output the workflow' followed to resolve a specific task, consisting of the actions (e.g., Verily identity) and their slot values (c.g., Janne55).
Figure 1:
added items policy in online shopping service may require modifying the established work- flows.
In this work, we focus on "workflow dis- covery" (WD) the extraction of work- flows that have either implicitly explic- itly guided task-oriented dialogues be- tween two people. Workflows extracted from a conversation consist of sum- mary of the key actions taken during the dialogue. These workflows consist. of pre-defined terms for actions and slots when possible, but our approach also al- lows for actions that be invented by the model online and used as new steps in the generated work- flow. Our approach is targeted toward the task analyzing chat transcripts be- tween real people, extracting workflows from transcripts, and using the extracted workflows to help design automated dia- logue systems to guide systems that are alrcady operational. Alternatively, extracted workflows might also be used for human agent training. One might imagine many scenarios where an analyst might use WD to understand if an un- resolved problem is due to divergence
Dialogue
Predicted Workflow
Can you help me book a table of Chinese restaurant that's in the center of town?
Book Chinese restaurant [2 people. 11 ::45
Great News! was able to reserve your table for 2 at the 11:45 time. They will hold the table for 15 minules. Reference number is GFIL3UG5
Return reference numbers [GFIL3UGS]
am looking for some lype entertainment
WD
same area as lhe restaurant, please.
not known to
What about museum The Cambridge contemporary ar is nice
Find attraction
[museum,
Cambridge)
Possible Actions
Modified Action
New Action
Book hotel. Find hotel, Book restaurant, Find attraction, Find train.
Known Action
Figure 2: Our conditioning approach allows for better zero-shot and few-shot. performance. Entirely new workflow actions (i.e, not the list of possible actions) can be proposed, as well those based on minor modifications to known actions.
2

---

Published in Transactions on Machine Learning Research (02/2023)
from formal workflow to understand if workflows have organically emerged, even in cases where no formal workflow documentation exists. WD is related to the well-established field of process mining, but process mining typically extracts workflow information from event logs as opposed to unstructured dialogues encoded as natural language. Our work shows that traditional process mining can be dramatically expanded using modern NLP and sequence modeling lechniques, as we propose and explore here. Our approach correspondingly allows traditional event. logging information to be combined with NLP allowing systems constructed using our approach benefit from recent advances in large language models (LLMs), including enhanced zero-shot and few-shot learning performance.
Our WD problem formulation here is related to dialogue act modeling in the sense that different steps along our workflows may be composed of dialogue acts in the sense of Stolcke et al. (2000), such as: posing yes or no questions, acknowledging responses, thanking, etc. Since WD is performed offline and focuses on workflow extraction, more fine-grained dialogue acts can help guide a WD model to extract more abstract workflow stops. WD is also different from standard (closod world) intent detection and slot filling Liu & Lane (2016), dialogue state tracking (DST) Williams al. (2014) and open world intent and slot induction Perkins & Yang (2019) in part because WD focuses extracting actions taken by an agent as opposed to intents of the user. We explore WD model variants that extract both agent actions and arguments or slots obtained from the user. WD could be seen as generalization of the recently proposed notion of Action State Tracking (AST) (Chen et al., 2021), but WD differs from AST in that: 1) WD is performed offline summarizing dialogues in terms of workflows; 2) unlike AST, WD does not require actions to be annotated at the turn level; 3) unlike AST, WD doesn't require known pre-defined actions and slots. One of the uses of WD is to create new names for new actions on the fly along with new slot types and extracted values; and, 4) these features of WD allow it to extract workflows containing new action and slot types which differ significantly from action and slot sequences seen during training. We sumuarrize these differences between AST and WD in Table | and we discuss all the relationships of WD to prior work in more detail in our Related Work section below.
Table A summary of the differences between AST and WD.
AST
WD
Performed offline
No
Yes
Requires annotated action turns
Yes
No
Requires known actions and slots types Significant deviation from known actions and slots possible
Yes
No
No
Yes
To address the challenges of WD introduced by dynamically changing actions, distributional shifts, and the challenges of transferring to completely new domains with limited labeled data, we propose a text-lo-text approach that can output the workflow consisting of the full set of actions and slot values from a task-oriented dialogue. Further, to better adapt to unseen domains, we condition our method on the full or partial set of possible actions as shown in Figure 2 enabling us to perform zero-shot and few-shot learning. Figure also illustrates the type of generalization possible with our approach in these regimes where we can propose new actions never seen during training. We investigate four scenarios for extracting workflows from dialogues under differing degrees of distributional shift: (1) no-aan workflows, where all workflow actions have been seen during training. (2) Cross-domain zero-shol, where actions from selected domains have been omitted during training but are present in the valid and test sets, (3) Cross-dataset zero-shot, where a model trained on all the domains of one dataset is applied to another domain a zero-shot setting, and (4) Cross-datasel few-shot, where model is trained on all the domains of one dataset, then trained on another domain in a few-shot setting.
Our contributions can be summaaiizd as follows:
We propose a new formulation to the problem of workflow extraction from dialogues, which, to our knowledge, has not been examined or framed as we present here. We cast the problem as summarizing dialogues with workflows and call the task Workflow Discovery (WD).
3

---

Published in Transactions on Machine Learning Research (02/2023)
We propose a text-to-text approach to solve the WD task that takes entire dialogues input and outputs workflows consisting of sequences of actions and slot values that can be used to guide future dialogues. We test our approach using various state-of-the-art text-to-text models and show its efficacy on the Action Based Conversations Dataset (ABCD) (Chen et al. 2021
We propose a conditioning mechanism for our text-to-text approach, providing the model with a set of possible actions to use as potential candidates. We show that this mechanism allows our method to be used in dramatically different domains from the MultiWOZ dataset (Budzianowski et al., 2018), yielding good cross-datasel zero-shot workflow extraction performance. Moreover, it allows important performance increase in the cross-dataset few-shot setting.
Using variant of our approach, achieve state-of-the-arr results on related but different and more standard tasks of Action State Tracking (AST) and Cascading Dialogue Success (CDS) on the ABCD evaluation.
2 Related Work
Our work intersects with three major groups of topics: Task Yrist-Oreented Dialogues, Sequence-to-Sequeece Text Generation, Intent/Slot induction, Action State Tracking (AST) and Process Mining & Discovery.
Task-Oriented Dialogues In lask-oriented dialogues, the system must grasp the users' intentions behind their utterances since this is the basis for selecting an appropriate system action, it should be able to understand and respond to wide range of user inputs, as well as handle complex tasks and maintain coherence in the dialogue Stolcke el al. (2000); Henderson et al. (2014). The two key tasks are intent detection and named entity identification, both of which have comparable components in task-oriented dialogues. Intent detection may be viewed as a classification task (action classification), with user utterances allocated to one or more intent categories. The goal of named entity recognition is to categorize entitics in given utterance, such as hotel names, room characteristies, and aftribute values Zang et al. (2020).
Although most systems perform intent detection and named entity identification for generic conversation systems Rafailidis & Manolopoulos (2018); Yan et al. (2017), the main goal of our method is to output a set of actions that resolve the task specified in the dialogue that abide by the system's 's guidelines.
Intent and Slot Induction. In human-human dialogues, the purpose of intent/slot induction is to de- termine user intents from user query utterances. In more recent years, there have been several efforts on intent/sla detection in conversation systems based on methods like capsule networks and relation networks Min el al. (2020); : Qin et al. (2020); Zhang el al. (2018); Qin et al. (2019); Niu el al. (2019). However, they all make the assumption that the intents are within a closed world in that the intents in the test also appear in the training set. In contrast, while work is about identifying the set of actions that address the task in dialogue consider the possibility that the actions in the test are novel.
Prior work has also sought to identify novel intents and slots. For example, Brycheín & Král (0016) proposed
an unsupervised method for identifying intents without utilizing prior knowledge of known intents. Ryu
let al.
2018) and Yu et al. (2017) proposed methods based on adversarial learning, whereas Kim & Kim (2018) proposed mothods based on an in-domain classifier and an out-of-domain detector to detect unknown
intents. Perkins & Yang (2019) used clustering with multi-view representation learning for intent induction,
whereas Zeng al. (2021) proposed method based on role-labeling, concept-mining and pattern-mining for discovering intents open-domain dialogues. Closer to our work is Yu et al. (2022) which leveraged large language models for discovering slot schemas for task-oricnted dialog in an unsupervised manner. They use unsupervised parsing to extract candidate slots followed by coarse-lo-fine clustering to induce slot types. In contrast, in our work, we use large language models and a prompting technique to detect and even label out-of-distribution actions. Rather than clustering embeddings or other structures, we leverage knowledge obtained during transformer pre- pre-training, and a special prompting scheme to let the LLM simply generate new workflow steps text. Our approach allows our models to propose new, plausible, and typically quite
4

---

Published in Transactions on Machine Learning Research (02/2023)
appropriate names for workflow steps that have not been seen in the workflow extraction training data. We present our analysis of these types of experiments in Section 6.3.3
Sequence-to-SSqquenc models Text generation has emerged as one of the most significant but difficult problems in natural language processing (NLP). RNNs, CNNs, GNNs, and pretrained language models (PLMs) have enabled end-to-end learning of semantic mappings from input to output (Raffel et al. 2020b; Lewis et al. 2019). The architectures of PLMs are divided into Encoder-Decoder (like BERT Devlin et al. (2018) and T5) and Decoder-only (like
et al. (2020)) architectures. We employ Encoder-Decodke based sequence-to-sequuen to-sequence models in this study since we want to accept an entire dialogue as input and produce a set of actions as workflow. Many Encoder-Decoder methods exist, in the literature and some of their main differences are how they were pretrained. For instance, PEGASUS Zhang et al. (2020) trains by /removinn important sentences from input document, T5 Raffel et al. (2020b) uses multi-task approach for pretraining, BART Lewis et al. (2019) uses static masking of sentences across epochs, and
RoBERTa Liu et al. (2019) uses dynamic masking, different parts of the sentences are masked across epochs.
These types of architectures form the basis of our evaluation.
Our work is connected to models developed for translation (Sutskever et al. 2014) and summarization (Gliwa et al. 2019 Zhang et al.] 2020), structured text generation such as Text-to-Code Wang et al.) (2021) and Text-to- SOLL Scholak et al. (2021). However, here we focus new task, where dialogues are transformed into sequences of steps that an agent has used to solve task.
Structured Text Generation. Structured text generation such as text-to-SQL, and text-to-code have received a lot of recent attention Warren & Pereira (1982); Zettlemoyer & Collins (2012); Finegan-Dollak
et al. 20188: Feng et al. (2020): Scholak et al. (2021): Wang et al. (2021): ): Yu et al. (2019). The majority of the past text-to-SQL. effort has been devoted to transforming a single, complicated question into its accompanying
SQL query. Only a few datasots have been created to link context-dependent inquiries to structured queries, but the recent conversational text-to-SQL. (CoSQL) work of Yu et al. (2019) has examined the generalion of SQL queries from multiple turns of a dialogue. Their work also involved creating dataset under WOZ setting involving interactions between two partics. This dataset consists of diverse semantics and discourse that encompass most sorts of conversational DB querying interactions; for example, the system will ask for clarification of unclear inquiries or advise the user of irrelevant questions. While this dataset is a great benchmark for SQL generation, here we are specifically interested in transforming dialogues into workflows consisting of much more diverse types of interactions. Correspondingly, we focus on the recently introduced ABCD dataset, Chen et al. (2021), which consists of dialogues where an agent's 's actions must accommodate both the desires expressed by customer and the constraints set by company policies and computer systems. Our goal is to predict sequences of agent actions and the arguments for those actions, which are often obtained from proviprovided information.
AST and CDS for Task-Oriented Dialogues. Action State Tracking (AST) is task proposed by Chen et al. (021)) which tries to predict relevant intents from customer utterances while taking Agent Guidelines into consideration, which goes steps beyond traditional dialog state tracking (DST) Lee et al. (2021). For instance, a customer's utterance might suggest that the action is to update an order. However, following the agent's guidelines, the agent might need to perform other actions prior to addressing the customer's 's intent (e.g,, validating the customer's identity). While AST shares some similarities with traditional DST tasks, its main advantage is that it parses both customer intents and agent guidelines to predict agent actions. However, AST relies on annotated action turns making it difficult to use existing dialogue datasets without substantial annotation effort. Further, AST relies on agent guidelines which require prior knowledge of all possible actions. Chen et al. (2021) also proposed the Cascading Dialogue Success (CDS) task that aims to access the model's ability to predict actions in context. CDS differs from AST since it does not assume that an action should occur in the current turn, but adds the possibility to take an action, respond with an utterance (c.g., requesting information from the customer), or terminate the conversation when the agent completed the task. Further, CDS is measured using modified version of the cascading evaluation
5

---

Published in Transactions on Machine Learning Research (02/2023)
metric that accesses success over successive turns opposed to AST which measures success in isolation single turn.
3 Workflow Discovery (WD)
3.1 WD Task Definition
We propose a novel task we call Workflow Discovery (WD) to extract the actual workflow followed by an agent from a Lask-oriented dialogue. A workflow is a sequence of actions with their respective slot values in the same order in which they appeared during the conversation. Formally, we define the WD task as follows:
Given dialogue D fui, u2: .... un), where n is the total namber of utterances in the dialogue, and an op- lional list of possible actions ō = {ai....,.z) where is the number of known actions, and 02 is unique work- flow action. A model should predict the target workflow W = {(01, [v/10 <= <= ni m,)),.., (ak, {ot, <= <= nx))); where a, E 6, vi is the jth slot value and n, is the ruumber of available slot values for action a,, and k is the number of workflow actions. Further, the model should also be able to formulate new compound keywords to characterize new actions as well as extract their slot values for actions that are not a part of the known action domain.
3.2 WD Compared to Existing Tasks
Workflow Discovery differs from dialogue state tracking (DST) where in WD, we are interested extracting the sequence of actions that have been followed to resolve an issue. We are particularly interested in the situation where the actions that are needed to resolve a problem associated with given intent task are not known priori and must be invented by the WD model. This is in sharp contrast to DST which generally requires dialogue states to be known and pre-defined. DST also generally focuses on tracking the state of one party in the conversation, e.g,, the user. In contrast, WD extracts act ions executed by the agent and slot values collected from user utterances. Furthermore, WD differs from Action State Tracking (AST), since WD aims to extract all actions and their matching slot values, using all dialogue utterances without any extra metadata like the annotated action turns at once (no online tracking). In contrast, AST predicts the next action at given turn, given the previous turns only. Models trained for AST help agents select the next action following the agent guidelines. In contrast, models trained for WD extract all actions from chat logs between real people, including those that might deviate from the agent guidelines. WD is aimed at agent training or workflow mining with the goal of formalizing the discovered processes for subsequent automation. WD can be applicd to completely new, organic tasks where possible actions are unknown. Some aspects of AST could be seen as a subset of WD when actions and slots are known. However, we believe that WD is more closely related to the concept of summarization using a specialized vocabulary for actions and slots, when possible, but inventing new terms when needed.
We compare and contrast WD, AST, and DST in Figure g We note that Figure shows situation where our WD approach has succeeded in inventing new term (i.e., 'Generate new code") which describes the exact action the agent performed. This result matches the agent guidelines (Chen et al. 2021), where there is a clear distinction between offering a promo code when a customer is unhappy in which our approach uses the known 'promo code" action, and generating a new code since the old one no longer working. A good WD method should exhibit compositional generalization when creating new terms for actions and their arguments. For example, if a system has been trained to summarize dialogues that involve granting a refund for ordering pair of pants, the WD summary of new dialogue involving the refund of shirt should use vocabulary consistent with the manner in which the refund of pair of pants has been expressed.
Finally, in contrast to policy learning, WD aims to extract a workflow from dialogue logs describing the sequence of agent actions that depend on slot values extracted from user utterances. The extracted sequence may differ significantly from the actions of an optimal or estimated policy. WD focuses on extracting potentially -nt-of-distribution workflows, where new action names and slot values derived from new domains must be invented. This makes WD very different from policy learning and more similar to dialogue policy
6

---

Published in Transactions on Machine Learning Research (02/2023)
act extraction, but where new acts and slot values must be invented on the fly to summarize new concepts at test time.
4 Baselines and Methodology
In this section, we describe our methodology for the WD task. Moreover, since there no existing baseline for our novel WD task, we included our text-to-text variants for both the AST and CDS tasks that we used to test our text-to-text task casting scheme against existing benchmarks.
4.1 Text-to- Text Workflow Discovery
We cast the WD task as text-to-text task where the input of the model PWD consists of all utterances and the list of possible actions, if available or partially available, formatted as
PWD = Dialogue: u1, th Actions: a1,...;
where un is dialogue utterance, is the total number of utterances in the dialogue, a, is workflow action, and a is the total number of available actions. 'Dialogue:' and "Actions: are delimiters that separate the dialogue utterances from the list of actions. Further, we omit the "Actions:' delimiter when possible actions are not provided. Adding the possible actions to the imput can be seen as a way to condition the model to use the provided actions rather than invent new ones. During training, the possible actions addod after "Actions:" in the input. is a shuffled set comprised of the current sample larget actions and randomly chosen number Tmin <=r
of actions, where mmin is hyper-parameter. This technique aims to make the model invariant to the position and the number of workflow actions, especially when adapting to new domain in the zero-shot and few-shot settings. For all other settings (i.e., during validation, testing, or the zero-shot setting), the list of possible actions contains all known actions without any modification. Finally, We use the source prefix Extract work workflow: The target workflow TwD formatted as
TWD ailel, ..., ef" : ax(vk,...,
where ag is a workflow action, &o is the number of actions, og is a slot value, and ny is the number of slot values for action k. "I" and "I' encapsulate the slot values. " and n.* are the separators for the slot values and actions. Moreover, if an action has no slot values, the target slot value list is set explicitly to [none]. An example of this casting scheme is shown in Figure 9a
Our text-to-text framework differs from other work since our prompts don't 't include slot descriptions value examples compared to Zhao et al.] (2022a). Moreover, Adding lists of possible actions to the prompt makes our method more flexible in the zero-shot setup (allowing new actions to be added on-the-fly) and improves performance in the few-shot setup. Finally, we don't prefix utterances with speaker prefixes (e.g., "User: ') compared Zhao et al.] (2022ah: Lin al.] (2021a), making our technique usable when this information is unavailable inaccurate; for example, in cases where the chat transcript originated from an upstream text-to-speech model.
4.2 Text-to- Text Action State Tracking
The goal of the AST task is to predict single action and its slot values given only the previous turns. This task is similar to the traditional DST with the difference that agent guidelines constrain the target actions. For example, an utterance might suggest that the target action is "validate-purchase" but the agent's proposed gold truth 'verify-identity' per the agent guideline because an agent needs to verify a customer's s identity before validating any purchase.
We cast the AST task a text-to-text task where the input of the model PAST consists of all the utterances formatted as
PAST = Context: Ui,... We
where is dialogue turn, including action turns, and t is the index of the current turn. "Context:" is delimiter. We use the source prefix Predict AST:". The target TAST is formatted as
TAST ar (eto,,..,,'""

---

Published in Transactions on Machine Learning Research (02/2023)
where a, is an action, tthh is slot value, and m is the number of slot values. "I" and "I" encapsulate the slot values. '," is the separator for the slot values. Moreover, if an action has no slot values, the target slot value list is set explicitly to [none]. An example of this casting scheme is shown in Figure 9b
4.3 Text-to-Tens Cascading Dialogue Success
The CDS Lask aims to evaluate a model's ability to predict the dialogue intent and the next step given the previous utterances. The next step can be to take action, respond with a text utterance, or end the conversation. Moreover, the CDS task is evaluated over successive turns. In contrast, AST assumes that the current turn is an action turn and is evaluated over individual turns.
We cast the CDS task text-to-text task where the model input P is formatted as
Pcns = Context: un, Wy Candidates:ei,... Co
where uy is dialogue turn, including action turns, and t is the index of the current turn. Gp E C is lext utterance from the current agent response candidate list. Ct, and v the size of C.. 'Context:" and "Candidates:" are delimiters. Finally, We use the source prefix PPredict CDS:'. The target ToDs is formatted differently depending on the target next step. When the next step is to take an action, the target is formatted as follows
T paction =; action; as at., of'!
where i is the dialogue intent, and values aelv), ..., v(r] is the step name and slots similar to the AST task formulation above. When the next step is to respond, the target is formatted as follows
TeDs = i; respond; c+11
where a+1 E C2 is the expected response utterance. When the next step is to terminate the dialogue, the target is formatted as follows
Tend = i; end
An example of this casting scheme is shown in Figure 9c
5 Experimental Setup
5.1 Implementation Details
In our experimentation, used the T5 (Rafeel et al., 2020b) BART Lewis et al. 2019); and PEGA- SUS (Zhang et al. 2020 models for the WD task, which call WD-T5, WD-BART, and WD-PEGASUS, respectively. Furt hermore, AST-T5, and CDS-T5, respectively. We use the Huggingface Transformers Pytorch implementation model variants and use the associated summarization checkpoint. fino-tuned on CNN MNilDailyMail Hermann et al. 2015) for all models. For T5, we use the small (60M parameters), base (220M parameters) and large 777MM parameters) variants, and the large variant for both BART (400M parameters) and PEGASUS (569M parameters). We fine-tuned all models on the WD tasks for 100 epochs for all experiments with a learning rate of 5c-5 with linear decay and batch size of 16. We set the maximum source length to 1024 and the maximum target length to 256. For the BART model, set the label smoothing factor to 0.1. We fine-tuned AST-T5, and CDS-T5 for 14 and 21 epochs, respectively, matching the original work of [Cen ct al. (2021), ; and used similar pyper-parameters used for the WD task. In all experiments, we use mmin = 10 as described Section 4.1 Finally, We ran all experiments on NVIDIA A100 GPUs with 80G memory, and the training time the longest experiment was under six hours.
use T5 model for the text-to-text AST and CDS lasks, which we call,
2 for all
Ahtter//luaginggeeco//rraaaa httpss//iaugiinffacccoommmddd
8

---

Published in Transactions on Machine Learning Research (02/2023)
5.2 Datasets
ABCD (Chen el al. 2021) contains over 10k human-to-human dialogues split over 8k training samples and 1k for each of the eval and test sets. The dialogues are divided across 30 domains, 231 associated slots, and 55 unique user intents from online shopping scenarios. Each intent requires a distinct flow of actions constrained by agent guidelines. To adapt ABCD to WD, we choose agent actions to represent the workflow actions. From each dialogue, we create the target workflow by extracting the actions and slot values from each action turn in the saine order they appear in the dataset. Then, we remove all action turns from the dialogue keeping only agent and customer utterances. We use the same data splits in ABCD. Furthermore, we use natural language action names (i.e.; using 'pull up customer account. instead of "pull-up-account" and we report the results of an ablation study in Section 6.3.5 showing the benefits of using natural language action names. Table 2 shows a subset of the names we used, and the full list can be found in Appendix
Table 2: Subset of ABCD workflow actions
Action Name
Natural Language Action Name
pull-up-account enter-details verify-identity
pull up customer account enter details
verify customer identity
MultiWOZ (Budiianwssi et al. 2018 contains over 10k dialogues with dataset splits similar to ABCD across eight domains: Restaurent, Attraction, Hotel, Tazi, Thain, Bus, Hospital, Police. We use MultiWOZ 2.2 (Zang et al.] 2020) as it contains annotated per turn user intents and applies additional annotations correction similar to Ye et al. (2022). In the MultiWOZ dataset, customer intents represent the workflow actions. We assume that the system will always perform a customer intent. To MultiWoz to the WD task, we create the target workflow by extracting the set of intents and slot values in the same order they appear in the dialogue. We don't make any modifications to the dialogue utterance since MutliWoz does not include action turns or any extra metadata. Similar to ABCD, subset of the natural language action names. See Appendix A.4.1 for the full list.
natural language action names. Table g shows
Table 3: Subset of MultiWOZ workflow actions
Action Name
Natural Language Action Name
find hotel
search for hotel
find_train
search for train
bbok_restauran
book table at restaurant
Our public code repository contains all the source code necessary to create the WD dataset from ABCD and MultiWoz. The repository also contains the annotated ABCD test subset for human evaluation. Further, we report dataset statistics in Appendix A.4.3
5.3 Metrics
We evaluate the WD task using the Exact Match (EM) and a variation of the Cascading Evaluation (CE) (Suhr et al., 2019) metrics similar to Chen et al. (2021). The EM metric only gives credit if the predicted workflow (i.e., actions and slot values) matches the ground truth. However, it is not good proxy of model performance on the WD task due to the sequential nature of workflows. For example, model that predicts all workflow actions correctly but one will have an EM score of 0, similar to another model that predicted all actions wrong. In contrast, the CE metric gives partial credit for each successive correctly predicted action and its slot values. Nonetheless, we kept the EM metric for applications where predicting the exact workflow critical. Furthermore, we report an action-only Exact Match (Action-only EM) and Cascading
9

---

Published in Transactions on Machine Learning Research (02/2023)
Evaluation (Action-only CE) for some experiments to help isolate the task complexity added by predicting the slot values. Finally, due to the text-to-text nature of our approach, we use stemming, and we ignore any failure that occurs due to a stop word miss-match (e.g,, we assume that the lensfield hotel is equivalent to lensfield hotel). Moreover, we use BERTScore (Zhang* ct al.) 2020) to evaluate the action in all zero-shot experiments. We assume that a predicted action is valid il has an ---orr above 95% (e.g., check customer identity is equivalent to verify customer identity).
We evaluate the AST task similar to Chen et al. (0021) where B-Slot and Value are accuracy measures for predicting action names and values, respectively. Action is a joint accuracy for both B-Slot and Value metrics. Furthermore, We evaluate the CDS task similar to Chen et al. (0021) using the Cascading Evaluation (CE) metric that relies on the Intent, NextStep, B-Slot, Value, and Recall@1 metrics that are accuracy measures for the dialogue intent, next step, action, value, and next utterance predictions. We omit the Recall@5, and Recall@10 metries results since our model only predict a single next utterance instead of ranking the top 5 or 10 utterances. However, the cascading evaluation calculation result remains valid since it only uses the Recall@1 scores.
6 Experimental Results
To validate the efficacy of our approach against existing baselines, we first compare our results to the current state-of-the-art on the AST and CDS tasks and show that our method achieves state-of-the-art results on both tasks. Then, we describe the experiments we used to evaluate our approach on the WD task, report the results, and discuss their conclusions.
6.1 Action State Tracking
Following the same evaluation method used in Chen et al. (2021), we compared our AST-T5 model against ABCD-RoBERTa (Chen et al.) (021)), the current ofate--ff-thearrs on the AST task. We report the results in Table 4 Moreover, we only report the results of AST-T5-Smal (60M parameters) variant for fair comparison since the ABCD-RoBERTa model is around 125M parameters while our AST-T5-Base is 220M parameters.
Table 4: Our AST-T5-Small results on the AST task using the ABCD test set.
Params
Models
B-Slot
Value
Action
124M
ABCD-RoBERTa
93.6%
67.2%
65.8%
60M
AST-T5-Small (Ours)
89.1%
89.2%
87.9%
Our AST-T5-Smin variant achieves state-of- sfate-of-the-art results on the AST task while using 50% less trainable parameters. Furthermore, text-to-text approach is casily adaptable to any domain without any model change. In contrast, an encoder-only approach like the one used by ABCD-RoBERTa requires update to the classification head to add new action or slot value.
6.2 Cascading Dialogue Success (CDS)
Following the same evaluation method used in Chen al. (2021), we compared our CDS-T5 models against the current otate-of-the-art the CDS task. In Table 5 report the best results (ABCD Ensemble) for each metric from Chen et al. (021).
Our CDS-T5-Small (60M parameters) outperforms the current stale-of-the-arl (with up to 345M parameters) while using 80% less trainable parameters. Furthermore, Our CDS-T5-Ban achieves a new state-of- the-trr on the CDS task while using 36% fewer trainable parameters, showing the advantage of our text-to-text approach. Specifically, our CDS-T5-Base outperformed the ABCD Ensemble on the next utterance prediction (recall@1) by 27.4 points. Furthermore, both models scored exceptionally predicting the next step.
10

---

Published in Transactions on Machine Learning Research (02/2023)
Table 5: CDS-T3 results on the CDS task using the ABCD test set. Intent, NextStep, B-Slot, Value, Re- call@1 are accuracy measures for the dialogue intent, next step, action, value, and next utterance predictions,
respectively. CE the cascading evaluation that uses all other metrics. ABCD Ensemble the ensemble with the best scores from ABCD.
Params
Models
Intent
Nextstep
B-Slot
Value
Recall@1
CE
345M
ABCD Ensemble
90.5%
87.8%
88.5%
73.3%
22.1
32.9%
60M
CDS-T3-Small (Ours)
85.7%
99.5%
85.9%
75.1%
40.7
38.3%
220M
CDS-T5-Base (Ours)
86.0%
99.6%
87.2%
77.3%
49.5
41.0%
Finally, CDS-15-Base outperforms the human performance points.
CCen al. 2021 on value accuracy by 1.8
6.3 Workflow Discovery (WD)
This section shows the experiments we performed to explore the WD task using several baseline models. First, we report the results in the following learning setups: in-domain, cross-domain zero-shot, cross-dataset zero-shot, and few-shot. Then, report the results of the experiments showing the performance improvement when using natural language action names and using models fine-tuned on summarization. It is worth noting that the idea behind using multiple model architectures and sizes is to show that our task formulation is model-independen and to understand the performance improvement as the model size scales in different learning setups.
6.3.1 In-Domain Workflow Actions
Table D shows the results of our methods trained on all ABCD domains and tested on the ABCD test set. We report each model variant's Cascading Evaluation (CE) and Exact Match (EM). Further, to understand the added complexity of predicting the slot values, we report the Action-only CE and Action-only EM results, where we are only interested the models' ability to predict the actions correctly regardless the predicted slot values.
Table 6: WD in-domain test results on ABCD. With and Without Possible Actions represents the results with and without the possible actions added to the input. EM and CE are the Exact Match and Cascading Evaluation, respectively. Action-only EM and CE are the EM and CE scores for Action-only prediction, excluding slot values.
EM/CE
Action-only EM/CE
Without
With
Without
With
Params
Models
Possible Actions
Possible Actions
Possible Actions
Possible Actions
60M
T5-Small
44.1/67.9
44.8/68.6
55.4/78.6
56.7/79.1
220M
T5-Base
47.7/69.0
49.5/72.3
56.2/799.4
57.5/79.2
406M
BART-Large
42.0/ 42.0/60.3 /6.0
44.9/64.3
61.9/68.5
64.3/70.1
568M
PEGASUS-
49.9 19.9//71.2
52.1/72.6
59.6/81.2
62.9/82.8 /2.8
770M
T5-Large
50.6/73.1
55.7 55.7/75.8
59.9/81.4
63.3/88 83.1
With and without adding the possible actions, the EM and CE scores show an expected improvement as we scale the model size except for the BART-Large variant. Our analysis showed that the performance gains scale the model size due to a better ability to extract slot values. The difficulty of predicting the slot values is because ABCD contains 30 unique actions with 231 associated slots, with each action having cither 0, 1, or 3 slots. Further, some actions use different slots depending on the customer's answer. For example, 'pull up customer account" can use either the account ID or the customer's full name. Our qualitative analysis showed that 37% the slot values failures arise when both the "pull up customer account" and "verify-identity" occur in the same dialogue. In such a situation, customer provides the account ID and the full name (present in 32.5% of test. dialogues), and puull up customer account! can either slot and "verify
11

---

Published in Transactions on Machine Learning Research (02/2023)
customer identity' uses both. Using the account ID or the full name for "pull up customer account" is valid from the ABCD's agent guidelines perspective. However, it might not match the gold truth. Further, larger models perform better on dialogues with turn counts larger than 20, which represents 38.2% of test dialogues. This improvement is related to large models' ability to handle longer context (Tay et al. 2021 Ainslic et al., 2020p. As for LaRg-Large variant, our qualitative analysis showed that it exhibits an interesting behavior where it performs poorly on extracting some slot values that represent either names, usernames, order IDs, or emails. For example, it predicts 'cryslm561' instead of "crystalm561", knowing that the former does not exist in the dataset. The fact that 70% of the test workflows contain slots of this type explains the drop in performance. This observation is also confirmed by the high Action-only EM score, where slot value predictions are ignored. Our analysis showed that BART-Large performs better on target workflow with at most two actions. However, we did not find a clear explanation for this behavior. Further, while T5-Large shows an improvement over the smallest model (i.e., T5-Small), the EM score is only slightly above 50%, which shows the difficulty of the WD task. Morcover, the significant difference between the EM and CE scores across all configurations shows that it is much harder to predict the actions and their slot values in the exact order, which shows the utility of CE as good metric for understanding the performance on the WD task.
Adding the possible actions improves the performance of all model variants, especially larger ones. For example, T5-Large improved by 5.1 on the FM score suggesting that the results should improve with even larger models (i.e., more than 770M parameters). However, the T5-Small variant improved by only 0.7 on the EM score. We believe that it is related to the fact that smaller models have difficulty utilizing larger inputs, known limitation of Transformer models (Tay et al., 2021 Ainslie et al.) 2020). Our qualitative analysis showed that most of the improvements gained by adding the possible actions value prediction, which can be counterintuitive. However, we believe that the models use more capacity to extract the slot values since providing the actions makes the action prediction an easier task. Further, adding the possible actions improved the action prediction performance on dialogues with turn counts larger than 25 (16.2% of test dialogues), which is shown by the higher Action-only EM and CE scores.
duc to better slot
To better understand the performance of our proposed baselines, we compared our results to human per- formance on 100 randomly selected test samples representing 10% of the ABCD test set. We hired two amnotators to label the samples and provided them with the complete list of actions with descriptions ex- tracted from the ABCD agent's guidelines. Moreover, we provided three fully annotated dialogues for each action from the ABCD training set. However, the Cohen kappa coefficient was 0.63, indicating a low agree- ment between the annotators. All the disagroed items were labeled by a third annotator Table 7 shows the EM and CE scores on the annotated subset.
Table 7: Human evaluation results on ABCD test subset. EM and CE are the Exact Match and Cascading Evaluation, respectively.
Methods
EM/CE 11.0/38.12 12
Human Performance
T5-Small Without Possible Actions
42.3/66.0
The results show that even our smallest model outperforms human performance by large margin. Our analysis showed that our method outperforms human annotators in cases where the agent performs multiple actions at the same turn. Further, our approach outperformed human performance in dialogues with more than 13 turns. However, since ABCD has 30 domains with 231 associated slots, further annotator training could be beneficial since the annotators reported difficulties when annotating the dialogues shown by the low Cohen kappa coefficient.
We also ran an ablation study to investigate the positive effect of shuffling and randomly selecting the actions added to the input described in Section 4.1 The result of this study is shown in Appendix A.I.I but in summary, this technique improves the performance of all model variants and avoids performance drop for our small model (i.e., Tmall).).
The annotated human subset available in our public code repository.
12

---

Published in Transactions on Machine Learning Research (02/2023)
6.3.2 ABCD Doms-D Zero-Shor
We performed a 'leave-onc-out" cross-domain zero-shot experiment similar to Lin et al. (2021b Hu et al. (022]; Zhao et al. (2022b) on ABCD. In this setup, we train a model on samples from all the domains except one. Then, we evaluate the performance on the test set that includes the omitted domain. In this experiment, we excluded the 'Shirt FAQ' and 'Promo Code' domains by removing all training samples that contain actions unique to each domain. The 'Shirt FAQ' domain consists of 34 dialogues representing 3.4% of the ABCD test set. It overlaps other domains, such as 'Boots FAQ' and "Jacket FAQ" on two actions (i.e., 'search fag" and 'select topic in faq'). However, while other domains use the selecct topic in fag' action, the 'Shirt FAQ' domain has eight unique slot values for this action since they represent existing FAQ questions that agents have to choose from (described in ABCD's agent guidelines). The "Promo Code' domain consists of 53 dialogues representing 5.3% of the ABCD test set. It overlaps other domains on three actions (i.c., "pull up cuslomer accourd", "ask oracle', and "check membership level'). However, the overlapped actions appear in an order unique to this domain, making it more difficult. since the order is important in the WD task. Table g shows the results of both experiments.
Table 8: WD leave-one-out cross-domain zero-shot results on ABCD with 'Shirt FAQ" and 'Promo Code" as target domains. With and Without Possible Actions represents the results with and without the possible actions added to the input. EM and CE are the Exact Match and Cascading Evaluation, respectively.
Promo Code
Shirt FAQ EM/CE
EM/CE
Without
With
Without
With
Params
Models
Possible Actions
Possible Actions
Possible Actions
Possible Actions
60M
T5-Small
42.3/65 1
42.5/66.5
41.0/64.8
41.0/65.0
220M
T5- 15-B
46.0/67.6
47.8/69.7
45.1/68.8
45.7/69.0
406M
BART-Large
41.5/58.3
43.6/62.2
40.0/59.1
42.3/61.7
568M
PEGASUS-Large
47.4/68.8
49.6/69
46.2/68.9
47.4/69.3
770M
T5-Large
48.1/70.7
51.8/72.3
49.9/72.4
50.2/72.4
The results show that our approach achieves good zero-shot transfer performance in both domains. Overall, the behavior of the different variants on both domains with and without the possible actions follows the same trends as the in-domain results in Table G. Similarly, BART-Large performs poorly on both domains, and our analysis showed that it suffers from the same issues described in Section 6.3.2 However, BART-Large has the least amount of performance drop compared to the in-domain results since the target domains do not contain the slots in which this model performs poorly, as described in Section 6.3.2 On average, the performance on the 'Shirt FAQ' is higher than the 'Promo Code" domain, where the latter has an EM score 3.5 points lower than the in-domain results, showing that the "Promo Gode" domain presents a more difficult cross-domain zero-shot setup.
Our results analysis showed that the "Promo Code' domain, all the failures relate to the "offer promo code" action. When the possible actions are not used, most of the failures when using T5- were caused by the model copying parts of the input. However, for larger models, 67% of the failures are caused by the models predicting "offer refund" instead of 'offer promo code". However, it is worth noting that dialogues from the "Offer Refund' domain unfold similarly to the "Promo Code" domain. In such situations, a customer is unhappy with a purchase, and the agent offers a refund, similar to offering a promo code. Further, the slot value (i.e., the promo code) is correctly predicted in 80% of these cases (i.e., when predicting 'offer refund"). We believe that this is caused by the fact that the slot value of offer refund' (..e. the refund amount) is close to the one of "offrr promo code'. However, this does not show in the results since both metrics require both the action and slot values to be correct. In the remaining, 33% of the failures, the larger models generated plausible action names such as "generale new code", 'creale new code", and "creale code" (See Appendix A.33 for more details plausible action names). These predictions were not deemed valid since their BERTscore was lower than the threshold. On the other hand, larger models predicted actions that are considered valid such as offer code', "offer the code", and "provide code'. When the possible actions are used, the failures due to the 'offer refund" confusion are reduced to 33%, showing the efficacy of adding the possible actions
13

---

Published in Transactions on Machine Learning Research (02/2023)
to the imput. The remaining errors were due to either invalid slot value prediction or invalid generated action. In this setup, 70% of valid predictions used the exact action name provided in the input, and the remaining were generated actions that are considered acceptable. On the 'Shirt FAQ" domain, When the possible actions are not used, 71% of the failures are due to confusion with similar actions such as 'search for jackets", or "search for jeans'. The remaining errors were due to bad slot. value prediction. Further, larger models generated plausible actions (e.g., "search t-shirt" or "information on shirts' and slot values (e.g., 'does this I-shirt shrink" instead of 'does this shirt shrink'). See Appendix AA.3] for more examples. When the possible actions are added to the imput, only 15% of the failures were due to confusion with similar domains (mostly 'Jacket FAQ'), and the remaining failures were due to invalid slot value predictions. In this setup, 83% of valid predictions used the exact action name provided in the input, and the remaining were generated actions that are considered acceptable.
We also performed an ablation study on the randomization technique and showed that it has a similar effect to the one seen in the in-domain setting. See Appendix A.2 for more details.
6.3.3 MultiWOZ Cross- Dataset Shro-Shot
In this experiment, we evaluate if model trained on one dataset can be used on another from different domain in a zero-shot setup similar to Zhao et al. (202b)). Therefore, we test our models trained on ABCD described in Section [6..22 on all MultiWOZ test set domains. In this setting, there is no inter-dataset overlap. All the eight MultiWOZ domains are different from the ones in ABCD, We report the results of this experiment in Table g
Table 9:
WD cross-dataset zero-shot test results on MultiWOZ. With and Without, Possible Actions repre- sents the results with and without the possible actions added to the imput. EM and CE are the Exact Match and Cascading Evaluation, respectivcly.
EM/CE
Params
Models
Without Possible Actions
With Possible Actions
60M
T5-Small
0.0/0.0
5.3/8.4
220M
T5- Base
3.6/10.0
23.0/ /3.0/38.
406M
BART-Large
5.1/12.2
24.1 / /39.9
568M
PEGASUS-Large
9.8/15.2
27.4/ 41.2
97.4/41.2
770M
T5-Large
9.0/13.1
06.9//4.0
As the results show, all model variants performed poorly when the list of actions was not included in the input. For example, T5-Small was not able to predict any valid workflow and our largest model achieved only a 9.0 EM score. For the T5-Small model, Our analysis showed that 91% of the failures are cases where the model uses ABCD's actions, and the remaining are caused by copying the input. For all other models, 87% of the failures are due to the models using ABCD's actions on average. The remaining failures are cases where the models copy part of the input invalid action slot value predictions. In this setting, all valid predictions were for cases with a single slot value, and the generated action names were considered valid using our evaluation method. For example, the models generated "searching for hotels", "Looking for hotel", and get a lan''. However, when adding the possible actions, the performance improved for all variants with an average EM score increase of 18.3 points, showing the utility of this technique in the zero-shot setting. In this setting, 65% of the valid predictions are exact matches
the one provided in the input, and the remaining are generated action names that are deemed valid. Our analysis showed that 80% of the failures are due to invalid slot value predictions, especially for actions with more than three slot values. We believe this is related to the fact that the source domains (i.e., ABCD) have at most three slot values per action. The remainder of the failures are due to using ABCD's action names. Further, PEGASUS-Large performs better than WD-T5-Large even if it has 200M fewer parameters. Our qualitative results ana malysss shows that WD-T5-Large uses actions from the source domains (i.e., ABCD) more often than WID-PEGASUS-Large For example, it uses search-timing' for cases where a customer asks about the train departure time. Morcover, BART-Large performs better in this setting since MultiWOZ does not contain usernames or email slot values. Furthermore, we noticed that some predicted slot, values more refined than the gold ones (e.g., museum
14

---

Published in Transactions on Machine Learning Research (02/2023)
of science instead of museum). However, our evaluation method does not mark such predictions valid. Further, while adding the possible actions improves the performance, it presents limitation for cases where this list is unknown and prompts future work to find an unsupervised method to extract it. The same limitation applics to the slot values, where one can solve the performance drop similar to Zhao et al.) (002bb) by providing the slot descriptions in the input. However, in some cases, this list is unknown or Trequently updated.
In both with and without possible actions, models larger than T5-Small were able to generate plausible action names. For example, the models generated the following: 'search for swimming pool', 'update booking information', 'search for a table", "make a reservation', "search for a museum', "search for guesthouse', and "search for a room'. While some of these predictions can be ambiguous (e.g., "make a reservation "), others are very plausible. For example, 'search for a room" or "search for guesthouse" are valid predictions for "search for hotel", and "search for a museum" is a valid prediction for 'search for attractions" since the museum is an attraction. While such predictions are not exact matches, they provide valuable domain knowledge, especially in zero-shot setting. However, our evaluation scheme cannol. mark these predictions as valid since they yield BERTscore lower than the threshold. While lowering the threshold could solve this issue, it will increase the false positives since the action names are short sentences. For example, 'search for hote" and buy hotel" has BERTScore of 89%, which shows one of the limitations of our proposed evaluation method.
Finally, is worth noting that in the zero-shot setting, when the list of actions is not provided, the models trained on WD perform a form of intent (or action) and slot induction. Here the models generate the action names and extract the matching slot values rather than just clustering utterances (Hu et al. 2022) and performing span extraction for slot values YYu et al.) 2022). However, one limitation of using WD as way to perform intent and slot induction is that using WD on multiple dialogues could yield action names that are semantically similar but use a different syntax (e.g., "searching for hotels' and "Looking for hotel"). One way of solving such an issue is to use a similarity measure to group the generated actions (or intent) and choosing one of them as the cluster label in a post-processing step. One could also feed the chosen action names to the model when performing the WD task, increasing performance as shown previously. However, we leave this exploration for future work.
6.3.4 MultiWOZ Cross Dataset Few-Shot
We consider the case where only few samples are available for each workflow action. To this end, we conducted a cross-dataset few-shot experiment, where we trained model on all ABCD domains, then fine- tuned it with randomly selected k samples per action from all MultiWOZ domains. We picked k = 1, k 5, and k = 10 that yielded a training set of 11, 55, and 106 samples, respectively. Due to the nature of workflows containing multiple actions, some of these actions might have more than k samples.
Table 10: WD cross-dataset few-shot test results on MultiWOZ. With and Without Possible Actions represents the results with and without the possible actions added to the input. EM and CE are the Exact Match and Cascading Evaluation, respectively.
EM/CE
k
# Samples
Without Possible Actions
With Possible Actions
1
11
5.9/54.8
8.2/58.7 61.7/84.4
5
55
10
106
43.2/73.0
72.2/89.1
We report the few-shot results of our WD-T5- Base in Table 10 Our approach shows a significant adaptation performance that keeps increasing as the number of training samples increases, reaching an EM score of 72.2. Moreover, adding the possible actions to the input shows an important performance gain in all the settings, where it improved the EM score by an average of 32.8 points. Our analysis showed that the results follow the same behavior on the cross-datasct zero-shot experiment with k = Then, all other values of h follow similar behaviors as in the ABCD in-domain experiment. Similar to the cross-dataset zero-shot experiment, the failures are mainly due to invalid slot values prediction. However, when & 10 the slot. values issues
15

---

Published in Transactions on Machine Learning Research (02/2023)
reduce considerably. Furthermore, the model generated 23 invalid actions as opposed to 5 when the list of actions is provided in the input. This behavior shows the utility of this conditioning mechanism and proves that it can restrict the model to the provided actions.
6.3.5 Using Natural Language Action Names
In ABCD, the action names are given in a label-like format (e.g., 'pull-up-accouut'). However, this format could have a negative effect on the WD performance when using pre-trained models that might not have seen such format during training since they are usually trained on well-written, well-structured text (Raffel et al. 2020a Karpukhin et al., 2020). To validate this claim, we manually generated natural language (NL) action names for each action. For example, we choose 'pull up customer account" as the NL action name for 'pull-up-accont' (See Appendix [..ee. for the full list). Then, we run an experiment similar to 663.3 to compare the performance of all the model variants using either the original label-like names (c.g., 'pull-up- account") or the NL action names (e.g., 'pull up customer account"). For this experiment, we only compare the Action-only EM and Action-only CE since we are interested in the performance of correctly generating the action names and not the slot values. Table II shows the results of this experiment.
Table 11: Results of using original label-like names compared to natural language action names. Action-only EM/CH are the EM and CE scores for action-only prediction, excluding slot values.
Action-only EM/CE
Params
Models
With Original Labels
With NL Names
60M
T5-Small
53.2/74.1
58.4/78.6
220M
T 15-B
52.8/74 /7.11
56.2 79.4
406M
BART-Large
59.2/66.3
61.9/68.4
568M
PEGASUS-
56.7/76.
59.6/81.2
770M
La-Ler
56.6/85.9
59.9/81.4
We observe that using NL names yields better performance on both metrics on all model variants, where it increased the Action-only EM by an average of 2.9. Our qualitative analysis showed that when using the original labels, most of the failures compared to using NL names are related to incorrectly generating the hyphen in the label. For example, the models generate 'pullup-accoum', or "pull-up account" instead of the target 'pull-up-account. Further, this behavior is more prominent for labels that consist of a single word (i.c, 'instructions' and 'membership') since they represent only 7% of the labels, where the models generate invalid labels like 'instructions or '-membership". Moreover, 60% of the errors made on these single-word labels are for the 'membership" label. We believe this is due to the existence of a similarly named label (i.e., "searsh-memmerrhiip".. Some invalid predictions include stop words (e.g., 'pull-up-the- account'). While extending training could resolve these issues, it will negatively affect domain adaptation due to overfitting. Also, while some of these invalid predictions can be handled in post-processing step, it will require additional effort to evaluate all the possible failures which can be a lengthy task, especially when adapting to a new domain. When using NL action names, the models still predict names that are not an exact match to the target. For example, the models predict 'pull up the customer account' instead of 'pull up customer account', or "ask customer try again" ' instead of 'ask customer to try again'. However, using NL names enables us to use techniques like stemming and ignoring stop words, avoiding the need to create claborate post-processing procedures. However, stemming and ignoring stop words were used most on 23% of the predictions (for T5-Small). The remaining improvements compared to using the original labels resulted from cases where the NL names brought semantic richness. For example, using NL action names, the models performed better on disferentiating "check membership level" from "gel memberships information' as opposed to using the original labels where the models confuse beaaers-mmmberhipp and "membership". However, manually generating the action names can be laborious task when dealing with a large or frequently updated domain, and we leave exploring more automated ways generate these names for future work.
16

---

Published in Transactions on Machine Learning Research (02/2023)
6.3.6 Using Models Fine-tuned Summarization
To understand the effect of using models fine-tuned on summarization, we performed an experiment following the same setup as in Section 6.3.1 without adding the possible action names to the imput, and we compare the performance using the same model with and without summarization fine-tuning. We report the results of this experiment in Table [2 on all model variants except PEGASUS-Lare since it does not have pre- trained model that has not been trained summarization tasks. Further, all models have been fine-tuned Mail (Hermann et al. 2015 dataset.
Table 12: Results of using original label-like names compared to natural language action names. With and Without Summarization Fine-tuning represents the results with and without the use models fine-tuned on summarization task. EM and CE are the Exact Match and Cascading Evaluation, respectively.
BM/CE
Without
With
Params
Models
Summarization Fine-tuning
Summarization Fine-tuning
60M
T5-Small
40.4/63.1
44.1/67.9
220M
T5-Base
44.2/65.9 65.9
47.7 /69.9
406M
BART-Large
42.0/60 /60.2
42.0/60 /60.3
770M
T5-Large
50.5/73.1 73.1
50.6/73.1
The results show that using models previously fine-tuned on summarization tasks improves the performance of models below 220 million parameters (i.e., T5-Small, and T5-Base). However, it did not have any effect on larger models' performance. Our result analysis showed that using fine-tuned models mostly improved the performance on slot value extraction for smaller models, especially for slot values with multiple words like full names (e-g., 'joseph banter'). Further, the same performance was attained for larger models with 4.8% fewer training iterations when using summarization checkpoints. These results show that using sunmarization fine-tuned models for the WD lask is beneficial.
7 Conclusion
We have formulated a new problem called Workflow Discovery, in which we aim to extract workflows from dialogues consisting of sequences of actions and slot values representing the steps taken throughout a di- alogue achieve a specific goal. We have examined models capable of performing three different tasks: Workflow Discovery, Action State Tracking, and Cascading Dialogue Success. Our experiments show that our sequeence-t--sequeecee approach can significantly outperform existing methods that use encoder-only lan- guage models for the AST and CDS tasks, achieving state-of-the-ae art results. Furthermore, our method is able to correctly predict both in-domain and out-of-distribution workflow actions, and our action condition- ing approach affords much better performance for out-of-domain few-shot and zero-shot learning. We hope that our work sparks further NLP research applied to the field of process mining, but allowing for the use logs, meta-data and dialogues as input. Moreover, we believe that this method of characterizing complex interactions between users and agents could have significant impact the way rescarchers and developers create automated agents, viewing workflow discovery a special type of summarization that produces form of natural language based program trace for task oriented dialogues.
References
Joshua Ainslic, Santiago Ontanon, Chris Alberti, Vaclav Cvicck, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Yang. ETC: Eucoding long and structured inputs in trans- formers. Empirical Methods in Natural Language Processing (EMNLP), November 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 2020.
17

---

Published in Transactions on Machine Learning Research (02/2023)
Tomáš Brychein and Pavel Král. Unsupervised dialogue act induction using gaussian mixtures. European Chapler of the Associalion for Compulational Linguistics (RACT.) 2016.
Pawel Budzianowski, Tsung- Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz-a large-scale multi-domain wizard-of-oz datasct for task-oriented dialogue modelling. Empirical Methods in Natural Language Processing (EMNLP), 2018.
Derek Chen, Howard Chen, Yi Yang, Alex Lin, and Zhou Yu. Action-based conversations datasct: A corpus for building more in-depth task-oriented dialogue systems. North American Chapler of the ACL (NAACL), 2021.
Jacob Devlin, Ming-Wia Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, North American Chapter of the ACL (NAACL), 2018.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. Association for Computational Lònguistics (ACL), 2020.
Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-sql evaluation methodology. Association for Computa- tional Lénguistics (ACL), 2018.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. Associalion for Compulational Linguislics (ACL), 2019.
Matthew Henderson, Blaise Thomson, and Jason D Williams. The second dialog state tracking challenge. Special interest group on discourse and dialogue (SIGDIAL), pp. 263-272, 2014.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. CoRR, has/1506.03344 2015.
Yushi Hu, Chia- Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. In-context learning for few-shot dialogue state tracking. arXiv, abs/2203.08522 088668 2022.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering. Empirical Methods in Natural Language Processing (EMNLP), 2020.
Joo-Kyung Kim and Young- Bum Kim. Joint learning of domain classification and out-of-domain detection with dynamic class weighting for satisficing false acceptance rates. International Speech Communicationn Association (INTERSPEECH), 2018.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Dialogue state tracking with a language model using schema-driven prompting. Empirical Melhods in Natural Language Processing (EMNLP)) 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising soquence-to-ssqquene pre-training for natural language generation, translation, and comprehension. Association for Computational Linguistics (ACL), 2019.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. Leveraging slot descript ions for zero-shot cross-domain dialogue StateTracking. North American Chapter of the ACL (NAACT.), 2021a.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A. Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. Leveraging slot descriptions for zcro-shot cross-domain dialogue statetracking. North American Chapler of the ACL (NAACL), 2021b.
Bing Liu and Ian Lane. Attention- recurrent neural network models for joint intent detection and slot filling. International Speech Communication Association (INTERSPEECH), 2016.
18

---

Published in Transactions on Machine Learning Research (02/2023)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXin, 2019.
Qingkai Min, Libo Qin, Zhiyang Teng, Xiao Liu, and Yue Zhang. Dialogue state induction using neural latent variable models. International Joint Conference on Artificial Intelligence (IJCAI), 2020.
Peiqing Niu, Zhongfu Chen, Meina Song, et al. A novel bi-directional interrelated model for joint intent detection and slot filling. Association for Computational Linguistics (ACL), 2019.
Hugh Perkins and Yi Yang. Dialog intent induction with deep multi-view clustering. Empirical Methods in Natural Language Processing (EMNLP), 2019.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. stock--ropngttion framework with token-level intent detection for spoken language understanding. Empirical Methods in Natural Language Processing (EMNLP), 2019.
Libo Qin, Wanxiang Che, Yangming Li, Mingheng Ni, and Ting Liu. Der-net: A deep co-interactive relation network for joint dialog act recognition and sentiment classification. Association for the Advancement of Arlificial Intelligence (AAAI), 2020.
Dimitrios Rafailidis and Yannis Manolopoulos. The technological gap between virtual assistants and recom- mendation systems. Xrv, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020a.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21:1 67, 2020b.
Seonghan Ryu, Sangiun Koo, Hwanjo Yu, and Gary Geunbae Lee. Out-of-domain detection based on generative adversarial network. Empirical Methods in Natural Language Processing (EMNLP), 2018.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. Empirical Methods in Natural Language Processing (EMNLP), 2021.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. Association for Computational Linguistics (ACL), 2000.
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. Executing instructions in situated collaborative interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-LJCNLP) 2019.
llya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence sequence learning with neural networks. Advances in neural information processing systems (NeurIPS), 27, 2014.
Yi Tay, Mostafa Dehghani, Samira Abuar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena A benchmark for efficient transformers. Inter- national Conference on Learning Representations (ICLR), 2021.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. Empirical Methods in Natural Language Processing (EMNLP), 2021.
19

---

Published in Transactions on Machine Learning Research (02/2023)
David HD Warren and Fernando CN Pereira. An efficient easily adaptable system for interpreting natural language queries. American journal of computalional linguistics, 8(3-4):110 10-122, 1982.
Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachan- dran. The dialog state tracking challenge series. Al Magazine, (5):):121 124, 2014.
Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. Building task orik-oriented dialogue systems for online shopping. Association for the Advancement of Artificial Intelligence (AAAI), 2017.
Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. Multiwoz 2.4: A multi-domain task-oriented dia-
logue dataset with essential annotation corrections to improve state tracking evaluation. Special interest group on discourse and dialogue (SIGDIAL), 2022.
Dian Yu, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent El Shafey, and Hagen Soltau. Unsupervised slot schema induction for task-oriented dialog. North American Chapler of the ACL (NAACL), 2022.
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. Cosal: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. Empirieal Methods in Natural Language Processing (EMNLP), 2019.
Yang Yu, Wei-Yang Qu, Nan Li, and Zimin Guo. Open-calegory classification by adversarial sample gener- ation. International Joint Conference on Artificial Intelligence (IJCAT), 2017.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupla, Jianguo Zhang, and Jindong Chen. Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. NLP4ConeAL, 2020.
Zengfeng Zeng, Dan Ma, Haiqin Yang, Zhen Gou, and Jianping Shen. Automatic intent-slot induction for dialogue systems. Proceedings of the Web Conference (WWW), 2021.
Luke S Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. Uncertainly in Artificial Intelligence (UAI), 2012.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S Yu. Joint slot filling and intent detection via
capsule neural networks. Association for Computational Linguistics (ACL), 2018.
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter Liu. Pogasus: Pre-training with extracted gap- sentences for abstractive summarization. International Conference on Machine Learning (ICML), 2020.
Tianyi Zhang*, Varsha Kishorc*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. International Conference on Learning Represenlations (ICLR), 2020.
Jelfrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. Description-driven task-oriented dialog modeling. arXiv, 2022a.
Jelfrey Zhao, Raghav Gupta, Yuanbin Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. Description-drieen task-oriented dialog modeling. arXiv, abs/2201.08904. 2022b.
A Appendices
A.1 Ablation on the Possible Actions Randomization
A.1.1
In-Domain Setting
We run an ablation study using all our model variants to understand the effect of the randomization technique described in Section 4.1 The use of randomization is motivated by the fact that the number of possible actions can be different when adapting a new domain or test time. The result of this study is shown
in Table 13
20

---

Published in Transactions on Machine Learning Research (02/2023)
Table 13: Randomization ablation results of in-domain WD on ABCD test set. w/ Possible Actions w/o randomization represents the results without randomization, and w/ Possible Actions the results with ran- EM and CE are the Exact the Match and Cascading Evaluation,
domization as described in Section
respectively
Model
EM/CE
WD-T5-Small
44.1/67.9
w/ Possible Actions w/o randomization
42.6/66.9
w/ Possible Actions
44.8/68.6
WD-T5 T5-Base
47.7/69.9
w/ Possible Actions w/o randomization
48.1/70.1
w/ Possible Actions
49.5/72.3
WD-BART-Larg.
42.0/60.3
w/ Possible Actions w/o randomization
43.2/61.0
w/ Possible Actions
44.9/64.3
W WD-PEGASUS-Larg
49.9/71.2
w/ Possible Actions w/o randomization
51.9/72.0
w/ Possible Actions
52.1/72.6
WD-T5-Large
50.6/73.1
w/ Possible Actions w/o randomization
54.6/74.8
w/ Possible Actions
55.7/75.8
The results show that adding the possible actions without randomization (i.e., w/ Possible Actions w/o ran- domization) improves the performance of all model variants except WD-T5-Small, where the EM score drops by 1.5 points. One explanation is that this model reaches its capacity since adding the possible actions lengthens the model input, which increases the complexity and reduces the performance, known limitation of Transformer models (Tay et al., 2021] Ainslie et al. 2020). Nonetheless, our randomization technique (i.e., w/ Possible Actions) resolves this issue and improves, even more, the performance of all other model variants, especially larger ones. For example, WD-T5-Large EM score increased by 5.1 points. The performance increase of WD-T5-Small is due to the fact that input length reduced at training time.
A.2 Cross-Domain Zero-Shot Setting
Similar to the in-domain experimental sctup, we performed an ablation study of the randomization technique described in Section in the cross-domain zero-shot setting. Table 14 shows the result of this study.
Randomization ablation results of cross-domain zero-shot WD ABCD test set with "Shirt FAQ' and "Promo Code" as target domains. w/ Possible Actions w/o randomization represents the results without randomization, and w/ Possible Actions the results with randomization as described in Section 4.1 EM and CE are the Exact Match and Cascading Evaluation metrics, respectively.
Table 14:
Shirt FAQ
Promo Code
Model
EM/CE
EM/CE
WD-T5-Small
42.3/65.1
41.0/64.8
w/ Possible Actions w/o randomization
41.2/64.4
40.9/64.5
w/ Possible Actions
42.5/66.5
41.0/65.0
WD-T5-Base
46.0/67.6
45.1/68.8
w/ Possible Actions
46.9/68.
45.3/68.8
w/ Possible Actions w/o randomization
47.8/69.7
45.7/69.0
Adding the possible actions (i.c., w/ Possible Actions) the input has a similar negative effect on the smaller model (i.e,, WD-T5-Small) as observed in the in-domain experiment results in Table and similarly, it increased the performance of the larger variant (i.e., WDD-T-Basse. However, randomized actions
21

---

Published in Transactions on Machine Learning Research (02/2023)
conditioning technique (i.e., w/ Possible Actions*) resolves this issue and improves performance both model variants.
A.3 Plausible Predictions in the Zero-Shot Setting
Task-Oriented Dialogue
Predicted Workflow
Hi. My name Alessandio Phoenix. My Order ID
Dialogue
Predicted Workflow
9602071 1593. The promo code fried these Michael Kors keeps scying it's Involid
aa litle upset checked on my order and found out that you have the same boots for $12 less
Generate new
Offer
This seems to be an enoo on our part. am going to generate you new promo code. just give me one moment.
Since seems you still upset with tris matter, what can do is offer a promo code that should ciffarence.
New Action
Know Action
(a)
(b)
Figure 3: Plausible (a) and known (b) workflow actions generated during the ABCD cross-domain zero-shot experiment.
Task Orientere Dialogue
Predicted Workflow
Could you also let me know the star rating the hotel, and also like to know they have wifi.
Check roting
The Aylesbray Lodge Guest House has four stars and free internet.
New Action
Figure 4: Plausible workflow action generated during the MultiWOZ cross dataset zero-shot experiment.
This section shows cases of plausible workflow actions predicted in the zero-shot setting that are not part of the possible actions. The generated actions are either "fine-grained" versions of existing actions or entirely new ones. For example, Figure shows case where the model predicted generate new code while the closet possible action is offer promo code. However, our analysis showed that there is a clear separation between cases in which agents offer promo code as shown in Figure 3b and when they generate new one the old promo code is no longer working, as shown in Figure 3a
Another example is shown in Figure 4 of entirely possible action. Here the customer is clearly asking for the rating of the hotel Hence the check rating action. Our dataset analysis showed that there are 10 Lest samples where customers requested to get the rating of hotels or restaurants.
action that has similarities with any other
A.4
WD Dataset
A.4.1
MultiWOZ Workflow Actions
Table 15 shows the MultiWOZ workflow natural language action names we used all our experiments.
A.4.2 ABCD Workflow Actions
Table 16 shows the ABCD workflow natural language actions names used in all our experiments.
22

---

Published in Transactions on Machine Learning Research (02/2023)
Table 15:
MultiWOZ workflow actions
Action Name
Natural Language Action Name
find hotel
search for hotel book hotel
book hotel
find_train
search for train book train ticket search for attractions search for restaurants book table at restaurant search for hospital book taxi search for taxi search for bus search for police station
book train
find attraccioon
jos_uuutaurant
book_restaurane
find hospital
book taxi
find taxi
find bus
find police
Table 16:
ABCD workflow actions
Action Name
Natural Language Action Name
pull-up-account
pull up customer account enter details
enter-details
verily-identity
verify customer identity
make pasks-p
create new password
search-timing
search timing
search-polies,
check policy
aalidate-purchase
validate purchase
search-fag
search faq check membership level search for boots ask customer to try again ask oracle update order information offer promo code
membership
search-bools
try-again
ssk-che-oracle
update-order
promo-code
update-account
update customer account get memberships information make purchase offer refund notify leam record reason search for jeans get shipping status search for shirt provide instructions scarch for jacket ask customer to log out log in select topic in faq get subscription status send link to eustomer check pricing
search-membership
make-purchase
offer-refund
notify-team
record-reason
search-jeans
shipping-statue
search-shirt
instructions
scarch-jacket
log-out-in
select-faq
subscription-status
send-link
search-pricing
23

---

Published in Transactions on Machine Learning Research (02/2023)
A.4.3 Dataset Statistics
This section shows the dataset set statistics for the WD task for ABCD and MultiWoz datasets. Table shows the size for each split, the average workflow length across each dataset, and the number of domains. Figure | and Figure 7 show the workflow length distribution for each data split on ABCD and MultiWoz, respectively. Figure 6 and Figure show the action distribution for cach data split on ABCD and MultiWoz, respectively.
Table 17:
WD dataset statistics for ABCD and MultiWoz datasets.
Average
# Train Samples
# Dev Samples
# Test Samples
# Domains
Workflow Length
ABCD
8034
1004
1004
18.5
30
MultiWoz
5048
527
544
3
8
Train
Dev
Test
2300
2210
2000
1300
174
2000
9101:121514
2324
113430
3101122334
154
Figure 5: Workflow length distribution for each ABCD dataset split.
24

---

Published in Transactions on Machine Learning Research (02/2023)
Train
5030
3030
2000
02
pullup account
valititte purchasa
enter det--ettle bits
hotify team
searchh
search-timug
salect-tac
shipping sttatus
update- arderr
verityy beaychh pricchpricng
update--cccous malate-accoune
tearch--boon Sharct--pputs
sharchh
instructions
try tryinn
bopocttin
affermafund
Settcch- jackett
tak-the oracto
mecord-resaoo
seircc--eans
Sibocciption Buloscigtian-ttttu
Gend-ini
Search parchhmenbbaarii
promo
Searrch ahire
mate make- parchase
Figure 6: Action distribution for each ABCD dataset split.
Train
Dev
Test
2054
1750
1500
1250
1000
500
258
250
174
183
51
5
1
5
2
Figure 7: Workflow length distribution for cach MultiWoz dataset split.
25

---

Published in Transactions on Machine Learning Research (02/2023)
Dev
1750
Test
1500
1250
1000
750
500
20
250
724
201
find. restaurant
book restaurant
find.
book hotel
fnd. train
find attraction
book train
find taxi
Figure Intent distribution for each MultiWoz dataset split.
26

---

Published in Transactions on Machine Learning Research (02/2023)
A.5 Prompt and target formats for WD, AST, and CDS tasks
PWD
'Hi there, am having trouble with promo code. keeps saying it is invalid" Have your account ID please? "Account ID is 123674" "okay. check system." "IIf seems that this is an error on our end. will generate you a new promo code," "Great thank you so much!" "Have great day' Actions: Offer refund, promo code, Get subscription status, Pull account, Ask oracle, notify team....
Dialogue:
T TWD
OM
Pull accouni [123674]; Ask oracle Generate new code [none]:...
(a) Workflow Discovery (WD). Illustration of prompt (PwD) consisting of an entire dialogue, and the target (TwD) consisting of a structured summary for the WD task. PWD is composed of the delimiters in bold that separate the dialogue utterances blue and the optional list possible actions green. Two composed of the predicted workflow actions, and matching slot values in brackets. The actions are either known (highlighted in green) or invented (highlighted orange).
PAST
Context 'Hi there, having trouble with my promo code, keeps saying it is invalid" "May Have your account ID please "Account is 123674"
TAST
Pull account [123674]
PAST
AST
Context: "Hi there. having trouble with my promo code, keeps saying it invalid" "May Have account ID please?" "Account ID is 123674" "Pull up eccount [123674]" ... 'It seems end. will generate you new
TAST Promo code [none]
that this is an error promo code."
(b) Action State Tracking (AST). Illustration of prompts (PAST) and targets (TAST) for the AST task at different turns in the dialogue. PAST is composed of the context delimiter in bold and the context utterances in blue that contain previous action turn text (underlined).
P CDD
Context: Hi there. am having trouble with my promo code, keeps saying it is invalid" Candidates: "hello. how may help you today?" "okay." " "May Have your account please?" "have nice day!"..
T
respond
CDS
Promo code invalid: respond; May Have your account ID please?
P PCDS
Taction
Context: Hi there. am having trouble with my promo code, keeps saying it invalid" "May Have your account ID plecse?""Account "Account ID 123674" Candidates: "have nice dayl" "Can you give me your usemame, please?".
CDS
Promo code invalid: action; Pull up account [123674]
P PCDS
Context: Hi there. am having trouble with my promo code, keeps saying involid" ... GGret thank you so much!" "Have a great day" Candidates: "sure!" "okay." "thank you so much," "welcome to acmebrands,, how may help?"
TCDS
Promo code invalid; end
"fhank"
(c) Cascading Dialogue Success (CDS). Illustration of prompts (Pens) and targets TeDS for the CDS task at different turns the dialogue. Peps composed the delimiters in bold that separate the dialogue utterances in blue and the list of possible agent response candidates in green. Treapond Taction and Tend are the targets when the next step is to respond with an utterance, perform an action, and end the conversation, respectively. 'Promo code invalid" is the entire dialogue intent.
Figure 9: Illustration of prompt and target formats for WD, AST, and CDS tasks.
27


Reviews:
[{"review_id": "h364JkInkP", "reviewer": "Reviewer_ST5G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n - First, the workflow task is an exciting direction that fits well with the spirit of recent NLP advances: more modularity in the task definitions. On a personal note, I think that it is an overstatement to call it workflow discovery, but it sounds nicer than sequential and conditional action tagging. I would encourage the authors to refine a bit the link between AST and WD to highlight better the core differences and the core similarities (e.g. a recap table). Overall, WD is more like a (welcome) generalization of AST.\n - Second, I appreciate that the authors assess their model on the AST tasks (which again demonstrate the close links between both methods) before testing them on the WD. It creates a proxy-baseline and gives trust in the next results.\n - Third, the paper flow is good. It is sometimes a bit wordy, but the reading is easy, and ideas are correctly articulated. It is easy to understand what the authors wanted to say. \n - The authors made some effort on the figures to ease intuitions, which is positive\n - The authors explore various WD settings with the inter/cross-domain and inter/cross dataset. As discussed below, those experiments may lack some in-depth analysis, but they go in the good directions\n\nWeaknesses:\n - One of the biggest weaknesses is the bibliography work. For instance, the introduction does not contain any citations! Differently, dialogue system is among the oldest field in ML... but they are only three papers that were cited that are written before 2015. I do not require a full survey of pre-deep learning papers, but this clearly hints that the related work section is incomplete. When introducing a new task, the quality of the bibliography is of paramount importance. Here, it is incomplete, ill-dispatched and it does not credit past work (e.g., no citation when introducing  AST or CDS). For instance, this could be a good start: https://web.stanford.edu/~jurafsky/ws97/CL-dialog.pdf\n - The second weakness is that the paper confuses the WF definition and the experimental implementation. Indeed, they describe the task by introducing a prompting mechanism, which is transformer-dependent. The WF task could also be solved with a retrieval model. Therefore, I strongly recommend the authors to clearly differentiate the task, i.e. set of actions, each dialogue may have different action options, and the WF implementation, using a prompting mechanism.\n - The authors listed the dataset used to define WF. However, I would be in trouble reproducing the setting (split, final workflows, etc.). I would recommend the authors release a file containing their experimental setting. In addition, the authors should also provide some statistical analysis on the WF setting (average sequence length, action distributions, inter-domain action overlaps, inter-dataset action overlap, etc.)\n - The paper sometimes mixes task definition and model-proposal, which I found quite confusing. Indeed, while it is interesting to test the proposed model to the AST task, there is no point of claiming Sota here. I would strongly recommend stating that your model is good enough to define a strong baseline for the WF task. Similarly, introducing randomization ablation blurs the message: the point of the paper is not to look for a better model to solve WF, but to define the research problem\n - Finally, while I like the diversity of WF tasks, I think they are not explored enough. Only results are provided, and little analysis is performed, e.g., what happens with incomplete actions set, and how often a new/incorrect action was outputted. How do inter-dataset scores correlate with action overlap between dataset? What is the impact of turning the actions into words instead of new trainable tokens when using the same dataset? They are many ways to challenge the WF tasks, but none were performed\n\n\nOverall, I would say that my core criticism of this paper is the lack of hindsight. While the authors introduce a task, there are too few references to the existing literature to rigorously frame their work in the community. Similarly, the author takes as granted the transformer prompt setting without abstracting the task by itself. Finally, the authors only compute final metrics and do not challenge their conclusion with additional experiments. While the paper has some potential, it needs more depth in the scientific background, task definition, and empirical analysis. However, the paper is good enough that a few rounds of iteration may solve these issues.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel variant of the action state tracking task that they named Workflow Discovery. \nIn short, the agent needs to output the workflow $T_{WD}$, i.e., a sequence of actions (with its labels), given a dialogue and a set of actions.\nHence, WD may be seen as an action state tracking task where the action tags are provided in the input (while AST actions are defined beforehand as an output). This task formulation allows for more flexibility -- few/zero-hot learning -- and may naturally leverage pretrained large language models.\n\nThe authors first attempt to frame the WD task within the literature and compare it with other dialogue tagging tasks (AST, DST).\nThen, they explain how to adapt the AST setting to the WD setting by defining a tokenization process for the input/output of a transformer.\nThen, they evaluate their WD models in an AST setting, assessing that the model output reasonable results.\nFinally, they assess diverse transformer architectures on different WD settings, e.g., in/across domain/dataset workflows, zero-shot, etc.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The core changes I would recommend are the following:\n - Redo the full related work section with more focused on the AST/Cascade setting, with its strenghes, weaknesses, past attempts etc. Have a broader view on similar settings, e.g., convert dialog into SQL. There are many facets to explore.\n - clean the BibTeX - the formatting is inconsistent!\n - Introduce an abstract definition of WD, and then describe the tokenization/prompt system in an experimental section\n - remove the sota claim; it is simply out of the scope here\n - Improve and complete the experimental section: fuse and rework the experimental sections, add back bad results, rework the table to ease reading (fully type with/without action set), provide some qualitative analysis, and further experiment analysis - especially on the cross-dataset evaluation (which is the most exciting part in my opinion)\n - the randomized conditioning techniques ablation should be done once for all in an ablation section and should never be mentioned again (no need for the + every time)\n\n(+ other see other remarks in previous paragraph)", "broader_impact_concerns": "Nan", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IwwieuiCmF", "reviewer": "Reviewer_dNS3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well structured. The new problem formulation looks interesting. However, whether this task is really useful needs to be explained better. Also, the used models are quite straightforward for the WD task.\n\nThe claim that workflow discovery is similar to summarization is not intuitive enough. How much summarization models give gains to the WD task is also not demonstrated.\n\nThe superiority between T5, BART, and PEGASUS is ambiguous. This is also the case with summarization datasets. However, can you provide why similar situations occur for these tasks? Also, considering this, can you provide a solution about which model practitioners should use for new datasets?\n\nI think the WD problem can be factorized into AST and CDS by dividing multiple turns. Do you have any thoughts on this? Also, there could be several variants of the problem formulation. Could you clarify why the proposed formulation is legit and most effective?\n\nFormats of inputs, outputs, and prefixes for each task are manually designed. I wonder how the results could change depending on different formats.\n\nThe baseline might need to be stronger to say that the proposed method is the-state-of-the-art.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper formulates a problem of workflow discovery (WD) from dialogues. WD could be closer to a realistic setting where all possible actions (or their sequences) are not fully defined beforehand. For the explanation, WD is compared with other tasks, action state tracking (AST) and cascading dialogue success (CDS). Based on the intuition that the WD problem is similar to summarization, text-to-text models (T5, BART, and PEGASUS) are used, fine-tuned to three tasks (WD, AST, and CDS), and evaluated on both an in-domain scenario (ABCD) and cross-domain scenario (MultiWOZ) based on a conditioning mechanism.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please include important baseline results and human accuracy to understand how models from the proposed method perform well relatively and absolutely.\n\nPlease add more literature on sequence-to-sequence models and few-shot/zero-shot approaches for AST and CDS.\n", "broader_impact_concerns": "The experiments are mainly conducted on the ABCD dataset (and MultiWOZ). Whether the proposed method could be extended to general or more realistic datasets are questionable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QEma4bpTtp", "reviewer": "Reviewer_TjxB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n  - Clearly explains the proposed task (WD) and its relation to similar tasks.\n  - Runs thorough experiments across a wide variety of model architectures, datasets, input setups and model sizes.  \n  - Strong state-of-the-art performance on the AST and CDS tasks for ABCD.\n\nWeaknesses\n  - Missing a comparison to intent and slot induction for dialogue.\n  - Authors mention that Workflow Discovery task includes an optional list of possible actions.  Operating with a known list of actions vs. not knowing this list is an important distinction.  The latter is a much more realistic setup that would be of much greater interest.  Accordingly, the difference in the two settings should be analyzed more closely.  \n  - Analysis of techniques and ablations are limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Introduce a task for Workflow Discovery (WD) that aims to extract the actions and associated slot-values from a dialogue. Authors evaluate their ability to complete this task on ABCD using various Seq2Seq models.\n\nAuthors also study cross-dataset workflow extraction in zero-shot and few-shot settings.  In particular, conditioning on the knowledge from ABCD transfers over to MultiWOZ\n\nFinally, authors offer SOTA results on AST and CDS, the two tasks proposed within the ABCD paper.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Authors should consider how workflow discovery is related to intent induction, which is a known task within dialogue settings.  Intent induction or slot-induction is more similar to WD than summarization, so a mention should certainly be added to Related Works and discussed in other areas.  Some examples include: \n  - https://aclanthology.org/E17-2078/\n  - https://aclanthology.org/D19-1413/\n  - https://arxiv.org/abs/2103.08886\n  - https://aclanthology.org/2022.naacl-main.86/\n\nSince the journal places less emphasis on novelty, the fact that the proposed WD task is similar to a known task (intent induction) is not a critical issue.  Mentioning the relationship somewhere is necessary, but should be straightforward enough to accomplish.  With that said, if it is possible to include an intent induction baseline for comparison to the text-to-text methods, that would significantly strengthen the work.\n\nClarify in Table 5 that EM*/CE* refer to actions only.  Perhaps in the description, but ideally somehow within the table.  Also, the table might benefit from being split, one to show different models (BART, Pegasus, T5) and one to show size (small, base, large).  Could use two columns, since there is enough space.\n\nMove some of the ablations into the main section and add more analysis around having the list of actions vs. no list of actions.  This is much more impactful than randomly shuffling the list of actions.  \nPossibly adding some line graphs, rather than just tables, since it might be easier to understand the trends.\n\nTypos\n - \"trained on another domain is a few-shot setting\" page 2\n - \"In the MutliWOZ dataset\" page 7", "broader_impact_concerns": "The models and topics do not touch on sensitive topics, so no issues with broader impact or ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers found the paper to be novel, interesting and well written and the empirical results to be quite convincing. 
There were initial concerns around comparison to prior work and the paper framing but the authors have comprehensively revised the paper to address these issues and the current version of the paper is publication worthy. 

Based on reviewer suggestions, I also recommend the Featured certification since this paper tackles the challenging task of dialog structure induction which has been a longstanding goal in dialog, plus has important applications in real-world settings. The paper is also well-written with comprehensive experiments and analyses, and is likely to substantially inform future work in this direction. As stated by one of the reviewers, "Workflow Discovery can be interesting to a broader audience since action-based dialogue is common in the real-world."
Decision: Accept (poster)