Title: ATTENTION OVER PARAMETERS FOR DIALOGUE SYSTEMS

Abstract: Dialogue systems require a great deal of different but complementary expertise to assist, inform, and entertain humans. For example, different domains (e.g., restaurant reservation, train ticket booking) of goal-oriented dialogue systems can be viewed as different skills, and so does ordinary chatting abilities of chitchat dialogue systems. In this paper, we propose to learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP). The experimental results show that this approach achieves competitive performance on a combined dataset of MultiWOZ (Budzianowski et al., 2018), In-Car Assistant (Eric et al., 2017), and Persona-Chat (Zhang et al., 2018). Finally, we demonstrate that each dialogue skill is effectively learned and can be combined with other skills to produce selective responses.

1. INTRODUCTION:

Unlike humans who can do both, goal-oriented dialogues (Williams & Young, 2007; Young et al., 2013) and chit-chat conversations (Serban et al., 2016a; Vinyals & Le, 2015) are often learned with separate models. A more desirable approach for the users would be to have a single chat interface that can handle both casual talk and tasks such as reservation or scheduling. This can be formulated as a problem of learning different conversational skills across multiple domains. A skill can be either querying a database, generating daily conversational utterances, or interacting with users in a particular task-domain (e.g. booking a restaurant). One challenge of having multiple skills is that existing datasets either focus only on chit-chat or on goal-oriented dialogues. This is due to the fact that traditional goal-oriented systems are modularized (Williams & Young, 2007; Hori et al., 2009; Lee et al., 2009; Levin et al., 2000; Young et al., 2013) ; thus, they cannot be jointly trained with end-to-end architecture as in chit-chat. However, recently proposed end-to-end trainable models (Eric & Manning, 2017; Madotto et al., 2018; Wu et al., 2019; Reddy et al., 2018; Yavuz et al., 2018) and datasets (Bordes & Weston, 2017; Eric et al., 2017) allow us to combine goal-oriented (Budzianowski et al., 2018; Eric et al., 2017) and chit-chat (Zhang et al., 2018) into a single benchmark dataset with multiple conversational skills as shown in Table 1 .

A straight forward solution would be to have a single model for all the conversational skills, which has shown to be effective to a certain extent by (Zhao et al., 2017) and (McCann et al., 2018) . Putting aside the performance in the tasks, such fixed shared-parameter framework, without any task-specific designs, would lose controllability and interpretability in the response generation. In this paper, instead, we propose to model multiple conversational skills using the Mixture of Experts (MoE) (Jacobs et al., 1991) paradigm, i.e., a model that learns and combine independent specialized experts using a gating function.

For instance, each expert could specialize in different dialogues domains (e.g., Hotel, Train, Chit-Chat etc.) and skills (e.g., generate SQL query). A popular implementation of MoE (Shazeer et al., 2017; Kaiser et al., 2017) uses a set of linear transformation (i.e., experts) in between two LSTM (Schmidhuber, 1987) layers. However, several problems arise with this implementation: 1) the model is computationally expensive as it has to decode multiple times each expert and make the combination at the representation-level; 2) no prior knowledge is injected in the expert selection (e.g., domains); 3) Seq2Seq model has limited ability in extracting information from a Knowledge Base (KB) (i.e., generated by the SQL query) (Eric et al., 2017) , as required in end-to-end task-oriented dialogues Table 1 : An example from the dataset which includes both chit-chat and task-oriented conversations. The model has to predict all the Sys turn, which includes SQL query and generating response from a the Memory content, which is dynamically updated with the queries results. The skills are the prior knowledge needed for the response, where Persona refers to chit-chat.

Spk. (Bordes & Weston, 2017) . The latter can be solved by using more advanced multi-hop models like the Transformer (Vaswani et al., 2017) , but the remaining two need to be addressed. Hence, in this paper we:

• propose a novel Transformer-based (Vaswani et al., 2017) architecture called Attention over Parameters (AoP). This model parameterize the conversational skills of end-to-end dialogue systems with independent decoder parameters (experts), and learns how to dynamically select and combine the appropriate decoder parameter sets by leveraging prior knowledge from the data such as domains and skill types;

• proof that AoP is algorithmically more efficient (Appendix A1) compared to forwarding all the Transformer decoders and then mix their output representation, like is normally done in MoE.

Figure 1 illustrates the high-level intuition of the difference;

• empirically show the effectiveness of using specialized parameters in a combined dataset of Multi-WOZ (Budzianowski et al., 2018) , In-Car Assistant (Eric et al., 2017) , and Persona-Chat (Zhang et al., 2018) , which to the best of our knowledge, is the first evaluation of this genre i.e. end-to-end large-scale multi-domains/skills. Moreover, we show that our model is highly interpretable and is able to combine different learned skills to produce compositional responses.

2. RELATED WORK:

Dialogue Task-oriented dialogue models (Gao et al., 2018) can be categorized in two types: modulebased (Williams & Young, 2007; Hori et al., 2009; Lee et al., 2009; Levin et al., 2000; Young et al., 2013) and end-to-end. In this paper, we focus on the latter which are systems that train a single model directly on text transcripts of dialogues. These tasks are tackled by selecting a set of predefined utterances (Bordes & Weston, 2017; Liu & Perez, 2017; Wu et al., 2017; 2018; Williams et al., 2017; Seo et al., 2017) or by generating a sequence of tokens (Madotto et al., 2018; Wen et al., 2017; Serban et al., 2016b; Zhao et al., 2017; Serban et al., 2017) . Especially in the latter, copy-augmented models (Eric & Manning, 2017; Madotto et al., 2018; Wu et al., 2019; Reddy et al., 2018; Yavuz et al., 2018) are very effective since extracting entities from a knowledge base is fundamental. On the other hand, end-to-end open domain chit-chat models have been widely studied (Serban et al., 2016a; Vinyals & Le, 2015; Wolf et al., 2019) . Several works improved on the initially reported baselines with various methodologies (Kulikov et al., 2018; Yavuz et al., 2018; Hancock et al., 2019; Lucas et al., 2009; Joshi et al., 2017; Zemlyanskiy & Sha, 2018; Dinan et al., 2019) . Finally, (Zhao et al., 2017) was the first attempt of having an end-to-end system for both task-oriented models and chit-chat. However, the dataset used for the evaluation was small, evaluated only in single domain, and the chit-chat ability was added manually through rules. 

Mixture of Expert & Conditional Computation:

The idea of having specialized parameters, or so-called experts, has been widely studied topics in the last two decades (Jacobs et al., 1991; Jordan & Jacobs, 1994) . For instance, different architecture and methodologies have been used such as SVM (Collobert et al., 2002) , Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015) , Dirichlet Processes (Shahbaba & Neal, 2009) , Hierarchical Experts (Yao et al., 2009) , Infinity Number of Experts (Rasmussen & Ghahramani, 2002) , and sequential expert addition (Aljundi et al., 2017) . More recently, the Mixture Of Expert (Shazeer et al., 2017; Kaiser et al., 2017) model was proposed which added a large number of experts between two LSTMs (Schmidhuber, 1987) . To the best of our knowledge, none of these previous works applied the results of the gating function to the parameters itself. On the other hand, there are Conditional Computational models which learn to dynamically select their computation graph (Bengio et al., 2013; Davis & Arel, 2013) .

Several methods have been used such as reinforcement learning (Bengio et al., 2016) , a halting function (Graves, 2016; Dehghani et al., 2019; Figurnov et al., 2017) , by pruning (Lin et al., 2017; He et al., 2018) and routing/controller function (Rosenbaum et al., 2018) . However, this line of work focuses more on optimizing the inference performance of the model more than specializing parts of it for computing a certain task.

Multi-task Learning Even though our model processes only input sequence and output sequences of text, it actually jointly learns multiple tasks (e.g. SQL and BOOK query, memory retrieval, and response generation), thus it is also related to multi-task learning (Caruana, 1997) . Interested readers may refer to (Ruder, 2017; Zhou et al., 2011) for a general overview on the topic. In Natural Language Processing, multi-task learning has been applied in a wide range of applications such as parsing (Collobert et al., 2011; Hashimoto et al., 2017; Ruder et al., 2017) , machine translation in multiple languages (Johnson et al., 2017) , and parsing image captioning and machine translation (Luong et al., 2016) . More interestingly, DecaNLP (McCann et al., 2018) has a large set of tasks that are cast to question answering (QA), and learned by a single model. In this work, we focus more on conversational data, but in future works, we plan to include these QA tasks.

3. METHODOLOGY:

We use the standard encoder-decoder architecture and avoid any task-specific designs (Madotto et al., 2018; Wu et al., 2019; Reddy et al., 2018) , as we aim to build a generic conversation model for both chit-chat and task-oriented dialogues. More specifically, we use a Transformer (Vaswani et al., 2017) for both encoder and decoder.

Let us define the sequence of tokens in the dialogue history as D = {d 1 , . . . , d m } and the dynamic memory content as a sequence of tokens M = {m 1 , . . . , m z }. The latter can be the result of a SQL query execution (e.g., table) or plain texts (e.g., persona description), depending on the task. The dialogue history D and the memory M are concatenated to obtain the final input denoted by X = [D; M ] = {x 1 , . . . , x n=m+z }. We then denote Y = {y 1 , . . . , y k } as the sequence of tokens that the model is expected to produce. Without loss of generality, Y can be both plain text and SQL-like queries. Hence, the model has to learn when to issue database queries and when to generate human-like responses. Finally, we define a binary skill vector V = {v 1 , . . . , v r } that specifies the type of skills required to generate Y . This can be considered as a prior vector for learning to select the correct expert during the training1 . For example, in Table 1 the first response is of type SQL in the Hotel domain, thus the skill vector V will have v SQL = 1 and v Hotel = 1, while all the other skill/domains are set to zero2 . More importantly, we may set the vector V to have multiple ones to enforce the model to compose skills to achieve a semantic compositionality of different experts.

3.1. ENCODER-DECODER:

To map the input sequence to the output sequence, we use a standard Transformer (Vaswani et al., 2017) and denote the encoder and decoder as TRS enc and TRS dec , respectively. The input of a Transformer is the embedded representation of the input words; thus, we define a word embedding matrix E ∈ R d×|V | where d is the embedding size and |V | is the cardinality of the vocabulary. The input X, with its positional embedding (Appendix A2 for more information), are encoded as the following equation:

EQUATION

where H ∈ R d model ×n , and E. Then the decoder receives the target sequence shifted by one Y :k-1 = {<SOS>, y 1 , . . . , y k } as the input. Using teacher-forcing (Williams & Zipser, 1989) , the model is trained to produce the correct sequence Y . The output of the decoder is produced as follow:

EQUATION

where O ∈ R d model ×k . Finally, a distribution over the vocabulary is generated for each token by an affine transformation W ∈ R d model ×|V | followed by a Softmax function.

EQUATION

In addition, P (Y |X) is mixed with the encoder-decoder attention distribution to enable to copy token from the input sequence as in (See et al., 2017) . The model is then trained to minimize a standard cross entropy loss function and at inference time to generate one token at the time in an auto-regressive manner (Graves, 2013) . Hence, the training loss is defined as:

EQUATION

3.2. ATTENTION OVER PARAMETERS:

The main idea is to produce a single set of parameters for decoder TRS dec by the weighted sum of r independently parameterized decoders. This process is similar to attention (Luong et al., 2015) where the memories are the parameters and the query is the encoded representation. Let us define Θ = [θ 1 , . . . , θ r ] as the list of parameters for r decoders, since a TRS dec is represented by its parameters θ. Since each θ can be sized in the order of millions, we assign the corresponding key vectors to each θ, similar to key-value memory networks (Miller et al., 2016) . Thus, we use a key matrix K ∈ R d model ×r and a Recurrent Neural Networks (RNN), in this instance a GRU (Cho et al., 2014) , to produce the query vector by processing the encoder output H. The attention weights for each decoders' parameters is computed as follow:

q = RNN(H) (5) α = Softmax(qK) (6)

where q ∈ R d model and α ∈ R r is the attention vectors where each α i is the score corresponding to θ i . Hence, the new set of parameters is computed as follow:

θ * = r i α i θ i (7)

The combined set of parameters θ * are then used to initialize a new TRS dec , and Equation 2will be applied to the input based on this. Equation 6 is similar to the gating function proposed in (Shazeer et al., 2017; Jacobs et al., 1991) , but the resulting scoring vector α is applied directly to the parameter instead of the output representation of each decoder. This indeed makes the computation algorithmically faster since the forward computation is done only once and summing r elements of size |θ| is linear compared to forwarding r multiplications with the input. Interested readers may refer to Appendix A1 for the proof. Importantly, if we apply α to each of the output representation O i generated by the TRS i dec , we end up having a Transformer-based implementation of MoE3 . We call this model as Attention over Representation (AoR).

Moreover, an additional loss term is used to supervise the attention vector α by using the prior knowledge vector V . Since multiple decoder parameters can be selected at the same time, we use a binary cross-entropy to train each α i . Thus a second loss is defined as:

EQUATION

The final loss is the summation of L P (Y |X) and L V .

Finally, in AoP, but in general in the MoE framework, stacking multiple layers (e.g., Transformer) leads to models with a large number of parameters, since multiple experts are repeated across layers. An elegant workaround is the Universal Transformer (Dehghani et al., 2019) , which loops over an unique layer and, as shown by (Dehghani et al., 2019) , holds similar or better performance than a multi-layer Transformer. In our experiment, we report a version of AoP that uses this architecture, which for instance does not add any further parameter to the model.

4. EXPERIMENTS AND RESULTS:



4.1. DATASET:

To evaluate the performance of our model for different conversational skills, we propose to combine three publicly available datasets: MultiWOZ (Budzianowski et al., 2018) , Stanford Multi-domain Dialogue (Eric et al., 2017) and Persona-Chat (Zhang et al., 2018) .

Figure 2 and two APIs interfaces: SQL and BOOK. The former is used to retrieve information about a certain domain and the latter is used to book restaurants, hotels, trains, and taxis. We refine this dataset to include SQL/BOOK queries and their outputs using the same annotations schema as (Bordes & Weston, 2017) . Hence, each response can either be plain text conversation with the user or SQL/BOOK queries, and the memory is dynamically populated with the results from the queries as the generated response is based on such information. This transformation allows us to train end-to-end models that learns how and when to produce SQL queries, to retrieve knowledge from a dynamic memory, and to produce plain text response. A detailed explanation is reported in Appendix A4, together with some samples.

Stanford Multi-domain Dialogue (SMD) is another human-to-human multi-domain goal-oriented dataset that is already designed for end-to-end training. There are three domains in this dataset (i.e., Point-of-Interest, Weather, Calendar) . The difference between this dataset and MWOZ is that each dialogue is associated with a set of records relevant to the dialogues. The memory is fixed in this case so the model does not need to issue any API calls. However, retrieving the correct entities from the memory is more challenging as the model has to compare different alternatives among records.

Persona-Chat is a multi-turn conversational dataset, in which two speakers are paired and different persona descriptions (4-5 sentences) are randomly assigned to each of them. For example, "I am an old man" and "I like to play football" are one of the possible persona descriptions provided to the system. Training models using this dataset results in a more persona consistent and fluent conversation compared to other existing datasets (Zhang et al., 2018) . Currently, this dataset has become one of the standard benchmarks for chit-chat systems, thus, we include it in our evaluation.

For all three datasets, we use the training/validation/test split provided by the authors and we keep all the real entities in input instead of using their delexicalized version as in (Budzianowski et al., 2018; Eric et al., 2017) . This makes the task more challenging, but at the same time more interesting since we force the model to produce real entities instead of generic and frequent placeholders. 

4.2. EVALUATION METRICS:

Goal-Oriented For both MWOZ and SMD, we follow the evaluation done by existing works (Eric & Manning, 2017; Zhao et al., 2017; Madotto et al., 2018; Wu et al., 2017) . We use BLEU4 score (Papineni et al., 2002) to measure the response fluency and Entity F1-Score (Wen et al., 2017; Zhao et al., 2017) to evaluates the ability of the model to generate relevant entities from the dynamic memory. Since MWOZ also includes SQL and BOOK queries, we compute the exact match accuracy (i.e., ACC SQL and ACC BOOK ) and BLEU score (i.e., BLEU SQL and BLEU BOOK ). Furthermore, we also report the F1-score for each domain in both MWOZ and SMD.

Chit-Chat We compare perplexity, BLEU score, F1-score (Dinan et al., 2019) , and Consistency score of the generate sentences with the human-generated prediction. The Consistency score is computed using a Natural Language Inference (NLI) model trained on dialogue NLI (Sean et al., 2018) , a recently proposed corpus based on Persona dataset. We fine-tune a pre-trained BERT model (Devlin et al., 2018) using the dialogue DNLI corpus and achieve a test set accuracy of 88.43%, which is similar to the best-reported model in (Sean et al., 2018) . The consistency score is defined as follow:

NLI(u, p j ) =    1 if u entails p j 0 if u is independent of p j -1 if u contradicts p j C(u) = m j NLI(u, p j ) (9)

where u is a generated utterance and p j is one sentence in the persona description. In Sean et al. (2018) , the authors showed that by re-ranking the beam search hypothesis using the DNLI score (i.e., C score), they achieved a substantial improvement in dialogue consistency. Intuitively, having a higher consistency C score means having a more persona consistent dialogue response.

4.3. BASELINES:

In our experiments, we compare Sequence-to-Sequence (Seq2Seq) (See et al., 2017) , Transformer (TRS) (Vaswani et al., 2017) , Mixture of Expert (MoE) (Shazeer et al., 2017) and Attention over Representation (AoR) with our proposed Attention over Parameters (AoP). In all the models, we used the same copy-mechanism as in (See et al., 2017) . In AoR instead of mixing the parameters as in Equation 7, we mix the output representation of each transformer decoder (i.e. Equation 2). For all AoP, AoR, and MoE, r = 13 is the number of decoders (experts): 2 skills of SQL and BOOK, 10 different domains for MWOZ+SMD, and 1 for Persona-Chat. Furthermore, we include also the following experiments: AoP that uses the gold attention vector V , which we refer as AoP w/ Oracle (or AoP + O); AoP trained by removing the L V from the optimization (AoP w/o L V ); and as aforementioned, the Universal Transformer for both AoP (AoP + U) and the standard Transformer (TRS + U) (i.e., 6 hops). All detailed model description and the full set of hyper-parameters used in the experiments are reported in Appendix A5.

4.4. RESULTS:

Table 2 and Table 3 show the respectively evaluation results in MWOZ+SMD and Persona-Chat datasets. From Table 2 , we can identify four patterns. 1) AoP and AoR perform consistently better then other baselines which shows the effectiveness of combining parameters by using the correct prior V ; 2) AoP performs consistently, but marginally, better than AoR, with the advantage of an algorithmic faster inference; 3) Using Oracle (AoP+O) gives the highest performance in all the measures, which shows the performance upper-bound for AoP. Hence, the performance gap when not using oracle attention is most likely due to the error in attention α (i.e., 2% error rate). Moreover, Table 2 shows that by removing L V (AoP w/o L V ) the model performance decreases, which confirms that good inductive bias is important for learning how to select and combine different parameters (experts). Additionally, in Appendix A6, we report the per-domain F1-Score for SQL, BOOK and sentences, and Table 3 and Table 2 with the standard deviation among the three runs. (Zhang et al., 2018) is lower since the vocabulary used in their experiments is smaller. In general, the difference in performance among different models is marginal except for the Consistency score; thus, we can conclude that all the models can learn this skill reasonably well. Consistently with the previous results, when L V is removed from the optimization, the models' performance decreases.

Finally, in both Table 2 and Table 3 , we report the results obtained by using the Universal Transformer, for both AoP and the Transformer. By adding the layer recursion, both models are able to consistently improve all the evaluated measures, in both Persona-Chat and the Task-Oriented tasks. Especially AoP, which achieves better performance than Oracle (i.e. single layer) in the SQL accuracy, and a consistently better performance in the Persona-Chat evaluation.

5. SKILL COMPOSITION:

To demonstrate the effectiveness of our model in learning independent skills and composing them together, we manually trigger skills by modifying α and generate 14 different responses for the same input dialogue context. This experiment allows us to verify whether the model accurately captures the meaning of each skill and whether it can properly learn to compose the selected parameters (skills). Table 3 first shows the dialogue history along with the response of AoP on the top, and then different responses generated by modifying α (i.e., black cells correspond to 1 in the vector, while the whites are 0). By analyzing Table 3 5 we can notice that: • The model learns the correct semantics of each skill. For instance, the AoP response is of type SQL and Train, and by deactivating the SQL skill and activating other domain-skills, including Train, we can see that the responses are grammatical and they are coherent with the selected skill semantics. For instance, by just selecting Train, the generated answer becomes "what time would you like to leave?" which is coherent with the dialogue context since such information has not been yet provided. Interestingly, when Persona skill is selected, the generated response is conversational and also coherent with the dialogue, even though it is less fluent.

• The model effectively learns how to compose multiple skills. For instance, when SQL or BOOK are triggered the response produces the correct SQL-syntax (e.g. "SELECT * FROM .." etc.). By also adding the corresponding domain-skill, the model generates the correct query format and attributes relative to the domain type (e.g. in SQL, Restaurant, the model queries with the relevant attribute food for restaurants).

6. CONCLUSION:

In this paper, we propose a novel way to train a single end-to-end dialogue model with multiple composable and interpretable skills. Unlike previous work, that mostly focused on the representationlevel mixing (Shazeer et al., 2017) , our proposed approach, Attention over Parameters, learns how to softly combine independent sets of specialized parameters (i.e., making SQL-Query, conversing with consistent persona, etc.) into a single set of parameters. By doing so, we not only achieve compositionality and interpretability but also gain algorithmically faster inference speed. To train and evaluate our model, we organize a multi-domain task-oriented datasets into end-to-end trainable formats and combine it with a conversational dataset (i.e. Persona-Chat). Our model learns to consider each task and domain as a separate skill that can be composed with each other, or used independently, and we verify the effectiveness of the interpretability and compositionality with competitive experimental results and thorough analysis.

A APPENDIX:

A.1 COMPUTATIONAL COST AOP Corollary A.0.1. The computation cost of Attention over Parameters (AoP) is always lower than Mixture Of Experts (MoE) as long as the processed sequence is longer than 1.

Proof. Let f θ : R d → R n a generic function parametrized by θ. Without loss of generality, we define θ as a affine transformation W ∈ R d×n . Let X ∈ R t×d a generic input sequence of length t and d dimensional size. Let the set F = [f θ1 , • • • , f θr ] be the set of r experts. Hence, the operation done by MoE are:

MoE(X) = f θ1 (X) + • • • + f θr (X) = XW 1 + • • • + XW r (10) Thus the computational cost in term of operation is O(rtdn + rtn) since the cost of f θi (X) is O(tdn)

and it is repeated r times, and the cost of summing the representation is O(rtn). On the other hand, the operation done by AoP are:

θ * = θ 1 + • • • + θ r = W 1 + • • • + W r (11) AoP(X) = f θ * (X) = XW * (12)

in this case the computational cost in term of operation is O((r + t)dn) since the cost of summing the parameters is O(rdn) and the cost of f θ * is O(tdn). Hence, it is easy to verify that if t > 1 then:

EQUATION

Furthermore, the assumption of using a simple affine transformation W is actually an optimal case. Indeed, assuming that the cost of parameters sum is equal to the number of operation is optimistic, for instance already by using attention the number of operations increases but the number of parameters remains constant. Since the model input may include structured data (e.g. DB records) we further define another embedding matrix for encoding the types and the segments as P ∈ R d×|S| where S is the set of positional tokens and |S| its cardinality. P is used to inform the model of the token types such as speaker information (e.g. Sys and Usr), the data-type for the memory content (e.g. Miles, Traffic etc.), and segment types like dialogue turn information and database record index (Wolf et al., 2019) . Figure 4 shows an example of the embedded representation of the input. Hence, we denote X T and X R as the type and segment tokens for each token in input X, respectively.

A.3 ATTENTION VISUALIZATION:

Figure 5 shows the attention vector α over parameters for different generated sentences. In this figure, and by analyzing more examples5 , we can identify two patterns:

• AoP learns to focus on the correct skills (i.e., SQL, BOOK) when API-calls are needed. From the first example in Figure 5 , we can see that the activations in α are consistent with those in the correct attention vector P . There are also false positives, in which AoP puts too high weights on BOOK when the correct response is plain text that should request more information from the user (i.e., i can help you with that. when would you like to leave the hotel?). However, we can notice that this example is, in fact, "almost correct" as triggering a booking API call may also be considered a valid response. Meanwhile, the third example also fails to attend to the correct skill, but, in fact, generates a very fluent and relevant response. This is most likely because the answer is simple and generic. • The attention often focuses on multiple skills not directly relevant to the task. We observe this pattern especially when there are other skill-related entities mentioned in the context or the response. For example, in the second dialog example in Figure 5 , we can notice that AoP not only accurately focuses on taxi domain, but also has non-negligible activations for restaurant and hotel. This is because the words "hotel" and "restaurant" are both mentioned in the dialogue context and the model has to produce two entities of the same type (i.e. finches bed and breakfast and ask).

A.4 DATA PRE-PROCESSING:

As mentioned in the main article, we convert MultiWOZ into an end-to-end trainable dataset. This requires to add sql-syntax queries when the system includes particular entities. To do so we leverage two annotations such as the state-tracker and the speech acts. The first is used to generate the a well-formed query, including key and attribute, the second instead to decide when to include the query. More details on the dialogue state-tracker slots and slots value, and the different speech acts can be found in (Budzianowski et al., 2018) .

A query is create by the slots, and its values, that has been updated in the latest turn. The SQL query uses the following syntax:

SELECT * FROM domain WHERE [slot_type = slot_value] *

Similarly for the booking api BOOK the syntax is the following:

BOOK FROM domain WHERE [slot_type = slot_value] *

In both cases the slot values are kept as real entities.

More challenging is to decide when to issue such apis. Speech acts are used to decide by using the "INFORM-DOMAIN" and "RECOMMEND-DOMAIN" tag. Thus any response that include those speech tag will trigger an api if and only if:

• there has been a change in the state-tracker from the previous turn • the produced query has never been issued before By a manual checking, this strategy results to be effective. However, as reported by (Budzianowski et al., 2018) the speech act annotation includes some noise, which is reflected also into our dataset.

The results from the SQL query can be of more that 1K records with multiple attributes. Following (Budzianowski et al., 2018) we use the following strategy:

• If no speech act INFORM or RECOMMEND and the number of records are more than 5, we use a special token in the memory < T M >.

• If no speech act INFORM or RECOMMEND and the number of records are less or equal than 5, we put all the records in memory. • If any speech act INFORM or RECOMMEND, we filter the records to include based on the act value. Notice that this is a fair strategy, since all the resulting record are correct possible answers and the annotators pick-up on of the record randomly (Budzianowski et al., 2018) .

Notice that the answer of a booking call instead, is only one record containing the booking information (e.g. reference number, taxi plate etc.) or "Not Available" token in case the booking cannot made.

A.5 HYPER-PARAMETERS AND TRAINING:

We used a standard Transformer architecture (Vaswani et al., 2017) with pre-trained Glove embedding (Pennington et al., 2014) . For the both Seq2Seq and MoE we use Adam (Kingma & Ba, 2014) optimizer with a learning rate of 1 × 10 -3 , where instead for the Transformer we used a warm-up learning rate strategy as in (Vaswani et al., 2017) . In both AoP and AoR we use an additional transformer layer on top the output of the model. Figure 6 ,7,8 shows the high level design MoE, AoR and AoP respectively. In all the model we used a batch size of 16, and we early stopped the model using the Validation set. All the experiments has been conducted using a single Nvidia 1080ti.

We used a small grid-search for tuning each model. The selected hyper-parameters are reported in 

:

the vector V will be absent during the testing

With the assumption that at each index in V is assigned a semantic skill (e.g. SQL position i)

but with the additional prior over the expert

Using the multi-bleu.perl script

Available in supplementary material and later online.

Figures and Tables:



Figure 1: Comparisons between Single model, Mixture of Experts (MoE)(Jacobs et al., 1991), and Attention over Parameters (AoP).

Figure 3: Results for the Persona-Chat dataset. Model Ppl. F1 C BLEU Seq2Seq 39.42 6.33 0.11 2.79 TRS 43.12 7.00 0.07 2.56 MoE 38.63 7.33 0.19 2.92 AoR 40.18 6.66 0.12 2.69 AoP 39.14 7.00 0.21 3.06 TRS + U 43.04 7.33 0.15 2.66 AoP + U 37.40 7.00 0.29 3.22 AoP w/o L V 42.81 6.66 0.12 2.85 AoP + O 40.16 7.33 0.21 2.91

Figure 4: Positional Embedding of the dialogue history and the memory content.

Figure Attention over Parameters visualization, vector α for different reference (Ref.) and AoP generated answers. Top rows (Usr) are the last utterances from each dialogue contexts.

Figure 7: Attention over Representation (AoR) consist of a transformer encoder which encode the source input and compute the attention over the skills. Then r transformer decoder layers computes r specialized representation and the output response is generated based on the weighted sum the representation. In the figure, we omitted the output layer.

Figure 8: Attention over Parameters (AoP) consist of a transformer encoder which encode the source input and compute the attention over the skills. Then, r specialized transformer decoder layers and a dummy transformer decoder layer parameterized by the weighted sum of the r specialized transformer decoder layers parameters. In the figure, we omitted the output layer.

I am good thank you! Do you like to ski? I really like it. Sys: No, I like to code in Haskell. Do you like to code? PERSONA systems

Table content: <table><tr><td/><td>Conversation</td><td>Skills</td></tr><tr><td colspan="2">Usr: Can you help me find a cheap 2 star hotel?</td><td/></tr><tr><td>Sys:</td><td>SELECT * FROM hotel WHERE pricerange='cheap' AND stars=2 AND type='hotel'</td><td>SQL+HOTEL</td></tr><tr><td colspan="2">Mem: (Result table from the SQL query, with real entities.)</td><td/></tr><tr><td colspan="2">Sys: Arms is a 2 star hotel in the centre. Shall I book a room for you?</td><td>HOTEL</td></tr><tr><td colspan="2">Usr: Yes, thank you!</td><td/></tr><tr><td colspan="2">Sys: Sure, for how many people should I book it for?</td><td>HOTEL</td></tr><tr><td/><td>• • •</td><td/></tr><tr><td colspan="2">Sys: Hello! How are you today?</td><td>PERSONA</td></tr><tr><td>Usr:</td><td/><td/></tr></table>

Results for the goal-oriented responses in both MWOZ and SMD. Last raw, and italicized, are the Oracle results, and bold-faced are best in each setting (w and w/o Universal). Results are averaged among three run (full table in Appendix A6). Model F1 BLEU SQL Acc SQL BLEU BOOK Acc BOOK BLEU

Table content: <table><tr><td>Seq2Seq 38.37</td><td>9.42</td><td>49.97</td><td>81.75</td><td>39.05</td><td>79.00</td></tr><tr><td>TRS 36.91</td><td>9.92</td><td>61.96</td><td>89.08</td><td>46.51</td><td>78.41</td></tr><tr><td>MoE 38.64</td><td>9.47</td><td>53.60</td><td>85.38</td><td>37.23</td><td>78.55</td></tr><tr><td colspan="2">AoR 40.36 10.66</td><td>69.39</td><td>90.64</td><td>52.15</td><td>81.15</td></tr><tr><td colspan="2">AoP 42.26 11.14</td><td>71.1</td><td>90.90</td><td>56.31</td><td>84.08</td></tr><tr><td>TRS + U 39.39</td><td>9.29</td><td>61.80</td><td>89.70</td><td>50.16</td><td>79.05</td></tr><tr><td colspan="2">AoP + U 44.04 11.26</td><td>74.83</td><td>91.90</td><td>56.37</td><td>84.15</td></tr><tr><td colspan="2">AoP w/o L V 38.50 10.50</td><td>61.47</td><td>88.28</td><td>52.61</td><td>80.34</td></tr><tr><td colspan="2">AoP+O 46.36 11.99</td><td>73.41</td><td>93.81</td><td>56.18</td><td>86.42</td></tr></table>

Table 2 summarizes the dataset statistics in terms of number of dialogues, turns, and unique tokens. Finally, we merge the three datasets obtaining 154,768/19,713/19,528 for training, validation and, test respectively, and a vocabulary size of 37,069 unique tokens.

Table content: <table/>

Selecting different skills thought the attention vector α results in a skill-consistent response. AoP response activates SQL and Train. There are lots of trains to choose from! Where are you departing from?Usr: I am departing from london heading to cambridge.AoP responses using different parameter combination I would be happy to book that for you.Ihave found you a few to choose from. what time would .. I have many options for you. is there a certain time you ... Is there a certain area you would like? what time would you like to leave? okay I have two trains what time would you like to cambridge? Where do you do for work ? BOOK FROM taxi WHERE leaveAt>"1530" AND destination=.. BOOK FROM restaurant WHERE time="1530" AND ... BOOK FROM hotel WHERE people="1" AND day="monday" BOOK FROM train WHERE people="1" AND id_booking=' SELECT * FROM restaurant WHERE food="1530" ... SELECT * FROM hotel WHERE type="1530" SELECT * FROM attraction WHERE name="departure"

Table content: <table><tr><td colspan="8">Sys: What time will you be travelling?</td><td/></tr><tr><td colspan="7">Usr: I need to arrive by 1530.</td><td/><td/></tr><tr><td colspan="9">AoP: SELECT * FROM train WHERE destination="cambridge" AND</td></tr><tr><td/><td colspan="8">day="monday" AND arriveBy &lt; "1530" and departure="london"</td></tr><tr><td>SQL</td><td>BOOK</td><td>Taxi</td><td>Restaurant</td><td>Hotel</td><td>Attraction</td><td>Train</td><td>Schedule</td><td>Persona</td></tr></table>

Thanks. I will also need a taxi from the hotel to the restaurant. Will you handle this? AoP: SELECT * FROM hotel WHERE parking='yes' AND pricerange='expensive' AND stars='3'Ref.: SELECT * FROM hotel WHERE parking='yes' AND pricerange='expensive' AND stars='3'Usr: I would like to stay in an expensive 3 star place with free parking. Is there anything like that?

Table content: <table><tr><td>Usr:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Ref.: I can help you with that . when would you like to leave the hotel?</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>AoP: BOOK FROM taxi WHERE destination='finches bed and breakfast' AND departure='ask'</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Usr: Thank you for your help!</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Ref.: Thank you and enjoy your stay!</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>AoP: You are welcome. Is there anything else I can help you with?</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>SQL</td><td>BOOK</td><td>taxi</td><td>police</td><td>restaurant</td><td>hospital</td><td>hotel</td><td>attraction</td><td>train</td><td>navigate</td><td>weather</td><td>schedule</td><td>persona</td></tr></table>

and we run each experiment 3 times and report the mean and standard deviation of each result.

Table content: <table><tr><td>Model</td><td>d</td><td colspan="7">d model Layers Head Depth Filter GloVe Experts</td></tr><tr><td colspan="2">Seq2Seq 100</td><td>100</td><td>1</td><td>-</td><td>-</td><td>-</td><td>Yes</td><td>-</td></tr><tr><td/><td>TRS 300</td><td>300</td><td>1</td><td>2</td><td>40</td><td>50</td><td>Yes</td><td>-</td></tr><tr><td/><td>MoE 100</td><td>100</td><td>2</td><td>-</td><td>-</td><td>-</td><td>Yes</td><td>13</td></tr><tr><td colspan="2">AoP,AoR 300</td><td>300</td><td>1</td><td>2</td><td>40</td><td>50</td><td>Yes</td><td>13</td></tr><tr><td colspan="2">TRS/AoP+U 300</td><td>300</td><td>6</td><td>2</td><td>40</td><td>50</td><td>Yes</td><td>13</td></tr></table>

Hyper-Parameters used for the evaluations. A.6 MWOZ AND SMD WITH STD. Seq2Seq 38.37 ± 1.69 9.42 ± 0.38 49.97 ± 3.49 81.75 ± 2.54 39.05 ± 9.52 79.00 ± 3.63 TRS 36.91 ± 1.24 9.92 ± 0.43 61.96 ± 3.95 89.08 ± 1.23 46.51 ± 5.46 78.41 ± 2.03 MoE 38.64 ± 1.11 9.47 ± 0.59 53.60 ± 4.62 85.38 ± 2.68 37.23 ± 3.89 78.55 ± 2.62 AoR 40.36 ± 1.39 10.66 ± 0.34 69.39 ± 1.05 90.64 ± 0.83 52.15 ± 2.22 81.15 ± 0.32 AoP 42.26 ± 2.39 11.14 ± 0.39 71.1 ± 0.47 90.90 ± 0.81 56.31 ± 0.46 84.08 ± 0.99 TRS + U 39.39 ± 1.23 9.29 ± 0.71 61.80± 4.82 89.70 ± 1.40 50.16±1.18 79.05± 1.42 AoP + U 44.04 ± 0.92 11.26 ± 0.07 74.83 ± 0.79 91.90 ± 1.03 56.37 ± 0.92 84.15 ± 0.32 AoP w/o L V 38.50 ± 1.15 10.50 ± 0.55 61.47 ± 0.15 88.28 ± 0.50 52.61 ± 0.56 80.34 ± 0.21 AoP+O 46.36 ± 0.92 11.99 ± 0.03 73.41 ± 0.59 93.81 ± 0.16 56.18± 1.55 86.42 ± 0.92 A.7 PERSONA RESULT WITH STD Seq2Seq 39.42 ± 1.54 6.33 ± 0.58 0.11 ± 0.06 2.80 ± 0.09 TRS 43.12 ± 1.46 7.00 ± 0.00 0.07 ± 0.16 2.56 ± 0.07 MoE 38.63 ± 0.20 7.33 ± 0.05 0.19 ± 0.16 2.92 ± 0.48 AoR 40.18 ± 0.74 6.66 ± 0.05 0.12 ± 0.14 2.69 ± 0.34 AoP 39.14 ± 0.48 7.00 ± 0.00 0.21 ± 0.05 3.06 ± 0.08 TRS + U 43.04± 1.78 7.33± 0.57 0.15± 0.02 2.66±0.43 AoP + O 40.16 ± 0.56 7.33 ± 0.58 0.21 ± 0.14 2.98 ± 0.05 A.8 DOMAIN F1-SCORE

Table content: <table><tr><td>Model</td><td>F1</td><td>BLEU</td><td>SQL Acc</td><td>SQL BLEU</td><td colspan="2">BOOK Acc</td><td>BOOK BLEU</td></tr><tr><td/><td>Model</td><td>Ppl.</td><td>F1</td><td>C</td><td/><td>BLEU</td></tr><tr><td/><td>AoP + U</td><td>37.40±0.08</td><td>7.00±0.00</td><td colspan="4">0.29± 0.07 3.22± 0.04</td></tr><tr><td/><td>AoP w/o L V</td><td>42.81±0.01</td><td>6.66±0.57</td><td colspan="4">0.12± 0.04 2.85± 0.21</td></tr><tr><td/><td>Sentence</td><td colspan="2">Seq2Seq MoE TRS</td><td>AoR</td><td colspan="2">AoP Aop+O</td></tr><tr><td/><td>Taxi</td><td>71.77</td><td colspan="3">75.97 73.92 76.07 76.58</td><td>78.30</td></tr><tr><td/><td>Police</td><td>49.73</td><td colspan="3">49.95 50.24 51.95 56.61</td><td>52.05</td></tr><tr><td/><td>Restaurant</td><td>50.20</td><td colspan="3">49.59 48.34 50.58 50.47</td><td>50.90</td></tr><tr><td/><td>Hotel</td><td>46.82</td><td colspan="3">45.37 43.38 45.51 46.40</td><td>44.47</td></tr><tr><td/><td>Attraction</td><td>37.87</td><td colspan="3">35.21 33.10 36.97 38.79</td><td>37.51</td></tr><tr><td/><td>Train</td><td>46.02</td><td colspan="3">41.72 41.28 44.33 46.32</td><td>45.93</td></tr><tr><td/><td>Weather</td><td>40.38</td><td colspan="3">27.06 18.97 44.77 51.94</td><td>55.23</td></tr><tr><td/><td>Schedule</td><td>35.98</td><td colspan="3">43.94 38.95 32.90 54.18</td><td>52.99</td></tr><tr><td/><td>Navigate</td><td>18.57</td><td colspan="3">21.34 6.96 12.69 12.18</td><td>16.56</td></tr><tr><td/><td/><td/><td>BOOK</td><td/><td/><td/></tr><tr><td/><td>Taxi</td><td>23.16</td><td colspan="3">32.28 30.70 41.93 46.66</td><td>43.15</td></tr><tr><td/><td>Restaurant</td><td>45.02</td><td colspan="3">28.26 49.72 55.70 58.51</td><td>57.70</td></tr><tr><td/><td>Hotel</td><td>49.22</td><td colspan="3">31.48 41.61 51.46 56.62</td><td>57.41</td></tr><tr><td/><td>Train</td><td>55.86</td><td colspan="3">56.38 57.51 57.51 59.15</td><td>60.80</td></tr><tr><td/><td/><td/><td>SQL</td><td/><td/><td/></tr><tr><td/><td>Police</td><td>81.33</td><td colspan="3">0.00 90.66 76.00 93.33</td><td>100.0</td></tr><tr><td/><td>Restaurant</td><td>71.58</td><td colspan="3">68.00 75.90 81.27 80.43</td><td>84.15</td></tr><tr><td/><td>Hospital</td><td>62.22</td><td colspan="3">15.55 58.89 71.11 76.67</td><td>83.33</td></tr><tr><td/><td>Hotel</td><td>45.25</td><td colspan="3">42.09 48.61 56.69 59.75</td><td>63.75</td></tr><tr><td/><td>Attraction</td><td>65.48</td><td colspan="3">67.69 65.91 70.61 76.22</td><td>74.93</td></tr><tr><td/><td>Train</td><td>30.02</td><td colspan="3">41.01 55.67 66.61 67.34</td><td>69.50</td></tr></table>

Per Domain F1 Score.

Table content: <table/>

Peer review (iclr2020):
1. Summary: The authors of this paper proposed an attention over parameters based model for dialogue response generation. They assume that different tasks or topics require different models to generate responses. To model this idea, they design a model that can generate an attention vector based on historical dialogue history. This attention vector can then be used to get a combination of parameters that focus on the right topic/task to generate the next response. 2. Overall assessment: This idea looks very interesting to me. It*s simple but effective. However, this paper can still be improved in several aspects and it is not qualified yet to be published in a very competitive conference such as ICLR. I hope the authors can keep improving this paper and makes it a stronger publication. 3. Comments: 3.1 It seems not necessary to model the whole dialogue history. The next response usually mostly depends on the most recent user queries. Meanwhile, with a long sequence of words to encode, the model may not be very efficient. How well does the model work if only a subset of the dialogue history, such as the most recent one or two rounds, is modeled? 3.2 Before formula (2), , is it or ? 3.3 How do the authors combine the three training datasets? This is very important information for other researchers to replicate this work. 3.4 How to get the prior knowledge vector? It is given in the training data? What if we need to work on a dataset without such information? Will the model still work well? What if we remove the supervision over the attention vector in formula (8)? I suggest the authors conduct some experiments and analysis to answer these questions. 3.5 From an engineering angle, how efficient is it to derive different decoder parameters for different instances and do the inference? As we know, having constant parameters at inference time enables us to do a lot of optimization and parallelizations. How is the efficiency of the proposed model compared with existing ones?