Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems
Year: 2023

Abstract:
Recently, self-learning methods based on user satisfaction metrics and contextual bandits have shown promising results to enable consistent improvements in conversational AI systems. However, directly targeting such metrics by off-policy bandit learning objectives often increases the risk of making abrupt policy changes that break the current user experience. In this study, we introduce a scalable framework for supporting fine-grained exploration targets for individual domains via user-defined constraints. For example, we may want to ensure fewer policy deviations in business-critical domains such as shopping, while allocating more exploration budget to domains such as music. We present a novel meta-gradient learning approach that is scalable and practical to address this problem. The proposed method adjusts constraint violation penalty terms adaptively through a meta objective that encourages balanced constraint satisfaction across domains. We conducted extensive experiments on a realworld conversational AI and using a set of realistic constraint benchmarks. The proposed approach has been deployed in production for a large-scale commercial assistant, enabling the best balance between the policy value and constraint satisfaction rate.

heading: 1 Introduction
text: Conversational AI systems such as Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana rely on multiple processing components for speech recognition, natural language understanding (NLU), taking proper actions, and generating a response to the user. In such a system, a skill routing block is responsible for selecting the right skill and NLU interpretation to serve the request. Skill routing is a challenging problem as thousands of skills are present in real-world conversational systems and new skills are being introduced every day. In such scenario, gathering human annotations is very expensive and suffers from high turn-around
times. Moreover, often more than one skill is capable of serving a request which makes human supervision even more challenging due to the lack of clear ground-truth assignments (Sarikaya, 2017).
Recently, self-learning methods have been proposed that leverage customer experience signals to define reward values and create a closed feedback loop (Karampatziakis et al., 2019). In contrast to more traditional methods that are based on replication of rule-based systems or defect relabeling (Park et al., 2020), self-learning methods continuously explore different routing alternatives and leverage user feedback to improve their decisions (Kachuee et al., 2022).
Despite their scalability and efficiency, because self-learning approaches directly optimize routing decisions to achieve highest rewards, they suffer from instability issues impacting the user experience. Specifically, off-policy contextual bandits frequently used as the policy learning algorithm are susceptible to off-policy optimization errors, resulting in potentially breaking the current user experience due to overestimation of action values or excessive explorations (Swaminathan et al., 2016; Joachims et al., 2018; Lopez et al., 2021). Such instabilities and drastic changes in the agent’s behavior not only regress user retention and trust, but also manifest as direct revenue loss for businesscritical domains such as shopping.
In a production system, it is crucial to not only estimate but also control the changes of behavior a new policy introduces when compared to the current production policy. In the literature, this problem has been studied under safe bandit updates (Jagerman et al., 2020; Daulton et al., 2019; Amani et al., 2019) and budgeted bandit learning (Hoffman et al., 2014; Guha and Munagala, 2007), usually targeting exploration budgets or encouraging a behavior resembling a baseline policy.
In the context of off-policy bandit updates, we define exploration as any change in the model be-
43
havior resulting from replacing a current production policy with a new updated policy. This definition is broad and encloses stochastic exploration actions as well as any behavior change when comparing the two consecutive policies. Furthermore, we consider the scenario in which samples are naturally classified into a set of domains, each representing a unique data segment. Note that in a task-oriented conversational agent, domains are typically defined based on NLU interpretation of the request (e.g. music, shopping, books).
While previous studies considered different aspects of constraining a bandit model, to the best of our knowledge the problem of controlling offpolicy bandit behavior changes across subsequent model updates with a fine-grained control on budgets for different data segments (domains) remains unaddressed. This study is the first to tackle the aforementioned issues by providing a scalable and practical approach. The main contributions of this paper are as follows: (i) Introducing a formulation for controlled exploration in the context of offpolicy contextual bandit learning considering finegrained control over domains based on user-defined constraints. (ii) Presenting a solution based on the primal-dual minimax constraint optimization method that is effective but requires adjusting a few hyperparameters. (iii) Proposing a novel meta gradient algorithm to balance constraint satisfaction and reward maximization that works out-of-the-box and outperforms other methods with no need for hyperparameter adjustment. (iv) Conducting extensive online and offline experiments on the skill routing problem in a real-world conversational AI agent using a set of realistic constraint benchmarks.

heading: 2 Related Work

heading: 2.1 Skill Routing in Conversational AIs
text: In contrast to traditional rule-based systems, modelbased skill routing approaches leverage machine learning models to understand a user request and predict the best action to serve the request (Li et al., 2021; Park et al., 2020).
To improve scalability, self-learning methods have been proposed that rely on user feedback rather than human annotations to learn and improve their skill routing policies in a closed-loop. The recent work by Kachuee et al. (2022) is an excellent example of such approach in which modelbased customer satisfaction metrics (Kachuee et al., 2021) are used to define the reward function, then a
stochastic mixture of replication and bandit models is used to control the exploration rate and safeguard the user experience. Nonetheless, such design may result in sub-optimal decisions as the bandit optimization does not consider the exploration budgets, the stochastic mixture may not be sufficiently finegrained to protect user experience in smaller traffic segments, and deploying such architecture requires dealing with additional complexity of maintaining a separate replication model.

heading: 2.2 Constrained Bandit Learning
text: The majority of studies on controlled bandit learning consider the case of simple multi-armed stochastic bandits (i.e., without context) with practical applications in experiment design (Guha and Munagala, 2007) and automated machine learning (Hoffman et al., 2014). Hoffman et al. (2014) suggested a Bayesian approach to two-phase exploration-exploitation bandit learning in which there is a pre-specified budget for exploration arm evaluations. Another aspect is to ensure safe exploration actions, which is especially useful for sensitive applications in industrial machine learning or healthcare. Amani et al. (2019) introduced a solution in which an initial set of exploration actions is defined, then the exploration set is gradually expanded to ensure minimal unexpected behavior.
For contextual bandits, constraints can be defined in the action space or in terms of model updates. For example, Daulton et al. (2019) solves a two-metric setting in which the reward is being maximized while enforcing a limit for regression on an auxiliary metric compared to a baseline status quo model. Balakrishnan et al. (2018) attempts to learn behavioral constraints by balancing between replication of the current baseline policy and making new actions that show promising rewards. In (Jagerman et al., 2020) authors define safety in terms of user experience metrics and suggest deciding on deploying a new model based on conservative confidence bounds on the off-policy estimates of such metrics.

heading: 3 Constrained Bandit Exploration

heading: 3.1 Problem Formulation
text: We consider the general framework of off-policy contextual bandits in which a policy Π is used to select an action a ∈ A given the observed context vector (x) to maximize the scalar reward (r) received from the environment. Here, we assume
stochastic policies of the form Πθ(a|x) in which a model parameterized by θ (e.g., a neural network) is used to assign action selection probabilities to each action given the context. Furthermore, we assume that each sample belongs to a domain denoted by k ∈ 1 . . .M provided as a feature in x.
In the off-policy setting, the policy is refreshed after collecting a dataset of samples from the current policy. We adopt a definition of exploration which considers any change in the agent behavior compared to the current policy as an exploration action. Alternatively, we can consider replication with respect to the current policy as the rate at which the new policy makes similar decisions to the current policy when both evaluated and sampled stochastically. We define replication for Πθ with respect to Π0 based on the L1-distance of their action propensities given a context x:
Rθ(x) = 1− |Πθ(x)−Π0(x)|1
2 . (1)
In a production system, it is desirable to precisely control the rate at which the new policy replicates the current policy for each domain. This ensures robust and controlled model updates for critical domains while enabling exploration for others that may benefit from an extra exploration budget. Accordingly, we define constraints to encourage the desired behavior for samples of each domain, while learning an off-policy bandit:
argmin θ Ex,a,r,k∼D LΠθ ,
s.t. cmink ≤ Rθ(x) ≤ cmaxk (2)
where context, action, reward, and domain (x, a, r, k) are sampled from a dataset collected from the current policy. In (2), we use cmink and cmaxk to indicate user-defined replication constraints for domain k. LΠθ can be any differentiable off-policy bandit learning objective, for simplicity of discussion, we consider the vanilla inverse propensity scoring (IPS) objective:
LΠθ(x, a, r) = −r Πθ(a|x) Π0(a|x) , (3)
where Π0 is the current policy and r is the observed reward for taking action a collected in the dataset.
A common approach to optimize constrained problems such as (2) is to use the penalty method,
translating constraints into penalty terms that encourage constraint satisfaction:
argmin θ Ex,a,r,k∼D [LΠθ(x, a, r) +
euk max(0, cmink −Rθ(x)) + evk max(0,Rθ(x)− cmaxk )] . (4)
Here, penalty terms are always non-negative and increase if the new policy assigns action probabilities that deviate from the current policy outside the desired boundary. u ∈ RM and v ∈ RM are variables that adjust the weight of each constraint violation term. The exponentiation improves the sensitivity to these parameters and ensures having nonnegative penalty terms. For (4) to actually solve the original constrained problem of (2), proper values for u and v need to be used that enable the best balance between constraint satisfaction and the policy value. In the constrained optimization literature, various methods have been suggested to solve this form of problem. In this paper, to solve this problem, we use the primal-dual minimax method suggested by Nandwani et al. (2019) (Section 3.2) as well as a novel meta-learning method (Section 3.3).

heading: 3.2 Minimax Primal-Dual Method
text: Nandwani et al. (2019) suggested a formulation of the augmented Lagrangian method that supports inequality constraints. They solve the dual problem which is optimizing the dual maximin problem to improve the scalability:
min θ max u,v Ex,a,r,k∼D [LΠθ(x, a, r) +
euk max(0, cmink −Rθ(x))+ evk max(0,Rθ(x)− cmaxk )] . (5)
Algorithm 1 shows an outline of the policy training using the minimax method. This method has four hyperparameters controlling the max player optimization via adjusting the update frequency, learning rate, and decay factors.
Intuitively, the min player is trying to update the policy while the max player is increasingly penalizing it for any constraint violation. A stable point of this algorithm would be to gradually reduce the max player update rate as the min player is getting better at satisfying the constraints, eventually satisfying all constraints resulting in a zero loss for the max player due to the zero hinge penalty terms.
Algorithm 1: Minimax constrained bandit input :D (dataset), η (max learning rate), γ (max
learning rate decay), τ (max update frequency), ξ (max update frequency decay)
u,v, t← 0; Initialize(Πθ) for x, a, r, k ∼ D do
/* loss function of (5) */ L← Loss(x, a, r, k, θ,u,v) if t%τ is 0 then
/* gradient ascent, max player */ u← u+ η∇uL v← v + η∇vL /* lr/update decay */ η ← γ × η τ ← ξ × τ end /* optimize Πθ, min player */ θ ← f(θ,∇θL) /* increment counter */ t← t+ 1
end

heading: 3.3 Meta Gradient Method
text: Theoretically, the primal-dual minimax method is capable of achieving Pareto optimal solutions (Jin et al., 2019; Nandwani et al., 2019). However, in practice, it is infeasible to train for an infinite number of iterations, and therefore approximate inner optimization loops are being used. To find the right balance between constraint satisfaction and policy improvement for the minimax algorithm, it is necessary to carefully adjust multiple hyperparameters. Note that an extensive hyperparameter search is undesirable in many real-world scenarios as it entails not only significant compute costs associated with the search but also increases the turn-around time to deploy refreshed models. To mitigate this issue, we suggest a meta-gradient optimization idea that adapts u and v based on a meta objective within the training process.
Specifically, we define the the meta objective as:
Lmeta = Ex,a,r,k∼D (1− λ)LΠθ(x, a, r)+
λ max(0, cmink −Rθ(x)) + max(0,Rθ(x)− cmaxk )
p(k) (6)
where λ is a hyperparameter to balance between the bandit objective and the constraint penalty terms. The second term is the macro average of violation penalties, in which p(k) is the prior probability of samples belonging to domain k that can be easily pre-computed for a large batch of samples.
Note that (6) is not directly dependent on u and v, instead we rely on online cross-validation (Sutton, 1992; Xu et al., 2018) to update these variables.
We define an inner objective the same as the min optimization problem of (5), do a differentiable optimization step, evaluate the meta objective on another batch of data, then update u and v by taking the derivative of the meta objective through the inner optimization trace.
Algorithm 2 presents an outline of the meta gradient optimization method. Due to practical issues of dealing with high-order gradients, we only consider the immediate impact of a single inner loop update on the meta objective. We found that discarding the vanilla gradient descent used for the inner optimization and using a more advanced optimizer (e.g., Adam) to update Πθ works best. Regarding the λ hyperparameter, we found that simply setting λ = 1 works well in practice. It effectively means that the meta-gradient solution does not require any hyperparameter adjustments (experimental evidence presented in Section 4.4).
Algorithm 2: Meta-grad constrained bandit input :D (dataset), η (learning rate), λ (penalty
weight) u,v← 0; Initialize(Πθ) for x, a, r, k ∼ D and x′, a′, r′, k′ ∼ D do
/* clone parameters */ θ′ ← clone(θ) /* inner loss with θ′ */ Linner ← Lossinner(x, a, r, k, θ′,u,v) /* grad. descent on cloned model */ θ′ ← θ′ − η∇θ′Linner /* compute meta loss */ Lmeta ← Lossmeta(x′, a′, r′, k′, θ′, λ) /* diff. through inner update */ Compute∇uLmeta and∇vLmeta /* use any optimizer for u,v */ u← f(u,∇uLmeta) v← f(v,∇vLmeta) /* inner loss with θ */ L← Lossinner(x, a, r, k, θ,u,v) /* use any optimizer for Πθ */ θ ← f(θ,∇θL)
end
Intuitively, at each training iteration, the inner objective naturally minimizes the bandit loss that is penalized by constraint violation terms proportional to the current u/v. Then, the meta objective computes a validation loss that measures the impact of the inner policy update and u/v on the macro-averaged constraint violations. Finally, by computing the meta-gradient of the meta objective through the inner optimization loop, u and v are updated to better encourage the constraint satisfaction for the next policy update iteration. Thanks to the online cross-validation update for u and v, the
meta-gradient method adjusts the penalty weights such that their value does not unnecessarily keep increasing when it does not result in further improvements to the constraint satisfaction.

heading: 4 Experiments

heading: 4.1 Setup
text: In a commercial dialogue agent, making controlled policy updates is crucial because any change in the skill routing policy directly impacts the user experience. Making abrupt policy changes may negatively impact user retention and in certain businesscritical domains may result in loss of revenue.
Figure 1 shows an overview of the model architecture used in our experiments. Input to the model is a set of routing candidates i.e., a combination of embedded ASR, NLU, and context vectors as well as skill embeddings. The output is the softmaxnormalized propensity of selecting each candidate to handle the user request. The final model has about 12M trainable parameters consisting of a language model to encode utterance, embeddings for contextual signals, and fully-connected layers. As our architecture closely follows the design choices from Kachuee et al. (2022), we refer interested readers to that paper for details.
To train and evaluate our models, we use logged data from a current production policy. The observed reward is based on a curated function of user satisfaction metrics (Kachuee et al., 2021). Our dataset consists of about 40M samples divided into 85% training, 10% validation, and 5% test sets covering 27 domains with imbalanced number of samples. Data used in this work was deidentified to comply with our customer privacy guidelines.

heading: 4.2 Benchmarks
text: In our experiments, we use three different benchmarks for the constraint settings: global, critical, and exploration. The global benchmark aims to constrain the new policy to be within an exploration limit for all domains. In addition to the global constraint, critical assert stronger limits for a set of critical domains defined based on the expert knowledge. The exploration benchmark extends the critical benchmark by adding constraints to encourage exploration for domains that may benefit from additional exploration. Each benchmark is a list of constraints consisting of a short description, applicable domain, and the desired replication range. We provide the exact constraint configurations in the appendix.

heading: 4.3 Baselines and Metrics
text: As the first baseline, we consider the vanilla IPS objective which disregards the constraints. Additionally, we build on the IPS baseline for constraint optimization approaches: quadratic (uniform constant penalty weight), minimax (Algorithm 1), and meta-gradient (Algorithm 2). Unless expressed otherwise, we use Adam optimizer with the default configuration (Kingma and Ba, 2014) .
Regarding hyperparameters, for the penalty weight of the quadratic method we use values from {0.1, 1, 10, 100, 1000}. For the minimax method (Algorithm 1), we found that setting τ and ξ to one while adjusting η and γ presents very similar results to adjusting all four hyperparameters. Consequently, we use a grid search of η ∈ {1, 0.1, 0.01} and γ ∈ {1, 0.999, 0.995} to find the best settings for each benchmark. For the meta-gradient method (Algorithm 2), we found that simply using λ equal to one in the meta objective (i.e., meta objective only focusing on the constraints) outperforms other works. As a result, it does not require adjusting any hyperparameter and the same setting is used across all benchmarks. We provide additional experiment details, sensitivity analysis, and the final hyperparametersin Appendix A.2, A.3, and A.4.
Regarding the evaluation metrics, we use the expected off-policy reward as well as the rate of change in constraint violations averaged over all samples (i.e., micro-averaged) and individual domains (i.e., macro-averaged). To comply with our privacy and business guidelines, in all instances, we only report relative and normalized results which do not represent the actual scales or metric values.

heading: 4.4 Results

heading: 4.4.1 Offline Experimentation
text: Table 1 shows a comparison of results for the IPS, quadratic, minimax, and meta-gradient methods on all benchmarks. For each case, we report the expected reward and the percentage of reduction in the rate of violations compared to the simple IPS objective. The meta-gradient approach consistently shows the best results across all benchmarks. The simple quadratic method behaves very competitively to minimax, except for the explore benchmark which requires a more fine-grained control on multiple constraints . The meta-gradient method, while having the highest reduction in constraints violations, also has very competitive performance in terms of the reward metric.

heading: 4.4.2 Online Experimentation
text: We conducted an A/B experiment to compare the proposed method with the stochastic gating method of Kachuee et al. (2022) for robust self-learning (indicated by RPDR in the table). We conducted our A/B in two phases, deploying and comparing each approach to a baseline skill routing production system. Each phase took one week and consisted of traffic from about 6M customers (3M control
and 3M treatment). For the RPDR method, we used a target replication rate of 99% for each domain. The meta-gradient model was trained with the global benchmark, constraining to a similar 99% replication. For both RPDR and MetaGrad models, we used the same training set which was collected from the control model behavior and followed the same model architecture.
Table 2 presents the results of the A/B experiment. For each method we report the percentage of changes in the achieved reward compared to the control model. For violation reduction, we report the percentage of reduction for MetaGrad compared to the RPDR method. For the replication metric, we simply report the percentage of time that each policy makes actions that replicate the control model decision. As we can see from the results, MetaGrad approach not only shows more stable behavior by better constraint satisfaction and replication rates, but it also achieves statistically significant improvements in the reward value.

heading: 5 Conclusion
text: This paper studied the problem of controlled exploration to control the policy updates in self-learning skill routing systems. We presented a constrained optimization formulation that enables defining the boundary of the desired exploration rate for individual domains. We proposed a scalable and practical solution based on meta-gradient learning which provides the highest constraint satisfaction rates without any extensive hyperparameter adjustment. Finally, we conducted experiments on a real-world conversation system for the skill routing problem. The proposed method was deployed in the production as it showed not only more control over policy changes but also gains in the policy value.
Limitations
While we conducted extensive experiments and demonstrated the effectiveness of the suggested approach for controlled bandit learning in the context of the skill routing problem, there are multiple directions of improvement for future studies. We believe one of the limitations of the suggested constrained optimization framework is that it relies on expert-defined conditions on an arbitrary segmentation of samples. It entails the need for human intervention and manual constraint definition/optimization which can be challenging. Another limitation we faced was during our experiments which showed additional compute overhead of between 2 to 3 times for different constrained optimization methods due to additional optimization objectives, inner loops, and backward passes.
Ethics Statement
The presented work is focused on improving robustness of off-policy bandit updates in conversational systems by introducing robustness constraints on the policy behavior. We do not believe there is any additional risk associated with this work when using the suggested platform on constraints that encourage controlled deviations from a current baseline. Regarding human data handling practices, we ensured anonymity of data samples used in this study and did not reveal any specifics that would violate our internal policies or our customer privacy policies.

heading: A Appendix
text: A.1 Constraint Benchmarks Figure 2 presents the definition of constraint benchmarks used in this paper: global, critical, and explore. The global benchmark sets a general minimum replication rate for all domains. The critical benchmark defines a tighter minimum replication rate for three business-critical domains (home automation, shopping, and notifications) and a more relaxed default case for all other domains. In the explore benchmark, we extend the critical benchmark to include exploration encouragement for the knowledge and music domains.
A.2 Training Details We train each model until convergence or reaching 32 epochs and take the best performing model based on the macro-averaged violation rate measured on the validation set. Each experiment was
run four times using different random seeds for data sampling and weight initialization to report the mean and standard deviation of each result. We used a cluster of 32 NVIDIA V100 GPUs to process a mini-batch size of 32K samples. Each individual run took between 4 to 24 hours.
A.3 Selected Hyperparameters
Table 3 shows the final selected hyperparameters for each benchmark and method. The definition of each hyper-parameter is presented in Algorithm 1 and 2.
A.4 Impact of Hyperparameters
To study the impact of hyperparameters, we conducted an experiment using the critical benchmark by training minimax and meta-gradient based models using different hyperparameter values. Specifically, we train minimax models (Algorithm 1) using η ∈ {1.0, 0.1, 0.01} and γ ∈ {1.0, 0.999, 0.995}. For the metagradient method (Algorithm 2), we use λ ∈ {0.01, 0.05, 0.1, 0.5, 0.75, 0.95, 1.0}. Figure 3 shows the results of such experiment. Based on this experiment, the minimax approach shows a much higher sensitivity to its two hyperparameters, showing a significant impact on both the reward and violation reduction metrics. However, the metagradient method shows much less sensitivity to the λ hyperparameter. We found that simply setting λ = 1 works very well in practice. It can be very desirable for real-world large-scale settings such as conversational systems which require frequent model updates as new features are on-boarded every day, and having a dependency on an extensive hyperparameter search is very costly, if not impractical.
0.886 0.887 0.888 0.889
0.89 0.891 0.892 0.893 0.894 0.895 0.896
η=1.0, γ=0.995
η=0.1, γ=0.999
η=1.0, γ=1.0
Reward(%)
0 1 2 3 4 5 6 7
Violation Reuc η=0.01, γ=0.995
η=0.01, γ=0.999
η=0.1, γ=0.995
η=0.01, γ=1.0
η=1.0, γ=0.999 η=0.1, γ=1.0
(a)
37%
24%
12%
0%
0 1 2 3 4 5 6 7
Violation Reduction(%)
(b)
0.887 0.888 0.889 0.89
0.891 0.892 0.893 0.894 0.895 0.896 0.897 0.898
0 1 2 3 4 5 6 7
Reward(%)
Reward(%)
Reward(%) λ=0.01
λ=0.05
λ=0.5 λ=0.1 λ=0.75 λ=0.95 λ=1.0
(c)
62% 50% 37% 24% 12%
0%
0 1 2 3 4 5 6 7
Violation Reduction(%)
(d)
Figure 3: Comparing the hyper-parameter sensitivity for the minimax and meta-gradient methods on the critical benchmark. For the minimax method: (a) reward and (b) macro violation reduction wrt. different η and γ settings. For the meta-gradient method: (c) reward and (d) macro violation reduction wrt. different λ settings.
A.5 Analysis of Penalty Weights To dive deeper into the reason behind the better performance for the meta-gradient algorithm compared to the minimax approach, we investigated the constraint penalty weight value for the first 3,000 iterations of training using the global benchmark. From Figure 4, we can see the minimax method is monotonically increasing the penalty weight with each iteration which is a behavior consistent with the gradient ascent update rule in Algorithm 1. In other words, as long as there are any constraint violations, minimax will keep increasing the penalty, which in our opinion is the reason for high sensitivity to the hyperparameters. On the other hand, the meta-gradient approach is using a validation signal to dynamically adjust the penalty weight. Consequently, it may keep the penalty term near zero for an initial phase, rapidly increase it, then decay when violations are reduced and getting a higher reward is preferred.

heading: Future Work
text: Limitations
While we conducted extensive experiments and demonstrated the effectiveness of the suggested approach for controlled bandit learning in the context of the skill routing problem, there are multiple directions of improvement for future studies. We believe one of the limitations of the suggested constrained optimization framework is that it relies on expert-defined conditions on an arbitrary segmentation of samples. It entails the need for human intervention and manual constraint definition/optimization which can be challenging. Another limitation we faced was during our experiments which showed additional compute overhead of between 2 to 3 times for different constrained optimization methods due to additional optimization objectives, inner loops, and backward passes