## 3 Proposed Method
### Variational Reward Estimator Bottleneck  
The variational information bottleneck [24; 25; 26] is an theoretical information approach that restricts unproductive information flow between the discriminator and inputs. Inspired by this approach, we propose a regularized objective that constrains the mutual information between encoded original inputs and state-action pairs, thereby ensuring highly informative internal representations and a robust adversarial model. Our proposed method trains an encoder that is maximally informative regarding human dialogues.  
To this end, we employ a stochastic encoder and an upper bound constraint on the mutual information between the dialogue states \(X\) and latent variables \(\mathbf{Z}\):  
\[\begin{split} L_{f,\mathbf{E}}(\zeta,\psi)&=\mathbb{E }_{x,a\sim\mathcal{D}}[\mathbb{E}_{\mathbf{z}\sim\mathbf{E}(\mathbf{z}|x_{t},x _{t+1})}[f_{\zeta,\psi}(\mathbf{z}_{g},\mathbf{z}^{\prime}_{h},\mathbf{z}_{h} )]\] \[-\mathbb{E}_{x,a\sim\pi}[\mathbb{E}_{\mathbf{z}\sim\mathbf{E}( \mathbf{z}|x_{t},x_{t+1})}[f_{\zeta,\psi}(\mathbf{z}_{g},\mathbf{z}^{\prime}_{ h},\mathbf{z}_{h})]]\] \[\text{s.t.}&\quad I(\mathbf{Z},X)\leq l_{c}\end{split} \tag{2}\]  
where \(f_{\zeta,\psi}(\mathbf{z}_{g},\mathbf{z}^{\prime}_{h},\mathbf{z}_{h})=D_{g}( \mathbf{z}_{g})+\gamma D_{h}(\mathbf{z}^{\prime}_{h})+D_{h}(\mathbf{z}_{h})\), and \(D\) is modeled with nonlinear function. Note that \(f_{\zeta,\psi}(\mathbf{z}_{g},\mathbf{z}^{\prime}_{h},\mathbf{z}_{h})\) is divided into the three terms \(D_{g}(\mathbf{z}_{g})\), \(\gamma D_{h}(\mathbf{z}^{\prime}_{h})\), and \(D_{h}(\mathbf{z}_{h})\), based on GANs [21], GAN-GCL [35], and AIRL [22]. \(D_{g}\) represents the encoded disentangled reward approximator with the parameter \(\zeta\), and \(D_{h}\) is the encoded shaping term with the parameter \(\psi\). Stochastic encoder \(\mathbf{E}(\mathbf{z}|x_{t},x_{t+1})\) can be defined as \(\mathbf{E}(\mathbf{z}|x_{t},x_{t+1})=\mathbf{E}_{g}(\mathbf{z}_{g}|x_{t}) \cdot\mathbf{E}_{h}(\mathbf{z}_{h}|x_{t})\cdot\mathbf{E}_{h}(\mathbf{z}^{ \prime}_{h}|x_{t+1})\), which maps states to a latent distribution \(\mathbf{z}\): \(\mathbf{E}(\mathbf{z}|x_{t})=\mathcal{N}(\mu_{\mathbf{E}}(x_{t}),\Sigma_{ \mathbf{E}}(x_{t}))\). \(r(\mathbf{z})=\mathcal{N}(0,I)\) is standard Gaussian, and \(l_{c}\) stands for an enforced upper bound on mutual information.  
To optimize \(L_{f,\mathbf{E}}(\zeta,\psi)\), VRB introduces a Lagrange multiplier \(\varphi\) as follows:  
\[\begin{split} L_{f,\mathbf{E}}(\zeta,\psi)&=\mathbb{E }_{x,a\sim\mathcal{D}}[\mathbb{E}_{\mathbf{z}\sim\mathbf{E}(\mathbf{z}|x_{t},x _{t+1})}[f_{\zeta,\psi}(\mathbf{z}_{g},\mathbf{z}^{\prime}_{h},\mathbf{z}_{h} )]\] \[-\mathbb{E}_{x,a\sim\pi}[\mathbb{E}_{\mathbf{z}\sim\mathbf{E}( \mathbf{z}|x_{t},x_{t+1})}[f_{\zeta,\psi}(\mathbf{z}_{g},\mathbf{z}^{\prime}_{ h},\mathbf{z}_{h})]\] \[+\varphi\left(\mathbb{E}_{x,a\sim\pi}\left[\operatorname{KL}[ \mathbf{E}(\mathbf{z}|x_{t},x_{t+1})]\right]||r(\mathbf{z})\right]-l_{c})\end{split} \tag{3}\]  
where the mutual information between dialogue states \(X\) and latent variable \(\mathbf{Z}\) is  
\[\begin{split} I(\mathbf{Z},X)&=\operatorname{KL}[p( \mathbf{z},x)||p(\mathbf{z})p(x)]\\ &=\int d\mathbf{z}\ dx\ p(\mathbf{z},x)\log\frac{p(\mathbf{z},x)}{ p(\mathbf{z})p(x)}\\ &=\int d\mathbf{z}\ dx\ p(x)\mathbf{E}(\mathbf{z}|x)\log\frac{ \mathbf{E}(\mathbf{z}|x)}{p(\mathbf{z})}\\ &\leq l_{c}=\int d\mathbf{z}\ dx\ \pi_{\theta}(x)\mathbf{E}( \mathbf{z}|x)\log\frac{\mathbf{E}(\mathbf{z}|x)}{r(\mathbf{z})}\\ &=\mathbb{E}_{x,a\sim\pi}[\operatorname{KL}[\mathbf{E}(\mathbf{ z}|x)||r(\mathbf{z})]]\end{split}\]  
In Equation (3), the VRB minimizes the mutual information with dialogue states to focus on discriminative features. The VRB also minimizes the KL divergence with the human dialogues, while maximizing the KL divergence with the generated dialogues, thereby distinguishing effectively between samples from dialogue policy and human dialogues. Our proposed model is summarized in Algorithm 1.