# An Adversarial Deep Hybrid Model for Text-Aware Recommendation with Convolutional Neural Networks
## 3 Adversarial Deep Collaborative Filtering
### Training
#### 3.4.2 Optimizing Generator  
On the other hand, the generative model \(p_{\theta}\) intends to minimize the objective in Equation (2). It fits the underlying true distribution by randomly generating samples to fool the discriminator. Formally, keeping the discriminator \(D_{w}\) fixed, we update the parameters \(\theta\) by maximizing the probability that samples from \(p_{\theta}\) are classified as real by the discriminator:  
\[ \begin{split}\theta^{*}&=\arg\min_{\theta}\mathrm{ E}_{\mathbf{s}\sim p_{\theta}}[\log(1-D_{w}(\mathbf{s}))].\\ &=\arg\max_{\theta}\mathrm{E}_{\mathbf{s}\sim p_{\theta}}[D_{w}( \mathbf{s})].\end{split}  \tag{13}\]From the perspective of reinforcement learning [51; 52; 53], \(p_{\theta}\) acts as the policy to select actions (samples), in which the reward for an action \(\mathbf{s}\) is given by the term \(\text{log}D_{w}(\mathbf{s})\). The goal of Equation (13) is to shift the policy through its parameters to increase the reward of generated samples, as judged by \(\text{log}D_{w}(\cdot)\). Following [39; 51; 52; 53], we adopt the policy gradient method and calculate the gradient with respect to the generator parameters as follows:  
\[\bigtriangledown\text{E}_{\mathbf{s}\sim p_{\theta}}[\text{log}D_{w }(\mathbf{s})] =\sum_{\mathbf{s}}\text{log}D_{w}(\mathbf{s})\bigtriangledown p_{ \theta}(\mathbf{s}) \tag{14}\] \[=\sum_{\mathbf{s}}\text{log}D_{w}(\mathbf{s})p_{\theta}(\mathbf{ s})\bigtriangledown\text{log}p_{\theta}(\mathbf{s})\] \[=\text{E}_{\mathbf{s}\sim p_{\theta}}[\text{log}D_{w}(\mathbf{s} )\bigtriangledown\text{log}p_{\theta}(\mathbf{s})].\]  
Here, \(\text{log}D_{w}(\cdot)\) based on the discriminator acts as learning weights and assigns relatively large weights (rewards) to those samples that are incorrectly modeled by the current \(p_{\theta}\). One typical solution for the last step is to perform Monte Carlo approximation and compute the average of multiple samples. However, this would distort the function of \(\text{log}D_{w}(\cdot)\), since the weights of different samples would counter-weigh each other in the training process. This is a direct result of the generation process of the generator. Therefore, we perform stochastic gradient decent and update the parameters using one sample at a time (shown in Algorithm 1). We note that the update rule has a very similar form to LambdaRank [54; 55] and the difference is that they use current item ranking list to compute the learning weights while we exploit signals from the discriminator. It is noteworthy that theoretically we can sample the unobserved item ranking pairs that are not in the training set \(S\), thus improving the model's generalization ability. Overall, the adversarial training is an alternating optimization with respect to the generator parameters \(\theta\) and the discriminator parameters \(w\), i.e., alternating between Equations (12) and (13). Following [39], we pre-train the generative model to ensure a stable model training process in experiments. The convergence of the proposed model ADHR is monitored through early stopping on a validation dataset. The overall optimization procedure is illustrated in Algorithm 1. ```
Input: generator \(p_{\theta}\); discriminator \(D_{w}\); and training set \(S\). Initialize \(p_{\theta}\) with a pre-trained model; repeat for train\(D_{w}\)do  Draw a sample \(\mathbf{s}=(u,j,j^{\prime})\) uniformly from the pairwise training data \(S\);  Run a Bernoulli trial with \(p_{\theta}(\mathbf{s})\) to generate a fake sample \(\mathbf{s}^{\prime}\);  Update \(w\gets w+\eta(\bigtriangledown\text{log}D_{w}(\mathbf{s})+\bigtriangledown \text{log}(1-D_{w}(\mathbf{s}^{\prime}))-\lambda w)\).