CaptainGAN: Navigate Through Embedding Space For Better Text Generation

1 Introduction . Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) have led to many advances in image generation , image editing , style transfer , and representation learning ( Karras et al. , 2017 ; Brock et al. , 2018 ; Karras et al. , 2019 ) . Unsurprisingly , much effort has been devoted to adopting the GAN framework for unsupervised text generation ( Yu et al. , 2017 ; Che et al. , 2017 ; Balagopalan et al. , 2018 ; Fedus et al. , 2018 ; Guo et al. , 2018 ; de Masson d ’ Autume et al. , 2019 ; Zhang et al. , 2017 ; Nie et al. , 2019 ) . However , as finding a Nash equilibrium is not as straightforward as finding a local optima , researchers have been forced to develop many ad-hoc tricks and techniques to make GAN training well-behaved . In the text generation setting , they are also faced with the additional obstacle of passing discrete tokens through a non-differentiable operation , which prohibits back-propagating the gradient signal to the generator . To address the issue of non-differentiablity , researchers and practitioners use score function gradient estimators such as REINFORCE to train GANs for text generation , where the discriminator is cast as a reward function for the generator . However , these methods still suffer from poor sample efficiency due to the credit assignment problem . We argue that it is disadvantageous to utilize the discriminator as simply a reward function when it is known that gradient-based backpropagation is a far more efficient way to perform credit assignment . In this paper , we propose a novel unsupervised text generation technique , called CaptainGAN , which propagates a modified gradient signal from discriminator to generator in order to improve the efficiency and accuracy of the estimator . Our contributions are as follows : • An update procedure for the generator to incorporate gradient information from the discriminator during generator training . • Lower memory and computational requirements than other RL-based counterparts . • Near SOTA results without maximum likelihood pretraining . Please see Appendix A for a detailed description of the notation used in this paper . 2 Background . The Generative Adversarial Network ( GAN ) proposed in Goodfellow et al . ( 2014 ) is an innovative approach to the generative modeling problem . Rather than using the maximum likelihood estimation ( MLE ) directly to learn a probabilistic model , the GAN is a two-player minimax game in which the goal of one player , the generator Gθ , is to generate samples x̂ from pθ , and the goal of the other player , the discriminator Dϕ , is to learn to classify whether or not a sample was generated from real data pdata or the generator . VD = Ex∼pdata [ logDϕ ( x ) ] + Ex̂∼pθ [ log ( 1−Dϕ ( x̂ ) ) ] ( 1 ) VG = Ex̂∼pθ [ log ( Dϕ ( x̂ ) ) ] ( 2 ) where VD and VG are respectively the objective functions of the discriminator Dϕ and the generator Gθ . Equation 2 is the alternative generator loss suggested by the original work as its gradient does not vanish when Dϕ ( x̂ ) is small . In the standard GAN architecture , the generator ’ s output is directly connected as the input to the discriminator in a fully differentiable manner , which means the gradients from the discriminator ’ s loss function can be back-propagated to the parameters of the generator . However , it requires sampling a sequence of tokens from a discrete distribution in text generation , which is essentially non-differentiable . In order to avoid the intractability of the gradients , text GANs resort to some sort of estimation . 2.1 Continuous Relaxation . Continuous relaxation approaches such as Gumbel-Softmax ( Jang et al. , 2016 ) approximate a stochastic categorical distribution in terms of a deterministic continuous function . While this apparently allows us to remove the non-differentiable discrete sampling altogether ( Kusner & Hernández-Lobato , 2016 ; Nie et al. , 2019 ) , it creates several serious issues . The continuous distribution generates the expectation of embeddings - a weighted sum with no direct correspondence to an exact word or token . This means all the discriminator has to do is spot the difference the actual word embeddings and expectation . In turn , the generator will try to compensate by producing extremely ” spiky ” predictions . Furthermore , this way of generating data creates a major inconsistency in that during inference , the generator has to sample a discrete sequence from distribution whereas during training , it ’ s only trained to generate an expectation that is feasible to discriminator . 2.2 Score-Function Gradient Estimator . The score-function gradient estimator ( Fu , 2006 ; Glynn , 1990 ) , also known as the REINFORCE ( Williams , 1992 ) is a common solution for non-differentiable issue as mentioned above . Applying the REINFORCE algorithm , the gradient of the expectation of reward function fϕ can be written as ∂ ∂θ Ex̂∼pθ [ fϕ ( x̂ ) ] = Ex̂∼pθ [ fϕ ( x̂ ) ∂ ∂θ log pθ ( x̂ ) ] . ( 3 ) Since it does not require fϕ to be differentiable or even continuous as a function of x , the gradient of Ex̂∼pθ [ fϕ ( x̂ ) ] can be back-propagated to the generator Gθ . In the context of GANs , Ex̂∼pθ [ fϕ ( x̂ ) ] can be seen as the objective function of the generator and the reward function fϕ can be replaced with Dϕ . Although REINFORCE is an unbiased estimator , it still has a number of disadvantages such as high variance , low sample efficiency and the credit assignment problem . Therefore , much effort is devoted to reducing the variance using special methods ( Gu et al. , 2016 ; Grathwohl et al. , 2017 ) , or to providing more dense rewards such as in Yu et al . ( 2017 ) ; Che et al . ( 2017 ) ; Fedus et al . ( 2018 ) ; de Masson d ’ Autume et al . ( 2019 ) , where Monte-Carlo roll-outs are used to obtain per-word rewards . 2.3 Straight-Through Gradient Estimator . Another approach is using a straight-through gradient estimator ( Bengio et al. , 2013 ; Jang et al. , 2016 ) . The basic idea is to perform non-differentiable operation during the forward pass , but approximate it with a differentiable proxy during the backward pass . Consider sampling from a categorical distribution ( notation borrowed from Jang et al . ( 2016 ) ) : The following operation is performed during the forward pass z = one-hot ( x̂ ) ( 4 ) where a categorical sample x̂ is encoded as a v-dimensional one-hot vector z ( v being the given vocabulary size ) . During the backward pass , the corresponding straight-through estimator of function f with respect to θ is ĝθ = ( ∂f ∂z ) ⊤ ∂mθ ∂θ ( 5 ) derived by using the approximation∂z∂θ ≈ ∂mθ ∂θ , where mθ is a differentiable proxy for the non-differentiable one-hot encoding vector z ( Jang et al. , 2016 ) . Moreover , Jang et al . ( 2016 ) suggest the choice of pθ = [ pθ ( x1 ) , . . . , pθ ( xv ) ] ⊤ ( 6 ) as proxy , where { x1 , . . . , xv } are all possible tokens in vocabulary V . Compared to the former method , this approach updates all the possible tokens ’ probability , even if the tokens were not sampled . 3 Proposed Method . The non-differentiable connection between the generator Gθ and the discriminator Dϕ is our primary focus in this paper ( Figure 1 ) . Starting from the straight through gradient estimator , we derive a re-centered estimator which helps the generator learn to select better tokens . 3.1 Gradient Estimation in the Generator . Consider a discriminator whose first layer is an embedding lookup layer . During forward propagation , the operation below is performed first : ex̂ = E.lookup ( x̂ ) = Ez ( 7 ) where E = [ e1 , . . . , ev ] and z is one-hot encoding of vocabulary . As described in Section 2.3 , this is a non-differentiable categorical sampling operation with a straight-through estimator of the gradient of VG with respect to θ , formulated as follows : ĝθ = T∑ t=1 ( ∂VG ∂zt ) ⊤ ∂pt , θ ∂θ = T∑ t=1 ( E⊤ ∂VG ∂e ( ex̂t ) ) ⊤ ∂pt , θ ∂θ ( 8 ) = T∑ t=1 ∑ x∈V ⟨ex , ∂VG ∂e ( ex̂t ) ⟩ ∂p ( x|x̂1 : t−1 ) ∂θ ( 9 ) where x̂t is the selected token at time-step t , T is the maximum time-step and V is the given vocabulary . The first term of Equation 8 can be derived from Equation 7 by applying the chain rule . 3.2 Re-centered Estimator . We turn our attention to the first term of Equation 9 : ⟨ex , ∂VG ∂e ( ex̂t ) ⟩ ( 10 ) Denoting it as δx̂t→x [ VG ] , Equation 9 becomes ĝθ = T∑ t=1 ∑ x∈V δx̂t→x [ VG ] ∂pθ ( x|x̂1 : t−1 ) ∂θ . ( 11 ) The coefficient δx̂t→x [ VG ] can be considered as an affinity factor indicating how much the discriminator is ‘ attracted ’ to the token x . Recall that the gradient ∂VG∂e ( ex̂ ) has the direction of greatest increase of the objective VG at ex̂ . Since this direction is relative to ex̂ , we suggest taking the inner product with the vector ex − ex̂ which has been re-centered relative to ex̂ ( Figure 2 ) . ⟨ex − ex̂t , ∂VG ∂e ( ex̂t ) ⟩ ( 12 ) This inner product approximates the increase of VG when x̂t is replaced by x . Because this approximation only holds in the neighborhood of ex̂t , |ex − ex̂t | ≪ 1 , we also add a kernel to dampen the affinity factor for ex that are far away . That is , δRCx̂t→x [ VG ] = K ( ex , ex̂t ) ⟨ex − ex̂t , ∂VG ∂e ( ex̂t ) ⟩ ( 13 ) where K ( u , v ) = 1√ 1 + |u − v|2 . ( 14 ) See Appendix B for more details . As a result , Equation 9 becomes ĝRCθ = T∑ t=1 ∑ x∈V δRCx̂t→x [ VG ] ∂p ( x|x̂1 : t−1 ) ∂θ . ( 15 ) In score-function based approaches , the generator is only given feedback on the tokens it samples , so it requires a large amount of trial-and-error in order to figure out how to write realistic sentences . To use an analogy , imagine a captain at sea ( embedding space ) searching for treasure . Each iteration of the game , the captain chooses random locations in the space and receives feedback regarding the choices . Score-function based approaches tell us how good the location is but say nothing about how to improve it . In contrast , our method , gives the captain a navigational direction to the next best location . Even if the direction is biased , it should allow the captain to find the treasures with fewer iterations . 4 Evaluation Metrics . We evaluate our model ’ s ability to generate realistic and diverse texts with BLEU-based metric , Fréchet Embedding Distance and language model based metric .

Summary: This paper attempts to solve the problem of non-differentiable connection between the generation and discriminator of a GAN. The authors come up with an estimator of the gradient for the generator from the gradient of the discriminator, which was disconnected previously. With this change, the model should be able to  select better tokens than random selection, which could leads to more robust training. The experiment results on both COCO Image Captions and EMNLP 2017 News datasets justify the authors' argument.