CaptainGAN: Navigate Through Embedding Space For Better Text Generation

1 Introduction . Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) have led to many advances in image generation , image editing , style transfer , and representation learning ( Karras et al. , 2017 ; Brock et al. , 2018 ; Karras et al. , 2019 ) . Unsurprisingly , much effort has been devoted to adopting the GAN framework for unsupervised text generation ( Yu et al. , 2017 ; Che et al. , 2017 ; Balagopalan et al. , 2018 ; Fedus et al. , 2018 ; Guo et al. , 2018 ; de Masson d ’ Autume et al. , 2019 ; Zhang et al. , 2017 ; Nie et al. , 2019 ) . However , as finding a Nash equilibrium is not as straightforward as finding a local optima , researchers have been forced to develop many ad-hoc tricks and techniques to make GAN training well-behaved . In the text generation setting , they are also faced with the additional obstacle of passing discrete tokens through a non-differentiable operation , which prohibits back-propagating the gradient signal to the generator . To address the issue of non-differentiablity , researchers and practitioners use score function gradient estimators such as REINFORCE to train GANs for text generation , where the discriminator is cast as a reward function for the generator . However , these methods still suffer from poor sample efficiency due to the credit assignment problem . We argue that it is disadvantageous to utilize the discriminator as simply a reward function when it is known that gradient-based backpropagation is a far more efficient way to perform credit assignment . In this paper , we propose a novel unsupervised text generation technique , called CaptainGAN , which propagates a modified gradient signal from discriminator to generator in order to improve the efficiency and accuracy of the estimator . Our contributions are as follows : • An update procedure for the generator to incorporate gradient information from the discriminator during generator training . • Lower memory and computational requirements than other RL-based counterparts . • Near SOTA results without maximum likelihood pretraining . Please see Appendix A for a detailed description of the notation used in this paper . 2 Background . The Generative Adversarial Network ( GAN ) proposed in Goodfellow et al . ( 2014 ) is an innovative approach to the generative modeling problem . Rather than using the maximum likelihood estimation ( MLE ) directly to learn a probabilistic model , the GAN is a two-player minimax game in which the goal of one player , the generator Gθ , is to generate samples x̂ from pθ , and the goal of the other player , the discriminator Dϕ , is to learn to classify whether or not a sample was generated from real data pdata or the generator . VD = Ex∼pdata [ logDϕ ( x ) ] + Ex̂∼pθ [ log ( 1−Dϕ ( x̂ ) ) ] ( 1 ) VG = Ex̂∼pθ [ log ( Dϕ ( x̂ ) ) ] ( 2 ) where VD and VG are respectively the objective functions of the discriminator Dϕ and the generator Gθ . Equation 2 is the alternative generator loss suggested by the original work as its gradient does not vanish when Dϕ ( x̂ ) is small . In the standard GAN architecture , the generator ’ s output is directly connected as the input to the discriminator in a fully differentiable manner , which means the gradients from the discriminator ’ s loss function can be back-propagated to the parameters of the generator . However , it requires sampling a sequence of tokens from a discrete distribution in text generation , which is essentially non-differentiable . In order to avoid the intractability of the gradients , text GANs resort to some sort of estimation . 2.1 Continuous Relaxation . Continuous relaxation approaches such as Gumbel-Softmax ( Jang et al. , 2016 ) approximate a stochastic categorical distribution in terms of a deterministic continuous function . While this apparently allows us to remove the non-differentiable discrete sampling altogether ( Kusner & Hernández-Lobato , 2016 ; Nie et al. , 2019 ) , it creates several serious issues . The continuous distribution generates the expectation of embeddings - a weighted sum with no direct correspondence to an exact word or token . This means all the discriminator has to do is spot the difference the actual word embeddings and expectation . In turn , the generator will try to compensate by producing extremely ” spiky ” predictions . Furthermore , this way of generating data creates a major inconsistency in that during inference , the generator has to sample a discrete sequence from distribution whereas during training , it ’ s only trained to generate an expectation that is feasible to discriminator . 2.2 Score-Function Gradient Estimator . The score-function gradient estimator ( Fu , 2006 ; Glynn , 1990 ) , also known as the REINFORCE ( Williams , 1992 ) is a common solution for non-differentiable issue as mentioned above . Applying the REINFORCE algorithm , the gradient of the expectation of reward function fϕ can be written as ∂ ∂θ Ex̂∼pθ [ fϕ ( x̂ ) ] = Ex̂∼pθ [ fϕ ( x̂ ) ∂ ∂θ log pθ ( x̂ ) ] . ( 3 ) Since it does not require fϕ to be differentiable or even continuous as a function of x , the gradient of Ex̂∼pθ [ fϕ ( x̂ ) ] can be back-propagated to the generator Gθ . In the context of GANs , Ex̂∼pθ [ fϕ ( x̂ ) ] can be seen as the objective function of the generator and the reward function fϕ can be replaced with Dϕ . Although REINFORCE is an unbiased estimator , it still has a number of disadvantages such as high variance , low sample efficiency and the credit assignment problem . Therefore , much effort is devoted to reducing the variance using special methods ( Gu et al. , 2016 ; Grathwohl et al. , 2017 ) , or to providing more dense rewards such as in Yu et al . ( 2017 ) ; Che et al . ( 2017 ) ; Fedus et al . ( 2018 ) ; de Masson d ’ Autume et al . ( 2019 ) , where Monte-Carlo roll-outs are used to obtain per-word rewards . 2.3 Straight-Through Gradient Estimator . Another approach is using a straight-through gradient estimator ( Bengio et al. , 2013 ; Jang et al. , 2016 ) . The basic idea is to perform non-differentiable operation during the forward pass , but approximate it with a differentiable proxy during the backward pass . Consider sampling from a categorical distribution ( notation borrowed from Jang et al . ( 2016 ) ) : The following operation is performed during the forward pass z = one-hot ( x̂ ) ( 4 ) where a categorical sample x̂ is encoded as a v-dimensional one-hot vector z ( v being the given vocabulary size ) . During the backward pass , the corresponding straight-through estimator of function f with respect to θ is ĝθ = ( ∂f ∂z ) ⊤ ∂mθ ∂θ ( 5 ) derived by using the approximation∂z∂θ ≈ ∂mθ ∂θ , where mθ is a differentiable proxy for the non-differentiable one-hot encoding vector z ( Jang et al. , 2016 ) . Moreover , Jang et al . ( 2016 ) suggest the choice of pθ = [ pθ ( x1 ) , . . . , pθ ( xv ) ] ⊤ ( 6 ) as proxy , where { x1 , . . . , xv } are all possible tokens in vocabulary V . Compared to the former method , this approach updates all the possible tokens ’ probability , even if the tokens were not sampled . 3 Proposed Method . The non-differentiable connection between the generator Gθ and the discriminator Dϕ is our primary focus in this paper ( Figure 1 ) . Starting from the straight through gradient estimator , we derive a re-centered estimator which helps the generator learn to select better tokens . 3.1 Gradient Estimation in the Generator . Consider a discriminator whose first layer is an embedding lookup layer . During forward propagation , the operation below is performed first : ex̂ = E.lookup ( x̂ ) = Ez ( 7 ) where E = [ e1 , . . . , ev ] and z is one-hot encoding of vocabulary . As described in Section 2.3 , this is a non-differentiable categorical sampling operation with a straight-through estimator of the gradient of VG with respect to θ , formulated as follows : ĝθ = T∑ t=1 ( ∂VG ∂zt ) ⊤ ∂pt , θ ∂θ = T∑ t=1 ( E⊤ ∂VG ∂e ( ex̂t ) ) ⊤ ∂pt , θ ∂θ ( 8 ) = T∑ t=1 ∑ x∈V ⟨ex , ∂VG ∂e ( ex̂t ) ⟩ ∂p ( x|x̂1 : t−1 ) ∂θ ( 9 ) where x̂t is the selected token at time-step t , T is the maximum time-step and V is the given vocabulary . The first term of Equation 8 can be derived from Equation 7 by applying the chain rule . 3.2 Re-centered Estimator . We turn our attention to the first term of Equation 9 : ⟨ex , ∂VG ∂e ( ex̂t ) ⟩ ( 10 ) Denoting it as δx̂t→x [ VG ] , Equation 9 becomes ĝθ = T∑ t=1 ∑ x∈V δx̂t→x [ VG ] ∂pθ ( x|x̂1 : t−1 ) ∂θ . ( 11 ) The coefficient δx̂t→x [ VG ] can be considered as an affinity factor indicating how much the discriminator is ‘ attracted ’ to the token x . Recall that the gradient ∂VG∂e ( ex̂ ) has the direction of greatest increase of the objective VG at ex̂ . Since this direction is relative to ex̂ , we suggest taking the inner product with the vector ex − ex̂ which has been re-centered relative to ex̂ ( Figure 2 ) . ⟨ex − ex̂t , ∂VG ∂e ( ex̂t ) ⟩ ( 12 ) This inner product approximates the increase of VG when x̂t is replaced by x . Because this approximation only holds in the neighborhood of ex̂t , |ex − ex̂t | ≪ 1 , we also add a kernel to dampen the affinity factor for ex that are far away . That is , δRCx̂t→x [ VG ] = K ( ex , ex̂t ) ⟨ex − ex̂t , ∂VG ∂e ( ex̂t ) ⟩ ( 13 ) where K ( u , v ) = 1√ 1 + |u − v|2 . ( 14 ) See Appendix B for more details . As a result , Equation 9 becomes ĝRCθ = T∑ t=1 ∑ x∈V δRCx̂t→x [ VG ] ∂p ( x|x̂1 : t−1 ) ∂θ . ( 15 ) In score-function based approaches , the generator is only given feedback on the tokens it samples , so it requires a large amount of trial-and-error in order to figure out how to write realistic sentences . To use an analogy , imagine a captain at sea ( embedding space ) searching for treasure . Each iteration of the game , the captain chooses random locations in the space and receives feedback regarding the choices . Score-function based approaches tell us how good the location is but say nothing about how to improve it . In contrast , our method , gives the captain a navigational direction to the next best location . Even if the direction is biased , it should allow the captain to find the treasures with fewer iterations . 4 Evaluation Metrics . We evaluate our model ’ s ability to generate realistic and diverse texts with BLEU-based metric , Fréchet Embedding Distance and language model based metric .

Summary: The submission proposes to train a GAN on discrete sequences using the straight-through Gumbel estimator introduced in Jang et al. (2016) in combination with gradient centering. The proposed approach is evaluated on COCO and EMNLP News in terms of BLEU and Self-BLEU scores, Fréchet Embedding Distance, Language Model Score, and Reverse Language Model Score.