Patent Publication Number: US-2022215594-A1

Title: Auto-regressive video generation neural networks

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a non-provisional of and claims priority to U.S. Provisional Patent Application No. 62/852,271, filed on May 23, 2019, the entire contents of which are hereby incorporated by reference. 
    
    
     BACKGROUND 
     This specification relates to a neural network system for video generation. 
     Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. 
     SUMMARY 
     This specification describes a neural network system implemented as computer programs on one or more computers in one or more locations that implement a video generation neural network system for efficient video generation. 
     The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. Generation of videos is a challenging task due to the large number of pixels that needs to be produced and the vast complexity of their joint distribution. The auto-regressive video neural network described in this specification can address the computational challenge posed to conventional video generation models by iteratively generating subscaled video slices in conjunction with an efficient implementation of a block-local self-attention mechanism. In particular, as the cost of splitting videos into blocks is negligible, the described auto-regressive video neural network can significantly reduce memory requirements by applying block-local self-attention on video blocks. The described auto-regressive video neural network can further reduce memory requirements by generating videos in spatial-temporal subscale order. As a result, the described auto-regressive video neural network can be scaled up substantially while retaining longer range spatio-temporal dependencies. This scalability allows a system that implements the auto-regressive video neural network to obtain state-of-the-art results across a range of popular video generation benchmarks and to model real-world videos of an unprecedented complexity. 
     The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows an example video generation neural network system. 
         FIG. 2  is a flow diagram of an example process for generating a video. 
         FIG. 3  is a flow diagram of an example process for generating an encoded conditioning channel slice using an encoder neural network. 
         FIG. 4  is a flow diagram of an example process for generating a next fully generated channel slice using a decoder neural network. 
     
    
    
     Like reference numbers and designations in the various drawings indicate like elements. 
     DETAILED DESCRIPTION 
     Video generation is an important task in many applications such as content creation, forecasting, transfer teaming and model-based reinforcement learning. However, due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural videos remains a challenging task. Existing video generation models attempt to address these issues by combining sometimes complex, often video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. 
     In contrast, the video generation neural network system described in this specification can achieve better results across multiple metrics in comparison to existing video generation models because it can produce videos of higher fidelity and realism. The ability to generate plausible videos in a computationally efficient manner make the video generation neural network system described herein particularly useful for real-world applications such as content creation, and camera motion and object or human movement prediction for reinforcement learning tasks such as robotic manipulation and self-driving car control. 
       FIG. 1  shows an example video generation neural network system  100 . The system  100  is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. 
     The system  100  includes an encoder neural network  102  (also referred to as “the encoder  102 ”), a decoder neural network  118  (also referred to as “the decoder  118 ”), and a subsystem  140 . The subsystem  140  is a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described herein can be implemented. 
     The system  100  aims to generate a desired output video, which can be denoted as video χ∈   T×H×W×N     c    with time, height, width and channel dimensions, respectively, N c  is the number of channels of the output video. For example, a video with RGB video frames has three channels: red, green, and blue. A distribution p(χ) over the video χ can be modeled by means of a pixel-channel level autoregressive factorization. That is, the joint distribution over pixels is factorized into a product of channel intensities for all N c  channels, for each of the N p =T×H×W pixels, with respect to an ordering π over pixels: 
     
       
         
           
             
               
                 
                   
                     p 
                     ⁡ 
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     
                       ∏ 
                       
                         i 
                         = 
                         0 
                       
                       
                         
                           N 
                           p 
                         
                         - 
                         1 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         ∏ 
                         
                           k 
                           = 
                           0 
                         
                         
                           
                             N 
                             c 
                           
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           p 
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   x 
                                   
                                     π 
                                     ⁡ 
                                     
                                       ( 
                                       i 
                                       ) 
                                     
                                   
                                   k 
                                 
                                 ❘ 
                                 
                                   x 
                                   
                                     π 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         &lt; 
                                         i 
                                       
                                       ) 
                                     
                                   
                                 
                               
                               , 
                               
                                 x 
                                 
                                   π 
                                   ⁡ 
                                   
                                     ( 
                                     i 
                                     ) 
                                   
                                 
                                 
                                   &lt; 
                                   k 
                                 
                               
                             
                             ) 
                           
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     The ordering π is given by a combination of a subscale- and raster-scan ordering as further detailed below. In some cases, the system  100  may generate the desired output video conditioned on an input. The input can be, for example, an input received from the user of the system  100  or from another system that specifies pixel values of one or more pixels of the output video. In some other implementations, the system  100  may generate the desired output video from scratch. 
     To generate the desired output video, the system  100  first generates an initial output video  132  using the subsystem  140 . The initial output video  132  includes multiple frames. Each of the frames has multiple channels. Each channel is a two-dimensional image and indexed by a respective channel index from a set of channel indices of the initial output video. For example, if each frame has three channels: red, green, and blue, then the set of channel indices is {R, G, B}. As another example, if each frame has four channels: cyan, magenta, yellow, and key, then the set of channel indices is {C, M, Y, K}. For each of the channels, each pixel in the channel is assigned a predetermined pixel value or is padded with a blank pixel. 
     For example, in some cases, the initial output video  132  is a fully padded video where every single pixel of the video  132  is padded with a blank pixel. In this case, the system  100  uses the decoder neural network  118  to generate the first fully-generated channel slice, pixel by pixel. This process will be described in further detail below. After generating the first channel slice, the system  100  can now partially fill the initial output video  132  with information (including pixel values) from the first fully-generated channel slice. 
     In some other cases, the system  100  may receive (e.g., from a user of the system  100  or from another system) an input that specifies predetermine pixel values for one or more of the pixels of the video  132 . For example, as shown in  FIG. 1 , the initial output video  132  has some black pixels, which are pixels that are padded with a blank pixel, and some gray pixels, which are pixels that are each assigned a respective predetermined pixel value specified by the input. 
     The subsystem  140  is configured to identify a partitioning of the initial output video  132  into a set of channel slices (e.g., channel slices  134 ,  136 ,  138 ,  140 , . . . ) that are indexed according to a particular slice order. Each channel slice is a down sampling of a channel stack from a set of channel stacks. Each channel stack in the set of channel stacks corresponds to a respective channel index (e.g., channel index R, G or B) and is a stack of channels having the same respective channel index according to time. 
     In particular, the subsystem  140  evenly divides the initial output video  132  with shape (T, H, W) into a set of smaller channel slices by using a given subscale factor s=(s t ,s h ,s w ). The subscale factor s divides the video  132  into s=(s t ·s h ·s w ) channel slices, each of resolution (T/s t ,H/s h , W/s w ), as illustrated in the bottom of  FIG. 1 . The subsystem  140  generates these channel slices in a particular order. For example, the subsystem  140  generates channel slices according to their respective offsets such that it first generates slice χ (0,0,0)  (slice  134  which a down sampling of a channel stack corresponding to channel index C), then χ (0,0,1)  (slice  136  which is a down sampling of a channel stack corresponding to channel index M), χ (0,1,0)  (slice  138  which is a down sampling of a channel stack corresponding to channel index Y), χ (1,0,0)  (slice  140  which is a down sampling of a channel stack corresponding to channel index K), up until slice χ (s     t     −1,s     h     −1,s     w     −1) . Generating all channel slices one at a time in this way can drastically reduce the number of pixels N p  in memory to N p /s, which enables scaling the architecture of the video generation neural network system  100  by a factor of s. 
     The subsystem  140  is configured to initialize, for each channel stack in the set of channel stacks, a set of fully-generated channel slices. For example, the subsystem  140  may initialize the set of fully-generated channel slices as an empty set, which means there is no fully-generated channel slice in the set at the beginning. 
     The system  100  repeatedly processes a current output video including the current set of fully-generated channel slices using the encoder  102  and decoder  118  to generate pixel values for each of the s channel slices according to the particular slice order. For the first iteration, the current output video is the initial output video  132 . After the channel slice is fully generated (e.g., all pixels in the channel slice are assigned respective values), the channel slice becomes the next fully-generated channel slice to be added to the set of fully-generated channel slices of the respective channel stack. 
     In particular, the encoder  102  is configured to process the current output video including the current set of fully-generated channel slices of all channel stacks to generate an encoded conditioning channel slice for the current channel slice. The current channel slice can be denoted as χ (a,b,c) , where (a, b, c) denotes the current channel slice index. The encoder  102  includes a 3D encoder convolution layer  104  followed by a stack of multiple encoding self-attention layers  114 . Each of the encoding self-attention layers includes an encoder layer-norm layer  106 , a block self-attention layer  110 , and one or more encoder feed-forward layers  112 . The one or more encoder feed-forward layers  112  may be, for example, a multilayer perceptron (MLP). 
     The encoder  102  processes the current output video using the 3D encoder convolution layer  104  to generate an initial encoder representation. The encoder  102  then transforms this initial encoder representation by a linear projection to a hidden size to generate an intermediate encoder representation and provides the intermediate encoder presentation as input to the stack of encoding self-attention layers  114 . The encoder  102  processes the intermediate encoder representation using the stack of encoding self-attention layers  114  to generate the encoded conditioning channel slice for the current channel slice. In some cases, each of the encoding self-attention layers can be parameterized by the same block size and number of attention heads. In some other cases, each of the encoding self-attention layers can be parameterized by a different block size and number of attention heads. 
     The process for generating the encoded conditioning channel slice using the encoder  102  is described in more detail below with reference to  FIG. 3 . 
     The decoder  118  is configured to receive the encoded conditioning channel slice from the encoder  102  and to process the encoded conditioning channel slice to generate pixel values for the current channel slice (i.e., to make the current channel slice become the next fully-generated channel slice). The decoder  118  can be initialized with pixels having predetermined pixel values from the initial output video  132  that are in the next fully-generated channel slice. The decoder  118  includes a 3D decoder convolution layer  120  followed by multiple decoding self-attention layers  128 . Each of the decoding self-attention layers includes a decoder layer-norm layer  122 , a masked block self-attention layer  124 , and one or more decoder feed-forward layers  126 . The one or more decoder feed-forward layers  126  may be, for example, a multilayer perceptron (MLP). The process for generating the next fully-generated channel slice using the decoder  118  is described in more detail below with reference to  FIG. 4 . 
     The subsystem  140  adds the next fully generated channel slice to the current set of fully-generated channel slices of the respective channel stack. 
     The system  100  repeats the above process of processing the current output video until all channel slices have been fully generated. 
     The system  100  generates, for each of the channel indices, a respective fully-generated channel stack using the respective fully generated channel slices. In particular, for each channel index, the subsystem  140  combines all fully-generated channel slices of the respective channel stack to generate the fully-generated channel stack. 
     The system  100  generates a fully-generated output video using the fully-generated channel stacks generated for the channel indices. In particular, the subsystem  140  combines all fully-generated channel stacks to generate the fully-generated output video (which is the desired output video). 
     In some implementations, the encoder neural network  102  and the decoder neural network  118  are implemented as computer programs on the same computer system. 
     In some other implementations, the encoder neural network  102  can be implemented on a first computer system. The encoder neural network  102  can generate next encoded conditioning channel slices for a desired output video and send these encoded conditioning channel slices to the decoder neural network  118  running on a second computer system different from the first computer system. The decoder neural network  118  can then use the encoded conditioning channel slices to generate next fully-generated channel slices that are used to reconstruct the desired output video. 
     In some implementations, the encoder  102  and the decoder  118  can be jointly trained using the same loss function. The loss function can be a negative log likelihood of all channel slices, 
       FIG. 2  is a flow diagram of an example process for generating an output video. For convenience, the process  200  will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation neural network system, e.g., the video generation neural network system  100  of  FIG. 1 , appropriately programmed in accordance with this specification, can perform the process  200 . 
     The system generates an initial output video including a plurality of frames (step  202 ). Each of the frames has a plurality of channels. Each channel is a two-dimensional image and indexed by a respective channel index from a set of channel indices of the initial output video. For each channel, each pixel in the channel is assigned a predetermined pixel value or is padded with a blank pixel. 
     The system identifies a partitioning of the initial output video into a set of channel slices that are indexed according to a particular slice order (step  204 ). Each channel slice is a down sampling of a channel stack from a set of channel stacks. Each channel stack in the set of channel stacks corresponds to a respective channel index (e.g., channel index R, G or B) and is a stack of channels having the same respective channel index according to time. 
     In particular, the system evenly divides the initial output video with shape (T, H, W) into a set of smaller channel slices by using a given subscale factor s=(s t , s h ,s w ). The subscale factor s divides the video  132  into s=(s t ·s h ·s w ) channel slices, each of resolution (T/s t ,H/s h ,W/s w ), as illustrated in the bottom of  FIG. 1 . The system generates these channel slices in a particular order. For example, the system generates channel slices according to their respective offsets such that it first generates slice χ (0,0,0)  (slice  134  which a down sampling of a channel stack corresponding to channel index Y for yellow), then χ (0,0,1)  (slice  136  which is a down sampling of a channel stack corresponding to channel index G for green), χ (0,1,0)  (slice  138  which is a down sampling of a channel stack corresponding to channel index R for red). χ (1,0,0)  (slice  140  which is a down sampling of a channel stack corresponding to channel index B for blue), up until slice χ (s     t     −1,s     h     −1,s     w     −1) . Generating all channel slices one at a time in this way can drastically reduce the number of pixels N p  in memory to N p /s, which enables scaling the architecture of the system by a factor of s. 
     The system initializes, for each channel stack in the set of channel stacks, a set of fully-generated channel slices (step  206 ). For example, the system may initialize the set of fully-generated channel slices as an empty set, which means there is no fully-generated channel slice in the set at the beginning. 
     The system repeatedly performs steps  208 - 212  according to the particular slice order. 
     The system processes, using an encoder neural network, a current output video comprising the current set of fully-generated channel slices of all channel stacks to generate an encoded conditioning channel slice (step  208 ). 
     The system processes, using a decoder neural network, the encoded conditioning channel slice to generate a next fully-generated channel slice (step  210 ). 
     The system adds the next fully generated channel slice to the current set of fully-generated channel slices of the channel stack (step  212 ). 
     After all channel slices have been fully generated, the system generates, for each of the channel indices, a respective fully-generated channel stack using the respective fully generated channel slices (step  214 ). 
     The system generates a fully-generated output video using the fully-generated channel stacks generated for the channel indices (step  216 ). 
       FIG. 3  is a flow diagram of an example process for generating an encoded conditioning channel slice for a current channel slice (denoted as χ (a,b,c) ) using an encoder neural network, for example, the encoder  102  of  FIG. 1 . 
     The encoder neural network (hereafter referred to as “the encoder” for simplicity) generates a partially masked video using the current set of fully generated channel slices of all channel stacks, in which only pixels from the current set of fully generated channel slices (i.e., only the pixels of preceding fully generated channel slices χ &lt;(a,b,c) ) are visible in the partially masked video (step  302 ). 
     The encoder generates an embedded partially masked video by concatenating the partially masked video with one-hot encodings of pixel intensities of all channels (step  304 ). The pixel intensities of each channel can be discretized pixel intensities. 
     The encoder processes the embedded partially masked video using the 3D encoder convolution layer to generate a down-sampled encoded video (step  306 ). For example, the encoder processes the embedded partially masked video using a 3D encoder convolution layer with kernel size k=(k 1 , k 2 , k 3 ) and stride s (the subscale factor) to generate a down-sampled encoded video of resolution (T/s t , H/s h , W/s w ). 
     The encoder applies convolution padding on the down-sampled encoded video to generate a padded down-sampled encoded video (step  308 ). The encoder applies convolution padding depending on the current slice index (a, b, c). In particular, the encoder can pad the down-sampled encoded video with (└k 1 /2┘−a, └k 2 /2┘−b, └k 3 /2┘−c), which “centers” the convolution kernel on the pixels of the current channel slice. 
     The encoder appends positional embeddings to the padded down-sampled encoded video to generate an initial encoder representation (step  310 ). More particularly, the encoder adds positional embeddings for each axis of the padded down-sampled encoded video, as well as embeddings for the current slice index (a, b, c), to the padded down-sampled encoded video. The result is the initial encoder representation 
         (a,b,c)   0 ∈   T/s     t     ×H/s     h     ×W/s     w     ×d     c   , where d e  is the embedding size. Optionally, the encoder can condition on auxiliary information (e.g., per-frame action values of a robot arm) by concatenating this auxiliary information to the initial encoder representation. 
     The encoder transforms the initial encoder representation to an intermediate encoder representation using a linear projection to a hidden size d(step  312 ). 
     The encoder processes the intermediate encoder representation using a stack of L encoding self-attention layers to generate the encoded conditioning channel slice (step  314 ), denoted as    (a,b,c)   L , which is then used as conditional input to a decoder neural network that generates values for the pixels of the current slice χ(a,b,c) . 
     In particular, each of the L encoding self-attention layers includes an encoder layer-norm layer, a block self-attention layer, and one or more encoder feed-forward layers. The one or more encoder feed-forward layers may be, for example, a multilayer perceptron (MLP). 
     Each encoding self-attention layer in a plurality encoding self-attention layers is configured to receive as input a padded video of shape (T, H, W) that includes a set of channel stacks. Each encoding self-attention layer then divides the padded video into a set of smaller video blocks of shape (t, h, w) of length n p =t, h, w. Then each encoding self-attention layer applies a self-attention (or block-local self-attention) mechanism on each of the video blocks independently. Given a block representation  ∈   n     p     ×d  of hidden size d as input, this amounts to: 
       [ q,k ,ν]=layernorm( ) W   qke    q,k,ν∈     n     p     ×d     a     , W   qkv ∈   d×3d     a   ,  (2)
 
         A =softmax( qk   T /√{square root over ( d   a   +B )})  A,B∈     n     p     ×d     a   ,  (3)
 
       attention( )= Aν.   (4)
 
     The attention mechanism can be applied to all video blocks in parallel. The input is first projected to query, key and value representations q, k, ν by the encoder layer-norm layer of the encoding self-attention layer (Eq. 2). An attention matrix A is then formed as the scaled dot-product between all query-key pairs qk adding a relative position bias B (Eq. 3). A bias B ij  is defined as the sum of per-dimension relative distance biases between element i and j, along each of the time- and spatial dimensions. Finally, the values ν are aggregated with respect to the attention weights of the attention matrix A to generate a self-attended output attention( ) (Eq. 4). It is noted that running block-local self-attention is very efficient in practice as the cost of splitting videos into block is negligible. 
     The encoding self-attention layer appends self-attended outputs of the set of video blocks to form a multi-headed self-attended output. In particular, the encoding self-attention layer concatenates the output of n a  parallel attention heads in the encoding self-attention layer and project the result by a linear transformation (Eq. 5) before applying a residual connection to form the multi-headed self-attended output . 
         =[attention 1 ( ); . . . ; attention n     a   ( )] W, +     W   p ∈   (n     a     ·d     a     )×d ,  (5)
 
     Finally, the encoding self-attention layer processes the multi-headed self-attended output using a final fully-connected neural network layer to generate an output of the block self-attention layer. For example, the multi-headed self-attended output  is passed thug a rectified linear unit (ReLU) activation, followed by a linear transformation and a residual connection as shown in Eq. 6: 
         ′=relu(layernorm( ) T   1 ) T   2   +     T   1   ,T   2 ∈   d×d ,  (6)
 
     where overloading notion, attention( ) denotes the blockwise application of self-attention to  , Applying layer normalization before each block, rather than after each block can improve training. 
     In some implementations, to avoid the need for overlaps to connect pixels across blocks, the block sizes can be varied between encoding self-attention layers, which is highly efficient and works well in practice. 
       FIG. 4  is a flow diagram of an example process for generating, using a decoder neural network, a next fully generated channel slice from an encoded conditioning channel slice provided by an encoder neural network. The decoder neural network can be, for example, the decoder neural network  118  of  FIG. 1 . The decoder neural network (also referred to as “the decoder” for simplicity) can be almost identical to the encoder neural network in structure, except for the use of masking in the decoder as defined by the particular order for generating fully-generated channel slices. 
     Generally, the decoder is configured to generate pixel values of the current slice χ (a,b,c)  conditioned on the encoded conditioning channel slice   (a,b,c)   L . Specifically, the decoder embeds the encoded conditioning channel slice by summing all channel embeddings of size d e  at every pixel to generate an embedded channel slice (step  402 ). 
     The decoder applies a masked convolution (for example, a 3×3×3 masked convolution) on the embedded channel slice, using the 3D decoder convolution layer, to generate a first masked channel slice (step  404 ). 
     The decoder adds positional embeddings to the first masked channel slice to generate an initial decoder channel representation (step  406 ). The initial decoder channel representation can be denoted as    (a,b,c)   0 ∈   T/s     t     ×H/s     h     ×W/s     w     ×d  where d is the embedding size. 
     The decoder adds the encoded conditioning channel slice to the initial decoder channel representation to generate a second masked channel slice (step  408 ). In particular, the decoder adds a linear projection of    (a,b,c)   L  to    (a,b,c)   0  to generate the second masked channel slice. 
     The decoder processes the second masked channel slice using a stack of L decoding self-attention layers with masking to generate pixel values for the current channel slice. The L decoding self-attention layers operate in the same manner as the L encoding self-attention layers as described above with reference to  FIG. 3 , except for the use of masking. In particular, the decoding self-attention layers are masked so that they do not attend over pixels in the channel slice that have not yet been generated. After all pixel values of the current channel slice are generated, the current channel slice becomes the next fully-generated channel slice, which can be denoted as    (a,b,c)   L  (step  410 ). The next fully-generated channel slice is then added to the set of fully-generated channel slices of the respective channel stack. 
     In some implementations, the system  100  of  FIG. 1  can predict per-pixel channel intensities χ (a.b,c)   k  (the channel slice index (a, b, c) is omitted in the following) for each channel k&lt;N c  by the encoder feed-forward layers  112  and decoder feed-forward layers  126  (e.g., MLPs) with a single hidden layer (Eq. 8), conditioned on the final decoder representation  L∈   n     p     ×d . The final decoder representation is itself conditioned on    (a,b,c)   L  and hence on prior slices χ &lt;(a,b,c)  as well as the preceding channels (χ j ) j=1 . . . k−1  for each pixel, encoded as one-hot vectors. 
         u   k =[layernorm(   L );onehot(χ 1 ); . . . ; onehot(χ k−1 )] U   k ,  (7)
 
         p (χ i   k |χ i   &lt;k ,χ &lt;i )=softmax(relu( u   i   k ) P ),
 
         P∈     d×N     u     , U   k ∈   (d+(k−1)·N     u     )×d ,  (8)
 
     The per video slice loss is defined as the negative log-likelihood as follows: 
     
       
         
           
             
               
                 
                   
                     ℒ 
                     ⁡ 
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     - 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           0 
                         
                         
                           
                             n 
                             p 
                           
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             k 
                             = 
                             0 
                           
                           
                             
                               N 
                               c 
                             
                             - 
                             1 
                           
                         
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           ln 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               p 
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     
                                       x 
                                       i 
                                       k 
                                     
                                     ❘ 
                                     
                                       x 
                                       i 
                                       
                                         &lt; 
                                         k 
                                       
                                     
                                   
                                   , 
                                   
                                     x 
                                     
                                       &lt; 
                                       i 
                                     
                                   
                                 
                                 ) 
                               
                             
                             . 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     This loss can be used to jointly train the encoder neural network and the decoder neural network. 
     This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. 
     Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal. 
     The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. 
     A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. 
     As used in this specification, an “engine,” or “software engine,” refers to a software implemented input/output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices. 
     The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). For example, the processes and logic flows can be performed by and apparatus can also be implemented as a graphics processing unit (GPU). 
     Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. 
     Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks. e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. 
     To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user, for example, by sending web pages to a web browser on a user&#39;s client device in response to requests received from the web browser. 
     Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet. 
     The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. 
     While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. 
     Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. 
     Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.