Patent Publication Number: US-2021193259-A1

Title: Method, device, and computer program for generating protein sequences with autoregressive neural networks

Description:
FIELD OF THE INVENTION 
     The invention generally relates to the field of protein sequences and of generating protein sequences. More particularly, the invention concerns a method, a device, and a computer program for generating protein sequences with autoregressive neural networks. 
     BACKGROUND OF THE INVENTION 
     Proteins are versatile molecules consisting of chains of amino acids which fold in 3D space to form molecular machines able to bind specific substrates, self-organize in larger structures, catalyze chemical reactions, and convert various sources of energy such as light and chemical gradients into chemical bonds or work, and more. 
     Thanks to their biological activity, recombinant proteins have found uses in medical applications including vaccines, antibodies, and growth factors. Their catalytic potential is also used in diverse industries ranging from paper, biofuel, food and detergent to chemical synthesis. 
     In many cases, it is desirable to modify the properties of the natural proteins used in these applications to improve stability, increase catalytic activity, modify substrate preferences etc. 
     As described in the article entitled “ Methods for the directed evolution of proteins ”, Packer M S et al., Nat. Rev. Genet., 2015, 16, 379-394, this can be achieved through the generation of more or less random mutations in the protein sequence and selection for the best performing variants. According to other solutions, when knowledge of the protein structure is available, computer aided rational design can help identify interesting modifications as described in the paper entitled “ Algorithms for protein design ”, Gainza P et al., Curr. Opin. Struct. Biol., 2016, 39, 16-26. Still according to other solutions, proteins can be designed from scratch as illustrated in the paper entitled “ The coming of age of de novo protein design ”, Huang P S et al., Nature, 2016, 537, 320-327. 
     However the systematic exploration of protein variants is made extremely challenging by the enormous space of possible sequences and the difficulty of accurately predicting protein fold and function. Indeed, proteins are composed of 20 different amino acids forming chains ranging from just a few, to thousands of amino acids. Accordingly, there are 20 300  possible proteins of length 300 (which corresponds to an average sized protein), i.e. vastly more than the number of atoms in the universe and what has been explored by the natural evolutionary process. 
     Accordingly, there is a need to explore the protein sequence space in a smart way to obtain interesting functional variants. 
     SUMMARY OF THE INVENTION 
     Faced with these constraints and limitations, the inventors provide a method, a device, and a computer program for generating protein sequences, i.e. ordered sequences of amino acids, with autoregressive neural networks. 
     It is a broad object of the invention to remedy the shortcomings of the prior art as described above. 
     According to a first aspect of the invention there is provided a computer method for generating a protein sequence in an autoregressive neural network comprising an encoder and a decoder, the decoder comprising an autoregressive module, the method comprising the steps of:
         obtaining a latent code and inputting the latent code into the autoregressive module;   obtaining probabilities from the autoregressive module, the obtained probabilities representing probabilities for amino acids to be selected at given locations of the sequence; and   generating an ordered sequence of amino acids from the obtained probabilities,       wherein the autoregressive module is such that, for a location of the sequence, the probabilities associated with amino acids to be selected for the location in the sequence are determined as a function of probabilities of amino acids to be selected for previous locations in the sequence and   wherein the autoregressive neural network has been trained end-to-end with the encoder, the encoder making it possible to encode protein sequences in a latent space as latent codes.   

     In particular, the claimed method makes it possible to sample new proteins, in particular proteins having specific physical or functional properties, to obtain variants of a sequence while conserving structural and functional properties of the initial protein, and to obtain sequence variants with desired modified physical or functional properties. 
     The autoregressive neural network learns transformations preserving similarities in structures and relatedness between amino acids. It is able to modify physical properties of existing proteins and to generate new ones having chosen properties with relative accuracy. 
     According to embodiments, the method further comprises a step of obtaining additional information representative of at least one characteristic of a protein sequence to be generated, the additional information being input along with the latent code into the autoregressive module, additional information representative of characteristics of protein sequences used for training the autoregressive neural network being input along with protein sequences into the encoder during training. 
     According to embodiments, the additional information is directed to physical characteristics of the protein sequences to be generated and of the protein sequences used for training the autoregressive neural network. 
     According to embodiments, the decoder further comprises an up-sampling module for increasing a number of dimensions of a latent code, the obtained latent code being input into the up-sampling module and the up-sampled latent code being input into the autoregressive module. 
     According to embodiments, the step of generating an ordered sequence of amino acids comprises a step of selecting sequentially each amino acid of the ordered sequence and, after an amino acid is selected to belong to the ordered sequence, a step of checking the likelihood of usefulness of the ordered sequence at least partially generated. 
     According to embodiments, the step of checking the likelihood of usefulness of the ordered sequence at least partially generated comprises a step of comparing the length of the ordered sequence at least partially generated with a threshold. 
     According to embodiments, the step of generating an ordered sequence of amino acids comprises a step of selecting sequentially each amino acid of the ordered sequence and the step of generating an ordered sequence of amino acids is stopped if a specific symbol is selected as an amino acid for a given position of the ordered sequence. 
     According to embodiments, the step of generating an ordered sequence of amino acids comprises a step of selecting one amino acid, for a given position of the ordered sequence, among a plurality of amino acids, the one amino acid being selected as a function of the probabilities associated with amino acids of the plurality of amino acids for the given position. 
     According to embodiments, the method further comprises a step of modifying probabilities obtained from the autoregressive neural network, denoted the first autoregressive neural network, as a function of probabilities obtained from a second autoregressive neural network different from the first autoregressive neural network. 
     According to embodiments, the method further comprises a learning phase for determining parameters of the autoregressive module. 
     According to embodiments, the learning phase is unsupervised. 
     According to embodiments, the autoregressive neural network is of the variational auto-encoder type or of the adversarial auto-encoder type. 
     According to embodiments, the autoregressive neural network is a recurrent neural network. 
     According to a second aspect of the invention there is provided an apparatus comprising means configured for carrying out each step of the method described above. The advantages provided by the second aspect of the present invention are similar to the ones provided by the first above-mentioned aspect. 
     Since parts of the present invention can be implemented in software, parts of the present invention can be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium. A tangible carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid state memory device and the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g. a microwave or RF signal. 
     In an embodiment, the computer code exploits graphic processing units (GPUs) that allow parallel processing of large matrix data. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. 
       Further advantages of the present invention will become apparent upon examination of the drawings and detailed description. It is intended that any additional advantages be incorporated herein. 
       Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which: 
         FIG. 1 a    and  FIG. 1 b    illustrate the logical architecture of a variational auto-encoder and of a conditional variational auto-encoder, respectively; 
         FIG. 2 a    and  FIG. 2 b    illustrate an example of the logical architecture of a conditional variational auto-encoder having an autoregressive decoder, used during the training phase and for generating new protein sequences, respectively; 
         FIG. 3  illustrates an example of the functional architecture of the encoder illustrated in  FIG. 2   a;    
         FIG. 4 a    and  FIG. 4 b    illustrate an example of the functional architecture of the autoregressive decoder illustrated in  FIGS. 2 a  and 2 b   , during training and during generation, respectively; 
         FIG. 5 a    and  FIG. 5 b    illustrates an example of the functional architecture of the up-sampler and of network blocks of the autoregressive module, respectively, illustrated in  FIGS. 2 a , 2 b , 4 a   , and  4   b;    
         FIG. 6  illustrates an example of steps of a method for generating a protein sequence with an autoregressive neural network such as the conditional variational auto-encoder illustrated in  FIGS. 2 a  and 2 b    or a (conditional) adversarial auto-encoder; 
         FIG. 7  illustrates a conditional probability distribution table wherein the probability of each amino acid is computed as a function of the previous amino acids in the sequence and of the up-sampled latent vector; 
         FIG. 8  makes it possible to compare the distribution of amino acids in generated sequences with patterns of amino acid occurrences of the training dataset; 
         FIG. 9  illustrates preservation of secondary structures in reconstructions by showing alignments between one protein and its reconstructions; 
         FIG. 10  illustrates 3D structures of the two proteins as predicted by Phyre2 for the initial protein (A) and for one of its reconstructions presented in  FIG. 9  (B); 
         FIG. 11  illustrates a BLOSUM62 matrix and BLOSUM-like generated matrix; 
         FIGS. 12 to 16  illustrate generation of protein sequences when conditioned on solubility, protein length, protein length and weight, charge, and molar extinction coefficient, respectively; 
         FIG. 17  illustrates generation of variants of a protein with desired properties; 
         FIG. 18  illustrates the projection of the training dataset in the latent space of an unconditioned model, colored as a function of the protein length; 
         FIG. 19  illustrates the projection of 5 Interpro families in the latent space; 
         FIG. 20  is a block diagram illustrating components of a processing device in which embodiments of the invention may be at least partially implemented; 
         FIG. 21  illustrates an example of the logical architecture of a conditional adversarial auto-encoder having an autoregressive decoder; 
         FIG. 22 a    and  FIG. 22 b    illustrate an example of the functional architecture of the encoder and of the discriminator illustrated in  FIG. 21 , respectively. 
         FIG. 23  illustrates the bioluminescence of LuxA variants relative to the wild-type LuxA protein as a function of the number of amino-acid substitutions (points at the bottom of the graph are below the limit of detection); 
         FIG. 24  illustrates the protein reconstruction accuracies for the neural network architectures “Non AR”, “No up-sampling”, “Reduced z” and “Up-sampling+AR”; 
         FIG. 25  illustrates the aligned secondary structures of sequences selected at random from training set. For ease of visualisation, the proteins are first aligned using mafft, and positions in the alignment with more than 50% gaps are removed. Y-axis coordinates refer to a sequence and X-axis to amino acid positions in the sequence. Patterns along each sequence are indicated with the following color code: Alpha helix in dark blue, Coil in light blue, Gap in orange and Beta strand in red; 
         FIG. 26  illustrates the aligned secondary structures of sequences generated by sampling from prior of the “Non AR”, “No up-sampling”, “Reduced z” and “Up-sampling+AR” models. For ease of visualisation, the proteins are first aligned using mafft, and positions in the alignment with more than 50% gaps are removed. Y-axis coordinates refer to a sequence and X-axis to amino acid positions in the sequence. Patterns along each sequence are indicated with the following color code: Alpha helix in dark blue, Coil in light blue, Gap in orange and Beta strand in red;  FIG. 27  illustrates the percentage of HMM hits for samples from the “Non AR”, “No up-sampling”, “Reduced z” and “Up-sampling+AR” models; and 
         FIG. 28  illustrates the evolution of KL loss term during training for the “Upsampling+AR (KL annealed)”, “RNN (KL annealed)” and “RNN” models. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION 
     According to the invention, a generative deep neural model is used to generate protein sequences. According to embodiments, the generative deep neural model is an autoregressive neural network such as a recurrent neural network (RNN), an autoregressive convolutional neural network (autoregressive CNN), for example a variational auto-encoder (VAE) or a conditional variational auto-encoder (CVAE), or an adversarial auto-encoder (AAE) or a conditional adversarial auto-encoder (CAAE). 
     Still according to embodiments, the generative deep neural model comprises a probabilistic encoder and a probabilistic decoder of the variational auto-encoder type, wherein the decoder is an autoregressive decoder that may comprise an up-sampler. It has proven extremely efficient at learning high level features of complex datasets in an unsupervised fashion, and at generating novel samples with similar properties to the data they are trained on. 
     During training, samples of the learning dataset are encoded into latent vectors (or latent codes or latent representations) using the encoder. The latent vectors are then used to train the decoder to predict the next element in the sequence being constructed, based on previous elements of this sequence. To generate new samples, latent vectors are used to stimulate the autoregressive sequential generation of a new sample, one amino acid at a time. 
     According to specific embodiments, conditioning vectors comprising additional information, also referred to as conditions, are given to the encoder during the training. Such conditions typically represent characteristics of protein sequences. In such a case, both the latent vectors and the conditioning vectors are then used to train the decoder to predict the next element in the sequence being constructed, based on previous elements of this sequence. Similarly, to generate new samples, latent vectors and conditioning vectors are used to stimulate the autoregressive sequential generation of a new sample one amino acid at a time. 
     Variational Auto-Encoder (VAE) 
     As described in the article “ Tutorial on variational autoencoders ”, Doersch C., 2016, variational auto-encoders are a deep learning technique for learning latent representations. They consist in a probabilistic encoder, a probabilistic decoder, and a loss function. The probabilistic encoder is a neural network that outputs a latent representation (hidden representation) from input data, for example from an image. The dimension of the latent space is much less than the one of the input space. Likewise, the probabilistic decoder is a neural network that outputs data, for example an image, from a latent representation. The dimension of the output space is typically the same as the one of the input space. 
     The latent space corresponds to latent variables that may be understood as unobserved variables that capture the most salient parts of the input data used for training the variational auto-encoder. They can be thought of as a compressed version of these data. 
       FIG. 1 a    and  FIG. 1 b    illustrate the logical architecture of a variational auto-encoder and of a conditional variational auto-encoder, respectively. 
     As illustrated in  FIG. 1   a,  variational auto-encoder  100  comprises a probabilistic encoder  105  and a probabilistic decoder  110 , referred to below as the encoder and the decoder, respectively. The input vector  115 , denoted x, is applied to encoder  105  to generate latent vector  120 , denoted z, that is in turn applied to decoder  110  to generate result  125 , denoted x′. After training, output x′ should be close to input x. As described above, the dimension of latent vector z is much less than that of input x and output x′. 
     As illustrated in  FIG. 1  b, a main difference between a variational auto-encoder and a conditional variational auto-encoder is that additional information  130  or conditions, denoted c, is input into the encoder in addition to input vector x and in the decoder in addition to the latent representation z 
     In a variational auto-encoder, the encoder and the decoder are jointly trained to reconstruct the sample the encoder takes as an input. During learning, the parameters of the variational auto-encoder are updated end-to-end using, for example, backpropagation with stochastic gradient descent to minimize a loss function, as described, in particular, in the article entitled “ Stochastic Backpropagation and Approximate Inference in Deep Generative Models ”, Rezende D J et al., 2014, arXiv:1401.4082v3 (see e.g. § 3), as well as in the article “ Tutorial on Variational Autoencoders ”, Carl Doersch, 2016, arXiv:1606.05908v2 (see e.g. § 2.2, equation 10, and the right panel of  FIG. 4 ). 
     The encoder learns a distribution over the latent representation z (which typically has a reduced number of dimensions compared to the input x, as mentioned above). This leads to an information bottleneck in the variational auto-encoders, forcing them to learn an identity function in a non-trivial way and to capture high level patterns about the data. The decoder learns to predict the sample that is the most likely to be expressed by a latent code. In addition to being trained to reconstruct their inputs, variational auto-encoders are forced to learn latent representations in a space that can be decoded into valid samples. This is achieved by imposing a prior (p(z)) over the distribution of the latent representation (p(z/x)). The encoder learns the parameters of a distribution (usually normal) that is then sampled and decoded. In addition, a penalty term is used to encourage all areas of the latent space to be decoded into valid samples. The resulting model has the advantage of learning a useful latent representation of the data. 
     For the sake of illustration, when trained on faces, variational auto-encoders represent interpretable high level features such as hair color, having glasses or wearing a hat. 
     This makes it possible to condition generated examples by smart sampling of the latent space, and to perform continuous interpolations and arithmetic operations on latent vectors. 
     When annotations (i.e. additional information or conditions) of the training data are available, it is possible to condition the learned representation on high level features by providing this information during training to the encoder and the decoder as illustrated in  FIG. 1 b   . Conditioning vector c can be seen as additional dimensions in the latent code that are not learned from the data but provided as inputs. After training, providing a latent vector and chosen conditions makes it possible to generate samples with the desired properties. 
     Variational Auto-Encoder Having an Autoregressive Decoder 
     Although variational auto-encoders effectively generate samples that capture the global structure of the data, these samples tend to be blurry. The inventors have observed that this results from each point in a generated sample being not conditioned on the other points but being only inferred from the latent space. To cope with this issue, embodiments of the invention use autoregressive decoders, for example an autoregressive decoder, in a variational auto-encoder. 
     Autoregressive models are used to predict the next element in a sequence, given the previous values in this sequence. The joint data distribution is modeled as a product of conditionals, as follows: 
     
       
         
           
             
               p 
                
               
                 ( 
                 X 
                 ) 
               
             
             = 
             
               
                 ∏ 
                 
                   i 
                   = 
                   1 
                 
                 N 
               
                
               
                   
               
                
               
                 p 
                  
                 
                   ( 
                   
                     
                       
                         x 
                         
                           i 
                           + 
                           1 
                         
                       
                        
                       
                         | 
                       
                        
                       
                         x 
                         1 
                       
                     
                     , 
                     … 
                      
                     
                         
                     
                     , 
                     
                       x 
                       o 
                     
                   
                   ) 
                 
               
             
           
         
       
     
       FIG. 2 a    and  FIG. 2 b    illustrate an example of the logical architecture of a conditional variational auto-encoder having an autoregressive decoder, used during the training phase and for generating new protein sequences, respectively. 
     As illustrated in  FIG. 2 a   , conditional variational auto-encoder  200  comprises a probabilistic encoder  205  and a probabilistic autoregressive decoder  210 , referred to as the encoder and the decoder, respectively, in the following. The input vector  215 , denoted x, as well as additional information  220  or conditions, denoted c, are applied to encoder  205  to generate latent vector  225 , denoted z. Encoder  205  may be similar to encoder  105  described in reference to  FIG. 1   b.  An example of encoder  205  is illustrated in  FIG. 3 . 
     For the sake of illustration, additional information  220  may comprise the length of the protein, its charge, its isoelectric point, its molar extinction coefficient, its solubility (e.g. probability of expression in  E.coli  inclusion bodies: the smaller this quantity, the more soluble the protein is), and/or its molecular weight, it being observed that some of these values may be correlated (for example, the molecular weight of a protein is highly correlated with its length). 
     According to the illustrated example, latent vector  225  and additional information  220  are then applied to up-sampler  230  to generate up-sampled latent vector  235 , denoted z′. Latent representation z is up-sampled in up-sampler  230  while being conditioned by additional information c to generate latent representation z′. The obtained up-sampled latent vector  235 , i.e. latent representation z′, and input vector x are then input in autoregressive module  240  to generated result  245 , denoted x′. 
     It is to be noted that although an up-sampling module is used in most of the illustrated examples, it is not requested in all embodiments. For example, the size of the latent code z may be increased by concatenation to reach the one of input vector x. 
     A more detailed example of autoregressive decoder  210  is described by reference to  FIGS. 4 a , 4 b , 5 a   , and  5   b.    
     It is to be noted that according to particular embodiments, no additional information is fed into encoder  205  and up-sampler  230 . 
     As illustrated in  FIG. 2 b   , only the decoder, comprising up-sampler  230  and autoregressive module  240 , is used for generating a protein sequence from a latent vector, additional information (conditions), and previously determined amino acids of the sequence being generated (x). 
     According to embodiments, encoder  205  and autoregressive decoder  210  comprise convolutional layers, with strided convolutions for down-sampling in the encoder and transposed convolution for up-sampling in the decoder. The kernel size of the convolutional layers is preferably set to two. 
     Still according to embodiments, all convolutional layers are followed by batch normalization and a parametric rectified linear unit (PReLU) activation (unless otherwise specified, in particular in description of  FIGS. 3, 4   a ,  4   b ,  5   a  and  5   b ). Indeed, batch normalization has been very effective at improving both the training speed and the reconstruction performance of the network. Dense layers are preferably followed by PReLU activation, except those that output the parameters of the Gaussian distribution learned by the encoder μ(x,c) (no activation) and σ(x,c) (softplus activation). Activations and batch normalization are not shown in  FIGS. 3, 4   a ,  4   b ,  5   a , and  5   b  to make them easier to read. 
     As described by reference to  FIG. 5 b   , autoregressive decoder  210  makes use of gated activation functions, for example the gated activation functions described in “ WaveNet: A Generative Model for Raw Audio ”, van den Oord A et al., 2016, arXiv:1609.03499v2 (see e.g. § 2.3, equation 2). Feature maps of convolutional layers may be split in two parts and two different activation functions may be applied on each half, followed by elementwise multiplication. The result is convolved upon and passed to the next stack, as well as kept to be used in skip connections. 
       FIG. 3  illustrates an example of the functional architecture of encoder  205  illustrated in  FIG. 2 a   . It is to be noted that other functional architectures may be implemented. 
     As illustrated, input vector x is fed in five successive 1D convolution layers referenced  300 - 1  to  300 - 5  (for the sake of clarity only two of them are represented). According to embodiments, the stride of these convolution layers is set to two, except for the first one which is set to one. The output of 1D convolution layer  300 - 5  is flattened (reference  305 ) so as to suppress one dimension. The convolution layers aim at establishing relations between parts of protein sequences. 
     In parallel, additional information or conditions c are fed into two successive dense layers  310 - 1  and  310 - 2 , i.e. into two fully connected neural networks that aim at identifying what should be in protein sequences. 
     The output of flattened layer  305  and the output of dense layer  310 - 2  are then concatenated (reference  315 ). As illustrated, the result of the concatenation is fed into two successive dense layers  320 - 1  and  320 - 2 , the result of the latter being fed into dense layers  325  and  330 . Again, these fully connected neural networks aim at identifying what should be in protein sequences, taking into account the conditions. 
     The output of dense layer  330  is then used as input of a stochastic layer comprising an activation function  335 , for example the softplus activation function, the square root function  340 , and the application of a Gaussian function (references  345  and  350 ). 
     The output of the stochastic layer is added (reference  355 ) to the output of dense layer  325  to create latent representation z 
     It is thus observed that the output of the encoder being stochastic (a Gaussian parametrized by the neural network), it has an additional input: random standard Gaussian noise. This makes it possible to backpropagate the gradients of the loss through the stochastic layer. 
     It is also to be noted that an encoder similar to the one illustrated in  FIG. 3  may be used when no additional information is used. In such a case, dense layers  310 - 1  and  310 - 2  are not implemented (as well as concatenation function  315 ). 
       FIG. 4 a    and  FIG. 4 b    illustrate an example of the functional architecture of autoregressive decoder  210  illustrated in  FIGS. 2 a  and 2 b   , during training and during generation, respectively. Again, it is to be noted that other functional architectures may be implemented. 
     As described in reference to  FIGS. 2 a  and 2 b    and according to the given example, autoregressive decoder  210  comprises up-sampler  230  and autoregressive module  240 . 
     Up-sampler  230  generates up-sampled latent vector z′ from latent vector z and additional information c. An example of the functional architecture of up-sampler  230  is described in reference with  FIG. 5 a   . Again, it is to be noted that up-sampler  230  is not required for carrying out the invention. 
     Up-sampled latent vector z′ is fed into a set of network blocks, for example a set of K network blocks of the WaveNet type, as illustrated. Such a type of network block is described, for example, in the paper entitled “ WaveNet: A Generative Model for Raw Audio ”, van den Oord A et al., 2016, arXiv:1609.03499v2 (see e.g. § 2.4). 
     As illustrated in  FIG. 4 a   , during learning, input vector x is fed into the first network block of which the output is fed into the second network block and so on until the last one. As described above, input vector x represents the input protein sequence and output vector x′ its reconstruction. 
     When generating a protein sequence from a latent representation and conditions, input vector x is initialized to zeros and previously determined amino acids of the sequence being generated (x′) are fed into the first network block of which the output is fed into the second network block and so on until the last one, as illustrated in  FIG. 4 b   . Sequentially, a reconstructed protein sequence is obtained by predicting the next amino acid in the sequence, as described in reference to Algorithm 1 given in the Appendix. At each step, an amino acid is predicted, added to input vector x, so that the next one can be predicted. 
     As described by reference to  FIG. 5 b   , the output of network block k is transmitted to network block k+1 and is kept to be added later to other residual connections. 
     Turning back to  FIGS. 4 a  and 4 b   , the outputs of network blocks  400 - k  are combined (reference  405 ) and filtered in convolution layer  410  that output is normalized (reference  415 ), for example using the softmax function, to generate output x′. An example of such an output is described by reference to  FIG. 7 . 
       FIG. 5 a    illustrates an example of the functional architecture of up-sampler  230  illustrated in  FIGS. 2 a , 2 b , 4 a , and 4 b   . Again, it is to be noted that other functional architectures may be implemented. 
     The up-sampler comprises two parts whose results are concatenated. 
     As illustrated, latent vector z, which may be obtained from the decoder during training or that may be obtained from another way, typically from a user, for generating new protein sequences, is fed into dense layer  500  to be up-sampled by a fully connected artificial neural network. The output of the latter is reshaped (reference  505 ) so as to add one dimension before being fed into four successive 1D deconvolution layers referenced  510 - 1  to  510 - 4  (for the sake of clarity only two of them are represented). The deconvolution layers add information about where something should be in the protein sequence while the fully connected neural network adds information about what should be in the protein sequence. 
     It is to be noted that a 1D deconvolution layer is to a 2D deconvolution layer the same as what a 1D convolution layer is to a 2D convolution layer. In other words, it is a 2D deconvolution layer applied on a 1D input to which one additional dimension has been added. For example, a tensor of shape (75, 21) is reshaped into (75, 1, 21) before being passed in the 2D deconvolution layer, thus becoming (150, 1, 21), which is finally reshaped into (150, 21). 
     In parallel, latent vector z is concatenated (reference  515 ) with additional information or conditions c. The result of this concatenation is fed into dense layer  520  the output of which being reshaped (reference  525 ) so as to add one dimension. Again, the fully connected neural network adds information about what should be in the protein sequence, taking into account the conditions. 
     Next, the output of 1D deconvolution layer  510 - 4  and of dense layer  520  (after it has been reshaped, reference  525 ) are concatenated (reference  530 ) to generate up-sampled latent vector z′. 
       FIG. 5 b    illustrates an example of the functional architecture of the network blocks of autoregressive module  240  illustrated in  FIGS. 2 a , 2 b , 4 a , and 4 b   . Again, it is to be noted that other functional architectures may be implemented. For the sake of illustration, only one network block ( 400 - k ) is illustrated. 
     Up-sampled latent vector z′, obtained from up-sampler  230 , is fed into convolution layer  550 . 
     In parallel, the (k−1) first values of the input vector x (during training) or of the protein sequence being generated, denoted x (k) , are fed into two successive causal convolution layers  555 - 1  and  555 - 2 , the dilation factor of the causal convolution being, for example, doubled each time so that the receptive field of the autoregressive model grows exponentially with the number of WaveNet-like stacks. After three stacks, a value that is advantageously used, the receptive field is 32 (each of the three network block comprising two blocks of causal convolution, which means six causal convolutional layers). 
     The output of convolution layer  550  and that of causal convolution layer  555 - 2  are added (reference  560 ) and an activation function is applied on the result (reference  565 ). For the sake of illustration, σ represents the sigmoid activation function. A gated activation may be used, for example the one described in the paper entitled “ WaveNet: A Generative Model for Raw Audio ”, van den Oord A et al., 2016, arXiv:1609.03499v2 (see e.g. § 2.3, equation 2). 
     The output of the activation function is then combined (reference  570 ) with the addition of the output of convolution layer  550  and of causal convolution layer  555 - 2  and the obtained result is fed into convolution layer  575  to update the protein sequence being reconstructed or generated with the k th  prediction. 
     The output of convolution layer  575  (i.e. x (k+1) ) is both passed to the (k+1) th  network block (i.e. network block  400 -( k+ 1)) and kept to be added later to the other residual connections. As described in the paper entitled “ WaveNet: A Generative Model for Raw Audio ”, van den Oord A et al., 2016, arXiv:1609.03499v2 (see e.g. § 2.4), these residual connections reduce the vanishing gradient problem, making training easier. 
     The output of network blocks  400 - 1  to  400 -K are combined (reference  405 ) and filtered in convolution layer  410  of which the output is normalized (reference  415 ), for example using the softmax function, to generate output x′. 
     Algorithm 1 given in the appendix illustrates an example of steps for generating protein sequences. As illustrated, each amino acid is determined one after the other in a loop the size of which may be determined by the length of the protein to be generated. At each iteration, a probability is determined for each amino acid to be the one used to generate the protein sequence. As described above, it is based on a latent code, on conditions, and on the probabilities associated with each amino acid in the protein sequence for the positions preceding the one that is processed. In this example, the amino acid that is selected, at a given position, is the one of higher probability. Other methods may be used. For example, sequence variants may be generated by choosing amino acids that do not have the highest probability but, for instance, choosing the second most probable amino acid. 
     Turning back to  FIG. 2 a   , variational auto-encoder  200  may be trained by minimizing the variational lower bound, as described in the paper entitled “Auto-Encoding Variational Bayes”, Kingma D P et al., 2013, arXiv:1312.6114v10 (see e.g. § 2, in particular § 2.2. to § 2.4). Accordingly, the loss function preferably comprises a cross-entropy term and a Kullback-Leibler divergence term. The cross-entropy is minimum when the model predicts the right amino acid with a probability of 1 and gives a probability of 0 to all other amino acids. The Kullback-Leibler divergence is a regularization-like term that is minimum when the data projected onto the latent space follows a standard normal distribution. According to particular embodiments, the loss function is minimized using the Adam optimization algorithm mentioned above, with default parameters for 50 epochs. 
       FIG. 6  illustrates an example of steps of a method for generating a protein sequence with an autoregressive neural network such as the conditional variational auto-encoder illustrated in  FIGS. 2 a  and 2 b    or a (conditional) adversarial auto-encoder. 
     As illustrated, a first step (step  600 ) is directed to obtaining a training dataset comprising protein sequences and, preferably, associated conditions. This training set is used to train the autoregressive neural network (step  605 ) until a loss error reached a threshold denoted θ error (step  610 ). 
     Step  605  comprises steps  615  to  650  described hereafter. 
     After training, a latent code and, preferably, associated conditions are obtained (step  615 ) and an index i representing a location within the protein sequence to be generated is initialized to 1 (step  620 ). Likewise, other variables are initialized, such as the probability for each amino acid to be selected at a given position of the sequence being generated. 
     If needed (as suggested with dotted line), the obtained latent code and conditions (if any) are up-sampled (step  625 ). 
     Then, the latent code and conditions (if any) or the up-sampled latent code and conditions (if any) as well as the probabilities previously obtained (i.e. if index i is greater than one) are input in the autoregressive module (step  630 ) for obtaining probabilities for amino acids to be selected at position i in the generated sequence (step  635 ). 
     Next, index i is incremented by one (step  640 ) and a test is carried out to determine whether or not the value of index i has reached the length of the protein sequence to be generated (step  645 ), i.e. to determine whether or not probabilities have been obtained for each location of the protein sequence to be generated, or whether or not another criterion is reached, indicating that the protein sequence has been generated. Such a criterion may be for example the selection of padding character instead of an amino acid for position i of the protein sequence. 
     If index i has not reached the length of the protein sequence to be generated or if the criterion is not reached, steps  630  to  645  are repeated. 
     On the contrary, if index i has reached the length of the protein sequence to be generated or if the criterion is reached, a protein sequence is generated as a function of the obtained probabilities (step  650 ). As mentioned above, for each location of the protein sequence, the amino acid associated with the higher probability may be selected. 
     Training such a neural network can be seen as unsupervised as the model is simply learning to reconstruct samples from the dataset while passing through an informational bottleneck. A variant of this training procedure is possible where in addition to performing reconstruction, the model is also asked to predict certain properties related to the protein. An error term reflecting how well these predictions are made can then be added to the loss function. Training such a model could be described as semi-supervised. 
     Generating Protein Sequences 
     The data used for training the variational auto-encoder  200  may be obtained from the databases known as InterPro and UniPro. 
     For the sake of illustration, all luciferase-like proteins (containing the Interpro domain IPR011251) having a length smaller than 600 amino acids may be chosen. There are about 72,000 of them. Some of these proteins have an easily testable activity: they emit light. Furthermore, the UniProt SPARQL endpoint may be queried to retrieve additional information about the sequences (such as Interpro families). The obtained sequences are advantageously padded with spaces at the end so that they all have the same length, for example a length of 600 amino acids. Still for the sake of illustration, the obtained sequences can be one-hot-encoded into matrices of shape 600×21 (20 amino acid, plus the padding character). Additional protein properties mays be obtained, for example they may be computed with the EMBOSS ‘pepstats’ software. 
     Accuracy 
     After being trained with such a luciferase-like protein dataset, variational auto-encoder  200  may predict the next amino acid in the sequence (based on the previous amino acids, the latent representation, and the conditions) with an 85.5% test accuracy. The variational auto-encoder being autoregressive, it learns quickly that after a padding character comes another one. It is almost 100% accurate at predicting padding characters and predicts the next amino acid in a sequence with a 78% accuracy. 
     A good accuracy of the reconstructions (i.e. protein sequence x′ generated by the decoder in an autoregressive fashion from projection z of protein x, given conditions c, as described by reference to Algorithm 1 and  FIG. 2 b   ) is also obtained. The decoder generates sequence x′ one amino acid at a time, the next amino acid in the sequence being predicted based on the previous predictions rather than on the input sequence (x). The accuracy of reconstructions is 82.5% overall, and 75% on the amino acid sequence (excluding padding). It is to be noted that these accuracies are lower than the test accuracy mentioned above, as errors accumulate during the reconstruction process. 
       FIG. 7  and table 1 in the Appendix illustrate protein reconstruction. 
       FIG. 7  illustrates a conditional probability distribution table wherein the probability of each amino acid is computed as a function of the previous amino acids and of the up-sampled latent vector z′. As illustrated, the most likely amino acid is added to the sequence which is constructed one amino acid at a time. For example, amino acid G (glycine) in the eighteenth position of the protein sequence is the one having the higher probability. Accordingly, it is selected as belonging to the protein sequence represented on top of the conditional probability distribution table. 
     Table 1 in the Appendix illustrates the alignments between a protein sequence used as input (denoted init) and the corresponding reconstructed protein sequence (denoted recons). In this example, the protein sequence that is used as input is Pyrimidine monooxygenase (RutA, UniProt ID: A1JMY1). (*) indicate identical amino acids, (:) very similar amino acids, and (.) similar amino acids. 
     Validity of Generated Protein Sequences 
     It is to be noted that to encode valid proteins, generated sequences should start with a methionine (M) (as all proteins from the dataset do) and must not have padding characters between amino acids. Out of 80,000 sequences randomly sampled from the latent space, 99.6% were valid. After having obtained these sequences, it has been assessed whether generated proteins would be found to have significant homologies to proteins present in the dataset. To that end, the HMM profile of the bacterial luciferase-like PFAM family (PF00296) have been used to identify generated proteins. As a result, 30% of the generated sequences were recognized as belonging to this family (E-value&lt;0.1). 
     Table 2 in the Appendix illustrates alignment between a generated protein randomly sampled from the latent space and the most similar known sequence (glucose-6-phosphate dehydrogenase, F420-dependent, UniProt ID: D9VBX0). The latter is identified with HMMER. Again, (*) indicate identical amino acids, (:) very similar amino acids, and (.) similar amino acids. 
     During training, the Kullback-Leibler divergence term pushes the data projected onto the latent space to follow a standard normal distribution. However, this is not perfect and some regions of the latent space are left “empty”. When sampling the latent space using a normal distribution to generate new proteins, points in the latent space that are too distant from actual projections might not be well decoded, explaining why a substantial number of the sequences generated above are not recognized as part of the PF00296 family. In contrast, reconstructions from projected points are almost always valid. As an example, 10,000 proteins from the dataset have been projected and reconstructed and it has been found that 95% of the reconstructed proteins were identified as part of the PF00296 family. 
     Distribution of N-Grams 
     It is observed that the distribution of amino acids in the generated sequences should match the natural distribution of amino acids in the dataset used for the training. As illustrated on graphs A and B in  FIG. 8 , the generative network is able to reproduce this distribution, but it tends to exaggerate it by predicting abundant amino acids such as Alanine (A) too frequently and rare amino acids such as Cysteine (C) not frequently enough. This is a common problem for classification tasks where categories have an imbalanced number of samples. This problem can be solved by weighting the loss associated with each class according to its abundance. 
     Beyond global amino acid frequencies, the network should also learn what amino acids usually follow other amino acids. The luciferase-like dataset contains 8,397 unique trigrams (3-mers). Their frequency in the dataset has been compared to their frequency in generated proteins. While the frequencies are nicely correlated, it has been observed, as illustrated on graph C in  FIG. 8 , the same tendency of the variational auto-encoder to produce abundant trigrams too frequently. 
     As mentioned above,  FIG. 8  makes it possible to compare the distribution of amino acids in generated sequences with patterns of amino acid occurrences of the training dataset. Graph A illustrates the distribution of the occurrence frequencies of amino acids in luciferase-like sequences, graph B illustrates the distribution of the occurrence frequencies of amino acids in generated sequences, and graph C illustrates the comparison of the occurrence log frequencies of the 8,397 trigrams in each dataset. 
     Higher Level Structures of Reconstructed Proteins 
     While preserving features of the primary structure such as 3-mers frequencies is of importance, the variational auto-encoder should also understand aspects of secondary and tertiary structures. To investigate this, nine reconstructions of a F420-dependent glucose-6-phosphate dehydrogenase protein (UniProt ID: A6W D69), which displayed low sequence identity to the initial protein (40-48%), have been selected. If the variational auto-encoder has some understanding of protein structure, it is expected that it preserves structure even when the primary sequence diverges, by selecting amino acids that will maintain a structure similar to that of the initial protein. For the sake of illustration, secondary structure predictions have been made with PSIPRED (Protein Sequence Analysis Workbench aggregating several UCL structure prediction methods into one location) and 3D structures have been predicted using Protein Homology/analogY Recognition Engine v 2.0 (Phyre2). 
       FIG. 9  illustrates preservation of secondary structures in reconstructions by showing alignments between one protein (Uniprot: A6WD69, denoted init in  FIG. 9 ) and its reconstructions (denoted 0 to 7 in  FIG. 9 ). Secondary structures as predicted by PSIPRED are highlighted as follow: Alpha helices are colored in purple, and beta sheets in yellow. Bold amino acids are the highly conserved positions from the HMM profile of Bacterial Luciferase (PF00296). Again, (*) indicate identical amino acids, (:) very similar amino acids and (.) similar amino acids. 
     It is apparent from  FIG. 9  that even divergent sequences reconstructed from the same point in the latent space tend to keep the same secondary structure as predicted by PSIPRED. Even with a reconstruction accuracy as low as 40%, the variational auto-encoder preserves the structure of the protein. All beta sheets and alpha helices roughly begin and start at the same place on both sequences. Phyre2 also predicts almost identical 3D structures for the initial and reconstructed proteins, as apparent from  FIG. 10 . 
       FIG. 10  illustrates 3D structures of the two proteins as predicted by Phyre2 for the initial protein (A) and for one of its reconstructions presented in  FIG. 9(B) . 
     Amino Acid Relatedness 
     Amino acids having similar properties (e.g. charge, polarity, or hydrophobicity) are more likely to preserve the activity and folding of the protein when swapped in a sequence. These similarities are usually represented in the form of a BLOSUM matrix (blocks substitution matrix). As apparent from  FIG. 7 , the variational auto-encoder seems to learn which amino acids can be swapped as seen by the fact that the few positions in a reconstructed protein that are different from the initial protein are usually very similar amino acids. This may be confirmed by comparing the knowledge obtained by the variational auto-encoder to the BLOSUM62 matrix. As mentioned above, the decoder predicts the probability that each amino acid will occupy a given position. According to embodiments, the chosen amino acid is the one with the highest probability but the other probabilities can be used to study how similar two amino acids are according to the variational auto-encoder and compute a BLOSUM-like matrix. For instance, it is expected that the probabilities for Leucine (L) and Isoleucine (I) to be relatively high when Valine (V) is predicted, since they all have hydrophobic side chains. 
     Algorithm 2 in the Appendix illustrates an example of steps for computing a BLOSUM-like matrix. 
       FIG. 11  illustrates a BLOSUM62 matrix (left) and BLOSUM-like generated matrix (right). Red indicates that high likelihood that these amino acids can be swapped and blue a low likelihood. 
     While the general aspect of the two matrices is different, it can be seen that the variational auto-encoder does correctly identify many of the similarities and dissimilarities between amino acids, with the notable exception of W and Y which the variational auto-encoder does not predict as easily swappable but are identified as such by the BLOSUM62 matrix. The variational auto-encoder thus learns what amino acids are related since they tend to appear in similar contexts, simply by being trained to perform reconstructions. 
     Conditioning 
     The encoder having a stochastic output (it predicts a distribution over the latent variables), the variational auto-encoder can reconstruct a given sequence in a variety of ways. 
     As described above, conditioning essentially consists in adding additional information to the encoder and to the decoder, which can then be used to control the generation process beyond random sampling. For the sake of illustration, the following additional information has been used as a conditioning vector (c): the length of the protein, its weight, charge, isoelectric point, molar extinction coefficient and probability of expression in inclusion bodies (a measure of solubility). These properties were computed using the pepstats tool from the EMBOSS suite. Interestingly, the information provided in conditioning vector c enabled a 2% gain in reconstruction accuracy. 
     To test the ability of the variational auto-encoder to condition on these properties, conditioning values have been modified linearly from −2 to +2 one at a time (quantities are standard scaled), while keeping the other values unchanged. For each set of conditioning values, 32 points have been randomly sampled in the latent space and the corresponding protein sequences have been generated. This enabled proteins to be generated which have, on average, chosen physical properties. This is exemplified, in particular, in  FIG. 12  where solubility has been modified. It is to be noted that properties that are strongly linked physically such as molecular weight and length cannot be conditioned upon separately, but conditioning is still possible when both are changed in the same direction. These experiments show that it is possible to generate sequences that have chosen physical properties. 
       FIGS. 12 to 16  illustrate generation of protein sequences when conditioned on solubility, protein length, protein length and weight, charge, and molar extinction coefficient, respectively. As mentioned above, proteins were generated while progressively changing the considered one(s) of these parameters from −2 to 2 and keeping the other physical properties unchanged. 
     In  FIG. 12 ,
         graph A represents solubility variation,   graph B represents length variation,   graph C represents charge variation,   graph D represents weight variation,   graph E represents molar extinction coefficient variation, and   graph F represents isoelectric point (P1) variation.       

     In  FIG. 13 ,
         graph A represents length variation,   graph B represents charge variation,   graph C represents insolubility variation,   graph D represents weight variation,   graph E represents molar extinction coefficient variation, and   graph F represents isoelectric point (P1) variation.       

     In  FIG. 14 ,
         graph A represents length variation,   graph B represents charge variation,   graph C represents solubility variation,   graph D represents weight variation,   graph E represents molar extinction coefficient variation, and   graph F represents isoelectric point (P1) variation.       

     In  FIG. 15 ,
         graph A represents charge variation,   graph B represents length variation,   graph C represents solubility variation,   graph D represents weight variation,   graph E represents molar extinction coefficient variation, and   graph F represents isoelectric point (P1) variation.       

     In  FIG. 16 ,
         graph A represents molar extinction coefficient variation,   graph B represents length variation,   graph C represents charge variation,   graph D represents weight variation,   graph E represents solubility variation, and   graph F represents isoelectric point (PI) variation.       

     The green line represents the value provided in the conditioning vector. 32 proteins were generated for each set of values. The points represent the mean and bars represent the standard deviation. 
     For the sake of illustration and to obtain variants of a specific protein with modified properties, an F420-dependent glucose-6-phosphate dehydrogenase was chosen and 100 different reconstructions were generated by sampling the latent space while conditioning for increased solubility, protein length, protein length and weight, charge, and molar extinction coefficient. 
     Regarding solubility, as expected, conditioned proteins had increased solubility on average, although not as much as expected if conditioning worked perfectly. Further, it was observed that conditioning on solubility did not affect other physical properties. Reconstructed sequences were on average 95% identical to the initial sequence, and the accuracy only dropped to 94% when conditioning. 
       FIG. 17  illustrates generation of variants of a protein with desired properties. Reconstructions of a F420-dependent glucose-6-phosphate dehydrogenase are generated while conditioning for increased solubility (decreased insolubility). The distributions over all the six physical properties (graph A: weight, graph B: charge, graph C: isoelectric point (PI), graph D: molar extinction coefficient, graph E: length, and graph F: insolubility) of 100 reconstructions are shown:
         blue shapes illustrate the distribution over the whole dataset,   green shapes illustrate the distribution over 100 reconstructions of the input protein,   red shapes illustrate the distribution over 100 modified reconstructions of the input protein (conditioning with insolubility=0.39),   purple shapes illustrate the distribution over 100 modified reconstructions of the input protein (conditioning with insolubility=0.22),   green lines represent the expected value for all reconstructions,   the red line is the expected insolubility for the insolubility red shape, and   purple line is the expected insolubility for the insolubility purple shape.       

     Latent Space Organization 
     When training the variational auto-encoder without conditioning, it may be observed that physical properties are encoded in the latent space, the principal component corresponding mostly to the protein size. 
       FIG. 18  illustrates the projection of the training set in the latent space of an unconditioned model, colored as a function of the protein length. The x-axis is the projection on the first principal component (PC) and the y-axis is that on the second principal component (PC). 
     Conditioning should in theory provide this physical information to the variational auto-encoder and thus free the latent space to encode other properties of the proteins. Indeed, in the variational auto-encoder trained with conditioning, it becomes impossible to predict protein size and the other properties from the latent space representation. 
     The latent space self-organizes so that proteins from the same families cluster together. For the sake of illustration, the luciferase-like domain (IPR011251) combines 88 Interpro families containing proteins with related functions, similarities in sequence, or similar secondary or tertiary structure. In the dataset described above, there are 24 non-overlapping Interpro families containing more than 100 samples. These 24 families are relatively balanced and represent about 50,000 samples. Training a simple multi-class logistic regression model on top of the latent representation of the 50,000 annotated samples makes it possible to classify proteins with a 92.7% accuracy. This shows that families are generally well separated in the latent space, which can be visualized in a projection. 
     Generating Protein Sequences: Example, Generation of LuxA Variants 
     To experimentally validate the ability of a conditional variational auto-encoder (CVAE) having an autoregressive decoder as illustrated in  FIGS. 2 a  to 5 b   , to generate functional protein variants according to the method illustrated in  FIG. 6 , variants of the LuxA protein from  Photorhabdus luminescens  (Uniprot P19839) were generated by randomly sampling the latent space around the representation of LuxA and decoding these points into protein sequences using the autoregressive decoder. 
     First, the CVAE was trained on 72,000 luciferase-like proteins (containing the Interpro domain IPR011251) with conditioning values corresponding to the following physical properties obtained from the EMBOSS ‘pepstats’ software (https://www.ebi.ac.uk/Tools/seqstats/emboss_pepstats/): length of the protein (Residues), its molecular weight, charge, isoelectric Point, A280 molar extinction coefficients and probability of expression in inclusion bodies. These conditioning values were not changed during the generation process of LuxA variants, they were kept to the values of the LuxA protein. The process for the generation corresponds to what is illustrated in  FIG. 4 b   , where the latent vector z was sampled from the distribution defined by the projection of the LuxA protein in the latent space. The variational auto-encoder doesn&#39;t simply learn a point in a latent space, it actually learns a normal distribution defined by two terms, a mean and a variance. This distribution was simply sampled to generate the variants of LuxA. 
     A set of 23 protein variants (SEQ ID NO: 30 to 52) more or less distant from the LuxA primary sequence spanning between 11 and 34 amino acid differences were chosen arbitrarily (Table 3). None of the variants generated belonged to the training set. 
     DNA sequences encoding these proteins were synthesized (SEQ ID NO: 53 to 75) and cloned on an expression vector designed to carry a bicistronic device in order to ensure that all variants are expressed at the same level (Mutalik et al, 2013 “ Precise and reliable gene expression via standard transcription and translation initiation elements. ” Nat. Methods 10: 354-360). The resulting plasmids were transformed in an  E. coli  strain expressing luxCDBE from another vector. 
     Bacterial Strains and Growth Conditions. 
       E. coli  DH5 alpha, used as a host for recombinant plasmids, was grown in LB media at 37° C. with shaking at 200 rpm. kanamycin (50 μg/ml) and chloramphenicol (20 μg/mL) were added to the medium to prevent loss of plasmids derived from pCM17 (luxCDABE) and pFD69 (pFAB3905ΩP14BCD1cat). 
     Construction of Plasmids 
     The luxA gene was deleted from plasmid pCM17 expressing the luxCDABE operon (Rhee et al, 2011 “ Determination of spatial and temporal colonization of enteropathogenic E. coli and enterohemorrhagic E. coli in mice using bioluminescent in vivo imaging. ” Gut Microbes 2: 34-41). by amplification using primers B731/LC545 (SEQ ID NO: 14 and 16) and B732/LC327 (SEQ ID NO: 15 and 17), followed by Gibson assembly (Gibson et al, 2009 “ Enzymatic assembly of DNA molecules up to several hundred kilobases. ” Nat Methods 6: 343-5) of the two PCR fragments, giving plasmid pDB283 (pCM17ΩΔluxA) (SEQ ID NO: 77). 
     The kanamycin gene of pFAB3905 (P14BCD1, KmR) (Mutalik et al, 2013) was replaced by the cat gene to allow the complementation by luxA in  E. coli  DH5/pDB283 (pCM17ΩΔluxA) which is already resistant to kanamycin. For this construction, the cat gene was amplified from pZA31-luc (Lutz &amp; Bujard, 1997 “ Independent and tight regulation of transcriptional units in Escherichia coli via the LacR/O, the TetR/O and AraC/I 1- I 2  regulatory elements. ” Nucleic Acids Res 25: 1203-1210) using primers F168/F169 (SEQ ID NO: 18 and 19) and assembled with primers F170/F171 (SEQ ID NO: 20 and 21) under the control of a P14BCD1 constitutive promoter in plasmid pFAB3905 giving plasmid pFD69 (pFAB3905ΩP14BCD1 cat). The luxA gene was amplified from pCM17 (luxCDABE) using primers F172/F173 (SEQ ID NO: 22 and 23) and assembled under the control of the P14BCD1 constitutive promoter in plasmid pFD69 amplified with primers F174/F175 (SEQ ID NO: 24 and 25), giving plasmid pFD72 (pFAB3905ΩP14BCD1luxAcat) (SEQ ID NO: 76). 
     The LuxA variants were encoded as a DNA sequence, corresponding genes were synthesized by Twist Bioscience (Table 4 and 5). The luxA variants were amplified from synthesized DNA fragments using primers F208/F209 (SEQ ID NO: 28 and 29) and cloned in plasmid pFD69, under the control of constitutive promoter and in a bicistronic device, amplified with primers F206/F207 (SEQ ID NO: 26 and 27) through Gibson assembly, giving plasmids pFD90-Vx (pFAB3905ΩP14BCD1luxA-Vxcat) where x corresponds to the number of the variant. All plasmids were verified by Sanger sequencing. 
     The resulting plasmids (pFD72 and pFD90-Vx) were transformed in  E. coli  DH5alpha carrying the plasmid pDB283 expressing the luxCDABE operon from  Photorhabdus luminescens  with gene luxA deleted (pCM17ΩΔluxA; SEQ ID NO: 77) and transformants were selected on LB agar containing kanamycin (50 μg/ml) and chloramphenicol (20 μg/ml). 
     Bioluminescence Measurements 
     Strains were grown in triplicate overnight at 37° C. and diluted 1:100 in 1 ml of LB broth in a 96-well microplate that was incubated at 37° C. with shaking in ThermoMixer (Eppendorf). After 5h of growth, luminescence monitoring was performed with 100 μl in white 96-well microplate (Greiner, Kremsmünster) on a Centro XS LB 960 microplate luminometer (Berthold Technologies, Thoiry). The luminescence was measured during 10 s for each well. In parallel, to study the growth of the strains, the overnight cultures were diluted 1:100 in 200 μl of LB broth in a 96-well microplate that was incubated at 37° C. with shaking in a Infinite M200 PRO reader (TECAN). Absorbance was measured at 600 nm every 10 min. None of the strains showed a growth defect. 
       FIG. 23  illustrates the bioluminescence of LuxA variants relative to the wild-type LuxA protein as a function of the number of amino-acid substitutions. Points at the bottom of the graph are below the limit of detection. 
     Luminescence could be measured for 18/23 (78%) of the LuxA variants ( FIG. 23 ), with the most distant variants still producing light carrying 27 mutations. 
     The probability that a random mutation in a protein sequence leads to its inactivation has been estimated to be ˜34% (Guo et al., 2004 “ Protein tolerance to random amino acid change ” PNAS Jun. 22, 2004 101 (25) 9205-9210;). Accordingly, one can make the crude estimate that 27 random mutations would only lead to a functional protein in 1 case out of 100,000. The ability of the conditional variational auto-encoder (CVAE) having an autoregressive decoder to generate functional distant variant thus demonstrates its ability to learn and decode meaningful representations of proteins. 
     Comparison of Different Autoregressive Neural Network Architectures 
     The choice of the details of the decoder architecture has an important role to play in determining whether the model mainly learns to use the latent space or mainly learns to use the conditional distribution in predicting amino acids, and, by extension, has an important effect on the way in which the model generates new proteins. The conditional variational auto-encoder (CVAE) architecture having an autoregressive decoder as illustrated in  FIGS. 2 a  to 5 b   , is chosen to balance the information in the latent space and in the conditional distribution. This choice is made possible by the inclusion of both an autoregressive and a feed-forward component in the decoder, and offers greater flexibility and control over the generation process than would be possible with a differently designed model. A model without latent variables which relied entirely on the conditional distribution would not offer control over the generation. A model which uses latent variables, but does not attempt to learn the conditional distribution, may struggle to generate coherent proteins. 
     Specifications of the Architecture According to Embodiments 
     Encoder 
     The encoder takes as input a fixed length, one-hot encoded protein sequence (padding characters are added to the end of sequences so that they are all of length 504) and outputs the parameters of the standard Normal distribution from which the latent code representing the protein is drawn. These parameters are the mean vector, and a vector of variances, each having the same number of dimensions as the latent code. To generate these parameter vectors, the one-hot encoded input is processed by a series of five 1D convolutional layers. The first layer has 21 filters and a stride size of 1. The subsequent layers all have a stride size of 2 meaning that the resolution of the representation is decreased by a factor of 2 by each of these layers. The number of filters is increased by a factor of 2 each time the resolution is decreased by a factor of 2. All convolutional layers have a kernel size of 2, PReLu activation functions and are batch normalized. Finally the output of the convolutional layers is fed separately through two output layers: a single fully connected layer with linear activation to produce the mean vector, and a fully connected layer with softplus activation to produce the variances. 
     Decoder 
     The decoder operates sequentially, with its outputs at each time step corresponding to probabilistic predictions of the identity of the amino acid at the next position in the sequence, given the latent code and the identities of the previous amino acids in the sequence. The decoder can be separated into two components: an up-sampling component and an autoregressive component. The autoregressive component acts sequentially on the output of the up-sampling component together with the input protein sequence to produce predictions for the next amino acid at each point in the sequence. 
     Up-Sampling Component 
     The up-sampler takes as input the latent vector and outputs a sequence of the same length as that of the padded proteins. It achieves this by inverting the sequence of operations in the encoder: the latent code is fed first through a fully connected layer, then through a series of deconvolutional layers, in each of which the resolution increases by a factor of 2 while the size of the feature vectors at each position is halved, until the representation is a sequence of the same length as the input proteins, with a 42-dimensional feature vector at each point in the sequence. 
     Autoregressive Component 
     The autoregressive component acts sequentially on the sequence of latent feature vectors output by the up-sampler together with the one-hot encoded input protein sequence. At time step t the autoregressive component takes as input the values of the two sequences up to position t and predicts the amino acid at position t+1. 
     The autoregressive processing is achieved by a series of dilated causal convolutions organized in the same manner as in a Wavenet. In detail, the one hot encoded protein sequence is first processed by a single causal convolution layer. Then the activations of this layer together with the latent feature sequence are passed through two residual blocks. In each residual block one or more dilated convolutions are applied to the activations from the previous layer or block. The resulting sequence of activations is then added to a linearly transformed version of the sequence of latent feature vectors, before being passed through a gated activation function as in Wavenet. Finally the output of the residual blocks is passed through a linear output layer with softmax activations to generate a prediction at each time step. The processing is such that the prediction at each timestep is a function only of the c preceding values of the input sequences, where c is a fixed context length which is controlled by the number of dilated convolutional layers. We use c=64. 
     PReLu activations and batch normalisation are used for the deconvolution and dilated convolution layers throughout the decoder. 
     Ablation Experiments 
     The design choices relating to the structure of the decoder outlined above play an important role in determining the ability of the model to generate interesting novel proteins. In order to highlight the effect of these choices, proteins generated by the model outlined above (referred as “Up-sampling+AR”) can be compared to those generated by models which instantiate different design choices. Of particular relevance are comparisons to models in which the roles of the latent variables or that of the conditional distribution of amino acids are either suppressed or removed entirely. This section reports the results of experiments on such models, in particular:
     (i) a “non autoregressive (non AR)” model in which the output of the up-sampler is used directly to predict the amino acid identities, with no information about prior amino acids,   (ii) a “no up-sampling” model in which the up-sampling component is removed, and   (iii) a “reduced z” model in which the information provided by the latent variables is heavily suppressed by setting the latent variable dimension to 2 and removing the up-sampling component, so that the model must rely almost entirely on conditional information in predicting amino acids (and, by extension, in generating new proteins).   

     Various forms of assessment were used to benchmark the behavior of generative models, each deriving from one of the following requirements on the performance of a good model: (i) a good model should be able to successfully (approximately) reconstruct known proteins having first projected them down to a lower dimensional representation, (ii) a good model should allow to generate novel proteins which share common properties or functions with other proteins from the same family but at the same time are not trivially similar to known proteins. 
     For comparison, the above generative models were trained with the same dataset used for the generation of LuxA variants. 
     Comparison of Prediction Accuracy of Amino Acid Identities 
     A model&#39;s reconstruction ability can be measured by the accuracy of its predictions of amino acid identities. That is, for each position (in a protein p), how accurately does the model predict the identity of the amino acid at that position in p given the latent vector z representing p and the c previous amino acids in p. This accuracy can be computed over the predictions in every position in a set of proteins (which may be a different set from those used for training), and refer to this quantity as the “amino acid reconstruction accuracy” on that set. This accuracy can be computed in two ways, corresponding to the training and generation modes: in the first, the context provided to the autoregressive component is the c previous amino acids in the protein being reconstructed, in the second the context provided the autoregressive component is the c previous predictions. We refer to these two modes as “true context” and “predicted context”. When using the predicted context rather than the true context, the model&#39;s accuracy will be much lower if insertions cause the straightforward alignment between positions in p and the reconstruction to be broken. To account for this possibility, a third score is also reported, the percentage identity of the reconstruction and p, after realigning the reconstruction to p. 
       FIG. 24  illustrates the protein reconstruction accuracies for the neural network architectures “Non AR”, “No up-sampling”, “Reduced z” and “Up-sampling +AR”. 
     These three quantities show significant differences in behavior of the differently designed models. The highest accuracy of any model trained with the “true context” is achieved by the model with both up-sampling and autoregression, due to its ability to exploit both contextual information and the global latent code. This accuracy comes at the price of reduced accuracy when reconstructing with only “predicted context”, due to the reliance on the context for important information allowing errors to accumulate. The reduction in accuracy is however mitigated by the use of up-sampling, with the “no up-sampling” model showing a much greater drop in accuracy when using the predicted context. 
     Comparison of the Properties of the Generated Sequences 
     A more direct assessment of the generative abilities of the models can be made by comparing the properties of the sequences that they generate. Sequences can be generated from a VAE in an unconstrained manner by sampling from the prior distribution over the latent variables. Given a latent vector sampled from the prior, a full sequence can be generated by simply passing the latent vector to the decoder (in “predicted context” mode). A KL term (Kullback-Leibler divergence) can be used as a measure of the distance between the prior distribution and the distribution of the projection of samples into the latent space The KL term in the optimization objective forces the encoder to map proteins from the training data distribution in such a way that the distribution of the latent vectors assigned to training data points should be close to the prior. It is this constraint which (if sufficiently closely obeyed) means that latent vectors drawn from the prior should be decoded to proteins with properties close to those of the training set. This is clearly desirable behavior for a generative model, since it provides an easy way of generating novel proteins which nonetheless correspond in important ways to the known proteins from a given family, and in particular, which might be expected to show significant structural or functional similarity. 
     A qualitative picture of whether the proteins generated by the model are similar to those of the training dataset can be obtained by visualising the predicted secondary structures of the generated sequences. 
       FIG. 25  illustrates the aligned secondary structures of sequences selected at random from training set and  FIG. 26  the aligned secondary structures of sequences generated by sampling from prior of the “Non AR”, “No up-sampling”, “Reduced z” and “Up-sampling+AR” models. 
     Proteins from the bacterial-luciferase like family display a consistent pattern of repeating alpha helices and beta strands in their predicted secondary structures. This is in agreement with knowledge of the secondary structures of solved proteins such as LuxA and LuxB from  Vibrio Harveyi,  which both display this pattern and fold into a TIM-barrel. A simple visualisation of the predicted secondary structures of a random sample of 3000 proteins from the training dataset makes this pattern clear ( FIG. 25 ). A visual assessment of the quality of the samples of protein-generating models can therefore be made by visualising the predicted secondary structures of the samples in the same way and looking for evidence of the same patterns that are present in the training set ( FIG. 26 ). Only the models with an autoregressive component are able to consistently generate proteins with the desired pattern of alternating alpha helices and beta strands. 
     Quantitative assessment of generation quality can be performed using the PFAM family HMM (Hidden Markov models). The HMM is a model designed to compute the probability that a sequence is a member of a given family. This is reported as an E-value, which measures the probability of a sequence achieving the same score as the sequence in question by chance. 
       FIG. 27  illustrates the percentage of HMM hits for samples from the “Non AR”, “No up-sampling”, “Reduced z” and “Up-sampling+AR” models. 
     The results of scoring the 3000 samples from the models visualised above confirm the observation that the autoregressive models are much more successful at generating sequences resembling those of the target family than the model which uses only an up-sampler in the decoder ( FIG. 27 ). 
     Comparison to RNN Decoder 
     In principle the autoregressive component of the decoder can be instantiated by any autoregressive network. However the use of an autoregressive convolutional neural network (autoregressive CNN) offers an important degree of flexibility in controlling the size of the context available at each time step. By limiting this context to a size less than that of the full protein the autoregressive component is encouraged to focus on local conditional details of the sequence while the latent code is responsible for global structural details. This encourages a balancing of contributions between the two components, which ensures both information in the conditional distribution and information in the latent code are exploited. Experiments where the autoregressive component was instead instantiated with a recurrent neural network indicated that in this case this balance would be harder to control. 
       FIG. 28  illustrates the evolution of KL loss term during training for the “Upsampling+AR (KL annealed)”, “RNN (KL annealed)” and “RNN” models. 
     When trained with an RNN in the decoder, the model showed a tendency to fail to place any information in the latent code, instead relying entirely on the contextual information to make amino acid predictions. This behavior can be observed by looking at the KL term in the objective function. This term measures the distance between the distribution over latent variables predicted for proteins in the training set by the encoder and the prior, a standard normal distribution. Models with an RNN in the decoder often minimize this distance early in training, by simply encouraging the encoder to output normally distributed latent codes, and focus later in training only on minimizing the reconstruction loss term ( FIG. 28 ). This behavior results in no information being stored in the latent code, as can be seen by the fact that in this case, samples taken from the model by first sampling a latent code from the prior distribution and then passing the code to the decoder show little or no diversity, irrespective of the sampled code. Ways of mitigating this kind of problem are discussed in Bowman et al. “ Generating Sentences from a Continuous Space. ” CoRR, 2015., and experiments suggest that both gradually increasing the weight of the KL term from a small value (typically 0-0.2) to 1, and applying a dropout mask to some percentage of the positions in the “true context” input can lead models with either an RNN or CNN autoregressive component to put more information in the latent code that they otherwise would.  FIG. 28  compares the behavior of the “Up-sampling+AR” model with CNN vs RNN autoregressive component with the KL term either constant at 1, or increased gradually from 0.2 to 1 (“KL annealed”). In both cases the model with a CNN autoregressive component relies more heavily on the latent code. The RNN without KL annealing does not use the latent code at all. 
     Conclusion 
     Experiments with the details of the VAE architecture reveal the extent to which the behavior of the trained model depends on the design choices made. The use of a hybrid decoder incorporating both up-sampling and autoregression is motivated by the observation that pure up-sampler/feedforward decoders, though able to achieve good reconstruction accuracies on held out proteins, make for bad generative models, with samples drawn from these models only rarely matching the HMM profile of the family, and failing to display the secondary structure characteristic of luciferase-like proteins. At the same time, pure autoregressive models show significant drops in reconstruction accuracy when generating with only the predicted rather than true context as signal for the autoregressive predictions. This diminishes the amount of information that can be specified about the nature of the protein to be generated via the latent variables, leading to a reduction in the control over the generative process that the model affords. 
     Thus, an architecture with a hybrid decoder incorporating both up-sampling and autoregression, for example the conditional variational auto-encoder (CVAE) having an autoregressive decoder as illustrated in  FIGS. 2 a  to 5 b   , is the architecture that achieves high performances on metrics designed to test both the ability to control the generative process through the specification of latent variables and the ability to sample novel proteins by sampling from the prior over the latent variables as desirable. 
       FIG. 19  illustrates the projection of five Interpro families in the latent space. The z-axis is a vector that is obtained by maximizing its projection on five normal vectors of the hyperplanes defined by the logistic regression. 
       FIG. 20  schematically illustrates a processing device  2000  configured to implement at least one embodiment of at least part of the invention, for example one or several of the algorithms described by reference to  FIGS. 3 to 9 . The processing device  2000  may be a device such as a micro-computer, a workstation, or a highly parallel computer. The device  2000  comprises a communication bus  2013  to which there are preferably connected:
         a central processing unit  2011 , such as a microprocessor, denoted CPU;   a read only memory  2007 , denoted ROM, for storing computer programs for implementing the invention;   a random access memory  2012 , denoted RAM, for storing the executable code of the method of embodiments of the invention as well as the registers adapted to record variables and parameters necessary for implementing the method for improving the assembling of raw images obtained by single molecule localization microscopy according to embodiments of the invention; and   a communication interface  2002  connected to a communication network  2003  over which digital data to be processed can be transmitted.       

     Optionally, the apparatus  2000  may also include the following components:
         a data storage means  2004  such as a hard disk, for storing computer programs for implementing methods of one or more embodiments of the invention and data used or produced during the implementation of one or more embodiments of the invention;   a disk drive  2005  for a disk  2006 , the disk drive being adapted to read data from the disk  2006  or to write data onto said disk;   a screen  2009  for displaying data and/or serving as a graphical interface with the user, by means of a keyboard  2010  or any other pointing means; and   graphic processing units (GPU), not represented, that allow parallel processing of large matrix data and that may be used, in particular, for carrying out operations of the artificial neural network(s). GPU may prove to be important for optimizing computation time.       

     The communication bus provides communication and interoperability between the various elements included in the apparatus  2000  or connected to it. The representation of the bus is not limiting and in particular the central processing unit is operable to communicate instructions to any element of the apparatus  2000  directly or by means of another element of the apparatus  2000 . 
     The disk  2006  can be replaced by any information medium such as for example a compact disk (CD-ROM), rewritable or not, a ZIP disk or a memory card and, in general terms, by an information storage means that can be read by a microcomputer or by a microprocessor, integrated or not into the apparatus, possibly removable and adapted to store one or more programs whose execution enables the method for improving the assembling of raw images obtained by single molecule localization microscopy according to embodiments of the invention to be implemented. 
     The executable code may be stored either in read only memory  2007 , on the hard disk  2004  or on a removable digital medium such as for example a disk  2006  as described previously. According to a variant, the executable code of the programs can be received by means of the communication network  2003 , via the interface  2002 , in order to be stored in one of the storage means of the apparatus  2000  before being executed, such as the hard disk  2004 . 
     The central processing unit  2011  is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to the invention, instructions that are stored in one of the aforementioned storage means. On powering up, the program or programs that are stored in a non-volatile memory, for example on the hard disk  2004  or in the read only memory  2007 , are transferred into the random access memory  2012 , which then contains the executable code of the program or programs, as well as registers for storing the variables and parameters necessary for implementing the invention. 
     In this embodiment, the apparatus is a programmable apparatus which uses software to implement the invention. However, alternatively, the present invention may be implemented in hardware (for example, in the form of an Application Specific Integrated Circuit or ASIC). 
     It is to be noted that while embodiments have been described by reference to variational auto-encoders, other generative deep neural models may be used. In particular, embodiments may use adversarial auto-encoders (AAE) or conditional adversarial auto-encoders (CAAE) that add a discriminator network to an auto-encoder. The discriminator is trained to differentiate randomly sampled points in the latent space from the projection of actual data. The encoder is trained to fool this discriminator, thus forcing projections to occupy the whole latent space and ensuring that all points of the latent space can be decoded into valid samples. The concept and training procedure for this type of architecture are detailed in “Adversarial Autoencoders”, Alireza Makhzani, Jonathon Shlens &amp; Navdeep Jaitly, Ian Goodfellow, Brendan Frey, arXiv:1511.05644v2 (see e.g. § 2). 
       FIG. 21  illustrates an example of the logical architecture of a conditional adversarial auto-encoder having an autoregressive decoder. 
     As illustrated, conditional adversarial auto-encoder  2100  comprises an encoder  2105  and an autoregressive decoder  2110 . The input vector  2115 , denoted x, as well as additional information  2120  or conditions, denoted c, are applied to encoder  2105  to generate latent vector  2125 , denoted z. An example of encoder  2105  is illustrated in  FIG. 22   a.    
     Again, for the sake of illustration, additional information  2120  may comprise the length of the protein, its charge, its isoelectric point, its molar extinction coefficient, its solubility (e.g. probability of expression in E.coli inclusion bodies: the smaller this quantity, the more soluble the protein is), and/or its molecular weight, it being observed that some of these values may be correlated (for example, the molecular weight of a protein is highly correlated with its length). 
     According to the illustrated example, latent vector  2125  and additional information  2120  are then applied to up-sampler  2130  to generate up-sampled latent vector  2135 , denoted z′. Latent representation z is up-sampled in up-sampler  2130  while being conditioned by additional information c to generate latent representation z′. The obtained up-sampled latent vector  2135 , i.e. latent representation z′, and input vector x are then input in autoregressive module  2140  to generated result  2145 , denoted x′. 
     Again, it is to be noted that the use of an up-sampling module is not requested in all embodiments. For example, the size of the latent code z may be increased by concatenation to reach the one of input vector x. 
     It is also to be noted that according to particular embodiments, no additional information is fed into encoder  2105  and up-sampler  2130 . 
     Autoregressive decoder  2110  may be similar to autoregressive decoder  210  described by reference to  FIG. 2   a.    
     Encoder  2105  and decoder  2140  can be similar to those described previously, with the difference that in the case of an adversarial auto-encoder architecture, the encoder is not probabilistic and simply makes a projection of vector x onto latent space z as illustrated in  FIG. 22   a.    
     Conditional adversarial auto-encoder  2100  further comprises a discriminator  2150  for discriminating projected samples (i.e. samples projected from input vectors) from drawn samples (i.e. samples obtained from the distribution within the latent space (p(z)). An example of discriminator is illustrated in  FIG. 22   b.    
       FIG. 22 a    illustrates an example of the functional architecture of encoder  2105  illustrated in  FIG. 21 . It is to be noted that other functional architectures may be implemented. 
     As illustrated, input vector x is fed in five successive 1D convolution layers referenced  2200 - 1  to  2200 - 5  (for the sake of clarity only two of them are represented). According to embodiments, the stride of these convolution layers is set to two, except for the first one which is set to one. The output of 1D convolution layer  2200 - 5  is flattened (reference  2205 ) so as to suppress one dimension. The convolution layers aim at establishing relations between parts of protein sequences. 
     In parallel, additional information or conditions c are fed into two successive dense layers  2210 - 1  and  2210 - 2 , i.e. into two fully connected neural networks that aim at identifying what should be in protein sequences. 
     The output of flattened layer  2205  and the output of dense layer  2210 - 2  are then concatenated (reference  2215 ). As illustrated, the result of the concatenation is fed into two successive dense layers  2220 - 1  and  2220 - 2 , the result of the latter being latent vector z 
       FIG. 22 b    illustrates an example of the functional architecture of discriminator  2150  illustrated in  FIG. 21 . It is to be noted that other functional architectures may be implemented. 
     As illustrated, discriminator  2150  is based on two dense layers  2250 - 1  and  2250 - 2  that are fully connected neural networks aiming at identifying what should be in protein sequences. 
     Training such a conditional adversarial auto-encoder and generating protein sequences with such a conditional adversarial auto-encoder may be based on the steps described by reference to  FIG. 6 . 
     Other generative deep neural models may use generative adversarial networks (GANs) that are trained to generate realistic data samples from noise using a generator network trained to fool a discriminator network. The discriminator is itself trained to distinguish generated samples from real samples. The architecture and training procedures for this type of networks are described in “ Generative Adversarial Networks ”, Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S et al., arXiv:1406.2661v1, and “ NIPS  2016  Tutorial: Generative Adversarial Networks ”, Goodfellow I arXiv:1701.00160v4. While GANs work very well with continuous data, it currently remains a challenge to train them on discrete data. Indeed, generating discrete data involves a discontinuous step making it impossible to train via backpropagation. A recent study proposed a strategy to train GANs on discrete data, and is another interesting avenue to generate proteins (see “ Boundary - Seeking Generative Adversarial Networks ”, Devon Hjelm R, Jacob A P, Che T, Cho K, Bengio Y., arXiv:1702.08431v2) 
     Other generative models can be constructed in which the encoder and decoder architectures can be modified to include attention mechanisms as described for instance in “ Convolutional Sequence to Sequence Learning ”, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, arXiv:1705.03122v3, or in “ Attention Is All You Need ”, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, arXiv:1706.03762v4. 
     Another possible improvement is to make the autoregressive module bi-directional. Two networks would be trained in parallel to predict the next amino-acid in a sequence, one from left to right and the other from right to left. The generation process would then happen in two steps. In a first step a protein sequence can be generated from left to right (or vice-versa), and in a second step the generated sequence can be modified according to the predictions made by the second model reading the sequence in the opposite direction. 
     Naturally, in order to satisfy local and specific requirements, a person skilled in the art may apply to the solution described above many modifications and alterations all of which, however, are included within the scope of protection of the invention as defined by the following claims. 
     APPENDIX 
       
     
       
         
           
               
             
               
                   
               
               
                 Algorithm 1: generation of a protein sequence from a latent code and conditions 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
            
               
                 Input: three arrays 
               
               
                  1. latent code z, 
               
               
                  2. conditions c, 
               
               
                  3. protein sequence x to be generated, it is padded with zeroes at initialization and is updated 
               
               
                   step by step to become a one-hot-encoded protein 
               
               
                 Output: a one-hot-encoded protein 
               
               
                 /* predict amino acids one after the other */ 
               
               
                 for i = 1 ... prot_size do 
               
               
                  pred ← Decoder([x,z,c]); 
               
               
                  pred ← pred[i,:]; // the model is autoregressive: only the i-th position is considered 
               
               
                  pred ← argmax pred; 
               
               
                    AA 
               
               
                  x[i,pred] ← 1; 
               
               
                 end 
               
               
                 return x 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                   
               
               
                 Algorithm 2: example of steps for computing a BLOSUM-like matrix 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
            
               
                 Input: three arrays 
               
               
                  n samples : the number of proteins to sample to estimate the BLOSUM scores 
               
               
                  n char : the number of amino acids (plus the padding characters) 
               
               
                  length: the length of proteins 
               
               
                 Output: a BLOSUM-like matrix 
               
               
                 BLOSUM ← 0n char ,n char   
               
               
                 count ← 0n char ,n char  // normalizing factors 
               
               
                 for i = 1 ... n samples  do 
               
               
                  z ← sample(N(0,1)); 
               
               
                  c ← sample(N(0,1)); 
               
               
                  x ← 0lenght,n char ; 
               
               
                  pred ← Decoder([x,z,c]); 
               
               
                  pred ← pred[i,:]; // the model is autoregressive: only the i-th position is considered 
               
               
                  most_likely ← argmax pred; 
               
               
                   AA 
               
               
                  x[i, most_likely] ← 1; 
               
               
                  BLOSUM[most_likely,:] ← BLOSUM[most_likely,:] + pred; 
               
               
                  BLOSUM[:,most_likely] ← BLOSUM[:,most_likely] + pred; 
               
               
                  Count[most_likely,:] ← Count[most_likely,:] + 1; 
               
               
                  Count[:,most_likely] ← Count[:,most_likely] + 1; 
               
               
                 End 
               
               
                 result ← log(BLOSUM/count); 
               
               
                 diagonal (result) ← mean(result); 
               
               
                 return result 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 1 
               
               
                   
               
               
                 alignment between an input protein and a reconstruction 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                 init 
                 MKIGVFIPIGNNGWLISSNAPQYMPSFELNKAIVQQAEHYQFDFALSMIKLRGFGGKTEF 60 
               
               
                 recons 
                 MKIGVFIPIGNNGWLISTNAPQYMPSFELNKAIVQKAEHYQFDFALSMIKLRGFGGKTEF 60 
               
               
                   
                 *****************:*****************:************************ 
               
            
           
           
               
            
               
                      
               
            
           
           
               
               
            
               
                 init 
                 WDHNLESFTLMAGLAAVTSRIKIYATAATLTLPPAIVARMASTTDSISNGPFGLNVVTGW 120 
               
               
                 recons 
                 WDHNLESFTLMAGLAAVTSRIKIYATAATLTLPPAIVARMASTTDSISNGPFGLNVVTGW 120 
               
               
                   
                 ************************************************************ 
               
               
                   
               
               
                 init 
                 QKPEYEQMGLWPGDEYFSRRYDYLSEYVEVLQQFWGTGQSDFNGEFFQMDDCRVSPQPQT 180 
               
               
                 recons 
                 QKPEYEQMGLWPGDEYFSRRYDYLSEYVQVLRDLWGTGKSDFKGEFFQMDDCRVSPQPQA 180 
               
               
                   
                 ****************************:**:*:****:***:****************: 
               
               
                   
               
               
                 init 
                 PIKLICAAQSDAGMAFSAKYADYNFCFGKGVNTPTAFAPTAARLQKAAEQAGREVSSYVL 240 
               
               
                 recons 
                 PIKVICAGQSDAGMAFSAKYADYNFCFGKGVNTPTAFAPTAARMKEAAEKAGRDVSSYVL 240 
               
               
                   
                 ***:***.***********************************:::***:***:****** 
               
               
                   
               
               
                 init 
                 FMIIADETDELARAKWESYKAGADTEALAWLTEQSGKDTQSGADTNVRQMADPTSAVNIN 300 
               
               
                 recons 
                 FMIIADETDEAARAKWESYKAGADTEALAWLTEQSGKDTKSGADTNVRQMADPTSAVNIN 300 
               
               
                   
                 ********** ****************************:******************** 
               
               
                   
               
               
                 init 
                 MGTLVGSYANVAKMMDDIATVPGTEGILLTFDDFLSGIENFGQHIQPLMNSRADTVDTLP 360 
               
               
                 recons 
                 MGTLVGSYANVARMMDETATVPGTEGILLTFDDFLSGIENFGQRIQPLMKSRADIIPTPP 360 
               
               
                   
                 ************:***:**************************:*****:*****: * * 
               
               
                   
               
               
                 init 
                 PAAREVA 367 
               
               
                 recons 
                 EAAREVA 367 
               
               
                   
                  ****** 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 2 
               
               
                   
               
               
                 alignment between a generated protein randomly sampled 
               
               
                 from the latent space and the most similar known sequence 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                 generated 
                 MPVSWLLLGHSSGVGLGGPDAAAAAEGAYGFRAAVVAERVGFDGVLLGARGSPEYP---- 56 
               
               
                 most_similar 
                 ------MDTRTSGITIGYKASAEQFGPRHLVELAVLAERRGFDSVLVSDHYQPWRHRNGH 54 
               
               
                   
                       :  ::**: :*   :*      : .. **:*** ***.**;. : .* 
               
               
                   
               
               
                 generated 
                 --ATVARLL-AEAALHQLRLGAG-VTATSTYAPAAGVEVLGTLASLFPRDIILGLGISTG 112 
               
               
                 most_similar 
                 APFSMAWLAAAGERTERVRLGTSVLTATERYHPAVVAQAFGTLGALCPGRVMLGLGTGEA 114 
               
               
                   
                    ::* *  *    .::***:. :***  * **. .:.:***.:* *  ::**** . . 
               
               
                   
               
               
                 generated 
                 ARHDEAGGRDRFQRWEGDFGDHYPAVRGALGLWTDEEVLREASFASPYYPRGVEG-W--- 168 
               
               
                 most_similar 
                 LNE-VAVARMEWPGFEERFARLREAIDLIRPLWTEERVS----FDGEYYRTENATVYDPP 169 
               
               
                   
                  ..  * .* .:  :*  *.    *:     ***:*.*     * . **       : 
               
               
                   
               
               
                 generated 
                 ----RVVVSPRPPRAGLWLGEKPEDVG-VLGDDVALL-HAFRPVFGRSGPAVAGLLGADV 222 
               
               
                 most_similar 
                 SRPVPVYVAAGGPVVAKYAGRIADGFICTSGKGMELYTEKLQPAVDA------G----AE 219 
               
               
                   
                      * *:   * .. : *.  :..  . *..: *  . ::*...       * 
               
               
                   
               
               
                 generated 
                 PAWAPAGLMPTVDVRLIWGGSPS----DAQFLADLERAEALAFARAEGMRGVEAEPTRRL 278 
               
               
                 most_similar 
                 AGREPADVARTIEIKLSYDTDAEAAAENTRFWAPLS-----------------------L 256 
               
               
                   
                  .  **.:   ::::* :. . .    :::* * *.                       * 
               
               
                   
               
               
                 generated 
                 LAEATSGAPDEAAAGTLAAQQVDVEIDGTPWWAET-PRT--------VSAGLGAAEL--- 326 
               
               
                 most_similar 
                 TADQKRGVSDPLAME-RAADELPMSQIASRWIVSSDPDEVVERIRPYVDAGFTDLVLHAP 315 
               
               
                   
                  *: . *. *  *    **::: :.  .: * ..: *          *.**:    * 
               
               
                   
               
               
                 generated 
                 ---GTRLVAVGLDAVATGATSVDGWSTVGGLAGGGDGFWIGPELYADGARRGAAFDGDVA 383 
               
               
                 most_similar 
                 GHDQAPFLELARQDL------LPRLSNLGG------------------------------ 339 
               
               
                   
                     :*:: :. : :      :    .:** 
               
               
                   
               
               
                 generated 
                 SLSPGRVADH 393 
               
               
                 most_similar 
                 ---------- 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 3 
               
             
            
               
                   
               
               
                 sequences of the primers used for 
               
               
                 plasmid construction 
               
            
           
           
               
               
               
            
               
                   
                   
                 SEQ 
               
               
                   
                   
                 ID 
               
               
                 Name 
                 Sequence 5′-3′ 
                 NO: 
               
               
                   
               
               
                 B731 
                 TAATATAATAGCGAACGTTGAGTACTAAAGTTTCCAAA 
                 14 
               
               
                   
                 TTTCATAGAGAGTCC 
                   
               
               
                   
               
               
                 B732 
                 CTATGAAATTTGGAAACTTTAGTACTCAACGTTCGCTA 
                 15 
               
               
                   
                 TTATATTAGCTAAGG 
                   
               
               
                   
               
               
                 LC545 
                 TCATTTCGAACCCCAGAGTC 
                 16 
               
               
                   
               
               
                 LC327 
                 CGCCTTCTTGACGAGTTCTT 
                 17 
               
               
                   
               
               
                 F168 
                 TCTATCAACAGGAGTCCAAGCGAGCTCGATATCAAATTAC 
                 18 
               
               
                   
               
               
                 F169 
                 TTTTGAATTCGACGTCGACGTCGATATCTGGCGAAAATGAG 
                 19 
               
               
                   
               
               
                 F170 
                 ATCGAGCTCGCTTGGACTCCTGTTGATAGATCCAGTAATG 
                 20 
               
               
                   
               
               
                 F171 
                 CCAGATATCGACGTCGACGTCGAATTCAAAAGATCTTAAG 
                 21 
               
               
                   
               
               
                 F172 
                 CACAGGAGACTTTCTAATGAAATTTGGAAACTTTTTGCT 
                 22 
               
               
                   
                 TAC 
                   
               
               
                   
               
               
                 F173 
                 CTCGACAACCGATCCCTAATATAATAGCGAACGTTGTTT 
                 23 
               
               
                   
               
               
                 F174 
                 GTTTCCAAATTTCATTAGAAAGTCTCCTGTGCATGATTAAG 
                 24 
               
               
                   
               
               
                 F175 
                 TCGCTATTATATTAGGGATCGGTTGTCGAGTAAGGATC 
                 25 
               
               
                   
               
               
                 F206 
                 CATTAGAAAGTCTCCTGTGCATGATTAAG 
                 26 
               
               
                   
               
               
                 F207 
                 GGATCGGTTGTCGAGTAAGGATC 
                 27 
               
               
                   
               
               
                 F208 
                 CAATTTTCGTACTGAAACATCTTAATCATGCAC 
                 28 
               
               
                   
               
               
                 F209 
                 CGTTTTATTTGATGCCTGGAGATCC 
                 29 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 4 
               
             
            
               
                   
               
               
                 protein sequences of the LuxA variants 
               
            
           
           
               
               
               
            
               
                   
                 Nb of 
                   
               
               
                 Protein sequence 
                 mismatches 
                 SEQ ID NO: 
               
               
                   
               
               
                 LuxAM1V0 Reconstruction 
                 13 
                 30 
               
               
                 LuxAM1V1 Variant 
                 23 
                 31 
               
               
                 LuxAM1V2 Variant 
                 16 
                 32 
               
               
                 LuxAM1V3 Variant 
                 19 
                 33 
               
               
                 LuxAM1V4 Variant 
                 27 
                 34 
               
               
                 LuxAM1V5 Variant 
                 14 
                 35 
               
               
                 LuxAM1V6 Variant 
                 18 
                 36 
               
               
                 LuxAM1V7 Variant 
                 13 
                 37 
               
               
                 LuxAM1V8 Variant 
                 16 
                 38 
               
               
                 LuxAM1V9 Variant 
                 15 
                 39 
               
               
                 LuxAM1V10 Variant 
                 27 
                 40 
               
               
                 LuxAM1V11 Variant 
                 16 
                 41 
               
               
                 LuxAM1V12 Variant 
                 20 
                 42 
               
               
                 LuxAM1V13 Variant 
                 10 
                 43 
               
               
                 LuxAM1V14 Variant 
                 27 
                 44 
               
               
                 LuxAM1V15 Variant 
                 20 
                 45 
               
               
                 LuxAM1V16 Variant 
                 25 
                 46 
               
               
                 LuxAM1V17 Variant 
                 25 
                 47 
               
               
                 LuxAM1V18 Variant 
                 15 
                 48 
               
               
                 LuxAM1V19 Variant 
                 23 
                 49 
               
               
                 LuxAM1V20 Variant 
                 34 
                 50 
               
               
                 LuxAM1V21 Variant 
                 18 
                 51 
               
               
                 LuxAM1V22 Variant 
                 11 
                 52 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 4 
               
             
            
               
                   
               
               
                 DNA sequences of luxA variants with homologies for  
               
               
                 assembly in plasmid pFD72 
               
            
           
           
               
               
               
            
               
                   
                 Nb of 
                   
               
               
                 DNA sequence 
                 mismatches 
                 SEQ ID NO: 
               
               
                   
               
               
                 luxAM1V0 Reconstruction 
                 13 
                 53 
               
               
                 luxAM1V1 Variant 
                 23 
                 54 
               
               
                 luxAM1V2 Variant 
                 16 
                 55 
               
               
                 luxAM1V3 Variant 
                 19 
                 56 
               
               
                 luxAM1V4 Variant 
                 27 
                 57 
               
               
                 luxAM1V5 Variant 
                 14 
                 58 
               
               
                 luxAM1V6 Variant 
                 18 
                 59 
               
               
                 luxAM1V7 Variant 
                 13 
                 60 
               
               
                 luxAM1V8 Variant 
                 16 
                 61 
               
               
                 luxAM1V9 Variant 
                 15 
                 62 
               
               
                 luxAM1V10 Variant 
                 27 
                 63 
               
               
                 luxAM1V11 Variant 
                 16 
                 64 
               
               
                 luxAM1V12 Variant 
                 20 
                 65 
               
               
                 luxAM1V13 Variant 
                 10 
                 66 
               
               
                 luxAM1V14 Variant 
                 27 
                 67 
               
               
                 luxAM1V15 Variant 
                 20 
                 68 
               
               
                 luxAM1V16 Variant 
                 25 
                 69 
               
               
                 luxAM1V17 Variant 
                 25 
                 70 
               
               
                 luxAM1V18 Variant 
                 15 
                 71 
               
               
                 luxAM1V19 Variant 
                 23 
                 72 
               
               
                 luxAM1V20 Variant 
                 34 
                 73 
               
               
                 luxAM1V21 Variant 
                 18 
                 74 
               
               
                 luxAM1V22 Variant 
                 11 
                 75 
               
               
                 pFD72 
                 na 
                 76