Update README.md (#2)

- Update README.md (0c9bb0a56d3e4a6b4912c76a2ffc5a53ea36e09a)


Co-authored-by: Sebastian Lindner <Bienenwolf655@users.noreply.huggingface.co>

README.md

@@ -4,70 +4,172 @@ pipeline_tag: translation
 tags:
 - chemistry
 - biology
+widget:
+- text: "r2sNC1=NC=NC2=C1N=CN2[C@@H]1O[C@H](COP(=O)([O-])OP(=O)([O-])OP(=O)([O-])[O-])[C@@H](O)[C@H]1O.*N[C@@H](CO)C(*)=O>>NC1=NC=NC2=C1N=CN2[C@@H]1O[C@H](COP(=O)([O-])OP(=O)([O-])[O-])[C@@H](O)[C@H]1O.[H+].*N[C@@H](COP(=O)([O-])[O-])C(*)=O</s>"
+inference:
+  parameters:
+    top_k: 15
+    top_p: 0.92
+    repetition_penalty: 1.2
 ---
 
 # **Contributors**
 
-- Sebastian Lindner (GitHub [@Bienenwolf655](https://www.google.com); Twitter @) 
-- Michael Heinzinger (GitHub @mheinzinger; Twitter @)
-- Noelia Ferruz (GitHub @noeliaferruz; Twitter @ferruz_noelia; Webpage: www.aiproteindesign.com )
+- Sebastian Lindner (GitHub [@Bienenwolf655](https://github.com/Bienenwolf655); Twitter [@lindner_seb](https://twitter.com/lindner_seb)) 
+- Michael Heinzinger (GitHub [@mheinzinger](https://github.com/@mheinzinger); Twitter [@HeinzingerM](https://twitter.com/lindner_seb))
+- Noelia Ferruz (GitHub [@noeliaferruz](https://github.com/@noeliaferruz); Twitter [@ferruz_noelia](https://twitter.com/ferruz_noelia); Webpage: [www.aiproteindesign.com](www.aiproteindesign.com) )
 
-# **REXyme: A Translation Machine for the Generation of New-to-Nature Enzymes**
+
+# **REXzyme: A Translation Machine for the Generation of New-to-Nature Enzymes**
 **Work in Progress**
 
-REXyme (Reaction to Enzyme) (manuscript in preparation) is a translation machine for the generation of enzyme that catalize user-defined reactions. 
+REXzyme (Reaction to Enzyme) (manuscript in preparation) is a translation machine for the generation of enzyme that catalize user-defined reactions.
+
+![Inference of REXzyme](./figures__.004.jpeg)
+
 It is possible to provide fine-grained input at the substrate level. 
 Akin to how translation machines have learned to translate between complex language pairs with great success, 
 often diverging in their representation at the character level, (Japanese - English), we posit that an advanced architecture will
-be able to translate between the chemical and sequence spaces. REXyme was trained on a set of xx reactions and yy enzyme pairs and it produces
+be able to translate between the chemical and sequence spaces. REXzyme was trained on a set of xx reactions and yy enzyme pairs and it produces
 sequences that putatitely perform their intended reactions.
 
-To run it, you will need to provide a reaction in the SMILE format (Simplified molecular-input line-entry system),
- which you can do online here: xxxx
+To run it, you will need to provide a reaction in the SMILE format (Simplified molecular-input line-entry system), which you can do online here: https://cactus.nci.nih.gov/chemical/structure.
 
-We are still working in the analysis of the model for different tasks, including experimental testing. 
-See below for information about the models' performance in different in-silico tasks and how to generate your own enzymes. 
+After converting each of the reaction components you should combine them in the following scheme : ReactantA.ReactantB>AgentA>ProductA.ProductB
+Additionally prepend the task suffix "r2s" and append the eos token "</s>"
+e.g. for the carbonic anhydrase "r2sO.COO>>HCOOO.[H+]</s>"
 
-## **Model description**
-REXyme is based on the [Efficient T5 Transformer](xx) architecture (which in turn is very similar to the current version of Google Translator)
-and contains xx layers
-with a model dimensionality of xx, totaling xx million parameters. 
+or via this simple python script:
 
-REXyme is a translation machine trained on the xx database containing xx reaction-enzyme pairs. 
-The pre-training was done on pairs of smiles and ... (fasta headers?),
+```python
+#  left reactants (seperated by +) seperated by a equal sign from the products (seperated by +)
+reactions =  "CO2 + H2O =  carbonic acid + H+"
+# agents (seperated by +) 
+agent = ""
 
-ZymCTRL was trained with an autoregressive objective (this is not right, check it ??) i.e., the model learns to predict a missing 
-token in the encoder's input. Hence,
-the model learns the dependencies among protein sequence features that enable a specific enzymatic reaction.
+# https://stackoverflow.com/questions/54930121/converting-molecule-name-to-smiles
+from urllib.request import urlopen
+from urllib.parse import quote
 
-Sebastian check if this applies?? There are stark differences in the number of members among EC classes, and for this reason, we also tokenized the EC numbers.
-In this manner, EC numbers '2.7.1.1' and '2.7.1.2' share the first three tokens (six, including separators), and hence the model can infer that
-there are relationships between the two classes.
+def CIRconvert(ids):
+    try:
+        url = 'http://cactus.nci.nih.gov/chemical/structure/' + quote(ids) + '/smiles'
+        ans = urlopen(url).read().decode('utf8')
+        return ans
+    except:
+        return 'Did not work'
 
-The figure below summarizes the process of training: (add figure)
+reagent = [CIRconvert(i) for i in reactions.replace(' ','').split('=')[0].split('+') if i != ""]
+agent = [CIRconvert(i) for i in agent.replace(' ','').split('+') if i != ""]
+product = [CIRconvert(i) for i in reactions.replace(' ','').split('=')[1].split('+') if i != ""]
+f"r2s{'.'.join(reagent)}>{'.'.join(agent)}>{'.'.join(product)}</s>"
+```
 
-## **Model Performance**
+We are still working in the analysis of the model for different tasks, including experimental testing. 
+See below for information about the models' performance in different in-silico tasks and how to generate your own enzymes. 
 
-- explain dataset curation
-- general descriptors (esmfold, iuored.. )
-- second pgp
-- mmseqs (Average?)
+## **Model description**
 
+REXzyme is based on the [Efficient T5 Large Transformer](https://huggingface.co/google/t5-efficient-large) architecture (which in turn is very similar to the current version of Google Translator)
+and contains 48 (24 el/ 24 dl) layers with a model dimensionality of 1024, totaling 737.72 million parameters. 
 
-## **How to generate from REXyme**
-REXyme can be used with the HuggingFace transformer python package.
-Detailed installation instructions can be found here: https://huggingface.co/docs/transformers/installation
+REXzyme is a translation machine trained on portion the RHEA database containing 31970152 reaction-enzyme pairs. 
+The pre-training was done on pairs of smiles and amino acid sequences, tokenized with a char-level 
+Sentencepiece tokenizer. Note that two seperate tokenizers were used for input and labels.
 
-Since REXyme has been trained on the objective of machine translation, users have to specify a chemical reaction, specified in the format of SMILES.
+REXzyme was pre-trained with a supervised translation objective  i.e., the model learned to use the continous representation of the reaction from the encoder to autoregressivly (causual language modeling) produce the output trying to match the shifted right target enzyme sequence. Hence, the model learns the dependencies among protein sequence features that enable a specific enzymatic reaction.
 
-[please seb include snippet to generate sequences]
+There are stark differences in the number of members among Reaction classes, and for this reason. Since we are tokenizing the reaction smiles on a char level, classes with few reactions can profit from the knwodledge gained for classes catalyzing similar reactions that have a lot of members.
 
+The figure below summarizes the process of training: (add figure) [STILL MISSING!]
 
-## **A word of caution** 
 
-- We have not yet fully tested the ability of the model for the generation of new-to-nature enzymes, i.e.,
-  with chemical reactions that do not appear in Nature (and hence neither in the training set). While this is the intended objective of our work,
-  it is very much work in progress. We'll uptadate the model and documentation shortly.
-  
+## **Model Performance**
 
+- **Dataset curation**
+<br/><br/>
+- **General descriptors**
+
+    | Method                | Natural     | Generated                |
+    | :---                  |    :----:   |          ---:            |
+    | **IUPRED3 (ordered)** | 99.9%       | 99.9%                    |
+    | **ESMFold**           | 85.03       | 71.59 (selected: 79.82)  |
+    | **FlDPnn**            | missing     | missing                  |
+    | **PSIpred**           | missing     | missing                  |
+<br/><br/>
+
+- **PGP pipeline**
+
+    | Method      | Natural | Generated |
+    | :---        | :----   |      :--- |
+    | **Disorder**      | 11.473       | 11.467   |
+    | **pggp3**   | L: 42%, H: 41%, E:18% | L: 45%, H: 39%, E: 16%|
+    | **pggp8**   | C:25%, H:38% T:10%, S:5%, I:0%, E:19%, G:2%, B:0% | C:29%, H:38% T:10%, S:4%, I:0%, E:17%, G:3%, B:0%|
+    | **CATH Classes**   | Mainly Beta: 6%, Alpha Beta: 78%, Mainly Alpha: 16%, Special: 0%, Few Secondary Structures: 0% |  Mainly Beta: 4%, Alpha Beta: 87%, Mainly Alpha: 9%, Special: 0%, Few Secondary Structures: 0%|
+    | **Transmembrane Prediction**  | Membrane: 9%, Soluble: 91% | Membrane: 9%, Soluble: 91%|
+    | **Conservation**      | High: 37%, Low: 33%      | High: 38%, Low: 33%  |
+    | **Localization**   | Cytop.: 66%, Nucleus: 4%, Extracellular: 6%, PM: 4%, ER: 11%, Lysosome/Vacuole: 1%, Mito.: 6%, Plastid: 1%, Golgi: 1%, Perox.: 1% | Cytop.: 85%, Nucleus: 2%, Extracellular: 6%, PM: 1%, ER: 6%, Lysosome/Vacuole: 0%, Mito.: 4%, Plastid: 0%, Golgi: 0%, Perox.: 0%|
+<br/><br/>
+
+- **Sequence similarity to the natural space**
+
+    | Syntax              | Identity    | Alignment length |
+    | :---                |    :----:   |          ---:    |
+    | **Generated**       | 74.29%      | 406.0            |
+    | **Selection (<70%)**| 57.20%      | 338.1            |
+<br/><br/>
+
+## **How to generate from REXzyme**
+REXzyme can be used with the HuggingFace transformer python package.
+Detailed installation instructions can be found [here](https://huggingface.co/docs/transformers/installation)
+
+Since REXzyme has been trained on the objective of machine translation, users have to specify a chemical reaction, specified in the format of SMILES.
+
+Disclaimer: Although the perplexity gets computed here it is not the best selection criteria. Usually the BLEU score is deployed for translation evaluation, but this score would enforce a high sequence similarity thus not *de novo* design. We recommend generating many sequences and selecting them by plDDT as well as low identity.
+
+```python
+from datasets import load_from_disk
+from transformers import AutoTokenizer
+from transformers import T5Tokenizer, T5ForConditionalGeneration
+import math
+import torch
+from tqdm import tqdm
+import pickle
+tokenizer_aa = AutoTokenizer.from_pretrained('/path/to//tokenizer_aa')
+tokenizer_smiles = AutoTokenizer.from_pretrained('/path/to//tokenizer_smiles')
+
+model = T5ForConditionalGeneration.from_pretrained("/path/to/REXzyme").cuda()
+print(model.generation_config)
+reactions = ["NC1=NC=NC2=C1N=CN2[C@@H]1O[C@H](COP(=O)([O-])OP(=O)([O-])OP(=O)([O-])[O-])[C@@H](O)[C@H]1O.*N[C@@H](CO)C(*)=O>>NC1=NC=NC2=C1N=CN2[C@@H]1O[C@H](COP(=O)([O-])OP(=O)([O-])[O-])[C@@H](O)[C@H]1O.[H+].*N[C@@H](COP(=O)([O-])[O-])C(*)=O"]
+
+def calculatePerplexity(inputs,model):
+    '''Function to compute perplexity'''
+    a=tokenizer_aa.decode(inputs)
+    b=tokenizer_aa(a, return_tensors="pt").input_ids.to(device='cuda')
+    b = torch.stack([[b[b!=tokenizer_aa.pad_token_id]] for label in b][0])
+    with torch.no_grad():
+        outputs = model(b, labels=b)
+    loss, logits = outputs[:2]
+    return math.exp(loss)
+
+
+for idx,i in tqdm(enumerate(reactions)):
+    input_ids = tokenizer_smiles(f"r2s{i}</s>", return_tensors="pt").input_ids.to(device='cuda')
+    print(f'Generating for {i}')
+    ppls_total = []
+    for _ in range(4):
+        outputs = model.generate(input_ids,
+                top_k=15,
+                top_p = 0.92,
+                repetition_penalty=1.2,
+                max_length=1024,
+                do_sample=True,
+                num_return_sequences=25)
+        ppls = [(tokenizer_aa.decode(output,skip_special_tokens=True), calculatePerplexity(output, model),len(tokenizer_aa.decode(output))) for output in tqdm(outputs)]
+        ppls_total.extend(ppls)
+```
+
+## **A word of caution** 
 
+- We have not yet fully tested the ability of the model for the generation of new-to-nature enzymes, i.e.,
+  with chemical reactions that do not appear in Nature (and hence neither in the training set). While this is the intended objective of our work, it is very much work in progress. We'll uptadate the model and documentation shortly.