<figure><img src="image_1.png" /></figure>

HAL
open science

# EnzBert: Deep attention network for enzyme class predictions

Nicolas Buton, Yann Le Cunff, François Coste

► To cite this version:

Nicolas Buton, Yann Le Cunff, François Coste. EnzBert: Deep attention network for enzyme class predictions. JOBIM 2022 - Journées Ouvertes en Biologie, Informatique et Mathématiques, Jul 2022, Rennes, France. pp.1-1. hal-03780557

HAL Id: hal-03780557

https://hal.science/hal-03780557

Submitted on 19 Sep 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# EnzBert: Deep attention network for enzyme class predictions

<figure><img src="image_3.png" /></figure>

## 1. Task

### 1.1 Functional annotation of protein sequences

<figure><img src="image_4.png" /></figure>

## 2. Methods

### 2.1 Attention mechanism and Transformer

Attention is the basis of Transformers, for each position it gives weights to other residues.

<figure><img src="image_5.png" /></figure>

Can account for long distance relations between the elements of the sequence

<figure><img src="image_6.png" /></figure>

### 1.2 Challenges

<figure><img src="image_7.png" /></figure>

* Sequence information (2.1)

<figure><img src="image_8.png" /></figure>

* Underlying 3D structure (2.1)

<figure><img src="image_9.png" /></figure>

* Few examples per category (2.2)

<figure><img src="image_10.png" /></figure>

* Interpretability (2.3)

### 2.2 Two phases training

Unsupervised pre-training on all proteins and supervised fine-tuning step on enzyme classification

Produces a meaningful embedding space for proteins and exploits this during the fine-tuning for better generalization

### 2.3 Interpretability method

Attention maps from multiple layers and multiple heads. A new interpretability method to get residues' importance scores : attention aggregation.

<figure><img src="image_11.png" /></figure>

## 3. Results

### 3.1 Enzyme classification

Comparison with state-of-the-art method at level 1 and 2 of the enzyme commission (EC) number on EC40 dataset

<table><thead><tr><th>Model</th><th>Accuracy at level 1</th><th>Accuracy at level 2</th></tr></thead><tbody><tr><td>UDSMProt<sup>1</sup></td><td>0.87</td><td>0.84</td></tr><tr><td><strong>EnzBert</strong></td><td><strong>0.97</strong></td><td><strong>0.95</strong></td></tr></tbody></table>

Macro metrics, comparison with state-of-the-art method at different levels of the enzyme commission (EC) number on ECPred40 dataset.

<table><thead><tr><th>Model</th><th>Level</th><th>Macro-precision</th><th>Macro-recall</th><th>Macro-f1</th></tr></thead><tbody><tr><td>ECPred<sup>2</sup></td><td>0</td><td>0.78</td><td>0.78</td><td>0.77</td></tr><tr><td><strong>EnzBert</strong></td><td>0</td><td><strong>0.80</strong></td><td><strong>0.80</strong></td><td><strong>0.80</strong></td></tr><tr><td><strong>ECPred</strong></td><td>1</td><td><strong>0.69</strong></td><td><strong>0.84</strong></td><td><strong>0.73</strong></td></tr><tr><td>EnzBert</td><td>1</td><td>0.62</td><td>0.55</td><td>0.55</td></tr><tr><td>ECPred</td><td>2</td><td>0.48</td><td><strong>0.61</strong></td><td>0.51</td></tr><tr><td><strong>EnzBert</strong></td><td>2</td><td><strong>0.57</strong></td><td>0.60</td><td><strong>0.55</strong></td></tr><tr><td>ECPred</td><td>3</td><td>0.50</td><td>0.56</td><td>0.50</td></tr><tr><td><strong>EnzBert</strong></td><td>3</td><td><strong>0.53</strong></td><td><strong>0.57</strong></td><td><strong>0.52</strong></td></tr><tr><td>ECPred</td><td>4</td><td>0.43</td><td>0.41</td><td>0.40</td></tr><tr><td><strong>EnzBert</strong></td><td>4</td><td><strong>0.49</strong></td><td><strong>0.49</strong></td><td><strong>0.47</strong></td></tr></tbody></table>

### 3.2 Interpretability

**Benchmark:** Evaluated on the identification of catalytic residues on enzymes with respect to different interpretability methods.

<table><thead><tr><th>Method type</th><th>PRG-AUC (x100)</th><th>max F-Gain (%)</th><th>Time (s)</th></tr></thead><tbody><tr><td>Random</td><td>42.54 ± 4.37</td><td>69.85 ± 1.04</td><td>–</td></tr><tr><td>Grad</td><td>75.01</td><td>81.27</td><td>4.64</td></tr><tr><td>Grad X input</td><td>63.62</td><td>78.66</td><td>7.74</td></tr><tr><td>Integrated grad</td><td>76.41</td><td>81.70</td><td>2.48 × 10<sup>2</sup></td></tr><tr><td>Attn last layer</td><td>87.80</td><td>85.62</td><td><strong>2.87</strong></td></tr><tr><td><strong>Attn aggregation</strong></td><td><strong>98.01</strong></td><td><strong>96.05</strong></td><td>3.72</td></tr><tr><td>Rollout</td><td>66.08</td><td>76.77</td><td>2.95</td></tr><tr><td>TGLRP</td><td>90.92</td><td>88.56</td><td>4.05 × 10<sup>1</sup></td></tr><tr><td>TGradCam</td><td>81.00</td><td>76.77</td><td>4.35 × 10<sup>1</sup></td></tr><tr><td>LIME</td><td>93.46</td><td>91.44</td><td>1.73 × 10<sup>4</sup></td></tr></tbody></table>

Attention-based interpretability performs best

## 4. Conclusion

* State-of-the-art prediction on enzymes' classes from sequences only for our model EnzBert

* New interpretability method for Transformers that works very well on enzymes

* Prospects: Considering the hierarchy of Enzyme Commission (EC) number may improve our results

## NH(3)-dependent NAD(+)-synthetase enzyme

<figure><img src="image_12.png" /><figcaption>0 MSMOEKIMRE LHVKP SIDPK QEIEDRVNFL KQYVKKTGAK GFV L G E S T L A R L A Q LAVESIREEG GDAQFIAVRL PHGTQQDEDD AQLALKFIKP
1 D KSWKFDIKS TVSAFSDQYQ QETGDQLTDF NKGNVKARTR MIAQYAIGGQ EGLLVL E T O H A A E AVTGFFT KYGDGGADLL PLTGLTKRQG RTLLKELGAP
2 ERLYLKEPTA DLLDEKPQQS DETELGIS EID DYLEGKE VSAKVSEALE KRYSMTEHKR QVPASMFDDW WK</figcaption></figure>

Fig. : Top 5% of residues' importance score of our interpretability method are highlighted in red and catalytic sites are represented by blue spheres.

The two highest scores on attention aggregation correspond to the two catalytic residues of the enzyme.

## Contacts

email : nicolas.beton@irisa.fr

## Bibliography

1. Strodthoff, N., Wagner, P., Wenzel, M., and Samek, W. (2020). UDSMProt: universal deep sequence models for protein classification. Bioinformatics, 36 (8), 2401–2409

2. Dalkiran, A., Rifaioglu, A. S., Martin, M. J., Cetin-Atalay, R., Atalay, V., and Doğan, T. (2018). ECPred: a tool for the prediction of the enzymatic functions of protein sequences based on the EC nomenclature. BMC Bioinformatics, 19 (1), 334

Nicolas Buton

Yann Le Cunff

François Coste

Univ. Rennes, Inria, CNRS, IRISA, Rennes, France