Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion
  Enhances Protein Representations
Authors: name: Hazem Alsamkary, name: Mohamed Elshaffei, name: Mohamed Elkerdawy, name: Ahmed Elnaggar

Protein language models (PLMs) have emerged as powerful tools to detect
complex patterns of protein sequences. However, the capability of PLMs to fully
capture information on protein sequences might be limited by focusing on single
pre-training tasks. Although adding data modalities or supervised objectives
can improve the performance of PLMs, pre-training often remains focused on
denoising corrupted sequences. To push the boundaries of PLMs, our research
investigated a multi-task pre-training strategy. We developed Ankh3, a model
jointly optimized on two objectives: masked language modeling with multiple
masking probabilities and protein sequence completion relying only on protein
sequences as input. This multi-task pre-training demonstrated that PLMs can
learn richer and more generalizable representations solely from protein
sequences. The results demonstrated improved performance in downstream tasks,
such as secondary structure prediction, fluorescence, GB1 fitness, and contact
prediction. The integration of multiple tasks gave the model a more
comprehensive understanding of protein properties, leading to more robust and
accurate predictions.