Unfolding AlphaFold. DeepMind AlphaFold Algorithm Explained... | by Naser Tamimi | Towards Data Science

On Monday, November 30th, 2020, the world received good news FINALLY. As you know, I am not talking about vaccine, but about a technology which eventually can lead to thousands of vaccines and treatments. This is a breakthrough not only for life sciences and biology but also for data science and artificial intelligence. Every data scientist should know about it and gets inspired by the impact that this technology could have in our life.
On Monday, DeepMind announced that in a major scientific breakthrough, the latest version of AlphaFold had been recognized as a solution to one of biology’s grand challenges — the “protein folding problem.”
In this article, I am going to explain the DeepMind team approach to the problem in layman’s language. I hope it encourages professional data scientists to attack more similar problems in life sciences and improve the quality of life for all human kinds.
Proteins carry out all the functions necessary for life. During transcription, the DNA serves as a template to form RNA. The resulting RNA is a single-stranded copy of the gene, which next must be translated into a protein molecule [read more]. Decoding DNA, RNA, and proteins can open unlimited doors to treat and cure diseases at their most basic level. DNA is a double-stranded, helical molecule composed of nucleotides known as A, C, G, and T. RNA is a single-stranded molecule composed of nucleotides known as A, C, G, and U. Although both DNA and RNA are important, the only way for them to control cell functionality is through proteins. Without proteins, they are like codes that have never been run.
Proteins are long chains or sequences of amino acids. Depending on the order of these 21 amino acids (ARNDCEQGHILKMFPSTUWYV), we can have different types of protein with different 3D structures (a unique sequence ==> a unique 3D structure). The 3D structure of the protein mainly determines its functionality. That is the reason why many researchers in life sciences and biology (sometimes called structural biologists) are interested in studying the structure of proteins and the relationship between amino acids sequence in a protein and its 3D structure.
Until now, the best way to obtain the 3D structure of a protein was through X-ray crystallography. It takes about a year and costs about $120,000 to obtain the structure of a single protein through X-ray crystallography [source]. On average, a protein is composed of 300 amino acids (residues). Therefore, at least, we have around 21299 (the first residue is fixed) possible proteins that each of them folds in a unique way and therefore has a unique functionality or application. Some of these proteins can cure the most complicated diseases, but we are unable to find them with time-consuming and expensive techniques such as X-ray crystallography. Besides X-ray crystallography, there is no straightforward and accurate solution to find the 3D structure of a protein from its sequence. In other words, there is no analytical solution available to predict the 3D structure of a protein from its sequence. There are some approximate solutions, but their accuracy is well below the acceptable range. That’s why the problem of protein folding is a 50-year old problem. This is such an important problem that every 2 years since 1994, top research teams around the world gather to showcase their prediction models in a competition called CASP (Critical Assessment of Protein Structure Prediction) [link].
The last CASP competition, known as CASP14, was held in November 2020, and the DeepMind team could present a stunning solution, called AlphaFold. This solution achieved a level of accuracy that is almost equivalent to the experimentally determined structures.
The DeepMind team has not published any paper about their new AlphaFold algorithm and its CASP14 approach yet. But in 2019, they published a full paper and released the full code for the previous AlphaFold (that won CASP13 in 2018). In this article, I call the initial 2018 version “AlphaFold” and I call the new 2020 version “AlphaFold2”. We expect these two algorithms will be very similar; therefore, I try to explain what was in the original AlphaFold algorithm and how they solved this problem back in 2018. After that, I compare their 2020 solution with the old one using a short blog that they recently published.
AlphaFold solves the problem in two steps. The first step involves a deep learning predictive modeling (convolutional neural network), and the second step is applying gradient descent optimization.
As I mentioned, the first step is predictive modeling using deep learning (deep convolutional neural networks). The goal of this step is to convert the protein sequence to two important matrices: distance distribution matrix and torsion angle matrix.
With having these two matrices, we can go to the second step, which is optimization (using an iterative gradient descent method). The goal of the second step is to translate these two matrices (distance distribution matrix and torsion angle matrix) into a 3D structure.
Now that you have a general (but probably not clear) idea about their approach let’s take a look at some details.
At this step, like any other machine learning problem, we need input data and target. The goal is to train a model that inputs the data and output the target with acceptable accuracy or error.
What is our input data here? For this problem, our input data are protein sequences. For example, something like (but much longer):
MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLS
It is called a protein sequence. As you know, in machine learning, the most successful predictive models are usually the ones that have good feature engineering. Feature engineering is the process of converting raw data (in this case, raw sequences) into more meaningful data. Feature engineering needs in field knowledge and a good understanding of the science behind the problem. In this case, the feature engineering was done mostly using MSA (multiple sequence alignment) techniques. Imagine you have a long chain of amino acids, and parts of this chain are similar to some parts of other known proteins. Using alignment, we can identify those similar subsets make a feature profile for that section of our sequence. This is a simple explanation of MSA feature engineering. The team did this kind of feature engineering for 31,247 proteins from Protein Data Bank (PDB) [link] and split them into the train and test sets (29,427 and 1,820 proteins, respectively). In addition, they calculated the distance and torsion angles between each pair of protein residues (i.e., amino acids). It will be the target for the predictive model. Now that we have our MSA features (as input) and distogram matrices as a target, we need a model to train and test on them. For this purpose DeepMind team decided to use a Deep Res-Net model (a deep convolutional neural network model). Here is a summary of neural network hyperparameters that they used.
Neural network hyperparameters>> 7 groups of 4 blocks with 256 channels, cycling through dilations 1, 2, 4, 8.>> 48 groups of 4 blocks with 128 channels, cycling through dilations 1, 2, 4, 8.>> Optimization: synchronized stochastic gradient descent>> Batch size: batch of 4 crops on each of 8 GPU workers.0.85 dropout keep probability.>> Nonlinearity: ELU.>> Learning rate: 0.06.>> Auxiliary loss weights: secondary structure: 0.005; accessible surface area: 0.001. These auxiliary losses were cut by a factor 10 after 100 000 steps.>> Learning rate decayed by 50% at 150,000, 200,000, 250,000 and 350,000 steps.>> Training time: about 5 days for 600,000 steps.
Step one outputs are two matrices that show the distance and torsion angles between different parts (or residues) of a protein. To have a 3D structure, we must translate these two matrices into a 3D structure. For example, we must show where inside this protein sequence we have a helix, strand, or any kind of torsion. The process of converting the distogram matrices into 3D structures is done with an iterative gradient descent method. To achieve this goal, the algorithm starts with a smooth 3D structure model updates the structure until the distogram (matrix of distances between different parts of a protein) of the predicted structure gets as close as possible to the distogram (or output) from the deep learning part. This is how the DeepMind team won the 2018 CASP13 competition. But how did they win the 2020 (CASP14) competition? Let’s see if we can find an answer to that.
Obviously, the DeepMind team achieved a breakthrough solution to this problem using their new algorithm. It suggests that the new algorithm must be more advanced than the previous one in different aspects. Unfortunately, we don’t know too much about their new approach, except a short blog post with the following flowchart.
Apparently, the DeepMind team did two important updates in their new approach. First, instead of taking two steps approach, they are solving the problem end to end. This end-to-end solution is based on an attention-based neural network method instead of a simple deep learning model. Attention-based neural networks became popular in 2015 after their encouraging results in natural language processing (NLP) applications. They are not only getting popular for NLP purposes, but also for computer vision purposes as well. DeepMind took advantage of this new deep learning approach in their model and could achieve this historic breakthrough.
I will keep you updated if they publish papers or codes regarding their new algorithm. Please follow me for updates.