# Capturing Protein Domain Structure and Function Using Self-Supervision on Domain Architectures
## 1 Introduction
First, researchers averaged out the protein amino acid vectors, where consequently, this qualitative evaluation is not related in a straightforward way with each learned embedding vector trained per amino-acid. In addition, this averaging-out operation may not reveal the function of the most important sites of a protein, meaning the comparative result holds a low degree of biological significance. Second, we argue that the presented qualitative evaluations lack the ability to assess different learned embeddings in a sophisticated manner. This is because there is no systematic way to _quantitatively_ compare 2-D plots of reduced embedding spaces, each produced by a protein-embedding method in investigation. Indeed for word embeddings, there has been an increase in methods to evaluate word representations intrinsically and in a quantitative manner, such as [26; 27]. Having such evaluation metrics allows us to validate the knowledge acquired _per each word vector_ and use the best-performing space for downstream tasks. However, intrinsic evaluations of current amino acid embedding representations are prevented by incomplete biological metadata at amino acid level, for all disposed proteins, in the UniProtKnowledgeBase (UniProtKB) [28]. To address this limitation in quantitative intrinsic evaluations of protein sequence embeddings, we present our approach with five major contributions:  
1. Our _dom2 vec_ approach is developed, in which words are InterPro annotations and sentences are the domain architectures. Then, we use the _word2 vec_ method to learn the embedding vector representation for each InterPro annotation. 2. A quantitative intrinsic evaluation method is established based on the most significant biological information for a domain--its structure and function. First, we evaluated the learned embedding space for domain hierarchy comparing known domain parent-children relations to cosine similarity of the parent domain. Then, we investigated the performance of a nearest neighbor classifier, \(C^{d}_{nearest}\), to predict the secondary structure class provided by SCOPe secondary structure class [29] and the Enzyme Commission (EC) primary class. Finally, we equally examined the performance of the \(C^{d}_{nearest}\) classifier to predict the GO molecular function class for three example model organisms and one human pathogen. 3. Strikingly, we observed that \(C^{d}_{nearest}\) reaches adequate accuracy, compared to \(C^{d}_{nearest}\) on randomized domains vectors, for secondary structure, enzymatic function, and GO molecular function. Thus, we hypothesized an analogy between word embedding clustering by local linguistic features and protein domains clustering by domain structure and function. 4. To evaluate our embeddings extrinsically, we inputted the learned domains embeddings to simple neural networks and compared their performance with state-of-the-art protein sequence embeddings in three full-protein tasks. We surpassed both SeqVec and ProtVec for the toxin presence and enzymatic primary function prediction task, and we reported comparable results in the cellular location prediction task. 5. The pre-trained protein domain embeddings are available online at [[https://doi.org/10.25835/0039431](https://doi.org/10.25835/0039431)]([https://doi.org/10.25835/0039431](https://doi.org/10.25835/0039431)), to be used by the research community. The remainder of the paper is organized as follows: related work on protein embeddings is reviewed in Section 2. The methodology used to train and evaluate dom2 vec embeddings is described in Section 3. The intrinsic and extrinsic evaluation results are presented in Section 4. In Section 5, we conclude.