# A Framework for Enhancing Big Data Integration in Biological Domain Using Distributed Processing
## 4 Methodology
### Environmental Setup  
Implementation and testing of the big data integration framework were conducted using the following settings and equipment:  
Figure 1: Big Data Integration Framework. 1. Equipment: * Dell PowerEdge T620 server with SATA (7.2K) hard drive is and with a VMware Workstation Pro 14 software to create a set of six virtual machines (VM); each machine runs on Ubuntu 16.04 LTS, Intel(r) Xeon(r) processor E5-2600 product family \(\times\) 4 processors. One VM works as a master with 14 GB of RAM and the rest work as slaves with 8 GB. * Samba file and print service, which is an open-source implementation of the Server Message Block/Common Internet File System (SMB/CIFS) protocols that provides the sharing of files and printers between master and slaves. * Protegeg [12] is an open-source ontology editor and knowledge management system. We will use it to validate or test the logical consistency of all ontologies. 2. Programming language: * JAVA programming language version 1.8. * Libraries: * Semantic measure library and toolkit (SML) [56] to read and process the GO. * JCIFS library [57] to access and manage shared data on a Samba Server installed on the master node using JAVA. * Jena is a Java-based programming toolkit. * Pellet and HermiT are used to check the ontology consistency and identifying subsumption relationships between classes. Pellet reasoner is an open-source based on OWL2 reasoner using Java programming language. It is used with Jena and OWL API libraries. HermiT limitations are based on OWL language. 3. Algorithms: * GO Split Algorithm to generate N GO Splits, where N ranged from 1 to 5, because in our settings we can have 2, 3, or 5 slaves. 4. Input: Due to the hardware limitation (hard drive size) in our system, we cannot integrate all the input data; therefore, a sample of input data was selected. Samples are generated based on collecting a line from an input sample file if its gene ID is in the NCBI genes list that has a relation with any gene in GO. To reduce the sample size, one line for each gene ID is taken because some gene IDs are repeated in many lines. Input sample files are: * GO [47] in Open Biomedical Ontologies (OBO) file format [48]; it is composed of 36,638 genes. * gene_info [58]: text file of information that has about 2,013,945 NCBI genes. A sample of 56,603 genes was selected. * gene2 go [59]: text file that reports about 2,070,137 relations between genes from GO and genes from NCBI. A sample of 55,859 relations was selected. * gene_neighbors [60]: text file that represents neighboring genes for all genes located on a given genomic sequence. A sample of 56,647 relations was selected. * gene2 ensembl [61]: text file of 1,907,407 matches between NCBI genes and Ensembl annotations based on the comparison of RNA and protein features. A sample of 56,647 relations was selected. * gene2 pubmed [62]: text file report that has about 11,165,891 relations to link genes from NCBI to PubMed ID. A sample of 56,044 relations was selected. * gene2 sts [63]: text file report that has about 1,173,647 relations to link genes from NCBI to UniSTS ID. A sample of 56,647 relations was selected. * gene2 accession [64]: text file of 18,142,094 accessions related to GeneID of the genes mentioned in the NCBI gene information file. It contains sequences from the international sequence collaboration, Swiss-Prot, and RefSeq. A sample of 56,498 accessions was selected. gene2 vega [65]: text file is composed of 84,828 matches between NCBI genes and Vega annotations. A sample of 29,496 matches was selected. * gene2 unigene [66]: text file report that has about 589,221 relations to link genes from NCBI to the UniGene cluster. A sample of 55,891 relations was selected. Evaluation Measures
* Logical Consistency measure: an ontology is marked as passing if ontology passes the logical consistency test and is marked as failing otherwise. Logical consistency tests are Jena, Pellet, and HermiT tests. The Jena test is done by loading ontology/sub-ontology in a Java program using the Jena library.