**Function Express Server**

**Software Design Description**

**Version 1.0**

**Approved 6/23/2005**

**Adopter: John Rux & Harold Riethman, Wistar**

**Architecture Reviewer: Arumani Manisundaram**

**V/CDE Reviewer: Brian Davis**

Document Change Record

+-----------+-------------+----------------------------+--------------+
| Version   | Date        | Description                | Author       |
| Number    |             |                            |              |
+-----------+-------------+----------------------------+--------------+
| 0.1       | 16^th^ May  | First draft of Function    | Srikanth     |
|           | 2005        | Express Server design      | Adiga        |
+-----------+-------------+----------------------------+--------------+
| 0.2       | 17^th^ May  | Added outline of design    | Pratibha     |
|           | 2005        | document                   | Dhok         |
+-----------+-------------+----------------------------+--------------+
| 0.3       | 18^th^ May  | Added design details of    | Anuj Tiwari  |
|           | 2005        | Entrez gene, UniSTS,       |              |
|           |             | UniGene parsers            |              |
+-----------+-------------+----------------------------+--------------+
| 0.4       | 19^th^ May  | Added design details of    | Sreekanth    |
|           | 2005        | Homologene, Gene Ontology, | Reddy        |
|           |             | PubMed parsers             |              |
+-----------+-------------+----------------------------+--------------+
| 0.5       | 20^th^ May  | Added design details of    | Pratibha     |
|           | 2005        | Postwork processing        | Dhok         |
+-----------+-------------+----------------------------+--------------+
| 0.6       | 23^rd^ May  | Integrated review comments | Pratibha     |
|           | 2005        | suggested by Srikanth      | Dhok         |
+-----------+-------------+----------------------------+--------------+
| 0.7       | 23^rd^ May  | Few more changes           | Srikanth     |
|           | 2005        |                            | Adiga        |
+-----------+-------------+----------------------------+--------------+
| 0.8       | 23^rd^ May  | Document Review            | Janet Stone  |
|           | 2005        |                            |              |
+-----------+-------------+----------------------------+--------------+
| 0.9       | 23^rd^ May  | Document Review            | Rakesh       |
|           | 2005        |                            | Nagarajan    |
+-----------+-------------+----------------------------+--------------+
| 1.0       | 24^th^ May  | Incorporated review        | Sreekanth    |
|           | 2005        | comments                   | Reddy,       |
|           |             |                            | Pratibha     |
|           |             |                            | Dhok,        |
|           |             |                            | Srikanth     |
|           |             |                            | Adiga        |
+-----------+-------------+----------------------------+--------------+
| 1.1       | 21^st^ June | 1.  Incorporated Wistar's  | Pratibha     |
|           | 2005        |     comments               | Dhok, Anuj   |
|           |             |                            | Tiwari,      |
|           |             | 2.  Added more detailed    | Srikanth     |
|           |             |     design for post work   | Adiga        |
|           |             |     component              |              |
|           |             |                            |              |
|           |             | 3.  Added description      |              |
|           |             |     about classes in the   |              |
|           |             |     Appendix               |              |
+-----------+-------------+----------------------------+--------------+
| 1.2       | 23^rd^ June | 4.  Incorporated review    | Pratibha     |
|           | 2005        |     comments suggested by  | Dhok         |
|           |             |     Rakesh                 |              |
+-----------+-------------+----------------------------+--------------+

TABLE OF CONTENTS

[1. Introduction 5](#introduction)

[1.1 References 5](#references)

[1.2 Document Overview 5](#document-overview)

[2. SYSTEM ARCHITECTURE 6](#system-architecture)

[2.1 System Overview 6](#system-overview)

[2.2 CONFIGURATION DETAILS 6](#configuration-details)

[2.2.1 Command file 7](#command-file)

[2.2.1.1 Retrieve Annotation Mode (Annotation Update)
7](#retrieve-annotation-mode-annotation-update)

[2.2.1.2 Add Chip library mode 8](#add-chip-library-mode)

[2.2.1.3 GO Properties file 8](#go-properties-file)

[2.2.1.4 Database Support 9](#database-support)

[3. COMPONENTS 10](#components)

[3.1 schema creation 10](#schema-creation)

[3.2 Retrive annotations 10](#retrive-annotations)

[3.2.1 Download and process 10](#download-and-process)

[3.2.1.1 Download manager 11](#download-manager)

[3.2.1.2 Parser manager 12](#parser-manager)

[3.2.1.3 Database Loader: 12](#database-loader)

[3.2.1.4 (UML) Universal Modeling Language Class diagrams
12](#uml-universal-modeling-language-class-diagrams)

[3.2.1.5 Data Sources downloaded and processed
15](#data-sources-downloaded-and-processed)

[3.2.1.5.1 Entrez Gene 15](#entrez-gene)

[3.2.1.5.2 UniGene 21](#unigene)

[3.2.1.5.3 UniSTS 25](#unists)

[3.2.1.5.4 Gene Ontology 28](#gene-ontology)

[3.2.1.5.5 HomoloGene 31](#homologene)

[3.2.1.5.6 PubMed 35](#pubmed)

[3.2.1.6 Revision History 39](#revision-history)

[3.2.2 Post data processing (Post work)
40](#post-data-processing-post-work)

[3.2.2.1 Chip-Description 41](#chip-description)

[3.2.2.2 Gene-Info 43](#gene-info)

[3.2.2.3 Gene-Info Summary 45](#gene-info-summary)

[3.2.2.4 Gene-Info Microarray Summary 46](#gene-info-microarray-summary)

[3.2.2.5 Chipinfo-Homolo 47](#chipinfo-homolo)

[3.2.2.6 Chipinfo-Term 48](#chipinfo-term)

[3.2.2.7 Chipinfo-OMIM 50](#chipinfo-omim)

[3.3 add chip library 50](#add-chip-library)

[3.3.1 Add Affymetrix chip library 52](#add-affymetrix-chip-library)

[3.3.2 Add caARRAY chip library 53](#add-caarray-chip-library)

[3.4 Query interface 53](#query-interface)

[3.4.1 Basics of Struts 53](#basics-of-struts)

[3.4.1 Basics of Struts 54](#basics-of-struts)

[3.4.1.1 Action class 54](#action-class)

[3.4.1.2 Controller 54](#controller)

[3.4.2 API Access 54](#api-access)

[3.4.3 Gene Search Query Interface 55](#gene-search-query-interface)

[3.4.3.1 Probe centric search 55](#probe-centric-search)

[3.4.3.2 Gene centric Search 55](#gene-centric-search)

[3.4.4 Literature Network Query Interface
55](#literature-network-query-interface)

[3.4.4.1 Probe centric search 55](#probe-centric-search-1)

[3.4.4.2 Gene centric search 56](#gene-centric-search-1)

[*APPENDIX A --* DOCUMENT TERMINOLOGIES AND ACRONYMS
57](#appendix-a-document-terminologies-and-acronyms)

[*APPENDIX B -- ORTHOLOG COMPUTATION DETAILS AND EXAMPLE*
58](#appendix-b-ortholog-computation-details-and-example)

[*APPENDIX C -- CLASSES* 60](#appendix-c-classes)

# Introduction

The Function Express (FE) Software Design Description (SDD) describes
the design decisions, architectural design, and the detailed design
needed to implement the FE software. It provides the required visibility
into the design and details information needed for software support.

## References

1.  [FE Server Use Case
    document](http://cabigcvs.nci.nih.gov/viewcvs/viewcvs.cgi/functionexpress/Deliverables/Use Cases/)

2.  [FE Server Requirements Specifications
    document](http://cabigcvs.nci.nih.gov/viewcvs/viewcvs.cgi/functionexpress/Deliverables/Software Requirements Specification/)

3.  [Other
    documents](http://cabigcvs.nci.nih.gov/viewcvs/viewcvs.cgi/functionexpress/)

## Document Overview

This document captures design descriptions for the FE project. The
remaining design sections are organized as follows:

-   SYSTEM ARCHITECTURE: Describes the overall architecture of the
    system.

-   COMPONENTS: Describes the design of all the components of FE Server.
    It includes individual parsers, post work, query interface and so
    forth.

-   APPENDIX A -- DOCUMENT TERMINOLOGIES AND ACRONYMS: Various
    abbreviations and terminologies used in this document.

-   APPENDIX B -- ORTHOLOG COMPUTATION DETAILS AND EXAMPLE

# SYSTEM ARCHITECTURE

## System Overview

This release of FE Server is comprised of the following three
components:

1.  A command-prompt based server application which runs in the
    following three modes:

    a.  Create database schema mode

    b.  Annotations Update mode

    c.  Add chip library mode

2.  A Java Struts based web interface to query annotations

3.  HTTP (API) Application Programming Interface based access to query
    annotations.

The following block diagram outlines the high level architecture of the
three components.

## CONFIGURATION DETAILS

The command prompt based server application takes the following
parameters as command line arguments:

\<mode\> \<dbUserID\> \<dbUserPassword\> \<dbName\> \<driverName\>
\<dbURL\> \<ftpCommandFile\>

The command line parameters specify the mode in which the server should
run.

  ---------------- ------------------------------------------------------
  Parameter        Value

  mode             mode \[1=Annotation Update , 2=Add
                   Chip,3=createDBSchema\]

  dbUserID         database user name

  dbUserPassword   database user\'s password

  dbName           database (SID) System Identifier

  drivername       Oracle driver name \[oracle.jdbc.driver.OracleDriver\]
                   (Similar, MySQL)

  dbURL            URL for database constructed with passed parameters

  ftpCommandFile   CommandFile.txt (Optional parameter- Annotation Update
                   / Add Chip mode require this parameter)
  ---------------- ------------------------------------------------------

### Command file

The section below describes the details of the CommandFile.txt file for
Annotation Update and Add Chip Library modes.

#### Retrieve Annotation Mode (Annotation Update)

The CommandFile.txt file contains the details about the annotation data
sources to download. For each data-source, it specifies protocol to use
for download, location of the data source, file name to download etc.
The record of each data source is terminated by ####.

The table below lists the details of parameters to be specified in the
Command file for each data source.

+-------------+--------------------------------+-----------------------+
| **Field**   | **Description**                | **Permissible         |
|             |                                | Values**              |
+-------------+--------------------------------+-----------------------+
| TYPE        | Data source type               | HTTP, FTP             |
+-------------+--------------------------------+-----------------------+
| SITE_URL    | URL of FTP/HTTP server         |                       |
+-------------+--------------------------------+-----------------------+
| BASE_DIR    | Directory on FTP/HTTP server   |                       |
|             | which holds the data files     |                       |
+-------------+--------------------------------+-----------------------+
| FILE_NAME   | The data file name             |                       |
+-------------+--------------------------------+-----------------------+
| USER_ID     | Username to access FTP data    |                       |
|             | files                          |                       |
+-------------+--------------------------------+-----------------------+
| PASSWD      | Password of the user to access |                       |
|             | remote data files              |                       |
+-------------+--------------------------------+-----------------------+
| LOCAL_FILE  | The data file name on the      |                       |
|             | local machine                  |                       |
|             |                                |                       |
|             | **Note:** This field is used   |                       |
|             | only while running the FE      |                       |
|             | Server on machines where the   |                       |
|             | data files to be processed are |                       |
|             | present on the same machine.   |                       |
+-------------+--------------------------------+-----------------------+
| PROXY_HOST  | Proxy machine name or IP       |                       |
|             | address                        |                       |
+-------------+--------------------------------+-----------------------+
| PROXY_PORT  | Proxy machine port number      |                       |
+-------------+--------------------------------+-----------------------+
| DATABASE    | The external database name     | CHIPINFORMATION       |
|             |                                |                       |
|             |                                | UNIGENE               |
|             |                                |                       |
|             |                                | ENTREZGENE            |
|             |                                |                       |
|             |                                | HOMOLOGENE            |
|             |                                |                       |
|             |                                | UNISTS                |
|             |                                |                       |
|             |                                | PUBMED                |
|             |                                |                       |
|             |                                | GO                    |
+-------------+--------------------------------+-----------------------+

#### Add Chip library mode

For add chip library mode, the command file contains the following
parameters.

+------------+------------------------+--------------------------------+
| **Field**  | **Description**        | **Permissible Values**         |
+------------+------------------------+--------------------------------+
| LOCAL_FILE | The chip library file  |                                |
|            | name with path         |                                |
+------------+------------------------+--------------------------------+
| DATABASE   | This is set to keyword | CHIPINFORMATION                |
|            | "CHIPINFORMATION" to   |                                |
|            | indicate that FE       |                                |
|            | Server should be run   |                                |
|            | in add chip library    |                                |
|            | mode.                  |                                |
+------------+------------------------+--------------------------------+
| IN         | Input format of Chip   | One or more of:                |
| PUT_FORMAT | library file           |                                |
|            |                        | A (Accession Number)           |
|            |                        |                                |
|            |                        | U (UniGene ID)                 |
|            |                        |                                |
|            |                        | G (Entrez GeneID)              |
+------------+------------------------+--------------------------------+

#### GO Properties file

The GO parser requires GO configuration file to get the term identifier
associated with the root node '**Gene_Ontology'**. The configuration
file has the following parameters.

+----------------+----------------------+------------------------------+
| **Field**      | **Description**      | **Permissible Values**       |
+----------------+----------------------+------------------------------+
| XML_ROOT       | The value root node  | ALL                          |
|                | in GO XML data       |                              |
|                | source file.         |                              |
+----------------+----------------------+------------------------------+
| PARENT_TERMID  | The term ID to be    | GO:0003673                   |
|                | associated with the  |                              |
|                | root node.           |                              |
|                |                      |                              |
|                | Note: This is set to |                              |
|                | a deprecated term ID |                              |
|                | so that it does not  |                              |
|                | clash with the       |                              |
|                | current term IDs.    |                              |
+----------------+----------------------+------------------------------+
| P              | The term name of the | Gene_Ontology                |
| ARENT_TERMNAME | root node            |                              |
+----------------+----------------------+------------------------------+

The following diagram outlines the high-level block diagram of FE Server
in the *update* mode

1.  Create intermediate base tables

    a.  Download and process raw data

    b.  Load data into base tables

2.  Create final summary tables

    a.  Read and correlate base tables' data

    b.  Load data into summary tables

#### Database Support

The FE Server will support MySQL 4.1 and Oracle 9i databases. All the
database specific queries in FE server will be moved to a properties
file. Also every database will have different set database creation
scripts.

# COMPONENTS

## schema creation

This component creates the database schema to store microarray chip
libraries information along with annotations. The table scripts required
to create Function Express server schema are in an XML format. Table
creation scripts differ from database to database. Hence according to
the choice of database this module will read appropriate schema creation
script, will parse them and execute DDL statements. This component also
adds default entries into tables.

When FE server runs in this mode it requires the following parameters as
command line arguments.

\<mode\> \<dbUserID\> \<dbUserPassword\> \<dbName\> \<driverName\>
\<dbURL\>

Where mode=3 for schema creation mode.

This FE Server reads the table creation scripts from the 'scripts'
folder under BASE_DIR where BASE_DIR is specified in the configuration
file.

## Retrive annotations

This component retrieves annotation data from public data sources and
populates the database tables appropriately. It has two main components:

-   Download and process

-   Post data processing

### Download and process

Some important points of this component:

-   Responsible for downloading, parsing and loading of data into the
    database base tables.

-   Handles different access mechanisms like HTTP, FTP, APIs

-   Handles different data formats like text, XML

The following block diagram outlines the high level architecture of this
component

-   The download manager downloads the data from different data sources
    and copies the data on the local machine.

-   The parser manager processes the downloaded data and writes the
    output into text files.

-   The database loader uploads the data into text files using high
    throughput load mechanisms like Oracle SQL Loader or MySQL Bulk
    Loader.

-   All the modules are multi-threaded.

-   The database loader will start its job after download manager and
    parser manager finish their jobs.

#### Download manager

-   The download manager is a multi-threaded component, responsible for
    downloading data from the configured data sources.

```{=html}
<!-- -->
```
-   Spawns a thread for every data source.

-   If a source has more than one file, a download thread is spawned for
    every file to be downloaded.

-   There are two types of downloaders: HTTPDownLoader and
    FTPDownLoader. The HTTPDownLoader can be used to download files from
    the HTTP Server while the FTPDownLoader downloads files from the
    (FTP) File Transfer Protocol Server.

-   The DownLoaderFactory can be used to give appropriate instances of
    different downloaders.

-   Once a file is downloaded, a signal is sent to the parser thread to
    indicate that it can start processing the files.

-   After downloading all files, a signal is sent to the database loader
    as well as the parser manager.

#### Parser manager 

> Parser manager is a multi-threaded component responsible for parsing
> the downloaded files.

-   Depending on the source of data (i.e. Entrez Gene, or UniGene etc),
    the appropriate parser thread is spawned.

-   For every data source, there is a corresponding parser.

-   The ParserFactory class is used to get instances for the appropriate
    parser.

-   The parser threads parse the files and write the output in a text
    file.

-   After parsing all source data files, a signal is sent to the
    database loader module.

#### Database Loader:

-   Responsible for uploading the data from text files into the database

-   Uses high throughput uploading mechanisms to load the data

-   There are two types of database loaders: OracleDatabaseLoader and
    MySQLDatabaseLoader. OracleDatabaseLoader uses Oracle **sqlldr**
    utility to load the data from a file and MySQLDatabaseLoader uses
    MySQL bulk loader.

#### (UML) Universal Modeling Language Class diagrams

The UML class diagrams below specifiy the interaction between the
different classes of download and process components.

![](media/image1.wmf){width="6.490972222222222in" height="5.99375in"}

![](media/image2.wmf)

![](media/image3.wmf){width="4.478472222222222in"
height="3.7402777777777776in"}

#### Data Sources downloaded and processed

This section of the design document describes the parser design of each
data source.

In this release, the following annotations sources are processed:

1.  Entrez Gene

2.  UniGene

3.  UniSTS

4.  Gene Ontology

5.  Homologene

6.  Pubmed

The following sections describe the design for processing each section
in detail.

##### Entrez Gene

The NCBIs' Entrez Gene data source file is in .ASN file format. Along
with this NCBI also provides a utility called '**gene2xml.exe'** which
converts the ASN file input into an XML file. The converted XML file can
be parsed easily.

An issue with this approach is when the input ASN file is converted to
an XML format, the resultant XML file size increases to 20GB. In some
cases, it might be difficult for the machine running the parser to
arrange for this large space. This problem is addressed by using JAVA's
'pipe' functionality available through
PipedInputStream/PipedOutputStream classes. The parser will connect the
OutputStream of the process converting the input .ASN file to XML, to
the InputStream in XmlParser class. By doing this, the ASN data will be
converted to XML format and will dynamically be fed to the parser. This
will eliminate the need to store the XML input file onto the hard disk.
Every \<Entrezgene\> node in XML file is parsed and corresponding values
are inserted into appropriate tables.

The command file of Function Express server has following entry for
Entrez Gene data source.

\#####################################################################

TYPE FTP

SITE_URL ftp.ncbi.nih.gov

USER_ID anonymous

PASSWD mail-ID of user

BASE_DIR /gene/DATA/ASN_BINARY/

FILE All_Data.ags.gz

FILE ../gene_history.gz

DATABASE ENTREZGENE

\#####################################################################

The fields that are extracted from the Entrez Gene data source file
'FILE All_Data.ags.gz ' are listed in the table below.

+--------------+--------------------+-------------------------+-------+
| **Field**    | Corresponding tag  | **Description**         | *     |
|              | in input file      |                         | *Data |
|              |                    |                         | T     |
|              |                    |                         | ype** |
+--------------+--------------------+-------------------------+-------+
| GeneID       | Gene-track_geneID  | The unique ID           | Long  |
+--------------+--------------------+-------------------------+-------+
| Taxname      | Org-ref_taxname    | Source species (Homo    | S     |
|              |                    | sapiens, Rattus         | tring |
|              |                    | norvegicus, etc.),      |       |
|              |                    | based on NCBI\'s        |       |
|              |                    | Taxonomy. The first     |       |
|              |                    | letter of each word in  |       |
|              |                    | the value is stored.    |       |
+--------------+--------------------+-------------------------+-------+
| PMID         | PubMedID           | A subset of             | Long  |
|              |                    | publications associated |       |
| (            |                    | with this gene. The     |       |
| multi-valued |                    | multiple values are     |       |
| field)       |                    | separated by ','.       |       |
+--------------+--------------------+-------------------------+-------+
| Post-text of | Prot-ref_name_E    | The name of the product | S     |
| Protein db   |                    | of this transcript      | tring |
| in Product   |                    |                         |       |
| (m           |                    |                         |       |
| ulti-valued) |                    |                         |       |
+--------------+--------------------+-------------------------+-------+
| Official     | Ge                 | Gene Name; The gene     | S     |
| full Name    | ne-commentary_text | description, validated  | tring |
|              | (if                | by the appropriate      |       |
|              | Gen                | nomenclature committee, |       |
|              | e-commentary_label | used for gene reports   |       |
|              | is 'Official Full  |                         |       |
|              | Name')             |                         |       |
+--------------+--------------------+-------------------------+-------+
| Official     | Ge                 | Gene symbol; validated  | S     |
| Symbol       | ne-commentary_text | by the appropriate      | tring |
|              | (if                | nomenclature committee, |       |
|              | Gen                | used for reports        |       |
|              | e-commentary_label |                         |       |
|              | is 'Official       |                         |       |
|              | Symbol')           |                         |       |
+--------------+--------------------+-------------------------+-------+
| Maploc       | Gene-ref_maploc    | Cytogenetic location on | S     |
|              |                    | which gene is located   | tring |
+--------------+--------------------+-------------------------+-------+
| Name of      | SubSource_name     | The chromosome where    | S     |
| Chromosome   |                    | the gene is found       | tring |
+--------------+--------------------+-------------------------+-------+
| Tag ID of    | Object_ID (If      | OMIM ID                 | Long  |
| MIM db       | Dbtag_db is 'MIM') |                         |       |
+--------------+--------------------+-------------------------+-------+
| Syn          | Gene-ref_syn_E     | Other symbols           | S     |
|              |                    | associated with the     | tring |
|              |                    | gene. This is a         |       |
|              |                    | multi-valued field,     |       |
|              |                    | with each value         |       |
|              |                    | separated by a ','      |       |
+--------------+--------------------+-------------------------+-------+
| Tag ID of GO | Object_ID (If      | GO term identifier.     | S     |
| db           | Gen                |                         | tring |
|              | e-commentary_label |                         |       |
| (            | is                 |                         |       |
| multi-valued | Process/           |                         |       |
| field)       | Component/Function |                         |       |
|              | & Dbtag_db is      |                         |       |
|              | 'GO')              |                         |       |
+--------------+--------------------+-------------------------+-------+
| Summary      | Entrezgene_summary | Summary description of  | S     |
|              |                    | the gene, its products, | tring |
|              |                    | its significance, and   |       |
|              |                    | mutant phenotypes       |       |
+--------------+--------------------+-------------------------+-------+
| Tag ID of    | Object_ID (if      | STS marker identifier   | Long  |
| UniSTS db    | Gene-              | in UniSTS database      |       |
| (            | commentary_heading |                         |       |
| multi-valued | is 'Markers        |                         |       |
| field)       | (Sequence Tagged   |                         |       |
|              | Sites/STS)' and    |                         |       |
|              | Dbtag_db is        |                         |       |
|              | 'UniSTS')          |                         |       |
+--------------+--------------------+-------------------------+-------+
| FlyBase ID   | Object-ID_str (If  | FlyBase identifier in   | S     |
|              | the Dbtag_db is    | Locus_Fly               | tring |
|              | 'FLYBASE')         |                         |       |
+--------------+--------------------+-------------------------+-------+
| Anchor       | O                  | STS Marker Name         | S     |
| (            | ther-source_anchor |                         | tring |
| multi-valued | (if                |                         |       |
| field)       | Gene-              |                         |       |
|              | commentary_heading |                         |       |
|              | is 'Markers        |                         |       |
|              | (Sequence Tagged   |                         |       |
|              | Sites/STS)')       |                         |       |
+--------------+--------------------+-------------------------+-------+
| Tag str of   | Xtra-Terms_value ( | UniGene Cluster ID      | S     |
| Unigene db   | if Xtra-Terms_Tag  | associated with this    | tring |
|              | is 'UNIGENE' &     | gene                    |       |
| (            | Ge                 |                         |       |
| multi-valued | ne-commentary_text |                         |       |
| field)       | is 'UniGene' )     |                         |       |
+--------------+--------------------+-------------------------+-------+
| Text of      | Ge                 | A phenotype associated  | S     |
| Phenotype    | ne-commentary_text | with a mutation in this | tring |
| (            | (if                | gene.                   |       |
| multi-valued | Gene-              |                         |       |
| field)       | commentary_heading |                         |       |
|              | is 'Phenotypes'    |                         |       |
|              | and                |                         |       |
|              | Ge                 |                         |       |
|              | ne-commentary_type |                         |       |
|              | value =            |                         |       |
|              | \"phenotype\".)    |                         |       |
+--------------+--------------------+-------------------------+-------+
| current-id   | Tag id of "GeneID" | New gene id assigned to | N     |
|              | db of the          | the retired gene        | umber |
|              | current-id node    |                         |       |
+--------------+--------------------+-------------------------+-------+

The tables that are populated from these data sources are listed below.

1\. ENTREZGENE

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EGE_GENEID                 Number                   Not null, Primary
                                                      key

  EGE_ORGANISM               Char                     Not null

  EGE_SYMBOL                 String                   

  EGE_GENE_NAME              String                   

  EGE_SUMMARY                String                   

  EGE_CHROMOSOME_MAP         String                   

  EGE_CHROMOSOME             String                   
  -------------------------- ------------------------ --------------------

2\. ENTREZ_UNIGENE

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EUG_GENEID                 Number                   Not null, Primary
                                                      key

  EUG_ORGANISM               String                   Not null

  EUG_UGID                   String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

3\. ENTREZ_MAP

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EMA_GENEID                 Number                   Not null, Primary
                                                      key

  EMA_ORGANISM               String                   Not null

  EMA_MAPID                  String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

4\. ENTREZ_STS

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EST_GENEID                 Number                   Not null, Primary
                                                      key

  EST_ORGANISM               String                   Not null

  EST_STSID                  String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

5\. ENTREZ_GOID

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EGO_GOID                   String                   Not null, Primary
                                                      key

  EGO_GENEID                 Number                   Not null, Primary
                                                      key

  EGO_ORGANISM               String                   Not null
  -------------------------- ------------------------ --------------------

6\. ENTREZ_OMIM

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EOM_GENEID                 Number                   Not null, Primary
                                                      key

  EOM_ORGANISM               String                   Not null

  EOM_OMIM                   String                   Not null
  -------------------------- ------------------------ --------------------

7\. ENTREZ_PMIDS

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EPI_GENEID                 Number                   Not null, Primary
                                                      key

  EPI_ORGANISM               String                   Not null

  EPI_PUBMEDID               String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

8\. ENTREZ_GENENAMES

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EGE_GENEID                 Number                   Not null, Primary
                                                      key

  EGE_ORGANISM               String                   Not null

  EGE_GENE_NAME              String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

9\. SYSTEM_TERMTREE

The values of \<Gene-ref_maploc\> and \< Object-ID_ID\> nodes are parsed
and corresponding entries are added to SYSTEM_TERMTREE.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  STT_CHILD_TERMID           String                   Not null, Primary
                                                      key

  STT_PARENT_TERMID          String                   Not null, Primary
                                                      key

  STT_ISPARENT               Char                     Not null
  -------------------------- ------------------------ --------------------

10\. SYSTEM_TERMDATA

The values of \<Gene-ref_maploc\> and \< Object-ID_ID\> nodes are parsed
and corresponding entries are added to SYSTEM_TERMDATA.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  STD_TERMID                 String                   Not null, Primary
                                                      key

  STD_TERM                   String                   Not null
  -------------------------- ------------------------ --------------------

11\. ENTREZ_FLY

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EFY_GENEID                 Number                   Not null, Primary
                                                      key

  EFY_FLYID                  String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

12\. ENTREZ_PHENOTYPE

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  EPH_GENEID                 Number                   Not null, Primary
                                                      key

  EPH_ORGANISM               String                   Not null

  EPH\_ PHENOTYPE            String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

**The** gene_history.gz file contains information about GeneIDs that are
no longer current. It's a tab-delimited file which contains following
fields

  -------------- -------------------------------------------- -------------
  Field          Description                                  Data Type

  tax_id         the unique identifier provided by NCBI       Number
                 Taxonomy for the species or strain/isolate   

  GeneID         the current unique identifier for a gene     Number

  Discontinued   the GeneID that is no longer current         Number
  GeneID                                                      

  Discontinued   the symbol that was assigned to the          String
  Symbol         discontinued GeneID, if the discontinued     
                 record was not replaced with another         
  -------------- -------------------------------------------- -------------

The GENE_HISTORY table is populated from this file by extracting
\<Discontinued GeneID, GeneID\> fields from the file.

### 

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  GHI_RETIRED_GENEID         Number                   Not null, Primary
                                                      Key

  GHI_NEW_GENEID             Number                   
  -------------------------- ------------------------ --------------------

**Class diagram:**

The UML class diagram for Entrez Gene is as follows.

![](media/image4.wmf){width="4.59375in" height="2.4881944444444444in"}

##### UniGene

The UniGene data source provides a separate file for each species where
the first two characters of the filename specify the organism name. The
UniGene data file for any species is a compressed (.gz) file, which
contains data for each UniGene cluster in text format. Each UniGene
entry consists of some key-value pairs. Each key is separated from its
value by '\\t' and each key-value pair is separated by '\\n'. While
every UniGene record is separated from the next record by a '\\\\'. The
UniGene parser reads input files for multiple organisms, parses the
key-value pairs stored in each file and populates the base tables. The
command file has the following entry for the UniGene data source.

\########################################################################

TYPE FTP

SITE_URL ftp.ncbi.nih.gov

USER_ID anonymous

PASSWD mail-ID of user

BASE_DIR /repository/UniGene/

FILE Dr.data.gz

FILE Mm.data.gz

FILE Hs.data.gz

FILE Rn.data.gz

FILE Xl.data.gz

FILE Bt.data.gz

FILE Dr.retired.lst.gz

FILE Mm.retired.lst.gzFILE Hs.retired.lst.gz

FILE Rn.retired.lst.gz

FILE XI.retired.lst.gz

FILE Bt.retired.lst.gz

DATABASE UNIGENE

\########################################################################

The algorithm used to parse a UniGene \*. data.gz file is as follows:

+-----------------------------------------------------------------------+
| 1.  If the key is one of 'ID','TITLE','GENE','CYTOBAND','ENTREZGENE', |
|     or 'CHROMOSOME' then store the corresponding values in the        |
|     UNIGENE table.                                                    |
|                                                                       |
| 2.  Else if the key is 'EXPRESS', then fetch all the unique           |
|     tissue-terms from the value field and store it in the UG_EXPRESS  |
|     table.                                                            |
|                                                                       |
| 3.  ELSE if the key is 'PROTSIM' then extract the values of ORG,      |
|     PROTGI, and PROTID sub fields. The first characters from each of  |
|     the words in ORG value are concatenated and stored in the         |
|     UGE_ORGANISM field of UG_PROTSIM table.                           |
|                                                                       |
| 4.  ELSE if the key is 'SEQUENCE', then extract values of ACC, NID    |
|     and PID sub fields. Store NULL for records not having a PID value |
|     into UG_SEQUENCE table.                                           |
+-----------------------------------------------------------------------+

The fields extracted from UniGene \*. data.gz files are listed below.

+------------+-------------------------------------------+------------+
| Field      | Description                               | Data Type  |
+------------+-------------------------------------------+------------+
| ID         | UniGene cluster ID                        | String     |
+------------+-------------------------------------------+------------+
| TITLE      | Title for the cluster                     | String     |
+------------+-------------------------------------------+------------+
| GENE       | Gene symbol                               | String     |
+------------+-------------------------------------------+------------+
| CYTOBAND   | Cytological band                          | String     |
+------------+-------------------------------------------+------------+
| EXPRESS    | \- Tissues of origin for ESTs in cluster  | String     |
|            |                                           |            |
|            | \- Multi value comma separated field      |            |
+------------+-------------------------------------------+------------+
| ENTREZGENE | EntrezGene identifier associated with at  | String     |
|            | least one sequence in this cluster (Hs    |            |
|            | only)                                     |            |
+------------+-------------------------------------------+------------+
| CHROMOSOME | Chromosome in which gene is found. For    | String     |
|            | plants, CHROMOSOME refers to mapping on   |            |
|            | the Arabidopsis genome.                   |            |
+------------+-------------------------------------------+------------+
| PROTSIM    |                                           |            |
| (M         |                                           |            |
| ulti-Value |                                           |            |
| field)     |                                           |            |
+------------+-------------------------------------------+------------+
| Protein    |                                           |            |
| similarity |                                           |            |
| data for   |                                           |            |
| the        |                                           |            |
| sequence   |                                           |            |
| with       |                                           |            |
| highe      |                                           |            |
| st-scoring |                                           |            |
| protein    |                                           |            |
| similarity |                                           |            |
| in this    |                                           |            |
| cluster.   |                                           |            |
|            |                                           |            |
| -   Is a   |                                           |            |
|     m      |                                           |            |
| ulti-value |                                           |            |
|     field  |                                           |            |
|     with   |                                           |            |
|     each   |                                           |            |
|            |                                           |            |
|  sub-value |                                           |            |
|     of the |                                           |            |
|     format |                                           |            |
|     "      |                                           |            |
| key=value" |                                           |            |
|            |                                           |            |
|    format. |                                           |            |
|            |                                           |            |
| -   Each   |                                           |            |
|     sub    |                                           |            |
|     value  |                                           |            |
|     is     |                                           |            |
|            |                                           |            |
|  separated |                                           |            |
|     by a   |                                           |            |
|     s      |                                           |            |
| emi-colon. |                                           |            |
+------------+-------------------------------------------+------------+
| ORG        | Organism                                  | String     |
+------------+-------------------------------------------+------------+
| PROTGI     | Sequence GI of protein                    | String     |
+------------+-------------------------------------------+------------+
| PROTID     | Sequence ID of protein and is one of      | String     |
|            |                                           |            |
|            | 1.  SP: SwissProt                         |            |
|            |                                           |            |
|            | 2.  Ref: RefSeq                           |            |
|            |                                           |            |
|            | 3.  PRF: Protein Research Foundation      |            |
|            |                                           |            |
|            | 4.  PIR: Protein Information Resource     |            |
|            |                                           |            |
|            | 5.  PDB: Protein Data Bank                |            |
|            |                                           |            |
|            | PID: GenPept                              |            |
+------------+-------------------------------------------+------------+
| SEQUENCE   |                                           |            |
| (M         |                                           |            |
| ulti-Value |                                           |            |
| field)     |                                           |            |
+------------+-------------------------------------------+------------+
| Sequence:  |                                           |            |
| Nucleotide |                                           |            |
| sequences  |                                           |            |
| belonging  |                                           |            |
| to each    |                                           |            |
| UniGene    |                                           |            |
| cluster.   |                                           |            |
|            |                                           |            |
| -   Is a   |                                           |            |
|     m      |                                           |            |
| ulti-value |                                           |            |
|     field  |                                           |            |
|     with   |                                           |            |
|     each   |                                           |            |
|            |                                           |            |
|  sub-value |                                           |            |
|     of the |                                           |            |
|     format |                                           |            |
|     "      |                                           |            |
| key=value" |                                           |            |
|            |                                           |            |
|    format. |                                           |            |
|            |                                           |            |
| -   Each   |                                           |            |
|     sub    |                                           |            |
|     value  |                                           |            |
|     is     |                                           |            |
|            |                                           |            |
|  separated |                                           |            |
|     by a   |                                           |            |
|     s      |                                           |            |
| emi-colon. |                                           |            |
+------------+-------------------------------------------+------------+
| ACC        | GenBank/EMBL/DDBJ accession number of     | String     |
|            | sequence                                  |            |
+------------+-------------------------------------------+------------+
| NID        | Unique nucleotide sequence identifier     | String     |
|            | (gi)                                      |            |
+------------+-------------------------------------------+------------+
| PID        | Unique protein sequence identifier (used  | String     |
|            | for non-ESTs)                             |            |
+------------+-------------------------------------------+------------+

The tables populated by the UniGene data source are listed below

1\. UNIGENE

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  UGE_UGID                   String                   Not null, Primary
                                                      key

  UGE_ORGANISM               Character(2)             Not null

  UGE_TITLE                  String                   

  UGE_GENE                   String                   

  UGE_CYTOBAND               String                   

  UGE_LOCUSID                String                   

  UGE_CHROMOSOME             String                   
  -------------------------- ------------------------ --------------------

2\. UG_EXPRESS

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  UEX_UGID                   String                   Not null, Primary
                                                      key

  UEX_TISSUEID               String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

3\. UG_SEQUENCE

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  USE_UGID                   String                   Not null, Primary
                                                      key

  USE_ACC_NO                 String                   Not null, Primary
                                                      key

  USE_NID                    String                   

  USE_PID                    String                   
  -------------------------- ------------------------ --------------------

4\. UG_PROTSIM

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  UPR_UGID                   String                   Not null

  UPR_ORGANISM               Character(2)             Not null

  UPR_PROTGI                 String                   

  UPR_PROTID                 String                   
  -------------------------- ------------------------ --------------------

5\. SYSTEM_TERMTREE

The value of 'EXPRESS' field is inserted into SYSTEM_TERMTREE.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  STT_CHILD_TERMID           String                   Not null, Primary
                                                      key

  STT_PARENT_TERMID          String                   Not null, Primary
                                                      key

  STT_ISPARENT               Character(1)             Not null
  -------------------------- ------------------------ --------------------

6.  SYSTEM_TERMDATA

The value of 'EXPRESS' field is inserted into SYSTEM_TERMDATA.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  STD_TERMID                 String                   Not null, Primary
                                                      key

  STD_TERM                   String                   Not null
  -------------------------- ------------------------ --------------------

The \*.retired.lst.gz files contain details about retired Unigene IDs.
These are the tab-delimited files which allow a comparison of the
current UniGene release with the previous version. It is a list of the
previous UniGene clusters, their composite sequences, and the current
UniGene cluster for each sequence. The UNIGENE_HISTORY table is
populated from these files.

UNIGENE_HISTORY

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  UGH_CURRENTUGID            String                   Not null

  UGH_PREVUGID               String                   Not null

  UGH_ACCESSIONNUMBER        String                   Not null
  -------------------------- ------------------------ --------------------

**Class diagram:**![](media/image5.wmf){width="6.750694444444444in"
height="7.019444444444445in"}

##### UniSTS

The input data for UniSTS comes in two text files (UniSTS.sts and
UniSTS.aliases). The UniSTS.sts contains all the markers in UniSTS that
have PCR product size information. The UniSTS.aliases file contains
unique identifiers and their aliases. Both the files are in text format
where each record is separated by a new line (\\n) character and each
field within a record is separated by TAB (\\t). The command file has
the following entry for a UniSTS data source.

\##################################################

TYPE FTP

SITE_URL ftp.ncbi.nih.gov

USER_ID anonymous

PASSWD email address of user

BASE_DIR repository/UniSTS/

FILE UniSTS.sts

FILE UniSTS.aliases

DATABASE UNISTS

\##################################################

The algorithm used to parse UniSTS data sources is shown below.

+-----------------------------------------------------------------------+
| 1.  For every record in UniSTS.sts file, tokenize the fields. Fields  |
|     are in \< Unique ID, Name, Organism, semicolon separated          |
|     Accession Numbers) format.                                        |
|                                                                       |
|     a.  Insert Unique ID, Name, Organism fields into the UniSTS table |
|                                                                       |
|     b.  For every accession number insert \< Unique ID, accession     |
|         number\> entry into UniSTS_ACCESSION table                    |
|                                                                       |
| 2.  For every record in UniSTS.aliases file                           |
|                                                                       |
|     a.  Tokenize the fields in the record. The fields are Unique ID,  |
|         semicolon separated alias names.                              |
|                                                                       |
|     b.  For every alias name insert \< Unique ID, alias\> entry into  |
|         UniSTS.aliases table                                          |
+-----------------------------------------------------------------------+

The fields extracted from UniSTS.sts file are listed in the following
table.

+------------+-------------------------------------------+------------+
| Field      | Description                               | Data Ttype |
+------------+-------------------------------------------+------------+
| Unique ID  | Unique ID of the (STS) Sequence Tag Site  | Number     |
+------------+-------------------------------------------+------------+
| Name       | Name of the STS                           | String     |
+------------+-------------------------------------------+------------+
| Organism   | Organism from which STS is derived        | String     |
+------------+-------------------------------------------+------------+
| Accession  | Accession number used in deriving STS     | String     |
| Number     | GenBank                                   |            |
| (multiple  |                                           |            |
| values)    | **NOTE:** Accession number values will be |            |
|            | separated by ';' in case more than one    |            |
|            | value is associated. In such cases all    |            |
|            | accession numbers should be extracted.    |            |
+------------+-------------------------------------------+------------+

The fields extracted from UniSTS.aliases are listed in the following
table.

+-------------+------------------------------------------+------------+
| **Field**   | **Description**                          | **Data     |
|             |                                          | Type**     |
+-------------+------------------------------------------+------------+
| Unique ID   | Unique ID of the STS                     | Number     |
+-------------+------------------------------------------+------------+
| Aliases     | The list of aliases for the given STS ID | String     |
| (multiple   |                                          |            |
| values)     | **NOTE:** Multiple aliases are separated |            |
|             | by ';' character.                        |            |
+-------------+------------------------------------------+------------+

The tables populated by UniSTS parser are listed below

1.  UniSTS

This table holds data obtained from the UniSTS.sts file.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  UST_STSID                  Number                   Not null, Primary
                                                      key

  UST_NAME                   String                   

  UST_ORGANISM               Character(2)             
  -------------------------- ------------------------ --------------------

2.  UniSTS_ACCESSION

This table contains UniSTS unique identifier to Access number mapping.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  UST_STSID                  Number                   Not null, Primary
                                                      key

  UST_ACC_NO                 String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

3.  UniSTS_Alias

This table holds data obtained from the UniSTS.aliases file.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  USA_STSID                  Number                   Not null, Primary
                                                      key

  USA_ALIAS                  String                   Not null, Primary
                                                      key
  -------------------------- ------------------------ --------------------

**Class diagram**

The following is the UML class diagram of UniSTS Parser.

![](media/image6.wmf){width="6.372222222222222in"
height="6.060416666666667in"}

##### Gene Ontology

The Gene Ontology (GO) parser processes the GO data source file
specified in the command file and stores processed data into the
database tables. The command file has the following entry for the GO
data source that specifies the protocol to download source file (HTTP),
the site URL from where to download the data file and other parameters
like proxy host and proxy port number.

\########################################################################

TYPE HTTP

SITE_URL
http://www.archive.godatabase.org/latest/go_200504-termdb.rdf-xml.gz

PROXY_HOST 128.252.198.254

PROXY_PORT 80

DATABASE GO

\########################################################################

The GO parser also reads the GO configuration file to get the term
identifier associated with the root node '**Gene_Ontology'**. The
configuration file has the following entry:

XML_ROOT=ALL

PARENT_TERMID=GO:0003673

PARENT_TERMNAME=Gene_Ontology

The fields that are extracted are listed in the following table.

  ---------------------- ----------------------------------- ------------
  Node                   Description                         Data type

  \<go:term\>: Parent                                        
  node in XML                                                

  GO controlled                                              
  vocabulary term                                            
  description                                                

  \<go:accession\>       GO term identifier                  String

  \<go:name\>            GO term name                        String

  \<go:is_a\>            The parent GO term identifier       String

  \<go:part_of\>         The parent GO term identifier       String
  ---------------------- ----------------------------------- ------------

The table below explains the algorithm used to parse the GO source file.

+-----------------------------------------------------------------------+
| 1.  For the root node,                                                |
|                                                                       |
|     a.  Insert PARENT_TERMID and PARENT_TERMNAME parameter values     |
|         read from the GO configuration file (example: GO:0003673,     |
|         Gene_Ontology) into the SYSTEM_TERMDATA table.                |
|                                                                       |
|     b.  Insert STT_CHILD_TERMID as termID read from the GO            |
|         configuration file, STT_PARENT_TERMID as 0 and STT_ISPARENT   |
|         as 1 into the SYSTEM_TERMTREE table.                          |
|                                                                       |
| 2.  For every go:term node                                            |
|                                                                       |
|     a.  Fetch go:accession node value as STT_CHILD_TERMID and         |
|         STD_TERMID                                                    |
|                                                                       |
|     b.  Fetch go:name value as STD_TERM                               |
|                                                                       |
|     c.  Insert STD_TERMID, STD_TERM into SYSTEM_TERMDATA table        |
|                                                                       |
|     d.  For every go:is_a or go:part_of node                          |
|                                                                       |
|         i.  Fetch rdf:resource attribute value. This value is a URL   |
|             having term ID string at the end separated by \# from     |
|             main URL (Example:                                        |
|             <http://www.geneontology.org/go#GO:0005634>). Parse the   |
|             attribute value to get STT_PARENT_TERMID (e.g.            |
|             GO:0005634).                                              |
|                                                                       |
|         ii. If STT_PARENT_TERMID is equal to XML_ROOT in GO           |
|             configuration file, then set STT_PARENT_TERMID to         |
|             PARENT_TERMID in GO configuration file.                   |
|                                                                       |
|         iii. If STT_PARENT_TERMID is not obsolete term, then insert   |
|              \<STT_CHILD_TERMID, STD_TERM, 1\> into SYSTEM_TERMTREE   |
|              table.                                                   |
+-----------------------------------------------------------------------+

The tables populated by the GO parser are listed below.

1.  SYSTEM_TERMDATA

This table holds GO term ID to term name mapping.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  STD_TERMID                 String                   Not null, Primary
                                                      key

  STD_TERM                   String                   Not null
  -------------------------- ------------------------ --------------------

2.  SYSTEM_TERMTREE

> This table holds parent child relationship between GO terms.

  -------------------------- ------------------------ --------------------
  Field                      Data Type                Constraints

  STT_CHILD_TERMID           String                   Not null, Primary
                                                      key

  STT_PARENT_TERMID          String                   Not null, Primary
                                                      key

  STT_ISPARENT               Character(1)             Not null
  -------------------------- ------------------------ --------------------

**Class Diagram:**

The following is the UML Class diagram of GO Parser.

![](media/image7.wmf){width="6.493055555555555in"
height="6.960416666666666in"}

##### HomoloGene

HomoloGene provides all the system information in a single file hosted
on an FTP Server. FE Server has to fetch this file, parse it and load
the data into the database. The file contains a GZip'ed dump of XML
data. After loading the data, orthologs are computed and stored in the
database.

The HomoloGene data file HomoloGene.xml.gz is a GZip\'ed file containing
an XML dump of the HomoloGene build. The Document Type Descriptor (DTD)
of the file can be found at
[ftp://ftp.ncbi.nih.gov/pub/HomoloGene/HomoloGene.dtd.tar](ftp://ftp.ncbi.nih.gov/pub/HomoloGene/homologene.dtd.tar).

The file contains many individual HomoloGene entries with data such as
gene ID, taxonomy ID, protein gene index etc of participating genes and
statistics like alignment similarity between two genes.

The following table describes the fields that will be extracted.

+-------------------+-------------------------------------+------------+
| Node              | Description                         | Data Type  |
+-------------------+-------------------------------------+------------+
| \<                | The details of HomoloGene Entry     | String     |
| HomoloGeneEntry\> |                                     |            |
+-------------------+-------------------------------------+------------+
| \<Homolo          | HomoloGene Entry ID                 | Number     |
| GeneEntry_hg-ID\> |                                     |            |
+-------------------+-------------------------------------+------------+
| \<Homolo          |                                     |            |
| GeneEntry_genes\> |                                     |            |
|                   |                                     |            |
| Parent node in    |                                     |            |
| XML               |                                     |            |
|                   |                                     |            |
| Set of genes in   |                                     |            |
| the corresponding |                                     |            |
| HomoloGene Entry  |                                     |            |
+-------------------+-------------------------------------+------------+
| \<Gene\>          | Description of the Gene             | String     |
+-------------------+-------------------------------------+------------+
| \<Gene_geneID\>   | Gene ID (Entrez Gene ID)            | Number     |
+-------------------+-------------------------------------+------------+
| \<Gene_taxID\>    | Taxonomy ID of the gene             | Number     |
+-------------------+-------------------------------------+------------+
| \<Gene_prot-gi\>  | Protein Gene Index                  | String     |
+-------------------+-------------------------------------+------------+
| \<HomoloGene      |                                     |            |
| Entry_distances\> |                                     |            |
|                   |                                     |            |
| Parent node in    |                                     |            |
| XML               |                                     |            |
|                   |                                     |            |
| HomoloGene Entry  |                                     |            |
| Distances         |                                     |            |
+-------------------+-------------------------------------+------------+
| \<Stats\>         | Statistical details of HomoloGene   |            |
|                   | Entry distances between a pair of   |            |
|                   | genes (gene 1, gene 2)              |            |
+-------------------+-------------------------------------+------------+
| \<Stats_gi1\>     | Gene Index of the gene 1            | String     |
+-------------------+-------------------------------------+------------+
| \<Stats_gi2\>     | Gene index of the gene 2            | String     |
+-------------------+-------------------------------------+------------+
| \<St              | Protein change value between gene 1 | Real       |
| ats_prot-change\> | and gene 2                          |            |
+-------------------+-------------------------------------+------------+
| \<S               | The reciprocality value             | boolean    |
| tats_recip-best\> |                                     |            |
+-------------------+-------------------------------------+------------+

**Parser details:**

The algorithm for parsing each HomoloGene group is as follows:

-   Create an in-memory associative array (map) to store mappings
    between protein gene indices and gene details.

> **Note**: This map is truncated at the beginning of the exploration of
> each homologene group.

-   Create a list of all records corresponding to one HomoloGene entry
    node with each record containing the following fields: gene ID1,
    taxonomyID1, gene ID2, taxonomy ID2, alignment and reciprocal.

> **Note:** The list is symmetric i.e. for any record \<geneID1,
> taxonomyID1, geneID2, taxonomyID2, alignment, reciprocal\> in the
> list, you will find its symmetrical record \<geneID2, taxonomyID2,
> geneID1, taxonomyID1, alignment, reciprocal\>.

-   Compute orthologs as described below

-   Write the list to the homologene_xml table

There are two types of ortholog groups:

1.  **Strict** **ortholog group:** Contains genes only with reciprocal
    value equal to true. These are genes in two or more organisms whose
    protein sequences are most similar to each other given all the other
    genes in all genomes.

2.  **Loose ortholog group:** Contains genes with reciprocal value
    either true or false. Additionally, if there are multiple genes from
    the same organism, then only the gene with the highest alignment
    value should be considered. These are genes where one or more
    gene-gene relationships is NOT reciprocal, indicating that at least
    one pair of genes is not most similar to each other in BOTH
    directions.

The algorithm for computing orthologs is as follows:

For each HomoloGene group in the input file:

1.  Ignore the following:

    a.  All paralogs i.e. the records which have the two participating
        genes from the same species.

    b.  All ortholog pairs with a reciprocal value of false AND
        alignment value less than the user configured cutoff.

2.  For genes with reciprocal value true, create *strict* ortholog
    groups.

3.  For genes with reciprocal value false:

    a.  Create *loose* ortholog groups (i.e. genes with reciprocal value
        equal to false plus the genes belonging to the corresponding
        *strict* ortholog group).

    b.  If multiple genes of the same organism are associated, consider
        only the gene with the highest alignment value to be part of the
        group.

The corresponding data is stored in the following tables.

1\. Homologene_XML

  ----------------------- ----------------------- -----------------------
  **Field**               **Type**                **Constraint**

  HGE_HOMOLOGENE_ID       Number                  not null

  HGE_GENEID1             Number                  not null

  HGE_TAXID1              Number                  not null

  HGE_GENEID1             Number                  not null

  HGE_TAXID1              Number                  not null

  HGE_ALIGNMENT           Number                  not null

  HGE_RECIPROCAL          Character(1)            not null
  ----------------------- ----------------------- -----------------------

2\. Ortholog

  ----------------------- ----------------------- -----------------------
  **Field**               **Type**                **Constraint**

  OTH_ORTHOLOG_ID         Number                  not null

  OTH_GENEID              Number                  not null
  ----------------------- ----------------------- -----------------------

3\. OrthologStartGene

  ----------------------- ----------------------- --------------------------
  **Field**               **Type**                **Constraint**

  OSL_ORTHOLOG_ID         Number                  not null references
                                                  Ortholog(oth_orthologID)

  OSL_GENEID              Number                  not null

  OSL_TRUEORTHOLOG        char(1)                 not null
  ----------------------- ----------------------- --------------------------

4\. Homologene_tmp

Homologene_tmp table is also filled during ortholog computation. The
HomoloGene records that constitute orthologs are inserted into the
homologene_tmp table. The paralogs are also inserted using the following
query. Namely, genes within the same HomoloGene group with the same
taxonomy ID having reciprocal equal to True or with their alignment
greater than or equal to the cutoff are inserted. The following query
accomplishes this:

*insert into homologene_tmp*

*select \* from homologene_xml where hge_taxID1 = hge_taxID2 and
(hge_reciprocal = 'T' or hge_alignment \> cutoff)*

  ----------------------- ----------------------- -----------------------
  **Field**               **Type**                **Constraint**

  HGT_GENEID1             Number                  not null

  HGT_ORGANISM1           Char                    not null

  HGT_GENEID2             Number                  not null

  HGT_ORGANISM2           Char                    not null

  HGT_ALIGNMENT           Number                  not null

  HGT_RECIPROCAL          Char                    not null
  ----------------------- ----------------------- -----------------------

5\. HomoloGene

The only difference between homologene_tmp and homologene table is that
the former has two more columns hge_ugID1, hge_ugID2. To fill these
UniGene ID values, the homologene_tmp table is joined to the
locus_unigene table using the Entrez Gene ID. Thus, following SQL query
is used to populate the homologene table.

*insert into homologene*

*Select hgt_geneID1, hgt_org1, a.lug_ugID, hgt_org1, hgt_geneID2,
hgt_org2, b.lug_ugID, hgt_alignment, hgt_reciprocal from (homologene_tmp
left join locus_unigene a on (hgt_geneID1 = a.lug_locusID)) left join
locus_unigene b on (hgt_geneID2 = b.lug_locusID)*

  ----------------------- ----------------------- -----------------------
  **Field**               **Type**                **Constraint**

  HGE_GENEID1             Number                  not null

  HGE_ORGANISM1           Char                    not null

  HGE_UGID1               String                  

  HGE_GENEID2             Number                  not null

  HGE_ORGANISM2           Char                    not null

  HGE_UGID2               String                  

  HGE_ALIGNMENT           Number                  not null

  HGE_RECIPROCAL          Char                    not null
  ----------------------- ----------------------- -----------------------

####### Class diagram {#class-diagram .unnumbered}

The following is the UML Class diagram of HomoloGene Parser.

![](media/image8.wmf){width="6.497916666666667in"
height="5.863194444444445in"}

##### PubMed

From Entrez Gene data source, for each gene, its official name, official
symbol, aliases, product names will be parsed and stored in the
database. For each gene, the PubMed database is queried for PMIDs of the
articles in which the gene is mentioned by one of its several names.
PubMed has provided a web based search utility called **esearch** for
querying. See
<http://eutils.ncbi.nlm.nih.gov/entrez/query/static/esearch_help.html>
for help.

.

Based on HomoloGene data, orthologous genes are grouped together and
PMIDs are associated with these groups. The correlation among gene
groups is computed and from these results correlations between two pairs
of genes is calculated.

For the incremental update of PubMed data, initially data of
ENTREZ_GENENAMES table is replicated into the ENTREZ_GENENAMES_COPY
table with Ege_lastupdated value set as NULL. The Ege_lastupdated field
indicates the date on which the PubMed data is fetched for that gene
name. For every subsequent runs data is fetched for the publications
having publication date later than Ege_lastupdated.

The implementation of the PubMed parser is also aimed at taking care of
the following two issues:

1.  Some of the names assigned to a gene can be dropped later.

2.  Some genes get retired.

###### Fetching PubMed data

The following table lists the various options that are used by the
**esearch** tool to fetch PubMed data.

+-------------+--------------------------------------------------------+
| Option      | Purpose                                                |
+-------------+--------------------------------------------------------+
| Database    | Restricts the search in the specified database.        |
|             |                                                        |
|             | Example: db=PubMed                                     |
+-------------+--------------------------------------------------------+
| Term        | Searches for the specified term in the database.       |
|             | Example term="Egr1"                                    |
+-------------+--------------------------------------------------------+
| Rettype     | To retrieve either count of PMIDs (*rettype=count*) or |
|             | list of PMIDs (*rettype=uilist*) for the given search  |
|             | criterion. Example ret=                                |
+-------------+--------------------------------------------------------+
| Retstart    | To specify to retrieve PMIDs starting from the given   |
|             | index. Example retstart=5000 will fetch 5000^th^ PMID  |
|             | onwards.                                               |
+-------------+--------------------------------------------------------+
| Retmax      | To specify the max number of PMIDs to retrieve in the  |
|             | search. Example retmax=5000                            |
+-------------+--------------------------------------------------------+
| Mindate     | Minimum publication date                               |
+-------------+--------------------------------------------------------+
| Maxdate     | Maximum publication date                               |
+-------------+--------------------------------------------------------+
| Retmode     | To specify either xml or html format of the result     |
|             |                                                        |
|             | Example retmode=xml                                    |
+-------------+--------------------------------------------------------+

The algorithm used to populate the Gene_pmids table is shown below.

+-----------------------------------------------------------------------+
| 1.  For each name(key word) of a Gene ID in entrez_genenames_copy     |
|     table, get Ege_lastupdated value (last_update)                    |
|                                                                       |
| 2.  If Ege_last_update is not NULL, get the data for the period post  |
|     last_update                                                       |
|                                                                       |
| > The URL is http://                                                  |
| > :                                                                   |
| //www.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=PubMed&term="key |
| > word"&minpubmedate="last_update"&retype=xml                         |
|                                                                       |
| 3.  Else, get ALL data                                                |
|                                                                       |
| > The URL is http://                                                  |
| > :                                                                   |
| //www.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=PubMed&term="key |
| > word"&retype=xml                                                    |
|                                                                       |
| 4.  Parse the output of the URL and insert Gene ID and PMIDs in the   |
|     output into the Gene_pmids table.                                 |
|                                                                       |
| 5.  Set Ege_last_updated value to the current date for each new row   |
|     in the Entrez_genenames_copy table.                               |
|                                                                       |
| 6.  Repeat above steps for all Gene IDs in the Entrez_genenames_copy  |
|     table.                                                            |
+-----------------------------------------------------------------------+

###### Data Synchronization

[The data in tables Entrez_genenames_copy and entrez_pmids used by
PubMed Parser needs to be synchronized with data in entrez_genenames and
gene_history tables populated by Entrez Gene Parser because some gene
names might be dropped and some genes might be retired. Synchronization
is done in the following two steps.]{.underline}

[Step #1]{.underline}

After parsing Entrez Gene data source, the Entrez_genenames table is
updated. The data in the Entrez_genenames_copy table needs to be
consistent with the Entrez_genenames table data because new names may
have been added to a Gene ID or some names may have been dropped. The
following algorithm will be used to synchronize data between the
Entrez_genenames and Entrez_genenames_copy tables.

+-----------------------------------------------------------------------+
| 1.  For each Gene ID in Entrez_genenames table, compare its gene      |
|     names set (say A) with the corresponding set of gene names        |
|     (say B) in Entrez_genenames_copy table.                           |
|                                                                       |
| *Check for NEW genes*                                                 |
|                                                                       |
| 2.  If A is a super set of B, add the gene names in A - B to          |
|     Entrez_gene_names_copy table and set the Ege_last_update to NULL. |
|                                                                       |
| > *Check for DROPPED genes*                                           |
|                                                                       |
| 3.  Else if A is a sub set of B, remove the records corresponding to  |
|     the Gene ID from the Entrez_genenames_copy table as well as from  |
|     the Gene_pmids table and add the gene names in A to               |
|     Entrez_genenames_copy table and set Ege_last_update to NULL.      |
|                                                                       |
| > *New Entrez Gene ID*                                                |
|                                                                       |
| 4.  From Entrez_genenames table, add to Entrez_genenames_copy table   |
|     all the records corresponding to Gene IDs that are not present in |
|     the latter table.                                                 |
+-----------------------------------------------------------------------+

[Step #2]{.underline}

The data in Entrez_genenames_copy and Gene_pmid tables needs to be
synchronized with Gene_history table so as to take care of retired Gene
IDs. The Gene_history table is updated after parsing Entrez Gene data
source. The following algorithm will be used to synchronize data between
Entrez_genenames_copy and Gene_history table.

+-----------------------------------------------------------------------+
| 1.  For each retired Gene ID G1, compare its set of gene names (A) in |
|     Entez_genenames_copy table with the set of gene names (B) of the  |
|     new Gene ID G2, to which the first Gene ID is retired, in the     |
|     Entrez_genenames_copy table.                                      |
|                                                                       |
| 2.  If A is subset set of B, add the gene names in B - A to           |
|     Entrez_gene_names_copy table for G1 with the Ege_last_update to   |
|     NULL. Delete records corresponding to G2 from both the tables     |
|     Entrez_genenames_copy and Gene_pmid and set the Gene ID value to  |
|     G2 for all records corresponding to G1 in both the tables.        |
|                                                                       |
| 3.  Else delete records corresponding to G1 from both the tables.     |
+-----------------------------------------------------------------------+

###### Correlation Computation

**Populating geneidgroups table**

Orthologs calculated using HomoloGene are used to group PMIDs. Thus, the
relationship of two orthologous genes in the literature will be the
same.

After synchronizing data between PubMed and Entrez Gene sources, PubMed
data is associated with GeneID groups using the following algorithm.

+-----------------------------------------------------------------------+
| 1.  For each GeneID in a group in the geneidgroups table, get PMIDs   |
|     from the gene_pmids table.                                        |
|                                                                       |
| 2.  Insert group ID, and PMIDS into geneidgroups_pmids.               |
+-----------------------------------------------------------------------+

For each gene ID group, the no of PMIDs associated is counted and stored
in the GeneIDgroup_pubmedcount table. The correlation weight between any
two GeneID groups is the number of PMIDs common in those two groups.
Thus the correlation weights are computed for all GeneID group pairs and
stored in GeneIDgroups_pmid_weights table. The correlation between two
gene IDs of the same organism is calculated as the correlation of the
corresponding equivalent gene ID groups and are stored in
Gene_pmid_weights.

The tables used / populated by PubMed data-source are listed below.

1\. Entrez_genenames (populated by EntrezGene data-source)

  ----------------- ------------------------ ----------------------------
  **Field**         **Data Type**            **Constraints**

  EGE_GENEID        Number                   Not null

  EGE_ORGANISM      Character(2)             Not null

  EGE_GENE_NAME     String                   Not null
  ----------------- ------------------------ ----------------------------

This table stores gene aliases, official gene name, official gene
symbol, and gene product name assigned for each gene ID.

2\. Entrez_genenames_copy

  ------------------- ------------------------ ---------------------------
  **Field**           **Data Type**            **Constraints**

  EGE_GENEID          Number                   Not null

  EGE_ORGANISM        Character(2)             Not null

  EGE_GENE_NAME       String                   Not null

  EGE_LASTUPDATED     DATE                     
  ------------------- ------------------------ ---------------------------

This table is similar to Entrez_genenames except that it stores the date
on which the corresponding PubMed data is fetched for the name.
Initially the entire Entrez_genenames table is replicated into this
table with Ege_lastupdated value set as NULL.

3\. Gene_history

  ----------------- ------------------------ ----------------------------
  **Field**         **Data Type**            **Constraints**

  GNH_OLDGENEID     Number                   Not null

  GNH_NEWGENEID     Number                   
  ----------------- ------------------------ ----------------------------

This table contains the history of retired Gene IDs.

4\. Gene_pmids

  ---------------- ------------------------- -----------------------------
  **Field**        **Data Type**             **Constraints**

  GPI_GENEID       Number                    Not null

  GPI_ORGANISM     CHAR (2)                  Not null

  GPI_PMID         NUMBER                    Not null
  ---------------- ------------------------- -----------------------------

5\. Geneidgroups

  ---------------- ------------------------- -----------------------------
  **Field**        **Data Type**             **Constraints**

  GEG_GROUPID      NUMBER                    NOT NULL

  GEG_GENEID       Number                    NOT NULL
  ---------------- ------------------------- -----------------------------

In this table, equivalent geneIDs are grouped together. Equivalency is
determined from the HomoloGene data. Population of this table is
explained in section "populating geneIDgroups table".

6\. Geneidgroup_pubmedids

  ---------------- ------------------------- -----------------------------
  **Field**        **Data Type**             **Constraints**

  GGP_GROUPID      NUMBER                    NOT NULL

  GGP_PMID         NUMBER                    NOT NULL
  ---------------- ------------------------- -----------------------------

7\. Genegroup_pmid_weights

  ------------------ ------------------------ ----------------------------
  **Field**          **Data Type**            **Constraints**

  GGW_FROMGROUP      NUMBER                   NOT NULL

  GGW_TOGROUP        NUMBER                   NOT NULL

  GGW_WEIGHT         NUMBER                   NOT NULL
  ------------------ ------------------------ ----------------------------

8\. Gene_pmid_weights

  ------------------- ----------------------- ---------------------------
  **Field**           **Data Type**           **Constraints**

  GPW_FROMGENEID      NUMBER                  NOT NULL

  GPW_FROMSYMBOL      String                  NOT NULL

  GPW_TOGENEID        NUMBER                  NOT NULL

  GPW_TOSYMBOL        String                  NOT NULL

  GPW_WEIGHT          NUMBER                  NOT NULL

  GPW_FROMPUBCT       NUMBER                  NOT NULL

  GPW_TOPUBCT         NUMBER                  NOT NULL
  ------------------- ----------------------- ---------------------------

9\. Geneidgroup_pubmedidcount

  ---------------- ------------------------- -----------------------------
  **Field**        **Data Type**             **Constraints**

  LPC_GROUPID      NUMBER                    NOT NULL

  LPC_COUNT        NUMBER                    NOT NULL
  ---------------- ------------------------- -----------------------------

#### Revision History

The revision history component keeps track of the version and the
release dates of the external databases used to refresh/populate the
annotation schema. The SERVER_FILE_STATUS table stores the revision
history of every data source. The revision number (REVISION_NUMBER)
field, wherever supplied by the data provider, will be the actual build
data of the data source or a sequence number generated by FE Server.

  ------------------ ------------------------ ----------------------------
  **Field**          **Data Type**            **Constraints**

  SFS_ID             NUMBER                   

  SFS_MODIFY_DATE    DATE                     

  SFS_FILENAME       STRING                   NOT NULL

  SFS_FILETYPE       STRING                   

  SFS_DATABASE       STRING                   

  REVISION_NUMBER    STRING                   
  ------------------ ------------------------ ----------------------------

The query interface and HTTP APIs will provide revision details for
every external data source.

### Post data processing (Post work)

This component performs the post--data processing of data populated by
the Download and Process components and then populates the summary
tables.

#### Chip-Description 

The chip library contains the mapping between probes and any combination
of \<accession numbers, UnigeneIDs, EntrezGeneID\> values.
CHIPINFORMATION table stores these mapping details. However, the details
provided by vendor and hence the details in chipinformation table may
not be complete and up-to-date. For instance, the chip-library file,
which is loaded into chipinformation table may or may not contain
complete mapping between accession number, Entrez Gene ID, UnigeneID and
organism value. Also, the fact that UnigeneIDs, Entrez Gene IDs do get
retired, it is necessary to take care of retired Entrez GeneIDs and
UniGeneIDs.

In view of above facts, the Chip-Description component creates a new
table called CHIPDESCRIPTION, which has the format similar to
chipinformation. The fields Entrez Gene ID, Unigene ID and organism in
chipdescription table are re-calculated by joining chipinformation table
with ug_sequence, unigene, entrezgene, and ENTREZ_fly tables. The
retired IDs are replaced with new IDs by joining chipinformation table
with GENE_history, unigene_history tables.

+-----------------------------------------------------------------------+
| **ALGORITHM**                                                         |
|                                                                       |
| 1.  Prepare an empty chipdescription record (*chipdesc_record*.)      |
|                                                                       |
| 2.  For each row in chipinformation table, follow the below given     |
|     steps:                                                            |
|                                                                       |
| > **I. If "Acc No is not null" and "UniGene ID is null or not null"   |
| > and "*Entrez Gene ID* is null or not null"**                        |
|                                                                       |
| 1.  If the accession number is a flybase identifier (starting with    |
|     'FB'),                                                            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| a.  Retrieve Entrez Gene ID from Entrez_fly table corresponding to    |
|     the accession number and store it into the *chipdesc_record*.     |
|                                                                       |
| b.  Store the organism value for this record as 'Dm'.                 |
|                                                                       |
| c.  Copy the rest of the information from current row of              |
|     chipinformation table into *chipdesc_record*.                     |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| 2.  Else:                                                             |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| a.  Retrieve UniGene ID from UG_SEQUENCE table corresponding to the   |
|     accession number                                                  |
|                                                                       |
| b.  Retrieve Entrez Gene ID and organism value from UniGene table for |
|     the UniGene ID calculated in previous step.                       |
|                                                                       |
| c.  Copy the rest of the information from the current row of the      |
|     chipinformation table into *chipdesc_record*.                     |
|                                                                       |
| > **II. If Acc No is null, UniGene ID is not null, Entrez Gene ID is  |
| > not null:**                                                         |
|                                                                       |
| 1.  From unigene table fetch Entrez Gene ID and organism value for    |
|     the corresponding UniGene ID.                                     |
|                                                                       |
| 2.  Populate *chipdesc_record* with above calculated information and  |
|     fill the remaining fields from current row of chipinformation     |
|     table.                                                            |
|                                                                       |
| > **III. If Acc No is null, UniGene ID is not null, Entrez Gene ID is |
| > null:**                                                             |
|                                                                       |
| 3.  From unigene table fetch Entrez Gene ID and organism value for    |
|     the corresponding UniGene ID.                                     |
|                                                                       |
| 4.  Populate *chipdesc_record* with above calculated information and  |
|     fill the remaining fields from current row of chipinformation     |
|     table.                                                            |
|                                                                       |
| > **IV. If Acc No is null, UniGene ID is null and Entrez Gene ID is   |
| > not null:**                                                         |
|                                                                       |
| 1.  From GENE_HISTORY table check if the Entrez Gene ID is retired.   |
|     If yes, then replace it with an updated Entrez Gene ID.           |
|                                                                       |
| 2.  From Entrez_uniGene table, fetch corresponding UniGene ID and     |
|     organism value for the given Entrez ID.                           |
|                                                                       |
| 3.  Populate *chipdesc_record* with above calculated information and  |
|     fill the remaining fields from current row of chipinformation     |
|     table.                                                            |
|                                                                       |
| > **V. If All Accession Number, UniGene ID and Entrez Gene ID are     |
| > null:**                                                             |
|                                                                       |
| 1.  Copy the entire chipinformation row to *chipdesc_record*.         |
|                                                                       |
| 2.  Insert the *chipdesc_record* populated in previous steps into     |
|     chipdescription table.                                            |
+-----------------------------------------------------------------------+

The table description for the CHIPDESCRIPTION table is shown below:

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  CDE_CHIPTYPEID         Number                 CHIPINFORMATION

  CDE_PROBESET           Number                 CHIPINFORMATION

  CDE_ACC_NO             String                 CHIPINFORMATION,
                                                UG_SEQUENCE

  CDE_UGID               String                 UNIGENE

  CDE_OGRANISM           Char                   UNIGENE, ENTREZ_FLY

  CDE_GENEID             Number                 UNIGENE, ENTREZ_FLY

  CDE_CHIP_DESCRIPTION   String                 CHIPINFORMATION
  ---------------------- ---------------------- --------------------------

#### Gene-Info

After the associations among probes, UniGene IDs, and Entrez Gene IDs
are computed (refer to section 3.2.2.1) the probes are then linked with
annotations from UniGene and Entrez Gene and the GENEINFO table is
populated to hold these details.

+-----------------------------------------------------------------------+
| **ALGORITHM**                                                         |
|                                                                       |
| Insert all the records from chipdescription table into geneinfo table |
| and fill Unigene ID, title, gene, cytoband details from unigene and   |
| chromosome, symbol, gene name, gene summary, summary function fields  |
| from EntrezGene table.                                                |
|                                                                       |
| ![](media/image9.wmf)                                                 |
+-----------------------------------------------------------------------+

The table description of GENEINFO is shown below.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  GIN_CHIPTYPEID         Number                 CHIPDESCRIPTION

  GIN_PROBESET           Number                 CHIPDESCRIPTION

  GIN_OGRANISM           Char                   CHIPDESCRIPTION

  GIN_ACC_NO             String                 CHIPDESCRIPTION

  GIN_UGID               String                 CHIPDESCRIPTION

  GIN_TITLE              String                 UNIGENE

  GIN_GENE               String                 UNIGENE

  GIN_CYTOBND            String                 UNIGENE

  GIN_CHROMOSOME         String                 UNIGENE

  GIN_GENEID             Number                 CHIPDESCRIPTION

  GIN_SYMBOL             String                 ENTREZ_GENE

  GIN_GENE_NAME          String                 ENTREZ_GENE

  GIN_SUMMARY            String                 ENTREZ_GENE

  GIN_SUM_FUNC           String                 ENTREZ_GENE

  GIN_CHROMOSOME_MAP     String                 ENTREZ_GENE

  GIN_CHIP_DESCRIPTION   String                 CHIPDESCRIPTION
  ---------------------- ---------------------- --------------------------

#### Gene-Info Summary

The Entrez Gene parser stores the mapping between Entrez Gene IDs and GO
term IDs. Using this data, the component provides a link between the
Entrez Gene ID, Symbol, Gene Name, ChromosomeMap and Gene Summary to GO
terms and UniGene. The summary table GENEINFO_SUMMARY is populated to
store these links.

![](media/image10.png)The table below describes the GENEINFO_SUMMARY
table details.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  GIN_GENEID             Number                 ENTREZ_GENE

  GIN_UGID               Number                 UNIGENE

  GIN_OGRANISM           Char                   UNIGENE

  GIN_GOTERM             String                 SYSTEM_TERMDATA

  GIN_GOID               String                 ENTREZ_GO

  GIN_SYMBOL             String                 ENTREZ_GENE

  GIN_GENE_NAME          String                 ENTREZ_GENE

  GIN_SUMMARY            String                 ENTREZ_GENE

  GIN_SUM_FUNC           String                 ENTREZ_GENE

  GIN_CHROMOSOME_MAP     String                 ENTREZ_GENE

  GIN_TITLE              String                 UNIGENE

  GIN_GENE               String                 UNIGENE

  GIN_CYTOBND            String                 UNIGENE

  GIN_CHROMOSOME         String                 UNIGENE
  ---------------------- ---------------------- --------------------------

#### Gene-Info Microarray Summary

The associations among Entrez Gene, UniGene, GO and probes is computed
by joining the GENEINFO and GENEINFO_SUMMARY tables.

The table description of GENEINFO_MICROARRAY_SUMMARY is shown below.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  GIN_CHIPTYPEID         Number                 CHIPDESCRIPTION

  GIN_PROBESET           Number                 CHIPDESCRIPTION

  GIN_OGRANISM           Char                   CHIPDESCRIPTION

  GIN_GOTERM             String                 GENEINFO_SUMMARY

  GIN_GOID               String                 GENEINFO_SUMMARY

  GIN_ACC_NO             String                 CHIPDESCRIPTION

  GIN_UGID               String                 GENEINFO_SUMMARY

  GIN_TITLE              String                 GENEINFO_SUMMARY

  GIN_GENE               String                 GENEINFO_SUMMARY

  GIN_CYTOBND            String                 GENEINFO_SUMMARY

  GIN_CHROMOSOME         String                 GENEINFO_SUMMARY

  GIN_GENEID             Number                 GENEINFO_SUMMARY

  GIN_SYMBOL             String                 GENEINFO_SUMMARY

  GIN_GENE_NAME          String                 GENEINFO_SUMMARY

  GIN_SUMMARY            String                 GENEINFO_SUMMARY

  GIN_SUM_FUNC           String                 GENEINFO_SUMMARY

  GIN_CHROMOSOME_MAP     String                 GENEINFO_SUMMARY

  GIN_CHIP_DESCRIPTION   String                 GENEINFO_SUMMARY
  ---------------------- ---------------------- --------------------------

#### Chipinfo-Homolo

The Homologene annotations are linked to probes by joining
CHIP_DESCRIPTION with the HOMOLOGENE table. This link is stored in the
CHIPINFO_HOMOLO table.

+-----------------------------------------------------------------------+
| **ALGORITHM**:                                                        |
|                                                                       |
| The chipinfo_homolo table is populated by joining CHIPDESCRIPTION     |
| table with HOMOLOGENE                                                 |
|                                                                       |
| 1.  **If CDE_GENEID field in** CHIPDESCRIPTION **is not null:**       |
|                                                                       |
| > Then join CHIPDESCRIPTION and homologene tables such that           |
| > CDE_GENEID = HGE_GENEID1.                                           |
|                                                                       |
| 2.  **If CDE_GENEID is null but CDE_UGID is not null:**               |
|                                                                       |
| > Then join chipdescription and homologene tables such that CDE_UGID  |
| > = HGE_UGID1.                                                        |
|                                                                       |
| 3.  **Populate the resulting information into** chipinfo_homolo       |
|     **table:**                                                        |
|                                                                       |
| > Insert into chipinfo_homolo table, the record (CDE_CHIPTYPEID,      |
| > CDE_PROBESET, HGE_ORGANISM1, HGE_GENEID1, HGE_UGID1, HGE_ORGANISM2, |
| > HGE_GENEID2 and HGE_UGID2.)                                         |
+-----------------------------------------------------------------------+

The table description of CHIPINFO_HOMOLO is shown below.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  CIH_CHIPTYPEID         Number                 CHIPDESCRIPTION,
                                                CHIPINFO_HOMOLO

  CIH_PROBESET           String                 CHIPDESCRIPTION,
                                                CHIPINFO_HOMOLO

  CIH_ORGANISM1          Character(2)           ENTREZ_GO, ENTREZ_MAP,
                                                UG_EXPRESS,
                                                CHIPINFO_HOMOLO

  CIH_GENEID1            Number                 ENTREZ_GO,
                                                ENTREZ_MAP,UG_EXPRESS,
                                                CHIPINFO_HOMOLO

  CIH_UGID1              String                 CHIPDESCRIPTION,
                                                CHIPINFO_HOMOLO

  CIH_ORGANISM2          String                 CHIPDESCRIPTION,
                                                CHIPINFO_HOMOLO

  CIH_GENEID2            Number                 ENTREZ_GO, ENTREZ_MAP,
                                                UG_EXPRESS,
                                                CHIPINFO_HOMOLO

  CIH_UGID2              String                 ENTREZ_GO,
                                                ENTREZ_MAP,UG_EXPRESS,
                                                CHIPINFO_HOMOLO
  ---------------------- ---------------------- --------------------------

#### Chipinfo-Term

> The CHIPINFO_TERM table stores the associations between probes GO
> terms, Chromosome Map and the Tissue Expression.

+-----------------------------------------------------------------------+
| **Algorithm**                                                         |
|                                                                       |
| 1\. Fill information from Entrez_goid table:                          |
|                                                                       |
| Insert records into chipinfo_term from chipdescription, ENTREZ_goid   |
| where geneid from chipdescription = GENEID from ENTREZ_GOID           |
|                                                                       |
| 2\. Fill information from Entrez_map table:                           |
|                                                                       |
| Insert records into chipinfo_term from chipdescription, locus_map     |
| where where geneid from chipdescription = GENEID from ENTREZ_MAP      |
|                                                                       |
| 3\. Fill information from ug_express table:                           |
|                                                                       |
| Insert records into chipinfo_term from from chipdescription,          |
| ug_express where UniGeneID from chipdescription = UniGeneID from      |
| ug_express                                                            |
|                                                                       |
| 4\. Fill information from homologene and entrez_goid:                 |
|                                                                       |
| Insert records into chipinfo_term from chipinfo_homolo, entrez_goid   |
| where geneid from chipinfo_homolo = geneid from entrez_goid           |
|                                                                       |
| 5\. Fill information from homologene and ug_express:                  |
|                                                                       |
| Insert records into chipinfo_term from from chipinfo_homolo,          |
| ug_express where UniGeneID from chipdescription = chipinfo_homolo     |
| from ug_express                                                       |
+-----------------------------------------------------------------------+

The table description of CHIPINFO_TERM is shown below.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  CIT_CHIPTYPEID         Number                 CHIPDESCRIPTION,
                                                CHIPINFO_HOMOLO

  CIT_PROBESET           String                 CHIPDESCRIPTION,
                                                CHIPINFO_HOMOLO

  CIT_ORGANISM           Character(2)           ENTREZ_GO, ENTREZ_MAP,
                                                UG_EXPRESS,
                                                CHIPINFO_HOMOLO

  CIT_TERMID             Number                 ENTREZ_GO,
                                                ENTREZ_MAP,UG_EXPRESS,
                                                CHIPINFO_HOMOLO
  ---------------------- ---------------------- --------------------------

#### Chipinfo-OMIM

The mapping between probes and OMIM IDs can be computed by joining the
entrez_omim table with chipdescription and CHIPINFO_homolo.

+-----------------------------------------------------------------------+
| Algorithm:                                                            |
|                                                                       |
| 1\. Fill information from Entrez_omim table:                          |
|                                                                       |
| Insert records into chipinfo_omim from chipdescription and locus_omim |
| where geneid from chipdescription table = geneid from entrez_omim     |
| table                                                                 |
|                                                                       |
| 2\. Fill information from chipinfo_homolo                             |
|                                                                       |
| Insert records into chipinfo_omim from chipinfo_homolo, entrez_omim   |
| where geneid from chipinfo_homolo table = geneid from entrez_omim     |
| table                                                                 |
+-----------------------------------------------------------------------+

The table description of CHIPINFO_OMIM is shown below.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Source Table Name**

  CIT_CHIPTYPEID         Number                 CHIPDESCRIPTION

  CIT_PROBESET           String                 CHIPDESCRIPTION

  CIT_ORGANISM           Character(2)           CHIPDESCRIPTION

  CIT_OMIMID             Number                 ENTREZ_OMIM
  ---------------------- ---------------------- --------------------------

## add chip library

This component of Function Express server adds details of new chip
libraries to the database. The Chip library contains details for probes,
any combination of \<accession numbers, UniGeneID, EntrezGeneID\> and
probe descriptions. This component parses the chip library file and
stores data into the database. Then probes get annotated with the
up-to-date information by the retrieve annotations component.

The new chip library is registered into the database by adding library
details into a CHIPTYPES table. Every chip library file should contain
the chip name and corresponding species in the first line. Function
Express server parser chip library file, reads the chip name, species
value and inserts a new row into the CHIPTYPES table by assigning a
unique identifier for this chip. The fields in the CHIPTYPES table are
mentioned in the table below.

  --------------------- ---------------------- --------------------------
  **Field**             **Data Type**          **Constraints**

  CTY_CHIPTYPEID        Number                 NOT NULL, Primary key

  CTY_CHIPNAME          String                 NOT NULL

  CTY_SPECIES           String                 

  CTY_INPRODUCTION      Character (2)          
  --------------------- ---------------------- --------------------------

The add chip library mode of the Function Express server populates the
following table.

-   CHIPINFORMATION

This table stores the \<probeset, chiptypeID, accession number, UniGene
ID, and/or Entrez Gene ID\> association for each chip library.

  ---------------------- ---------------------- --------------------------
  **Field**              **Data Type**          **Constraints**

  CIN_CHIPTYPEID         Number                 NOT NULL, Primary key

  CIN_PROBESET           Number                 NOT NULL, Primary key

  CIN_ACC_NO             String                 

  CIN_UGID               String                 

  CIN_ORGANISM           Character(2)           

  CIN_GENEID             Number                 

  CIN_CHIP_DESCRIPTION   String                 
  ---------------------- ---------------------- --------------------------

The diagram below shows the workflow of the add chip library component.

### Add Affymetrix chip library

The Affymetrix chip library file is a structured text file, which has
one record per line, and the record holds the values of probe
identifier, accession number, and/or UniGene ID, and/or Entrez Gene ID
and probe description fields that are separated by TAB.

Following steps are followed to add an Affymetrix chip library.

1.  Register Chip library by inserting a new row into the CHIPTYPES
    table by reading first line of chip library file.

2.  Parse the chip library file and insert details into the
    CHIPINFORMATION table. The CIN_CHIPTYPEID value for this library
    comes from the CHIPTYPES table.

3.  Annotated newly added probes with current information by running
    server in the update mode.

### Add caARRAY chip library

caARRAY informatics provides MAGE-OM (Object Model) APIs to connect to
the caArray database and search for the list of arrays. The result
returned is loaded into the CHIPINFORMATION table. Then the retrieve
annotations module links these probes to the available annotations.

###  {#section-1 .unnumbered}

## Query interface

The following diagram outlines the high level architectural design of
the query interface.

###   {#section-2 .unnumbered}

### Basics of Struts

The query interface is a web based API implemented using a Java Struts
framework. This section briefly describes various facets of the Struts
framework.

In a Java Struts framework, the typical flow of events is as follows:

1.  User enters the information into the web page.

2.  Upon submission, the action form containing the information is
    passed onto the action class.

3.  The action class performs the necessary validation and creates the
    domain objects.

4.  The domain objects are then passed onto the data access class which
    retrieves the data in the database.

The following sections introduce some of the basic components of Struts:

#### Action class

The action class is the link between the *model* and the *view*. The
primary responsibility of this class is to:

1.  Retrieve information from the web page (ActionForm object) and
    create the appropriate domain objects.

2.  Perform the action (e.g. searching the database based on an
    identifier).

3.  Return results back to the web page.

#### Controller

The Controller component of the Struts Framework is the backbone of all
Struts based applications. It is implemented using a Servlet named
ActionServlet.

-   This servlet receives all client requests, and delegates the control
    of each request to a RequestProcessor class.

-   The RequestProcessor processes each request by initializing the
    ActionForm associated with the web page and then delegates the
    control of the request to a user defined Action class.

**ActionForm:** The ActionForm object contains inputs supplied by the
user. The RequestProcessor prepares an ActionForm object from the
HttpRequest and forwards it to the action classes for validation of the
business requirements.

**Domain Object:** The domain objects are value objects of the
application e.g. Gene, Probeset etc. The domain objects are populated by
the Action classes, and are then passed to the Data Access Object class.

### API Access

The main goal of caBIG is to create a network or grid of applications,
sharing data and tools, thereby creating a *World Wide Web* of cancer
research. In order to achieve this, the FE server application provides
an API so that others can access meaningful data, correlate data with
other applications or create new applications.

**HTTPClient:** HttpClient is a wrapper class over an HTTP API that
provides APIs to connect and access the FE Application through other
applications. To use a FE server API through Java applications, the
HttpClient follows two methods:

1.  connect:

-   This method connects to the server and authenticates the user.

-   After successful authentication of the user, the server sends a
    session ID to the client.

-   HttpClient class stores the session ID for subsequent use.

2.  doOperation: This method performs the specified action on a given
    form object. The user creates a new instance of the desired
    ActionForm object and passes this object and action name to this
    method. The method then performs the following steps:

-   Sends previously stored session ID and ActionForm to the server.

-   On successful completion of the operation it returns a list of
    resultant objects.

In case of error it throws an exception.

### Gene Search Query Interface

The gene search query interface provides a case insensitive search,
which can be gene centric or probe centric.

#### Probe centric search

-   The probe centric search results are displayed as a list of all
    probes satisfying the search criteria entered by the user. Search
    results also contain links to all associated annotations. The
    annotations are obtained by querying the GENEINFO_MICROARRAY_SUMMARY
    table.

-   Clicking on a specific probe identifier show detailed annotations
    about the selected probe. The details page also provides links to
    the respective data source such as UniGene, Entrez Gene, and Gene
    Ontology. The tables used for getting these details are the
    UG_EXPRESS (to get tissue expression details), ENTREZ_STS (to get
    STS data), ENTREZ_PHYNOTYPE (phenotype details), and ENTREZ_OMIM
    table (to get associated OMIM ID).

-   Clicking on the Pubmed link will display the literature network
    query interface page. This is used to get literature network details
    for a probe.

#### Gene centric Search

-   The gene centric search results are displayed as a list of all
    probes satisfying the search criteria entered by the user. Search
    results also contain links to all associated annotationsThe
    annotations are obtained by querying to the GENEINFO_SUMMARY table.

-   Clicking on a specific GeneID shows the detailed annotations about
    the selected gene. The details page also provides links to the
    respective data source such as UniGene, Entrez Gene, and Gene
    Ontology. The tables used for getting these details are UG_EXPRESS
    (to get tissue expression details), ENTREZ_STS (to get STS data),
    ENTREZ_PHYNOTYPE (phenotype details), ENTREZ_OMIM (to get associated
    OMIM ID).

-   Clicking on Pubmed link, the literature network query interface page
    is displayed to get the literature network details for a gene.

### Literature Network Query Interface

#### Probe centric search

1.  Get the user specified values for chip name, unique probe
    identifier, minimum correlation and depth of the network.

2.  Find the corresponding Entrez Gene ID from the GENEINFO table and
    add it to the genes List. Current depth is initialized to 1.

3.  Get records from the Gene_pmid_weights table whose from_geneid is in
    the genes List with the correlation greater than the
    minimum_correlation.

4.  For each pair of genes in the above result set, get the common PMIDs
    from genegroup_pmids table and show them.

5.  Truncate the genes list and add all to_geneids from the result set
    of step #3. Increment current depth.

6.  Repeat steps 3 through 4 until current depth is equal to depth of
    the network.

#### Gene centric search

1.  Get the user specified values for gene name/ gene id, minimum
    correlation and depth of the network.

2.  Find the corresponding Entrez Gene ID from the GENEINFO table and
    add it to the genes list. Current depth is initialized to 1.

3.  Get the records from Gene_pmid_weights table whose from_geneid is in
    genes list and correlation is greater than minimum correlation.

4.  For each pair of genes in the above result set, get the common PMIDs
    from genegroup_pmids table and show them.

5.  Truncate genes list and add all to_geneids from the result set of
    step #3. Increment current depth.

6.  Repeat steps 3 through 4 until current depth is equal to depth of
    the network.

# *APPENDIX A --* DOCUMENT TERMINOLOGIES AND ACRONYMS {#appendix-a-document-terminologies-and-acronyms .unnumbered}

  ------------------------ ----------------------------------------------
  **Acronym**              **Description**

  ACC                      Accession Number

  API                      Application Programming Interface

  caDSR                    Cancer Data Standards Repository

  CDE                      Common Data Element

  DAO                      Data Access Object

  EVS                      Enterprise Vocabulary Services

  FE                       Functional Express

  FTP                      File Transfer Protocol

  GO                       Gene Ontology

  ID                       Identifier

  NCICB                    National Cancer Institute Center for
                           Bioinformatics

  NID                      Nucleotide Sequence Identifier

  OM                       Object Model

  PID                      Protein Sequence Identifier

  PV                       Permissible Value

  SDD                      Software Design Description

  SID                      System Identifier

  STS                      Sequence Tag Site

  UML                      Universal Modeling Language
  ------------------------ ----------------------------------------------

# *APPENDIX B -- ORTHOLOG COMPUTATION DETAILS AND EXAMPLE* {#appendix-b-ortholog-computation-details-and-example .unnumbered}

The following algorithm is used to compute ortholog groups from the
sorted list.

+-----------------------------------------------------------------------+
| 1.  Create a list of sets of gene IDs call it geneID_list. Initially  |
|     geneID_list is empty.                                             |
|                                                                       |
| 2.  Create a map ortholog_map that stores mappings between Ortholog   |
|     Set and OrthologID. Initially ortholog_map is empty.              |
|                                                                       |
| 3.  ortholog_ID is a static variable whose initial value is 1. This   |
|     is a FE Server generated ID for each ortholog.                    |
|                                                                       |
| 4.  For each geneID1 get the list of geneID2 with reciprocal value    |
|     TRUE and create an ortholog_set that includes geneID1 and the     |
|     list of geneID2                                                   |
|                                                                       |
| > **COMPUTE STRICT ORTHOLOGS**                                        |
|                                                                       |
| 5.  If strict_ortholog_set exists in geneID_list,                     |
|                                                                       |
|     a.  Find its ortholog_ID                                          |
|                                                                       |
|     b.  insert into table orthologstartlocus (ortholog_ID, geneID1,   |
|         'T')                                                          |
|                                                                       |
| > ELSE                                                                |
|                                                                       |
| a.  Insert strict_ortholog_set, ortholog_ID into ortholog_map         |
|                                                                       |
| b.  Insert all genes in strict_ortholog_set and ortholog_ID into      |
|     ortholog table                                                    |
|                                                                       |
| c.  Insert into orthologstartlocus (ortholog_ID, geneID1, 'T')        |
|                                                                       |
| d.  Increment ortholog_ID;                                            |
|                                                                       |
| > **COMPUTE LOOSE ORTHOLOGS**                                         |
|                                                                       |
| 6.  For each geneID1 get the list of geneID2 with reciprocal value    |
|     FALSE and create a loose_ortholog_set.                            |
|                                                                       |
| 7.  If there are multiple gene IDs whose organism is same in the list |
|     of geneID2, consider only the one with highest alignment value.   |
|                                                                       |
| 8.  Add geneID1, list of geneID2 and each gene ID in                  |
|     strict_ortholog_set to lose_ortholog_set.                         |
|                                                                       |
| 9.  If loose_ortholog_set exists in geneID_list,                      |
|                                                                       |
|     a.  Find its ortholog_ID                                          |
|                                                                       |
|     b.  insert into table orthologstartlocus (ortholog_ID, geneID1,   |
|         'T')                                                          |
|                                                                       |
| > ELSE                                                                |
|                                                                       |
| e.  Insert loose_ortholog_set, ortholog_ID into ortholog_map          |
|                                                                       |
| f.  Insert all genes in loose_ortholog_set and ortholog_ID into       |
|     ortholog table                                                    |
|                                                                       |
| g.  Insert into orthologstartlocus (ortholog_ID, geneID1, 'F')        |
|                                                                       |
| h.  Increment ortholog_ID;                                            |
+-----------------------------------------------------------------------+

A sample-sorted list is shown in the following table.

**NOTE:** Each color indicates an ortholog group.

  --------- ---------- ------------ ---------- ------------ ---------------
  GeneID1   TaxID1     GeneID2      TaxID2     Reciprocal   Alignment

  355       9606       395274       9031       true         0.36024845

  355       9606       14102        10090      true         0.506135

  355       9606       466149       9598       false        0.97342193

  14102     10090      395274       9031       true         0.34628975

  14102     10090      466149       9598       true         0.48333335

  14102     10090      355          9606       true         0.506135

  14102     10090      365454       10116      true         0.6292517

  365454    10116      395274       9031       true         0.36904764

  365454    10116      14102        10090      true         0.6292517

  365454    10116      466149       9598       true         0.50530034

  395274    9031       355          9606       true         0.36024845

  395274    9031       466149       9598       true         0.42510122

  395274    9031       14102        10090      true         0.34628975

  395274    9031       365454       10116      true         0.36904764

  466149    9598       395274       9031       true         0.42510122

  466149    9598       14102        10090      true         0.48333335

  466149    9598       365454       10116      true         0.50530034

  466149    9598       355          9606       false        0.97342193
  --------- ---------- ------------ ---------- ------------ ---------------

# *APPENDIX C -- CLASSES* {#appendix-c-classes .unnumbered}

1.  **DOWNLOAD MANAGER**

The input data files are downloaded from public sources using FTP and
HTTP protocols. Both HttpDownLoader and FTPDownLoader make use of a
queue to hold and efficiently use the downloaded files.

-   **HttpDownLoader**

> This class downloads the files from public data sources using HTTP
> protocol.
>
> **checkDate():** This function checks the file date on the HTTP server
> and compares it to the date stored in the database.
>
> **getData(**FileInfo httpInfo,FileHolder fileHolder):This function is
> used to download the data file from Http Source. fileHolder object
> holds the downloaded file and httpInfo object holds all the HTTP
> details for the file to be downloaded.

-   **FTPDownLoader**

> This class downloads the data files from public data sources using FTP
> protocol.
>
> **downloadSite(**FileInfo ftpInfo, FileHolder fileHolder**):** It
> downloads the data file from FTP source.
>
> **boolean ftpFileDateCheck(**FileInfo ftpInfo**):** determines if
> files of a particular db need to be ftp\'d or not.

-   **DownLoadFactory**

> **getDownloader(**FileInfo, DPQueue): Returns a Downloader object for
> files stored in FileInfo object. The function also accepts as input a
> DPQueue object to manage the downloaded files.

2.  **PARSER MANAGER**

It's a multithreaded component which parses the downloaded files.

-   **ParserManager**

> **getParser(**String, FileHolder**):** This function accepts the type
> of data source and a FileHolder which holds the downloaded files and
> returns a Parser Instance.

-   **Parser**

> Parses the database and passes the parsed record to the database
> loader.
>
> **parse(**String fileName**):** It is an abstract function declared by
> Parser which must be defined by each individual parser.

-   **XmlParser**

> This class extends Parser and provides XML parsing functionality using
> SAX API.
>
> **startElement(**String namespaceURI,String localName,String qName,
> Attributes atts**):** Specifies the action to be taken when the start
> of a tag is encountered.
>
> **endElement(**String namespaceURI,String localName,String qName**):**
> Specifies the action to be taken when the end of a tag is encountered.
>
> **characters(**char\[\] ch,int start,int length**):** Specifies the
> action to be taken when the content of a tag is encountered.

-   **HomologeneXMLParser**

> Parser for the HomoloGene data files in XML format. It extends the
> XmlParser class and overrides the functions like startElement(),
> endElement(), charactors().
>
> **addParalogs():** Adds paralogs to Homologene_Tmp table.
>
> **initializeTables**(): Initialise homologene database table.
>
> **fillHomoloGeneTable**(): Populates the homologene table from
> homologene_tmp table.

-   **HomologeneRecComparator**

> **compare**(Object o1, Object o2): This function compares two
> homologene records and is used to sort the collection of
> HomologeneRecords order by geneid1, reciprocal, taxid2, alignment

-   **Ortholog**

> **computeOrthologs**(ArrayList list): Calculates the orthologs among
> the list of HomologeneRec tuples sorted in the the order of geneid1,
> reciprocal, taxid2 and alignment.

-   **GOXMLParser**

> It parses the Gene Ontology data stored in XML input files. And
> populates the tables system_termdata and system_termtree. It extends
> the XmlParser class and overrides the functions like startElement(),
> endElement(), charactors().
>
> **insertRootNode**(): Inserts the root nodes in system_termdata and
> system_termtree tables.
>
> **initializeTables**(): Initialises system_termdata and
> system_termtree tables.
>
> **readConfigFile**(): Reads the GOConfig file and determines the
> parent term id and parent term name.

-   **EntrezGeneXMLParser**

> Parses the Entrez gene input file in XML format and populates the
> associated base tables. It extends the XmlParser class and overrides
> the functions like startElement(), endElement(), & charactors().
>
> **initializeTables**(): Initialises all the base tables like
> EntrezGene, Entrez_Fly, Entrez_GOID etc.

-   **UnigeneParser**

> It parses the unigene text input-data coming in separate files for
> different organism.
>
> **parseExpress**(): Tokenizes the input line into Express records and
> populates them into ug_express table.
>
> **initializeTables**(): Initialises the tables
> unigene,ug_sequence,ug_protsim, ug_express,system_termdata and
> system_termtree tables
>
> **parseProtsim**()**:** Parses the prot-sim record.
>
> **getTissueID**(String tissue) :Returns an id for a tissue.

-   **UniStsParser**

> It parses the STS tags data available in the text input files and
> populates UniSTS and UniSTS.aliases.
>
> **getShortOrgName**(String org): returns the two-character short name
> for Organism
>
> **initializeTables**(): Initialises the tables UniSTS and UniSTS_alias

**findFormat**(String fileName): This Function returns true if file is
UNISTS false for UNISTS.Alias