# Presentation: 163538

## P.F. Lemkin
LECB, CCR, NCI/FCRDC
mail: lemkin@ncifcrf.gov
[This document is under construction]

Revised: 06-19-2002

- LECB, CCR, NCI/FCRDC
- mail: lemkin@ncifcrf.gov
- [_**This document is under construction**_]
- Revised: 06-19-2002

-  **Software Design** of the MicroArray Explorer Data Mining Tool

**Home: **_**http://www.lecb.ncifcrf.gov/MAExplorer**_**Open Source: **_**http://maexplorer.sourceforge.net/**_

##  Mozilla 1.1 public license

-     This work was produced by Peter Lemkin of the National Cancer Institute, an agency of the United States Government.  As a work of the United States Government there is no associated copyright.  It is offered as open source software under the Mozilla Public License (version 1.1) subject to the limitations noted in the accompanying LEGAL file. Both files are available on http://maexplorer.sourceforge.net/.

##  Java MAExplorer program design issues

- 1. Overview
- 2. Design decisions, Client-centric vs. Server-Centric
- 3. MAExplorer GUI (graphical user interface)
- 4. Web database I/O
- 5. Genes and Gene lists
- 6. Gene data filter
- 7. Multiple plot windows
- 8. Other plot window implementations
- 9. Reports & access to other Web databases
- 10. Synchronizing windows
- 11. Dumping text and plot windows to .txt and .gif files
- 12. Saving and Restoring the MAExplorer state
- 13. Miscellaneous classes
- 14. MAEPlugin design
- 15. MaeJavaAPI design
- 16. Examples (links): MAExplorer menu organization

##  Contents of this document

- This document discusses the _software design_ of MAExplorer
- MAExplorer is _Open Source_ with the source code and a collaborative environment for improving the code available at http://maexplorer.sourceforge.net/
- The first part contains the _primary design discussion_
- The second part contains_ Examples of computer screens_ for many of the windows illustrating these data structures and classes

## 1. Overview - MicroArray Explorer

**MAExplorer is a flexible Java microarray data-mining tool                                                                                                                                                                                                                        **
- Handles multiple cDNA or oligo array samples
- Handles duplicate spots/array and replicate samples
- Handles intensity or ratio (Cy3/Cy5) quantified array data
- Data organized by 2- (X vs Y) and N-condition lists expression profiles, sample lists, data structures could support ordered lists of condition lists
- Gene data-filters gene set computed by statistics, clustering, gene sets
- Direct data manipulation in pseudoarray image, graphics, spreadsheets
- Access genomic Web servers from plots and reports
- Oriented toward mRNA data, could extend to protein arrays
- Stand-alone (off-line) or applet  (Web-based)
- User data converted using Cvt2Mae “wizard” tool
- MAEPlugins allow users to add new analysis methods

## 1.1 Array data mining - finding patterns of genes

- Quantified array spot data for multiple samples

- Organize by: sample, gene expression, gene sets

- Change views: normalization, data filters

- Visualize and query: plots, cluster, reports

- Explore: external genomic databases

- Specific
- Results

- General

- Data
- Reduction

- Subset of genes for further analysis

**Notes:**
I. Microarray Data Mining Overview

* Data mining lets us find patterns of genes related to experimental conditions that result in gene subset(s) that can be subjected to further analysis

 Data consists of quantified spot data

 Flow of an analysis

 Results are subjected to further analysis

## 1.2  Data preparation for MAExplorer 

**Notes:**
 MAExplorer uses quantified spot data (brown box)

 Data preparation proceeds as follows ...

## 1.3 Paradigm: local & genomic-Web databases

**Notes:**
 Quantified array data is read by MAExplorer

 During analysis Plots, Reports may be generated and saved

 The State may be saved for later continuation

 Genomic databases can be access for particular genes using a popup Web brower

## 1.4 Overview of MAExplorer database system 

- (Data preparation steps prior to MAExplorer analysis are in cyan)

## 1.5 MAExplorer analysis environment

- _Stand-alone Java application_ for user data
- Download MAExplorer program from Web site
- _Installers_: Windows, MacOS/-X, Solaris, Linux, Unix, etc.
- Documentation, tutorials, MGAP demo database on Web site
- _Cvt2Mae_ “wizard” tool converts user data for use with MAExplorer
- May download data from NCI/CIT mAdb microarray data server RDBMS for direct use with MAExplorer

## 1.6 MAExplorer user interface - main Analysis menu

**1.6 **_**MAExplorer user interface**_** - main Analysis menu**

**Pseudoarray image is ratio of samples:**
**pregnancy (HP-X) / lactation (HP-Y)**

**Genes passing data filter have white circles**

**Current gene has yellow circle**

**Select X and Y samples**

**Notes:**
 Go over Pseudoarray Image, Current Gene, Data Filter
 Analysis Menu organized as a first approximation of an analysis - actual analyses obviously more complex
    1. File - database, file access operations 
     2. Samples - select lists of hybridized sample conditions 
     3. Edit - edit gene and condition subsets, E.G.L. and preferences
     4. Analysis - primary analysis menus
        4.1 GeneClass - select gene subset for gene class data Filter
        4.2 Normalization - select gene intensity normalization mode
        4.3 Filter - select data filters to compute gene subset of interest
        4.4 Plot - pseudoarray image, scattter, histograms, expression                                                                                                                                                                                                       profile popup plots
        4.5 Cluster - perform cluster analysis on data filtered genes
        4.6 Report - popup spreadsheet reports of genes & samples
    5. View - setup genomic gene data views preferences
     6. Plugins - add and execute new MAEPlugin methods 
     7. Help - popup documentation on MAExplorer and specific DB 

## 1.7 Data conversion “Wizard” for MAExplorer

- _Cvt2Mae_ “Wizard” converts commercial and user-defined array formats (e.g. Affymetrix, GenePix, Scanalyze, etc)
- Users may create and save _Array Layout_ descriptions for subsequent conversions
- Conversion generates a MAExplorer _project directory tree_ of files that are ready to analyze
- After MAExplorer installed, click on project’s _*MAE/Start.mae*_ file to start analysis
- See Cvt2Mae home page for details on operation

## 1.7.1 Cvt2Mae array data conversion “wizard”

**Database of built-in and user- defined array layout formats**

**Notes:**
 Cvt2Mae “Wizard” application converts user’s commercial and custom array formats (e.g.                                                                                                                                                                                                                    Affymetrix, GenePix, Scanalyze, etc).

 Users may create and save Array Layout                                                                                                                                                                                                                          descriptions of their array data for subsequent data                                                                                                                                                                                                           conversions

 Will add generic “MGED-ML” portable data                                                                                                                                                                                                                       compatibility (data standard for microarray XML)

## 1.8 MAExplorer Home Page http://www.lecb.ncifcrf.gov/MAExplorer

**1.8 MAExplorer Home Page http://www.lecb.ncifcrf.gov/MAExplorer**

**Reference Manual**

**Tutorials**

**Download MAExplorer**

**Data conversion wizard**

**Plugins**

**Examples**

**Notes:**
 MAExplorer Web Site Home Page resources

 On-line or down-loadable hyper-linked Reference Manual

 Simple and advanced “How-To” tutorials that work with any data

 PDF “How-To” slide show documents

 Download stand-alone MAExplorer Java application for Microsoft windows, Macintosh, Sun, Linux, Unix

 Download Cvt2Mae array data conversion “wizard”

 MAExplugins to extend analytic methods

## 2. Our design decisions

- _Computation is done in the Java application _for better user interaction
- Initially used Java _JDK1.1_, since not all Web browsers handled JDK1.2/1.3.  JDK1.1 is the least common denominator for applet portability. Not important with stand-alone version which uses JDK1.3. The JDK1.3 is packaged with the Installers
- All data files (except images) are _tab-delimited_ ASCII files - Excel-compatible. The Cvt2Mae data converter translates user data files to MAExplorer formatted files
- _Minimize amount of data_ that needs to be read initially
- _Use object-oriented design_ with new base classes and extended classes as required.  Write custom GUI classes to better control the user interface
- Use _Integrated Development Environment_ such as Sun’s free “Forte for Java” for rapid debugging
- Optimize code and garbage-collect data structures often ...

## 2.1 Client-Centric computations - advantages/disadvantages

**Client-centric approach uses stand-alone programs**
** **
-  +     Java runs on all operating systems as either stand-alone or browser applets
-  +     handles rapid response required for direct manipulation on desktop computers
-  +    stand-alone version may be restarted quickly from local or cached data
-  +    size limitations are not a problem with stand-alone Java applications
-  +    Java plug-ins allows prototyping new analysis methods by any group of users
-  **+    **easy to build large stable stand-alone programs handling very large data sets
-   -    for applet version, slow startup since program & data downloaded when run
-   **-    **difficult to build large stable Web-applets handling very large data sets
-   **-    **for stand-alone application, must be installed on client's computer

## 2.2 Server-Centric (CGI or Applet) computations - advantages/disadvantages

**Server-centric approach uses **_**mix of HTML, CGI, Java Applets**_
** **
** **+   may have better resources for very large data sets but with dependence on server
-  +   faster startup than full applet since minimal GUI required and little data is downloaded
-  +   easier to prototype and distribute new functionality using centralized CGI or servlets
-   **- **  susceptible to Internet traffic bandwidth problems for large numbers of users
-   **-**   susceptible to server-load dependencies for large numbers of users
-   **-**   difficult to get very rapid response for direct manipulation for data mining

## 2.3 The MAExplorer project

- A database resides in a _project directory_ which contains all samples the user may wish to analyze
- _Multiple MAExplorer projects_ may exist on a local disk (or Web server) - each having a standard project directory tree  (shown in next slide)
- A projects database file (_maeProjects.txt_) in the stand-alone installation directory tracks the names of these projects and disk location and last active database
- The (File | Database | Open file DB) menu command specifies a particular  database startup file within a project directory
- MAExplorer is started by _opening a .mae startup file_ that specifies the subset of samples to use (Note: .mae is the _file extension_ - not the complete file name!)
- Clicking on a .mae file (e.g. in Windows) will start MAExplorer on that database

## 2.3.1 MAExplorer Project Directory Tree

**      ****/Cache**** - **(optional)  cached data saved from initial download from Web DB server
**       ****/Config****  - **project databases for** **Configuration DB, GIPO DB, Samples DB files
**       ****/MAE   **** - **set of .mae stand-alone startup files for subsets of the project samples
-        **/Plugins** - (optional) set of MAEPlugins written by the user. [Normally, it checks                                                                                                                                                                                           the /Plugins directory in the MAExplorer installation directory]
**       ****/Quant**** - **quantified spot data files for each hybridized sample
**       ****/Report**** - **(optional) directory where text and GIF files are saved by user with
-                           _SaveAs Report_ and _SaveAs GIF_  commands
**       ****/State**** - (optional) **the GeneBitSets (.cbs) and SampleSets (.hpl) “SaveAs DB”  files

## 2.3.2 Required project directories

- All data files (except images) are tab-delimited files
- The .mae startup file is a physical file on a local disk or a Web server virtual CGI file  in the _MAE/_ directory
- There may be any number of .mae startup files. They all end with “.mae”
- Each .mae startup file points may point to a specific_ configuration_ file
- The _Config/_ directory contains:
-       1. _Configuration database_ file describing the array architecture and other DB files
-       2. _Samples database_ file listing all of the samples available in the database
-       3. _GIPO (Gene In Plate Order) print table database_ file mapping spot position to                                                                                                                                                                             geneomic information through a spot identifier
- The  _Quant/_ directory contains quantified spot data files for each hybridized sample that includes channel(s) intensity and background, good spot flags (QualCheck), spot identifier

## 2.3.3 Optional/Generated project directories

- The _Cache/_ directory can be used to saving all data files downloaded from a Web server to avoid returning to the server  when accessing that data in subsequent data mining sessions
- The _State/_ directory is used for saving the named gene sets (.cbs files) that are shared between all startup files.
- The _State/_ directory is also used for saving the named lists of hybridized samples (.hpl) files that are shared between all startup files
- When saving a data mining session using (File | Database | SaveAs ... DB) menu command, the named gene sets and sample lists are written to the State/ directory as .cbs and .hpl files
- Saving the database also writes the _complete state_ into a .mae startup file that overides the configuration file data the next time MAExplorer is started
- The _Report/_ directory is used for saving text reports as .txt files and all plots as .gif files

## 2.3.4 Opening a database from local disk

- In stand-alone mode, you can browse a project database containing a set of startup databases.

##  2.4 Notation : JDK vs. new MAExplorer classes

- Classes which are part of the _SUN JDK library_ are in **Green**,              classes which we wrote specifically for _MAExplorer_ are in **Red**.
- The stand-alone main() method is in MAExplorer
- There are about 140 classes including the MAEPlugins support
- The MAExplorer class contains instances of the single instantiations of all classes that may require global access (for speed) from any subsequent processing
- These global instances are created in *MAExlorer.init()* at startup

##  2.4.1 New MAExplorer base classes

- Guesser  - scrollable text area for selecting one or more items using prefix (e.g. “carbonic”) or wild card (e.g. “*onco*) notation. Includes: PopupGeneGuesser, PopupHPmenuGuesser, PopupProjDirGuesser.
- Chooser  - specialized Chooser that lets users move objects from a “remainder” list to a “selected List. Includes: PopupHPChooser for selecting a the HP-X, HP-Y sets and HP-E list.
- Table  - tab-delimited table constructor and file reader. Includes: ConfigTable, GipoTable, MaHPquantTable, MaInfoTable, SamplesTable. Table is also used to compute intermediate data structures for computing reports

##   2.4.1 New MAExplorer base classes (continued)

- Table is extended from SimpleTable that can be used (and is used) to make short-lived temporary tables for various purposes including reports
- Draw2Dplot  - create scrollable 2D plots. The DrawScatterPlot, ExprProfileOverlay extend Draw2Dplot
- Gene  - create Gene instances. It is used in *GeneList.mList[]* that may contain ordered lists of Genes
- GeneBitSet - creates efficient 64-bit/word bit-sets representing Gene sets and operations on these sets. It is normally accessed through *GeneList.bitSet*

##  2.4.2 Data structures: genes, hybridized samples

- Any class instance that may need to be accessed from other classes has a class instance kept in MAExplorer. Most classes have an “initialization” constructor that captures the MAExplorer instance (typically called ‘mae’) for future use by that class. This lets methods in that class access any other classes required
- All variables and methods are private  unless they need to be accessed from outside of the class
- _State data_ is found primarily in two classes in **MAExplorer** and **Config**
- Fundamental objects are genes (**Gene**) and hybridized samples (**MaHybridSample**) which are then used in other classes and lists

##  2.4.3 Data structures: HPs, MIDs, GIDs, gang GIDs

- A _HP_ is all of the array data for a _hybridized sample_. It contains data for multiple spot intensity channels (e.g. F1&F2, or Cy3&Cy5), background, QualCheck flags, etc.
- A _GID is a Grid index ID_ and uniquely defines a spot in the array database. Corresponding spots in different samples have the same GIDs
- Replicate spotted grid in the  array have _Gang GID_s
- A _MID is a Master gene ID_ and uniquely defines a gene in the database. All GIDs representing the same gene have the same MID
- There is one copy of a Gene instance in the database and it _has all gene specific data_ (gene name, GenBank ID, Clone ID, etc.)

##  2.4.4 Data structures: Maps

- The **Map** class defines maps between MIDs, GIDs, GridCoords, and Genes
- The master gene list, *Map.midStaticCL*, has a list of all Genes instances indexed by MID as *Map.midStaticCL.mList[mid]*
-  The *Map.gidStaticCL.mList[gid]* accesses corresponding Gene instances by GID
- The *Map.gid2mid[gid]* looks up the MID given the GID
- The *Map.mid2gid[mid]* looks up the GID given the MID
- The *Map.gidToGangGid[gid]* looks up the Gang GID given the GID
- There are other maps between lists of spot **GridCoord** (field,grid,row,column) and GIDs

##  2.4.5 Data structures: Gene

- The Gene Gene is the base class used to define a single gene (clone or oligo) data  structure consisting of sample-specific data fields and sample independent genomic identifiers and name fields. The latter is represented by the Master Gene ID (MID) which is unique for any number of spots for that gene and the Grid coordinate ID (GID) which corresponds to a particular spot for that gene
- The *Gene.midList[0:nMid-1]* is a list of all other Gene instance MIDs that are the same gene (i.e. replicates)
- This identifies replicate genes on the array that are available for computing statistics
- Quantified data may be temporarily stored in (data, data1, data2, pValue, geneDist, etc.) variables
-  Generally, F1 (Cy3) is data1, F2 (Cy5) is data2 and F1/F2 or Cy3/Cy5 is data

##  2.4.5 Data structures: Gene (continued)

- The _Genomic IDs_ include: Clone_ID, GenBankAcc, GenBankAcc3, GenBankAcc5, Unigene_ID, dbEST3, dbEST5, SwissProt, RefSeqID, LocusID.
- The _Master_ID_ is set to one of these.
- The arrays GenomicID[] and nGenomicID[] may be used for specifying external identifiers for particular user databases
- _ __Gene names_ include: Gene_Name, UGclusterName. The _MasterGeneName_ is set to one of these
- Additional identifiers include: Gene_Class, plate, plate_row, plate_col
- Each gene has various _properties_ indicated the inclusive-or of C_xxxx constants

## 2.5 Algorithms: initialization and event handling

***MAExplorer.init()***
**1. Read (name,value) parameters from Applet PARAMs or ****Config**** file**
**2. Read database files and set up database structures**
**3. Create GUI with scrollable pseudoarray image with**
**     ****ArrayScroller, ScrollableImageCanvas, DrawPseudoImage**** **
**4. Create pull-down ****MenuBar**** with ****MenuBarFrame**

**EventMenu**** - pull-down menu event handling**
**1. Menu item command *****EventMenu.handleActions() *****- eval menu command**
**2. Menu checkbox item *****EventMenu.handleItemStateChanged() *****- eval menu **
**                                                                                             ****checkbox command**

**ScrollableImageCanvas**** - pseudoarray direct manipulation event handling**
**1. Select spots invokes ****PopupRegistry**** current gene change **
**2. Select sample invokes ****PopupRegistry**** change current HP sample and Filter **

## 2.6 Startup (name,value) Parameters: .mae file or Applet PARAMs

- The Config class contains many of the state variables (MAExplorer contains most of the rest)
- The parameters are set in the Config class using the GetParam class to get (Name,Value) data definitions if they exist. They are defined in an overide hierarchy:
- 1) Parameters are initially defined by reading the Config file using the ConfigTable class. If they are not defined, then either the variables are not defined or use hardwired values
- 2.a) These are overidden using <APPLET> PARAM values if they exist when using an _applet_, or
- 2.s) They are overidden using (Name,Value) data from the .mae startup file when used in _stand-alone_ mode

## 2.7 Data structures: hybridized samples (HP)

- The MaHybridSample class contains the un-normalized data for a particular sample read by MAExplorer. It uses a one-time instance of MaHPquantTable class to read and parse the .quant data file
- The SampleSets class contains the working HP-X, HP-Y and HP-E lists of MaMybridSample instances used by the data Filter. It also contains the menu sample names (parsed from SamplesTable class by StageNames class)
- The HPxyData class contains the the current HP-X and HP-Y sample sets data for a particular gene MID including the statistics for each set for use in computations on a single gene across X-Y samples
- New set statistics are computed for a new MID using HPxyData.updateData()

## 2.7.1 Data structures: sample condition sets

- The Condition class contains named lists of named hybridized samples
- These may be copied to the SampleSets data structure working lists for the HP-X set, HP-Y set and HP-E list
- It also contains ordered lists of Conditions that could be used with expression lists of averaged samples

## 2.8 Data structures: spot data for genes

- The SpotData is a data-only class holds the raw and normalized data copied from a single Gene for a single sample
- The SpotData instance is loaded using the *MaHybridSample.get...Data()* methods. There are a number of different methods
- The SpotFeature class methods computes a pretty-print string _summary line_ or 3 lines for data for a single Gene for one HP, HP-(X,-Y),  HP-(X,Y) ‘sets’, or HP-E
- This summary line is modified for single channel (F1 or F2) and (Cy3 or Cy5) intensity data or ratio data (Cy3/Cy5), HP-X/HP-Y etc. taking the normalization mode into account

## 2.8.1 Data structures: lists of spot data for genes

- Arrays of a specific sample HP (F1,F2)  (I.e. Cy3, Cy5) gene spot data passing the data Filter are loaded using the *MaHybridSample.getF1F2data()* method
- Arrays of single HP-(X,Y) samples data passing the data Filter are accessed using the *CompositeDatabase.getHP_XandYdata() *methods
- Arrays of sets of HP-(X,Y) data passing the data Filter are accessed using the *CompositeDatabase.getHP_XandYsetData()* method

## 2.9 Data structures: expression profile

- The ExprProfile class contains the expression profile data structure for a single gene
- Data may be changed using *ExprProfile.updateData()*
- The samples (HP1, ..., HPn) used in the expression profile are defined by the HP-E list of samples with n = |HP-E|
- For intensity data from each sample j (j=1:n) for gene g (g=1:maxGenes) it computes:  mean(j,g),  stdDev(j,g), CV(j,g)
- Then ExprProfile instances serve as the data structure basis for other expression profile plot classes (ExprProfilePlot, ExprProfilePanel, ExprProfileScrollPane, ExprProfileCanvas, ExprProfileOverlay)

## 3. MAExplorer GUI: Graphical User Interface

- The initial GUI is created during startup using the MenuBarFrame class that is a separate popup frame
- It creates the _pull-down menu_ created by the class that is an instance of a **MenuBar** may is attached. This lets us use cascading menus for a reasonably easy to use look and feel
- The EventMenu class handles all events from the pull-down menu. It is the main dispatching area for calling the actual functionality to service these events
- Note: The instances of _menu  checkboxes_ are defined in MenuBarFrame but are used in EventMenu
- The MenuBarFrame also creates a _control panel for buttons and messages_ at the top
- Messages are written into status areas by the ***Util.showMsg()*** methods

## 3.1 MAExplorer GUI: status messages and logging

- Messages are written into status areas by the *Util.showMsg()* methods into the three status lines at the top of the GUI
- If _command history logging_ is enabled, then commands (from both the menu selections and clicking on genes etc.) are logged into a popup log window by *Util.saveCmdHistory()*
- If _message logging_ is enabled, then all messages that go to the status lines as are logged into a popup log window by *Util.saveMsgHistory()*
- The command history and message logging text may be saved into log files

## 3.2 MAExplorer GUI - the pseudoarray image

- The _pseudoarray image_ is recomputed only as needed (e.g. samples makeup or normalization changed) using the DrawPseudoImage class
- The image may or may not represent the actual physical array layout depending on whether the data was available to the configuration database
- Spot color is either grayscale (spot intensity) or a pseudocolor representing the ratio of (F1/F2, Cy3/Cy5, HP-X/HP-Y, HP-X ‘set’/HP-Y ‘set’, p-Value) or the sum of two channels (red+green)
- The pseudo microarray image is then painted in a ScrollableImageCanvas class contained as an instance of  the ArrayScroller class in the main frame.
- Whenever the pseudoarray image is redrawn, various _overlay graphics_ are added (e.g. current gene, E.G.L, and cluster boxes or circles, etc) in different colors

## 3.3 MAExplorer GUI: selecting genes and samples

- Users select genes using _direct manipulation_ by clicking or by typing in pseudoarray image, plots or reports
- Users  type sub-strings of names of things (e.g. genes, samples) into a pop up “guesser” window:
-       1. The PopupGeneGuesser extends the Guesser base class to select a single gene or a set of genes (into the E.G.L) based on gene name, genomic ID, etc.
-      2. The PopupHPChooser uses collections of the ChooserGUI class to select the HP-X, and HP-Y ‘sets’ and HP-E ‘list’ from the set of all samples. These sample sets are stored in SampleSet lists of samples

## 3.5 MAExplorer GUI: state sliders to set thresholds 

- The StateScroller class implements a popup window containing the threshold state scrollers
- Only threshold scrollers that are being actively used (i.e. selected in the data filter) are normally presented. This can be overridden to force all threshold sliders to be presented
- Changing a scroller invokes the *PopupRegistry.updateSlider() *that calls all active windows that have registered for a callback when the scroller value changed
- Changing the _normalization method_ changes the state of the scroller since the dynamic range of some threshold will also change.
- The popup window is different from all other MAExplorer popup windows in that you may not close it by clicking on the close window button. Rather, you must deselect all active filters to make it close. However, you may minimize it

## 3.6 MAExplorer GUI: dialog popups  

- There are two types of text dialog popups: a single variable popup PopupDialogQuery class and a triple operand PopupBinOprDialogQuery class
- The PopupDialogQuery class has both a pull-down list of items to select and a text area. Selecting a pull-down item enters it into the text area
- The PopupBinOprDialogQuery class has three instances of the PopupDialogQuery class in its GUI. It is desiged for doing Boolean set or list operations (e.g. dstOperand3= srcOperand1 OPR srcOperand2)
- Gene set operations are done in the GeneClass class
- Hybrized sample set operations are done in the Condition class

## 3.7 MAExplorer GUI: Menu organization

**Note: The Analysis Menu **_**organized**_** as a first approximation of an analysis **
-     1. **File** - database, file access operations
-      2. **Samples** - select lists of hybridized sample conditions
-      3. **Edit** - edit gene and condition subsets, E.G.L. and preferences
-      4. **Analysis** - primary analysis menus
-         4.1 **GeneClass** - select gene subset for gene class data Filter
-         4.2 **Normalization** - select gene intensity normalization mode
-         4.3 **Filter** - select data filters to compute gene subset of interest
-         4.4 **Plot** - pseudoarray image, scattter, histograms, expression                                                                                                                                                                                                       profile popup plots
-         4.5 **Cluster** - perform cluster analysis on data filtered genes
-         4.6 **Report** - popup spreadsheet reports of genes & samples
-     5. **View** - setup genomic gene data views preferences
-      6. **Plugins** - add and execute new MAEPlugin methods
-      7. **Help** - popup documentation on MAExplorer and specific DB

**Notes:**
 Go over Pseudoarray Image, Current Gene, Data Filter
 Analysis Menu organized as a first approximation of an analysis - actual analyses obviously more complex
    1. File - database, file access operations 
     2. Samples - select lists of hybridized sample conditions 
     3. Edit - edit gene and condition subsets, E.G.L. and preferences
     4. Analysis - primary analysis menus
        4.1 GeneClass - select gene subset for gene class data Filter
        4.2 Normalization - select gene intensity normalization mode
        4.3 Filter - select data filters to compute gene subset of interest
        4.4 Plot - pseudoarray image, scattter, histograms, expression                                                                                                                                                                                                       profile popup plots
        4.5 Cluster - perform cluster analysis on data filtered genes
        4.6 Report - popup spreadsheet reports of genes & samples
    5. View - setup genomic gene data views preferences
     6. Plugins - add and execute new MAEPlugin methods 
     7. Help - popup documentation on MAExplorer and specific DB 

## 4. Database I/O: file and Web I/O

- All I/O goes through a single FileIO class that determines if _local disk_ **file://**  or _Web_  **http://**  I/O needs is needed
- Startup using either <APPLET> HTML code or .mae startup files
- Web **http://** I/O is read using  the JavaCGIbridge class
- All data files are _tab-delimited tables_ read by a Table base class
- Different types of specialized tables are extended from the Table base class (e.g. ConfigTable,  SamplesTable,  GipoTable,  MaHPquantTable,  etc.)
- The MaHybridSample base class contains quantitative data used only for input
- Then, a particular _project_ DB subset, is a set of MaHybridSample instances
- Used as an applet, protected projects require a user login validation by a CGI server program - modifying the JavaCGIbridge class

## 4.1 Expression Data Used in MAExplorer

**Database configuration data table **for specific array layout and content** ****
**Hybridized array samples table **describing their experimental** **conditions ******
**Gene-In-Plate-Order table** listing Clone Ids, gene names, genomic DB Ids, spot and source plate coordinates ******
**Quantified array spot data** **table** for samples from quantification software such as GenePix, ScanAnalyze, Research Genetics’ PathwaysTM, Molecular Dynamics’ ImageQuantTM, etc. ******
**Data is optionally cached **from a microarray Web database server data to the local computer.  Future analysis of this data is then independent of the Web database server
**External Web genomic databases** corresponding to probes and Clone IDs are accessed as needed:  I.M.A.G.E, GeneBank, dbEST, UniGene, LocusLink, NCI/CIT mAdb Clone DB, GeneCard, MGAP histology and model DBs, etc*.                                                                                                                                                                                                                                                                                                                                                                             ***** _Required tab-delimited data files for MAExplorer are indicated in blue_ ** **

## 5. Data structures: Genes and gene lists

- The  basic unit of discourse is the _gene_.  This is defined by the Gene base class. Sets or lists of genes are defined by the GeneList class for named gene lists
- _Named gene sets_ may be saved and used in subsequent set operations
- The GeneBitSet class is a specialized efficient implementation of  the Java BitSet class (that it does _not_ use)
- The GeneList _always_ implements the GeneBitSet class - an _unordered_ _set_
- If an _ordered list_ of genes is required, then an explicit list of Gene[s] is defined in the *GeneList.mList[] *
- _Set operations_ (Union, Intersection, Difference) are implemented using logical 64-bit operations (that are VERY FAST) on GeneBitSet[s].

## 6.1 The gene data “Filter”

- The gene data _“Filter”_ is the intersection of  various tests selected from the menu by the user and is implemented as a Filter class
- If  a particular test requires _direct manipulation of parameter values_, then scrollers are added to a StateScroller class pop up window and removed when the filter is disabled
- The StateScroller window contains a variable number of active adjustment scrollers for all active data Filter test parameters
- _The data Filter is run_ whenever parameters change and resulting gene sets for the active tests are intersected (GeneBitSets) to create the _working gene list_
- The _working gene list_ is displayed as colored overlays in the pseudo image and scatter plots (and is the *Filter.workingCL** *GeneList)

## 6.1 Gene data Filter is intersection of tests

- Working set of genes is **intersection** of gene sets each passing selected filter tests
- Filtered gene subset is used as **pre-filter** for subsequent clustering, plots, and tables
- Changing any filter parameters causes the data filter to be re-computed

## 7. Multiple pop up windows

- Multiple  pop up windows (e.g. Expression profiles, scatter plots in different modalities, etc.) are sometimes allowed so that _different interpretations_ of the data would be available at the same time
- All windows are registered with the PopupRegistry that tracks refreshing (see 10. Synchronizing windows)
- Plots use _data from the working gene list_.  If it changes, then the plots will change.
- The user may _select genes_ or sets of genes from plots
- All plots _track both genes and their x,y positions in the plots_.  This helps the event handlers quickly determine  which gene was selected

## 7.1 E.g. Multiple Expression Profile plots

## 7.2 Two-dimensional plots: scatter plots and expression profile overlays

- The Draw2Dplot class extends Canvas
- The DrawScatterPlot and ExprProfileOverlay classes are extended from Draw2Dplot
- Draw2Dplot class _zoom scroll bars_ compute scale factors used to clip data and scale it to the actual size of the plot window
- Tracks (x,y,gene) for drawn points so event handler can track points clicked on to determine corresponding gene

## 7.3 E.g. Scatter Plots of Two Conditions

## 8.  Other plot window classes

- A scrollable _list of expression profile instances_ is implemented by creating a grid-layout in a ScrollPane that was contained in a pop up Frame
- Then, individual ExprProfilePanel plots are inserted in the ScrollPane grid elements
- The functionality of the ExprProfilePopup, DrawRatioHistogram, DrawHistogram, and ClusterGram  (includes dendrograms) classes were so specialized that we created new classes instead of extending the Draw2Dplot class (used for DrawScatterPlot, ExprProfileOverlay)
- The event handlers for the histogram plots _track histogram bins_ and may be used for setting additional tests in the data Filter
- The histogram bin Filter tests track the _range of gene intensities or ratios_ selected for a particular histogram bin

## 8.1 Scrollable list of Expression Profile plots

## 8.2 Expression Profile Overlay plots

## 8.3 Ratio Histogram plot and data filter 

## 8.4 Intensity Histogram plot and data filter 

## 8.5 ClusterGram and DendroGram plots

## 9. Reports and access to other Web databases

- _Sample and gene Reports_ are available in Report class instances as either
-      1) dynamic selectable spreadsheets or
-      2) tab-delimited TextArea’s for export via cut & paste to Excel
- The _dynamic scrollable spreadsheet_ uses a SpreadSheet class with separate event handlers that enable
-     1) scrolling cells,
-     2) Web access if enabled using popupViewer() method in Util class,
-     3) sorting rows by data in a particular column
- Applet access to other Web genomic DBs uses a _proxy server_ residing on the _same_ Web server as array data.  Stand-alone access is done directly
- _Web data_ is requested by selecting a spot in the array, a point in a scatter plot, or a cell in a report. Data is displayed in a new browser window

## 9.1 Scrollable Dynamic Gene Reports

-       Scrollable gene report of highest ratio genes & NCI mAdb pop up Web browser page (foreground) of particular gene. Clicking on blue hypertext cell in gene report (middle) invokes pop up web pages

## 9.2 Tab-delimited report for cut & paste to Excel

- Clones clustered with similar expression profiles

## 9.3 Saving Reports of Results

- _Plot Results_ saved as: GIF image files
- _Table Report Results_ saved as: tab-delimited text files
- Reports:                                                                                                                                                                                                                                              1.  Web-accessible _dynamic spreadsheets_ or                                                                                                                                                                                                                     2.  _tab-delimited text_ exportable to Excel
- _Gene set_ reports - linked to _pop-up Web browser_ reports from genomic Web databases
- _Array sample_ reports
-      1. information on samples
-      2. _pop-up Web browser_ reports linked to histology and                                                                                                                                                                                                          mouse model Web pages,
-      3. data filtered sample vs sample correlation coefficients

## 10. Synchronizing Windows

- _Direct manipulation_ of data in plots and reports is synchronized to highlight the same current gene and Filtered genes if they are changed in any of these windows
- E.g. _selecting_ (clicking on) a gene in the pseudo array image, scatter plots and gene report windows, will change it in the other visible windows
- The PopupRegistry class is used to register all windows when they are created, refresh them when the system state changes (normalization, Filter, current clone, E.G.L, etc.), and remove them when the user is finished with them
- The _system state_ changes if the current gene, current cluster, or Filter changes. The popup registry is called and in turn notifies all relevant plots and reports that the state has changed so they can update themselves if needed

## 10.1 PopupRegistry  plot types and callbacks

**Types of windows that may be registered with the ****PopupRegistry****:**
** **1. ShowPlotPopup                     for plot popups
-  2. ShowSpreadsheetPopup        for dynamic spreadsheet report popups
-  3.  ShowStringPopup                 for String report popups
-  4.  ShowExprProfilesPopup      for expression profile Expression Profile popups
-  5.  MAEPluginUpdateListener  for active MAEPlugins
-  6.  Object                                   also for active Popup Window Object's Object
**Types of **_**callback methods**_** available for registered popup windows**
-  1. *updateCurGene(mid)*            if the current gene has changed to mid
-  2. *updateFilter(gene list)*           if data filter has changed the working gene list
-  3. *updateSlider()*                        if any threshold slider has changed
-  4. *updateLabels()*                      if any label names (eg. Class names,etc) have changed

## 11.  Saving text & plot windows to .txt & .gif files

- Text  popup windows are written out in stand-alone mode as tab-delimited .txt files when servicing SaveAs commands. Data is written using *FileIO.writeFileToDisk()*
- Later, tab-delimited text files may be read into Excel
- The WriteGifEncoder class is used to convert repainted Images to Gif encoded .gif files in stand-alone mode when servicing “SaveAs” commands in plot windows
- All .txt and .gif files saved by the user doing a “SaveAs” command are saved in the **/Report** directory

## 12. MAExplorer state: Saving and Restoring

- The UpdateState class is used to read and write state information files including: .mae state startup files, .cbs GeneBitSet files, .hpl SampleSet files
- All  .mae files are saved in the **/MAE** directory
- All .cbs and .hpl files are saved in the **/State** directory
- The UpdateState*.readMAEstartupFile()* method is used to read the specified .mae startup file. This overides (Name,Value) pairs setup initially through the Config class
- When (File | Databases | SaveAs ... DB) is invoked, it writes out the current state with UpdateState.w*riteMAEstartupFile()* method. This puts the names of the current .cbs and .hpl files in the .mae file - not the data itself. Then it writes out the .cbs and .hpl files

## 13. Miscellaneous Classes

- The MathMAE class contains various specialized math functions
- The Statistics class contains various specialized statistics and probability functions (t-test, etc)
- The SortMAE class contains various sorting methods specific for MAExplorer
- The Util class contains various utility methods uses in multiple modules for data structure conversion, display, string conversion, colormap conversion, etc.

## 14. MAEPlugin Paradigm

- Plugins are installed and dynamically loaded when MAExplorer starts or when it is running using the MAEPlugin class and package
- Plugins are invoked from pull-down menus
- Client-server plugins may invoke MAExplorer from other programs or vice versa
- Plugins may access _any_ MAExplorer data structures, but in a portable structured way using the MaeJavaAPI and MJAxxxx classes (Open Java API)
- Plugins may provide their own GUI interfaces or save data back into MAExplorer and use it’s plot and report capabilities

## 14.1 MAEPlugin Extensions for MAExplorer

- _Java plugins_ allow investigators to extend capabilities of core MAExplorer program to new analysis methods
- _Web site contents:_ Open Java API, Java open-source examples, donated plugins & links will be published and freely available
- _MAEPlugin types_: normalization, metrics, Filters, PCA, clustering, client-server, functional genomic analysis of cluster results, etc.
- _MAEPlugins_ will have three types of implementations:                                                                                                                                                                                                           1. Using 100% Java code                                                                                                                                                                                                                       2. Access local programs written in any language  (e.g. ‘R’)                                                                                                                                                                                                   3. Web-CGI or client-server to specialized genomic DBs

**Notes:**
 Users could add new analytic methods

 (API) Applications Programming Interface

 (PCA) Principal Components Analysis

 Various types of analysis plugins: local methods and      client- server

 Implementations can access other genomic DBs 

 These may have additional gene function analysis that could be performed on gene sets separated  by gene expression

## 14.2 Montage MAEPlugin of Current Gene for Visual Verification

**14.2 Montage **_**MAEPlugin**_** of Current Gene ****for Visual Verification**

**1. Load Montage MAEPlugin**

**2. Select current gene**

**3. Show Montage  on current gene**

**4. Montage MAEPlugin showing original image regions of current gene**

**Notes:**

 Example of a MAEPlugin to perform visual verification of original array images around particular spots

Process: 1-4

 Plugins may be installed within particular menus where most appropriate

## 14.2 MAEPlugin Design

**Notes:**
 User plugins may be dynamically loaded at startup or any time when running

 “Cluster” is an example of a new method

 (JAR) file is loaded by MAExplorer

 The plugin accesses data through Open Java API

 This in turn accesses local array data

 New computations by the plugin may be “put” back into the MAExplorer DB

## 14.3 Open Java API for Writing MAEPlugins

## 15.  MaeJavaAPI Open Java API

- The MaeJavaAPI class implements methods for MAEPlugin.MAEstub class
- Typical user written plugins can adopt the following convention:
-   1. E.g. DemoPlugin.java implements the MAEstub methods DemoPlugin.pluginMain(), DemoPlugin.updateCurGene(), DemoPlugin.updateFilter(), DemoPlugin.updateSliders(), DemoPlugin.updateLabels()
-   2. It creates an instance of a Demo class which actually implements the plugin analysis
-   3. DemoPlugin passes down the instance of *MAExplorer.mja* to the Demo() constructor so it can access all of the MJAxxxx classes
-   Data is accessed through MJAxxxx class methods
-   See sample plugins and MAExplorer javadocs for more details (on Web site)

## MJAbase               constants used by other MJAxxxx classes
MJAcluster            cluster data structures and methods
MJAcondition       condition lists of samples & ordered lists of condition lists
MJAeval                command interpreter for use with client-server access
MJAexprProfile     expression profiles data
MJAfilter               gene data filters
MJAgene               single gene data
MJAgeneList         lists of genes and get sets
MJAgenomicDB    genomic databases on the Internet 
MJAgeometry        array geometry, spot to gene maps, etc.
MJAhelp                 popup browser help methods
MJAhistogram        histograms and histogram plots
MJAmath                built-in math functions
MJAnormalization  normalization data and methods

- MJAcluster            cluster data structures and methods
- MJAcondition       condition lists of samples & ordered lists of condition lists
- MJAeval                command interpreter for use with client-server access
- MJAexprProfile     expression profiles data
- MJAfilter               gene data filters
- MJAgene               single gene data
- MJAgeneList         lists of genes and get sets
- MJAgenomicDB    genomic databases on the Internet
- MJAgeometry        array geometry, spot to gene maps, etc.
- MJAhelp                 popup browser help methods
- MJAhistogram        histograms and histogram plots
- MJAmath                built-in math functions
- MJAnormalization  normalization data and methods

**15.1  MaeJavaAPI **_**MJAxxxx classes**_

## 15.2  MaeJavaAPI MJAxxxx classes

-  MJAproperty          get and put individual MAExplorer state properties
-  MJApropList          get lists of properties
-  MJAsample            get and put single sample top-level data
-  MJAsampleList      get lists of samples top-level data
-  MJAsort                  built-in sort methods oriented toward MAExplorer
-  MJAstatistics          built-in statistics methods
-  MJAstate                get and save state, get additional state info
-  MJAutil                  built-in utility methods

## 16. Examples (links): MAExplorer menu organization

**File menu**

**Gene Class menu**

**Sample menu**

**Edit menu**

**Analysis menu**

**View menu**

**Plugin menu**

**Plot menu**

**Help menu**

**Normalization menu**

**Cluster menu**

**Pseudoarray menu**

**Report menu**

**Scatter plot menu**

**Histograms menu**

**Expression Profiles menu**

**Sample reports menu**

**Gene set reports menu**

**Filter menu**

**Notes:**
 Go over Pseudoarray Image, Current Gene, Data Filter
 Analysis Menu organized as a first approximation of an analysis - actual analyses obviously more complex
    1. File - database, file access operations 
     2. Samples - select lists of hybridized sample conditions 
     3. Edit - edit gene and condition subsets, E.G.L. and preferences
     4. Analysis - primary analysis menus
        4.1 GeneClass - select gene subset for gene class data Filter
        4.2 Normalization - select gene intensity normalization mode
        4.3 Filter - select data filters to compute gene subset of interest
        4.4 Plot - pseudoarray image, scattter, histograms, expression                                                                                                                                                                                                       profile popup plots
        4.5 Cluster - perform cluster analysis on data filtered genes
        4.6 Report - popup spreadsheet reports of genes & samples
    5. View - setup genomic gene data views preferences
     6. Plugins - add and execute new MAEPlugin methods 
     7. Help - popup documentation on MAExplorer and specific DB 

## 15.1 Files - Open and Save Exploration State

## 15.2 Samples - Select Condition Lists

## 15.2.1 Lists of Hybridized Sample Conditions

- _Database_ may consist of multiple conditions with replicates
- _Organize as lists of conditions:_                                                                                                                                                                                                                          1.  X and Y pairs of individual samples                                                                                                                                                                                                                        2.  Sets of X and Y replicates (**X-set**, **Y-set**)                                                                                                                                                                                                                  3.  Ordered expression profile list (**E-list**)
- Manipulate _named sample lists_ with AND, OR, DIFFERENCE
- _Save and restore named sample lists_ during analysis and between sessions

## 15.2.2 Sample Condition HP-X &-Y Sets, HP-E Lists

## 15.3 Edit Menu - Gene Sets & Condition Lists

## 15.3.1 Gene Set Operations help manage data and search results

- _All gene sets are “named”_  - view directory of current sets
- _Set operations_ (AND, OR, DIFFERENCE) may be used to create new derived named sets
- Special sets:                                                                                                   1. _Filtered genes set_     holds genes passing the data filter                                                                                            2. _Edited Gene List_      holds results of clustering or editing                         3. _Normalization set_     may be used as normalization method                               4. _User data filter set_    may be used as a data filter
- Genes sets are _saved_ when the session is saved, _restored_ when MAExplorer restarted

## 15.3.2 Gene Set Operations - e.g. ‘AND’ of Two Sets

## 15.3.3 Find Gene - Enter gene name, then press Done

## 15.3.4 Find Gene Subset by Enter gene name,then Press Set E.G.L.

## 15.4 Analysis Menu - Organized by Task

**Notes:**
 Go over Pseudoarray Image, Current Gene, Data Filter
 Analysis Menu organized as a first approximation of an analysis - actual analyses obviously more complex
    1. File - database, file access operations 
     2. Samples - select lists of hybridized sample conditions 
     3. Edit - edit gene and condition subsets, E.G.L. and preferences
     4. Analysis - primary analysis menus
        4.1 GeneClass - select gene subset for gene class data Filter
        4.2 Normalization - select gene intensity normalization mode
        4.3 Filter - select data filters to compute gene subset of interest
        4.4 Plot - pseudoarray image, scattter, histograms, expression                                                                                                                                                                                                       profile popup plots
        4.5 Cluster - perform cluster analysis on data filtered genes
        4.6 Report - popup spreadsheet reports of genes & samples
    5. View - setup genomic gene data views preferences
     6. Plugins - add and execute new MAEPlugin methods 
     7. Help - popup documentation on MAExplorer and specific DB 

## 15.4.1 Gene Class Menu

## 15.4.2 Normalization Menu

## 15.4.3 Filter gene data Operations By Multiple Tests

## 15.4.3.1 Gene data filters - finding a gene subset

- The selected _data-filters_ tests are applied to all genes
- Creates working _subset of genes_ used for subsequent clustering, plots, and reports
- _Data filters_ compute _intersection_ (AND) of sets from tests:                                                                                                                                                                                                    1. Gene subsets from previous operations                                                                                                                                                                                                                 2. Spot intensity and ratio ranges                                                                                                                                                                                                                             3. Statistics: CV, t-Test                                                                                                                                                                                                                                      4. Clustering: similar expression profiles, K-means, hierarchical
- Changing filter parameters _recomputes_ active data filter and updates active plot or cluster displays

## 15.4.4 Plot menu - pseudoarrays, zoomable scatter plots, histograms, expression profile plots

## 15.4.4.1 Direct manipulation of gene data in plots

-    1.  _Pseudoarray image_ - intensity, ratio (X/Y) or (Cy3/Cy5),                                                                                                                                                                                                    sums (red+green) of (Cy5**+**Cy5) or (Y+X), etc.
-    2.  _Zoomable scatter plots_ - X vs Y, Cy3 vs Cy5, duplicate                                                                                                                                                                                                      spots, labeled K-means clusters
-    3.  _Histograms_ - ratio and intensity, select bins for filtering
-    4.  _Expression profiles_ - individual genes, lists, overlay plots

## 15.4.4.1 Plots - Pseudoarrays of functions of samples (Show Microarray)

## 15.4.4.1.1 Pseudoarray of individual X/Y samples

## 15.4.4.1.2 Pseudoarray of (mean X / meanY) sample sets

## 15.4.4.1.3 Pseudoarray sum Cy3+Cy5 per sample

## 15.4.4.1.4 Pseudoarray of intensity of one sample

## 15.4.4.1.5 Pseudoarray of X- vs Y-sets p-value (t-test)

## 15.4.4.2 Zoomable Scatter Plots

## 15.4.4.3 Plots - Histograms 

## 15.4.4.3.1 Plots - Ratio Histogram HP-X/HP-Y

**15.4.4.3.1 Plots - **_**Ratio Histogram**_** HP-X/HP-Y**

## 15.4.4.4 Plots - Expression Profile plots

## 15.4.4.5 Clustering Similar Genes, K-means, Hierarchical

## 15.4.4.5.1 Cluster operations on filtered genes 

-    1. _Find genes similar to specified gene_ - sorted list and                                                                                                                                                                                                      _silhouette_ _plots_, gene reports, save cluster as set
-    2. K-means clustering given # (K) of clusters - sorted lists and                                                                                                                                                                                               Expression Profile (EP) plots, _silhouette plots_, gene reports,                                                                                                                                                                                                 save clusters as sets
-    3. _Hierarchical clustering of genes_ - clustergram, dendrogram,                                                                                                                                                                                                 list of EP plots, gene reports

## 15.4.4.6 Reports - Samples and Gene Subsets

## 15.4.4.6.1 Report Menu - Samples Reports

## 15.4.4.6.2 Report Menu - Gene Reports

## 15.4.4.6.3 Scrollable Dynamic Gene Reports

## 15.4.4.6.4 Scrollable Dynamic Gene Reports

## 15.4.5 View Menu

## 15.6 Plugins Menu

## 15.7 Help Menu