-------------------------- --------------------------------------------
  Last Revised:              10.17.2007

  Produced By:               NCICB Development Team

  Version:                   0.4
  -------------------------- --------------------------------------------

Revision History

  ------------ --------- ------------------------------------ ---------------
  Date         Version   Description                          Revised by

  10/15/2007   0.1       Initial Draft                        Bill Mason

  10/17/2007   0.2       Updated with details of workloads    Rashmi
                         and test cases. Also reorganized     Srinivasa
                         document.                            

  10/18/2007   0.3       Fleshed out the API and search test  Rashmi
                         cases and cleaned up document.       Srinivasa

  11/19/2007   0.4       Added all remaining tests and        Rashmi
                         revised the search and download      Srinivasa
                         tests.                               

                                                              

                                                              

                                                              
  ------------ --------- ------------------------------------ ---------------

Table of Contents

1\. Introduction [iv](#introduction)

2\. Document Terminology [iv](#document-terminology)

2.1.1. Load Testing and Stress Testing
[iv](#load-testing-and-stress-testing)

2.1.2. Workload [iv](#workload)

3\. Approach [iv](#approach)

3.1. Architecture and Test Environment
[iv](#architecture-and-test-environment)

3.2. Baselining and Benchmarking the Tests
[v](#baselining-and-benchmarking-the-tests)

3.3. Performance Metrics [v](#performance-metrics)

3.4. Performance Report [v](#performance-report)

4\. Workloads [v](#workloads)

4.1.1. CaArray Upload Workload [v](#caarray-upload-workload)

4.1.2. CaArray Import Workload [vi](#caarray-import-workload)

4.1.3. CaArray Download Workload [vi](#caarray-download-workload)

4.1.4. CaArray Search Workload [vi](#caarray-search-workload)

4.1.5. CaArray API Download Workload
[vii](#caarray-api-download-workload)

4.1.6. CaArray API Search Workload [vii](#caarray-api-search-workload)

4.1.7. CaArray Browse Workload [viii](#caarray-browse-workload)

4.1.8. CaArray Browse Experiment Workload
[viii](#caarray-browse-experiment-workload)

5\. Load Test Cases [viii](#load-test-cases)

5.1.1. CaArray Upload Test [viii](#caarray-upload-test)

5.1.2. CaArray Import Test [ix](#caarray-import-test)

5.1.3. CaArray Download Test [ix](#caarray-download-test)

5.1.4. CaArray Search Test [ix](#caarray-search-test)

5.1.5. CaArray Remote Java API Download Test
[x](#caarray-remote-java-api-download-test)

5.1.6. CaArray GRID API Download Test
[x](#caarray-grid-api-download-test)

5.1.7. CaArray Remote Java API Search Test
[xi](#caarray-remote-java-api-search-test)

5.1.8. CaArray GRID API Search Test [xi](#caarray-grid-api-search-test)

5.1.9. CaArray Browse Test [xi](#caarray-browse-test)

5.1.10. CaArray Browse Experiment Test
[xii](#caarray-browse-experiment-test)

6\. Stress Test Cases [xii](#stress-test-cases)

6.1.1. CaArray Upload Stress Test [xii](#caarray-upload-stress-test)

6.1.2. CaArray Import Stress Test [xiii](#caarray-import-stress-test)

6.1.3. CaArray Download Stress Test
[xiii](#caarray-download-stress-test)

6.1.4. CaArray Search Stress Test [xiii](#caarray-search-stress-test)

6.1.5. CaArray Remote Java API Download Stress Test
[xiv](#caarray-remote-java-api-download-stress-test)

6.1.6. CaArray GRID API Download Stress Test
[xiv](#caarray-grid-api-download-stress-test)

6.1.7. CaArray Remote Java API Search Stress Test
[xv](#caarray-remote-java-api-search-stress-test)

6.1.8. CaArray GRID API Search Stress Test
[xv](#caarray-grid-api-search-stress-test)

6.1.9. CaArray Browse Stress Test [xv](#caarray-browse-stress-test)

6.1.10. CaArray Browse Experiment Stress Test
[xvi](#caarray-browse-experiment-stress-test)

6.1.11. CaArray Heterogeneous Workload Stress Test
[xvi](#caarray-heterogeneous-workload-stress-test)

> Performance Test Plan

# Introduction

The goal of performance testing is not to find bugs, but to eliminate
bottlenecks and establish a baseline for future regression testing. To
conduct performance testing is to engage in a carefully controlled
process of measurement and analysis.

# Document Terminology

### Load Testing and Stress Testing

-   A *load test* is a simulation of real-life use of a Web application,
    performed to assess how the application will work under actual
    conditions of use. The test is performed under expected conditions
    both prior to application deployment and during the life cycle of
    the application.

-   A *stress test* is designed to determine how heavy a load the Web
    application can handle. Stress testing does not aim to break the
    system by overwhelming it, but instead tries to keep the system
    constantly humming like a well-oiled machine. Stress testing will
    expose bugs that do not surface in cursory testing, such as memory
    management bugs, memory leaks, buffer overflows etc.

### Workload

A workload needs to be established before testing begins. A workload is
the total burden of activity placed on the Web application to be tested.
This burden consists of a certain number of virtual users who process a
defined set of transactions in a specified time period. Assigning a
proper workload is the crux of performance testing.

# Approach

## Architecture and Test Environment

Before choosing the right model for performance tests, it is important
for us to understand the architecture. Testing in an environment that
does not mimic production provides wrong results. It is crucial to know
the specifications of the web server, databases, or any other external
dependencies the application might have. Building the test environment
as close as possible to production is the key to providing accurate
results. Currently, the Dev and QA environments are using VMware in a
shared environment. Each application is given an execution priority,
which will have a significant impact on performance testing. It is
planned to get a dedicated VMware box to perform these tests.

## Baselining and Benchmarking the Tests

A benchmark test is conducted to profile performance for different
software releases. Benchmarking helps us to run tests under similar
conditions in order to compare results for different software, to verify
that new features and functions do not impact response times of the Web
application under test, and to find system bottlenecks. Defining what
each virtual user does and measuring response times for a single virtual
user performing a certain transaction is what baselining is all about.
Once we have data for a single user or workload, we can then ramp up the
number of users linearly to \"n\" users and measure the response times
to look for deviations.

## Performance Metrics

Most performance tools measure round-trip response times. \"Round trip\"
can be defined as the time taken between the request being sent from the
client and the response received from the server. Round-trip time
includes the network time that is not a constant depending on the
environment in which the test is conducted, the external dependencies on
databases, and network latency. Hence, round-trip response times are not
an accurate indicator of the performance of the application. We will
measure the server-side response times from the server logs and compare
them with the round-trip times to see how the response times vary
depending on the network traffic.

## Performance Report

Performance numbers are typically reported as average, minimum, and
maximum response times. We will run the performance tests for a
specified time, and the average response time will be the most important
indicator of performance. We will report maximum and minimum response
times to provide a means to investigate why a certain transaction with a
certain input indicated a good or bad response time.

# Workloads

The following are the basic workloads that will be used by the test
cases.

### CaArray Upload Workload

-   Description: One user uploads files for an experiment through the
    CaArray UI.

-   Setup

    -   User logs in.

    -   User creates and opens an experiment.

-   Steps:

    -   User uploads one or more files (total size of data = 1KB-4GB).

### CaArray Import Workload

-   Description: One user imports already-uploaded files into an
    experiment through the CaArray UI. This workload exercises the
    validation functionality and the import of the data into the
    database.

-   Setup

    -   User logs in.

    -   User creates and opens an experiment.

    -   User uploads one or more files (total size of data = 1KB-4GB).

-   Steps:

    -   User imports the uploaded files.

### CaArray Download Workload

-   Description: One user downloads array data files through the CaArray
    UI.

-   Setup:

    -   An experiment with appropriate size of hybridization data files
        exists and is viewable by the user.

    -   User logs in.

    -   User opens an experiment with data files.

-   Steps:

    -   User downloads array data files from an Experiment (size of
        uncompressed data = 1KB-10GB):

### CaArray Search Workload

-   Description: One user executes an experiment search through the
    CaArray UI.

-   Setup:

    -   The appropriate experiments matching the search criteria exist
        and are viewable by the user.

-   Steps:

    -   User enters search criteria which will match 5-30% of the total
        number of experiments in caArray.

    -   The experiments resulting from the search are returned.

    -   The above steps are repeated for different database sizes
        (100-4000 experiments).

### CaArray API Download Workload

-   Description: One external user downloads array data using the Remote
    Java API or the GRID API.

-   Setup:

    -   An experiment with the appropriate size of hybridization data
        files exists and is viewable by the user.

    -   User connects to the CaArray server and logs on (either through
        the remote Java API or through the CaArray Grid Service).

    -   User selects the hybridization(s) and quantitation type(s) for
        an experiment.

-   Steps:

    -   User downloads array data for the selected hybridization(s) and
        quantitation type(s) (size of data = 1KB-4GB).

    -   Repeat the above step for all types of arrays:

        -   Affymetrix raw data (.cel)

        -   Affymetrix derived data (.chp)

        -   Illumina raw data

        -   Illumina derived data

        -   Genepix

### CaArray API Search Workload

-   Description: One external user executes an experiment search using
    the Remote Java API or the GRID API.

-   Setup:

    -   The appropriate experiments matching the search criteria exist
        and are viewable by the user.

    -   User connects to the CaArray server (either through the remote
        Java API or through the CaArray Grid Service).

-   Steps:

    -   User enters search criteria which will match 5-30% of the total
        number of experiments in caArray.

    -   The experiments resulting from the search are returned.

    -   The above steps are repeated for different database sizes
        (100-4000 experiments).

    -   The above steps are repeated for the following search APIs:

        -   CQL search

        -   Search by example

### CaArray Browse Workload

-   Description: One user browses through public experiments in the
    CaArray UI.

-   Setup:

    -   The appropriate number of experiments exists, and the
        experiments are viewable by the user.

-   Steps:

    -   User browses to the public experiments page in caArray.

    -   User clicks on a few pages of the paginated results.

    -   The above steps are repeated for different database sizes
        (100-4000 experiments).

### CaArray Browse Experiment Workload

-   Description: One user browses through the details of a single
    experiment in the CaArray UI.

-   Setup:

    -   An experiment exists and is viewable by the user.

-   Steps:

    -   User clicks on an experiment in caArray.

    -   User clicks on a few tabs describing the annotations and data
        files associated with the experiment.

    -   The above steps are repeated for different database sizes
        (100-4000 experiments).

# Load Test Cases

Using JMeter, tests will be run to simulate 10 concurrent users
exercising the basic workloads.

### CaArray Upload Test

-   Description: This tests concurrent users uploading files into
    experiments using the CaArray UI. The basic CaArray Upload workload
    is exercised by 10 concurrent users. The total file size being
    uploaded is varied from 1KB to 4GB. The response time is captured.

-   Steps:

    -   10 concurrent users run the CaArray Upload workload.

    -   The above step is repeated for a total file size varying from
        1KB to 4GB.

-   Expected Results

    -   The upload response time does not degrade unacceptably as the
        file size increases.

### CaArray Import Test

-   Description: This tests concurrent users importing already-uploaded
    files into experiments using the CaArray UI. It exercises the
    validation functionality and the import of the data into the
    database. The basic CaArray Import workload is exercised by 10
    concurrent users. The total file size being uploaded is varied from
    1KB to 4GB. The response time is captured.

-   Steps:

    -   10 concurrent users run the CaArray Import workload.

    -   The above step is repeated for a total file size varying from
        1KB to 4GB.

-   Expected Results

    -   The import response time does not degrade unacceptably as the
        file size increases.

###  CaArray Download Test

-   Description: This tests concurrent users downloading array data from
    experiments using the CaArray UI. The basic CaArray Download
    workload is exercised by 10 concurrent users. The total data size
    being downloaded is varied from 1KB to 10GB. The response time is
    captured.

-   Steps:

    -   10 concurrent users run the CaArray Download workload.

    -   The above step is repeated for a total data size varying from
        1KB to 10GB.

    -   The test is run for all the alternatives described in the
        CaArray Download workload (number of hybridizations, number of
        quantitation types, different array types).

-   Expected Results

    -   The download time does not degrade unacceptably as the data size
        increases.

### CaArray Search Test

-   Description: This tests concurrent users searching for experiments
    using the CaArray UI. The basic CaArray Search workload is exercised
    by 10 concurrent users. The size of the database (number of
    experiments) is varied from 100 to 4000. The response time is
    captured.

-   Steps:

    -   10 concurrent users run the CaArray Search workload.

    -   The above step is repeated for a database size of 100, 500,
        1000, 2000 and 4000 experiments.

-   Expected Results

    -   The response time does not degrade unacceptably as the database
        size increases.

### CaArray Remote Java API Download Test

-   Description: This tests concurrent users downloading array data from
    experiments using the CaArray Remote Java API. The basic CaArray API
    Download workload is exercised by 10 concurrent users connecting
    through the remote Java API. The total data size being downloaded is
    varied from 1KB to 4GB. The response time is captured.

-   Steps:

    -   10 concurrent users connect through the remote Java API and run
        the CaArray API Download workload.

    -   The above step is repeated for a total data size varying from
        1KB to 4GB.

    -   The test is run for all the array types described in the CaArray
        API Download workload.

-   Expected Results

    -   The download time does not degrade unacceptably as the data size
        increases.

### CaArray GRID API Download Test

-   Description: This tests concurrent users downloading array data from
    experiments through the CaArray GRID Service. The basic CaArray API
    Download workload is exercised by 10 concurrent users connecting
    through the CaArray GRID Service. The total data size being
    downloaded is varied from 1KB to 4GB. The response time is captured.

-   Steps:

    -   10 concurrent users connect through the CaArray GRID Service and
        run the CaArray API Download workload.

    -   The above step is repeated for a total data size varying from
        1KB to 4GB.

    -   The test is run for all the array types described in the CaArray
        API Download workload.

-   Expected Results

    -   The download time does not degrade unacceptably as the data size
        increases.

### CaArray Remote Java API Search Test

-   Description: This tests concurrent users searching for experiments
    using the CaArray Remote Java API. The basic CaArray API Search
    workload is exercised by 10 concurrent users connecting through the
    remote Java API. The size of the database (number of experiments) is
    varied from 100 to 4000. The response time is captured.

-   Steps:

    -   10 concurrent users connect through the remote Java API and run
        the CaArray API Search workload.

    -   The above step is repeated for a database size of 100, 500,
        1000, 2000 and 4000 experiments.

    -   The test is run for the two search types described in the
        CaArray API Search workload (CQL search and search by example).

-   Expected Results

    -   The response time does not degrade unacceptably as the database
        size increases.

### CaArray GRID API Search Test

-   Description: This tests concurrent users searching for experiments
    through the CaArray GRID Service. The basic CaArray API Search
    workload is exercised by 10 concurrent users connecting through the
    CaArray GRID Service. The size of the database (number of
    experiments) is varied from 100 to 4000. The response time is
    captured.

-   Steps:

    -   10 concurrent users connect through the CaArray GRID Service and
        run the CaArray API Search workload using a CQL query.

    -   The above step is repeated for a database size of 100, 500,
        1000, 2000 and 4000 experiments.

-   Expected Results

    -   The response time does not degrade unacceptably as the database
        size increases.

### CaArray Browse Test

-   Description: This tests concurrent users browsing through
    experiments using the CaArray UI. The basic CaArray Browse workload
    is exercised by 10 concurrent users. The size of the database
    (number of experiments) is varied from 100 to 4000. The response
    time is captured.

-   Steps:

    -   10 concurrent users run the CaArray Browse workload.

    -   The above step is repeated for a database size of 100, 500,
        1000, 2000 and 4000 experiments.

-   Expected Results

    -   The response time does not degrade unacceptably as the database
        size increases.

### CaArray Browse Experiment Test

-   Description: This tests concurrent users browsing through the
    details of a single experiment using the CaArray UI. The basic
    CaArray Browse Experiment workload is exercised by 10 concurrent
    users. The size of the database (number of experiments) is varied
    from 100 to 4000. The response time is captured.

-   Steps:

    -   10 concurrent users run the CaArray Browse Experiment workload.

    -   The above step is repeated for a database size of 100, 500,
        1000, 2000 and 4000 experiments.

-   Expected Results

    -   The response time does not degrade unacceptably as the database
        size increases.

# Stress Test Cases

Using JMeter, tests will be run to simulate 10 concurrent users
exercising the basic workloads with the highest load or largest data
set. The number of concurrent users will then be escalated to find the
maximum user load under which the application still performs acceptably
well.

### CaArray Upload Stress Test

-   Description: This tests concurrent users uploading files into
    experiments using the CaArray UI. The basic CaArray Upload workload
    is exercised with a file size of 4GB (maximum) by 10 concurrent
    users. The number of concurrent users is increased progressively
    until the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users run the CaArray Upload workload with a file
        size of 4GB.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users uploading files that can
        be supported by CaArray.

### CaArray Import Stress Test

-   Description: This tests concurrent users importing already-uploaded
    files into experiments using the CaArray UI. It exercises the
    validation functionality and the import of the data into the
    database. The basic CaArray Import workload is exercised with a file
    size of 4GB (maximum) by 10 concurrent users. The number of
    concurrent users is increased progressively until the response time
    becomes unacceptably high.

-   Steps:

    -   10 concurrent users run the CaArray Import workload with a file
        size of 4GB.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users importing files that can
        be supported by CaArray.

###  CaArray Download Stress Test

-   Description: This tests concurrent users downloading array data
    files from experiments using the CaArray UI. The basic CaArray
    Download workload is exercised with a total data size of 10GB
    (maximum) by 10 concurrent users. The number of concurrent users is
    increased progressively until the response time becomes unacceptably
    high.

-   Steps:

    -   10 concurrent users run the CaArray Download workload with a
        data size of 10GB.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users downloading files that
        can be supported by CaArray through the UI.

### CaArray Search Stress Test

-   Description: This tests concurrent users searching for experiments
    using the CaArray UI. The basic CaArray Search workload is exercised
    with a database size of 4000 experiments (maximum) by 10 concurrent
    users. The number of concurrent users is increased progressively
    until the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users run the CaArray Search workload with a
        database size of 4000 experiments.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users searching for experiments
        that can be supported by CaArray through the UI.

### CaArray Remote Java API Download Stress Test

-   Description: This tests concurrent users downloading array data from
    experiments using the CaArray Remote Java API. The basic CaArray API
    Download workload is exercised with a total data size of 4GB
    (maximum) by 10 concurrent users connecting through the Remote Java
    API. The number of concurrent users is increased progressively until
    the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users connect through the CaArray Remote Java API
        and run the CaArray API Download workload with a data size of
        4GB.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

    -   The test is run for all the array types described in the CaArray
        API Download workload.

-   Expected Results

    -   The maximum number of concurrent users downloading files that
        can be supported by CaArray through the Remote Java API.

### CaArray GRID API Download Stress Test

-   Description: This tests concurrent users downloading array data from
    experiments using the CaArray GRID Service. The basic CaArray API
    Download workload is exercised with a total data size of 4GB
    (maximum) by 10 concurrent users connecting through the CaArray GRID
    Service. The number of concurrent users is increased progressively
    until the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users connect through the CaArray GRID Service and
        run the CaArray API Download workload with a data size of 4GB.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

    -   The test is run for all the array types described in the CaArray
        API Download workload.

-   Expected Results

    -   The maximum number of concurrent users downloading files that
        can be supported by the CaArray GRID Service.

### CaArray Remote Java API Search Stress Test

-   Description: This tests concurrent users searching for experiments
    through the CaArray Remote Java API. The basic CaArray API Search
    workload is exercised with a database size of 4000 experiments
    (maximum) by 10 concurrent users connecting through the CaArray
    Remote Java API. The number of concurrent users is increased
    progressively until the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users connect through the CaArray Remote Java API
        and run the CaArray API Search workload with a database size of
        4000 experiments.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

    -   The test is run for the two search types described in the
        CaArray API Search workload (CQL search and search by example).

-   Expected Results

    -   The maximum number of concurrent users searching for experiments
        that can be supported by CaArray through the Remote Java API.

### CaArray GRID API Search Stress Test

-   Description: This tests concurrent users searching for experiments
    through the CaArray GRID Service. The basic CaArray API Search
    workload is exercised with a database size of 4000 experiments
    (maximum) by 10 concurrent users connecting through the CaArray GRID
    Service. The number of concurrent users is increased progressively
    until the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users connect through the CaArray GRID Service and
        run the CaArray API Search workload (CQL query) with a database
        size of 4000 experiments.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users searching for experiments
        that can be supported by the CaArray GRID Service.

### CaArray Browse Stress Test

-   Description: This tests concurrent users browsing through
    experiments using the CaArray UI. The basic CaArray Browse workload
    is exercised with a database size of 4000 experiments (maximum) by
    10 concurrent users. The number of concurrent users is increased
    progressively until the response time becomes unacceptably high.

-   Steps:

    -   10 concurrent users run the CaArray Browse workload with a
        database size of 4000 experiments.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users browsing through
        experiments that can be supported by CaArray through the UI.

### CaArray Browse Experiment Stress Test

-   Description: This tests concurrent users browsing through the
    details of a single experiment using the CaArray UI. The basic
    CaArray Browse Experiment workload is exercised with a database size
    of 4000 experiments (maximum) by 10 concurrent users. The number of
    concurrent users is increased progressively until the response time
    becomes unacceptably high.

-   Steps:

    -   10 concurrent users run the CaArray Browse Experiment workload
        with a database size of 4000 experiments.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent browsing through the details of
        an experiment that can be supported by CaArray through the UI.

### CaArray Heterogeneous Workload Stress Test

-   Description: This tests concurrent users running a heterogeneous
    workload using the CaArray UI and API. Each basic CaArray workload
    is exercised with the largest load possible by 1 user, and all
    workloads are exercised concurrently. The number of concurrent users
    is increased progressively until the response time becomes
    unacceptably high.

-   Steps:

    -   The basic CaArray workloads are run concurrently with the
        largest load by 1 user each. In other words, 1 user runs the
        CaArray Upload/Import workload, 1 user runs the CaArray Download
        workload, 1 user runs the CaArray Search workload, 1 user runs
        the CaArray API Download workload, 1 user runs the CaArray API
        Search workload, 1 user runs the CaArray Browse workload, and 1
        user runs the CaArray Browse Experiment workload.

    -   The above step is repeated with a progressively-increasing
        number of concurrent users until the response time becomes
        unacceptably high.

-   Expected Results

    -   The maximum number of concurrent users executing a heterogeneous
        workload that can be supported by CaArray.