**Manage Experiment Data Files Use Case Specification\
caArray**

  ---------------------------------------- ------------------------------
  Last Revised:                            February 02, 2009

  Produced By:                             NCICB Development

  Version:                                 1.7
  ---------------------------------------- ------------------------------

Document Approvals

The list contains the name and contact information for the core project
team and any key stakeholders who have an interest in the success of the
project. An "S" identifies persons responsible for approval from the
stakeholder groups. Sign off of the document would be required when a
decision is made not to take action for defined gaps.

  --- ---------------- ---------------- --- ---------------- ----------------
  S   Name             Role             S   Name             Role

  S   Anand Basu       NCICB            S   Juli Klemm,      Product Manager
                       Engineering          Ph.D.            
                       Manager                               

      Brent Gendleman  Development          George           CBIIT
                       Project Manager      Komatsoulis,     Independent QA
                                            Ph.D.            

      Eric Tavela      Development          Don Swan         NCICB
                       Architect                             Application
                                                             Support

                                                             
  --- ---------------- ---------------- --- ---------------- ----------------

Revision History

When you make a change to a document, you must add an entry to this
Revision History table and you must manually type the Last Revised Date
on the front cover.

  --------- --------- --------------------------------------- ---------------
  Date      Version   Description                             Revised by

  6/18/07   0.1       Initial Draft -- Basic Flow             Brent Gendleman

  6/22/07   0.2       Split original use case up into Manage  Rashmi
                      Project Data, Validate Files and Import Srinivasa
                      Project Data. Fleshed out details of    
                      each sub-component.                     

  6/26/07   0.3       Updated to reflect use case analysis of Brent Gendleman
                      all 3 uses cases mentioned above and    
                      include "upload" as part of the basic   
                      flow                                    

  6/30/07   0.4       Changed file name to be consistent with Brent Gendleman
                      use case analysis                       

  7/09/07   0.5       Updated Special Requirements based on   Rashmi
                      group review.                           Srinivasa

  8/3/07    0.6       Revised based on community feedback     Brent Gendleman

  9/21/07   0.7       Modified Title to reflect Experiment    Brent Gendleman
                      focus                                   

  10/2/07   0.8       Updated with additional alternate flow  Brent Gendleman
                      to support the uploading of .zip files  
                      and presentation of extracted files for 
                      management                              

  11/8/07   0.9       Updated Pre-conditions, modified Basic  Brent Gendleman
                      Flow, Change File Alt Flow, still need  
                      to add available file types             

  1/4/08    1.0       Updated per Bill Mason peer review,     Brent Gendleman
                      modified basic flow, added messaging    

  2/22/08   1.1       Updated for 2.0.1, reflecting defects   Rashmi
                      11884 and 10457 and feature request     Srinivasa
                      12344. Also cleaned up invalid          
                      hyperlinks etc.                         

  3/11/08   1.2       Updated for 2.0.1, reflecting           Rashmi
                      background upload.                      Srinivasa

  6/27/08   1.3       Updated for 2.1.0, reflecting deletion  Rashmi
                      of imported files and invalid zip       Srinivasa
                      formats.                                

  7/1/08    1.4       Changing file type in bulk.             Rashmi
                                                              Srinivasa

  9/22/08   1.5       Added file-filtering requirement.       Rashmi
                                                              Srinivasa

  1/2/09    1.6       Updated supported data file types for   Rashmi
                      2.2.0.                                  Srinivasa

  2/2/09    1.7       Added the "Select Referenced Files"     Rashmi
                      alternative flow.                       Srinivasa
  --------- --------- --------------------------------------- ---------------

Copyrights and Trademarks

© Copyright 2007 by NCICB, caBIG^TM^. All rights reserved.

Table of Contents

[**1.** **Brief Description 1**](#brief-description)

[1.1. References 1](#references)

[2. Deferred from 2.0 1](#deferred-from-2.0)

[2.1. Affymetrix .EXP files 1](#affymetrix-.exp-files)

[2.2. Real Time Progress Meter 2](#real-time-progress-meter)

[2.3. Resuming Cancelled or Interrupted Uploads
2](#resuming-cancelled-or-interrupted-uploads)

[3. Preconditions 2](#preconditions)

[3.1. Data Owner 2](#data-owner)

[3.2. An Available Experiment Exists in a Saved State
2](#an-available-experiment-exists-in-a-saved-state)

[3.3. The Experiment Service and Array Types are Known
2](#the-experiment-service-and-array-types-are-known)

[3.4. Compressed File Formats 2](#compressed-file-formats)

[4. Basic Flow of Events 3](#basic-flow-of-events)

[5. Alternative Flows 4](#alternative-flows)

[5.1. Upload a Compressed File 4](#upload-a-compressed-file)

[5.2. In Progress Cancellation - Deferred
4](#in-progress-cancellation---deferred)

[5.3. Delete Files 4](#delete-files)

[5.4. Change File Type 4](#change-file-type)

[5.5. Delete Imported Files 4](#delete-imported-files)

[5.6. Select Referenced Files 5](#select-referenced-files)

[6. Exception Flows 5](#exception-flows)

[6.1. Connection to Server is Lost 5](#connection-to-server-is-lost)

[6.2. Data Owner Closes Browser 5](#data-owner-closes-browser)

[6.3. Data Owner Changes Page View 5](#data-owner-changes-page-view)

[6.4. Invalid Compressed File 5](#invalid-compressed-file)

[7. Post-conditions 6](#post-conditions)

[7.1. Uploaded Data Files are Saved 6](#uploaded-data-files-are-saved)

[8. Special Requirements 6](#special-requirements)

[8.1. Progress Meter 6](#progress-meter)

[8.2. Ability for File Upload to Continue when Navigating Elsewhere
6](#ability-for-file-upload-to-continue-when-navigating-elsewhere)

[8.3. Inability to import an Array Design within an Experiment
6](#inability-to-import-an-array-design-within-an-experiment)

[8.4. Expected File Types 6](#expected-file-types)

[8.5. User Messaging 7](#user-messaging)

[8.5.1. Upon Upload 7](#upon-upload)

[8.5.2. File Already Exists 7](#file-already-exists)

# Brief Description

Initiated by a Data Owner, this use case enables the uploading of
annotation and array content collectively and independently into a
caArray project and then provides the ability to validate, import and/or
delete the uploaded files and records each action taken. Due to the
large file size of array data and to a lesser extent -- the annotation,
the upload may take minutes or even hours to complete; so the ability to
offer to run the upload in the background (allowing the user to perform
other functions inside caArray) is desirable.

![](media/image2.wmf){width="4.467361111111111in" height="3.175in"}

Figure 1. Data Owner Actor Depiction

## References

caArray Vision -- functional requirements

caArray Supplementary Specifications -- non-functional requirements

caArray Use Case Summary -- actor definitions and context

# Deferred from 2.0

## Affymetrix .EXP files

It has been suggested that Affymetrix .EXP files would support quality
control metrics across a given project. This is deferred from 2.0 but we
should revisit shortly to support improved quality control data and
workflow for biostatisticians.

## Real Time Progress Meter

The ability to see the status of files (states: uploading, validating,
importing) are deferred. Several off-the-shelf open source options exist
for supporting this requirement but are dependent either on applets or
Flash 8. We will revisit to determine if we perform a browser check and
if Flash 8 is installed provide this additional feature.

## Resuming Cancelled or Interrupted Uploads

For 2.0, there is NO expectation that the ability to resume will be
available. This is primarily to remain consistent with the no applet and
open source supplemental requirement.

# Preconditions

## Data Owner

A Data Owner must be logged in before he/she can manage experiment data.
A Data Owner is an experiment submitter from the Principal
Investigator's lab (typically a Research Scientist), or an experiment
executor from the core facility or lab (a Lab Scientist or
Biostatistician).

## An Available Experiment Exists in a Saved State

At least one experiment owned by the contributor is available to have
data uploaded, validated and imported into it. If no experiments exist,
the feature will be unavailable to the user.

## The Experiment Service and Array Types are Known

The service type (complete, partial, analysis or publish) and the assay
type (e.g., gene expression vs. SNP) is known so the system can expect
what data types it will receive.

## Compressed File Formats

The ability to upload a .zip file is available in 2.0. Additional
compression formats may be added with the same behavior expected in
later releases.

# Basic Flow of Events

#### The Data Owner navigates to and selects the experiment of interest from their experiment workspace in the System that contains no uploaded data. 

#### The System opens the selected experiment and indicates the ability to upload data to the project. 

#### The Data Owner chooses to upload data files.

#### The System opens a new window (hereafter referred to as the "Upload window") in which it presents a dialogue that allows for a single or multiple files to be selected from a local or networked location without restriction on the size or type (file extension independent) of files.

#### The Data Owner selects the single file of interest or a compressed file (see [Alternative Flow: Upload a Compressed File](#upload-a-compressed-file)).

#### The System presents the ability to Upload the file or add an additional file.

#### The Data Owner chooses to add an additional file and can repeat this action as often as desired (Steps 5 and 6) and chooses to upload.

#### The System [messages](#upon-upload) the user that the upload progress has begun, and warns the user that the upload process will fail of the Upload window is closed prematurely. The System gives the Data Owner the ability to navigate through the experiment in the old window.

#### The System indicates the progress of the upload in the Upload window. See Special Requirements: Progress Meter.

#### The System indicates to the Data Owner when all files have made it and displays the list of files, the file extension, the type of file (if recognized by the System -- see [Special Requirements: Expected File Types](#expected-file-types)) and their status (Uploaded), and the ability to:

#### Navigate Experiment

#### Upload Additional Files -- This returns the Data Owner to Step 3 of the Basic Flow.

#### Select or deselect any one or all of the files to support the following actions:  Delete Files -- see [Alternative Flow: Delete Files](#delete-files)  Change File Type -- see Change File Type  Validate -- see [Validate Data Files Use Case](https://gforge.nci.nih.gov/svnroot/caarray2/trunk/docs/requirements/use_cases/validate_data_files_use_case_specification.doc)  Import -- see [Import Experiment Data Use Case](https://gforge.nci.nih.gov/svnroot/caarray2/trunk/docs/requirements/use_cases/import_experiment_data_use_case_specification.doc)

#### Filter the view of the files by file type and/or status.

#### The Data Owner selects to navigate the experiment, choosing any aspect of the experiment displayed.

# Alternative Flows

## Upload a Compressed File

#### In Step 5 of the Basic Flow, the Data Owner chooses a file that has been zipped (.zip) for upload among the files selected for upload.

#### The flow continues through Step 10 of the Basic Flow.

#### At Step 10, however, the System extracts the compressed file, discards the original and presents each of the files that were contained in the compressed file to the user, indicating that each one has been uploaded and the actions indicated in Step 10 of the Basic Flow are available on these distinct files.

## In Progress Cancellation - Deferred

#### From Step 8 in the Basic Flow, the Data Owner chooses to cancel the upload job. 

#### The System halts the uploading of the file or set of files and presents the Data Owner the list of files that were uploaded, and moves the Data Owner forward to Step 10 in the Basic Flow.

## Delete Files 

#### From Step 10c in the Basic Flow, the Data Owner chooses the file or files to be deleted and selects "delete."

#### The System physically deletes the selected file or files and returns the Data Owner to Step 10 in the Basic Flow without displaying the deleted files. 

## Change File Type

#### From Step 10c in the Basic Flow, the Data Owner chooses all files to be changed to a particular type, and selects "change file type."

#### The System presents the appropriate drop down set for a given experiment or to cancel the action.

#### The Data Owner chooses the file type that is appropriate for the selected file and saves.

#### The System changes the file type for all selected files and returns the Data Owner to Step 10 in the Basic Flow with updated file types.

## Delete Imported Files

#### The Data Owner navigates to the Imported Data Files view.

#### The Data Owner chooses the file or files to be deleted and selects "delete." Only files that are not associated with hybridizations can be selected for delete. (E.g., MAGE-TAB idf and sdrf files can be selected, but data files cannot.)

#### The System physically deletes the selected file or files and returns the Data Owner to the Imported Data Files view which does not display the deleted files. 

## Select Referenced Files 

#### After Step 10b in the Basic Flow, the Data Owner chooses a MAGE-TAB IDF file and selects "Select Referenced Files".

#### The System selects the SDRF file(s) referenced by the IDF, and all data files referenced by each SDRF.

#### The System allows all the actions listed in step 10c of the Basic Flow.

#### The Data Owner proceeds as in Step 10c of the Basic Flow, choosing to Validate or Import the selected files as a set. 

# Exception Flows

## Connection to Server is Lost

If at any point prior to the entirety of the file or set of files chosen
to be uploaded the connection is lost, the data will not be uploaded and
must be done over. See Alternative Flow: In Progress Cancellation -
Deferred.

## Data Owner Closes Browser

If the Data Owner chooses to or inadvertently closes the Upload window,
the data won't be uploaded. See Alternative Flow: In Progress
Cancellation - Deferred.

## Data Owner Changes Page View

If the Data Owner browses elsewhere in the Upload window, the data will
not be uploaded. See Alternative Flow: In Progress Cancellation -
Deferred.

## Invalid Compressed File

Attempting to upload a corrupt or invalid zip file in Alternative Flow:
Upload a Compressed File will result in an upload failure and the System
will present an error message indicating the same. A zip containing a
directory is considered invalid.

# Post-conditions

## Uploaded Data Files are Saved

After a successful execution of this use case, any completely uploaded
files are available in caArray with a status of "Uploaded", and are
available for validation and import. Any deleted files are completely
removed from caArray.

# Special Requirements

## Progress Meter

The ability to see the total number of files selected for upload and the
status of each file is provided.

The status indicator also indicates the percentage of the upload process
complete.

## Ability for File Upload to Continue when Navigating Elsewhere

The upload process should occur in the background in a new window,
allowing the user to navigate elsewhere in the application while uploads
are occurring.

## Inability to import an Array Design within an Experiment

The Import Array Design Use Case provides the functionality for managing
Array Designs independently of an experiment.

## Expected File Types

The following types of MAGE-TAB files will be supported:

***Imported after validation and processing:***

IDF (Investigation Description Format)

SDRF (Sample and Data Relationship Format)

***Imported without validation and parsing:***

Data matrix

ADF (Array Design Format)

Raw/processed data files of the following types will be supported:

***Imported after validation and processing:***

Affymetrix CEL, CHP

GenePix GPR

Illumina CSV

***Imported without validation and parsing:***

Affymetrix DAT, RPT, TXT, and EXP

Agilent TSV, TXT

Illumina IDAT, TXT

ImaGene TIF, TXT

Nimblegen GFF, TXT

ScanArray CSV

GEO SOFT and GSM

Generic files like Word documents, Excel spreadsheets, Powerpoint slides
and PDFs are capable of being uploaded and imported though the only
validation is that they are present. These files are not associated with
any sample(s), but are uploaded at the Experiment level. They will be
found under the Supplemental Documents tab for the experiment.

## User Messaging

### Upon Upload

*Your file(s) are currently being uploaded. Depending on the speed of
your connection and the size of the file or set of files, this may take
significant time while the data is transferred.*

### File Already Exists

For every file that is uploaded that already exists in the manage file
data set or that has been previously imported into the experiment, the
system will produce the following message:

*\<\<file name\>\> was not uploaded because a file with the same name
already exists in this experiment.\
\# of files uploaded* (takes into account how many were requested minus
the ones which didn't)