# Presentation: 723747

## Bulk Data Transfer Issues in Development of caGrid Analytical Services

- Jahangheer Shaik
- Washington University School of Medicine

## Analytical Services at WashU

## Unsupported Associations

- Assumption: Object graph when passed as method parameters by client can be marshaled, transported to the server and reconstructed to obtain original object
- Observation: Not all associations are supported. Only one-to-one and one-to-many associations are supported.
- Bi-directional, cyclic,self loop, many-to-one and many-to-many associations are not supported
- Bi-directional/cyclic: StackOverFlowException
- Many-to-one: Not all associations are passed to server

## Why do we have unsupported Associations?

- Web-services depend on SOAP to form associations in XML form.
- XML in its native form can support one-to-one and one-to-many
- We can control the software that makes use of XML and XSD to accomplish what we need. Needs change in XSDs and XML

## Convert Graphs to trees

## Modifications to XML and XSDs

- Introduce References

- Modify XSD complex types

| 1: <a name=”A” _id_=”o1”>
2: <b name=”B” _id_=”o2”>
3: <a _idref_=”o1”/>
4: </b>
5: </b> |
| --- |

- Pseudo nodes are supported by only those objects that contain attributes with name _id_ and _idref_.

## Large Objects, Memory based restrictions

- Gene Pattern Example: Approximately 0.1 million ProbeCopyNumber objects are present in object CopyNumberOutputForSample
- Assuming there are 20 samples in an average experiment, we need to handle 2 million objects
- Assuming each object is 256 bytes, 512MB RAM is required per session. If there are 10 such concurrent sessions, we need 5GB RAM space alone. Not to mention Operating system, application server, analytical tool which demand extra RAM

## Serialization and De-serialization of Large Objects

- Serialization/De-serialization of 0.3 million objects succeeded but for 0.5 million objects, the process fails
- Introduce 1.2 employs AXIS by default. An attempt was made using CASTOR but it failed for 0.5 million objects.

## Serializer, De-serializer

- Custom written transformer loads entire text or XML data into beans.
- 	This essentially is loading the entire input data into the memory.
- 	For XML data, APIs such as SAX, StAX or DOM are required to read the input data.
- Invoke the web method on the service that passes created beans as method parameters.
- Beans are serialized automatically to XML and will be part of method invocation on analytical service.
- 	All the limitations of default serialization will apply
- Method implementation on analytical service receives de-serialized beans as parameters automatically.
- 	Entire data is present in memory as beans instances
- A transformer is then required to convert beans into a text file presented as input to the analytical tool.

## Serializer, De-serializer

- Use JAXP's StAX API for writing data objects directly in XML.
- 	Read input text files using Java Buffer-Readers which use String.split with Regular expression
- 	support for reading input line by line.
- 	For XML use SAX or StAX but NOT DOM as it tends to load the entire data into the memory
-     If there is one-to-one correspondence between input data line and output XML-element, then this conversion
-     does NOT require more than a megabyte of memory irrespective of the input size. If direct conversion is not possible,
-      try to develop a logic that requires one or two passes on input data. The first pass populates essential set and
-      maps where as the second pass performs direct conversion using earlier populated set and maps.
- The output XML is stored in a Zip file and passed to service using putter methods and caGridTransfer service.
- Similar to an analytical-tool, wrapped by analytical-service that takes input in the text format,
-    a custom converter is written to convert received XML file into a text file.
-   In service implementation, typically the XML is read using StAX API.
-   Therefore, loading the entire data into memory is avoided. This makes handling of large objects easier.

## Data exists in memory in compressed format

| fileZip = new File(“input.xml.zip”);
fis = new FileInputStream(fileZip);
bis = new BufferedInputStream(fis);
zis = new ZipInputStream(bis);
ze = zis.getNextEntry(); |
| --- |

- Reading XML content without uncompressing

| fos = new FileOutputStream(fileZipOutput);
zos = new ZipOutputStream(fos);
ZipEntry ze = new ZipEntry(“filename.xml”);
zos.putNextEntry(ze);
XMLOutputFactory xof = XMLOutputFactory.newInstance();
xstWr = xof.createXMLStreamWriter(zos); |
| --- |

- Writing XML content into compressed file

## Java Analytical Service Container JASC

- Analytical Service= IntroduceGeneratedSkeleton+ AnalyzerBean+ JASC Container
- JASC is a pseudo container for hosting Analyzer beans
- JASC separates non-data processing tasks from Analyzer-bean
- Integration of JASC with introduce generated skeleton is performed by developer

## JASC

- Tomcat Web server

- Globus

- Properties file

- JASC loads properties file and makes it available to the bean.
- If two analytical services are to be developed, two analyzer beans are created and separately deployed
- When TomCat is started, JASC will search for each properties file for each analyzer-bean
- This provides better session-management

## Common Non-Functional Requirementsaddressed by JASC

- Automatic Multi-session (one session one client)
- Developers develops service assuming one client existence
- Working directory for each session (easy to invoke analytical-tool)
- Compressed (Zip) XML – input and output
- Low bandwidth requirement, faster transmission
- Less temporary storage required for concurrent sessions
- Automatic Exception handling; also when no method-call is in progress, but analytical-tool fails mid-way
- Auto Tracking of analysis progress, and informing client
- Auto Resource clean-up (remove analytical-tool’s temporary files, working directory, and objects)

- 

## Advantages for Service Developers

**No change in WSDL or XSD**
- No need to use context resource properties – instead class instance variables may be used
- Program simple Java class (no need to learn Globus/Introduce API)
- No need to handle Exceptions by programmer
- Deferred exception delivery (as client may disconnect after invoking analysis)
- Exception Confinement, with no impact on other sessions
- If one session fails due to excepted or unexpected exceptions
- Out-of-memory faults experienced by a session
- Trapping of Illegal calls from client (like request for output if analysis is in progress)

## Advantages for Service Developers – 2

- No need to implement for multiple client or concurrency – developer implements for only one client
- Starvation Prevention (responsive application, thread priorities NOT managed by developer)
- Lesser maintenance due to non-redundant code
- Prevents state of sessions being mixed (like: output from one session wrongly delivered to another session)
- Automatic type checking on received XML content via caGrid-Transfer
- Promotes interoperability by monitoring XML content in and out of service – should be WSDL & XSD compliant
- Easily returns multiple output files after analysis (using caGrid-Transfer)
- No test code written by developers for non-functional concerns described above

## Execution Environment

- Analyzer Bean
- Java class implementing Analyzer interface

**Instance****Variables**

- Separate Thread of Execution

- Service Specific Deployment Properties

- Service Name

- Analysis Progress Status

- Exception History

- Shared data in Hash Maps for:
- Input Objects
- Input Files (XSD type Verified)
- Output Objects
- Output Files (XSD type Verified)

*Available due to:*
***“******One Client - One Analyzer Bean Instance” policy***
*(Introduce-1.2 by default does NOT support this)*

***Access to Environment is provided by JASC’s Session***