# Presentation: 709356

## ICR Working Groups Overview

**Baris Suzek**
**Protein Information Resource (PIR)**
**Georgetown University – Lombardi Comprehensive Cancer Center**
- _[bes23@georgetown.edu](mailto:bes23@georgetown.edu)_
**August 7, 2007**

## Working Groups

**ICR Workspace Working Groups:**
**Workflow**
- Implement workflows to bring analytical and data services together to address scientific use cases
**High Throughput Data Transfer**
- Address issues regarding large data set exchanges
**Analytical Services Best Practices**
- Provide best practices for development of new analytical services
**Related Architecture Workspace Working Group:**
**caGrid/TeraGrid **
- Provide access to HPC resources of TeraGrid for computationally intensive tasks

## Big picture

- CQL

- caArray

- MAGE

- caBioconductor

- MAGE

- normalize

- MicroarrayTranslator

- mageToMicroarraySet

- mageToStatML

- MicroarraySet

- StatML

- MicroarraySet

- StatML

- query

- geWorkbench

- GenePattern

- cluster

- cluster

- Cluster

- Cluster

- HierarchicalCluster

- HierarchicalCluster

- ClusterTranslator

- clusterToTree

- hClusterToTree

- cdt

- gtr

- atr

- TreeViewer

- mageToMicroarraySet

- mageToStatML

**caGrid/TeraGrid Working Group** -Provide access to TeraGrid HPC facilities to support analysis

**caGrid/TeraGrid Working Group -**will provide access to TeraGrid HPC facilities to support analysis

**HTP Data Transfer Working Group** -Address exchange of large objects generated by analytical services

**HTP Data Transfer Working Group** -Address exchange of large objects generated by analytical services

**Analytical Services Best Practices Working Group- **Provide best practices for developing analytical services and requirements for additional functionality/tools to support analytical service development

**HTP Data Transfer Working Group** -Address exchange of large objects generated by data services

**Workflow Working Group- **Identify workflow authoring tools, provide requirements for workflow tools/standards, implement scientific workflows and document experience/issues

## Goal

**Provide details on activities of WG – What was done?**
**Present Architecture/VCDE related issues WGs experiencing – What can’t be done or is hard to do?**
**Lead discussions and provide recommendations on how to address the issues – How Architecture/VCDE WSs can help?**

## Agenda

**Workflow WG – ****Kiran Keshav**, **Patrick McConnell**
**High Throughput Data – Shannon Hastings**
**Analytical Services Best Practices – Baris Suzek**

## ICR Workflow Working Group

***Working Group Activities***
- Kiran Keshav, Columbia University
- Patrick McConnell, Duke Comprehensive Cancer Center

## Agenda

**Background**
- Goals, workflows
**Issues**
- Bottlenecks, authoring, datatypes, submission
**Activities**
- Where we’ve been, where we’re going
**Discussion**

## Group Participants

**Kiran Keshav (WG lead)**
**Curtis Lockshin (facilitator)**
**Patrick McConnell**
**Ravi Madduri**
**Baris Suzek**
**Martin Morgan**
**Ted Liefeld**
**Aris Floratos**
**Denise Warzel**
**Brian Davis**
**Subhashree Madhavan**
**Juli Klemm**

## BackgroundWhat Are We Trying to Accomplish?

**Execute grid analytical and data services to accomplish a pre-defined task.**
**Authoring of workflows should be ****easy****.**
**Submission of workflows should be ****easy****.**

## BackgroundWhat Does This Look Like?

- This includes run A, get results from A, input results from A into B, get results from B, etc.

## Background... What Does This Look Like?

- This BPEL workflow could also be parallel depending on the problem:
**	**

## IssuesCurrent Bottlenecks

**Authoring Workflows**
- Who is authoring workflows?
- What tools are they using?  Are the tools graphical?
- What workflow features are supported?
**Data Type Compatibility**
- For workflows to “work”, the output of one service needs to be compatible with the input of another service
- Semantic equivalence is required
- Syntactic equivalence is required in lieu of a transformation
- We should not have to write transformers for every step of a workflow

## IssuesAuthoring Workflows

**A “simple” workflow looks like**
- <process name="Simple" suppressJoinFailure="yes" targetNamespace="http://Simple" xmlns="http://schemas.xmlsoap.org/ws/2003/03/business-process/"  xmlns:simple="http://workflow.cagrid.nci.nih.gov/SampleService1" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
-    <partnerLinks>
-       <partnerLink myRole="sample1" name="Sample1PartnerLinkType" 	partnerLinkType="simple:Sample1PartnerLinkType" partnerRole="sample1"/>
-    </partnerLinks>
-    <variables>
-       <variable messageType="simple:invokeInputMessage" name="invokeInputMessage"/>
-       <variable messageType="simple:invokeOutputMessage" name="invokeOutputMessage"/>
-    </variables>
-    <sequence>
-       <receive createInstance="yes" operation="invoke" partnerLink="Sample1PartnerLinkType" portType="simple:SampleService1PortType" variable="invokeInputMessage"/>
-       <invoke inputVariable="invokeInputMessage" operation="invoke" outputVariable="invokeOutputMessage" 	partnerLink="Sample1PartnerLinkType" portType="simple:SampleService1PortType"/>
-      <reply operation="invoke" partnerLink="Sample1PartnerLinkType" portType="simple:SampleService1PortType" variable="invokeOutputMessage"/>
-   </sequence>
- </process>

## Issues... Authoring Workflows

**Need an easy way to author workflows.**
**Tools exist, but are they easy to use?**
- Who are the actors (Wetlab Researcher/Bioinformatician)?
- Active BPEL Designer
- Can we abstract the workflow creation from the researcher with a simple GUI?
- Introduce hooks to workflow creation?

## IssuesData Type Compatibility

**To invoke multiple analytical services, ****data types must be compatible. **
**This will require groups to communicate ****with other groups up front.  Reference ****implementers last year found this requires ****lots of effort outside of the VCDE ****infrastructure.**

## IssuesWorkflow Submission

**API is easy to use.**
**API looks like:**
_**Send BPEL Document**_
- url = "https://cagrid-workflow.nci.nih.gov:8443/wsrf/services/cagrid/WorkflowFactoryService";
- fileName = "Simple.bpel";
- WorkflowFactoryServiceClient client = **new** WorkflowFactoryServiceClient(url);
- WMSInputType input = *createInput*(fileName);
- WMSOutputType output = client.createWorkflow(input);
- EndpointReferenceType epr = output.getWorkflowEPR();
- writer = **new** FileWriter("workflow_" + input.getWorkflowName() + "_epr");
- writer.write(ObjectSerializer.*toString*(epr,**new** QName("", "WMS_EPR")));
_**Add Arguments**_
- WorkflowServiceImplClient wclient = **new** WorkflowServiceImplClient(epr);
- StartInputType startInput = **new** StartInputType();
- WorkflowInputType inputArgs = **new** WorkflowInputType();
- FileInputStream in = **new** FileInputStream("input.xml");
- Element e2 = XmlUtils.*newDocument*(in).getDocumentElement();
- MessageElement anyContent = AnyHelper.*toAny*(**new** MessageElement(e2));
- inputArgs.set_any(**new** MessageElement[] { anyContent });
- startInput.setInputArgs(inputArgs);
_**Start Workflow**_
- WorkflowStatusType workflowStatusType = wclient.start(startInput);

## ActivitiesWhat We’ve Done

**Reviewed existing workflow ****authoring tools (see website).**
**Discussed what features from ****each tool were necessary vs. ****desired.**
**Prioritization of features.**

## Activities... What We’ve Done

**Taverna is the defacto standard bioinformatics authoring tool**
**Scufl vs. BPEL:**
- _[http://www.mygrid.org.uk/wiki/Mygrid/TavernaFaq#Why_does_Taverna_use_scufl_and_n](http://www.mygrid.org.uk/wiki/Mygrid/TavernaFaq#Why_does_Taverna_use_scufl_and_n)_
**Does Taverna have any interest in BPEL?**
**Should we have interest in Scufl?  Do we need to write a Scufl – BPEL converter?**

## ActivitiesWhere We’re Going

**Pass feature list to development team**
- See if the caGrid team agrees with these features.
- Is it feasible based on time constraints.  What has to be built?
**Review potential scientific workflows ****in caBIG.**
- What services exist now that can be reviewed.

## ActivitiesAdministrative

**Meeting 4****th**** Friday of every month**
- Next Meeting: August 24 @ 3:00pm
**Website: **_**[http://gforge.nci.nih.gov/projects/workflow/](http://gforge.nci.nih.gov/projects/workflow/)**__** **_
**Contact**
- Email: _[keshav@c2b2.columbia.edu](mailto:keshav@c2b2.columbia.edu)_
- Phone: 212-851-4664

## Discussion

**Authoring workflows**
- Workflow authoring tools – level of abstraction
- Who is authoring tools? Wetlab researchers?  Bioinformaticians?
**Service datatypes in workflows**
- Inputs/outputs semantic and syntactic equivalence
**What about thinking about workflows upfront, as opposed to at the end?**
- “Sketching out” workflows in the beginning will force compatibility with data types from other services.
**What about making this a requirement for Silver or Gold Level Compatibility as opposed to a feature?**
**Scufl verse BPEL**

## ICRHTP-WGUpdate

***August 6-8******th******, 2007***

**ICR****HTP-WG**Update

- Shannon hastings(_[hastings@bmi.osu.edu](mailto:hastings@bmi.osu.edu)_) - Ohio State University

## Team

- Martin Morgan
- Curtis LockshinPatrick McConnell
- Shannon Hastings
- Kiran Keshav
- Rakesh Nagarajan
- Michael Keller

## Charter

**“****This WG aims to design and implement solutions to overcome performance issues related to accessing high-throughput (HTP) genomic, proteomic and translational datasets via caGrid services.”**

## Plan

- Document use cases driven from ICR for leveraging High Throughput Data Transportation.
- Evaluate current caGrid Bulk Data Transportation technologies such as WS-Enumeration and GridFTP support.
- Define gap analysis and proposed road map and/or additions to help cover described use cases.

## Use Cases: Data

- CGEMS
**Potentially 1000’s of small records**
- Microarrays
**Expression arrays: 10’s of arrays, ‘.CEL’ files or MAGE-ML. Total 100’s of MB**
**SNP Biomarkers: 1000’s of arrays, >500k markers, GBs total data**
- Proteomics
**Very large files, 10’s of replicates**

## Use Cases: Service Use Scenarios

- Data movement
**Simple client request to analytic service**
**Client mediates interaction between data and analytic service**
- Avoid as much as possible discussion of complicated work flows

## Evaluations

- Group is developing instances of sample usage scenarios leveraging caGrid 1.0 technologies for high throughput data transfer.
- Currently 2 scenarios are underway and preliminary feedback has been received.

## Evaluations

**Utilization of caGrid WS-Enumeration support (Patrick McConnell)**
- Reported findings are as expected.  Enumeration will help to reduce/balance the load on the client by allowing the client to pull the data as needed.  Also speedup can be achieved by server asynchronously staging the data while client is pulling it.

## Evaluations

**Utilization of 3****rd**** party delivery using caGrid Bulk Data Transfer with gridFTP (Martin Morgan)**
- Still in progress.  Please check the HTP project on gforge to read about results.
- Found the lack of metadata in the BDTService to be somewhat cumbersome to know what to expect from the service if you only have the EPR to the results.
- Lack of caGrid GridFTP client implementation may lead to duplicated effort / incompatible services

## Future Plans

**Group will be finalizing trials and gathering information this month and plan to have a report out at the next ICR face to face.**
**The group will be delivering the feedback to the architecture group soon after and work with them to suggest/derive solutions or modifications where necessary to help better support the identified needs.**

## Analytical Services Best Practices Working Group

**Baris Suzek**
**Protein Information Resource (PIR)**
**Georgetown University – Lombardi Comprehensive Cancer Center**** **
- _[bes23@georgetown.edu](mailto:bes23@georgetown.edu)_
**August 7, 2007**

## Team

- Ted Liefeld (Lead)
- Kiran Keshav
- Patrick McConnell
- Martin Morgan
- Salvatore Mungal
- Rakesh Nagarajan
- Baris Suzek (VCDE Liaison)
- Juli Klemm
- Curtis Lockshin
- Denise Warzel
- Claire Wolfe

## Agenda

**Issues**
- Mismatch between XML schema and information models
- Reannotation/reregistration of reused classes as an impeding factor for development
- Insufficient documentation for analytical service development
**Background **
- Progress so far
- Issues as they have materialized
**Proposal to Cross-cutting Workspaces **
**Discussion/Crosscutting WS Implications**
**Next steps**** **

## Progress so far	

- Setting mission and objectives for the working group
- Identification and discussion on interoperability issue between caArray and GenePattern (Issue 1)
- Documentation of a demo analytical service development:
- Determining the function, inputs and outputs
- MAGE-OM ->log transform -> MAGE-OM
- Establishment of a task force and task allocation
- Modeling interfaces and parameter classes
- Trying to reuse/reannotate existing classes from MAGE-OM  (Issue 2)

## Issue 1: Mismatch between XML schema and information models

- Problem found when GenePattern tried to use MAGE-OM format outputted by caArray
- GenePattern and caArray use different XML schemas so generate different beans:
- caArray (gov.nih.nci.mageom.domain.xsd)
- GenePattern (bioAssay.xml)

## Issue 1: Mismatch between XML schema and information models

- XML schema generation process is error-prone:
- Import portion/full model into EA
**Generate schema from EA**
**Fix/handcraft schema using XML visualization tool**
- Load schema to GME
- Import schema into Introduce
- XML schemas should be shared/reused to facilitate interoperability

## Issue 2: Reannotation/reregistration of reused classes as an impeding factor for development

- Analytical services typically reuse the classes generated by data services
- Process involved reannotation and reregistration of reused classes:
- Model the interface/parameter classes for the service
- Include all the reused classes/packages to the model
**Annotate model (including reused classes) using SIW **
- Load annotated model to caDSR, ensuring mapped correctly based on annotations

## Issue 2: Reannotation/reregistration of reused classes as an impeding factor for development

- Right tools/processes might not have been available/accessible at the time of development e.g. SIW Roundtrip, Analytical Services Loader, refactored SIW process to eliminate fix-ea task
- Reannotation/reregistration of reused classes is time-consuming and impedes analytical service development
- Can SIW’s improved roundtrip function help ? Currently not. Issues include:
- Missing annotations in classes without attributes (will be fixed in version 3.2.1.1)
- CDE mappings for inherited attributes (will be fixed in version 3.2.1.1)
- Matching value domains to existing caDSR value domains (not critical but will be fixed in future releases)

## Issue 3: Insufficient documentation for analytical service development

- Most of the of the documentation is data services oriented
- More analytical services oriented documentation is required

## Proposal to Cross-cutting WorkspacesProposed Solutions to Issue 1

- Provide guidelines/SOP to analytical service developers to  import/reuse XSDs:
- GME Client API
- GME View
- Introduce
- Include XSD in compatibility submission package and store at Gforge site – probably a requirement for Gold Compatibility review

## Proposal to Cross-cutting WorkspacesProposed Solutions to Issue 2

- The beta version of the new tool called “Service Loader” will be used/tested
- “Service Loader” designed to be a lightweight “registration” system and works with existing registered classes, so the process will be:
- Service-unique classes will be modeled/annotated and loaded to caDSR
- Introduce toolkit will be used to create the Analytical Service
**serviceMetadata.xml generated by Introduce will be sent to NCI and loaded to caDSR**
- The feature requests or bugs identified during development will be reported

## Proposal to Cross-cutting WorkspacesProposed Solutions to Issue 3

- Analytical Services Best Practices working group will document the process of developing an analytical service using the most recent toolset
- Documentation on tools such as “Service Loader” should become available soon

## Discussion/Crosscutting WS Implications

- Mentor/reviewer training to help analytical service developers – new tools/processes, emphasis on class/model reuse
- Reflecting the significance of XML schemas in compatibility checklists for gold or grid-enabled silver level systems
- Developing/designing procedures/tools to ensure consistency of GME, Index services and caDSR registrations

## Next Steps

- Demo on “Service Loader” to working group
- Continuing developing the demo analytical service
- Evaluating approaches for semantic annotation of analytical services for effective discovery and advertisement – default, proposed attributes, myGrid/cancerGrid
- Providing use cases to Architecture/VCDE teams to improve existing tools/processes