**ICR Face-to-Face Meeting #4**

Broad Institute, MIT, Cambridge, MA

September 18-19, 2006

*Meeting Notes*

**Summary of Key Issues Raised/Discussed:**

-   In caBIG Year 1 and Year 2 we focused on learning how to create
    metadata. We are now trying to shift our focus to the re-use of
    metatdata to forward the sematic interoperability goals of caBIG.

-   An assessment of "usability" is not part of the caBIG Silver
    compatibility review. However, this needs to be an area of focus for
    caBIG and part of the evaluation criteria for projects' continuation
    funding.

-   In the coming months, caBIG will be transitioning from the Pilot
    phase to the long-term Program. This will involve engagement of
    additional communities, stronger focus on adopters, stronger support
    infrastructure (among other changes) but the overall structure of
    the program is anticipated to remain the same.

-   From the Workflow discussion, the group focused on the sharing and
    reuse of workflows and provenance tracking associated with
    workflows.

-   Strategic Plan review: The progress against the goals is mixed. Key
    areas to focus on in the near term include:

    -   Develop a methodology for gaining feedback on the usage of
        caBIG™ resources. Due 12/07

    -   Continue to propose standards for common data elements to
        promote semantic interoperability.

    -   Grid enable Year 1 ICR Projects.

    -   Query the cancer research community to identify tools they would
        like to have on the Grid. Due 10/06

-   The Gap Analysis conducted in January was reviewed. It was the
    consensus of the group that future effort should focus on
    integration, hardening and adoption of existing resources.

**Review Meeting Agenda & Goals -- Juli Klemm**

<http://gforge.nci.nih.gov/frs/download.php/940/ICR_F2F_Sept06_Agenda_Final.pdf>)

**Welcome -- Jill Mesirov**

(<http://gforge.nci.nih.gov/frs/download.php/990/cabig.091806_jill_mesirov.pdf>)

-   Jill gave an overview of the Broad Institute, its organization,
    programs, and initiatives.

-   The Broad has a computational community and has participation from
    all over the community.

-   Specifically, the Broad works heavily in cancer genomics: including
    development of analysis tools, development of algorithms to support
    computational research, and disease studies.

**Scientific Talk: Signatures of Cancer Delivery -- Todd Golub**

-   Some computational considerations for cancer molecular diagnostics

    -   Which genes have statistical significance?

    -   How do you visualize the results?

    -   Has this pattern ever been seen before?

    -   Generalizability -- Reuse the same approach so you don't have to
        re-invent the wheel each time

    -   Challenge -- scoring high-dimensionality readout

-   Overall goal: A complete molecular taxonomy of cancer molecular
    diagnostics.

-   TCGA is showing that you should be able to identify all the
    mutations that underlay cancers.

-   Todd spoke of gene expression predictors of the Revlimid response.
    The question was how do you determine which genes reach statistical
    significance. Signature amplification and detection was done by
    ligation-mediated amplification (LMA) and luminex beads,
    respectively. A streptavidin-phcoerythrin fluorescence product was
    measured.

-   Comparative gene expression profiling allows for proteomics profiles
    to be generated (many people are doing this). It allows one to
    visualize different clinical states of interest (state A vs. state
    B). But now what? What we really want to do is recognize these
    signatures if we've seen this before (i.e. computationally look up
    these signatures---have these been seen in a clinical setting?)

-   Current drug discovery process: Disease processgenomic
    signaturemolecular understandingtargetssmall molecules. You often
    get stuck in the molecular understanding and targets stage if you
    want to push this to a drug discovery process). Can we skip those
    steps? What if we start the small molecule screen by using the
    genomic signature as the readout?

-   Gene-expression-based high throughput screening (GE-HTS). You then
    get an induction of different signatures. For example, when ALS
    cells were treated with these compounds, they became like
    neutrophils and then indeed go on to die. This allows you to see
    which drugs have an effect.

-   Inhibiting androgen receptor activity in prostate cancer. Performed
    a GE-HTS screen and found celastrol and gedunin (both natural
    products with long history of medicinal use). Now you can do
    signatures in optimization mode and not identification mode. Dose
    response: as you increase celastrol, you turn it off...so it does
    what you want.

-   Use this data to create a Connectivity Map and implement as a
    searchable Signature Database, with a simple, "Google-like"
    interface

> [Question]{.underline}: What are the structural differences in ligand
> binding? Have these been characterized structurally?
>
> [Answer]{.underline}: They bind and interact differently and in
> various ways.
>
> [Question]{.underline}: For the Signature database, do you correct the
> platform differences?
>
> [Answer]{.underline}: This is version 1 and may change. We do gene
> expression profiling in different conditions and make a rank-ordered
> list by degree of differences in expression. Signatures are used to
> query databases..

**Approaches to Model Harmonization and CDE Reuse -- Dianne Reeves,
Tommie Curtis, Steve Sandberg**

(<http://gforge.nci.nih.gov/frs/download.php/939/caBIG_Model_Harmonization_Report.ppt>)

(<http://gforge.nci.nih.gov/frs/download.php/987/Reusev2forICRF2FBoston.ppt>)

Comments/Questions

-   How does process for development of the BRIDG model within CTMS fit
    with the process for development of the über-model? Will there be
    collaboration between these groups?

    -   Yes. Continue analyzing overlap with BRIDG.

-   What is the level of abstraction for the über-model?

    -   Focus is for application developers, to create a model that is
        understandable by and re-usable for them.

    -   BRIDG is an intermediate level of abstraction.

        -   Focused on protocol driven research domain

    -   Question of domain models:We are not defining a single domain
        model in caBIG™, each Workspace is defining a domain

    -   ICR is quite broad, we may want to break the domain down into
        smaller sub-sets

    -   Need to identify touch points among models, which the large
        scale harmonization group is doing

-   The question was raised as to whether creation of an über-model is
    the way to go. Or, would it be better to focus on improving the
    tooling to support harmonization and re-use. Does an über-model
    provide an unncecessary layer of complexity?

-   caDSR is mapping at the data element level, we need tools map at the
    object level

-   UML loader identifies harmonized data elements

-   The OBO (Open Biomedical Ontologies) group has combined
    bioinformatics experts with "field expertise". EVS is active in OBO
    organization.

    -   Genome ontologies

    -   Sequence ontologies

    -   RNAi ontologies

-   Need to develop systems to get work done. Although we want to get
    everything to work together, there is still the tension of what we
    need now to do our work vs. standards-based interoperable systems
    that are coming.

    -   How do we handle "legacy" systems that are being created now as
        we wait for the standards-based systems.

-   ICR is always developing new technology, we need to keep these
    systems as flexible as possible

-   Note that CDEs can only be re-used if their status is "Released" on
    the caDSR Production server. Therefore, it is important that
    developers make sure their CDEs are promoted to this status.

**Overview of Silver Compatibility Reviews: What Reviewers are Looking
for -- Sal Mungal, Bob Freimuth, Tahsin Kurc**

(<http://gforge.nci.nih.gov/frs/download.php/966/caBIG_ICR_F2F_2006-09-18_Silver_Compat_Review.ppt>)

Comments/Questions:

-   How many compatibility reviews have been performed to date?

    -   31 models on stage, approximately 12 done

-   How to handle the fact that models are being developed in parallel
    with standards being approved?

    -   May need to version apps to incorporate approved data standards

    -   Do we have set timelines when applications have to update app to
        conform to approved standards?

        -   We have not yet addressed that.

        -   However metadata is still there and usable

        -   Will also address what is expected at gold level
            compatibility

-   Once gold criteria established and formalized, issues around CDE
    re-use will come into play even more

-   Is it possible/reasonable to have 2 different APIs modeled for same
    application and same data? Course-grained client friendly API?

    -   Should be possible as long as clearly defined and documented

    -   Only the API that is being exposed and exchanging objects is the
        one that needs to be reviewed.

-   Hope to see standardization of APIs as well

    -   As caBIG™ grows, standardization on APIs is expected.

-   Assessment of usability \-- How are the tools really being used in
    the community? Is this part of the Silver Compatibility review?

    -   Outside of scope of Crosscutting Workspaces.

    -   Distinct topic from technical caBIG™ interoperability.

    -   Have not yet developed usability criteria

-   Concern was raised that specialists who use CDEs are not being asked
    to see if they are being used properly. Specialists should be
    included throughout the reviews

    -   Adopters review annotated UML models to confirm scientific
        validity

-   What is the estimate to complete a full Silver Compatibility review?

    -   5-6 weeks to complete the review

-   As more projects are coming on to the grid, how easy is it for
    people to use tools. Need to highlight successful uses of the tools
    at the annual meeting.

-   In the pilot we relied on bioinformatics groups from the cancer
    centers to engage the end-user community. Is it the consensus of
    this group that we should find other routes to engage end users at
    the cancer centers rather than going through the current caBIG™
    participants (i.e. the bioinformatics groups at the cancer centers)?

-   There is a perceived communication gap between developers and end
    users that needs to be addressed.

**NCICB Welcome -- Peter Covitz**

-   NCI sees caBIG™ as a success and thanks to all participants for
    making it that.

-   NCICB is constantly seeking constructive feedback.

Questions/Comments:

-   What are the specific steps to transition from the Pilot phase of
    caBIG™ to the Enterprise?

    -   caBIG™ is looking to expand its community and to have the caBIG
        resources used to support scientific endeavors and to promote
        collaboration

-   Will usability be part of the criteria for what's funded?

    -   Some projects will be carried forward based on the value
        contribution. Some that reached completion phase may or may not
        be actively funded through caBIG™.

    -   Usability and user acceptance are extremely important criteria.

-   What is the NCI requirement of other research when publishing data?
    This language is now showing up in grants from the NCI

    -   Data should be published to a caBIG™ compatible system.

    -   Should define a caBIG™ data repository or format or one that can
        be made caBIG™ compatible. A key challenge is that there aren't
        caBIG systems for all data types at this time.

-   How do you measure user acceptance?

    -   We need to identify relevant stakeholders and focus on their
        needs.

**Use of External Standards Lessons Learned -- Tahsin Kurc, Ted Liefeld,
Patrick McConnell**

(<http://gforge.nci.nih.gov/frs/download.php/947/icr_f2f_ext_stand_2006_09_18.ppt>)

-   The goal is to determine if external data models can be used for
    caBIG™ and identify and resolve any issues.

Questions/Comments

-   In the case of cPath, an XML RDF stream (BioPAX) is was created
    within a single large object, but to use the data the receiving
    application has to know to use (parse) the incoming document.

-   Are there criteria that others can use to compare an external
    standard and make a decision to use the blob or take the trouble to
    map it fully to a caBIG™ object model?

    -   It should be based on a use case. Adopters can make a decision
        on what will contribute to their benefit or which portion of the
        external standard is relevant to use.

-   Question to Patrick McConnell: What would you do different for the
    RProteomics project if you had options?

    -   RProteomics project was more of an analytics project (rather
        than a data service); if we had it to do over again, we may have
        modeled mzXML as a single large object (like the BioPAX case).

-   What is the nature of the following restriction: the inability to
    import classes and attributes with associations.

    -   Not a requirement of infrastructure but more of a best practice.
        The caDSR is CDE-oriented so you can register a class with no
        attributes, but you will not have any CDEs---which will not be
        useful for VCDE. Still an open issue.

-   Is it more efficient to implement the whole model?

    -   Would like to adopt entire standards. The question is about
        resource allocation. If it's important to caBIG™, a group of
        people should be assigned to it.

**Review Functionality for Workflow in caGrid 1.0 and Discuss Future
Requirements -- Patrick McConnell**

(<http://gforge.nci.nih.gov/frs/download.php/958/workflow_icr_f2f_2006_09_18.ppt>)

-   Patrick spoke about caGRID components and gave a conceptual view of
    the GRID.

-   Review and demo of caGRID 1.0 workflow implementation

-   Feedback and thoughts from the audience:

    -   Use workflow to compare outputs from the use of various routines
        and/or various parameters

    -   Need more resources on the grid to take advantage of workflow

    -   Advertisement (Index service) -- "Use caBIG a lot"

    -   Would be useful to have a repository of workflows -- a workflow
        "data service"

    -   Instantiation of a workflow needs to be accessible to users

    -   Workflows as a first class service -- workflows composed of
        other workflows

    -   The need for dynamic service discovery was questioned -- limited
        number of uniform services

    -   Would be useful to created a new "piece" or "module" of an
        application that could be plugged in

    -   caBIG should identify common workflows for the community
        (coarse-grained functionality)

    -   Provenance tracking needed -- mapping results and parameters to
        the workflow that generated it

    -   Store the protocol with the results

    -   Access to intermediate results

        -   Discussed client side vs. server side (many advocated for
            client-side)

        -   GenePattern team's approach -- temporary server-side storage

    -   Need to optimize how data (esp. large data) is passed

**caArray Update -- Anand Basu**

(<http://gforge.nci.nih.gov/frs/download.php/954/caArray_ICR_F2F_Sept.ppt>)

Comments/Questions:

-   What is the status of MAGE OM registration in caDSR?

    -   It is on the staging server and will move to production in a
        week or so.

    -   Will not coincide with 1.4 but the model will be available in
        1.5

-   Validation levels....Are all the errors detected and displayed, or
    just the first error that is encountered reported?

    -   Only the first error of a give type that is encountered is
        reported.

-   Is there validation for MAGE ML files?

    -   Will take any valid MAGE ML file and/or external data.

-   What MAGE ML sources have been used for testing?

    -   Affymetrix, GeneTraffic, Agilent (some massaging was needed)

-   Is there a limit on the doc size?

    -   caAMEL been tested on files up to 1.6GB

    -   In the future it may be useful/necessary to provide a mechanism
        to divide the file into smaller chunks

-   What is the status of migrating caArray to caGRID?

    -   caArray will be migrated to caGRID 1.0.

-   Do we have to generate a MAGE ML file? Is this the only way to get
    data into caArray?

    -   That's the only annotation file format supported. The portal is
        the other way annotation data can be loaded and the expression
        values can be loaded in their native formats.

**GenePattern Update -- Ted Liefeld**

<http://gforge.nci.nih.gov/frs/download.php/1035/ICR_F2F_2006_GenePatternGridExtensions.ppt>

**Comments/Questions**

-   Introduce toolkit

    -   We are generating own XML rather than going through the GUI.

-   Any plans for reviewing code for people submit that they wrote
    themselves

    -   Modules contributed by the community are reviewed by the
        GenePattern team before they are made available publicly

    -   Users can independently integrate their own modules with
        GenePattern

-   I am interested in connecting to JavaScript to create AJAX APIs. Is
    this doable?

    -   Yes

-   There are currently no plans to release GenePattern server (created
    independently of and before caBIG). The analytical executables are
    open source, anything built for caBIG™ is open source

**Pilot to Enterprise -- Mark Adams**

**Comments/Questions**

-   What is the effect on the caBIG™ budget as we move to enterprise?

    -   Change in focus, not necessarily a change in funding

    -   Use of resources to provide platforms to support broader
        deployment of caBIG tools

    -   Resources may come for other venues

        -   NCI's portfolio is planned to move toward caBIG™
            compatibility.

        -   More RFP's and RFA's are including caBIG™ compatibility
            requirements

-   New support mechanisms (desktop, infrastructure) and newcomers
    coming into caBIG™

    -   Encouragement of caBIG™ user groups at institutions

    -   TBPT WS has monthly newcomers calls to introduce people to the
        tools

-   Timeline to make some of these decisions on DSIC issues?

    -   Intellectual property and security make this a challenge. Based
        on DSIC security policy project

-   Is it possible to access tissue from caTissue?

    -   Any instance of caTissue Core is accessible via a web browser.
        If you have permission to access tissue, then yes. It is
        controlled access.

**Review ICR Strategic Plan -- Juli Klemm**

<http://gforge.nci.nih.gov/frs/download.php/1049/20060918_Day1Day2_JKlemmSlides.ppt>

Comments/Questions

-   CPAS

    -   What is special about CPAS? It's the only project outside of
        caBIG™ that is mentioned explicitly in the ICR Strategic Plan

        -   At the time this strategic plan was written, a key priority
            was to identify a way to engage this group.

-   Population Sciences is still housed in ICR but it's like a mini WS.
    We need to determine the best way to engage this group.

-   What's the problem with having the cancer community put tools up on
    the web instead of having them certified?

    -   You don't need a Grid for that; you have the internet

**caTRIP Update -- Patrick McConnell**

(<http://gforge.nci.nih.gov/frs/download.php/962/catrip_icr_f2f_2006_09_18.ppt>)

Comments/Questions

-   How current is the data in the Duke tumor registry?

    -   Approximately current to within the last six months -- need to
        verify

-   Different hospitals use different record numbers so two people can
    have the same record number but be different people. Linking by
    Medical Record Number can be potentially problematic when you start
    to link across institutions -- MRN's can be duplicated across
    institutions. May need to consider using a combination of attributes
    as the primary key.

    -   Looking to the CTMS WS to propose solutions to these kinds of
        problems.

-   Where are the tissue samples?

    -   The data in caTRIP will only be from the Duke tissue repository.

-   Does caTRIP allow a user to link together systems of a given type
    (for example, all caTissueCORE instances) or does it just link
    across given instances of a system type?

    -   The latter is the use case at the present time.

**Bioconductor Update -- Martin Morgan and Pan Du**

(<http://gforge.nci.nih.gov/frs/download.php/963/20060919_Day2_Bioconductor.ppt>)

Comments/Questions

-   In today's demo, the data was passed in as Affy files. Any plans to
    write them out as MAGE objects?

    -   Yes, that's the plan, for both input and output.

-   Will this release result in a toolkit for grid-enabling Bioconductor
    modules in a way that could be included with the Introduce toolkit?

    -   We will have such a toolkit and incorporating with the rest of
        the caGrid toolkit would be feasible.

-   Are there any efforts to standardize interfaces between tools --
    i.e., have common interfaces; For example, a common interface for
    all clustering tools, regardless of what type of data they take in?

    -   Within the R community, there is an interface package whereby
        tools that use R can be put together with a common interface.

**GeneConnect Update -- Rakesh Nagarajan**

(<http://gforge.nci.nih.gov/frs/download.php/957/GeneConnect_F2F_Sept_06.ppt>)

Comments/Questions

-   How does GeneConnect relate to caFE?

    -   It's a different project. caFE is an automated annotation tool.
        We've re-used the majority of the design in the caFE server to
        do the parsing. That's where the commonality lies.

-   Is there a biological basis for confidence values calculated through
    GeneConnect?

    -   The biological basis is only as reliable as the reliability
        between the link of the databases. A higher confidence value
        certainly means it's a higher rank.

-   How long does it take to execute a query? Can GeneConnect handle
    50-60,000 identifiers at the same time?

    -   Performance issues are TBD. No limit on how much can be taken as
        input. If you want to put a whole array worth of probes, caFE
        might be a better way.

-   The pipeline of new genomes coming out is huge. There are new
    releases coming out all the time. What ideas are there to make sure
    things stay up-to-date?

    -   The GeneConnect database can be updated with whatever frequency
        makes sense.

-   Why do you have separate services to annotate microarrays and not
    just do it at once?

    -   GeneConnect allows to do it in many different ways and not just
        microarrays. There are additional services on the Grid that can
        go through GeneConnect.

-   Is this something we'd run all the time or are we getting the maps
    from you?

    -   Either way can work and it's certainly up to the user.

**geWorkbench Update - Aris Floratos, Craig Street**

(<http://gforge.nci.nih.gov/frs/download.php/945/20060919_Day2_geWorkbench_grid_enablement.ppt>)

Comments/Questions

-   How will caScript work with BPEL?

    -   caScript is needed on top of BPEL in order to orchestrate
        internal services with caGrid services.

-   Are you developing your own scripting language?

    -   Yes, we are. But we are considering using a generic scripting
        language.

    -   Our scripting language is less complicated than Java and should
        simplify workflow specification.

    -   Might want to look at python as another possibility

-   What is the data format that is returned from clustering?

    -   A registered data type in grid service.

-   Is the internal structure represented in the model?

    -   We have a formal framework to represent data types internally --
        this is not exposed through the caBIG API. .

**Review ICR Gap Analysis and Review Workflow Diagram for ICR Tools --
Moderated by Juli Klemm**

<http://gforge.nci.nih.gov/frs/download.php/1049/20060918_Day1Day2_JKlemmSlides.ppt>

Comments/Questions

-   "The key is the edges" -- ICR should focus on integration between
    tools rather than development of new tools

-   Eliminate user intervention between the edges (seamless integration)

-   Do more work on existing projects; focus on user adoption

-   Why aren't certain tools being used?

    -   Reluctance to change

    -   Need to show value

-   Expand adoption

    -   Fund multi-tool adoption

-   More training needed

-   Announcement of tool availability -- journal article?

-   Give a lot of weight to researchers' voices

-   Non-caBIG resources -- Integrate with legacy systems using standards

-   Get more data into caArray -- e.g. Oncomine data, ArrayExpress data

-   Comprehensive and flexible interface to grid services -- don't
    reinvent the wheel

-   Harden existing projects, show value, advertise

-   Provide elegant ways of getting data in and out

-   Feedback on tools diagram -- Focus on "modules" of interoperability,
    rather than individuals

-   Provide a forum for user experience with caBIG tools

-   Engage cooperative groups and program projects as adopters

-   Need to get the Silver resources on the grid -- need to determine
    where the data and resources reside

-   Easy installers needed

-   Sell the vision of caBIG with real scientific success stories

-   Need to be clear about the ultimate plan, the big picture

-   Focus on the needs of the scientific community

    -   Find better ways of engaging scientists face-to-face, such as
        focus groups

    -   Follow the example of TBPT

-   Be clear about the distinction between resources that support
    science (but are not directly used by scientists) and those that are
    used directly by bench scientists

-   Who will perform the training for these tools? Primary developers
    and adopters, initially

**DAS2, HapMap and VPD Update - Brian Gilman, John Rux, Maureen
Higgins**

Comments/Questions

-   Why did you choose that particular method for identifying binding
    sites instead of the typical promoter?

    -   The binding site is already determined from statistical methods.

-   Will you have to download all the genes?

    -   Yes,this will be a one-time analysis.

-   Will you be bringing in SNPs from sources other than HapMap?

    -   No, the scope of this project is only the HapMap data.

-   Will the caBIG interface to the HapMap and VPD datasets be the
    caCORE API?

    -   Yes

-   What security is set up in DAS?

    -   None yet as we use public data.

**caB2B update -- Rakesh Nagarajan**

(<http://gforge.nci.nih.gov/frs/download.php/970/caB2B_F2F_0906.ppt>)

Comments/Questions

-   How does the I2B2 project (Informatics for Integrating Biology and
    the Bedside) relate to the caB2B project? It is trying to do things
    that are very similar to what you are describing. How are you guys
    collaborating? It's one of the roadmap initiatives.

    -   We're not aware of them but would love to talk to them.

-   How do you envision displaying microarray data once it's in the
    shopping cart---use caFE or how is stored or visualized?

    -   Yes, everything will be in caB2B. There will be a central area
        for visualization---analogous to geWorkbench. The local database
        would store all the process results and anything you've brought
        back.

-   From the first integration, when you get to the Grid, will the
    application work with all the centers?

    -   First iteration will only be deployed locally but the code will
        allow it to go on the Grid 1.0. The real trick will be to
        populate the different data centers.