Informational Meeting: caBIG™ Service Provider Program

[Summary Notes]{.underline}

**Meeting Information:**

When: Tuesday, August 14, 2007, 9:00AM -- 2:15PM

Where: Conference Room C - 6001 Executive Blvd. Rockville, MD 20852

This meeting was designed to gather input from the interested service
provider community to develop criteria for an NCI-sponsored "caBIG™
Service Providers Program." Additionally, this meeting communicated
background information on the caBIG™ Enterprise Support Network, NCIs
expected role in that network, and explained how the Service Provider
Program (SPP) will be developed and rolled out. Finally, this meeting
provided a forum for presenting points of view from clients of IT
services, and gathered program suggestions from interested service
providers, with a specific emphasis on gathering detailed
characteristics requisite to making such a program a success.

**The caBIG™ Enterprise Support Network and Service Providers Program
(9:15-10:15)**

Ken Buetow (NCI) opened the meeting by welcoming attendees, and
introducing the goals of the session. This opening presentation is
captured in slides entitled "Overview of caBIG™ and the Enterprise
Support Network." Key points/questions not reflected in the slides
include:

-   Q: Concerning the licenses and their boundaries: How will you define
    the boundaries of license usage, and how will you enforce the
    boundaries? A: There will be a detailed set of services, and
    definitions of those services -- those definitions will define the
    boundaries of licensing and who is qualified to be within those
    boundaries. If groups provide services outside that scope, and claim
    licensing, we will defend the trademark using the means available,
    both in terms of communication with the offending party and legal
    actions if needed.

-   Q: Some of the tools aren't really complete. What if vendors take
    one and build it out? Is it still "a genuine caBIG™ part" once it's
    evolved? A: Providers could continue to extend tools - as long as
    you are compatible, it is fine. The compatibility
    guidelines/standards define compliance -- it is a "caBIG™ part" if
    it is compatible, and meets the guidelines.

-   Q: Will service providers be encouraged to do open source or is
    proprietary development allowed? A: Tools developed with government
    money must be open source. If it funded from other sources, you are
    free to develop proprietary tools -- this is a non-viral license --
    if the criteria for compatibility for a closed software tool are
    met, it is compatible. You can be silver compatible without being
    open source.

-   Q: Are compatibility reviews themselves a service? A: This is a
    possibility we are talking about.

-   Q: Where are the lines between what the Center developer does and
    what the service provider will do? The lines need to be clear
    between what Centers do, versus what the providers do -- funding two
    groups to do the same thing is a problem. If you already have teams
    funded to develop x, y, and z, what do we get to do? What's funded
    by NCI as directed support, and what comes to providers? How do we
    encourage buyers to use providers, versus NCI's directly funded
    services? A: NCI will continue to provide resources to the
    community, but there will be a difference in the quality levels --
    you will find that NCI's capability to support audiences is quite
    limited -- we can't offer the responsiveness to the full community
    that you can. That said, we hear the concern. Where do providers
    think the boundaries should be? We are not in a programmatic
    position to close off support -- but right now, the boundaries are
    not entirely unclear.

-   Q: What will NCI be doing to improve the information infrastructure
    about the tools themselves? Gforge is not a good option -- it's very
    hard to find things there. The tools are not learnable given the
    existing infrastructure. A: The comment was duly noted.

-   Q: What are the criteria for adopters to meet the caBIG
    compatibility requirements? A: Groups on the path to compliance will
    be demonstrating the use of compatibility infrastructure at silver
    and gold levels. This can be done through adoption or modification
    of existing tools.

-   Q: Will there be exclusive agreements, or is this a competitive
    set-up? A: We are not anticipating exclusive licenses. There will be
    lots of providers -- some will be more narrow in scope than others.

-   Q: What will NCI's role be in providing information about service
    providers to the community? We need help to get Cancer Centers to
    provide funding to engage service providers. A: We hope that
    licensed providers will market their services. We will be
    aggressively communicating the benefits of the licensing process and
    the availability of service providers to the centers -- Directors
    and CIO's. When they ask, "Who can help us," we will communicate the
    provider list to them. We CAN NOT market and endorse products.
    Commercial providers are generally welcomed by CIO's -- it's harder
    to sell at the grassroots, to the folks who would prefer to build
    their own tools.

-   Can we create a listserv for this group? Yes -- materials will also
    be posted on the website.

**Prospective Service Buyer's Point of View (10:15-11:05)**

Panel: Martin L. Ferguson, David Fenstermacher (Moffitt), Robert
Annechiarico (Duke)

During this section, representatives of both commercial and academic
entities, with experience as "buyers" of IT support services, presented
key issues in decisions to procure services in a technology-centric
emerging market situation.

[Robert Annechiarico (Duke)]{.underline}

-   Provided overview of Duke informatics organization and
    infrastructure, focusing particularly on the clinical trials
    elements of infrastructure, which extends beyond the science into
    financial areas as well.

-   Duke set-up is not all caBIG™ - there are other pieces hanging off
    of it, and it all needs to connect to Duke's larger infrastructure

-   Wish List: Supplement staff capability, extend capabilities -- need
    "go to" technical support and programming resources. Need
    continuity, and patch control. Need to make sure that future
    iterations are monitored and that patches are available. Need to
    help maintain the interoperability element over the long term, as
    pieces evolve and become increasingly interconnected.

[Martin L. Ferguson\
]{.underline}

-   Speaking from the perspective of a procurer of services, seeking
    "work for hire" software development and hosting services.

-   Providers need domain-specific technical knowledge: know both IT
    (three tier technology) and biomedicine (human subjects awareness,
    biomolecular knowledge) -- need a strong understanding of regulatory
    realities and impacts in this field.

-   Other key criteria: Provider corporate stability: financial
    stability, credit/financials, M&A.

-   Contractual dynamics: flexible contracting mechanisms (fixed price,
    time and materials), service level agreements, willingness to
    barter, quality guarantees, willingness to train buyer, source code
    deposition into escrow.

-   Need to have an understanding of boundaries between academic and
    industry collaborations -- different interests can be coordinated
    (open source versus proprietary).

-   Value of license: technical competence, engagement with community.

[David Fenstermacher (Moffitt)]{.underline}

-   Representing non-profit view... We like to tinker with software, so
    don't want to give all that up.... BUT, we also need
    commercial-grade software to serve our entire community.

-   Our case study: caTissue Core. We did an initial evaluation of
    requirements -- and found we needed the entire CAP list, barcoding,
    and certain fields (container type). Having a structured project
    plan was also important to us. After doing our assessment, we
    decided NOT to go with caTissue, and procured a tool that is not
    caBIG™ compatible. Now, we need to figure out how to GET compatible,
    even though what we bought was not caBIG™ compatible (and vendor is
    not planning on making it compatible).

-   Needs: onsite installation and configuration (ETL, integration, data
    quality and data migration/integration), and acceptance testing?
    Need ongoing support, help desk, bug reporting, tiered support --
    different options for responsiveness levels. Need support for end
    user directly, and we need training.

-   Need caBIG™ compatibility certification, so that if we extend
    software, we can then get it checked. Need external groups to do
    this -- existing VCDE review process is too slow and limited in
    scope/coverage.

-   We need to be able to compare licensed providers to other providers:
    cost, risk assessments, ongoing support plans, conflict resolution
    mechanisms and warranties.

[Questions/Answers for Buyers:]{.underline}

-   Q: Small businesses will have a tough time competing against these
    requirements, and the large companies are going to cost too much.
    How are you going to deal with this? A: Combinations of large
    companies with small subcontractors is a possibility here. There's a
    diversity of companies and company sizes -- there may be value in
    partnerships among you. Different users will have different needs;
    different services will require different levels of support. There
    are going to be tradeoffs that will be worked out in the competitive
    landscape. You have to build your own business model -- what
    services do you want to provide that fit your own size and business
    model?

-   Q: There's not just competition between service providers, but also
    between cancer centers. How do you see that playing out? A: There
    may be less competition than you think -- there's an even playing
    field to start from. Research is not a barrier to collaboration --
    in fact, take the SPORES as an example -- the collaborative
    environment is now a criterion for successful competition of grants
    -- the rewards are shifting from competition to collaboration. The
    whole is greater than the pieces. Science is getting bigger -- and
    the competition isn't in the infrastructure.

-   Q: How many providers do buyers want to have? A: As few as possible,
    but there won't be just one that can do everything -- we need to
    make decisions based on what is needed. No one provider is going to
    be good at everything, but there's no optimal number. You want
    healthy competition, it's an emerging market -- partnerships will be
    important to help both provide a market and allow for competition.

-   Q: What is the procurement process in your organization? How
    streamlined is it? How centralized is it, or is it distributed
    throughout organization? A: Every organization is different --
    ranging from central IT shops to highly distributed groups in
    different departments. Knowing who the buyer groups are is hard
    sometimes. Some are at level of department, some at the cancer
    center, some at the medical center, and some are at the
    university-wide level. It varies by both center and need.

-   Q: There's a balance and choice between products and services --
    will they be done together? A: These are unifying principles that
    cancer centers look at -- each center will have a slightly different
    path, and there needs to be a mix. Individual companies will build
    reputations that will get spread throughout the community -- we ask
    each other, "What are you using? What works?" We don't trust
    brochure-ware.

-   Q: What are the funding mechanisms? Where does money come from? A:
    It will have to come from customers. Funding is a matrix -- there's
    a core grant, there's revenue from pharma trials, SPORE grants, DoD
    funding, etc -- there's some fluidity there from different sources
    that can be leveraged here.

-   Q: What's the size of the market? A: Hard to estimate - caBIG™
    services will eventually be asked for in areas where government
    money is not involved -- e.g., philanthropic funding,
    pharmaceuticals that will need to take data in for certain studies
    -- there are lots of possibilities.

**Working Session 1 -- Developing criteria for caBIG™ service providers
to be designated as licensees (11:20-12:05)**

Peter Covitz (NCI) introduced this session by discussing the need for a
set of criteria against which potential service providers can be
evaluated to determine eligibility for a caBIG™ license. This
introduction included an overview of service areas desired from
providers, and an initial set of requirements/criteria, captured in the
slides from this presentation. Here is a high level summary of the
services and criteria:

+-------------------------------+--------------------------------------+
| **Key service areas likely to | **Criteria Categories:**             |
| include:**                    |                                      |
|                               | -   Technical capabilities           |
| -   Help Desk/Phone Support   |                                      |
|                               | -   Staffing                         |
| -   Software Development      |                                      |
|                               | -   Geographic coverage              |
| -   Implementation Support    |                                      |
|                               | -   Experience with caBIG™           |
| -   Training Development and  |     Program/Technology               |
|     Delivery                  |                                      |
|                               | -   Biomedical domain expertise      |
|                               |     (cultural awareness)             |
+-------------------------------+--------------------------------------+

A discussion followed the presentation. The following bullets summarize
the questions and recommendations from this discussion.

-   Q: Will certification be done at the level of the service category,
    or at a more specific tools level? Buyers likely to want
    certification at the level of the tool; providers would likely
    prefer at the service level. A: We agree that will be hard to manage
    at a highly granular level, however, recognize that providers are
    more likely to focus on a subset of tools. Cancer Centers will
    ultimately evaluate provider capability for their specific needs.

-   Q: What is the role of boot camps for providers to create diversity
    of choice? A: NCI boot camps are available to everyone including
    providers. There's a touch balance between investing and building
    the market vs letting the market define itself.

-   Audience proposed two additional service categories: (1) Operations
    and management, including on-going evaluations of software; (2)
    caBIG compatibility reviews by external parties.

-   Proposal: Should there be two kinds of licenses? First kind would be
    application specific, providing life cycle support within a domain;
    second kind would provide service specific support (training, help
    desk). Case Study: Licensed provider is an expert in tool A, but
    dabbles in other areas. Are they really at the level that caBIG
    desires? A: The buyers will help regulate this to a point.

-   Proposal: You probably need to ask applicants to specify any
    limits/focus areas related to their license to help focus the
    evaluation - different business should be able to stress different
    pieces. This could be done even though the licensing is done at the
    service category level, allowing both growth and specialization.

-   Q: What other biomedical areas are likely to be engaged in caBIG
    down the road? VA? (What's the potential size of this market?) A:
    There are many potential growth areas - this is a moving target that
    you can help shape through your services.

-   Q: What's the role of competitive grant requirements in helping
    build the market? A: This is evolving. We need to balance actual
    provider capability with demanding that the capability be used.

-   Proposal: Consider a "distinguished" mark to help delineate
    expertise in a specific domain or service area. This would help ease
    the evaluation process.

-   Q: What about competitive positioning? I provide caBIG services, but
    I also sell products that compete with caBIG tools. Is this OK? A.
    This would probably lead to some internal struggles for the
    provider, but as long as you are excelling at caBIG, it's probably
    not a problem. It is NOT OK to just use license to get in the door
    with your product.

-   Q: Can services be billed to a grant? A: Depends on the grant. This
    is very situational.

-   Proposal: Recommend ability to expand beyond the originally licensed
    service area. Case Study: I started in software development; now, I
    want to add training. This should be possible. Response: We see
    implementing a periodic renewal process, where this could be done
    (annually?).

-   You should incorporate buyer requirements into application questions
    and criteria.

-   Q: What are the criteria for having your license dropped? What if
    providers are in a long term relationship with a center, and their
    license is lost due to performance elsewhere? Would the center have
    to get a new provider? A: This would have to be determined by the
    contractual language established between provider and buyer.

-   Q: How will applications be scored? Can I reapply if I don't get in,
    and will I get feedback about my application if I don't pass? A: We
    need to figure that out, but at present there are limited government
    resources/manpower to administer this program.

-   Q: When developing software track application, consider looking at
    external certification models for ideas (SEI/CMMi)

-   Q: Can I "do caBIG" without a license? A: Yes, you can provide
    services without a license. This is up to vendors and buyers --
    buyers will ultimately set the demand for licensing.

-   Q: Is licensing a procurement activity? If not, be careful not to
    include legal or government procurement requirements, as these
    should be driven at the buyer level. A: NO, this is NOT a
    procurement activity.

-   Q: Who pays for the application evaluation? A: Right now, caBIG will
    cover this. Eventually, the process could be facilitated by a third
    party.

-   Proposal: Need a practical element in the assessment process. Past
    experience is important - it's not just theory. The broadness of the
    service categories makes it tough for small businesses. Requiring a
    demonstration of capability would allow small business to prove that
    they could perform, even with a small staff.

-   Note: Licensing is not a guarantee of performance. Be careful not to
    require things in the application or renewal that buyers don't
    actually want. Could set up conflicts later on if provider focused
    on license requirements rather than user needs.

**Session 2 -- Developing evaluation criteria to measure success in
provision of service (1:10-1:25)**

Martin L. Ferguson opened this discussion by noting that once the
program is launched, NCI will need to receive feedback from both
providers and clients to gauge overall program success and continually
consider "tweaks" that enhance the program. Furthermore, these criteria
should be specific enough to measure the performance of individual
providers and provide a basis for determining continued grant of the
caBIG™ license. Initial key categories for evaluation include: client
satisfaction, domain knowledge, capacity and responsiveness, and
product/service line expansion.

A discussion followed the presentation. The following bullets summarize
the questions and recommendations from this discussion.

-   It's good to have standards related to client satisfaction, but
    surveys can be problematic. Surveys would have to be done with
    buyers that the provider has done business with under the license.
    You also need to allow a fix-it period.

-   How will you distinguish satisfaction with the tool vs. the program
    vs. the provider themselves? Did the program do a good job? Did the
    company do a good job? This is a joint responsibility -- answers
    from customer surveys may not always actually match the question...
    important to separate out the issues.

-   Consider looking at the GSA Schedule client survey questions for
    ideas here.

-   Consider a mechanism for sharing contract SOW with NCI, so
    evaluators would know what provider was actually contracted to
    provide. This would place customer feedback in better context.

-   Client satisfaction is a bell curve. It's not just a count of
    issues - it's overall, based on total performance.

-   Responsiveness metrics are a bit easier than customer satisfaction
    to develop.

-   What about licensing individuals rather than companies? (e.g.,
    license caBIG certified people).

-   Need to watch out for too much detail here -- the more granular the
    measures, the harder it will be to evaluate and prove performance.

-   Q: Who tracks all this? Would knowledge centers be responsible for
    evaluating domain expertise? A: Boundaries are not yet clear, we'll
    have to avoid conflict of interest issues.

-   Each license category will need different measures of success.

-   Be sure to include project management elements (e.g., scope, time,
    cost management) as part of the software track.

-   Don't make this too difficult. There are lots of nuances. The E-Bay
    model may be best... Was the experience positive or negative, with
    an allowance for comments. There's merit in public displays if the
    right questions are used. Is that enough?

-   Success criteria should be tied back to the licensing evaluation
    itself.

-   Avoid Likert scales. Responses should be forced choice (yes or no)
    with comments.

-   Proposed examples of domain knowledge: Did provider understand the
    relationship of the clients world to the caBIG world? Did provider
    perform what they advertised? Did the provider help clarify the
    client's own requirements?

**Wrap Up and Next Steps (1:25-1:35)**

Leslie Derr (NCI) closed by thanking participants for their input, and
summarizing the key topics completed during this meeting.

Next steps:

-   NCI will develop evaluation criteria based on inputs from today and
    from other sources

-   There will be an announcement for caBIG™ Service Providers through
    regular caBIG™ channels , with time for questions

-   Applications will be evaluated based on defined criteria

-   Those who have been successfully approved will be given guidelines
    for branding use.

Applications will include: service category - with responses to
criteria, and a plan for gathering customer feedback.

After a year of activity, we will evaluate to see if the program is
working -- Are demands being filled? Are there gaps in the program? Over
time, providers will be reevaluated, and the set of applicants may
broaden. This will be an evolving and iterative process.