**NC Collaboratory:**

**Domains:**

-   **Biomolecular Simulation**

-   **Biology**

-   **Climate Change Scenarios**

**Biomolecular Simulation:**

-   *Discussion/Issues:*

-   MD (applying newton's rules of motion to protein/dna structure)

-   Computationally intense, but not much data

-   Want to make predictions

-   Use case 1: iteratively build model of indiv. Prot/dna
    sequence/structure.

    -   Initially collect data -

        -   Nmr

        -   Xray

        -   Crosses domain boundaries and is distributed

    -   Input to initial model

    -   Simulate

        -   Scaling:

            -   Tightly coupled: 256 cpus

            -   Loosely coupled (scale)

        -   May want to visualize

        -   May run on order of months:

            -   Preprocess locally,

            -   move data to hpc resource

            -   Run and evaluate RT, monitor, track

            -   70 GB

            -   10 nanosecond simulations

            -   1 month on 128 cpus, high level of communications

    -   Refine hypothesis

    -   Possible collect more data

    -   Typically this is done on the phone, and not shared so want a
        better collaboration env

    -   Issues:

        -   Need better resource discovery for cluster scavenging

        -   Wants to sit on desktop, drive everything from single script

```{=html}
<!-- -->
```
-   Use case 2: protein structure

    -   Small, independent jobs (1 day)

    -   Good cycle harvesting candidate

    -   Want to look at data, perform queries while data is coming in

        -   Use this output to seed future simulations

    -   Want a framework that everyone has installed on systems that can
        be used for resource discovery

    -   Indexing and data manipulation --

        -   Confidence intervals

        -   Merge

        -   Data flow

        -   Auditing and notebooking

-   *Requirements*:

    -   Workflows that allow users to transition from batch oriented
        computing to 'personal grids'

    -   Engineering system from bottom to account for transient failures
        of all types

    -   Better API's, more reliable and robust

    -   Interactive/remote access/control while simulation is running --

        -   New tools that will augment what is typically consider to be
            computational steering

            -   Decision support

            -   Managing large workflows, multiple machine

            -   Steering becomes more complex

            -   Feature based decision

        -   Don't' want to just run a simulation and wait for it to
            finish. If something going wrong, want hooks/diagnostics
            that allow them to evaluate how computation is progressing
            and then kill/change/etc.

**Biology:**

-   *Discussion/Issues:*

-   What they can do today:

    -   Independent instrument data stream (log 1/sec over 1000 hours)

    -   Problem is taking data to calculate analysis; using excel, but
        limitied by data set sizes;

    -   Use telnet to monitor/access data; cannot do much bcs of work
        firewall

    -   Distributed operation and control of an instrument:

        -   Have 20 reactors running, they are \$\$\$, so trying to
            establish a user facility so that an EDU person can access
            and use remote instrumentation access; and then want to
            monitor and manage remote

        -   Have different sample types and they have to snail mail
            around to different instrument centers; they run and get
            data and store; community needs to access the data; spectra
            generated needs to be analyzed; need to do initial 'blessing
            and reduction of data"; need metadata associated with this

        -   Collaborators can do analysis on raw data

        -   Need to retrieve all analyzed data sets on a
            sample/collection of samples and wholistically analyze;
            tracking of a samples "life-line" or tree/path is done
            manually; data is put into TB size drives and manually
            shipped around.

        -   Workfow of sorts that is labor intensive;

    -   New domains:

        -   Comparative biology and genomics: how do they compare, etc.

        -   Done by hand, ad-hoc

        -   Different data formats, no central database, few integration
            tools

        -   PCMDI @ LLNL: comparison project

        -   Kyoto has a database that is searchable, and can do
            comparative analysis; need more sources like this

        -   Sociological issue: global collaboration vs global
            leadership?

-   *Requirements*:

    -   High speed data transfer networks so can stop shipping disks;

    -   Digital metadata associated with data so can track a sample
        lifeline

        -   Data is in different formats, so need uniform way to view it

        -   Want to know what software, model, configuration,
            annotations are used in collecting the data or analyzing it.

        -   Data is always changing, want to continuously update their
            data banks;

    -   Connection of instrumentation with remote access and monitoring

    -   Interactions and integrating data sets:

        -   Integrate data from these 4 or 5 analyses

        -   RT collaboration tools to complement f2f, web ex.

        -   Challenge to compare data from cross-domain data sets; need
            to be searched and matched

    -   Collaborations:

        -   Communities have not tools

    -   Publication of this information

        -   Trust/quality factor

        -   Format of data

        -   Protection of access

**Climate Change Scenarios:**

-   *Discussion/Issues:*

    -   In past: making everything by hand: data catalogues,

    -   Partnered with ESG, deliberately distributed the problems then
        were faced with need to integrate data and publish is

    -   Need to be able to interact with development, analysis,
        production collaborators

    -   Work related issues:

        -   Individual components are there but there is not a whole
            picture; feels like it is to discreet

        -   Would like the BIRN approach bcs is looks more integrated

    -   Systems are complex, need better tools to diagnose them

        -   If the system was self aware of its own working -- it could
            provide logging and answers, predictions

        -   Real-time flow of processes are needed to be modeled; how
            long is a process running, is it working

        -   These systems give you new abstractions that you are not
            familiar with

    -   Question: are there other communities they would like to
        interact with?

    -   Tactical: what we cannot do today

        -   Collaborating with non-DOE scientists, and international

        -   What does this mean? Putting up an AG node

        -   Sharing of data, models, run parts of the model, not all of
            it

        -   Enable other users to use part of their models

        -   Impossible to track who is doing what on the data

        -   Codes do not scale to large number of processors -- need
            more memory

    -   What can be done today (ESG model): you run a simulation, you go
        to portal, get data and analyze

        -   Current community model is to large to run on small systems

        -   Use IBM SP's in DOE; managing job runs is a huge issue; need
            dedicated access for several months -- 4 total on project

        -   Management is done via cron jobs written by user,
            notification by phone to user which notifies him when things
            go wrong.

        -   Have monitors for people to see status

    -   Vision that would break open field:

        -   Bring in people who don't have access to the resources, are
            part of different science areas or have new models

        -   Currently couple atmosphere, land, ice, etc.

        -   Big challenge has been to get international community to
            agree to couple models and share data; doing ensembles and
            coupled models;

        -   Would like to extend current tools to run beyond DOE (get
            ESG out) and bringing more resources

-   *Requirements*:

    -   Computation:

        -   to be able to submit to a generic location

        -   Universal allocations across resources

        -   Perform more computations or faster -- reduce cycle

    -   Improvements in data and information management

    -   Collaboration and interactions with other communities

        -   Might be able to get other elses' model or data

        -   Enabling smaller groups to do big science

        -   Enabling them to participate

    -   Wholistic view of the overall system including:

        -   Better encapsulation of fuction \-- don't' really want to
            even know that they are using a grid

        -   Status/Manage/monitor, better logging/streaming of
            information, notification

        -   Need/want to move away from hand crafting specialized
            solutions (e.g. cron jobs)

        -   Easy to publish to community and client

        -   better, distributed tools: distributed debugger, how well
            did things run? who has the data?

        -   View t he codebase is a facility, the 'grid world' is a
            facility

        -   As sytems get large in science, we need to iintroduce
            separation of roles and responsibilities