Algorithms, Frameworks and Toolsets for High Performance, Remote and
Distributed Scientific Visualization

#### Wes Bethel and John Shalf {#wes-bethel-and-john-shalf .unnumbered}

#### Lawrence Berkeley National Laboratory {#lawrence-berkeley-national-laboratory .unnumbered}

Principal Investigators

####  {#section .unnumbered}

*Bernd Hamann, Ken Joy* *and Kwan-Liu Ma*

*University of California - Davis,* *and Lawrence Berkeley National
Laboratory*

Academic Co-Investigators

A Field Work Proposal

to

The U.S. Department of Energy

For the period

FY2005-FY2007

February 2004

  ---------------------------------------------------- ----------------------------------------------------
  ![](media/image1.png){width="1.1354166666666667in"   ![](media/image2.png){width="1.9270833333333333in"
  height="0.6854166666666667in"}                       height="0.6256944444444444in"}

  ---------------------------------------------------- ----------------------------------------------------

###### Table of Contents {#table-of-contents .unnumbered}

[1 Abstract 4](#abstract)

[2 Project Vision 4](#__RefHeading___Toc65574774)

[3 Project Benefits 5](#project-benefits)

[4 Background and Motivation 6](#background-and-motivation)

[5 Research Focus Areas 8](#research-focus-areas)

[5.1 DiVA Framework 9](#diva-framework)

[5.2 Domain-Neutral Multiresolution Data Model
10](#domain-neutral-multiresolution-data-model)

[5.3 Component-Based Visualization Pipelines: Automated Component
Placement and Dynamic Optimization 11](#__RefHeading___Toc65574781)

[5.4 Visualization- and Graphics-Centric Network Transport
13](#visualization--and-graphics-centric-network-transport)

[5.5 Grid-Enabled Remote and Distributed Visualization Infrastructure
15](#grid-enabled-remote-and-distributed-visualization-infrastructure)

[5.6 Multiresolution, Latency-Tolerant Remote & Distributed
Visualization Algorithms.
17](#multiresolution-latency-tolerant-remote-distributed-visualization-algorithms.)

[5.7 Realizing Grid-based RDV Applications
18](#__RefHeading___Toc65574786)

[5.8 Remote and Distributed Visualization and Scientific Data Management
22](#__RefHeading___Toc65574787)

[6 Statement of Work 24](#__RefHeading___Toc65574788)

[6.1 Overview 24](#__RefHeading___Toc65574789)

[6.2 Approach 24](#__RefHeading___Toc65574790)

[6.3 Tasks, Milestones, Deliverables 25](#__RefHeading___Toc65574791)

[6.3.1 FY05 25](#__RefHeading___Toc65574792)

[6.3.2 FY06 26](#__RefHeading___Toc65574793)

[6.3.3 FY07 27](#__RefHeading___Toc65574794)

[7 Bibliography 27](#__RefHeading___Toc65574799)

[8 Biographical Sketches 29](#__RefHeading___Toc65574800)

[8.1 Bethel (LBNL) 29](#__RefHeading___Toc65574801)

[8.2 Shalf (LBNL) 32](#__RefHeading___Toc65574802)

[8.3 Hamann (UCD) 34](#__RefHeading___Toc65574803)

[8.4 Joy (UCD) 36](#__RefHeading___Toc65574804)

[8.5 Ma (UCD) 38](#__RefHeading___Toc65574805)

[9 Facilities and Resources 39](#__RefHeading___Toc65574806)

[10 Current and Pending Support 40](#__RefHeading___Toc65574807)

[10.1 LBNL Visualization Group 40](#__RefHeading___Toc65574808)

[10.2 Visualization and Graphics Research at UC Davis
40](#__RefHeading___Toc65574809)

[10.2.1 Hamann 41](#__RefHeading___Toc65574810)

[10.2.2 Joy 43](#__RefHeading___Toc65574811)

[10.2.3 Ma 44](#__RefHeading___Toc65574812)

[11 Appendix A -- Justification For Equipment Funding (Contains
Proprietary Information) (Remove this Section Before Sending to UC
Davis) 47](#__RefHeading___Toc65574813)

[11.1 Summary 47](#__RefHeading___Toc65574814)

[11.2 Background and Existing Infrastructure
47](#__RefHeading___Toc65574815)

[11.3 Technical System Requirements 48](#__RefHeading___Toc65574816)

[11.4 Intended Use 49](#__RefHeading___Toc65574817)

[11.4.1 Visualization Research Projects 49](#__RefHeading___Toc65574818)

[11.4.2 Future Technology Group Projects
50](#__RefHeading___Toc65574819)

[11.5 Altix Architecture 50](#__RefHeading___Toc65574820)

[11.6 Alternatives 51](#__RefHeading___Toc65574821)

[11.6.1 SGI Onyx 51](#__RefHeading___Toc65574822)

[11.6.2 A "White Box" Cluster 52](#__RefHeading___Toc65574823)

[11.6.3 Some Other Branded Cluster 52](#__RefHeading___Toc65574824)

[11.6.4 Sun Microsystems: Sun Fire 53](#__RefHeading___Toc65574825)

[11.6.5 Hewlett-Packard: Scalable Visualization with Sepia
53](#__RefHeading___Toc65574826)

[11.6.6 Summary of Alternatives 55](#__RefHeading___Toc65574827)

[11.7 Purchase and Deployment Plans 56](#__RefHeading___Toc65574828)

[11.7.1 Deployment 56](#__RefHeading___Toc65574829)

[11.7.2 System Administration and Management
56](#__RefHeading___Toc65574830)

[11.7.3 Recurring Costs: Maintenance 56](#__RefHeading___Toc65574831)

[11.7.4 Expansion and Growth 56](#__RefHeading___Toc65574832)

[11.8 SGI Altix Quote 57](#__RefHeading___Toc65574833)

[11.9 Dell IA64 Quote 58](#__RefHeading___Toc65574834)

[11.10 Sun Microsystems Quote A and Architecture Diagram
59](#__RefHeading___Toc65574835)

[11.11 Sun Microsystems Quote B and Architecture Diagram
61](#__RefHeading___Toc65574836)

# Abstract

The primary goal of this project is to foster the definition and
emergence of a "virtual standard" for high performance scientific
visualization. Our research focuses on visualization algorithms,
infrastructure and architecture as applied to DOE-funded science
projects with visualization needs unfulfilled by any commercial or
research technologies. We will undertake our research as an end-to-end
process, driven by close interactions with stakeholders and related
research communities. Our primary stakeholders are DOE-funded SciDAC
projects with specific remote and distributed visualization (RDV) needs.
Generally speaking, the unfulfilled needs can be characterized as the
ability to perform scientific visualization and data analysis of large
and complex scientific data using remote and distributed resources. Our
project combines research activities from two complementary areas that
will result in new capabilities needed by science programs. The two
research areas are (1) graphics and visualization algorithms targeted
especially for RDV environments, and (2) the infrastructure needed to
effectively deploy these new RDV capabilities. We will work closely with
individual scientific research groups to ensure the technology is well
targeted and to validate the efficacy of the new methods.

[]{#__RefHeading___Toc65574774 .anchor}Project Vision

Scientific visualization -- the transformation of abstract data into
images -- is an integral part of the modern scientific process. Modern
scientific visualization, as well as modern scientific practices, both
face similar challenges posed by increasingly complex "environmental
considerations." Large scientific simulations, which are often performed
at centers with unique computational capabilities, generate data of
immense size and complexity. In order to perform remote visualization of
large scientific data, research scientists must become masters of
several different types of skills. They must become well versed in
site-specific access policies for each different location. They must
have command of an encyclopedic knowledge of which visualization tool is
best for a specific visualization task. They must be able to diagnose
network problems and know whom to call to fix the problem. They must log
in to many different machines with different operating environments and
resource usage policies in order to access, move, and analyze their
simulation data. They must be adept at translating files from their
simulation data format to the format required by their preferred
visualization tool. Then, after these myriad hurdles have been cleared,
they are then at liberty to examine their data and gain insight into the
scientific phenomenon or process being studied.

Our vision is for scientific researchers to be able to perform
visualization anytime, anywhere, using any of a large collection of
resources, and without the need to be master of many different
disciplines. Furthermore, we envision a stable environment for
visualization and data analysis that can be used effectively by all
scientific researchers as well as for visualization research by the
scientific visualization community. We envision high performance
visualization tools having the same ease of use and prevalence as common
office productivity software. We strive for the ability to bring to bear
the combined resources of many remotely distributed computational
resources upon a single scientific visualization task that exceeds the
capabilities of any one platform.

# Project Benefits

We are fortunate to have a close relationship with a number of
computational science projects that have substantial visualization needs
as well as missions important to the Office of Science. Phil Colella,
the PI for the APDEC SciDAC ISIC[^1], and his team are located at
Berkeley Lab. We have worked closely with his team on Adaptive Mesh
Refinement (AMR) visualization in recent years (\[1\], \[2\], \[3\]).
Robert Ryne, the PI for the Accelerator SciDAC ISIC[^2], is also located
at Berkeley Lab. We have collaborated with Ryne's project on several
different projects that aim to improve visualization capabilities and
capacities specifically for Accelerator design applications as well as
for general-purpose high-energy particle physics applications.[^3] Ryne
is in the process of updating his codes to use particle-in-cell methods
on adaptive grids through collaboration with the APDEC project. Other
collaborative partners include computational astrophysicists (Peter
Nugent and Julian Borrill, both of LBNL) who are affiliated with SciDAC
efforts. We also have an ongoing set of relationships in the Fusion
community: the National Fusion Collaboratory SciDAC Project[^4], led by
David Schissel at General Atomics in San Diego, which uses considerable
computing resources at NERSC and has a substantial user base at the
Princeton Plasma Physics Laboratory in Princeton, NJ.

Our past experience with these and similar projects has led to the
appreciation and understanding of a set of needs for remote and
distributed visualization capabilities. Our proposal targets a
multi-faceted research agenda that is intended to make remote and
distributed visualization a reality for these projects initially, and
the broader scientific community later in time.

The work we propose provides the following major benefits:

1.  The work will provide new graphics and visualization capabilities
    acutely needed by DOE scientific research programs. Characteristic
    needs include (1) the ability to perform visual data analysis of
    very large scientific data (2) from arbitrary locations (3) by
    collaborative teams of researchers (4) using a heterogeneous
    collection of software tools (5) located on distributed resources.
    These capabilities are simply not available from any source -- they
    must be conceived, designed and implemented.

2.  Cost-efficiency and leveraging of resources. The cost of providing
    such new capabilities is considerably less than if our customers set
    out to develop the technologies on their own. Additionally, the
    proposal PIs have a substantial amount of accumulated expertise in
    developing algorithms and software in the stated research focus
    areas. Therefore, it would be difficult to reproduce such a
    combination in a domain-specific application project, such as one of
    the SciDAC ISICs.

3.  Close coupling to the domain scientists (our customers) focuses our
    development efforts on relevant capabilities and accelerates the
    transfer of RDV expertise to their applications. Likewise, our
    project benefits from direct input from customers in this
    complementary relationship.

4.  Reduced duplication of effort across visualization research
    programs. A stable visualization test and deployment environment
    reduces redundancy by providing a consistent software infrastructure
    for research that cuts across different visualization
    subdisciplines.

5.  Modular, interoperable interfaces and data models for visualization
    components that will prolong use of legacy applications in new
    environments and implicitly foster compatibility (interoperability)
    between research prototypes and other infrastructure components,
    such as data I/O libraries and legacy visualization tools.

Beyond the needs of computational science customers, the work we propose
will have a substantial impact on the visualization community itself.
Our work will create the opportunity for increased interaction between
visualization programs by fostering interoperability of the different
technologies that comprise a "visualization application." Increased
sharing and reuse of technology will reduce duplicated effort amongst
visualization programs and create synergistic software development
relationships that span scientific domains and institutional boundaries.

# Background and Motivation

Prior to the 1990s, visualization was usually performed by applications
that made subroutine calls to specialized graphics or visualization
APIs. The NCAR Graphics library is an early example of such an API.

During the 1990s, visual programming and dataflow environments
simplified the process of creating visualization applications. These
environments (\[4\], \[5\]) consisted of (1) a collection of software
components, (2) a "flow executive" that would manage execution and data
flow between the components, and (3) an interface whereby a user created
a visualization application. These environments were immensely
successful for several reasons. First, a non-programmer could build
applications using the point-and-click interface. With a small amount of
training, scientists could create custom visualization applications on
high performance platforms. Second, these environments were extensible:
developers could add new software components that would run in these
environments, effectively extending the capabilities of the system.
Third, communities of developers and users began to form around these
environments. Developers could share source code for custom modules, and
users could share module networks that realize a visualization
application.

During the later part of 1990s, these environments began to suffer from
unanticipated limitations caused by increasing size and complexity of
scientific data, and the trend towards distributed-memory high
performance computing architectures. Underlying each dataflow
environment is a notion of a data model, which in turn was used to
enforce strong data typing between the software components that comprise
the toolset for the package. The evolution of multigrid approaches for
simulations, such as Berger-Colella Adaptive Mesh Refinement \[6\],
posed insurmountable challenges for these packages. The data and
execution models were too brittle to accommodate the demands posed by
large and complex data types. As data increases in size, other design
limitations of the data flow environments were encountered, namely the
fact that it became impossible to load one, or multiple copies of data
into available core memory of a single workstation. As distributed
processing environments became a necessity to address performance issues
in scientific computing, dataflow visualization applications proved to
be ill suited for use outside of their single-machine design. These
problems range from mere access to remote hosts, to more troublesome
issues like scatter-gather and load balancing operations common in
parallel applications.

Late in the 1990s and early 2000s, the visualization research community
has approached the subject of RDV in a somewhat oblique and piecemeal
fashion. Parallel processing frameworks and APIs like MPI[^5] and
OpenMP[^6] provide the means for creating what is logically a single
application from a number of disjoint by functionally identical building
blocks. Parallel processing frameworks do not provide the means to
implement a dataflow architecture that is representative of the
processing steps in a visualization application. As a result, remote and
distributed visualization applications are typically highly customized
for a specific set of architectures, data sets, specific environments
and uses. A trend of divergence within the visualization community is
partly fueled by the need to show immediate research results (e.g.,
through publications), which is most quickly accomplished through the
use of ad-hoc and highly specialized approaches to RDV. Such an approach
occurs at the expense of generality and longevity. In other words,
research programs are rewarded for publications rather than working and
long-lived tools.

Our aim is not to change the way research programs are run, but instead
to do a service to the research community as well as the user community.
The vision of a stable framework for RDV helps the research community by
providing the infrastructure upon which prototypes are built: data
modeling and I/O tools, fundamental rendering tools, the ability to
quickly deploy parallel and distributed software, and so forth. The
vision helps the user community by establishing a stable target
infrastructure where research prototypes can be quickly used on new
datasets.

In 1998, Berkeley Lab held a workshop on the subject of interoperability
between the various visualization programs within DOE[^7]. The purpose
of the workshop was to identify ways in which visualization efforts
within DOE could work together more effectively. The workshop findings
identified a progressive sequence of ways in which programs could work
together. The lowest common denominator is a common data model. Relying
upon the data model are software components that are able to read and
write data in these formats. Frameworks are used to coordinate the
execution and data flow between software components. In addition to
sharing tools and toolsets, the workshop indicated the need to share
resources, such as facilities, as well as the need for a stronger,
well-defined direction within the community to help guide the overall
programmatic direction, and help reduce divergence.

The work described in this proposal is intended to alter the divergent
course within our community by performing research and development in
several technology areas that will tend to foster cooperation. The
elements of our research agenda take steps to meet the objectives of
interoperability listed in the 1998 workshop. The elements of our
research agenda cover many technical areas, ranging from data modeling
to network encodings through use of nascent Grid technology to gain
access to remote resources in a secure and standardized fashion. The
many topics are representative of the fact that RDV encompasses many
technologies. Our fundamental approach is to be inclusive of existing
visualization technology, so that the substantial DOE investment in
visualization technology can be used effectively in an RDV context.

We must quickly point out that the scope of our vision is large. For
example, the "common data format" problem has been tackled before with
inconclusive results. The common data format problem is in itself very
difficult, but it is but a subset of our vision. We do not believe we
have any "silver bullets" to solve these large problems that have
heretofore been unsolvable by large teams of motivated researchers.
Instead, we are going to focus our activities on projects that will
directly benefit several different DOE scientific research programs. Our
aim is to show some early initial successes in key areas -- common data
models, consistent component interfaces, execution framework suitable
for use in heterogeneous and wide-area environments -- and use those
early successes as "condensation nuclei" around which other
visualization efforts will begin to coalesce.

This approach is similar to how wide-area networks are built. Early
attempts to build wide-area networks focused on building large trunk
lines, then slowly adding branches to local organizations. This approach
was prone to failure due to difficulties establishing the "last-mile
connectivity" to sites. These lessons have led to a more innovative and
scalable approach: build the local loops first, then slowly connect the
local loops with trunk lines into broader, more comprehensive networks.
This approach works better because the local loops ensure greater local
control of the infrastructure and establish the wider area connections
only after policy issues have been resolved.

# Research Focus Areas

Our research agenda focuses upon the technologies needed to realize
effective and pervasive RDV. As stated in the Introduction, we envision
two broad categories of research: fundamental graphics and visualization
technology, and the infrastructure necessary to deploy new visualization
and graphics technologies in an RDV context. These two broad topics
unfold into a number of subtopics. The topic of deployment covers many
separate areas, including: (1) a framework for efficient and orderly
execution of component-based visualization software tools organized as a
single application that spans distributed resources; (2) shaping and
adoption of a data model and data management techniques that form the
core of a collection of component-based visualization tools; (3)
performance modeling and optimization of visualization applications
composed of component-based visualization tools deployed in a
distributed fashion; (4) visualization- and graphics-centric encodings
for high performance network transport between distributed components;
(5) exploration and use of emerging Grid services to mediate and broker
access to and use of remote and distributed resources for the purposes
of visualization; (6) realization of working applications built from
components and frameworks. The thrust area of research of fundamental
graphics and visualization algorithms for remote use requires an
intimate understanding of and close relationship with the RDV
infrastructure and deployment methodology. The two thrust areas --
deployment and fundamental graphics and visualization algorithms -- are
closely related parts of the broader topic of RDV.

## DiVA Framework 

The Distributed Visualization Architecture (DiVA) is a collection of
technologies that are used to realize deployment of RDV applications
built from software components. The framework serves the dual role of
application delivery and as a testbed for conducting fundamental RDV
research. The framework combines and uses the elements of the research
topics discussed in other sections in this proposal.

The high level objective is to be able to quickly create RDV
applications from components, and to deploy the application on
distributed resources. DiVA will make it possible to quickly prototype
and test new technologies, such as new visualization algorithms, new
network encodings for data transport between distributed components, new
Grid services for resource location and use, and so forth.

As a framework, DiVA encompasses many of the architectural
characteristics of its framework predecessors. These include execution
model, data model, and software components. DiVA goes far beyond its
predecessors by providing features not imaginable or realizable during
the mid-1990s. DiVA will enable a collection of software components to
execute in pipelined fashion over distributed resources using Grid
technology. Grid technology is used to address issues related to
security, resource discovery, launching of and fair-use policies for
remote and distributed components. Also, unlike its predecessors, DiVA's
technologies are separable -- a developer may use only portions of DiVA
in order to realize an RDV application. A primary objective for DiVA is
the ability to allow use of any visualization technology in an RDV
setting, regardless of origin. This is an important point -- DiVA does
not exclude existing visualization technology. Instead, it serves as a
catalyst to use existing technology in new ways, and is therefore a
complement to existing DOE visualization investments.

It is our opinion, based upon experience with many visualization
technologies, that the demands posed by RDV environments far exceed what
can possibly be provided by any existing visualization framework. This
opinion is substantiated by the numerous projects over the years that
have attempted to modify an existing framework to an RDV environment,
but have fallen short of their objectives.

Research Objectives:

1.  Define the boundaries of DiVA in terms of provided services, define
    potential interfaces with existing visualization and deployment
    tools, define a potential "distributable product" for use by the
    visualization community, define the mechanism for a community-based
    effort.

2.  Framework-mediated execution of a visualization application composed
    of multiple, distributed components.

3.  Explicit support for execution patterns required by multiresolution,
    hierarchical, data-parallel, and progressively refined data models.

4.  Interface with pipeline optimization technology to dynamically
    change the placement of components onto resources depending upon
    performance data or performance estimates.

5.  Expand the scope of the framework to include "arbitrary"
    visualization tools. For example, use the DiVA data mover to provide
    data to a remote visualization application, such as AVS/Express
    running at PPPL using data stored at NERSC.

6.  Expose key framework capabilities as Grid Services in order to take
    advantage of Grid software infrastructure being deployed across the
    DOE as part of the DOE Science Grid. The precise set of framework
    capabilities to expose as Grid Services will be defined as the
    project evolves.

7.  Demonstrate the efficacy of the DiVA technology by providing new
    capabilities to DOE scientific research programs.

## Domain-Neutral Multiresolution Data Model 

Component-based visualization tools require a domain-neutral abstraction
layer to enable interoperability between their many disparate
implementations. New technology must be developed to support this
requirement because current data model layers employed by visualization
frameworks lack a robust representation for multiresolution datasets.
The lack of such a data model has generally impeded growth of
multiresolution visualization techniques, and has resulted in many
unique and highly specialized implementations that do not lend
themselves well to widespread use. We will limit the scope of this
potentially large problem by focusing upon needs of visualization and
data analysis, along with a clear definition of those grid types that
can be supported. We do not expect to be able to provide multiresolution
coverage for every imaginable grid type. Therefore, we will use the
needs of our "customers" -- collaborative DOE-funded science projects --
to keep the effort focused. We will also limit the scope of our effort
by leveraging and reusing existing technology, wherever feasible.

The multiresolution data model will be implemented as a layer on top of
existing and proven technology to perform low-level, platform neutral
I/O and provide an elementary scheme for expressing complex data
representations. The most promising existing technology for such
capabilities is HDF5[^8], a widely used I/O library for scientific data.
The multiresolution metadata model and API will be layered upon existing
technologies to provide domain-neutral, multiresolution access to
scientific data.

Research Objectives:

1.  Define common multiresolution capabilities needed by a subset of
    SciDAC efforts, with a particular focus on visualization and data
    analysis.

2.  Define, implement and test interfaces between multiresolution
    metadata layer and underlying data I/O technology. We will use HDF5
    as a starting point, and expand into netCDF on a time-permitting
    basis over the course of the project.

3.  Define, implement and test processes that facilitate derivation of
    multiresolution metadata layer from existing scientific data,
    without copies, for the purposes of extending the shelf-life of
    these data sets by ensuring compatibility with next-generation
    visualization tools. This activity will be done in concert with
    domain scientists and other visualization researchers through a
    series of workshops and collaborative development activities.

4.  Explore and exploit use of the new multiresolution data model with
    evolving indexing approaches used for large data sets located on
    tertiary storage.

5.  Test efficacy of design through use in component-based,
    multiresolution-aware visualization tools deployed in a remote and
    distributed context.

6.  

7.  Use novel network encodings and protocols to achieve the most
    efficient use possible of available bandwidth when data is moved
    across the network between distributed software components. Our aim
    is to reuse research efforts as much as practical, although some
    research and development in this area may be needed to support the
    needs of RDV.

8.  

Component-Based Visualization Pipelines: Automated Component Placement
and Dynamic Optimization

Many visualization component architectures use a dataflow pipeline
paradigm for their distributed execution model. Such an organization has
provided the underpinnings of the most successful visualization
packages, such as AVS, IBM's Data Explorer, and the Visualization
Toolkit (VTK). In their simplest form, all components that comprise the
application pipeline reside on a single platform. Early attempts at
distributed execution required a user to manually partition components
into remote and local groups. In such a partitioning, some of the
components run locally, and the other collection runs on a remote host.
The partitioning is static, which means the partitioning never changes
in response to changing application needs or environmental conditions.

However, there is no a-priori way to select an optimal (or even
tolerable) pipeline distribution at startup without first being able to
accurately predict the performance of the individual components. Because
no such performance models exist, placement of components to date has
been entirely heuristic. When we have a performance model associated
with each component that comprises a pipeline, we can create a composite
parametric model that enables us to accurately predict the overall
performance of the pipeline and therefore can make quantifiably optimal
selections for the distribution of the components across resources. The
ability to optimally place components will be a core requirement for
resource selection mechanisms needed for effective Grid computing. Note
that "pipeline elements" consist not only of individual software
components, but also include the "pipes" through which data flows
between components.

However, it is also true that performance of a visualization pipeline
varies dynamically as a function of input data, user parameters, and
environmental conditions (e.g., competing but measurable traffic on
shared network links). Therefore, the performance model must
continuously estimate pipeline performance of visualization
"applications" consisting of multiple software components deployed in a
heterogeneous and distributed environment. A resilient pipeline flow
executive must be able to obtain a quantitative measure or estimate of
performance for each element in the distributed application, but should
also be able to modify at run time the use of resources in a
heterogeneous and distributed environment to achieve optimal
performance. The algorithm must be able to take into account the cost of
redefining the dataflow topology as well as with the potential impact to
user interactivity requirements \[20\].

Efforts to support such dynamic redeployment of Grid-based simulation
codes have been hampered by the cost of migrating the entire simulation
state to a new resource in response to a "contract violation" \[21\].
The costs of migrating the application in this situation often far
exceed the benefits of the improved performance of the new deployment.
Visualization offers a unique opportunity to support greater flexibility
in dynamic job migration because, unlike simulation codes, each time a
new dataset is loaded from disk, the entire visualization pipeline state
is "flushed," thereby permitting dramatic re-distribution of the
components without the need to migrate large amounts of state or
checkpoint-restart information.

Accurate pipeline performance prediction also requires accurate models
of the network performance and current capacities. There are a number of
network monitoring systems in development, such as LBL's Network
Characterization Service (NCS)[^9] and Web100[^10], which use indirect
methods (testing at the endpoints of the network) to estimate network
topology, link bandwidth, and available capacity. We have also been
investigating direct methods of extracting these parameters using tools
like Network Ferret[^11]. A SciDAC project -- the Bandwidth Estimation
project[^12] -- aims to provide the means for applications to detect and
respond to changes in network performance characteristics. We will
investigate the relative advantages and disadvantages of these different
methods for the purpose of accurately estimating pipeline performance.

There is an opportunity to become more deeply involved with the Grid
community to define a set of "standards" whereby Grid-enabled components
can "publish" performance estimates. An external process can then use
the published performance estimates to derive an estimate of aggregate
performance. We will use performance estimates of the pipeline
components to determine if a given pipeline distribution is optimal, or
if the distribution should be reconfigured to improve performance. The
performance model must include the cost of resource migration and
balance it against the nature of the contract violation that has
prompted the need for redeploying the pipeline. We will investigate the
use of empirical/history-based, extrapolated, and (if possible) fully
parameterized performance models for some comparatively simple
visualization components and the network resources that tie them
together.

This information will likely take the form of a new data schema that is
published through a standard Grid information service like the Globus
MDS (\[7\], \[8\]). We will also explore a way to expose the performance
modeling information through custom MDS information providers and
whether such data can be tied into the emerging CIM data model[^13] for
resource description. Such information will be useful to virtually any
distributed application architecture. Therefore, this development work
has the potential to impact a wide array of Grid computing activities
that are far outside its original scope.

Research objectives.

1.  Define scope of variables needed to accurately model the performance
    of a visualization application consisting of software components
    deployed in a wide-area fashion.

2.  Investigate ways to obtain data for the variables. Foster
    collaborative ties with other appropriate research groups to define
    the means to obtain data.

3.  Investigate network performance monitoring and modeling frameworks
    in development by the network research community. Accurate and
    current network performance estimation is a critical component of
    the process of pipeline performance estimation.

4.  Using performance data and performance estimates, perform dynamic
    pipeline reconfiguration and optimization.

5.  Establish a collaborative relationship with the Global Grid forum
    ACE (Advanced Collaborative Environments)[^14] working group to
    foster two-way exchange of ideas and standards for performance
    modeling and estimation. Visualization is a key Grid application
    that will be used to test and refine performance measurement and
    optimization concepts.

## Visualization- and Graphics-Centric Network Transport

Improving the end-to-end performance of network-intensive applications
is often considered the domain of networking protocol specialists.
However, the LBNL Visualization group has dominated the annual
Supercomputing Bandwidth Challenge event since its inception three years
ago, beating teams that have a considerably narrower focus on networking
protocols and communication performance. The key reason for our success
is a very strict focus on the application performance, while still
working closely with network protocol and performance specialists, such
as Mike Bennett of LBLnet and Brian Tierney of the LBL distributed
systems group. This application-driven research orientation is essential
for ensuring that results target the most critical impediments to
performance delivered to end-users.

To this end, we are identifying areas where the information encoding of
off-the-shelf TCP/IP protocols can be specialized to improve performance
metrics that are important to interactive visualization application
\[9\]. Our Bandwidth Challenge application has demonstrated the
effectiveness of custom UDP transport encodings that dramatically
increase the throughput of remote data accesses for a distributed
visualization application; peaking at 16.8 Gigabits/sec for a
global-scale distributed program during our SC02 run. While our
accomplishments focus on movement of raw scientific data, the same
principle can and should be extended to new classes of data that have
different delivery requirements and performance metrics. Improved use of
network resources will benefit the interactive response and data
throughput of an RDV application.

Reliable transport is built atop concept of a stream encoding, which is
defined as a contiguous stream of bytes that arrive in exactly the order
they are sent. However, in the case of interactive applications, such as
visualization, the stream paradigm is overly restrictive. In order to
maintain the notion of a stream of consecutive bytes, TCP does a
considerable amount of buffering on both the sending and the receiving
side of a data connection. This means that the operating system may
delay delivery of information that has arrived over the network for a
considerable amount of time, leading to "jerky" and inconsistent
responsiveness for end-users of the RDV system. Such responsiveness is
often incorrectly attributed to network "jitter;" a Layer-2 phenomenon
that leads to delay inconsistencies on the order of microseconds to
milliseconds. The perceived jitter is induced by the buffering employed
by the protocol (a Layer-3 phenomenon) rather than the transport
infrastructure, and can lead to delays that are orders of magnitude
worse than that of network jitter (milliseconds to seconds).

Solving the "jitter problem" requires implementing custom UDP protocols,
or a TCP-like reliable protocol, that eliminates Layer-3 jitter by
eliminating the dependency on stream encoding. An unreliable, non-stream
protocol is simply UDP. We refer to our non-stream reliable protocol as
RIPP (Reliable Independent Packet Protocol). RIPP guarantees delivery of
packets, but presents them to the application the moment they are
received rather than buffering them. Such delivery greatly improves
interactivity, but requires considerable attention to the manner in
which information is encoded since the receiving end must be able to
accept the data in any order. Extending data models and transport
mechanisms to accommodate out-of-order arrival of data is a deep
research issue.

Another area of considerable interest is the development of non-stream
encodings that are fault-tolerant or fault-resilient. A loss-tolerant
encoding has the potential to eliminate buffering on the sending side as
well as the receiving side, and will greatly reduce end-to-end transport
latencies. However, such encodings can only be considered in the context
of the application that they support (there is no general-purpose
method). For instance, a simplistic approach to fault-tolerance in a
continuous frame-oriented data transport situation is to use data from
the previous timestep to fill in lost data. A more advanced method would
use wavelet or frequency domain encoding of the data to hide loss as
missing spatial frequencies (similar to JPEG compression). For transport
of geometric models, we can look at packet encodings that support
progressively refined meshes using triangle bisection. Such techniques
make packet loss less visually distracting and reduce the need for long
latency-increasing buffer queues on the receiving side. The UDP
transport can still be reliable, but must be able to present information
immediately upon arrival rather than buffering them until the missing
packets arrive. Such direct presentation is essential for supporting
interactive applications, whereas protocols like TCP, SABUL[^15], and
GridFTP[^16] are entirely oriented around throughput at the expense of
interactivity.

Research Objectives

1.  Finalize and release the RIPP (Reliable Independent Packet Protocol)
    for reliable transport of data as well as an efficient
    rate-controlled packet engine used for strictly unreliable
    transport.

2.  Implement and release replacement-based and wavelet-based encodings
    for buffer-oriented block data. This is data for a finite-length
    buffer that is continuously updated by replacement over a network
    buffer. A 1D analogy is an audio stream, and a 2D analogy is a movie
    stream. Our interest lies in generalizing these methods to deal with
    3D data (both single-step and time-series) in a variety of domain
    decompositions. This work will directly support Visapult, but will
    be packaged so that it can be applied to other applications
    developed by other groups.

3.  Explore non-stream progressive encodings for geometry data. This
    will have application for Chromium \[10\] and even multicasting of
    geometric data for use in shared environments like the Access Grid.
    We will look at order-independent encodings for the triangle
    bisection progressive-mesh-refinement method that are suitable for
    use in both reliable and unreliable non-stream transport methods.

## Grid-Enabled Remote and Distributed Visualization Infrastructure

There is a need to make Grid technology more accessible to visualization
application developers. Grid middleware, and its APIs and nomenclature,
present a moving target that can be confusing and difficult to use. Due
to the potential value of Grid services and the concerted effort within
the DOE to deploy a Grid infrastructure across all sites, the level of
effort needed to Grid-enable RDV is warranted. We intend to work through
the Global Grid Forum[^17] to create appropriate abstractions for
high-level visualization services using Grid service description
languages. In addition, we investigate and possibly adapt application
APIs and design patterns that will make the use of Grid middleware more
accessible to visualization application developers. Consequently,
visualization frameworks and applications will become more pervasive in
the Grid community. "Grid-enabled visualization" is woven into many
aspects of our research, roadmap including the data modeling and network
transport enhancements described in other sections of this document. In
this section we outline our plans for improving integration of
information services, data management, and security into a set of APIs
that make it much easier for visualization developers to incorporate
Grid features into RDV frameworks and applications.

The first step in effective use of distributed components is being able
to locate resources, which in this case are remotely located
visualization components. Current distributed visualization applications
use a variety of essentially ad-hoc methods to manage distributed
objects that comprise the application. We will explore ways to use the
Globus MDS/GIIS/GRIS (\[7\], \[8\]) and the emerging CIM standards[^18]
as a directory interface to locate components and services that comprise
a distributed visualization applications in a given Virtual
Organization. The GIIS can be used to index statically placed software
components or mediate access to shared services. The GRIS on each host
can also be used as a registry for active components for the
visualization system's application scheduler (the "flow executive") to
aid in efficient resource selection and facilitate automatic restart of
components that die or become unresponsive due to hardware or software
failure. We also intend to integrate these capabilities with resource
discovery and management systems that manage the intermediate network
resources.

The next issue is having the ability to launch remote components. RDV
applications need some mechanism to launch the various distributed
components on remote computational resources. The current security model
employed by RDV applications typically relies on SSH or the Globus GRAM
\[11\] to launch software components that subsequently establish socket
connections between peer components to create the working application.
However, this last step of establishing network connections
("unregulated" socket connections) contradicts well-established security
practices for network applications. Namely, the individual socket
connections lack an authentication that ensures that some rogue
application cannot connect to that socket and thereby "hijack" a
component of a RDV distributed application. Nothing prevents such a
breach from occurring in any of the popular visualization applications
within the DOE that are capable of limited distributed operation,
including SCIrun, VTK, and even AVS Express.

It is essential to create a lightweight API for establishing
authenticated TCP and UDP information streams between distributed
components that makes use of existing Globus GSI security infrastructure
and X.509 certificates. Such an API would make it much easier to
retrofit existing visualization frameworks to securely use the Grid
infrastructure, and facilitate future visualization application
development using Grid infrastructure.

The research agenda for these grid services is as follows;

1.  Participate in the ACE-WG on security requirements for Grid
    applications, and focus on creating an API that implements simple
    SSL authentication (but not encryption) of newly established TCP
    sockets. We will also work with Keith Jackson of the DOE Science
    Grid on a method to enable authentication of UDP packet streams
    using some form of shared secret through a peer TCP control socket.

2.  Develop schemas for new visualization-oriented information providers
    for the MDS. These include information providers for identifying and
    locating shared services like off-screen rendering pipes and storage
    resource managers; information providers for locating availability
    of components that can be used for distributed applications on
    remote machines and their estimated performance; and finally an
    information provider that can track running components of an active
    distributed visualization application and relaunch them
    automatically if they die or otherwise become unresponsive. Such
    fault-tolerance is essential for interactive applications that
    employ such widely distributed resources.

3.  Follow the development of the DMTF Common Information Model (CIM)
    data schema that will be deployed with Globus GT3 and modify our
    work on MDS information providers to mirror this data model as much
    as possible.

4.  Work closely with experts in Storage Resource Management, such as
    Arie Shoshani at LBL, on closely integrating data management systems
    and their associated preprocessing agents with the visualization
    workflow. In particular, interactive visualization benefits
    considerably from very expensive data reorganizations that must
    necessarily be performed offline. Storage Resource Management[^19]
    systems have a concept of data processing "agents" that can be
    invoked as part of the file retrieval process, and can perform
    various visualization-related tasks such as data-reorganization,
    feature detection and data subsetting/extraction based on user
    criteria. We will work closely to provide visualization-oriented
    data filtering agents that can be plugged into storage management
    frameworks being developed under the auspices of DOE and EU Data
    Grid efforts.

## Multiresolution, Latency-Tolerant Remote & Distributed Visualization Algorithms. 

Any visualization tool implementation is intimately tied to a data
model. Thus, the multiresolution capabilities of any visualization tool
depend, to a large degree, upon the robustness of the underlying data
model, and also upon the effectiveness of the fundamental algorithm for
visually representing abstract data. Latency-tolerant algorithms are
those that achieve interactivity even in the presence of network-induced
latency in the input data flow.

The most common approaches for remote visualization typically use one of
two strategies: (1) render all data remotely, send images over the
network to the viewer, or (2) send subsets of data over the network,
where all visualization and rendering is performed locally. More
recently, hybrid approaches have emerged, in which visualization results
(e.g., geometry) are sent over the network for rendering on the viewer
platform. Another hybrid approach that achieves latency tolerance is
Visapult \[12\], which uses Image-Based rendering (IBR) techniques
combined with traditional geometry-based techniques \[13\]. Other points
along this continuum have yet to be fully explored, such as combining
height-fields with image-maps to achieve 2.5 degree of freedom
interactions. IBR and IBR-geometry techniques have proven useful for
fast rendering of extremely complex scenes, such as outdoor and
natural-light environments \[14\], but have yet to be fully exploited
for use in scientific visualization.

Current and expected computational environments, with petaflop computing
capabilities, will beget the need for extremely high performance in
visualization and data analysis. One promising approach for interactive
exploration of such large datasets relies on multiresolution data
models, and complementary multiresolution-aware visualization tools.
Implementations of these tools will by necessity make extensive use of
parallelism and pipelining to achieve maximum possible throughput. Other
promising approaches will leverage "shared rendering", or "co-rendering"
pipelining between large parallel visualization engines and powerful
desktop workstations. In these approaches, the most bandwidth-intensive
visualization is performed close to the data source, while the most
latency-sensitive part of the visualization and rendering is performed
on the desktop workstation. Multiresolution methods provide a framework
for progressive refinement through the visualization pipeline, from data
source to the user's desktop machine. These new approaches hold much
promise to help meet challenging visualization needs arising from
scientific research projects.

Research Objectives:

1.  Multiresolution visualization of particle-based datasets.

2.  Image based/multistage shared rendering techniques that are amenable
    to pipeline parallelism.

3.  Efficient distributed memory and loosely coupled parallel algorithms
    for clusters and wide-area computing environments.

4.  Algorithms that take advantage of novel data encodings for transport
    or storage including space-filling curves, multilevel-block
    decompositions, data encodings that support progressive data
    transmission, and non-stream (discrete) transport encodings that
    tolerate loss gracefully (i.e., algorithms designed for UDP
    transport).

5.  Algorithms that take advantage of emerging data representation
    methods to accelerate performance, such as so-called "bitmap index"
    approaches for representing results of boolean query operations
    \[15\].

Realizing Grid-based RDV Applications

The Grid and its associated middleware is useful as a concept only if
its services can be used to create the illusion that all of the
resources are centralized or local to the user\'s workstation.
Paradoxically, the most successful distributed applications on the Grid
will be those where the user is not aware that she is operating using
distributed components. It is essential that the DiVA be completely
decoupled from the user interface paradigm so that a variety of
interface methodologies can be supported; from desktop computers, to
CAVEs, to AccessGrids, to web browsers on cell phones. For this purpose,
there are a number of technologies for separating the interface
definition from the back-end logic that implements those operations.
Such separation is the foundation of the service specifications,
examples of which include Open Grid Services Architecture (OGSA)[^20]
and the Web Services Resource Framework[^21] (WSRF), and interface
specifications like the Web Service Definition Language (WSDL)[^22] and
gSOAP[^23]. For performance oriented applications, it may be necessary
to look at other specification methods like the Common Component
Architecture's[^24] SIDL \[16\] (an analogue to CORBA), or the relative
merits of a more custom system built atop the RIPP protocol The
separation between presentation and implementation is so critically
important that we will make it a fundamental element feature of our
research activities from the very start.

Despite such decoupling of the GUI from the compute engine, some
centralized resources are required for coordinating resource access and
providing a unified view to the user's Virtual Organization. The
motivation for producing a web-based Grid Portal interface to
visualization services and Problem Solving Environments (PSE's) is
derived from the desire to hide the complex software architectures and
access to remote resources behind a single point of presence that is
accessible through comparatively simple client-side interfaces

A portal is a single point of presence, typically hosted on the web that
can be customized for a particular user and remembers particular aspects
of the customizations regardless of of the location of the user's
access. Yahoo and HotMail are typical consumer-oriented examples of this
capability, and are the origin of the meaning for the term "portal."
Regardless of your location when you login to the URL of these portals,
you get access to the same view of your personalized environment and
data (ie. in the case of HotMail, your email). Like the Yahoo example, a
Grid portal, such as the nascent LBL VisPortal \[22\], provides a
personalized environment that enables access to sophisticated
distributed data analysis tools that support the NERSC HPC environment
and its users. We feel that a web-portal architecture is a very natural
way to provide the location independence, consolidation of tools, and
the level automation necessary to support HPC activities. A portal is
just one way to present a collection of services, but it is highly
effective.

Portal Implementation:

The LBL VisPortal (visportal.lbl.gov) is our Alpha-release deployment of
a web portal GUI that leverages off technology that was originally
developed for e-commerce applications. The architecture utilizes a
commercial-grade Apache web server, and offers SSL encryption using a
site certificate from the DOE Science Grid Certificate Authority.
Running side-by-side with the web server is a TomCat Java Server
Pages[^25] (JSP) engine, which offers the means to manage automation in
an elegant and easily maintainable fashion. JSP allows us to directly
execute methods of server-side java beans, rather than the typical
CGI-script methodology of parsing the state of form elements
individually after an http \'post\' event. JSP is much easier to
maintain than Perl- or Python-based CGI scripts, and has considerable
industry support in commercial products like IBM WebSphere and Sun's
Java Enterprise Edition products. Java beans directly call Java
CoG[^26], a pure Java implementation of the Globus toolkit in order to
extend the automation to Grid resources. The basic interface between JSP
and the Java CoG is part of a framework developed by Jason Novotny of
LBL, called the Grid Portal Development Kit[^27] (GPDK). We have used a
tailored version of GPDK to act as the interface to a number of services
important to visualization and data analysis.

As part of our future activites, we will investigate emerging
JSP-portlet-based architectures like the GridSphere[^28] project. The
GridSphere technology is compliant with the Portlet Java Specification
Request (JSR-18), an open standard for Web Portal Development and
Frameworks. Portals developed in this framework will be portable across
many commercial and Open Source web application servers, including IBM
WebSphere, Sun JavaOne and Tomcat Jakarta. GridSphere is the logical
successor to GPDK, the current substrate for our portal and therefore
requires a critical technology review.

Future Directions:

The implementation we described is focused entirely on the
lowest-common-denominator user interface implementation; DHTML. We chose
to attack the low-end first in order to maximize access to these new
visualization services on the Grid. A truly full-featured portal design
will require a more comprehensive, multi-faceted approach to interface
design. We divide these interfaces into 3 categories of web-portal
clients:

*Thin Clients*: A pure DHTML interface to the portal automation. This is
the least-common-denominator GUI, and works on virtually any web
browser. In particular, DHTML is great for wireless devices and other
means of remote access.

The user interfaces afforded by DHTML are not as expressive as
traditional windowing GUIs. In order to work in this restrictive
environment, we are exploring new GUI paradigms, like visualization
spreadsheets \[17\] and simultaneously focusing on very specialized
application interfaces. One of the most common complaints by users,
emphasized repeatedly in the NERSC Visualization Greenbook \[18\] is
that most visualization tool user interfaces are far too complex for
scientists. General-purpose dataflow GUI's in particular offer a
powerful programming paradigm that is exciting for visualization
experts, but daunting for the scientists. The feedback we get from many
of our users indicates they would prefer a "customized dashboard"
interface to each data analysis task they would want to perform. Over
time, they may be enticed to invest the effort to learn how to use more
powerful user interfaces, but our initial focus should be to create a
lower the threshold of investment required for new users of Grid
services

Specialization greatly simplifies the requirements of the user interface
and also addresses issues of user interface complexity. For example, one
DHTML application is a single-web-page interface for converting image
files into movies. Another page is used only to transfer files between
machines to stage them for interactive data analysis. The limited scope
of dashboard interfaces can greatly improve user performance of basic,
commonly used tasks by eliminating the confusion and inefficiency
associated with deeply nested menus.

We are also working closely with UC Davis researchers to explore new GUI
paradigms that are more efficient in high-latency environments like the
web. The visualization spreadsheets interface allows a user to see a
matrix of exploration possibilities in a single mouse-click. This
approach takes advantage of the capabilities of the web's hyperlinks;
allowing a user to drill down into their dataset by simply clicking on
the element of the spreadsheet that offers them the most interesting or
compelling view of their dataset.

*Fat Clients*: This is a standalone application that interacts with the
portal automation. The portal acts as a simple broker of remote and
distributed resource management on behalf of the application. The portal
also enables synchronization of state information between the standalone
clients and their Peer thin-client interfaces.

The Fat Client paradigm involves client machine running a standalone
application program that interoperates with the portal. For instance,
the client application may be a full AVS application that has a specific
module to communicate with the portal to locate remote data files or
compute resources. In this situation, the portal merely coordinates
direct connections between desktop application and Grid distributed data
and/or services. This approach is most effective if existing
applications like AVS or SCIrun are retrofitted with Portal-aware
components, rather then developing complete systems from scratch.

*Slender Clients*: A Java applet or a signed Java application, or
simplified binary is downloaded to the client side on every invocation
and communicates directly with the portal. Slender clients offer GUI
capabilities and interactive performance, but also require more
configuration effort on the users part to ensure an appropriate Java
runtime environment is installed on their workstation. The slender
client is neither thin nor fat, but in its own category.

In order to support a more flexible, capable, and responsive user
interface, we must eventually move beyond DHTML. However, in doing so,
we open up a host of complex issues relating to effective deployment of
tools, software porting efforts, and even the burden of installation
support for potential users of the software. The slender-client paradigm
attempts to reduce the complexity of this issue by making the client
side of the application (the part that must run on the user's
workstation) as small as possible. This means the client exists only to
provide a GUI, and to act as a conduit for displaying the images and
geometry computed by a remotely located collection of back-end
components. A slender client provides the sophisticated widgets and
capabilities offered by modern GUI frameworks. Yet, the client code is
simple enough that maintenance and porting issues are minimized.

A more robust implementation of the slender client methodology will use
applets or signed applications that are downloaded to the client machine
upon invocation. An advanced system could use the browser identity
string to determine the client type and upload a native binary
implementation to the client. This method offers a novel way around
current versioning and deployment problems that afflict multi-component
distributed applications because the latest version of the client
component is downloaded on every use. The effect can be compared to the
Auto-Update feature included in modern operating systems, but the update
is always fully automated.

Slender clients still require some investment on the part of the user,
albeit minimal. For instance, if we choose Java for our slender-client
deployments, we have found that the JDK implementations commonly
incorporated into web browsers tend to be either very old, or very weak
implementations. Worse yet, we\'ve seen situations where minor revision
changes in JDK technology incorporated in the browsers can result in
major changes to the Java rendering engine or changes to the GUI API
(particularly with older versions of SWING). In the worst case, you must
download swing.jar with your application in order to guarantee some
degree of consistency in an otherwise uncontrolled environment. These
inconsistencies are well documented in trade magazines and have been
experienced first hand by many experienced Java developers.

Research Agenda

1.  Deploy initial sample portal-hosted applications on the LBL/NERSC
    Visportal system, which is based on GPDK and the Java CoG.

```{=html}
<!-- -->
```
6.  Explore use of the next generation JSP-Portlet-based portal
    architectures using GridSphere.

7.  Explore new GUI paradigms that are more amenable to web-based
    visualization and interaction techniques, which aim to overcome the
    limitations of HTML.

8.  Develop a simplified protocol to support communication of user
    interface events between a slender client and its various backend
    components. We will investigate the relative merits of using
    interface definition languages like SIDL or WSDL as opposed to
    direct specification using a protocol like gSOAP.

9.  Seek ways to maximize use of the thin and slender client interfaces
    for visualization tasks.

10. Explore means for better integrating more complex fat-client
    interfaces, like dataflow GUIs, so that they can communicate with
    and interoperate with the Portal system.

## Remote and Distributed Visualization and Scientific Data Management

The close proximity of the LBNL Scientific Data Management (SDM) and
Visualization groups creates a unique opportunity for multidisciplinary
collaboration. The LBNL Visualization group\'s research focus is on
remote and distributed visualization architectures, frameworks and
algorithms. Their work is predicated upon the existence of data made
available to visualization components. The SDM group\'s research focus
is on efficient storage, retrieval and classification mechanisms that
provide the means for researchers to perform analysis on data. The
project described below is intended to bring both groups together on a
focused, science driven project that will result in new software
capabilities acutely needed by researchers.

From a high level, the functional requirements for the application,
which we\'ll call "Dex" (short for dexterous), center about visual data
exploration and analysis. One conundrum often encountered in scientific
research is the quest for "interesting features" hidden in data, but the
inability to qualitatively or quantitatively define "interesting."
Visual presentation of multivariate data is often the first step on the
road towards characterizing interesting features. As characteristics of
interesting features are discovered -- an "ignition kernel" in a
combustion data set might be the confluence of several variables;
temperature in excess of a certain threshold, presence of certain
species; flow rate constrained within a range -- these factors can all
be codified into queries exercised upon massive datasets. A "hit"
returned from the query indicates the presence of conditions
characteristic of "interesting features." Further analysis on
interesting regions of data is then often domain specific. Combustion
researchers will in some cases strive to understand the curvature of the
flame front, which is an "interesting feature," with respect to
variation in the temperature along the flame front. That kind of
relationship offers insights into the combustion process.

The visualization challenges in realizing Dex are directly in line with
the LBNL Visualization Group\'s research effort funded under the Base
Program grant entitled "Algorithms, Frameworks and Toolsets for High
Performance, Remote and Distributed Scientific Visualization." First,
rather than create a monolithic application, we will build up an
application from individual components. Such a focus on components and
an execution framework is the subject of DiVA, the Distributed
Visualization Architecture. Second, Dex will integrate the bitmap based
indexing and searching technologies developed by the SDM group, thereby
making the technology readily available to scientific research programs
in a software application. The SDM group has conducted the initial work
to demonstrate the effectiveness of their bitmap-based technologies on
structured uniform grids \[19\]. Our design goals will build on these
earlier initial bitmap technology accomplishments to accommodate use on
Adaptive Mesh Refinement (AMR) grids. Our existing relationships with
LBNL's Combustion Programs (Bell) and the APDEC SciDAC (Colella) will
serve to provide data sources and feedback directly from scientific
research programs using adaptive and multiresolution methods. Another
focus area is remote and distributed visualization in a Grid-enabled
fashion. The SDM group has early success in this area in the form of the
Earth Systems Grid. Multivariate visualization is an ongoing
visualization challenge, and was directly identified as a critical need
in the 2002 NERSC Visualization Greenbook Workshop \[18\].

Research Agenda (NB -- the following research items encompass both
Visualization and Scientific Data Management; no attempt is made to
partition them into separate categories since both SDM and Visualization
will be working on the project together):

1.  Workflow analysis -- how do science researchers go about "hunting
    for interesting things in data?" Translate workflow patterns into
    software components and processes. Define and specify features
    (components) needed to implement a workflow within the context of
    Combustion research.

2.  Simultaneous visualization of multiple variables in 2D and 3D.

3.  Generate and display statistical information for user-defined
    regions of interest.

4.  Use statistical information and conditions resulting from data
    values or ranges to define regions of interest in time varying AMR
    data.

5.  Adapt bitmap-indexing technology for use with time varying AMR data.

6.  Interface specification and implementation between visualization and
    scientific data management components/processes.

7.  Develop labeling strategies to track regions of interest over time
    for use in fast indexing applications.

8.  Search and track derived fields.

9.  Produce usable tools that don't require an expert.

# Statement of Work

## Overview

This statement of work defines the work objectives -- the tasks,
milestones and allocation of responsibilities for the combined efforts
of the two collaborating institutions: the Visualization Group at
Lawrence Berkeley National Laboratory and researchers from the
Visualization and Graphics Research Laboratory at the Center for Image
Processing and Integrated Computing (CIPIC) at University of Davis
(UCD).

The two institutions have a history of successful and fruitful
collaborations that spans almost six years. In the past, the
collaboration has used a model whereby LBNL has been the funding
recipient, and has in turn hired graduate students from UCD. The UCD
students have been assigned focused research projects defined by LBNL
staff, and have been supervised jointly by LBNL staff and UCD faculty.
This Statement of Work represents an attempt to change how the financial
aspect of a collaborative LBNL-UCD project is executed, with funding for
UCD activities made available directly to UCD rather than via a
subcontract through LBNL.

## Approach

The two institutions serve uniquely different but complementary roles on
this project. Generally speaking, the role of LBNL is to define a
research and development agenda that results in technology capabilities
that help advance scientific research, and to ensure that research
results -- to which it plays a central and catalyzing role -- are
deployed in a manner as to have impact on scientific research. In
contrast, the role of UCD is to undertake focused, relatively short-term
and potentially high-risk research projects that dovetail with the needs
of DOE-sponsored scientific research. The ongoing collaborative effort
between LBNL and UCD represents a successful division of duties that are
partitioned along these lines.

The visualization research activities discussed in the main proposal
body will be applied to four specific DOE-funded science project areas.
These projects include: (1) the Advanced Computing for 21^st^ Century
Accelerator Science and Technology SciDAC project, led by Rob Ryne of
LBNL; (2) several related Fusion projects that include Center for
Extended MHD Modeling led by Steve Jardin of PPPL and the National
Fusion Collaboratory led by Dave Schissel of General Atomics; (3)
several astrophysics projects that include the study of multi-group
flux-limited diffusion radiation-hydrodynamic equations, led by Doug
Swesty at the State University of New York, Stony Brook, which is part
of the larger SciDAC-funded TeraScale Supernova Initiative led by Tony
Mezzacappa at Oak Ridge National Laboratory; (4) combustion and reactive
chemistry in projects that include the Applied Partial Differential
Equations SciDAC ISIC led by Phil Colella of LBNL and related projects
run by John Bell of LBNL.

Our research efforts are intended to provide new and acutely needed
visualization capabilities to these specific science projects. The
project-centric focus ensures that the visualization research is
grounded by the needs of science. To streamline the presentation of
yearly objectives, we omit references to specific science projects.

## Tasks, Milestones, Deliverables

### FY05

-   DiVA Framework Requirements Specification. Identify characteristics
    and needs common in workflow patterns across multiple science
    projects (LBNL). Undertake research to survey, compare and contrast
    current component-based technologies (UCD). Define requirements for
    framework interoperability of components in visualization and data
    analysis requirements (LBNL, UCD).

-   Multiresolution Data Modeling for Visualization. Define
    generalization of "multiresolution data models" across multiple
    science projects. Develop early prototype multiresolution data model
    atop existing technology (LBNL).

-   Dynamic Pipeline Performance Modeling and Optimization. Develop
    performance model for individual components (UCD, LBNL). Develop
    visualization techniques for component and pipeline performance
    models (UCD).

-   Visualization and Graphics Centric Network Transport. Requirements
    specification and early prototype or adaptation of existing
    technology to provide high performance network transport of graphics
    and visualization data (LBNL).

-   Grid-Enabled Remote and Distributed Visualization Infrastructure.
    Participate in Grid community activities to ensure that
    visualization needs are heard. Define requirements for cataloguing
    and locating remote visualization resources. (LBNL).

-   Multiresolution and Latency Tolerant Graphics and Visualization
    Algorithms. Multiresolution visualization and rendering of
    particle-based data (UCD, LBNL). Research and early prototypes of
    high-dimensional and information visualization technologies (LBNL,
    UCD). Design, specification and early prototype of multiresolution,
    stream-based "sort-last" rendering architecture for use in remote
    and distributed contexts (LBNL).

-   Scientific Data Management and Visualization. Workflow analysis and
    prototype implementation of visualization workflow combining Storage
    Resource Management (SRM), Bitmap-Index technology and visualization
    components (LBNL). Multivariate visual data analysis and mining
    using SRMs as data sources (LBNL, UCD).

### FY06

Unlike the FY05 objectives, which are reasonably clear as of the time of
this writing, specific tasks and assignments of responsibilities for
tasks in the out years are more amorphous. The spirit of allocation
across institutional boundaries is quite clear, however. The items
listed below for FY06 and FY07 are high-level objectives that are a
natural bridge between FY05 activities and long-term project objectives.
We anticipate the milestones for the out years will be revised and more
clearly defined as the project proceeds.

-   DiVA Framework. "Compete" alternative approaches for defining
    component interfaces and execution control. Demonstrate
    "componentizing" legacy applications. Demonstrate cross-institution
    distributed execution using legacy applications and new
    visualization components. Early specification for execution flow
    control with multiresolution data models. Define "DiVA Compliant" in
    terms of component interfaces, execution control and data model.

-   Multiresolution Data Modeling for Visualization. Use a
    domain-neutral multiresolution data model for visualization in
    multiple science project applications. Study and develop new
    multiresolution techniques for "information visualization" data
    modeling. Where feasible, adapt multiresolution concepts to existing
    visualization applications. Requirements and specification for
    multiresolution "visualization pipelines."

-   Dynamic Pipeline Performance Modeling and Optimization. Characterize
    pipeline performance using serial, parallel and distributed
    components. Expand performance model to include cost of data
    movement and process migration. Where feasible, obtain measurements
    of non-component pipeline performance from Grid services.
    Requirements and specification for pipeline optimization. Early
    research into multiparticipant pipelines.

-   Visualization and Graphics Centric Network Transport. Architecture
    and preliminary implementation of highly efficient network transport
    of graphics and visualization data between components in a single
    pipeline, as well as components in multiparticipant pipelines.

-   Grid-Enabled Remote and Distributed Visualization Infrastructure.
    Grid-deployment of multiresolution particle visualization and
    "hybrid rendering" components. Grid-brokered discovery and launching
    of distributed visualization components. Deploy application-specific
    interfaces (e.g., "dashboards") that encapsulate common tasks in
    simple workflows.

-   Multiresolution and Latency Tolerant Graphics and Visualization
    Algorithms. Requirements, specifications and where feasible, early
    implementations of families of multiresolution-aware visualization
    components. Demonstrate multi-participant wide-area visualization of
    multi-terabyte data sets. Separately demonstrate use of data
    streamed from running codes as well as precomputed and stored in
    data caches.

-   Scientific Data Management and Visualization. Interactive and
    offline region selection using SRM-brokered constructs of both
    spatial and non-spatial data. Visualization of results using
    prototype components.

### FY07

-   DiVA Framework Requirements Specification. Finalize "DiVA
    Compliance" specification. Define minimum set of tools, components
    and technologies that embody a "DiVA Reference Implementation."
    Demonstrate multi-participant, multi-site distributed visualization
    using "DiVA-compliant" components and applications.

-   Multiresolution Data Modeling for Visualization. Demonstrate
    multiresolution data modeling and visualization in multiple
    applications as applied to multiple science projects.

-   Dynamic Pipeline Performance Modeling and Optimization.
    Characterization of performance of visualization pipelines that
    include wide-area distributed and parallel components. Preliminary
    implementation of wide-area and parallel dynamic pipeline
    optimization.

-   Visualization and Graphics Centric Network Transport. Release
    infrastructure library for efficient network transport of
    visualization and graphics data. The library is one of the elements
    of the reference DiVA implementation.

-   Grid-Enabled Remote and Distributed Visualization Infrastructure.
    Adapt DiVA Reference Implementation and demonstration to use Grid
    services to locate and launch components.

-   Multiresolution and Latency Tolerant Graphics and Visualization
    Algorithms. Research into use of hybrid techniques combining
    Image-Based Rendering acceleration, multiresolution geometry and
    progressive streaming of data for latency-tolerant rendering of
    visualization results.

-   Scientific Data Management and Visualization. Integrated feature
    identification, tracking and visualization of multi-terabyte data
    sets.

## 

## 

## 

1.  

## 

1.  

```{=html}
<!-- -->
```
1.  

##  

##  {#section-6 .unnumbered}

## 

## 

## 

-   
-   
-   
-   
-   
-   

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

-   
-   

## Bibliography

\[1\] J. Chen, Z. Bai, B. Hamann , and T. Ligocki, *A Normalized-Cut
Algorithm for Hierarchical Vector Field Data Segmentation*, In
Proceedings of 2003 SPIE/IS&T Visual Data Exploration and Analysis
Conference , San Jose, CA, January 2003. LBNL-52024.

\[2\] O. Kreylos, G. Weber, W. Bethel, J. Shalf, B. Hamann, and K. Joy,
*Remote Interactive Direct Volume Rendering of AMR Data*, LBNL-49954.

\[3\] T. Ligocki, *Implementing a Visualization Tool for Adaptive Mesh
Refinement Data using VTK*, in Proceedings of Visualization Development
Environments 2000. LBNL-45394.

\[4\] C. Upson, T. Faulhaber Jr,, D. Kamins, D. Laidlaw, D. Schlegel, J.
Vroom, R. Gurwitz, R., A. van Dam, et al. *The Application Visualization
System: A Computational Environment for Scientific Visualization*. IEEE
Computer Graphics & Applications, 9 (4), July 1989, 30-42.

\[5\] J. Rasure and M. Young, *An Open Environment for Image Processing
Software Development*, 1992 SPIE/IS&T Symposium on Electronic Imaging,
SPIE Proceedings Vol. 1659, February 14, 1992.

\[6\] M. Berger and P. Colella, *Local Adaptive Mesh Refinement for
Shock Hydrodynamics*, J. Comp. Phys., vol. 82, pp. 64-84, May 1989.

\[7\] K. Czajkowski, S. Fitzgerald, I. Foster, C. Kesselman. *Grid
Information Services for Distributed Resource Sharing*. Proceedings of
the Tenth IEEE International Symposium on High-Performance Distributed
Computing (HPDC-10), IEEE Press, August 2001.

\[8\] S. Fitzgerald, I. Foster, C. Kesselman, G. von Laszewski, W.
Smith, S. Tuecke. *A Directory Service for Configuring High-Performance
Distributed Computations.* Proc. 6th IEEE Symposium on High-Performance
Distributed Computing, pp. 365-375, 1997.

\[9\] J. Shalf and W. Bethel, *Cactus and Visapult: An Ultra-High
Performance Grid-Distributed Visualization Architecture Using
Connectionless Protocols*. To appear in the IEEE CG&A Mar/April 2003
special issue on Grid Visualization. LBNL-51564.

\[10\] G. Humphreys, M. Eldridge, I. Buck, G. Stoll, M. Everett and P.
Hanrahan. *WireGL: A Scalable Graphics System for Clusters."* In
Proceedings of ACM Siggraph 2001, July 2001, Los Angeles, CA, USA.

\[11\] K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W.
Smith, S. Tuecke. *A Resource Management Architecture for Metacomputing
Systems*. Proc. IPPS/SPDP \'98 Workshop on Job Scheduling Strategies for
Parallel Processing, pg. 62-82, 1998.

\[12\] W. Bethel, B. Tierney, J. Lee, D. Gunter, S. Lau, *Using
High-Speed WANs and Network Data Caches to Enable Remote and Distributed
Visualization*, in Proceedings of IEEE/ACM 2000 Conference on
Supercomputing (CDROM), November 04-10, Dallas, Texas, USA.

\[13\] K. Müller, N. Shareef, J. Huang, R. Crawfis, "IBR-Assisted Volume
Rendering." In Late Breaking Hot Topics Proceedings of IEEE
Visualization 99, IEEE Press, October 27-29, 1999, San Francisco, CA,
USA. pp 5-8.

\[14\] W. Chen, J. Bourget, M. Chu, R. Grzeszczuk, "Light Field Mapping:
Efficient Representation and Hardware Rendering of Surface Light
Fields." In ACM Transactions on Graphics, 21(3), Proceedings of ACM
Siggraph 2002, July 2002.

\[15\] M-C. Wu and A. P. Buchmann, *Encoded Bitmap Indexing for Data
Warehouses*, Proceedings of the 14th IEEE International Conference on
Data Engineering, 1998, pp. 220-230.

\[16\] E. Vega, *SIDL: Design and Specification of a Compositional
Object Oriented Model for Generalized Data Management*. PhD Thesis,
Computing Laboratory, University of Kent at Canterbury, December 1996.

\[17\] T.J. Jankun-Kelly, O. Kreylos, J. Shalf, Kwan-Liu Ma, B. Hamann,
K. Joy, E.W. Bethel. *Deploying Web-based Visual Exporation Tools on the
Grid*. To appear in the IEEE CG&A Mar/April 2003 special issue on Grid
Visualization. LBNL-51563.

\[18\] B. Hamann, W. Bethel, H. Simon, and J. Meza. *NERSC Visualization
Greenbook: Future Visualization Needs of the DOE Computational Science
Community Hosted at NERSC*. Submitted to International Journal of High
Performance Computing Applications. November, 2002. LBNL-51699.

\[19\] K. Wu, W. Koegler, J. Chen and A. Shoshani, Using Bitmap Index
for Interactive Exploration of Large Datasets. In Proceedings of SSDBM
2003. Preprint as LBNL-52535.

\[20\] D. Angulo, R. Aydt, F. Berman, A. Chien, K. Cooper, H. Dail, J.
Dongarra, I. Foster, D. Gannon, L. Johnsson, K. Kennedy, C. Kesselman,
M. Mazina, J. Mellor-Crummey, D. Reed, O. Siervert, L. Torczon, S.
Vadhiyar, and R. Wolski. "Toward a Framework for Preparing and Executing
Adpative Grid Programs." IPDPS, 2002.

\[21\] G. Allen, D. Angulo, I. Foster, G. Lanfermann, C. Liu, T. Radke,
E. Seidel and J. Shalf. "The Cactus Worm: Experiments with Dynamic
Resource Discover and Allocation in a Grid Environment." International
Journal of High Performance Computer Applications and Supercomputing,
15(4), Winter, 2001.

\[22\] C. Siegerist, P. Shetty, W. Bethel, T.J. Jankun-Kelly, O.
Kreylos, K.L. Ma, J. Shalf. "VisPortal: Deploying Grid-Enabled
Visualization Tools Through a Web-Portal Interface." Wide Area
Collaborative Environments Workshop, Seattle, WA., June 2003.
LBNL-52940.

# Biographical Sketches

## Bethel (LBNL)

![](media/image3.png){width="5.996527777777778in"
height="6.916666666666667in"}

![](media/image4.png){width="5.997222222222222in"
height="7.763194444444444in"}

## Shalf (LBNL)

![](media/image5.png){width="5.992361111111111in"
height="5.700694444444444in"}

![](media/image6.png){width="5.992361111111111in"
height="5.0055555555555555in"}

## Hamann (UCD)

**Bernd Hamann**

Co-Director, Center for Image Processing and Integrated Computing

University of California

One Shields Avenue

Davis, CA 95616-8553

\(530\) 754-9157; fax: (530) 752-8894

hamann@cs.ucdavis.edu

**Professional Preparation**

Technical University of Braunschweig, Germany Computer Science B.S. 1985

Technical University of Braunschweig, Germany Mathematics B.S. 1986

Technical University of Braunschweig, Germany Computer Science M.S. 1988

Arizona State University, Tempe, Arizona Computer Science Ph.D. 1991

**Appointments**

2003-Present Associate Vice Chancellor for Research, University of
California, Davis

2000-Present Professor, Department of Computer Science, University of
California, Davis

1999-Present Faculty Computer Scientist, National Energy Research
Scientific Computing Center (NERSC), Lawrence Berkeley National
Laboratory (LBNL)

1998-Present Participating Guest Researcher, Center for Applied
Scientific Computing (CASC), Lawrence Livermore National Laboratory
(LLNL)

1997- 2000 Associate Professor with tenure, Department of Computer
Science, University of California, Davis

1997--Present Co-Director, Center for Image Processing and Integrated
Computing (CIPIC), University of California, Davis

1995-Present Acting Associate Professor, Department of Computer Science,
University of California, Davis

1995-Present Adjunct Professor, Department of Computer Science,
Mississippi State University

1994-1995 Associate Professor, Department of Computer Science,
Mississippi State University

1991-1995 Research faculty member, Scientific Visualization Thrust at
the NSF Engineering Research Center for Computational Field Simulation,
Mississippi State University

1991-1994 Assistant Professor, Department of Computer Science,
Mississippi State University

1988-1991 Research/teaching assistant, Department of Computer Science,
Arizona State University

1985-1986 Programmer, Stahlwerke Peine-Salzgitter AG, Salzgitter,
Germany

**Distinctive Awards And Nominations**

Associate editor, IEEE Transactions on Visualization and Computer
Graphics, 1999-2003

Papers co-chair and proceedings co-editor, IEEE Visualization
conferences 1999 and 2000

Member, Committee of Visitors (evaluation of NSF's Advanced
Computational Research Program), 2001

NSF Career Award, NSF, 1996

Hearin-Hess Distinguished Professor of Engineering, College of
Engineering Faculty Award Program, Mississippi State University, May
1995

NSF Research Initiation Award, NSF, 1992

**Publications**

Bertram, M., Duchaineau, M.A., Hamann, B., and Joy, K.I. (2003)
Generalizing lifted tensor-product wavelets to irregular polygonal
domains, in Post, F.H., Nielson, G.M. and Bonneau, G.-P., eds., Data
Visualization: the State of the Art, Kluwer Academic Publishers,
Norwell, Massachusetts, pp. 289-300

Bonnell, K.S., Duchaineau, M.A., Schikore, D.R., Hamann, B. and Joy,
K.I. (2003), Material interface reconstruction", IEEE Transactions on
Visualization and Computer Graphics 9(4), pp. 500-511

Bremer, P.-T., Edelsbrunner, H., Hamann, B. and Pascucci, V. (2003), A
multi-resolution data structure for two-dimensional Morse-Smale
functions, in: Turk, G., van Wijk, J.J. and Moorhead, R.J., eds., IEEE
Visualization 2003, IEEE Computer Society Press, Los Alamitos,
California, pp. 139-146

Co, C.S., Heckel, B., Hagen, H., Hamann, B. and Joy, K.I. (2003),
Hierarchical clustering for unstructured volumetric scalar fields, in:
Turk, G., van Wijk, J.J. and Moorhead, R.J., eds., IEEE Visualization
2003, IEEE Computer Society Press, Los Alamitos, California, pp. 325-332

Hamann, B., Bethel, E.W., Simon, H.D. and Meza, J.C. (2003), NERSC
\`\`Visualization Greenbook\'\' \-- Future visualization needs of the
DOE computational science community hosted at NERSC", The International
Journal of High Performance Computing Applications 17(2), SAGE
Publications Ltd., pp. 97-123

Kreylos, O., Max, N.L., Hamann, B., Crivelli, S.N. and Bethel, E.W.
(2003), Interactive protein manipulation, in: Turk, G., van Wijk, J.J.
and Moorhead, R.J., eds., IEEE Visualization 2003,  IEEE Computer
Society Press, Los Alamitos, California, pp. 581-588

Mahrous, K.M., Bennett, J.C., Hamann, B. and Joy, K.I. (2003), Improving
topological segmentation of three-dimensional vector fields, in:
Bonneau, G.-P., Hahmann, S. and Hansen, C.D., eds., Data Visualization
2003 (Proceedings of \`\`VisSym \'03\'\'), Association for Computing
Machinery, New York, New York, pp. 203-212

Scheuermann, G., Hamann, B., Joy, K.I. and Kollmann, W. (2003),
Localizing vector field topology, in: Post, F.H., Nielson, G.M. and
Bonneau, G.-P., eds., Data Visualization: the State of the Art, Kluwer
Academic Publishers, Norwell, Massachusetts, pp. 19-35

Vivodtzev, F., Linsen, L., Bonneau, G.-P., Hamann, B., Joy, K.I. and
Olshausen, B.A. (2003), Hierarchical isosurface segmentation based on
discrete curvature, in: Bonneau, G.-P., Hahmann, S. and Hansen, C.D.,
eds., Data Visualization 2003 (Proceedings of \`\`VisSym \'03\'\'),
Association for Computing Machinery, New York, New York, pp. 249-258

Weber, G.H., Scheuermann, G. and Hamann, B. (2003), Detecting critical
regions in scalar fields, in: Bonneau, G.-P., Hahmann, S. and Hansen,
C.D., eds., Data Visualization 2003 (Proceedings of \`\`VisSym
\'03\'\'), Association for Computing Machinery, New York, New York, pp.
85-94

###### Collaborators {#collaborators .unnumbered}

B.C. Banks, Mississippi State University

E.W. Bethel, Lawrence Berkeley National Laboratory

R. Crawfis, Ohio State University

M.A. Duchaineau, Lawrence Livermore National Laboratory

G. Farin, Arizona State University

H. Hagen, University of Kaiserslautern, Germany

K.I. Joy, University of California, Davis

T.J. Ligocki, Lawrence Berkeley National Laboratory

N.L. Max, University of California, Davis/Lawrence Livermore National
Laboratory

V. Pascucci, Lawrence Livermore National Laboratory

S.P. Uselton, Lawrence Livermore National Laboratory

G. Brunnett, Chemnitz University of Technology, Germany

H. Mueller, University of Dortmund, Germany

**Graduate Students and Post-Doctoral Fellows**

M. Bertram, University of Kaiserslautern, Germany

M.L. Chen,

A.B. Forgang

B. Heckel, Purple Yogi, Mountain University

G. Hong

S.R. Jolly

F. Kuester, University of California, Irvine

L. Linsen, University of California, Davis

S.I. Lundy, Intel

J. Meyer, Mississippi State University

C. Nuber, University of California, Davis

G. Patel

H.-C Rodrian, University of Kaiserslautern, Germany

E.S. Rosenthal, SONY Imageworks

G. Scheuermann, University of Kaiserslautern, Germany

I Shih, ACME, Taiwan

A.M. Tesdall, University of California, Davis

A.E. Thompson

J.M. Westerburg

S. Xie

###### Graduate Advisors {#graduate-advisors .unnumbered}

H. Hagen, University of Kaiserslautern, Germany

G.M. Nielson, Arizona State University

## Joy (UCD)

Dr. Kenneth I. Joy

Professor

Center for Image Processing and Integrated Computing

Department of Computer Science

University of California

Davis, CA 95616-8562

Phone: (530) 752-1077

FAX: (530) 752-4767

Email: joy@cs.ucdavis.edu

Professional Preparation:

University of California, Los Angeles Mathematics B.A. 1968

University of California, Los Angeles Mathematics M.A. 1972

University of Colorado, Boulder CO Mathematics Ph. D. 1977

Professional Experience:

1999 -- Present Professor, Department of Computer Science, University of
California, Davis

1997 -- Present Research Faculty, Center for Image Processing and
Integrated Computing, University of California, Davis, September

1987 -- 1999 Associate Professor, Department of Computer Science,
University of California, Davis

1983 -- 1987 Assistant Professor, Division of Computer Science,
University of California, Davis

1980 -- 1983 Assistant Professor, Department of Mathematics, University
of California, Davis

1977 -- 1980 Assistant Professor, Department of Mathematics, Northern
Michigan University, Marquette, Michigan

1973 -- 1977 Teaching Assistant, Department of Mathematics, University
of Colorado, Boulder

1968 -- 1971 Systems Analyst, Systems Development Corporation, Los
Angeles, California

Publications Closely Related to this Project:

Kathleen S. Bonnell, Mark A. Duchaineau, Daniel A. Schikore, Bernd
Hamann and Kenneth I. Joy, "Material Interface Reconstruction," *IEEE
Transactions on Visualization and Computer Graphics,* 2003, (to appear).

Martin Bertram, Daniel E. Laney, Mark A. Duchaineau, Charles D. Hansen,
Bernd Hamann, and Kenneth I. Joy, \"Wavelet Representation of Contour
Sets,\" Proceedings of IEEE Visualization 2001, Ertl, T., Joy, K.I. and
Varshney, A., eds., IEEE Computer Society Press, Los Alamitos,
California, 303-310.

Bertram, M., Duchaineau, M.A., Hamann, B., and Joy, K.I., \"Bicubic
Subdivision-Surface Wavelets for Large-Scale Isosurface Representation
and Visualization,\" Proceedings of IEEE Visualization 2000, T. Ertl, B.
Hamann and A. Varshney, eds., Salt Lake City, Utah, October 9-13, 2000,
389-396.

Bonnell, K., Duchaineau, M.A., Schikore, D., Hamann, B., and Joy, K.I.,
\"Constructing Material Interfaces from Data Sets Containing Volume
Fraction Information,\" Proceedings of IEEE Visualization 2000, T. Ertl,
B. Hamann and A. Varshney, eds., Salt Lake City, Utah, October 9-13,
2000, 367-372.

Trotts, I.J., Hamann, B. and Joy, K.I., "Simplification of Tetrahedral
Meshes with Error Bounds", *IEEE Transactions on Visualization and
Computer Graphics*, Vol. 5, No. 2, 224-237.

**Other Significant Publications:**

Gieng, T.S., Hamann, B., Joy, K.I., Schussman, G.L., and Trotts, I.J.,
"Constructing Hierarchies for Triangle Meshes," *IEEE Transactions on
Visualization and Computer Graphics*, Vol. 4, No. 2, April-June 1998,
145-161.

La Mar, E.C., Duchaineau, M.A., Hamann, B., and Joy, K.I.,
\"Multiresolution Techniques for Interactive Texture-Based Rendering of
Arbitrarily Oriented Cutting Planes\"in van Liere, R., Hermann, I., and
Ribarsky, W., eds., Proceedings of VisSim \'00 \-- The Joint
Eurographics and IEEE TVCG Conference on Visualization, Amsterdam, The
Netherlands, May 2000, 105-114.

Joy, K.I., and Duchaineau, M.A., "Boundary Determination for Trivariate
Solids," *Proceedings of the 1999 Pacific Graphics Conference*, Seoul,
Korea, October 5-7, 1999, 82-91.

La Mar, E.C., Hamann, B., and Joy, K.I., "Multiresolution techniques for
Interactive Texture-Based Volume Visualization*," Proceedings of IEEE
Visualization '99*, San Francisco, CA, October 25-29, 1999. 355-362.

MacCracken, R.A and K.I., Joy, "Free-form deformations with lattices of
arbitrary topology," *Proceedings of SIGGRAPH \'96*, Annual Conference
Series, August 1996, 181\--188.

Joy, K.I., "Utilizing Parametric Hyperpatch Methods for Modeling and
Display of Free- Form Solids," *International Journal of Computational
Geometry and Applications* 1, No. 4, (December 1991), 455-472.

**List of Collaborators**

Bernd Hamann, University of California, Davis, Nelson Max, University of
California, Davis, Kenneth L. Verosub, University of California, Davis,
Mark Duchaineau, Lawrence Livermore National Laboratory, Joerg Meyer,
Mississippi State University, Hans Hagen, University of Kaiserslautern,
Germany, Helmut Pottmann, Technical University of Vienna, Austria, Eric
La Mar, Lawrence Livermore National Laboratory, Falko Kuester,
University of California Irvine.

Graduate and Post-Graduate Advisors and Advisees

Advisor: Dr. Lawrence Baggett, University of Colorado, Boulder;
Advisees: C.F. Borges, M. Bhetanabhotla, B. Cabral, J. Chen, M.
Duchaineau, D. Fox, R. Gross, S. Janus, L. Jurrus, F. Kuester, R.
MacCracken, R. Moster, M. Miller, J. Reus, S. Stoneking, C. Madrigal, E.
Jurrus, F. Kuester, E. La Mar, K. Bonnell, A. Pomeranz, S. Konkle, J.
Conkey; number of graduate students 30 (including current advisees);
post-doctoral fellows sponsored: 4 (J. Meyer, G. Scheuermann, L. Linsen,
C. Nuber).

## Ma (UCD)

![](media/image7.png){width="5.9944444444444445in"
height="7.158333333333333in"}

![](media/image8.png){width="5.99375in" height="6.542361111111111in"}

Facilities and Resources

The work described in this proposal will be conducted on site at
Lawrence Berkeley National Laboratory. For the purposes of conducting
research activities, we have full and unlimited access to the following
equipment:

-   Desktop workstations consisting of single- and dual-processor x86
    hardware, Linux OS, nVidia-based graphics accelerators.

-   SGI Onyx 2000, eight R10K MIPS processors, (four at 250Mhz, four at
    195Mhz), 5 GB of RAM, dual InfiniteReality2 graphics pipes, 1TB of
    attached storage, and dual Gigabit Ethernet NICs.

-   A six-node "white box" cluster of dual Athlon systems equipped with
    nVidia NV25 graphics accelerators and gigabit Ethernet.

-   Access to a 3x2 tiled projector array connected to the SGI and
    "white box" cluster via video switches.

-   Single tile "Powerwall" display consisting of a Christie Digital
    Mirage 2000 projector for high-resolution, field-sequential
    stereo-capable, single pipe visualization.

-   Tiled projection array consisting of six Barco Atlas XVGA projectors
    arranged into a three by two configuration, providing an aggregate
    resolution of 3840x2048 pixels. These display tiles are shared with
    LBL's Access Grid node, which is located in building 50B.

Nearby in the NERSC center, we have constrained access to:

-   The NERSC IBM SP RS/6000, named *Seaborg*, is a distributed memory
    machine with 2,944 compute processors. Each processor has a peak
    performance of 1.5 GFlops. The processors are distributed among 184
    compute *nodes* with 16 processors per node. Each node has between
    16 and 64 GBytes of memory.

-   HPSS is an archival storage system. At NERSC, we currently have
    approximately 18 Terabytes of disk cache and 2 petabytes of tape
    storage.

-   The NERSC Visualization Server, escher.nersc.gov, allows NERSC users
    to perform visualizations from remote locations. Escher is a Silicon
    Graphics Onyx 3400 with 12 600 MHz IP35 processors, 24 GB memory, 2
    Infinite Reality 4 graphics pipes, 3 gigabit Ethernet interfaces,
    and a 4TG, RAID-5 disk array.

```{=html}
<!-- -->
```
-   
-   

Lawrence Berkeley Laboratory is home of ESnet, the DOE's production
network for scientific research. LBL has an OC-48 connection to ESnet as
of early 2003.

# Current and Pending Support

## LBNL Visualization Group

###### Current Support {#current-support .unnumbered}

1.  "Adaptive Mesh Refinement Data Visualization." Bethel (LBNL), Shalf
    (LBNL), Ligocki (LBNL), Hamann (UC Davis). 10/1/2001 through
    9/30/2004. \$450K/yr.

2.  NERSC Program, Production Visualization. Provides for 2.5 full-time
    equivalent in staffing as opposed to a specific dollar amount.

###### Pending {#pending .unnumbered}

1.  NERSC Program, Production Visualization. Provides for 2.5 full-time
    equivalent in staffing as opposed to a specific dollar amount.

## Visualization and Graphics Research at UC Davis

### Hamann

![](media/image9.png){width="5.996527777777778in"
height="6.376388888888889in"}

![](media/image10.png){width="5.997916666666667in"
height="6.261805555555555in"}

![](media/image11.png){width="5.999305555555556in"
height="6.270833333333333in"}

### Joy

[CURRENT]{.underline}

Title: Effective and Efficient Segmentation Frameworks for Scientific
Data Exploration

Source: NSF

Amount: 250,000 Project Period: 10/1/02 -- 9/30/05

Title: Approximation and Visualization of Curved, High-Degree Finite
Element Data

Source: LLNL

Amount: \$255,648 Project Period: 4/01/02 -- 3/31/04

Title: Massive Data Analysis and Exploration

Source: LLNL

Amount: \$525,000 Project Period: 7/1/02 -- 9/30/04

Title: Interactive Visualization and Exploration of Massive Multisource
Data Streams in Collaboration Environments Using Parallel & Distributed
Computing

Source: NSF/UC Berkeley

Amount: \$525,000 Project Period: 10/1/01 -- 9/30/04

Title: Interactive Visualization Methods for Exploration and Comparison
of Multi-Billion Base Pair Sequence Data

Source: LBNL

Amount: \$584,000 Project Period: 1/1/02 -- 9/30/04

Title: VISUS: Visualization Streams for Ultimate Scalability

Source: LLNL

Amount: \$300,000 Project Period: 3/16/00 -- 3/15/04

Title: Clustering of Adaptive Mesh Refinement Data and Exploration of
Comparative Genomics Data

Source: LBNL

Amount: \$110,000 Project Period: 1/1/04 -- 9/30/04

[PENDING]{.underline}

Title: Visualization-based Interactive Exploration of Microarray Data

Source: NSF

Amount: \$2,281321 Project Period: 10/01/04 -- 09/30/09

### Ma

**[CURRENT]{.underline}**

Title: PECASE: Parallel Visualization and Interaction Techniques or
Exploring Large Scale Volume Data

Source: NSF

Amount: \$500,000 Project Period: 2/1/00 -- 12/31/04

Title: Multiresolution and Topology-Based Visualization of Large
Scientific Data Sets in Parallel and Distributed Computing Environments

Source: NSF

Amount: \$769,629 Project Period: 6/1/00 -- 5/31/04

Title: Partnership for Advanced Computing Infrastructure, Data Intensive
Computing (with UC San Diego)

Source: NSF/NPACI

Amount: \$375,447 Project Period: 10/1/97 -- 9/30/04

Title: Informatics of Human and Monkey Brain Atlases

Source: NIH

Amount: \$6,605,744 Project Period: 06/01/01 -- 6/30/06

Title: A Metadata-Driven Visualization Interface Technology for
Scientific Data Exploration

Source: NSF

Amount: \$265,054 Project Period: 9/1/02 -- 8/31/05

Title: Advanced Computing for 21^st^ Center Accelerator Science and
Technology

Source: DOE

Amount: \$160,000 Project Period: 8/15/01 -- 8/14/03

Title: ITR for Gleaning Insight in Large Time-Varying Scientific and
Engineering Data

Source: NSF

Amount: \$3,535,325 Project Period: 9/15/03 -- 9/14/08

Title: Interactive Visualization of Large-Mixed Resolution Volume Data

Source: LLNL

Amount: \$150,000 Project Period: 7/1/02 - 6/30/05

Title: Large Graph Visualization for DOW & NNSA

Source: LLNL

Amount: \$51,965 Project Period: 2/6/04 -- 2/8/04

Title: Interactive Visualization for Nondestructive Evaluation

Source: LANL/CARE

Amount: \$45,046 Project Period: 7/1/03 -- 6/30/04

Title: Multi-resolution Interactive Rendering of Large Scientific
Datasets Using and Image Cache

Source: LLNL

Amount: \$16,657 Project Period: 2/18/04 -- 8/30/04

**[PENDING]{.underline}**

Title: Advanced Computing for 21^st^ Century Accelerator Science and
Technology

Source: DOE

Amount: \$258,623 Project Period: 8/15/04 -- 8/14/06

# Appendix A -- Justification For Equipment Funding (Contains Proprietary Information) (Remove this Section Before Sending to UC Davis)

## Summary

This proposal requests for equipment funding to acquire visualization
hardware needed to ensure the success of the DOE-funded visualization
research program at LBNL. We request \$340K to purchase a 16 CPU, 6-pipe
Altix system with a modest RAID for secondary storage. Our request is
motivated by the fact our primary visualization research platform -- an
aging and dying SGI Onyx2 system purchased in 1996 -- is inadequate to
carry out the visualization activities for which we have been funded.
This proposal describes (1) our existing infrastructure and why it is
inadequate for current needs; (2) an overview of our technical
requirements for a visualization research platform; (3) how we intend to
make use of the new system; (4) an overview of the architecture of the
proposed system; (5) alternative systems and architectures we considered
before concluding the Altix was the most appropriate choice and best use
of DOE funds; (6) our purchase and deployment plan, along with plans for
accommodating future needs.

## Background and Existing Infrastructure

The MICS Office in DOE's Office of Science funds the LBNL Visualization
Group to perform research and development in the areas of
multiresolution, high performance, remote and distributed scientific
visualization. The Group's LBNL location and affiliation with the NERSC
center affords unique opportunities for collaborative interactions with
computational scientists. The needs of the scientific computing
community provide the impetus for much of the Group's work, and in turn,
the scientific computing community benefits in the form of tools with
new capabilities that were developed in and tailored for remote and
distributed environments.

One of the hallmarks of many modern computational science projects is
large and complex data. In 2001, the Group collaborated with Dr. Ed
Seidel, then of the Max Planck Institute, who was performing novel
computational simulations to model the gravitational waves generated
during the collision of two black holes. The size and complexity of the
data generated by this particular simulation was profound: the
simulation, characterized by Dr. Seidel as "modest," produced dataset
sizes on the order of tens of terabytes and required use of 1024
processors on NERSC's SP2 system. Whereas a large amount of
computational resources were dedicated to generating the data, the
visualization was performed on a machine with less than 1% of the
computational capacity of the simulation platform.

Our last major equipment purchase was in 1996 -- a full eight years ago
-- for an SGI Onyx2 system, consisting of eight 195MHz CPUs, two
Infinite Reality2 graphics pipes, 5GB of memory, and 400GB of secondary
storage capacity. The current annual maintenance costs on this hardware
exceed its market value, and the system does not have the graphics or
computational power needed to meet our research program's needs, which
are driven by the demands of modern computational science.

Our existing platform falls far short of providing minimal required
resources in all system categories: FLOPs, memory capacity, graphics
capacity, storage capacity and networking capacity are all inadequate.
Due to its age, the system cannot be "upgraded" with the addition of
more processors, graphics hardware and memory. Ideally, a visualization
research platform would have capabilities commensurate with the
simulation platform to "impedance match" between the data producer and
consumer. In our view, the amount of funding required to procure such a
platform that would be used for visualization research is not practical.
As a compromise, our proposal targets "middle ground" in terms of cost,
capability, expandability and the absolute minimum amount of system
capabilities required for performing visualization research at one of
DOE's leading institutions.

## Technical System Requirements

LBNL\'s visualization research program focuses upon algorithms and
architectures for multiresolution visual data analysis of large and
complex datasets in a remote and distributed context. The research
program requires development platform with specific characteristics that
are listed below.

1.  64-bit processors and operating system support. The need for 64-bit
    addressing capability is a requirement when working with data sizes
    that exceed 2GB in size.

2.  A system architecture that supports scalable I/O and graphics
    capabilities, where incremental improvements can be purchased at a
    relatively low cost. The cost of adding a graphics pipe to an Onyx
    system is typically on the order of about \$50K. The cost of adding
    an extra graphics accelerator to an Altix system will be
    substantially less, although no exact cost figure is available at
    this time (we are using a figure of \$10K per pipe in this proposal
    for the purposes of discussion). The Altix "G-brick" supports two
    graphics cards per brick. If we are simply adding a new card to an
    existing G-brick, the incremental cost is simply that of the
    graphics accelerator. In the worst case, an additional G-brick would
    be needed along with the graphics accelerator. The worst case
    scenario on the Altix is still much more financially attractive than
    the alternatives for the Onyx.

3.  Support for both shared- and distributed-memory algorithmic
    development. The typical visualization research and development
    cycle involves creating early prototypes that are serial. Such
    development and testing in SMP environments is substantially less
    complex then on distributed memory systems. One of our research foci
    is frameworks for weaving together component-based visualization
    tools into aggregate applications. In many instances, "recycled"
    components are legacy in nature, which means they are most likely
    serial in implementation. Having support for both SMP and
    distributed memory processing capabilities on the same platform buys
    us a great deal of flexibility in how we approach and execute our
    research program.

4.  Vendor support for both the hardware and system software. We require
    direct vendor support for our research platform's hardware and
    software. Having direct vendor support allows us to spend more time
    doing research, and less time performing system administration
    tasks. A corollary to the requirement for vendor support is that the
    new platform should be early in its life cycle, rather than close to
    extinct. A product that is about to be discontinued means that we
    can expect difficulty in obtaining support and maintenance from the
    vendor in the future. A "do-it-yourself" cluster is not a viable
    alternative as we do not have the internal support staff to provide
    in-house support.

5.  High-speed internode system bandwidth. The "visualization pipeline,"
    which starts with a data source, proceeds through memory, system
    processors, graphics pipes and often through network links, imposes
    extraordinary bandwidth demands upon a system. An 8x AGP graphics
    accelerator, which can consume 2GB/s of graphics data, will be
    "starved" for data if the system's internal interconnect fabric
    supports only 100MB/s of transfer bandwidth. If the system is
    equipped with multiple graphics accelerators, the requirement for
    internal bandwidth grows accordingly in order to maintain balanced
    system performance.

6.  Extremely high internal system bandwidth. As compared to the
    Onyx-class systems, the Altix offers nearly an order of magnitude
    more internal bandwidth to the graphics devices (compare 8x AGP at
    2GB/sec with XIO at 300MB/sec). In addition, the NUMAlink
    interconnect fabric bandwidth exceeds the capabilities of Gigabit
    Ethernet, Myranet and Infiniband solutions hosted on a PCI bus. the
    SGI solution connects NUMAlink to the processors via chipsets,
    rather than routing such traffic through a PCI bus). The challenges
    posed by visual data analysis of terascale data require substantial
    amounts of bandwidth.

## Intended Use

The intended use of the system is to support visualization research. We
describe a typical-use scenario in the following section. In addition to
visualization research, we anticipate sharing the machine with other LBL
groups who are also funded by DOE.

### Visualization Research Projects

A recurrent theme from the user community is "we want to find and see
interesting things in our data, but we can't give you a precise
definition of *interesting*." Instead, what we do is to work closely
with the user to understand their workflow patterns, and then develop
new tools and techniques that help to achieve the desired workflow.

Looking at combustion research, two of the common themes are "what
factors influence flame efficiency?" and "how can we refine and improve
the fundamental model of a flame?" The problem is that the size of the
state space is huge and complex: simulation results produce time varying
3D AMR data containing 50 chemical species and 80 chemical reactions.

The approach we will use in close cooperation with the users (CCSE and
ANAG at LBNL) involves first defining then finding all regions of
interest, then performing more detailed analysis on those regions. A
region of interest -- a flame front -- might be defined as a condition
expressed over one or more variables. A flame front might be
characterized as "regions where temperature is in excess of 1500F and
the concentration of CH exceeds 10%."

Currently, combustion researchers write small "disposable codes" that
rifle through large datasets to extract regions of interest. These codes
in turn produce smaller data files that undergo more detailed analysis.
One such analysis is to generate normals to the flame front, then
perform numerical integration of chemical species along the normal. This
process helps to better characterize the regions in and around the flame
front, which in turn produce a better understanding of the flame itself.

In our visualization work, we pursue multiple simultaneous approaches.
On the one hand, we use the same tools currently employed by the user to
reproduce their results. These are serial codes that read from large
(multiple TBs) datasets and consume vast amounts of RAM (10s of GB). On
the other hand, we aim to improve the workflow by streamlining
operations, and to produce new algorithms that generate more meaningful
output.

In this reasonably typical scenario, we have the following system
demands and requirements:

1.  Large amount of secondary storage directly attached to the research
    platform of sufficient size to hold a modest dataset.

2.  Large amount of RAM available to single-threaded operations.

3.  Multiple processors to support research of parallel visualization
    algorithms.

4.  Hardware accelerated graphics to support fast rendering of
    primitive-based visualization results (points, polygons, textures,
    etc.).

5.  High internal system bandwidth to support visualization of typical
    data sets.

### Future Technology Group Projects

The LBNL Future Technologies group has expressed the desire to use an
Altix system shared with the Visualization Group as a research platform
on two different projects. The Checkpoint/Restart project[^29] currently
does not have a 64-bit Linux platform for use in research in
development. Similarly, the Unified Parallel C (UPC) project[^30] is in
need of a 64-bit Linux platform.

## Altix Architecture

SGI's new Altix system provides all of capabilities we described in the
previous section detailing our requirements. In brief, the Altix system
is constructed from Intel IA64 processors, and system components are
connected via a high-speed interconnect fabric, called "NUMAlink." SGI's
"value add" to a "garden variety" cluster composed of IA64 nodes is the
NUMAlink interconnect fabric and a CC-NUMA memory subsystem integrated
with an SGI-specific Linux kernel. The combination effectively
transforms a distributed memory cluster into a large symmetric
multiprocessor machine. SGI's NUMAlink bus provides approximately six
gigabytes/second in aggregate, bi-directional data bandwidth between
processors and memory. The Altix uses a modular architecture, similar to
the older Onyx product line, whereby system components attach directly
to the NUMAlink bus, allowing for extremely high bandwidth between
components while retaining the ability to grow the base system without
having to purchase an entire new backplane at a later time. In terms of
graphics capabilities, SGI provides graphics modules that attach
directly to the NUMAlink bus to capitalize upon the high NUMAlink
bandwidth. The CC-NUMA architecture and the NUMAlink-attached graphics
pipes permit access to any graphics unit from any processor. Each
graphics unit employs commodity GPUs for the graphics hardware. This
"best of both worlds" approach allows us to take advantage of high
performance, low-cost commodity graphics hardware that is hosted on a
stable platform with extremely high internal bandwidth.

What Altix does not provide at this point is the rich set of media
libraries and utilities that have been included with SGI products over
the years. It remains to be seen if basic system utilities, like MPI,
work well.

In addition, there is some risk because we are not aware of whether or
not vendors of graphics and visualization applications will be porting
their products to the Itanium2 architecture. We feel the risk is
relatively minimal, as Linux is widely supported by vendors, and because
64-bit versions of Linux for both the IA64 and AMD64 are readily
available.

## Alternatives

In this section, we describe some of the potential alternative
candidates for a visualization research platform. The list is not
exhaustive. Instead, it represents a reasonable cross-section of
different technologies that contrast to the Altix system.

### SGI Onyx

Historically speaking, the preferred platform for visualization research
has been the SGI Onyx. The Onyx possesses a number of characteristics
that are conducive to visualization research, many of which are present
in our list of requirements. The Onyx supports 64-bit operation, scales
to a large number of processors and graphics pipes, provides a
cache-coherent shared memory architecture, and has strong vendor
support. In addition, the applications shipped with the operating system
are "friendly" towards graphics and visualization. These applications
include libraries and utilities for manipulating digital media, for
controlling the graphics display parameters, for creating real-time
visual simulation applications, and so forth. In addition, the system
software is generally of very high quality in terms of implementation as
well as performance.

There are three substantive drawbacks to an Onyx in this day and age.
First is the fact that the Onyx graphics system is based upon the
Infinite Reality custom ASIC hardware, which is quite expensive and of
inferior performance to other contemporary graphics engines. Related,
the MIPS line of processors, which can be considered as "custom
hardware," do not enjoy the same market segment size as AMD and Intel
processors, and as a result are more expensive. Second is the fact that
SGI is deemphasizing the Onyx product line as the company as a whole
moves towards systems comprised of commodity components. Third, the
overall price/performance ratio of the Onyx does not compare favorably
to other products.

While the price-to-performance ratio is an important part of any
purchase decision, the primary strength of the Onyx platform is the
combination of tightly integrated hardware and software, and a rich set
of software offerings that are very useful in visualization research.

### A "White Box" Cluster

As an alternative to the SGI Altix, we consider a "do-it-yourself"
cluster built from generic, "White box" Itanium2 units. The nominal
configuration for the White Box cluster is generic IA64 workstations
with commodity graphics hardware and commodity (Gigabit Ethernet)
interconnect fabric. The primary reason for considering a White Box
solution is to gain a better understanding of the the "low end" of the
pricing scale on Itanium clusters that have a minimal level of hardware
and software support.

Our online quote from Dell for a single IA64 node is attached as an
Appendix to this document. Each node consists of dual 1.5Ghz 6MB
Itanium2 processors, and 4GB of memory per node. The GSA price for each
node is \$17,803, for a 16CPU price of approximately \$142K. The cost
does not include storage, Gigabit Ethernet switches, graphics hardware,
RAID, setup, and so forth. The cost does include 3-year support on
RedHat Advanced Server (which adds about \$3K to the cost of each node).
Each node has three 100Mhz PCI-X slots, one of which could be used for a
commodity graphics accelerator, although we would need to add PCI riser
cards to the system and purchase graphics hardware separately.

In contrast, the attached quote from SGI lists the base price of the
system, without graphics, as approximately \$186K. That quote includes
the a 500GB TP9100 FCAL RAID. The line items for the RAID total about
\$15K. Therefore, the system cost is closer to about \$171K. The \$30K
difference between a "white box" cluster and the SGI base system
represents the cost of the NUMAlink backplane, power supplies,
enclosures, and so forth. In our view, the financial outlay is about the
same, yet the white box approach will require substantially more effort
on our part. The extra effort detracts from our primary mission, which
is visualization research.

### Some Other Branded Cluster

A middle ground in the "do-it-yourself" strategy is to use a branded
cluster. An example is the technology offered by GraphStream[^31].
GraphStream's computer systems are commodity PC clusters (either IA32 or
AMD64) that use an Infiniband fabric for system interconnect, as well as
for connecting to storage devices. GraphStream provides turnkey
solutions that include OS, integrated drivers for the Infiniband
hardware, and if necessary, modified versions of common utility software
(MPI, Chromium) that can take advantage of Infiniband interconnect.
Infiniband at the 4x level (current technology) offers 1GB/sec
bandwidth, and which has lower latency than Gigabit IP fabric.

Unlike the Dell solution, GraphStream appears to be sensitive to the
needs of the parallel graphics and visualization community. For example,
they provide an Infiniband-enabled version of Chromium. They
demonstrated a parallel visualization application on an Infiniband
cluster at IEEE Visualization 2003 in Seattle and at SC03 in Phoenix. We
do not have any pricing data from GraphStream to include with this
discussion.

### Sun Microsystems: Sun Fire

The current approach Sun Microsystems uses for scalable visualization
and rendering systems involves connecting what is basically a "PC
cluster" to a large Sun Fire server using Myrinet interconnects. The Sun
Fire server, which is an SMP platform, provides the computational part
of the visualization pipeline. The "PC cluster" provides scalable
rendering capabilities. Two separate quotes were provided by Sun for 16
CPU, six graphics pipe systems, and these are included as an Appendix to
this document. The two quotes, which are for architecturally similar
systems, range from about \$450K to about \$550K. The price difference
is a function of configuration of the SMP host. The PC graphics cluster
uses commodity graphics cards, rather than the more expensive Sun
proprietary solution (Shaka Zulu).

From an architectural standpoint, we have several concerns about Sun's
architectural design. The choice of Myrinet as the fabric for heavy
payload transfers is a single-source, expensive, proprietary solution.
An Infiniband solution would allow a buyer or system builder/integrator
to choose from multiple vendors. It would also leave open the
possibility for taking advantage of increases bandwidth in the future.
Presently, a single 4x Infiniband card, which costs around nine hundred
dollars, provides better I/O performance than the striped Myrinet
solution proposed by Sun. Another concern is that there appear to be too
many "data locks" between main memory and the graphics card. In other
words, the path that data must traverse includes an I/O boat on the
server side, then through a PCI-based Myrinet card, over Myrinet,
through another PCI-based Myrinet card, into host memory on a graphics
cluster node, then into another PCI adapter, which is the graphics
accelerator. Finally, since the graphics PC cluster is used as a
rendering slave, those processors are effectively unavailable for
general-purpose use.

### Hewlett-Packard: Scalable Visualization with Sepia

Like SGI, Hewlett-Packard's scalable visualization platform uses Linux
on Itanium clusters equipped with commodity graphics hardware. HP adds
to the basic cluster configuration custom hardware to perform fast image
compositing. The custom hardware is known as *Sepia*. The combination of
Linux on Itanium with commodity graphics hardware and Sepia compositing
cards is also known as "Golden Gate[^32]." The information on HP's
website indicates the product is in the "advanced development" stage and
solicits collaborative partners.

A number of white papers linked from HP's website describe the
characteristics of the Sepia architecture in greater detail. Through an
API, it is possible to overcome one of the most common limitations of
hardware-based sort-last compositing networks: the composite reordering
problem. When performing view dependent operations, it is necessary to
alter the order in which image fragments are composited to produce
correct results. One of the Sepia papers discusses an implementation
that uses sort-first parallelism to render images of volume subsets
using commodity hardware, then uses Sepia hardware compositing to
reorder the compositing as a function of view parameters.

Sepia's interface to the graphics hardware uses a DMA engine on the
Sepia hardware[^33]. This approach provides for PCI-X bus-rate access to
the framebuffer. On the other hand, the DMA approach prohibits access to
framebuffer memory on AGP devices. On the image compositing network
side, Sepia operates at the "tile" level, as opposed to the scanline
level as in Lighting-2, or at the pixel level as in PixelPlanes. Sepia
uses an interface to ServerNet-1 using FPGAs. According to literature we
could find on HP's website, ServerNet II - which is intended for use in
Sepia Phase II -- uses Gigabit fabric.

Beyond the Sepia literature, HP fails to provide information about the
architecture of a parallel HP visualization platform. The previous
scalable HP visualization platform, the *sv6*, appears to have been
discontinued and is not considered as part of our analysis. Based upon
publicly available literature, it would appear that HP's parallel
visualization platform consists of a "do-it-yourself" Itanium cluster
equipped with commodity PCI-X graphics accelerators and alpha-quality
Sepia-2 hardware for compositing. In terms of system software, HP
appears to be posturing itself to rely on the Gelato consortium[^34] to
provide elements of a Linux distribution, including an SMP kernel.

In summary, HP does not appear to have a commercial-quality scalable
visualization and rendering platform at this time.

### Summary of Alternatives

The following table provides a summary of capabilities and price (where
available) by platform.

  -------------- --------------- -------------- ------------ ---------------- --------------- ------------- -------------
                 64-bit Support  Scalable       Supports     Vendor support   Interconnect    GPU           Estimated
                                 Architecture   shared       for hardware and Fabric                        Price for
                                                memory       software         Bandwidth                     16CPU, 6GPU,
                                                operation.                                                  32GB RAM
                                                                                                            system

  SGI/Altix      Yes -- IA64     Yes[^35]       Yes          Yes/Yes          6.4GB/s         Commodity     \$263K

  SGI/Onyx4      Yes -- MIPS     Up to 64 CPUs. Yes          Yes/Yes          1.2GB/s         Proprietary   \$750K
                 R12K                                                                                       (estimated)

  "White Box"    Possibly[^36]   Yes            No.          Yes/Maybe[^37]   0.1GB/s[^38],   Commodity     \$180K
  IA64 Cluster                                                                0.25GB/s[^39]                 
  from Dell                                                                                                 

  GraphStream    Yes -- Opteron  Yes            No           Yes/Maybe        1.0GB/s         Commodity     Unknown
  Infiniband                                                                                                
  Cluster                                                                                                   

  Sun            Yes -                          Yes          Yes/Yes          250MB/s         Proprietary   \$394K -
  Microsystems   UltraSparc                                                   (Myrinet)                     \$553K
  SunFire                                                                                                   

  HP Sepia[^40]  Yes -- IA64     Yes            No           Yes/Yes          Gigabit         Commodity     Unknown --
                                                                              Ethernet                      not a
                                                                                                            product.
  -------------- --------------- -------------- ------------ ---------------- --------------- ------------- -------------

The conclusion we draw from the comparison of Altix with potential
alternatives is that the Altix is the clear winner both in terms of
capabilities as well as cost effectiveness. There is a small amount of
risk in terms of whether or not vendors of commercial visualization
software will provide product support for Linux on IA64 platforms. There
is also a small amount of risk in terms of whether or not SGI will port
its very useful media conversion and manipulation tools to Linux from
IRIX. We view both of these risk areas as being relatively small.

## Purchase and Deployment Plans

### Deployment

The new Altix system will be located in the Building 50 complex at
Lawrence Berkeley National Laboratory. Specifically, the machine will be
located in 50B-1275, which is a raised-floor, climate-controlled
facility that is shared by many computing resource stakeholders at LBNL.
Through existing video switching hardware, the Altix will be used to
drive several different types of display technology. These include a
(type of projector in the vislab), a three-by-two tiled array shared
with LBNL's Access Grid node in building 50B, and a three-by-two tiled
display array built from LCD panels that is used for development
purposes. The LBNL Visualization group, as part of the Computational
Research Division, shares access to a Gigabit subnet at LBNL that is
managed by LBLnet. No facilities or local infrastructure improvements
will be required -- the Altix can be immediately deployed.

### System Administration and Management

The Altix system will be administered and managed by highly experienced
staff at LBNL. George Smith III was hired in 2001 by NERSC as a system
administrator, and has recently assumed responsibility for system
administration of nearly all systems in the High Performance Computing
Research Department, which includes the Visualization Group (Bethel et.
al.), the Scientific Data Management Group (Shoshani et. al.), the
Imaging Group (Parvin et. al.), the Scientific Computing Group (Ng et.
al.), the Future Technologies Group (Gorda et. al.), the Applied
Numerical Algorithms Group (Colella et. al.) and the Center for
Computational Sciences and Engineering (Bell et. al.). Smith has
extensive experience building, configuring and managing Linux systems,
and has proven to be an invaluable resource to HPCRD. Smith is well
qualified to lead the system administration tasks on the new Altix, and
his effort is provided by departmental overhead.

### Recurring Costs: Maintenance

We do not have a specific quote for maintenance from SGI for the
proposed system. Typically, hardware and software maintenance charges
range from 5-10% per year of the total system cost. We estimate a
worst-case recurring cost of around \$30K/year for maintenance, and
would pay those charges from our DOE research funding.

### Expansion and Growth

One of the attractive qualities of the Altix is the ability for
reasonably low-cost upgrades. As new graphics accelerators become
available, replacing the first-generation commodity accelerators with
newer ones will be substantially less expensive than replacing an
InfiniteReality pipe on an Onyx system. This feature -- replacing a
low-cost commodity accelerator with a newer, low-cost commodity
accelerator -- is not unique to the Altix, but applies to any of the
systems that use commodity accelerators.

Adding more processors to the Altix is a straightforward and relatively
inexpensive proposition. In contrast, adding processors to, say, the
SunFire or the Onyx incurs a substantially greater expense.

Adding more drives to the TP9100 RAID represents a relatively small
incremental cost. The total amount we are requesting for a purchase
includes approximately \$40K above and beyond the SGI quote so that we
can populate the TP9100 with additional drives to bring its storage
capacity up to 5TB from 500GB.

## SGI Altix Quote

![](media/image12.png){width="4.625in" height="6.552083333333333in"}

![](media/image13.png){width="4.583333333333333in" height="3.3125in"}

Pages 3 and 4 of the SGI quote are passages of legalese, and are
intentionally omitted from this narrantive.

## Dell IA64 Quote

![](media/image14.png){width="5.998611111111111in"
height="7.763194444444444in"}

## Sun Microsystems Quote A and Architecture Diagram

![](media/image15.png){width="6.052083333333333in"
height="4.690277777777778in"}

![](media/image16.png){width="5.99375in" height="5.618055555555555in"}

![](media/image17.png){width="5.996527777777778in"
height="2.8930555555555557in"}

## Sun Microsystems Quote B and Architecture Diagram

![](media/image18.png){width="5.99375in" height="4.613194444444445in"}

![](media/image19.png){width="5.995138888888889in"
height="5.621527777777778in"}

![](media/image20.png){width="5.993055555555555in"
height="3.5694444444444446in"}

[^1]: An Algorithmic and Software Framework for Applied Partial
    Differential Equations, http://davis.lbl.gov/APDEC/.

[^2]: Advanced Computing for 21st Century Accelerator Science and
    Technology, http://scidac.nersc.gov/accelerator/.

[^3]: See <http://vis.lbl.gov/Vignettes/> for visualization examples.

[^4]: See http://www.fusiongrid.org/

[^5]: See www.mpi-forum.org.

[^6]: See www.openmp.org.

[^7]: http://vis.lbl.gov/Events/DOEworkshop-98/index.html

[^8]: http://hdf.ncsa.uiuc.edu/HDF5/

[^9]: See http://www-didc.lbl.gov/NCS

[^10]: See http://www.web100.org/

[^11]: Network Ferret is a comprehensive, Java-based IP discovery engine
    that not only discovers basic node information but also complex
    layer 2, layer 3 and ATM connectivity. Network Ferret uses both
    standard MIBs and vendor-specific MIBs to find this information. See
    www.panix.com/\~logikos/.

[^12]: http://www.osti.gov/scidac/computing/projects/claffy.html

[^13]: <http://www.gridforum.org/1_GIS/CIM.htm>

[^14]: See calder.ncsa.uiuc.edu/ACE-grid/.

[^15]: SABUL is an application-level protocol that uses UDP to move
    payload data, and TCP for control messages. See
    http://www.dataspaceweb.net/sabul.htm.

[^16]: GridFTP is a high-performance, secure, reliable data transfer
    protocol optimized for high-bandwidth, wide area networks. See
    http://www.globus.org/datagrid/gridftp.html.

[^17]: The Global Grid Forum (GGF) is a community initiated forum of
    5000+ individual researchers and practitioners working on
    distributed computing, or "grid" technologies. See
    www.gridforum.org.

[^18]: See http://www.gridforum.org/1_GIS/CIM.htm.

[^19]: Storage Resource Managers (SRMs) are grid middleware components
    that dynamically manage client access to storage resources under
    their control. See
    <http://sdm.lbl.gov/srm/documents/srm-paper-032101.pdf>.

[^20]: See http://www.globus.org/osga/.

[^21]: http://www.globus.org/wsrf/

[^22]: http://www.w3.org/TR/wsdl/

[^23]: gSOAP is a set of compiler tools that provide SOAP/XML-to-C/C++
    binding to ease the development of SOAP/XML web services and client
    application in C or C++. See gsoap.sourceforge.net.

[^24]: The Common Component Architecture (CCA) Forum, funded by DOE
    through SciDAC, is a group of researchers from national labs and
    academic institutions committed to defining component architecture
    standards for high performance computing. The objective of the CCA
    Forum is to define a minimal set of standard interfaces that a
    high-performance component framework has to provide to components,
    and can expect from them, in order to allow disparate components to
    be composed together to build a running application. See
    www.cca-forum.org.

[^25]: See http://java.sun.com/products/jsp/download.html.

[^26]: See http://www-unix.globus.org/cog/java/.

[^27]: http://doesciencegrid.org/projects/GPDK/.

[^28]: http://www.gridsphere.org/

[^29]: http://ftg.lbl.gov/twiki/bin/view/FTG/CheckpointRestart

[^30]: http://upc.lbl.gov/

[^31]: http://www.graphstream.com/

[^32]: http://www.hp.com/techservers/hpccn/sci_vis/index.html

[^33]: http://www.hp.com/techservers/hpccn/sci_vis/sepia_arch.html

[^34]: http://www.gelato.org/

[^35]: Single system image up to 64 CPUs, single system size up to
    "hundreds" of processors.

[^36]: "Possibly" means that building and installing a 64-bit Linux
    kernel would be the responsibility of the user. In addition to the
    kernel itself, 64-bit compliant drivers and utilities would also
    have to be obtained or built from scratch. While such an endeavor is
    possible, it clearly lies outside the scope of our group's mission,
    and we do not have dedicated system staff to undertake such a thing.

[^37]: "Maybe" for vendor software support recognizes potential
    difficulties that arise when OEM integrators load 3^rd^ party
    software. Such a model has proven to be effective for Windows
    systems. Dell proffers the new RedHat Enterprise distribution, which
    replaces the RedHat workstation products. At this time, the jury is
    still out on the quality of that distribution and the quality of
    support provided by RedHat.

[^38]: Gigabit Ethernet interconnect.

[^39]: Myrinet 2.0 unidirectional measured transfer rates; 64-bit,
    133MHz PCI-X 225MHz RISC & Memory. See
    http://www.myricom.com/myrinet/performance/

[^40]: As far as we can tell, Sepia is not a "real" product. We include
    it here for completeness.