# Presentation: 211960

## DANSE Software Architecture

- Overview and short demonstration
-  of the DANSE analysis software

- Mike McKerns, Michael Aivazis
- California Institute of Technology
- NeSSI Workshop
- October 13-15, 2003

## Ontology

- DANSE is a *software architecture*:
*a specification of the organization of the software system*
*a description of the crucial structural elements and their interfaces*
*a specification for the possible collaborations of these elements *
*a strategy for the composition of structural and behavioral elements*
- DANSE is multi-layered
*flexibility*
*complexity management*
*robustness under evolutionary pressures*
- Contemplate the disconnect between a remote, parallel computation and your ability to control it from your laptop

## Stereotypical DANSE users

- Visiting scientist
*occasional user of prepackaged and specialized analysis tools*
- Instrument specialist
*author of prepackaged specialized tools*
- Expert scientist
*prospective author/reviewer of PRL paper*
- Analysis expert
*author of analysis, modeling or simulation software*
- Software integrator
*responsible for extending software with new technology*
- Framework maintainer
*responsible for maintaining and extending the DANSE infrastructure*

## Application generic user interface

- Workflow graphs: the natural interface
*good correspondence between logical and physical descriptions of the computation*
- Multiple views of the computation
*data flow*
*control flow*
*deployment of distributed components*
- Interactive editing of component state
*property editors*
*dynamic interface generation from component supplied specifications*
- High end graphics
*data plotting in two and three dimensions*
*three dimensional visualization*

## Distributed computing

- Enabled by design
*careful orchestration of the interactions between the computational engines and their environment*
- Issues
*authentication*
*security*
*remote deployment, control and monitoring*
- Two pronged implementation strategy:
*use currently available, robust (but primitive) technologies*
*initial authentication and deployment based on ssh*
*authentication and security using pyre services*
*take advantage of grid services as they become available*
- User interfaces are harder to write correctly!

## Component anatomy

- Core: encapsulation of computational engines
*three-tier middleware that manages the interaction between the framework and analysis modules written in low level languages*
*lowest  level UI: python scripts*
- Harness: an intermediary between a component’s core and the external world
*Function:*
*collects and delivers user configurable parameters*
*interacts with the data transport mechanisms*
*guides the core through the various stages of its lifecycle*
*provides monitoring services*
*Parallelism and distributed computing are achieved by specialized harness implementations*
*The harness enables the second level of integration*
* **adding constraints makes code interaction more predictable*
*provides complete support for an application generic interface*

## Pyre architecture

- component

- bindings

- library

- extension

- component

- bindings

- custom code

- core

- facility

- framework

- facility

- facility

- facility

- component

- bindings

- custom code

- service

*requirement*

*implementation*

*package*

- The integration framework is a set of co-operating abstract services

*FORTRAN/C/C++*

*python*

## Data analysis engines

- Normal engine life cycle:
*deployment*
*staging, instantiation, static initialization, dynamic initialization, resource allocation*
*launching*
*input delivery, execution control, hauling of output*
*teardown*
*resource de-allocation, archiving, execution statistics*
- Exceptional events
*core dumps, resource allocation failures*
*diagnostics: errors, warnings, informational messages*
*monitoring: debugging information, self consistency checks*
- Crossing the single process boundaries
*parallel processing*
*distributed computing*

## Ports and pipes

- Ports further enable the physical decoupling of components by encapsulating data exchange
- Runtime connectivity implies a two stage negotiation process
*when the connection is first established, the IO ports exchange abstract descriptions of their requirements*
*appropriate encoding and decoding takes place during data flow*
- Pipes are data transport mechanisms chosen for efficiency
*intra-process or inter-process*
*components need not be aware of the location of their neighbors*
- Standardized data types obviate the need for a complicated runtime typing system
*meta-data in a format that is easy to parse (XML)*
*tables*
*histograms*

## Component implementation strategy

- Write data analysis engine
*custom code, third party libraries*
*modularize by providing explicit support for life cycle management*
*implement handling of exceptional events*
- Construct python bindings
*select entry points to expose*
- Integrate into framework
*construct object oriented veneer*
*extend and leverage framework services*
- Cast as a component
*provide object that implements component interface*
*describe user configurable parameters*
*provide meta data that specify the IO port characteristics*
*code custom conversions from standard data streams into lower level data structures*
- All steps are well localized!

## Application-specific interfaces

- Provide custom UI for select analysis workflows
- User interface should
*be simple, easy to use*
*treat the data analysis process at the right level of abstraction*
*have reasonable defaults for most choices*
- Forms or wizards?
- As users get more familiar they become more demanding
*transparent access to the underlying application-generic interface*
*access to the scripts?*

## Status

- Leveraging **pyre**:
*large number of existing services*
*component cores*
*integration strategy well understood*
*component harnesses*
*support for user configurable properties is complete*
*data transport*
- User interfaces
*two application-generic prototypes under construction*
- Missing pieces
*engines for analysis, simulation and modeling*
*application-specific UI*
*tests, of all sorts...*
*better plotting*
*documentation, training materials*