# Presentation: 082959

## Desktop Access to Remote Resources

- An International HPCC/HPCN Community Activity to establish Interoperability Standards
- Second Working Group MeetingSandia February 15,16 99
- http://www-fp.mcs.anl.gov/~gregor/datorr
- datorr@mcs.anl.gov

## Logistics

- Thanks to Sandia -- Judy Beiriger, ...
- and to Gregor von Laszewski
- Conference Fee paid during lunch time
- Dinner tonight at Gardunos -- sign up sheet at door

## Motivation for DatorrWhy desktop access? 

- Very many machines, platforms, Operating Systems
- Though a good/necessary thing, using them in changing environment becomes difficult for users.
- Goal:
- Enable Uniform or seamless access to general backend compute, communication and information resources
- Hide details and satisfy “majority of” customers
- Datorr will help metacomputing but goals are focussed on a subset of issues and aimed at establishing interoperability standards -- reference implementations may follow but are not direct goal

## Seamless Interfaces

**Seamless****Interface**

**Database1**

**Database2**

**User View**

**System View**

- 2 workinggroups at
- last meeting

- 3rd workinggroups was
- a particularservice --security

## What is Scope of Datorr

**Desktop**
- Workstation UNIX/MS, WWW Browsers
- PalmPilot, Pager :-)
- Which submit “**tasks**” to:
**Remote Resources** in multi-tier systems
- compute/communication resources:
- supercomputers, workstations, PC’s
- Data Resources:
- databases, information services
- compute/communication services

## Some Cosmic Issues

- We are roughly defining properties and methods of those distributed objects associated with computing services -- these include jobs and resources
- This could lead to CORBA facilities or Java frameworks for computing services which was motivation for interest by **Java Grande Forum **at** www.javagrande.org**
- At a recent meeting at Argonne organized by NCSA Alliance, Datorr appeared to be very relevant to building reusable “Problem Solving Environment” tools a.k.a. components for domain specific “workbenchs”

## Datorr Process

- Identification and documentation of existing relevant projects
- Set of meetings leading to agreement on common standards
- International community input such as SC98 BoF
- Preliminary documents (on web) leading to (draft) final proposed standards
- Target date for draft proposal is SC99 (another BoF) where we can have appropriate broad discussion
- Note Metacomputing Project Symposium in Juelich, Germany Sept 20-21 1999
- Selected community projects e.g. reference implementations

## Workshop Activities:Details of Datorr Process

- What is the current state?
- What projects are out there?
- What are their goals?
- What can we reuse?
- Where do we have to expand/focus?
- Most importantly workshop participants agreed that it was indeed useful to define common interoperable interfaces
- Mailing List datorr@mcs.anl.gov
- http://www-fp.mcs.anl.gov/~gregor/datorr
- Proceedings of first workshop available

## Contents of Proceedings

- 32 (15 more to add!) Existing Projects surveyed covering metacomputing, distributed scheduling, seamless interface, particular tools and technologies in areas such as resource management/monitoring/collaboration/security
- Has grown from 18 at SC98 -- add your project!
- http://www-fp.mcs.anl.gov/~gregor/datorr
- http://www-fp.mcs.anl.gov/~gregor/datorr/report/datorr-report.html
- http://www-fp.mcs.anl.gov/~gregor/datorr/report/datorr-security.html

## What Should we do this meeting?

- First we should have an update from nifty projects which were not discussed at the first meeting
- Then we should break into working groups in areas which aren’t too controversial so we lay a solid foundation for future Datorr activities
- Suggested topics:
- What is a task (user view)
- What is a remote resource (back-end view)
- What are services (linking users and resources)
- Architecture Issues

## Sources of Remote Resource Taxonomies

- Globus
- Condor
- Legion
- Netsolve, Ninf
- GRD (Genias/Raytheon)
- NOW Millenium (UCB)
- Ninja (as services)
- Jini
- Gateway, DRM (must be done)
- Dutch ASCI (Bal)
- Harness
- TOP500 and other Linpack registration and publication services
- PetaSIM and other performance estimators

## Resources

- CPUs
- System Architecture
- Databases
- Software -- libraries, licenses, applications, versions
- Network QoS, Bandwidth
- Storage
- Parallel I/O
- Visualization
- Memory
- Event Handlers
- Peripherals such as instruments
- Printers
- Cost: Money (accounting) and User Pleasure

## Choices and Issues I

- Extensibility
- Resource management (temporal versus spatial)
- What are criteria of successful scheduling
- Millenium is user preference oriented
- Interactive / Computational Steering
- Availability
- Services imply resources and vice versa
- Queue Length depends on dynamic issues

## Choices and Issues II

- Express user needs, preferences, bank balance
- Language to compose resources into constraints etc.
- Condor is not very relatively unique?
- Mariposa is a database system
- Jini has limited capabilities
- even Globus has it!
- Look up and discovery
- Security characteristics are difficult to express
- How to express standard?
- C++ API is not essential
- CORBA or Java RMI Interfaces
- Web Interfaces
- Cell Phone (except in DoE facilities)
- Separate control path (discover resource) from data path (how you use)
- Standard should specify nature of API

## Choices and Issues III

- So access method is part of resource definition
- cf. Javelin from UCSB
- High Level Principles?
- Platform Independence -- required at least for resource discovery
- Linkages to common approaches
- Separate resource discovery from resource use
- Define protocols that resources respect
- Hierarchical definitions
- is user’s bank balance a resource?

## Choices and Issues IV

- A Legion is a resource
- Globus views Legion as a software resource
- Legion views Globus .....
- Note Globus could refuse to let you ask what machines where in its resource pool
- What is architecture of a resource
- all resources have a natural service attached to them -- a printer is a printing service exporting the printer interface?
- Need strong typing of interface
- Sometimes discovery is trivial -- you remember your money is stored under your bed etc.
- What resources will the real world define
- Some we should ignore (e.g. cars)
- Others we should follow the big boys (e.g. printers?)

## Choices and Issues V

- What is on our list
- Either because we can’t describe them well
- Or we don’t need them
- We should focus -- decide on limited goals
- Discovery is not performance critical -- can use standards even if non optimal
- How does one query resource list
- Need both required and optional arguments
- Should document why certain things are there e.g. maybe some fields only there for some special computers

## Choices and Issues VI

- Look into relationships with MIME types
- A given resource will have multiple XML descriptions depending on type of query
- e.g. some queries need just high level descriptions of MPP; others detailed hierarchical definitions
- So need to specify query
- Grand Challenge in XML is (courtesy of NPACI/UCB) distinguish Tera T90 old SP new SP, cluster of PC’s WS’s
- prize is a discarded Paragon ...
- Ninja is broader than Datorr

## Choices and Issues VII

- Do we support multiple interface cards with different network interfaces
- Identify successes and failures of previous systems
- www.datorr.org appears a more modern “better” model than Globus Legion etc.
- Scope of www.datorr.org is “computing” as the commodity market is not addressing
- cf. HLA standard in DoD

## Issues in prototyping XML definition of resource subset

- What is the subset? One suggestion is
- Submit to a batch queues managing N supercomputers
- Submit interactively
- Supercomputers require memory model and notion of collection of nodes?
- Or are they an abstract compute resource without detailed knowledge
- What is query mechanism?
- Access www.datorr.org with XML tools ....? Record world’s supercomputers  (TOP500)
- Map into Legion, Globus, or Condor syntax
- Too hard -- do this later

## Process Guidelines

- Goal: Need XML definition of resources according to previous principles
- Choose a subset -- implement quickly e.g. one month
- Test or evaluate it (also a month)
- User and system builder requirements from DoE DoD etc.
- Globus Unicore Legion Condor Ninja etc. can ask if this proposed resource definition would work
- Cover lookup and use of resource
- Iterate process extending subset to other resources
- DATORR 1.0 at SC99 does not preclude DATORR 1.1 etc. later (i.e. must be extensible)
- SC99 demo of its use at www.datorr.org includes task definition (the other working group)

## Process Implementation

- Identify “people” with responsibilities
- Cover definition ( ~2), Preparation of tests(many), Clarifying general principles(2)
- General principles activity includes building XML base infrastructure supporting extensibility, multiple views, hierarchy

## Resource Prototype I

- Specify a given collection of computers: subset of
- IBM SP
- NOW/CPLANT
- Tera
- Sun E10000
- Origin 2000
- T3E
- include multiprocessor nodes (include digital SMP’s) and node linkage
- Query XML database for “linpack” performance
- Must build on the XML base infrastructure supporting extensibility, multiple views, hierarchy
- Registration service to add resources to www.datorr.org comes together with lookup service

## What Prototype leaves Out

- Access method, security, scheduling including current load
- storage, database

## Early Success Possibilities

- Legion Globus Condor ... use www.datorr.org
- Download interface to remote resource with query
- Legion Globus etc. supply different interfaces to given remote resources

## DATORR:TASK Object

## The TASK Abstraction

- A *Task *is a static representation of “what” is to be done.
- A *Job *is an instantiation of a task for execution.
- A task:
- has a core set of attributes
- is extensible
- is recursive

## A Simple Task

*A Simple task object contains*:
- a set of resources:
- name, type, constraints, ...
- a set of constraints - an arbitrary expression using resource attributes which returns *a ternerary *value (*true, false, undefined); *these define cross resource constraints
- a preference/rank - an arbitrary expression using resource attributes which provides a ranking for resources matching the constraints
- other task attributes:
- set of inputs/outputs
- triggers
- action
- executable
- command line args
- environment variables/contexts
- owner
- ...

## A Task

*A task object contains*:
- a set of resources:
- name, type, constraints, ...
- a set of subtasks
- a set of constraints - cross resource/task constraints
- a preference/rank
- a set of dependencies:
- temporal
- data dependency
- concurrency
- asynch (absence of dependency)
- control
- other task attributes

## Action Items

- Projects need to provide their concept of a “task”
- Focus: a simple task *start a remote process*
- Collect attributes needed for a remote process - send comments to pm@icase.edu (Feb 26th)
- XML definition (T. Tannenbaum, T. Haupt, P. Mehrotra end March)
- Comments from:
- users
- system designers
- Iterate on definition
- Reference implementation:
- build a simple task
- find a matching resource
- execute the task