# Presentation: 447260

## Unified Robotic Software Development using CLARAty

- Issa A.D. Nesnas
- Mobility and Robotic Systems Section
- Autonomous Systems Division
- July 20, 2005
- _[http:](http://claraty.jpl.nasa.gov/)__[//claraty](http://claraty.jpl.nasa.gov/)__[.jpl.](http://claraty.jpl.nasa.gov/)__[nasa](http://claraty.jpl.nasa.gov/)__[.gov](http://claraty.jpl.nasa.gov/)_
- Briefing to the Office of the Chief Technologist
- Sponsors: Mars Technology ProgramIntelligent Systems Program

- Theme: The Long Road to Technology Infusion

## Presentation Overview

- Historical Antecedents
- Problem and Objectives
- What is CLARAty?
- Interoperability Success Stories
- Navigation
- Single-cycle Instrument Placement
- Challenges of Software Interoperability
- Challenges in Technology Infusion
- Architecture Overview
- Challenges Ahead

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Some JPL Robots

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Historical Antecedents

**Late 80’s - Early 90’s:** parallel robotic developments
- RSI, MOTES, Satellite Servicing, Robby, Mircorover
- No shared *hardware* or *software*
**Mid 90’s:** Mars rover research centralized with Rocky 7
- First flight rover
**Late 90’s:** Expansion and diversification of rover work
- No software interoperability (Rocky 7, FIDO, Athena, DARPA)
- Autonomy demonstration of Remote Agent Experiment (ARC and JPL)
- MDS investigates reusable software for spacecraft control.
**‘****99-Early 00:** Exploration Technology Program develops concept for a unifying autonomy architecture
- Unifying autonomy and robotic control
- Started the CLARAty task
**Today:**
- Unification of several robotic developments at JPL, ARC, and CMU
- Two flight rovers with several new robotic capabilities

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Problem and Approach

- Problem:
- Difficult to share software/algorithms across systems
- Different hardware/software infrastructure
- No standard protocols and APIs
- No flexible code base of robotic capabilities
- Objectives
- Unify robotic infrastructure and framework
- Capture and integrate legacy algorithms
- Simplify integration of new  technology
- Operate heterogeneous robots

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## What is CLARAty?

- CLARAty is a *unified *and *reusable ***software** that provides robotic functionality and simplifies the integration of new technologies on robotic platforms

*A tool for technology development *
*and maturation*

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Interoperability: Software & Hardware

- JPL_STEREO

- VxWorks

- Linux

- QNX

- CLARAty Reusable Software

- Robot Adaptation

- Pose Estimation
- MER_SAPP

- Obstacle Avoidance
- MORPHIN

- Obstacle Avoidance
- MORPHIN

- Obstacle Avoidance
- MORPHIN

- GESTALT

- Drivemaps

- Stereovision
- JPL_STEREO

- Stereovision
- JPL_STEREO

- Pose Estimation
- MER_SAPP

- Pose Estimation
- MER_SAPP

- Pose Estimation
- MER_SAPP

- SRI Stereo

- ARC Stereo

- Sojouner PoseFIDO 3DEKF
- 6D EKF

**CAPABILITY: Navigation**

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Interoperability Success Stories

- Navigation
- Single-cycle Instrument Placement

## Capturing Flight Algorithms: MER GESTALT on FIDO

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Navigation Interoperability on FIDO and Rocky 8

**Complex Algorithms on different Platforms**
**I/O, motion control**
**Trajectory Generation**
**Rough Terrain Locomotion**
**Odometry Pose Estimation**
**Stereo Processing**
**Visual Odometry**
**Navigation (Morphin)**
**Obstacle avoidance **
**Path Planning**

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Interoperability Success Stories

- Navigation
- Single-cycle Instrument Placement

## Single-cycle Instrument Placement

- Changes in FOV

- 1st Frame

- 37th Frame after 10 m

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## FALCON Visual Target Tracker on Rocky 8

**Integration of Complex Algorithms**
**I/O, motion control**
**Trajectory Generation**
**Rough Terrain Locomotion**
**Odometry Pose Estimation**
**Stereo Processing**
**Visual Odometry**
***Obstacle avoidance ***
**Mast Control**
**Visual Tracking**

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Integrated Single-Cycle Instrument Placement

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## CLARAty Framework for Single-cycle Instrument Placement

- 2D/3D Visual Tracking

- R8 Mast Pointing

- Morphin Navigator

- KIM
- Hand-off

- MSL Base Placement

- HIPS

- Visual Odometry

- Maestro
- Ground
- System

- Visual Tracker

- Visual
- Odometry

- SIFT
- Tracker

- Gaze Pointing

- K9 Mast Pointing

- Locomotor

- R8 Locomotor

-  Wheel Odometry

- Pose Estimator

- GESTALT

- Drivemaps

- FIDO EKF

- 6DOF EKF

- Visual Odometry

- Camera Hand-off

- Mesh Registration

- Bundle Adjustment

- Base Placement

- Haz Camera Tracking

- 2D/3D Visual Tracking

- SIFT
- Tracker

- Obstacle Avoider

- Possible Alternatives

- Vision-guided Manip

- On -board Rover
- Software
- Infrastructure

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Challenges in Interoperability

- Mechanisms and Sensors
- Hardware Architecture
- Software Algorithms

## Different Mobility Mechanisms

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Different Sensors and Appendages

- 4 DOF Mast

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Challenges in Interoperability

- Mechanisms and Sensors
- Hardware Architecture
- Software Algorithms

## Centralized Hardware Architecture

***FIDO***

- Actuator/Encoders

- Potentiometers

- PID Control in Software

- Video Switcher

- IMU

- RS232
- Serial

**PC104+**
- x86 Arch
- Framegrabbers
- Digital I/O
- Analog I/O
- Wireless Ethernet

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Distributed Hardware Architecture

**Compact PCI**
- - x86 Arch
-  Wireless E/net
-  1394 FireWire
- - I2C Bus

- Actuator/Encoders

- Potentio-
- meters

- I2C

- 1394 Bus

- IMU

- Sun Sensor

***Rocky Widgets***
- Single-axis controllers
- Current sensing
- Digital I/O
- Analog I/O

- RS232

- Distributed Motion
- Control and Vision

***Rocky 8***

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Challenges in Interoperability

- Mechanisms and Sensors
- Hardware Architecture
- Software Algorithms

## Software Challenges for Algorithm Infusion

- The *new* algorithms to be integrated may:
- Have architectural mismatches with the framework
- Include multiple orthogonal functionalities
- Make implicit assumptions about the platform
- Duplicate functionality in the framework
- Use incompatible data structures
- Are complex and hard to tune
- Require highly specialized domain expertise
- Are poorly implemented

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Architecture and Process

## A Two-Layered Architecture

_***THE DECISION LAYER:***_ Declarative model-based Global planning

**        **CLARAty = Coupled Layer Architecture for Robotic Autonomy

***INTERFACE:*** Access to various levelsCommanding and updates

_***THE FUNCTIONAL LAYER:***__*** ***_Object-oriented abstractionsAutonomous behaviorBasic system functionality

**A Two-Layered Architecture**

- Adaptation to a system

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Multi-level Abstraction Model

- Use abstractions

- Interface at different levels

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Collaborations & Interactions

- Tasks

**CLARAty**

- Jet Propulsion Lab

- CMU

- NASA ARC

- U. Minnesota

- R&TD, MDS, DRDF

**Competed ****Mars Technology**
**Program**

**Other NASA Programs**

- Rover Simulation
- ROAMS

- Rover Hardware

**JPL Internal Programs**

**Flight Focused****Technology Programs**

- Science InstrumentsSimulation

- Operator Interface

- Legacy AlgorithmsFlight Algorithms

- NASA Centers
- andUniversities
- Technology Tasks

- NASA Centers
- andUniversities
- Technology Tasks

- NASA Centers
- andUniversities
- Technology Tasks

- NASA Centers
- andUniversities
- Technology Tasks

- TechnologyValidation Tasks

- TechnologyValidation Tasks

- Technology
- Tasks

- Technology
- Tasks

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## CLARAty Test Bed for Regression Testing

- ATRV Jr.

- FIDO2 Stack

- Dexter ManipulatorBench top

- Rocky 8 PPC Bench top

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Challenges Ahead

- Mature framework and robotic capabilities
- Investigate relevance to flight and define migration path
- Develop regression tests for long-term maintainability of robotic capabilities - *very hard and open research topic *
- Maintain current capabilities on existing platforms
- Develop new capabilities (e.g. continuous motion)
- Integrate new technologies from competed programs
- Develop a releasable version
- Develop formal documentation and tutorials
- Identify and deploy on low-cost rover platform
- Open source

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Summary

- Developed a unified and reusable software framework
- Deployed at multiple institutions
- Deployed on multiple heterogeneous robots
- Integrated multiple technologies from different institutions
- Delivered algorithms for formal validation
- Enabled new technology developments on multiple platforms
- Integrated flight algorithms for detailed performance characterization and operation on research rovers.
- Taking a technology from inception, to development in CLARAty, to validation, and now to integration into flight

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Current CLARAty Core Team

**NASA Ames Research Center**
- Clay Kunz
- Eric Park
- Susan Lee
**Carnegie Mellon University**
- David Apelfaum
- Nick Melchior
- Reid Simmons
**University of Minnesota**
- Stergios Roumeliotis

**Jet Propulsion Laboratory**
- Antonio Diaz Calderon
- Tara Estlin
- John Guineau
- Won Soo Kim
- Richard Madison
- Michael McHenry
- Mihail Pivtoraiko
- Issa A.D. Nesnas
- Babak Sapir
- I-hsiang Shu
**OphirTech**
- Hari Das Nayar

- Full Credits for all Developers and Contributors at:http://keuka.jpl.nasa.gov/main/project/team/index.shtml

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Questions?

## Distributed Software Development

*AFS Backbone*

- CMU

- JPL

- CLARAty

- 3rd Party

- Web

- UW

- ...

- VxWorks

- ARC

- FIDO

- K9

*Authentication*

- U. Minnesota

- Rocky 8

- Number of employees and not FTEs

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Unify Mechanism Model

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Some CLARAty Statistics

- ~320 modules in repository (increase of 6% from FY03) – goal is to limit modules
- ~60 modules are researched technology algorithms (~20%)
- About 500,000 lines of C++ code – revise and reduce
- Five adaptations: Rocky 8, FIDO, Rocky 7, ATRV, K9
- Most technology modules are at Level III
- None are at Level IV or Level V (formally reviewed, documented, and open source)

- CLARAty Integration Levels
- Level I – Deposited
- Level II – Encapsulated
- Level III – Refactored
- Level IV – Formally reviewed
- Level V – Open source and fully                 documented

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Relevance to the Missions

- Why is this work relevant to the missions?
- Provides a  *common environment* for development, test, and comparison of advanced robotic technologies
- Provides an infusion path for robotics technologies into flight missions
- Demonstrates technologies on relevant robotic systems
- Makes research rovers viable test platforms for flight algorithms (e.g. navigation)
- Is robust to changes in rover hardware designs
- Can be easily adapted to flight and new research rovers

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005

## Measuring Success or Failure

- We succeed **IF **we:
- Significantly reduce integration time of new technology software onto real robotic systems
- Support multiple platforms with different hardware architectures
- Provide a service that is enabling for technologists
- Simplify the development/integrate/debug/test cycle for current and next generation NASA rovers
- Have people other than the developers using and “*like*” the system

- Briefing to the Chief Technologist - I.A.N.

- July 20, 2005