# Presentation: 409735

## Rover Technology Infusion (incl CLARAty)

**Issa A.D. Nesnas**
** *****Jet Propulsion Laboratory***
***NASA Ames Research Center***
***Carnegie Mellon***
***University of Minnesota***

## Rover Technology Test beds with CLARAty

**Funding Profile ($K):**

**Objectives:**
- Facilitate infusion of performance-enhancing navigation and manipulation technologies into MSL flight system
- Provide a flexible framework for integrating and comparing competing technologies on all research rovers: Rocky8, FIDO, Rocky7, K9, and FIDO5

*CLARAty **– Coupled Layer Architecture for Robotic Autonomy*

**FY03-FY05 Milestones:**
**FY03** mobility and navigation for long traverse
**FY04**	pose estimation, tracking, and manipulation for instrument placement
**FY05**	complete simulations, onboard science and health monitoring

**Task Manager:**
- 	Issa A. D. Nesnas	(818) 354-9709
- 	nesnas@jpl.nasa.gov
**Participating Organizations:**
- 	JPL, Ames Research Center, Carnegie Mellon, U. of Minnesota, RMSA Participating Universities
**Facilities:**
- 	Rocky 8, Rocky 7, K9, FIDO, FIDO 5, ATRVs, CLARAty test bed, ROAMS, WITS, JPL Mars Yard

## Problem Statement

**Problem:**
- Non-integrated robotic infrastructure for validation and infusion into flight
- Non-reusable and redundant robotic infrastructure for each rover
- Non-unified framework requiring technology integration for each robot
- Non-common infrastructure for infusion of technologies from universities
- Non-interchangeable software for: Rocky 8, FIDO, Rocky 7, K9, ATRVs
- Absence of repository that captures domain expertise – (JPL and  other centers)
**Key Challenges**
- Collaborative software development with multiple institutions
- Software accessibility and intellectual property
- Large legacy code bases
- Continuous working software for all platforms
- Different hardware architecture for each rover
- Large customer base: users, contributors, and collaborators
- Rover access to remote developer
- Flexible framework for advanced research

## Mission Relevance and State-of-the-Art

**Mission Relevance**
- Enables transfer of technology to flight from a single integrated source
- Enables integration and validation of competing technologies for selection by missions
- Enables usage of several robotic platforms
- Captures competed technologies for future flight missions
- Makes research rovers viable test bed for flight algorithms
- Adapts easily to future rovers with different hardware architectures
- Relevant to MSL, future rover missions, and future robotic research
**State-of-the-art**
- Separate disparate robotic software systems
- Interchangeable robotic software limited to high-level encapsulation
- Several efforts attempted to build common infrastructure – DARPA (Jaus), Intel (Robotics Engineering Task Force)

## Technical Approach

**Capture legacy **algorithms and infrastructure
**Decompose** robotics domain into subsystems
- Use **domain knowledge** to guide design of each subsystem
- Identify **common** **infrastructure** needs across and within subsystems
- Make all implicit **assumptions** explicit
- Define proper **interfaces** for each component
- Develop **generic framework** to support various implementations
**Adapt** multiple legacy implementations to validate framework
**Encapsulate **large components where re-factoring is not yet possible
**Test** on multiple robotic platforms and study limitations
- Enable learned experience from **users** to** feedback** into the design
- Project potential **advances** to these components
**Review** and update implementation
- Modify/**extend**/redesign to address limitations
- Provide **building blocks** for new development efforts
***After several iterations - your generic base is reusable***

## Statement of Work – FY00-FY02

- Study various robotic architectures including legacy JPL/ARC/CMU
- Develop an architecture to address robotic domain problems
- Integrate model-based programming (Decision Layer) and procedural-based programming (Functional Layer)
- Design and implement first prototype of architecture
- Develop generic framework for: I/O control, motion control, communication, locomotion, manipulation, and rover control
- Demonstrate encapsulation of local rover navigation on Rocky 8
- Demonstrate on Rocky 8 and Rocky 7 – locomotion, navigation, pose estimation, and science scenarios (visiting multiple science targets)
- Develop an adaptation to FIDO rover and demonstrate locomotion
- Develop generic framework for: vision, estimation, navigation, path planning
- Demonstrate long-range traverse using the integration of global path planning and local navigation on Rocky 8
- Demonstrate CLARAty locomotion with ROAMS simulation
- Build a CLARAty Test bed for all research rovers

## Statement of Work – FY03

- Procurements
**ATRV Jr for efficient university collaborations**
**Rocky 8 Bench top II x86, K9 bench top**
- CLARAty Deliverables

| Date | To | Task | Description |
| --- | --- | --- | --- |
| 12/1/02 | All | All | CLARAty Test bed |
| 12/17/02 | Backes | WITS | DL/FL documentation and interfaces |
| 12/26/02 | Jain | MDS/ROAMS | API for CLARAty/ROAMS interface |
| 1/1/03 | Huntsberger | LT | FIDO EKF on Rocky 8 |
| 2/1/03 | Kim | IP | Stereo vision processing on all platforms |
| 3/1/03 | Backes | Manipulation | Dexter platform  |
| 4/7/03 | Kim | IP | 2D Tracking on Rocky 8  |
| 5/1/03 | Backes | Manipulation | Arm and mast control on Dexter |
| 6/9/03 | Huntsberger | LT | Visual odometry on FIDO |
| 7/1/03 | All | All | CLARAty/WITS commanding rover motion |
| 7/1/03 | All | All | CLARAty/ROAMS with IMU and locomotion |
| 8/1/03 | All | All | CLARAty/ROAMS image acquisition |

## Statement of Work – FY04

- Capture Technologies from competed RMSA tasks
- Integrate functionality and encapsulate with proper CLARAty APIs
- Deliver mature technologies to validation tasks
**Long Range Traverse**
**Instrument Placement**
**Autonomous Science**
- Support upcoming NRA competed tasks

| Date | From | Description |
| --- | --- | --- |
| 11/15/03 | Nesnas | 2D/3D Visual Tracker |
| 1/1/04 | Li | Multi-view image registration |
| 1/1/04 | Matthies/Roumeliotis | 6DOF EKF algorithms |
| 2/1/04 | Roush/Glymour | Bayesian spectral classifier |
| 3/1/04 | Jain | ROAMS with manipulation |
| 4/1/04 | Olson | Wide Baseline Stereo |
| 5/1/04 | Deardren | Motor fault detection |
| 6/1/04 | Stentz | TEMPEST long range path planning |

## Statement of Work – FY04

- Improve tools and processes to allow rapid but stable development
- Review and cleanup all CLARAty APIs and module distribution
- Provide an integrated end-to-end WITS/CLARAty/ROAMS for developers
- Reduce learning curve currently necessary for get people up to speed
- Develop procedure for 3rd parties to integrate software into CLARAty
- Properly refactor frequently used CLARAty modules

| Date | To | Description |
| --- | --- | --- |
| 12/1/03 | Huntsberger | GESTALT-derivative (Morphin) navigator on Rocky 8 |
| 1/15/04 | Kim | Integrated 2D/3D Visual Tracker on Rocky 8 |
| 2/1/04 | GNC/MDS | FIDO EKF with ROAMS and MSL rover model |
| 2/15/04 | Gat | Layer detection algorithm (Roush’s) |
| 3/1/04 | Huntsberger | Sun sensor algorithm |
| 4/1/04 | Kim | Multi-view registration on Rocky 8 |
| 4/15/04 | Kim | 5DOF arm control and straight line trajectory on FIDO |
| 5/1/04 | Huntsberger | 6DOF EKF on FIDO |
| 5/15/04 | Kim | Arm collision prediction |
| 6/1/04 | Backes/Kim | CLARAty/ROAMS with manipulation |
| 7/1/04 | Huntsberger | Wide baseline stereo |
| 8/1/04 | Gat | Bayesian spectral classifier |

## Statement of Work – FY05

- Deliver integrated competed technologies based on MSL needs
- Continue integration of newly selected competed NRA tasks

| Date | To | Description |
| --- | --- | --- |
| 12/1/04 | Gat | Motor fault detection |
| 12/1/04 | Kim | Manipulation contact sensing |
| 1/1/05 | Huntsberger | WITS/CLARAty/ROAMS navigation |
| 3/1/05 | Gat | Environment perturbation fault detection |
| 4/1/05 | Kim | Rover-based placement with obstacle detection |
| 5/1/05 | Huntsberger | Self calibration for pose estimation (Borenstein) |
| 6/1/05 | Huntsberger | TEMPEST global path planning |

## Milestones

_**FY03: **_
- Delivery of several technologies to validation tasks and to MDS/MSL
_**Theme I:**_ *demonstrate multiple algorithms on multiple rovers*
- _Extended Kalman Filter (FIDO algorithm) on (1) __*Rocky 8*__, (2) __*FIDO*__, and (3) __*ROAMS*_
- _Replace with pure odometry on (1) __*Rocky 8*__, (2) __*FIDO*__, and (3) __*ROAMS *_
- _Replace with visual odometry on (1) __*Rocky 8*__ and (2) __*FIDO*_
_**Theme II:**__ __*integrate multiple complex algorithms *_
- _Morphin Navigator + local D* + wheel odometry on (1) __*Rocky 8*__ and (2) __*FIDO*__ _
- _Morphin Navigator + local D* + visual odometry on (1) __*Rocky 8*__ and (2) __*FIDO *_
- Demonstrate integrated capabilities (WITS, CLARAty, ROAMS)
_**FY02:**_
- Demonstrate 60 m traverse using path planning and navigation in **rough terrain**
- Demonstrate locomotion onto FIDO rover platform
- Demonstrate mission-like scenario using improved position estimation
- Demonstrate higher fidelity interface to ROAMS
_**FY01:**_
- Demonstrate visiting multiple targets using both  Decision and Functional Layers on Rocky 7, Rocky 8 (in Mars Yard) and on ROAMS.
- Demonstrate reusability by running software on: Rocky 8, K9, and Rocky 7
_**FY00**_
- Year-end report on the two-layer architecture
- Peer review of the CLARAty architecture

## FY03 Accomplishments

## CLARAty Team

- NASA Ames Research Center
- Maria Bualat
- Sal Desiano
- Clay Kunz (*Data Structure Lead*)
- Eric Park
- Randy Sargent
- Anne Wright *(Cog-E & Core lead)*
- Carnegie Mellon University
- David Apelfaum
- Reid Simmons *(Navigation lead)*
- Chris Urmson
- David Wettergreen
- University of Minnesota
- Stergios Roumeliotis
- Yukikazu Hidaka
- MIT
- Brian Williams
- Greg Sullivan

- Jet Propulsion Laboratory
- Max Bajracharya (34) *(Cog-E &  Vision lead)*
- Edward Barlow (34)
- Antonio Diaz Calderon (34)
- Caroline Chouinard (36)
- Gene Chalfant (34)
- Tara Estlin (36) *(Deputy Manager & Decision Layer lead)*
- Erann Gat (36)
- Dan Gaines (36) *(Estimation Lead)*
- Mehran Gangianpour (34)
- Won Soo Kim (34) *(Motion lead)*
- Michael Mossey (31)
- Issa A.D. Nesnas (34) *(Task Manager)*
- Richard Petras (34) *(Adaptation lead)*
- Marsette Vona (34)
- Barry Werger (34)
- OphirTech
- Hari Das Nayar

## A Two-Layered Architecture

_***THE DECISION LAYER:***_ Declarative model-based Mission and system constraintsGlobal planning

***INTERFACE:*** Access to various levelsCommanding and updates

_***THE FUNCTIONAL LAYER:***__*** ***_Object-oriented abstractionsAutonomous behaviorBasic system functionality

- Adaptation to a system

## Supported Platforms

**x86**

- Ames

- JPL

***Rocky 8***

**x86**

**VxWorks**

**Linux**

***Rocky 7***

***K9***

***ROAMS***

***FIDO***

- CMU

**Linux**

***ATRV***

**x86**

- JPL

***FIDO***

**VxWorks**

**x86**

- JPL

**ppc**

**VxWorks**

- JPL

**Linux**

## Collaborations

**CLARAty**

- MIT
*Dubowsky*

- RMSA Competed Tasks

- CMU
*Stentz*

- U. Michigan
*Borenstein*

- JPL
*Matthies*

- ARC
*Roush*

- U. Washington
*Olson*

- Jet Propulsion Lab

- CMU

- NASA ARC

- U. Minnesota

- MDS
*Lai*

- Inst. Placement
*Kim*

- Long Traverse
*Huntsberger*

- Validation Tasks

- Auto. Science
*Gat*

- New NRA
- Competed Tasks

- WITS
*Backes*

- ROAMS
*Jain*

- Science Sim
*M. Lee*

**MTP**

**CICT**

- ALERT
*Matthies*

- OASIS
*Castano, Judd*

- Adv. Avionics
*Bolotin*

**R&TD**

- Craters
*Cheng*

- MIT
*Williams*

- CMU
*Glymour/**Ramsey*

- JPL
*Nesnas*

- Ohio State U.
*Li*

- ARC
*Dearden*

- CLEaR
*Estlin/Fisher*

- Instr. Placement
*Pedersen*

- U. Survey
*Tunstel*

## General Highlights

**Two-day CLARAty Workshop – Jan 2003**
- Attended by 36 individuals from 4 institutions
- First face-to-face meeting for entire team
- Included management and technical presentations, discussion topics, users feedback, and tutorials
**ITAR **
- CLARAty was cleared by State Department for ITAR and was classified by Commerce Department enabling most foreign countries access to CLARAty
**Distribution**
- Worked with General Counsel’s and International Affairs office to setup a process to enable access to most foreign nationals
**IT**
- Working with JPL institutional IT to:
- Simplify process to allow collaborator access to CLARAty
- Improve JPL infrastructure to no longer require local JPL accounts

## General Highlights

**Robotic Engineering Task Force (RETF) and Intel **
- Participated in Intel’s Workshop on Robotics
- Hosted Intel for a day of interactions on RETF collaboration
- Invited CLARAty paper to IROS
**ASTEP (**Astrobiology Science and Technology for Exploring Planets**)**
- CMU plans to use CLARAty now that ITAR has cleared
- ARC currently using CLARAty for ASTEP work
**Information and Processes**
- New CLARAty site with comprehensive information
- Overview – objectives, publications, presentations, movies
- Project – team, working groups, schedule, milestones, meetings, access process, intellectual property,
- Software - FAQ, procedures, software modules, documentation, conventions, development tools,
- Hardware – concepts, technology, data sheets, upgrades
- Test bed –  test targets, build targets, hosts, vxWorks, FAQ and sign up

## Is CLARAty Paying Off?

**Some Data Points**
- After integrating and tuning EKF into new estimation framework on **Rocky 8** (3 months), integrating algorithm for **FIDO** took **1 day**
- After getting locomotion working on **FIDO**, moving to **K9** took **4 days**
- After testing CLARAty/Morphin navigator on **Rocky 8**
- Integrating and testing on **FIDO** took **2 days**
- Integrating on **K9** ** **took** 4 days **and testing **2 weeks**
- Integrating on **ROAMS** took** 1 day**
- After getting mast software working on **Dexter**
- Integrating on **Rocky 8** took **2 weeks**
- Integrating on **FIDO** took **4 days**
- Adaptation of entire locomotion, motion control, I/O control and communication onto completely new avionics **FPGA** hardware with PPC405 for controlling **Rocky 8** took **2 weeks**
**N.B. These results are from professional CLARAty developers and should not be used to assess future development costs for new users**
**(do not try this at home)**

## We Don’t Always Get it Right

**The price for high reusability**
- CLARAty core is becoming harder to change because many tasks at JPL, ARC, CMU, U. Washington, depend on it. Making changes, building and testing is complex and tedious.
- Process needs improvement – very high on FY04 priority list
**Other Data Points**
- Adaptation to ROAMS after Rocky 8, FIDO, Rocky 7 and K9 expected to take 5 weeks took **3 months**
- Sun sensor delivery expected to take 2 months has taken **3 months** and further debugging is needed – subtle integration bug
- Several software bugs experienced and reported by validation tasks. Some are easy to fix, others are harder to reproduce
- Turn around debug and redeliver cycle requires some improvements
**Observation**
- Code developed using CLARAty structures requires much shorter debugging cycle and shorter time to port to different platforms (significant cost savings)

## Package Development Highlights

**Infrastructure**
- New improved tools in place to simplify collaborative development
**Estimation**
- New framework with EKF adapted to Rocky 8, FIDO, and ROAMS
- Visual odometry integrated on Rocky 7, Rocky 8 and FIDO
**Manipulation**
- Mast code tested on Dexter, Rocky 8, and FIDO, and arm code on Dexter
**Navigation**
- Navigation integrated with VO and D*. Test on Rocky 8, FIDO, and ROAMS
**Vision**
- Vision adaptations to all rovers and addition of many image operations
**Sensing**
- Developed infrastructure and integrated sun sensing capabilities
**Decision Layer**
- Tested on multiple, randomly generated scenarios and support for onboard science
**I/O and Motion Control**
- Progress towards merging two major branches
**Data Structure**
- Flexible data marshalling for CLARAty structures

## Levels of Integration

_**Level I - Deposited**_** **
- Code in CLARAty repository - all Intellectual Property items cleared
- Compiles as a standalone application - no dependencies to other modules
- Have test programs and user documentation for getting started
_**Level II - Encapsulated**_
- Integrated with other CLARAty modules
- Uses CLARAty components to interact with rover
- Does not support a CLARAty API
- Runs on at least one robot platform
_**Level III - Integrated**_
- Conforms to a generic CLARAty API (or parent class)
- Has no unsupported 3rd party dependencies
- Runs on all applicable rover platforms
_**Level IV - Refactored **_
- Uses all applicable CLARAty classes
- Internally conforms to CLARAty conventions and coding standard
_**Level V - Reviewed**_
- Software reviewed by committee to ensure internal/external consistency

_**+ Reused**_
- Re-used by other modules in  CLARAty - dependent module
- Provides access to all internal data products

## Technology Algorithms in CLARAty

## Serving the Customer

**Closer interaction with MDS on:**
- Interfaces and behavior of robotic specific components
- Planning configuration for rovers
- Test bed support and access to CLARAty software
**Formal Delivery Documentation**
- Executive summary
- Algorithm description
- How to check out and use
- References
**CLARAty Documentation **
- API and behavior definitions
- Web-based access to most information

## Software Development Process

- Repository

*AFS Backbone*

- CMU

- JPL

- CLARAty

- Repository

- 3rd Party

- Releases

- Web

- UW

- ...

- VxWorks

- ARC

- Rocky 8

- FIDO

- Rocky 7

- Bench tops

- Bench tops

- Bench tops

- K9

- Repository

- ATRV

*Authentication*

- U. Minnesota

## Some CLARAty Statistics

**About 300 modules in Repository – goal is to limit modules**
**About 500,000 lines of C++ code – revise and reduce**
**Five adaptations to the following rovers:**
- Rocky 8, FIDO, Rocky 7
- ATRV
- K9
**Most technology modules are at Level II and Level III**
**None are at Level IV or Level V (fully compliant, documented, and formally reviewed)**

## CLARAty Test Bed

- FIDO Benchtop

- Rocky 8  Bench top

- Dexter  ManipulatorsRocky 7  Bench top

## CLARAty Test Bed

**Used by:**
- MDS/MSL
- RMSA university tasks
- CLARAty Developers
- JPL
- ARC
- CMU
- Manipulation task
- Validation tasks
**Remote Access**
- Web camera
- Remote power cycle

## FY03 Accomplishments

**Publications:**
- C. Urmson, R. Simmons, "Approaches for Heuristically Biasing RRT Growth," Proceedings IROS 2003, October, 2003
- I.A. Nesnas, A. Wright, M. Bajracharya, R. Simmons, T. Estlin,_[ ](http://claraty.jpl.nasa.gov/new_site/overview/publications/03_nesnas_challenges_iros.pdf)_Won Soo Kim, "CLARAty: An Architecture for Reusable Robotic Software," SPIE Aerosense Conference, Orlando, Florida, April 2003. (730 KB)
- I.A. Nesnas, A. Wright, M. Bajracharya, R. Simmons, T. Estlin, "CLARAty and Challenges of Developing Interoperable Robotic Software," invited to *International Conference on Intelligent Robots and Systems (IROS),* Nevada, October 2003. (410 KB)
- C. Urmson, R. Simmons, I. Nesnas,** **"A Generic Framework for Robotic Navigation," *Proceedings of the IEEE Aerospace Conference*, Montana, March 2003. (8 pages, 730KB)
- C. M. Chouinard, F. Fisher, D. M. Gaines, T.A. Estlin, S.R. Schaffer, "An Approach to Autonomous Operations for Remote Mobile Robotic Exploration," *Proceedings of the IEEE Aerospace Conference*, Montana, March 2003 (277 KB)
- T. Estlin, F. Fisher, D. Gaines, C. Chouinard, S. Schaffer, I. Nesnas,** **"Continuous Planning and Execution for an Autonomous Rover,"** ***Proceedings of the Third International NASA Workshop on Planning and Scheduling for Space*, Houston, TX, Oct 2002. (168 KB)

## Significant Events

## Integration of Multiple Algorithms on Multiple Rovers

**Provides multiple inter-changeable algorithms to compare performance against mission requirements**
- Demonstrated multiple pose estimation algorithms on multiple rovers
- Pose Estimation Algorithms:
- FIDO EKF 3D pose estimation
- Wheel odometry
- Visual odometry
- Platforms demonstrated on:
- Rocky 8
- FIDO
- ROAMS (without visual odometry)

## Integration of Multiple Complex Algorithms 

**Provides an end-to-end capability enabling evaluation of multiple integrated technologies.**
**Important for evaluation against mission requirements**
**Demonstrated the integration:**
- CLARAty Morphin navigator
- D* path planner
- Wheel odometry pose estimator
- Visual odometry pose estimator
**Platforms demonstrated on:**
- Rocky 8 (all)
- FIDO (all)
- ROAMS
- All except visual odometry
- With real scanned Mars Yard terrain
- Stereo on synthetic images
- Very preliminary tests results

## 1st CLARAty & Rocky 8 Field Trial

**1st CLARAty & Rocky 8 Field Trial**

**Demonstrated a field-ready development and test environment using CLARAty and Rocky 8 rover**** **
**1st Field Trial for CLARAty and Rocky 8**
- Johnson Valley – Mojave Desert
- Sept 22 – 25, 2003
- In collaboration with Slope Navigation task
**Demonstrated CLARAty off-line**
- Ability to rebuild software in-field
- Ability to run tests and collect data
**Collected data to support**
- Wide Baseline Stereo task
- Visual Tracking task
- Slope Navigation task

## Year-End Review Action Item Response

**MTP Annual Review**
**A16 “**The task manager and the MTP are strongly encouraged to research other successful NASA software distribution activities to understand what customer support and funding arrangements (or even licensing) arrangements have been made to sustain these ongoing developments, in anticipation of a continually growing demand for CLARAty support ”
**Response**
- MTP created a “University Robotics Survey” task
- Close CLARAty interactions with above task
- University Survey task investigated software distribution for IGOAL at JSC
- Joint development between JSC and Muniz Engineering – initial distribution through Cosmic.
- No IP restrictions (civil servants) – IP transferred to Muniz
- Only other example was RCS from NIST
- NIST developing and supporting product
- Adopted at some universities (e.g. Ohio State University)

## Issues/Concerns

- Task encountered setbacks due to inability to share software with foreign national personnel on the task before ITAR clearance
- IP concerns with respect to sharing technology algorithms with the community at large
- Due to limited resources and increased level of collaborations and activities, some infrastructure work has not received the proper attention

## Measuring Success or Failure

**We succeed **IF **we:**
**Significantly reduce integration time of new technology software onto real robotic systems**
**Support multiple platforms with different hardware architectures**
**Provide a service that is enabling for technologists**
**Simplify the development/integrate/debug/test cycle for current and next generation NASA rovers**
**Have people other than the developers using and “*****liking*****” the system**

## Video of Morphin, VO, D* on FIDO and Rocky 8