Fourth Workshop for the Autonomy Levels for Unmanned Systems (ALFUS)

Meeting Report

*Date and Time:*

Titan System Conference Room; February 25-26, 2004

***Executive Summary:***

The workshop has achieved consensus on a representation that summarizes
the detailed autonomy level metrics. It is a three-column table used for
identifying UMS (and its major functions) autonomy levels from the three
perspectives. The "user" participants of the workshop commented that the
form should be helpful to the program specification, testing and
evaluation processes.

The subgroup reports showed that significant progress has been made in
the development of the metrics for the ALFUS detailed model for autonomy
levels. Metrics sets and the corresponding measures have been developed
for the three axes.

We felt that the ALFUS framework has been sufficiently established to
serve as a main reference for interactions with FCS LSI. They would be
invited to the next workshop.

STANAG 4586 was discussed. Resolution was for the group to review the
document before we determine a course to take.

It was stressed, again, that the Framework should be generic and
flexible enough to be instantiated for specific uses and to be
integrated to individual users' lifecycle processes.

Main action items for participants and interested practitioners are to:

-   participate in the subgroups to continue developing metrics,

-   identify and define terms for Terminology version 2.0,

-   help the WG to disseminate its results via identifying user
    communities, distributing documents, raising awareness within
    organizations, especially up the command chain, presenting papers,
    and applying results to their programs, and

-   pursue articles in AUVSI magazine publicizing our work.

***Specific Meeting Discussions:***

Format: The following meeting report is organized per Meeting agenda.
*Italic + Bold* indicates the agenda items, with the corresponding
discussions follow.

### Day 1, February 25, 2004

***Welcome, Meeting Objectives Albus/Huang 1300 -- 1330***

## A brief overview of ALFUS was given by Hui. General Meeting objectives were depicted in the Agenda. Woody English served as the meeting facilitator.

> The ALFUS web site has been established as:
>
> <http://www.isd.mel.nist.gov/projects/autonomy_levels/>

***Terminology Update 1330 -- 1345***

> Version 1.0 was reported as published.
>
> **Actions:**

-   Participants, as well as interested practitioners in general, were
    requested to identify and submit to the Group terms and definitions
    for the Version 2.0 effort. The three metrics development subgroups
    were further requested to focus on terms that were used in their
    respective axes.

-   Kerry to send updated Controls definition to Hui for distribution
    for comments. He needs the input back in two weeks so that he could
    reference it by the end of April, per his FCS ORD cycle.

***The HRI Axis Report 1345 -- 1430***

> Brian Novak presented updated metrics. See attached. General criteria
> for prioritizing the initial, large list of metrics were that, the
> metrics unique to HRI, specific, and easily measurable are selected
> and other deferred for future versions. This resulted in four
> manageable groups of metrics.
>
> Questions were raised on the two measurement scales developed for the
> metric Intervention Frequency. Further investigation was planned.

The metric Robot Initiation was discussed. It was considered a positive
attribute for autonomy levels when a UMS is able to identify its
problems, know when to ask for help, and present the situation correctly
and efficiently.

Which axis would have a metric covering the prognosis/diagnosis
capability? Suggestions included the mission complexity axis' mission
planning and analysis metric.

Please contact Brian Novak at NovakB@tacom.army.mil or Dennis Overstreet
at <dennis.overstreet@wg.srs.com> for joining the subgroup.

***Break 1430 -- 1445***

***The Mission Complexity Axis Report 1445 -- 1530***

Hui Huang presented the updated metrics. See attached. The subgroup
focused on providing flexibility for users via weights for metrics,
alternate types of measures, and different ways of combining the
metrics. A concept similar to ACL is, in general, around level 8 is
regarded as performance from a competent human.

> Kerry presented an example showing how a user would document
> Requirements for an autonomous lawn mower. The presentation posted a
> critical question of how the autonomy metrics may be applied and may
> help the process. The group planned to further address the question by
> applying its identified metrics to the set of requirements. One
> envisioned benefit of involving the ALFUS results in the requirements
> specification process is that standard definitions and metrics
> facilitates concise requirements statements and make them unambiguous.
> Presentation slides attached.
>
> As a first test for feasibility and consistency, Bob Smith presented
> how the metrics were used to develop the ACL-like autonomy level
> specification for the autonomous lawn mower. This demonstrated, at a
> conceptual level, consistency between the ALFUS metrics and existent
> ACL from AFRL. Presentation slides attached.
>
> Please contact Hui-Min Huang (<hui-min.huang@nist.gov>) for joining
> the subgroup.

***The Environmental Difficulty Axis Report 1530 -- 1615***

> Woody English presented the updated metrics. See attached. The metrics
> were categorized into static/dynamic, electronic/electromagnetic,
> mobility, mapping and navigation, urban/rural, weather, and
> operational. Suggestions were made to provide generic descriptions for
> the metrics, which could be very useful.
>
> Please contact Woody English at <woody.english@us.army.mil> for
> joining the subgroup.

***Adjourn 1730***

#### Day 2, February 26, 2004

***Issue Discussions---NATO STANAG 4586 1620 -- 1650***

David Knichel gave a presentation on the subject. The highlights were:

-   STANAG 4586 specifies a common ground station for UAV\'s used by
    NATO countries.

-   Implementation of the agreement will enable information between
    different national UAV\'s to be collated and shared via common
    ground stations.

-   STANAG is linked to the US Joint Technical Architecture, both
    attempt to achieve interoperability.

-   

David recommended the group to review the document and consider
adopting, in some form, several key terms. The group concurred. Some
concerns/comments were:

-   STANAG 4586 is, obviously, for UAV only, whereas ALFUS covers a
    wider scope, needing to include at least UUV and non-Defense
    perspectives.

-   The STANAG was not supported by Air Force due to inconsistence with
    its doctrine.

-   Its definition on the levels of control is inconsistent with the
    FCS's in that the high levels do not include the low level
    capabilities.

-   Since the STANAG is about interoperability, JAUS should also take a
    deep look at it.

**Actions:**

-   Group to review the STANAG before we discuss a course to take:
    whether to adopt the terms, how otherwise to collaborate, etc.

-   David to email STANAG contact list. We need to contact them and be
    involved in corresponding "committees."

##### Framework Development Common Issues 0830 -- 1000

-   ***Targets by the end of the day.***

No time left to discuss this issue.

-   ***A common example/scenario?***

No time left to discuss this issue.

-   ***Integration of the metrics and the three axes.***

A simple method of integrating all the metric measurements may be
weighed average. However, different types of users, such as procurement
or evaluation, may be better served with different integration methods.
These methods include picking the highest or lowest number, graphing the
measurements and determining a "central" value, etc.

The following question demonstrated another aspect of the issue: a
mission that was executed as pre-planned and pre-programmed and resulted
in efficient performance vs. not pre-planned and applying real-time
planning, resulting in a less efficient system, which one of the
following is considered having a higher level of autonomy?

-   ***The systems/logistics "phantom" axis?***

> ***(HRI Group/Novak to lead)***

No time left to discuss this issue.

-   ***Design a test/measure process---***

> ***statistically and scientifically consistent***
>
> ***(HRI Group/Novak to lead)***

No time left to discuss this issue.

-   ***User*** ***Issues***

The Group should focus on the issues of: what does the end product look
like? Do they fit users' needs? Are they presented in a format that is
compatible with the users' existent toolsets, environments, and
cultures? Do they provide value-added to the users or add to their
burdens?

There are different types of "users," such as end users, PM, S & T, test
& evaluation (T & E), contractors, etc. PM takes requirements, as
described by users, goes through detailed analysis based on the
knowledge of actual performance data, experimental results, and
objective capabilities, and specifies the system. The analysis process
is iterative between PM and the users. It is conceivable that the
resulting specification includes a particular autonomy level for the
subject UMS, which further dictates levels along the three axes. PMs do
not want over-constraining tools that may limit their ability to specify
systems. Well-defined metrics and definitions could help producing
concise specifications. A cited example of successful products is, in
the Army, when one specifies "cold," people pretty much knows what it
means. A term like semi-autonomous is useful in spec's, but it covers a
wide range of possible capabilities. It would be useful when everyone
knows what it means upon seeing autonomy level N in the spec.
Contractors know exactly how to manufacture the system.

For T & E community, detailed metrics and measures are helpful. Tests
need to be designed for the definitions---how does one test whether a
UMS meets level N as specified?

For S & T, they can assess technical capability and availability and
project future upgrades in terms of levels.

A concern is whether the ALFUS metrics would cause existent programs
such as FCS to have to re-spec and create contractual problems. It was
felt that the ALFUS framework is aimed at providing generic metrics to
help system specification and evaluation. The latter would be applicable
to existent programs (need additional input on this discussion).

***Workshop Summary, and Actions 1330 -- 1600***

-   ***Integrated View-Framework Format, a Workshop Consensus***

> We converged on a notion that the detailed ALFUS metrics may be
> represented as a table, as shown in Figure 1.

![](media/image1.jpeg){width="1.7145833333333333in"
height="2.3520833333333333in"}

Figure 1: Detailed autonomy level matrix

> A UMS may have a number of major functions, including mobility,
> sensors, weapons, embedded testing equipment, etc. Each may use such a
> table, as shown in Figure 2.

![](media/image2.jpeg){width="2.270138888888889in" height="2.28125in"}

Figure 2: Applying ALFUS Detailed Model

> This tool may be helpful to allow PM to break down requirements and
> identify autonomy levels at the three axes, which should be helpful.
>
> These level assessment needs to be specific to the operations and not
> generic.

We, however, also discussed other possible uses of the detailed metrics
in the three axes. For example, instead of using levels, just use the
metrics sets to profile the UMS capability, for example, what kinds of
environments that the vehicle is capable of handling.

-   ***Are we ready to open to LSI and industry in general?***

The WG's plan, since its first workshop, was to open to everyone when
the main structure of the framework has been established. We felt that
we were very close, but not 100% ready yet. However, since FCS was our
immediate concern, the WG decided to invite LSI to our next workshop.
Also to be invited were the Defense Acquisition University (?)
personnel.

-   ***Campaign strategy***

> The issue of a formal, government-wide adoption of ALFUS results was
> raised again. This would involve a significant amount of effort and
> costs. One issue is how high levels in agencies should we seek the
> approval. A suggestion was to be adopted in JTA, in the defense
> domain. Another suggestion was for ALFUS to be corresponded to NATO
> STANAG's in certain forms (participation, collaboration, cross
> referencing, adoption, etc.). Participants were urged to forward our
> results up their individual chain of command to start the process.
> However, this might be pursued much more effectively once the metrics
> for the framework has been integrated. At that time, perhaps need a
> broad Call for Comments announcement. NIST is to look into how the
> formal process can be pursued and the accompanying resource
> requirements.
>
> Kerry reported that he was on the panel for an American nuclear
> engineering society meeting and he planned to describe ALFUS work.

David's feedback from his community was that they were surprised that
autonomy level metrics has not been in existence yet. So we need to get
words out. Kerry: robotics IPT is interested, LSI is anxious to work
with us.

> AUVSI should be a good forum to publicize the ALFUS work. Should
> pursue articles in its magazine describing our work, participate in
> AUVSI conferences, etc.
>
> The ALFUS web site should allow for viewers to send feedback.
> Currently the site contains Contacts page, to look into dialogue boxes
> and the like to allow direct feedback.
>
> Hui and Woody to complete the paper submitted to SPIE Defense and
> Security Symposium.

-   ***Long-term objectives***

A long-term objective of this WG could be Government-wide unmanned
system performance metrics framework for procurement, testing &
evaluation, etc.

-   ***Next meeting***

First week in May, Atlanta, Airport area.

***Adjourn 1600***