# Presentation: 509855

## Jean Scholtz 

- Human Robot Interaction

- Assessment panel    2/2005

## Human Robot Interaction

- “Basic taxonomies and research issues must be identified._ _Research-related issues which should be addressed within the next 3 years are: metrics, toolboxes for interfaces, establishment of principles of user-centered design, and how to incorporate the contributions from broader communities (AI, Engineering, Psychology, etc.)”
- One of five findings from DARPA/NSF sponsored workshop on HRI, Sept. 2001

- Assessment panel    2/2005

## NIST HRI work 

- On-road autonomous driving (DARPA MARS; fy02,fy03)    ~600K
- Off-road autonomous driving (ARL, fy03)  ~70K
- USAR (DHS, fy04,fy05)  ~100K
- EOD (NIJ, DHS, fy04,fy05)  ~350K
- NASA P2P (fy05- fy08)  ~140K
- Baseline development (strs, fy04,fy05)

- Assessment panel    2/2005

## HRI vs. HCI

- User is often remote from the robot
- User’s actions depend on task + information about the environment
- Operation continues even with degraded sensors/conditions on the robot
- Users sometimes need to interact with varying levels of autonomy
- Difficult to replicate exact paths of robots for experiments
- Safety issues with robots
- Dynamic system
- Robot, world and user’s tasks

- Assessment panel    2/2005

## Where is the information?

- Robots are being used to find information in dangerous environments and to transmit that information back to allow planning and decision making
- Examples:
- USAR:  structural damage, victim triage
- EOD:  type of IED
- On-road/Off-road driving:  surveillance
- NASA: status of task

- Assessment panel    2/2005

## Metrics and Methodologies

- Situation awareness
- Critical incident analysis
- Workload measures
- Usability requirements evaluation specifications
- GOMS analysis

- Assessment panel    2/2005

## Situation Awareness

- Critical to evaluate how well user interfaces facilitate situation awareness
- SAGAT
- Situation awareness global assessment technique (Endsley)
- Implemented this for on-road driving
- Have a baseline UI and the simulation, data collection tool
- Time to acquire SA
- Data collected during a technical readiness level assessment
- Compared effects of terrain on SA acquisition time/ intervention time
- Useful in predicting manpower needs

- Assessment panel    2/2005

## SAGAT Implementation

- Results look promising
- 	Need refinement for SA level 2
- 	Need to validate, move to multiple vehicles

- Assessment panel    2/2005

## Critical Incident Analysis

- Defined categories of critical incidents for USAR
- Navigation (local, global)
- Obstacle extraction
- Victim identification
- Collected data during USAR competitions
- Developed guidelines for user interface design

- Assessment panel    2/2005

## Guidelines for HRIfrom studies conducted at USAR competitions

- _Enhance awareness._  Provide a map of where the robot has been.  Provide more spatial information about the robot in the environment to make operators more aware of their robots’ immediate surroundings.
- _Lower cognitive load_. Provide fused sensor information to avoid making the user fuse the data mentally.
- _Increase efficiency_.  Provide user interfaces that support multiple robots in a single window, if possible.  In general, minimize the use of multiple windows.
- _Provide help in choosing robot modality_.  Provide the operator assistance in determining the most appropriate level of robotic autonomy at any given time.

- Assessment panel    2/2005

## Workload Measures

- Data collected from EOD tasks
- Found relationship between NASA TLX scores for mental workload, # clicks and mode changes
- Developed set of primitive EOD tasks
- Conducting more evaluations to refine these
- Using data from civilian EOD reports for further refinement

- Assessment panel    2/2005

## Usability requirements evaluation specifications

- Collecting data from first responders in workshop settings
- Using different types of scenarios; recon and primary search tasks
- Developing
- Content of use
- Initial training/continual training requirements
- Tasks that have to be supported
- Effectiveness specifications
- Using this data as a case study for Requirements CIF (you’ve already heard about this)

- Assessment panel    2/2005

## Human-Robot Dialogue

- Metrics under consideration include:
- # turns between user and robot to accomplish “task”
- % content turns/ % clarification turns
- Types of clarification
- What
- When
- Where
- How
- % time human is in control/ % time robot is in control
- Task flow:  appropriateness of interruptions by robot
- Comparison to human-human team (time, dialogue, success of task)
- User satisfaction
- Task time
- Task success
- Will develop coding schemes for analyzing video data collected during exercises
- May need a taxonomy of “tasks”

- Assessment panel    2/2005

## GOMS Analysis(Goals, Operators, Methods, Selection Rules)

- GOMS is used to predict time for expert behavior (# mental operations, # physical operations)
- Modifications are needed to different GOMS methods (keystroke, NGLGOMS, CPM GOMS)
- Due to decision making of operator based on video and other sensory data
- Doing several operations in parallel (driving and watching sensory data)
- Approach
- Using data collected from USAR and EOD to develop the modifications needed
- If-then statements, goto statements
- Dealing with timing for analyzing complex video scenes

- Assessment panel    2/2005

## Developing USAR “state of the art” Baseline

- Modifying an ActivMedia robot
- with 360 degree camera controlled by head tracking unit
- Haptic feedback for victim id and obstacle avoidance
- Game boy controller to allow simultaneous driving and camera manipulation
- Fusion of sensor data in UI
- Development of interaction techniques for levels of autonomy
- Will introduce at USAR competitions in an actual “mission”

- Assessment panel    2/2005

## STRS HRI Research

- USAR workshop, Italy, 2004
- Goal is to have off-the –shelf components for new teams to jump start their work
- Developed baseline UI in USAR workshop in 12 hours
- Learned a visual language
- Designed and implemented UI
- Hooked up components to robot
- Drove in small USAR arena

- Assessment panel    2/2005

## Impact

- New field
- First HRI only conference to be held in 2006
- NIST is recognized as a major player in the evaluation of human-robot interaction
- Important to recognize the need for evaluation early
- Faster progress
- Ability to learn from other research

- Assessment panel    2/2005

## Publications

- Drury, J.  Scholtz, J.  and Yanco, H. 2003.  “Awareness in Human-Robot Interactions.”  *Proceedings of the IEEE Conference on Systems, Man, and Cybernetics*, Washington, DC, Oct. 2003.
- Drury, J., Hestand, D., Yanco, H., and Scholtz, J.  2004.  Design Guidelines for Improved Human-Robot Interaction.  Poster at CHI 2004, Vienna, Austria, April 24-29.
- Scholtz, J.  2004.  The Effect of Situation Awareness Acquisition in Determining the Ratio of Operators to Semi-Autonomous Driving Vehicles.  SPIE Interaction Symposium:  Optics Ease.  Oct. 25-28, Philadelphia, PA.
- Scholtz, J.   2003.  Theory and Evaluation of Human Robot Interactions, *Hawaii International Conference on System Science 36 (HICSS 36)*.  Hawaii, Jan., 2003.
- Scholtz, J.  2002.  Human-robot Interactions:  Creating Synergistic Cyberforces, in Alan C. Schultz and Lynne E. Parker (eds.), *Multi-Robot Systems: From Swarms to Intelligent Automata*, in Kluwer,
- Scholtz, J. 2002.  “Evaluation methods for human-system performance of intelligent systems.” In *Proceedings of the 2002 Performance Metrics for Intelligent Systems (PerMIS) Workshop*.
- Scholtz, J., Antonishek, B., and Young, J.  2004.  Operator Interventions in autonomous Off-road Driving:  Effects of Terrain.   System, Man, and Cybernetics, Oct., Den Hague.
- Scholtz, J., Antonishek, B., and Young, J.  2004.  Evaluation of Human-robot Interaction in the NIST Reference Search and Rescue Test Arenas.  In Proceedings of the Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, MD, August. 2004.
- Scholtz, J., Young, J., Drury, J., and Yanco, H.   2004.  Evaluation of Human-Robot Interaction Awareness in Search and Rescue.  International Conference on Robotics and Automation (ICRA 2004), New Orleans, April 26- May1. 2004.
- Scholtz, J., Antonishek, B., and Young, J.  2004.  Evaluating human-robot interfaces:  development of a situational awareness assessment methodology, *Hawaii International Conference on System Science 37 (HICSS 37)*.  Hawaii, Jan. 2004.
- Scholtz, J., Antonishek, B., and Young, J.  2003.  Evaluation of Operator Interventions in Autonomous Off-Road Driving. In *Proceedings of the Workshop on* *Performance Metrics for Intelligent Systems, (PERMIS).*  Gaithersburg, MD,  Sept. 2003.
- Scholtz, J. and Bahrami, S.  2003.  Human-Robot Interaction:  Development of  an Evaluation Methodology for the Bystander Role of Interaction.    *Proceedings of the IEEE Conference on System, Man, and Cybernetics, 2003*.  Washington, DC.  Oct. 2003.
- Yanco, H., Drury, J. and Scholtz, J. 2004.    “Beyond Usability Evaluation:  Analysis of Human-Robot Interaction at a Major Robotics Competition.”. *Human-Computer Interactio.,*Vol. 19 (#1&2), 117-150.

- Assessment panel    2/2005

## Other Activities

- CHI 2004 and CHI 2005 tutorial on HRI
- Instructor at HRI summer school, 2004
- Instructor at USAR Robotics workshop, 2004 for HRI
- Editor, special issue of ACM Interactions on HRI (March/April 2005 issue)
- HRI 2006 conference committee
- Associate chair, CHI 2004  Robotics and Transportation

- Assessment panel    2/2005

## Backups

- Assessment panel    2/2005

## Other DARPA/NSF workshop findings

- More extensive interdisciplinary interaction must be motivated. HRI is an intrinsically cross-displinary endeavor. There is a perceived need for cross-disciplinary education and joint work.
- Social informatics is a critical, unexplored arena. While emotional intelligence is needed from some applications, it may be inappropriate for others; therefore, both the issues of how to embody emotional intelligence and when it is useful was suggested as technical goals for the next 3 years
- It is essential to define a small number of common application domains._ _Research in HRI has reached the point where appropriate domains are needed for rigorous evaluation and comparison of results. Sponsor the development of an HRI community infrastructure through online websites and supporting workshops/tutorials for the next three years at major discipline-related conferences.
- Members of the HRI community need field experience.
- http://www.csc.calpoly.edu/~erogers/HRI/HRI-report-final.html

- Assessment panel    2/2005

## SAGAT methodology

- Developed UI, scenarios, questions for different levels of SA, simulation and data collection tools
- Expertise for SA questions extracted from driving training program
- Conducted experiment one
- Redesigned collection mechanism for SA level 2
- Conducted experiment two- validation for SA 1 and SA 3, improvements in SA 2?
- Redesigned UI – sensitivity test

- Assessment panel    2/2005

## Scenarios used

| Scenario # | Speed limit | # Lanes | Vehicle in Lane  | Risks at freeze point during Experiment 3 | Conditions encountered during scenario |
| --- | --- | --- | --- | --- | --- |
| 1 | 45 | 4 | right | Red light, slow speed | Red light; another car |
| 2 | 55 | 6 | right | Excess speed | Over speed limit entire time |
| 3 | 45 | 4 | left | Congestion, slow speed, 2 cars in other lane | Speed limit exceeded; two cars encountered |
| 4 | 45 | 4 | right | Debris blocking lane; car beside, slow speed | Cars encountered |
| 5 | 55 | 6 | right  | Car in front | Speed limit exceeded |
| 6 | 55 | 4 | right | Car to left; car in front | nothing |
| 7 | 55 | 4 | right | Exit in a mile;  slow speed | Low fuel; exit at 2 miles |
| 8 | 45 | 4 | right | Lane blocked by debris; debris on left roadside; car beside, slow speed, congestion | Car passing; traffic stopping |
| 9 | 45 | 4 | left | Debris in front; car behind, slow speed, congestion | Pedestrian and two cars; shifts lanes;  |
| 10 | 45, 35, 45 | 4 | left | Pedestrian on roadside | Speed limits change; right turn made |

- Assessment panel    2/2005

## SA Levels for graphical user interface

| Scenario | Vehicle | Environment | Route |
| --- | --- | --- | --- |
| 1 |  |  | 12.5 |
| 2 | 12.5 |  |  |
| 3 |  |  | 25 |
| 4 |  |  | 37.5 |
| 5 |  | 12.5 | 25 |
| 6 |  |  | 12.5 |
| 7 | 25 |  |  |
| 8 |  | 25 | 25 |
| 9 |  | 12.5 |  |
| 10 |  |  |  |

- SA level 1

- Vehicle and Environment awareness is good
- Route information is more difficult.  Possibly due to textual presentation.

- Assessment panel    2/2005

## SA Level 2

| Scenario | # of risks | omissions– Mean(SD) | additions– Mean(SD) |
| --- | --- | --- | --- |
| 1 | 2 | .38(.52) | .00(.00) |
| 2 | 1 | .00(.00) | .00(.00) |
| 3 | 6 | 1.13(.64) | .50(.53) |
| 4 | 6 | 1.00(1.31) | .25(.46) |
| 5 | 3 | .38(.52) | .25(.46) |
| 6 | 5 | .50(.93) | .00(.00) |
| 7 | 2 | .88(.83) | .13(.35) |
| 8 | 9 | 2.75(1.16) | .25(.46) |
| 9 | 7 | 2.13(1.36) | .13(.35) |
| 10 | 1 | .00(.00) | .00(.00) |

- SA level 2 awareness decreases as number of risk indicators increase.  We need to evaluate how we assess this.

- Assessment panel    2/2005

## SA Level 3

| Scenario | % Correct | % Didn’t know | Correct Rationale (# of responses) |
| --- | --- | --- | --- |
| 1 | 75 | 25 | 4(5) |
| 2 | 100 |  | 7(7) |
| 3 | 100 |  | 8(8) |
| 4 | 87 |  | 8(8) |
| 5 | 100 |  | 8(8) |
| 6 | 100 |  | 7(8) |
| 7 | 100 |  | 7(7) |
| 8 | 87 |  | 7(8) |
| 9 | 100 |  | 8(8) |
| 10 | 87 |  | 7(8) |

- SA level 3 shows good awareness.  This implies to us that we are not assessing SA level 2 correctly.
- We have an agenda to investigate this:
- Ensure that scenarios and SA level 2 and SA level 3 are compatible
- Devise graphical assessment technique

- Assessment panel    2/2005

## Critical Incident Analysis

- Assessment panel    2/2005

## Critical Incident Analysis

| Critical Incident | Overall | Team A | Team B | Team C |
| --- | --- | --- | --- | --- |
| Local Navigation | 15 | 4 | 1 | 10 |
| Global Navigation | 0 |  |  |  |
| Obstacle Encounter | 26 | 6 | 9 | 11 |
| Victim Identification | 0 |  |  |  |
| Vehicle State | 12 | 5 | 2 | 5 |

- Assessment panel    2/2005

## Guidelines

- UIs for USAR should include:
- a frame of reference to determine position of robot relative to environment (and provide awareness of the robot’s surroundings)
- indicators of robot health/state, including which camera is being used, the position(s) of camera(s), traction information, and pitch/roll indicators (to provide better awareness of the robot’s status)
- information from multiple sensors presented in an integrated fashion (to avoid relying  on the operator devising strategies to overcome information fragmentation and facilitate better awareness of the robot’s location and surroundings)
- the ability to self inspect the robot body for damage or entangled obstacles (to provide enhanced awareness of the robot’s status)
- automatic presentation of contextually-appropriate information, such as automatically switching to a rear camera view if the robot is backing up

- Assessment panel    2/2005

## EOD workload

| Task | EOD tech P1 | EOD tech P2 |
| --- | --- | --- |
| 1.  Open door with lever | 12 | 7.7 |
| 2.  Negotiate through large doorway | 1 | 1 |
| 3.  Measure a target with grippers | 12.4 | 7.1 |
| 4.  Inspect under furniture | 20 | 14 |
| 5.  Lift and relocate objects | 20 | 8.9 |
| 6.  Negotiate over tree trunks | 2.7
(this tech did not try the larger objects) | 7.1 |
| 7.  Search under a box truck | 1 | 6.6 |
| 8.  Negotiate a slope |  | 11 |
| 9.  Maneuver in adequate space: 
Maneuver in tight space:  | 9.7 | 5.6 |
| 11. Negotiate steps | 17 | 7.1 |
| 12. Negotiate curbs | 9.9 | 8.1 |
| 13. and 14. Negotiate doorways and knobs  | 1.0  
(again this tech only went through the corridors, did not try the doors) | 14.7 |

- Assessment panel    2/2005

## EOD modes and clicks

| Tasks | Number of modes- P2 | Number of clicks – P2 | Number of modes – P1 | Number of clicks – P1 |
| --- | --- | --- | --- | --- |
| 1.  Open door with lever | 13 | 25 | 29 | 70 |
| 2.  Negotiate through large doorway | 0 | 0 | 0 | 0 |
| 3.  Measure a target with grippers | 8 | 20 | 34 | 85 |
| 4.  Inspect under furniture | 13 | 31 | 33 | 72 |
| 5.  Lift and relocate objects | 13 | 90 | 94 | 240 |
| 6.  Negotiate over tree trunks | 3 | 7 | 3 | 7 |
| 7.  Search under a box truck | 29 | 63 | 6 | 14 |
| 8.  Negotiate a slope | 12 | 32 |  |  |
| 9.  Maneuver in adequate space: 
Maneuver in tight space:  | 3
4 | 7
9 | 3
5 | 7
13 |
| 11. Negotiate steps | 7 | 19 | 8 | 20 |
| 12. Negotiate curbs | 8 | 24 |  |  |
| 13. and 14. Negotiate doorways and knobs  | 48- lever
32 - knob | 129
75 |  |  |

- Assessment panel    2/2005

## KLM model fragment

| Label  | Description | Operator |
| --- | --- | --- |
|  | View video window | V |
|  | Comprehend video window scene | M |
|  | View map | V |
|  | Comprehend map | M |
|  | View proximity sensor display | V |
|  | Comprehend proximity sensor display | M |
|  | Determine which way to move | M |
|  | Home hands to screen | H |
|  | Tap teleoperate button | K |
|  | Home hands to joystick | H |
| Push | Push joystick simultaneously with | E |
|  | 	View video window | V	 |

- Assessment panel    2/2005

## NLGOMSL fragment

- Method for goal: Move robot.
- Determine which direction to move in (M).
- If hands are not already on joystick,
- Home hands on joystick (H).
- Press joystick in correct direction (E) simultaneously with looking at video (V).
- Comprehend video information (M).
- If operator has insufficient situation awareness as a result of motion,
- 	Accomplish goal: Obtain information [about situation].
- If robot mobility is severely constrained by obstacles nearby,
- Accomplish goal: Change mode [to escape mode]
- Accomplish goal: Determine if more motion is needed.
- Return with goal accomplished.
- Method for goal: Determine if more motion is needed.
- Obtain information [about need for moving robot]
- If more motion is needed,
- 	Accomplish goal:  Move robot.
- Return with goal accomplished.

- Assessment panel    2/2005

## CPM GOMS fragment

- Assessment panel    2/2005