This presentation is released under the terms of theCreative Commons Attribution-Share Alike license.
You are free to reuse it and modify it as much as you want as long as(1) you mention me as being the original author,(2) you re-share your presentation under the same terms.

You can download the sources of this presentation here:
github.com/severin-lemaignan/presentation-ros4hri

ROS for Human-Robot Interaction
What is the new REP-155?
ROSCon | Oct 2022

Séverin Lemaignan
PAL Robotics Senior Scientist AI & Social Interactions

situation assessment

symbolic grounding

symbolic reasoningSYMBOLIC SOCIAL COGNITION FOR ROBOTS

ontologies
cognitive architectures

joint action

perspective takingsocial situation assessment
natural language processing

ROS4HRI

DATA-DRIVEN
HRI

REAL-WORLD
SOCIAL AUTONOMY

large datasets

learning of social policies
theory of mind
robotics forlearning

group dynamics

human-in-the-loop ML

experimental robotics

CHILD-ROBOTtrust
HUMAN FACTORSINTERACTION
engagement
anthropomorphism
responsible AIsocial robotics

participatory design

persuasion

WHY ROS4HRI?

◦ Dealing with humans is actually hard: they keep on disappearing/reappearing;
hard to predict where/when; ‘shape’ known at run-time only, etc.

[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

4

WHY ROS4HRI?

◦ Dealing with humans is actually hard: they keep on disappearing/reappearing;
hard to predict where/when; ‘shape’ known at run-time only, etc.
◦ Widely different requirements depending on application: from ‘2D points’, to fullonline kinematic model[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

4

WHY ROS4HRI?

◦ Dealing with humans is actually hard: they keep on disappearing/reappearing;
hard to predict where/when; ‘shape’ known at run-time only, etc.
◦ Widely different requirements depending on application: from ‘2D points’, to fullonline kinematic model◦ No ROS standard for HRI (nothing, nada, rien!)[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

4

DESIGN REQUIREMENTS◦ Representations application-agnostic: from point-like crowd simulation, tokineastetic teaching, to social interaction[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

5

DESIGN REQUIREMENTS◦ Representations application-agnostic: from point-like crowd simulation, tokineastetic teaching, to social interaction◦ Does not enforce any specific algorithm or perception pipeline[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

5

DESIGN REQUIREMENTS◦ Representations application-agnostic: from point-like crowd simulation, tokineastetic teaching, to social interaction◦ Does not enforce any specific algorithm or perception pipeline◦ However, takes into account what current algorithms can or can not do (eg:kinematic model of human)[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

5

DESIGN REQUIREMENTS◦ Representations application-agnostic: from point-like crowd simulation, tokineastetic teaching, to social interaction◦ Does not enforce any specific algorithm or perception pipeline◦ However, takes into account what current algorithms can or can not do (eg:kinematic model of human)◦ Integrates as much as possible with existing ROS conventions (eg:robot_state_publisher for human forward kinematics)[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

5

SO: WHAT IS ‘ROS4HRI’?

REP-155

specificationcore interfacescore libraries
visualisation and toolingopen ecosystem[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

6

SO: WHAT IS ‘ROS4HRI’?

tf frames

topicshri_msgs

REP-155

specificationcore interfaces

human_description

global parameters

core libraries
visualisation and toolingopen ecosystem[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

6

SO: WHAT IS ‘ROS4HRI’?

libhritf frames

topicshri_msgs

REP-155

specificationcore interfacescore libraries
visualisation and toolinghuman_description

global parameterspyhri

open ecosystem[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

6

SO: WHAT IS ‘ROS4HRI’?

libhritf frames

hri_rvizroshumans (TBD)
topicshri_msgs

REP-155

specificationcore interfacescore libraries
visualisation and toolingopen ecosystem[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

human_description

global parameterspyhri
rqt_human_radar6SO: WHAT IS ‘ROS4HRI’?
hri_person_manager

your node!
libhritf frames

hri_rvizroshumans (TBD)
topicshri_msgs

REP-155
hri_face_detectspecificationcore interfaces

your node!
human_description

core libraries

pyhri

visualisation and toolingopen ecosystem[Mohamed, Lemaignan, ROS for Human-Robot Interaction, IROS 2021]

global parameters

hri_fullbody
your node!

rqt_human_radar6

LIMITED SCOPE

For now, focus on perception only
(research on the ‘behaviours’ side just starting)7

LIMITED SCOPE

For now, focus on perception only
(research on the ‘behaviours’ side just starting)

Initially, ROS1 only
(ROS2 support planned for next year)

7

HUMAN REPRESENTATION: PERMANENT VS TRANSIENT IDS

8

HUMAN REPRESENTATION: PERMANENT VS TRANSIENT IDS

8

HUMAN REPRESENTATION: PERMANENT VS TRANSIENT IDS

8

HUMAN REPRESENTATION: PERMANENT VS TRANSIENT IDS

8

HUMAN REPRESENTATION: PERMANENT VS TRANSIENT IDS

8HUMAN REPRESENTATION: PERMANENT VS TRANSIENT IDS

8

TOPICS STRUCTURE: FACES

Currently tracked faces: /humans/faces/tracked [hri_msgs/IdsList]

9

TOPICS STRUCTURE: FACES

Currently tracked faces: /humans/faces/tracked [hri_msgs/IdsList]
Then, under /humans/faces/<faceID>/ (eg /humans/faces/bf3d):
NameMessage type

Description/roi/cropped/frontalized/landmarks/facs

hri_msgs/NormalizedRoI2D
sensor_msgs/Image
sensor_msgs/Image
hri_msgs/FacialLandmarks
hri_msgs/FacialActionUnits/expression/softbiometrics

hri_msgs/Expression
hri_msgs/SoftBiometrics

Region of the face in the source image
Cropped face
Frontalised face
The 2D facial landmarks extracted from the face
The presence and intensity of facial action units
found in the face
The expression recognised from the face
Soft biometrics like age and gender of the face

9TOPICS STRUCTURE: BODIES/humans/bodies/tracked [hri_msgs/IdsList]/humans/bodies/<bodyID>/ (eg /humans/bodies/5e4d):
Name

Message type

Description/roi/cropped/joint_states/skeleton2d/posture/gesturehri_msgs/NormalizedRoI2D
sensor_msgs/Image
sensor_msgs/JointState
hri_msgs/Skeleton2D
hri_msgs/BodyPosturehri_msgs/Gesture

Region of the whole body in the source image
Cropped image of the body
The joint state of the human body
The 2D points of the detected skeleton
Recognised body posture (sitting, standing)Recognised symbolic gesture

3D pose? tf frames from joint state + human URDF! I’ll come to it in a minute.
10

TOPICS STRUCTURE: VOICES/humans/voices/tracked [hri_msgs/IdsList]/humans/voices/<voiceID>/ (eg /humans/voices/dde2):Name

Message type

Description/audio/features

audio_common_msgs/AudioData
hri_msgs/AudioFeatures/is_speaking

std_msgs/Bool/speech

hri_msgs/LiveSpeech

Separated audio stream for this voice
INTERSPEECH’09 Emotion challenge low-level
audio features
Whether or not speech is recognised from thisvoice
The live stream of speech recognized via an ASR
engine11

TOPICS STRUCTURE: PERSONS/humans/persons/known [hri_msgs/IdsList]/humans/persons/tracked [hri_msgs/IdsList]

12

TOPICS STRUCTURE: PERSONS/humans/persons/known [hri_msgs/IdsList]/humans/persons/tracked [hri_msgs/IdsList]/humans/persons/<personID>/ (eg /humans/persons/45ff):
Name

Message type

Description/face_id/body_id/voice_id/alias/anonymous/engagement_status/location_confidence

std_msgs/String (latched)
std_msgs/String (latched)
std_msgs/String (latched)
std_msgs/String (latched)
std_msgs/Bool (latched)
hri_msgs/EngagementLevel
std_msgs/Float32/name/native_language

std_msgs/String
std_msgs/String

Face matched to that person (if any)
Body matched to that person (if any)
Voice matched to that person (if any)
ID of other person, if alias
if true, anonymous person (not permanent ID)
engagement status of the person with the robot
Location confidence; 1 means ’person currently
seen’, 0 means ’person location unknown’Name, if known
IETF language codes like EN_gb, if known

12

TOPICS STRUCTURE: GROUPS/humans/groups/tracked [hri_msgs/IdsList]/humans/groups/<groupID>/ (eg /humans/groups/56ef2):
Name

Message type

Description/members

hri_msgs/IdLists

Person ID of the members of the group

Attention: extension not yet in the REP-155!

13

TOPICS STRUCTURE: INTERACTIONS

Under /humans/interactions/:
Name

Message type

Description/gaze

hri_msgs/Gazeestimated gazing behaviours

14

HUMAN PHYSICAL REPRESENTATION◦ standard ROS pipeline: jointstate (eg OpenPose,mediapipe) ->robot_state_publisher +URDF◦ URDF generated on the fly,based on person’s height(xacro params)◦ Follows REP-120 as much aspossible.
15

TOOLING

16

NEXT STEPS FOR THE REP?

◦ REP-155 is currently an open REP: to become an accepted recommendation, we
need community feedback.

17

NEXT STEPS FOR THE REP?

◦ REP-155 is currently an open REP: to become an accepted recommendation, we
need community feedback.
◦ several PRs are under discussion: specification of groups, HRIHeader instead ofone topic per ID,...

17

NEXT STEPS FOR THE REP?

◦ REP-155 is currently an open REP: to become an accepted recommendation, we
need community feedback.
◦ several PRs are under discussion: specification of groups, HRIHeader instead of
one topic per ID,...
◦ However, right time to make your nodes REP-155-conformant! (and to add themto wiki.ros.org/hri)17

NEXT STEPS FOR THE REP?

◦ REP-155 is currently an open REP: to become an accepted recommendation, we
need community feedback.
◦ several PRs are under discussion: specification of groups, HRIHeader instead ofone topic per ID,...
◦ However, right time to make your nodes REP-155-conformant! (and to add themto wiki.ros.org/hri)◦ ...and we miss a good illustration for ROS4HRI! Josh? :wink: :wink:

17

START BUILDING YOUR PIPELINES!

18

THREE LINKS TO RULE THEM ALL◦ Code: github.com/ros4hri◦ Documentation: wiki.ros.org/hri◦ Feedback: github.com/ros-infrastructure/rep#ROS4HRI(and on Sunday: ROS4HRI tutorial @IROS. Join if you can!)20