HI RICHARD - PERSONALIZE YOUR ROBOTWITH THE COB_PEOPLE_PERCEPTION STACK
Richard Bormann, Thomas Zwölfer, Jan Fischer
Fraunhofer-Institute for Manufacturing Engineering and Automation IPAFolie 1© Fraunhofer

Introduction

Folie 2
© Fraunhofer

Introduction
 personalized greeting appropriate behavior according to the user’s preferences usercenteredservices,reminders,etc.

Folie 3
© Fraunhofer

HelloMrs. Ebner.

Outline
This talk explains the cob_people_perception stack to detect and identifypeople. Details on implementation, interfaces and usage will be provided.

Folie 4
© Fraunhofer

Approach

Implementation

Interfaces

Usage for Applications

Approach for Person Detection and Identification
 3 steps: 1. head detection, 2. face detection and 3. face identification

2.

1.
DanielFolie 5
© Fraunhofer

3.

Approach for Person Detection
 head detection in depth image ofthe scene with Viola-Jonesclassifier cob_people_perception comeswith a well-trained classifiercascade for depth data from theKinect

Folie 6
© Fraunhofer face detection in color image
patches of the head regions withViola-Jones classifier cob_people_perception uses
trained OpenCV classifier

Approach for Person Identification
 Fisherfaces on gray image of the face basically a projection-based method generates the basis of a “face space” from training samples
 minimizes intra-class variance and maximizes inter-class variance face similarity judged by similarity of face space vectors
 intuition from similar method Eigenfaces:
known people=
average image + weighted sum of eigenfaces

averageimage

eigenfaces(face space)
Folie 7
© Fraunhofer

images from: Matthew Turk and Alex Pentland
Journal of Cognitive Neuroscience 1991 3:1, 71-86

Approach for Person Identification
 increased robustness of identification through illumination normalization,e.g. by gamma transformcombined withdiscrete cosine transform
coefficient scaling head pose alignment,e.g. by detection of eyes andnose and image warping

Folie 8© FraunhoferVideo – Person Detection and Identification

Folie 9© Fraunhofer

Implementation
 functionality
 person detection and identification continuously or on demand capture training data for face identification
 build face recognition models for all known people or just a subset
 update or delete data for face identification
 division into specialized modules that run as nodes advantages and drawbacks of high modularity

 + exchangeable algorithms
 + stable interfaces
 + uncomplicated team work on different nodes
 - message transfer overhead

Folie 10
© Fraunhofer

Implementation

SensorPointCloud2
Image

Sensor Message
Gateway
PointCloud2
Image

Head Detector

Actions :add_data_server
update_data_server
delete_data_server

ColorDepthImageArray
Face Capture
NodeFace Detector

DynamicReconfigureColorDepthImageArray
Actions :load_model_server

S erv ices :start_recognition
stop_recognition

Folie 11
© Fraunhofer

Face Recognizer

DetectionArray

Actions :get_detections_server
Coordinator

S erv ices :capture_image
finish_recording

DetectionArray

Detection
Tracker

People Detection
Display

Interfaces – Recognizing People
 continuous stream of detections and identifications
 open camera message gateway with service start_recognition
message type: recognitionTrigger.srv:# request message
float32 target_frame_rate--# response message
 close camera message gateway with service stop_recognition
message type: Empty.srv

Folie 12
© Fraunhofer

Interfaces – Recognizing People
 single detection and identification on demand request via action get_detections_server (opens message gateway
automatically if closed and resets its status afterwards)
message type: getDetections.action#goal
float32 maximum_message_age
float32 timeout--#resultcob_people_detection_msgs/DetectionArray detections--#feedback

Folie 13
© Fraunhofer

Interfaces – Capturing Training Images of new Persons
 m anual capturing mode: call service capture_image to record animage, and finish_recording to quit
 autom atic capturing mode: automatic recording of x images with atimeout of y in between
 request via action add_data_server
message type: addData.action# goal message
string label
int32 capture_mode# 0=manual, 1=continuous
int32 continuous_mode_images_to_capture
float32 continuous_mode_delay--# result message--# feedback message
int32 images_captured
 update labels or delete training data with actions
update_data_server and delete_data_server
Folie 14
© Fraunhofer

Interfaces – Construct Recognition Model build a model for face recognition with Fisherfaces for a specified list ofpersons that have been recorded to the database before request via action load_model_server
message type: loadModel.action# goal message
string[] labels# list of persons to be recognized--# result message--# feedback message

Folie 15
© FraunhoferVideo – Demo Application “Search for a Certain Person”

Folie 16
© FraunhoferUsage get the software from www.ros.org/wiki/cob_people_perception
 compile run a roscore start the Kinect driverroslaunch openni_launch openni.launch launch people detection
roslaunch cob_people_detection people_detection.launch start the client for manual usage …
rosrun cob_people_detection people_detection_client … or just start your own script or state machine to communicate with theperson identification module

Folie 17
© Fraunhofer

Thank you for your interest!
and thanks to the major collaborators within this project
Jan Fischer

Thomas Zwölfer

for more infos visit www.ros.org/wiki/cob_people_perception
Folie 18
© Fraunhofer