Parametric Human Movements: Learning, Synthesis, Recognition, and Tracking

This thesis aims at the learning of action primitives and their application on the perceptive side (tracking and action recognition) and the generative side (synthesizing movements for robot control). A major motivation is to use a unified primitive representation which represents the primitives in a context-dependent way and can be used for both, perception and generation. A motivation for a unified model comes from neuroscience: certain regions in the human brain are active during recognition and generation of certain actions. Suitable models in such a context are generative stochastic models such as the hidden Markov model (HMM). This thesis considers arm actions in a table-top scenario. The actions are for example: pointing to, reaching for, and relocating an object. These actions are highly context dependent, i.e. they depend on the actual locations of the objects. A promising extension of the commonly used HMM is the parametric HMM (PHMM), which has been introduced previously in the context of parametric gestures. The PHMM can learn and generalize the dependency of a movement trajectory on a set of parameters, as, for example, object locations. This capability is crucial in both cases: the recognition and the generation of arm movements. Besides the training of the PHMMs and the accuracy of this action representation, the following two applications are considered: imitation and intertwined tracking and recognition. In the robotics imitation application, a humanoid robot is enabled to relocate objects placed on a table-top. An important aspect is to synthesize movements on the robot such that the robot reaches for the right object in order to grasp it. The learned parametric HMM enables the robot to generate an action for specific parameters given by the location of the object for which it should reach. Thus, the robot is able to achieve the desired effect by its actions as required in the current context. An interesting aspect of the imitation application is that the actions and their effects are learned on the basis of another embodiment, i.e. the PHMMs are trained on human performances. Both, the actions and their corresponding effects need to be mapped to the robot’s embodiment. In the tracking and recognition application, the action primitives are used to define a space of possible actions and action sequences, where the sequences are defined by a grammar. The tracking is performed in the action space which reduces the dimensionality of the tracking problem and allows for recognition. From the tracking perspective, it is crucial that the parameters of the action primitive can be used to adapt the primitive to the actual appearance of the tracked motion, since the actions visually appear different when applied to different object locations. From the recognition perspective, it is necessary to recognize that an action has been performed, but in order to understand the full semantic of an action, also the recovery of the action parameters is important. For example, in order to identify an object to which a person is pointing, it is necessary to identify the location to which the person is pointing. This is provided by the estimate of the action parameters. Findings of the thesis are: a method is developed thus that PHMMs can be utilized to represent parametric actions/primitives not only for recognition but also for the synthesis of accurate trajectories. This is evaluated by experiments. The experiments with the humanoid robot show that the proposed methods of reproducing the actions on the robot and synthesizing actions from the parametric models (trained on demonstrations by a person) enable the robot to accomplish tasks by generating appropriate actions with the desired effect. In a rule-learning application the robot learns what to do with several objects. The implemented framework for recognition and tracking makes use of several action models, which are trained on demonstrations. The framework runs in real time and allows one to recover the pose and recognize actions and their parameters online. The experiments show that different actions can be distinguished and that the primitive actions of complex actions being tracked can be recognized even from a single view in an view invariant manner. Acknowledgments I would like to take this opportunity to thank all the people who have supported me during my PhD thesis. First, I want to thank my supervisor Assoc. Prof. Dr. Volker Krüger for all his guidance, fruitful discussions, advice and, especially, his patience over the past 4 years. I also thank my former college Prof. Daniel Grest for his support and collaboration in the beginning of my work. I appreciated very much the collaboration with Prof. Dr. Aleš Ude during my three month residence in Slovenia, where I could use a humanoid robot for research. I also thank my current and past group members Pradeep Kumar Reddy, Baby Sanmohan, Gaurav Sharma and Kasper Broegaard Simonsen for their company and discussions. Special thanks go to my parents, my Aunt Jane, and Uncle Zig for their moral support.