Articulated Human Pose Estimation in Unconstrained Images and Videos

Recent advances in Artificial Intelligence (AI) have resulted in the integration of an ever-increasing number of AI technologies in our daily lives. One of the quests in this direction is to develop autonomous systems that can integrate into our daily lives and interact with us as naturally as another human could do. Humans interact with each other and with their environment in complex ways. Even if we do not speak, our bodies transmit a lot of information such as our behavior, intention, or activity via hand and body gestures, facial expressions and body movements. While the humans can effortlessly detect and interpret such information from subtle and complex body signals, we need to endow these autonomous systems with similar capabilities to integrate them naturally in real-world human environments. Supporting such detailed reasoning would eventually need highly sophisticated computer vision systems – able to estimate human body pose, hand articulations, facial landmarks, and perceive their motion, all in the three-dimensional world space. Besides, the developed methods can also be utilized in numerous practical applications e.g . , robotics, activity recognition, surveillance systems, self-driving cars, driver-assistance, sign-language recognition, human monitoring, sports video analytic, gaming, entertainment industry, etc. Some example applications of the human body understanding can be seen in Figure 1. However, to be applicable in complex real-world scenarios, the developed methods must operate under varied background and lighting conditions, scenes with an unknown number of persons, sever body part occlusions and truncations, and large appearance variations exhibited by the human body. Further, they should be robust against noise due to sensing modalities such as motion blur, depth and scale ambiguities, low image resolution, and compression artifacts. The small functional parts of the human body, such as hands, present additional challenges due to the small size and heavy occlusions. While the problem of articulated human pose estimation has been studied for a long time in the literature, most of the earlier works make strong assumptions and ignore the aforementioned challenges posed by the realistic scenarios. For example, a large number of earlier works for 2D body pose estimation assumes that only a single, pre-localized person is visible in the image [Felzenszwalb and Huttenlocher, 2005, Peng et al., 2018]. For 3D body pose estimation, the used training data is often recorded in indoor settings. Hence, the trained methods are only applicable in similar constrained scenarios. In contrast to body pose estimation, hand pose estimation has received much less attention in the literature and only a few works exist that address this challenging problem from only RGB images [Zimmermann and Brox, 2017, Mueller et al., 2018]. The goal of this thesis, therefore, is to address the challenges mentioned above and fill the gaps in the literature. To this end, the thesis focuses on developing computational methods for articulated pose estimation of humans from completely unconstrained images and videos. It starts by presenting a method for 2D body pose estimation of multiple, potentially truncated and occluded, people (Section 2). As a natural next step, it then addresses the problem of multi-person pose estimation and tracking in videos. The methods developed for pose estimation in images cannot be applied directly to this problem since in this case persons should also be associated over time. To this end, it presents an approach that jointly models pose estimation and tracking in a single formulation (Section 3). Furthermore, it introduces the PoseTrack benchmark which is a new large-scale benchmark for videobased multi-person human pose estimation and tracking. To date, the proposed dataset contains more than 1300 annotated videos and comprises more than 215K pose annotations. It also presents an extensive experimental study on the recent approaches for multi-person pose estimation and tracking and provides an analysis of the strengths and weaknesses of the state-of-the-art (Section 4). Given the 2D pose trajectories of a person, the thesis then presents a method to refine the pose trajectories by exploiting the information about human activities (Section 5).