Unlocking artificial intelligent video understanding through object-centric relational reasoning
Creators: name: Zhou, Honglu
Publisher: name: No Publisher Supplied
Year: 2023

description: Modern visual systems have achieved high precision in tasks such as face detection, scene segmentation, and object localization, through their emphasis on analyzing static images and low-level pixel data. However, our experience as humans is deeply shaped by our perception of objects, their relations, and what happens to the objects in the visual world. Therefore, rather than building a machine that attempts to attain artificial visual intelligence solely from the static image or directly from the low-level pixels, we may need to accomplish that from dynamic relational reasoning about objects in videos. Relational reasoning reasons the saliency of objects and their dynamic interactions. In this context, object refers to an entity, such as a discrete concept, a physical object, or a person. Towards intelligent video understanding, which is crucial for machines to interpret and interact with the dynamic visual world that we humans live in, we propose using relational reasoning about objects in videos to address the key challenges in video understanding: (i) spatiotemporal reasoning, (ii) high-level semantic understanding, and (iii) rudimentary video representation learning. In this dissertation, we begin with considering the problem of spatiotemporal reasoning, and the proxy task to evaluate on is Snitch Localization, where multiple objects move and are frequently occluded by one another. We propose Hopper, which uses a Multi-hop Transformer for reasoning object permanence through multi-step relational modeling. Hopper demonstrates the effectiveness of multi-step relational reasoning for spatiotemporal reasoning. Moving towards the non-trivial higher-level semantic understanding of videos, we tackle the task of human Group Activity Recognition that detects the activity collectively performed by a group of actors in a video. We propose COMPOSER, a Multiscale Transformer based architecture that performs multi-scale relational reasoning to learn group activity compositionally. COMPOSER shows relational reasoning additionally armed with multi-scale modeling that explicitly models hierarchy and compositionality, facilitates an accurate high-level semantic understanding of videos. Last but foremost, video representations are the basis of video understanding; improved video representations could potentially enhance video system performance. We propose Paprika, a pre- training procedure that leverages graph-based relational reasoning to learn a generalizable video representation, targeting an improved downstream procedural understanding ability in instructional videos, and thus paving the way for robot learning from human-provided demonstrations to provide humans with better assistance and interactions. Contributions: Our solutions outperform the current state-of-the-art methods on several video datasets. Our results indicate that object-centric relational reasoning is the key towards intelligent video understanding. By introducing inductive biases such as multi-step reasoning, multi-hierarchy, graph-structured knowledge, suitable modalities, and external sources of information, relational reasoning can be further enhanced, ultimately leading to a machine-learning-based video system that is more efficient to develop, a more comprehensive understanding of videos, and a more effective ability to solve a variety of video-related tasks.