Project on Visual Commonsense Reasoning

1 Credits 2 Introduction Human understand the world by recognizing objects in the context and reasoning their relationships. Trying to mimic the ways human brain recognize objects in visual scene, deep neural networks utilizing convolutional layers and pooling layers to extract visual features for various tasks, e.g. object recognition, object detection, semantic segmentation, etc. Similarly, some models learn word embeddings for neural language processing tasks with recurrent neural networks and conditional random field, e.g. name entity recognition, relation extraction, etc. However, one significant drawback of such deep feature learning scheme is that the models suffer from poor generalization and reasoning power, which leads to difficulties in transferable feature learning and multi-model learning. To solve this problem, it is important to go beyond recognition tasks to cognition tasks, which aim to understand the relationships on the basis of certain attributions learnt by well established recognition models. Towards this approach, some methods try to build the connection of objects by learning feature embeddings and relation networks. Others try to leverage triplet relationship or external knowledge base to construct and model commonsense knowledge to understand the interaction of objects. These models are performed on either visual question answering problem or commonsense problem until the introduction of visual commonsense reasoning problem has been proposed. The problem visual commonsense reasoning explicitly requires three types of evaluation: question to answering (Q → A), question and answering to reasoning (QA → R) and question to answering and reasoning (Q → AR). These three problems gradually increase the need of reasoning power of the model, especially when sometimes the answer could not directly be reflected from the appearance of the image itself, e.g. when a waiter serves the coffee, the image does not contain any information about where the coffee is going to be placed, however, from other people’s reaction we are able to infer it, which requires commonsense knowledge outside of the image. We argue that there are two main challenges in VCR problems compared to VQA and commonsense problems: 1) an explicit knowledge graph is needed to build to model the background commonsense knowledge outside of the images; 2) the visual features and the semantic features needs to be aligned in the knowledge graph to form a uniform knowledge source so that the model could directly connect the questions with the visual scenes. Thus we propose cross-modal relation graph method, a novel knowledge graph based visual commonsense reasoning method that contains three component: 1) commonsense knowledge graph that learns commonsense concepts based on triplet relationships; 2) visual relation graph that built on the input image with each object and its learnt attribute descriptors; 3) question relation graphs that built on the input question. The model then uses graph matching techniques to match both the visual and question relation graphs to the commonsense knowledge graph and returns the confidence of the matching. After that, the model could trace back to the commonsense knowledge graph to find the reasoning of the matching. This model could naturally address the three tasks proposed by VCR. For Q → A, we seek for correct answer to be the answer with highest matching confidence; for QA → R, we trace back to the commonsense knowledge graph given by the matching