Abstract: Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language. Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents. To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks. Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures. Afterwards, we employ dynamic memory networks to attend to the large body of facts in the knowledge graph and images, and then perform reasoning over these facts to generate corresponding answers. Extensive experiments demonstrate that our model not only achieves the state-of-the-art performance in the visual question answering task, but can also answer open-domain questions effectively by leveraging the external knowledge.

Introduction: Visual Question Answering (VQA) is a ladder towards a better understanding of the visual world, which pushes forward the boundaries of both computer vision and natural language processing. A system in VQA tasks is given a text-based question about an image, which is expected to generate a correct answer corresponding to the question. In general, VQA is a kind of Visual Turing Test, which rigorously assesses whether a system is able to achieve human-level semantic analysis of images BIBREF0 , BIBREF1 . A system could solve most of the tasks in computer vision if it performs as well as or better than humans in VQA. In this case, it has garnered increasing attentions due to its numerous potential applications BIBREF2 , such as providing a more natural way to improve human-computer interaction, enabling the visually impaired individuals to get information about images, etc. To fulfill VQA tasks, it requires to endow the responder to understand intention of the question, reason over visual elements of the image, and sometimes have general knowledge about the world. Most of the present methods solve VQA by jointly learning interactions and performing inference over the question and image contents based on the recent success of deep learning BIBREF3 , BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , which can be further improved by introducing the attention mechanisms BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . However, most of questions in the current VQA dataset are quite simple, which are answerable by analyzing the question and image alone BIBREF2 , BIBREF13 . It can be debated whether the system can answer questions that require prior knowledge ranging common sense to subject-specific and even expert-level knowledge. It is attractive to develop methods that are capable of deeper image understanding by answering open-domain questions BIBREF13 , which requires the system to have the mechanisms in connecting VQA with structured knowledge, as is shown in Fig. 1 . Some efforts have been made in this direction, but most of them can only handle a limited number of predefined types of questions BIBREF14 , BIBREF15 . Different from the text-based QA problem, it is unfavourable to conduct the open-domain VQA based on the knowledge-based reasoning, since it is inevitably incomplete to describe an image with structured forms BIBREF16 . The recent availability of large training datasets BIBREF13 makes it feasible to train a complex model in an end-to-end fashion by leveraging the recent advances in deep neural networks (DNN) BIBREF2 , BIBREF5 , BIBREF7 , BIBREF10 , BIBREF12 . Nevertheless, it is non-trivial to integrate knowledge into DNN-based methods, since the knowledge are usually represented in a symbol-based or graph-based manner (e.g., Freebase BIBREF17 , DBPedia BIBREF18 ), which is intrinsically different from the DNN-based features. A few attempts are made in this direction BIBREF19 , but it may involve much irrelevant information and fail to implement multi-hop reasoning over several facts. The memory networks BIBREF20 , BIBREF21 , BIBREF22 offer an opportunity to address these challenges by reading from and writing to the external memory module, which is modeled by the actions of neural networks. Recently, it has demonstrated the state-of-the-art performance in numerous NLP applications, including the reading comprehension BIBREF23 and textual question answering BIBREF24 , BIBREF22 . Some seminal efforts are also made to implement VQA based on dynamic memory networks BIBREF25 , but it does not involve the mechanism to incorporate the external knowledge, making it incapable of answering open-domain visual questions. Nevertheless, the attractive characteristics motivate us to leverage the memory structures to encode the large-scale structured knowledge and fuse it with the image features, which offers an approach to answer open domain visual questions.

Question: What are the baselines for this paper?
Answer: Ablated versions of the full model (without external knowledge, without memory network); alternative VQA methods: LSTM-Att, MemAUG, MCB+Att, MLAN