## 1 Introduction
### Motivation  
Learning relations between entities is a complex problem given the large number of entity relationship combinations possible. The long-tailed distribution of such triplets makes straightforward classifications extremely difficult. Furthermore, the creation of supervised models for these kinds of problems requires datasets containing millions of images that have graph-based annotations. Graph convolutional networks [4] have been implemented before, but they operate best on a very restricted set of nodes and edges and tend to fall apart for larger class problems such as natural language representation, which can possess large vocabularies and corpora defining the problem space. Throughout the last decade, we have seen the tremendous performance of deep learning approaches in several sectors of computer vision that require rich feature extraction. Scene graph generation also uses this ability of CNNs for building descriptive visual features that can be used for the prediction of several objects and actions. However, most of the time, relation detection depends on other techniques such as message passing [5], probabilistic graphical models [6] or statistical correlations [7] within the datasets. One major issue is that these approaches are not completely explainable. However, our approach is based on the hypothesis that the relational representation is already encoded inside CNNs and can be extracted without other learning techniques. We have already seen how CNNs can be used for generating image captions, and we know for a fact that natural language captions demonstrate relational aspects of an image. Thus, CNNs are capable of encoding relational information in some way that may not be directly represented as graphs, but can be expressed through sequential outputs such as natural language texts. Our approach aims to extract relational information from such image captions instead of directly extracting it from the image. Though the caption generation may not be completely explainable, the proposed  
Figure 1: An example showing generated holistic scene graphs (**right**) from images (**left**) using region-specific dependency parse trees. graph generation from image captions is completely rule-based, hence explainable. As the image becomes more and more complex, automatically generated image captions fail to capture all the concepts hidden within the image. In the work of [PERSON] et al. (2018), it was shown that dense captioning can yield much more extensive information as compared to standard captions. As compared to normal captions, dense captions are much more localized in nature. This approach tends to generate several simple captions for different regions of the image. Finally, for the extraction of relational information from natural language captions, we used natural language processing concepts such as dependency graphs. To summarize, instead of building dedicated models for scene graph generation, our model tends to extract the latent relational information that is inherently encoded in image captioning models and uses detection techniques for region propositions for creating dense concept graphs for images. A block diagram of the proposed approach is given in Figure 2.  
The paper is organized into seven sections. In Section 1, we define the objective of the work and the motivation that guided the proposed approaches. Following that, in Section 2, we talk about the various literature that already exists in the domain of scene graph generation and the respective methodologies and shortcomings. In Section 3, we define the desired characteristics of the holistic scene graph. In Section 4, we introduce the first phase of our approach, which is the generation of graphs from image captions. In Section 5, we discuss the graph merging technique that results in the holistic scene graph. In Section 6, we provide the statistical characteristics of the generated graphs along with the qualitative analysis.