## 4 Graph Conversion from Dependency Tree  
To locate regions of interest generating local captions, we drew inspiration from dense captioning approaches [57]. However, unlike these, we simply made use of pre-trained models for object detection [1] and automatic image captioning [2] algorithms. The regional caption generation model consists of two distinct phases. One is an object detection phase that locates the key objects in the image; we used a pre-trained YOLO-V3 model for this purpose. The second phase, which is the image captioning model, takes as the input the regions of the original image that are cropped out according to the YOLO-V3 bounding box predictions. This implementation is very straightforward. However, there is one key factor that must be taken into account: the YOLO-V3 model takes an input image of size \(416\times 416\), but the bounding box predictions come in a variety of sizes; moreover, the image captioning model works only for a specific size of image of \(224\times 224\). Based on the works of [58], our approaches use the centered zero-padding technique. The reasons are that it is one of the fastest techniques, the aspect ratio of the objects in the scenes is not distorted,and, for our purposes, randomized reshaping techniques may lead to different outputs from the image captioning model. Hence, to ensure consistent high-quality outputs from the pre-trained image captioning module, the centered zero-padding technique was used. In this work, we considered a single generated caption per region to avoid unnecessary redundancies. An example of a set of captions for various regions across the image is shown in Figure 3.  
Once we have a set of captions with respect to detected regions of interest in the image, we can generate rich conceptual graphs using the graph construction method proposed in Section 4.1. The graph construction first creates the set of primary entities followed by the edges based on the dependencies; secondary entities are created on an ad hoc basis to accommodate hanging edges. The dependency tree provides rich information about relations between the various words in a sentence. These relations can reveal several aspects relevant to our purposes such as the dependence between subjects, predicates, and objects, modifiers of various objects, actions, and spatial relations based on the type of prepositions used to connect phrases and also demonstrate a hierarchical concept of the scene. For this work, we used the enhanced++ typed dependencies from the Stanford dependency parser [59] and the default English language model for parsing ([[https://nlp.stanford.edu/software/lex-parser.shtml](https://nlp.stanford.edu/software/lex-parser.shtml)]([https://nlp.stanford.edu/software/lex-parser.shtml](https://nlp.stanford.edu/software/lex-parser.shtml)) (accessed on: 25 December 2022)). The Stanford dependency parser was created to provide a simplified depiction of the grammatical links in a phrase such that persons without linguistic knowledge can understand and use them to extract the textual relations. It expresses all sentence relationships as typed dependency relations, instead of the phrase structure representations that have been traditionally used in the computational linguistic research community. All dependencies are expressed using specific tags. The basic version of the Stanford dependency parser introduced 55 unique tags, which were further improved to include more complex grammatical constructs in future versions, namely the enhanced and enhanced++ dependencies. For our task, we created a simplified intermediate representation. A sentence \(S\) is defined as a sequence of words \(\{word_{1},word_{2},\dots,word_{n}\}\). For each word \(word_{i}\), we have an intermediate representation \(h_{i}\):  
\[h_{i}=\langle id_{i},t_{i},l_{i},p_{i},\text{LIST}[r_{k},id_{k},t_{k},|) \tag{1}\]  
Figure 3: An example regional caption generation. Regions are detected using YOLO-V3 [1], and captions are generated using Show, Attend, and Tell [2].