# Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning
## 1 Introduction  
Image captioning has proved its advantage at the naive level by being a virtual assistant or a means of understanding for visually challenged people [1, 2]. The task of image captioning has turned out to be difficult due to varied interpretations of the image by different individuals; thus, making it complex enough to come out with an accurate description of the image with the best possible use of words, and this is even a definitive way of caption generation for images [3]. The model's output is envisioned to describe what is shown in the image utilising the object's properties, but replicating this behaviour in an artificial system is a tedious and lengthy task; hence the incorporation of intricate techniques, such as deep learning, to solve the task comes into application [4]. Image captioning as compared to image recognition is a complex process, as the former allows the machine to present a sentence in the correct manner encompassing the relevant details of the scene [5]. Advancement in the process of image captioning leads to a better development or usage of a model to achieve significant results and analysis. When the objects present in the considered image are detected and their relationship has been identified, then a caption is generated for the image. The sequence of the words matters the most from which the captions are generated since a readable and grammatically correct sentence is important for understanding [6]. To perform a comparative analysis and concise review of different image captioning methodologies, a detailed study and implementation of both attention-based and non-attention-based image captioning methodologies had been performed. The BLEU score or Bilingual Evaluation Understudy has been taken as the evaluation criteria, which is one of the most trusted evaluation metrics for the quality of text generated by the machine [7,8]. Flicker 30k dataset has also been used on the above architecture that includes about 30,000 images with each image used for five different captions leading to a total of 158k captions providing a benchmark in the domain of sentence-based image captioning [9]. A simple attention-based model, as previously mentioned, can be gruelling at times purposely when the objective laid has abstract word identification in it, which leads to a semantic gap between the words and the vision. To bridge this gap, the transformer model has come into extraction, which can attain semantic and scenic information simultaneously [10]. Hence, an approach for image captioning based on dual-model transformer architecture is proposed, with iterative answer prediction using a dynamic pointer network (DPN) [11]. Given an image, n-dimensional feature vectors are extracted from two modalities, i.e., the text and object. These feature vectors are then passed as an input to a multi-layer transformer which outputs a reinforced representation of these vectors in a common semantic space through intra and inter interactions among the two modalities [12]. Thus, a caption is then generated word-by-word in an auto-regressive manner through the transformer using the dynamic pointer approach. The proposed work concatenates two types of embeddings: one is generated using the image and another is word embedding of that image to feed a given encoder. This dual mode approach creates a highly extensive feature vector for the training of the encoders. The major novelty lies in the extraction of feature vectors fed into the encoder. To form the encoding from the image, two embeddings, one based on the objects in the image and another by using an Inception-V3 model are concatenated and a final image-based embedding is generated. Further, a Faster Regions with CNN (FRCNN) has been used along with a region prediction network (RPN) to generate object-based encoding from the images. This generates a detailed feature vector.