CLMIU: Commonsense Learning in Multimodal Image Understanding.
Authors: Sergio Sanchez Santiesteban
Muhammad Awais
Sara Atito
Yi-Zhe Song
Josef Kittler
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision and language pretraining
Image captioning
Commonsense knowledge
Transformers
Graph attention networks
Group masked model learning

Abstract: The problem of automatically describing the content of an image through accurate and meaningful captions has been attracting considerable attention among computer vision researchers. Recently, Transformers have been applied to image captioning to encode cross-modal information, in conjunction with Convolutional Neural Networks, which provide image region descriptions in terms of embeddings and object labels as input. However, the generated captions sometimes fail to capture the intentions, relationships, and abstract concepts that rely on general or commonsense knowledge. In this work we propose a novel network design, combining the strengths of Transformer models with graph-based models conveying external (common sense) knowledge. Our proposed architecture is a pure vision transformer-based image captioning model, with sequences of image patches used directly as input, without extracting any regional features. In particular, unlike the prior work, our architecture incorporates a knowledge-augmented encoder with a Transformer backbone to inject the external knowledge extracted from a knowledge graph. Furthermore, the bidirectional training on a vision-language corpus of image-text pairs, using modality specific self-supervised learning objectives, achieves promising results compared to the state-of-the-art. Our method has been trained from scratch on a small dataset, achieving a 3.8%, 2.7%, 3.2% and 6.3% improvement in BLEU@4, Meteor, Rouge and Cider scores respectively. We also reported competitive results on the NoCaps dataset, showing that the model generalizes to unseen object categories.

Under review as a conference paper at ICLR 2023
CLMIU: COMMONSENSE LEARNING IN MULTIMODAL IMAGE UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of automatically describing the content of an image through accu- rate and meaningful captions has been attracting considerable attention among computer vision researchers. Recently, Transformers have been applied to image captioning to encode cross-modal information, in conjunction with Convolutional Neural Networks, which provide image region descriptions terms of embed- dings and object labels as input. However, the generated captions sometimes fail to capture the intentions, relationships, and abstract concepts that rely on general commonsense knowledge. In this work we propose a novel network design, combining the strengths of Transformer models with graph-based models convey- ing external (common sense) knowledge. Our proposed architecture is a pure vi- sion transformer-basee image captioning model, with sequences of image patches used directly as input, without extracting any regional features. In particular, un- like the prior work, our architecture incorporates knowledge augmented encoder with a Transformer backbone to inject the external knowledge extracted from a knowledge graph. Furthermore, the bidirectional training on dision-languaal corpus of image-text pairs, using modality specific self-supervised learning ob- jectives, achieves promising results compared to the state-of-the-arr. Our method has been trained from scratch on a small dataset, achieving a 3,8%, 2.7%, 3.2% and 6.3% improvement in BLEU@4, Meteor, Rouge and Cider scores respec- tively. We also reported competitive results on the NoCaps dataset, showing that the model generalizes to unseen object categories.
knowledge-augmented
INTRODUCTION
Image captioning (IC) is an important research area of Computer Vision (CV) which addresses the problem of automatically describing the content of an image. The generated description includes the global scene, the objects contained in the image, their relationships as well their attributes and the activities they are involved in. Training multimodal models on manually annotated paired image and text corpora aims to learn cross- modsl representations that capture rich image and language semantics. Factual and commonsense knowledge are essential to how humans understand the world around them and learn about it. Factual knowledge refers to the specific details elements of subject (e.g. "London is the capital of the United Kingdom"" "). Commonsense knowledge includes information about events and their effects, about physical objects and how they are perceived, and about their properties and their relations to one another McCarthy et al. (1960). A large amount of common-sense".
this knowledge is common to all humans, hence the term "common"
Commonsense knowledge is hard to compute/learn by machine learning models. Therefore, in- corporating commonsense information is present a key problem facing machine learning re- search Klein & Nabi (2019); Zhou et al. (2019); Zhang et al. (2019): Wang et al. (2020): Liu et al. 2220 Even the tf-h-ar-th (SOTA) models image captioning ignore this type of knowl- edge Li et al. (2019a by: Lu et al. (2019); 1 Tan & Bansal (2019); Chen et al. (2020): Desal & Johnson 2020 Li et al. (2020b): Hu et al. (2020): Zhang et al. (2021b). Though some captions might hint at learning elaborated abstract concepts, is not obvious that even training on, e.g., 1.8B image/text pairs Wang et al. 2021) will result in models capable of acquiring commonsense knowledge. We argue that using exclusively pre-trained language models and the concepts learned by them cannot provide sufficient information for image captioning. Incorporating external commonsense knowl- edge into the image captioning methods relies primarily on the intuition that human beings produce

---

Under review as a conference paper at ICLR 2023
Oscar: A person blowing our candles on colie
Concept Vett NG reletionships
blow
Types of birthday
bake Things motivated by-
Ours: A person slowing out candles on birthday cake
Human descriptions
old woman and a birthday cale with candles A person at a birthday partyy
A woman about to blew the candles on the care
Figure I: This is an example image showing a case when identifying semantic concepts, not explic- itly represented in the scene, would help to provide better description. Commonsense knowledge used to relate the elements, namely people, cake and candle, to the concept of birthday. The comparison between the baseline caption and the generated by method shows the benefit of incorporating external knowledge. The relationship information extracted from the Concept- Net Speer et al. knowledge graph.
accurate and semantically rich descriptions of scenes by exploiting this source of information. In Figure [ it can be seen that the woman is getting ready to blow out the candles on the cake. We can infer that the occasion might be a birthday celebration just by looking at the cake, the candles and the overall composition of the scene. This is the type of knowledge that is very relevant for the task of captioning. It typically encoded in Knowledge Graph (KG) in the form of nodes repre- senting objects and their relationships. As can be seen in the figure, the incorporation of external knowledge from a knowledge graph improves the caption by suggesting the semantic concept of "birthday". Furthermore, current vision and language models often leverage frozen object detector, like Faster R-CNN, pre-trained on labelled object detection datasets. Such approaches are limited by the granularity of these datasets, and so, are less scalable.
In this paper, we introduce CLMIU, short for Commonsense Learning Multimodal Image Understanding. CLMIU's objective is to incorporate factual and commonsense knowledge in mul- timodal network by training on a small dataset with access to external information. Aiming to learn accurate object-detector free image descriptions beyond explicit image content, we train CLMIU to a) recover masked language and transformed vision tokens from its contextualized vector repre- sentations, and to b). identify the correct alignment of the textual concepts to their corresponding image regions in the image-text pairs. We show both quantitatively and qualitatively that CLMIU has strong understanding of factual and commonsense elements needed to accurately describe natural images. When using considerably less image-text data for training, CLMIU outperforms strong baselines like Oscar Li et al. (2020b), Vin ZZang et al. (202Tb) and Vil et al. (202T). Specifically, our method achieves 3.8% 2.7%, 3.2% and 6.3% improvement in BLEU BLEU@44Pppli Papineni al. (2002). METEOR Denkowski & Lavie (2014), Rouge Lin (2004) and CIDEr Vedantam et al. (2015) scores respectively on the MS COCO Captions dataset et al. (2015). This improvement in caption generation emerges during pre- pre-training with an external knowledge source. The results obtained on the NoCaps Agrawal al. (2019) dataset show that the model generalizes to unseen object categories. An analysis of CLMIU's 's attention patterns (supplementary material Appendix Figure 4 shows that image regions attend text tokens that are conceptually related.
Finally, ablation studies of CLMIU show that 1) external knowledge injection works better when added the last layer(s) rather than from the start of the multimodal encoder, 2) existing methods improve by re-training with the external knowledge encoder, 3) using Group Mask Model Learning (GMML) Atito et al. (202Tb) for the vision encoding leads better image representations and 4) CLMIU's performance steadily improves by training longer. The combination of these results suggests that incorporating factual and commonsense knowledge into image captioning models is a promising path forward for future research. In summary, our main contributions are:
1. CLMIU, performant end-to-end vision and language model, that learns multimodal im- age representations from images and their captions incorporating external knowledge. To the best of knowledge, CLMIU is the first to propose the use of external knowl-
2

---

Under review as a conference paper at ICLR 2023
edge graph to learn knowledge- informmed multimodal image and text representations using transformers.
2. A detector-free multimodal model able to produce multi-concept image representations by leveraging the strengths of GMML.
3. A set of experiments/ablation studies demonstrating the strong performance CLMIU in image captioning. It improves the performance of VLP models by producing semantically better captions.
2 RELATED WORK
Viagua-Language Pre-training, Inspired by the success of large scale pre-trained language models such as BERT Devlin et al. (2019) and GPT Radford et al. (2019). there is a growing interest in extending these models for cross-modal representation learning. Vision-language tasks, such as image captioning. benefit from larger training corpus, which usually requires tedious human work for the data labelling. The goal of VLP is learn a cross cross-modal representation of image-text pairs in self-supervised manner, which can be adapted to down-stream tasks via fine-tuning Li et al. (2020b). VLP methods commonly use multi-layer self-attention Transformers to learn cross-modal contextualised representations, based on a separate embedding of each modality. A common model architecture and training paradigm in Tan & Bansal (2019); Li et al. (2019a b); Yu et al. (2019); x Lu al. (20T9); Tran al. 22020); SS al. (2020); Qi al. 2020b); Zhou al. (2020); Chen al. (2020) is to take visual region features of an image, and the word embeddings of its paired text as input. It relies on the self-attention mechanism learn image-text alignments and produces cross- modal contextual representations. Luowei Zhou et al et al. (2020) proposed a unified encoder- decoder model, which can be fine-tuned for both vision-language generation and understanding tasks. However, the VLP models rely heavily to heavy computational load and require bounding box annotations.
external pre-trained object detector, which leads
Attributes, relationships and context. The task of producing richer and diverse captions by includ- ing object attributes, their relationships and the global scene context, defines an important research direction in image captioning. One way achieve this is through the use of dense captions Krishna ctt al. (2117) during model training Desai & Johnson (020). Attribute-driven approaches aim at boosting the performance of captioning model using the high-level semantic information present
in images. Ting Yao et Yao et al.] (2017) are one of the firsts to devise an attribute augmented architecture by integrating the object attributes into CNN LeCun et al. (1989) plus RNN(LSTM) image captioning framework. Hui Chen et al et al. (201)) also propose an attribute-based approach by developing an attribute-driven attention model Yao al. (2018).
For vision and language tasks, the authors of Knowledge Augmented Transformer (KAT) Gui et al. (2021) propose method to solve the Visual Question Answering (VQA) task. KAT focuses on Vi- sual Question Answering (VQA) with a rather complex method that could not be easily adapted for other vision and language tasks. For other NLP tasks involving only text, such as generative com- monsense reasoning, there are recent works that explore the idea of integrating external knowledge via knowledge graphs. For instance, KG-BART Liu et al. (2020) used knowledge graphs to pro- vide relational information among the commonsense concepts, which helped them to produce more logical and natural sentences. ERNIE Yu et al. [202I) a single modality (text) method, which en- codes the graph structure of the knowledge graph using only knowledge embedding algorithms like Trans Bordes et al. 2013), while we propose using a knowledge encoder based on Graph Attention Networks (GAT) Veličković et al. (017) which capture the richer connectivity structure in the graph.
3 METHODOLOGGY
The goal of image captioning is to provide a description an image / in the form of a textual sequence S, where 5 = {wi, W2, Wr. consists of words. Existing image captioning models usually consist of an OD module to extract regional features (v) from the raw image (I), and a multi- modal module (MM) to generate textual description (t). Recent works Li et al. (2020b) show that the object tags (a) extracted from the detector can serve as anchoring points across modalities. This procedure can be expressed as follows: (v, a) OD(T),
MM(
3

---

Under review as a conference paper at ICLR 2023
WPA
GMML Recomstruttio Head
Text
Patch distribtion
CLS
MAAK
Bidirectional Transformer (Vi))
Self attention
Graph Attention Network
GAT Layer
Graph
Node embeddings
Text tokenizer (Bert)
Patch projection (VIT)
connectivity
[CIS] horse carrying a [SEP]
ConceptNett 5.0
Original image Comuutte image for GMMLL
Figure 2: The model architecture for our proposed method. A textual description of the image and the linear projections of image patches are embedded using ViT Devlin et al. (2019) based Vision and Language encoder. Then relevant external information from the knowledge graph is incorporated to this encoder through cross attention. ITM: Image Text Matching, MTL: Masked Token Loss, GMML: Group Mask Model Learning and WPA: Word Patch Alignment.
These methods usually use pre-trained models to encode the ground truth captions associated with the images based textual information alone. In doing so, important relations between the semantic concepts, which are not explicitly annotated the captions, are ignored. In order to overcome this drawback, propose incorporate external knowledge into the multimodal model. Specifically, we construct a knowledge graph G to embed external information from knowledge base. An overview of our model architecture is presented in Figure 2 A textual description of the image and the linear projections of image patches are embedded using ViT Devlin et al. (2019) based Vision and Language encoder. It cannot be overemphasised that we do not need any object detectors to provide pre-computed features into the transformer. We work directly with image patches. The relevant external information from the knowledge graph incorporated to this encoder through cross attention. In Section 3.1 explain the construction process of our KG and its formalisation. The method leverages knowledge graph built from ConceptNet Speer et al.) ConceptNet labelled graph which connects words and phrases of natural language connected by edges that denote commonsense relationships between them. Most of the recent Vision and Language (VL) models are based on the Transformer architecture Vaswani al. (017). One of the most important components Transformer is the self-attention mechanism. This involves computing mapping between query and set of key-value pairs an output, where the query Q, keys K, values V, and output all vectors.
QKT
Q = H:W,,, K HWW,
V = H.W. A = softmar
Vd V
(I)
Hi Nxd d is the input hidden vector the th Transformer layer, Wg. WR. W, E Rdxd projection matrices, N is the sequence length and d is the hidden vector's 's dimension.
Image encoding. Naive application self-attention images would require that each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes. VIT Dosovitskiy et al. (2020) proposed reshape the image into sequence of flattened 2D patches. Following their approach, the input image E CxY xW is sliced into patches and flattened to (P*(C))) where (Y, W ) is the resolution of the original image, C. is the number of channels,

---

Under review as a conference paper at ICLR 2023
(P, P) is the patch resolution and J YW/P2 is the resulting number of patches. Followed by linear projection V E R(P and position embedding Vios E R(J+1)xY, u is embedded into E R'xY, Patch projection, in contrast with multi- object detectors, translates the visual em- bedding step to the level of textual embedding by tokenization. The text and image embeddings are summed with their corresponding modal-type embedding vectors ('upe, v'upe, and concatenated into a combined sequence z". The contextualised vector is iteratively updated through former layers up to the final contextualised sequence zP. The role of GMML is to give consistent representation of lata-tokens on an object which helps to overcome the challenges of detector free designs. The absence of commonsense and self-supervised pretraining in SiMVLM (ViIT, CPTR) can only be overcome by the use of a larger corpus to outperform detector-based frameworks. For all experiments, we use weights from ViT-B/16 pre-trained on ImageNet. Hidden size 768, layer depth D is 12, patch size is 16, MLP size is 3072, and the number of attention heads is 12.
depth trans-
3.1 KNOWLEDGE GRAPH CONSTRUCTION
In this section, we explain how to construct and learn the embedding of the information conveyed by the knowledge graph from the large commonsense knowledge base ConceptNet 5.0 Speer al.) We use I' to denote the semantic entities vocabulary, which is the set of all entities extracted from the image captions (objects and actions). The knowledge graph is denoted as G = (T, E, R). where Y is the set of semantic entities, E is the set of edges and R is the set of relations among entities. For pair of entities v E (subjeet)) and v/ E (objject), linked by the relation, rij E R. the edge esj- E E can be represented as a triplet (a4, U2). Specifically, we assume that the semantic vocabulary is a subset of knowledge graph entities, namely I C Y. Given an unordered set of commonsense entities = {71 72: .......77 }. where each entity 7/4 I' is an object (noun) or action (verb), we want to allow the flow of information through cross attention from the knowledge graph encoder into the multimodal model. Our aim is to boost the performance of the IC task with the help of the commonsense knowledge database G. which can be treated as auxiliary information.
More formally, the problem can be stated as follows: h : (T. - {G} that takes the semantic entities sets 7 E I and the knowledge graph G as the input to first learn a commonsense knowledge graph G.,. and then g {T' G + Y to generate the final token embeddings with the common- sense knowledge incorporated. Specifically, G. C G consists of all concept triplets (u,, Tij, Uy), where vú and U/ E I' and rij E R the relation between the semantic entity pair. Most of the semantic entities in image captions correspond to a knowledge graph unigram entity, so we can di- rectly match the semantic entities set the entities from the knowledge graph to generate G.. We rank the corresponding G nodes for each semantic entity according to the word similarity scores and select their potential top-k similar nodes adding them to Gy. To calculate the word similarity scores, we use the pre-trained Gl PPennington al. (2014) embedding as the representation of each entity node in the knowledge graph. Since some of the semantic entitics do not have a direct connection in the knowledge graph, instead of using G, directly, method named TransE Bordes et al. (2013) to learn their entity and relation embeddings. Part of the proposed methodology for extracting the sub-graph of interest is inspired by the one followed in KG- BART Liu et al. (2020). But KG-BART is a single modality (text) processing method. There are also architectural differences. KG-BART needs an encoder-decoder architecture with knowledge injection in two stages, while ours proposes an encoder only model with an external (one-stage) knowledge encoder. The incorporation of external information through cross-attention is more flex- ible than what KG-BART proposed. Other methods such as Kim et al. (021). CPTR [iu et al.) (202T) and VLM Wang et al. (202T) already introduced a detector free image interpretation method, but (to the best of our knowledge) we are the first one to approach this task by masked image model learning and report the benefits of using the pre-text task introduced by GMML Atito et al. (2021b) in multimodal analysis.
knowledge embedding
3.2 KNOWLEDGE INJECTION
We incorporate external knowledge into the main vision and language model during training by allowing cross attention into a transformer encoder, specifically a GAT model. In the last ViT layer, after the original self-attention module add a cross-attention module where the queries come from the previous ViT layer while the keys and values come from the output of a corresponding GAT layer, so that the KG encoder's knowledge information flow into the VIT. Figure 2 shows this
5

---

Under review as a conference paper at ICLR 2023
process. The KG encoder architecture follows the GAT model proposed by Veličkovie et al. (011)) The input to each layer of the GAT is a set of node features, h = {h1, fg,,.. hy J. h4 E R³, where N is the number of nodes, and F is the number of features in each node. The layer produces new set of node features, h' = { R2 ..., h2y J: Mr E Rf, as its output. A shared linear transformation, parametrized by weight matrix, W E RPXF, is applied to every node. We then perform self- attention the nodes. The shared attentional mechanism computes attention coefficients: Gij aWh,, Why) that indicate the importance of node j's features to node i. We inject the graph structure into the mechanism by performing masked attention, that is, we only compute eij for nodes j E N; where N; is some neighbourhood of node in the graph. In all our experiments, we use the elements of the first-order neighbourhood of (including i). The connectivity information is extracted from the ConceptNet sub-graph corresponding to each set of nodes, built as explained in Section 3
3.2.1 TRAINING OBJECTIVES
Optimizing the alignment between text tokens in the caption and all image features is a crucial objective for tasks involving cross-modal representation learning, Contrastive methods to learn an instance-level alignment between the whole image and the caption have been successfully used for this purpose in the past Chen et al. 2020): Li et al. (2020b); Kim et al. (202T). In this work, we specifically use the Image Text Matching (TTM) Chen al. (2020) loss. We randomly sample a set of "polluted" image representations by replacing the aligned image, with probability 50%, with a different image randomly sampled from the dataset. The encoder output on the special token (CL.S) is assumed as the aggregated vision-language representation of v). A single linear layer projects the pooled output feature to logits, acting as a binary classifier f()). Finally, we compute the negative log-likelihood. The output is a binary label y {0, 1), indicating if the sampled pair is a match. This ITM loss is defined as: LITM = - -log.)) (c.
Furthermore, in order to provide more fine-grained grained alignment between word tokens and image gions, we use the Word Patch A Alignment (WPA) Kim et al. (022) loss. This objective is based on Optimal Transport, which effectively calculates the minimum cost of transporting the contextualized image embeddings from the local patches to the word embeddings (and vice versa). We compute the alignment score between two subsets of 20 z zDIt (textual subset) and 2D (u (visual subset), using the inexact proximal point method for optimal transports (IPOT) Xie et al. (2020). We set the hy- perparameters of IPOT following Chen et al. (020)) and add the approximate Wasserstein distance multiplied by 0.1 to the ITM loss. Learning robust representations for each individual modality (language and text) is another important factor in multimodal tasks. Predicting masked words based on the observation of their surrounding words is an effective task for language modelling Devlin jet al. (2018), as it helps to model contextual relationships. We apply the Masked Token (MTL) loss as formulated by Li et al. (2020b). Each input token in the caption is randomly masked with probability 15,, with a maximum 3 tokens per caption to be masked out. The masked ones are replaced with a special token [MASK]. The model is trained to predict these masked tokens from its contextualized vector d and all image features v by minimising the negative log-likelihood
LMTE
For learning useful and robust image representations, self-supervised approaches have reported state of the art results vision only tasks, e.g. image classification, while being more scalable and in- terpretable Atito et al. (2021a). We adopt the Group Mask Model Learning (GMML. objective as introduced in Atito al. (2021b). We apply a transformation to local patches of the image in blocks of neighbouring tokens arranged spatially. SiT Atito et al. (2021b) authors showed that it critical to transform neighbouring tokens in order to extract the context provided by neighbour- ing patches in the image. In this work, the local transformation applied involved random drop of connected patches. The objective of the image reconstruction to restore the original image E HR x@xxWxx xC from the GMML manipulated image &. For this task, we use the él-loss between the original and the reconstructed image: L(W) = M [ Σ M x [x - -x,,)) where
M.j 1, if is manipulated
W denotes the parameters to be learned during training, N is
0, otherwise
the batch size, M is a binary mask with 1 indicating the manipulated pixels, and x is the recon- structed image. The reconstructed image x is obtained by averaging the output features from the intermediate blocks of the transformer encoder F.() and feeding the output a light decoder D()):

---

Under review as a conference paper at ICLR 2023
x D (EDEB (x)). Where (s(.) is the output features from block and B is pre-defined index set of transformer blocks that are included in the decoding process. In this work, we set B to {8, 10). Furthermore, MTL. and GMML losses serve in our architecture two purposes: the aforementioned benefit for individual modality robustness and their contribution to cross modal representation learn- ing. The latter is possible as we allow attention over both modalities while optimizing these losses. The full training objective is:
== LMTE + LITM + 0.ICwPA + LGMML
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We use Adam Loshchilov & Hutter (0117 optimizer with base learning rate of 10 and weight decay of 10-2 The learning rate was warmed up for 10% of the total training steps and was decayed linearly zero for the rest of the training. We resize the shorter edge of input images to 224 and limit the longer edge to under 320 while preserving the aspect ratio. Patch projection of our model, using ViT-B/16 yields 14 x 20 280 patches for an image with a resolution of 224 x 320. We interpolate the learned positional embeddings of ViT-B/16 to fit the size of each image and pad the patches for batch training. We use the bert-base-und uncased tokenizer to tokenize text inputs. We use Part of Speech Tagger (POS) Akbik al. (2019) to detect nouns and verbs in the captions and then these were matched with the top-k nodes from ConceptNet. We pre-train all our model variants for 20 epochs on 4 NVIDIA V100 GPUs with batch size of 128. For fine tuning the image captioning task, we train for another 10 epochs.
Model configurations. On Table D we report the performance of several model configurations. A (ITM+ + MTL GMML): here we do not use the WPA loss based on OT but the GMML. objective is included for the first time. B (ITM MTL GMML WPA): this configuration uses all the optimization objectives represented in Figure 2
C (ITM + MTL): this variant uses the same procedure introduced by Liu et al. (2020) to inject the external knowledge into the multimodal encoder. Instead of allowing cross attention, the embeddings from the GAT are directly added into the intermediate language tokens on the the ViT based encoder. D (ITM + MTL): this configuration uses the cross attention mechanism to allow the flow of information from the knowledge graph encoder to the multimodal model. E (ITM + MTL + GMML + WPA): in this variant we change the layers used by GMML from the last layer of the multimodal encoder to layers 8th and 10th combined. E+20: we also report the same configuration but trained for 20 more epochs. We re- implemented the methods in CPTR Liu et al. (202I) and Sim VLM Zhang et al. (2021a) following their instructions. The rest of the implementations were the ones released by their authors.
4.1.1 DATASETS
We use the Microsoft COCO Captions Chen et al. (2015) dataset for pre-training and training. No additional data is used. For the COCO Captions dataset, adopt the widely used Karpathy split Karpathy & Feil (2015) to conduct our experiments. Specifically, the dataset consists of 113 287 images for training. 5000 images for validation, and 5000 images for testing. Each im- age has 5 annotated captions with an average caption length of 1 11.81 + 2.81 after being tokenized tokenizer. Besides, we report evaluation results on the eet al. (2019) dataset. The NoCaps dataset provides benchmark with images from the Open Images dataset Kuznetsova et al. (2020) to test models' capability of describing novel objects which are not seen in the training corpus.
with the bert-base-
4.2 RESULTS AND DISCUSSION
We perform comparisons of the proposed method with state of the art models. Following other work in image captioning, we use several widely-used automatic metrics to assess the performance; such BLEU Papineni et al. (2002). METEOR Denkowski & Lavie (2014). Rouge Lin (2004) and CIDEr Vedantam et al. (2015) Table [ presents the experimental results achieved in the proposed model evaluation the COCO Captions and NoCaps datasets. We can see that our method out-
7

---

Under review as a conference paper at ICLR 2023
Table 1: The image captioning evaluation results (single model) obtained on the MSCOCO Captions dataset. All models are trained on the training set of the COCO Captions dataset following the Karpathy split Karpathy Fei-Fei 2015). B: BLEU@4, M: Meteor, R: Rouge and C: Cider.
Methods
COCO Captions
NoCaps
B@4
M
R
C
@@4
M
C
Up-Down Anderson
2018
28.5
20.4
48.3
111.5
19.2
23.0
50.9
54.3
Oscar
20206
34.3
27.6
54.7
115.2
21.0
24.8
53.1
57.2
Sim VLM Wang ct
2021
32.9
26.5
53.9
110.2
20.5
24.6
52.0
55.1
CPTR
31.3
25.2
53.1
108.6
19.8
24.0
51.2
54.9
ViIT
Kim al
2021
33.0
26.8
54.1
113.3
20.9
24.0
52.7
56.0
Oscar (with KG)
35.8
28.3
56.0
116.1
Ours (A: ITM + MTI (IMMI)
30.2
25.0
51.0
112.7
Ours MTT. GMME. WPA)
33.7
27.0
54.5
114.2
Ours (C+KG (KgBan like MTL)
34.8
27.9
55.2
116.0
Ours (D+KG: ITM MTL)
35.7
28.3
56.0
118.5
Ours (B+KG: ITM MTL GMML+ + WPA)
36.1
29.0
56.2
118.5
Ours (E: IIM + MIL + GMML + WPA)
36.5
29.4
56.8
118.5
Ours (E+20: ITM MIL GMML
38.1
30.3
57.9
121.5
22.3
26.7
55.2
59.1
best our
3.8
1 2.7
3.2
1 6.3
1.3
1.9
1 2.1
1.9
Table 2: The image-text retrieval evaluation results obtained on the test set of MSCOCO Cap- tions dataset. All models are trained exclusively on the MSCOCO training set using the Karpathy split Karpathy & Fei-Fei (2015).
Methods
Rank @
Rank @5
Rank @ 10
Oscar Li al. (202bb)
35.1
66.2
76.8
Ours (E+)
36.5
67.4
77.9
A difference
E+
t 1.4
1 1.2
1 1.1
performs the baseline by significant margin in most of the metrics. The improvement is observed more clearly when measured by the CIDEr score with 3.3% improvement. Table shows the text retrieval results on the Ik test set of MSCOCO. Following the Oscar Li al. (2020b) methodol- ogy, during training, we formulate this task as binary classification problem. Given an aligned image-text pair, we randomly select different image pair. The final representation of the (CLS] token is used as the input to a binary classifier to predict whether the given pair aligned or not. We decided use the binary classification loss instead of the ranking losses following the findings of Oscar Li et al. (2020b) and Imagebert Qi et al. (2020a), which reported better performance for the binary loss. In the testing stage, the probability scores are used to rank the given image-text pairs of a query. Following Li et al. (2020a), we reported the set. We included results on image- text retrieval show
different caption form unrelated
top-K retrieval results on the MSCOCO how CLMIU is able to perform other vision and language tasks. The improvement over the baseline shows that incorporating external knowledge also boosts the performance of image-text retrieval. The results on the NoCaps dataset show that the model generalizes to unseen object categories. The size of the images for this task was the same as in all previous image captioning experiments (224x320). We anticipate that using bigger image size, e.g. 640x480, would have positive impact on performance.
The results suggest that incorporating external commonsense knowledge boosts the performance of the VLP model. The fact that the improvement more apparent the CIDEr metric owes to the nature of this metric. It takes into account the semantic correctness and quality of the generated
8

---

Under review as a conference paper at ICLR 2023
captions better. The other metries were adapted from pre-existing NLP tasks and such they more focused on analysing the presence of overlapping n n-grams between the candidate and reference sentences. An n-gram overlap is neither necessary nor sufficient for two sentences to convey the same meaning or the same semantic correspondence.
4.2.1 LIMITATIONS
There are several potential limitations of CLMIU that will be addressed in future work, including: 1) exploring dynamic knowledge representations that be adjusted by feedback during training; and 2) reducing the need for paired data during training by leveraging language and vision only pre-training and/or self-supervised learning. The curation process involved in the creation of the original COCO captions dataset could cause the models trained on it to reproduce undesirable social biases. For this reason, mitigation measures should be employed before any production deployment of this method. Furthermore, we advise against the use of CLMIU or any its variants in automated scenarios where could influence critical decisions with consequences for human beings. We do not advocate the use of CLMIU as part of any mass surveillance system.
4.2.2 ABLATIONS
The following ablation study aims to investigate the relative contributions of architectural and train- ing decisions the final model performance.
Into which layers is best to inject the KG information?: adding a cross attention module after every layer on the ViT to allow information flow from the KG, does not lead to significant im- provements. We argue that: the concepts and relationships encoded in the KG sit on the top of the abstraction hierarchy. Therefore, this information is better exploited and integrated by the multi- modal ViT encoder in its final layers. Do existing methods improve by just coupling the KG encoder?: the cross attention module allows to couple the KG encoder into any Transformer-base multimodal encoder. We retrained existing IC SOTA methods, such as: Oscar Li et al.) (2020b), after incorporating the KG encoder through cross attention. The improvements reported show that they do benefit from this external information. The fact that the image encoder is frozen pre-trained object detector limits the overall gain. We expect that other similar SOTA models (e.g. VIn /hang et al.) (2021a)) will also benefit from this strategy. GMML objective leads better image representa- tions: previous attempts Fang et al. (202)) to use object detector free IC models had to incorporate additional training objectives designed to model the objects the image. SiT Atito et al. (202Tb) Atito and MC-SSLO.0 Atito et al. (2021a) succeeded in modelling multiple concepts present in an image through sulf-suuprvised pre- traitrainngg. They showed that these learned concepts or object categories are consistent across different images and across instances of the same category. We also found that this training strategy boosted performance and we attributed to better image representations being learned. Effect of Longer Training: as reported in supplementary material Appendix Figure 3 increasing the number of training epochs causes the performance to steadily increase. This results suggest that the approach is scalable with respect to training time available.
CONCLUSIONS
We addressed the problem of image captioning and presented two major contributions. The first is the proposed methodology for incorporating commonsense knowledge into the captioning process boost the performance of the learnt vision-language model. This has been achieved by injecting the semantic relations among concepts conveyed by a knowledge graph using a transformer archi- tecture. The experimental results confirm that this auxiliary information improves the performance of existing image captioning systems as well as the novel architecture proposed in the paper. The key innovation of the proposed image captioning system its object detector free design, which has been enabled by new method of vision transformer training using the Group Masked Model Learning (GMML). This revolutionary pre-training method has major advantages, namely: a) en- abling end to end training; b) requiring no bounding box annotation of training images, and most importantly; c) greater efficiency in extracting contextual information, resulting in much better im- age representation. The image captioning system based the proposed architecture and incorpo- rating the common sense knowledge has defined a new sfate-of-the- the-tar performance on the COCO Captions dataset.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Harsh Agrawal, Karan Desai, Yufci Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8948-8957, 2019.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. Flair: An easy-to-use framework for ofate-of-the-ar nlp. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (demonstrations), pp. 54-59, 2019.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, Salt Lake City, UT, June 2018. IEEE. ISBN 978- 1-5386-6440-6- doi: 109/ CVPR.2018.0066 URL
leee. www.met.otn/n////oooooommmm /B57 /B5787347
Sara Atito, Muhammad Awais, Ammarah Farooq, Zhenhua Feng, and Josef Kittle. Mc-ssl0. O: Towards multi-concept self-supervised learning. arXiv preprint arXiv:2111. 2021a.
Sara Atito, Muhammad Awais, and Josef' Kittler. Sit: Self-supervised vision transformer. arXiv
preprint arXiv:2104.00 03600, 2021b.
Antoine Bordes, Nicolas Usumier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information pro- cessing systems, 26, 2013.
Hui Chen, Guiguang Ding, Zijia Lin, Sicheng Zhao, and Jungong Han. Show, Observe and Tell: Attribute-driven Attention Model for Image Captioning. Proceedings of the Twenty- Artificial Intelligence, pp. 606-612, Stockholm, Sweden, July 2018 International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-9992241-2277. doi: 00224933/jcci.2011883 URL https:/ www ijcaiiiooo. org/ pgoceedings/220 220.//011///
Seventh International Joint Conference
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv: 003550034555... [cs], April 2015. URL 1504.00325.
00325
arXiv:
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image-TExt Representation Learning. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12375, pp. 104-120. Springer International Publishing, Cham, 2020. ISBN 978-3- 030-58576-1 978-3-030-58577-4 springer. wttp:............//././//
doi: 10.10077978-3-000-887777
3-030-5857788.77 URL http:://inkk.
10. 3-0-0007777788-00---------- 3-00788---0000-757---
Series Title: Lecture Notes in Com-
puter Science.
Michael Denkowski and Alon Lavie. Meteor Universal:
Language Specific Translation Evalua- tion for Any Target Language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 376-380. Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115//1//144-34888 W14-3 3348
URL https::///// www. acwweb. wwtp://.....//aaaoogg......
Karan Desai and Justin Johnson. VirTex: Learning Visual Representations from Textual Annota- tions. arXiv:2006. 06666 [cs], June 2020. URL hta.:/ppp//arrv..o///////200 arXiv: 2006.06666.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Ming-Wei
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019. URL httt:/tt: wttpaa/wwww.orra//////aa/// 04805 arXiv: 1810.04805.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010. / 1929 [cs], October 2020. URL http:://arxil..rr/abbb... 11929 arXiv: 2010.11929.
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang. and Zicheng Liu. Injecting semantic concepts into end-to-end image captioning. arXiv preprint arXiv:2112.052500 2021.
Liangke Gui, Borui Wang. Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. KAT: A Knowledge Augmented Transformer for Yision-and-Language wrXiv:2112.080 [cs), December 2021.
phttp:://axxtae wwwp:/wwwwww..../////.xeaaa/ /2122//1111 022224886444
arXiv: 2112.08614.
Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-
Training, arXiv:2009. 13682 [cs], September 2020. URL http: arxiv. wexxiv.org/aoo orggabs//asssssssssssssgg//////
13682
arXiv:
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-lange transformer without convo- lution or region supervision. In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021.
Tassilo Klein and Moin Nabi. Attention (not) All You Need for Commonsense Reasoning. 1905 [cs], May 2019. URL 1909.13497.
13497 arXiv:
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Vi- sual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. International Journal of Computer Vision, 123(1)332-73, May 2017. ISSN 1573-1405, 1007//11263-001--9988-11- URL https: /oi. org. 10. 1 1007/s112210
doi: 10. 016 0981-7
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-1))) 1981- 2020.
Shahab
Yann LeCunn et al. Generalization and network design strategies. Connectionism in perspective, pp- 143-155, 1989.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder- VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training. arXiv:1908 1990.000066 [cs], December 2019a. URL
org/ abs/ /wtp//90..... 06066
arXiv: 1908.06066.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11 11336-1111 1344, 2020a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv:1908. 03557 [cs], August 2019b. URL
org/ abs / 1908. 03557 arXiv: 1908.03557.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object Abjjet-Semantics Aligned Pre-training for Vision-Language Tasks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, volume 12375, pp. 121-137. Springer International Publishing, Cham, 2020b. ISBN 978-3-030-58576-1 978-3-030-58577- 8. doi: 011177997788-00-58577777 8. URL http:://Iat springer.... com/ 100o/110.100)
978 0-0--0000-88777-- Series Title: Lecture Notes in Computer Science.

---

Under review as a conference paper at ICLR 2023
Chin- Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summa- rization Branches Out, pp. 84, Barcelona, Spain, July 2004, Association for Computational Linguisties. URL https: www www.waalwee..oo. htt.d//www...........oo.ooo/////
Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu. CPTR: Full Transformer Network for Image Captioning. arX1v:2101.10004 [cs], January 2021. URL abs foss/2101.0 10804 arXiv: 2101.10804.
Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S Yu. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. arXiv preprint urXiv:2009.1266 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Vi- siolinguistic Representations for Vision-and- Language Tasks. Advances in Neural Information Processing Systems, 32:13-23, 2019. URL https: //aperrs wniss.iiii.....................m www.nniisscpiaaa.c cttppp/scccpapp/11./9////// 9d5aadee wbsstrrctaattttt....e....aa
hash//774397601eae222 A
html
John McCarthy et al. Programs with common sense. RLE and MIT computation center Cambridge, MA, USA, 1960.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Au- tomatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia. Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1077083.1071355.. URL https:/ http::///www... ac web www.:www........................ htt../orrrraatooogggPP....
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data, arXiv preprint arXiv:2001.07966, 2020a.
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-TexI Text Data. arXiv:2001.0777 0.7966 [cs], January 2020b. URL /ttt:://rxxvv.oo wttp://////aaaaaa///..oa.e a20///11./22001... 07966 arXiv: 2001.07966.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. (GPT-2) Language Models are Unsupervised Multitask Learners. pp. 24, 2019.
Robyn Speer. Joshua Chin, and Catherine Havasi. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. pp. 8.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BBRTT Pre- training of Generic Visual-Linguistic Representations. arXiv: 08530 [cs], February 2020. arXiv: 1908.08530.
URL
org/ abs/1908 08530
Hao Tan and Mohit Bansal. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 5 5100-5111. Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-11 1514- D19-1514
/www www.eb.ww......... wtt.///www..r//tth.oooo..oooo
Alasdair Tran, Alexander Mathews, and Lexing Xie. Transform and Tell: Entity-Aware News Im- age Captioning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13032-13042. Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-7288 168-5. doi:
http::ttts:/:eeeeeee htttps:/eeeexpplxee... leee hwwp:/ogggooooooooo....ee document/
91564567
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. pp. 11, 2017.
Ramakrishna Vedantam, C. Lawrence Zitmick, and Devi Parikh. CIDEr: Consensus-basee image description evaluation. In 2015 IEEE Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 4566-4575, Boston, MA, USA, June 2015. IEEE ISBN 978-1-4673-6964-) URL httpttttt//eeeeppeee/e///
doi: 10.1 201/////..21..../... 72990877]
ieee. org documet//
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.1090. 2017.
Tan Wang. Jiangiang Huang, Hanwang Zhang, and Qianru Sun. Visual Commonsense R- CNN. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10757- Seattle, WA, USA, June 2020. IEEE ISBN 978-1-72817-168-5. doi: 10111090CVP-22260000000000 91563477
WRLLhttp::/// htttps://heeexplorsttt ieee wrg/docuner/..
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- plc visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in artificial intelligence, pp. 433-453. PMLR, 2020.
Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting Image Captioning With Attributes. pp. 9, 2017.
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring Visual Relationship for Image Caption- ing. pp. 16, 2018.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emie-vil: Knowl- edge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 35, pp. 3208-3216, 2021.
Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. Multimodal Transformer with Multi-View Vi- sual Representation for Image Captioning. arXiv: 07841 [cs]. May 2019. URL http:
//arxiv. wwwp:///rraaba/b/a/ooooo/...ss 00011905588848
arXiv: 1905.07841.
Mingxing Zhang, Yang Yang, Hanwang Zhang, Yanli Ji, Heng Tao Shen, and Tat-Seng Chua. More is Better: Precise and Detailed Image Captioning Using Online Positive Recall and Missing Con- cepts Mining. IEEE Transactions Image Processing, 28(1):32 44, January 2019. ISSN 1057- 7149, 1941-0042. doi: 1..11099112201828855511 URL htttp:: /ttp::/:/eeeeeeorreeee 1eee.
org/ httt//occmet////// /00105/448882
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021a.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinVI: Making Visual Representations Matter in Vision- Language Mod- els. 1tti::/0110005:0 [cs], January 2021b. URLhttpp//arxxy arXiv: 2101.00529.
00529
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified Vision-Language Pre-Trainin for Image Captioning and VQA. 1rXiv:199..11059 [cs], 2020.
URL URLt:/tp://arii org/ http://a/s//1111111 19099.1099.119 arXiv: 1909.
Yimin Zhou, Yiwei Sun, and Vasant Honavar. Improving Image Captioning by Leveraging Knowl- edge Graphs. arXiv: 1901.08942 [cs], January 2019. URL wtt.hhtt:///axiiiii httpaaawwwww.orrbb/////oaa/////e 08 08942| arXiv: 1901 .08942.
13

---

Under review as a conference paper at ICLR 2023
B1
57.5
Seug4
56.5
20.5
36
26000
55000
44000
26000
39000
44000
Trainingsteps
Trainingsteps
30.5
122
29.5
Meteor
Cider
28.5
26000
35000
44000
26000
35000
44000
Training steps
Training steps
Figure 3: Effect of longer training on performance improvement.
A APPENDIX
Appendix A.1 shows an analysis of CLMIU's attention patterns, where can be seen that image regions attend text tokens that conceptually related. Appendix A.2 shows the effect of longer training on the performance of the model. Appendix A.3 shows visualisations of the self-attention heads in the last layer of the model for different images.
A.1 CROSS-MODAL ALIGNMENT
In Figure 4 the transportation plan learned by the Word Patch Aligmment objective is manifest in heatmap for each of the text tokens highlighted red colour. Each square tile represents a patch, and its opacity indicates how much mass is transported from the highlighted word token. The figure shows how specific objects are aligned with their textual identifiers. This is the case the candles in image
the ball in image Abstract concepts involved in their definition. In image a the concept of birthday is connected with most of the objects related it. Besides, in image b the concept of riding (a bicycle) attends similar than the term bicycle, as these are semantically related.
shown be distributed over the objects
A.2 EFFECT OF LONGER TRAINING
Figure shows that increasing number of training steps produces a steady increase on perfor- mance all of the metrics reported. Besides, this also shows that training metries still do not saturate and they should continue to improve with longer training time. The longer training benefits classic metrics (Figure B1) and more semantic metries (Figure B2) as well.
A.3 SELF-ATTENTION HEADS
S the self-attention heads from the last layer. We visualise the attention maps computed using the [CLS] token as a query for the different heads in the last layer. We found that the main objects or parts of the scene, which are necessary for generating description, are identified and preserved across the attention heads.
14

---

Under review as a conference paper at ICLR 2023
a) A woman blowing out candles on a birthday cake.
woman
candles
birthday
b) A woman and a child riding a bicycle with dog behind them.
woman
riding
bicycle
baseball player swinging at a during a game.
swinging
ball
game
d) A cat is sleeping on laptop.
cat
sleeping
laptop
Figure 4: Visualizations of the transportation plan for the Word Patch Alignment. The images from a) the internet, (b, e and d) MSCOCO dataset.
15

---

Under review as a conference paper at ICLR 2023
Figure 5: Self-attention heads from the last layer.
16


Reviews:
[{"review_id": "R3T6j45_3I", "reviewer": "Reviewer_C2N5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to pretrain a vision language transformer with external commonsense knowledge to improve on downstream tasks where external knowledge is relevant. They validate their approach on the downstream task of image captioning and image retrieval and show improvement when the model is trained with commonsense knowledge. ", "summary_of_the_review": "The paper proposes an interesting idea but the contributions are not sufficient and the experimental results do not fully convey the addition of commonsense knowledge and its effetcs. Other downstream tasks such as VCR or VQA might be more reasonable the effect of commonsense knowledge on pretraining. \n", "strength_and_weaknesses": "Strengths - \nThe motivation and intuition behind incorporating external commonsense knowledge during pretraining is interesting .The authors also use  a detector free approach to image modeling thus training an end to end system. \n\nWeaknesses -\n\nThe role of GMML is not very well experimentally evaluated. What happens if GMML is not used at all? And the only losses used are ITM + MTL + WPA ? Is it not very convincing what advantages it brings when knowledge is incorporated . \n\nQualitative visualization of what commonsense knowledge nodes were important for the downstream task ?\n\nThe contributions of the paper are not significant and not well illustrated in experimental results.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "DnoOdinwI7N", "reviewer": "Reviewer_yTLW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a model for image captioning, utilizing external knowledge with a graph-based model. The full model contains a transformer (a text tokenizer + a ViT based patch tokenizer), and a graph attention network to get the external knowledge from conceptnet 5.0. The paper claims to achieve state-of-the-art performance on smaller dataset, and can generalize to unseen object categories.\n", "summary_of_the_review": "I think the paper passed the threshold for acceptance, based on its novelty.\n", "strength_and_weaknesses": " - The paper is clear overall.\n - The authors provides good ablation study in table 1 to validate their major claim.\n - It is interesting and novel to me to incorporate external knowledge to image captioning tasks, and achieve competitive results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A5WvyPHLzv", "reviewer": "Reviewer_3AwF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to identify semantic concepts that are not explicitly represented in the given image for caption generation by incorporating knowledge from an external knowledge graph (e.g., ConceptNet). Specifically, a ViT is trained to extract multimodal representations from image-text pairs with multiple pre-training objectives (i.e., Image Text Matching, Word Patch Alignment, Masked Toke Loss and Group Mask Model Learning), and the external information from a pre-trained knowledge graph built on the basis of ConceptNet is injected into the last layer of the ViT through cross attention. Compared with state-of-the-art models of vison-and-language pre-training, the proposed method achieves better captioning performances on MSCOCO.", "summary_of_the_review": "I lean to negative for the reasons as follows: \n\n1) The idea of leveraging external knowledge graph for multimodal understanding tasks is not new and has been adopted in previous works. \n\n2) No new architecture designs or training strategies are proposed in the paper. \n\n3) Though the authors claim that the proposed model is a vision and language model which can learn multimodal image representations from image-text pairs, evaluations on more multimodal understanding tasks are not included in the paper.", "strength_and_weaknesses": "Strengths:\n\n1.\tThis paper proposes to leverage useful information from an external knowledge graph for identifying underlying semantic concepts not represented in the images for caption generation.\n\n2.\tThe proposed method has achieved better results in image captioning by training on a relatively small dataset compared with several state-of-the-art models of vision-and-language pre-training.\n\nWeaknesses:\n\n1.\tIt is unclear that the proposed model should be considered as a captioning model or a vision-and-language pre-training model. If it is a captioning model, it should be compared with state-of-the-art baselines of image captioning. Otherwise, it should be evaluated on more cross-modal understanding tasks like the other pre-training models. However, none of these is included in the paper.\n\n2.\tThe idea of leveraging external knowledge graph for cross-modal understanding tasks is not new and has been adopted in other works like [R1-2]. The authors should discuss the differences between the proposed method and these works. Moreover, for me, the key idea between KG-BART and the proposed method is almost the same except that KG-BART is for NLP tasks.\n\n[R1] Relational Reasoning using Prior Knowledge for Visual Captioning, arxiv 1906. 01290v1.\n[R2] Concept Propagation via Attentional Knowledge Graph Reasoning for Video-Text Retrieval, ACM MM 2022.\n\n3.\tFor efficiency, it seems unpractical to cross attention over all the nodes in the knowledge graph in the last layer of ViT.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7l6TjUssVS6", "reviewer": "Reviewer_hmfQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a Vision language model that can use the external knowledge graph ConceptNet. The model consists of a transformer model with a graph-based model to embed graph information. The authors claim the proposed model can outperform state-of-the-art models in image captioning.\n", "summary_of_the_review": "There are many problems in this paper as I described in the weaknesses, thus I suggest reject.", "strength_and_weaknesses": "Strengths:\n- The model design is simple and intuitive.\n- The proposed method does not require a pretrained object detector.\n- The authors provide some ablation studies.\n\nWeaknesses:\n- Experiments:\n    - No qualitative results.\n    - No analysis of how this added knowledge actually increases the \"common sense\" of the captioning model.\n    - Some of the ablation experiments results are only described but not shown in any table. For example, the VinVL + knowledge graph result.\n    - The authors only show performance on two tasks, whereas, for such a general vision language model, it is important to see the generalization ability on many different vision language tasks.\n    - The result table of image caption retrieval is too simple. Should include more baselines and ablations.\n    - The authors say the experiments show that GMML is helpful however, if we look at Table 1, for all methods with GMML, there is no method without GMML to compare to. This seems suspicious to me.\n    - The numbers in the tables are weird and the baselines’ scores are far lower than the numbers reported in the original papers. The SOTA performance is lower than what was 5 years ago.\n- Paper presentation\n    - I am seeing many superscripts in which I am expecting footnotes. For example, for method C, and Oscar in Table 1. However, I don’t see any footnotes. Is that expected?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject