Distilled Cross-Combination Transformer for Image Captioning with Dual Refined Visual Features

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:09:55

Abstract: Transformer-based encoders that encode both region and grid features are the preferred choice for the image captioning task due to their multi-head self-attention mechanism. This mechanism ensures superior capture of relationships and contextual information between various regions in an image. However, because of the Transformer block stacking, self-attention computes the visual features several times, increasing computing costs and producing a great deal of redundant feature calculation. In this paper, we propose a novel Distilled Cross-Combination Transformer (DCCT) network. Specifically, we first design a distillation cascade fusion encoder(DCFE) to filter out redundant features in visual features that affect attentional focus, obtaining refined features. Additionally, we introduce a parallel cross-fusion attention module (PCFA) that fully utilizes the complementarity and correlation between grid and region features to better fuse the encoded dual visual features. Extensive experiments on the MSCOCO dataset demonstrate that the proposed DCCT strategy outperforms many state-of-the-art techniques and attains exceptional performance.

Main paper: 
 introduction: 
 Image captioning is an interdisciplinary research field at the intersection of computer vision and natural language processing. Its goal is to automatically understand the content of an image and generate natural language descriptions closely related to it. It is a challenging task that involves analyzing cross-modal data. With the significant success of region-based features in image captioning tasks [2] , many researchers have continued to improve the performance of image caption generation based on these features [6, 18, 27] . Despite their tremendous contribution as the sole visual feature in image captioning, critiqued for their lack of fine-grained details and contextual information, these methods stand in contrast to grid features [23, 25] , which retain all the information within a given image. Incorporating the triumph of Transformers in natural language processing [35] , numerous NLP endeavors have leveraged its distinctive encoder-decoder structure, a trend that has garnered significant interest within the realm of image captioning. Transformer-based image captioning models [6, 8, 41, 42, 45] have excellent modeling capabilities, employing multi-head attention mechanisms to simultaneously attend to different parts of both the image and text while passing global contextual information between the encoder and decoder. Having demonstrated exceptional performance, Transformers has become the dominant approach to captioning over the past few years.However, during the stacking process of Transformer blocks, multiple self-attentions are performed on all region features and grid features. This may result in the inclusion of many irrelevant features, affecting the attention focus on important features. Represented in (a) of Figure 1 , the key values of visual features post-linear projection are depicted on the horizontal axis, while the vertical axis illustrates the attention weights. The curvature of the curve indicates the level of activation for the corresponding q-value. The more curved the curve, the more active the q-value, indicating the higher value of the corresponding feature. Conversely, a less curved curve indicates a lower value of the corresponding feature. For 𝑞 12 , it has similar and low attention weights with all key values, indicating that it does not correlate with most features. On the other hand, 𝑞 2 is different; it has higher attention weights with some key values, demonstrating its correlation with many features. Therefore, inspired by the time-series prediction model Informer [52] , we attempt to replace the original Transformer encoder with an improved Informer encoder.In addition, to fully utilize the region and grid features obtained from the encoding layer during the decoding stage, they are fused in the cross-modal attention module. There are various methods for designing cross-modal attention in the middle part. Some common methods include sequential cross-modal attention and cascaded cross-modal attention. As shown in (b) in Figure 1 , in cascaded crossmodal attention, the grid and region features obtained from the encoding stage are concatenated and then fed into the cross-modal attention module. Sequential cross-modal attention calculates two independent cross-modal attention for region features and grid features separately and then combines them. Although sequential cross-modal attention can independently process region features and grid features through sequential design, it still has two main problems: 1) The former may not fully exploit the interaction between different features, thereby limiting the model's expressive power and performance. 2) The order of the two cross-modal attention in the latter may affect the final performance. If the region features are computed first, there may be insufficient attention on the grid features, and vice versa.To address the aforementioned issues, we present a unique Distilled Cross-Combination Transformer (DCCT) to enhance the efficiency and accuracy of image captioning. Firstly, we put forward a Distillation Cascade Fusion Encoder (DCFE), which filters out redundant visual features like 𝑞 12 to make attention focus more on important features like 𝑞 2 , resulting in more refined feature representations. Secondly, we propose a Parallel Cross-modal Fusion Attention (PCFA) module, which cross-attends to processed grid features and region features, fully utilizing the correlation and complementarity between them. Then, a gating mechanism is applied to adjust the importance and contribution of the two multimodal features to caption generation, enabling the model to better utilize the information from different features and achieve comprehensive multimodal fusion. The contributions made in this article can be summarised in the following way:• We present a Distillation Cascade Fusion Encoder (DCFE), which enhances encoding efficiency by filtering out redundant features from the images to produce more refined visual representations. • We introduce a novel Parallel Cross-modal Fusion Attention (PCFA) module that fully exploits the complementarity and correlation between dual visual data to obtain more informative multimodal feature representations. [6] prior knowledge through a mesh-like connection between memory vectors and encoding-decoding modules. The DLCT model proposed by Luo et al. [23] achieves complementary features of region and grid in image captioning. Recently, large-scale data pretrained visual-language models (VLMs) [10, 31, 33] have achieved great success in various visual-language tasks, such as video description generation [33, 33, 36] , visual question answering [7, 14, 15] , and image captioning [10, 22, 46] . Visual features extracted by different large models have unique advantages and powerful representation capabilities. For example, in VinVL [50] , the authors use stronger object detectors to extract diverse region features, providing compact object-level representations in images. In CLIP [31] , the authors obtained cross-modal grid features that contain rich semantic scenes and fine-grained details through contrastive pre-training. These advanced visual features are used for image captioning tasks, helping the model to perform complex reasoning and improve the accuracy of generated descriptions.Visual-Semantic Interaction In the task of image captioning, the interaction between vision and language during the decoding stage is a crucial step [6, 16, 19, 49] . Lu et al. [21] introduced the concept of an additional visual sentinel and extended the previous LSTM architecture. This adaptive mechanism allows the model to determine whether to focus attention on the language or visual part during the decoding process, thereby enhancing the interaction between vision and language. Chen et al. [4] aimed to bridge the semantic gap between different modalities. They designed a novel encoder-decoder attention mechanism with an unsaturated calibration gate function to control the interaction between vision and language. This mechanism helps to achieve a balance between the two modalities. Li et al. [17] utilized CLIP to extract grid features and employed cross-modal retrieval to identify essential semantic clues. They incorporated cross-attention in the decoder to facilitate the interaction between vision and language, enabling modality fusion. Zhang et al. [48] recognized that the semantic information obtained from offline detectors often contains irrelevant objects. They proposed a novel constrained weakly supervised learning module, which provides more relevant semantic-enhanced information to improve the model's visual-semantic interaction capability.In general, although the above-mentioned methods have partially alleviated the semantic gap and achieved basic visual-semantic interaction, they still fail to fully integrate the semantic information in visual content with textual information. In this paper, we propose a novel Distilled Cross-Combination Transformer (DCCT) for image caption generation. Detailed explanations will be provided in Section 3. 
methodolgy: 
 This article introduces a novel Distilled Cross-Combination Transformer (DCCT), as shown in the overall framework diagram in Figure 3 . During the encoding stage, given an image, we first extract grid features using the widely adopted pre-trained model CLIP [31] , and region features using the object detector from the pre-trained model VinVL [50] . To unify the feature dimensions, we project these features onto a specified dimension 𝑑, indicated by𝑉 𝐿 𝐺 𝐼 = {𝑣 𝑖 } 𝐿 𝐺 and 𝑉 𝐿 𝑅 𝐼 = {𝑣 𝑖 } 𝐿 𝑅 individually.Here, 𝐿 𝐺 and 𝐿 𝑅 represent grid and region feature numbers, respectively. Afterward, we input them into the DCFE for encoding. After passing through N self-attention distillation modules and a cascading layer, we obtain the grid output features 𝑉 𝑁 𝐺 𝑂 and region output features 𝑉 𝑁 𝑅 𝑂 , defined as follows:EQUATIONDuring the decoding stage, these refined grid features and region features are fed into a Parallel Cross-Fusion Attention (PCFA) module to be combined with the word features 𝑤 𝑖+1 𝑡 through a multi-head masked self-attention layer, resulting in the fused feature 𝑝 𝑖+1 𝑡 . The corresponding definition is as follows:EQUATION)The details of PCFA will be described in Section 3.2. Afterward, we pass rich multimodal representation 𝑝 𝑖+1 𝑡 through a positionwise feed-forward layer, followed by residual connections and layer normalization, to obtain the output 𝑦 𝑖+1 𝑡 . The corresponding definition is as follows:EQUATIONFinally, the output of the 𝑁 -th layer is fed into a vocabulary classifier for predicting the next word.Next, we will explain the two most important modules in DCCT: the Distillation Cascade Fusion Encoder and the Parallel Cross-Fusion Attention module.In this paper, we propose a Distillation Cascade Fusion Encoder (DCFE), which consists of N multi-head probability sparse attention layers and a cascade layer. The probability sparse attention layer includes probability sparse attention, position-wise feed-forward layer, and convolutional pooling layer. The two primary sections of this encoder component are: multi-head probability sparse selfattention and encoding visual features using DCFE. We will provide a detailed introduction to these in the following.Multi-head ProbSparse Self-Attention. Regarding the Prob-Sparse Self-Attention, we will use grid features 𝑉 𝐿 𝑅 𝐼 to illustrate since the encoding process for grid features and region features is similar. We represent the visual features as 𝑄, 𝐾, and 𝑉 . Here, the visual features are vector representations obtained through linear transformations of grid features 𝑉 𝐿 𝑅 𝐼 . In an image, not every position's attention is necessarily important. For each query 𝑄, only a small portion of 𝐾 has a strong relationship with it. Calculating every 𝑄 with every 𝐾 would be inefficient. As shown in Figure 2 (a), for the attention weight matrix heat map of grid features, it can be observed that only a small portion of 𝑄 and 𝐾 have relatively large attention weights, appearing brighter in the image, while the majority of attention weights are small or even zero, shown as black in the image. Similarly, the region feature heat map in Figure 2 (b) , follows a similar pattern to the grid features, which will not be reiterated here.Specifically, in the first step, we sample the queries 𝑄 to determine which ones are more correlated with each other and which ones have lower correlations. We begin by randomly sampling 𝐾 to obtain 𝑟 _𝑘. The number of sampled 𝐾 is given by 𝑐 • ⌊log 𝑀 𝐾 ⌋, where 𝑐 is a constant and 𝑀 𝐾 represents the number of keys 𝐾.Next, we calculate the similarity between 𝑄 and the sampled 𝑟 _𝑘 as 𝑞 𝑚 𝑘 ⊤ 𝑛 / √ 𝑑. To improve computational efficiency, we measure the difference between the maximum value among these similarities and a uniform distribution. A larger difference indicates a more significant relationship. The corresponding definitions are as follows:EQUATIONNext, we sort the calculated similarity differences S (𝑞 𝑚 , 𝐾) in descending order and select the top 𝑐 • ⌊log 𝑀 𝑄 ⌋ features 𝑄. We then compute attention between these selected features 𝑄 and all 𝐾 and 𝑉 vectors. The corresponding definitions are as follows:EQUATIONwhere Q represents the selected features 𝑄 after sorting. For the remaining features 𝑄, since their correlations with other features 𝑄 are relatively low and their activity levels are lower, we do not compute their attention. Instead, we replace their similarity relationships with other features 𝑄 by using the mean of all 𝑉 vectors. Afterward, we fill the selected 𝑄 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 features based on their indices into the original positions in 𝑉 , replacing the original mean vectors. Then, we apply residual connections and normalization to obtain the output visual features. We thus finish the Multi-head ProbSparse Self-Attention process on the original visual features. The corresponding definitions are as follows:EQUATIONEQUATIONEQUATIONwhere the mean function is used to calculate the mean vector, the fill function represents filling, and LayerNorm represents layer normalization.The process of encoding visual features with DCFE. For the entire encoding process, specifically, given the initial grid features EQUATIONEQUATIONwhere MPS represents Multi-head ProbSparse Self-Attention. Then we pass it through a position-wise feed-forward layer, followed by residual connections and layer normalization, to obtain the outputs Ḡ 𝑗+1 and R𝑗+1 . The corresponding definitions are as follows:EQUATIONEQUATIONWe keep stacking layers after one cycle of position-wise feedforward layer and ProbSparse self-attention. However, unlike traditional transformers, due to the presence of redundant combinations of 𝑉 in the visual feature map of the encoder, we perform a distillation operation on these visual features obtained through probabilistic sparse attention. This operation prioritizes selecting dominant and active features and forms a refined self-attention feature map in the next layer. The distillation process is carried out from the 𝑗-th layer to the ( 𝑗 + 1)-th layer through the following steps:EQUATIONEQUATIONwhere 𝐶𝑜𝑛𝑣1𝑑 (•) represents the convolution operation, which performs a 1-D convolution filter (kernel width = 3) on the feature dimension. BN stands for batch normalization, ELU represents the ReLU activation function, and MaxPool indicates maximum pooling. Finally, we concatenate the visual features obtained from each layer of ProbSparse self-attention to obtain the final visual features𝑉 𝑁 𝐺 𝑂 = [𝐺 𝑖 | 𝑁 𝐺 𝑖=1 ] and 𝑉 𝑁 𝑅 𝑂 = [𝑅 𝑖 | 𝑁 𝑅 𝑖=1 ].As a result, we can combine feature data from several levels and produce a more complete feature representation.The cross-modal parallel cross-fusion attention module we propose mainly consists of three inputs: the grid features 𝑉 𝑁 𝐺 𝑂 encoded by the encoder, the region features 𝑉 𝑁 𝑅 𝑂 encoded by the encoder, and the hidden state 𝑤 (𝑖+1) 𝑡 obtained from the decoder's masked self-attention sub-layer, where we describe the (𝑖 + 1) block of the decoder. First, the grid features 𝑉 𝑁 𝐺 𝑂 obtained from the encoder are fed into the cross-attention module as keys and values. Combined with the hidden state 𝑤 (𝑖+1) 𝑡 , a cross-attention mechanism is applied to capture the relationship between word features and region features. Combined with residual connections and layer normalization, the resulting attended features yield the multimodal features ḡ𝑖+1 𝑡 . Similarly, the modeling of cross-attention between grid features and word characteristics is similar to that of region features. After the same module calculation, the multimodal feature r𝑖+1𝑡 is obtained. Therefore, their respective operations are as follows:EQUATIONEQUATIONEQUATIONr𝑖+1EQUATIONwhere CA represents Cross-Attention and LayerNorm represents Layer Normalization. We concatenate the multimodal features ḡ𝑖+1 𝑡 and r𝑖+1 𝑡 with the word features 𝑤 𝑖+1 𝑡 , and then use a linear transformation to convert into 𝑑-dimensional vectors, obtaining the integrated features g𝑖+1 𝑡 and r𝑖+1𝑡 . The purpose of this is to further integrate the information of word characteristics, grid characteristics, and region characteristics, and transform the integrated information features into a specific dimension for subsequent modules to learn and compute, to generate more comprehensive and rich representations. The corresponding definitions are as follows:EQUATIONr𝑖+1EQUATIONwhere [.] denotes concatenation, 𝑊 𝑔 and 𝑊 𝑟 represent the corresponding weight matrices, and 𝑏 𝑔 and 𝑏 𝑟 represent the bias terms. We cross-fuse the integrated dual visual features, which helps the model better understand and utilize both image and text information at a macro level, reducing dependence on individual features. In addition, compared to cascading fusion and sequential fusion, this cross-fusion approach can explore deeper-level visual and semantic connections, effectively compensating for the limitations of region features and enriching the representation of multimodal features, resulting in more accurate descriptions. Specifically, for the grid feature branch, we use the integrated region feature r𝑖+1 𝑡 as keys and values, and the integrated grid feature g𝑖+1 𝑡 as queries. We then compute the cross-attention to thoroughly explore their correlations. The resulting cross-attention features are then passed through residual connections and layer normalization to obtain enriched multimodal features 𝑔 𝑅 (𝑖+1) 𝑡 . Similarly, for the region feature branch, the learning process is similar to the grid feature branch, where after a series of computations, the feature 𝑟EQUATION𝑔EQUATION𝑟EQUATION𝑟EQUATIONFinally, we concatenate these enriched multimodal features 𝑔 separately. This weighted combination method is used to adjust the importance and contribution of the two multimodal features to the generation of descriptions, enabling the model to better utilize information from different features.Following this, to further model the deep relationship between language context and multimodal features, we add the result of the weighted sum of multimodal features to the word feature 𝑤 𝑖+1 𝑡 and then normalize the result to obtain the final comprehensive feature representation 𝑝 𝑖+1 𝑡 with enhanced expressiveness and adaptability. The following are the matching definitions:EQUATION𝛼EQUATIONEQUATIONThe multimodal comprehensive feature 𝑝 𝑖+1 𝑡 expands the capability for complex cross-modal reasoning, and it will be fed into subsequent feedforward neural networks and decoding layers for further decoding.As is customary in image captioning research [6, 32] , we use crossentropy loss (XE) for pre-training the model and reinforcement learning for fine-tuning. Specifically, during the XE training phase, with the target true sequence 𝑤 * 1:𝑇 provided, the goal is to reduce the cross-entropy loss (XE) defined as follows.:EQUATIONwhere 𝜃 is our model's parameter. Next, we use the self-critical sequence training (SCST) [32] approach to constantly optimize the non-differentiable CIDEr-D score during the reinforcement learning phase as follows [6] :EQUATIONwhere 𝑘 is the beam search size, 𝑟 is the CIDEr-D score function, and 𝑏 = ( 𝑖 𝑟 (𝑤 𝑖 1:𝑇 ))/𝑘 is the baseline. 
experiments 4.1 experimental settings: 
 Dataset. We conducted experiments on the MSCOCO 2014 dataset [20] , which consists of 123,287 images, including 82,783 training images, 40,504 validation images, and 40,775 test images, each with 5 different annotations. To ensure a fair comparison with most existing techniques, we utilized the splits provided by Karpathy et al. [12] , where 5,000 images were used for validation, 5,000 images were used for testing. The remaining images were used for training.Additionally, MSCOCO provides 40,775 images for online evaluation, with their annotations not publicly available. During the training process, we converted all training captions to lowercase and removed words that appeared less than 5 times, resulting in a vocabulary of 10,201 words. Evaluation Metrics. The effectiveness of image descriptions is evaluated using a variety of captioning metrics, encompassing BLEU (B@1-4) [28] , METEOR (M) [3] , ROUGE (R) [5] , CIDEr (C) [38] , and SPICE (S) [1] .Implementation Details. In DCCT, we used the Faster R-CNN object detector provided by VinVL [50] and the pre-trained CLIP-RN50×4 [31] model to extract region features and grid features, respectively. The grid size was 9 by 9, and the maximum number of items in region features was 50. The dimensionality of the grid features was 2560, while the dimensionality of the extracted region features was 2048. We set the number of cascade layers to 1, the number of probability sparse attention layers in the encoder to 3, and the number of layers in the decoder to 3. In addition, we set the hyperparameters for DCCT training by implementing the Transformer model as suggested in [6] . The feed-forward neural network (FFN) layer had an inner dimension of 2048, the multi-head attention mechanism was configured with 8 heads, the dimensionality 𝑑 of each layer was set to 512, and the dropout after each multi-head attention and FFN layer was set to 0.1. During the cross-entropy training stage, we used the Adam optimizer to train our model until the CIDEr metric continuously decreased for five epochs. At that point, we switched to self-critical sequence training, i.e., reinforcement learning stage. The batch size was set to 20. Additionally, the learning rate scheduling approach was implemented using the following [51] : 𝑙𝑟 =              𝑏𝑎𝑠𝑒_𝑙𝑟 * 𝑒/where 𝑒 is the number of iterations that are currently being performed, starting at 0, and the base learning rate (𝑏𝑎𝑠𝑒_𝑙𝑟 ) was set to 0.0001. During the reinforcement learning phase, we optimized the model employing the Adam optimizer with a batch size of 100 and a fixed learning rate of 5 × 10 -6 . The training process was stopped after five epochs of progressively decreasing CIDEr metrics. We employed a beam search approach with a beam size of five during inference.To show the efficacy of the suggested distillation cascade fusion encoder (DCFE) and parallel cross-fusion attention module (PCFA) and their effects on the overall performance of DCCT on the MS COCO dataset, we carried out some ablation experiments in this section. The outcomes of our ablation trials are displayed in Table 1 .As can be seen in Table 1 , the Transformer-based Base model performs worse overall with a CIDEr score of only about 122.9 when the dual visual features are not fused, as demonstrated in the second and third rows of the table, i.e., when either grid features or region features are used for image captioning. In the cross-attention module, we usually fuse both region and grid features when they are used for captioning images.We initially fixed the encoder section as the basic Transformer encoder module "Base". Then we compared the overall performance of various cross-attention modules with our suggested PCFA to confirm the efficacy of our proposed PCFA. This is shown in rows 4, 5, and 6 of Table 1 . "Concat Fusion" denotes the cascade fusion of dual visual features, which are subsequently supplied into the decoder to decode. "Sequential Fusion" represents sequential fusion, where cross-attention is separately computed for region and grid features, and their attention results are added together for subsequent decoding. As indicated by the experimental results in the table, the model's overall performance is higher after utilizing feature fusion techniques in the cross-attention module than when using only one feature type. Furthermore, when using only cascade fusion, the CIDEr score is 1.3 points higher than when using only grid features. However, whether using cascade fusion or sequential fusion, the performance improvement is minimal, with a difference of only 0.7 between the two. Nevertheless, when we replace cascade fusion and sequential fusion with PCFA, the CIDEr score increases by 1.7 compared to cascade fusion and by 1.0 compared to sequential fusion. This indicates that our PCFA can fully utilize the correlation and complementarity between grid and region features, thereby achieving better characteristic fusion and expressive capability.To analyze the effectiveness of DCFE in the encoding phase, we kept the cross-attention part of the decoder fixed, using cascade fusion, sequential fusion, and PCFA, as indicated in rows 4 and 7 of Table 1 . When the cross-attention part of the decoder was fixed as cascade fusion, our encoder DCFE significantly improved the model's overall performance in comparison with the Transformerbased encoder, with the CIDEr score increasing by approximately 1.0. This is because our encoder, during feature encoding, filters out some redundant features through the probabilistic sparse attention layer, allowing attention to focus more on important feature parts and making subsequent decoding simpler and more efficient. Subsequently, when the cross-attention module was fixed as sequential fusion, there was also an improvement in model performance. We consider the model architecture in row 4 as our baseline model. When we equipped our DCFE and PCFA on the baseline model, the CIDEr score achieved the best performance at 127.5, further demonstrating the effectiveness of our proposed encoder and crossattention module for image caption generation. Online Evaluation. To better confirm the effectiveness of our image captioning model DCCT, we submitted it to the MS COCO online test server for evaluation and compared its performance with other online models. For the online evaluation, since most top-performing methods on the leaderboard use ensemble models, we also used the aforementioned ensemble configuration for a fair comparison. In Table 4 , we respectively recorded the overall effectiveness of the model employing five reference descriptions(c5) along with forty standard descriptions(c40). Observing that our approach surpasses the current advanced methods across all metrics, it attains the highest performance level. Compared to COS-Net, our model achieves scores of 76.4 and 140.2 in BLEU-4 (c40) and CIDEr (c40) respectively, which are 1.7 and 1.9 points higher than the best-performing methods.In this section, we conducted a detailed qualitative analysis of the proposed DCCT model, with Figure 4 showing examples of descriptive sentences generated by DCCT and the Transformer baseline, along with basic factual sentences annotated by humans (GT). It can be observed that both DCCT and the Transformer baseline models are capable of generating coherent language descriptions. However, when redundant feature information is filtered out, some finergrained information becomes more prominent, making it easier for our DCCT model to capture these refined details and generate more accurate descriptions. For example, in the first instance, due to the inability of the baseline model to filter out certain redundant features and capture finer-grained information, an incorrect key descriptor "balloon" is generated, whereas we produce "kite," which matches the ground truth annotation. Simultaneously, to mitigate the negative effects of regional features, DCCT combines grid features to compensate for them, resulting in more accurate positioning and fine-grained content. For example, in the first instance, the model generates "next to" and in the fourth example it generates "broccoli". 
 
 Appendix:  


Decision: The paper presents the Distilled Cross-Combination Transformer (DCCT) for image captioning, which uses region and grid visual features. It introduces a Distillation Cascade Fusion Encoder (DCFE) to refine visual features by filtering out redundancies and a Parallel Cross-Fusion Attention (PCFA) module to combine the features effectively. The related experiments have demonstrated the effectiveness of the method. Some reviewers expressed concerns about the paper's novelty, but after one rebuttal round, the authors provided convincing responses, and the reviewers acknowledged its contribution to the image captioning field. Hopefully, the authors will further improve the paper based on the reviewers' feedback.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel Transformer-based image captioning framework. The proposed method compresses tokens with less impact to focus the attention on important features. The proposed method combines grid features and region features, achieving advanced performance on MSCOCO dataset.
strength_weakness: strength_weakness: 1. This paper introduces distillation into the image captioning task, and the proposed method compresses tokens with less impact to focus the attention of the model on important features. This method demonstrates high innovation.
2.The proposed method utilizes cross-attention to integrate grid features and region features. The simplicity and effectiveness of this method have also been demonstrated through comparison studies.
3. Sufficient experiments have been conducted on the MSCOCO dataset to demonstrate the effectiveness of this method.
limitations: limitations: 1.The Motivation expression reflected in Figure 1 (b) is slightly inferior. The rationality of the two proposed questions needs to be verified and supported by relevant papers. Additionally, there is ambiguity between Figure 1 (b) and the description of the problem.
(i)The first type of understanding is that according to Figure 1 (b), both G and R of Sequential Fusion are cross-attention to each other's V, Q, and K. For the first question, since both G and R of Sequential Fusion have cross-attention as V, Q, and K respectively, why is it that, as stated in the paper, "the interaction between different features cannot be fully utilized"? For the second question, although the order will affect the final performance, the model still considers both orders, so there is still a lack of argumentation and persuasiveness.
(ii)The second type of understanding, according to the textual expression in the paper, is that both G and R in Sequential Fusion are self-attention rather than cross-attention in Figure 1 (b). So, for the second question, since the Transformer block uses a position encoding sequence and there is no forgetting problem as in LSTM, there will be no issue of insufficient attention on the other due to the flipping of the order.
2. The paper lacks proof for the first motivation about distillation. We’d like to investigate the changes in network attention before and after distillation in order to verify the effectiveness of the first motivation.
3. The paper lacks visualization results for word generation to prove the second motivation. This type of visualization results are important as it demonstrates the grid features and region features that each generated word focuses on.
paper_summary: paper_summary: This paper focuses on the effectiveness and efficiency of the image caption, where an architecture named DCCT is proposed. Specifically, DCCT contains the DCFE and PCFA, the former is designed to filter out redundant visual features based on the attention weights, and the PCFA is proposed to fuse the multi-modal features better.
strength_weakness: strength_weakness: 1. The motivation of the paper is clear and the proposed modules are effective.
limitations: limitations: 1. The novelty of the proposed method is limited, for example [1] has already proposed a gated-attention mechanism to select a subset of elements to attend. The PCFA also seems trivial with only interactions across multi-modal features;
2. The performance of the proposed method is not the best in METEOR, the bold is not appropriate;
3. It is better to provide  a performance of CLIP as a baseline, since the proposed method adopts the CLIP as a feature extractor.


[1] Not All Attention Is Needed: Gated Attention Network for Sequence Data, AAAI, 2020
paper_summary: paper_summary: The paper presents the Distilled Cross-Combination Transformer (DCCT) for image captioning, which uses both region and grid visual features. It introduces a Distillation Cascade Fusion Encoder (DCFE) to refine visual features by filtering out redundancies and a Parallel Cross-Fusion Attention (PCFA) module to combine the features effectively. Experiments on the MSCOCO dataset show that DCCT performs well across various metrics compared to existing methods. The approach addresses the integration of multiple types of visual features and their fusion in generating image captions. The complexity of the model and its validation on other datasets are mentioned as considerations for further research.
strength_weakness: strength_weakness: -The figure are very clear and easy to understand.
-The writing is straightforward and easy to follow.
-The model achieves high performance across various metrics on the MSCOCO dataset.
limitations: limitations: -PCFA is a lot like [a]; you need to talk more about the differences, like how the features interact.
-Show some actual numbers to prove that DCFE speeds up inference after filtering tokens.
-Include visualizations of DCFE's effects so we can see the results clearly.
-The performance improvement is pretty minor, and the bolding in the first column of Table 2 looks wrong.
-You missed discussing several important works like [b]; you should compare them in the related work and in Tables 2, 3, and 4.

If the authors address these issues, I will consider raising the score.

[a] Dual-Level Collaborative Transformer for Image Captioning. AAAI 2021.
[b] Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network. AAAI 2021.