Visual-linguistic Cross-domain Feature Learning with Group Attention and Gamma-correct Gated Fusion for Extracting Commonsense Knowledge

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-15 13:15:08

Abstract: Acquiring commonsense knowledge about entity-pairs from images is crucial across diverse applications.  Distantly supervised learning has made significant advancements by automatically retrieving images containing entity pairs and summarizing commonsense knowledge from the bag of images. However, the retrieved images may not always cover all possible relations, and the informative features across the bag of images are often overlooked. To address these challenges, a Multi-modal Cross-domain Feature Learning framework is proposed to incorporate the general domain knowledge from a large vision-text foundation model, ViT-GPT2, to handle unseen relations and exploit complementary information from multiple sources. Then, a Group Attention module is designed to exploit the attentive information from other instances of the same bag to boost the informative features of individual instances. Finally, a Gamma-corrected Gated Fusion is designed to select a subset of informative instances for a comprehensive summarization of commonsense entity relations. Extensive experimental results demonstrate the superiority of the proposed method over state-of-the-art models for extracting commonsense knowledge.

Main paper: 
 related work: 
 Commonsense Knowledge Extraction automatically gathers implicit knowledge from diverse sources [34] . However, the construction of large-scale commonsense knowledge bases [27] heavily relies on manual annotations, greatly limiting their applications. Many models have been designed to automatically extract common relations between entities, which can be broadly divided into language-based, vision-based, and multi-modal models [34] . Language-based CKE Models. Early CKE models often utilize semi/unstructured text [12] or pre-trained language models [2] to identify entity relations. Schuster et al. [28] employed a Relation Triplet Parser (RTP) to extract relation triplets from image descriptions. Pre-trained Language Models [5] have driven the development of natural language processing, which could learn rich features from raw texts. Petroni's [25] designed a model named LAnguage Model Analysis (LAMA) to evaluate large language models such as BERT [5] for extracting commonsense knowledge. Peng et al. [23] applied BERT [5] on contexts and entities to predict their relations. Lin et al. [14] evaluated BERT [5] for encoding numerical commonsense knowledge. Lu et al. [18] introduced a knowledge-evolving framework by iterative consolidation and expansion with the guidance of PLMs and devised a rule generator by prompt-tuning to stimulate the rich knowledge in PLMs. Despite the progress, it has been observed that the commonsense in PLMs suffers from low consistency where small changes in queries may lead to significantly different predictions [22] . Vision-based CKE Models. More evidence suggests that visual perception brings commonsense that text may not reveal [34] . Yatskar et al. [36] uncovered general facts from image captions by leveraging WordNet and submodular k-coverage. Dai et al. [4] employed scene graphs to understand visual interactions between objects. But most image-based approaches are confined to visual commonsense of specific attributes [17, 40] , or require large amounts of labeled data [39] . Distantly supervised learning [15, 35] has been designed to automatically create a large volume of commonsense relations without costly manual annotations. For example, Yao et al. [35] employed a distant supervision strategy to align relational facts in existing knowledge bases with real-world image bags. However, distantly supervised learning often results in inaccurately labeled samples, which greatly hinders the performance of CKE models. Multi-modal CKE Models. Distant supervision and utilizing information from diverse modalities are shown beneficial for commonsense extraction [7, 34] . To mitigate mislabeled instances in distant supervision, Zeng et al. [39] developed Piecewise Convolutional Neural Networks with multi-instance learning. Chen et al. [3] designed a hierarchical multi-modal fusion with a dynamic gated aggregation strategy to remove irrelevant object-text pairs. Lin et al. [15] utilized sentence-level attention to reduce the influence of noisy instances. Knowledge graphs are often utilized to represent entity relations. Zheng et al. [43] aligned different modalities through a dual graph structure, which better correlates visual relations among objects to textual relations. Feng et al. [7] constructed two cross-modal knowledge graphs to bridge the semantic gap and adopted a cross-attention mechanism to learn the cross-modal knowledge representations. Ma et al. [20] developed a multi-source knowledge reasoning graph network to learn multimodal correlations, intra-event object relations, and inter-event semantic associations for event-centered commonsense inference. Various attention mechanisms have been designed to capture crossmodality interactions, e.g., bi-linear attention [44] , cross-modal attention network [42] , etc. In particular, Yao et al. [34] utilized a pre-training vision-language model to analyze images and implemented a contrastive attention mechanism to choose descriptive images for commonsense relation summarization from entity pairs. 
proposed method 3.1 overview of proposed method: 
 To tackle the challenges of commonsense knowledge extraction, Multi-modal Cross-domain Feature Learning with Group Attention and Gated Fusion (MCFL-GAGF) is proposed. The overall structure is illustrated in Fig. 2 . First of all, to extract commonsense facts about a pair of entities, a bag of images containing the specified entity pair is retrieved from a large image corpus, Visual Genome [10] , which contains relational triplets about entities derived from realworld images. Then, three modules are designed to extract common entity relations. 1) Multi-modal Cross-domain Feature Learning (MCFL) module, which extracts instance features in two stages: Single-modal Learning to extract visual features of the retrieved images by utilizing VinVL [41] , and linguistic features of entity labels and captions generated from ViT-GPT2 [1, 29] using word embedding [40] ; and Cross-modal learning to utilize Caption-Bert [40] to exploit interactions between modalities and modal alignment from three pairs of visual-language features. 2) Group Attention (GA) module, which exploits the semantic interactions across different instances to enhance the features of individual instances with the attentive support of other images in the same bag. For each instance, we first extract the co-attention features from every other instance and then aggregate these features to boost the features of the current instance. 3) Gamma-corrected Gated Fusion (GGF) module, which mitigates the negative effects of mislabeled relations for instances in distant supervision and optimally fuse the group-attended features of individual instances through a set of Gamma-corrected gated ratios adaptively determined by all instance features through a set of gated nets. Finally, the discriminative commonsense knowledge derived from each instance is aggregated together with bag-level features representing collective group knowledge to produce the final prediction of entity relations.Given a query entity pair (𝑠, 𝑜), the commonsense relations between entities 𝑠 and 𝑜 can be summarized from a bag of 𝑁 images {𝑰 𝑖 } 𝑁 𝑖=1 . However, the retrieved images may not cover all the possible entity relations and it is challenging for a model to handle entities and relations that are not covered in the training data [7] . Summarizing such knowledge from diverse sources as complementary information [34] and incorporating the general domain knowledge as additional knowledge [7, 37] are two potential solutions. The proposed MCFL seamlessly integrates the two advantageous methods. Single-modal Learning. Single-modal learning is designed to extract linguistic features and visual features separately. Linguistic features for entity tags are encoded by leveraging the GloVe vectors [24] as,EQUATIONwhere E 𝑖 denotes the set of 𝐸 entity tags for the 𝑖-th image and F 𝐺 denotes the word embedding operation in GloVe.𝑬 𝑖 = [𝒆 1 𝑖 , . . . , 𝒆 𝐸 𝑖 ],where 𝒆 𝑗 𝑖 is feature vector of the 𝑗-th entity. Linguistic features for image captions are extracted using GloVe vectors [24] , where captions are generated by ViT-GPT2 [1] , a large vision-language model pre-trained with extensive image-text pairs. It contains a large amount of commonsense relations in the general domain. For each image 𝑰 𝑖 , a set of 𝑊 caption words W 𝑖 = F 𝑉 𝐺 (𝑰 𝑖 ) are generated, where F 𝑉 𝐺 denotes the caption-generation operation of ViT-GPT2. W 𝑖 is encoded into linguistic features using GloVe vectors as,EQUATIONwhere 𝑪 𝑖 = {𝒄 1 𝑖 , . . . , 𝒄 𝑊 𝑖 }, and 𝒄 𝑗 𝑖 denotes the 𝑗-th word embedding. Visual features are extracted using VinVL [41] , a visual-language model pre-trained on the Visual Genome dataset [10] ,EQUATIONwhere F 𝑉 denotes the feature extraction process in VinVL. 𝑽 𝑖 = {𝒗 1 𝑖 , . . . , 𝒗 𝐸 𝑖 }, where 𝒗 𝑗 𝑖 denotes the visual vector for the 𝑗-th entity. Cross-modal Learning. Cross-modal learning is designed to exploit the interactions between visual features and linguistic features. Single-modal features are combined into three different pairings, i.e., {𝒗 𝑖 , 𝒆 𝑖 }, {𝒗 𝑖 , 𝒆 𝑖 , 𝒄 𝑖 }, and {𝒆 𝑖 , 𝒄 𝑖 }, corresponding to the visuallinguistic pairing between images and entities, multi-modal crossdomain feature pairing, and cross-domain linguistic feature pairing for entities and captions, respectively. A recent vision-language model that performs well in aligning features of different modalities, CaptionBert [40] , is utilized to model the complex instance-level feature interactions as,EQUATIONEQUATIONEQUATIONMulti-Modal Learning where F 𝐶𝐵 represents the multi-head self-attention mechanism in CaptionBert [40] . Then, these features are jointly encoded using an MLP F 𝑀𝐿𝑃 as,𝒉 ! 𝒉 " Interaction Matrix 𝑮 !! 𝑮 #! 𝑮 !# 𝑮 ## 𝒉 # ! 𝒉 # #GenerateEQUATIONwhere 𝒉 𝑖 are the single-instance features for the 𝑖-th image. The proposed MCFL incorporates the general domain knowledge from pre-trained models, extracts relation features from multiple sources, and exploits the multi-modal cross-domain interactions among these features.To mitigate the mislabeled instances due to distant supervision, multi-instance learning [34] has been designed to collectively consider a group of instances and jointly exploit the information from the group to discern commonsense knowledge at the bag level. However, multi-instance learning often overlooks the discriminant information embedded in individual instances. To exploit both the discriminant information in individual instances and the collective attention information in a group of instances, a Group Attention module is proposed to extract informative features. Specifically, for each instance 𝒉 𝑖 , it queries every other instance 𝒉 𝑗 to determine their co-attention information, where the interaction matrix 𝑮 𝑖 𝑗 contains the attention weights indicating which part in 𝒉 𝑖 is more important with respective to 𝒉 𝑗 ,EQUATIONwhere 𝑓 𝑠 is the softmax function, and 𝑾 𝐺 is a learnable weight matrix. The co-attention features 𝒉 𝑖 𝑗 from 𝒉 𝑖 to 𝒉 𝑗 are then calculated as,EQUATIONwhere 𝑅𝑒𝐿𝑢 is the ReLu function.Finally, we derive the group-attended features as,EQUATIONwhere 𝜎 is the sigmoid function, 𝑾 𝑖 and 𝑏 𝑖 are the learnable weights and bias. The group-attended features ĥ𝑖 represent a dynamic focus on key common relations between entities. By exploiting the attentive information from each instance in the group, the proposed Group Attention module greatly enhances the features of individual instances.Given the group-attended features of individual instances, some instances may contain more common relations while some contain less frequent relations which are potentially mislabeled. Thus, it is important to design an effective fusion scheme to uncover the common entity relations in these instances while mitigating the negative efforts of noisy instances. To achieve this, a Gammacorrected Gated Fusion module is proposed to automatically focus on a subset of instances indicating reasonable entity relations while ignoring less informative instances. Specifically, all instance features { ĥ𝑖 } 𝑁 𝑖=1 are fit to an MLP to excavate support evidence and Visual-linguistic Cross-domain Feature Learning with Group Attention and Gamma-correct Gated Fusion for CKE ACM MM, 2024, Melbourne, Australia fused to construct the gated ratio 𝑟 𝑖 for the 𝑖-th instance,EQUATIONwhere 𝒘 𝑖 𝑗 and 𝑏 are the learnable weights and bias. It is witnessed that the derived gated ratios 𝒓 = [𝑟 1 , . . . , 𝑟 𝑁 ] for different instances do not differ significantly, possibly due to the too many instances to fuse, resulting in insignificant contrast between the informative instances and less informative ones. To amplify the contrast, the Gamma correction is applied to the gated ratios. The fused features are then derived as,EQUATION)where 𝛾 is the hyper-parameter for Gamma correction. Finally, the fused features are mapped to the answer space as,EQUATIONwhere W are learnable mapping weights. The proposed Gammacorrected Gated Fusion (GGF) aims to derive an optimal subset of instances containing commonsense knowledge, which differs from the existing gated fusion [26] that aims to combine multiple feature maps into one. In addition, the proposed Gamma correction better distinguishes informative instances from non-informative ones and mitigates the latter during fusion, thereby effectively extracting commonsense knowledge. Finally, the binary cross-entropy loss is utilized in this paper,EQUATION)where 𝐶 is the number of classes, ȳ = { ȳ𝑖 } 𝐶 𝑖=1 are the predicted labels and 𝒚 = {𝑦 𝑖 } 𝐶 𝑖=1 are the ground-truth labels. 
experimental settings: 
 Dataset Description. Very recently, Yao et al. [34] developed a large dataset for multi-modal commonsense knowledge extraction, which is a subset of the Visual Genome dataset [10] , denoted as VG-CKE. The VG-CKE dataset [34] focuses on the top 100 entities and relations, comprising 6,443/1,964/678 entity pairs and 13,780/3,496/1,166 corresponding commonsense facts for training, testing, and validation, respectively. These relation facts are automatically aligned using distant supervision with image bags, resulting in 55,911 images for training, 13,722 for testing, and 5,224 for validation. Some instances may be mislabeled due to distant supervision [34] .Compared Methods. The proposed method is compared to nine CKE models. 1) Four language-based models. RTP [28] utilizes dependency trees to extract commonsense triplets from captions, whereas LAMA [25] , Vanilla-FT [23] and Prompt-FT [14] summarize relational facts by leveraging linguistic knowledge in pretrained language models. 2) Five multi-modal models. ONE [39] , ATT [15] , AVG [15] and CLEVER [34] adopt multi-instance learning to synthesize commonsense relations from multiple modalities, which extract visual features using VinVL [41] and linguistic features using Glove [24] , and employ CaptionBert [40] to extract vision-text features. They differ from the strategies of combining instance features, e.g., max pooling for ONE [39] , weighted sum for ATT [15] , average pooling for AVG [15] , and contrastive attention for CLEVER [34] . In addition, CLEVER 𝐸𝑛𝑠. [34] combines the results of RTP [28] , Vanilla-FT [23] , and CLEVER [34] through score-level fusion. The results of these methods are obtained from [34] . Implementation Details. Following [34] , the bag size is set to 𝑁 = 50. 𝛾 in Gamma-corrected Gated Fusion is empirically set to 3. The AdamW optimizer is employed for training, with an initial learning rate of 5e-6 and a decay rate of 0.01. Training stops when no significant gain is observed in 10 consecutive epochs. Evaluation Metrics. For a fair comparison, the same evaluation metrics used in [15, 34, 39] are employed, including Area Under the Curve (AUC and mAUC), F1 score (F1 and mF1), as well as Precision@2% (P@2% and mP@2%), where 'm' denotes 'perrelation' macro evaluation obtained by averaging corresponding scores across different relations, and 'P@2%' represents the precision of the top 2% retrieved candidates. The AUC and F1 scores offer one-value evaluations for the model, which are principally utilized for result analysis in this study.The experimental results of all the compared CKE methods on the VG-CKE dataset are summarized in Table 1 . We have the following observations. 1) The proposed method significantly outperforms all compared methods in terms of all the evaluation metrics.Compared to the second-best method, CLEVER 𝐸𝑛𝑠 [34] , the proposed method achieves the performance gain of 3.48%, 1.37%, 1.14%, 1.71%, 1.27%, and 1.90% in terms of AUC, F1, P@2%, mAUC, mF1, and mP@2%, respectively. The large performance gains clearly demonstrate the effectiveness of the proposed method in extracting common entity relations. 2) Note that CLEVER 𝐸𝑛𝑠 performs a score level fusion of three models, RTP [28] , Vanilla-FT [23] , and CLEVER [34] . Compared to the previous best-performing single model, CLEVER [34] , the performance gains are much more significant, e.g., 7.24%, 2.34%, 2.39%, 2.52%, 0.78%, and 2.68% in terms of six evaluation metrics.3) The previous two best-performing languagebased methods, Vanilla-FT [23] and Prompt-FT [14] , which utilize the inherent linguistic understanding in pre-trained language models, yield a poor performance compared to the previous best performing multi-modal model, CLEVER [34] . Such observation not only demonstrates the effectiveness of integrating visual and linguistic information for CKE but also validates the design of the proposed Multi-modal Cross-domain Feature Learning framework. The proposed method is visually compared to the previous bestperforming CLEVER [34] in two scenarios. 1) The proposed model successfully uncovers the entity relations that CLEVER cannot. As shown in the first column of Fig. 3 , the presence of 'man', 'under', and 'umbrella' in both captions enables the proposed method to successfully predict the 'under' relation, while CLEVER [34] fails to do so. Similar examples can be observed in the next two columns, which demonstrate the benefits of exploiting general domain knowledge in generated captions. 2) The proposed method correctly predicts the relations that CLEVER [34] inaccurately extracts. As depicted in the fourth column, the proposed method correctly [34] . The proposed method accurately extracts more relational facts of the specified entity pairs. Facts that were missed by CLEVER but correctly summarized by the proposed methods are highlighted in green, while relation facts that are inaccurately abstracted by CLEVER are highlighted in red.identifies that there is no relevant relation between the 'paper' and 'mouth' entities, while CLEVER [34] wrongly identifies the relation as 'in'. Similar examples can be observed in the fifth column. The improved accuracy can be attributed to the proposed Group Attention module, which significantly enhances single-instance features, coupled with the Gamma-corrected Gated Fusion to mitigate noisy instances.A set of ablation studies have been carried out to validate the effectiveness of every novel contribution. Ablation Study of Major Modules. An ablation study is carried out on the VG-CKE dataset [34] to evaluate the three major proposed modules. The baseline method utilizes CaptionBert [40] to extract the vision-text features from images and entities, concatenation of all instance features, and an MLP for prediction. We gradually replace the feature extraction module with the proposed Multi-modal Cross-domain Feature Learning (MCFL), incorporate the Group Attention module (GA), and replace the concatenation with the proposed Gammacorrected Gated Fusion (GGF). The results are summarized in Table 2. The following can be observed. 1) Compared to the baseline, by integrating the general domain knowledge embedded in captions and exploiting the interactions with other features, significant performance gains of 2.17%, 0.49%, 0.91%, and 1.02% in AUC, F1, mAUC, and mF1 are achieved, respectively. 2) By exploiting the Group Attention, the attention information across images is aggregated based on their semantic interactions and relational similarity. The AUC, F1, mAUC, and mF1 are hence further improved by 4.86%, 1.57%, 3.29%, and 6.26%, respectively, which are mainly attributed to the attentive support from other instances to enhance the instance features. 3) By incorporating the proposed Gamma-corrected Gated Fusion, the AUC, F1, mAUC, and mF1 are further boosted by 0.43%, 0.95%, 2.23% and 0.59%, respectively, as the proposed GGF better fuses these instance features for commonsense knowledge extraction and mitigates the influence of the mislabeled instances. These ablation results demonstrate the effectiveness of all three proposed major modules. 2 , where we demonstrate that the proposed GA effectively exploits the attentive information from a group of instances to enhance the description ability of individual instances.To further demonstrate the effectiveness of the Group Attention module, we compare it with different cross-instance methods and summarize the experimental results in Table 4 . 'MLP-Instance' [38] applies MLPs on features of each individual instance to boost its description power. 'MLP-Token' [6] applies MLPs to each individual feature across instances. 'MLP-All' [32] combines the two aforementioned methods. 'BasicATT' [16] aggregates information from other instances with weights determined by feature similarity through an attention mechanism. As shown in Table 4 , the proposed GA significantly surpasses all the compared methods.Ablation Study of Fusion Methods. The proposed fusion method is compared with four feature-level fusion methods and two scorelevel fusion methods, e.g., 'Feature-Concat' represents the concatenation of all instance features, and 'Feature-GatedFusion' represents gated fusion [26] . The remaining components and settings remain We further analyze the failure cases and categorize them into two types. 1) Type-I failures comprise triplets that are not labeled in the VG-CKE dataset but can be inferred from the images. For instance, in the first case, the relational triplets (woman, 'in front of'/'with', 'elephant') are reasonably summarized by the proposed method, yet not predicted by CLEVER [34] nor included in the ground-truth annotations. In the second case, the proposed method soundly infers the relations between the 'sign' and 'fence' as 'in front of', and 'on', while CLEVER [34] only identifies the additional relationship of 'in front of'. The instances are retrieved by automatically aligning relational facts from knowledge bases to the Visual Genome dataset [10] through distant supervision, and hence there may be missing labels. Despite the challenges, the proposed method correctly identifies these relations from images. Similar cases can be observed in columns 3 to 5 in Fig. 4 . 2) Type-II failures consist of incorrectly recognized relational facts by the proposed method. As shown in the first column of Fig. 5 , the relation 'has', 'on', 'under', and 'with' are summarized by our model, but they are not labeled in the VG-CKE dataset. Other relational facts of 'behind' and 'of' for the entity pair (track, man), 'in' for the entity pair (bike, car), 'on' for (person, book) and 'has' for (mountain, face) are also erroneously inferred by both the proposed model and CLEVER [34] , as evident from the respective images in Fig. 5 .  
: 
 Figure 1 : The proposed model integrates linguistic features of captions generated from the large pre-trained model, ViT-GPT2, to provide the general domain knowledge on unseen relations, together with linguistic features from entity labels and visual features from retrieved images, to exploit multimodal cross-domain interactions within individual instances and attentive support from a group of instances to extract commonsense knowledge from these sources.ATOMIC [27] often store abundant structured human commonsense. However, the high costs and reliance on human annotations for building these CKBs limit their scale and concept coverage [21] .Automatic Commonsense Knowledge Extraction (CKE) has received increasing research attention [13, 30, 34] . Early approaches primarily utilize text [12] or Pre-trained Language Models (PLMs) [2, 25] to obtain rich and diverse commonsense knowledge, but the inherent bias of documenting more unusual circumstances than common occurrences in textual data leads to possible over-report situations in large corpora [22, 40] . Recently, CKE from images has demonstrated promising potentials, e.g., 83% of relation facts in visual relation learning datasets [34] are not covered in the large-scale text-based dataset, ConceptNet [31] . However, most image-based approaches are confined to visual commonsense in attributes such as spatial relation, color, and relative size. Recent models combine visual and textual modalities as complementary signals [34, 35] , developing cross-modality learning solutions to capture more commonsense knowledge.Despite the recent advancements, CKE still poses several challenges. 1) It often requires extracting unseen relations between entities. However, it is hard for a model to handle entities and relations not covered in the training data. Incorporating general domain knowledge could help identify unseen entity relations, while existing models often lack such general domain knowledge [8] .2) Many methods [14, 23] assume that each instance carries sufficient information to independently determine the relation of an entity pair. However, commonsense is aggregated over a group of instances, and a single instance alone is insufficient to derive a common relation. 3) The presence of inconsistent labels for the same entity relation may confuse the model to accurately derive the commonsense relations [34] . As a result, directly aggregating all the single instances that may contain noisy labels may not accurately extract common entity relations.To better identify unseen entity relations, a Multi-modal Crossdomain Feature Learning (MCFL) framework is proposed, which integrates ViT-GPT2 [1, 29] , a pre-trained large-scale visual-text model containing rich general domain knowledge, to reduce the knowledge gap between sparse objects and their rich relations. Specifically, given an entity pair, we first retrieve a bag of images containing the entities from a large image corpus, extract visual features by using VinVL [41] , and derive linguistic features of entity labels through word embedding using GloVe vectors [24] . Due to the limited number of retrieved images, these features may not cover all the feasible relations between entities, while ViT-GPT2 supplements more linguistic information by generating image captions containing rich relations not seen in the retrieved images. Next, the proposed MCFL utilizes CaptionBert [40] to efficiently extract vision-text features from the pairings between visual features and entities (and captions) embeddings, as well as the linguistic feature pairings between entities and captions.Then, to boost the features of individual instances, a Group Attention (GA) module is proposed. In single-instance learning [14, 23] , each instance is treated independently to determine the relation of a specific entity pair. While effectively exploring the entity relations in an instance, this approach often overlooks the inter-instance relations and the collective insights gained from a bag of instances, which are crucial for extracting commonsense knowledge. In contrast, multi-instance learning [34] exploits the collective information from these groups to discern commonsense knowledge at the bag level, but the entity relations residing in individual instances are often overlooked. To tackle the problem, the proposed Group Attention module first derives the interaction matrix between every pair of instances, utilizes it to construct the co-attention features between every pair, and aggregates the attentive support across the whole group of instances. In such a way, the features of individual instances are greatly enhanced by exploiting partial attentive information of the whole group.Due to automatic annotations in distant supervision with existing knowledge bases [34] , instances may be assigned the wrong labels. In addition, it remains challenging to optimally aggregate common relations from multiple instances, especially when these instances are noisy. To tackle these, a Gamma-corrected Gated Fusion (GGF) module is proposed to optimally aggregate the groupattended single-instance features. Specifically, a gated net is applied to all instances to determine the adaptive weight of each instance. Intuitively, an instance with more similar instances in the group should be assigned a higher weight as it more likely contains common relations. As the gated ratios derived across many instances tend to be similar, a gamma correction is hence designed to enlarge the differences between gated ratios. As a result, the instances containing fewer occurred relations are assigned smaller weights, while the instances containing common relations are assigned larger weights. The final relation features of all instances are fed to a multi-layer perceptron (MLP) for answer space mapping.Our contributions can be summarized as follows. 1) To exploit unseen relations and entities from images and texts, a Multi-modal Cross-domain Feature Learning framework is proposed, which effectively integrates the general domain knowledge embedded in large pre-trained models to uncover novel relations that are missing in the retrieved images. 2) To boost the features of individual instances, a novel Group Attention module is proposed, which is capable of collaboratively exploiting the attentive support from a group of instances to enhance the description power of individual instances. 3) To adaptively select the best set of instances for extracting commonsense knowledge while mitigating noisy instances, a Gamma-Corrected Gated Fusion module is designed to assign higher weights to instances containing more common relations and alleviate the influence of noisy instances. 4) Experimental results on the large benchmark dataset demonstrate substantial performance gains of the proposed method over state-of-the-art CKE models. 
 
 Appendix:  


Decision: The reviewers have come up to the following strengths and limitations of the paper

STRENGTHS
- Effectiveness of the Proposed Framework: Both extensive and ablation experiments underscore the framework's effectiveness.
- Cross-Modal Learning: Integration of visual and textual features through a cross-modal learning approach is emphasized.
- Innovative Modules and Models: Introduction of novel elements like the Group Attention module and the use of advanced models (ViT-GPT2 and VinVL) are highlighted.


LIMITATIONS
- Weak Technical Contribution and Unclear Main Contribution: The main contribution of the proposed framework is not well defined, and the technical contribution is considered weak.
- Lack of Comparative Experiments: The absence of comparative experiments with large language models is noted.
- Incomplete Ablation Studies: Ablation experiments are not comprehensive enough to fully understand the impact of individual modules.
- Complexity and Motivation: Concerns about the complexity of the model and confusion regarding certain processes and motivations.
- Unclear Purpose of Features: The necessity and performance impact of using multiple types of features are questioned.
- Limited Generalizability and Dataset Bias: Focus on a specific task (commonsense knowledge extraction from images) limits generalizability, and the primary evaluation dataset (Visual Genome) may introduce biases.
- Lack of Interpretability and Computational Cost Analysis: The interpretability of the framework's outputs and the computational cost are not adequately addressed.

Reviews: review: paper_summary: paper_summary: This paper proposes a multi-modal cross-domain feature learning framework aimed at improving the identification of unseen entity relations in the task of automatic commonsense knowledge extraction. Overall, the paper is well-motivated and well-written. The experimental results are promising.
strength_weakness: strength_weakness: The paper is well-motivated and well-written. The experimental results are promising.
limitations: limitations: Detailed comments:
1)In equation (1), why do you use Glove for word embedding? Is the word embedding consistent with the captions?
2)What is VinVL after Equation (3)?
paper_summary: paper_summary: This paper proposes a novel framework that consists of MCFL module, Group Attention module and Gamma-corrected Gated Fusion module for Commonsense Knowledge Extraction. The authors also conduct extensive experiments to evaluate the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1. The writing is good which makes readers easy to follow.
2. The extensive experiments are conducted to demonstrate the effectiveness of the proposed framework.
3. The ablation experiments show that each part in the proposed framework is effective.
limitations: limitations: 1. The technique contribution is weak.
2. Missing comparative experiments with large language models (LLMs).
3. Some ablation experiments are not comprehensive enough.

- Details:
1. What are the main differences between "Commonsense Knowledge Extraction" (CKE) in this paper and "Relation Classification in Multimodal Knowledge Graph Completion"? Both tasks supplement the relations between entity pairs. However, CKE mainly supplements location-related relations, while MKGC can supplement more complex relations, making it more meaningful.
2. The main contribution of the proposed framework is not clear. If it is about using large-scale pre-trained models to contain rich general domain knowledge, existing work [1] has already utilized PLMs to supplement relations between entity pairs. If it is about employing gate mechanism to filter noise images, the work RSME [2] for MKGC has also used gates for processing. If it is about the Group Attention Module, what is the main difference between it and self-attention?
3. In Group Attention Module, h_i will interact with all other h_j. What are the differences between it and directly performing self-attention on h_1 to h_n, and how do their performances differ?
4. The purpose of needing three types of features (VE, VC, VCE) is unclear since VCE theoretically already contains all the feature information. It is better for Table 3 to include the performance of using only VCE. Adding the two features, VE and VC, might introduce significant time consumption while providing only limited performance improvement.
5. LLMs (e.g., GPT-4) have demonstrated excellent performance across multiple tasks. It is suggested to supplement experiments related to LLMs in Table 1 to better illustrate the effectiveness of the proposed framework.

[1] Chen X, Zhang N, Li L, et al. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion[C]//Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2022: 904-915.
[2] Wang M, Wang S, Yang H, et al. Is visual context really helpful for knowledge graph? A representation learning perspective[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 2735-2743.
paper_summary: paper_summary: The paper introduces a novel framework for visual-linguistic cross-domain feature learning aimed at extracting commonsense knowledge from images. The proposed method, Multi-modal Cross-domain Feature Learning with Group Attention and Gamma-corrected Gated Fusion (MCFL-GAGF), leverages large-scale pre-trained models to enrich the representation of entity relations in images. The methodology is evaluated against alternatives on several benchmarks, demonstrating competitive performance in commonsense knowledge extraction.
strength_weakness: strength_weakness: The paper effectively combines visual and textual features using a cross-modal learning approach that integrates insights from both domains, enhancing the model’s ability to interpret complex entity relations.

The introduction of a Group Attention module allows the model to consider the context provided by other instances, providing a more nuanced understanding of each instance.
limitations: limitations: I feel quite confused about the process of generating interaction matrix and co-attention with GPT, whose motivation needs more explanation.

The complexity of the model raises my concerns, especially with the integration of multiple advanced components like group attention and gated fusion.

Given the model's sophisticated architecture, the ablation study might be better explained with more combinations.
paper_summary: paper_summary: This paper proposes a Multi-modal Cross-domain Feature Learning framework with Group Attention and Gamma-corrected Gated Fusion for extracting commonsense knowledge from images. The framework integrates visual and linguistic features, leveraging a large vision-text foundation model, ViT-GPT2, to handle unseen relations and exploit complementary information from multiple sources. A Group Attention module enhances the features of individual instances by exploiting information from other instances in the same bag, while a Gamma-corrected Gated Fusion module optimally aggregates features to extract commonsense entity relations. Experimental results demonstrate the superiority of the proposed method over state-of-the-art models.
strength_weakness: strength_weakness: 1. The paper introduces a novel approach that integrates visual and linguistic features for commonsense knowledge extraction, addressing the limitations of existing methods.
2. The framework leverages advanced models like ViT-GPT2 and VinVL, showcasing a sophisticated approach to feature learning and fusion.
3. The paper provides extensive experimental results demonstrating the effectiveness of the proposed method, showcasing its superiority over existing models.
limitations: limitations: 1. While the paper provides an overview of the proposed method and its components, a more detailed analysis of the individual modules and their impact on performance could enhance the understanding of the approach.
2. The paper focuses on commonsense knowledge extraction from images, which limits its generalizability to other domains or tasks. Exploring the applicability of the proposed framework to broader contexts could strengthen its impact.
3. While the framework achieves superior performance, the interpretability of the extracted commonsense knowledge is not explicitly addressed. Understanding how the framework generates and summarizes commonsense relations could enhance its usability.
4. The computational cost of the proposed framework, especially in terms of memory and processing requirements, is not thoroughly analyzed. This aspect is crucial for practical deployment in resource-constrained environments.
5. The paper primarily evaluates the framework on the Visual Genome dataset, which may have inherent biases or limitations. A more comprehensive evaluation on diverse datasets could provide a better understanding of the framework's robustness and generalizability.