Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing
Authors: Ying Zhang
Wenbo Fan
Kehui Song
Yu Zhao
Xuhui Sui
Xiaojie Yuan
Venue: EMNLP 2023 Findings (2023)
Keywords: Fine-Grained Entity Typing
Multimodal Learning.

Abstract: Fine-grained entity typing (FGET) aims to assign appropriate fine-grained types to entity mentions within their context, which is an important foundational task in natural language processing. Previous approaches for FGET only utilized textual context information. However, in the form of short text, the contextual semantic information is often insufficient for FGET. In many real-world scenarios, text is often accompanied by images, and the visual context is valuable for FGET. To this end, we firstly propose a new task called multimodal fine-grained entity typing (MFGET). Then we construct a large-scale dataset for multimodal fine-grained entity typing called MFIGER based on FIGER. To fully leverage both textual and visual information, we propose a novel Multimodal Object-Level Visual Context Network (MOVCNet). MOVCNet can capture fine-grained semantic information by detecting objects in images, and effectively merge both textual and visual context. Experimental results demonstrate that our approach achieves superior classification performance compared to previous text-based approaches.

Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing
Ying Zhang, Wenbo Fan, Kehui Song, Yu Zhao, Xuhui Sui, Xiaojie Yuan College of Computer Science, VCIP, TMCC, TBI Center, Nankai University, China [yingzhang, yuanx J@nankai edu cn (fanwenbo,songkeeuu zhaoyu, suixuhui j Lankai.edu cn
Abstract
Fine-grained entity typing (FGET) aims to as- sign appropriate fine-grained types to entity mentions within their context, which is im- portant foundational task in natural language processing. Previous approaches for FGET only utilized textual context information. How- ever, in the form of short text, the contex- tual semantic information often insufficient for FGET. In many real-wort world scenarios, text is often accompanied by images, and the vi- sual context is valuable for FGET. To this end, we firstly propose new task called mul- timodal fine- fine-grained entity typing (MFGET). Then construct a large-scale dataset for multimodal fine teee-graied entity typing called MFIGER based on FIGER. To fully leverage both textual and visual information, pro- pose novel Multimodal Object-Level Visual Context Network (MOVCNet). MOVCNet capture fine-grained semantic information by detecting objects in images, and effectively merge both textual and visual context. Exper- imental results demonstrate that our approach achieves superior classification performance compared to previous approaches.
10
(a) Lionel Messi [Person, Athlete] won the cham- pionship of 2022 FIFA World Cup.
(b) Ronald Reagan [Per- son, Actor, Politician] au- ditioned for the movie The Philadelphia Story.
Figure 1: Two examples for Multimodal Entity Typing (MFGET). Entity mentions and their fine- grained types brackets are highlighted.
structure (Ling and Weld, 2012; Gillick et al., 2014; Choi et al., 2018). Without taking into ac- count labels' interdependencies, it's hard to classify entity mention in isolation due to the large label set. Some recent works attempt to address this issue by leveraging label structures or statistics (Lin and Ji, 2019; Chen et al., 2020; Liu et al., 2021). Most of the previous methods only focus on textual content. However, in some certain situations, text can not provide sufficient contextual information to serve as the basis for classification, making it difficult to directly determine the ground truth label of the entity mention.
1 Introduction
Fine-Grained Entity Typing (FGET) aims to clas- sify an entity mention with its context into one more fine-grained types. For example, given a sentence "Lionel Messi won the championship of 2022 FIFA World Cup", the mention "Lionel Messi" should be classified as Person as its coarse- grained type and Athlete as its fine-graime type. FGET serves many down-stream NLP applications, such as relation extraction (Liu et al., 2014) and entity linking (Onoe and Durrett, 2020; Sui et al., 2022), thus is the foundation for building knowl- edge graphs.
In many real-world scenarios, texts are often companied by images and the images also contain rich semantic information, which provides addi- tional help to the FGET task. The advantages incorporating visual contexts of images into textual contexts for FGET are summarized as follows:
One major challenge of FGET lies in its rich and fine-grained labels with some kind of hierarchi-
1. Visual context could enhance the indicative ability textual context. As shown in Fig- ure ((a), given the sentence "Lionel Messi won
"Corresponding author.

---

the championship of 2022 FIFA World Cup". we cannot accurately determine from the tex- tual context whether the mention "Lionel Messi" refers to a person or a sports team. However, with the help of visual context, we can accu- rately classify the ambiguous "Lionel Messi" as Person and Athlete based on the detected ob- jects in the image, c.g. sneakers and football.
text into fine-grained types.
We evaluate the proposed method MOVCNet on our constructed dataset MFIGER. Com- pared with previous baselines, our method can effectively improve the performance of fine- grained entity typing.
2 Related Work
2. Visual context provides complementary se- manties to the textual context. As shown in Figure 1(b), given the sentence "Ronald Rea- gan auditioned for the movie The Philadelphia Story", we can classify the mention "Ronald Reagan" as Person and Actor using the textual context "auditioned" and "movie". However, based on the visual context of objects like the American flag and suit, we could also infer its category as Politician. Together we infer all the ground truth labels of (Person, Actor, Politician).
2.1 Fine-Grained Entity Typing
Two major challenges of FGET have been exten- sively studied by researchers. One challenge is that distant supervision introduces significant amount of noise to FGET. Some researches divide the dataset into clean set and noisy set, and model them separately (Ren et al., 2016; Abhishek et al., 2017; Xu and Barbosa, 2018). Onoe and Durrett (2019) proposed a two-stage denoising method in- cluding filtering and relabeling function. Zhang et al. (2020) proposed probabilistic automatic relabeling method with pseudo-truth label distri- bution estimation. Pan et al. (2022) proposed method to correct and identify noisy labels. Pang et al. (2022) tried mitigate the effect of noise by feature clustering and loss correction.
Therefore, considering that visual context is helpful for FGET, try introduce images into FGET, proposing a new task called Multi- modal Fine-Grained Entity Typing (MFGET). In the mcanwhile, we construct an MFGET dataset with a corresponding image for each sentence. The images are derived from Wikidata.' To in- corporate visual information, we propose a multi- modal object-level visual context network MOVC- Net. MOVCNet can effectively extract local ob- ject features in the image that are relevant to the text. Through the object-based attention mecha- nism, MOVCNet can better fuse the text and image context and further improve the performance of fine-grained classification. Experimental results demonstrate the effectiveness of MOVCNet com- pared with previous approaches.
Another challenge in FGET label hierarchy and interdependency. Xu and Barbosa (2018) intro- duced hierarchical loss normalization to deal with type hierarchy. Lin and Ji (2019) proposed a hybrid classification model to utilize type hicrarchy. Chen et al. (2020) proposed novel model with multi- level learning-to- to-nnkk loss and coarse-to- fonf de- coder. Liu et al. (2021) proposed label reasoning network to capture extrinsic and intrinsic depen- dencies. Zuo et al. (2022) modeled type hierar- chy by hierarchical contrastive strategy. Moreover, some works attempted to model FGET in hyper- bolic space (López et al., 2019) or box space (Onoe et al., 2021) instead of traditional vector space. However, images often co-occur with text in many real-world scenarios, yet no one has inves- tigated the impact of images on FGET. Therefore, we introduce images to FGET and propose a new task called multimodal fine-grained entity typing. and then study the effect of images on FGET.
The main contributions of this work can be sum- marized as follows:
To the best of our knowledge, we are the first to propose the task called Multimodal Fine- Grained Entity Typing (MFGET). Based on widely used dataset FIGER for FGET, We construct a new dataset MFIGER for MFGET.
We propose a multimodal object-level vi- sual context network MOVCNet for MFGET, MOVCNet can effectively identify objects in images and fuse visual and textual context, aid- ing in classifying entity mention with its con-
2.2 Multimodal Information Extraction
Multimodal information extraction aims to extract structured knowledge from various modalities, in- cluding unstructured and semi-structured text, im- ages, videos, etc. There exists some tasks like multimodal named entity recognition( (Wang et al.,
1tttps: //wwwwwwwwwwwccco wi ki data.org

---

HCM
Known Type Classifier
Latent Type Classifier
Fusion
N6
Xn+1
An+2
depresentaaie
BERT
VinVL
Ficas]
Esione
Essessi
E [SSP1]
Fiionee
Rxtessat
Fwon
Egup
El Eisapt
RA
Ea
Ea
Ex
Ep
Fi
&
Es
F+
Fx
Fi
Exerd
Eesz
[CLS]
Lionel
[SEP]
Lionel
won
Cup
[SEP]
Figure 2: The overall architecture of our proposed model MOVCNet. "Representation denotes the text and visual representation. "Fusion" denotes text-guided multimodal fusion. "HCM" denotes hybrid classification model.
2022b; Chen et al., 2022; et al., 2022), multi- modal relation extraction(Zhao et al., 2023; Chen et al., 2022), and multimodal entity linking(Zhang et al., 2023; Wang et al., 2022c). Information ex- traction techniques that incorporate multimodality form the foundation for constructing multimodal knowledge bases, providing ample data support for applications such question-answering systems, information retrieval, and more.
precise characterization of the entities. Therefore, when applying them to MFGET, they cannot ac- curately classify entity mentions into their ground truth fine- fine-grined labels. To this end, we propose novel multimodal object-level visual context net- work MOVCNe for MFGET.
Methodolgy
In this section, we first introduce the definition of FGET and MFGET tasks. Next, we provide de- tailed explanation of how to construct the MFIGER dataset. Then, we elaborate on the implementation details of our proposed model MOVCNet in four parts. Figure 2 shows the comprehensive architec- ture of MOVCNet.
Multimodal named entity recognition (MNER) aims to detect named entities and determine their corresponding entity types based (sentence, image) pair (Moon al., 2018; Zhang et al., 2018: Lu et al., 2018). Multimodal relation ex- traction (MRE) aims to predict relations berween two named entities in a sentence with the help of images (Zheng et al., 2021a.b). Multimodal entity linking (MEL) aims to map an ambiguous mention in sentence to an entity in knowledge base with textual and visual information (Adjali et al., 2020; Wang et al., 2022a). Most previous multimodal information extraction works focus extracting better representations of both textual and visual modalities and designing better task-specific fusion models of two modalities.
3.1 Definition
Traditional fine-graines entity typing datasets con- sist of massive collection of (mention, context) tuples: D {(m,, ( c1). 02),... (m,, G.))- Given an instance (m, c), the FGET task aims predict its appropriate types y C T, where (t,, t2, ..... } is the pre-defined fine-grained type set and |T| is the number of candidate types. Different from FGET, multimodal fine-grained entity typing datasets consist of a massive collec- tion of (mention, context, image) tuples: D {m : C1, v1 (1).(mp) C22 u2).... (myy Gns 2n)). The additional image information helps models predict corresponding types more accurately.
However, existing multimodal information traction methods cannot be directly applied mul- timodal fine-grained entity typing. For example, existing MNER methods only consider coarse- grained labels like Person, Location and Organiza- tion, these coarse- -raiined labels can not provide a

---

3.2 Dataset Construction
MFGET are fine-grained with some kind of hier- archy, so is crucial to extract highly informative and nuanced features from images. The objects an image can be used to deduce the fine- grained type of an entity mention, e.g. a badminton racket implies that he is an athlete. So by extracting local object features in an image, we can improve the precision and effectiveness of MFGET.
FIGER was proposed by (Ling and Weld, 2012), which contains 113 types with a 2-level hierar- chy. Based on FIGER, we construct new dataset MFIGER for Multimodal Fine-Grained Entity Typing. The detailed procedure consists of the following four steps:
1. Based on a mapping file from Wikipedia titles to Freebase mids, we first retrieve the Free- base mid of each mention within its context.
Thus, we adopt the object detector of VinVL (Zhang al., 2021) as our visual encoder, which is a large pre-trained vision-language model and contains a large-scale object-attribute detec- tion model based on ResNeXt- C4 architecture. Given an image v, we extract top m local visual objects as follows:
2. Based on a mapping file from Freebase enti- ties to Wikidata entities, we get the Wikidata id of each entity mention within its context.
3. We get images from the Wikidata webpage ac- cording to the Wikidata id, ultimately obtain- ing one image corresponding to each entity mention within its context.
f = VinVL(D)) E Rmxdd
(2)
where m is the number of objects, d, 2048 is the dimension of object feature representation.
4.
The original size of FIGER is 2,010,563, and after the above three steps, 935,744 instances have images. We divide them into the training set, validation set, and test set by 7:1:2.
3.5 Text-Guided Multimodal Fusion
We use to effectively fuse textual and visual context infor- mation. To align the dimensions of the both textual and visual representation, we add a fully connected layer on the visual representation f as follows:
Attention (Vaswani et al., 2017)
MFIGER contains 102 types with 2-1evel hier- archy like FIGER. And the statistical information is presented in Scction 4.1.
p- - E Rudd
(3)
3.3 Text Representation
where W/ E RR xds is a trainable parameter. Specifically, we treat the textual representation t as the query, and the transformed visual represen- tation p the key and value. We get text-aware visual representation g as follows:
Following (Onoe et al., 2021; Ding et al., 2022; Pan al., 2022), we adopt BERT (Devlin et al., 2019) our text encoder, the input of BERT is the sen- tence represented as $ = [CLS] mention [SEP] con- text [SEP]. The output is X = [ro, #1. #n+2] including 3 special tokens, where x; E IRd, denotes the contextualized representation of the i-th word in the sentence, dy denotes the dimension of hid- den layer in BERT, and denotes the length of the input sentence. Finally, by taking the hidden vector at [CLS] token, we encode the whole sequence into a single vector t:
g MultiHcad(
= -Conca(heedd,, heads) Wo (4)
where h is the number of attention head, head, represents the output of the i-th attention head, and WO is the output transformation matrix. The out- put of each head head, can be calculated as fol- lows:
t = BERTTS: beort) E Rd4
(1)
where ghort is the parameter of BERT encoder, d3 = 768 is the dimension of BERT hidden state.
head, = Attention Qi, K,,
(5)
where Attention is the calculation function of atten- tion, and
3.4 Object-Level Visual Representation
For the visual representation, traditional CNNs like VGG (Simonyan and Zisserman, 2015) and ResNet (He et al., 2016) can effectively extract global features of an image. However, the types in
Q: = Qw
K; = KW. K
(6)
V, VWV i 1,..,

---

where W W w. are the transformation matrix of the i-th query. key and value respectively,
Coarse
#Fine
Train
Dev
Person
14
217.430
30,971
61,992
C2
Location
13
314,283
44,948
90,045
We get the final representation a, which is the concatenation of textual representation and text- aware visual object representation g.
Organization
12
118,016
16,820
33,607
C4
Art
4
19,444
2,727
5,414
C5
Event
6
27,140
3,899
7,778
C6
Building
6
27,370
3,889
7,802
Product
12,904
1,885
3,634
Concat(t,
Others
30
62,320
8,744
Total
8
95
655,022
93,574
187,148
3.6 Hybrid Classification Model
Table 1: MFIGER type statistics.
Following Lin and (2019), we use a hybrid type classification model consisting of two classifiers: known type classifier and latent type classifier.
Finally, we combine the above two classifiers using the following formula:
Our known type classifier trains a linear transfor- mation matrix 4 to independently predict each type without considering their interdependencies:
(12)
a
(8)
where ÿ. is the overall predicted probability for the i-th type of the two classifiers. À is scalar with an initial value of 0.1, and is dynamically adjusted during the training phase.
where of is the predicted probability for the i-th type, W c Rdax2d, and dn is the number of types.
We regard MFGET as a multi-label classifica- tion problem, so a multi-label training objective is needed. We optimize a multi-label binary cross- entropy-based objective:
To fully leverage the type interdependency and hierarchy, we use latent type classifier motivated by Principle Label Space Transformation (Tai and Lin, 2012). Based on the hypercube sparsity as- sumption, where 2dh is significantly larger than the size of the training set, Tai and Lin (2012) utilize Singular Value Decomposition (SVD) to reduce the dimensionality of high-ding digh-ddmensional type vec- tors by projecting them into lower-dimensional space. This projection allows us to uncover the underlying type correlations that go beyond first- order co-occurrence. The formula of SVD is as follows:
C= - yr log Us + 34) 34) ū:)
(13)
where y/ is set the value if the mention is classi- fied as the i-th type, N is the number of types. During the test phase, we make predictions for each type based on the probability ū > 0.5. If all probabilities are lower than 0.5, we select the type with the highest probability using arg max ÿt-
(9) where U E Rdaxd: DE EERaxd, E Rxd, and dn >> dj. The resulting low-dimensional space
Y Y = UEL
Experiments
In this section, compare proposed method with previous state-of-the-an approaches to vali- date the effectiveness of our model. We first in- troduce the dataset and the statistical information that we use. Then, we provide brief overview of the baseline models for comparison and our imple- mentation details. Finally, we present the overall results and analysis on the performance comparison between our model and others.
resembles the hidden concept space found in La- tent Semantic Analysis (Deerwester et al., 1990). Each row of the matrix L represents the latent rep-
resentation of a specific type vector. Subsequently, we predict the latent type representation from the feature vector:
I V'a
(10)
where Vlg E x2d; is trainable parameter. Us- ing a linear transformation matrix Wi, we recon- struct the type vector based on l:
4.1 Datasets
We evaluate our model on our own constructed mul- timodal dataset MFIGER, which comprises pairs of sentences with annotated entity mentions and their associated images. The detailed explanation
w - UEI
(11)
where 5 is the predicted probability for the i-th type, W E Rd Xd is a trainable parameter.
Note that an entity mention may have multiple types.

---

Total
Coarse
Fine
Model
Acc
Ma-F1
Mi-Fi
Acc
Ma-F1
Mi-F1
Acc
Ma-FI
Mi-F1
NFETC (2018)
54.23
85,53
79.21
82.04
93.25
89.26
57.12
78.69
69.78
Lin and (2019)
72.80
92.13
91.58
93.59
96.97
96.23
75.45
88.49
87.91
ML-L2R (2020)
47.99
84.19
79.44
84,43
93.69
90.88
52.43
75.06
68.20
Box (2021)
79.72
94.20
93.67
95.63
97.88
97.39
81.21
91.27
90.72
NFETC-FCLC (2022)
48.93
83.15
76.32
82.05
93.34
89,32
53.10
73.07
63.29
DenoiseFET (2022)
67.83
90.87
89.72
93.05
96.68
95.87
69.91
86.48
85.03
UMT (2020)
88.56
95.93
96,09
96.95
98.29
98.05
89,43
94.05
94.56
MAF (2022)
77.17
92.07
92.04
94.11
96.70
96.22
78.69
88.67
88.69
MOVCNet
90.99
96.80
96.92
97.54
98.65
98.44
91.67
95.27
95.70
Table 2: Overall results of different label granularity on MFIGER test set. "Coarse" represents the results on 8 coarse-grained types, "Fine" represents the results on fine-grained types. The best results are highlighted.
of the dataset construction process is in Section 3.2. We have summarized coarse-grained categories and 95 fine-grained categories. Table presents the type statistics of the dataset. We can see that train, dev, and test set share similar type distribution.
MAF (Xu et al., 2022) proposes matching and alignment framework to make text and image consistent.
4.3 Implementation Details
We adopt BERT- ((ERT-Base(cased) (Devlin al., 2019) as encoder, Adam optimizer (Kingma and Ba, 2015) with a learning rate of BERT at 5e-5. The training batch size is 32, the hidden size of BERT encoder is 768, and the dropout rate is 0.1. For VinVL (Zhang et al., 2021), we set the number of local visual objects = 10. For Multi-Has Heali At- tention, set the number of attention head h 4. Our experiments are conducted on NVIDIA RTX 2080 Ti GPUs, and all models are implemented using PyTorch. Our experimental code is available here
4.2 Baselines
For multimodal fine-grained entity typing task, compare our model with the following text-based and multimodal state-of-the-art approaches:
NFETC (Xu and Barbosa, 2018) tries to solve noisy label problems by variant of the cross- entropy loss function, and deals with type hi- erarchy by hicrarchical loss normalization.
Lin and Ji (Lin and Ji, 2019) presents two- step attention mechanism and hybrid classi- fication method utilize label co-occurrence.
Following previous work (Ling and Weld, 2012), we use strict accuracy(Acor macro-averaged FI Score(Ma-FI), and micro-averaged F1 score(Mi- Fl) to evaluate the performance of models.
ML-L2R (Chen et al., 2020) proposes a novel multi-level learning-to to-rank especially for hi- erarchical classification problems.
4.4 Overall Results
Box (Onoe et al., 2021) is the first to intro- duce box space FGET instead of traditional vector space.
Table 2 shows the overall results of different la- bel granularity all baselines and our method on our own constructed dataset MFIGER test set. We can clearly see that our proposed method MOVC- Net significantly outperforms previous methods, whether at the total level, at the coarse-grained level, or at the fine-grained level.
NFETC-FCLC (Pang et al., 2022) designs a feature-clustering method with loss correction on each cluster.
At the total level, get the evaluation sults of all baselines and our method on total of 102 labels, including coarse-grained labels and 95 fine- fine-grained labels. Compared with previous SOTA method UMT (Yu et al., 2020), MOVC- Net achieves 2.43% improvement in strict accuracy (from 88.56% to 90.99%), 0.87% improvement on
DenoiseFET (Pan et al., 2022) proposes a method to automatically correct and identify noisy labels.
UMT (Yu et al., 2020) designs a unified mul- timodal transformer with an entity span de- tection module which can better capture the intrinsic correlations between modalities.
Pht / thub. com/ com/Web- Wow//WebFFN//OVOCe FAN/HOVCNEt

---

Total
Coarse
Fine
Model
Acc
Ma-F1
Mi-F1
Ace
Ma-F1
Mi-F1
Acc
Ma-F1
Mi-F1
MOVCNet
90.99
96.80
96.92
97.54
98.65
98.44
91.67
95.27
95.70
w/o BERT 87.60
95.71
95.91
96.70
98.19
97.94
88.69
93.78
94.34
w/o object 89.98
96.06
96.29
97.11
98.27
98. 11
90.62
94.33
94.93
w/o attention
90.75
96.37
96.60
97.23
98.39
98.21
91.40
94.84
95.38
Table 3: Ablation study of different label granularity on MFIGER. "woo BERT" denotes replacing BERT with ELMo. "w/o object" denotes replacing object features with global image features from VGG16. "w/o attention" denotes replacing multi-head attention with average pooling.
macro-averaged Fl score (from 95.93% to 96.80%) 0.83% improvement on micro-averaged score (from 96.09% to 96.92%)
categories, ultimately improving the performance of fine-grained entity typing, including strict accu- racy, micro-averaged F1 score, and micro-averaged Fl score.
At the coarse-grained level, we get the evalua- tion results of all baselines and our method only on 8 goarse-grained labels. Compared with previous SOTA method UMT (Yu et al., 2020), MOVCNet achieves 0.59% improvement on strict accuracy (from 96.95% to 97. ),,54%), 0.36% improvement on macro-averaged Fl score (from 98.29% to 98.65%). 0.39% improvement micro-averaged FI score (from 98.05% to 98.44%). Besides, most base- line models perform well on coarse-grained entity typing, achieving an accuracy of over 80%, and some models even surpass 90%. This indicates that the coarse-grained entity typing task is relatively simple, as the coarse-graine types of entity men- tions can be accurately inferred from the contextual information contained in the text alone.
4.5 Ablation Study
To study the effects of different modules in model, design three variants of MOVCNet. Ta- ble 3 shows the results of the ablation study of different label granularity on our dataset MFIGER. Effect of BERT. We replace BERT with ELMo (Peters et al., 2018) as our sentence en- coder, (Lin and Ji, 2019) is the first to use ELMo to get contextualized word representations. Com- pared with ELMo, BERT brings 3.39% improve- ment on strict accuracy, 1.09% improvement on macro-averaged F1 score. This demonstrates that in the MFGET task, BERT more capable of ob- taining better text representations, which in turn facilitates the fusion with image representations. Effect of Object Feature. We replace the local object features detected by VinVL with global im- age features extracted by traditional Convolutional Neural Network (CNN) VGG16 (Simonyan and Zisserman, 2015). Compared with VGG16, the object feature brings 1.01 % improvement in strict accuracy, 0.74% improvement in macro-averaged FI score. VGG can extract global features from im- ages, but it may not capture certain details. On the other hand, VIn can effectively extract objects in the image, and these objects provide fine-grained visual context to aid fine-grained entity typing. Effect of Attention Mechanism. We replace Multi-Han Head Attention with the average representa- tion over top objects. Compared with average representation, Multi-Head Attention brings 0.24% improvement on strict accuracy, 0.43% improve- ment in macro-averaged Fl score. With the atten- tion mechanism, we can determine which objects are relevant to the entity mentions in the sentence, and thereby obtain image representations that are
At the fine-grained level, we get the evaluation results of all baselines and our method only on 95 fine-gra grained labels. Compared with the previ- ous SOTA method UMT (Yu et al., 2020), MOVC- Net achieves 2.24% improvement in strict accuracy (from 89.43% 91.67%), 1.22% improvement on macro-averaged FI score (from 94.05% to 95.27%), 1.14% improvement on micro-averaged Fl score (from 94.56% to 95.70%)
Compared to the coarse-grained level, our multi- modal model MOVCNet shows greater improve- ment at the fine-grained level. The fine-grained entity typing task is relatively complex, as it is diffi- cult to accurately infer the fine-grained category of an entity mention based solely on the textual con- text or a combination of textual context and global image fcatures. Our multimodal model introduces visual context from images. effectively leveraging the objects contained in the images and interact- ing with the textual context for fusion. Through this approach, the images can effectively assist in classifying the entity mentions into -rained

---

Lin and (2019)
UMT (2020)
Box (2021)
MOVCNet
100
90
(a) It is song by southern rock band Lynyrd Skynyrd [Person, Musician] released on its 1974 album.
(b) Justine Henin [Person, Athlete] won her first tourn-- ment since her comehack at the Tennis Grand.
70
60
Figure 3: Two examples of attention visualization. The more the color tends towards red, the higher the weight of attention.
50
Figure 4: Results across coarse-grained types respec- tively. C1-C8 refers to the 8 coarse-grained types de- scribed Table Y-axis refers to the strict classifica- tion accuracy (%), begins from 50%.
perceptually aligned with the text. This allows the most relevant objects assist in fine-grate fine-grained entity typing, improving the classification performance.
Based on the analysis of the above three variants, we can draw the following conclusions. In our model, the module that has the greatest positive impact multimodal fine-grained entity typing is BERT, followed by the detected objects, and finally the attention mechanism.
coarse-grained type.
Compared with previous baselines, our model achieves significant improvements in the strict clas- sification accuracy on 8 coarse-grained types. The gain in classification performance, from highest to lowest, is follows: Person (C1), Building (C6), Art (C4), Organization (C3), Product (C7), Others (C8), Location (C2), Event (C5).
4.6 Attention Visualization
To study the effectiveness of the attention mecha- nism in model MOVCNet, visualized the attention in Figure 3. figure 3(a), the ground truth type is Person and Musician. We can see that the guitars are given more attention, providing strong visual cues for the classification of Musician. In figure 3(b), the ground truth type is Person and Athlete. We can see that the tennis ball, the tennis given more attention,
We infer that the fine-grained types under the coarse granned type Person relatively com- plex, such as Actor, Musician and Politician, as it requires strong textual cues to indicate a person's s profession. Given an entity mention within a sen- tence with Person as its ground truth label, we can hardly classify it into specific fine-grained types di- rectly. Because the textual context alone is always limited, it may not contain sufficient fine- grained information assist in fine-grained classification. Besides, the global features of an image do not contain sufficient fine-grained object information. Similarly, for mentions belonging to Building or Art, is also hard to determine their fine-grained types from the textual context or global image fea- tures.
racket, and the sportswear providing useful visual context for the classification of Athlete. These two examples demonstrate that the Multi-Head Attention used in our model can ef- fectively extract visual contextual information from the images, and the visual cues are relevant to the entity mention in the sentence. Therefore, MOVC- Net can achieve better classification performance than previous models.
Under the circumstances, our model introduces object-level image information to FGET. MOVC- Net can extract the most relevant local objects in images and integrate information from both modal- ities effectively. So MOVCNet can provide impor- tant visual cues the textual context, e.g. guitar racket for Person, parking apron for Building, piano sheet music for Art. These objects are valuable for identifying the profession of a person, the type of a
4.7 Results on Different Categories
To further analyze the effect of images in fine- grained entity typing task, conduct compara- tive experiment the classification performance of our model and three baseline models. Figure shows the fine-grained classification results sum- marized across 8 coarse -grained types respectively, there are several fine- -rainer types under each

---

Although our model MOVCNet has achieved ex- cellent results, it should be noted that we have used a simple off-the-shelf object detection tool VLnVL. that can effectively extract the objects from the im- building or an art, so the classification performance of our model can significantly surpass that of the compared baseline models across coarse-grained types respectively.
Omar Adjali, Romaric Besançon, Olivier Ferret, Hervé Le Borgne, and Brigitte Grau. 2020. Mul- timodal entity linking for tweets. In Advances in Information Retrieval 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal. April 14-17, 2020, Proceedings, Part I, volume 12035 of Lecture Notes in Computer Science, pages 463-478. Springer.
5 Conclusion
this paper, we propose a new task called multi- modal fine-grained entity typing (MFGET). Based on FIGER, we construct a multimodal dataset MFIGER with both text and accompanying images for MFGET. We propose novel multimodal model MOVCNet to incorporate object-level visual con- text for FGET. Specifically, MOVCNet can capture relevant objects in images and merge visual and textual contexts effectively. Experimental results on MFIGER demonstrate that our proposed model achieves the best performance compared with com- petitive existing models.
Tongfei Chen, Yunmo Chen, and Benjamin Van Durme. 2020. Hierarchical entity typing via multi-level learn- ing to rank. In Proceedings of the 58th Aanual Meet- ing of the Association for Computational Linguisties, pages 8465 8475, Online. Association for Computa- tional Linguisties.
Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao. Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Good visual guidance make a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction. In Findings of the Association for Computational Lin- guistics; NAACL 2022, pages 1607-1618, Seattle, United States. Association for Computational Lin- guistics.
Limitations
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 87-96, Mclbourne, Australia. Association for Computational Linguistics.
ages and get their features. However, there may be better methods for object detection, or we can design a dedicated object detection method specifi- cally for multimodal fine-grained entity typing to better extract local object features that are relevant to the text. These areas can be further explored in future work.
Scott C. Deerwester, Susan T. Dumais, Thomas Lan- dauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci., 41(6):391-407
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota. Association for Computational Linguistics.
Acknowledgements
This research is supported by the National Natu- ral Science Foundation of China (No. 62272250), the Natural Science Foundation of Tianjin, China (No. 22JCIQIC00150, 22JCQNJC01580), the Fundamental Research Funds for the Central Uni- versities (No. 63231 149), Tianjin Research In- novation Project for Postgraduate Students (No. 20225KY7332).
Ning Ding. Yulin Chen, Xu Han. Guangwei Xu, Xi- aobin Wang, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li, and Hong- Gee Kim. 2022. Prompt- learning for fine-gre mma-grained entity typing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6888-690 Abu Dhabi, United Arab Emirates. Association for Computational Lin- guisties.
References
Abhishek Abhishek, Ashish Anand, and Amit Awekar. 2017. Fine-grained entity type classification by jointly learning representations and label embeddings. In Proceedings of the 15th Conference of the Euro-
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context- dependent fine-grained entity type tagging. CoRR, abs/1412.1820.
pean Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. pages 797-807, Valencia, Spain. Association for Computational Lin- guistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Viston

---

and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Yasumasa Onoe, Michael Boratko, Andrew McCallum. and Greg Durrett. 2021. Modeling fine- aine-grined entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume I: Long Papers), pages 2051-2064, Online. As- sociation for Computational Linguistios.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Reprexentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2407-20 -417-2417. Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Ying Lin and Heng Ji. 2019. An attentive fine- tre-grained entity typing model with latent type representation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natural Lan- guage Processing (EMNI.P-IJCNTP)) pages 6197- 6202, Hong Kong, China. Association for Computa- tional Linguistics.
Yasumasa Onoe and Greg Durrett. 2020. Fine-grained entity typing for domain independent entity linking. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8576-88 8583, AAAI Press.
Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In Proceedings of the Twenry- Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada. AAAI Press.
Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. 2021. Fine-grained entity typing via label reasoning.
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4611-4622, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Weiran Pan, Wei Wei, and Feida 7hu. 2022. Automatic noisy label correction for fine-grained entity typing. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 43 4317-4323 ijcai.org.
Yang Liu, Kang Liu, Liheng Xu, and Jun Zhao 2014, Exploring fine-grained entity type constraints for dis- tantly supervised relation extraction. In COLING 2014, 25th International Conference on Computa- tional Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ire- land, pages 2107-2116. ACL.
Kunyuan Pang, Haoyu Zhang, Jie Zhou, and Ting Wang. 2022. Divide and denoise: Learning from noisy la- bels in fine-grained entity typing with cluster-wise loss correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 1997-2006, Dublin, Ireland. Association for Computational Lin- guistics.
Federico López, Benjamin Heinzerling, and Michael Strube. 2019. Fine-graines entity typing in hyper- bolic space. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANL.-20)) pages
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Florence, Italy, Association for Com- putational Linguistics.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang. and Heng Ji. 2018. Visual attention model for name tagging multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1990-1999, Melbourne, Australia. Association for Computational Linguistics.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016. AFET: Automatic fine- grained entity typing by hierarchical partial-label em- bedding. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Process- ing, pages 1669,1378, Austin, Texas. Association for Computational Linguistics.
Seungwhan Moon, Leonardo Neves, and Vitor Carvalho, 2018. Multimodal named entity recognition for short social media posts. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long Papers), pages 852-860, New Orleans, Louisiana. Association for Computational Linguistics.
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on

---

Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings.
Orleans, Louisiana, Association for Computational Linguistics.
Jianfei Yu, Jing Jiang. Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal trans- former. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 134223352, Online. Association for Computational Linguistics.
Xuhui Sui, Ying Zhang, Kehui Song, Baohang Zhou, Guoqing Zhao, Xin Wei, and Xiaojie Yuan. 2022. Im- proving zero-shot entity linking candidate generation with ultra-fine entity type information. In Proceed- ings of the 29th International Conference on Compu- tational Linguistics, COLING 2022, Gyeongju. Re- public of Korea, October 12-17, 2022, pages 2429- 2437. International Committee on Computational Linguistics.
Gongrui Zhang, Chenghuan Jiang. Zhongheng Guan, and Peng Wang. 2023, Multimodal entity linking with mixed fusion mechanism. In Database Systems for Advanced Applications 28th International Con- ference, DASFAA 2023, Tianjin, China, April 17-20, 2023, Proceedings, Part III, volume 13945 of Lecture Notes in Computer Science, pages 607-622. Springer.
Farbound Tai and Hsuan-Tien Lin. 2012. Multilabel classification with principal label space transforma- tion. Neural Comput., (4(9):2508-254)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 5998-6008, 6008.
Haoyu Zhang, Dingkun Long, Guangwei Xu, Muhua Zhu, Pengjun Xie. Fei Huang. and Ji Wang. 2020. Learning with noise: Improving distamly-supervised fine-grained entity typing via automatic relabeling. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3808-3815, 15. ijcai.org.
Peng Wang, Jiangheng Wu, and Xiaohang Chen. 2022a. Multimodal entity linking with gated hierarchical fu- sion and contrastive training. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022, pages 938-948. ACM.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. Vinvl: Revisiting visual representa- tions in vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579-5588. Computer Vision Foundation IEEE.
Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, and Kewei Tu. 2022b. ITA: Image-text alignments for multi- modal named entity recognition. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3176-3189 Seattle, United States. Association for Computational Linguisties.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named en- tity recognition in tweets. In Proceedings of the AAAI conference on artificial intelligence, volume 32,
Qihui Zhao, Tianhan Gao, and Nan Guo. 2023. TSVEN: two-stage visual fusion network for multimodal rela- tion extraction. Inf. Process. Manag., 1033):003264
Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Rui Wang, Ming Yan, Lihan Chen, and Yanghua Xiao. 2022c. WikiDiverse: A multimodal entity linking dataset with diversified contextual topics and entity types. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 4785-4797, Dublin, Ireland. Association for Computational Linguistics.
Changmeng Zheng, Junhao Feng, Ze Fu, Yi Cai, Qing Li, and Tao Wang. 2021a. Multimodal relation ex- traction with efficient graph alignment. In MM '21: ACM Multimedia Conference, Virtual Event, China, October 20- 24, 2021, pages 5298 5306. ACM.
Changmeng Zheng, Zhiwei Wu, Junhao Feng. Ze Fu, and Yi Cai. 2021b. MNRE: A challenge multimodal dataset for neural relation extraction with visual evi- dence in social media posts. In 2021 IEEE Interna- tional Conference Multimedia and Expo, ICME 2021, Shenzhen, China, July 5-9, 2021, pages 1-6. IEEE
Bo Xu, Shizhou Huang, Chaofeng Sha, and Hongya Wang. 2022. MAF: A general matching and align- ment framework for multimodal named entity recog- nition. In WSDM '22: The Fifteensh ACM Interna- tional Conference on Web Search and Data Mining. Virtual Event / Tempe, A7, USA, February 21- 25, 2022, pages 1215-1223. ACM.
Xinyu Zuo, Haijin Liang, Ning Jing, Shuang Zeng- Zhou Fang, and Yu Luo. 2022. Type-enriched hi- erarchical contrastive strategy for fine- fine-graieed entity typing. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 2405-2417. International Committee on Computational Linguisties.
Peng Xu and Denilson Barbosa. 2018 Neural fine- grained entity type classification with hierarchy- aware loss. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long Papers), pages 16-25, New


Reviews:
[{"review_id": "1FpAMtSrOL", "reviewer": "Reviewer_MZdU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and sincerely appreciate your feedback. We would like to address the specific questions below.\n\n1.Missing ablation study on the latent type classifier\n\nLin and Ji (2019) exploits type interdependency with latent type representation and improves classification performance significantly, so we adopt the latent type classifier following Lin and Ji (2019). Our main contribution is to introduce visual information into fine-grained entity typing and the latent type classifier is not the technical innovation of our model. Therefore, in the ablation study, we are more focused on our contribution including object information and object-level attention, rather than the latent type classifier.\n\n2.Missing comparison experiments of baselines\n\nAs shown in Table 3, \"w/o BERT\" denotes introducing VinVL features into Lin and Ji (2019). The performance of \"w/o BERT\" is lower than our model MOVCNet, but higher than Lin and Ji (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and is easy to follow.\n- The attention visualization enhances the interpretability of the proposed model.", "reasons_to_reject": "- Missing ablation study on the latent type classifier. Does the latent type classifier help to improve the classification accuracy? \n- Missing some comparison experiments of baselines. For example, for other text-only methods, can VinVL features be introduced into text features through attention? I think such a comparison is more fair.", "questions_to_authors": "", "paper_topic_and_main_contributions": "- This paper propose a novel task called Multimodal Fine-Grained Entity Typing (MFGET), and construct a new dataset MFIGER for MFGET task.\n- This paper propose a novel MFGET method called MOVCNet, which can identify objects in images and fuse visual and textual context, aiding in classifying entity mention with its context into fine-grained types.\n- Compared with previous text-only baselines, MOVCNet achieves better performances in the proposed MFIGER dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2upMjTBqJU", "reviewer": "Reviewer_mhGN", "summary": "", "questions": "Question A: Recent years there have been a few multimodal models such as OpenFlamingo [1] or LLaVA [2]. I wonder what might be the zero shot performance of those model on coarse level, e.g. simply asking the model which one of the label does the entity belong to. However I understand that running those models might take a lot of compute so my score will not be affected by the results.\n\n[1]: https://github.com/mlfoundations/open_flamingo\n\n[2]: https://arxiv.org/abs/2304.08485", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive suggestions and sincerely appreciate your positive feedback. We would like to address the specific questions below.\n\n1.Zero-shot performance of the multimodal models on coarse level\n\nLarge multimodal models require a lot of compute, and they focus more on unsupervised and zero-shot scenarios. Our model focuses on supervised scenarios, and will get better performance than those multimodal models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The first to purpose multimodal fine-grained entity typing task\n- 11.27%, 1.91%, and 10.46% strict accuracy improvement over baselines on total level, coarse level and fine-grained level on MFIGER\n- MFIGER datasets can be a useful benchmark for future research and is beneficial for the community.\n- Motivation is clear and I like the careful model design", "reasons_to_reject": "No such thing, I believe this is a sound paper.", "questions_to_authors": "Question A: Recent years there have been a few multimodal models such as OpenFlamingo [1] or LLaVA [2]. I wonder what might be the zero shot performance of those model on coarse level, e.g. simply asking the model which one of the label does the entity belong to. However I understand that running those models might take a lot of compute so my score will not be affected by the results.\n\n[1]: https://github.com/mlfoundations/open_flamingo\n\n[2]: https://arxiv.org/abs/2304.08485", "paper_topic_and_main_contributions": "- The first work to present this multimodal fine-grained entity typing which predicts entity types with the aid of visual context\n- Curate MFIGER, dataset on this task with 95 fine-grained categories\n- Purpose MOVCNet that leverages object-level attention mechanism\n- Strong performance gain on MFIGER compared to baselines", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "wF5kcuEPNc", "reviewer": "Reviewer_JYT1", "summary": "", "questions": "A. For section 3.2 dataset construction. What's the quality of the collected images? Is there any method to evaluate your images quality? Does there exist any noise for the images?\nB. Line 302, is the attention calculation following self-attention mechanism, where q is from text and k and v are from images?\nC. Line 356-357, could you please explain more about how you dynamically adjust the parameter \\lambda? what's the final value for \\lambda?\nD. For your experiments, all baselines are unimodal. In other words, they only use textual information. There are lots of existing multimodal fusion models, I would like to see how your model compare with those multimodal fusion models. You could refer to the missing references, or even some simpliest fusion techniques such as concatenation, circulant fusion, etc. If there is no such comparison, the paper only proves that adding object-level visual information works better than only using textual information. In general, if the assumption is you have high quality inputs (additional information such as images), it always help improve the performance (regardless of efficiency). In this way, it is hard to get the contribution of your proposed multimodal object-level visual context network. Because in your ablation study, using your attention-based fusion method, the performance improves less than 1 percentage. The most performance contribution is provided by adding additional information for the inputs.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and sincerely appreciate your feedback. We would like to address the specific questions below.\n\n1.Commonly used object information fusion and the hybrid classification layer\n\n(1) For multi-modal fusion, although fusing object information by cross-modal attention layers is commonly used for multi-modal information extraction, no one has introduced object information into fine-grained entity typing task. Besides, entity mentions are required to be classified into many fine-grained types, detected objects in images will provide strong visual cues for fine-grained classification. (2)For hybrid classification layer, Lin and Ji (2019) exploits type interdependency with latent type representation and improves classification performance significantly. So we adopt the latent type classifier following Lin and Ji (2019). However, our main contribution is to introduce visual information into fine-grained entity typing and propose a new task. So we are more focused on the effect of visual information and the fusion of two modalities, rather than the hybrid type classifier.\n\n2.Experiment on only one dataset\n\nWe are the first to propose the new task, and the dataset construction process is complex. So we only conduct experiments on our own constructed dataset.\n\n3.Dataset quality and noise\n\nWe get images from Wikidata webpages, so the image quality is relatively high, while inevitably there will be some noise. We can evaluate image quality by calculating the similarity between images and texts.\n\n4.Attention calculation\n\nYes, we use multi-head attention, and q is from text and k and v are from images.\n\n5.Parameter lambda\n\nLambda is a nn.Parameter(), it's fed into the optimizer and updates with training. The final value is 0.1474.\n\n6.Missing multi-modal baselines for performance comparison\n\nOur main contribution is to introduce images into fine-grained entity typing. As shown in Table 2 and Table 3, the classification performance of our model is better than others, this indicates the effectiveness of our model. As shown in Table 3 in “4.5 Ablation Study”, “w/o object” denotes concatenating textual and VGG16 visual representation, “w/o attention” denotes concatenating textual and average pooling object visual representation, they are both simple concatenation. Besides, we have implemented some representative multi-modal baselines like UMT and MAF, the performance of our model is better than them. This further validates the effectiveness of our model. We will include this part of comparison experiment in the revised version.\n\n| Model | Total_Acc | Total_Ma-F1 | Total_Mi-F1 | Coarse_Acc | Coarse_Ma-F1 | Coarse_Mi-F1 | Fine_Acc | Fine_Ma-F1 | Fine_Mi-F1 |\n\n| UMT | 88.56    |     95.93        |      96.09      |      96.95       |        98.29         |          98.05      |    89.43    |       94.05     |       94.56    |\n\n| MAF   |    77.17    |     92.07        |      92.04      |      94.11       |        96.70         |          96.22      |    78.69    |       88.67     |       88.69    |\n\n| MOVCNet  |    90.99    |     96.80        |      96.92      |      97.54       |        98.65         |          98.44      |    91.67    |       95.27     |       95.70    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new dataset is constructed for multimodal fine-grained entity typing task and their proposed model shows that adding object-level visual information can improve the performance a lot. The attention visualization helps better understand how object-level visual information works.", "reasons_to_reject": "The main concerns of this paper are:\n1. Fusing object information by cross-modal attention layeres are commonly used for multi-modal information extraction. Please refer to missing references section. And the hybrid classification layer is also following previous works. \n2. The experiment is only conducted on one dataset. Need to implement more multi-modal baselines for performance comparison. Please see details in question D.", "questions_to_authors": "A. For section 3.2 dataset construction. What's the quality of the collected images? Is there any method to evaluate your images quality? Does there exist any noise for the images?\nB. Line 302, is the attention calculation following self-attention mechanism, where q is from text and k and v are from images?\nC. Line 356-357, could you please explain more about how you dynamically adjust the parameter \\lambda? what's the final value for \\lambda?\nD. For your experiments, all baselines are unimodal. In other words, they only use textual information. There are lots of existing multimodal fusion models, I would like to see how your model compare with those multimodal fusion models. You could refer to the missing references, or even some simpliest fusion techniques such as concatenation, circulant fusion, etc. If there is no such comparison, the paper only proves that adding object-level visual information works better than only using textual information. In general, if the assumption is you have high quality inputs (additional information such as images), it always help improve the performance (regardless of efficiency). In this way, it is hard to get the contribution of your proposed multimodal object-level visual context network. Because in your ablation study, using your attention-based fusion method, the performance improves less than 1 percentage. The most performance contribution is provided by adding additional information for the inputs.\n", "paper_topic_and_main_contributions": "This paper first construct a new dataset for multimodal fine-grained entity typing based on FIGER. They propose a multimodal object-level visual context network, which fuses textual information and object-level visual information through attention layers, to capture fine-grained semantic information. Experiments compared with several existing textual-level models show that fusing object-level visual information can help improve the performance.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Chen, Xiang, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. \"Good visual guidance makes a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction.\"NAACL 2022.\n2. Bo Xu, Shizhou Huang, Chaofeng Sha, and Hongya Wang. 2022. MAF: A General Matching and Alignment Framework for Multimodal Named Entity Recognition. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM '22). Association for Computing Machinery, New York, NY, USA, 1215–1223. https://doi.org/10.1145/3488560.3498475\n3. C. Zheng, Z. Wu, T. Wang, Y. Cai and Q. Li, \"Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning,\" in IEEE Transactions on Multimedia, vol. 23, pp. 2520-2532, 2021, doi: 10.1109/TMM.2020.3013398.\n4. Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020. Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts. In Proceedings of the 28th ACM International Conference on Multimedia (MM '20). Association for Computing Machinery, New York, NY, USA, 1038–1046. https://doi.org/10.1145/3394171.3413650\n"}}]

Author rebuttal:

Meta-review: Summary:
This paper begins by creating a new dataset for multimodal fine-grained entity typing, building upon FIGER. The authors introduce a multimodal object-level visual context network, which employs attention layers to fuse textual information and object-level visual data, facilitating the capture of fine-grained semantic information. Comparative experiments with several existing textual-level models demonstrate that the integration of object-level visual information enhances performance.

Reason To Accept:
1.A new dataset has been created for the multimodal fine-grained entity typing task, and the proposed model demonstrates a significant improvement in performance through the incorporation of object-level visual information. The MFIGER dataset holds promise as a valuable benchmark for future research and is a beneficial contribution to the community.
2.The motivation behind this work is clear. It marks the first attempt to introduce the multimodal fine-grained entity typing task, and the paper is well-written and easy to follow. 
3.The reported results showcase substantial improvements: 11.27% improvement in total level, 1.91% in coarse level, and 10.46% in fine-grained level strict accuracy over baselines on MFIGER. Additionally, the attention visualization enhances the interpretability of the proposed model.

Reason To Reject:
1.The use of cross-modal attention layers for fusing object information is a common practice in multi-modal information extraction. It's advisable to reference relevant literature in the missing references section. Additionally, the hybrid classification layer appears to align with previous work in the field.
2.Several comparison experiments with baselines are lacking. 1)  For other text-only methods, it would be valuable to assess whether VinVL features can enhance text features through attention, ensuring a fairer comparison. 2) The experiments are conducted on a single dataset, and it's advisable to implement more multi-modal baselines for a comprehensive performance comparison.
3.Missing ablation study on the latent type classifier. It is necessary to investigate whether the inclusion of the latent type classifier contributes to an improvement in classification accuracy.
Decision: Accept-Findings