Venue: NAACL 2024

# SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal

Named Entity Recognition of Unseen Entities

Hyunjong Ok\({}^{1,3}\) , Taeho Kil\({}^{2}\)+, Sukmin Seo\({}^{2}\), Jaeho Lee\({}^{1}\)

\({}^{1}\) POSTECH \({}^{2}\) NAVER Cloud \({}^{3}\) HJ AILAB hyunjong.ok@gmail.com, taeho.kil@navercorp.com

 Work done as an intern at NAVER Cloud.To be corresponded with.

###### Abstract

Recent advances in named entity recognition (NER) have pushed the boundary of the task to incorporate visual signals, leading to many variants, including multi-modal NER (MNER) or grounded MNER (GMNER). A key challenge to these tasks is that the model should be able to generalize to the entities unseen during the training, and should be able to handle the training samples with noisy annotations. To address this obstacle, we propose SCANNER (Span CANdidate detection and recognition for NER), a model capable of effectively handling all three NER variants. SCANNER is a two-stage structure; we extract entity candidates in the first stage and use it as a query to get knowledge, effectively pulling knowledge from various sources. We can boost our performance by utilizing this entity-centric extracted knowledge to address unseen entities. Furthermore, to tackle the challenges arising from noisy annotations in NER datasets, we introduce a novel self-distillation method, enhancing the robustness and accuracy of our model in processing training data with inherent uncertainties. Our approach demonstrates competitive performance on the NER benchmark and surpasses existing methods on both MNER and GMNER benchmarks. Further analysis shows that the proposed distillation and knowledge utilization methods improve the performance of our model on various benchmarks.

## 1 Introduction

Named entity recognition (NER) is a fundamental task in natural language processing to identify textual spans that correspond to named entities in the given text, and classify them into pre-defined categories, such as persons, locations, and organizations Li et al. (2020). The extracted information can be utilized for various downstream tasks, including entity linking and relation extraction.

The rapid growth of the amount of multi-modal contents on social media platforms has given rise to the multi-modal variants of NER. The most prominent example is multi-modal NER MNER Zhang et al. (2018) , which extends traditional NER to identifying named entities in the text based on additional image input paired with the text (Fig. 0(b)). Another recent example is the grounded MNER GMNER Yu et al. (2023)); here, one additionally aims to predict the bounding boxes of named entities appearing in the given image (Fig. 0(c)).

A major challenge in NER, MNER, and GMNER tasks is the presence of unseen entities in the test datasets, which are not found in the training datasets. Traditional models often struggle with low performance on these unseen entities (see Table 1). To tackle this problem effectively, it is important to use knowledge about unseen entities in a way that boosts ability of the model to generalize

Figure 1: Illustrations of NER, MNER, and GMNER tasks. The NER task aims to identify named entities from the given text. MNER extends this task to utilize additional image informations. GMNER additionally requires the model to predict entity bounding boxes in the given image, if they are present.



and perform well across different types of data. In this paper, we introduce SCANNER, which stands for Span CANiddate detection and recognition for Named Entity Recognition. Our approach is designed to effectively use knowledge about unseen entities, addressing NER, MNER, and GMNER tasks with improved robustness. SCANNER adopts a two-stage structure, comprising a span candidate detection module and entity recognition module. The span candidate detection module identifies named entity candidates within sentences. Following this, the entity recognition module uses these candidates as queries to extract relevant knowledge from various sources, effectively recognizing the class of the entity candidate. As illustrated in Fig. 2, we were able to accurately identify 'Kroger' as an 'organization' by utilizing object knowledge. SCANNER effectively gathers and uses knowledge from various sources, boosting its performance in the challenging NER, MNER, and GMNER benchmarks. Notably, the GMNER challenge involves the intricate process of identifying entities and determining their bounding boxes within images. The architecture of SCANNER, leveraging its comprehensive knowledge, is effective in addressing the GMNER task. The effectiveness of SCANNER in the GMNER task is highlighted by establishing a new baseline that is over \(21\%\) higher than the previous standard, as measured by the F1 score. Additionally, we introduce the novel self-distillation method, called as Trust Your Teacher. The NER task faces challenges with noisy annotations Wang et al. (2019); Zhu and Li (2022), particularly at entity boundaries where exact span matching is crucial and ambiguity often leads to increased noise (see Table 2). Our distillation method, which softly utilizes both the prediction of the teacher model and ground truth (GT) logit, addresses the challenges of noisy annotations.

Our approach demonstrates competitive performance on NER and surpasses existing methods on both MNER and GMNER. Further analysis shows that the proposed distillation and knowledge utilization methods improve the performance of our model on various benchmarks.

The contributions of SCANNER are summarized in three key aspects:

* We propose a new distillation method that softly blends the predictions of the teacher model with ground truth annotations to enhance data quality and model training.
* We develop SCANNER, a two-stage structured model that effectively utilizes knowledge to improve performance, particularly in recognizing unseen entities.
* The SCANNER model shows competitive performance in NER benchmarks and demonstrates higher performance than existing methods in MNER and GMNER benchmarks.

## 2 Related work

Prior works on MNER typically operates by first extracting the NER-related features from the image, and then combining these features with text features to recognize name entities. Roughly, ex

\begin{table}
\begin{tabular}{l l c c} \hline Datasets & Methods & Seen entities & Unseen entities \\ \hline CoNLL2003 & BERT-base & 93.78 & 80.90 \\  & Ours (w/o Knowledge) & 96.29 & 89.68 \\ \hline Twitter-2015 & BERT-base & 79.81 & 57.81 \\  & Ours (w/o Knowledge) & 87.18 & 73.84 \\ \hline Twitter-2017 & BERT-base & 93.81 & 67.76 \\  & Ours (w/o Knowledge) & 95.68 & 82.96 \\ \hline \end{tabular}
\end{table}
Table 1: A comparison of test F1 scores for the named entities that have appeared at least once in the training dataset, versus the entities that have not appeared.

\begin{table}
\begin{tabular}{l l} \hline Text & Dataset \\ \hline [The [Oval]\({}_{\text{loss}}\)]\({}_{\text{loss}}\) & CoNLL2003 \\ \hline [The [World Cup]\({}_{\text{HSC}}\)]\({}_{\text{HSC}}\) & Twitter-2015 \\ \hline [Taste of [Toronto]\({}_{\text{LRC}}\)]\({}_{\text{HSC}}\) & Twitter-2015 \\ \hline [Mrs. [Brozik]\({}_{\text{PRR}}\)] & Twitter-2017 \\ \hline [[Robert Downey]\({}_{\text{PRR}}\) Jr]\({}_{\text{PRR}}\) & Twitter-2017 \\ \hline \end{tabular}
\end{table}
Table 2: Examples of gold annotation and potential alternatives. The gold annotations are marked in blue [*], whereas the alternative annotations are in red [*].

Figure 2: ‘Kroger’ is an unseen entity that is hard to recognize as an Organization or Location. By our knowledge base model, it brings to successful prediction.

 

isting works fall into two categories according to how they extract image features.

**Textual features.** Several works extract the textual metadata from the given image and utilize them as features for the subsequent NER task Wang et al. (2022); Ma et al. (2023); Li et al. (2023). For instance, ITA Wang et al. (2022) extracts object tags, image captions, and OCR results from the given image. Similarly, Li et al. (2023) also extracts image captions, but additionally utilizes large language model as an implicit knowledge source to further refine the features. MoRe Wang et al. (2022) takes a slightly different approach, using an image-based retrieval system to retrieve textual descriptions of the closest images in the database.

**Visual encoders.** Another line of work attempts to extract the image features using a visual encoder, such as pre-trained ResNets, ViTs, or CLIP vision encoder Wang et al. (2022); Zhang et al. (2023); Chen et al. (2023). The extracted features are then combined with the text features extracted from a separate text encoder, which often involves additional alignment via cross-modal attention Chen et al. (2022); Lu et al. (2022); Wang et al. (2022); Zhang et al. (2023); Chen et al. (2023). Notably, PromptMNER Wang et al. (2022) calculates the similarity between visual features and various text prompts to extract visual cues that are loosely related to the input text.

In this paper, we take a different path and extract the image features _conditioned_ on the information extracted from the given text. Up to our knowledge, even though it has related works in NER Wang et al. (2021, 2022); Tan et al. (2023), it is the first such attempt in the context of MNER, which is a more challenging task.

In addition, a new task has been introduced, which not only incorporates image inputs but also actively addresses the task of grounding entity locations within images Yu et al. (2023).

## 3 Method

In this section, we first introduce the architecture of the proposed method, which comprises the span candidate detection module and the named entity recognition module (Sec. 3.1). Then, we describe the named entity recognition module, which performs entity recognition and visual grounding in the image for each entity candidate (Sec. 3.2). Finally, we explain a novel distillation method, named Trust Your Teacher, which is designed to robustly train our model even in the presence of noisy dataset annotations (Sec. 3.3).

### SCANNER Architecture

The primary focus of this paper is to perform MNER using both knowledge extracted from within images and external knowledge, even for entities not encountered during training. To achieve this, as illustrated in Fig. 3, we propose a two-stage architecture, known for its efficiency in extracting and searching for knowledge from various sources. In the first stage, we extract named entity candidates, and in the second stage, we efficiently search and extract only knowledge relevant to these candidates. This acquired knowledge is then utilized for entity recognition.

**Stage 1: Span Candidate Detection Module.** In the first stage of SCANNER, the transformer encoder Liu et al. (2019) is employed to detect entity candidates from the input text. During this phase, we utilizes BIO Beginning, Inside, Outside) tagging to classify each token in the input text, determining whether it corresponds to the beginning, inside, or outside of an entity span. The classification process is guided by cross-entropy loss.

**Stage 2: Entity Recognition Module.** In Stage 2, SCANNER performs named entity recognition and visual grounding for each entity candidate detected in Stage 1. It utilizes each entity candidate as a query to extract and leverage the necessary knowledge for the tasks. During this process, SCANNER efficiently searches and extracts knowledge by focusing on the initially detected entity candidates rather than the entire input text. SCANNER utilizes both internal (image-based) and external (e.g., Wikipedia) knowl

Figure 3: The overall architecture of the proposed SCANNER method. The two-stage structure allows for efficient extraction and utilization of knowledge, as knowledge is extracted only for those entity candidates that were filtered through in stage 1.

 

MNER on unseen entities, not encountered in training. Detailed information about these modules will be provided in Section 3.2.

### Entity Recognition Module

For each entity candidate identified by the span candidate detection module, the entity recognition module processes a text prompt that includes both the entity candidate and associated knowledge. This knowledge, extracted from images and external knowledge sources, allows for performing MNER on unseen entities that were not encountered during training. Our methodology involves extracting this knowledge from a variety of sources, utilizing the identified entity candidates as the basis for the extraction process. Then, this module classifies the class of each entity candidate and performs grounding to determine which object in the image corresponds to the entity. A detailed illustration is shown in Fig. 4.

#### 3.2.1 Prompt construction with knowledge

The entity recognition module extracts and utilizes useful knowledge from various sources when constructing the text prompt corresponding to the input. The knowledge applied for constructing text prompts in our method includes the following.

**Wikipedia knowledge.** Initially, information is searched using the entity candidate as a query in external knowledge source, which is Wikipedia. This information can be valuable for classifying the type of entity for each candidate and, moreover, enables the model to classify unseen entities that were not encountered during training. As illustrated in Fig. 4, for entity candidates like 'Steve Kerr', it enhances entity recognition performance by providing valuable information for classification as an American basketball player and coach.

**Image caption.** To effectively utilize visual information, image captioning results are also used. We use the BLIP-2 Li et al. (2023) to extract synthetic captions for the whole image.

**Object knowledge.** In addition to global information about the image, object-level information is also beneficial for entity recognition. To achieve this, results obtained from the object detector are employed as knowledge. Initially, object classes are converted into text format and used as knowledge. Then, synthetic captions for each object region are also utilized in conjunction with class names. This information is structured as details corresponding to each object, along with a special token denoted as \([{\rm obj}]\), as shown in Fig. 4. Additionally, during this process, the visual-language similarity between each object and entity candidate is calculated, and objects are arranged in order of high similarity, which is then included in the text prompt. One of the problems with existing meth

Figure 4: An illustration of the entity recognition module (stage 2). Based on the entity candidates (extracted in stage 1), SCANNER utilizes various knowledge sources such as Wikipedia, image captioner, and object knowledge extractor. The knowledge collected from these sources are then processed by RoBERTa to give the final prediction.

 

ods for the MNER task is that the model sometimes references objects in the image that are irrelevant to the entity, leading to incorrect recognition. By arranging the object details in the text prompt according to the visual-language similarity order with the entity, our model can focus more on the object regions that are highly related to the entity. In this paper, CLIP (Radford et al., 2021) is employed for visual-language similarity, specifically calculating the similarity between the text representation of the entity candidate and the visual representation of each Region of Interest (RoI).

All such knowledge mentioned above is converted into a textual format and integrated with the text prompt for entity recognition and visual grounding.

The text prompt, structured to include entity candidates, the entire input text sentence, and extracted knowledge, is presented as _"The entity is [mask] for [entity] in this sentence. [original sentence] {Wikipedia} [image caption] [obj] [object 1] [obj] [object 2] ..."_.

#### 3.2.2 Encoder and Objective

The prompts constructed for each entity candidate are input into a transformer encoder model (Liu et al., 2019). For entity recognition, the output token representation of the [mask] token in the text prompt \(x_{i}\) for the \(i\)-th entity candidate is fed into a linear layer to predict the probability distribution \(\hat{y}_{i}\). Given the ground truth \(y\), the objective function is to minimize the cross-entropy loss between the predicted entity class distribution and the ground truth logit:

\[\mathcal{L}_{c}=-\sum_{i=1}^{N}y_{i}\log\hat{y}_{i}, \tag{1}\]

where \(N\) is the total number of the entity candidates.

Additionally, the visual grounding is performed by feeding the output token representation of the \(j\)-th [obj] token from the text prompt \(x_{i}\) into a linear layer. This is followed by a sigmoid function, which aids in predicting the overlap score \(\hat{o}_{ij}\) between the ground truth image region grounding entity candidate \(i\) and object \(j\). The objective function of visual grounding is calculated based on the binary cross-entropy loss between the overlap score and the ground truth Intersection over Union (IoU):

\[\begin{split}\mathcal{L}_{g}&=-\sum_{i=1}^{N}\sum_{ j}o_{ij}\log\hat{o}_{ij}\\ &\qquad+(1-o_{ij})\log(1-\hat{o}_{ij}),\end{split} \tag{2}\]

where \(o_{ij}\) is the ground truth IoU between the ground truth image region of the entity \(i\) and object region \(j\).

In training stage, we combine two losses as the final loss of our model:

\[\mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{g}, \tag{3}\]

where \(\lambda\) is the weighting coefficient, we set \(\lambda\) to 1 for the GMNER task and to 0 for the NER and MNER tasks in this paper.

### Trust Your Teacher

We introduce the novel self-distillation method, called as Trust Your Teacher (TYT). Our distillation method, which softly utilizes both the prediction of the teacher model and ground truth (GT) logit, addresses the challenges of noisy annotations. First, we train the teacher model using equation 1, and then train the final student model using both the predictions of the teacher model and the ground truth labels. The most significant feature of our proposed method is that it assesses the reliability of each sample by utilizing the prediction of the teacher model to determine if it is trustworthy or noisy. Based on this assessment, the method sets the weights between the model prediction and the gt label, which are then reflected in the loss calculation. The objective of the our proposed distillation method composes a cross-entropy loss with ground truth and Kullback-Leibler Divergence (KLD) loss with teacher predictions:

\[\begin{split}&\mathcal{L}_{\text{TYT}}=\sum_{i}a_{i}\,\mathcal{ L}_{\mathrm{CE}}\big{(}y_{i},S(x_{i},\theta_{S})\big{)}\\ &\quad+(1-a_{i})\mathcal{L}_{\mathrm{KLD}}\big{(}S(x_{i},\theta_ {S}),T(x_{i},\theta_{T})\big{)},\end{split} \tag{4}\]

where \(x_{i}\) is the input sample, \(\theta_{S}\) and \(\theta_{T}\) are the model parameters of the student and teacher, \(S\) and \(T\) are the prediction distributions of the student and teacher and \(a_{i}\) is a balancing factor proposed in this paper. In detail, \(a_{i}\) determines whether to trust the teacher model prediction or the ground truth, and it represents the prediction score of the teacher model for the ground truth class index, which is \(T(x_{i},\theta_{T})[y_{i}]\). This implies that since the teacher model is well-trained, if the score for the ground 

truth class is high, then the sample is considered reliable and more weight is given to the cross-entropy with the ground truth label. Conversely, if the score is low, the sample is assumed to be an unreliable, noisy sample, and more weight is placed on the KLD loss with the prediction of the teacher model, rather than the ground truth label.

To demonstrate the significant impact of our TYT approach, we have carried out some experiments. Fig. 5 illustrates our experiments on a text classification task in MNLI dataset. We extract about 30% of the train set for experimental efficiency and intentionally added label noise at rates of 10% and 20% to this subset. We then compare the performance of the model trained with our TYT method on the train set with added label noise against the baseline that does not use distillation. Fig. 5 indicates that using TYT demonstrates relatively robust performance under moderate noise conditions. Additionally, we compare our method with the conventional soft distillation methods that do not dynamically vary the \(a_{i}\) parameter in the entity detection task, stage 1 of MNER and GMNER. Table 3 shows that our method has better performance on MNER and GMNER benchmarks, and adaptively varying the \(a_{i}\) is more effective than keeping it fixed.

We apply the TYT to both stages 1 and 2. But in NER, we only use it in stage 1. The loss from the TYT is applied only to the classification loss and not to the loss for visual grounding.

## 4 Experiment

### Dataset

Our methodology's efficacy was assessed using widely used datasets for each task. We utilize CoNLL2003Tjong Kim Sang and De Meulder (2003) for NER, Twitter-2015Zhang et al. (2018) and Twitter-2017Lu et al. (2018) for MNER, and Twitter-GMNER Yu et al. (2023) for GMNER. Details are in appendix B.

### Experimental Setups

**Evaluation metrics.** To evaluate our method, we use Entity-wise F1, precision, and recall scores for NER and MNER tasks. For the GMNER task, there is an additional evaluation of the visual grounding. For instances, where the visual grounding is unforoundable, a prediction is correct if it is classified as 'None.' For others, correctness hinges on the IoU metric. A prediction is considered correct if the IoU score between the predicted visual region and the ground truth bounding boxes exceeds a threshold of 0.5. We use F1, precision, and recall scores, which are calculated based on the aggregate correctness across entity, type, and visual region predictions. Our primary focus is on the F1 score in line with numerous preceding studies.

**Implementation details.** Following most recent works, we implement our model utilizing RoBERTa-large in NER, XLM-RoBERtLarge Conneau et al. (2020) for MNER, GMNER both in stage 1 and stage 2. For the object detector, we use VinVL Zhang et al. (2021b) following the settings with ITA Wang et al. (2022). To address the requirements of visual-language similarity and image caption, we use each of them CLIP 1 and BLIP-2 2 models respectively. Detailed hyper-parameter settings are shown in appendix A. All experiments were done on a single GeForce RTX 4090 GPU or NVIDIA H100 GPU, and we report the average score from 5 runs with different random seeds for each setting.

Footnote 1: openai/clip-vit-large-patch14

Also we applied several minor methods to enhance performance. In the second stage, we incorporated a 'non-entity' la

Figure 5: Experiments of text classification task in MNLI datasets. ‘matched’ is in-domain, and ’ mismatched’ is out-domain.

\begin{table}
\begin{tabular}{l|c c c c c c c c c} \hline \hline Methods & \multicolumn{3}{c|}{Twitters-2015} & \multicolumn{3}{c|}{Twitter-2017} & \multicolumn{3}{c}{Twitter-GMNER} \\  & Pro. & Rec. & F1 & Pre. & Rec. & F1 & Pre. & Rec. & F1 \\ \hline Base & 83.28 & 87.68 & 85.43 & **90.96** & 93.23 & 92.08 & 86.60 & 87.59 & 87.09 \\ Half & 83.36 & 87.69 & 85.47 & 90.31 & 92.92 & 91.60 & **87.08** & 87.96 & 87.47 \\ Full & **83.63** & 87.27 & 85.63 & 90.53 & 92.95 & 91.97 & 86.90 & 87.80 & 87.35 \\ TYT & 83.59 & **87.98** & **85.73** & 90.94 & **93.28** & **92.09** & 86.82 & **88.37** & **87.59** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation study on the MNER dataset in first stage. ‘Half’ is when \(a_{i}\) is 0.5 and ‘Full’ is 0. In ‘TYT’, \(a_{i}\) is adjusted through the trust your teacher method.

 

where the model erroneously predicts entity candidates not present in the dataset. That allowed for more accurate handling of such cases. We augmented it with non-entity data by dividing the training set into four folds in stage 1 and validating each fold. Secondly, we employed adversarial weight perturbation (AWP) Wu et al. (2020) in stage 1 , which enhances the robustness and generalization capabilities of the model. We initiated AWP from an intermediate stage of our training process.

### Experimental results in various NER tasks

**Experimental results in NER.** To evaluate the effectiveness of our approach in NER, we primarily compared our model against the existing methods in Table 5. It shows that SCANNER exhibits a competitive performance compared to the existing NER methods.

**Experimental results in MNER.** In assessing the effectiveness of SCANNER in MNER, we conducted comparative analyses against various leading models in this task. The results, detailed in Table 4, reveal that our model achieves superior performance in Twitter-2015 and exhibits markedly impressive results in Twitter-2017. Notably, while PGIM shows outstanding performance on Twitter-2017, it utilizes large language models (LLM) like ChatGPT, which incurs API costs, a notable drawback. In contrast, our model does not rely on LLM knowledge, freeing it from such disadvantages and demonstrating better performance on Twitter-2015. Additionally, we conduct experience using the same LLM knowledge as PGIM, which is in appendix C.

**Experimental results in GMNER.** To show our effectiveness in GMNER, we make broad compara

\begin{table}
\begin{tabular}{l c c c} \hline \hline Methods & \multicolumn{3}{c}{**Twitter-GMNER**} \\  & Pre. & Rec. & F1 \\ \hline \multicolumn{3}{c}{Text} \\ \hline HBILSTM-CRF-None Liu et al. (2018) & 43.56 & 40.69 & 42.07 \\ BERT-None Devlin et al. (2019) & 42.18 & 43.76 & 42.96 \\ BERT-CRF-None & 42.73 & 44.88 & 43.78 \\ BARTNER-None Yan et al. (2021a) & 44.61 & 45.04 & 44.82 \\ \hline \multicolumn{3}{c}{Text+Image} \\ \hline GVATT-RCNN-EVG Luo et al. (2018) & 49.36 & 47.80 & 48.57 \\ UMT-RCNN-EVG Yu et al. (2020) & 49.16 & 51.48 & 50.29 \\ UMF-ViLM-EVG Yi et al. (2020) & 50.15 & 52.52 & 51.31 \\ UMCG-ViLM-EVG Yi et al. (2021a) & 51.62 & 51.72 & 51.67 \\ T-Vin-LV-EVG Yi et al. (2022) & 52.37 & 50.77 & 51.56 \\ BARTMENT-ViLM-EVG Yi et al. (2023) & 52.47 & 52.43 & 52.45 \\ H-Index Wei et al. (2023) & 56.16 & 56.67 & 56.41 \\
**Scannker** (Ours) & **68.34** & **68.71** & **68.52** \\  & \(\pm\) 0.73 & \(\pm\) 0.61 & \(\pm\) 0.67 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Experiment results on the Twitter-GMNER. The reported figures for the baseline models are taken from Yu et al. (2023).

\begin{table}
\begin{tabular}{l|r r r r r r r r r r r r r} \hline \hline \multirow{2}{*}{Methods} & \multicolumn{6}{c}{**Twitter-2015**} & \multicolumn{6}{c}{**Twitter-2017**} \\  & \multicolumn{6}{c}{**Single Type(F1)**} & \multicolumn{6}{c}{**Overall**} & \multicolumn{6}{c}{**Single Type(F1)**} & \multicolumn{6}{c}{**Overall**} \\  & PER & LOC & ORG & OTR. & Pre. & Rec. & F1 & PER & LOC & ORG & OTR. & Pre. & Rec. & F1 \\ \hline \multicolumn{10}{c}{Text} \\ \hline BERT-CRF\({}^{\dagger}\) & 85.37 & 81.82 & 63.26 & 44.13 & 75.56 & 73.88 & 74.71 & 90.66 & 84.89 & 83.71 & 66.86 & 86.10 & 83.85 & 84.96 \\ BERT-SPAN\({}^{\dagger}\) Yamada et al. (2020) & 85.35 & 81.88 & 62.06 & 43.23 & 75.52 & 73.83 & 74.76 & 90.84 & 85.55 & 81.99 & 69.77 & 85.68 & 84.60 & 85.14 \\ RoBERTa-SPAN\({}^{\dagger}\) Yamada et al. (2020) & 87.20 & 83.58 & 66.33 & 50.66 & 77.48 & 77.43 & 77.45 & 94.27 & 86.23 & 87.22 & 74.94 & 88.71 & 89.44 & 89.06 \\ \hline \multicolumn{10}{c}{vision+LML (w/ zero-shot)} \\ \hline Gemini-pro-vision & 73.12 & 65.53 & 35.80 & 20.27 & 48.24 & 64.88 & 55.34 & 84.36 & 71.65 & 61.24 & 22.02 & 64.02 & 69.90 & 66.83 \\ GPT4-V & 80.00 & 75.26 & 40.53 & 25.26 & 51.46 & 70.42 & 59.46 & 85.63 & 78.62 & 73.68 & 36.63 & 67.63 & 74.90 & 71.08 \\ \hline \multicolumn{10}{c}{Text+Image} \\ \hline UMT Yin et al. (2020) & 85.24 & 81.58 & 63.03 & 39.45 & 71.67 & 75.23 & 73.41 & 91.56 & 84.73 & 82.24 & 70.10 & 85.28 & 85.34 & 85.31 \\ UMZ Zhang et al. (2021) & 84.26 & 83.17 & 62.42 & 42.44 & 74.79 & 75.21 & 74.85 & 91.92 & 85.22 & 83.13 & 69.83 & 86.54 & 84.50 & 85.51 \\ MNEAR-OR Qi et al. (2023) & 85.68 & 81.42 & 63.42 & 45.13 & 77.76 & 22.31 & 79.44 & 93.41 & 86.02 & 84.64 & 71.83 & 88.57 & 85.96 & 87.25 \\ R-GCN Zhao et al. (2022) & 86.36 & 82.08 & 60.78 & 41.56 & 73.95 & 76.18 & 75.00 & 92.86 & 86.10 & 84.05 & 72.38 & 86.72 & 87.53 & 87.11 \\ ITA Wang et al. (2022b) & - & - & - & - & - & - & 78.03 & - & - & - & - & - & 89.75 \\ PromoMBNet Wang et al. (2022d) & - & - & - & - & 78.03 & 79.71 & 78.60 & - & - & - & - & 89.93 & 90.60 & 90.27 \\ CAT-NERR Wang et al. (2022e) & 88.04 & 84.70 & 68.04 & 52.33 & 78.75 & 78.69 & 78.72 & 94.61 & 88.40 & 88.14 & 80.50 & 90.27 & 90.67 & 90.47 \\ Moke Wang et al. (2022a) & - & - & - & - & 79.21 & - & - & - & - & - & - & - & - & - \\ POIM \({}^{\ddagger}\) Li et al. (2023b) & 88.34 & 84.22 & 70.15 & 52.34 & 79.21 & **79.45** & 79.33 & 96.46 & 89.89 & 89.03 & 79.62 & **90.86** & **92.01** & **91.43** \\
**SCANNER** (Ours) & 88.24 & 85.16 & 69.86 & 52.23 & **79.72** & 90.03 & **79.38** & 95.18 & 88.52 & 88.45 & 79.71 & 90.40 & 90.67 & 90.54 \\ \(\downarrow\) 0.27 \(\uparrow\) 0.22 \(\rightarrow\) 0.31 \(\downarrow\) 1.39 \(\downarrow\) 0.56 \(\downarrow\) 0.64 \(\downarrow\) 0.14 \(\downarrow\) 0.23 \(\rightarrow\) 0.26 \(\downarrow\) 0.66 \(\downarrow\) 2.98 & \(\uparrow\)0.19 \(\downarrow\) 0.53 \(\rightarrow\) 0.32 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Experiment results on the Twitter-2015 and Twitter-2017. The results for methods marked with \({}^{\dagger}\) are from Wang et al. (2022e). The methods marked with \({}^{\ddagger}\) denotes that they utilize LLMs (of ChatGPT scale) 

isons with all existing methods. Text-only models made to predict the visual groundings all 'None'. The Table 6 shows that our model achieves significant performance improvements over prior research and establishes a new powerful baseline for future GMNER studies.

### Ablation study

**Ablation study in MNER.** We conduct ablation experiments on the MNER task to evaluate the effectiveness of the proposed method. These results are shown in Table 7. We observe that removing the Trust Your Teacher method led to a decrease in performance. Our proposed distillation method effectively alleviates the dataset noise issue, making our model more robust to learning from noisy dataset. Additionally, to verify the effectiveness of the various types of knowledge used in our study, we compare the results with experiments where each type of knowledge was removed. We confirm that the object knowledge, Wikipedia knowledge, and image caption knowledge used in our paper all contribute to the performance improvement of the MNER task.

**Case study.** As shown in Fig. 6, all three types of knowledge can be utilized as useful information for named entity recognition. In the case of the first image, knowledge from Wikipedia such as "American retail company" and object knowledge containing the logo information of "Kroger" both help in predicting the "Kroger" entity as an organization. For the image on the bottom left, image caption and object knowledge aided in named entity recognition. Moreover, in the image on the bottom right, vision information like image caption and object knowledge led to incorrect entity recognition results, but it was corrected through external knowledge from Wikipedia. Thus, the three types of knowledge proposed in this paper complement each other, enabling accurate MNER performance.

**Effectiveness in unseen entity.** Table 8 shows the effectiveness of knowledge in unseen entities. As

\begin{table}
\begin{tabular}{c c c|c c c} \hline \multirow{2}{*}{Methods} & \multicolumn{3}{c|}{**Twitter-2015**} & \multicolumn{3}{c}{**Twitter-2017**} \\  & Pre. & Rec. & F1 & Pre. & Rec. & F1 \\ \hline SCANNER & 79.72 & 79.03 & 79.38 & 90.40 & 90.67 & 90.54 \\ - TYT & -0.26 & -0.17 & -0.21 & -0.24 & -0.11 & -0.18 \\ - OBK & +0.11 & -0.60 & -0.26 & -0.23 & -0.22 & -0.22 \\ - WKK & -1.12 & -0.14 & -0.64 & -0.51 & -0.44 & -0.48 \\ - ICK & -0.08 & -0.54 & -0.31 & -0.29 & -0.31 & -0.29 \\ \hline \end{tabular}
\end{table}
Table 7: Ablation studies on MNER datasets. ‘-TYT’ is without trust your teacher method. ‘-OBK’ is without object knowledge. ’-WKK’ is without Wikipedia knowledge. ‘-ICK’ is without image caption knowledge.

Figure 6: Visualization results showing how various types of knowledge are brought in and utilized differently to perform the MNER task. Knowledge highlighted in blue positively influences correct predictions.

 

SCANNER utilizes various knowledge in MNER, it greatly increases performance in unseen entities. In NER, lack of various knowledge causes there to be no image, which slightly improves the performance.

## 5 Conclusions

We introduce SCANNER, a novel approach for performing NER tasks by utilizing knowledge from various sources. To efficiently fetch diverse knowledge, SCANNER employs a two-stage structure, which detects entity candidates first, and performs named entity recognition and visual grounding on these candidates. Additionally, we propose the novel distillation method, which robustly trains the model against dataset noise, demonstrating superior performance in various NER benchmarks. We believe that our method can be easily extended to utilize knowledge from multiple sources that were not covered in this paper.

### Limitations

In this study, we extract knowledge from various sources and utilize it to perform MNER tasks. By leveraging several vision experts such as CLIP, and also fetching external knowledge, our method takes relatively longer inference time compared to approaches that do not use knowledge. However, the use of vision experts and knowledge is essential for a MNER model that functions well even with unseen entities, and we efficiently extract information through a two-stage structure.

## Ethics statement

All experimental results we provide in this paper is based on publicly available datasets and open-source models.

## Acknowledgments

This work was partly supported by Artificial Intelligence Industrial Convergence Cluster Development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City, and the National Research Foundation of Korea (NRF) grant (RS-2023-00213710, Neural Network Optimization with Minimal Optimization Costs), and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-01906, Artificial Intelligence Graduate School Program(POSTECH))

## References

* F. Chen, J. Liu, K. Ji, W. Ren, J. Wang, and J. Chen (2023)Learning implicit entity-object relations by bidirectional generative alignment for multimodal ncmm. In ACM MM, Cited by: SS4.
* X. Chen, N. Zhang, L. Li, S. Deng, C. Tan, C. Xu, F. Huang, L. Si, and H. Chen (2022)Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In SIGIR, Cited by: SS1, SS2.
* A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020)Unsupervised cross-lingual representation learning at scale. In ACL, Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In ACL, Cited by: SS1, SS2.
* M. Jia, L. Shen, X. Shen, L. Liao, M. Chen, X. He, Z. Chen, and J. Li (2023)MNER-QG: an end-to-end mrc framework for multimodal named entity recognition with query grounding. In AAAI, Cited by: SS1, SS2.
* J. Li, H. Fei, J. Liu, S. Wu, M. Zhang, C. Teng, D. Ji, and F. Li (2022)Unified named entity recognition as word-word relation classification. In AAAI, Cited by: SS1, SS2.
* J. Li, D. Li, S. Savarese, and S. Hoi (2023)BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597. External Links: Link Cited by: SS1, SS2.
* J. Li, H. Li, Z. Pan, and G. Pan (2023)Prompt ChatGPT in MNER: improved multimodal named entity recognition method based on auxiliary refining knowledge from chatgpt. arXiv preprint arXiv:2305.12212. Cited by: SS1, SS2.
* J. Li, A. Sun, J. Han, and C. Li (2020)A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering34. Cited by: SS1, SS2.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)RoBERTa: a robustly optimized BERT pre-training approach. arXiv preprint arXiv:1907.11692. Cited by: SS1, SS2.
* I. Loshchilov and . Hutter (2019)Decoupled weight decay regularization. In ICLR, Cited by: SS1, SS2.

\begin{table}
\begin{tabular}{l|c c c c} \hline \multirow{2}{*}{Datasets} & **w/o Knowledge** & **w/ Knowledge** \\  & Seen & Unseen & Seen & Unseen \\ \hline CoNLL2003 & 96.29 & 89.68 & 96.35 & 89.70 \\ Twitter-2015 & 87.18 & 73.84 & 87.50 & 75.45 \\ Twitter-2017 & 95.68 & 82.96 & 95.90 & 83.71 \\ \hline \end{tabular}
\end{table}
Table 8: The result comparing the test F1 scores in unseen entities of knowledge extracted and baseline.

 

D. Lu, L. Neves, V. Carvalho, N. Zhang, and H. Ji (2018)Visual attention model for name tagging in multimodal social media. In ACL, Cited by: SS4.
* J. Lu, D. Zhang, J. Zhang, and P. Zhang (2022)Flat multi-modal interaction transformer for named entity recognition. In COLING, Cited by: SS3.
* M. E. Peters, W. Ammar, C. Bhagavatula, and R. Power (2017)Semi-supervised sequence tagging with bidirectional language models. In ACL, Cited by: SS5.
* A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021)Learning transferable visual models from natural language supervision. In ICML, Cited by: SS6.
* R. R Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017)Grad-CAM: Visual explanations from deep networks via gradient-based localization. In ICCV, Cited by: SS1, SS2.
* Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang (2023a)DiffusionNER: Boundary diffusion for named entity recognition. In ACL, Cited by: SS1, SS2.
* Y. Shen, Z. Tan, S. Wu, W. Zhang, R. Zhang, Y. Xi, W. Lu, and Y. Zhuang (2023b)PromptNER: prompt locating and typing for named entity recognition. In ACL, Cited by: SS1, SS2.
* Z. Tan, S. Huang, Z. Jia, J. Cai, Y. Li, W. Lu, Y. Zhuang, K. Tu, P. Xie, F. Huang, and Y. Jiang (2023c)DAO-NLP at SemEval-2023 task 2: a unified retrieval-augmented system for multilingual named entity recognition. In SemEval, Cited by: SS1, SS2.
* E. F. Tjong Kim Sang and F. De Meulder (2003)Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In NAACL, Cited by: SS1, SS2.
* X. Wang, J. Cai, Y. Jiang, P. Xie, K. Tu, and W. Lu (2022a)Named entity and relation extraction with multi-modal retrieval. In EMNLP findings, Cited by: SS1, SS2.
* X. Wang, M. Gui, Y. Jiang, Z. Jia, N. Bach, T. Wang, Z. Huang, and K. Tu (2022b)ITA: image-text alignments for multi-modal named entity recognition. In NAACL-HLT, Cited by: SS1, SS2.
* X. Wang, Y. Jiang, N. Bach, T. Wang, Z. Huang, F. Huang, and K. Tu (2021)Improving named entity recognition by external context retrieving and cooperative learning. In ACL, Cited by: SS1, SS2.
* X. Wang, J. Tian, M. Gui, Z. Li, J. Ye, M. Yan, and Y. Xiao (2022d)PromptMNER: prompt-based entity-related visual clue extraction and integration for multimodal named entity recognition. In DASFAA, Cited by: SS1, SS2.
* X. Wang, J. Ye, Z. Li, L. Lu, J. Liu, and J. Han (2019)Crossweigh: Training named entity tagger from imperfect annotations. In EMNLP-IJCNLP, Cited by: SS1, SS2.
* D. Wu, S. Xia, and Y. Wang (2020)Adversarial weight perturbation helps robust generalization. In NeurIPS, Cited by: SS1, SS2.
* I. Yamada, A. Asai, H. Shindo, H. Takeda, and Y. Matsumoto (2020)LUKE: Deep contextualized entity representations with entity-aware self-attention. In EMNLP, Cited by: SS1, SS2.
* H. Yan, T. Gui, J. Dai, Q. Guo, Z. Zhang, and X. Qiu (2021a)A unified generative framework for variousner subtasks. In ACL, Cited by: SS1, SS2.
* H. Yan, T. Gui, J. Dai, Q. Guo, Z. Zhang, and X. Qiu (2021b)A unified generative framework for various NER subtasks. In ACL, Cited by: SS1, SS2.
* J. Yu, J. Jiang, L. Yang, and R. Xia (2020)Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In ACL, Cited by: SS1, SS2.
* J. Yu, Z. Li, J. Wang, and R. Xia (2023)Grounded multimodal named entity recognition on social media. In ACL, Cited by: SS1, SS2.
* D. Zhang, S. Wei, S. Li, H. Wu, Q. Zhu, and G. Zhou (2021a)Multi-modal graph fusion for named entity recognition with targeted visual guidance. AAAI, Cited by: SS1, SS2.
* P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao (2021b)PinVL: Revisiting visual representations in vision-language models. In CVPR, Cited by: SS1, SS2.
* Q. Zhang, J. Fu, X. Liu, and X. Huang (2018)Adaptive co-attention network for named entity recognition in tweets. In AAAI, Cited by: SS1, SS2.
* Xin Zhang, Jingling Yuan, Lin Li, and Jianquan Liu (2023)Reducing the bias of visual objects in multimodal named entity recognition. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 958-966. Cited by: SS1, SS2.
* F. Zhao, C. Li, Z. Wu, S. Xing, and X. Dai (2022)Learning from different text-image pairs: a relation-enhanced graph convolutional network for multimodal ner. In ACM MM, Cited by: SS1, SS2.
* E. Zhu and J. Li (2022)Boundary smoothing for named entity recognition. In ACL, Cited by: SS1, SS2.

 



## Appendix A Hyper-parameter settings

We conducted our experiments with hyper-parameter settings as outlined in the following Table 9 and Table 10, and we utilize AdamW Loshchilov and Hutter (2019) optimizer for all tasks. 'max objects' refers to the maximum number of object knowledge inputs. We performed a grid search for the learning rate within the range of \([5\times 10^{-6},1\times 10^{-5}]\). We tested batch sizes of 4, 8, and 16 to determine the optimal value, and we explored weight decay within a range of \([0.01,2]\).

## Appendix B Detailed dataset statistics

To demonstrate the superiority of our method for various NER tasks, we conduct experiments on a range of datasets. The overall dataset statistics are shown in Table 11, and each task description is in below.

**NER dataset.** CoNLL2003 Tjong Kim Sang De Meulder (2003), a dataset with four named entities: PER, LOC, ORG, and MISC. We follow the standard setting Peters et al. (2017); Yan et al. (2021b); Shen et al. (2023a): use both the train set and dev set for training and evaluate with the test set

**MNER dataset.** Twitter-2015 Zhang et al. (2018) and Twitter-2017 Lu et al. (2018); collected from social network service posts. Like CoNLL2003, it consists of the same four named entity types. We operate a train set for training and hyper-parameter tuning using a dev set and evaluate it with the test set.

**GMNER dataset.** Twitter-GMNER Yu et al. (2023), a dataset collected by extracting some of the data from Twitter-2015 and Twitter-2017, and employ bounding box annotation. We operate same validate strategy as MNER.

## Appendix C Compare with MoRe and PGIM

**Using LLM knowledge.** With the advancements in LLM, we conducted experiments using GPT knowledge in SCANNER instead of Wikipedia knowledge. We utilize the same knowledge used in PGIM, and the results are in Table 12. We can observe that while applying GPT knowledge in SCANNER increases time and API-related costs, it also enhances performance. Notably, despite the GPT knowledge being tailored for PGIM, its performance in SCANNER is superior.

**Time cost in retrieval knowledge.** We conduct time cost comparisons using retrieval knowledge. As shown in Table 13, our entity-centric approach for integrating Wikipedia information demonstrates a speed advantage over MoRe's BM25 and K-NN based information retrieval and PGIM's GPT-based knowledge creation. This speed benefit arises from our direct method of identifying entity candidates, which allows for immediate retrieval of relevant Wikipedia articles without any bottlenecks.

## Appendix D Ablation study in GMNER

We conduct additional ablation experiments on the Twitter-GMNER task to evaluate the effectiveness of the proposed method. Effects of each method are shown in Table 14, and ablation studies on the number of object tokens are shown in Table 15. These tables substantiate the efficacy of our pro

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Datasets & \multicolumn{4}{c}{**Stage 2**} \\ \cline{2-5}  & epochs & batch size & lr & weight decay \\ \hline ConvIL2003 & 20 & 8 & \(3\times 10^{-6}\) & 0.01 & - \\ Twitter-2015 & 5 & 8 & \(1\times 10^{-5}\) & 0.01 & 15 \\ Twitter-2017 & 7 & 8 & \(5\times 10^{-4}\) & 0.01 & 15 \\ Twitter-GMNER & 5 & 8 & \(5\times 10^{-4}\) & 2 & 18 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyper-parameter settings in Stage 2 were used in the experiments for NER, MNER, and GMNER.

\begin{table}
\begin{tabular}{l|r r r r r} \hline \hline \multirow{2}{*}{Methods} & \multicolumn{3}{c}{**Twitter-2015**} & \multicolumn{3}{c}{**Twitter-2017**} \\  & Pre. & Rec. & F1 & Pre. & Rec. & F1 \\ \hline MoRe Wang et al. (2022) & - & - & 79.21 & - & 90.67 \\ PGDI Li et al. (2023b) & 79.21 & 79.45 & 79.33 & **90.86** & 92.01 & **91.43** \\ SCANNER & **79.72** & 79.03 & 79.38 & 90.40 & 90.67 & 90.54 \\ SCANNER (w/ GPT knowledge) & 79.24 & **80.97** & **80.10** & 90.22 & **92.23** & 91.22 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Dataset statistics of NER, MNER, and GMNER benchmarks

\begin{table}
\begin{tabular}{l l c c c} \hline \hline \multirow{2}{*}{Datasets} & \multicolumn{4}{c}{**Stage 1**} \\ \cline{2-5}  & epochs & batch size & lr & weight decay \\ \hline CoNLL2003 & 5 & 8 & \(5\times 10^{-6}\) & 1 \\ Twitter-2015 & 10 & 4 & \(1\times 10^{-5}\) & 2 \\ Twitter-2017 & 10 & 8 & \(1\times 10^{-5}\) & 2 \\ Twitter-GMNER & 10 & 8 & \(1\times 10^{-5}\) & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Hyper-parameter settings in Stage 1 were used in the experiments for NER, MNER, and GMNER.



posed methodologies in the context of GMNER and show that it is optimal when the number of object tokens is 18.

**Effect of CLIP knowledge.** CLIP is a practical module extracting entities from an image. As visualized in Fig 7, CLIP utilizes its knowledge and attention to the entity location in the image and improves the model's capability in GMNER tasks.

\begin{table}
\begin{tabular}{l r r} \hline \hline Methods & Knowledge base & Sentences / Sec. \\ \hline MoRe & Wiki (Text) & 64.6 \\ MoRe & Wiki (Image) & 650.1 \\ PGIM & GPT & 0.92 \\ SCANNER (Ours) & Wiki (Text) & **1,300.09** \\ \hline \hline \end{tabular}
\end{table}
Table XIII: Throughput of knowledge extractor. Performance metrics for MoRe are sourced directly from the MoRe paper, while those for PGIM and SCANNER are obtained from our measurements on a Ryzen 7900 CPU. Bigger is faster. For each 100 sentences, PGIM paid $0.2 for using ChatGPT API.

\begin{table}
\begin{tabular}{l|r r r} \hline \hline \multirow{2}{*}{Object tokens} & \multicolumn{3}{c}{**Twitter-GMNER**} \\  & Pre. & Rec. & F1 \\ \hline
9 & 67.50 & 67.85 & 67.67 \\
12 & 68.06 & 68.43 & 68.24 \\
15 & 68.22 & 68.55 & 68.38 \\
18 & **68.34** & **68.71** & **68.52** \\
21 & **68.34** & 68.68 & 68.51 \\ \hline \hline \end{tabular}
\end{table}
Table XII: Ablation studies on the number of object tokens.

\begin{table}
\begin{tabular}{l l c c} \hline \hline \multirow{2}{*}{Methods} & \multicolumn{3}{c}{**Twitter-GMNER**} \\  & Pre. & Rec. & F1 \\ \hline SCANNER & 68.34 & 68.71 & 68.52 \\ - TYT & -0.46 & -0.44 & -0.45 \\ - SOC & -0.14 & -0.16 & -0.15 \\ - WKK & -0.11 & -0.17 & -0.14 \\ - ICK & -1.09 & -1.09 & -1.09 \\ \hline \hline \end{tabular}
\end{table}
Table XIX: Ablation studies on GMNER datasets. ‘-TYT’ is without trust your teacher method. ‘-SOC’ is without sorting objects by clip. ’-WKK’ is without Wikipedia knowledge.‘-ICK’ is without image caption knowledge.

 

Figure 7: A visualization of which location CLIP focuses on by using Grad-CAM (Selvaraju et al., 2017).