Venue: NAACL 2018

# Multimodal Named Entity Recognition for Short Social Media Posts

Seungwhan Moon1,2, Leonardo Neves2, Vitor Carvalho3

###### Abstract

We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapach captions, which comprise short text with accompanying images. These social media posts often come in inconsistent or incomplete syntax and lexical notations with very limited surrounding textual contexts, bringing significant challenges for NER. To this end, we create a new dataset for MNER called SnapCaptions (Snapchat image-caption pairs submitted to public and crowd-sourced stories with fully annotated named entities). We then build upon the state-of-the-art Bi-LSTM word/character based NER models with 1) a deep image network which incorporates relevant visual context to augment textual information, and 2) a generic _modality-attention_ module which learns to attenuate irrelevant modalities while amplifying the most informative ones to extract contexts from, adaptive to each sample and token. The proposed MNER model with modality attention significantly outperforms the state-of-the-art text-only NER models by successfully leveraging provided visual contexts, opening up potential applications of MNER on myriads of social media platforms.

languageresourceLanguage Technologies Institute, Carnegie Mellon University

\({}^{2}\) Snap Research

\({}^{3}\) Intuit seungwhm@cs.cmu.edu, lneves@snap.com, vitor_carvalho@intuit.com

## 1 Introduction

Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individuals. These insights are primarily hidden in unstructured forms of social media posts, such as in free-form text or images without tags. Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a critical step for building structural information, allowing for its use in personalized assistance, recommendations, advertisement, etc.

While many previous approaches [9, 46, 50] can be used to evaluate the impact of various methods on the topic of interest in social media posts. In addition, the authors have shown success for well-formed text in recognizing named entities via word context resolution (_e.g._ LSTM with word embeddings) combined with character-level features (_e.g._ CharLSTM/CNN), several additional challenges remain for recognizing named entities from extremely short and coarse text found in social media posts. For instance, short social media posts often do not provide enough textual contexts to resolve polysemous entities (_e.g._ "monopoly is da best", where 'monopoly' may refer to a board game (named entity) or a term in economics). In addition, noisy text includes a huge number of unknown tokens due to inconsistent lexical notations and frequent mentions of various newly trending entities (_e.g._ "xoxo Marshallloooo", where 'Marshmellooooo' is a mis-spelling of a known entity 'Marshmello', a

Figure 1: **Multimodal NER + modality attention.** (a) Visual contexts help recognizing polysemous entity names (‘Monopoly’ as in a board game versus an economics term). (b) Modality attention successfully suppresses word embeddings of a unknown token ( ‘Marshellocooo’ with erroneously trailing ’o’s), and focuses on character-based context (_e.g._ capitalized first letter, and lexical similarity to a known named entity ( ‘Marshello’, a music producer)) for correct prediction.



music producer), making word embeddings based neural networks NER models vulnerable.

To address the challenges above for social media posts, we build upon the state-of-the-art neural architecture for NER with the following two novel approaches (Figure 1). First, we propose to leverage auxiliary modalities for additional context resolution of entities. For example, many popular social media platforms now provide ways to compose a post in multiple modalities - specifically image and text (_e.g. Snapchat_ captions, Twitter posts with image URLs), from which we can obtain additional context for understanding posts. While "monopoly" in the previous example is ambiguous in its textual form, an accompanying snap image of a board game can help disambiguate among polysemous entities, thereby correctly recognizing it as a named entity.

Second, we also propose a general modality attention module which chooses per decoding step the most informative modality among available ones (in our case, word embeddings, character embeddings, or visual features) to extract context from. For example, the modality attention module lets the decoder attenuate the word-level signals for unknown word tokens (_e.g._ "Marshmellooooo" with trailing 'o's) and amplifies character-level features instead (_e.g._ capitalized first letter, lexical similarity to other known named entity token 'Marshmello', etc.), thereby suppressing noise information ("UNK" token embedding) in decoding steps. Note that most of the previous literature in NER or other NLP tasks combine word and character-level information with naive concatenation, which is vulnerable to noisy social media posts. When an auxiliary image is available, the modality attention module determines to amplify this visual context _e.g._ in disambiguating polysemous entities, or to attenuate visual contexts when they are irrelevant to target named entities, _e.g._ selfies, etc. Note that the proposed modality attention module is distinct from how attention is used in other sequence-to-sequence literature (_e.g._ attending to a specific token within an input sequence). Section 2 provides the detailed literature review.

**Our contributions** are three-fold: we propose (1) an LSTM-CNN hybrid multimodal NER network that takes as input both image and text for recognition of a named entity in text input. To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks. (2) We propose a general _modality attention_ module that selectively chooses modalities to extract primary context from, maximizing information gain and suppressing irrelevant contexts from each modality (we treat words, characters, and images as separate modalities). (3) We show that the proposed approaches outperform the state-of-the-art NER models (**both** with and without using additional visual contexts) on our new MNER dataset _SnapCaptions_, a large collection of informal and extremely short social media posts paired with unique images.

## 2 Related Work

**Neural models for NER** have been recently proposed, producing state-of-the-art performance on standard NER tasks. For example, some of the end-to-end NER systems Passos et al. (2014); Chiu and Nichols (2015); Huang et al. (2015); Lample et al. (2016); Ma and Hovy (2016) use a recurrent neural network usually with a CRF Lafferty et al. (2001); McCallum and Li (2003) for sequence labeling, accompanied with feature extractors for words and characters (CNN, LSTMs, etc.), and achieve the state-of-the-art performance mostly without any use of gazetteers information. Note that most of these work aggregate textual contexts via concatenation of word embeddings and character embeddings. Recently, several work have addressed the NER task specifically on noisy short text segments such as Tweets, etc. Baldwin et al. (2015); Aguilar et al. (2017). They report performance gains from leveraging external sources of information such as lexical information (_e.g._ POS tags, etc.) and/or from several preprocessing steps (_e.g._ token substitution, etc.). Our model builds upon these state-of-the-art neural models for NER tasks, and improves the model in two critical ways: (1) incorporation of visual contexts to provide auxiliary information for short media posts, and (2) addition of the modality attention module, which better incorporates word embeddings and character embeddings, especially when there are many missing tokens in the given word embedding matrix. Note that we do not explore the use of gazetteers information or other auxiliary information (POS tags, etc.) Ratinov and Roth (2009) as it is not the focus of our study.

**Attention** modules are widely applied in several deep learning tasks Xu et al. (2015); Chan 

et al., 2015; Sukhbaatar et al., 2015; Yao et al., 2015). For example, they use an attention module to attend to a subset within a single input (a part/region of an image, a specific token in an input sequence of tokens, etc.) at each decoding step in an encoder-decoder framework for image captioning tasks, etc. Rei et al. (2016) explore various attention mechanisms in NLP tasks, but do not incorporate visual components or investigate the impact of such models on noisy social media data. Moon and Carbonell (2017) propose to use attention for a subset of discrete source samples in transfer learning settings. Our modality attention differs from the previous approaches in that we attenuate or amplifies each modality input as a whole among multiple available modalities, and that we use the attention mechanism essentially to map heterogeneous modalities in a single joint embedding space. Our approach also allows for re-use of the same model for predicting labels even when some of the modalities are missing in input, as other modalities would still preserve the same semantics in the embeddings space.

**Multimodal learning** is studied in various domains and applications, aimed at building a joint model that extracts contextual information from multiple modalities (views) of parallel datasets.

The most relevant task to our multimodal NER system is the task of multimodal machine translation Elliott et al. (2015); Specia et al. (2016), which aims at building a better machine translation system by taking as input a sentence in a source language as well as a corresponding image. Several standard sequence-to-sequence architectures are explored (_e.g_. a target-language LSTM decoder that takes as input an image first).

Other previous literature include study of Canonical Correlation Analysis (CCA) Dhillon et al. (2011) to learn feature correlations among multiple modalities, which is widely used in many applications. Other applications include image captioning Xu et al. (2015), audio-visual recognition Moon et al. (2015), visual question answering systems Antol et al. (2015), etc.

To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks.

## 3 Proposed Methods

Figure 2 illustrates the proposed multimodal NER (MNER) model. First, we obtain word embeddings, character embeddings, and visual features (Section 3.1). A Bi-LSTM-CRF model then takes as input a sequence of tokens, each of which comprises a word token, a character sequence, and an image, in their respective representation (Section 3.2). At each decoding step, representations from each modality are combined via the modality attention module to produce an entity label for each token (3.3). We formulate each component of the model in the following subsections.

**Notations**: Let \({\bf x}=\{{\bf x}_{t}\}^{T}_{t=1}\) a sequence of input tokens with length \(T\), with a corresponding label sequence \({\bf y}=\{{\bf y}_{t}\}^{T}_{t=1}\) indicating named entities (_e.g_. in standard BIO formats). Each input token is composed of three modalities: \({\bf x}_{t}=\{{\bf x}^{(w)}_{t},{\bf x}^{(c)}_{t},{\bf x}^{(v)}_{t}\}\) for word embeddings, character embeddings, and visual embeddings representations, respectively.

### Features

Similar to the state-of-the-art NER approaches Lample et al. (2016); Ma and Hovy (2016); Aguilar et al. (2017); Passos et al. (2014); Chiu and Nichols (2015); Huang et al. (2015), we use both word embeddings and character embeddings.

**Word embeddings** are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics Mikolov et al. (2013). Specifically, we use pre-trained embeddings from GloVE Pennington et al. (2014).

Figure 2: The main architecture for our multimodal NER (MNER) network with modality attention. At each decoding step, word embeddings, character embeddings, and visual features are merged with modality attention. Bi-LSTM/CRF takes as input each token and produces an entity label.

 



**Character embeddings** are obtained from a Bi-LSTM which takes as input a sequence of characters of each token, similarly to (Lample et al., 2016). An alternative approach for obtaining character embeddings is using a convolutional neural network as in (Ma and Hovy, 2016), but we find that Bi-LSTM representation of characters yields empirically better results in our experiments.

**Visual embeddings**: To extract features from an image, we take the final hidden layer representation of a modified version of the convolutional network model called Inception (GoogLeNet) (Szegedy et al., 2014, 2015) trained on the ImageNet dataset (Russakovsky et al., 2015) to classify multiple objects in the scene. Our implementation of the Inception model has deep 22 layers, training of which is made possible via "network in network" principles and several dimension reduction techniques to improve computing resource utilization. The final layer representation encodes discriminative information describing what objects are shown in an image, which provide auxiliary contexts for understanding textual tokens and entities in accompanying captions.

Incorporating this visual information onto the traditional NER system is an open challenge, and multiple approaches can be considered. For instance, one may provide visual contexts only as an initial input to decoder as in some encoder-decoder image captioning systems (Vinyals et al., 2015). However, we empirically observe that an NER decoder which takes as input the visual embeddings at every decoding step (Section 3.2), combined with the modality attention module (Section 3.3), yields better results.

Lastly, we add a transform layer for each feature _e.g._\({\bf x}^{(w)}_{t},{\bf x}^{(c)}_{t},{\bf x}^{(v)}_{t}:=\sigma_{w}({\bf x}^{( w)}_{t}),\sigma_{c}({\bf x}^{(c)}_{t}),\sigma_{v}({\bf x}^{(v)}_{t})\) before it is fed to the NER entity LSTM.

### Bi-LSTM + CRF for Multimodal NER

Our MNER model is built on a Bi-LSTM and CRF hybrid model. We use the following implementation for the entity Bi-LSTM.

\[\begin{split}&{\bf i}_{t}=\sigma({\bf W}_{xi}{\bf h}_{t-1}+{\bf W }_{ci}{\bf c}_{t-1})\\ &{\bf c}_{t}=(1-{\bf i}_{t})\odot{\bf c}_{t-1}\\ &+{\bf i}_{t}\odot\text{tanh}({\bf W}_{xc}{\overline{{\bf x}}}_{ t}+{\bf W}_{hc}{\bf h}_{t-1})\\ &{\bf o}_{t}=\sigma({\bf W}_{zo}{\overline{{\bf x}}}_{t}+{\bf W} _{ho}{\bf h}_{t-1}+{\bf W}_{co}{\bf c}_{t})\\ &{\bf h}_{t}=\text{LSTM}({\overline{{\bf x}}}_{t})\\ &={\bf o}_{t}\odot\text{tanh}({\bf c}_{t})\end{split} \tag{1}\]

where \({\overline{{\bf x}}}_{t}\) is a weighted average of three modalities \({\bf x}_{t}=\{{\bf x}^{(w)}_{t};{\bf x}^{(c)}_{t};{\bf x}^{(v)}_{t}\}\) via the modality attention module, which will be defined in Section 3.3. Bias terms for gates are omitted here for simplicity of notation.

We then obtain bi-directional entity token representations \(\overleftrightarrow{{\bf h}_{t}}=[{\overrightarrow{{\bf h}_{t}}};{ \overleftarrow{{\bf h}_{t}}}]\) by concatenating its left and right context representations. To enforce structural correlations between labels in sequence decoding, \(\overleftrightarrow{{\bf h}_{t}}\) is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective.

\[{\bf y}^{*}=\operatornamewithlimits{argmax}_{\mathbf{y}}p(\mathbf{y}| \overleftrightarrow{{\bf h}};{\bf W}_{\text{CRF}}) \tag{2}\]

\[p(\mathbf{y}|\overleftrightarrow{{\bf h}};{\bf W}_{\text{CRF}})=\frac{\prod_{ t}\psi_{t}({\bf y}_{t-1},{\bf y}_{t};\overleftrightarrow{{\bf h}})}{\sum_{ \mathbf{y}^{\prime}}\prod_{t}\psi_{t}({\bf y^{\prime}}_{t-1},{\bf y^{\prime}}_ {t};\overleftrightarrow{{\bf h}})}\]

where \(\psi_{t}({\bf y}^{\prime},{\bf y}^{\prime};\overleftrightarrow{{\bf h}})\) is a potential function, \({\bf W}_{\text{CRF}}\) is a set of parameters that defines the potential functions and weight vectors for label pairs \(({\bf y}^{\prime},{\bf y}^{\prime})\). Bias terms are omitted for brevity of formulation.

The model can be trained via log-likelihood maximization for the training set \(\{({\bf x}_{i},{\bf y}_{i})\}\):

\[\mathcal{L}({\bf W}_{\text{CRF}})=\sum_{i}\log p(\mathbf{y}|\overleftrightarrow {{\bf h}};{\bf W}) \tag{3}\]

### Modality Attention

The modality attention module learns a unified representation space for multiple available modalities (_e.g._ words, characters, images, etc.), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance. We motivate this module from the following observations.

A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, _e.g._\({\bf h}_{t}=\text{LSTM}([{\bf x}^{(w)}_{t};{\bf x}^{(c)}_{t}])\) in Eq.1. However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (_e.g._ an all-zero vector \({\bf x}^{(w)}_{t}={\bf 0}\) or a random vector \({\bf x}^{(w)}_{t}=\boldsymbol{\epsilon}\sim U(-\sigma,+\sigma)\) is assigned for any unknown token \({\bf x}_{t}\), thus \({\bf h}_{t}=\text{LSTM}([{\bf 0};{\bf x}^{(c)}_{t}])\) or \(\text{LSTM}([{\bf\epsilon};{\bf x}^{(c)}_{t}])\)). While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post

datasets which contain a significant number of missing tokens.

Similarly, naive merging of textual and visual information (_e.g._\({\mathbf{h}}_{t}=\text{LSTM}([{\mathbf{x}}^{(w)}_{t};{\mathbf{x}}^{(c)}_{t};{ \mathbf{x}}^{(v)}_{t}])\)) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities. Hence, ideally there needs a mechanism in which the model can effectively turn the _switch_ on and off the modalities adaptive to each sample.

To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step \(t\), and produces a soft-attended context vector \(\overline{\mathbf{x}_{t}}\) as an input token for the entity LSTM.

\[\begin{split}&[{\mathbf{a}}^{(w)}_{t},{\mathbf{a}}^{(c)}_{t},{ \mathbf{a}}^{(v)}_{t}]=\sigma\big{(}{\mathbf{W}}_{m}\cdot[{\mathbf{x}}^{(w)}_{ t};{\mathbf{x}}^{(c)}_{t};{\mathbf{x}}^{(v)}_{t}]+{\mathbf{b}}_{m}\big{)}\\ &\alpha^{(m)}_{t}=\frac{\exp({\mathbf{a}}^{(m)}_{t})}{\sum\limits \limits_{m^{\prime}\in\{w,c,v\}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \! 

* Bi-LSTM/CRF + Bi-CharLSTM (W+C) [16]: takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM. Entity LSTM takes concatenated vectors of word and character embeddings as input tokens.
* Bi-LSTM/CRF + CharCNN (W+C) [17]: uses character embeddings extracted from a CNN instead.
* Bi-LSTM/CRF + CharCNN (W+C) + Multitask [2]: trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks.
* **(proposed)** Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings.
* **(proposed)** Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors.
* **(proposed)** Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM.

### Results: SnapCaptions Dataset

Table 1 shows the NER performance on the _Snap Captions_ dataset. We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results.

**Parameters**: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, **150**, 200, 300}, word embeddings size: {25, 50, 100, **150**, 200, 300}, LSTM hidden states: {25, 50, **100**, 150, 200, 300}, and \(\overline{x}\) dimension: {25, 50, 100, **150**, 200, 300}. We optimize the parameters with Adagrad [4] with batch size 10, learning rate 0.02, epsilon \(10^{-8}\), and decay 0.0.

**Main Results**: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks. In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V). This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain. Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines [29, 17, 16] that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well.

**Error Analysis**: Table 2 shows example cases where incorporation of visual contexts affects prediction of named entities. For example, the token 'curry' in the caption _"The curry's_ \(\raisebox{-0.86pt}{\scalebox{0.8}{$\stackrel{{\mbox{\tiny{$ \Box$}}}}{{=}}$}}\)" is polysemous and may refer to either a type of food or a famous basketball player 'Stephen Curry', and the surrounding textual contexts do not provide

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \multirow{2}{*}{Modalities} & \multicolumn{2}{c}{Model} & \multicolumn{2}{c}{4 Entity Types (\%)} & \multicolumn{2}{c}{Segmentation (\%)} \\ \cline{3-8}  & & Prec. & Recall & F1 & Prec. & Recall & F1 \\ \hline \hline C & Bi-LSTM/CRF + Bi-CharLSTM & 5.0 & 28.1 & 8.5 & 68.6 & 10.8 & 18.6 \\ W & Bi-LSTM/CRF & 38.2 & 53.3 & 44.6 & 82.5 & 50.1 & 62.4 \\ \hline W + C & [29] & 45.9 & 48.9 & 47.4 & 74.0 & 61.7 & 67.3 \\ W + C & [16] & 46.0 & 51.9 & 48.7 & 76.8 & 61.0 & 68.0 \\ W + C & [16] & 47.7 & 49.9 & 48.8 & 74.4 & 63.3 & 68.4 \\ W + C & Bi-LSTM/CRF + Bi-CharLSTM w/ Modality Attention & 49.4 & 51.7 & 50.5 & 75.7 & 63.3 & 68.9 \\ \hline W + C + V & Bi-LSTM/CRF + Bi-CharLSTM + Inception & **50.5** & 52.3 & 51.4 & 71.9 & **66.5** & **69.1** \\ W + C + V & Bi-LSTM/CRF + Bi-CharLSTM + Inception w/ Modality Attention & 48.7 & **58.7** & **52.4** & **77.4** & 60.6 & 68.0 \\ \hline \end{tabular}
\end{table}
Table 1: NER performance on the _SnapCaptions_ dataset with varying modalities (W: word, C: char, V: visual). We report precision, recall, and F1 score for both entity types recognition (PER, LOC, ORG, MISC) and entity segmentation (untyped recognition - named entity or not) tasks.

 

enough information to disambiguate it. On the other hand, visual contexts (visual tags: 'parade', 'urban area', ...) provide similarities to the token's distributional semantics from other training examples (_e.g_. snaps from "NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity. Similarly, while the text-only model erroneously predicts 'Apple' in the caption _"Grandma w dat lit Apple Crisp"_ as an organization (_e.g_. Apple Inc.), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit). Trending entities (musicians or DJs such as 'CID', 'Duke Dumont', 'Marshmello', etc.) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts. A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts.

**Visualization of Modality Attention**: Figure 3 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color.

For the image-aided model (W+C+V; upper row in Figure 3), we confirm that the modality attention successfully attenuates irrelevant signals (_e.g_. selfies, etc.) and amplifies relevant modality-based contexts in prediction of a given token. In the example of "_disney word essential = coffee_" with visual tags _selfie, phone, person_, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions. The named entities in the examples of "_Beautiful night atop The Space Needle_" and "_Splash Mountain_" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict. In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals.

For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix. For example, while _WaRriOoOrs_ and _Kooler Matic_ are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (_e.g_. capitalized first letters, similarity to known entities 'Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens _e.g_. 'WaRriOoOrs'), leading to correct predictions. This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely.

Figure 3 (b) shows the cases where the modality attention led to incorrect predictions. For example, the model predicts missing tokens _HUUUGE_ and _Shampooer_ incorrectly as named entities by amplifying misleading character-based contexts (_e.g_. capitalized first letters) or visual contexts (_e.g_.

\begin{table}
\begin{tabular}{l l l c c c} \hline \hline  & Caption (target) & Visual Tags & GT & \multicolumn{2}{c}{Prediction} \\ \cline{4-6}  & & & & (W+C+V) & (W+C) \\ \hline \hline \multirow{5}{*}{_“The curry’s_} & parade, marching, urban area, ... & B-PER & B-PER & O \\  & _“Grandma w dat lit Apple Crisp”_ & funnel cake, melting, frozen, ... & O & O & B-ORG \\  & _“Okay duke dumont_ & DJ, guitarist, circus, ... & BJ-PER & BJ-PER & O,O \\ \cline{1-1}   & _“CSI with my hubby”_ & TV, movie, television, ... & B-MISC & B-MISC & B-ORG \\  & _“Twin day at angel stadium”_ & stadium, arena, stampede, ... & BJ-LOC & BJ-LOC & O,O \\  & _“LETS GO CID”_ & drum, DJ, drummer, ... & B-PER & B-PER & O \\  & _“MARSHMELLOOOOOOOOS”_ & DJ, night, marrini, ... & B-PER & B-PER & O \\ \hline \multirow{7}{*}{_“Yall come see me at bojangles._} & & & & \\  & _“If u ’re not watching this season of_ & & floor, tile, airport terminal, ... & B-ORG & O & B-ORG \\ \cline{1-1}  & _“If u ’re not watching this season of_ & monitor, suite, cubicle, ... & B-MISC & O & B-MISC \\ \cline{1-1}  & bachelorte ur doing LIFE WRONG”_ & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Error analysis: **when do images help NER?** Ground-truth labels (GT) and predictions of our model with vision input (W+C+V) and the one without (W+C) for the underlined named entities (or false positives) are shown. For interpretability, visual tags (label output of InceptionNet) are presented instead of actual feature vectors used.

 

concert scenes, associated contexts of which often include named entities in the training dataset).

**Sensitivity to Word Embeddings Vocabulary Size**: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table 3. By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation. Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix. This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets.

## 5 Conclusions

We proposed a new multimodal NER (MNER: image + text) task on short social media posts. We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines. Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images. In addition, we proposed the _modality attention_ module, a new neural mechanism which learns optimal integration of different modes of correlated information. In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations. We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information.

Figure 3: **Visualization of modality attention** (a) successful cases and (b) unsuccessful ones from _SnapCaptions_ test data. For each decoding step of a token (column), the modality attention module amplifies the most relevant modality (darker) while attenuating irrelevant modalities (lighter). The model makes final predictions based on weighted signals from all modalities. For interpretability, visual tags (label output of InceptionNet) are presented instead of actual feature vectors used. GT: ground-truth, Pred: prediction by our model. Modalities– W: words, C: characters, V: visual.

 



## References

* Aguilar et al. (2017) Gustavo Aguilar, Suraj Maharjan, A. Pastor Lopez-Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. _ACL WNUT Workshop_ .
* Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In _ICCV_.
* Baldwin et al. (2015) Timothy Baldwin, Marie-Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In _Proceedings of the Workshop on Noisy User-generated Text_.
* Chan et al. (2015) William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. _arXiv:1508.01211_ .
* Chiu and Nichols (2015) Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. _arXiv:1511.08308_ .
* Dhillon et al. (2011) Paramveer Dhillon, Dean P Foster, and Lyle H Ungar. 2011. Multi-view learning of word embeddings via cca. In _NIPS_. pages 199-207.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _JMLR_ .
* Elliott et al. (2015) Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural sequence models. _CoRR, abs/1510.04709_ .
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv:1508.01991_ .
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data .
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. _NAACL_ .
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. _arXiv preprint arXiv:1603.01354_ .
* McCallum and Li (2003) Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In _NAACL_.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _ICALR_ .
* Moon and Carbonell (2017) Seungwhan Moon and Jaime Carbonell. 2017. Completely heterogeneous transfer learning with attention: What and what not to transfer. _IJCAI_ .
* Moon et al. (2015) Seungwhan Moon, Suyoun Kim, and Haohan Wang. 2015. Multimodal transfer deep learning with applications in audio-visual recognition. In _NIPS MMML Workshop_.
* Passos et al. (2014) Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. _aarXiv:1404.5367_ .
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_.
* Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In _CoNLL_.
* Rei et al. (2016) Marek Rei, Gamal KO Crichton, and Sampo Pyysalo. 2016. Attending to characters in neural sequence labeling models. _COLING_ .
* Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. _IJCV_ .
* Sang and Venestra (1999) Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. In _EACL_.
* Specia et al. (2016) Lucia Specia, Stella Frank, Khalil Sima'an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In _WMT_.
* Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In _NIPS_.
* Szegedy et al. (2014) C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2014. Going deeper with convolutions. _CVPR_ .
* Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. _CoRR_ .
* Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In _CVPR_.
* Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. _ICML_ .
* Yao et al. (2015) Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Balas, Christopher Pal, Hugo Larochelle, and Aaron Courville. 2015. Describing videos by exploiting temporal structure. In _ICCV_