prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is imbalance in analyzed corpora?
Context 1: Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres  Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0  In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis  The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%  Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2 
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community  Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3  Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies  This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8 
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have  We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community  Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system 
From gender representation in data to gender bias in AI ::: On the importance of data
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available  The quality of a system now depends mostly on the quality and quantity of the data it has been trained on  If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource  Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development  This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9 
Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems  The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s  The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time  Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used  Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation  For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date 
From gender representation in data to gender bias in AI ::: From data to bias
The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address  Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7)  The impact of the training data used within these deep-learning algorithms is therefore questioned 
Bias can be found at different levels as pointed out by BIBREF19  BIBREF20 defines bias as a skew that produces a type of harm  She distinguishes two types of harms that are allocation harm and representation harm  The allocation harm occurs when a system is performing better or worse for a certain group while representational harm contributes to the perpetuation of stereotypes  Both types of harm are the results of bias in machine learning that often comes from the data systems are trained on  Disparities in representation in our social structures is captured and reflected by the training data, through statistical patterns  The GenderShades study is a striking example of what data disparity and lack of representation can produce: the authors tested several gender recognition modules used by facial recognition tools and found difference in error-rate as high as 34 percentage points between recognition of white male and black female faces  The scarce presence of women and colored people in training set resulted in bias in performance towards these two categories, with a strong intersectional bias  As written by BIBREF21 "A data set may have many millions of pieces of data, but this does not mean it is random or representative  To make statistical claims about a data set, we need to know where data is coming from; it is similarly important to know and account for the weaknesses in that data " (p 668) 
Regarding ASR technology, little work has explored the presence of gender bias within the systems and no consensus has been reached  BIBREF22 found that speech recognizers perform better on female voice on a broadcast news and telephone corpus  They proposed several explanations to this observation, such as the larger presence of non-professional male speech in the broadcast data, implying a less prepared speech for these speakers or a more normative language and standard pronunciation for women linked to the traditional role of women in language acquisition and education  The same trend was observed by BIBREF23  More recently, BIBREF24 discovered a gender bias within YouTube's automatic captioning system but this bias was not observed in a second study evaluating Bing Speech system and YouTube Automatic Captions on a larger data set BIBREF8  However race and dialect bias were found  General American speakers and white speakers had the lowest error rate for both systems  If the better performance on General American speakers could be explained by the fact that they are all voice professionals, producing clear and articulated speech, but no explanation is provided for biases towards non-white speakers 
Gender bias in ASR technology is still an open research question as no clear answer has been reached so far  It seems that many parameters are to take into account to achieve a general agreement  As we established the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have, the following content of this paper is structured as this: we first describe statistically the gender representation of a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community, introducing the notion of speaker's role to refine our analysis in terms of voice professionalism  We then question the impact of such a representation on a ASR system trained on these data  BIBREF25
Methodology
This section is organized as follows: we first present the data we are working on  In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role  The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance 
Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system  Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR  Show duration varies between 10min and an hour  As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed  ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition 
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech  Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech  Training data by show, medium and speech type is summarized in Table and evaluation data in Table   Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set) 
Methodology ::: Methodology for descriptive analysis of gender representation in training data
We first describe the gender representation in training data  Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16)  Each speech turn was mapped to its speaker in order to associate it with a gender 
As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions  It is clear that a small number of speakers is responsible for a large number of speech turns  Most of these speakers are journalists, politicians, presenters and such, who are representative of a show  Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role  Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show  Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length  We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show  The same process is applied to identify speakers talking for a long period from those who do not  We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time 
These roles are defined at the show level  They could be roughly assimilated to the categorization “host/guest” in radio and TV shows  Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people"  The concept of speaker's role makes sense at both sociological and technical levels  An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25  It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe  The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table   Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows  A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27  The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K)  It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode  It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation
Word Error Rate (WER) is a common metric to evaluate ASR performance  It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription  As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level  Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on  Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show  This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0 001) BIBREF30  The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data 
Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table )  Women represent 33 16% of the speakers, confirming the figures given by the GMMP report BIBREF0  However, it is worth noticing that women account for only 22 57% of the total speech time, which leads us to conclude that women also speak less than men 
Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3 79%), they nevertheless concentrate 35 71 % of the total speech time 
Results ::: Descriptive analysis of gender representation in training data ::: Role and gender interaction
When crossing both parameters, we can observe that the gender distribution is not constant throughout roles  Women represent 29 47% of the speakers within the Anchor category, even less than among the Punctual speakers  Their percentage of speech is also smaller  When calculating the average speech time uttered by a female Anchor, we obtain a value of 15 9 min against 25 2 min for a male Anchor, which suggests that even within the Anchor category men tend to speak more  This confirms the existence of gender disparities within French media  It corroborates with the analysis of the CSA BIBREF1, which shows that women were less present during high-audience hours  Our study shows that they are also less present in important roles  These results legitimate our initial questioning on the impact of gender balance on ASR performance trained on broadcast recordings 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference  The higher the WER, the poorer the system performance  Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42 9% for women and 34 3% for men  This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0 001) 
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21  For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume  The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30 8% for the Anchor speakers and 42 23% for the Punctual speakers  This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0 001)  
Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender  It is worth noticing that the gender difference is only significant within the Punctual speakers group  The average WER is of 49 04% for the women and 38 56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0 001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0 173)  This could be explained by the quantity of data available per speaker 
Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results  BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance 
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis  As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech  WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0 005) whereas they are worse for women (61 29%) than for men (46 51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0 001) 
Discussion
We find a clear disparity in terms of women presence and speech quantity in French media  Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation  One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2  Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems 
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition  However the following study on read speech does not replicate these results  Yet a performance degradation is observed across dialect and race BIBREF8  BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French  The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution  We observed that the higher the degree of normativity of speech the smaller the gender difference  No significant gender bias is observed for prepared speech nor within the Anchor category  Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers 
Another explanation could be the use of adaptation within the pipeline  Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system  An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model  On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data  Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table )  Considering that Punctual speakers represent 92 78% of the speakers, this explains why gender difference is significant over our entire data set  A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step 
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders  But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech)  Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4 2 3) 
Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time 
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed  We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias 
Gender bias varies across speaker's role and speech spontaneity level  Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27 2% respectively 31 8% between male and female speakers 
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance  Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data  Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies  This is in line with the concept of “Fairness by Design" proposed by BIBREF31 
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design  Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance  Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop 
Context 2: Introduction
This work is licensed under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Entity linking (EL), mapping entity mentions in texts to a given knowledge base (KB), serves as a fundamental role in many fields, such as question answering BIBREF0 , semantic search BIBREF1 , and information extraction BIBREF2 , BIBREF3   However, this task is non-trivial because entity mentions are usually ambiguous  As shown in Figure FIGREF1 , the mention England refers to three entities in KB, and an entity linking system should be capable of identifying the correct entity as England cricket team rather than England and England national football team 
Entity linking is typically broken down into two main phases: (i) candidate generation obtains a set of referent entities in KB for each mention, and (ii) named entity disambiguation selects the possible candidate entity by solving a ranking problem  The key challenge lies in the ranking model that computes the relevance between candidates and the corresponding mentions based on the information both in texts and KBs BIBREF4   In terms of the features used for ranking, we classify existing EL models into two groups: local models to resolve mentions independently relying on textual context information from the surrounding words BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , and global (collective) models, which are the main focus of this paper, that encourage the target entities of all mentions in a document to be topically coherent BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14  
Global models usually build an entity graph based on KBs to capture coherent entities for all identified mentions in a document, where the nodes are entities, and edges denote their relations  The graph provides highly discriminative semantic signals (e g , entity relatedness) that are unavailable to local model BIBREF15   For example (Figure FIGREF1 ), an EL model seemly cannot find sufficient disambiguation clues for the mention England from its surrounding words, unless it utilizes the coherence information of consistent topic “cricket" among adjacent mentions England, Hussain, and Essex  Although the global model has achieved significant improvements, its limitation is threefold:
To mitigate the first limitation, recent EL studies introduce neural network (NN) models due to its amazing feature abstraction and generalization ability  In such models, words/entities are represented by low dimensional vectors in a continuous space, and features for mention as well as candidate entities are automatically learned from data BIBREF4   However, existing NN-based methods for EL are either local models BIBREF16 , BIBREF17 or merely use word/entity embeddings for feature extraction and rely on another modules for collective disambiguation, which thus cannot fully utilize the power of NN models for collective EL BIBREF18 , BIBREF19 , BIBREF20  
The second drawback of the global approach has been alleviated through approximate optimization techniques, such as PageRank/random walks BIBREF21 , graph pruning BIBREF22 , ranking SVMs BIBREF23 , or loopy belief propagation (LBP) BIBREF18 , BIBREF24   However, these methods are not differentiable and thus difficult to be integrated into neural network models (the solution for the first limitation) 
To overcome the third issue of inadequate training data, BIBREF17 has explored a massive amount of hyperlinks in Wikipedia, but these potential annotations for EL contain much noise, which may distract a naive disambiguation model BIBREF6  
In this paper, we propose a novel Neural Collective Entity Linking model (NCEL), which performs global EL combining deep neural networks with Graph Convolutional Network (GCN) BIBREF25 , BIBREF26 that allows flexible encoding of entity graphs  It integrates both local contextual information and global interdependence of mentions in a document, and is efficiently trainable in an end-to-end fashion  Particularly, we introduce attention mechanism to robustly model local contextual information by selecting informative words and filtering out the noise  On the other hand, we apply GCNs to improve discriminative signals of candidate entities by exploiting the rich structure underlying the correct entities  To alleviate the global computations, we propose to convolute on the subgraph of adjacent mentions  Thus, the overall coherence shall be achieved in a chain-like way via a sliding window over the document  To the best of our knowledge, this is the first effort to develop a unified model for neural collective entity linking 
In experiments, we first verify the efficiency of NCEL via theoretically comparing its time complexity with other collective alternatives  Afterwards, we train our neural model using collected Wikipedia hyperlinks instead of dataset-specific annotations, and perform evaluations on five public available benchmarks  The results show that NCEL consistently outperforms various baselines with a favorable generalization ability  Finally, we further present the performance on a challenging dataset WW BIBREF19 as well as qualitative results, investigating the effectiveness of each key module 
Preliminaries and Framework
We denote INLINEFORM0 as a set of entity mentions in a document INLINEFORM1 , where INLINEFORM2 is either a word INLINEFORM3 or a mention INLINEFORM4   INLINEFORM5 is the entity graph for document INLINEFORM6 derived from the given knowledge base, where INLINEFORM7 is a set of entities, INLINEFORM8 denotes the relatedness between INLINEFORM9 and higher values indicate stronger relations  Based on INLINEFORM10 , we extract a subgraph INLINEFORM11 for INLINEFORM12 , where INLINEFORM13 denotes the set of candidate entities for INLINEFORM14   Note that we don't include the relations among candidates of the same mention in INLINEFORM15 because these candidates are mutually exclusive in disambiguation 
Formally, we define the entity linking problem as follows: Given a set of mentions INLINEFORM0 in a document INLINEFORM1 , and an entity graph INLINEFORM2 , the goal is to find an assignment INLINEFORM3  
To collectively find the best assignment, NCEL aims to improve the discriminability of candidates' local features by using entity relatedness within a document via GCN, which is capable of learning a function of features on the graph through shared parameters over all nodes  Figure FIGREF10 shows the framework of NCEL including three main components:
Example As shown in Figure FIGREF10 , for the current mention England, we utilize its surrounding words as local contexts (e g , surplus), and adjacent mentions (e g , Hussian) as global information  Collectively, we utilize the candidates of England INLINEFORM0 as well as those entities of its adjacencies INLINEFORM1 to construct feature vectors for INLINEFORM2 and the subgraph of relatedness as inputs of our neural model  Let darker blue indicate higher probability of being predicted, the correct candidate INLINEFORM3 becomes bluer due to its bluer neighbor nodes of other mentions INLINEFORM4   The dashed lines denote entity relations that have indirect impacts through the sliding adjacent window , and the overall structure shall be achieved via multiple sub-graphs by traversing all mentions 
Before introducing our model, we first describe the component of candidate generation 
Candidate Generation
Similar to previous work BIBREF24 , we use the prior probability INLINEFORM0 of entity INLINEFORM1 conditioned on mention INLINEFORM2 both as a local feature and to generate candidate entities: INLINEFORM3   We compute INLINEFORM4 based on statistics of mention-entity pairs from: (i) Wikipedia page titles, redirect titles and hyperlinks, (ii) the dictionary derived from a large Web Corpus BIBREF27 , and (iii) the YAGO dictionary with a uniform distribution BIBREF22   We pick up the maximal prior if a mention-entity pair occurs in different resources  In experiments, to optimize for memory and run time, we keep only top INLINEFORM5 entities based on INLINEFORM6   In the following two sections, we will present the key components of NECL, namely feature extraction and neural network for collective entity linking 
Feature Extraction
The main goal of NCEL is to find a solution for collective entity linking using an end-to-end neural model, rather than to improve the measurements of local textual similarity or global mention/entity relatedness  Therefore, we use joint embeddings of words and entities at sense level BIBREF28 to represent mentions and its contexts for feature extraction  In this section, we give a brief description of our embeddings followed by our features used in the neural model 
Learning Joint Embeddings of Word and Entity
Following BIBREF28 , we use Wikipedia articles, hyperlinks, and entity outlinks to jointly learn word/mention and entity embeddings in a unified vector space, so that similar words/mentions and entities have similar vectors  To address the ambiguity of words/mentions, BIBREF28 represents each word/mention with multiple vectors, and each vector denotes a sense referring to an entity in KB  The quality of the embeddings is verified on both textual similarity and entity relatedness tasks 
Formally, each word/mention has a global embedding INLINEFORM0 , and multiple sense embeddings INLINEFORM1   Each sense embedding INLINEFORM2 refers to an entity embedding INLINEFORM3 , while the difference between INLINEFORM4 and INLINEFORM5 is that INLINEFORM6 models the co-occurrence information of an entity in texts (via hyperlinks) and INLINEFORM7 encodes the structured entity relations in KBs  More details can be found in the original paper 
Local Features
Local features focus on how compatible the entity is mentioned in a piece of text (i e , the mention and the context words)  Except for the prior probability (Section SECREF9 ), we define two types of local features for each candidate entity INLINEFORM0 :
String Similarity Similar to BIBREF16 , we define string based features as follows: the edit distance between mention's surface form and entity title, and boolean features indicating whether they are equivalent, whether the mention is inside, starts with or ends with entity title and vice versa 
Compatibility We also measure the compatibility of INLINEFORM0 with the mention's context words INLINEFORM1 by computing their similarities based on joint embeddings: INLINEFORM2 and INLINEFORM3 , where INLINEFORM4 is the context embedding of INLINEFORM5 conditioned on candidate INLINEFORM6 and is defined as the average sum of word global vectors weighted by attentions: INLINEFORM7
where INLINEFORM0 is the INLINEFORM1 -th word's attention from INLINEFORM2   In this way, we automatically select informative words by assigning higher attention weights, and filter out irrelevant noise through small weights  The attention INLINEFORM3 is computed as follows: INLINEFORM4
where INLINEFORM0 is the similarity measurement, and we use cosine similarity in the presented work  We concatenate the prior probability, string based similarities, compatibility similarities and the embeddings of contexts as well as the entity as the local feature vectors 
Global Features
The key idea of collective EL is to utilize the topical coherence throughout the entire document  The consistency assumption behind it is that: all mentions in a document shall be on the same topic  However, this leads to exhaustive computations if the number of mentions is large  Based on the observation that the consistency attenuates along with the distance between two mentions, we argue that the adjacent mentions might be sufficient for supporting the assumption efficiently 
Formally, we define neighbor mentions as INLINEFORM0 adjacent mentions before and after current mention INLINEFORM1 : INLINEFORM2 , where INLINEFORM3 is the pre-defined window size  Thus, the topical coherence at document level shall be achieved in a chain-like way  As shown in Figure FIGREF10 ( INLINEFORM4 ), mentions Hussain and Essex, a cricket player and the cricket club, provide adequate disambiguation clues to induce the underlying topic “cricket" for the current mention England, which impacts positively on identifying the mention surrey as another cricket club via the common neighbor mention Essex 
A degraded case happens if INLINEFORM0 is large enough to cover the entire document, and the mentions used for global features become the same as the previous work, such as BIBREF21   In experiments, we heuristically found a suitable INLINEFORM1 which is much smaller than the total number of mentions  The benefits of efficiency are in two ways: (i) to decrease time complexity, and (ii) to trim the entity graph into a fixed size of subgraph that facilitates computation acceleration through GPUs and batch techniques, which will be discussed in Section SECREF24  
Given neighbor mentions INLINEFORM0 , we extract two types of vectorial global features and structured global features for each candidate INLINEFORM1 :
Neighbor Mention Compatibility Suppose neighbor mentions are topical coherent, a candidate entity shall also be compatible with neighbor mentions if it has a high compatibility score with the current mention, otherwise not  That is, we extract the vectorial global features by computing the similarities between INLINEFORM0 and all neighbor mentions: INLINEFORM1 , where INLINEFORM2 is the mention embedding by averaging the global vectors of words in its surface form: INLINEFORM3 , where INLINEFORM4 are tokenized words of mention INLINEFORM5  
Subgraph Structure The above features reflect the consistent semantics in texts (i e , mentions)  We now extract structured global features using the relations in KB, which facilitates the inference among candidates to find the most topical coherent subset

ground_truth: Women represent 33.16% of the speakers

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How effective is their NCEL approach overall?
Context 2:  For each document, we obtain the entity graph INLINEFORM0 by taking candidate entities of all mentions INLINEFORM1 as nodes, and using entity embeddings to compute their similarities as edges INLINEFORM2   Then, we extract the subgraph structured features INLINEFORM3 for each entity INLINEFORM4 for efficiency 
Formally, we define the subgraph as: INLINEFORM0 , where INLINEFORM1   For example (Figure FIGREF1 ), for entity England cricket team, the subgraph contains the relation from it to all candidates of neighbor mentions: England cricket team, Nasser Hussain (rugby union), Nasser Hussain, Essex, Essex County Cricket Club and Essex, New York  To support batch-wise acceleration, we represent INLINEFORM2 in the form of adjacency table based vectors: INLINEFORM3 , where INLINEFORM4 is the number of candidates per mention 
Finally, for each candidate INLINEFORM0 , we concatenate local features and neighbor mention compatibility scores as the feature vector INLINEFORM1 , and construct the subgraph structure representation INLINEFORM2 as the inputs of NCEL 
Neural Collective Entity Linking
NCEL incorporates GCN into a deep neural network to utilize structured graph information for collectively feature abstraction, while differs from conventional GCN in the way of applying the graph  Instead of the entire graph, only a subset of nodes is “visible" to each node in our proposed method, and then the overall structured information shall be reached in a chain-like way  Fixing the size of the subset, NCEL is further speeded up by batch techniques and GPUs, and is efficient to large-scale data 
Graph Convolutional Network
GCNs are a type of neural network model that deals with structured data  It takes a graph as an input and output labels for each node  As a simplification of spectral graph convolutions, the main idea of BIBREF26 is similar to a propagation model: to enhance the features of a node according to its neighbor nodes  The formulation is as follows: INLINEFORM0
where INLINEFORM0 is a normalized adjacent matrix of the input graph with self-connection, INLINEFORM1 and INLINEFORM2 are the hidden states and weights in the INLINEFORM3 -th layer, and INLINEFORM4 is a non-linear activation, such as ReLu 
Model Architecture
As shown in Figure FIGREF10 , NCEL identifies the correct candidate INLINEFORM0 for the mention INLINEFORM1 by using vectorial features as well as structured relatedness with candidates of neighbor mentions INLINEFORM2   Given feature vector INLINEFORM3 and subgraph representation INLINEFORM4 of each candidate INLINEFORM5 , we stack them as inputs for mention INLINEFORM6 : INLINEFORM7 , and the adjacent matrix INLINEFORM8 , where INLINEFORM9 denotes the subgraph with self-connection  We normalize INLINEFORM10 such that all rows sum to one, denoted as INLINEFORM11 , avoiding the change in the scale of the feature vectors 
Given INLINEFORM0 and INLINEFORM1 , the goal of NCEL is to find the best assignment: INLINEFORM2
where INLINEFORM0 is the output variable of candidates, and INLINEFORM1 is a probability function as follows: INLINEFORM2
where INLINEFORM0 is the score function parameters by INLINEFORM1   NCEL learns the mapping INLINEFORM2 through a neural network including three main modules: encoder, sub-graph convolution network (sub-GCN) and decoder  Next, we introduce them in turn 
Encoder The function of this module is to integrate different features by a multi-layer perceptron (MLP): INLINEFORM0
where INLINEFORM0 is the hidden states of the current mention, INLINEFORM1 and INLINEFORM2 are trainable parameters and bias  We use ReLu as the non-linear activation INLINEFORM3  
Sub-Graph Convolution Network Similar to GCN, this module learns to abstract features from the hidden state of the mention itself as well as its neighbors  Suppose INLINEFORM0 is the hidden states of the neighbor INLINEFORM1 , we stack them to expand the current hidden states of INLINEFORM2 as INLINEFORM3 , such that each row corresponds to that in the subgraph adjacent matrix INLINEFORM4   We define sub-graph convolution as: INLINEFORM5
where INLINEFORM0 is a trainable parameter 
Decoder After INLINEFORM0 iterations of sub-graph convolution, the hidden states integrate both features of INLINEFORM1 and its neighbors  A fully connected decoder maps INLINEFORM2 to the number of candidates as follows: INLINEFORM3
where INLINEFORM0  
Training
The parameters of network are trained to minimize cross-entropy of the predicted and ground truth INLINEFORM0 : INLINEFORM1
Suppose there are INLINEFORM0 documents in training corpus, each document has a set of mentions INLINEFORM1 , leading to totally INLINEFORM2 mention sets  The overall objective function is as follows: INLINEFORM3
Experiments
To avoid overfitting with some dataset, we train NCEL using collected Wikipedia hyperlinks instead of specific annotated data  We then evaluate the trained model on five different benchmarks to verify the linking precision as well as the generalization ability  Furthermore, we investigate the effectiveness of key modules in NCEL and give qualitative results for comprehensive analysis 
Baselines and Datasets
We compare NCEL with the following state-of-the-art EL methods including three local models and three types of global models:
Local models: He BIBREF29 and Chisholm BIBREF6 beat many global models by using auto-encoders and web links, respectively, and NTEE BIBREF16 achieves the best performance based on joint embeddings of words and entities 
Iterative model: AIDA BIBREF22 links entities by iteratively finding a dense subgraph 
Loopy Belief Propagation: Globerson BIBREF18 and PBoH BIBREF30 introduce LBP BIBREF31 techniques for collective inference, and Ganea BIBREF24 solves the global training problem via truncated fitting LBP 
PageRank/Random Walk: Boosting BIBREF32 , AGDISTISG BIBREF33 , Babelfy BIBREF34 , WAT BIBREF35 , xLisa BIBREF36 and WNED BIBREF19 performs PageRank BIBREF37 or random walk BIBREF38 on the mention-entity graph and use the convergence score for disambiguation 
For fairly comparison, we report the original scores of the baselines in the papers  Following these methods, we evaluate NCEL on the following five datasets: (1) CoNLL-YAGO BIBREF22 : the CoNLL 2003 shared task including testa of 4791 mentions in 216 documents, and testb of 4485 mentions in 213 documents  (2) TAC2010 BIBREF39 : constructed for the Text Analysis Conference that comprises 676 mentions in 352 documents for testing  (3) ACE2004 BIBREF23 : a subset of ACE2004 co-reference documents including 248 mentions in 35 documents, which is annotated by Amazon Mechanical Turk  (4) AQUAINT BIBREF40 : 50 news articles including 699 mentions from three different news agencies  (5) WW BIBREF19 : a new benchmark with balanced prior distributions of mentions, leading to a hard case of disambiguation  It has 6374 mentions in 310 documents automatically extracted from Wikipedia 
Training Details and Running Time Analysis
Training We collect 50,000 Wikipedia articles according to the number of its hyperlinks as our training data  For efficiency, we trim the articles to the first three paragraphs leading to 1,035,665 mentions in total  Using CoNLL-Test A as the development set, we evaluate the trained NCEL on the above benchmarks  We set context window to 20, neighbor mention window to 6, and top INLINEFORM0 candidates for each mention  We use two layers with 2000 and 1 hidden units in MLP encoder, and 3 layers in sub-GCN  We use early stop and fine tune the embeddings  With a batch size of 16, nearly 3 epochs cost less than 15 minutes on the server with 20 core CPU and the GeForce GTX 1080Ti GPU with 12Gb memory  We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements 
Complexity Analysis Compared with local methods, the main disadvantage of collective methods is high complexity and expensive costs  Suppose there are INLINEFORM0 mentions in documents on average, among these global models, NCEL not surprisingly has the lowest time complexity INLINEFORM1 since it only considers adjacent mentions, where INLINEFORM2 is the number of sub-GCN layers indicating the iterations until convergence  AIDA has the highest time complexity INLINEFORM3 in worst case due to exhaustive iteratively finding and sorting the graph  The LBP and PageRank/random walk based methods achieve similar high time complexity of INLINEFORM4 mainly because of the inference on the entire graph 
Results on GERBIL
GERBIL BIBREF41 is a benchmark entity annotation framework that aims to provide a unified comparison among different EL methods across datasets including ACE2004, AQUAINT and CoNLL  We compare NCEL with the global models that report the performance on GERBIL 
As shown in Table TABREF26 , NCEL achieves the best performance in most cases with an average gain of 2% on Micro F1 and 3% Macro F1  The baseline methods also achieve competitive results on some datasets but fail to adapt to the others  For example, AIDA and xLisa perform quite well on ACE2004 but poorly on other datasets, or WAT, PBoH, and WNED have a favorable performance on CoNLL but lower values on ACE2004 and AQUAINT  Our proposed method performs consistently well on all datasets that demonstrates the good generalization ability 
Results on TAC2010 and WW
In this section, we investigate the effectiveness of NCEL in the “easy" and “hard" datasets, respectively  Particularly, TAC2010, which has two mentions per document on average (Section SECREF19 ) and high prior probabilities of correct candidates (Figure FIGREF28 ), is regarded as the “easy" case for EL, and WW is the “hard" case since it has the most mentions with balanced prior probabilities BIBREF19   Besides, we further compare the impact of key modules by removing the following part from NCEL: global features (NCEL-local), attention (NCEL-noatt), embedding features (NCEL-noemb), and the impact of the prior probability (prior) 
The results are shown in Table FIGREF28 and Table FIGREF28   We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases  In the “easy" case, local models have similar performance with global models since only little global information is available (2 mentions per document)  Besides, NN-based models, NTEE and NCEL-local, perform significantly better than others including most global models, demonstrating that the effectiveness of neural models deals with the first limitation in the introduction 
Impact of NCEL Modules
As shown in Figure FIGREF28 , the prior probability performs quite well in TAC2010 but poorly in WW  Compared with NCEL-local, the global module in NCEL brings more improvements in the “hard" case than that for “easy" dataset, because local features are discriminative enough in most cases of TAC2010, and global information becomes quite helpful when local features cannot handle  That is, our propose collective model is robust and shows a good generalization ability to difficult EL  The improvements by each main module are relatively small in TAC2010, while the modules of attention and embedding features show non-negligible impacts in WW (even worse than local model), mainly because WW contains much noise, and these two modules are effective in improving the robustness to noise and the ability of generalization by selecting informative words and providing more accurate semantics, respectively 
Qualitative Analysis
The results of example in Figure FIGREF1 are shown in Table TABREF30 , which is from CoNLL testa dataset  For mention Essex, although both NCEL and NCEL-local correctly identify entity Essex County Cricket Club, NCEL outputs higher probability due to the enhancement of neighbor mentions  Moreover, for mention England, NCEL-local cannot find enough disambiguation clues from its context words, such as surplus and requirements, and thus assigns a higher probability of 0 42 to the country England according to the prior probability  Collectively, NCEL correctly identifies England cricket team with a probability of 0 72 as compared with 0 20 in NCEL-local with the help of its neighbor mention Essex 
Conclusion
In this paper, we propose a neural model for collective entity linking that is end-to-end trainable  It applies GCN on subgraphs instead of the entire entity graph to efficiently learn features from both local and global information  We design an attention mechanism that endows NCEL robust to noisy data  Trained on collected Wikipedia hyperlinks, NCEL outperforms the state-of-the-art collective methods across five different datasets  Besides, further analysis of the impacts of main modules as well as qualitative results demonstrates its effectiveness 
In the future, we will extend our method into cross-lingual settings to help link entities in low-resourced languages by exploiting rich knowledge from high-resourced languages, and deal with NIL entities to facilitate specific applications 
Acknowledgments
The work is supported by National Key Research and Development Program of China (2017YFB1002101), NSFC key project (U1736204, 61661146007), and THUNUS NExT Co-Lab 

ground_truth: NCEL consistently outperforms various baselines with a favorable generalization ability

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Does their NER model learn NER from both text and images?
Context 3: Introduction
Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individuals  These insights are primarily hidden in unstructured forms of social media posts, such as in free-form text or images without tags  Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a critical step for building structural information, allowing for its use in personalized assistance, recommendations, advertisement, etc 
While many previous approaches BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 on NER have shown success for well-formed text in recognizing named entities via word context resolution (e g  LSTM with word embeddings) combined with character-level features (e g  CharLSTM/CNN), several additional challenges remain for recognizing named entities from extremely short and coarse text found in social media posts  For instance, short social media posts often do not provide enough textual contexts to resolve polysemous entities (e g  “monopoly is da best ", where `monopoly' may refer to a board game (named entity) or a term in economics)  In addition, noisy text includes a huge number of unknown tokens due to inconsistent lexical notations and frequent mentions of various newly trending entities (e g  “xoxo Marshmelloooo ", where `Marshmelloooo' is a mis-spelling of a known entity `Marshmello', a music producer), making word embeddings based neural networks NER models vulnerable 
To address the challenges above for social media posts, we build upon the state-of-the-art neural architecture for NER with the following two novel approaches (Figure FIGREF1 )  First, we propose to leverage auxiliary modalities for additional context resolution of entities  For example, many popular social media platforms now provide ways to compose a post in multiple modalities - specifically image and text (e g  Snapchat captions, Twitter posts with image URLs), from which we can obtain additional context for understanding posts  While “monopoly" in the previous example is ambiguous in its textual form, an accompanying snap image of a board game can help disambiguate among polysemous entities, thereby correctly recognizing it as a named entity 
Second, we also propose a general modality attention module which chooses per decoding step the most informative modality among available ones (in our case, word embeddings, character embeddings, or visual features) to extract context from  For example, the modality attention module lets the decoder attenuate the word-level signals for unknown word tokens (“Marshmellooooo" with trailing `o's) and amplifies character-level features intsead (capitalized first letter, lexical similarity to other known named entity token `Marshmello', etc ), thereby suppressing noise information (“UNK" token embedding) in decoding steps  Note that most of the previous literature in NER or other NLP tasks combine word and character-level information with naive concatenation, which is vulnerable to noisy social media posts  When an auxiliary image is available, the modality attention module determines to amplify this visual context in disambiguating polysemous entities, or to attenuate visual contexts when they are irrelevant to target named entities, selfies, etc  Note that the proposed modality attention module is distinct from how attention is used in other sequence-to-sequence literature (e g  attending to a specific token within an input sequence)  Section SECREF2 provides the detailed literature review 
Our contributions are three-fold: we propose (1) an LSTM-CNN hybrid multimodal NER network that takes as input both image and text for recognition of a named entity in text input  To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks  (2) We propose a general modality attention module that selectively chooses modalities to extract primary context from, maximizing information gain and suppressing irrelevant contexts from each modality (we treat words, characters, and images as separate modalities)  (3) We show that the proposed approaches outperform the state-of-the-art NER models (both with and without using additional visual contexts) on our new MNER dataset SnapCaptions, a large collection of informal and extremely short social media posts paired with unique images 
Related Work
Neural models for NER have been recently proposed, producing state-of-the-art performance on standard NER tasks  For example, some of the end-to-end NER systems BIBREF4 , BIBREF2 , BIBREF3 , BIBREF0 , BIBREF1 use a recurrent neural network usually with a CRF BIBREF5 , BIBREF6 for sequence labeling, accompanied with feature extractors for words and characters (CNN, LSTMs, etc ), and achieve the state-of-the-art performance mostly without any use of gazetteers information  Note that most of these work aggregate textual contexts via concatenation of word embeddings and character embeddings  Recently, several work have addressed the NER task specifically on noisy short text segments such as Tweets, etc  BIBREF7 , BIBREF8   They report performance gains from leveraging external sources of information such as lexical information (POS tags, etc ) and/or from several preprocessing steps (token substitution, etc )  Our model builds upon these state-of-the-art neural models for NER tasks, and improves the model in two critical ways: (1) incorporation of visual contexts to provide auxiliary information for short media posts, and (2) addition of the modality attention module, which better incorporates word embeddings and character embeddings, especially when there are many missing tokens in the given word embedding matrix  Note that we do not explore the use of gazetteers information or other auxiliary information (POS tags, etc ) BIBREF9 as it is not the focus of our study 
Attention modules are widely applied in several deep learning tasks BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13   For example, they use an attention module to attend to a subset within a single input (a part/region of an image, a specific token in an input sequence of tokens, etc ) at each decoding step in an encoder-decoder framework for image captioning tasks, etc  BIBREF14 explore various attention mechanisms in NLP tasks, but do not incorporate visual components or investigate the impact of such models on noisy social media data  BIBREF15 propose to use attention for a subset of discrete source samples in transfer learning settings  Our modality attention differs from the previous approaches in that we attenuate or amplifies each modality input as a whole among multiple available modalities, and that we use the attention mechanism essentially to map heterogeneous modalities in a single joint embedding space  Our approach also allows for re-use of the same model for predicting labels even when some of the modalities are missing in input, as other modalities would still preserve the same semantics in the embeddings space 
Multimodal learning is studied in various domains and applications, aimed at building a joint model that extracts contextual information from multiple modalities (views) of parallel datasets 
The most relevant task to our multimodal NER system is the task of multimodal machine translation BIBREF16 , BIBREF17 , which aims at building a better machine translation system by taking as input a sentence in a source language as well as a corresponding image  Several standard sequence-to-sequence architectures are explored (a target-language LSTM decoder that takes as input an image first)  Other previous literature include study of Canonical Correlation Analysis (CCA) BIBREF18 to learn feature correlations among multiple modalities, which is widely used in many applications  Other applications include image captioning BIBREF10 , audio-visual recognition BIBREF19 , visual question answering systems BIBREF20 , etc 
To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks 
Proposed Methods
Figure FIGREF2 illustrates the proposed multimodal NER (MNER) model  First, we obtain word embeddings, character embeddings, and visual features (Section SECREF3 )  A Bi-LSTM-CRF model then takes as input a sequence of tokens, each of which comprises a word token, a character sequence, and an image, in their respective representation (Section SECREF4 )  At each decoding step, representations from each modality are combined via the modality attention module to produce an entity label for each token ( SECREF5 )  We formulate each component of the model in the following subsections 
Notations: Let INLINEFORM0 a sequence of input tokens with length INLINEFORM1 , with a corresponding label sequence INLINEFORM2 indicating named entities (e g  in standard BIO formats)  Each input token is composed of three modalities: INLINEFORM3 for word embeddings, character embeddings, and visual embeddings representations, respectively 
Features
Similar to the state-of-the-art NER approaches BIBREF0 , BIBREF1 , BIBREF8 , BIBREF4 , BIBREF2 , BIBREF3 , we use both word embeddings and character embeddings 
Word embeddings are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics BIBREF21   Specifically, we use pre-trained embeddings from GloVE BIBREF22  
Character embeddings are obtained from a Bi-LSTM which takes as input a sequence of characters of each token, similarly to BIBREF0   An alternative approach for obtaining character embeddings is using a convolutional neural network as in BIBREF1 , but we find that Bi-LSTM representation of characters yields empirically better results in our experiments 
Visual embeddings: To extract features from an image, we take the final hidden layer representation of a modified version of the convolutional network model called Inception (GoogLeNet) BIBREF23 , BIBREF24 trained on the ImageNet dataset BIBREF25 to classify multiple objects in the scene  Our implementation of the Inception model has deep 22 layers, training of which is made possible via “network in network" principles and several dimension reduction techniques to improve computing resource utilization  The final layer representation encodes discriminative information describing what objects are shown in an image, which provide auxiliary contexts for understanding textual tokens and entities in accompanying captions 
Incorporating this visual information onto the traditional NER system is an open challenge, and multiple approaches can be considered  For instance, one may provide visual contexts only as an initial input to decoder as in some encoder-decoder image captioning systems BIBREF26   However, we empirically observe that an NER decoder which takes as input the visual embeddings at every decoding step (Section SECREF4 ), combined with the modality attention module (Section SECREF5 ), yields better results 
Lastly, we add a transform layer for each feature INLINEFORM0 before it is fed to the NER entity LSTM 
Bi-LSTM + CRF for Multimodal NER
Our MNER model is built on a Bi-LSTM and CRF hybrid model  We use the following implementation for the entity Bi-LSTM 
it = (Wxiht-1 + Wcict-1)
ct = (1-it) ct-1
+ it tanh(Wxcxt + Whcht-1)
ot = (Wxoxt + Whoht-1 + Wcoct)
ht = LSTM(xt)
= ot tanh(ct)
where INLINEFORM0 is a weighted average of three modalities INLINEFORM1 via the modality attention module, which will be defined in Section SECREF5   Bias terms for gates are omitted here for simplicity of notation 
We then obtain bi-directional entity token representations INLINEFORM0 by concatenating its left and right context representations  To enforce structural correlations between labels in sequence decoding, INLINEFORM1 is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective  y* = y p(y|h; WCRF)
p(y|h; WCRF) = t t (yt-1,yt;h) y' t t (y't-1,y't;h)
where INLINEFORM0 is a potential function, INLINEFORM1 is a set of parameters that defines the potential functions and weight vectors for label pairs ( INLINEFORM2 )  Bias terms are omitted for brevity of formulation 
The model can be trained via log-likelihood maximization for the training set INLINEFORM0 :
L(WCRF) = i p(y|h; W)
Modality Attention
The modality attention module learns a unified representation space for multiple available modalities (words, characters, images, etc ), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance  We motivate this module from the following observations 
A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, e g  INLINEFORM0 in Eq  SECREF4   However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (an all-zero vector INLINEFORM1 or a random vector INLINEFORM2 is assigned for any unknown token INLINEFORM3 , thus INLINEFORM4 or INLINEFORM5 )  While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a significant number of missing tokens 
Similarly, naive merging of textual and visual information ( INLINEFORM0 ) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities  Hence, ideally there needs a mechanism in which the model can effectively turn the switch on and off the modalities adaptive to each sample 
To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step INLINEFORM0 , and produces a soft-attended context vector INLINEFORM1 as an input token for the entity LSTM  [at(w),at(c),at(v)] = (Wm[xt(w); xt(c); xt(v)] + bm )
t(m) = (at(m))m'{w,c,v}(at(m')) m {w,c,v}
xt = m{w,c,v} t(m)xt(m)
where INLINEFORM0 is an attention vector at each decoding step INLINEFORM1 , and INLINEFORM2 is a final context vector at INLINEFORM3 that maximizes information gain for INLINEFORM4   Note that the optimization of the objective function (Eq  SECREF4 ) with modality attention (Eq  SECREF5 ) requires each modality to have the same dimension ( INLINEFORM5 ), and that the transformation via INLINEFORM6 essentially enforces each modality to be mapped into the same unified subspace, where the weighted average of which encodes discrimitive features for recognition of named entities 
When visual context is not provided with each token (as in the traditional NER task), we can define the modality attention for word and character embeddings only in a similar way: [at(w),at(c)] = (Wm[xt(w); xt(c)] + bm )
t(m) = (at(m))m'{w,c}(at(m')) m {w,c}
xt = m{w,c} t(m)xt(m)
Note that while we apply this modality attention module to the Bi-LSTM+CRF architecture (Section SECREF4 ) for its empirical superiority, the module itself is flexible and thus can work with other NER architectures or for other multimodal applications 
SnapCaptions Dataset
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC)  These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories)  Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc  All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available)  We split the dataset into train (70%), validation (15%), and test sets (15%)  The captions data have average length of 30 7 characters (5 81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22   Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc ) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities 
Baselines
Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27   We report the performance of the following state-of-the-art NER models as baselines, as well as several configurations of our proposed approach to examine contributions of each component (W: word, C: char, V: visual) 
Bi-LSTM/CRF (W only): only takes word token embeddings (Stanford GloVE) as input  The rest of the architecture is kept the same 
Bi-LSTM/CRF + Bi-CharLSTM (C only): only takes a character sequence of each word token as input  (No word embeddings)
Bi-LSTM/CRF + Bi-CharLSTM (W+C) BIBREF0 : takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM  Entity LSTM takes concatenated vectors of word and character embeddings as input tokens 
Bi-LSTM/CRF + CharCNN (W+C) BIBREF1 : uses character embeddings extracted from a CNN instead 
Bi-LSTM/CRF + CharCNN (W+C) + Multi-task BIBREF8 : trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM 
Results: SnapCaptions Dataset
Table TABREF6 shows the NER performance on the Snap Captions dataset  We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results 
Parameters: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, 150, 200, 300}, word embeddings size: {25, 50, 100, 150, 200, 300}, LSTM hidden states: {25, 50, 100, 150, 200, 300}, and INLINEFORM0 dimension: {25, 50, 100, 150, 200, 300}  We optimize the parameters with Adagrad BIBREF28 with batch size 10, learning rate 0 02, epsilon INLINEFORM1 , and decay 0 0 
Main Results: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks  In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V)  This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain  Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines BIBREF8 , BIBREF1 , BIBREF0 that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well 
Error Analysis: Table TABREF17 shows example cases where incorporation of visual contexts affects prediction of named entities  For example, the token `curry' in the caption “The curry's " is polysemous and may refer to either a type of food or a famous basketball player `Stephen Curry', and the surrounding textual contexts do not provide enough information to disambiguate it  On the other hand, visual contexts (visual tags: `parade', `urban area',    ) provide similarities to the token's distributional semantics from other training examples (snaps from “NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity  Similarly, while the text-only model erroneously predicts `Apple' in the caption “Grandma w dat lit Apple Crisp" as an organization (Apple Inc ), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit)  Trending entities (musicians or DJs such as `CID', `Duke Dumont', `Marshmello', etc ) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts  A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts 
Visualization of Modality Attention: Figure FIGREF19 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color 
For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc ) and amplifies relevant modality-based contexts in prediction of a given token  In the example of “disney word essential = coffee" with visual tags selfie, phone, person, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions  The named entities in the examples of “Beautiful night atop The Space Needle" and “Splash Mountain" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict  In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals 
For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix  For example, while WaRriOoOrs and Kooler Matic are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (capitalized first letters, similarity to known entities `Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens `WaRriOoOrs'), leading to correct predictions  This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely 
Figure FIGREF19 (b) shows the cases where the modality attention led to incorrect predictions  For example, the model predicts missing tokens HUUUGE and Shampooer incorrectly as named entities by amplifying misleading character-based contexts (capitalized first letters) or visual contexts (concert scenes, associated contexts of which often include named entities in the training dataset) 
Sensitivity to Word Embeddings Vocabulary Size: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table TABREF20   By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation  Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix  This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets 
Conclusions
We proposed a new multimodal NER (MNER: image + text) task on short social media posts  We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines  Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images  In addition, we proposed the modality attention module, a new neural mechanism which learns optimal integration of different modes of correlated information  In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations  We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information 

ground_truth: Yes

Yes