Instead of using RNN, authors used LSTM. The long and short term memory capability of these networks helps them to keep in memory what is important and forget what is not necessary to remember. Equation DISPLAY_FORM6 formulates forget-gate of an LSTM neural network, eq. shows input-gate, eq. notes output-gate and eq. presents memory-cell. Finally, eq. shows the hidden part of an LSTM unit , . for all these equations, $\sigma $ is activation function (sigmoid or tanh are commonly used for LSTM) and $\circ $ is concatenation operation. $W$ and $U$ are weights and $b$ is the bias which should be learned over training process. LSTM is useful for capturing the relation of tokens in a forward sequential form, however in natural language processing tasks, it is required to know the upcoming token. To overcome this problem, the authors have used a backward and forward LSTM combining output of both. In a different approach, character embedding followed by a convolution layer is proposed in  for sequence labeling. The utilized architecture is followed by a bidirectional LSTM layer that ends in a CRF layer.

Experimental Evaluation ::: Dataset	In  a refined collection of tweets gathered from twitter is presented. Their dataset which is labeled for named entity recognition task contains 8,257 tweets. There are 12,784 entities in total in this dataset. Table TABREF19 shows statistics related to each named entity in training, development and test sets.

To have a better understanding of MNER, equation DISPLAY_FORM9 formulates the available multimodal data while equations and are true for this task. $i$ refers to image and the rest goes same as equation DISPLAY_FORM2 for word token sequence. In  pioneering research was conducted using feature extraction from both image and textual data. The extracted features were fed to decision trees in order to output the named entity classes. Researchers have used multiple datasets ranging from buildings to human face images to train their image feature extractor (object detector and k-means clustering) and a text classifier has been trained on texts acquired from DBPedia. Researchers in  proposed a MNER model with regards to triplet embedding of words, characters and image. Modality attention applied to this triplet indicates the importance of each embedding and their impact on the output while reducing the impact of irrelevant modals. Modality attention layer is applied to all embedding vectors for each modal, however the investigation of fine-grained attention mechanism is still unclear . The proposed method with Inception feature extraction  and pretrained GloVe word vectors shows good results on the dataset that the authors aggregated from Snapchat.

Experimental Evaluation	The present section provides evaluation results of our model against baselines. Before diving into our results, a brief description of dataset and its statistics are provided.

Conclusion	In this article we have proposed a novel named entity recognizer based on multimodal deep learning. In our proposed model, we have used a new architecture in character feature extraction that has helped our model to overcome the issue of noise. Instead of using direct image features from near last layers of image feature extractors such as Inception, we have used the direct output of the last layer. This last layer which is 1000 classes of diverse objects that is result of InceptionV3 trained on ImageNet dataset. We used top 5 classes out of these and converted them to one-hot vectors. The resulting image feature embedding out of these high probability one-hot vectors helped our model to overcome the issue of noise in images posted by social media users. Evaluation results of our proposed model compared to other state of the art methods show its superiority to these methods in overall while in two categories (Person and Miscellaneous) our model outperformed others.

This method shows around 0.5 for precision and F-measure for four entity types (person, location, organization and misc) while for segmentation tasks (distinguishing between a named entity and a non-named entity) it shows around 0.7 for the metrics mentioned. An adaptive co-attention neural network with four generations are proposed in . The adaptive co-attention part is similar to the multimodal attention proposed in  that enabled the authors to have better results over the dataset they collected from Twitter. In their main proposal, convolutional layers are used for word representation, BiLSTM is utilized to combine word and character embeddings and an attention layer combines the best of the triplet (word, character and image features). VGG-Net16  is used as a feature extractor for image while the impact of other deep image feature extractors on the proposed solution is unclear, however the results show its superiority over related unimodal methods.

Character embedding is a useful technique that the authors tried to use it in a combination with word embedding. Character embedding with the use of convolution as feature extractor from character level, captures relations between characters that form a word and reduces spelling noise. It also helps the model to have an embedding when pretrained word embedding is empty or initialized as random for new words. These words are encountered when they were not present in the training set, thus, in the test phase, model fails to provide a useful embedding.

Many models were available as first part of image feature extraction, however the main reason we used InceptionV3 as feature extractor backbone is better performance of it on ImageNet and the results obtained by this particular model were slightly better compared to others. Instead of using headless version of InceptionV3 for image feature extraction, we have used the full model which outputs the 1000 classes of ImageNet. Each of these classes resembles an item, the set of these items can present a person, location or anything that is identified as a whole. To have better features extracted from the image, we have used an embedding layer. In other words, we looked at the top 5 extracted probabilities as words that is shown in eq. DISPLAY_FORM16; Based on our assumption, these five words present textual keywords related to the image and combination of these words should provide useful information about the objects in visual data. An LSTM unit has been used to output the final image features. These combined embeddings from the most probable items in image are the key to have extra information from a social media post. where $IW$ is image-word vector, $x$ is output of InceptionV3 and $i$ is the image.

Rather than using the one-hot representation of characters, a randomly initialized (uniform distribution) embedding layer is used. The first three convolution layers are followed by a one dimensional pooling layer. In each layer, kernel size is increased incrementally from 2 to 4 while the number of kernels are doubled starting from 16. Just like the first part, the second segment of this feature extractor uses three layers but with slight changes. Kernel size is reduced starting from 4 to 2 and the number of kernels is halved starting from 64. In this part, $\otimes $ sign shows concatenation operation. TD + GN + SineRelu note targeted dropout, group normalization and sine-relu , , . These layers prevent the character feature extractor from overfitting. Equation DISPLAY_FORM12 defines SineRelu activation function which is slightly different from Relu. Instead of using zero in the second part of this equation, $\epsilon (\sin {x}-\cos {x})$ has been used for negative inputs, $\epsilon $ is a hyperparameter that controls the amplitude of $\sin {x}-\cos {x}$ wave.

Experimental Evaluation ::: Experimental Setup	In order to obtain the best results in tab. TABREF20 for our first model (CWI), we have used the following setup in tables TABREF22, TABREF23, TABREF24 and TABREF25. For the second proposed method, the same parameter settings have been used with an additional attention layer. This additional layer has been added after layer 31 in table TABREF25 and before the final CRF layer, indexed as 32. $Adam$ optimizer with $8\times 10^{-5}$ has been used in training phase with 10 epochs.

Related Work ::: Unimodal Named Entity Recognition	The recognition of named entities from only textual data (unimodal learning approach) is a well studied and explored research criteria. For a prominent example of this category, the Stanford NER is a widely used baseline for many applications . The incorporation of non-local information in information extraction is proposed by the authors using of Gibbs sampling. The conditional random field (CRF) approach used in this article, creates a chain of cliques, where each clique represents the probabilistic relationship between two adjacent states. Also, Viterbi algorithm has been used to infer the most likely state in the CRF output sequence. Equation DISPLAY_FORM5 shows the proposed CRF method. where $\phi $ is the potential function. CRF finds the most probable likelihood by modeling the input sequence of tokens $s$ as a normalized product of feature functions. In a simpler explanation, CRF outputs the most probable tags that follow each other. For example it is more likely to have an I-PER, O or any other that that starts with B- after B-PER rather than encountering tags that start with I-.

This slight change prevents network from having dead-neurons and unlike Relu, it is differentiable everywhere. On the other hand, it has been proven that using GroupNormalization provides better results than BatchNormalization on various tasks . However the dropout has major improvement on the neural network as an overfitting prevention technique , in our setup the TargtedDropout shows to provide better results. TargetedDropout randomly drops neurons whose output is over a threshold. Word Feature Extraction is presented in the middle part of figure FIGREF11. Joint embeddings from pretrained word vectors of GloVe  and fastText  by concatenation operation results in 500 dimensional word embedding. In order to have forward and backward information for each hidden layer, we used a bidirectional long-short term memory , . For the words which were not in the pretrained tokens, we used a random initialization (uniform initialization) between -0.25 and 0.25 at each embedding. The result of this phase is extracted features for each word. Image Feature Extraction is shown in the right part of figure FIGREF11. For this part, we have used InceptionV3 pretrained on ImageNet .

As an example, consider a sequence of tokens where a new token invented by social media users gets trended. This trending new word is misspelled and is used in a sequence along with other tokens in which the whole sequence does not follow known linguistic grammar. For this special case, classical methods and those which use engineered features do not perform well. Using the sequence $s$ itself or adding more information to it divides two approaches to overcome this problem: unimodal and multimodal. Although many approaches for NER have been proposed and reviewing them is not in the scope of this article, we focus on foremost analogues classical and deep learning approaches for named entity recognition in two subsections. In subsection SECREF4 unimodal approaches for named entity recognition are presented while in subsection SECREF7 emerging multimodal solutions are described.

T-NER is another approach that is specifically aimed to answer NER task in twitter . A set of algorithms in their original work have been published to answer tasks such as POS (part of speech tagging), named entity segmentation and NER. Labeled LDA has been used by the authors in order to outperform baseline in  for NER task. Their approach strongly relies on dictionary, contextual and orthographic features. Deep learning techniques use distributed word or character representation rather than raw one-hot vectors. Most of this research in NLP field use pretrained word embeddings such as Word2Vec , GloVe  or fastText . These low dimensional real valued dense vectors have proved to provide better representation for words compared to one-hot vector or other space vector models. The combination of word embedding along with bidirectional long-short term memory (LSTM) neural networks are examined in . The authors also propose to add a CRF layer at the end of their neural network architecture in order to preserve output tag relativity. Utilization of recurrent neural networks (RNN) provides better sequential modeling over data. However, only using sequential information does not result in major improvements because these networks tend to rely on the most recent tokens.

$x$ is in domain of [0,1] and $\sum \limits _{\forall k\in x}k=1$ holds true, while $\sum \limits _{\forall k\in IW}k\le 1$. Multimodal Fusion in our work is presented as concatenation of three feature sets extracted from words, characters and images. Unlike previous methods, our original work does not include an attention layer to remove noisy features. Instead, we stacked LSTM units from word and image feature extractors to have better results. The last layer presented at the top right side of figure FIGREF11 shows this part. In our second proposed method, we have used attention layer applied to this triplet. Our proposed attention mechanism is able to detect on which modality to increase or decrease focus. Equations DISPLAY_FORM17, and show attention mechanism related to second proposed model. Conditional Random Field is the last layer in our setup which forms the final output. The same implementation explained in eq. DISPLAY_FORM5 is used for our method.

Sometimes different entities are called exactly the same, for example "Micheal Jordan" refers to a basketball player and also a computer scientist in the field of artificial intelligence. The only thing that divides both of these is the context in which entity appeared. If the context refers to something related to AI, the reader can conclude "Micheal Jordan" is the scientist, and if the context is refers to sports and basketball then he is the basketball player. The task of distinguishing between different named entities that appear to have the same textual appearance is called named entity disambiguation. There is more useful data on the subject rather than on plain text. For example images and visual data are more descriptive than just text for tasks such as named entity recognition and disambiguation  while some methods only use the textual data . The provided extra information is closely related to the textual data. As a clear example, figure FIGREF1 shows a tweet containing an image. The combination of these multimodal data in order to achieve better performance in NLP related tasks is a promising alternative explored recently.

Introduction	A common social media delivery system such as Twitter supports various media types like video, image and text. This media allows users to share their short posts called Tweets. Users are able to share their tweets with other users that are usually following the source user. Hovewer there are rules to protect the privacy of users from unauthorized access to their timeline . The very nature of user interactions in Twitter micro-blogging social media is oriented towards their daily life, first witness news-reporting and engaging in various events (sports, political stands etc.). According to studies, news in twitter is propagated and reported faster than conventional news media . Thus, extracting first hand news and entities occurring in this fast and versatile online media gives valuable information. However, abridged and noisy content of Tweets makes it even more difficult and challenging for tasks such as named entity recognition and information retrieval . The task of tracking and recovering information from social media posts is a concise definition of information retrieval in social media , . However many challenges are blocking useful solutions to this issue, namely, the noisy nature of user generated content and the perplexity of words used in short posts.

An NLP task such as named entity recognition in social media is a most challenging task because users tend to invent, mistype and epitomize words. Sometimes these words correspond to named entities which makes the recognition task even more difficult . In some cases, the context that carries the entity (surrounding words and related image) is more descriptive than the entity word presentation . To find a solution to the issues at hand, and keeping multimodal data in mind, recognition of named entities from social media has become a research interest which utilizes image compared to NER task in a conventional text. Researchers in this field have tried to propose multimodal architectures based on deep neural networks with multimodal input that are capable of combining text and image , , . In this paper we draw a better solution in terms of performance by proposing a new novel method called CWI (Character-Word-Image model). We used multimodal deep neural network to overcome the NER task in micro-blogging social media.

Related Work ::: Multimodal Named Entity Recognition	Multimodal learning has become an emerging research interest and with the rise of deep learning techniques, it has become more visible in different research areas ranging from medical imaging to image segmentation and natural language processing , , , , , , , , , , , , , , , , , , . On the other hand, very little research has been focused on the extraction of named entities with joint image and textual data concerning short and noisy content , , ,  while several studies have been explored in textual named entity recognition using neural models , , , , , , , . State-of-the-art methods have shown acceptable evaluation on structured and well formatted short texts. Techniques based on deep learning such as utilization of convolutional neural networks , , recurrent neural networks  and long short term memory neural networks ,  are aimed to solve NER problem. The multimodal named entity recognizers can be categorized in two categories based on the tasks at hand, one tries to improve NER task with utilization of visual data , , , and the other tries to give further information about the task at hand such as disambiguation of named entities . We will refer to both of these tasks as MNER.

Related Work	Many algorithms and methods have been proposed to detect, classify or extract information from single type of data such as audio, text, image etc. However, in the case of social media, data comes in a variety of types such as text, image, video or audio in a bounded style. Most of the time, it is very common to caption a video or image with textual information. This information about the video or image can refer to a person, location etc. From a multimodal learning perspective, jointly computing such data is considered to be more valuable in terms of representation and evaluation. Named entity recognition task, on the other hand, is the task of recognizing named entities from a sentence or group of sentences in a document format. Named entity is formally defined as a word or phrase that clearly identifies an item from set of other similar items , . Equation DISPLAY_FORM2 expresses a sequence of tokens. From this equation, the NER task is defined as recognition of tokens that correspond to interesting items. These items from natural language processing perspective are known as named entity categories; BIO2 proposes four major categories, namely, organization, person, location and miscellaneous .

The rest of the paper is organized as follows: section SECREF2 provides an insight view of previous methods; section SECREF3 describes the method we propose; section SECREF4 shows experimental evaluation and test results; finally, section SECREF5 concludes the whole article.

From the biomedical domain, gene, protein, drug and disease names are known as named entities , . Output of NER task is formulated in . $I_s\in [1,N]$ and $I_e\in [1,N]$ is the start and end indices of each named entity and $t$ is named entity type . BIO2 tagging for named entity recognition is defined in equation . Table TABREF3 shows BIO2 tags and their respective meanings; B and I indicate beginning and inside of the entity respectively, while O shows the outside of it. Even though many tagging standards have been proposed for NER task, BIO is the foremost accepted by many real world applications . A named entity recognizer gets $s$ as input and provides entity tags for each token. This sequential process requires information from the whole sentence rather than only tokens and for that reason, it is also considered to be a sequence tagging problem. Another analogous problem to this issue is part of speech tagging and some methods are capable of doing both. However, in cases where noise is present and input sequence has linguistic typos, many methods fail to overcome the problem.

Experimental Evaluation ::: Evaluation Results	Table TABREF20 presents evaluation results of our proposed models. Compared to other state of the art methods, our first proposed model shows $1\%$ improvement on f1 score. The effect of different word embedding sizes on our proposed method is presented in TABREF26. Sensitivity to TD+SineRelu+GN is presented in tab. TABREF28.

The Proposed Approach	In the present work, we propose a new multimodal deep approach (CWI) that is able to handle noise by co-learning semantics from three modalities, character, word and image. Our method is composed of three parts, convolutional character embedding, joint word embedding (fastText-GloVe) and InceptionV3 image feature extraction , , . Figure FIGREF11 shows CWI architecture in more detail. Character Feature Extraction shown in the left part of figure FIGREF11 is a composition of six layers. Each sequence of words from a single tweet, $\langle w_1, w_2, \dots , w_n \rangle $ is converted to a sequence of character representation $\langle [c_{(0,0)}, c_{(0,1)}, \dots , c_{(0,k)}], \dots , [c_{(n,0)}, c_{(n,1)}, \dots , c_{(n,k)}] \rangle $ and in order to apply one dimensional convolution, it is required to be in a fixed length. $k$ shows the fixed length of the character sequence representing each word.

A multimodal deep learning approach for named entity recognition from social media	Named Entity Recognition (NER) from social media posts is a challenging task. User generated content which forms the nature of social media, is noisy and contains grammatical and linguistic errors. This noisy content makes it much harder for tasks such as named entity recognition. However some applications like automatic journalism or information retrieval from social media, require more information about entities mentioned in groups of social media posts. Conventional methods applied to structured and well typed documents provide acceptable results while compared to new user generated media, these methods are not satisfactory. One valuable piece of information about an entity is the related image to the text. Combining this multimodal data reduces ambiguity and provides wider information about the entities mentioned. In order to address this issue, we propose a novel deep learning approach utilizing multimodal deep learning. Our solution is able to provide more accurate results on named entity recognition task. Experimental results, namely the precision, recall and F1 score metrics show the superiority of our work compared to other state-of-the-art NER solutions.

Question: What datasets did they use?
Answer:  a refined collection of tweets gathered from twitter