prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What evaluation metrics are looked at for classification tasks?
Context 1: Introduction
Performance appraisal (PA) is an important HR process, particularly for modern organizations that crucially depend on the skills and expertise of their workforce  The PA process enables an organization to periodically measure and evaluate every employee's performance  It also provides a mechanism to link the goals established by the organization to its each employee's day-to-day activities and performance  Design and analysis of PA processes is a lively area of research within the HR community BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3  
The PA process in any modern organization is nowadays implemented and tracked through an IT system (the PA system) that records the interactions that happen in various steps  Availability of this data in a computer-readable database opens up opportunities to analyze it using automated statistical, data-mining and text-mining techniques, to generate novel and actionable insights / patterns and to help in improving the quality and effectiveness of the PA process BIBREF4 , BIBREF5 , BIBREF6   Automated analysis of large-scale PA data is now facilitated by technological and algorithmic advances, and is becoming essential for large organizations containing thousands of geographically distributed employees handling a wide variety of roles and tasks 
A typical PA process involves purposeful multi-step multi-modal communication between employees, their supervisors and their peers  In most PA processes, the communication includes the following steps: (i) in self-appraisal, an employee records his/her achievements, activities, tasks handled etc ; (ii) in supervisor assessment, the supervisor provides the criticism, evaluation and suggestions for improvement of performance etc ; and (iii) in peer feedback (aka INLINEFORM0 view), the peers of the employee provide their feedback  There are several business questions that managers are interested in  Examples:
In this paper, we develop text mining techniques that can automatically produce answers to these questions  Since the intended users are HR executives, ideally, the techniques should work with minimum training data and experimentation with parameter setting  These techniques have been implemented and are being used in a PA system in a large multi-national IT company 
The rest of the paper is organized as follows  Section SECREF2 summarizes related work  Section SECREF3 summarizes the PA dataset used in this paper  Section SECREF4 applies sentence classification algorithms to automatically discover three important classes of sentences in the PA corpus viz , sentences that discuss strengths, weaknesses of employees and contain suggestions for improving her performance  Section SECREF5 considers the problem of mapping the actual targets mentioned in strengths, weaknesses and suggestions to a fixed set of attributes  In Section SECREF6 , we discuss how the feedback from peers for a particular employee can be summarized  In Section SECREF7 we draw conclusions and identify some further work 
Related Work
We first review some work related to sentence classification  Semantically classifying sentences (based on the sentence's purpose) is a much harder task, and is gaining increasing attention from linguists and NLP researchers  McKnight and Srinivasan BIBREF7 and Yamamoto and Takagi BIBREF8 used SVM to classify sentences in biomedical abstracts into classes such as INTRODUCTION, BACKGROUND, PURPOSE, METHOD, RESULT, CONCLUSION  Cohen et al  BIBREF9 applied SVM and other techniques to learn classifiers for sentences in emails into classes, which are speech acts defined by a verb-noun pair, with verbs such as request, propose, amend, commit, deliver and nouns such as meeting, document, committee; see also BIBREF10   Khoo et al  BIBREF11 uses various classifiers to classify sentences in emails into classes such as APOLOGY, INSTRUCTION, QUESTION, REQUEST, SALUTATION, STATEMENT, SUGGESTION, THANKING etc  Qadir and Riloff BIBREF12 proposes several filters and classifiers to classify sentences on message boards (community QA systems) into 4 speech acts: COMMISSIVE (speaker commits to a future action), DIRECTIVE (speaker expects listener to take some action), EXPRESSIVE (speaker expresses his or her psychological state to the listener), REPRESENTATIVE (represents the speaker's belief of something)  Hachey and Grover BIBREF13 used SVM and maximum entropy classifiers to classify sentences in legal documents into classes such as FACT, PROCEEDINGS, BACKGROUND, FRAMING, DISPOSAL; see also BIBREF14   Deshpande et al  BIBREF15 proposes unsupervised linguistic patterns to classify sentences into classes SUGGESTION, COMPLAINT 
There is much work on a closely related problem viz , classifying sentences in dialogues through dialogue-specific categories called dialogue acts BIBREF16 , which we will not review here  Just as one example, Cotterill BIBREF17 classifies questions in emails into the dialogue acts of YES_NO_QUESTION, WH_QUESTION, ACTION_REQUEST, RHETORICAL, MULTIPLE_CHOICE etc 
We could not find much work related to mining of performance appraisals data  Pawar et al  BIBREF18 uses kernel-based classification to classify sentences in both performance appraisal text and product reviews into classes SUGGESTION, APPRECIATION, COMPLAINT  Apte et al  BIBREF6 provides two algorithms for matching the descriptions of goals or tasks assigned to employees to a standard template of model goals  One algorithm is based on the co-training framework and uses goal descriptions and self-appraisal comments as two separate perspectives  The second approach uses semantic similarity under a weak supervision framework  Ramrakhiyani et al  BIBREF5 proposes label propagation algorithms to discover aspects in supervisor assessments in performance appraisals, where an aspect is modelled as a verb-noun pair (e g  conduct training, improve coding) 
Dataset
In this paper, we used the supervisor assessment and peer feedback text produced during the performance appraisal of 4528 employees in a large multi-national IT company  The corpus of supervisor assessment has 26972 sentences  The summary statistics about the number of words in a sentence is: min:4 max:217 average:15 5 STDEV:9 2 Q1:9 Q2:14 Q3:19 
Sentence Classification
The PA corpus contains several classes of sentences that are of interest  In this paper, we focus on three important classes of sentences viz , sentences that discuss strengths (class STRENGTH), weaknesses of employees (class WEAKNESS) and suggestions for improving her performance (class SUGGESTION)  The strengths or weaknesses are mostly about the performance in work carried out, but sometimes they can be about the working style or other personal qualities  The classes WEAKNESS and SUGGESTION are somewhat overlapping; e g , a suggestion may address a perceived weakness  Following are two example sentences in each class 
STRENGTH:
WEAKNESS:
SUGGESTION:
Several linguistic aspects of these classes of sentences are apparent  The subject is implicit in many sentences  The strengths are often mentioned as either noun phrases (NP) with positive adjectives (Excellent technology leadership) or positive nouns (engineering strength) or through verbs with positive polarity (dedicated) or as verb phrases containing positive adjectives (delivers innovative solutions)  Similarly for weaknesses, where negation is more frequently used (presentations are not his forte), or alternatively, the polarities of verbs (avoid) or adjectives (poor) tend to be negative  However, sometimes the form of both the strengths and weaknesses is the same, typically a stand-alone sentiment-neutral NP, making it difficult to distinguish between them; e g , adherence to timing or timely closure  Suggestions often have an imperative mood and contain secondary verbs such as need to, should, has to  Suggestions are sometimes expressed using comparatives (better process compliance)  We built a simple set of patterns for each of the 3 classes on the POS-tagged form of the sentences  We use each set of these patterns as an unsupervised sentence classifier for that class  If a particular sentence matched with patterns for multiple classes, then we have simple tie-breaking rules for picking the final class  The pattern for the STRENGTH class looks for the presence of positive words / phrases like takes ownership, excellent, hard working, commitment, etc  Similarly, the pattern for the WEAKNESS class looks for the presence of negative words / phrases like lacking, diffident, slow learner, less focused, etc  The SUGGESTION pattern not only looks for keywords like should, needs to but also for POS based pattern like “a verb in the base form (VB) in the beginning of a sentence” 
We randomly selected 2000 sentences from the supervisor assessment corpus and manually tagged them (dataset D1)  This labelled dataset contained 705, 103, 822 and 370 sentences having the class labels STRENGTH, WEAKNESS, SUGGESTION or OTHER respectively  We trained several multi-class classifiers on this dataset  Table TABREF10 shows the results of 5-fold cross-validation experiments on dataset D1  For the first 5 classifiers, we used their implementation from the SciKit Learn library in Python (scikit-learn org)  The features used for these classifiers were simply the sentence words along with their frequencies  For the last 2 classifiers (in Table TABREF10 ), we used our own implementation  The overall accuracy for a classifier is defined as INLINEFORM0 , where the denominator is 2000 for dataset D1  Note that the pattern-based approach is unsupervised i e , it did not use any training data  Hence, the results shown for it are for the entire dataset and not based on cross-validation 
Comparison with Sentiment Analyzer
We also explored whether a sentiment analyzer can be used as a baseline for identifying the class labels STRENGTH and WEAKNESS  We used an implementation of sentiment analyzer from TextBlob to get a polarity score for each sentence  Table TABREF13 shows the distribution of positive, negative and neutral sentiments across the 3 class labels STRENGTH, WEAKNESS and SUGGESTION  It can be observed that distribution of positive and negative sentiments is almost similar in STRENGTH as well as SUGGESTION sentences, hence we can conclude that the information about sentiments is not much useful for our classification problem 
Discovering Clusters within Sentence Classes
After identifying sentences in each class, we can now answer question (1) in Section SECREF1   From 12742 sentences predicted to have label STRENGTH, we extract nouns that indicate the actual strength, and cluster them using a simple clustering algorithm which uses the cosine similarity between word embeddings of these nouns  We repeat this for the 9160 sentences with predicted label WEAKNESS or SUGGESTION as a single class  Tables TABREF15 and TABREF16 show a few representative clusters in strengths and in weaknesses, respectively  We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms  Carrot2 Lingo discovered 167 clusters and also assigned labels to these clusters  We then generated 167 clusters using CLUTO as well  CLUTO does not generate cluster labels automatically, hence we used 5 most frequent words within the cluster as its labels  Table TABREF19 shows the largest 5 clusters by both the algorithms  It was observed that the clusters created by CLUTO were more meaningful and informative as compared to those by Carrot2 Lingo  Also, it was observed that there is some correspondence between noun clusters and sentence clusters  E g  the nouns cluster motivation expertise knowledge talent skill (Table TABREF15 ) corresponds to the CLUTO sentence cluster skill customer management knowledge team (Table TABREF19 )  But overall, users found the nouns clusters to be more meaningful than the sentence clusters 
PA along Attributes
In many organizations, PA is done from a predefined set of perspectives, which we call attributes  Each attribute covers one specific aspect of the work done by the employees  This has the advantage that we can easily compare the performance of any two employees (or groups of employees) along any given attribute  We can correlate various performance attributes and find dependencies among them  We can also cluster employees in the workforce using their supervisor ratings for each attribute to discover interesting insights into the workforce  The HR managers in the organization considered in this paper have defined 15 attributes (Table TABREF20 )  Each attribute is essentially a work item or work category described at an abstract level  For example, FUNCTIONAL_EXCELLENCE covers any tasks, goals or activities related to the software engineering life-cycle (e g , requirements analysis, design, coding, testing etc ) as well as technologies such as databases, web services and GUI 
In the example in Section SECREF4 , the first sentence (which has class STRENGTH) can be mapped to two attributes: FUNCTIONAL_EXCELLENCE and BUILDING_EFFECTIVE_TEAMS  Similarly, the third sentence (which has class WEAKNESS) can be mapped to the attribute INTERPERSONAL_EFFECTIVENESS and so forth  Thus, in order to answer the second question in Section SECREF1 , we need to map each sentence in each of the 3 classes to zero, one, two or more attributes, which is a multi-class multi-label classification problem 
We manually tagged the same 2000 sentences in Dataset D1 with attributes, where each sentence may get 0, 1, 2, etc  up to 15 class labels (this is dataset D2)  This labelled dataset contained 749, 206, 289, 207, 91, 223, 191, 144, 103, 80, 82, 42, 29, 15, 24 sentences having the class labels listed in Table TABREF20 in the same order  The number of sentences having 0, 1, 2, or more than 2 attributes are: 321, 1070, 470 and 139 respectively  We trained several multi-class multi-label classifiers on this dataset  Table TABREF21 shows the results of 5-fold cross-validation experiments on dataset D2 
Precision, Recall and F-measure for this multi-label classification are computed using a strategy similar to the one described in BIBREF21   Let INLINEFORM0 be the set of predicted labels and INLINEFORM1 be the set of actual labels for the INLINEFORM2 instance  Precision and recall for this instance are computed as follows: INLINEFORM3
It can be observed that INLINEFORM0 would be undefined if INLINEFORM1 is empty and similarly INLINEFORM2 would be undefined when INLINEFORM3 is empty  Hence, overall precision and recall are computed by averaging over all the instances except where they are undefined  Instance-level F-measure can not be computed for instances where either precision or recall are undefined  Therefore, overall F-measure is computed using the overall precision and recall 
Summarization of Peer Feedback using ILP
The PA system includes a set of peer feedback comments for each employee  To answer the third question in Section SECREF1 , we need to create a summary of all the peer feedback comments about a given employee  As an example, following are the feedback comments from 5 peers of an employee 
The individual sentences in the comments written by each peer are first identified and then POS tags are assigned to each sentence  We hypothesize that a good summary of these multiple comments can be constructed by identifying a set of important text fragments or phrases  Initially, a set of candidate phrases is extracted from these comments and a subset of these candidate phrases is chosen as the final summary, using Integer Linear Programming (ILP)  The details of the ILP formulation are shown in Table TABREF36   As an example, following is the summary generated for the above 5 peer comments 
humble nature, effective communication, technical expertise, always supportive, vast knowledge
Following rules are used to identify candidate phrases:
Various parameters are used to evaluate a candidate phrase for its importance  A candidate phrase is more important:
A complete list of parameters is described in detail in Table TABREF36  
There is a trivial constraint INLINEFORM0 which makes sure that only INLINEFORM1 out of INLINEFORM2 candidate phrases are chosen  A suitable value of INLINEFORM3 is used for each employee depending on number of candidate phrases identified across all peers (see Algorithm SECREF6 )  Another set of constraints ( INLINEFORM4 to INLINEFORM5 ) make sure that at least one phrase is selected for each of the leadership attributes  The constraint INLINEFORM6 makes sure that multiple phrases sharing the same headword are not chosen at a time  Also, single word candidate phrases are chosen only if they are adjectives or nouns with lexical category noun attribute  This is imposed by the constraint INLINEFORM7   It is important to note that all the constraints except INLINEFORM8 are soft constraints, i e  there may be feasible solutions which do not satisfy some of these constraints  But each constraint which is not satisfied, results in a penalty through the use of slack variables  These constraints are described in detail in Table TABREF36  
The objective function maximizes the total importance score of the selected candidate phrases  At the same time, it also minimizes the sum of all slack variables so that the minimum number of constraints are broken 
INLINEFORM0 : No  of candidate phrases INLINEFORM1 : No  of phrases to select as part of summary
INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8
INLINEFORM0 and INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6
INLINEFORM0 (For determining number of phrases to select to include in summary)
Evaluation of auto-generated summaries
We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded  Also, for each employee, a manual summary was generated by an HR personnel  The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score  For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package  A common parameter which is required by all these algorithms is number of sentences keep in the final summary  ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases  Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3  Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries  The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference  Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries 
Conclusions and Further Work
In this paper, we presented an analysis of the text generated in Performance Appraisal (PA) process in a large multi-national IT company  We performed sentence classification to identify strengths, weaknesses and suggestions for improvements found in the supervisor assessments and then used clustering to discover broad categories among them  As this is non-topical classification, we found that SVM with ADWS kernel BIBREF18 produced the best results  We also used multi-class multi-label classification techniques to match supervisor assessments to predefined broad perspectives on performance  Logistic Regression classifier was observed to produce the best results for this topical classification  Finally, we proposed an ILP-based summarization technique to produce a summary of peer feedback comments for a given employee and compared it with manual summaries 
The PA process also generates much structured data, such as supervisor ratings  It is an interesting problem to compare and combine the insights from discovered from structured data and unstructured text  Also, we are planning to automatically discover any additional performance attributes to the list of 15 attributes currently used by HR 
Context 3: Introduction
Chinese word segmentation (CWS) is a task for Chinese natural language process to delimit word boundary  CWS is a basic and essential task for Chinese which is written without explicit word delimiters and different from alphabetical languages like English  BIBREF0 treats Chinese word segmentation (CWS) as a sequence labeling task with character position tags, which is followed by BIBREF1, BIBREF2, BIBREF3  Traditional CWS models depend on the design of features heavily which effects the performance of model  To minimize the effort in feature engineering, some CWS models BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11 are developed following neural network architecture for sequence labeling tasks BIBREF12  Neural CWS models perform strong ability of feature representation, employing unigram and bigram character embedding as input and approach good performance 
The CWS task is often modeled as one graph model based on a scoring model that means it is composed of two parts, one part is an encoder which is used to generate the representation of characters from the input sequence, the other part is a decoder which performs segmentation according to the encoder scoring  Table TABREF1 summarizes typical CWS models according to their decoding ways for both traditional and neural models  Markov models such as BIBREF13 and BIBREF4 depend on the maximum entropy model or maximum entropy Markov model both with a Viterbi decoder  Besides, conditional random field (CRF) or Semi-CRF for sequence labeling has been used for both traditional and neural models though with different representations BIBREF2, BIBREF15, BIBREF10, BIBREF17, BIBREF18  Generally speaking, the major difference between traditional and neural network models is about the way to represent input sentences 
Recent works about neural CWS which focus on benchmark dataset, namely SIGHAN Bakeoff BIBREF21, may be put into the following three categories roughly 
Encoder  Practice in various natural language processing tasks has been shown that effective representation is essential to the performance improvement  Thus for better CWS, it is crucial to encode the input character, word or sentence into effective representation  Table TABREF2 summarizes regular feature sets for typical CWS models including ours as well  The building blocks that encoders use include recurrent neural network (RNN) and convolutional neural network (CNN), and long-term memory network (LSTM) 
Graph model  As CWS is a kind of structure learning task, the graph model determines which type of decoder should be adopted for segmentation, also it may limit the capability of defining feature, as shown in Table 2, not all graph models can support the word features  Thus recent work focused on finding more general or flexible graph model to make model learn the representation of segmentation more effective as BIBREF9, BIBREF11 
External data and pre-trained embedding  Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself  Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23  SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21  In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement 
Shown in Table TABREF1, different decoders have particular decoding algorithms to match the respective CWS models  Markov models and CRF-based models often use Viterbi decoders with polynomial time complexity  In general graph model, search space may be too large for model to search  Thus it forces graph models to use an approximate beam search strategy  Beam search algorithm has a kind low-order polynomial time complexity  Especially, when beam width $b$=1, the beam search algorithm will reduce to greedy algorithm with a better time complexity $O(Mn)$ against the general beam search time complexity $O(Mnb^2)$, where $n$ is the number of units in one sentences, $M$ is a constant representing the model complexity  Greedy decoding algorithm can bring the fastest speed of decoding while it is not easy to guarantee the precision of decoding when the encoder is not strong enough 
In this paper, we focus on more effective encoder design which is capable of offering fast and accurate Chinese word segmentation with only unigram feature and greedy decoding  Our proposed encoder will only consist of attention mechanisms as building blocks but nothing else  Motivated by the Transformer BIBREF24 and its strength of capturing long-range dependencies of input sentences, we use a self-attention network to generate the representation of input which makes the model encode sentences at once without feeding input iteratively  Considering the weakness of the Transformer to model relative and absolute position information directly BIBREF25 and the importance of localness information, position information and directional information for CWS, we further improve the architecture of standard multi-head self-attention of the Transformer with a directional Gaussian mask and get a variant called Gaussian-masked directional multi-head attention  Based on the newly improved attention mechanism, we expand the encoder of the Transformer to capture different directional information  With our powerful encoder, our model uses only simple unigram features to generate representation of sentences 
For decoder which directly performs the segmentation, we use the bi-affinal attention scorer, which has been used in dependency parsing BIBREF26 and semantic role labeling BIBREF27, to implement greedy decoding on finding the boundaries of words  In our proposed model, greedy decoding ensures a fast segmentation while powerful encoder design ensures a good enough segmentation performance even working with greedy decoder together  Our model will be strictly evaluated on benchmark datasets from SIGHAN Bakeoff shared task on CWS in terms of closed test setting, and the experimental results show that our proposed model achieves new state-of-the-art 
The technical contributions of this paper can be summarized as follows 
We propose a CWS model with only attention structure  The encoder and decoder are both based on attention structure 
With a powerful enough encoder, we for the first time show that unigram (character) featues can help yield strong performance instead of diverse $n$-gram (character and word) features in most of previous work 
To capture the representation of localness information and directional information, we propose a variant of directional multi-head self-attention to further enhance the state-of-the-art Transformer encoder 
Models
The CWS task is often modelled as one graph model based on an encoder-based scoring model  The model for CWS task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation  Figure FIGREF6 is the architecture of our model  The model feeds sentence into encoder  Embedding captures the vector $e=(e_1,   ,e_n)$ of the input character sequences of $c=(c_1,   ,c_n)$  The encoder maps vector sequences of $ {e}=(e_1,  ,e_n)$ to two sequences of vector which are $ {v^b}=(v_1^b,   ,v_n^b)$ and ${v^f}=(v_1^f,   v_n^f)$ as the representation of sentences  With $v^b$ and $v^f$, the bi-affinal scorer calculates the probability of each segmentation gaps and predicts the word boundaries of input

ground_truth: Precision, Recall, F-measure, accuracy

Precision, Recall and F-measure

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Does their NER model learn NER from both text and images?
Context 2: Introduction
Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individuals  These insights are primarily hidden in unstructured forms of social media posts, such as in free-form text or images without tags  Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a critical step for building structural information, allowing for its use in personalized assistance, recommendations, advertisement, etc 
While many previous approaches BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 on NER have shown success for well-formed text in recognizing named entities via word context resolution (e g  LSTM with word embeddings) combined with character-level features (e g  CharLSTM/CNN), several additional challenges remain for recognizing named entities from extremely short and coarse text found in social media posts  For instance, short social media posts often do not provide enough textual contexts to resolve polysemous entities (e g  “monopoly is da best ", where `monopoly' may refer to a board game (named entity) or a term in economics)  In addition, noisy text includes a huge number of unknown tokens due to inconsistent lexical notations and frequent mentions of various newly trending entities (e g  “xoxo Marshmelloooo ", where `Marshmelloooo' is a mis-spelling of a known entity `Marshmello', a music producer), making word embeddings based neural networks NER models vulnerable 
To address the challenges above for social media posts, we build upon the state-of-the-art neural architecture for NER with the following two novel approaches (Figure FIGREF1 )  First, we propose to leverage auxiliary modalities for additional context resolution of entities  For example, many popular social media platforms now provide ways to compose a post in multiple modalities - specifically image and text (e g  Snapchat captions, Twitter posts with image URLs), from which we can obtain additional context for understanding posts  While “monopoly" in the previous example is ambiguous in its textual form, an accompanying snap image of a board game can help disambiguate among polysemous entities, thereby correctly recognizing it as a named entity 
Second, we also propose a general modality attention module which chooses per decoding step the most informative modality among available ones (in our case, word embeddings, character embeddings, or visual features) to extract context from  For example, the modality attention module lets the decoder attenuate the word-level signals for unknown word tokens (“Marshmellooooo" with trailing `o's) and amplifies character-level features intsead (capitalized first letter, lexical similarity to other known named entity token `Marshmello', etc ), thereby suppressing noise information (“UNK" token embedding) in decoding steps  Note that most of the previous literature in NER or other NLP tasks combine word and character-level information with naive concatenation, which is vulnerable to noisy social media posts  When an auxiliary image is available, the modality attention module determines to amplify this visual context in disambiguating polysemous entities, or to attenuate visual contexts when they are irrelevant to target named entities, selfies, etc  Note that the proposed modality attention module is distinct from how attention is used in other sequence-to-sequence literature (e g  attending to a specific token within an input sequence)  Section SECREF2 provides the detailed literature review 
Our contributions are three-fold: we propose (1) an LSTM-CNN hybrid multimodal NER network that takes as input both image and text for recognition of a named entity in text input  To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks  (2) We propose a general modality attention module that selectively chooses modalities to extract primary context from, maximizing information gain and suppressing irrelevant contexts from each modality (we treat words, characters, and images as separate modalities)  (3) We show that the proposed approaches outperform the state-of-the-art NER models (both with and without using additional visual contexts) on our new MNER dataset SnapCaptions, a large collection of informal and extremely short social media posts paired with unique images 
Related Work
Neural models for NER have been recently proposed, producing state-of-the-art performance on standard NER tasks  For example, some of the end-to-end NER systems BIBREF4 , BIBREF2 , BIBREF3 , BIBREF0 , BIBREF1 use a recurrent neural network usually with a CRF BIBREF5 , BIBREF6 for sequence labeling, accompanied with feature extractors for words and characters (CNN, LSTMs, etc ), and achieve the state-of-the-art performance mostly without any use of gazetteers information  Note that most of these work aggregate textual contexts via concatenation of word embeddings and character embeddings  Recently, several work have addressed the NER task specifically on noisy short text segments such as Tweets, etc  BIBREF7 , BIBREF8   They report performance gains from leveraging external sources of information such as lexical information (POS tags, etc ) and/or from several preprocessing steps (token substitution, etc )  Our model builds upon these state-of-the-art neural models for NER tasks, and improves the model in two critical ways: (1) incorporation of visual contexts to provide auxiliary information for short media posts, and (2) addition of the modality attention module, which better incorporates word embeddings and character embeddings, especially when there are many missing tokens in the given word embedding matrix  Note that we do not explore the use of gazetteers information or other auxiliary information (POS tags, etc ) BIBREF9 as it is not the focus of our study 
Attention modules are widely applied in several deep learning tasks BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13   For example, they use an attention module to attend to a subset within a single input (a part/region of an image, a specific token in an input sequence of tokens, etc ) at each decoding step in an encoder-decoder framework for image captioning tasks, etc  BIBREF14 explore various attention mechanisms in NLP tasks, but do not incorporate visual components or investigate the impact of such models on noisy social media data  BIBREF15 propose to use attention for a subset of discrete source samples in transfer learning settings  Our modality attention differs from the previous approaches in that we attenuate or amplifies each modality input as a whole among multiple available modalities, and that we use the attention mechanism essentially to map heterogeneous modalities in a single joint embedding space  Our approach also allows for re-use of the same model for predicting labels even when some of the modalities are missing in input, as other modalities would still preserve the same semantics in the embeddings space 
Multimodal learning is studied in various domains and applications, aimed at building a joint model that extracts contextual information from multiple modalities (views) of parallel datasets 
The most relevant task to our multimodal NER system is the task of multimodal machine translation BIBREF16 , BIBREF17 , which aims at building a better machine translation system by taking as input a sentence in a source language as well as a corresponding image  Several standard sequence-to-sequence architectures are explored (a target-language LSTM decoder that takes as input an image first)  Other previous literature include study of Canonical Correlation Analysis (CCA) BIBREF18 to learn feature correlations among multiple modalities, which is widely used in many applications  Other applications include image captioning BIBREF10 , audio-visual recognition BIBREF19 , visual question answering systems BIBREF20 , etc 
To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks 
Proposed Methods
Figure FIGREF2 illustrates the proposed multimodal NER (MNER) model  First, we obtain word embeddings, character embeddings, and visual features (Section SECREF3 )  A Bi-LSTM-CRF model then takes as input a sequence of tokens, each of which comprises a word token, a character sequence, and an image, in their respective representation (Section SECREF4 )  At each decoding step, representations from each modality are combined via the modality attention module to produce an entity label for each token ( SECREF5 )  We formulate each component of the model in the following subsections 
Notations: Let INLINEFORM0 a sequence of input tokens with length INLINEFORM1 , with a corresponding label sequence INLINEFORM2 indicating named entities (e g  in standard BIO formats)  Each input token is composed of three modalities: INLINEFORM3 for word embeddings, character embeddings, and visual embeddings representations, respectively 
Features
Similar to the state-of-the-art NER approaches BIBREF0 , BIBREF1 , BIBREF8 , BIBREF4 , BIBREF2 , BIBREF3 , we use both word embeddings and character embeddings 
Word embeddings are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics BIBREF21   Specifically, we use pre-trained embeddings from GloVE BIBREF22  
Character embeddings are obtained from a Bi-LSTM which takes as input a sequence of characters of each token, similarly to BIBREF0   An alternative approach for obtaining character embeddings is using a convolutional neural network as in BIBREF1 , but we find that Bi-LSTM representation of characters yields empirically better results in our experiments 
Visual embeddings: To extract features from an image, we take the final hidden layer representation of a modified version of the convolutional network model called Inception (GoogLeNet) BIBREF23 , BIBREF24 trained on the ImageNet dataset BIBREF25 to classify multiple objects in the scene  Our implementation of the Inception model has deep 22 layers, training of which is made possible via “network in network" principles and several dimension reduction techniques to improve computing resource utilization  The final layer representation encodes discriminative information describing what objects are shown in an image, which provide auxiliary contexts for understanding textual tokens and entities in accompanying captions 
Incorporating this visual information onto the traditional NER system is an open challenge, and multiple approaches can be considered  For instance, one may provide visual contexts only as an initial input to decoder as in some encoder-decoder image captioning systems BIBREF26   However, we empirically observe that an NER decoder which takes as input the visual embeddings at every decoding step (Section SECREF4 ), combined with the modality attention module (Section SECREF5 ), yields better results 
Lastly, we add a transform layer for each feature INLINEFORM0 before it is fed to the NER entity LSTM 
Bi-LSTM + CRF for Multimodal NER
Our MNER model is built on a Bi-LSTM and CRF hybrid model  We use the following implementation for the entity Bi-LSTM 
it = (Wxiht-1 + Wcict-1)
ct = (1-it) ct-1
+ it tanh(Wxcxt + Whcht-1)
ot = (Wxoxt + Whoht-1 + Wcoct)
ht = LSTM(xt)
= ot tanh(ct)
where INLINEFORM0 is a weighted average of three modalities INLINEFORM1 via the modality attention module, which will be defined in Section SECREF5   Bias terms for gates are omitted here for simplicity of notation 
We then obtain bi-directional entity token representations INLINEFORM0 by concatenating its left and right context representations  To enforce structural correlations between labels in sequence decoding, INLINEFORM1 is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective  y* = y p(y|h; WCRF)
p(y|h; WCRF) = t t (yt-1,yt;h) y' t t (y't-1,y't;h)
where INLINEFORM0 is a potential function, INLINEFORM1 is a set of parameters that defines the potential functions and weight vectors for label pairs ( INLINEFORM2 )  Bias terms are omitted for brevity of formulation 
The model can be trained via log-likelihood maximization for the training set INLINEFORM0 :
L(WCRF) = i p(y|h; W)
Modality Attention
The modality attention module learns a unified representation space for multiple available modalities (words, characters, images, etc ), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance  We motivate this module from the following observations 
A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, e g  INLINEFORM0 in Eq  SECREF4   However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (an all-zero vector INLINEFORM1 or a random vector INLINEFORM2 is assigned for any unknown token INLINEFORM3 , thus INLINEFORM4 or INLINEFORM5 )  While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a significant number of missing tokens 
Similarly, naive merging of textual and visual information ( INLINEFORM0 ) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities  Hence, ideally there needs a mechanism in which the model can effectively turn the switch on and off the modalities adaptive to each sample 
To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step INLINEFORM0 , and produces a soft-attended context vector INLINEFORM1 as an input token for the entity LSTM  [at(w),at(c),at(v)] = (Wm[xt(w); xt(c); xt(v)] + bm )
t(m) = (at(m))m'{w,c,v}(at(m')) m {w,c,v}
xt = m{w,c,v} t(m)xt(m)
where INLINEFORM0 is an attention vector at each decoding step INLINEFORM1 , and INLINEFORM2 is a final context vector at INLINEFORM3 that maximizes information gain for INLINEFORM4   Note that the optimization of the objective function (Eq  SECREF4 ) with modality attention (Eq  SECREF5 ) requires each modality to have the same dimension ( INLINEFORM5 ), and that the transformation via INLINEFORM6 essentially enforces each modality to be mapped into the same unified subspace, where the weighted average of which encodes discrimitive features for recognition of named entities 
When visual context is not provided with each token (as in the traditional NER task), we can define the modality attention for word and character embeddings only in a similar way: [at(w),at(c)] = (Wm[xt(w); xt(c)] + bm )
t(m) = (at(m))m'{w,c}(at(m')) m {w,c}
xt = m{w,c} t(m)xt(m)
Note that while we apply this modality attention module to the Bi-LSTM+CRF architecture (Section SECREF4 ) for its empirical superiority, the module itself is flexible and thus can work with other NER architectures or for other multimodal applications 
SnapCaptions Dataset
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC)  These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories)  Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc  All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available)  We split the dataset into train (70%), validation (15%), and test sets (15%)  The captions data have average length of 30 7 characters (5 81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22   Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc ) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities 
Baselines
Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27   We report the performance of the following state-of-the-art NER models as baselines, as well as several configurations of our proposed approach to examine contributions of each component (W: word, C: char, V: visual) 
Bi-LSTM/CRF (W only): only takes word token embeddings (Stanford GloVE) as input  The rest of the architecture is kept the same 
Bi-LSTM/CRF + Bi-CharLSTM (C only): only takes a character sequence of each word token as input  (No word embeddings)
Bi-LSTM/CRF + Bi-CharLSTM (W+C) BIBREF0 : takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM  Entity LSTM takes concatenated vectors of word and character embeddings as input tokens 
Bi-LSTM/CRF + CharCNN (W+C) BIBREF1 : uses character embeddings extracted from a CNN instead 
Bi-LSTM/CRF + CharCNN (W+C) + Multi-task BIBREF8 : trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM 
Results: SnapCaptions Dataset
Table TABREF6 shows the NER performance on the Snap Captions dataset  We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results 
Parameters: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, 150, 200, 300}, word embeddings size: {25, 50, 100, 150, 200, 300}, LSTM hidden states: {25, 50, 100, 150, 200, 300}, and INLINEFORM0 dimension: {25, 50, 100, 150, 200, 300}  We optimize the parameters with Adagrad BIBREF28 with batch size 10, learning rate 0 02, epsilon INLINEFORM1 , and decay 0 0 
Main Results: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks  In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V)  This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain  Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines BIBREF8 , BIBREF1 , BIBREF0 that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well 
Error Analysis: Table TABREF17 shows example cases where incorporation of visual contexts affects prediction of named entities  For example, the token `curry' in the caption “The curry's " is polysemous and may refer to either a type of food or a famous basketball player `Stephen Curry', and the surrounding textual contexts do not provide enough information to disambiguate it  On the other hand, visual contexts (visual tags: `parade', `urban area',    ) provide similarities to the token's distributional semantics from other training examples (snaps from “NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity  Similarly, while the text-only model erroneously predicts `Apple' in the caption “Grandma w dat lit Apple Crisp" as an organization (Apple Inc ), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit)  Trending entities (musicians or DJs such as `CID', `Duke Dumont', `Marshmello', etc ) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts  A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts 
Visualization of Modality Attention: Figure FIGREF19 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color 
For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc ) and amplifies relevant modality-based contexts in prediction of a given token  In the example of “disney word essential = coffee" with visual tags selfie, phone, person, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions  The named entities in the examples of “Beautiful night atop The Space Needle" and “Splash Mountain" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict  In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals 
For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix  For example, while WaRriOoOrs and Kooler Matic are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (capitalized first letters, similarity to known entities `Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens `WaRriOoOrs'), leading to correct predictions  This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely 
Figure FIGREF19 (b) shows the cases where the modality attention led to incorrect predictions  For example, the model predicts missing tokens HUUUGE and Shampooer incorrectly as named entities by amplifying misleading character-based contexts (capitalized first letters) or visual contexts (concert scenes, associated contexts of which often include named entities in the training dataset) 
Sensitivity to Word Embeddings Vocabulary Size: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table TABREF20   By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation  Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix  This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets 
Conclusions
We proposed a new multimodal NER (MNER: image + text) task on short social media posts  We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines  Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images  In addition, we proposed the modality attention module, a new neural mechanism which learns optimal integration of different modes of correlated information  In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations  We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information 
Context 3:  Similar as the Transformer, the encoder is an attention network with stacked self-attention and point-wise, fully connected layers while our encoder includes three independent directional encoders 
Models ::: Encoder Stacks
In the Transformer, the encoder is composed of a stack of N identical layers and each layer has one multi-head self-attention layer and one position-wise fully connected feed-forward layer  One residual connection is around two sub-layers and followed by layer normalization BIBREF24  This architecture provides the Transformer a good ability to generate representation of sentence 
With the variant of multi-head self-attention, we design a Gaussian-masked directional encoder to capture representation of different directions to improve the ability of capturing the localness information and position information for the importance of adjacent characters  One unidirectional encoder can capture information of one particular direction 
For CWS tasks, one gap of characters, which is from a word boundary, can divide one sequence into two parts, one part in front of the gap and one part in the rear of it  The forward encoder and backward encoder are used to capture information of two directions which correspond to two parts divided by the gap 
One central encoder is paralleled with forward and backward encoders to capture the information of entire sentences  The central encoder is a special directional encoder for forward and backward information of sentences  The central encoder can fuse the information and enable the encoder to capture the global information 
The encoder outputs one forward information and one backward information of each positions  The representation of sentence generated by center encoder will be added to these information directly:
where $v^{b}=(v^b_1,   ,v^b_n)$ is the backward information, $v^{f}=(v^f_1,   ,v^f_n)$ is the forward information, $r^{b}=(r^b_1,   ,r^b_n)$ is the output of backward encoder, $r^{c}=(r^c_1,   ,r^c_n)$ is the output of center encoder and $r^{f}=(r^f_1,   ,r^f_n)$ is the output of forward encoder 
Models ::: Gaussian-Masked Directional Multi-Head Attention
Similar as scaled dot-product attention BIBREF24, Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input  Here queries, keys and values are all vectors  Standard scaled dot-product attention is calculated by dotting query $Q$ with all keys $K$, dividing each values by $\sqrt{d_k}$, where $\sqrt{d_k}$ is the dimension of keys, and apply a softmax function to generate the weights in the attention:
Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention  We assume that the Gaussian weight only relys on the distance between characters 
Firstly we introduce the Gaussian weight matrix $G$ which presents the localness relationship between each two characters:
where $g_{ij}$ is the Gaussian weight between character $i$ and $j$, $dis_{ij}$ is the distance between character $i$ and $j$, $\Phi (x)$ is the cumulative distribution function of Gaussian, $\sigma $ is the standard deviation of Gaussian function and it is a hyperparameter in our method  Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0  The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters 
To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$
where $AG$ is the Gaussian-masked attention  It ensures that the relationship between two characters with long distances is weaker than adjacent characters 
The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence  For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly  The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters 
For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights:
where $pos_i$ is the position of character $c_i$  The triangular matrix for forward and backward encode are:
$\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$
Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention  With multi-head attention architecture, the representation of input can be captured by
where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head 
Models ::: Bi-affinal Attention Scorer
Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task  Different from character labeling task, gap labeling task requires information of two adjacent characters  The relationship between adjacent characters can be represented as the type of gap  The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task 
Bi-affinal attention scorer is the component that we use to label the gap  Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27  The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27  Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention  The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine 
Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units  In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by:
where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$  In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training  $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels 
In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters  Figure FIGREF22 is an example of labeling gap  The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information  The score vector of the gap is formed by the probability of being a boundary of word  Further, the model generates all boundaries using activation function in a greedy decoding way 
Experiments ::: Experimental Settings ::: Data
We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU  Table TABREF23 shows the statistics of train data  We use F-score to evaluate CWS models  To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese 
Experiments ::: Experimental Settings ::: Pre-trained Embedding
We only use unigram feature so we only trained character embeddings  Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are strong baselines model is compared to?
Context 3:  The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented  On closed test, we use embeddings initialized randomly 
Experiments ::: Experimental Settings ::: Hyperparameters
For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24  We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS)  We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2  Each training batch contains sentences with at most 4096 tokens 
Experiments ::: Experimental Settings ::: Optimizer
To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0 9$, $\beta _2=0 98$ and $\epsilon =10^{-9}$  The learning rate schedule is the same as BIBREF24:
where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup  When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases 
Experiments ::: Hardware and Implements
We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU  We implement our model in Python with Pytorch 1 0 
Experiments ::: Results
Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting  Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU  Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones 
With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result  Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU  The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff 
Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used  In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable 
However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding  Not all models may use the same method and data to pre-train  Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources  It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better 
Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU  Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs  For small corpus, the model tends to be overfitting 
Tables TABREF25 and TABREF26 also show the decoding time in different datasets  Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block 
Related Work ::: Chinese Word Segmentation
CWS is a task for Chinese natural language process to delimit word boundary  BIBREF0 for the first time formulize CWS as a sequence labeling task  BIBREF3 show that different character tag sets can make essential impact for CWS  BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art  Works of statistical CWS has built the basis for neural CWS 
Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS  BIBREF4 introduce the neural model with sliding-window based sequence labeling  BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features  BIBREF7 use LSTM to learn long distance information  BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history  BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously  BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging  BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously  BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings  BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs  BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data  BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS  BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria 
Related Work ::: Transformer
Transformer BIBREF24 is an attention-based neural machine translation model  The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38  Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer  Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer  The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer 
Scaled dot-product attention is the key component in the Transformer  The input of attention contains queries, keys, and values of input sequences  The attention is generated using queries and keys like Equation (DISPLAY_FORM11)  Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one  Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask  BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces 
Conclusion
In this paper, we propose an attention mechanism only based Chinese word segmentation model  Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps  To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention  We also extend the Transformer encoder to capture directional features  Our model uses only unigram features instead of multiple $n$-gram features in previous work  Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models 

ground_truth: Baseline models are:
- Chen et al., 2015a
- Chen et al., 2015b
- Liu et al., 2016
- Cai and Zhao, 2016
- Cai et al., 2017
- Zhou et al., 2017
- Ma et al., 2018
- Wang et al., 2019