SynonymNet: Multi-context Bilateral Matching for Entity Synonyms

1 INTRODUCTION . Discovering synonymous entities from a massive corpus is an indispensable task for automated knowledge discovery . For each entity , its synonyms refer to the entities that can be used interchangeably under certain contexts . For example , Clogged Nose and Nasal Congestion are synonyms relative to the context in which they are mentioned . Given two entities , the synonym discovery task determines how likely these two entities are synonym with each other . The main goal of synonym discovery is to learn a metric that distinguishes synonym entities from non-synonym ones . The synonym discovery task is challenging to deal with , a part of which due to the various entity expressions . For example , U.S.A/ United States of America/ United States/ U.S. refer to the same entity but expressed quite differently . Recent works on synonym discovery focus on learning the similarity from entities and their character-level features ( Neculoiu et al. , 2016 ; Mueller & Thyagarajan , 2016 ) . These methods work well for synonyms that share a lot of character-level features like airplane/ aeroplane or an entity and its abbreviation like Acquired Immune Deficiency Syndrome/ AIDS . However , a much larger number of synonym entities in the real world do not share a lot of character-level features , such as JD/ law degree , or clogged nose/ nasal congestion . With only character-level features being used , these models hardly obtain the ability to discriminate entities that share similar semantics but are not alike verbatim . Context information is helpful in indicating entity synonymity , as the meaning of an entity can be better reflected by the contexts in which it appears . Modeling the context for entity synonym usually suffers from following challenges : 1 ) Semantic Structure . Context , as a snippet of natural language sentence , is essentially semantically structured . Some existing models encode the semantic structures in the contexts implicitly during the entity representation learning ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Peters et al. , 2018 ) . The context-aware entity representations embody meaningful semantics : entities with similar contexts are likely to live in proximity in the embedding space . Some other works extract and model contexts in an explicit manner with structured annotations . Structured annotations such as dependency parsing ( Qu et al. , 2017 ) , user click information ( Wei et al. , 2009 ) , or signed heterogeneous graphs ( Ren & Cheng , 2015 ) are introduced to guide synonym discovery . 2 ) Diverse Contexts . An entity can be mentioned under a wide range of circumstances . Previous works on context-based synonym discovery either focus on entity information only ( Neculoiu et al. , 2016 ; Mueller & Thyagarajan , 2016 ) , or use a single piece of context for each entity ( Liao et al. , 2017 ; Qu et al. , 2017 ) to learn a similarity function for entity matching . While in practice , similar context is only a sufficient but not necessary condition for context matching . Notably , in some domains such as medical , the context expression preference varies a lot from individuals . For example , sinus congestion is usually referred by medical professionals in the medical literature , while patients often use stuffy nose on social media . It is not practical to assume that each piece of context is equally informative to represent the meaning of an entity : a context may contribute differently when matched with different contexts of other entities . Thus it is imperative to focus on multiple pieces of contexts with a dynamic matching schema for accuracy and robustness . In light of these challenges , we propose a framework to discover synonym entities from a massive corpus without additional structured annotation . Candidate entities are obtained from a massive text corpus unsupervisely . A novel neural network model SYNONYMNET is proposed to detect entity synonyms based on two given entities via a bilateral matching among multiple pieces of contexts in which each entity appears . A leaky unit is designed to explicitly alleviate the noises from uninformative context during the matching process . The contribution of this work is summarized as follows : • We propose SYNONYMNET , a context-aware bilateral matching model to detect entity synonyms . SYNONYMNET utilizes multiple pieces of contexts in which each entity appears , and a bilateral matching schema with leaky units to determine entity synonymity . • We introduce a synonym discovery framework that adopts SYNONYMNET to obtain synonym entities from a free-text corpus without additional structured annotation . • Experiments on generic and domain-specific real-world datasets in English and Chinese demonstrate the effectiveness of the proposed model for synonym discovery . 2 SYNONYMNET . We introduce SYNONYMNET , our proposed model that detects whether or not two entities are synonyms to each other based on a bilateral matching between multiple pieces of contexts in which entities appear . Figure 1 gives an overview of the proposed model . 2.1 CONTEXT RETRIEVER . For each entity e , the context retriever randomly fetches P pieces of contexts from the corpus D in which the entity appears . We denote the retrieved contexts for e as a set C = { c0 , c1 , ... , cP } , where P is the number of context pieces . Each piece of context cp ∈ C contains a sequence of words cp = ( w ( 0 ) p , w ( 1 ) p , ... , w ( T ) p ) , where T is the length of the context , which varies from one instance to another . w ( t ) p is the t-th word in the p-th context retrieved for an entity e . 2.2 CONFLUENCE CONTEXT ENCODER . For the p-th context cp , an encoder tries to learn a continuous vector that represents the context . For example , a recurrent neural network ( RNN ) such as a bidirectional LSTM ( Bi-LSTM ) ( Hochreiter & Schmidhuber , 1997 ) can be applied to sequentially encode the context into hidden states : → h ( t ) p = LSTMfw ( w ( t ) p , → h ( t−1 ) p ) , ← h ( t ) p = LSTMbw ( w ( t ) p , ← h ( t+1 ) p ) , ( 1 ) where w ( t ) p is the word embedding vector used for the word w ( t ) p . We could concatenate the last hidden state → h ( T ) p in the forward LSTMfw with the first hidden state ← h ( 0 ) p from the backward LSTMbw to obtain the context vector hp for cp : hp = [ → h ( T ) p , ← h ( 0 ) p ] . However , such approach does not explicitly consider the location where the entity is mentioned in the context . As the context becomes longer , it is getting risky to simply rely on the gate functions of LSTM to properly encode the context . We introduce an encoder architecture that models contexts for synonym discovery , namely the confluence context encoder . The confluence context encoder learns to encode the local information around the entity from the raw context , without utilizing additional structured annotations . It focuses on both forward and backward directions . However , the encoding process for each direction ceases immediately after it goes beyond the entity word in the context : hp = [ → h ( te ) p , ← h ( te ) p ] , where te is the index of the entity word e in the context and hp ∈ R1×dCE . By doing this , the confluence context encoder summarizes the context while explicitly considers the entity ’ s location in the context , where no additional computation cost is introduced . Comparing with existing works for context modeling ( Cambria et al. , 2018 ) where the left context and right context are modeled separately , but with the entity word being discarded , the confluence context encoder preserves entity mention information as well as the inter-dependencies between the left and right contexts . 2.3 BILATERAL MATCHING WITH LEAKY UNIT . Considering the base case , where we want to identify whether or not two entities , say e and k , are synonyms with each other , we propose to find the consensus information from multiple pieces of contexts via a bilateral matching schema . Recall that for entity e , P pieces of contexts H = { h1 , h2 , ... , hP } are randomly fetched and encoded . And for entity k , we denote Q pieces of contexts being fetched and encoded as G = { g1 , g2 , ... , gQ } . Instead of focusing on a single piece of context to determine entity synonymity , we adopt a bilateral matching between multiple pieces of encoded contexts for both accuracy and robustness . H→G matching phrase : For each hp in H and gq in G , the matching score mp→q is calculated as : mp→q = exp ( hpWBMg T q ) ∑ p′∈P exp ( hp′WBMg T q ) , where WBM ∈ RdCE×dCE is a bi-linear weight matrix . Similarly , the H←G matching phrase considers how much each context gq ∈ G could be useful to hp ∈ H : mp←q = exp ( gqWBMh T p ) ∑ q′∈Q exp ( gq′WBMh T p ) . Note that P × Q matching needs to be conducted in total for each entity pair . We write the equations for each hp ∈ H and gq ∈ G for clarity . Regarding the implementation , the bilateral matching can be easily written and effectively computed in a matrix form , where a matrix multiplication is used HWBMGT ∈ RP×Q where H ∈ RP×DCE and G ∈ RQ×DCE . The matching score matrix M can be obtained by taking softmax on the HWBMGT matrix over certain axis ( over 0-axis for Mp→q , 1-axis for Mp←q ) . Not all contexts are informative during the matching for two given entities . For example , some contexts may contain intricate contextual information even if they mention the entity explicitly . In this work , we introduce a leaky unit during the bilateral matching , so that uninformative contexts can be routed via the leaky unit rather than forced to be matched with any informative contexts . The leaky unit is a domain-dependent vector l ∈ R1×dCE learned with the model . For simplicity , we keep l as a zero vector . If we use the H→G matching phrase as an example , the matching score from the leaky unit l to the q-th encoded context in gq is : ml→q = exp ( lWBMg T q ) exp ( lWBMgTq ) + ∑ p′∈P exp ( hp′WBMgTq ) . ( 2 ) Then , if there is any uninformative context in H , say the p̃-th encoded context , hp̃ will contribute less when matched with gq due to the leaky effect : when hp̃ is less informative than the leaky unit l. mp̃→q = exp ( hp̃WBMg T q ) exp ( lWBMgTq ) + ∑ p′∈P exp ( hp′WBMgTq ) . ( 3 ) 2.4 CONTEXT AGGREGATION . The informativeness of a context for an entity should not be a fixed value : it heavily depends on the other entity and the other entity ’ s contexts that we are comparing with . The bilateral matching scores indicate the matching among multiple pieces of encoded contexts for two entities . For each piece of encoded context , say gq for the entity k , we use the highest matched score with its counterpart as the relative informativeness score of gq to k , denote as aq = max ( mp→q|p ∈ P ) . Then , we aggregate multiple pieces of encoded contexts for each entity to a global context based on the relative informativeness scores : for entity e : h̄ = ∑ p∈P aphp , for entity k : ḡ = ∑ q∈Q aqgq . ( 4 ) Note that due to the leaky effect , less informative contexts are not forced to be heavily involved during the aggregation : the leaky unit may be more competitive than contexts that are less informative , thus assigned with larger matching scores . However , as the leaky unit is not used for aggregation , scores on informative contexts become more salient during context aggregation .

Summary: This paper studies the problem of identifying (discovering) synonymous entities. The paper proposes using the "contexts" of the entities as they occur in associated text corpora (e.g. Wiki) in the proposed neural-network based embedding approach for this task. The key novelties of the approach lie in the "matching" system used, where contexts of one entity are matched with that for the other entity to see how well they align with each other (which effectively determines the similarity of the two entities). Experiments are conducted on three different datasets to show the efficacy of the proposed approach.