SynonymNet: Multi-context Bilateral Matching for Entity Synonyms

1 INTRODUCTION . Discovering synonymous entities from a massive corpus is an indispensable task for automated knowledge discovery . For each entity , its synonyms refer to the entities that can be used interchangeably under certain contexts . For example , Clogged Nose and Nasal Congestion are synonyms relative to the context in which they are mentioned . Given two entities , the synonym discovery task determines how likely these two entities are synonym with each other . The main goal of synonym discovery is to learn a metric that distinguishes synonym entities from non-synonym ones . The synonym discovery task is challenging to deal with , a part of which due to the various entity expressions . For example , U.S.A/ United States of America/ United States/ U.S. refer to the same entity but expressed quite differently . Recent works on synonym discovery focus on learning the similarity from entities and their character-level features ( Neculoiu et al. , 2016 ; Mueller & Thyagarajan , 2016 ) . These methods work well for synonyms that share a lot of character-level features like airplane/ aeroplane or an entity and its abbreviation like Acquired Immune Deficiency Syndrome/ AIDS . However , a much larger number of synonym entities in the real world do not share a lot of character-level features , such as JD/ law degree , or clogged nose/ nasal congestion . With only character-level features being used , these models hardly obtain the ability to discriminate entities that share similar semantics but are not alike verbatim . Context information is helpful in indicating entity synonymity , as the meaning of an entity can be better reflected by the contexts in which it appears . Modeling the context for entity synonym usually suffers from following challenges : 1 ) Semantic Structure . Context , as a snippet of natural language sentence , is essentially semantically structured . Some existing models encode the semantic structures in the contexts implicitly during the entity representation learning ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Peters et al. , 2018 ) . The context-aware entity representations embody meaningful semantics : entities with similar contexts are likely to live in proximity in the embedding space . Some other works extract and model contexts in an explicit manner with structured annotations . Structured annotations such as dependency parsing ( Qu et al. , 2017 ) , user click information ( Wei et al. , 2009 ) , or signed heterogeneous graphs ( Ren & Cheng , 2015 ) are introduced to guide synonym discovery . 2 ) Diverse Contexts . An entity can be mentioned under a wide range of circumstances . Previous works on context-based synonym discovery either focus on entity information only ( Neculoiu et al. , 2016 ; Mueller & Thyagarajan , 2016 ) , or use a single piece of context for each entity ( Liao et al. , 2017 ; Qu et al. , 2017 ) to learn a similarity function for entity matching . While in practice , similar context is only a sufficient but not necessary condition for context matching . Notably , in some domains such as medical , the context expression preference varies a lot from individuals . For example , sinus congestion is usually referred by medical professionals in the medical literature , while patients often use stuffy nose on social media . It is not practical to assume that each piece of context is equally informative to represent the meaning of an entity : a context may contribute differently when matched with different contexts of other entities . Thus it is imperative to focus on multiple pieces of contexts with a dynamic matching schema for accuracy and robustness . In light of these challenges , we propose a framework to discover synonym entities from a massive corpus without additional structured annotation . Candidate entities are obtained from a massive text corpus unsupervisely . A novel neural network model SYNONYMNET is proposed to detect entity synonyms based on two given entities via a bilateral matching among multiple pieces of contexts in which each entity appears . A leaky unit is designed to explicitly alleviate the noises from uninformative context during the matching process . The contribution of this work is summarized as follows : • We propose SYNONYMNET , a context-aware bilateral matching model to detect entity synonyms . SYNONYMNET utilizes multiple pieces of contexts in which each entity appears , and a bilateral matching schema with leaky units to determine entity synonymity . • We introduce a synonym discovery framework that adopts SYNONYMNET to obtain synonym entities from a free-text corpus without additional structured annotation . • Experiments on generic and domain-specific real-world datasets in English and Chinese demonstrate the effectiveness of the proposed model for synonym discovery . 2 SYNONYMNET . We introduce SYNONYMNET , our proposed model that detects whether or not two entities are synonyms to each other based on a bilateral matching between multiple pieces of contexts in which entities appear . Figure 1 gives an overview of the proposed model . 2.1 CONTEXT RETRIEVER . For each entity e , the context retriever randomly fetches P pieces of contexts from the corpus D in which the entity appears . We denote the retrieved contexts for e as a set C = { c0 , c1 , ... , cP } , where P is the number of context pieces . Each piece of context cp ∈ C contains a sequence of words cp = ( w ( 0 ) p , w ( 1 ) p , ... , w ( T ) p ) , where T is the length of the context , which varies from one instance to another . w ( t ) p is the t-th word in the p-th context retrieved for an entity e . 2.2 CONFLUENCE CONTEXT ENCODER . For the p-th context cp , an encoder tries to learn a continuous vector that represents the context . For example , a recurrent neural network ( RNN ) such as a bidirectional LSTM ( Bi-LSTM ) ( Hochreiter & Schmidhuber , 1997 ) can be applied to sequentially encode the context into hidden states : → h ( t ) p = LSTMfw ( w ( t ) p , → h ( t−1 ) p ) , ← h ( t ) p = LSTMbw ( w ( t ) p , ← h ( t+1 ) p ) , ( 1 ) where w ( t ) p is the word embedding vector used for the word w ( t ) p . We could concatenate the last hidden state → h ( T ) p in the forward LSTMfw with the first hidden state ← h ( 0 ) p from the backward LSTMbw to obtain the context vector hp for cp : hp = [ → h ( T ) p , ← h ( 0 ) p ] . However , such approach does not explicitly consider the location where the entity is mentioned in the context . As the context becomes longer , it is getting risky to simply rely on the gate functions of LSTM to properly encode the context . We introduce an encoder architecture that models contexts for synonym discovery , namely the confluence context encoder . The confluence context encoder learns to encode the local information around the entity from the raw context , without utilizing additional structured annotations . It focuses on both forward and backward directions . However , the encoding process for each direction ceases immediately after it goes beyond the entity word in the context : hp = [ → h ( te ) p , ← h ( te ) p ] , where te is the index of the entity word e in the context and hp ∈ R1×dCE . By doing this , the confluence context encoder summarizes the context while explicitly considers the entity ’ s location in the context , where no additional computation cost is introduced . Comparing with existing works for context modeling ( Cambria et al. , 2018 ) where the left context and right context are modeled separately , but with the entity word being discarded , the confluence context encoder preserves entity mention information as well as the inter-dependencies between the left and right contexts . 2.3 BILATERAL MATCHING WITH LEAKY UNIT . Considering the base case , where we want to identify whether or not two entities , say e and k , are synonyms with each other , we propose to find the consensus information from multiple pieces of contexts via a bilateral matching schema . Recall that for entity e , P pieces of contexts H = { h1 , h2 , ... , hP } are randomly fetched and encoded . And for entity k , we denote Q pieces of contexts being fetched and encoded as G = { g1 , g2 , ... , gQ } . Instead of focusing on a single piece of context to determine entity synonymity , we adopt a bilateral matching between multiple pieces of encoded contexts for both accuracy and robustness . H→G matching phrase : For each hp in H and gq in G , the matching score mp→q is calculated as : mp→q = exp ( hpWBMg T q ) ∑ p′∈P exp ( hp′WBMg T q ) , where WBM ∈ RdCE×dCE is a bi-linear weight matrix . Similarly , the H←G matching phrase considers how much each context gq ∈ G could be useful to hp ∈ H : mp←q = exp ( gqWBMh T p ) ∑ q′∈Q exp ( gq′WBMh T p ) . Note that P × Q matching needs to be conducted in total for each entity pair . We write the equations for each hp ∈ H and gq ∈ G for clarity . Regarding the implementation , the bilateral matching can be easily written and effectively computed in a matrix form , where a matrix multiplication is used HWBMGT ∈ RP×Q where H ∈ RP×DCE and G ∈ RQ×DCE . The matching score matrix M can be obtained by taking softmax on the HWBMGT matrix over certain axis ( over 0-axis for Mp→q , 1-axis for Mp←q ) . Not all contexts are informative during the matching for two given entities . For example , some contexts may contain intricate contextual information even if they mention the entity explicitly . In this work , we introduce a leaky unit during the bilateral matching , so that uninformative contexts can be routed via the leaky unit rather than forced to be matched with any informative contexts . The leaky unit is a domain-dependent vector l ∈ R1×dCE learned with the model . For simplicity , we keep l as a zero vector . If we use the H→G matching phrase as an example , the matching score from the leaky unit l to the q-th encoded context in gq is : ml→q = exp ( lWBMg T q ) exp ( lWBMgTq ) + ∑ p′∈P exp ( hp′WBMgTq ) . ( 2 ) Then , if there is any uninformative context in H , say the p̃-th encoded context , hp̃ will contribute less when matched with gq due to the leaky effect : when hp̃ is less informative than the leaky unit l. mp̃→q = exp ( hp̃WBMg T q ) exp ( lWBMgTq ) + ∑ p′∈P exp ( hp′WBMgTq ) . ( 3 ) 2.4 CONTEXT AGGREGATION . The informativeness of a context for an entity should not be a fixed value : it heavily depends on the other entity and the other entity ’ s contexts that we are comparing with . The bilateral matching scores indicate the matching among multiple pieces of encoded contexts for two entities . For each piece of encoded context , say gq for the entity k , we use the highest matched score with its counterpart as the relative informativeness score of gq to k , denote as aq = max ( mp→q|p ∈ P ) . Then , we aggregate multiple pieces of encoded contexts for each entity to a global context based on the relative informativeness scores : for entity e : h̄ = ∑ p∈P aphp , for entity k : ḡ = ∑ q∈Q aqgq . ( 4 ) Note that due to the leaky effect , less informative contexts are not forced to be heavily involved during the aggregation : the leaky unit may be more competitive than contexts that are less informative , thus assigned with larger matching scores . However , as the leaky unit is not used for aggregation , scores on informative contexts become more salient during context aggregation .

Summary: The paper presents a neural network model (SYNONYMNET) for automatically discovering synonymous entities from a large free-text corpus with minimal human annotation. The solution is fairly natural in the form of a siamese network, a class of neural network architectures that contain two or more identical subnetworks, which are an obvious approach for such a task, even though this task's SotA does not cover such architectures. even though the abstract consists the word novel, the chosen architecture is not a novel one but attached to this task, it can be considered as if.