Exploiting Semantic Relations for Fine-grained Entity Typing

1 . Introduction . Given a sentence and an entity mention in it , fine-grained entity typing ( FET ) aims to recognize the fine-grained types of this mention that matches the context [ Ling and Weld , 2012 ] . For example , in the sentence “ Trump ’ s trade wars and ethanol policy hurt farmers , ” the entity mention “ Trump ” should be identified as a politician . Other types of Donald Trump such as businessman , TV personality are incorrect since they do not fit the context . FET provides more information for the entity mentions to downstream tasks than the coarse-grained typing used in named entity recognition [ Borthwick et al. , 1998 ] . Its results can be used as important information for the referred entities while constructing knowledge bases . It can also be applied to relation extraction [ Koch et al. , 2014 ] , question answering [ Lee et al. , 2006 ] , entity linking [ Ling and Weld , 2012 ] , etc . Though there have been many studies on FET , the neural models used to approach this problem are usually similar . Typically , they [ Shimaoka et al. , 2017 , Onoe and Durrett , 2019 , Dai et al. , 2019 ] first obtain two representations from the mention and the context , respectively . Then , they concatenate them to form a feature representation that is used for the final prediction . While obtaining the representation for the context , all words are uniformly treated as the input for a neural model such as LSTM . We find that such models may overlook some semantic relations between the mention and the words or phrases in the sentence that are important for determining the entity type . To illustrate this problem , we show two examples in Table 1 , along with the predictions made by the state-of-the-art FET model used in [ Dai et al. , 2019 ] , which is a deep neural model that follows the typical architecture described above . It achieves a strict accuracy of 0.805 on the BBN dataset [ Weischedel and Brunstein , 2005 ] . In the first example , although the sentence clearly says that “ Connaught ” is a “ concern ” ( company ) , the model only predicts it as /ORGANIZATION . This problem can be resolved if we can extract “ concern ” as the hypernym of “ Connaught ” , and correspond it to the type /ORGANIZATION/CORPORATION . The hypernyms of mentions occur frequently when the referred entities are newly occurred or not known by a lot of people , which makes the typing results based on them especially valuable . In the second example , the model incorrectly predicts the mention “ Salty Dogs ” as /WORK OF ART , likely because of the terms in the context such as “ Characters ” , “ Bugs Bunny ” . However , since “ Salty Dogs ” is the patient of the verb “ drink ” , we know that the type of this mention should probably be /SUBSTANCE/FOOD ( drinks are categorized as /SUBSTANCE/FOOD in BBN ) . Such verb-argument relations can be obtained through Semantic Role Labeling ( SRL ) [ Palmer et al. , 2005 ] . Some other types of semantic relations can also help . For example , from the relation between “ head coach ” and “ Stanford ” in “ the head coach of Stanford ” , we can infer that this “ Stanford ” probably refers to a sports team . Thus , we believe that it is possible to improve neural FET models by using some semantic relations between the mention and the context words or phrases . In this paper , we investigate how the two types of semantic relations that is already discussed above can be exploited in FET : hypernym relations , and the verb-argument relations obtained with semantic role labeling . To utilize them effectively , we design two individual components that performs FET based on these two types of relations respectively . Since such semantic relations can only be found for some mentions , an existing neural FET model is also used as a base model . Then , we use a stacking approach similar to mixture of experts [ Jacobs et al. , 1991 ] to combine these three components , so that the final results are better than the predictions made by each individual component . Such a design makes our approach very flexible . It allows other types of semantic relations to be incorporated easily , and has little restriction on how each of them are modeled . We conduct experiments on two commonly used FET datasets : FIGER ( GOLD ) [ Ling and Weld , 2012 ] and BBN . Experimental results show that our approach effectively improves FET performance . Both hypernym relations and verb-argument relations are helpful . Our main contributions are : • We are the first to propose using the semantic relations between the mention and some words or phrases in the sentence to improve fine-grained entity typing . A framework that can incorporate different types of such relations is proposed . • We investigate the use of hypernym relations and verb-argument relations for FET . • We conduct experiments on two commonly used FET datasets to verify the effectiveness of our approach . Our code is available at https : //github.com/HKUST-KnowComp/SRFET . 2 . Related Work . Most recent studies on FET use neural models to approach this problem . They try different ways to improve the performance . For example , denoising the training data [ Onoe and Durrett , 2019 , Chen et al. , 2019 , Ren et al. , 2016b ] , utilizing the inter-dependencies between different types [ Lin and Ji , 2019 , Xu and Barbosa , 2018 ] , using external information [ Xin et al. , 2018 , Dai et al. , 2019 ] , etc . Before neural models become popular , there were also some approaches [ Yogatama et al. , 2015 , Ling and Weld , 2012 , Ren et al. , 2016a ] that use manually designed features . However , they usually use lexical and syntactic features instead of semantic features . The performance of these approaches is also inferior than the neural FET models proposed more recently . There are several studies [ Choi et al. , 2018 , Yogatama et al. , 2015 , Ren et al. , 2016a ] that make use of the syntactic head of the mention strings . They use the head words as features or try to create more training data by mapping them to the types used by the dataset . This is different from the hypernym relation based FET used in our approach . Head words are only extracted from the mention strings , and they do not necessarily indicate the finegrained types of the mentions . But we extract hypernyms from the whole sentence to help FET . They are almost always the terms that describe the types of the mentions . Our work is also related to using semantic annotation results to improve NLP tasks . For example , semantic role labeling have been used to help question answering [ Hommel et al. , 2019 , Khashabi et al. , 2018 ] , machine translation [ Marcheggiani et al. , 2018 ] , etc . 3 . Framework . The framework of our approach is shown in Figure 1 . Firstly , there are three individual components that makes FET predictions : Base FET Model , Hypernym Relation Based FET , and Verb-argument Relation Based FET . For each mention , each component outputs a vector whose dimensionality equals to the number of fine-grained type tags . Each element in the vector corresponds to a type , with positive value indicates that the mention belongs to this type , negative value indicates that the mention does not belong to this type . The Base FET Model is a neural model with a structure similar to existing FET models . It simply takes the mention and its context sentence as input , and outputs a prediction result . The other two components are only active when the corresponding semantic relation is found for the mention . They output zero vectors when inactive . We use a gating network [ Jacobs et al. , 1991 ] to combine these three components . It takes some features we extract from the three components as input , and outputs three weights . Then , they are used to take a weighted sum of the output vectors of the three components to obtain the final prediction . Next , we will introduce Hypernym Relation Based FET in Section 4 , Verb-argument Relation Based FET in Section 5 . The gating network will be described in Section 6 . 4 . Hypernym Relation Based FET . There are many existing studies for hypernym detection . However , almost all of them focus on classifying whether a pair of words are in hypernym relation after processing a large textual corpus [ Camacho-Collados et al. , 2018 ] . It is common for them to use statistical information that is obtained based on all the co-occurrences of this word pair in the corpus . This does not satisfy our requirement , since we need an approach that extracts the hypernym of a mention only from the sentence it occurs in , instead of using the whole corpus . Also , we need the extracted hypernyms to be of high precision , since incorrect hypernyms will hurt the performance of our FET system . Moreover , we need to be able to map the extracted hypernyms to the type labels used by the FET datasets . To this end , we propose a two-step hypernym extraction approach : 1 ) Hypernym Candidate Extraction . For each mention , use rules or string matching to extract words or phrases from the sentence that are possible to be its hypernyms . We call these words or phrases hypernym candidates . 2 ) Hypernym Verification . For each hypernym candidate found in step 1 , we use a neural model to predict whether it truly is a hypernym of the mention . 4.1 Hypernym Candidate Extraction . To perform hypernym candidate extraction , we first construct a dictionary of type terms by using the Wikidata knowledge base . In Wikidata , each entity or concept is represented as an item . Each entity item may have an attribute called instance of . For example , the entity Harvard University has an instance of value private university , the entity Google has an instance of value enterprise . Items such as private university and enterprise may have an attribute called subclass of . For example , private university has an subclass of value university , enterprise has an subclass of value organization . We process a Wikidata dump and find the items that appear as instance of values or subclass of values of some other items as type terms . We also use Wikidata to construct a mapping from the type terms obtained above to the type tags used by the target dataset . For each type term s , we perform a depth-limited breadth first search starting from the its corresponding Wikidata item , along the subclass of attribute . Denote all the visited items as H ( s ) . H ( s ) contains types or concepts that are more general than s. Then , make a series of attempts to map s to the target tags based on H ( s ) . Bellow is a snippet of the process : IF 〈Q6256 , country〉 ∈ H ( s ) ∪ { s } THEN RETURN /GPE/COUNTRY IF 〈Q6266 , nation〉 ∈ H ( s ) ∪ { s } THEN RETURN /GPE/COUNTRY IF 〈Q515 , city〉 ∈ H ( s ) ∪ { s } THEN RETURN /GPE/CITY ... Where 〈Q6256 , country〉 represents the item in Wikidata with ID “ Q6256 ” and label “ country ” . The rest of the process is similar , and can be obtained with the help of an FET training set . The type terms that can not be mapped are removed from the dictionary . We denote the final set of type terms in the dictionary as S , and the resultant mapping as τ . For example , for term “ city ” ∈ S , τ ( “ city ” ) = /GPE/CITY . Note that τ is only used for hypernym extraction and verification , it maps a term to one type tag . While performing FET based on the extracted term , a slightly different mapping is used , this will be discussed later . After obtaining the type term set S and the mapping τ . For each mention , we extract term s ∈ S as its candidate hypernym if it occurs in the context sentence , and matches any one of a list of patterns we create . The full list of patterns can be found in Appendix A . Below are three of them : • M ( and|or ) ( any|some ) other s . E.g. , “ Microsoft and other companies ” . • M ( is| , ) ( a|an ) ( NN |NNS |NNP |JJ |CC | , ) * s . E.g. , “ Connaught is a biotechnology research company ” . • s M . E.g. , “ actor Leonardo DiCaprio ... ” . where M denotes the mention string ; NN , NNS , NNP , JJ , and CC matches words with the corresponding POS tags . The first pattern above is from Hearst Patterns [ Hearst , 1992 , Roller et al. , 2018 ] . Besides these patterns , we also extract the terms in S that occurs within the mention string itself . For example , we extract the term “ university ” from “ Harvard University ” .

Summary: This work addresses fine-grained entity typing by leveraging semantic relations of the mention with other words in the same sentence. Specifically, the authors showed how to use hypernym relations and verb-argument relations. For the first, they used Wikidata to train a string match based candidate extraction and BERT-based verification model. For the second, they used an existing SRL system to extract relations between the verb and the mention. Then the two system each produce a prediction that is then combined with the base model through a gating network. The proposed method significantly improves over baselines. And they performed ablations studies to show the help from hypernym and verb-argument.