Domain Specific Word Extraction from Hierarchical Web Documents: A First Step Toward Building Lexicon Trees from Web Corpora

Domain specific words and ontological information among words are important resources for general natural language applications. This paper proposes a statistical model for finding domain specific words (DSW’s) in particular domains, and thus building the association among them. When applying this model to the hierarchical structure of the web directories node-by-node, the document tree can potentially be converted into a large semantically annotated lexicon tree. Some preliminary results show that the current approach is better than a conventional TF-IDF approach for measuring domain specificity. An average precision of 65.4% and an average recall of 36.3% are observed if the top-10% candidates are extracted as domain-specific words. 1 Domain Specific Words and Lexicon Trees as Important NLP Resources Domain specific words (DSW’s) are important “anchoring words” for natural language processing applications that involve word sense disambiguation (WSD). It is appreciated that multi-sense words appearing in the same document tend to be tagged with the same word sense if they belong to the same common domain in the semantic hierarchy (Yarowsky, 1995). The existence of some DSW’s in a document will therefore be a strong evidence of a specific sense for words within the document. For instance, the existence of “basketball”in a document would strongly suggest the “sport”sense of the word “活塞”(“Pistons”), rather than its “mechanics” sense. It is also a personal belief that DSW-based sense disambiguation, document classification and many similar applications would be easier than sense-based models since sense-tagged documents are rare while domain-aware training documents are abundant on the Web. DSW identification is therefore an important issue. On the other hand, the semantics hierarchy among words (especially among sets of domain specific words) as well as the membership of domain specific words are also important resources for general natural language processing applications, since the hierarchy will provide semantic links and ontological information (such as “is-A”and “part-of”relationships) for words, and, domain specific words belonging to the same domain may have the “synonym” or “antonym”relationships. A hierarchical lexicon tree (or a network, in general) (Fellbaum, 1998; Jurafsky and Martin, 2000), indicative of sets of highly associated domain specific words and their hierarchy, is therefore invaluable for NLP applications. Manually constructing such a lexicon hierarchy and acquiring the associated words for each node in the hierarchy, however, is most likely unaffordable both in terms of time and cost. In addition, new words (or new usages of words) are dynamically produced day by day. For instance, the Chinese word “活塞”(pistons) is more frequently used as the “sport” or “basketball” sense (referring to the “Detroit Pistons”) in Chinese web pages rather than the “mechanics” or “automobile” sense. It is therefore desirable to find an automatic and inexpensive way to construct the whole hierarchy. Since the hierarchical web pages provide semantic tag information (explicitly from the HTML/XML tags or implicitly from the directory names) and useful semantic links, it is desirable that the lexicon construction process could be conducted using the web corpora. Actually, the directory hierarchy of the Web can be regarded as a kind of classification tree for web documents, which assigns an implicit hidden tag (represented by the directory name) to each document and hence the embedded domain specific words. Converting such a hierarchy into a lexicon tree is therefore feasible, provided that we can remove non-specific terms from the associated document sets. For instance, the domain-specific words for documents under the “sport”hierarchy are likely to be tagged with a “sport”tag. These tags, in turn, can be used in various word sense disambiguation (WSD) tasks and other hot applications like anti-spamming mail filters. Such rich annotation provides a useful knowledge source for mining various semantic links among words. We therefore will explore a non-conventional view for constructing a lexicon tree from the web hierarchy, where domain-specific word identification turns out to be a key issue and the first step toward such a construction process. An inter-domain entropy (IDE) measure will be proposed for this purpose. 2 Conventional Clustering View for Constructing Lexicon Trees One conventional way to construct the lexicon hierarchy from web corpora is to collect the terms in all web documents and measure the degree of word association between word pairs using some well-known association metrics (Church and Hanks, 1989; Smadja et al., 1996) as the distance measure. Terms of high association are then clustered bottom-up using some clustering techniques to build the hierarchy. The clustered hierarchy is then submitted to lexicographers to assign a semantic label to each sub-cluster. The cost will be reduced in this way, but could still be unaffordable. Besides, it still depends on the lexicographers to assign appropriate semantic tags to the list of highly associated words. There are several disadvantages with this approach. Firstly, the hierarchical relationship among the web documents, and hence the embedded DSW’s, is lost during the document collection process, since the words are collected without considering where they come from in the document hierarchy. The loss of such hierarchical information implies that the clustered one will not match human perception quite well. Secondly, the word association metric and the clustering criteria used by the clustering algorithm are not directly related to human perception. Therefore, the lexicographers may not be able to adjust the clustered hierarchy comfortably. Thirdly, most clustering algorithms merge terms in a binary way; this may not match human perception as well. As far as the computation cost is concerned, computation of word association based on pairwise word association metrics will be time consuming. Actually, such an approach may not be the only option today, thanks to the large number of web documents, which are natively arranged in a hierarchical manner. 3 Lexicon Tree Construction as Domain Specific Word Detection from Web Hierarchy Since the web documents virtually form an extremely huge document classification tree, we propose here a simple approach to convert it into a lexicon tree, and assign implicit semantic tags to the domain specific words in the web documents automatically. This simple approach is inspired by the fact that most text materials (webpages) in websites are already classified in a hierarchical manner; the hierarchical directory structures implicitly suggest that the domain specific terms in the text materials of a particular subdirectory are closely related to a common subject, which is identified by the name of the subdirectory. If we can detect domain specific words within each document, and remove words that are non-specific, and tag the DSW’s thus acquired with the directory name (or any appropriate tag), then we virtually get a hierarchical lexicon tree. In such a tree, each node is semantically linked by the original web document hierarchy, and each node has a set of domain specific words associated with it. For instance, a subdirectory entitled ‘entertainment’ is likely to have a large number of web pages containing domain specific terms like ‘singer’, ‘pop songs’, ‘rock & rol’, ‘Ah-Mei’(nickname of a pop song singer), ‘album’, and so on. Since these words are highly associated with the ‘entertainment’domain, we will be able to collect the domain specific words of the ‘entertainment’domain from such a directory. In the extraction process, the directory names can be regarded as implicit sense labels or implicit semantic tags (which may be different from linguistically motivated semantic tags), and the action to put the web pages into properly named directories can be regarded as an implicit tagging process by the webmasters. And, the hierarchical directory itself provides information on the hierarchy of the semantic tags. From a well-organized web site, we will then be able to acquire an implicitly tagged corpus from that site. Thanks to the webmasters, whose daily work include the implicit tagging of the corpora in their websites, there is almost no cost to extract DSW’s from such web corpora. This idea actually extends equally well for other Internet resources, such as news groups and BBS articles, that are associated with hierarchical group names. Extending the idea to well organized book chapters, encyclopedia and things like that would not be surprised too. The advantages of such a construction process, by removing non-specific terms, are many folds. First, the original hierarchical structure reflects human perception on document (and term) classification. Therefore, the need for adjustment may be rare, and the lexicographers may be more comfortable to adjust the hierarchy even if necessary. Second, the directory names may have higher correlation with linguistically motivated sense tags than those assigned by a clustering algorithm, since the web hierarchy was created by a human tagger (i.e., the webmaster). As far as the computation cost is concerned, pairwise word association computation is now replaced by the computation of “domain specificity” of words against domains. The reduction is significant, from O(|W|x|W|) to O(|W|x|D|), where |W| and |D| represent the vocabulary size and number of domains, respectively. 4 Domain Specific Word Extraction as the Key Technology: An Inter-Domain Entropy Approach Since the terms (words or compound words) in the documents include general terms as well as domain-specific terms, the only problem then is an effective model to exclude those domain-independent terms from the implicit tagging process. The degree of domain independency can be measured with the inter-domain entropy (IDE) a