SEMANTIC SEARCH USING ANNOTATIONS BY NATURAL LANGUAGE PROCESSING : PAPER SEARCH BASED ON EVENTS IN BIOMEDICAL SCIENCE

This dissertation aims to implement a high-accuracy semantic search system by incorporating various natural language processing (NLP) techniques, and we proposes a retrieval framework for textbases which are tag-annotated with the result of natural language processing (NLP) technique in order to achieve the semantic search system. Basic NLP systems, such as part-of-speech taggers, named entity recognizers, syntactic/semantic parsers, etc. has been developed in recent years, and the accuracy of these systems is sufficient to use in more advance NLP applications. However, because there is no effective and efficient frameworks for storing and using the results from these basic NLP systems comprehensively, one usually applies linguistic processing on the fly to a set of text after they are retrieved by simple method, such as keyword-based search. On the other hand, there is an emerging trend of enriching text with various kinds of information as annotations and using or sharing the annotations of information or knowledge in various research area not only the NLP area. This dissertation proposes a high-accuracy semantic search system incorporating NLP technologies, and shows effectiveness of NLP technologies for information retrieval. As a framework that implements the semantic search system, we propose a a semantic retrieval framework which realized advanced retrieval as compared with existing keyword-based retrieval system by specifying annotated information of the result of various types of NLP modules. In order to realize a semantic retrieval system using annotations of NLP results, we extend a region algebra and its algorithm, which is a search framework for textbases structured with tag-annotations. The extended framework can be applied to the annotations of NLP results which contain nesting structure in the annotations, and enables us to search using references by variable. Moreover, we proposes ranking retrieval for by extending the probabilistic model for keyword-based retrieval into a retrieval framework in which direct dependency is supposed among a set of structured queries. As a real world application, we implements a semantic retrieval system for MEDLINE databases, which is a database of paper abstracts in the biomedical area. We apply a deep parser and a named entity recognizer as a basic NLP modules, and recognizers of events in biomedical research area, such as a protein-protein interaction recognizer, as advanced NLP modules to the documents in MEDLINE in order to annotate the NLP informations. By realizing semantic retrieval which using annotated information from NLP modules, we construct a retrieval system which enables us to search events like protein-protein interaction which is important in biomedical area, and demonstrate the effectiveness of the proposing search framework. In the experiments, we evaluated accuracy improvement of our semantic search system on our own test data and the publically usable test collection, which is created in TREC Genomic Tracks, and shows effectiveness of incorporating NLP technologies into search systems. Moreover, we declare effectiveness of our framework by compare the framework with existing XML databases and evaluate our framework and algorithms as a retrieval framework for annotated documents.