The ALVIS Document Model for a Semantic Search Engine

ALVIS is a research project in the design, use and interoperability of topic-specific search engines with the goal of developing an open source prototype of a peer-to-peer, semanticbased search engine. Existing search engines provide poor foundations for semantic-based web operations. Our approach is not the traditional Semantic Web approach with coded metadata, but rather an engine that can build on content through semi-automatic analysis. This paper describes the ALVIS document processing architecture that supports the processing of documents for these purposes. 1. THE ALVIS OBJECTIVE AND RATIONALE The ALVIS objectives are: to provide a powerful, free, stand-alone semantic-based search system so that applicationdomain experts can readily build topic-specific search sites without needing to become information retrieval experts or computer systems gurus; and also to also develop complementary distributed components, together with bridges to existing topic-specific search sites, so that the individual sites can be linked up to form a search network. The semantic-based search engine is intended to automatically build and maintain its own semantic structure with named entities, topics and so forth, and to input primitive ontologies. It is not a Semantic Web engine, and does not rely on the existence of Semantic Web ontologies or build its own ontologies. The semantic structure is created semiautomatically using statistical and machine learning methods for the purpose of returning better search results. The distributed system is intended to be able to operate with heterogeneous search servers, using query topics as a routing mechanism, and using distributed methods for ranking and semantic-based processing. There are four main factors that motivate this design  The individual search engines we provide (1) must have more capabilities than existing major commercial search engines. A user must gain significant advantage from ALVIS serPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ESWC ’05 Heraklion,  harness the efforts and imagination of talented groups and individuals in the research and development community, then the effort can be maintained after the end of the funded project’s lifespan. Open Source effort is essential to match the significant resources of the commercial engines. The so-called deep web provides a large, rich set of resources across many areas both commercially and in digital libraries. The (3) deep web is naturally accessed by a distributed system, and not by a single centralized indexer. Finally, in the light of the distributed open source development of Linux, and the Open Directory Project, (4) we view open source as a healthy business model for this particular service, important in terms of encouraging participation. 2. STANDARDS FOR OPEN SOURCE PROCESSING OF DOCUMENTS The basis of ALVIS, then, is the individual peer, capable of taking in documents, analysing them semantically and building suitable indexes, then making those documents available across the wider ALVIS network. Within each peer, a processing pipeline generates the semantic information. Document processing is then summarised in the arFigure 1: Overall architecture chitecture of Figure 1. Routine document processing occurs before hand over to the run-time search system, and subsequently the peer-to-peer system. Collection-wide analysis and maintenance of the system’s semantic and linguistic resources then occurs in on off-line maintenance system, that uses the document representation. It is in the maintenance system that tools for semi-automated ontology learning would be used, for instance, we are starting to experiment with tools for simple ISA-hierarchy discovery [4]. Note we currently restrict our semantic links to the ISA type and use RDF as the standard for representation. We are currently in the process of readying and documenting the XML formats used in the pipeline so that we can integrate a wide variety of linguistic and annotation tools, and allow others to do so as well. 2.1 Document Acquisition and Canonicalisation Documents may in principle be of any type and acquired from any source: HTML documents harvested by a Web crawler, local PDF files scanned from the local disk, MSWord files from a company-wide repository, etc. In order that they can be handled uniformly by the later started in the pipeline, document acquisition models are required to generate a canonicalised version of each document, conforming to a simple XML schema. The details of how this canonical version is generated vary depending on the sourcedocument’s format. The simple scheme only recognises sections and lists, eliminating other information at present. Conversion from HTML to this form is simpler than full application of W3C’s tidy for conversion to XHTML, known to fail on a significant percentage of documents. Thus HTML such as: Tell me, Alvis! You’re the dwarf who knows everything ... ...