# Presentation: 070763

## Intelligent Meta-Search and Clustering Technologyhttp://tamas.nlm.nih.gov/metasearch/ http://toxseek.nlm.nih.gov Tamas Doszkocs, Ph.D.Computer Scientist National Library of Medicine doszkocs@nlm.nih.gov

## Characteristics of Web Searching

- Content is created by _diverse organizations and individuals_
- Information on the Web is inherently _heterogeneous_
- _Content is distributed_ on _multiple servers_ in _multiple locations_ and _multiple formats_ and _languages__ _aimed for _diverse audiences_ and _purposes_
- 	(In its **April 2005** survey NetCraft received responses from **62,286,451** web sites)
- The “_Open Web_” of billions of static Web pages is indexed and searched via multiple search engines and directories

## Problems in Web Searching

- Even the largest of the current search engines        _ index only a fraction of all Web pages_
-        (The WayBackMacine of Internet Archive has_ _indexed 40 billion pages, Google about 8.1 billion, Yahoo about 20.8 billion  -- August 2005)
- The not so “_Hidden Web_” of content databases (e.g. PubMed, Web of Science) is estimated to be thousands of times larger than the Open Web.
-  Both the Open Web and the Hidden Web are characterized by _problems of information coverage_, _quality_, _overload_, _relevancy_, _currency and completeness_, as well as_ inherent language ambiguity_ and _incompatible user interfaces_

## Meta-Searching

- Meta-Search Engines may simultaneously search multiple_ Open Web_ and _Hidden Web_  sites in order to _increase content coverage_, _precision, relevance_ and/or _search efficiency_ and_ effectiveness_.

## Overlap Among 3 Major Search Engineshttp://missingpieces.dogpile.com/whitepaper.pdfhttp://comparesearchengines.dogpile.com/OverlapAnalysis.pdf

## Overlap Among AskJeeves, Google, MSN and YahooGoogle Isn’t Everything!http://www.forbes.com/business/free_forbes/2005/0815/056.html?partner=yahoomag 

## Generations of Meta-Search Engines

- First Generation
- Second Generation
- Third Generation
- Next Generation

**“****Broadcast” or “Federated” search**
**List of results**
** ****Merging and Ranking**
**Increased coverage**
** ****Result Clustering**
**Focused drill-down**
**Dynamic Query Mods**
** ****Semantic and         Pragmatic Intelligence**
**tamas.nlm.nih.gov/metasearch/**
**toxseek.nlm.nih.gov**
**http://bestmeta.com**

## Moving Targets:Nine Search Engines ComparedBy Ben Patterson (May 9, 2005) http://reviews.cnet.com/4520-10572_7-6219242-2.html?tag=txt

## Moving Targetsand the need for Automatic Change Detection and MonitoringandIntegrating New Capabilities 

## The ToxSeek Meta-Search and ClusteringProject

- Goals:
- Integrate best practices _Information Retrieval_ and _Natural Language Processing_ techniques with AI _heuristics_ to create an advanced general purpose meta-search, result clustering and knowledge discovery tool
- Apply ToxSeek to efficiently _access diverse biomedical _and _environmental health information resources_
- Create _specialized applications_ for accessing quality information sources on HIV/AIDS, consumer health, homeland security, public health law, library research and other applications

## ToxSeek Features

- Integrates _multiple spellcheckers_ and _sophisticated lexical_, _morphologic_, _syntactic_ and _semantic resources_
- _Merges_ and _ranks_ the results from heterogeneous information sources
- Employs efficient _Natural Language Phrase Parser_ and _AI heuristics_ to automatically identify _Key Concepts_ and their _Associations_ in queries and retrieved documents
- Uses the automatically identified Key Concepts and Associations to create topical_ __Result Clusters_
- Supports _focused multi-concept drill-down_, _dynamic query refinement_, _multi-media_ and _limited question answering_

## ToxSeek Implementation

- Production applications and research prototypes have been implemented for meta-searching diverse content on:
_**Toxicology and Environmental Health**_
- Consumer Health
- Library Catalogs and Proprietary Databases
- HIV/AIDS
- BioDefense
- _Homeland Security_
- “Shift Happens...”
- http://library.nps.navy.mil/home/staff/gmarlatt/HSDL%20ALI%20April%202005%20%20final%20rev%207%20april.ppt

## ToxSeek Web Search Query: “terrorism”

## ToxSeek Query: “police state”

##  Win the Search Engine Wars with Intelligent Meta-Search and Clustering Technologyhttp://tamas.nlm.nih.gov/metasearch/ http://toxseek.nlm.nih.gov Tamas Doszkocs, Ph.D.Computer Scientist National Library of Medicine doszkocs@nlm.nih.gov