# Presentation: 145088

## Extending Access: A Study of Search Engine Technologies(A joint COSPO/CENDI sponsored project)

**Dr. Inderjeet Mani **
**imani@mitre.org**

**N.B. This study was completed in September 1996. **
**Some of the datapoints may have changed since then. **
**Hopefully, the methodology will stand the test of time**

## COSPO/CENDI Text Search Engine Study  

**Objective**
**Recommend specific vendors and products as candidates for joint-agency investments**
**Motivation**
**leverage resources across agencies to influence vendors**
**leverage MITRE expertise and knowledge about text search and indexing**
**Scope: 3 months**
**Contributors: Adrienne Kleiboemer, Dave Mattox, Kevin Anderson, Inderjeet Mani**

## Participants 

**CENDI Agencies **
**DTIC**
**NAIC**
**NBS **
**NLM **
**NTIS**
**Intelligence Community Agencies**
**FBIS**
**NSA**
**OIR**

## Study Methodology 

**Gather common and minority requirements through interviews and discussions with agency POCs**
**Identify candidate search engines**
**Exploit MITRE knowledge from assessing/evaluating/transitioning/using search engines in mission-critical government programs**
**Match requirements, obtain feedback, publish report**

## Methodology Task Breakdown 

**Gather requirements through interviews with agency POCs**
**Prepare interview summaries; develop  requirements matrix**
**Identify candidate search engines **
**Prepare preliminary draft report and obtain feedback  **
**Establish common requirements from matrix, interviews, and feedback**
**Gather additional search engine information**
**Match requirements against  product capabilities**
**Develop recommendations**
**Prepare final report**

## Some Key Search Engine Components

_**Query Processor **_**(to allow flexible expression of information need)**
_**Analyzer**_** (tokenizes text, extracts content such as content words, stems, phrases, bigrams, named entities, etc.)**
_**Indexer**_**  (for efficient search)**
_**Matcher**_** (assesses similarity between information need and document content)**
_**Document Manager**_**   (for document storage, incl. DBMS integration, personalization/security, indexing, format conversion, handling compound/multimedia documents, etc.)**

- User

- Query
- Processor

- Matcher

- Retrieved Hits

- Index

- Collections

- Analyzer

- Indexer

- Result
- Presenter

- User Interface

- Query

- Feedback

- Document Manager

## Search Engine Requirements Categories

**Broad categories established via interviews**
**Categories further grouped here for presentation as:**
_**Query Formulation**_
**Data Formats**
**Result Presentation**
_**System Architecture**_
_**Accuracy**_
**Miscellaneous**
**Requirements which turn out to be common are underlined**

## Query Formulation

_***Boolean***_**  operators (AND, OR, NOT) -  may or may not have Boolean semantics**
_**Fielded**_** (author, title, publication date)**
**Hierarchical**
**Iteration**
**NL Query  **
_**Proximity**_** (adjacency, before/after, n words)**
_**Fuzzy**_** (e.g., single/multiple-char wildcards, soundex, pattern matching)**
_**Thesaurus Expansion**_
**Query By Example, Relevance Feedback, Pseudo RF**

## Data Formats

_**Multimedia Data**_
**storing and searching for imagery, video, audio (text-only --> ignoring embedded multimedia --> searching on textual meta-info --> searching on multimedia content)**
_**Native Formats**_
**processing documents which are in MS Word, etc. (search engine may convert, but retains link)**
_**SGML**_
**standard notation for rep. doc structure and content (many engines process SGML only to a limited extent)**
**Foreign Languages**

## Result Presentation

_**Relevance Ranking **_**(usually based on # of query terms in doc, importance of each one (tf.idf), locations in text,  relative proximity, and size of document)**
**Clustering**
**Summarization (e.g., TIPSTER)**
**Machine-Aided Indexing (e.g., TIPSTER, MUC-6)**
**Visualization Tools **
**Passage-Level Retrieval (retrieve paras, sections, regions - esp. within long docs)**

## System Architecture

_**Scalability**_
**Index  large volumes (e.g., 1 GB a day) plus search large collections (e.g., > 100GB) at practical speeds**
**Support for parallel execution, distributed search servers, load balancing**
_**Integration with Structured Databases**_
**Ability to map information to and from structured dbs**
_**API**_
**allows product to be adapted to different applications**
**customized GUI, help, and usability highly dependent on this**
**Interoperability - Protocol Standards**
**e.g., Z39.50**** **

## Miscellaneous

_**Profiling**_
**filtering using standing queries - relevant information is automatically identified and routed as it arrives**
_**Security**_
**restricting access to documents or collections or functionality based on authenticated access level or permissions**
_**Web Access**_
**allowing issuing requests to and receiving results from a search engine using a Web browser; also, ability to browse document collections using a Web browser.**
**Workspace Management**
**allowing modifying, saving, retrieving and editing of queries, search results, creating virtual private collections**

## Comparative Requirements Matrix

## Comparative Requirements Matrix (contd.)

## Observations on Common Requirements

**No surprises: **
_**Accuracy*, Scalability*, Profiling*, Relevance Ranking**_
**Reflects need for efficiency in handling mushrooming of information**
_**Security***_
**need to provide classified info to trans- or intra-firewall customers, or just concerns about unauthorized access**
_**Web Access*, flexible Query Formulation*, Thesaurus*, API***_
**affects end-user usability**
**Surprises: (did not emerge as common requirement)**
**NL Query, Machine-Aided Indexing**
**Other:**
_**Document Formats (Multimedia, SGML), Integration with Structured DBs**_
**Browser, Visualization Tools, Passage-Level retrieval**

- * => high priority for some agencies

## Search Engines considered

**BASISPlus - Information Dimensions Inc.**
**CLARIT - Claritech**
**Oracle ConText - Clarit**
**Fulcrum SearchServer/Surfboard - Fulcrum**
**INQUERY - Sovereign Hill (Also in Infoseek)**
**InTEXT - InTEXT Systems**
**OpenText Livelink Search - Open Text**
**PLS/PLWeb (on MII ) - Personal Library Software**
**RetrievalWare - Excalibur**
**TOPIC - Verity**

## Search Engine Matrix - Common Requirements

## Search Engines Matrix - Minority Requirements

## Observations on Search Engines meeting requirements

_**Accuracy**_** is hard to establish, though statistical systems are in general superior to Boolean ones**
**Only RetrievalWare allows you to search for images by color and texture**
**applies to severely degraded OCR text**
**Several systems offer**_** integration with structured dbs **_**(TOPIC, PLS, OpenText , Oracle)**
**“****tuples” versus “docs”: Substantial issues in terms of integrated queries and indexing  (cf. MITRE’s current DIA project)**

## Observations on Search Engines meeting requirements (continued)

_**Relevance Ranking **_**+ **_**Boolean**_** operator queries often very useful  **
**e.g., Inquery - can be mixed**
**e.g., RetrievalWare - Boolean and Smart/Expert modes couldn’t be mixed**
**Web Access is still  a moving target **
**CGI scripts: APIs vary, maintenance issue, interacts with integration with structured dbs (security: certificate passing not handled; transaction logic: lack of state maintenance; scalability: need for threading of cgi processes where many clients access a server)**

## Group I and II systems

**Group I: not deficient in more than one common requirement (3 systems) **
**These were further compared in terms of product use/experience, government relationships, and company strength and experience.**
**Group II: deficient in 2-3 common requirements (6 systems)**

## Recommendations for joint-agency investments

**Satisfying common requirements: choose a Group I system**
**Agency -Specific Recommendations**
**To determine search effectiveness (accuracy, query formulation, relevance ranking, etc.) test all three Group 1 systems on actual retrieval tasks and data collections with operational users. **
**Benchmark these systems on appropriate hardware configurations and full-size collections**
**If all else is equal, leverage existing investments by some agencies in particular products**
**For agencies whose individual requirements concerns outweigh benefits of Group1 solution, consult  Agency-Specific Recommendations.**

## Conclusions

**Agencies surveyed share many common requirements for text search engines**
**Even though there are differences, there are sufficient commonalities that can and should be exploited for mutual benefit**