MEMORANDUM

To: Chair, Interagency Committee on Government Information,\
Office of Management and Budget,\
Executive Office of the President

From: Chair, Categorization of Government Information Working Group

Subject: Transmittal of Search Interoperability Recommendation

On behalf of the Categorization of Government Information Working Group,
I am pleased to transmit our Search Interoperability Recommendation to
the Interagency Committee on Government Information, pursuant to the
E‑Government Act of 2002, Section 207, \"Accessibility, Usability, And
Preservation of Government Information\".

This recommendation builds on the long standing principle in law and
policy that government information must have an appropriate
bibliographic treatment so that items are citable, whether those items
are electronic or otherwise. Citations are composed of bibliographic
attributes (title, author, date, etc.), and such attributes also include
various categories that are useful for information discovery (subject,
place, audience, identifier, etc.). With contemporary information
technology, Federal agencies have flexibility in achieving these
requirements for bibliographic treatment at reasonable cost. Human
catalogers are now supported by information technology that can
reformat, generate, or suggest some of the bibliographic values and so
deliver information citations as part of the search and retrieval
process.

This recommendation specifically proposes that the U.S. Federal
Government adopt the ISO 23950 international standard to enhance
interoperability among networked systems that aid in the discovery of
and access to government information. This mature search and retrieval
standard allows for traditional bibliographic catalogs to be integrated
as appropriate with electronic information resources of many kinds and
different formats.

Although this recommendation does not require new law or policy, the
role of GSA should be clarified as necessary to assure that future
search technology procured by Federal agencies will be compliant with
the ISO 23950 international standard. The oversight role of OMB under
existing policy should also be updated to reflect the effects of the
E-Government Act and other changes.

This recommendation supports existing law and policy, and aligns with
other ICGI recommendations required under the E‑Government Act, Section
207(d)(1). ICGI will recommend an overall definition of Government
information, and this search interoperability recommendation applies to
Federal Government information services providing search of such
information. ICGI will recommend an open standard for searchable
identifiers, and this search interoperability recommendation provides
for search by such identifiers. ICGI will also recommend a standard set
of categories for all government information, and this search
interoperability recommendation provides for search by such categories.

A Summary of Comments Received on the Search Interoperability
Recommendation is provided as Attachment 1. The Search Interoperability
Recommendation as it was posted for public comment is provided as
Attachment 2.

/signed/

Eliot Christian

Chair, Categorization of Government Information Working Group

Attachments

The following is a categorized summary of comments on the Search
Interoperability Recommendation (posted at
http://www.cio.gov/documents/ICGI/recommendation.html ). In addition to
the written comments received during the announced public review period
through September 27, a few noteworthy conversations are also summarized
here.

### Alignment with Law and Policy

**Patrice McDermott** (Deputy Director, Office of Government Relations,
American Library Association) commented:

I am writing on behalf of the American Library Association to commend
the Working Group on its recommendation that the federal government
should adopt a search service standard to enhance interoperability among
networked systems. This recommendation is completely in line with what
those of us who worked on the E‑Government Act from its very beginning
had in mind. Our intent in working for the legislation was to make it
possible for the public to search, find, get access to and use
government information products across the government. While a search
service standard will not accomplish all these things, it is essential
that the public have faith that it is finding all government information
on a topic or concern.\
\
As the recommendation notes in Section 3, the library community has
worked together to provide seamless access to information through search
service standards. The section succinctly lays out all the reasons for
the government to adopt this recommendation: this is good government
both in the sense of accountability to the public and in the sense of
effective government.\
\
We strongly urge OMB to adopt and implement this recommendation.

**Lori Lisowski** (Director, Policy and Communications Staff, NARA)
stated:

NARA supports this recommendation. It makes practical sense to have a
government-wide search standard. We support leveraging existing
standards and technologies for search interoperability of online
government information and of a scalable and flexible approach.

Following a discussion at an open meeting of the CGI WG, **Andy
Hoskinson** (Industry Advisory Council) developed a definition of
\"search technology\" as that term might be used in the context of
Federal acquisitions. He reports that IAC members did not object to his
proposed definition of search technology as:

Any software application or information system that has the primary
purpose of facilitating discovery and retrieval of information.

### Feasibility and Cost

**Jon Venverloh** (Senior Manager, Google Federal) commented:

Google agrees that the additional cost to government of the \'additional
support for the search service standard\' costs \'thousands per year.\'
Such work likely includes modifying front end applications to compose
search queries in different ways to interoperate with each search
engine.

In conversation with the editor, Jon further clarified that vendors
would not make major architectural changes to their software, but would
readily support the common standard through a software gateway. The
editor notes that such a gateway approach to implementation is
completely consistent with the recommendation. Gateways are already a
very common technique for technology vendors to support this and many
other standard interfaces.

**John Joaquin** (Convera) stated:

Convera is committed to continuing its support of the government market
and would quickly support this effort.

**Beverly Godwin** (GSA) recommended that CGI WG take a look at the
search mechanism provided by the \"Science.Gov\" initiative. This prompt
led to contact with **Carl Sylvia** at Deep Web Technologies, which
provides the search technology underlying Science.Gov. Carl was asked to
review the recommended search standard and provided this response:

I have taken a few moments to review the Z39.50 \[ISO 23950\] SRW/SRU
standards, and I am quite excited by what I have discovered. I agree
that with a minimal amount of effort our current search technology will
be able to take advantage of the SRU functionality. In addition, are
currently in the process of re-architecting our deep-web search
technology solution, and this architecture will be able to make full use
of the full Z39.50 SRW implementation (as it will be a web-services
based implementation).

I would be very interested in discussing the possibility of applying our
search technologies to collections that are currently supporting the
Z39.50 standards, both with our current implementation, and in the
future as we begin deployment of our new web-services based
architecture.  In particular, I would be interested in discussing Z39.50
support on collections that we currently support through science.gov,
and any additional collections that we may include in future search
portals.

**Walt Warnick** (Office of Scientific and Technical Information at the
Department of Energy) discussed search interoperability issues with the
CGI WG. After discussion, all agreed that the recommendation would
recognize that a mix of human and machine-aided categorization is
appropriate with contemporary information technology. There is also an
expectation that more innovations will be forthcoming in the automated
analysis of text to support even more effective searching in the future.

**John Shirey** (Environmental Protection Agency) initially expressed a
concern about recommending the ISO 23950 search standard because most
ISO 23950 implementations in libraries have used a special Internet
protocol rather than the ubiquitous \"HTTP\" (HyperText Transfer
Protocol). John withdrew his objections when he looked into the new ISO
23950 profile that does in fact use HTTP, and also provides a simplified
and human-readable query syntax for search.

### Benefits in Cost Savings

**Raul Valdes-Perez** (President and co-Founder of Vivisimo, Inc.)
stated that implementation of these recommendations would provide
\"substantial information-delivery and cost benefits\" by simplifying
the task of creating and maintaining cross-agency search facilities.

**Andy Hoskinson** asserted: \"From a benefits standpoint, the cost
savings achieved through improved information sharing between federal
agencies and other entities (state and local governments, private sector
entities, etc.) would be staggering.\"

**Lori Lisowski** asked: \"what are the costs of not doing this,
especially with free commercial search tools already available?\"\
\
The editor responded: \"the costs of continuing the current lack of
interoperability include various lost opportunities for better
efficiency across agencies, across levels of government, and in managing
technology evolution.\" The editor also pointed out that free commercial
search tools will continue to play a major role in public access to
government information, but those search tools operated by Federal
government agencies should also be interoperable (typically via software
gateways).

### Benefits in Information Accessibility

**Jon Venverloh** stated:

We at Google think the CGI Working Group\'s mission is important and
agree that improved search capabilities will reap significant rewards
for citizens and civil servants.

**Andy Hoskinson** predicted that implementing the recommendation would

greatly enhance the effectiveness, accountability, and transparency of
government through improved citizen access to government information. It
will accomplish this by providing open, non-proprietary, and uniform
interfaces to government search services that make it easy for both
public and private sector entities to \'connect the dots\' between
information maintained by different federal government agencies

**Gladys Cotter** (Associate Chief Biologist for Information, USGS)
commented:

The search interoperability standard will greatly aid access to data and
information produced by all sectors of society. Implementation of this
standard will move us closer to the vision of fully integrated
information access and retrieval.

**Lori Lisowski** asked: \"Would this interoperability function provide
greater levels of service, such as digging into the data in
databases?\"\
\
The editor responded: \"this interoperability function can be used to
provide greater levels of service. However, it should be understood that
\'digging into the data in databases\' can be achieved with the
recommended standard only with respect to those databases holding
certain types of information.\"

### Role of Government

**John Joaquin** (Convera) stated:

Search interoperability is a much needed capability for government
users. Proprietary search technology, like that offered by Convera and
others, provides considerable value to the knowledge worker.
Unfortunately the value collected in product specific systems (indexes)
is difficult to access from other search technologies. Given the nature
of the search market it very unlikely that vendors will come together to
solve this problem using a standards-based approach.

A customer-driven approach is, in my opinion, the best way to a
solution. This approach leverages the government\'s buying power.
Ideally policy-setting organizations would promulgate a standard that
would require/recommend agencies to support when making investments in
search tools. This approach will get the attention of industry and
provide a business reason to adopt said standard.

**Andy Hoskinson** commented:

there is no monetary incentive that I am aware of for private sector
companies to collaborate on a search interoperability standard \[\...\]
it is appropriate for the Government to take the lead in this area.

### Intergovernmental Implications

**John Borras** (Director, Technology Policy, e-Government Unit, UK)
stated his full support for the recommendation. John noted that this
recommendation builds on work done in the OASIS e‑Government Technical
Committee wherein there was

lengthy debate within the committee and amongst our various technical
advisors about the all the issues and the committee has now published
it\'s own similar recommendation for the attention of all
Governments\...Clearly the more we in government across the globe
implement this standard then the better chance we have of meeting our
stated goal of true interoperability.

**Ivan DeLoatch** (Staff Director, Federal Geographic Data Committee)
expressed

strong endorsement of this recommendation for standard-based
interoperable search. In particular, the use of the ISO 23950 standard
is critical to leveraging the broad interoperability currently
implemented in many levels of government in the context of geospatial
data. This includes: the US National Spatial Data Infrastructure as
mandated by law and directed by policy, the international Global Spatial
Data Infrastructure that now includes more than fifty national spatial
data infrastructure, and the more recent Global Earth Observations
System of Systems currently being planned for international use.

### Effect on Search Technology

**Raul Valdes-Perez** stated that providers of search services will be
able to

independently evolve an individual search resource without the
substantial overhead of worrying how changes to the resource will impact
cross-agency search engines that depend on it

**Andy Hoskinson** drew attention to

the likelihood that this recommendation, if implemented, will spawn a
robust private-sector "cottage industry\" around the discovery,
mediation, and aggregation of federal government information and related
information obtained through state/local and private sector channels

### Technology Approach

**David McDonald** (Microsoft Consulting Services, Public Sector) stated

I am pleased to see this initiative and am a strong supporter of
government search interoperability. Inclusion of the Search/Retrieve Web
Service Profile to FIPS 192 is strongly recommended. A standard web
services interface enables software developers to more easily create
solutions that include search and retrieval of government information.
Standardizing the URL and query syntax using CQL as stipulated by the
SRU service as described in this recommendation is a good first step
\[\...\] Government agencies should be encouraged to implement the full
XML/SOAP interfaces.

**John Barden** (Head, Reference & Research Services, William T. Muse
Law Library, University of Richmond School of Law) commented:

This is a very promising initiative. Is it possible, to go one-step
beyond the search standards identified here to incorporate \[\...\]
proximity searching

The editor responded that the recommended standard does support
proximity searching, among other useful features not discussed in this
document.

**Anne Washington** (Librarian, Office of Legislative Information,
Congressional Research Service, Library of Congress) stated:

There is a prevalence of structured and semi-structured information held
by governments. Here on Capitol Hill, there are several projects
underway to deliver legislation in XML. I know that we aren\'t alone in
our effort to make full-text government documents available in XML.
\[The Requirements document\] refers to common query mechanisms that the
interoperable search standard should address. This might be a place to
refer to the XML Query language (XQUERY). It is a single mechanism that
addresses full-text, markup and fielded documents. XQuery is a standard
being developed by the W3C

The editor responded:

From the perspective of interoperable search, we cannot expect a
searcher to know the details of any particular document schema such as a
convention for markup of legislation. Indeed, we cannot expect a
searcher to know which markup language is used with any particular
collection of documents. In the case of XQuery, the searcher needs to
know that the documents being searched are to be treated as XML
documents, and further that they are structured according to a
particular XML Schema.

\[\...\] interoperable search is directed at the problem of broad-scale
interoperability. It should be seen as supplementing, not supplanting,
other search mechanisms such as the use of XQuery on legislative
documents in a particular schema.

**Jon Venverloh** commented:

the international standards referenced in section 3 do not appear to
address relevance of documents that have been retrieved. That is, these
standards may address recall but do not address precision. How relevance
is determined is the primary differentiate among search engines and
represents the bulk of research and development costs for the major
search engines such as Google, Yahoo, and others.

The editor responded that the recommended standard intentionally
excludes standardization of relevance ranking. Search services compliant
with this standard do in fact use a wide range of ranking approaches and
ranking is an area where search technology vendors compete actively.

**Jon Venverloh** further commented that:

query analysis, syntax and results are handled differently from one
search engine to another. Search engines compete to some degree on
usability. Google spends a substantial amount of its R&D resources on
continually improving usability.

The editor responded that the recommendation is supportive of active
competition among vendors with respect to query analysis and results
handling as that is the primary way that progress is made in balancing
\"precision versus recall\" in information search. The standard
recommended here does not overly constrain those different approaches.
It only provides a standard syntax for those certain components of a
query and result set that are common across the great majority of search
technologies.

**Jon Venverloh** also contended that:

a standardized query format \[\...\] would limit unique features and
innovations among some search engines \[\...\] the proposed standards do
not seem to anticipate the different technological approaches that are
inherent\--and beneficial to end users\--in a market with multiple
competing search engines.

The editor corrected this misimpression and pointed out that the
recommended standard does provides support for unique features and
innovations, and that such extensions are widely used in practice.

### Agency Implementation Specifics 

**Lori Lisowski** asked: \"What are the practicalities of being able to
accomplish this?\"\
\
The editor responded: \"As a practical matter, search interoperability
is typically provided by adding a standard gateway service onto whatever
similar service already exists. This is usually a matter for the search
technology vendor to address.\"

**Lori Lisowski** also asked: \"How much preparatory work would agencies
be required to do in order to enable such a service?\"\
\
The editor responded: \"Agencies would usually need to work with the
vendor to provide the \'semantic map\' which relates locally-known
elements to the standard search concepts. This is the same type of work
that agencies already perform whenever they customize a search to use
any agency-specific features.\"

**Lori Lisowski** further asked: \"How would it deal with false
positives?\"\
\
The editor responded:

A \'false positive\' might exist when an item in any result set is
perceived to satisfy the given search criteria and yet be regarded as
not a very relevant item. In the discipline of \'information
retrieval\', this aspect is measured as \'precision\' and it is
understood that any relevance ranking makes a particular trade-off
between precision and \'recall\'. This standard is silent with regard to
the variety of ranking approaches and algorithms. This is currently an
active area of competition among information retrieval technologies.

### General Expressions of Support

**Kate Dolan** (The Enterprise Standards Program, National Security
Agency): \"I just wanted to say that we are happy to see the support of
standards usage.\"

**Aline Dolan** (Naval Hospital, Jacksonville): \"Excellent. Definitely
the right way to go. Thank you very much for your efforts to keep our
services accessible to all.\"

### 1 Recommendation

The U.S. Federal Government should adopt a search service standard to
enhance interoperability among networked systems that aid in the
discovery of and access to government information. The adopted search
service standard should be the ISO 23950 international standard, thereby
providing a high degree of interoperability across many communities of
practice and types of data and information holdings. This recommendation
follows existing law and policies of the U.S. Federal Government,
positioning the standard search service as a supplement to other search
mechanisms as may be needed for reasons other than broad scale
interoperability.

### 2 Implications 

**Policy** - No additional policy action is needed to implement the
standard search service recommended here. The U.S. Federal Government
already has law and policy mandating a standard search service as part
of the Government Information Locator Service established by law (United
States Code Chapter 44, Section 3511). Corresponding Federal policy (OMB
Memorandum 98-5) required a standard search service to be used for
locating government information. That standard search service is
described in Federal Information Processing Standard (FIPS) 192-1, which
is required to be cited in procurements of search technology by Federal
agencies. FIPS 192 adopted a profile \[GILS\] of the international
standard recommended here, ISO 23950. Similar law, policy, and standards
exist for geospatial data in the United States (i.e., E-Government Act
section 216, OMB Circular A-16, and the Geospatial profile \[GEO\] of
ISO 23950).

**Oversight** - As noted in existing policy, GSA and OMB oversight
should be exercised to assess and enforce existing law and policy
requiring search technology procured by Federal agencies to comply
with FIPS 192. FIPS 192 should be updated to include the newly available
\"Web services\" profile of ISO 23950 known as \[SRW\].

**Cost** - There is an ongoing operational cost to government in
supporting any standardized search service, but this would be
essentially the same as what is entailed in setting up non-standard
search services. When first introduced, the support of a search service
standard may prompt an add-on cost in acquisitions of search technology,
but such a cost would be a small percentage of what the U.S. Federal
government spends on disseminating government information. For example,
an Internet portal for government information may cost millions per
year, while the additional support for the search service standard on
that portal may be thousands per year. Also, after search technology
vendors have implemented the standard interface once, their costs for
supporting additional implementations should be very minor.

### 3 Background 

An information index helps a searcher to find resources. Such an index
usually covers just one collection of resources, yet searchers often
want to search across multiple collections. The ability to search
multiple, separately operated indexes is called \"search
interoperability\". Amazingly, libraries worldwide already offer
interoperable search across their many thousands of collections. This
search interoperability is based on a carefully negotiated international
standard supported by the major vendors of information retrieval
technology. And, the standard addresses far more than mere \"word in
text\" search\--the standard includes sophisticated methods needed for
precise searching of collections holding millions of diverse resources.

Clearly, libraries worldwide are meeting their customers\' needs to have
seamless access to information. The same cannot be said of governments.
Agencies no longer rely on libraries as the backbone of information
dissemination\--they offer information directly to the public via their
own indexes and directories of Web pages, databases, and a diverse range
of specialized services. It is true that the amount of accessible
government information is growing at a healthy rate, but the need for
people to confidently search for government information across agencies
and levels of government is not being met. The problem is not so much
the amount of information. The problem is that few governments have yet
focused on search interoperability.

Because public access to government information is the basis of
effective, accountable and transparent government, interoperability of
government search facilities is essential. Adoption of a search service
standard would serve the public interest by making government
information more readily accessible through the diverse community of
government information providers. Search interoperability also generates
government-wide efficiencies: from increased information sharing, and
from lowered costs for mechanisms needed to merge information from
multiple government sources. Efficiencies accrue within each single
government organization, as well. For instance, a search service
standard provides some \"future-proofing\" against changes in search
technology. With standards-based search, the periodic migration to new
search technology is not so disruptive, and it is also easier to
maintain access to legacy holdings.

Governments at all levels worldwide are major producers and consumers of
data and information, encompassing many communities of practice and
types of data and information holdings. Because governments both depend
upon and foster a competitive intermediary market for information
dissemination and service delivery, government support of broad scale,
standards-based interoperability is essential. In that regard,
governments must promote an information search interface that is
non-proprietary, fair, and stable. By acquiring products that support an
international standard search service, governments will encourage a fair
and competitive market for products, and maximize agency choice.

The E-Government Act of 2002 requires the U.S. Federal Government to
enhance search interoperability by adopting a common standard. Section
207 (\"Accessibility, Usability, And Preservation of Government
Information\"), requires that the Interagency Committee on Government
Information submit recommendations to the Director of the Office of
Management and Budget (OMB) on \"the adoption of standards, which are
open to the maximum extent feasible, to enable the organization and
categorization of Government information in a way that is searchable
electronically, including by searchable identifiers; and in ways that
are interoperable across agencies\".

### 4 Base Requirements

This Recommendation satisfies all of the mandatory and desirable
requirements as given in the Statement of Requirements for Search
Interoperability, posted for public comments and revised over the period
February - April, 2004 ( http://www.search.gov/interop/requirements.html
).

**Paraphrased Statement of Requirement:**

-   Supports different levels of access control, such as restrictions by
    service, session, distributed resource, database, record, or data
    element.

-   Supports authentication of user identity through an ancillary
    service (e.g., e‑Authentication).

-   Supports verification of the integrity of delivered data, metadata,
    or other information must be able to be verified as well.

-   Supports the search service standard for library catalogs accessible
    over network technologies, \[ISO 23950\] (identical to ANSI/NISO
    Z39.50)

-   Supports the library standard for catalog records, Machine-Readable
    Cataloging

-   Supports access to data without mandating proprietary technologies,
    nor proprietary vocabularies or thesauri

-   Can be readily accommodated by leading search products, including
    Internet search engines

-   Supports search of information that may be unstructured (often
    called \"full-text\"), semi‑structured (typically represented with
    inline \"markup\"), or structured (sometimes known as \"fielded\").

-   Supports search of HTML meta element contents and other varieties of
    metadata embedded within particular types of files (e.g., PDF,
    e-mail, etc).

-   Supports customizable search of other varieties of structured
    metadata through common mechanisms such as SQL and LDAP.

-   Provides for interoperable search across locators for information
    and collections of information

-   Interoperable with the international standard search service
    supporting the U.S. National Spatial Data Infrastructure
    Clearinghouse of geospatial data

-   Implementable over the Internet using TCP/IP, HTTP/HTTPS, HTTP GET
    and HTTP POST

-   Precisely defined as to how searches are expressed and communicated
    between a client component and a server component, including a query
    language, a query syntax, and standardization of a result set schema

-   Specified in an interface definition language such as Web Services
    Definition Language

-   Supports searching of structured information using a nested Boolean
    query, e.g., (date \> \'20040101\') AND ((subject = \'earthquake\')
    OR (subject = \'temblor\')).

-   Supports the usual sets of data structures (word, phrase, date,
    URL.) and relations (equal, greater than, less than).

-   Includes a query evaluation function to handle \"abstract concepts\"
    (e.g., name, category, date) according to what they mean
    semantically rather than merely how they may be labeled
    syntactically.

-   Supports abstract concepts that are produced by semantic mapping
    without requiring any particular semantic mapping technique

-   Supports gateway to Internet Anonymous FTP Archive (IAFA) file
    system catalogs and Distributed Authoring and Versioning for the Web
    (WebDAV)

-   Adopts readily to the underlying data model of named properties and
    property sets that is defined for objects addressable by software

-   Already in production use for searching metadata variants such as
    Dublin Core Metadata Initiative, ISO 15836 Encoded Archival
    Description, and ISO 8879Standard Generalized Markup Language (SGML)

-   Compatible with many and diverse approaches to compiling collections
    of information, without mandating any particular approach

-   Supports interoperable search of business and services registries,
    modeled on ISO 11179 Metadata Registries, ebXML, or the Universal
    Description, Discovery, and Integration (UDDI) model

-   Scalable in terms of supporting arbitrarily complex searches

-   Scalable in not foreclosing concurrent searches on multiple servers

-   Extensible to search tasks with unusual data structures and
    relations, definable through profiles or equivalent

-   Provides extension mechanisms to nurture innovation in areas not yet
    ready for the broadest level of standardization

-   Has been in use worldwide in many languages

-   Supports negotiation between client and server as to each other\'s
    language capabilities for the session

-   Supports character set negotiation, with Latin-1 as a minimum for
    U.S. Federal Government applications

### 5 ISO 23950 Overview 

ISO 23950, the international standard for information search and
retrieval, defines a particular set of network client-server
\"services\". The definition is powerful enough to accommodate the most
commonly required search functions over a broad range of search
facilities, including the requirements stated above. This section
provides an general overview of ISO 23950 using the particular variety
of ISO 23950 known as the \[SRW\] (Search and Retrieve for the Web)
profile.

By analogy to a restaurant, a network service in operation is like a
waiter handling a dinner order from a customer. Just as a customer is
not expected to give step-by-step instructions to the kitchen, the
ISO 23950 service allows the client to precisely specify the request but
does not allow the client to specify the exact procedure for satisfying
the request. This is an important feature for security as well as for
broad interoperability. Clients have no more control than necessary, and
clients need not know execution details.

For example, a searcher who wants to find what the Library of Congress
may have on \"fruit\" can send an ISO 23950 search request that looks
like this:

http://z3950.loc.gov:7090/voyager?operation=searchRetrieve&version=1.1&query=fruit

This SRW search syntax uses the Internet standard for URL\'s (RFC 1738).
The search request has two component parts: a \"base URL\" and a
\"searchpart\", separated by a question mark (\"?\"). The base
URL identifies the server host and port (here, \"z3950.loc.gov:7090\")
and the ISO 23950 service (here, \"voyager\"). The searchpart consists
of parameters separated by \"&\", each with the structure
\"key= value\". The names of the parameters from the ISO 23950 service
description are the \"key\" strings within the URL. (In this example,
the keys are \"operation\", \"version\", and \"query\".)

The ISO 23950 definition of a standard search syntax provides an obvious
level of interoperability. The example search statement could be applied
to several popular Internet search services in this way:

http://www.google.com/search?operation=searchRetrieve&version=1.1&query=fruit

http://search.yahoo.com/search?operation=searchRetrieve&version=1.1&query=fruit

http://alltheweb.com/search?operation=searchRetrieve&version=1.1&query=fruit

http://www.altavista.com/web/results?operation=searchRetrieve&version=1.1&query=fruit

http://vivisimo.com/search?operation=searchRetrieve&version=1.1&query=fruit

Without ISO 23950, a searcher would need to use the particular syntax
invented by each search technology vendor:

http://www.google.com/search?hl=en&ie=UTF‑8&oe=UTF‑8&q=fruit&\
btnG=Google+Search

http://search.yahoo.com/search?fr=fp-pull-web-t&p=fruit

http://alltheweb.com/search?cat=web&cs=utf8&q=fruit&\_sb_lang=pref

http://www.altavista.com/web/results?q=fruit&kgs=1&kls=0

http://vivisimo.com/search?query=fruit&v%3Asources=Web&x=0&y=0

This is only a trivial example of the variety of search syntaxes
supported by technology vendors, especially as most also support Boolean
operations with fielded searches. The bewildering variety of search
syntax has become a major barrier to search interoperability among
Internet search vendors, just as it was among library catalog search
vendors before agreements were reached on the international search
standard in the 1990\'s.

Following here is a bit more detail about the \"Common Query Language\"
\[CQL\] syntax used in the \"searchpart\" of an SRW search URL
introduced above. In CQL, a query can be as simple as an unqualified
single terms (\"fruit\" in the example above). Queries also may be
joined together using the Boolean \"and\", \"or\" operators, as in the
following example:

(bird or dinosaur) and (feathers or scales)

The Boolean \"not\" is used as a binary operator, finding records which
contain \"this but not that\". For example,

dinosaur not reptile

would find records which contain the word \"dinosaur\" but not the word
\"reptile\"\'.

In addition to queries targeted at whole records, queries can be limited
to a particular part of the records being searched. These searchable
parts are called \"indexes\" in CQL. For example, limiting a search to
the \"author\" index would find matches on the names of authors. An
index is specified in CQL as part of a set of indexes, in recognition
that different communities of practice sometimes have unique indexes.
For instance, both the bibliographic and the heraldry communities might
wish to name a \"title\" index, but those indexes would have different
meanings.

In ISO 23950 and CQL, an \"index\" is an abstract concept. A CQL query
that limits a search to the \"author\" index can be executed in various
ways by the server application. For an e-mail collection, the author
index may contain values taken out of the \"from\" field of e-mail
messages; For a news clipping collection, the author index may contain
values taken out of the \"by-line\" field in the news stories. This
abstraction is very important for achieving search interoperability.

### 6 Alternatives Considered 

**Single Portal** - From a management control perspective, there is a
certain attraction to having just one system encompassing all relevant
information. The manager of a portal might then focus just on
\"operability\" issues within the manager\'s own control, relegating
\"interoperability\" to \"someone else\'s problem\". Such a single
portal could be physically distributed, using various mechanisms for
\"pulling\" or \"pushing\" information, metadata, and update signals
among distributed components of a logically centralized system. Yet,
from a public policy perspective, the very idea of a single, master
portal is unrealistic. Any effective government organization must
accommodate relationships with other levels of government and with other
public and private sector information sources. Consequently, any single
portal must co-exist with other information portals, and so must support
a degree of interoperability. From a technology perspective also,
interoperability is a more appropriate approach. There are simply too
many mutually incompatible search mechanisms already in place to imagine
that any single solution could provide customized interfaces to all of
them. The need for interoperability is even more obvious when one
considers that many of those custom interfaces would be shaped by
distinct vendors who can change interface specifications at will.

**Common Data Model** - In the early days of mainframe computer systems,
it was common to envision an enterprise-wide \"management information
system\" that mandated a common data model applied to all enterprise
information systems. This approach is less stringent than subordinating
all systems into some master, all-encompassing system, but it still does
require central administration of an abstract and complex model shared
by all interacting systems. In practice, this approach suffers much the
same difficulties found in the \"single portal\" approach. Today\'s
reality is that any government organization must accommodate a great
variety of in‑house and external actors who evolve their component
systems independently. These largely independent systems already have
their own data models that often have little in common, even when a
single vendor has supplied the systems software.

**Applications Programming Interface (API)** - Software is an integral
part of most government information systems and software is implemented
through programs. Designers of complex systems usually divide software
into modules that are each provided with a published interface with
well-defined entry points for application programmers. Unfortunately,
such an API approach must be tailored to each distinct programming
environment. Now that there are many operating platforms and programming
languages, the programming interfaces needed for broad interoperability
have become too numerous to be manageable. However, the current
\"services‑oriented architecture\" approach underlying the present
recommendation does build on the programming discipline of the API
approach. The important difference is that \"services\" are based on the
characteristics of a network interface between interacting systems,
rather than being based on characteristics of the programming interface.
This is a great advantage for the set of problems encountered in
information search and retrieval.

**Structured Query Language (SQL)** - SQL has a long history of use,
starting with the first management information system efforts several
decades ago. When combined with an appropriate network service such as
ODBC (Open DataBase Connectivity), SQL can be used as part of a
services-oriented architecture. However, SQL by itself does not include
the essential idea of search indexes as an abstract mapping against
actual content structures. Also, SQL is oriented toward query of
database tables rather than Information Retrieval against very large
collections. An SQL query would result in a table having all records
that satisfy the search constraints; Information Retrieval would build a
\"result set\" giving a rank-ordered listing of records that satisfy a
search request, any of which might be actually retrieved in a separate
operation. Nevertheless, SQL is often used very effectively in
combination with the ISO 23950 international standard search service
recommended here.

### 7 Review Process Used

An iterative approach identified requirements for a search service
standard. A draft Statement of Requirements was posted publicly, with
comments of the major stakeholders invited, on the Internet
at http://www.search.gov/interop/requirements.html . Several revisions
were made over the months February - April, 2004. The initial version of
the Statement of Requirements was based on a recommendation in February
2003 of the E-Government Technical Committee of the Organization
for the Advancement of Structured Information Standards \[OASIS\]. The
OASIS recommendation was informed by, among others, an April 2002 white
paper titled \"Interoperability Strategy: Concepts, Challenges, and
Recommendations\" by the Industry Advisory Council \[IAC\], Enterprise
Architecture Shared Interest Group. (Focused on promoting
government-industry partnerships, the Industry Advisory Council
represents professionals from over 400 leading information technology
companies.)

### 8 Notes and References

\[CQL\] Common Query Language is described at
http://www.loc.gov/z3950/agency/zing/cql/

\[GILS\] E. Christian, Application Profile For The Government
Information Locator Service (GILS), http://www.gils.net/prof_v2.html ,
April 1997.

\[GEO\] D. Nebert, Z39.50 Application Profile for Geospatial Metadata or
\"GEO\", http://www.blueangeltech.com/Standards/GeoProfile/geo22.htm ,
May 2000.

\[IAC\] Dodd, John, at al, Interoperability Strategy: Concepts,
Challenges, and Recommendations,
http://www.search.gov/IAC-Interoperability.pdf , April 2003.

\[ISO 23950\] ANSI/NISO Z39.50-1995, Information Retrieval (Z39.50):
Application Service Definition and Protocol Specification,
http://lcweb.loc.gov/z3950/agency , 1995.

\[OASIS\] The OASIS E-Government Technical Committee recommendation
concerning Search Service interoperability is available at
http://www.oasis-open.org/committees/download.php/5846/wd-egov-searchservice-CD.pdf

\[SRW\] Search/Retrieve Web Service home page is
http://www.loc.gov/z3950/agency/zing/srw/