# Presentation: 465359

## Implementing a Government-wide Semantic Solution to Thesauri

- _[Kenneth B. Sall](mailto:sallk@saic.com)_, Science Applications International Corporation (SAIC) and
- _[Ronald P. Reck](mailto:rreck@rrecktek.com)_, RRecktek LLC
- April 20, 2006
- _[XML Community of Practice ](http://xml.gov/)_(XML CoP)
- Town Hall at the _[eGov](http://www.e-gov.com/)__[ Institute](http://www.e-gov.com/)_'s _[KM Conference](http://events.fcw.com/EventOverview.aspx?Event=KM06&NoCache=632744861000156250)_.

## Agenda

- Problem
- Goals and Requirements
- Basic Thesaurus Terminology and IC Examples
- SKOS (Simple Knowledge Organisation System)
- Our SKOS Element Subset and Extensions
- SKOSaurus Pilot
- DTIC Thesaurus Examples
- Potential Next Steps

## Problem Statement

- Government agencies need common vocabulary of (technical) terminology.
- Communication and data sharing is greatly enhanced when the semantics are clear.
- Various government groups approach this in different ways -- Microsoft Word, Excel, HTML, databases, and wiki pages: bulleted lists, tables, spreadsheets, acronym lists, etc.
- Need to focus on a common formats and standards that enable reuse and harmonization across Communities of Interest (COIs).

## Goals and Requirements

- Allow constraining terms to one COI or sharing across COIs.
- Should benefit from ISO standards for thesauri.
- Enable term authors to use familiar tools (e.g., Excel).
- Leverage existing Microsoft formats to access expressive backend data stores.
- XML-based (RDF) solution with few required elements but many optional and/or repeatable elements.
- Multiple definitions of the same term must be permitted, with either same or different subject/context.
- Should support semantic relationships between terms: “search thesaurus”.

## % Thesauri Standards and Specifications

**ISO 2788:1986** – Documentation - Guidelines for the establishment and development of monolingual thesauri
- Developing a Thesaurus (mono-lingual)
- ISO 5964:1986 – multi-lingual version
- ISO 1087:2000 - Vocabulary of Terminology
- ISO 704:2000 - Principles and Methods
**ANSI/NISO Z39.19-2003** - Construction, Format, and Management
- ISO 15836:2003 - The Dublin Core metadata element set
- [Many more listed in paper.]

## % Basic Thesaurus Terminology (1) [ISO 2788:1986]

*Thesaurus* – list of concepts in a particular domain of knowledge together with explicit relationships
*Concept* - unit of thought that exists in the mind as an abstract entity, independent of the term(s) that identify it  (i.e., human language independent)
*Concept Scheme* -  set of concepts, optionally including statements about semantic relationships between those concepts.
- Thesauri, classification schemes, subject heading lists, taxonomies, terminologies, glossaries and other types of controlled vocabularies

## Basic Thesaurus Terminology (2) [ISO 2788:1986]

- ISO 2788 defines abbreviations for each thesaurus construct. These “generally recognized conventions” are useful for compactness and in automated processing.
- USE (or SEE) – *preferred* label for this concept follows
- UF = USE FOR – *alternate* label follows, may be a synonym but less preferred
- e.g., ELECTRONIC INTELLIGENCE **UF** ELINT

- Preferred label

- Alternate label

## Basic Thesaurus Terminology (3) [ISO 2788:1986]

- SN = Scope Note - to clarify or constrain the meaning
- sometimes contains the concept’s definition
- BT = Broader Than – another concept more *general* than this concept
- NT = Narrower Than – more *specialized* than this concept
- RT = Related To – concept that is *similar* in some way

## Thesaurus Concept Example

- Source: GAO Thesaurus, Feb. 2005

## Example: DTIC Thesaurus (1)

*INTELLIGENCE*
-  NT  ACOUSTIC INTELLIGENCE
-  NT  COUNTERINTELLIGENCE
-  NT *ELECTRONIC INTELLIGENCE
-  NT  INTELLIGENCE(HUMANS)
-  NT *MILITARY INTELLIGENCE
-  NT  PHOTOGRAPHIC INTELLIGENCE
*INTELLIGENCE(MILITARY)  *
-  use MILITARY INTELLIGENCE

- _[DTIC ](http://stinet.dtic.mil/str/thesaurus.html)_
- _[Thesaurus](http://stinet.dtic.mil/str/thesaurus.html)_
- Italics added

- Source:
- Defense Technical Information Center

## DTIC Thesaurus (2)

*MILITARY INTELLIGENCE   *
-   EVALUATED INFORMATION CONCERNING AN ACTU
-   AL OR POSSIBLE ENEMY THEATER OF OPERATIO
-   NS.
-  UF  INTELLIGENCE(MILITARY)
-  BT  INTELLIGENCE
-  NT  AIR INTELLIGENCE
-  NT  ARMY INTELLIGENCE
-  NT  COMMUNICATIONS INTELLIGENCE
-  NT  ESPIONAGE
-  NT  NAVAL INTELLIGENCE
-  NT  STRATEGIC INTELLIGENCE
-  NT *TACTICAL INTELLIGENCE

## DTIC Thesaurus (3)

*ARMY INTELLIGENCE   **	*
-   INCLUDES EVERY PHASE AND HANDLING OF INF
-   ORMATION FROM ITS EVALUATION COLLATION,
-   SYNTHESIS, INTERPRETATION AND PRESENTATI
-   ON, TO ITS DISSEMINATION BY THE ARMY.
-  BT *MILITARY INTELLIGENCE
*COMINT  *
-  use COMMUNICATIONS INTELLIGENCE
*COMMUNICATIONS INTELLIGENCE *
-   TECHNICAL AND INTELLIGENCE INFORMATION D
-   ERIVED FROM FOREIGN COMMUNICATIONS BY OT
-   HER THAN THE INTENDED RECIPIENTS.
-  UF  COMINT
-  BT *MILITARY INTELLIGENCE

## DTIC Thesaurus (4)

*COUNTERINTELLIGENCE *
-  BT  INTELLIGENCE
*ELECTRONIC INTELLIGENCE *
-   THE TECHNICAL AND INTELLIGENCE INFORMATI
-   ON DERIVED FROM FOREIGN NONCOMMUNICATION
-   S ELECTROMAGNETIC RADIATIONS EMANATING F
-   ROM OTHER THAN NUCLEAR DETONATIONS OR RA
-   DIOACTIVE SOURCES.
-  UF  ELINT
-  BT  INTELLIGENCE
-  NT  RADAR INTELLIGENCE
*ELINT   *
-  use ELECTRONIC INTELLIGENCE

## DTIC Thesaurus (5)

*NAVAL INTELLIGENCE  *
-  BT *MILITARY INTELLIGENCE
*RADAR INTELLIGENCE  *
-   INTELLIGENCE CONCERNING RADAR OR INTELLI
-   GENCE DERIVED FROM THE USE OF RADAR EQUI
-   PMENT.
-  BT *ELECTRONIC INTELLIGENCE
*STRATEGIC INTELLIGENCE  *
-  BT *MILITARY INTELLIGENCE
*TACTICAL INTELLIGENCE   *
-  BT *MILITARY INTELLIGENCE
-  NT  TERRAIN INTELLIGENCE

- Note: We will see the data from these DTIC slides later in SKOSaurus.

## DTIC Thesaurus (6) – Navigation Interface

- _[DTIC ](http://stinet.dtic.mil/str/thesaurus.html)_
- _[Thesaurus](http://stinet.dtic.mil/str/thesaurus.html)_

## Example: CALL Thesaurus (1)

- _[CALL Thesaurus](http://call.army.mil/thesaurus.asp)_

- Concept: _[intelligence](http://call.army.mil/products/thesaur_e/00002397.asp)_

## CALL Thesaurus (2)

- _[CALL Thesaurus](http://call.army.mil/thesaurus.asp)_

- Concept: _[finished ](http://call.army.mil/products/thesaur_e/00019191.asp)_
- _[intelligence](http://call.army.mil/products/thesaur_e/00019191.asp)_

## FEA Business Reference Model (BRM)

- _[Intelligence](http://www.whitehouse.gov/omb/egov/a-3-2-services-intell.html)_
- _[Operations](http://www.whitehouse.gov/omb/egov/a-3-2-services-intell.html)_
- in the BRM

## SKOS (Simple Knowledge Organisation System)

- Leverages ISO 2788 (and ISO 5964) by defining an RDF vocabulary based on the ISO standards [implied].
- Defines an XML element (SKOS property) for each thesaurus construct (USE, UF, BT, NT, SN, etc.) and many more.
- _[Semantic Web Best Practices and Deployment Working Group](http://www.w3.org/2001/sw/BestPractices/)_ – W3C
- _[SKOS Working Drafts](http://www.w3.org/2004/02/skos/)_ (W3C) and Related Efforts
- SKOS Core Guide
- SKOS Core Vocabulary Specification
- Quick Guide to Publishing a Thesaurus on the Semantic Web
- Also: SKOS Mapping, Extensions, API, Development Wiki

## SKOS Core

- _[SKOS Core ](http://www.w3.org/TR/swbp-skos-core-guide)_- model for expressing structure and content of *concept schemes*:
- Thesauri
- Classification schemes
- Subject heading lists
- Taxonomies
- “_[Folksonomies](http://en.wikipedia.org/wiki/Folksonomy)_”
- Other types of controlled vocabulary
- Concept schemes are also embedded in glossaries and terminologies.

- Source: SKOS Core Guide, November 2005

## SKOS Core Vocabulary

- _[SKOS Core Vocabulary](http://www.w3.org/TR/swbp-skos-core-spec)_
- Application of _[Resource Description Framework (RDF)](http://www.w3.org/RDF/)__ _
- Can be used to express a concept scheme as an RDF graph
- Can be linked to and/or merged with other RDF data by semantic web applications
- Uses _[RDFS Classes ](http://www.w3.org/TR/rdf-schema/#ch_classes)__and RDF Properties to describe Concepts and Concept Schemes_
- 26 Properties and 5 Classes

- Source: SKOS Core Guide, November 2005

## % SKOS Vocabulary: 5 Classes

- 	_[CollectableProperty](http://www.w3.org/TR/swbp-skos-core-spec/#CollectableProperty)__[Collection](http://www.w3.org/TR/swbp-skos-core-spec/#Collection)__[Concept](http://www.w3.org/TR/swbp-skos-core-spec/#Concept)_   *_[ConceptScheme](http://www.w3.org/TR/swbp-skos-core-spec/#ConceptScheme)_   (*)_[OrderedCollection](http://www.w3.org/TR/swbp-skos-core-spec/#OrderedCollection)_

- * = the classes implemented in SKOSaurus pilot
- Source: SKOS Core Vocabulary, November 2005

## % SKOS Vocabulary: 26 Properties

- 	_[altLabel](http://www.w3.org/TR/swbp-skos-core-spec/#altLabel)_   *_[altSymbol](http://www.w3.org/TR/swbp-skos-core-spec/#altSymbol)__[broader](http://www.w3.org/TR/swbp-skos-core-spec/#broader)_   *_[changeNote](http://www.w3.org/TR/swbp-skos-core-spec/#changeNote)__[definition](http://www.w3.org/TR/swbp-skos-core-spec/#definition)_   *_[editorialNote](http://www.w3.org/TR/swbp-skos-core-spec/#editorialNote)__[example](http://www.w3.org/TR/swbp-skos-core-spec/#example)_   *_[hasTopConcept](http://www.w3.org/TR/swbp-skos-core-spec/#hasTopConcept)__[hiddenLabel](http://www.w3.org/TR/swbp-skos-core-spec/#hiddenLabel)__[historyNote](http://www.w3.org/TR/swbp-skos-core-spec/#historyNote)__[inScheme](http://www.w3.org/TR/swbp-skos-core-spec/#inScheme)__[isPrimarySubjectOf](http://www.w3.org/TR/swbp-skos-core-spec/#isPrimarySubjectOf)_
- 	_[isSubjectOf](http://www.w3.org/TR/swbp-skos-core-spec/#isSubjectOf)_

- 	_[member](http://www.w3.org/TR/swbp-skos-core-spec/#member)__[memberList](http://www.w3.org/TR/swbp-skos-core-spec/#memberList)__[narrower](http://www.w3.org/TR/swbp-skos-core-spec/#narrower)_   *_[note](http://www.w3.org/TR/swbp-skos-core-spec/#note)__[prefLabel](http://www.w3.org/TR/swbp-skos-core-spec/#prefLabel)_   *_[prefSymbol](http://www.w3.org/TR/swbp-skos-core-spec/#prefSymbol)__[primarySubject](http://www.w3.org/TR/swbp-skos-core-spec/#primarySubject)__[related](http://www.w3.org/TR/swbp-skos-core-spec/#related)_   *_[scopeNote](http://www.w3.org/TR/swbp-skos-core-spec/#scopeNote)_  *_[semanticRelation](http://www.w3.org/TR/swbp-skos-core-spec/#semanticRelation)__[subject](http://www.w3.org/TR/swbp-skos-core-spec/#subject)_   *_[subjectIndicator](http://www.w3.org/TR/swbp-skos-core-spec/#subjectIndicator)__[symbol](http://www.w3.org/TR/swbp-skos-core-spec/#symbol)_

- * = 9 properties implemented in SKOSaurus pilot
-            Source: SKOS Core Vocabulary, November 2005

## Our SKOS Element Subset and Extensions (1)

**skos:Concept** – contains all statements about properties for a given concept
**skos:prefLabel** – USE; preferred handle for this concept; designator. [In SKOS, no two concepts in the same *concept scheme* may have same prefLabel.]
**skos:altLabel** – UF; alternate handle; spelling variants; *can* be used for abbreviations or acronyms (but we don’t)
**skos:related**, **skos:narrower**, **skos:broader** – associated with, more specific, or more general than this concept

## Our SKOS Element Subset and Extensions (2)

**skos:scopeNote** – constrains meaning; ISO 2788 allows definitions to appear here (but we don’t)
**skos:definition** – statement or formal explanation of the meaning of a concept
**skos:example** – used in a sentence
**skos:subject** – topic; can be a skos:broader

## Our SKOS Element Subset and Extensions (3)

- Pilot Extensions (non-SKOS)
**ABBREVIATON_OR_ACRONYM** – very common government need (could define as rdfs:subPropertyOf skos:altLabel)
**SOURCE** - official document names and URLs are preferred, but specific names of people or agencies are acceptable; (probably could define as rdfs:subPropertyOf skos:note)
**COI** – essentially a skos:Collection (with a potential skos:ConceptScheme)

## SKOS Fragment: Military Intelligence

-  <skos:Concept rdf:about="http://ex.com/MILITARY_INTELLIGENCE#concept">
-   <skos:prefLabel xml:lang="en">MILITARY INTELLIGENCE</skos:prefLabel>
-   <skos:altLabel xml:lang="en">INTELLIGENCE(MILITARY)</skos:altLabel>
-   <skos:definition xml:lang="en">EVALUATED INFORMATION CONCERNING AN ACTUAL OR POSSIBLE ENEMY THEATER OF OPERATIONS</skos:definition>
-   <skos:scopenote xml:lang="en">IC</skos:scopenote>
-   <skos:source xml:lang="en">DTIC Thesaurus</skos:source>
-   <skos:subject rdf:resource="http://ex.com/intelligence#concept"/>
-   <skos:narrower rdf:resource="http://ex.com/AIR_INTELLIGENCE#concept"/>
-   <skos:narrower rdf:resource="http://ex.com/ARMY_INTELLIGENCE#concept"/>
-   <skos:narrower rdf:resource="http://ex.com/ESPIONAGE#concept"/>
-   <skos:narrower rdf:resource="http://ex.com/NAVAL_INTELLIGENCE#concept"/>
-   <!-- etc (several NTs omitted for space reasons) -->
- </skos:Concept>

## SKOSaurus Pilot

- Proof of concept
- Many simplifying assumptions
- Fabricated data (except for DTIC)
- About 100 man hours
- Ron Reck and Ken Sall
- Presented at XML 2005 Conference (Nov. 2005)

## % SKOSaurus Pilot: Environment

- CGI script issues SOAP requests and uses RMI.
- The host operating system is Microsoft Windows XP with Service Pack 2.
- Dell Latitude D800 (1.69GHz) with 1G of RAM.
- The Windows XP host runs VMware 5.0 build 13124 to emulate a machine onto which the Solaris X86 operating system version 10 is installed.
- This is referred to as the guest operating system which runs the SKOSaurus system, consisting of:
- Perl version 5.8.7 and various Perl modules
- Java version 1.4.2.08
- Kowari server 1.1.0 Pre2
- XSLT stylesheets

## Main Use Cases (for Pilot)

- Concept Entry via Web Form
- File Upload of Excel Spreadsheet (as CSV)
- File Upload of SKOS (or RDF)
- Query of Concept Data Store

## RDF Graph: Bird Example

**RDF Graph: Bird Example**

**Back to **_[SKOSaurus](slide14.xml)_

## % Illustrative Statements from RDF Graph

- An alternate label (skos:altLabel) for "bird" is "Aves".
- The concepts with the preferred label "vertebrate" and "animal" are broader than the concept with the preferred label "bird".
- There are four specializations of birds listed ("robin", "hawk", "sparrow" and "eagle"), each indicated as skos:narrower than "bird".
- The concepts "lizard" and "reptile" are skos:related to the "bird" concept in some way.
- Among various concepts which might have the skos:prefLabel of "bird", the one illustrated is constrained to ornithology, according to skos:scopeNote. This distinguishes the concept from "bird", such as in the informal term for a (young) woman.

## % OWL Statements About SKOS

**skos:broader owl:inverseOf skos:narrower**
**skos:narrower owl:inverseOf skos:broader**
- and
**skos:broader ****is an**** owl:TransitiveProperty**
**skos:narrower ****is an**** owl:TransitiveProperty**
- RDF/OWL version of SKOS Core

## Example Spreadsheet: birds

## Single Row of Spreadsheet

## % Spreadsheet Conventions (Pilot)

- One row per concept, sparse or densely populated.
- New row for different definition or homonym (e.g., bird). [SKOS conflict: no duplicate prefLabels.]
- The heading row should not be removed or modified.
- Column order is invariant.
- Since several elements are repeatable, use semi-colon to indicate iteration. [Configurable.]
- A limitation in our pilot parser requires the author to use the pipe symbol ("|") instead of a comma within a cell. [Config.]
- Any number of rows can be included, but there must be no blank rows or separator rows.
- File > Save As Comma Separated Values (*.csv).

## SKOSaurus: Home Page

## SKOSaurus: Manage COIs

## SKOSaurus: Upload CSV or SKOS

## SKOSaurus: Upload Feedback

- Generated SKOS files

- Datastore for COI

## SKOSaurus: Generated SKOS Excerpt (1)

## SKOSaurus: Generated SKOS Excerpt (2)

## SKOSaurus: Web Form

## SKOSaurus: Kowari Model Dump: Query

## SKOSaurus: Kowari Model Dump: Result

## SKOSaurus: Intuitive Search

## SKOSaurus: DTIC Data: 17K concepts (in 73K lines)  70K SKOS statements

- Source: Defense Technical Information Center

- DTIC
- Thesaurus

## Intuitive Search 1:  Organizations (prefLabel)

## Organizations: Military Organizations (NT)

## Organizations: Mil Orgs: Military Reserves (NT)

## SKOS “definition” Property

- Note: Definitions not shown in other screenshots.

## Intuitive Search 2: Intelligence (prefLabel)

## Intelligence:  Military Intelligence 

## Intelligence:  Military Intelligence: narrower 

## (Intelligence: MILINTEL) or (Unconventional Warfare) : Subversion: Espionage

## Unconventional Warfare: Subversion: Terrorism

- So What?

## Now, Connect the Dots!

**Intelligence**

**Military Intelligence**

- Subversion

- Terrorism

**Unconventional Warfare**

- Espionage

- Air
- Intel

- Terrain Intel

- Strategic
- Intel

- Naval Intel

- Army
- Intel

- Communications
- Intel

- Tactical Intel

- Acoustic
- Intel

- Electronic
- Intel

- Photographic
- Intel

- Intelligence(Human)

- Counterintel

- Sabotage

- Learning

## Graphical Interface: Arlington Library – AcornWeb.org

## How Does This Help Solve IC Problems?

- Allows concept descriptions in human-friendly Microsoft Office formats.
- Converts relationships to XML-based format; can manipulate with common XML tools.
- XML is really RDF and SKOS, which are *machine-friendly* formats. RDF was designed to be manipulated by machines.
- Semantic Web – moving from humans to machines; *we want computers to do the work for us.*
- SKOSaurus concepts are ripe for integration with commercial semantic tools: METS, Content Analyst, Siderean, Factiva, Images, etc.

## Potential Next Steps

- Class of problems with CONOPS (next slide)
- Graphical interface (topicmap-like?)
- Add on Reporting Tools (paper and graphics) – analytic searching, display portions, etc.
- Search across COIs
- Access control mechanisms
- Edit existing concepts
- Ingesting other common formats
- Integration with commercial semantic products

## Potential Next Steps: Class of Problems

- Glossary management
- De-confliction (detection and resolution)
- Data Reference Model (DRM) artifact problem
- Conceptual Taxonomies
- Conceptual Model to Logical Model
- CONOPS to be developed for specific applications of these concepts

## Summary and Conclusions

- Semantic concepts help reveal less than obvious associations.
- SKOS is a useful vocabulary for implementing a thesaurus.
- The U.S. Government would benefit from a unified approach to thesauri, especially when sharing terminology within and across Communities of Interest.
- Our approach assumes government term authors want to work in Excel, not XML/RDF/SKOS (although we permit SKOS upload).
- Other SKOS implementations are worth considering (e.g., Java-based NBII SKOS Thesaurus client).
- We hope W3C considers SKOS for the Recommendation Track.

## Resources: Semantics and Thesauri

- SKOS home page     http://www.w3.org/2004/02/skos/
- XML 2005 Conference Proceedings and Slides
- W3C Semantic Web Activity home page
- Willpower Glossary of Terms Related to Thesauri
- W3C Semantic Web News and Events Archive
- SKOS: A language to describe simple knowledge structures for the web – A. Miles, XTech 2005 or paper
- SKOS Core Tutorial for DCMI 2005 – A. Miles, or PDF
- NBII SKOS Thesaurus
- SICoP: Semantic Interoperability (XML Web Services) Community of Practice – Brand Niemann et al
- Sall’s Earlier Glossary Work