# Presentation: 445200

## Information Integration the Web Way

- Andrew Schain, Kendall Clark

## Background

- It all started innocently enough, I was at a conference 8 months ago and heard about a problem Jeanne was having...
- It is an interesting problem.

## The Problem

- It is virtually impossible to discover critical information that you did not know existed and extremely difficult to find relevant information you *are* aware of.
- Our data problem exists within at least 5 dimensions; size, complexity, diversity, rate of growth and trust.
- Use-case scenarios and requirements change _all the time_.
***We cannot anticipate in advance what the next collection of information elements need to be or for what purpose!!***

## The challenge

- Integrate information from disjoint data sources, ad hoc’ly, to solve customer needs
- Without upsetting delicate info-ecologies (data owners, curators, extant policies & procedures)
- Without requiring major investment in time or $$

**Notes:**
Traditional forms-driven, single-purpose database applications are difficult to validate, expensive to maintain, and resistant to integration with new data sources. 

Must “fit” within the NASA technical and social environment. Must be reusable!!!

## The inspiration...

## The Goal

- Alleviate NASA’s data management problem by making information discoverable with machine assistance, and retrieved as an integrated response across different databases, repositories, sources and systems.

## The design principles

## Aggregate and federate information
Deploy a service that makes the whole infrastructure smarter
Leverage public standards
Innovate in the user interface
Formalize information models

- Deploy a _*service*_ that makes the whole infrastructure _*smarter*_
- Leverage public _*standards*_
- Innovate in the _*user*_ _*interface*_
_*Formalize*_ information models

**Notes:**
So it can be discoverable and reused

The service - takes information that was locked in databases and makes it available for for others 

We make the data smarter - the whole is greater than the sum of the parts

## Info Federation

- Must leave data *in situ*, close to those who know it best
- Must not upset *delicate info-niches* by alienating curators, owners, or violating policies politics
- Use a sufficiently expressive federation technology (in this case, W3C’s RDF)

## POPS is an expertise locator app, but...
Also a service, deployed in the fabric of NASA application infrastructure
Thus, the POPS data is reusable by other apps
Lower barriers to ad hoc reuse

- Also a service, deployed in the fabric of NASA application infrastructure
- Thus, the POPS data is reusable by other apps
- Lower barriers to *ad hoc* reuse

- Deploy a service

**Notes:**
So, adding 6 in a federation it becomes easier to export and link to other stuff

## Leverage public standards

- Why? *Humility*, *laziness*, & *tiny budget!*
- Promotes reuse, cohesion with existing technologies
- Open Source software is our friend
- Return on Investment
- They work! :)

## HTTP, SOAP, WSDL, SPARQL Protocol for RDF
XML, RDF, RDFS, JSON, OWL
SPARQL Query Language for RDF
FOAF, DOAP
Atom Syndication Format

- XML, RDF, RDFS, JSON, OWL
- SPARQL Query Language for RDF
- FOAF, DOAP
- Atom Syndication Format

*Some* candidate public standards

**Notes:**
Owl for same as - inverser functional properties 

## Be different? Look-feel-and-act different.
JSpace, a Polyarchical Visual Query Builder for Federated RDF Stores
Social Network visualizations
What the hell is a polyarchy?

- JSpace, a Polyarchical Visual Query Builder for Federated RDF Stores
- Social Network visualizations
- What the hell is a polyarchy?

- Innovative UI

## JSpace

- A *polyarchy* is a means of interacting with “multiple intersecting hierarchies”
- Which is precisely what many information integration problems are (people + orgs + projects + skills)
- The backend is only _*half*_ the problem

**Notes:**
The front end is the other half!! mSpace solved that in large part - - 

## Visual Query Builder?

- Folks can learn a QL, but why?
- Get the machine to build queries based on _*regular and customary*_ user input: **browsing**
- Browsing better than searching
- Add query-by-example (find another thing like this thing except with this difference...)
- Propinquity!!!

**Notes:**
Q-by-Example So, you find something like “me” and then do a match against me and change or highlight the one property and resubmits the query 

## Formalizing Information

- Use OWL (Web Ontology Language) to formalize the problem domain(s)
- Why?
_*Correctness*_, create shared understanding, regulatory compliance (DRM)
- To prepare for the eventual semantic _*upshift*_

**Notes:**
Whats the up shift? 
Everyone is on the same page

Eventually we willl want to do the same-as 

## Other stuff we probably need:

- Model Libraries
- Data Access agreements
- Data assurance
- Model assurance
- Good “go to” application models
- Desire & commitment
*Lots more*

## What’s next for POPS?

- We have buy-in and project plan with the OCE. We will validate our agreement (plan)for implementation within the NEN and have it done before I go on vacation in August
- Continue working with Clark Parsia; Kendall, Bijan, Jen Golbeck, Mike Grove, Chris Shenton & others including  folks on my SAIC  team to build a similar service at HQ for EA “as-builts”.

## What about the rest of us?

- Let’s throw a party!
- For our comrades who are current practitioners
- Give them a blank piece of paper and write down stuff that would make things easier and stuff that makes things really hard
- Invite some folks we want to make friends with and work the list together

## Examples

## Models in Federated Libraries

- Domain specific references that can be used by developers
- Domain specific information representations (complete with logic, cardinality, etc) that can be used to form queryable information that cuts across sources
- Code repositories
- Web Services repositories so that task-orientated computing services can be discovered, assessed, choreographed, and orchestrated

## Data access agreements

- Between who and who (and who is keeping track?)
- Valid?
- Has it gone thru a validity checker y/n?
- Current?
- Is it fresh? (may not need to be) but we need to know
- Provenance?
- Who is the responsible person for the system and for the data?
- Access Permissions?
- Given the set of data required, does the access permission change?

**Notes:**
Actually, you know what would be great...if there was a standard request/security/permission process for gaining access to those NASA "data storage" systems that people frequently want to reuse.  It seems like there could be a reasonable list that could be considered the standard set that would be manageable.Also, if a data storage system wanted to be considered to be a universal NASA Agency system, the "owners" would need to agree: 1) to sharing data through out the Agency and 2) to keep the data as clean and accurate as possible.  This would really cut down the IT costs for developing and managing redundant database systems.This is EXACTLY what we will need to figure out how best to have a federated information model. We will need agreements as well as an understanding of currency (some data can be cyclical some must be fresh all must be understood), validity (has the source data been validated, has the information model been validated), and provenance (something regarding the authorship and hereditary aspects are useful).

## But really to:

- 	Articulate the goal
- 	Develop the planning, including gathering requirements, prioritizing tasks, identifying resources, and setting up a road map for the next few years.
- 	Some of you will be invited, if not to the first one (April) to the lollapalooza InterOp

## Backups

- We define five types of relationships: 1) same facility, 2) same department, 3) same skill and department, 4) same skill and project, 5) same skill, project, and facility. Call these r1 - r5.
- rixy indicates a relationship of type i between person x (px) and person y (py)
- There is a direct connection between users pu and ps if there exists an rmus
- If there is not a direct connection, we search for a path from pu to ps by finding pa such that there exists rmua, rnas. Then, we add (pu, ps, pa, rmua, rnas) to the graph.
- For example, if Alice is the user and Bob is the selected person, we will look for a direct relationship between them, such as if Alice and Bob both work in the same department (i.e. find rmalice,bob).
- If the direct relationship does not exist, we look at all the people Alice has relationships with, and check to see if any of them also have relationships with Bob. For example, Alice may work in the same facility as Chuck (r1alice,chuck). Chuck, in turn, may have the same skill and work on the same project as Bob (r4Chuck,Bob). Chuck then becomes a connection between Alice and Bob. All three people and their relationships are added to the graph.

- Mathematics of the who-knows-who relationship visualization