# Presentation: 717365

## XMDR Prototype Progress Report:11179 metadata & inference interface

**XMDR Prototype Progress Report:****11179 metadata & inference interface**

**John McCarthy and Karlo Berket**
**XMDR Project Quarterly Meeting**
**26 April, 2006**
**UC Berkeley Faculty Club**

**Notes:**
A new name on the marque/masthead/title page credits
Karlo Berket joined LBL’s XMDR Project group as a part time programmer in January
He comes to us from the Cybersecurity and Network :& Distributed Systems Research Group at LBNL,
which continues to occupy and pay for the other half of his working hours.
I might add that both Karlo and Kevin are serious telecommuters – Karlo from MA and Kevin from NY,
But they keep in close touch with us via email, weekly teleconferences, and our project wiki

## Progress Report Outline

**Summarize recent progress**
**Fixed bugs and limitations of earlier prototype version**
**Created XMDR Prototype distribution package**
**And began process for open source licensing**
**Began including 11179-ed 2 registry metadata in XMDR prototype**
**tabular specifications for mappings from external files into XMDR**
**Began mapping for EPA Environmental Data Registry database**
**Began steps to load some caDSR metadata **
**Created web interface for inference queries**
**Inference query results share output displays**
**Including forward and backward links from results**
**Demonstrate inference interface & queries**
**Discuss next steps & major challenges**

**Notes:**
Prepared this with Larry in mind as the architype for primary audience

Tell story about painful upgrade from T23 to T43
Analogy to upgrading 11179 to include more formal semantic information
We’ve been hearing about inference from Frank for a number of years (Prolog, etc.)

Problem that had a silver lining
Karlo machine crashed, so he had to re-install—did it on xmdr server and revised recipe

## Fixed bugs & limitations

**linux and Postgres upgrades **
**Enabled incorporation of GEMET metadata**
**System fix**
**Bugs that arose for NCI Thesaurus metadata**
**Special character “+” in URI eliminated**
**Data fix**
**Detailed results for elements referenced as URI fragments (with “#”)**
**Used to show results for root element of URI**
**Worked around special character “#” in URIs**
**Application fix**

**Notes:**
Each of these bug fixes was accomplished by “fixing” different aspects of the XMDR prototype:
 -the first was specific to the computer system on which the software runs on (and the dependencies of the XMDR software)
 -the second was specific to the data loaded into the software - the NCI Thesaurus files on disk had a space in their name => the base namespace had a space in it; Lucene does not like this, so whoever created that data changed the space in the base namespace to a +; browsers do not like this (+ is a reserved character) => I changed the + to a _ in the base namespace and changed the space to a _ in the file names (otherwise inconsistencies between the text and semantic index happen) --- this is data specific since the + character should not arise in URIs that point to non-dynamically generated data (assuming we’re not trying to deal with that)

 -the third was specific to the application - the # character is common in URIs that describe our data => had to fix it in the app => split the URI around # and added additional parameter for passing the URI fragment (string after #)

## Created XMDR Prototype Distribution Package

**Revised & expanded documentation**
**Began assembling package of introductory materials**
**e.g., for Chris Mungall**
**See list on next page**
**Streamlined installation procedures**
_**[erdos](http://erdos.lbl.gov/mediawiki/index.php/)**__**[.](http://erdos.lbl.gov/mediawiki/index.php/)**__**[lbl](http://erdos.lbl.gov/mediawiki/index.php/)**__**[.](http://erdos.lbl.gov/mediawiki/index.php/)**__**[gov](http://erdos.lbl.gov/mediawiki/index.php/)**__**[/](http://erdos.lbl.gov/mediawiki/index.php/)**__**[mediawiki](http://erdos.lbl.gov/mediawiki/index.php/)**__**[/index.](http://erdos.lbl.gov/mediawiki/index.php/)**__**[php](http://erdos.lbl.gov/mediawiki/index.php/)**__**[/](http://erdos.lbl.gov/mediawiki/index.php/)**_** **_**Build_and_Installation**_**    (login/password required)**
**demonstrate on-line**
**Notes on known bugs, new data & features**
**Included as bottom link in new interface**
**Karlo will discuss licensing later**

## Introduction to the XMDR Project: selected overview documents

_**[www.xmdr.org/](http://www.xmdr.org/)**_** **
_**[hpcrd](http://hpcrd.lbl.gov/SDM/XMDR/overview.html)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/overview.html)**__**[lbl](http://hpcrd.lbl.gov/SDM/XMDR/overview.html)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/overview.html)**__**[gov](http://hpcrd.lbl.gov/SDM/XMDR/overview.html)**__**[/SDM/XMDR/overview.html](http://hpcrd.lbl.gov/SDM/XMDR/overview.html)**_** (link from xmdr.org)**
_**[hpcrd](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**__**[lbl](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**__**[gov](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**__**[/SDM/XMDR/presentations/XMDR_Elevator_ Summary_rough_draft.](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**__**[ppt](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR_Elevator_Summary_rough_draft.ppt)**_**  (overview) **
**xmdr.lbl.gov/xmdr/    (prototype system) **
_**[hpcrd](http://hpcrd.lbl.gov/SDM/XMDR/arch/index.html)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/arch/index.html)**__**[lbl](http://hpcrd.lbl.gov/SDM/XMDR/arch/index.html)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/arch/index.html)**__**[gov](http://hpcrd.lbl.gov/SDM/XMDR/arch/index.html)**__**[/SDM/XMDR/arch/index.html](http://hpcrd.lbl.gov/SDM/XMDR/arch/index.html)**_**   (architecture) **
_**[erdos](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[.](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[lbl](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[.](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[gov](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[/](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[mediawiki](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[/index.](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[php](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**__**[/Main_Page](http://erdos.lbl.gov/mediawiki/index.php/Main_Page)**_** (project wiki)**
_**[hpcrd](http://hpcrd.lbl.gov/SDM/XMDR/presentations/)**__**[.lbl.gov/SDM/XMDR/presentations/](http://hpcrd.lbl.gov/SDM/XMDR/presentations/)**_**   (esp recent ones) **
_**[hpcrd](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**__**[lbl](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**__**[.](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**__**[gov](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**__**[/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**__**[ppt](http://hpcrd.lbl.gov/SDM/XMDR/presentations/XMDR-Prototype-Status-Oct-2005.ppt)**_**   (status report) **

## XMDR Content: need to add Data Element Metadata to Ontologies 

## Began incorporation of metadata from EPA’s Environmental Data Registry

**Sucessfully loaded Oracle .dmp file for EDR db**
**Oracle 9.1 .dmp file loaded into Oracle 10.1 database**
**110 tables, 1300 columns**
**Contacted technical liaison at EDR contractor**
**Ready to help if we need additional information**
**Found ER diagrams for 11179 metadata objects**
**(thanks to Doug Mann!)**
**Next steps will be to construct draft mappings**
**ala mappings for caDSR**
**more challenging because EDR is not in XML**
**but we can create XML from VIEWs in Oracle 10.1 **

## High level 11179 ER diagram forEnvironmental Data Registry database 

## More high level 11179 ER diagram forEnvironmental Data Registry database

## Began mapping & loading caDSR metadata into XMDR prototype

**xmdr.lbl.gov/caDSR-xmdr-mapping/**

## caDSR mapping details

**Demonstrate active outline at **_**[xmdr](http://xmdr.lbl.gov/caDSR-xmdr-mapping/)**__**[.lbl.gov/caDSR-xmdr-mapping/](http://xmdr.lbl.gov/caDSR-xmdr-mapping/)**_

## caDSR and EDR registry metadata:next steps & proposed approach 

**caDSR web interface can get selected 11179 metadata**
**but not full set of objects and attributes**
**caDSR API can access full 11179 registry metadata**
**but will take a bit longer to implement**
**EPA Environmental Data Registry will take longer**
**use Oracle tool for creating XML from views?**
**Upgrade XMDR metamodel before loading more data?**
**Tabular specification of mapping from caDSR to XMDR**
**For high level, understandable description/documentation**
**XSL-T scripts to do mappings from caDSR and EDR**

## Designed & implemented web interface for inference queries

- http://xmdr.lbl.gov/xmdr/

## Notable features of XMDR Advanced Inference Search

**You don’t have to know SPARQL**
**but you can see the generated SPARQL query**
**Each search component has pop-up help screen **
**Choice of reasoners **
***None, Jena transitive, OWL micro, RDFS simple, etc*****.**
**Restrict search to target object type**
***e.g., ontology, data element, concept, value domain, etc. ***
**Restrict search by object attributes or links**
***e.g., administrativeStatus, EffectiveDate, etc.***
**Combines some elements of XMDR text search**
***phrases, words (all, at least one, without), strings***
**Simple output summary & control**
***Result count, specify number displayed per screen***
***Show results as web addresses, literals, or both***

## Inference query results now share output display with text queries

## Info & backward links lead to more information with additional links

## DEMO & DISCUSS

**Segue to on-line demo**

## What should near term priorities be for XMDR Prototype work?

**More data – esp 11179 registries, i.e. EPA-EDR, caDSR**
**Update XMDR metamodel & data to reflect 11179 revisions**
**and explore how to handle future model revisions**
**Improve tools & procedures for data loading & mapping**
**Further enhancements to XMDR User Interface **
**FRESNEL (from MIT Simile project, works with Jena) **
** ****Longwell – faceted browser from the Simile project **
**selective transitive closure queries for ****i. exact match; ****ii. all nodes above or below current node; ****iii. all nodes within specified number of arcs; **
**Use Subversion to store xmdr data (rather than unix files) **
**connect subversion to interface & compare performance vs files**
**Version control for updates & additions using Subversion **
**Impact of incremental data additions on models, etc.?**

## Longer term challenges & future goals for XMDR prototype system

**Additional prototype features**
**Extensions to 11179 metamodel**
**Acquire & develop complementary tools**
**Analysis and evaluation of component modules**
**Expand relationships with other organizations**

## Extend prototype features

**Mapping (among concepts, data elements, etc.)**
**axiomatic methods, graph & query based representations**
**Validation**
**Use of externally maintained sources (data, ontologies)**
**Common Logic, Web Services, etc**
**Ontology Lifecycle Management**** – versions & semantic drift**
**Generate schemas from axiomitized ontologies**
**Harmonization with ODM and MMF**
**Integrate management of semantics, data, and content**
**Version management**
**Service oriented architecture & semantic grid**** **

## Extend 11179 Metamodel

**Type system -- semantic types & type constraints on relationship roles**
**Axioms for specification of type semantics**
**Relationships among codesets (set-based on conceptual domain)**
**Recommended set of semantic relations  (is-a, part-of, spatial, ...) **
**Support for derived data (e.g., arithmetic syntax)**
**More support for constraints**
**Key constraints (functional dependencies)**
**Inclusion dependencies**
**Arithmetic constraints (within object, among objects, aggregates****)**
**More formal specification of registration processes**
**Semantic specification of services, i/o types, pre & post conditions**
**Support for composite data elements,  collection data elements**
**Support for power sets of values  (for TC 184) **
**Elaboration of contexts**
**Detailed treatment of measurement units**

## Tools

**User-friendly interface for RDF inference queries**
**Something like EDR UI with link labels & inverse references**
**Form interface for registration & uploading metadata?**
**Registry access services, query facilities, etc.**
**RDF tools won’t create XMDR files (add wrappers?)**
**Handling multiple registries within single registry server **
**MDR for federated databases**
**Extraction, Translation &Loading (ETL) metadata **
**aggregation operators for derived tables (statistical/OLAP) **
**XBRL support for tables, etc.**

## Analysis and evaluation

**Identify more technologies**** for use with XMDR**
**Test more complex ontologies & metadata**
**Scalability & performance **
**currently everything runs in main memory**
**Prototype currently includes ~2.5 million triples**
**Compare different search engines & reasoners?**
**Alternative technologies for other modules**
**Explore quads for support of reification in RDF **
**Revision of RDF standard to support quads **

## Expand relationships with other organizations & standards

**Current active developers**
**U MD (Joel Sachs, Sandor Dornbush)**
**NCI – Mayo Clinic (Denise, Harold)**
**Other collaborating organizations**
**EPA**
**USGS**
**DOD**
**Potential future collaborators**
**?European Ecoinformatics (Palle Haastrup)?**
**CWM,  now IMM standards/registries from OMG **
**NCBO (Chris Mungall)**
**HL7 (Harold Solbrig)**
**NCBI**
**CL **
**Cheshire (Ray Larson)**

## END, Thanks, Acknowledgements

**This is the end of the presentation**
**Slides following this one can be **
**folded back into the mainline presentation,**
**Held in reserve if questions arise they can help**
**Dropped altogether**

## Other Topics? Extra Slides...

## *XMDR Prototype Architecture: Initial Implemented Modules

**11179 Metamodel as OWL Ontology**

***XMDR Prototype Architecture: Initial Implemented Modules**

**Registry**

**External**
**Interface**

**Java**

**RetrievalIndex**

**FullTextIndex**

**Lucene**

**LogicBasedIndex**

**Jena**
**RDFS,etc.**

**RegistryStore**
**(possibly remote)**

## *XMDR RDF graph query facilities complement text query capabilities

**SQL-like structured queries**
*e.g., **SELECT ?x  WHERE (?x rdf:type xmdr:ValueDomain)*** **
**Spans items that are only indirectly connected**
*e.g., data elements associated with a conceptual domain*
**Expand queries to subsumed classes in hierarchy**
*e.g., all cities within state and states within countries*
**Transitivity**
*e.g., all subclasses subsumed by a higher order class*
*e.g., all superclasses (ancestors) of a particular class*
**Least common ancestor**
*e.g., closest subsuming concept for 2 concepts*
*(minimal generalization)*
**Siblings**
*e.g., other airport codes comparable to “SFO”*

## *Reasoners use OWL ontologies to augment RDF graph queries

**RDF Query**
- (rdql/nrdql/_SPARQL_)

**Reasoners**
- Jena
- (knows RDF & OWL)
- (main memory)

**result set**
*Includes tuples*
*With subclasses,*
*inverses, etc.*

- Jena is a Java framework for building Semantic Web applications.
- Jena provides a programmatic environment for RDF, RDFS and OWL, and a rule-based inference engine. Jena is open source and grown out of work with the HP Labs Semantic Web Programme.
- Introduction: _[www-128.ibm.com/developerworks/java/library/j-jena/](http://www-128.ibm.com/developerworks/java/library/j-jena/)_
- Jena API Overview: www.jdocs.com/jena/2.3/api/overview-summary.html

- 11179 metadata (xml/rdf/owl files)

- OWL built-in rules

- OWL 11179 Metamodel Ontology

- Several choices

## *XMDR example content has been loadedfrom diverse sources via lexgrid & XSLT

- Original Source A

- Lexgrid Source A

- XSLT
- script

- Harold Solbrig
- (Mayo Clinic)

***XMDR example content has been loaded****from diverse sources via lexgrid & XSLT**

- Concept System A

- A Concepts

- A Relationships

- [temporary problems loading GEMET]

## Other XMDR example content has been loaded from EDR and SWEET

***screen ***
***scrape ***
***(perl)***

**concepts & relationships**

**XMDR ontology**

**SWEET (OWL)**

***java***

**EDR**

**XMDR files**
**(includes ontologies)**

**XMDR files**
**(ontologies)**

## Results are URI’s to xmdr tuples with LINKS to related object tuples

**<****DataElement**** ****[...] ****   ****xml:base="**_**erdos.lbl.gov/xmdr2/data/**_**DEALL.1.5394.1.xml">****[...] ****        <sign>Country Name</sign> ****[...] ****  <type rdf:resource="RCDIS.1.12116.1.xml"/>****RepresentationClass link****  <domain rdf:resource="VDALL.1.15147.1.xml"/> ****ValueDomain link**** ****  <meaning rdf:resource="DCDIS.1.12800.1.xml"/> ****DataElementConcept link****  <example rdf:datatype="&xsd;string">United States</example> ****</DataElement>**** **

**<****ValueDomain**** ****[...] ****   ****xml:base="**_**erdos.lbl.gov/xmdr2/data/ **_**VDALL.1.15147.1.xml">**

**Metadata schema includes relationships that specify which attributes can or must link to other entity-types**

**Index contains names of entities & links**

**Kevin: show HTML rendition**

## Reasoner software is still somewhat “bleeding edge”

**OWL Web Ontology Language**
**W3C Recommendation February 2004**
**W3C Reasoners investigation (March 2004)**
_**[www.w3.org/2003/08/owl-systems/test-results-out](http://www.w3.org/2003/08/owl-systems/test-results-out)**_
**No pre-OWL reasoners passed all of the tests**
**Pre-OWL reasoners took some time to modify **
**New tools are just starting to be available**
**Jena may have some serious limitations for XMDR**
**Loads everything into main memory**
**We may also investigate alternatives (e.g., Kowari)**

## Preliminary conclusions from comparative analysis to date

**Jena does not construct the full graph for InfModel (model plus reasoner) **
**until the results of the first  query posed to that model are iterated**
**	****Hence first query posed  against an InfModel can take time **
** ****	****	****(depends on size of data and  type of reasoner)**
**     ****But subsequent queries on an InfModel are fast (equal to DB  queries) **
**     ****So... Priming of InfModel is a good idea.**
**Investigated 2 ways of merging Models in Jena: **
**	****subsequent  unions of models **
**	****create a model from the union of the underlying graphs. **
** ****Which is optimal depends on reasoner that will be  used, **
**	****if using RDFS simple then 2nd way is faster, **
**	****if  using OWL micro then 1st is better (2nd does not complete)**
**Hybrid approach may be optimal for when using all reasoners**
**	****take graph  union of all but one model & create model from it**
**	****then union of that model with remaining model**

## Extending the XMDR Advanced Search Interface to include inference

**Extending the XMDR Advanced Search Interface to include inference**

- More Results>>

- XMDR Web Interface 0.4, LBNL

*Search for* "any:(+country +(code name))"

**xmdr.lbl.gov/xmdr/**

## How to deal with objects that have both OWL and XMDR encodings? 

## ***(meta)data used for XMDR inference testing and analysis

**What do we call the entitites/info globs/instances of classes?       2.6 million triples**
**Show on OWL class diagram or outline**
**Many per file, each as XML structure or 1/ file?**
**"EDR Data Elements", "DCDIS?", "DEALL?", and "VDALL?" are actually all part of EDR.  **
** ****EDR ConceptSystem derived from the EDR.  **
**DEALL = Data Elements, one in each file (499).  **
** ****one ConceptualDomain (CDDIS), **
**two RepresentationClasses (RCDIS), and **
**three Contacts (CONDIS).  **
** ****total number of files for EDR is 1051. ****/srv/www/tomcat5/base/webapps/xmdr> **
**"Wetland?" is a ConceptualDomain and a ValueDomain that I created myself in order to illustrate a use of the Concepts defined in SWEET. **