# Presentation: 009267

## Guy Divita
Chris Lu

- Chris Lu

## SPECIALISTLexicon

**SPECIALIST****Lexicon**

**Lexical Tools**

**Term Lookup**

**Spelling Suggestion**

**Corpus ****and Document Based Tools**

**Text-to-Concept Mapping Tools**

**Indexing Tools**

- Term Based Tools

**Text Categorization**

**Sentence/Phrase/ Term/Word Tokenizers**

**POS Tagging**

## SPECIALIST.nlm.nih.gov

**About**

_**Projects**_

_**Documents**_

- The Lexicon

- Lexicon
- Term Lookup

-  Term Manipulation Tools

- Spelling Suggestion

- Document Tokenization Tools

- Text Categorization Tool

- POS Tagger

- MetaMap Transfer (MMTx)
*mmtx.nlm.nih.gov*

## {base=child 

- {base=child
- entry=E0016427
- 	cat=noun
- 	variants=irreg|child|children|
- }
- {base=Child
- entry=E0355216
- 	cat=noun
- 	variants=reg
- 	variants=uncount
- 	proper
- }

- Warning: This record’s content has been  modified to fit this screen.

- Lexicon

##       SPECIALIST Lexical Tools

- _[Word Indexing](http://umlslex.nlm.nih.gov:8100/WebLvg.2005/jsp/wordInd/wordInd.jsp?type=WordInd)_

- _[Term Normalization](http://umlslex.nlm.nih.gov:8100/WebLvg.2005/jsp/norm/norm.jsp?type=Norm)_

- _[Variant Generation](http://umlslex.nlm.nih.gov:8100/WebLvg.2005/jsp/lvg/lvg.jsp?type=Lvg)_

## Uses

- Term Transformation
- Query Expansion
- Term normalization
- Building indexes
- Component of controlled vocabulary building tools
- Syntactic parsing
- Component of search engines
- Component of text-to-concept mapping tools
- Component of automated document indexing tools
- Component of  text summarization tools
- Component of data-mining tools

##      Lexical Variant Generation

- treat

- treatment

- treatments

- treats

- treatability

- treated

- treating

- treatable

- treaty

- treater

**inflections**

**nominalizations**

**derivations**

**combinations**

##   Lexical Variant Generation

***  ******Lexical Variant Generation***

- color

- colors

- colored

- colorless

- colorful

- inflections

- nominalizations

- derivations

- combinations

- colour

**Spelling variants**

- chromatic

**synonyms**

- coloring

- colorant

- colorlessness

- Chromaticness

##   Lexical Variant Generation

***  ******Lexical Variant Generation***

- second

- seconded

- secondary

- inflections

- nominalizations

- derivations

- combinations

- s

**acronyms**

- synonyms

- seconds

- secondly

- secant

- SOR

- s’s

- Ser

- serous

- sec

**Acronym**
**expansions**

##   Lexical Variant Generation

**Input **
**term**

**Remove possessive**

**lowercase**

**Strip punctuation**

**Remove stop words**

**Strip diacritics**

**Word order sort**

**Output **
**term**

**The tools can be arranged so that the output of one is the input to another.**

***  ******Lexical Variant Generation***

**Notes:**
Here’s an example of such a serial combination. That is, 
where the output of one tool serves as
 the input to the next tool.
In this example, we are first lowercasing the input, then stripping diacritics, removing possessives, removing stop words, stripping punctuation, and finally sorting the resulting words in ASCII sort order.  
The example shows just how this flow is invoked on the command line. 

## Norm abstracts away from: 
case
punctuation
word order
stop words
possessive forms
inflectional variation
spelling variation
 normalizes diacritics/ligatures/symbols

- case
- punctuation
- word order
- stop words
- possessive forms
- inflectional variation
- spelling variation
-  normalizes diacritics/ligatures/symbols

***Term Normalization***

**Notes:**
Our form of normalization involves the following:

It abstracts away from case, punctuation, and word order. 
It abstracts away from inflectional variations. 
It abstracts away from possessive forms.

## Word order
Upper left lobe of lung
Left upper lobe of lung 
Possessive Forms
Grave’s Disease
Graves Disease
Graves’ Disease
Diacritic/Ligature/Symbol Normalization
entrée, anæsthesia, β-blockers, Medline®

- Upper left lobe of lung
- Left upper lobe of lung
- Possessive Forms
- Grave’s Disease
- Graves Disease
- Graves’ Disease
- Diacritic/Ligature/Symbol Normalization
- entrée, anæsthesia, β-blockers, Medline®

-       ***Term Normalization Examples***

## Hodgkin Disease
HODGKINS DISEASE
Hodgkin's Disease
Disease, Hodgkin's
HODGKIN'S DISEASE
Hodgkin's disease
Hodgkins Disease
Hodgkin's disease NOS

- Hodgkin Disease
- HODGKINS DISEASE
- Hodgkin's Disease
- Disease, Hodgkin's
- HODGKIN'S DISEASE
- Hodgkin's disease
- Hodgkins Disease
- Hodgkin's disease NOS

- disease hodgkin

- Note:  A normalized form is not necessarily itself a readable term. It is a hash.

- Normalization web tool

-  ***Term Normalization Example***

- Hodgkin's disease, NOS
- Disease, Hodgkins
- Diseases, Hodgkins
- Hodgkins Diseases
- Hodgkins disease
- hodgkin's disease
- Disease;Hodgkins
- Disease, Hodgkin

##  Lgt, a GUI Example

##  Command Line Example

**> lvg –f:i**
**leave**
**leave****|****leave****|128|1|i|1|**
**leave|****leave****|128|512|i|1|**
**leave|****leaves****|128|8|i|1|**
**leave|****left****|1024|64|i|1|**
**leave|****left****|1024|32|i|1|**
**leave|****leave****|1024|1|i|1|**
**leave|****leave****|1024|262144|**
**leave|****leave****|1024|1024|i|1**
**leave|****leaves****|1024|128|i|1**
**leave|****leaving****|1024|16|i|1|**

**> lvg –f:i –SC -SI**
**leave**
**leave****|****leave****|<****noun****>|<****base****>|i|****1****|**
**leave|****leave****|<****noun****>|<****singular****>|i|**
**leave|****leaves****|<****noun****>|<****plural****>|i|1|**
**leave|****left****|<****verb****>|<****past****>|i|1|**
**leave|****left****|<****verb****>|<****pastPart****>|i|1|**
**leave|****leave****|<****verb****>|<****base****>|i|1|**
**leave|****leave****|<****verb****>|<****pres1p23p****>|**
**leave|****leave****|<****verb****>|<****infinitive****>|i|**
**leave|****leaves****|<****verb****>|<****pres3s****>|i|1**
**leave|****leaving****|<****verb****>|<****presPart****>|**

## leave

**Output Term**

- _[Categories](http://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/current/docs/designDoc/UDF/bitMask/category.html)_

- _[Inflections](http://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/current/docs/designDoc/UDF/bitMask/inflection.html)_

**Flow history**

**Flow Number**

-  128

- 1

-   1

-  i

- |

- |

- |

- |

- |

- > lvg –f:i

***Command Line Example:***
***Output Fields Explained***

** ****Outline of the needed pieces:**
-  import gov.nih.nlm.nls.lvg.Api.*;
-  NormApi api = new NormApi();
-  Vector<String> out = api.Mutate(inStr);
-  api.CleanUp();

***SPECIALIST Lexical Tools: API’s***

**import java.util.*;**
**import gov.nih.nlm.nls.lvg.Api.*;**
**public class simplestApi**
**{**
**	****public static void main(String[ ] args)**
**	****{****	****	****	**
**	****	****NormApi api = new NormApi( );    // instantiate a NormApi object**
**	****	****try  // Process**
**	****	****{**
**	****	****	****Vector<String> out = api.Mutate(“inputs”);**
**	****	****	****for(int i = 0; i < out.size( ); i++) // print out result**
**	****	****	****{**
**	****	****	****	****System.out.println(out.elementAt(i));**
**	****	****	****}**
**	****	****}**
**	****	****catch (Exception e) {  }**
**	****	****api.CleanUp(); ****	****// make sure to clean up**
**	****}**
**}**

***Norm API Example***

## Norm API Example (2)

** ****To compile and run:**
**CLASSPATH = ${CLASSPATH}:**
**    ****${LVG_DIR}:**
**    ****${LVG_DIR}/lib/*****lvg2007dist.jar:***
**    **

***Norm API Example (2)***

## Application

**Metathesaurus**
**English**
**Strings**

**norm**

**Normalized string index**

**Normalized word index**

**WordInd**

**MRXNS.ENG**

**MRXNW.ENG**

**Notes:**
Here’s where the lexical tools fit in with the UMLS Metathesaurus.
The norm program normalizes the Metathesaurus strings to produce the normalized string index.
Each normalized string is also associated with three types of identifiers, a concept identifier (CUI), a lexical term identifier (LUI), and a string identifier (SUI). These ids are keys into all the other tables.  
The normalized string index is then broken into a normalized word index via wordInd, carrying along the identifiers.

## Application

**norm**

**Normalized string index**

**Normalized word index**

**Metathesaurus**
**Concepts**

**Query**

**Normed**
**term**

**SUIS**

**Metathesaurus**
**concepts that match**
**the normalized query**

**Notes:**
Here is a example of the application. 
Application developers put the norm program between queries and the Metathesaurus. 
First of all, user input a term to search for Metathesaurus concept.
Term are normalized and form the query.
Normalized queries are then used get entries from either the normalized string or word index to get the ids to retrieve Metathesaurus concept information. 
Metathesaurus concept information is then returned to satisfy the request.

## Example

**norm**

**dry eye syndrome**

**Dry Eyes Syndrome**

**Notes:**
Here’s a simple minded example. Suppose the query is Dry eyes syndrome. 
The query gets normalized.

## Example (Cont.)

- ENG|**dry eye syndrome**|C0013238|L0013238|**S0035652**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0090228**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0090454**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0220550**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0368350**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S1459074**|

**Normed**
**term**

_**Example (Cont.)**_

**Notes:**
All entries that share the normalized form are retrieved from the normalized string index. 
The string unique identifiers to these rows are used as key to retrieve the String name

## Example (Cont.)

- C0013238|ENG|P|L0013238|VS    |S0004019|Dry eye syndrome
- C0013238|ENG|P|L0013238|VS    |S0368350|Dry Eye Syndrome
- C0013238|ENG|P|L0013238|VS    |S1459074|dry eye syndrome
- C0013238|ENG|P|L0013238|VWS|S0090228|Syndrome, Dry Eye
- C0013238|ENG|P|L0013238|VWS|S0220550|Dry, eye syndrome
- C0013238|ENG|P|L0013238|VW  |S0090454|Syndromes, Dry Eye

**SUIS**

**MRCON**

**C0013238|ENG|P|L0013238|****PF****  |S0035652| Dry Eye Syndromes**

_**Example (Cont.)**_

**Notes:**
To retrieve Metathesurus concept information. 
If one of the things you want to return are the concept names, not the string names for the query, you can review all the returned strings to see which ones are the concept names. 
I’ve got one of these rows highlighted that indicates that it’s the preferred term for that group of strings, and that group of strings is considered the preferred group for that concept. 
All told, this is what we consider the concept name.

## Slide 25

**Notes:**
GSpell. This is a set of spelling suggestion resources. Two tools are included in this set: GSpell which is a word and term based spelling suggestion tool, and Bag-O-WordsPlus, which is a phrase retrieval tool that uses GSpell when it needs to, but, as the name implies, uses a bag of words approach to retrieve phrases from a target, and only relies on gspell, when none of the words of the query are found in the target as spelled.  

Both tool’s usage and api will be reviewed and the output of GSpell will be elaborated upon to show what kind of information is returned. 

##  Spelling Retrieval Tools

**GSpell**
- A term retrieval tool
- N-gram nearest neighbor algorithm
- MetaPhone phonetic spelling normalization
- Homophones
- Common misspellings
- Candidates sorted by an edit distance and frequency of occurrence from a corpus
**BagOWordsPlus**
- a phrase retrieval tool
-  uses correctly spelled words within the phrase to limit possible candidates
- uses GSpell only when it has to.

## GSpell: Usage

*Usage*
- 	**GSpellFind**.[*sh*|*bat*]
- 		--**dictionary**=*NameOfDictionary*
- 		[--**inputFile**=*Source*] [--**outputFile**=*target*]
- 		[--**truncate**=*N*] [--**considerNCandidates**=*N*]
- 		[--**maxEditDistance**=*N*]
**   ****GSpellIndex**.[*sh*|*bat*]
- 	  --**dictionary**=*NameOfDictionary*
- 	  --**inputFile**=*SourceFile*
- 	  [--reportTime] [--version][--help]

**Notes:**
Here’s GSpell’s Usage.  This emplies a dictionary has been previously indexed, and you know the name of that index. GSpell is distributed with the current year’s lexicon as an index.

GSpell has options to fiddle with to tune at will. 

## GSpell: Output

-  anonomous|**anonymous**|1.0|0.87|NGrams|
-  anonomous|**allonomous**|2.0|0.58|NGrams|
-  anonomous|**autonomous**|2.0|0.58|NGrams|
-  anonomous|**anadromous**|3.0|0.29|NGrams|
-  anonomous|**analogous**|3.0|0.29|NGrams|
-  anonomous|**anomalous**|3.0|0.29|NGrams|
-  anonomous|**anonymously**|3.0|0.29|NGrams|
-  anonomous|**anonymes**|3.0|0.29|Metaphone|
-  anonomous|**anonyms**|3.0|0.29|Metaphone|
-  anonomous|**acoprous**|4.0|0.11|NGrams|

| 
Input Term | 
Suggestion | Edit
Distance | 
Rank | 
Method | 
Message |
| --- | --- | --- | --- | --- | --- |

**Notes:**
Here is a sample output from GSpell.  The output consists of the input term, the suggestion, the edit distance, a rank, the method that retrieved the suggestion and a message.  

Edit distance is the minimum number of edits or transfermations that transforms the query term into the target term. These transformations include character removal, character substitution, and the swapping of two characters.  Each transformation has a cost of 1. The Edit distance is the sum of the transformations necessary.

A rank is returned as well. This rank is the result of an equation employing the edit distance, with the properties that it has the range from 0 to 1, and the rank gets quickly  smaller after two edit distances quickly approaching 0 by 4 edit distances.

A message, such as “correct” is given when a correctly spelled term is put in. 

## GSpell: API

**import** gov.nih.nlm.nls.gspell.**GSpell**;        // <-------These come from the gspell.jar
**import** gov.nih.nlm.nls.gspell.**Candidate**;
**GSpell** gspell = new **GSpell**( _dictionaryName, 					     GSpell.READ_ONLY );
**Vector **candidates = **gspell.*****find*****( aTerm );**
**if** ( candidates != null )
-   for ( int i = 0; i < candidates.length; i++ )
- 	  System.out.println(candidates[i].toString());
-  **else**
-     System.out.println("**No Suggestions**");
- gspell.cleanup();

**Notes:**
Here’s how you might embed GSpell into your app. Like the other samples we’ve shone, a new instance is needed. 
The GSpell constructor needs to have the name of the dictionary passed in, and a read-only flag passed in. 

The find method takes a string as input, and returns a vector of candidates. A Candidate object contains all those things that we just saw on the output of the prior slide, with methods to get at each. See the java doc for the details.  

## SPECIALIST Text Tools

- LexicalLookup

- NpParser

- VariantLookup

- POS tagger

- Document
-  Indexing

## SPECIALIST TextTools

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

- Title

- Abstract

**Segments text into:**
- Sections
- Sentences
- Phrases
- Terms
- Words
- Term variants

- Java Objects

## Specialist TextToolsContainer Classes: Entity Diagramgov.nih.nlm.nls.nlp.textfeatures

**Sentence**

- Vector getPhrases()
- Vector getLexicalElem
- Vector getTokens()

- Contains many relationship

**Phrase**

- Vector getLexicalElemements()
- Vector getTokens()

**Lexical Element**

- getTokens()

- These are terms

**Document**

- Vector getSections()
- Vector getSentences()
- Vector getPhrases()
- Vector getLexicalElem
- Vector getTokens()

- These are words

## Specialist TextToolsContainer Classes: Entity Diagram, more detailsgov.nih.nlm.nls.nlp.textfeatures

**Variant**

- String getTerm()
- Vector getTokens()
- int getCategories()
- int getDistance()
- int getHistory()
- String getNormalizedTerm()
- int getOrigCat()
- String getOrigTerm()
- LexicalElement getParent()

**Document (continued)**

- Doccument()
- Document( File pFile)
- Document(GlobalBehavior pSettings)
- Document(String pFileName)

**Collection**

- Vector getDocuments()

- Collection()
- Collection(GlobalBehavior pSettings)
- Collection(StringBuffer pText)
- Collection(String pFileName,
-                   GlobalBehavior pSettings)

## SPECIALIST TextToolsLexicalLookup

- Title

- Abstract

***LexicalLookup***
- segments text into
- Sections
- Sentences
***Terms***
***Lexical Entries***
- Words

## SPECIALIST TextToolsNPParser

- Title

- Abstract

- Segments text into
- Sections
- Sentences
***Noun Phrases***
- Terms
- Words
- Lexical Entries

- Java Objects

## NpParser --fileName=PMID14700470.txt
...
Section: 8|Title Words|
Sentence|0|0|212|The knowledge and expectations of parents about the role of antibiotic treatment in upper respiratory tract infection - a survey among parents attending the primary physician with their sick child.
Phrase|0|0|12|The knowledge|knowledge|2|true|NOUN PHRASE
	Lexical Element|0|LEXICON|det|The|0|2|false
	Lexical Element|1|LEXICON|noun|knowledge|4|12|true
Phrase|1|14|16|and|and|1|false|CONJUNCTION_PHRASE
	Lexical Element|2|LEXICON|conj|and|14|16|false
Phrase|2|18|29|expectations|expectation

- ...
**Section**: 8|Title Words|
**Sentence***|0|0|212|The knowledge and expectations of parents about the role of antibiotic treatment in upper respiratory tract infection - a survey among parents attending the primary physician with their sick child*.
**Phrase**|0|0|12|The knowledge|**knowledge**|2|true|NOUN PHRASE
- 	**Lexical Element**|0|LEXICON|det|The|0|2|false
- 	**Lexical Element**|1|LEXICON|noun|knowledge|4|12|true
**Phrase**|1|14|16|and|and|1|false|CONJUNCTION_PHRASE
- 	**Lexical Element**|2|LEXICON|conj|and|14|16|false
**Phrase**|2|18|29|expectations|**expectation**

*SPECIALIST TextTools**NPParser Example*

## SPECIALIST TextToolsClasses

## SPECIALIST TextTools Common Methods

- processCollection(Collection pCollection )

- Method Summary

- void

- processDocument(Document pDocument )

- void

- processSentence(Sentence pSentence )

- void

- processSentence(String pString )

- Sentence

## SPECIALIST TextToolsSpecial Sauce

**NLPRegistry.cfg:  Example Contents:**
-                -015|**--annotationFormat1**|boolean|false|Simple Annotation format

**Command line arguments: Example Contents:**
-                                                       **--annotationFormat1**

## SPECIALIST TextToolsExtract Terms from Documents

**// =============+ Create a LexicalLookupAPI object +==**
**LexicalLookupAPI** **look** = **new** **LexicalLookupAPI**(args);
**// =============================+ Chunk the file +==**
**Document** aDocument = look.***processDocument***( aFile );
**List** terms = aDocument.***getLexicalElements***();
**LexicalElement** **aTerm** = **null**;
**// ================+ Print the LexicalElements out +==**
**for** (**Iterator** **i** = terms.iterator(); i.hasNext(); ) {
- 	aTerm = (LexicalElement) i.next();
- 	System.out.println(aTerm.***toPipedString***());
- }

## SPECIALIST TextToolsVariantLookup

- This is LVG’s fruitful variants index available as an API within the textTools
- Is used to generate variants from noun phrases extracted from documents

- gov.nih.nlm.nlp.lexicon.**VariantLookup**

- VariantLookup( gov.nih.nlm.nls.utils.GlobalBehavior pSettings )

- Constructor Summary

- Variant[] find( String pTerm)

- Method Summary

- Variant[] find( String pTerm, int pCats, int pVarTableType)

## SPECIALIST TextToolstaggerClient

- Assigns Parts of Speech (POS) to words in text
- NP parsers need terms with Parts of Speech assigned to determine phrase breaks and head assignment
- Includes LexicalLookup

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

- noun

- verb

- prep

- adj

- adv

- det

- conj

- shape

- pron

*Legend:*

## SPECIALIST TextToolsTaggerClient

## SPECIALIST TextToolstaggerservices

- gov.nih.nlm.nlp.taggerservices
**Interface TaggerInterface**

- tag( Sentence pSentence )

- Method Summary

- build(GlobalBehavior pSettings )

- Class gov.nih.nlm.nlp.taggerservices.**TaggerFactory**

- static TaggerInterface

**NLPRegistry.cfg:  Example Contents:**
-       -043|**--tagger**|String|*medpostskr*|Name of tagger hooked in

## Extracts UMLS concepts from text
Java Implementation of MetaMap

- Retinoblastoma is a rare type of eye cancer that develops in the retina, which is the part of the eye that detects light and color. Although this disorder can occur at any age, it usually develops in young children.

- Extracts UMLS concepts from text
- Java Implementation of MetaMap

- Meta Mapping (1000):
-   C0496836
-  (Malignant neoplasm of eye, unspecified)
- [Neoplastic Process]

- MetaMap Transfer (MMTx)

**Notes:**
MetaMap, a tool developed by the National Library of Medicine, extracts concepts from free-text documents, such as MEDLINE abstracts, and maps them to candidate concepts in NLM’s UMLS Metathesaurus. MetaMap is a rese
MetaMap Transfer (or MMTx) is a Java implementation of MetaMap.  MMTx is the version of MetaMap that is distributed.
As illustrated here, the term “eye cancer” has been mapped to the UMLS concept represented by the term “Malignant neoplasm of eye, unspecified.” 
Because these tools map extracted terms to the UMLS Metathesaurus, the most comprehensive repository of medical knowledge available, I’ll now briefly describe the structure of the Metathesaurus.

## Specialist TextToolsContainer Classes: Entity Diagramgov.nih.nlm.nls.nlp.textfeatures

**Candidate**

- String getCUI()
- String getUMLSConceptName()
- String getCandidateScore()
- UMLS_SemanticTypePointer[]
-                      getSemanticTypes()
- String getSuis()
- UMLSSourceInfo[] getSources()

**FinalMapping**

- List getCandidates()
- int   getScore()

**Phrase (cont.)**

- Vector getLexicalElements()
- Vector getTokens()
- Vector getAllVariants()
- ArrayList getBestMappings()
- Vector getCandidateList()

## MetaMap Transfer (MMTx)

- _[Tokenization](slide21.xml)__[ ](slide21.xml)_
- Section, Sentence, Phrase, Term, Word

**UMLS Concept Retrieval**

**UMLS Concept to Phrase Evaluation**

- _[Variant Generation](slide11.xml)_

**Final Mapping of Good Candidates to best cover the phrase**

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- Abstract. The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

## MetaMap Transfer (MMTx)Display Mappings

**// ======================+ Display Phrase and Concepts +==**
**String** **displayPhrase**( **Phrase** **aPhrase** ) throws **Exception** {
-    ** **
**// =================================+ Get the Mappings +==**
**List** **finalMappings** = aPhrase.*getFinalMappings*();
- if ( finalMappings != null ) {
**  ****Iterator** **mappingIterator** = finalMappings.iterator();
**  ****// ===================+ Iterate through the Mappings +==**
-   **while** (mappingIterator.hasNext()) {
-    **FinalMapping** **aMapping**=(**FinalMapping**) mappingIterator.next();
-      System.out.println( aMapping );
-   }
-  }
- }

## MetaMap Transfer (MMTx)MMTxAPI

**// ==========================+ Create a MMTxAPI object +==**
**MMTxAPI** **mmtx** = **new** **MMTxAPI**(  );
**// =============================+ Analyze the Sentence +==**
**Sentence** **aSentence** = mmtx.***processSentence***("Insomnia is a symptom of a sleep disorder" );
**Iterator** **phraseIterator**=aSentence.***getPhrases***().iterator();
**// ======================+ Iterate through the Phrases +==**
**while** ( phraseIterator.hasNext() ) {
-   **Phrase **aPhrase = (**Phrase**) phraseIterator.next();
-   System.out.println( displayPhrase( aPhrase) );
- }

## MetaMap Transfer (MMTx)MMTxAPI

- Phrase: "non-hodgkin's lymphoma"
- Meta Candidates (5)
-   1000 ***Lymphoma, Non Hodgkin's*** [Neoplastic Process]
-     861 hodgkin's lymphoma (***HODGKINS DISEASE***)
-                                                                     [Neoplastic Process]
-    812 Lymphoma (***Germinoblastoma***) [Neoplastic Process]
-    812 ***Lymphoma*** [Neoplastic Process]
-    805 NON (***NON Mouse***) [Mammal]
- Meta Mapping (1000)
-   1000 ***Lymphoma, Non Hodgkin's*** [Neoplastic Process]

## Training

- Lexical Lookup

- Tagging

- Updating

- Features

- Introduction

- Handling Unknowns

- Error Analysis

- Work Still to Do

- _[specialist.nlm.nih.gov](http://specialist.nlm.nih.gov/)_

## Motivation

- Why another POS tagger?
- SPECIALIST Lexicon
- Arbitrary tag set
- Supervised and unsupervised training and updating
- True multi-word (term based) tagger
- Generalizable to other languages

- Word based
-  taggers

- Term based
-  tools

## Features 

- Tag set specified as a configurable file
- Just make sure Lexicon/tagset/corpus use the
- same tags.
- Lexical Information (from .lex files)
- SPECIAST lexicon
- Pseudo lexicon (number words, roman numerals)
- Verbs as adjectives
- Local lexicon

## Features (2)

- Lexical lookup, pattern recognition to tokeninize into terms
- Unknown words handled with several strategies
- Hidden Markov Model, Viterbi Model used for training and tagging
- Probability of correct tag assignment reported back

## Lexical Lookup

- diabetes mellitus
- diabetes Atherosclerosis Intervention Study
**diabetes gravidarum**
- diabetes insipidus

- .. on  diabetes gravidarum diagnosis and treatment

- on

- Longest spanning matches from Lexicon

- diabetes gravidarum

- on

- diagnosis

- and

- treatment

## Handling Unknowns

- Overall probability of the next word being an unknown word gathered for (open class) words with low frequency
- Suffix statistics gathered from annotated corpus or from Lexicon (for unsupervised training) for open class terms.

## Handling Unknowns: Shape Identification

## Handling Unknowns: Shape Identification (2)

- Units of Measure

- Number

- Word Number

- Level of Significance

- Experiment Size

- EmailAddress

- URL

- Telephone Number

- Range

- Real Number

- Percent

- Fraction

- Glob

- Handling Unknowns: Shape Identification (2)

- Equation

- Gene Name

- Name

- Address

- Chemical

- Proper Name

- Delimiter

- *Next or future release

## Unsupervised Updating and Training

- Unsupervised Updating
- With a prior model and lexicon and an un-annotated corpus
- Even 10 hand annotated sentences as the
- initial model gives a big boost
-  Unsupervised Training
- With only the lexicon and no prior model
- Suffix statistics gleaned from lexicon

## dTagger Available Programs

- TrainWithTaggedText
- Tag
- UpdateWithUntaggedText
- TrainWithUntaggedText
- MorphologyDiscovery
- AnalizeTaggedCorpus (next release)

## Unsupervised Updating and Training

| Option | Description | Default Value |
| --- | --- | --- |
| --dirName= | Where the input files will be found | the current directory |
| --fileName= | The input file or input file pattern | standard input |
| --modelName= | Saved Hidden Markov model name | “default” |
| --Rev= | Revision number useful for tracking multiple updates to a model | “0” |

## Unsupervised Updating and Training

| Option | Description | Default Value |
| --- | --- | --- |
| --corpusName= | This is echoed out in output stats | “MedPost” |
| --sentenceIDMarker | Sentence ID in the training corpus. | “P” |
| --overwrite | Overwrite the current rev’s probs when updating | false |

## Error Analysis 

- Verbs tagged as nouns
- Adj/noun and noun/adj’s
- Odd usages
- Human tagging inconsistencies
- Conflicts with what the lexicon says

## The dTagger Tag Class

- processCollection(Collection pCollection )

- Method Summary

- void

- processDocument(Document pDocument )

- void

- processSentence(Sentence pSentence )

- void

- tag(LexicalElement[]  pTerms )

- double

- Tag(GlobalBehavior pSettings )

-  gov.nih.nlm.nls.dtagger.**Tag** Constructor Summary

- tag(Sentence pSentence )

- double

- tag(String pSentence )

- LexicalElement[]

## Work In Progress

- Put the latest and greatest on the website (includes textTools w/ dTagger now)
- More evaluation
- Single vs. multi-word comparison
- Using different tag sets
- Applied to Spanish to help build Spanish lexical resources
- “fix it, Make it better, make it faster” tasks

## Lexical Systems Group

| Websites | http://specialist.nlm.nih.gov
http://mmtx.nlm.nih.gov |
| --- | --- |
| Contacts | Allen Browne       browne@nlm.nih.gov
Guy Divita               divita@nlm.nih.gov
Chris Lu                        lu@nlm.nih.gov
Susanne M. Humphrey       humphrey@nlm.nih.gov
Lexical Systems Group     
                            umlslex@nlm.nih.gov |

## This Space to be filled in in Brisbane

## Hidden Markov Model for POS Tagging

- Det

- Adj

- Verb

- The

- a

- acute

- quick

- heart

- process

- run

- run

- heal

- process

- Noun

**Notes:**
A Markov Model (MM) consists of a set of states with probabilities of transitions between them.
In a Hidden Markov Model (HMM), the states have probabilities of emitting observations.  When the same observation can be emitted by multiple states, it becomes ambiguous which state produced the observation.  This problem never goes away, but the ambiguity can be quantified.  The HMM makes it is possible to calculate the probability that a given state emitted a given observation.
For POS tagging, the states represent POS tags and the observations are words.  Any given sentence is considered a sequence of observations.  For each sentence, there are many possible sequences of POS tags (also called paths through state space).  The HMM formalism allows us to calculate the probabilities that a given sequence is correct.  We can also find the most likely, or best, POS sequence.
The figure is a simplified example of a HMM model for POS tagging.  Circles are POS states.  Rectangles are observed words.  Arcs and arrows have a probability that they will be traversed.  Note that the words "run" and "process" are emitted by two states.

## Training (1)

- Hidden Markov Model
- Emission counts gathered from annotated corpus

|  | Noun | Adj | Adv | Verb | Det | Conj | ... |
| --- | --- | --- | --- | --- | --- | --- | --- |
| the | 0 | 0 | 0 | 0 | 7406 | 0 |  |
| cause | 7 | 0 | 0 | 13 | 0 | 0 |  |
| ... |  |  |  |  |  |  |  |

## Training (2)

- Transition counts gathered from annotated corpus

|  | Noun | Adj | Adv | Verb | Det | Conj | ... |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Noun | 14242 | 1235 | 713 | 3231 | 3100 | 424 |  |
| Adj | 11421 | 1703 | 113 | 11 | 4 | 1 |  |
| Adv | 140 | 631 | 285 | 2014 | 359 | 9 |  |
| Verb | 1273 | 758 | 794 | 97 | 19 | 3 |  |
| ... |  |  |  |  |  |  |  |

## Tagging

- Viterbi Algorithm:

|  | Do | viruses | cause | cancer |
| --- | --- | --- | --- | --- |
| Noun |  | 97 % | 45 % | 97 % |
| Adj |  |  |  |  |
| Adv |  |  |  |  |
| Verb | 77 % |  | 52 % |  |
| Aux | 23 % |  |  |  |
| Unknown |  | .07 % | .07 % | .07% |
| ... |  |  |  |  |

- State Probabilities

-      **|Do  |viruses|cause|cancer|**
**34.92****|verb|noun   |verb |noun  |**