# Presentation: 009265

## Lexical Tools for UMLS Developers

**Allen C. Browne, Guy Divita, **
**Chris Lu **
**Lister Hill National Center for Biomedical Communications**
** **National Library of Medicine

**Lexical Tools for UMLS Developers**

## Slide 2

## Lexical Tools for UMLS Developers

- The SPECIALIST lexicon – Browne
- The lexical tools – Divita/Lu
- Coffee Break– 10:00 - 10:30
- Lexical tools cont. – Divita/Lu

## Lexical tools

**Lexical tools**

- Text
-  processing

**SPECIALIST**
**LEXICON**

## The SPECIALIST Lexicon

- A syntactic lexicon
- Biomedical and general English
- Over 160,000 records

## The SPECIALIST Lexicon

- General English:
- 10,000 most frequent words from the American Heritage word frequency list
- 2,000 words used by Longman’s Dictionary of Contemporary English
- Verbs and adjectives identified by heuristics

## Lexicon Growth

## George A. Miller

- _The Science of Words_
- 1991

## The SPECIALIST Lexicon

- Morphology
- Inflection
- Derivation
- Orthography
- Spelling variants
- Syntax
- Complementation for verbs, nouns, and adjectives

## Morphology

- Inflectional
- nucleus -- nuclei
- cauterize, cauterizes, cauterized, cauterizing
- red, redder reddest
- Derivational
- laryngeal -- larynx
- transport -- transportation

## Orthography

**align -- aline**
**Grave’s disease -- Graves’s disease -- Graves’ disease**
**anesthetize -- anesthetise**
**esophagus -- oesophagus**

**Spelling Variation**

## British and American Spelling

- Criticise -- criticize
- naturalise --naturalize
- centre -- center
- foetus -- fetus

## Try This Test

**Which of the variant spellings below do you, accept**
**as standard American English? After making your**
**choices, consult the pages of this book to see what the dictionaries say. You are due for some astonishments.**

_**A Comparative Study of Spellings in four major collegiate dictionaries**_**, **
**by Lee C Deighton, **
**Hardscrabble Press 1972**.

## Syntax -- Verb Complements

- Intran
- I’ll treat.
- tran=np
- He treated the patient.
- ditran=np,pphr(with,np)
- She treated the patient with the drug.

## Syntax -- Verb Complements

- {base=treat
- entry=E0061964
- 	cat=verb
- 	variants=reg
- 	intran
- 	tran=np
- 	tran=pphr(with,np)
- 	tran=pphr(of,np)
- 	ditran=np,pphr(to,np)
- 	ditran=np,pphr(with,np)
- 	ditran=np,pphr(for,np)
- 	cplxtran=np,advbl
- 	nominalization=treatment|noun|E0061968
- }

## The 2002 SPECIALIST Lexicon

## Slide 17

## Lexicon Unit Records

**spelling_variant**=Kaposi sarcoma
**entry**=E0003576
- 	**cat**=noun
- 	variants=uncount
- 	variants=reg
- 	variants=glreg
- }

- {**base**=chronic
**entry**=E0016869
- 	**cat**=adj
- 	variants=inv
- 	position=attrib(1)
- 	position=pred
- 	stative
- }

- {**base**=aspirate
**entry**=E0010803
-         **cat**=verb
-         variants=reg
-         tran=np
-         nominalization=aspiration|noun|E0010804
- }

**Lexicon Unit Records**

- {**base**=in
**entry**=E0033870
-         **cat**=prep
- }

## Noun Variants

- Kaposi’s sarcoma
- Kaposi’s sarcomas
- Kaposi’s sarcomata
- Kaposi sarcoma
- Kaposi sarcomas
- Kaposi sarcomata

- {base**=**Kaposi's sarcoma
- spelling_variant=Kaposi sarcoma
- entry=E0003576
- 	cat=noun
- 	variants=uncount
- 	variants=reg
- 	variants=glreg
- }

## Regular Nouns

- The plural suffix is *s*.
*y* becomes *ie* following a consonant before *s*.
- e is inserted before s if the base ends in *s*, *z*, *x*, *ch*, or *s*

## Regular Nouns 

**Regular Nouns **

## Greco-latin Regular nouns

## Uncount Nouns(abstract or mass)

- * a smallpox
- * two smallpoxes
- much smallpox
- * a potassium
- * two potassiums
- much potassium

- {base=smallpox
- entry=E0056359
- 	cat=noun
- 	variants=uncount
- }
- {base=potassium
- entry=E0049387
- 	cat=noun
- 	variants=uncount
- }

## Fixed Plural Nouns

- {base=police
- entry=E0048616
- 	cat=noun
- 	variants=plur
- }

- {base=scissors
- entry=E0054633
-         cat=noun
-         variants=plur
- }

## Irregular Nouns

- {base=corpus
- entry=E0019113
-         cat=noun
-         variants=irreg|corpora|
-         variants=reg
- }

- {base=larynx
- entry=E0036919
-         cat=noun
-         variants=irreg|larynges|
-         variants=reg
- }

## Regular Verbs

- The third person present tense suffix is *s.*
*y* becomes *ie* following a consonant before *s.*
*e *is inserted between z, x, ch, or sh and *s*.
- The past tense suffix is *ed*.
*y* becomes *ie* following a consonant before *ed.*
- Final* e *is deleted before* ed.*

## Regular Verbs

- dismiss:dismisses, dismissed, dismissing
- agree: agrees; agreed; agreeing
- dry: dries, dried, drying

## Regular Doubling Verbs

- End in a CVC pattern
- Double the final consonant before *ed* and *ing*.
- Are otherwise regular
- variants=regd
- e.g. control: controls, controlled, controlling

## Irregular Verbs

- {base=dive
- cat=verb
-         variants=reg
-         variants=irreg|dives|dove|dove|diving|
-         intran
-         intran;part(in)
-         ...
- }

## Defective Verb

_**Defective Verb**_
**   ****sightsee**
** *****sightsees**
** *****sightsaw**
** *****sightseen**
**   ****sightseeing**

## Regular Adjectives and Adverbs

- The comparative suffix is *er*.
- The superlative suffix is *est*.
*y* become *ie* after a consonant before *er* or *est.*
- Final* e *is deleted before* er *or* est.*
- e.g. green: greener, greenest

## Regular Doubling Adjectives and Adverbs

- CVC final pattern
- Final consonant is doubled before ed or est.
- Otherwise regular
- e.g. red: redder, reddest

## Ancillary Data Bases

- Synonymy
- sm.db
- Derivation
- dm.db, dm.rules
- Inflection
- im.rules
- Neoclassical compounds
- nc.db

## Derivational Facts and Rules

**dm.facts**
**treatment|noun|treat|verb**
**prohibition|noun|prohibitive|adj**
**cell lineage|noun|cell line|noun**
**photochemotherapeutic|adj|photochemotherapy|noun**
**pharmacotherapeutic|adj|pharmacotherapy|noun**

## Derivational Facts and Rules

**dm.rules**
**# e.g. alienation|alienate**
**ation$|noun|ate|verb**
**	****ration|rate; station|state**;

## Inflectional Facts and Rules

**im.rules**** **
**# Noun rules (glreg)**
**us$|noun|singular|i$|noun|plural**
**    ****antus|anti;**
**ma$|noun|singular|mata$|noun|plural**
**a$|noun|singular|ae$|noun|plural**
**um$|noun|singular|a$|noun|plural**
**on$|noun|singular|a$|noun|plural**
**sis$|noun|singular|ses$|noun|plural**
**is$|noun|singular|ides$|noun|plural**
**men$|noun|singular|mina$|noun|plural**
**ex$|noun|singular|ices$|noun|plural**
**x$|noun|singular|ces$|noun|plural**

## Neoclassical compounds

**nc.db**
- abdomin(o)|abdomen|root
- ab|away from|prefix
- acanth(o)|prickle|root
- acar(o)|mite|root
- acetabul(o)|acetabulum|root
- ad|towards|prefix
- agogue|inducing|terminal
- albumin(o)|albumin|root
- sis|condition|terminal
- stomy|surgical opening|terminal

## Synonyms

**sm.db**
- alar|adj|wing|noun
- amygdaline|adj|tonsil|noun
- articular|adj|joint|noun
- bulbar|adj|medulla oblongata|noun
- fununcular|adj|boil|noun
- genicular|adj|knee|noun
- hepatocellular|adj|liver cells|noun
- lazar|adj|leprosy|noun
- lenticular|adj|crystalline lens|noun
- ypsiliform|adj|upsiloid|adj
- wolfram|noun|tungsten|noun
- double vision|noun|diplopia|noun

## Relational Tables

- One line records
- Pipe separated Fields -- “|”
- Keyed to EUI
- LRAGR matches forms to EUIs
- Word index: LRWD

## Relational Tables

- LRAGR - Agreement
- LRCMP - Complements
- LRFIL - Files
- LRFLD - Fields
- LRMOD - Modification
- LRNOM - Nominalization
- LRPRN - Pronouns
- LRPRP - Properties
- LRSPL - Spelling
- LRTRM - Trademarks
- LRWD - Word index

## LRAGR

- EUI - Entry ID
- STR - Inflected form
- SCA - Syntactic category
- AGR - agreement information
- BAS  - Base form (morphological)
- CIT - Citation form (base=)

**Agreement and Inflection**

## LRAGR

**E0003576|Kaposi sarcomas|noun|count(thr_plur)|Kaposi sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi sarcomata|noun|count(thr_plur)|Kaposi sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi sarcoma|noun|count(thr_sing)|Kaposi sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi sarcoma|noun|uncount(thr_sing)|Kaposi sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi's sarcomas|noun|count(thr_plur)|Kaposi's sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi's sarcomata|noun|count(thr_plur)|Kaposi's sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi's sarcoma|noun|count(thr_sing)|Kaposi's sarcoma|Kaposi's sarcoma|**
**E0003576|Kaposi's sarcoma|noun|uncount(thr_sing)|Kaposi's sarcoma|Kaposi's sarcoma**|

## Lexical tools

**Lexical tools**

- Text
-  processing

**SPECIALIST**
**LEXICON**

## Lexical Tools

- _Wordind_ -- breaks strings into words
-  Produces the Metathesaurus word indexes (MRXW)
- _LVG_ -- performs various lexical transformations
- _NORM_ -- a selection of LVG transformations,
- Used for Metathesaurus indexing
- Produces the Metathesaurus Normalized word and string indexes (MRXNW & MRXNS)
- Used to access those indexes

## Hodgkin Disease
HODGKINS DISEASE
Hodgkin's Disease
Disease, Hodgkin's
HODGKIN'S DISEASE
Hodgkin's disease
Hodgkins Disease
Hodgkin's disease NOS
Hodgkin's disease, NOS
Disease, Hodgkins
Diseases, Hodgkins
Hodgkins Diseases
Hodgkins disease
hodgkin's disease
Disease;Hodgkins
Disease, Hodgkin

**HODGKINS DISEASE**
**Hodgkin's Disease**
**Disease, Hodgkin's**
**HODGKIN'S DISEASE**
**Hodgkin's disease**
**Hodgkins Disease**
**Hodgkin's disease NOS**
**Hodgkin's disease, NOS**
**Disease, Hodgkins**
**Diseases, Hodgkins**
**Hodgkins Diseases**
**Hodgkins disease**
**hodgkin's disease**
**Disease;Hodgkins**
**Disease, Hodgkin**

**Disease hodgkin**

- Normalization

## Lexical Tools

**Notes:**

The lexical tools include a word and string normalization tool, 
a word tokenization tool, 
and a suite of text utilities,  otherwise known as lexical variant generation , or LVG.
We often refer to the lexical tools as the Swiss army knife of text utilities. It’s got 54 different tools, some sharp, some blunt. 
Warning. This presentation contains material that is suitable for grad students and medical informatics trained professionals. There are some unseemly details with gratuitous code inserted just for developers.
Chris

## The Lexical Tools

- Introduction
- Norm/WordInd/Lvg
- Details, Details and More Details ...
- Installation
- Embedding This into Your Application
- Using The Lexical Tools with The Metathesaurus
- Building an Index Using The Lexical Tools
- Plans for 2003 and Beyond

**Notes:**
I’d like to cover the details of the command line tools. These include  Norm, Wordind,  and lvg.
Next covered, uncovered? is installalation. If things go right, we might even install the tools while we cover that section.
 We’ll get into some details of how to embed these tools into your application. 
At which point, those non-developers heads will explode with details they will never need to know.
But wait, the next section we’ll cover involves how these tools fit in with the Metathesaurus. 
I’m going to end my presentation, if there is time, by talking about building customized indexes into the Metathesaurus. 
And, oh, there are always plans for things we just haven’t snuck in the package yet.

Chris

## Lexical Tools: Introduction

- These tools:
- Includes Norm,WordInd, and Lvg
- Pure Java based
- Command line tools
- Java APIs

**Notes:**
What ARE these tools, AND  what are they good for? 
These tools are command line tools. Built from java code.
I used to say that these are built from C code and that binaries are distributed for Solaris and NT. 
Thank the good lord I no longer need to say that. 
The source code is also distributed. 
These tools are intended to be embedded into your applications. We ARE assuming that REAL developers use JAVA.
As such, there are an untold number of public JAVA API’s that were developed to be called within your app.

I still have to give the warning for those Windows only people that the command line tool runs from 
dos console windows. They do not have separate windows to set parameters!  Don’t just  click on the executable icons. No command arguments get set, and consequently, nothing happens. All the tools require setting some parameters to tell the tool what to do. You’ll see ...
Chris

## Lexical Tools: Introduction

- These tools are good for
- aggressive text pattern matching
- making word, term, phrase indexes
- matching queries with indexed entries
- increasing recall and/or precision

**Notes:**
What are these tools good for?
These tools are good for text matching that is more powerful than exact match.
The tools are used to normalize terms, as well as generate expanded terms.
These tools are used to create word, term, and phrase indexes. They are  embedded  between queries and  target indexes. 

If your purpose is to increase recall over exact matches, the lexical tools can be   configured to do that.  
If your purpose is to increase precision over some loose matching algorithm, the lexical tools can be configured to do that.

## Lexical Tools: Introduction

- Characteristics of all the command line tools
- take input from the screen or a file
- put their results to the screen or a file
- Interpret fielded text
- Can be told which fields contain what type of information

**Notes:**
Some general characteristics of all of these tools:
They all take input from standard in, and put results on standard out.
They all are configured via command line arguments.

They all interpret fielded text, using by default, the UNIX’sh Pipe character    as a field separator, and a newline (natively defined) as the end of record, or end of term indicator.
That is, all the tools can be told 
what field to do the text transformation on,
what fields to pass through to the output,                 and
what fields contain other relevant information.

Chris

## Lexical Tools: Norm

**Notes:**
Let’s begin with the most popular of the tools, NORM.  

The norm program normalizes terms. 

In general,  normalization involves transforming something that’s not uniform to something that is uniform.

Norm is used to create the UMLS Metathesaurus normalized word and string indexes. 

Applications that use these indexes, 
also use norm to mediate between their queries and 
entries in these indexes.

## Lexical Tools: Norm

- Norm abstracts away from:
- case
- Punctuation
- word order
- possessive forms
- inflectional variation

**Notes:**
Our form of normalization involves the following:

It abstracts away from case, punctuation, and wordorder. 
It abstracts away from inflectional variations. 
It abstracts away from possessive forms.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**lowercase**

**uninflect each word**

**word order sort**

**replace punctuation with spaces**

**Hodgkin's **
**Diseases, **
**NOS**

**Notes:**
Norm in some more detail. 

Let’s take the example Hodgkin’s Diseases, NOS. Here’s a term that could be found in text an awful lot of ways. 
This program will transform a fair number of those variant ways into a uniform representation that will enable applications to recognize all those variants belonging to the same term.

## Lexical Tools: Norm

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

- Lexical Tools: Norm

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Genitives,  those apostrophe ses, or the possessive forms are removed.
Medical terminology is replete with terms that include possessive forms, 
Such as Parkinson’s Disease. 
There are a lot of variation caused by forms that differ by possessive forms. 
In a lot of cases, as Allen had mentioned,  terms evolve from possessive forms to plural forms.
It turns out to be a good idea to abstract away from possessive forms for retrieval purposes. 

In our example, Hodgkin’s gets transformed to Hodgkin. 

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Punctuation is replaced by spaces.

It is important to identify the apostrophe s’s 
before punctuation is replaced.  

The comma is removed in this example.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Stop words are stripped. 
Stop words are those words that you really don’t want to query on, 
such as determiners and frequently used prepositions. 
For example, words like “THE”, and ”OF”.  

We consider NOS as one of our stop words. It gets stripped.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
For obvious reasons, it’s a big win to either lower case your input or uppercase it.

We’ve settled on lowercasing the input to abstract away from case variation in terms. and everything is lowercased

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**hodgkin disease **

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Each word of the term is uninflected. 
Uninflection is the what we would consider an entry form in our Lexicon. These are, for the most part,
   the singular forms of nouns,  
   the infinitive forms for verbs, and
   the positive forms for adjectives and adverbs. 
Uninflected forms are similar to, but are not the same as Stemmed forms.  We don’t use the “S” word.  

Stemming involves suffix stripping, not bothering to heed the distinction between derivational suffixes and inflectional suffixes. Stemmers often refer to the “stems” as the root of the word, regardless whether the meaning had changed when the suffix was removed or added.

Stemmers often only involve suffix replacement and may or may not filter to a word list      after the suffix transformations. 

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**disease hodgkin **

**disease**
** ****hodgkin**

**hodgkin disease **

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
As a last step, all the resulting words are sorted in ASCII order. 
Why?
There are terms in the Metathesaurus in inverted form. That is, where the important part of the term is at the front of the term.  “Cancer, Breast”. Inversion is a useful cataloguing technique.

There are terms in the Metathesaurus in uninverted form. And, there are terms in the Metathesaurus that are in inverted form, but, where the inversion points, typically commas, are missing.  

We found that stripping punctuation and sorting in word order, in general helps rather than hurts recall. There rare exceptions to this, such as the distinction between school nursing and nursing school. 

## Lexical Tools: Norm

| Down’s Syndrome
Down Syndrome | down syndrome |
| --- | --- |
| Acetolyses
acetolysis | acetolysis |
| Lung cancer
Cancer, lung | cancer lung |

**Norm to**

**Notes:**
Here are some examples of normalized forms. 
Possessive forms are removed in Down’s syndrome. It is common to see Down’s Syndrome with and without the S and with and without the apostrophe. 

Acetolyses and acetolysis normalize to their uninflected form, and 

lung cancer, and 
cancer, lung
have it’s punctuation stripped, lowercased,  and words sorted to come up with
its normalized form.  

## Lexical Tools: Norm

**> **norm
- Paget’s disease-scapula
- Paget's disease-scapula|**disease paget scapula**
- Scapula, Paget Disease
- Scapula, Paget Disease|**disease paget scapula**

**Notes:**
Here’s how it looks from the command line. 

## Lexical Tools: WordInd

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together. It’s like the centermeters/inches problem we heard about not too long ago.  
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- Breaks words into tokens
- Passes other fields to output, untouched
- Lowercases
- Removes white space and punctuation

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together. It’s like the centermeters/inches problem we heard about not too long ago.  
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- Useful command line options for wordInd

| -t[:Num] | Defines what field to tokenize |
| --- | --- |
| -f[:Num[:Num]] | Defines what fields get passed through |

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together. It’s like the centermeters/inches problem we heard about not too long ago.  
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- > wordInd –t:7 –F:1:6
- C0185495|S0298948|**denis**
- C0185495|S0298948|**browne**
- C0185495|S0298948|**splint**
- C0185495|S0298948|**strapping**

- C0185495|ENG|P|L0223844|PF|S0298948**|Denis-Browne splint strapping|**3**|**

**Notes:**
Here’s an example of what wordind does to a row from the Metathesaurus’ MRCON. 
The command line arguments to WORDIND include telling wordind to use the seventh field to find the term, and to pass through fields one and six to the output.

## Lexical Tools: Lvg

**Notes:**

Lvg is a command line tool that does a lot, and 
as a consequence 

has lots of options and filters. 

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| A | Return known acronyms  |
| a | Return known acronym expansions  |
| B | Uninflect words in a term  |
| b | Uninflect a term  |
| C | Canonicalize  |
| Ct | Retrieve the citation term  |

**Notes:**
I’m going to switch terminology at this point from talking about text  utilities or tool and introduce and refer to each utility as this funny thing called a 
“FLOW COMPONENT” I’ll explain later.
There are 54 tools, or FLOW COMPONENTS with more on the way. 
I’m going to flip through an alphabetical list of some of these tools.  (Not all are listed here. They are fully specified in the documentation).

The mnemonic on the left is what you would specify on the command line. 
These include tools to grab acronyms and their expansions.
Uninflection tools.

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| c | Tokenize a term into "words"  |
| ca | Tokenize, keep everything  |
| ch | Tokenize without breaking hyphens  |
| d | Generate derivational variants  |
| dc~N | Generate derivational variants with specifying output categories  |

**Notes:**
Several tokenization tools.
Derivation tools.

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| E | Retrieve the unique EUI for a term  |
| f | Filter output to contain only forms from lexicon  |
| Gn | Generate known fruitful variants  |
| g | Remove genitive  |
| i | Generate inflectional variants  |
| ici~Cats+Infls | Generate inflectional variants, by Categories and inflections  |

**Notes:**
Tools that filter output to words from the lexicon,
A useful variants generation tool
A genitive stripping tool
Inflect tools.

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| L | Retrieve category and inflection for a term  |
| l | Lowercase  |
| N | Normalize the input text in a non-canonical way (Norm)  |
| o | Replace punctuations with spaces  |
| p | Strip Punctuation |
| q | Strip diacritics  |
| R | Generate derivational variants, recursively  |
| r | Generate synonyms, recursively  |

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| s | Generate known spelling variants  |
| t | Strip stop words  |
| u | Uninvert the input phrase around commas  |
| w | Sort words by order  |
| y | Generate synonyms  |

## Lexical Tools: Flows

**leave**

**leave**

**leaves**

**leaving**

**left**

**inflect**

**Notes:**
Some characteristics about the lexical tools:

Each of the utilities can be used separately. 
Here is an example of the inflect tool. 
This tool generates all known inflections of a word.  
The notion of a flow is a serial combination of these tools put together. 
Thus the name for each tool: a flow component. 

## Lexical Tools: Flows

- leave
- leave|**leave**|128|1|i|1|
- leave|**leave**|128|512|i|1|
- leave|**leaves**|128|8|i|1|
- leave|**left**|1024|64|i|1|
- leave|**left**|1024|32|i|1|
- leave|**leave**|1024|1|i|1|
- leave|**leave**|1024|262144|i|1|
- leave|**leave**|1024|1024|i|1|
- leave|**leaves**|1024|128|i|1|
- leave|**leaving**|1024|16|i|1|

**Notes:**
The example shows just how such a tool is invoked from the command line. 
Don’t pay any attention to those pesky fields with numbers in them right now.
Pay attention to the fact that we put leave into this tool and leave, leaves, left, and leaving were returned. 

[That same information should be shown in the GUI tool].

## Lexical Tools: A Serial Flow

**Input **
**term**

**Remove possessive**

**lowercase**

**Strip punctuation**

**Remove stop words**

**Strip diacritics**

**Word order sort**

**Output **
**term**

**Flow components can be arranged so that the output of one is the input to another.**

**Notes:**
Here’s an example of such a serial combination. That is, 
where the output of one tool serves as
 the input to the next tool.
In this example, we are first lowercasing the input, then stripping diacritics, removing possessives, removing stop words, stripping punctuation, and finally sorting the resulting words in ASCII sort order.  
The example shows just how this flow is invoked on the command line. 

## Lexical Tools: A Serial Flow

**The Gougerot-Sjögren's Syndrome**
- The Gougerot-Sjögren's Syndrome**|**
**         ****gougerotsjogren syndrome**|2047|16777215|
-                                                                   l+q+g+t+p+w|1|

**Notes:**
Here’s how that sequence would be put together.  
The “-f” on the command line is an indication that what follows is to be taken as a sequence of flow components to execute in a serial order.
In this example, that sequence boils down to the sequence of the single character mnemonic forms: “L” for lowercase, q for strip diacritics, g for strip possessives, t for strip stop words, p for strip punctuation, and finally w for word order sort the input. 
In this example, The Gougerot-Sjögren's Syndrome, “the” is stripped off, the o umlaut is normalized to “o”, the hyphen is removed, and the resulting form is sorted.

## Lexical Tools: Parallel Flows

**term**

**Output **
**term**

**Multiple flows can be defined**

**noOperation**

**Uninflect**

**synonyms**

**Output **
**terms**

**Notes:**
The flow components can be put together into separate flows that produce transformations that are only dependant on that flow. 

In the example, we want both the original term and the synonyms of the term.

## Lexical Tools: Parallel Flows

- ear
- ear|**ear**|2047|1048575|n|1|
- ear|**aural**|1|1|B+y|2|
- ear|**auricularis**|1|1|B+y|2|
- ear|**otic**|1|1|B+y|2|
- ear|**otor**|1|1|B+y|2|

**Second Flow**

**First Flow**

**Notes:**
This is an example of how one would invoke such a parallel set of flows. 

The output consists of the combined results from the output of each flow. 
Note the last field of each output line, 
the number in red. 

That number indicates which flow it came from. 

## Lexical Tools: Fielded Output

**Input Term**

**Output Term**

**Categories**

**Inflections**

**Flow history**

**Flow Number**

- leaves

- leaves

- 1152

- 1

- 136

- L

- |

- |

- |

- |

- |

- > lvg –f:L
- leaves

**Notes:**
Just what are you getting on the output?  Much more than you bargained for!  

On each output line, five additional fields are added to the input fields. 

The output or transformed term, 
what categories and inflections the output term can be, 
what transformations or flow components  produced it. This is also known as the flow history, 
And, 
the flow number that produced it. 

## Lexical Tools: Fielded Output

**Categories**

**verb**

**plural**

**pres3ps**

**Output**
**term**

**Inflections**

**leaves**

**noun**

**Notes:**
Imagine, if you will, that some flow produced the output term “leaves”.
Leaves can be either the plural form of a noun or the present third person singular of a verb.

That’s what conceptually will be part of the output for this term.

## Lexical Tools: Categories

**1152**

- adj

- adv

- aux

- compl

- conj

- det

- modal

**noun**

- prep

- pron

- verb

**0**

**0**

**0**

**1**

**0**

**1**

**0**

**0**

**0**

**0**

**0**

**Category bit vector**

**Notes:**
But, that’s not easy to stuff into a concise, machine parsable, single line representation that can be used by other tools. A somewhat less readable bit vector representation of category and inflection was adopted to be the concise, parsable form that could be used by other tools.

Let’s concentrate on category for a minute. The bit vector for category consists of 11 bits in this order. To represent the categories that leaves take on, the bits for noun and verb are set to one. 

What appears on the command line output is the decimal representation of this bit vector. 

## Lexical Tools: Categories

- adv

- aux

- compl

- conj

- det

- modal

- noun

- prep

- pron

- verb

**0**

**6**

**8**

**7**

**9**

**10**

**3**

**4**

**5**

**1**

**2**

**Bit Vector**
**positions**

| Adjective | 1 |
| --- | --- |
| Adverb | 2 |
| Auxiliary  | 4 |
| Complement | 8 |
| Conjunction | 16 |
| Determiner | 32 |

| Modal | 64 |
| --- | --- |
| Noun | 128 |
| Preposition  | 256 |
| Pronoun | 512 |
| Verb | 1024 |

**Notes:**
Here is the decimal representation of these categories.

## Lexical Tools: Inflections

**136**

- base

- comparative

- superlative

- plural

- Present participle

- Present 3ps

**0**

**1**

**0**

**0**

**0**

**0**

- Present 3ps

**1**

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

**Notes:**
Likewise, inflections are stored in a bit vector representing inflections. Here, the base form is a simplified form for singular if the term is a noun, infinitive if the term is a verb, and positive if the term is an adjective or adverb.  

Leaves in the example can be either the plural form of a noun or the present third person singular of a verb. 
Those bits are set, which, when represented as a decimal,  add up to be 136. 

## Lexical Tools: Inflections

| Base | 1 |
| --- | --- |
| Comparative | 2 |
| Superlative | 4 |
| Plural | 8 |
| Present Participle | 16 |
| Past | 32 |
| Past Participle | 64 |
| Present 3rd Person Singular | 128 |

| Positive | 256 |
| --- | --- |
| Singular | 512 |
| Infinitive | 1024 |
| Pres 123p | 2048 |
| Past Neg | 4096 |
| pres123pNeg | 8192 |
| Pres1s | 16384 |
| past1p23pNeg | 32768 |
| past1s3sNeg | 65536  |

| pres1p23p | 131072  |
| --- | --- |
| pres1p23p | 262144  |
| pres1p23pNeg | 524288  |
| past1s3s | 1048576  |
| pres | 2097152  |
| pres3sNeg | 4194304  |
| presNeg | 8388608  |
| all | 16777215  |

**Notes:**
Here are the inflections viewed in their decimal representation. 

## Lexical Tools: Fielded Output

**Categories**

**1152**

**Output**
**term**

**Inflections**

**leaves**

**136**

**Input**
**term**

**leaves**

**Flow**
**history**

**Flow**
**number**

**L**

**1**

**Notes:**
Finally, this is what you get on the output.  
Note that the flow that produced this is the “L”  flow component, which looks up the part of speech and inflection information from the LEXICON. 

## Lexical Tools: Fielded Input

** **

**Categories**

**Input**
**term**

**leaves**

**noun**

**Inflections**

**Notes:**
If the output knows about category and infection, it would reason that we should be able to tell each tool any category and inflection information about an input term. 

We can. 
We can put on the input, via fielded input, category and inflection information. 
Suppose we have leaves as an input term, and we know we are talking about the leaves, the noun, as in “the leaves of a tree”.  

## Lexical Tools: Fielded Input

- > lvg –f:L **–t**:1 **–cf**:2
- leaves|128
- leaves|leaves|128|8|L|1

**Input category field**

**Input term field**

**Notes:**
On the command line, we tell the tool which field to look for the term, and which field to look for the category information. 

On the input, we then can put the categories in the field, using the bit vector form. 

## Hierarchical structure

- Lexical Tools: Command Line Syntax

**-****Option**

**:****Option Attribute**

**:****Option Attribute**

**:****Option Attribute**

**~****1’st **
**Attribute**
**Parameter**

**+****2’nd **
**Attribute**
**Parameter**

**+****N’th**
**Attribute**
**Parameter**

**...**

**Notes:**
We are about to delve into more details about the tool then you ever wanted to know. To get the tool to do what we want, we needed to come up with a more expressive syntax than the POSIX standard. (That word was inserted just for developers)  

The command line syntax for all these tools involves having options start with hyphens. 
Options may have attributes associated with them. Those attributes are separated by colons.
Attributes may have parameters. 
Attributes that have one parameter have the parameter separated by a tilda. 
Additional parameters are separated by a plus sign. 

Make sure there are not spaces between attributes and their parameters. There should be spaces between options. 

## Lexical Tools: Command Line Syntax

**-f:L:ici~128+ALL:w**

**-f**

**:ici**

**:w**

**:L**

**~128**

**+ALL**

**Notes:**
Here’s an example of the parse of the command line for Looking up lexical information, then inflecting only the nouns of those, then word order sorting them. 

A flow is an command line option.  A flow option is represented by an –f. 

Flows are made up of flow components (The tools). Each flow component of a flow is separated by a colon. 
 flow components can have parameters associated with them.  The :ici flow component has two attributes associated with it.

## Lexical Tools: Post Flow Options

| SC | Show category names |
| --- | --- |
| SI | Show inflection names |
| ccgi | Mark the end of the set of variants returned |
| F:Int[:Int] | Specify fields for outputs |
| ti | Display the only input term in the output when using fielded input |
| R:Int | Restrict the number of variants returned  |

**Notes:**
The lexical tools have some useful post processing options that are of note. 
For the most part these options are filters. Though some of these options add information to the end of the processing.
Here are some miscillanious but useful options.

Oh, my favorites: Show the categories and inflections in string form, not in bit vector form.

-ccgi, an end of processing marker.

-F followed by the fields you want to have in your output.

-ti Forget about passing through all the input fields, only pass through the field you are picking up the term from. 

-R followed by a number to restrict the number of output rows. This is useful, say when you are generating lots of forms, but you only need the top 10. 

## Lexical Tools: Post Flow Options

**Show category names**

**Show inflection names**

- > lvg -f:L -**SC -SI**
*phosphoprotein*
- phosphoprotein|phosphoprotein|**<noun>**|**<base+singular>**|L|1|
*sclerosing*
- sclerosing|sclerosing|**<adj+verb>**|**<base+presPart+positive>**|L|1|

**Show the category and inflection names**

**Notes:**
These options are necessary for those of us who cannot do the math.
 It displays the categories and inflections as strings rather than bit vectors. 
I generally use these for debugging purposes. 
The fields can get quite long for terms that can be many categories and inflections. 

## Lexical Tools: Post Flow Options

**Mark the end of the set of variants returned**

- > lvg -f:L **-ccgi**
- behavior
- behavior|behavior|128|513|L|1|
- __THE_END__

**Mark the end of processing**

**Notes:**
Here’s the –ccgi option.
 This option is useful when embedding this into applications where you need the tool to tell you that it’s done processing an input.  This tells the calling app. To stop waiting for more output.

## Lexical Tools: Post Flow Options

- _[Specify fields for outputs](file:///docs/designDoc/requirement/lvgOptions/outFieldOption.html)_

- > lvg -f:u -t:7 **-F:8:6**
- C0035440|ENG|S|L0035434|VW|S0003894|Rheumatic carditis, acute
***acute Rheumatic carditis***|***S0003894***

**Display only the 8****th**** and 6****th**** field from the output **

**Notes:**
Oh, yea, the –F option. 
I use this option frequently,
 to cull down the output to only those fields I need.
 In this example, I only want the output field and 
the field that has the SUI in it. 

## Lexical Tools: Post Flow Options

- _[Display only the input term field when using fielded input](file:///docs/designDoc/requirement/lvgOptions/filterInputOption.html)_

- > lvg -f:u -t:7 **–ti**
- C0035440|S0003894|***Rheumatic carditis, acute***
***Rheumatic carditis, acute***|acute Rheumatic carditis|2047|16777215|u|1|

**Display only the input term from the fielded input to**
**the output**

**Notes:**
The –ti option is always used in conjunction with the option that indicates the input field that contains the term.
An equivalent way of doing this would also be using the –F option with the input term field.

## Lexical Tools: Post Flow Options

**Restrict the number of variants returned **

- > lvg -f:i **-R:**2
- foo
- foo|foo|128|1|i|1|
- foo|foos|128|8|i|1|

**Limit the number of output terms to 2**

- Note:  The unrestricted output would
-  have produced 12 rows otherwise

**Note: Dangerous!**
**Do not try this at home!**

**Notes:**
Here’s an example using the truncation filter. Here I’m passing an unknown word to be inflected. If I did not have this filter on, the tools would generate 12 forms.

## Lexical Tools: Post Flow Options

| EC:Long | Display variants: exclude categories specified  |
| --- | --- |
| EI:Long | Display variants: exclude inflections specified  |
| DC:Long | Display variants that only contain the categories specified  |
| DI:Long | Display variants that only contain the inflections specified  |

**Notes:**

The lexical tools have some useful post processing options that are of note. 
For the most part these options are filters. Though some of these options add information to the end of the processing.

Worth mentioning are the filters that exclude categories or only include categories.

## Lexical Tools: Post Flow Options

- _[Display variants: exclude categories specified](file:///docs/designDoc/requirement/lvgOptions/excludeCatOption.html)_

- > lvg –f:i **–EC:**1919
- sleep
- sleep|sleep|128|1|i|1
- sleep|sleep|128|512|i|1

**Display variants, but exclude all terms other than nouns**

**Notes:**
For example, suppose because of some twisted reason, you want only to get noun variants.
Given this option with the bit vector set to everything set on but nouns will do it.

## Lexical Tools: Post Flow Options

- _[Display variants exclude inflection specified](file:///docs/designDoc/requirement/lvgOptions/excludeInflOption.html)_

- > lvg –f:i –EC:1919 **–EI:**1
- sleep
- sleep|sleep|128|512|i|1

**Display variants, but exclude base forms**

**Notes:**
In addition to nouns, we only want to see non base form nouns.

## Lexical Tools: Post Flow Options

- _[Display variants contain category specified](file:///docs/designDoc/requirement/lvgOptions/catOption.html)_

- > lvg –f:i **–DC:**128
- sleep
- sleep|sleep|128|1|i|1
- sleep|sleep|128|512|i|1

**Display variants, but only include nouns in the output**

**Notes:**
An easier way to only get nouns from a flow would be to use the “display categories”
Filter with nouns (128)

## Lexical Tools: Post Flow Options

- _[Display variants contain inflection specified](file:///docs/designDoc/requirement/lvgOptions/inflOption.html)__ _

- > lvg –f:i **–DI:**255
- sleep
- sleep|sleep|128|1|i|1|
- sleep|sleep|1024|1|i|1|
- sleep|slept|1024|64|i|1|
- sleep|slept|1024|32|i|1|
- sleep|sleeps|1024|128|i|1|
- sleep|sleeping|1024|16|i|1|

**Display variants, but only include “simplified” inflections**

**Notes:**
Here is a favorite filter of mine. There are many instances where I want the set of inflections where the base form is listed, but not its nearly redundant form. I call these the simplified inflections. 
For example, I want the base form for a noun, but not the nearly redundant row that contains the singular row.
It turns out that all those forms are inflectional forms with bit vector representations lower than 256 decimal. 
Creating a filter that allows all inflections below 256 does that.

## Lexical Tools: Post Flow Options

| CR:o | Combine record by output term  |
| --- | --- |
| CR:oc | Combine record by output term and category  |
| CR:oi | Combine record by output term and inflection  |
| St:o | Sort outputs by terms in an alphabetical order  |
| St:oc | Sort outputs by term and category  |
| St:oci | Sort outputs by term, category, and inflection  |

**Notes:**

There are grouping and sorting post flow options that we have found useful. These involve combining and sorting the output  rows by the criteria of similar term forms, categories and/or inflections. 

## Lexical Tools: Post Flow Options

- _[Combine record by output term](file:///docs/designDoc/requirement/lvgOptions/combTermOption.html)_

- > lvg -f:i **-CR:o**
- sleep
- sleep|sleep|**1152**|263681|i|1|
- sleep|slept|1024|96|i|1|
- sleep|sleeps|1024|128|i|1|
- sleep|sleeping|1024|16|i|1|

**Combine records by term**

**Note: this is a noun+verb**

**Notes:**
Here’s an example of combining records by term. Notice that the category and inflection bit vectors contain multiple categories and inflections. 

## Lexical Tools: Post Flow Options

- sleep
- sleep|sleep|128|513|i|1|
- sleep|sleep|1024|263169|i|1|
- sleep|slept|1024|96|i|1|
- sleep|sleeps|1024|128|i|1|
- sleep|sleeping|1024|16|i|1|

- _[Combine record by output term and category](file:///docs/designDoc/requirement/lvgOptions/combTermCatOption.html)_

**Combine records by term and category.**

**Note: this is both the **
**base+singular inflections**
**combined**

**Notes:**
Here’s an example that combines by term and category. Note that the inflection field indicates multiple inflections for the form “sleep”

## Lexical Tools: Post Flow Options

- _[Combine record by output term and inflection](file:///docs/designDoc/requirement/lvgOptions/combTermInflOption.html)__ _

- lvg -f:b **-CR:oi**
- left
- left|left|131|1|b|1|
- left|leave|1024|1|b|1|

**Combine records by term and inflection**

**Note: This is an adj+adv+noun**

**Notes:**
An example of combining by inflection.

## Lexical Tools: Post Flow Options

- _[Sort outputs by terms in an alphabetical order](file:///docs/designDoc/requirement/lvgOptions/sortByTermOption.htm)__ _

- >lvg -f:i **-St:o**
- see
- see|saw|1024|32|i|1|
- see|see|1024|1|i|1|
- see|see|1024|262144|i|1|
- see|see|1024|1024|i|1|
- see|seeing|1024|16|i|1|
- see|seen|1024|64|i|1|
- see|sees|1024|128|i|1|

**Sort by the output term**

**Notes:**
It’s sometimes necessary to sort the output by various criteria.

In this example, the output is sorted by term. 

## Lexical Tools: Post Flow Options

- _[Sort outputs by term and category](file:///docs/designDoc/requirement/lvgOptions/sortByTermCatOption.htm)_

- > lvg -f:i -**St:oc**
- left
- ...
- left|left|1|1|i|1|
- left|left|1|256|i|1|
- left|left|2|1|i|1|
- left|left|2|256|i|1|
- left|left|128|1|i|1|
- left|left|128|512|i|1|
- left|left|128|8|i|1|
- left|left|1024|64|i|1|
- left|left|1024|32|i|1|
- left|lefts|128|8|i|1|

**Sort by the output term and category**

**Notes:**
Here’s an example of sorting by term and category. 

## Lexical Tools: Post Flow Options

- _[Sort out](http://cgsb9.nlm.nih.gov:8757/lvg/designDoc/requirement/lvgOptions/sortByTermCatInflOption.html)__[p](file:///docs/designDoc/requirement/lvgOptions/sortByTermCatInflOption.htm)__[uts](http://cgsb9.nlm.nih.gov:8757/lvg/designDoc/requirement/lvgOptions/sortByTermCatInflOption.html)__[ by term, category, and inflection](http://cgsb9.nlm.nih.gov:8757/lvg/designDoc/requirement/lvgOptions/sortByTermCatInflOption.html)_** **

- > lvg -f:i **-St:oci**
- see
- see|saw|1024|32|i|1|
- see|see|1024|1|i|1|
- see|see|1024|1024|i|1|
- see|see|1024|262144|i|1|
- see|seeing|1024|16|i|1|
- see|seen|1024|64|i|1|
- see|sees|1024|128|i|1|

**Sort by the output term, category and inflection**

**Notes:**
Here’s an example of sorting by term, category and inflection. 

## Lexical Tools: Why Base+Singular

- Base form for Nouns:
- Singular form
- There are exceptions:
- Police
- lvg -f:i -SC -SI -DC:128
- police
- police|police|<noun>|<base>|i|1|
- police|police|<noun>|<plural>|i|1|

**Notes:**
Here’s a public service message. It’s inserted here for lack of a better place to mention it.
Allen noted that we we generally consider the singular forms of nouns as the base form, and infinitive form for verbs as the base or uninflected forms.  For years we’ve had inflection bit vectors that have combined the base form with the singular form because it’s redundant.  
When we re-implemented the tools in java, they kept on breaking occasionally because of a class of words that breaks these rules. 
Here’s an example of a noun that is plural, and has NO singular form.  We had to account for such instances, so the tool will print out a row for the uninflected form AND a row for it’s singular form even though they are next to always the same. In the general case, separating the base from the singular or infinitive form means that in cases where it matters, you don’t have to do the inferencing to compute that base means singular or infinitive or positive.
If you don’t like this, then you can use the –Di:255 filter to filter back to the “simplified” inflectional forms.

## Lexical Tools: Global Behaviors

| i:filename | Define input file name  |
| --- | --- |
| o:filename | Define output file name  |
| x:filename | Loading an alternative configuration file  |
| p | Interactive prompt |
| m | Print extra information of flow mutations  |
| s:Str | Defines a field separator.  |

**Output **
**terms**

**Input **
**term**

**Output **
**terms**

**Notes:**
Here are some options that are worth mentioning.

The tools can take a file as input and can put the output to a file.
The tools can be configured in strange and divient ways by altering the config file.
You can tell the tools at startup time to use an alternative config file rather than the default one.

When using the tool interactively on the command line, it’s sometimes useful to have a prompt.

## Lexical Tools: Global Behaviors

- _[Interactive prompt](file:///docs/designDoc/requirement/lvgOptions/promptOption.html)_

- > lvg -f:s -CR:oc  **-p**
***- Please input a term (type "Ctl-d" to Quit) >***
- color
- color|color|128|513|s|1|
- color|color|1024|263169|s|1|
- color|colour|128|513|s|1|
- color|colour|1024|263169|s|1|
***- Please input a term (type "Ctl-d" to Quit) >***

**Set the prompt on**

**Notes:**
Here’s what the prompt looks like.

## Lexical Tools: Global Behaviors

| K:dInt | Configure the derivation morphology behavior  |
| --- | --- |
| K:iInt | Configure the inflection morphology behavior |

**Output **
**terms**

**Input **
**term**

**Output **
**terms**

**Notes:**
Finally, there are two options that control the generation of morphological forms. These options affect the general behavior of the inflection and derivation and related flow components.  

## Lexical Tools: Global Behaviors

- _[Configure the derivation morphology behavior](file:///docs/designDoc/requirement/lvgOptions/derivationOption.html)_

| 1 | Restrict the output to those variants which are known to the lexicon (default).  |
| --- | --- |
| 2 | Restrict the output to those variants which are known to the lexicon, unless none of the variants are found in the lexicon, in which case the entire (rule-generated) list is returned.  |
| 3 | No restriction on the output of the morphology. Both facts and rules generated variants are displayed. |

**Notes:**
You can alter the derivational variant generation behavior to restrict those forms produced to words that we know about via the lexicon. In general this is restriction is a good thing to do, because the tool blindly adds suffixes to the ends of words to make new words. In a lot of cases those new strings are not words at all. Having this filter on is the default. 
At the other end of the spectrum, you can have the tool report all the forms generated whether or not they are words. You might want to do this in an environment where there is not a lot of overlap between the words in our lexicon and the words that you are working with. 

## Lexical Tools: Global Behaviors

- _[Configure the derivation morphology behavior](file:///docs/designDoc/requirement/lvgOptions/derivationOption.html)_

- > lvg -f:d **-kd:**1  -m
- breath
- breath|breather|128|1|d|1|RULE|$|verb|base|er$|noun|base|
- breath|breathy|1|1|d|1|RULE|$|noun|base|y$|adj|base|
- breath|breathless|1|1|d|1|FACT|breath|128|breathless|1|

**Restrict the derivations to terms that are known to the Lexicon**

**Notes:**
Here’s an example of turning on the filter.

## Lexical Tools: Global Behaviors

- _[Configure the inflection morphology behavior](file:///docs/designDoc/requirement/lvgOptions/inflectionOption.html)_

| 1 | Restrict the output to those variants which are known to the lexicon.  |
| --- | --- |
| 2 | Restrict the output to those variants which are known to the lexicon, unless none of the variants are found in the lexicon, in which case the entire (rule-generated) list is returned (Default). |
| 3 | No restriction on the output of the morphology. Both facts and rules generated variants are displayed |

**Notes:**
Inflectional morphology can be altered in similar ways to constrain the output to only include forms known to the lexicon, or at the other end of the spectrum to return all forms, known and rule generated. 

The inflectional rules are less aggressive than the derivational rules. If an input form is not in the lexicon, it has been found to be useful to go ahead and use the rule generated forms.  This is the default behavior for inflections.

## Lexical Tools: Global Behaviors

- _[Configure the inflection morphology behavior](file:///docs/designDoc/requirement/lvgOptions/inflectionOption.html)_

- > lvg -f:i -m **-ki:3** -R:5 -F:1:2:7:3:4 -SC -SI
- Amish
- Amish|Amish|FACT|<noun>|<base>
- Amish|Amish|FACT|<noun>|<plural>
- Amish|Amishs|RULE|<noun>|<plural>
- Amish|Amishes|RULE|<noun>|<plural>
- Amish|Amishs|RULE|<verb>|<pres>

**Return all inflected variants, both fact and rule generated**

**Notes:**
You may, I don’t know why, but you may, want to create an index that has all inflectional forms, fact AND rule generated. Here’s an example of doing that with the term Amish.  (Oh, there’s another  of those non-singular noun words)

## Lexical Tools: No Operation

**-f:n**

**Copies the input term to the output with no transformation**

- > lvg -f:**n** -f:d -f:y -SC -SI
- force
- force|**force**|<all>|<all>|**n**|1|
- force|forcefully|<adv>|<base>|d|2|
- force|forceful|<adj>|<base>|d|2|
- force|forcible|<adj>|<base>|d|2|
- force|dynamic|<adj>|<base>|y|3|

**Notes:**
Now, some specific flow components.
I’m going to cover only a few of them. Refer to the documentation for the rest of them. 
The most trivial, but really useful flow component is just a pass through. The –f:n flow component just repeats the input term as an output term. This turns out to be useful, for instance, when you want to have the original term as one of the outputs.

In this example, force, the input term, is on the output of the “make me an index of terms, their derivations and their synonyms” flows only because of that pass through flow component,     “-f:n”.

## Lexical Tools: Inflect

**-f:i**

**Generate inflectional variants **

- > lvg  -f**:i** -SC -SI
- West Nile Virus
- West Nile Virus|**West Nile virus**|<noun>|<base>|i|1|
- West Nile Virus|**West Nile virus**|<noun>|<singular>|i|1|
- West Nile Virus|**West Nile viruses**|<noun>|<plural>|i|1|

**Notes:**
The inflect flow component. This flow component uses the LEXICON to find all the known inflectional forms of a term. 
Here’s a pretty simple example. West Nile Virus, a term popular these days in Maryland.
In fact, for a given term, this utility will uninflect the term, then retrieve all the inflected forms for that term, including the form that came in.  

If the term is not in the LEXICON, inflectional suffix adding and replacement rules based on the regular rules of inflection are applied to the input form to generate potential inflectional forms, in the hopes that at least some of resulting forms are useful. The rules come with exceptions to prune out forms that we’ve noticed go wrong. 
The inflection flow component is just one configuration of the morphology module that produces both inflections, uninflections and derivations.

## Lexical Tools: Inflect

**Generate inflections, filter by cat and/or inflection**

**-f:ici~cats+infls**

- > lvg -f:**ici~128+ALL** -SC -SI
- bioassay
- bioassay|**bioassay**|<noun>|<base>|ici|1|
- bioassay|**bioassay**|<noun>|<singular>|ici|1|
- bioassay|**bioassays**|<noun>|<plural>|ici|1|

**Notes:**
But wait! There’s more to inflection. Inflections can be filtered to include or exclude certain categories and inflections. 

For example, you may only want to build an index of noun forms, or build an index of all nouns being in their plural form.

The options to the –f:i flow component include the decimal representation of the categories and inflections to include. 

## Lexical Tools: Uninflect by Term

**-f:b**

**Uninflect by term**

- > lvg -f:**b** -SC -SI
- left atria
- left atria|**left atrium**|<noun>|<base>|b|1|

**Notes:**
The uninflect by term flow component. 
You can pass in terms from the input that contain multiple words, but where you want to treat as a single entity. 
Here’s an example where uninflecting the term would be different than uninflecting each word. Why?
If we consider left atriums as a term, and uninflect it as a term, we get “left atrium” as the resulting form. If we were to uninflect by each word, left would uninflect to both left atrium and leave atrium as well. Not what we had intended.
This flow component is useful when you know you are working with terms.
There is a flow component that uninflects by word, when you don’t have that luxury.

## Lexical Tools: Derivations

**-f:d**

**Generate derivations**

- > lvg -f:**d** -SC -SI
- diagnostic
- diagnostic|**diagnosis**|<noun>|<base>|d|1|
- diagnostic|**diagnostics**|<noun>|<base>|d|1|
- diagnostic|**diagnose**|<verb>|<base>|d|1|
- diagnostic|**diagnostical**|<adj>|<base>|d|1|

**Notes:**
Generate derivational forms.
Derivational forms are terms that are somewhat related to the original term, but change meaning in some way. They almost always involve shifting category as well.
By facts, such as nominalizations of verbs, adjectives and nouns from the LEXICON. 
By suffix replacement rules that also have exceptions. These rules are by-directional, and include things like add or remove the suffix “tion”, or “ly” from the input term. Here are some of the derivational forms of diagnostic.

## Lexical Tools: Derivations

**-f:dc~cats**

**Generate derivations, filter by category**

- > lvg -f:**dc~129** -SC -SI
- reduce
- reduce|**reduction**|<noun>|<base>|d|1|
- reduce|**reducible**|<adj>|<base>|d|1|

**Notes:**
Like inflection, derivational forms can be filtered by category.
Derivational forms can be used when creating related variants when doing query expansion. Derivational variants, by their nature, involve some change in meaning and category. This can be over aggressive. One way to constrain some of this over generation, is to filter to reasonable categories. 
For instance, you know you are working with nouns, you may want to constrain the derivations to nouns and adjectives. 

## Lexical Tools: Synonyms

**-f:y**

**Generate synonyms**

- > lvg -f:**y** -SC -SI
- kidney
- kidney|**nephric**|<adj>|<base>|y|1|
- kidney|**nephritic**|<adj>|<base>|y|1|
- kidney|**renal**|<adj>|<base>|y|1|

**Notes:**
Generate synonyms. 
This is a table lookup operation that retrieve synonyms from the table of synonyms that come from one of the LEXICON’s supplemental data sources.
These synonyms are intended to complement, not duplicate, the Metathesaurus’s set of synonymy. The synonyms that are on this list are more on the word rather than the “Metathesaurus term” level. Metathesaurus terms can be whole sentences.
The set of synonyms is small, and were created on a somewhat ad-hoc basis to cover word level synonymy that the Metathesaurus does not, or had not had at the time. One example is chest for breast.

## Lexical Tools: Normalize (norm)

**-f:N**

**Remove stop words, then remove genitives, then replace punctuation with spaces, then lowercase, then uninflect each word, then take each of the uninflected words, then word order sort. **

- > lvg -f:N
- Syndrome, Dry Eyes
- Syndrome, Dry Eyes|**dry eye syndrome**|2047|1|g+o+t+l+B+w|1|

**Notes:**
Generate normalized forms. This is the Norm  transformation. We know this tool also as the “Norm” tool.
I think that I’ve talked enough about norm for you to get the idea. 
The only other thing I’ll add is to repeat that norm, luiNorm, and wordind are all just utilities that are flow components from LVG.

## Lexical Tools: Installation

- Requirements
- 1.6 gigabytes of space
- Solaris/NT/Linux
- Tar, gzip or WinZip
- Minimum of 36 MB Memory
- Java JREs included

**Notes:**
The lexical tools do a lot, and they take some heafty resourses. 160 megabytes of space is needed
Before installation. A somewhat smaller amount is used after installation.

We are distributing Java programs.  

For non developers, this means that the tool will work without having to do strange and dark deeds that developers are prone to do like compile code.

GONE are the days where I say that you will need an ANCI C compiler. Wee!

For the most popular platforms, we even provide the program that runs the java programs.
For those that don’t know, the program that runs java programs is called the Java Runtime Environment (JRE)

Experience has taught us that there are less phone calls when we distribute the JRE’s and automatically have them installed,  rather than leave it up to the end user to acquire and install a JRE. 

The Four JRE distributions actually account for over half the space required. Once the program is installed, the non-necessary JRE distributions are removed. 

Here’s a developer’s note: 
The tool comes with a pure java database. It’s a good database, but it’s not going to win any performance awards.
We developed the tool so that other databases could be swapped in if performance is a concern. 
Note: you have to be a trained professional to do this. Do not attempt this at home. Go over to someone else’s house. 

At NLM, we use the public domain MySQL database for a variety of projects.  We’ve included all the necessary steps in our installation to use MySQL as the back end db, if you are already using MySQL.  
Don’t count on this to be a permanent feature though, we are likely to change this feature. 

## Lexical Tools: Installation

**> ./install/bin/install_solaris_sparc.sh**
**> .\install\bin\install_win**

**Notes:**
Installation is relatively easy. 
Just acquire the distribution archive.  You can get it either from the CD, or download the UMLS, or get it from the UMLS Knowledge source Server’s Lexicon and Lexical Resources page.
Once gotten, un-archive it.  On Unix machines this involves using tar and gzip. On the PC, winzip or pkarchive can be used.  
Un-archive this to the location where you want to have this installed to.  The un-archived folders and files will be contained in the folder named “lvg200X”.  

We’ll call this the LVG_DIR directory. 

Installation starts from the LVG_DIR directory. On the pc, you will have to open a command (or dos) window.  (eeeuu). 
Change directories (cd) to this LVG_DIR directory.
From here, type in the command bin install_ and your platform. That kicks off the install process.
Someday we’ll make a truly PC look and feel install. But not this year.

## Lexical Tools: Installation

**Notes:**
The first thing that happens is the JRE gets installed.  
On the PC, the JRE gets put into a standard place. If you already have the same JRE, it’s going to be overwritten. The registry does get updated with this version of the JRE. Sorry. 
For you developers, that cannot deal with this,  comment this out in the install script.  You have to be a trained professional to do this. Oh, and you need to belong to the union.
Anyway, the JRE installation is going to ask that you aggree to some things. Agree to them if you will.
Once that is done ...

## Lexical Tools: Installation

-  **Welcome to the Java Lexical Tools Installation!**
-  -----------------------------------------------
-  Please read the ***installationNotes.html***
-  prior to invoking this script.
-  This script will configure the
-  _LVG_DIR_/data/config/lvg.properties
-  file.
-  This script will create configured
-  lvg, norm, luiNorm, and wordind
-  scripts in the _LVG_DIR_ directory.
-  This script, as an option, will configure
-  and load the lexical tools into a pre-existing
-  MySql database. This option is only available
-  if you already have a MySql database running.
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The Java Lexical tools use or (may use) the
- following third party software packages:
-  - Instant DB from Enhydra.org
-    (see http://instantdb.enhydra.org)
-    This package is covered under an Enhydra
-    Public License, which allows for redistribution
-    and use with a "use notice".
-  - The installation script uses some GNU CYGWIN
-    programs that have been distributed along with
-    this script. These commands are only used when
-    this installation is on a Windows platform.
-    The commands that have been distributed
-    include tar, gunzip, rm, find, hosthame,
-    tee, and cmp.
-    The CYGWIN package is licensed under the
-    GNU Public License. The entire CYGWIN package,
-    along with the sources to these commands can be
-    found at http://sources.redhat.com/cygwin/
- Enhydra is a trademark of Lutris Technologies, Inc.

**Notes:**
The installation continues.  I’ve got to put up the required stuff about what wonderful products we’ve used within our program.  We use the IDB database, and for installation only,  Unix tools written for the PC called CYGWIN.

## Lexical Tools: Installation

- +----------------------Note---------------------------+
- |Although we do not support this feature, and it      |
- |is very likely that we will change the underlying    |
- |implementation, we have found it useful for our      |
- |applications to use an existing database system      |
- |other than IDB. We often use Mysql as the underlying |
- |database for our applications. Since we needed a     |
- |way to get the lexical tools data into MySql, we     |
- |figured that we would incorporate the MYSQL loader   |
- |into this install.                                   |
- |                                                     |
- |This option requires that an existing mysql database |
- |(version 3.23 or higher) have already been installed,|
- |that you have database root access privileges,       |
- |that the database has enough room to load this       |
- |data, and that the database is running now.          |
- +-----------------------------------------------------+
- +--------------- **Question** -------------------+
- |  Do you want to have the lexical tools use |
- |  your mysql database [y/n]  [n]? **n**

**Notes:**
The program is going to ask if you would like to use an already existing MySQL database. You must be a trained professional to say yes to that question. If you say yes to that question, you must have ready, a running MySQL database running, and a root or administrators account with root privileges along with its password. You will also need to make sure you have enough space to house the tables that will be loaded into this database. These will be asked of you.
One other note, it takes about 40 minutes or so to load all the tables into MySQL via this install.

But we are not going to get into that. We will say no for this demonstration.

## Lexical Tools: Installation

- +-----------------------------------------------+
- | Verifying the installation ...                |
- +-----------------------------------------------+
- Enhydra InstantDB - Version 3.26
- The Initial Developer of the Original Code is Lutris Technologies
- Inc. Portions created by Lutris are Copyright (C) 1997-2001
- Lutris Technologies, Inc.  All Rights Reserved.
- Database sample is shutting down...
- Database sample shutdown complete.
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Notes:**
At the tail end of the installation, we check to see that everything is installed and working properly ...

## Lexical Tools: Installation

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-                 **Congratulations!**
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-   This script has completed configuring the java
-   lexical tools. You may invoke these tools from a
-   command line. These tools are found in the
-   __LVG_DIR__/bin directory.
-   You can add this __LVG_DIR__/bin path to your
-   $PATH environment variable. This would enable
-   you to find and run these tools from any location.
-   In UNIX, this would be done by adding this
-   path to your ~/.cshrc or ~/.profile startup script.
-   In Windows, this would be done by appending
-   this path to the PATH variable from the
-   control panel/System/Environment variables menus.
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-   **The lexical tools are ready to be used! **
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Notes:**
If things go as they are expected to go, you should see a failure note. (Just kidding)
You should see a Congratulations note.

Upon a successful installation, it will be up to you to alter your environment to be able to run
This from anywhere on the machine. On unix machines, this involves adding the path to LVG’s
Bin directory into your cshrc, or .profile or .login file’s PATH environment variable. 

On the pc, this involved adding this path to the control panel’s system, environment variable 
Settings to the PATH environment variable.

## Classpath
NormApi()
LvgCmdApi()
LexItem
LvgLexItemApi()

- NormApi()
- LvgCmdApi()
- LexItem
- LvgLexItemApi()

- Lexical Tools: **Embedding These Tools into Your Application**

**Notes:**

Here’s the section for developers and those that want to embed the lexical tools into their apps.
I’m going to cover the prerequisites, and the basic classes to use, from the simplest to the more featured interfaces.

The full code to the next examples can be found in the appendix to this tutorial, and are distributed as examples within the source code distribution.

- Lexical Tools: **Embedding Norm into Your Application**

**Notes:**

Most applications are looking to embed norm into their application. We’ve got an interface for you.
The NormApi. This is an instantiated class that you can send terms through and get back
Normalized strings. It takes as input a string, and returns a vector of Strings.

## Lexical Tools: Embedding These Tools into Your Application

**CLASSPATH = ${CLASSPATH}:**
**    ****${LVG_DIR}:**
**    ****${LVG_DIR}/classes/*****lvg2002.jar:***
**    ****${LVG_DIR}/classes/IDB/jta-spec1_0_1.jar:**
**    ****${LVG_DIR}/classes/IDB/idb.jar:**
**    ****${LVG_DIR}/classes/jdbcDrivers/mm.mysql-2.0.6**

- Lexical Tools: **Embedding These Tools into Your Application**

**If you **
**are using **
**MySQL**

**If you are **
**using IDB**

**Notes:**
But first, some prerequisates: Classpath, classpath, classpath. Here’s what you need to add to your classpath to use our tools. 

The LVG_DIR, the lvg2002.jar file, and one of the following depending on if you are using the IDB database or MySQL database.

- import Lvg.Api.*;
- NormApi        normalize = new NormApi();
- String            *input2Norm* = null;
- Vector  outputFromNorm = null;

- Lexical Tools: **Embedding Norm into Your Application**

**Notes:**
Here are the important pieces of code that are needed to imbed this into your app. 

You need to import the Lvg.Api.* packages.
You need to instantiate the NormApi class. 
Better have handy a string and a vector variable ...

- while ( (*input2Norm* = stdIn.readLine() ) != null ) {
-    outputFromNorm= **normalize.Mutate**(*input2Norm*);
-    for ( int i = 0; i < outputFromNorm.size(); i++ ) {
-         System.out.println((String) outputFromNorm.get(i));
-     }
- }
**normalize.CleanUp**();

- Lexical Tools: **Embedding Norm into Your Application**

**Notes:**
I’m going to assume that you know how to read terms in from some stream, or method. 
This bit of code shows that you can call the normalize.Mutate method over and over again
In a loop.  This Mutate method (yes it’s an uppercase M, by design) takes a string as input,
And returns a vector of strings. 

This example shows looping through the resulting output vector, printing out the resulting
Normalized terms.

When your done, be nice, clean up. This method closes open files, open database connections.

**Your application**

**LvgCmd**
**Api**

**Input **
**term**

**Output **
**terms**

- Lexical Tools: **Embedding Lvg into Your Application**

**Notes:**
Now, for a simple way to embed other lexical tools into your application. 
For simple tasks, there is the LvgCmdApi() that takes a term from the input
And returns a vector of strings. It’s similar in structure to the NormApi() 

- import Lvg.Api.*;
- LvgCmdApi lvgApi = new **LvgCmdApi**(“-f:b –CR:o –SC –SI”);
- String            *input2Lvg* = null;
- Vector   outputFromLvg = null;

- Lexical Tools: **Embedding Lvg into Your Application**

**Notes:**
Again, you need to import the LvgApi packages. 

You will need to instantiate the LvgCmdApi with an argument. The argument is a string. 
That string should contain what you would have put on the command line if you were
Running the process from the command line.

- Lexical Tools: **Embedding Lvg into Your Application**

- while ( (*input2Lvg* = stdIn.readLine() ) != null ) {
-    outputFromLvg= **lvgApi.Mutate**(*input2Lvg*);
-     for ( int i = 0; i < outputFromLvg.size(); i++ ) {
-         System.out.println((String) outputFromLvg.get(i));
-     }
- }
- lvgApi.CleanUp();

**Notes:**
This bit of code shows that you can call the normalize.Mutate method over and over again
In a loop.  This Mutate method (yes it’s an uppercase M, by design) takes a string as input,
And returns a vector of strings. 

This example shows looping through the resulting output vector, printing out the resulting
mutated terms. The output is the same as what would have gone out to standard out
From the command line.

When your done, be nice, clean up. This method closes open files, open database connections.

**Your application**

**LvgLexItem**
**Api**

- Lexical Tools: **Embedding Lvg into Your Application**

**Notes:**
Now,  for the real Api. This LvgLexItemApi is a bit more featured than the other two. 

**LvgLexItem**
**Api**

**Input **
**term**

- Lexical Tools: **Embedding Lvg into Your Application**

- f:g:o:t:l:i

**LexItems**

**Notes:**
It can take as input a term. Its output is not a vector of strings however, it is a vector of an object or structure  we call lexItem.  Internally we use these lexItems to carry around each term’s information.

## Lexical Tools: The LexItem Class 

- Lexical Tools: **The LexItem Class**

**Notes:**
Here’s how I like to think of LexItems. LexItems contain information about an incoming or source term, including it’s category, inflection, and the actual string. LexItems contain information about the term once it’s been mutated. We call this information the target information. This includes the target term and its categories and inflections. LexItems also carry around some residual information including the mutations this term has gone through so far, the very first original term before any mutations were applied to it, and asorted extra information. 

You should note that there is a strong correspondence between this structure and what information gets returned on every output row.  

Suffice to say that there are getter and setter methods that come with this object.

## import Lvg.Api.*;
import Lvg.Lib.*;

LvgCmdApi    lvgApi = new LvgLexItemApi(“-f:g:o:t:l:i”);
String          input2Lvg = null;
Vector outputFromLvg = null;
LexItem         aLexItem = null;

- import Lvg.Lib.*;
- LvgCmdApi    lvgApi = new LvgLexItemApi(“-f:g:o:t:l:i”);
- String          ***input2Lvg*** = null;
- Vector outputFromLvg = null;
- LexItem         aLexItem = null;

- Lexical Tools:**Embedding Lvg into Your Application**

**Notes:**

Note that an additional set of packages are needed to use this method. The Lvg.Lib packages contain the lexItem. 

Again, you need to instantiate the LvgLexItemApi with an argument that is the command line settings.

Please keep handy a vector of LexItems for the output.
A temparary LexItem will come in handy as well.

## while ( (input2Lvg = stdIn.readLine() ) != null ) {
   outputFromLvg=lvgApi.MutateLexItem(input2Lvg);
    for ( int i = 0; i < outputFromLvg.size(); i++ ) {
       aLexItem = (LexItem) outputFromLvg.get(i);  
       System.out.println(aLexItem.GetSourceTerm() + "|" + 
		  aLexItem.GetTargetTerm() + "|" +
		  aLexItem.GetTargetCategory().GetName()+ "|" +
		  aLexItem.GetTargetInflection().GetName() + "|" +
		  aLexItem.GetTargetInflection().GetValue() );
    }
}
lvgApi.CleanUp();

-    outputFromLvg=**lvgApi.MutateLexItem**(*input2Lvg*);
-     for ( int i = 0; i < outputFromLvg.size(); i++ ) {
-        aLexItem = (LexItem) outputFromLvg.get(i);
-        System.out.println(aLexItem.*GetSourceTerm()* + "|" +
- 		  aLexItem.*GetTargetTerm()* + "|" +
- 		  aLexItem.*GetTargetCategory().*GetName()+ "|" +
- 		  aLexItem.*GetTargetInflection().*GetName() + "|" +
- 		  aLexItem.*GetTargetInflection().*GetValue() );
-     }
- }
- lvgApi.CleanUp();

- Lexical Tools:**Embedding Lvg into Your Application**

**Notes:**
This example shows that you can call the Mutator over and over again. The MutateLexItem() method takes a string as input, and returns a vector of LexItem.  Here are some examples of the get methods that can be called from the lexItem.

You’re going to have to assume that there exists a Category class and an Inflection class that
Become instantiated objects, AND that these objects have methods GetName() and GetValue()
To get both the string representation and their integer (ops, long) representation for category and inflection.

Once again, be kind and clean up after your finished processing.

## Lexical Tools: Java API Documentation

| Packages |  |
| --- | --- |
| Lvg.Api | This package provides APIs |
| Lvg.CmdLineSyntax | Provides Java classes necessary to create a command line system. |
| Lvg.Db | Provides a higher level interface to LVG database. |
| Lvg.Flows | This package provides API of all Lvg flow components. |
| Lvg.Lib | Contains LVG general library classes of BitMaskBase, Category, Gender, Inflection, Configuration, CombineRecords, OutputFilter, Flow, GlobalBehavior, LexItem, LexItemComparator |
| Lvg.Trie | Provides the classes necessary to generate inflections, uninflections, and derivations using LVG rules tries |
| Lvg.Util | Contains LVG general utility classes of comparators, Bit operations, Case, In, Out, Strip operations, token operation, Char, Str, and Word. |

**Notes:**
These are but a few of the many api’s available to be used. Refer to the JavaDoc for details.

## Lexical Resources

**Notes:**
Where to get the tools, the lexicon, the documentation, the updates. 
Where else? the Resources page from the knowledge source server.

## References

- McCray AT, Loane RF, Browne AC, Bangalore AK. Terminology Issues in User Access to Web-based Medical Information. Proc AMIA Symp. 1998;:107-111
- Divita G, Browne AC, Rindflesch T. Evaluating Lexical Variant Generation to Improve Information Retrieval, Proc AMIA Symp. 1998;:775-9
- McCray AT, Browne AC. Discovering the modifiers in a terminology data set. Proc AMIA Symp. 1998;:780-4
- McCray AT. The nature of lexical knowledge. Methods Inf Med. 1998 Nov;37(4-5):353-60.
- McCray AT, Cheh ML, Bangalore AK, Rafei K, Razi AM, Divita G, Stavri PZ. Conducting the NLM/AHCPR Large Scale Vocabulary Test: a distributed Internet-based experiment. Proc AMIA Annu Fall Symp. 1997;:560-4.
- McCray AT, Razi, AM, Bangalore AK, Browne AC, Stavri PZ. The UMLS Knowledge Source Server:  A versatile Internet-based research tool. Proc AMIA Symp. 1996;164-8.
- McCray AT, Srinivasan S, Browne AC. Lexical methods for managing variation in biomedical terminologies.  Proc Annu Symp Comput Appl Med Care. 1994;:235-9.

## Using The Lexical Tools with The Metathesaurus

**Metathesaurus**
**English**
**Strings**

**norm**

**Normalized string index**

**Normalized word index**

**WordInd**

**MRXNS.ENG**

**MRXNW.ENG**

**Notes:**
Here’s where the lexical tools fit in with the UMLS Metathesaurus.
The norm program normalizes the Metathesaurus strings to produce the normalized string index. Each normalized string is also associated with three types of identifiers, a concept identifier, a lexical term identifier, and a string identifier. These ids are keys into all the other tables.  The normalized string index is then broken into a normalized word index via wordind, carrying along the identifiers.

## Using The Lexical Tools with The Metathesaurus

**norm**

**Normalized string index**

**Normalized word index**

**Metathesaurus**
**Concepts**

**Query**

**Normed**
**term**

**SUIS**

**Metathesaurus**
**Concepts that match**
**The normalized query**

**Notes:**
Application developers put the norm program between queries and the Metathesaurus. 
Normalized queries are then used get entries from either the normalized string or word index to get the ids to retrieve Metathesaurus concept information. 
Metathesaurus concept information is then returned to satisfy the request.

## Using The Lexical Tools with The Metathesaurus

**norm**

**dry eye syndrome**

**Dry Eyes Syndrome**

**Notes:**
Here’s a simple minded example. Suppose the query is Dry eyes syndrome. 
The query gets normalized.

## Using The Lexical Tools with The Metathesaurus

- ENG|**dry eye syndrome**|C0013238|L0013238|**S0035652**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0090228**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0090454**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0220550**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0368350**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S1459074**|

**Normed**
**term**

**Notes:**
All entries that share the normalized form are retreived from the normalized string index. The string unique identifiers to these rows are used ...

## Using The Lexical Tools with The Metathesaurus

- C0013238|ENG|P|L0013238|VS    |S0004019|Dry eye syndrome
- C0013238|ENG|P|L0013238|VS    |S0368350|Dry Eye Syndrome
- C0013238|ENG|P|L0013238|VS    |S1459074|dry eye syndrome
- C0013238|ENG|P|L0013238|VWS|S0090228|Syndrome, Dry Eye
- C0013238|ENG|P|L0013238|VWS|S0220550|Dry, eye syndrome
- C0013238|ENG|P|L0013238|VW  |S0090454|Syndromes, Dry Eye

**SUIS**

**MRCON**

**C0013238|ENG|P|L0013238|PF  |S0035652| Dry Eye Syndromes**

**Notes:**
To retrieve Metathesurus concept information. 
If one of the things you want to return are the the concept names, not the string names for the query, you can review all the returned strings to see which ones are the concept names. I’ve got one of these rows highlighted that indicates that it’s the preferred term for that group of strings, and that group of strings is considered the preferred group for that concept. All told, this is what we consider the concept name. 

## Building an Index Using The Lexical Tools

- Can we build a tool that increases recall?
- Can we build a tool that increases precision?

**Notes:**
Can we build a tool that increases recall?
Can we build a tool that increases precision?

## Building an Index Using The Lexical Tools

- Can we build a tool that increases precision?
- Case
- Constrain by part of speech
- Filter to the lexicon

**Notes:**
We may want to have case be significant. 
Or we may only want to handle words that we know about.

## Building an Index Using The Lexical Tools

- Can we a tool that increases recall?
- - Include
- 		synonyms
- 		derivations
- 		acronyms and their expansions
- 		spelling variants

**Notes:**
We may want to include additional word level synonymy, 
or derivations,
or acronyms and abbreviations,
or spelling variants to increase recall.

## Building an Index Using The Lexical Tools

**Metathesaurus**
**Strings**

**index**

**word index**

**Wordind**

**Notes:**
Just as norm was used to create the normalized word and string index, we can come up with our own.
I’ve got a really small set of rows from the Metatheaurus, that we can use to create an index on, load into mysql, then retrieve via a web page that uses our norm.
We can create our norm by putting our flavor in script that is called by the indexing script as well as the web page behind the query.

## Building an Index Using The Lexical Tools

**       **

**word index**

**Metathesaurus**
**Concepts**

**Query**

**Transformed**
**query**

**SUIS**

**Metathesaurus**
**Concepts that match**
**The transformed query**

**Notes:**
On the retrieval side, I’ve created simple java program that calls our norm script to transform the query, retrieves entries from our norm’d index, gets the suis, and uses the suis to get strings from the Metathesaurus. 
I’ve got it set up as word index. 
Now to demonstrate that ....

## Lexical Tools: 2003 and Beyond

- Performance issues addressed
- Graphic User Interface
- Lexical GUI tool
- On-line Web based tool
- Enhance flow components
- Norm improved
- New flow components
- Nominalize, Complementation
- Additional functionalities
- XML output option

## Lexical Tools for UMLS Developers

- November 4, 2001
**Allen C. Browne, Guy Divita, **
**Chris Lu **
**Lister Hill National Center for Biomedical Communications**
** **National Library of Medicine

**Lexical Tools for UMLS Developers**

**Email:                                   umlslex.nlm.nih.gov**
**Knowledge Source Server: http://umlsks.nlm.nih.gov**
**UMLS Information:            http://umlsInfo.nlm.nih.gov**

## Appendix 

- NormExample.java
- LvgExampleEasy.java
- LvgExampleHarder.java
- LvgExampleEvenHarder.java