# Presentation: 477659

## Lexical Tools

**Notes:**

The lexical tools include a word and string normalization tool, 
a word tokenization tool, 
and a suite of text utilities,  otherwise known as lexical variant generation , or LVG.
We often refer to the lexical tools as the Swiss army knife of text utilities. It’s got 51 different tools, some sharp, some blunt. 
Warning. This presentation contains material that is suitable for grad students and medical informatics trained professionals. There are some unseemly details with gratuitous code inserted just for developers.
Chris

## The Lexical Tools

- Introduction
- Norm
- WordInd
- Lvg
- Additional Tools Developed by NLM

**Notes:**
I’d like to cover the details of the command line tools. These include  Norm, Wordind,  and lvg.
Next covered, uncovered? is installalation. If things go right, we might even install the tools while we cover that section.
 We’ll get into some details of how to embed these tools into your application. 
At which point, those non-developers heads will explode with details they will never need to know.
But wait, the next section we’ll cover involves how these tools fit in with the Metathesaurus. 
I’m going to end my presentation, if there is time, by talking about building customized indexes into the Metathesaurus. 
And, oh, there are always plans for things we just haven’t snuck in the package yet.

Chris

## Lexical Tools: Introduction

- Command line tools
- norm
- lvg
- wordInd
- _[Web GUI](http://umlslex.nlm.nih.gov:8888/WebLvg/jsp/norm/norm.jsp?type=Norm)__, Lexical Gui Tool (lgt)_
- _[Embeddable Java API’s](http://umlslex.nlm.nih.gov/lvg/2002/docs/userDoc/api.html)_

**Notes:**
What ARE these tools, AND  what are they good for? 
These tools are command line tools. Built from java code.
I used to say that these are built from C code and that binaries are distributed for Solaris and NT. 
Thank the good lord I no longer need to say that. 
The source code is also distributed. 
These tools are intended to be embedded into your applications. We ARE assuming that REAL developers use JAVA.
As such, there are an untold number of public JAVA API’s that were developed to be called within your app.

I still have to give the warning for those Windows only people that the command line tool runs from 
dos console windows. They do not have separate windows to set parameters!  Don’t just  click on the executable icons. No command arguments get set, and consequently, nothing happens. All the tools require setting some parameters to tell the tool what to do. You’ll see ...
Chris

## Lexical Tools: Introduction

- These tools are good for
- aggressive text pattern matching
- making word, term, phrase indexes
- matching queries with indexed entries
- increasing recall and/or precision

**Notes:**
What are these tools good for?
These tools are good for text matching that is more powerful than exact match.
The tools are used to normalize terms, as well as generate expanded terms.
These tools are used to create word, term, and phrase indexes. They are  embedded  between queries and  target indexes. 

If your purpose is to increase recall over exact matches, the lexical tools can be   configured to do that.  
If your purpose is to increase precision over some loose matching algorithm, the lexical tools can be configured to do that.

## Lexical Tools: Introduction

- Characteristics of all the command line tools
- take input from the screen or a file
- put their results to the screen or a file
- Interpret fielded text
- Can be told which fields contain what type of information

**Notes:**
Some general characteristics of all of these tools:
They all take input from standard in, and put results on standard out.
They all are configured via command line arguments.

They all interpret fielded text, using by default, the UNIX’sh Pipe character    as a field separator, and a newline (natively defined) as the end of record, or end of term indicator.
That is, all the tools can be told 
what field to do the text transformation on,
what fields to pass through to the output,                 and
what fields contain other relevant information.

Chris

## Lexical Tools: Norm

**Metathesaurus**
**English**
**Strings**

**norm**

**Normalized string index**

**Normalized word index**

**WordInd**

**MRXNS.ENG**

**MRXNW.ENG**

**Notes:**
Here’s where the lexical tools fit in with the UMLS Metathesaurus.
The norm program normalizes the Metathesaurus strings to produce the normalized string index. Each normalized string is also associated with three types of identifiers, a concept identifier, a lexical term identifier, and a string identifier. These ids are keys into all the other tables.  The normalized string index is then broken into a normalized word index via wordind, carrying along the identifiers.

## Lexical Tools: Norm

**norm**

**Normalized string index**

**Normalized word index**

**Metathesaurus**
**Concepts**

**Query**

**Normed**
**term**

**SUIS**

**Metathesaurus**
**Concepts that match**
**The normalized query**

**Notes:**

Let’s begin with the most popular of the tools, NORM.  

The norm program normalizes terms. 

In general,  normalization involves transforming something that’s not uniform to something that is uniform.

Norm is used to create the UMLS Metathesaurus normalized word and string indexes. 

Applications that use these indexes, 
also use norm to mediate between their queries and 
entries in these indexes.

Application developers put the norm program between queries and the Metathesaurus. 
Normalized queries are then used get entries from either the normalized string or word index to get the ids to retrieve Metathesaurus concept information. 
Metathesaurus concept information is then returned to satisfy the request.

## Lexical Tools: Norm

- Norm abstracts away from:
- case
- punctuation
- word order
- possessive forms
- inflectional variation

**Notes:**
Our form of normalization involves the following:

It abstracts away from case, punctuation, and wordorder. 
It abstracts away from inflectional variations. 
It abstracts away from possessive forms.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**lowercase**

**uninflect each word**

**word order sort**

**replace punctuation with spaces**

**Hodgkin's **
**Diseases, **
**NOS**

**Notes:**
Norm in some more detail. 

Let’s take the example Hodgkin’s Diseases, NOS. Here’s a term that could be found in text an awful lot of ways. 
This program will transform a fair number of those variant ways into a uniform representation that will enable applications to recognize all those variants belonging to the same term.

## Lexical Tools: Norm

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

- Lexical Tools: Norm

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Genitives,  those apostrophe ses, or the possessive forms are removed.
Medical terminology is replete with terms that include possessive forms, 
Such as Parkinson’s Disease. 
There are a lot of variation caused by forms that differ by possessive forms. 
In a lot of cases, as Allen had mentioned,  terms evolve from possessive forms to plural forms.
It turns out to be a good idea to abstract away from possessive forms for retrieval purposes. 

In our example, Hodgkin’s gets transformed to Hodgkin. 

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Punctuation is replaced by spaces.

It is important to identify the apostrophe s’s 
before punctuation is replaced.  

The comma is removed in this example.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Stop words are stripped. 
Stop words are those words that you really don’t want to query on, 
such as determiners and frequently used prepositions. 
For example, words like “THE”, and ”OF”.  

We consider NOS as one of our stop words. It gets stripped.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
For obvious reasons, it’s a big win to either lower case your input or uppercase it.

We’ve settled on lowercasing the input to abstract away from case variation in terms. and everything is lowercased

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**hodgkin disease **

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Each word of the term is uninflected. 
Uninflection is the what we would consider an entry form in our Lexicon. These are, for the most part,
   the singular forms of nouns,  
   the infinitive forms for verbs, and
   the positive forms for adjectives and adverbs. 
Uninflected forms are similar to, but are not the same as Stemmed forms.  We don’t use the “S” word.  

Stemming involves suffix stripping, not bothering to heed the distinction between derivational suffixes and inflectional suffixes. Stemmers often refer to the “stems” as the root of the word, regardless whether the meaning had changed when the suffix was removed or added.

Stemmers often only involve suffix replacement and may or may not filter to a word list      after the suffix transformations. 

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**disease hodgkin **

**disease**
** ****hodgkin**

**hodgkin disease **

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
As a last step, all the resulting words are sorted in ASCII order. 
Why?
There are terms in the Metathesaurus in inverted form. That is, where the important part of the term is at the front of the term.  “Cancer, Breast”. Inversion is a useful cataloguing technique.

There are terms in the Metathesaurus in uninverted form. And, there are terms in the Metathesaurus that are in inverted form, but, where the inversion points, typically commas, are missing.  

We found that stripping punctuation and sorting in word order, in general helps rather than hurts recall. There rare exceptions to this, such as the distinction between school nursing and nursing school. 

## Lexical Tools: Norm

| Down’s Syndrome
Down Syndrome | down syndrome |
| --- | --- |
| Acetolyses
acetolysis | acetolysis |
| Lung cancer
Cancer, lung | cancer lung |

**Norm to**

**Notes:**
Here are some examples of normalized forms. 
Possessive forms are removed in Down’s syndrome. It is common to see Down’s Syndrome with and without the S and with and without the apostrophe. 

Acetolyses and acetolysis normalize to their uninflected form, and 

lung cancer, and 
cancer, lung
have it’s punctuation stripped, lowercased,  and words sorted to come up with
its normalized form.  

Chis: Example Here

## Lexical Tools: WordInd

## Lexical Tools: WordInd

**wordind**
- is
- a
- tool
- to
- break
- terms
- into
- words
- it
- is
- used
- to

**Wordind** is a tool to break terms into words.
- It is used to take a row from a Metathesaurus table that contains** **a term, sentence, paragraph, story,
- and break the text part of that row into it’s constituent words.

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together. It’s like the centermeters/inches problem we heard about not too long ago.  
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- Breaks words into tokens
- Passes other fields to output, untouched
- Lowercases
- Removes white space and punctuation

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together. It’s like the centermeters/inches problem we heard about not too long ago.  
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- Useful command line options for wordInd

| -t[:Num] | Defines what field to tokenize |
| --- | --- |
| -f[:Num[:Num]] | Defines what fields get passed through |

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together. It’s like the centermeters/inches problem we heard about not too long ago.  
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- > wordInd –t:7 –F:1:6
- C0185495|S0298948|**denis**
- C0185495|S0298948|**browne**
- C0185495|S0298948|**splint**
- C0185495|S0298948|**strapping**

- C0185495|ENG|P|L0223844|PF|S0298948**|Denis-Browne splint strapping|**3**|**

**Notes:**
Here’s an example of what wordind does to a row from the Metathesaurus’ MRCON. 
The command line arguments to WORDIND include telling wordind to use the seventh field to find the term, and to pass through fields one and six to the output.

Chis: Example Here

## Lexical Tools: Lvg

**Notes:**

Lvg is a command line tool that does a lot, and 
as a consequence 

has lots of options and filters. 

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| A | Return known acronyms  |
| a | Return known acronym expansions  |
| B | Uninflect words in a term  |
| b | Uninflect a term  |
| C | Canonicalize  |
| c | Tokenize a term into "words"  |
| ca | Tokenize, keep everything  |
| ch | Tokenize without breaking hyphens  |

**Notes:**
I’m going to switch terminology at this point from talking about text  utilities or tool and introduce and refer to each utility as this funny thing called a 
“FLOW COMPONENT” I’ll explain later.
There are 51 tools, or FLOW COMPONENTS with more on the way. 
I’m going to flip through an alphabetical list of some of these tools.  (Not all are listed here. They are fully specified in the documentation).

The mnemonic on the left is what you would specify on the command line. 
These include tools to grab acronyms and their expansions.
Uninflection tools.

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| Ct | Retrieve the citation term  |
| d | Generate derivational variants  |
| dc~N | Generate derivational variants with specifying output categories  |
| R | Generate derivational variants, recursively  |
| E | Retrieve the unique EUI for a term  |
| f | Filter output to contain only forms from lexicon  |

**Notes:**
Several tokenization tools.
Derivation tools.

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| Gn | Generate known fruitful variants  |
| g | Remove genitive  |
| i | Generate inflectional variants  |
| ici~Cats+Infls | Generate inflectional variants, by Categories and inflections  |
| is | Generate inflectional variants, by Categories and inflections  |
| Si | Inflections to simplified inflections  |

**Notes:**
Tools that filter output to words from the lexicon,
A useful variants generation tool
A genitive stripping tool
Inflect tools.

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| L | Retrieve category and inflection for a term  |
| l | Lowercase  |
| N | Normalize the input text in a non-canonical way (Norm)  |
| o | Replace punctuations with spaces  |
| p | Strip Punctuation |
| q | Strip diacritics  |

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| s | Generate known spelling variants  |
| t | Strip stop words  |
| u | Uninvert the input phrase around commas  |
| w | Sort words by order  |
| y | Generate synonyms  |
| r | Generate synonyms, recursively  |

## Lexical Tools: Flows

**leave**

**leave**

**leaves**

**leaving**

**left**

**inflect**

**Notes:**
Some characteristics about the lexical tools:

Each of the utilities can be used separately. 
Here is an example of the inflect tool. 
This tool generates all known inflections of a word.  
The notion of a flow is a serial combination of these tools put together. 
Thus the name for each tool: a flow component. 

## Lexical Tools: Flows

- leave
- leave|**leave**|128|1|i|1|
- leave|**leave**|128|512|i|1|
- leave|**leaves**|128|8|i|1|
- leave|**left**|1024|64|i|1|
- leave|**left**|1024|32|i|1|
- leave|**leave**|1024|1|i|1|
- leave|**leave**|1024|262144|i|1|
- leave|**leave**|1024|1024|i|1|
- leave|**leaves**|1024|128|i|1|
- leave|**leaving**|1024|16|i|1|

**Notes:**
The example shows just how such a tool is invoked from the command line. 
Don’t pay any attention to those pesky fields with numbers in them right now.
Pay attention to the fact that we put leave into this tool and leave, leaves, left, and leaving were returned. 

[That same information should be shown in the GUI tool].

Chis: Example Here

## Lexical Tools: A Serial Flow

**Input **
**term**

**Remove possessive**

**lowercase**

**Strip punctuation**

**Remove stop words**

**Strip diacritics**

**Word order sort**

**Output **
**term**

**Flow components can be arranged so that the output of one is the input to another.**

**Notes:**
Here’s an example of such a serial combination. That is, 
where the output of one tool serves as
 the input to the next tool.
In this example, we are first lowercasing the input, then stripping diacritics, removing possessives, removing stop words, stripping punctuation, and finally sorting the resulting words in ASCII sort order.  
The example shows just how this flow is invoked on the command line. 

## Lexical Tools: A Serial Flow

**The Gougerot-Sjögren's Syndrome**
- The Gougerot-Sjögren's Syndrome**|**
**         ****gougerotsjogren syndrome**|2047|16777215|
-                                                                   l+q+g+t+p+w|1|

**Notes:**
Here’s how that sequence would be put together.  
The “-f” on the command line is an indication that what follows is to be taken as a sequence of flow components to execute in a serial order.
In this example, that sequence boils down to the sequence of the single character mnemonic forms: “L” for lowercase, q for strip diacritics, g for strip possessives, t for strip stop words, p for strip punctuation, and finally w for word order sort the input. 
In this example, The Gougerot-Sjögren's Syndrome, “the” is stripped off, the o umlaut is normalized to “o”, the hyphen is removed, and the resulting form is sorted.

## Lexical Tools: Parallel Flows

**term**

**Output **
**term**

**Multiple flows can be defined**

**noOperation**

**Uninflect**

**synonyms**

**Output **
**terms**

**Notes:**
The flow components can be put together into separate flows that produce transformations that are only dependant on that flow. 

In the example, we want both the original term and the synonyms of the term.

## Lexical Tools: Parallel Flows

- ear
- ear|**ear**|2047|1048575|n|1|
- ear|**aural**|1|1|B+y|2|
- ear|**auricularis**|1|1|B+y|2|
- ear|**otic**|1|1|B+y|2|
- ear|**otor**|1|1|B+y|2|

**Second Flow**

**First Flow**

**Notes:**
This is an example of how one would invoke such a parallel set of flows. 

The output consists of the combined results from the output of each flow. 
Note the last field of each output line, 
the number in red. 

That number indicates which flow it came from. 

Chris: Example Here

## Lexical Tools: Fielded Output

**Input Term**

**Output Term**

**Categories**

**Inflections**

**Flow history**

**Flow Number**

- leaves

- leaves

- 1152

- 1

- 136

- L

- |

- |

- |

- |

- |

- > lvg –f:L
- leaves

**Notes:**
Just what are you getting on the output?  Much more than you bargained for!  

On each output line, five additional fields are added to the input fields. 

The output or transformed term, 
what categories and inflections the output term can be, 
what transformations or flow components  produced it. This is also known as the flow history, 
And, 
the flow number that produced it. 

## Lexical Tools: Fielded Output

**Categories**

**verb**

**plural**

**pres3ps**

**Output**
**term**

**Inflections**

**leaves**

**noun**

**Notes:**
Imagine, if you will, that some flow produced the output term “leaves”.
Leaves can be either the plural form of a noun or the present third person singular of a verb.

That’s what conceptually will be part of the output for this term.

## Lexical Tools: Categories

**1152**

- adj

- adv

- aux

- compl

- conj

- det

- modal

**noun**

- prep

- pron

- verb

**0**

**0**

**0**

**1**

**0**

**1**

**0**

**0**

**0**

**0**

**0**

**Category bit vector**

**Notes:**
But, that’s not easy to stuff into a concise, machine parsable, single line representation that can be used by other tools. A somewhat less readable bit vector representation of category and inflection was adopted to be the concise, parsable form that could be used by other tools.

Let’s concentrate on category for a minute. The bit vector for category consists of 11 bits in this order. To represent the categories that leaves take on, the bits for noun and verb are set to one. 

What appears on the command line output is the decimal representation of this bit vector. 

## Lexical Tools: Categories

- adv

- aux

- compl

- conj

- det

- modal

- noun

- prep

- pron

- verb

**0**

**6**

**8**

**7**

**9**

**10**

**3**

**4**

**5**

**1**

**2**

**Bit Vector**
**positions**

| Adjective | 1 |
| --- | --- |
| Adverb | 2 |
| Auxiliary  | 4 |
| Complement | 8 |
| Conjunction | 16 |
| Determiner | 32 |

| Modal | 64 |
| --- | --- |
| Noun | 128 |
| Preposition  | 256 |
| Pronoun | 512 |
| Verb | 1024 |

**Notes:**
Here is the decimal representation of these categories.

## Lexical Tools: Inflections

**136**

- base

- comparative

- superlative

- plural

- Present participle

- Present 3ps

**0**

**1**

**0**

**0**

**0**

**0**

- Present 3ps

**1**

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

**Notes:**
Likewise, inflections are stored in a bit vector representing inflections. Here, the base form is a simplified form for singular if the term is a noun, infinitive if the term is a verb, and positive if the term is an adjective or adverb.  

Leaves in the example can be either the plural form of a noun or the present third person singular of a verb. 
Those bits are set, which, when represented as a decimal,  add up to be 136. 

## Lexical Tools: Inflections

| Base | 1 |
| --- | --- |
| Comparative | 2 |
| Superlative | 4 |
| Plural | 8 |
| Present Participle | 16 |
| Past | 32 |
| Past Participle | 64 |
| Present 3rd Person Singular | 128 |

**Notes:**
Here are the basic inflections viewed in their decimal representation. 

## Lexical Tools: Inflections

| Inflection | Expanded | Mapped |
| --- | --- | --- |
| Positive | 256 | 1 |
| Singular | 512 | 1 |
| Infinitive | 1024 | 1 |
| Pres 123p | 2048 | 128 |
| Past Neg | 4096 | 32 |
| pres123pNeg | 8192 | 128 |
| Pres1s | 16384 | 128 |
| past1p23pNeg | 32768 | 32 |
| past1s3sNeg | 65536  | 32 |

| Inflection | Expanded | Mapped |
| --- | --- | --- |
| Pres1p23p | 131072  | 32 |
| pres1p23p | 262144  | 128 |
| pres1p23pNeg | 524288  | 128 |
| past1s3s | 1048576  | 32 |
| pres | 2097152  | 128 |
| pres3sNeg | 4194304  | 128 |
| presNeg | 8388608  | 128 |
| all | 16777215  | 255 |

**Notes:**
Here are the inflections viewed in their decimal representation. 

## Lexical Tools: Post Flow Options

| SC | Show category names |
| --- | --- |
| SI | Show inflection names |
| ccgi | Mark the end of the set of variants returned |
| F:Int[:Int] | Specify fields for outputs |
| ti | Display the only input term in the output when using fielded input |
| R:Int | Restrict the number of variants returned  |

**Notes:**
The lexical tools have some useful post processing options that are of note. 
For the most part these options are filters. Though some of these options add information to the end of the processing.
Here are some miscillanious but useful options.

Oh, my favorites: Show the categories and inflections in string form, not in bit vector form.

-ccgi, an end of processing marker.

-F followed by the fields you want to have in your output.

-ti Forget about passing through all the input fields, only pass through the field you are picking up the term from. 

-R followed by a number to restrict the number of output rows. This is useful, say when you are generating lots of forms, but you only need the top 10. 

## Lexical Tools: Post Flow Options

- _[Show category names](file:///../lvg2003/docs/designDoc/requirement/lvgOptions/catStrOption.html)_

- _[Show inflection names](file:///../lvg2003/docs/designDoc/requirement/lvgOptions/inflStrOption.html)_

- > lvg -f:L -**SC -SI**
*phosphoprotein*
- phosphoprotein|phosphoprotein|**<noun>**|**<base+singular>**|L|1|
*sclerosing*
- sclerosing|sclerosing|**<adj+verb>**|**<base+presPart+positive>**|L|1|

**Show the category and inflection names**

**Notes:**
These options are necessary for those of us who cannot do the math.
 It displays the categories and inflections as strings rather than bit vectors. 
I generally use these for debugging purposes. 
The fields can get quite long for terms that can be many categories and inflections. 

## Lexical Tools: Post Flow Options

- _[Mark the end of the set of variants returned](file:///lvg2003/docs/designDoc/requirement/lvgOptions/endMarkOption.html)_

- > lvg -f:L **-ccgi**
- behavior
- behavior|behavior|128|513|L|1|
- __THE_END__

**Mark the end of processing**

**Notes:**
Here’s the –ccgi option.
 This option is useful when embedding this into applications where you need the tool to tell you that it’s done processing an input.  This tells the calling app. To stop waiting for more output.

## Lexical Tools: Post Flow Options

- _[Specify fields for outputs](file:///lvg2003/docs/designDoc/requirement/lvgOptions/outFieldOption.html)_

- > lvg -f:u -t:7 **-F:8:6**
- C0035440|ENG|S|L0035434|VW|S0003894|Rheumatic carditis, acute
***acute Rheumatic carditis***|***S0003894***

**Display only the 8****th**** and 6****th**** field from the output **

**Notes:**
Oh, yea, the –F option. 
I use this option frequently,
 to cull down the output to only those fields I need.
 In this example, I only want the output field and 
the field that has the SUI in it. 

## Lexical Tools: Post Flow Options

- _[Display only the input term field when using fielded input](file:///lvg2003/docs/designDoc/requirement/lvgOptions/filterInputOption.html)_

- > lvg -f:u -t:7 **–ti**
- C0035440|S0003894|***Rheumatic carditis, acute***
***Rheumatic carditis, acute***|acute Rheumatic carditis|2047|16777215|u|1|

**Display only the input term from the fielded input to**
**the output**

**Notes:**
The –ti option is always used in conjunction with the option that indicates the input field that contains the term.
An equivalent way of doing this would also be using the –F option with the input term field.

## Lexical Tools: Post Flow Options

- _[Restrict the number of variants returned ](file:///lvg2003/docs/designDoc/requirement/lvgOptions/numberOption.html)_

- > lvg -f:i **-R:**2
- foo
- foo|foo|128|1|i|1|
- foo|foos|128|8|i|1|

**Limit the number of output terms to 2**

- Note:  The unrestricted output would
-  have produced 12 rows otherwise

**Note: Dangerous!**
**Do not try this at home!**

**Notes:**
Here’s an example using the truncation filter. Here I’m passing an unknown word to be inflected. If I did not have this filter on, the tools would generate 12 forms.

## Lexical Tools: Global Behaviors

| i:filename | Define input file name  |
| --- | --- |
| o:filename | Define output file name  |
| x:filename | Loading an alternative configuration file  |
| p | Interactive prompt |
| m | Print extra information of flow mutations  |
| s:Str | Defines a field separator.  |

**Output **
**terms**

**Input **
**term**

**Output **
**terms**

**Notes:**
Here are some options that are worth mentioning.

The tools can take a file as input and can put the output to a file.
The tools can be configured in strange and divient ways by altering the config file.
You can tell the tools at startup time to use an alternative config file rather than the default one.

When using the tool interactively on the command line, it’s sometimes useful to have a prompt.

## Lexical Tools: No Operation

**-f:n**

**Copies the input term to the output with no transformation**

- > lvg -f:**n** -f:d -f:y -SC -SI
- force
- force|**force**|<all>|<all>|**n**|1|
- force|forcefully|<adv>|<base>|d|2|
- force|forceful|<adj>|<base>|d|2|
- force|forcible|<adj>|<base>|d|2|
- force|dynamic|<adj>|<base>|y|3|

**Notes:**
Now, some specific flow components.
I’m going to cover only a few of them. Refer to the documentation for the rest of them. 
The most trivial, but really useful flow component is just a pass through. The –f:n flow component just repeats the input term as an output term. This turns out to be useful, for instance, when you want to have the original term as one of the outputs.

In this example, force, the input term, is on the output of the “make me an index of terms, their derivations and their synonyms” flows only because of that pass through flow component,     “-f:n”.

## Lexical Tools: Inflect

**-f:i**

**Generate inflectional variants **

- > lvg  -f**:i** -SC -SI
- West Nile Virus
- West Nile Virus|**West Nile virus**|<noun>|<base>|i|1|
- West Nile Virus|**West Nile virus**|<noun>|<singular>|i|1|
- West Nile Virus|**West Nile viruses**|<noun>|<plural>|i|1|

**Notes:**
The inflect flow component. This flow component uses the LEXICON to find all the known inflectional forms of a term. 
Here’s a pretty simple example. West Nile Virus, a term popular these days in Maryland.
In fact, for a given term, this utility will uninflect the term, then retrieve all the inflected forms for that term, including the form that came in.  

If the term is not in the LEXICON, inflectional suffix adding and replacement rules based on the regular rules of inflection are applied to the input form to generate potential inflectional forms, in the hopes that at least some of resulting forms are useful. The rules come with exceptions to prune out forms that we’ve noticed go wrong. 
The inflection flow component is just one configuration of the morphology module that produces both inflections, uninflections and derivations.

## Lexical Tools: Inflect

**Generate inflections, filter by cat and/or inflection**

**-f:ici~cats+infls**

- > lvg -f:**ici~128+ALL** -SC -SI
- bioassay
- bioassay|**bioassay**|<noun>|<base>|ici|1|
- bioassay|**bioassay**|<noun>|<singular>|ici|1|
- bioassay|**bioassays**|<noun>|<plural>|ici|1|

**Notes:**
But wait! There’s more to inflection. Inflections can be filtered to include or exclude certain categories and inflections. 

For example, you may only want to build an index of noun forms, or build an index of all nouns being in their plural form.

The options to the –f:i flow component include the decimal representation of the categories and inflections to include. 

## Lexical Tools: Inflect Simple

**Generate inflections noting simplified inflection tags**

**-f:is**

- > lvg -f:**is**
- cut|cut|1024|1|is|1|
- cut|cutting|1024|16|is|1|
- cut|cut|1024|32|is|1|
- cut|cut|1024|64|is|1|
- cut|cut|1024|128|is|1|
- cut|cuts|1024|128|is|1|
- cut|cut|128|1|is|1|
- cut|cuts|128|8|is|1|

**Notes:**
But wait! There’s more to inflection. Inflections can be filtered to include or exclude certain categories and inflections. 

For example, you may only want to build an index of noun forms, or build an index of all nouns being in their plural form.

The options to the –f:i flow component include the decimal representation of the categories and inflections to include. 

## Lexical Tools: Uninflect by Term

**-f:b**

**Uninflect by term**

- > lvg -f:**b** -SC -SI
- left atria
- left atria|**left atrium**|<noun>|<base>|b|1|

**Notes:**
The uninflect by term flow component. 
You can pass in terms from the input that contain multiple words, but where you want to treat as a single entity. 
Here’s an example where uninflecting the term would be different than uninflecting each word. Why?
If we consider left atriums as a term, and uninflect it as a term, we get “left atrium” as the resulting form. If we were to uninflect by each word, left would uninflect to both left atrium and leave atrium as well. Not what we had intended.
This flow component is useful when you know you are working with terms.
There is a flow component that uninflects by word, when you don’t have that luxury.

## Lexical Tools: Derivations

**-f:d**

**Generate derivations**

- > lvg -f:**d** -SC -SI
- diagnostic
- diagnostic|**diagnosis**|<noun>|<base>|d|1|
- diagnostic|**diagnostics**|<noun>|<base>|d|1|
- diagnostic|**diagnose**|<verb>|<base>|d|1|
- diagnostic|**diagnostical**|<adj>|<base>|d|1|

**Notes:**
Generate derivational forms.
Derivational forms are terms that are somewhat related to the original term, but change meaning in some way. They almost always involve shifting category as well.
By facts, such as nominalizations of verbs, adjectives and nouns from the LEXICON. 
By suffix replacement rules that also have exceptions. These rules are by-directional, and include things like add or remove the suffix “tion”, or “ly” from the input term. Here are some of the derivational forms of diagnostic.

## Lexical Tools: Derivations

**-f:dc~cats**

**Generate derivations, filter by category**

- > lvg -f:**dc~129** -SC -SI
- Reduce
- reduce|**reducer**|<noun>|<base>|d|1|
- reduce|**reduction**|<noun>|<base>|d|1|
- reduce|**reducible**|<adj>|<base>|d|1|

**Notes:**
Like inflection, derivational forms can be filtered by category.
Derivational forms can be used when creating related variants when doing query expansion. Derivational variants, by their nature, involve some change in meaning and category. This can be over aggressive. One way to constrain some of this over generation, is to filter to reasonable categories. 
For instance, you know you are working with nouns, you may want to constrain the derivations to nouns and adjectives. 

## Lexical Tools: Synonyms

**-f:y**

**Generate synonyms**

- > lvg -f:**y** -SC -SI
- kidney
- kidney|**nephric**|<adj>|<base>|y|1|
- kidney|**nephritic**|<adj>|<base>|y|1|
- kidney|**renal**|<adj>|<base>|y|1|

**Notes:**
Generate synonyms. 
This is a table lookup operation that retrieve synonyms from the table of synonyms that come from one of the LEXICON’s supplemental data sources.
These synonyms are intended to complement, not duplicate, the Metathesaurus’s set of synonymy. The synonyms that are on this list are more on the word rather than the “Metathesaurus term” level. Metathesaurus terms can be whole sentences.
The set of synonyms is small, and were created on a somewhat ad-hoc basis to cover word level synonymy that the Metathesaurus does not, or had not had at the time. One example is chest for breast.

## Lexical Tools: Normalize (norm)

**-f:N**

**Remove stop words, then remove genitives, then replace punctuation with spaces, then lowercase, then uninflect each word, then take each of the uninflected words, then word order sort. **

- > lvg -f:N
- Syndrome, Dry Eyes
- Syndrome, Dry Eyes|**dry eye syndrome**|2047|1|g+o+t+l+B+w|1|

**Notes:**
Generate normalized forms. This is the Norm  transformation. We know this tool also as the “Norm” tool.
I think that I’ve talked enough about norm for you to get the idea. 
The only other thing I’ll add is to repeat that norm, luiNorm, and wordind are all just utilities that are flow components from LVG.

## Lexical Resources

**Notes:**
Where to get the tools, the lexicon, the documentation, the updates. 
Where else? the Resources page from the knowledge source server.

## Building an Index Using The Lexical Tools

- Can we build a tool that increases recall?
- Can we build a tool that increases precision?

**Notes:**
Can we build a tool that increases recall?
Can we build a tool that increases precision?

## Building an Index Using The Lexical Tools

- Can we build a tool that increases precision?
- Case
- Constrain by part of speech
- Filter to the lexicon

**Notes:**
We may want to have case be significant. 
Or we may only want to handle words that we know about.

Chris: LVG Example Here

## Building an Index Using The Lexical Tools

- Can we a tool that increases recall?
- - Include
- 		synonyms
- 		derivations
- 		acronyms and their expansions
- 		spelling variants

**Notes:**
We may want to include additional word level synonymy, 
or derivations,
or acronyms and abbreviations,
or spelling variants to increase recall.

Chris: LVG Example Here