# Presentation: 009264

## Slide 1

## Slide 2

## Lexical Tools: Introduction

- Command line tools
- norm
- Lvg
- wordInd
- Web GUI
- Pure Java Application
- Embeddable Java API’s

**Notes:**
What ARE these tools, AND  what are they good for? 
These tools are primarily command line tools. Built from java code.
There are  GUI’s that run these tools.
The source code is distributed. 
These tools are intended to be embedded into your applications. 
As such, there are an untold number of public JAVA API’s that 
were developed to be called within your app.

Chris*

## Lexical Tools: Introduction

- These tools are good for
- aggressive text pattern matching
- making word, term, phrase indexes
- matching queries with indexed entries
- increasing recall and/or precision

**Notes:**
What are these tools good for?
These tools are good for text matching that is more powerful than exact match.
The tools are used to normalize terms, as well as generate expanded terms.
These tools are used to create word, term, and phrase indexes. They are  embedded  between queries and  target indexes. 

If your purpose is to increase recall over exact matches, the lexical tools can be   configured to do that.  
If your purpose is to increase precision over some loose matching algorithm, the lexical tools can be configured to do that.

## Lexical Tools: Introduction

- Characteristics of all the command line tools
- take input from the screen or a file
- put their results to the screen or a file
- Interpret fielded text
- Can be told which fields contain what type of information

**Notes:**
Some general characteristics of all of the command line tools:
They all take input from standard in, and put results on standard out.
They all are configured via command line arguments.

They all interpret fielded text, using by default, the UNIX’sh Pipe character 
 as a field separator, and a newline (natively defined) as the end of record, 
or end of term indicator.
That is, all the tools can be told 
what field to do the text transformation on,
what fields to pass through to the output,                 and
what fields contain other relevant information.

Chris

## Lexical Tools: WordInd

**wordind**
- is
- a
- tool
- to
- break
- terms
- into
- words
- it
- is
- used
- to

**Wordind** is a tool to break terms into words.
- It is used to take a row from a Metathesaurus table that contains** **a term, sentence, paragraph, story,
- and break the text part of that row into it’s constituent words.

**Notes:**
Wordind is a tool to break terms into words. 
It is used to take a row from a Metathesaurus table that contains a term, sentence, paragraph, story, 
and break the text part of that row into it’s constituent words,
while passing along, duplicating the other fields in that row. 
From the very beginning we needed a way to standardize what we meant what a word is. This turns out to be critically important for fitting programs together.
We consider a word what wordInd says it is, for better or worse.  WordInd, considers as a word a sequence of alpha-numeric tokens broken by white space or punctuation. Whitespace and puncutation is subsequently thrown out. 

## Lexical Tools: WordInd

- Breaks words into tokens
- Passes other fields to output, untouched
- Lowercases
- Removes white space and punctuation

**Notes:**
WordInd is primarily used to create word indexes. As such, it does a bit more than tokenizing words. It also lowercases the tokenized words as well. 
So, it tokenizes, ,
    it strips punctuation, ,
         it lowercases, ,
           and it passes other fields through to the output, untouched.

## Lexical Tools: WordInd

- Useful command line options for wordInd

| -t[:Num] | Defines what field to tokenize |
| --- | --- |
| -f[:Num[:Num]] | Defines what fields get passed through |

**Notes:**
Here are two well used command line options that wordInd uses. The –t option tells wordInd what field to tokenize, and the –f option tells word in what fields to pass through to the output untouched.

## Lexical Tools: WordInd

- > wordInd –t:7 –F:1:6
- C0185495|S0298948|**denis**
- C0185495|S0298948|**browne**
- C0185495|S0298948|**splint**
- C0185495|S0298948|**strapping**

**C0185495**|ENG|P|L0223844|PF|**S0298948****|Denis-Browne splint strapping|**3**|**

**Notes:**
Here’s an example of what wordind does to a row from the Metathesaurus’ MRCON. 
The command line arguments to WORDIND include telling wordind to use the seventh field to find the term, and to pass through fields one and six to the output.

## Lexical Tools: Norm

**Metathesaurus**
**English**
**Strings**

**norm**

**Normalized string index**

**Normalized word index**

**WordInd**

**MRXNS.ENG**

**MRXNW.ENG**

**Notes:**
Here’s where the lexical tools fit in with the UMLS Metathesaurus.
The norm program normalizes the Metathesaurus strings to produce the normalized string index. 
In general,  normalization involves transforming something that’s not uniform to something that is uniform.

The wordInd program tokenizes the normalized strings to create the normalized word index.

## Lexical Tools: Norm

**norm**

**Normalized string index**

**Normalized word index**

**Metathesaurus**
**Concepts**

**Query**

**Normed**
**term**

**SUIS**

**Metathesaurus**
**Concepts that match**
**The normalized query**

**Notes:**

These indexes provide great recall enhansed access to the content of the Metathesaurus.

To take advantage of these indexes, queries need to use the same program to transform the query to mediate between the query and entries in the normalized indexes.

## Lexical Tools: Norm

- Norm abstracts away from:
- case
- punctuation
- word order
- possessive forms
- inflectional variation

**Notes:**
Our form of normalization involves the following:

It abstracts away from case, punctuation, and word order. 
It abstracts away from inflectional variations. 
It abstracts away from possessive forms.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**lowercase**

**uninflect each word**

**word order sort**

**replace punctuation with spaces**

**Hodgkin's **
**Diseases, **
**NOS**

**Notes:**
Norm in some more detail. 

Let’s take the example Hodgkin’s Diseases, NOS. Here’s a term that could be found in text an awful lot of ways. 
This program will transform a fair number of those variant ways into a uniform representation that will enable applications to recognize all those variants belonging to the same term.

## Lexical Tools: Norm

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

- Lexical Tools: Norm

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Genitives,  those apostrophe ses, or the possessive forms are removed.
Medical terminology is replete with terms that include possessive forms, 
Such as Parkinson’s Disease. 
There are a lot of variation caused by forms that differ by possessive forms. 
In a lot of cases, as Allen had mentioned,  terms evolve from possessive forms to plural forms.
It turns out to be a good idea to abstract away from possessive forms for retrieval purposes. 

In our example, Hodgkin’s gets transformed to Hodgkin. 

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Punctuation is replaced by spaces.

It is important to identify the apostrophe s’s 
before punctuation is replaced.  

The comma is removed in this example.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Stop words are stripped. 
Stop words are those words that you really don’t want to query on, 
such as determiners and frequently used prepositions. 
For example, words like “THE”, and ”OF”.  

We consider NOS as one of our stop words. It gets stripped.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
For obvious reasons, it’s a big win to either lower case your input or uppercase it.

We’ve settled on lowercasing the input to abstract away from case variation in terms. In this example, everything is lowercased.

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**hodgkin disease **

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
Each word of the term is uninflected. 
Uninflection is the what we would consider an entry form in our Lexicon. These are, for the most part,
   the singular forms of nouns,  
   the infinitive forms for verbs, and
   the positive forms for adjectives and adverbs. 
Uninflected forms are similar to, but are not the same as Stemmed forms. Stemming involves suffix stripping, not bothering to heed the distinction between derivational suffixes and inflectional suffixes. Stemmers often refer to the “stems” as the root of the word, regardless whether the meaning had changed when the suffix was removed or added.

Stemmers often only involve suffix replacement and may or may not filter to a word list after the suffix transformations. 

## Lexical Tools: Norm

**remove genitives**

**remove stop words**

**replace punctuation with spaces**

**lowercase**

**uninflect each word**

**word order sort**

**Hodgkin's **
**Diseases, **
**NOS**

**disease hodgkin **

**disease**
** ****hodgkin**

**hodgkin disease **

**hodgkin diseases **

**Hodgkin Diseases **

**Hodgkin Diseases **
**NOS**

**Hodgkin Diseases, **
**NOS**

**Hodgkin'sDiseases, **
**NOS**

**Notes:**
As a last step, all the resulting words are sorted in ASCII order. 
Why?
There are terms in the Metathesaurus in inverted form. That is, where the important part of the term is at the front of the term.  “Cancer, Breast”. Inversion is a useful cataloguing technique.

There are terms in the Metathesaurus in uninverted form. And, there are terms in the Metathesaurus that are in inverted form, but, where the inversion points, typically commas, are missing.  

We found that stripping punctuation and sorting in word order, in general helps rather than hurts recall. There rare exceptions to this, such as the distinction between school nursing and nursing school. 

## Lexical Tools

**Notes:**

The lexical tools include a word and string normalization tool, 
a word tokenization tool, 
and a suite of text utilities,  otherwise known as lexical variant generation , or LVG.
We often refer to the lexical tools as the Swiss army knife of text utilities. It’s got 54 different tools, some sharp, some blunt. 

This presentation contains material that is suitable for end-users and developers. This presentation contains excruciating detail on the tools wtih some gratuitous code examples thrown in.
Chris

## Lexical Tools: Flow Components

| Mnemonic | Tool |
| --- | --- |
| A | Return known acronyms  |
| a | Return known acronym expansions  |
| b | Uninflect a term  |
| c | Tokenize a term into "words"  |
| Ct | Retrieve the citation term  |
| d | Generate derivational variants  |
| g | Remove genitive  |
| i | Generate inflectional variants  |
| L | Retrieve category and inflection for a term  |

**Notes:**
I’m going to switch terminology at this point from talking about text  utilities or tool and introduce and refer to each utility as this funny thing called a 
“FLOW COMPONENT” I’ll explain later.
There are 54 tools, or FLOW COMPONENTS with more on the way. 
I’m going to flip through an alphabetical list of some of these tools.  (Not all are listed here. They are fully specified in the documentation).

The mnemonic on the left is what you would specify on the command line. 
These include tools to grab acronyms and their expansions.
Uninflection tools, tokenxization tools.

## Lexical Tools: Flows

**leave**

**leave**

**leaves**

**leaving**

**left**

**inflect**

**Notes:**
Some characteristics about the lexical tools:

Each of the utilities can be used separately. 
Here is an example of the inflect tool. 
This tool generates all known inflections of a word.  
There is a  notion of a flow is a serial combination of these tools put together. 
Thus the name for each tool: a flow component. 

## Lexical Tools: Flows

- leave
- leave|**leave**|128|1|i|1|
- leave|**leave**|128|512|i|1|
- leave|**leaves**|128|8|i|1|
- leave|**left**|1024|64|i|1|
- leave|**left**|1024|32|i|1|
- leave|**leave**|1024|1|i|1|
- leave|**leave**|1024|262144|i|1|
- leave|**leave**|1024|1024|i|1|
- leave|**leaves**|1024|128|i|1|
- leave|**leaving**|1024|16|i|1|

**Notes:**
The example shows just how such a tool is invoked from the command line. 
Don’t pay any attention to those pesky fields with numbers in them right now.
Pay attention to the fact that we put leave into this tool and leave, leaves, left, and leaving were returned. 

[That same information should be shown in the GUI tool].

## Lexical Tools: Fielded Output

**Input Term**

**Output Term**

**Categories**

**Inflections**

**Flow history**

**Flow Number**

- leaves

- leaves

- 1152

- 1

- 136

- L

- |

- |

- |

- |

- |

- > lvg –f:L
- leaves

**Notes:**
On each output line, five additional fields are added to the input fields. 

The output or transformed term, 
what categories and inflections the output term can be, 
what transformations or flow components  produced it. This is also known as the flow history, 
And, 
the flow number that produced it. 

## Lexical Tools: Fielded Output

**Categories**

**verb**

**plural**

**pres3ps**

**Output**
**term**

**Inflections**

**leaves**

**noun**

**Notes:**
Imagine, if you will, that some flow produced the output term “leaves”.
Leaves can be either the plural form of a noun or the present third person singular of a verb.

That’s what conceptually will be part of the output for this term.

## Lexical Tools: Categories

**1152**

- adj

- adv

- aux

- compl

- conj

- det

- modal

**noun**

- prep

- pron

- verb

**0**

**0**

**0**

**1**

**0**

**1**

**0**

**0**

**0**

**0**

**0**

**Category bit vector**

**Notes:**
But, that’s not easy to stuff into a concise, machine parsable, single line representation that can be used by other tools. A somewhat less readable bit vector representation of category and inflection was adopted to be the concise, parsable form that could be used by other tools.

Let’s concentrate on category for a minute. The bit vector for category consists of 11 bits in this order. To represent the categories that leaves take on, the bits for noun and verb are set to one. 

What appears on the command line output is the decimal representation of this bit vector. 

## Lexical Tools: Categories

- adv

- aux

- compl

- conj

- det

- modal

- noun

- prep

- pron

- verb

**0**

**6**

**8**

**7**

**9**

**10**

**3**

**4**

**5**

**1**

**2**

**Bit Vector**
**positions**

| Adjective | 1 |
| --- | --- |
| Adverb | 2 |
| Auxiliary  | 4 |
| Complement | 8 |
| Conjunction | 16 |
| Determiner | 32 |

| Modal | 64 |
| --- | --- |
| Noun | 128 |
| Preposition  | 256 |
| Pronoun | 512 |
| Verb | 1024 |

**Notes:**
Here is the decimal representation of these categories.

## Lexical Tools: Inflections

**136**

- base

- comparative

- superlative

- plural

- Present participle

- Present 3ps

**0**

**1**

**0**

**0**

**0**

**0**

- Present 3ps

**1**

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

- 0

**Notes:**
Likewise, inflections are stored in a bit vector representing inflections. Here, the base form is a simplified form for singular if the term is a noun, infinitive if the term is a verb, and positive if the term is an adjective or adverb.  

Leaves in the example can be either the plural form of a noun or the present third person singular of a verb. 
Those bits are set, which, when represented as a decimal,  add up to be 136. 

## Lexical Tools: Inflections

| Base | 1 |
| --- | --- |
| Comparative | 2 |
| Superlative | 4 |
| Plural | 8 |
| Present Participle | 16 |
| Past | 32 |
| Past Participle | 64 |
| Present 3rd Person Singular | 128 |

**Notes:**
Here are the basic inflections viewed in their decimal representation. 

## Lexical Tools: Fielded Input

** **

**Categories**

**Input**
**term**

**leaves**

**noun**

**Inflections**

**Notes:**
If the output knows about category and infection, it would reason that we should be able to tell each tool any category and inflection information about an input term. 

We can. 
We can put on the input, via fielded input, category and inflection information. 
Suppose we have leaves as an input term, and we know we are talking about the leaves, the noun, as in “the leaves of a tree”.  

## Lexical Tools: Fielded Input

- > lvg –f:L **–t**:1 **–cf**:2
- leaves|128
- leaves|leaves|128|8|L|1

**Input category field**

**Input term field**

**Notes:**
On the command line, we tell the tool which field to look for the term, and which field to look for the category information. 

On the input, we then can put the categories in the field, using the bit vector form. 

## Lexical Tools: Post Flow Options

| SC | Show category names |
| --- | --- |
| SI | Show inflection names |
| ccgi | Mark the end of the set of variants returned |
| F:Int[:Int] | Specify fields for outputs |
| ti | Display the only input term in the output when using fielded input |
| R:Int | Restrict the number of variants returned  |

**Notes:**
The lexical tools have some useful post processing options that are of note. 
For the most part these options are filters. Though some of these options add information to the end of the processing.
Here are some miscellaneous but useful options.

 Show the categories and inflections in string form, not in bit vector form.

-ccgi, an end of processing marker.

-F followed by the fields you want to have in your output.

-ti Forget about passing through all the input fields, only pass through the field you are picking up the term from. 

-R followed by a number to restrict the number of output rows. This is useful, say when you are generating lots of forms, but you only need the top 10. 

## Lexical Tools: Post Flow Options

**Show category names**

**Show inflection names**

- > lvg -f:L -**SC -SI**
*phosphoprotein*
- phosphoprotein|phosphoprotein|**<noun>**|**<base+singular>**|L|1|
*sclerosing*
- sclerosing|sclerosing|**<adj+verb>**|**<base+presPart+positive>**|L|1|

**Show the category and inflection names**

**Notes:**
These options are necessary for those of us who cannot do the math.
 It displays the categories and inflections as strings rather than bit vectors. 
I generally use these for debugging purposes. 
The fields can get quite long for terms that can be many categories and inflections. 

## Lexical Tools: A Serial Flow

**Input **
**term**

**Remove possessive**

**lowercase**

**Strip punctuation**

**Remove stop words**

**Strip diacritics**

**Word order sort**

**Output **
**term**

**Flow components can be arranged so that the output of one is the input to another.**

**Notes:**
Here’s an example of such a serial combination. That is, 
where the output of one tool serves as
 the input to the next tool.
In this example, we are first lowercasing the input, then stripping diacritics, removing possessives, removing stop words, stripping punctuation, and finally sorting the resulting words in ASCII sort order.  
The example shows just how this flow is invoked on the command line. 

## Classpath
NormApi()
LvgCmdApi()

- NormApi()
- LvgCmdApi()

- Lexical Tools: **Embedding These Tools into Your Application**

**Notes:**

Here’s the section for developers and those that want to embed the lexical tools into their apps.
I’m going to cover the prerequisites, and the basic classes to use, from the simplest to the more featured interfaces.

The full code to the next examples can be found in the appendix to this tutorial, and are distributed as examples within the source code distribution.

## Lexical Tools: Embedding These Tools into Your Application

**CLASSPATH = ${CLASSPATH}:**
**    ****${LVG_DIR}:**
**    ****${LVG_DIR}/lib/*****lvg2003dist.jar:***
**    **

- Lexical Tools: **Embedding These Tools into Your Application**

**Notes:**
But first, some prerequisates: Classpath, classpath, classpath. Here’s what you need to add to your classpath to use our tools. 

The LVG_DIR, the lvg2002.jar file, and one of the following depending on if you are using the IDB database or MySQL database.

- import gov.nih.nlm.nls.lvg.Api.*;
- NormApi        normalize = new NormApi();
- String            *input2Norm* = null;
- Vector  outputFromNorm = null;

- Lexical Tools: **Embedding Norm into Your Application**

**Notes:**
Here are the important pieces of code that are needed to imbed this into your app. 

You need to import the Lvg.Api.* packages.
You need to instantiate the NormApi class. 
Better have handy a string and a vector variable ...

- while ( (*input2Norm* = stdIn.readLine() ) != null ) {
-    outputFromNorm= **normalize.Mutate**(*input2Norm*);
-    for ( int i = 0; i < outputFromNorm.size(); i++ ) {
-         System.out.println((String) outputFromNorm.get(i));
-     }
- }
**normalize.CleanUp**();

- Lexical Tools: **Embedding Norm into Your Application**

**Notes:**
I’m going to assume that you know how to read terms in from some stream, or method. 
This bit of code shows that you can call the normalize.Mutate method over and over again
In a loop.  This Mutate method (yes it’s an uppercase M, by design) takes a string as input,
And returns a vector of strings. 

This example shows looping through the resulting output vector, printing out the resulting
Normalized terms.

When your done, be nice, clean up. This method closes open files, open database connections.

**Your application**

**LvgCmd**
**Api**

**Input **
**term**

**Output **
**terms**

- Lexical Tools: **Embedding Lvg into Your Application**

**Notes:**
Now, for a simple way to embed other lexical tools into your application. 
For simple tasks, there is the LvgCmdApi() that takes a term from the input
And returns a vector of strings. It’s similar in structure to the NormApi() 

- import gov.nih.nlm.nls.lvg.Api.*;
- LvgCmdApi lvgApi = new **LvgCmdApi**(“-f:b –CR:o –SC –SI”);
- String            *input2Lvg* = null;
- Vector   outputFromLvg = null;

- Lexical Tools: **Embedding Lvg into Your Application**

**Notes:**
Again, you need to import the LvgApi packages. 

You will need to instantiate the LvgCmdApi with an argument. The argument is a string. 
That string should contain what you would have put on the command line if you were
Running the process from the command line.

- Lexical Tools: **Embedding Lvg into Your Application**

- while ( (*input2Lvg* = stdIn.readLine() ) != null ) {
-    outputFromLvg= **lvgApi.MutateToString**(*input2Lvg*);
-     for ( int i = 0; i < outputFromLvg.size(); i++ ) {
-         System.out.println((String) outputFromLvg.get(i));
-     }
- }
- lvgApi.CleanUp();

**Notes:**
This bit of code shows that you can call the normalize.Mutate method over and over again
In a loop.  This Mutate method (yes it’s an uppercase M, by design) takes a string as input,
And returns a vector of strings. 

This example shows looping through the resulting output vector, printing out the resulting
mutated terms. The output is the same as what would have gone out to standard out
From the command line.

When your done, be nice, clean up. This method closes open files, open database connections.

## Using The Lexical Tools with The Metathesaurus

**Metathesaurus**
**English**
**Strings**

**norm**

**Normalized string index**

**Normalized word index**

**WordInd**

**MRXNS.ENG**

**MRXNW.ENG**

**Notes:**
Here’s where the lexical tools fit in with the UMLS Metathesaurus.
The norm program normalizes the Metathesaurus strings to produce the normalized string index. Each normalized string is also associated with three types of identifiers, a concept identifier, a lexical term identifier, and a string identifier. These ids are keys into all the other tables.  The normalized string index is then broken into a normalized word index via wordind, carrying along the identifiers.

## Using The Lexical Tools with The Metathesaurus

**norm**

**Normalized string index**

**Normalized word index**

**Metathesaurus**
**Concepts**

**Query**

**Normed**
**term**

**SUIS**

**Metathesaurus**
**Concepts that match**
**The normalized query**

**Notes:**
Application developers put the norm program between queries and the Metathesaurus. 
Normalized queries are then used get entries from either the normalized string or word index to get the ids to retrieve Metathesaurus concept information. 
Metathesaurus concept information is then returned to satisfy the request.

## Using The Lexical Tools with The Metathesaurus

**norm**

**dry eye syndrome**

**Dry Eyes Syndrome**

**Notes:**
Here’s a simple minded example. Suppose the query is Dry eyes syndrome. 
The query gets normalized.

## Using The Lexical Tools with The Metathesaurus

- ENG|**dry eye syndrome**|C0013238|L0013238|**S0035652**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0090228**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0090454**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0220550**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S0368350**|
- ENG|**dry eye syndrome**|C0013238|L0013238|**S1459074**|

**Normed**
**term**

**Notes:**
All entries that share the normalized form are retreived from the normalized string index. The string unique identifiers to these rows are used ...

## Using The Lexical Tools with The Metathesaurus

- C0013238|ENG|P|L0013238|VS    |S0004019|Dry eye syndrome
- C0013238|ENG|P|L0013238|VS    |S0368350|Dry Eye Syndrome
- C0013238|ENG|P|L0013238|VS    |S1459074|dry eye syndrome
- C0013238|ENG|P|L0013238|VWS|S0090228|Syndrome, Dry Eye
- C0013238|ENG|P|L0013238|VWS|S0220550|Dry, eye syndrome
- C0013238|ENG|P|L0013238|VW  |S0090454|Syndromes, Dry Eye

**SUIS**

**MRCON**

**C0013238|ENG|P|L0013238|PF  |S0035652| Dry Eye Syndromes**

**Notes:**
To retrieve Metathesurus concept information. 
If one of the things you want to return are the the concept names, not the string names for the query, you can review all the returned strings to see which ones are the concept names. I’ve got one of these rows highlighted that indicates that it’s the preferred term for that group of strings, and that group of strings is considered the preferred group for that concept. All told, this is what we consider the concept name. 

## Building an Index Using The Lexical Tools

- Can we build a tool that increases precision?
- 	Case
- 	Constrain by part of speech
- 	Filter to the lexicon
- Can we a tool that increases recall?
- 		synonyms
- 		derivations
- 		acronyms and their expansions
- 		spelling variants

**Notes:**
We may want to have case be significant. 
Or we may only want to handle words that we know about.

## Building an Index Using The Lexical Tools

**Metathesaurus**
**Strings**

**String index**

**Word index**

**Wordind**

**Notes:**
Just as norm was used to create the normalized word and string index, we can come up with our own.
I’ve got a really small set of rows from the Metatheaurus, that we can use to create an index on, load into mysql, then retrieve via a web page that uses our norm.
We can create our norm by putting our flavor in script that is called by the indexing script as well as the web page behind the query.

## Building an Index Using The Lexical Tools

**       **

**String index**

**Metathesaurus**
**Concepts**

**Query**

**Transformed**
**query**

**SUIS**

**Metathesaurus**
**Concepts that match**
**the transformed query**

**Notes:**
On the retrieval side, I’ve created simple java program that calls our norm script to transform the query, retrieves entries from our norm’d index, gets the suis, and uses the suis to get strings from the Metathesaurus. 
I’ve got it set up as word index. 
Now to demonstrate that ....

## Slide 50

## Gspell: Introduction

- The GSpell program is a spelling suggestion tool that uses a mix of algorithms to retrieve close neighbors. This application is best suited to applications that index at the word or term level of tokenization.
- BagOWordsPlus is a phrase retrieval tool. This tool is useful to retrieve closest matching phrases to data such as strings from the Metathesaurus.

## GSpell: Usage

*Usage*
- 	**GSpellFind**.[*sh*|*bat*]
- 		--**dictionary**=*NameOfDictionary*
- 		[--**inputFile**=*Source*] [--**outputFile**=*target*]
- 		[--**truncate**=*N*] [--**considerNCandidates**=*N*]
- 		[--**maxEditDistance**=*N*]

## GSpell: Indexing

*Usage*
**GSpellIndex**.[*sh*|*bat*]
- 	--**dictionary**=*NameOfDictionary*
- 	--**inputFile**=*SourceFile*
- 	[--reportTime] [--version][--help]
- Format for the input file
- One word per line

## GSpell: Output

-  anonomous|**anonymous**|1.0|0.87|NGrams|
-  anonomous|**allonomous**|2.0|0.58|NGrams|
-  anonomous|**autonomous**|2.0|0.58|NGrams|
-  anonomous|**anadromous**|3.0|0.29|NGrams|
-  anonomous|**analogous**|3.0|0.29|NGrams|
-  anonomous|**anomalous**|3.0|0.29|NGrams|
-  anonomous|**anonymously**|3.0|0.29|NGrams|
-  anonomous|**anonymes**|3.0|0.29|Metaphone|
-  anonomous|**anonyms**|3.0|0.29|Metaphone|
-  anonomous|**acoprous**|4.0|0.11|NGrams|

| 
Input Term | 
Suggestion | Edit
Distance | 
Rank | 
Method | 
Message |
| --- | --- | --- | --- | --- | --- |

## GSpell: API

**import** gov.nih.nlm.nls.gspell.**GSpell**;        // <-------These come from the gspell.jar
**import** gov.nih.nlm.nls.gspell.**Candidate**;
**GSpell** gspell = new **GSpell**( _dictionaryName, 					     GSpell.READ_ONLY );
**Vector **candidates = gspell.***find***( aTerm );
**if** ( candidates != null )
-   for ( int i = 0; i < candidates.length; i++ )
- 	  System.out.println(candidates[i].toString());
-  **else**
-     System.out.println("**No Suggestions**");
- gspell.cleanup();

## BagOWordsPlus: Usage

*Usage*
- 	**BagOWordsPlusFind**.[*sh*|*bat*]
- 		--**dictionary**=*NameOfDictionary*
- 		[--**inputFile**=*Source*] [--**outputFile**=*target*]
- 		[--**truncate**=*N*] [--**considerNCandidates**=*N*]
- 		[--**maxEditDistance**=*N*]

## BagOWordsPlus: Indexing

*Usage*
**BagOWordsPlusIndex**.[*sh*|*bat*]
- 	--**dictionary**=*NameOfDictionary*
- 	--**inputFile**=*SourceFile*
- 	[--reportTime] [--version][--help]
- Format for the input file
- One phrase per line

## BagOWordsPlus: Output

- sleep|sleep|0.0
- sleep|S-sleep|2.0
- sleep|S sleep|2.0
- sleep|REM sleep|4.0
- sleep|deep sleep|5.0

| 
Input Term | 
Suggestion | Edit
Distance |
| --- | --- | --- |

## BagOWordsPlus: API

**import** gov.nih.nlm.nls.gspell.BagOWordsPlus;  // <-------These come from the gspell.jar
**import** gov.nih.nlm.nls.gspell.**Candidate**;
**BagOWordsPlus**  ir = new **BagOWordsPlus**( args );
**Vector** candidates = ir.***get***( aTerm );
**if** ( candidates != null )
-   for ( int i = 0; i < candidates.length; i++ )
- 	  System.out.println(candidates[i].toString());
-  **else**
-     System.out.println("**No Suggestions**");
- ir.cleanup();

## SPECIALIST NLP Tools:Table of Contents

- Logical/physical views of the functionalities
- The tools as stand-alone applications
- Command line options
- Example output
- API functionalities
- The model of a document
- Parts list
- Structure
- Details: Lexical Element
- Details: Token

## Table of Contents

- Example applications using the API's
- Sentence Tokenizer
- Noun Phrase Tokenizer

## Introduction: Logical View

- Word Tokenizer
- Term Tokenizer
- Phrase Tokenizer
- Sentence Tokenizer
- Section Tokenizer

## Introduction: Physical View

- Section/Sentence/Word Tokenizer
- Term Tokenizer
- a.k.a lexical lookup, term recognizer
- Phrase Tokenizer
- a.k.a phrase chunker, noun phrase extractor, parser

## Section/Sentence/Word Tokenizer

- Tokenizes text into
- Sections (paragraphs)
- Sentences
- Tokens

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- Abstract. The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

**Tokenizer**

## SPECIALIST NLP Tools: Tokenizer

*Usage*
- 	**tokenize**.[*bat*|*sh*] [*Options*]
- 		--**fileName**=*fileName*
- 		--**outputFileName**=*fileName*
- 		--**inputType**=[*freeText*|*medlineCitations*]
- 		--**sections**
- 		--**sentences**
- 		--**tokens**
- 		--**pipedOutput**
- 		--**indicate_citation_end**

## SPECIALIST NLP Tools: Tokenizer

**Sentence**|1|97|182|But those follow-up tests have been inconclusive, state 		and federal officials said.
- Token|16|97|99|0|0|**But**|||
- Token|17|101|105|1|0|**those**|||
- Token|18|108|113|2|0|**follow**|||
- Token|19|114|114|2|0|**-**|||
- Token|20|115|116|3|0|**up**|||
- Token|21|118|122|4|0|**tests**|||
- Token|22|124|127|5|0|**have**|||
- Token|23|129|132|6|0|**been**|||
- Token|24|134|145|7|0|**inconclusive**|||

- tokenize.bat --inputFile=5.txt --inputType=freeText --sentences --tokens
-                      --pipedOutput

## SPECIALIST NLP Tools: Tokenizer

**// =============+ Create a TokenizeAPI object +==**
**TokenizeAPI** **tokenizer** = **new** **TokenizeAPI**( argv );
**// =======================+ Tokenize the file +==**
**Document** aDocument =
- 			tokenizer.***processDocument***( aFile);
**Vector**      **tokens** = aDocumen**t**.***getTokens***() ;
**int** **numberOfTokens** = tokens.size();
**Token**       **aToken** = **null**;
**// ====================+ Print the tokens out +==**
**for** ( **int** **i** = 0; i < numberOfTokens; i++ ) {
- 	aToken = (**Token**) tokens.get(i);
- 	System.out.println( aToken.***toPipedString***() );
- }

## SPECIALIST NLP Tools: Term Tokenizer

- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- Abstract. The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

- SPECIALIST NLP Tools: Term Tokenizer

- Chunks tokens into
- 	 terms
- From SPECIALIST
- 	Lexicon
- From regular
- 	expressions

**Term**
**Tokenizer**

## Usage
	LexicalLookup.[bat|sh] [Options] 
		--fileName=fileName
		--outputFileName=fileName
		--inputType=[freeText|HTML|medlineCitations]
		--sections
		--sentences
		--lexicalElements
		--lexicalEntries
		--tokens
		--pipedOutput

- 	**LexicalLookup**.[*bat*|*sh*] [*Options*]
- 		--**fileName**=*fileName*
- 		--**outputFileName**=*fileName*
- 		--**inputType**=[*freeText*|*HTML*|*medlineCitations*]
- 		--**sections**
- 		--**sentences**
- 		--**lexicalElements**
- 		--**lexicalEntries**
- 		--**tokens**
- 		--**pipedOutput**

- SPECIALIST NLP Tools: Term Tokenizer

## SPECIALIST NLP Tools: Term Tokenizer

- Lexical Element|12|SHAPE:Unlabeled|*unknown*|**Richmond**|67|74
- Lexical Element|13|LEXICON|prep|**for**|76|78
- Lexical Element|14|LEXICON|adj|**further**|80|86
- Lexical Element|15|LEXICON|verb|**testing**|88|94
- Lexical Element |16|PUNCTUATION|punctuation|**.**|95|95
- Lexical Element |17|LEXICON|prep|**But**|97|99
- Lexical Element |18|LEXICON|det|**those**|101|105
- Lexical Element |20|LEXICON|adj|**follow-up**|108|116
- Lexical Element |23|LEXICON|noun|**tests**|118|122
- Lexical Element |24|LEXICON|aux|**have**|124|127

- LexicalLookup.bat --inputFile=5.txt --inputType=freeText                      			  --lexicalElements --lexicalEntries --pipedOutput

## SPECIALIST NLP Tools: Term Tokenizer

- Lexical Element|17|LEXICON|prep|**But**|97|99
***LexicalEntry***|but|conj|base|E0014465
***LexicalEntry***|but|prep|base|E0014464
- Lexical Element|18|LEXICON|det|**those**|101|105
***LexicalEntry***|those|det|plural|E0060728
***LexicalEntry***|those|pron|base|E0060729
- Lexical Element|20|LEXICON|adj|**follow-up**|108|116
***LexicalEntry***|follow-up|adj|base|E0028422
- Lexical Element|23|LEXICON|noun|**tests**|118|122
***LexicalEntry***|tests|verb|pres3s|E0060349
***LexicalEntry***|tests|noun|plural|E0060348

- LexicalLookup.bat --inputFile=5.txt --inputType=freeText                      			  --lexicalElements --lexicalEntries --pipedOutput

## SPECIALIST NLP Tools: Term Tokenizer

**// ==========+ Create a LexicalLookupAPI object +==**
**LexicalLookupAPI** **look** = **new** **LexicalLookupAPI**(argv);
**// ============================+ Chunk the file +==**
**Document** aDocument = look.***processDocument***( aFile );
**Vector** les = aDocumen**t**.***getLexicalElements***();
**int** **numberOfLexElements** = les.size();
**LexicalElement** **aLexElement** = **null**;
**// =============+ Print the LexicalElements out +==**
**for** (**int** **i** = 0; i< numberOfLexElements; i++ ) {
- 	aLexElement = (LexicalElement) les.get(i);
- 	System.out.println(aLexElement.***toPipedString***());
- }

## SPECIALIST NLP Tools: Phrase Tokenizer

- Chunks sentences
- 	into simple phrases

**Phrase**
**Tokenizer**

## SPECIALIST NLP Tools: Phrase Tokenizer

*Usage*
- 	**npParser**.[*bat*|*sh*] [*Options*]
- 		--**fileName**=*fileName*
- 		--**outputFileName**=*fileName*
- 		--**inputType**=[*freeText*|*HTML*|*medlineCitations*]
- 		--**sections**
- 		--**sentences**
- 		--**phrases**|--**nps**|--**mincoMan**
- 		--**lexicalElements**
- 		--**lexicalEntries**
- 		--**tokens**
- 		--**pipedOutput**

## SPECIALIST NLP Tools: Phrase Tokenizer

- Phrase|0|0|10|**The company**|*company*
- Phrase|1|12|14|**has**|
- Phrase|2|16|24|**forwarded**|
- Phrase|3|26|39|**some materials**|*materials*
- Phrase|4|41|62|**to a state laboratory**|*state laboratory*
- Phrase|5|64|74|**in Richmond**|*Richmond*
- Phrase|6|76|86|**for further**|*further*
- Phrase|7|88|94|**testing**|

- npParser.bat --inputFile=5.txt --inputType=freeText --phrases                 		       --pipedOutput

## SPECIALIST NLP Tools: Phrase Tokenizer

**// ====================+ Create a Parser object +==**
- Parser parser = **new** Parser( argv );
**// ============================+ Parse the file +==**
**Document** aDocument = parser.***processDocument***(aFile);
**Vector**     phrases = aDocumen**t**.***getPhrase***() ;
**Int ****numberOfPhrases** = phrases.size();
- Phrase     **aPhrase** = **null**;
**// =====================+ Print the Phrases out +==**
**for** ( **int** **i** = 0; i < numberOfPhrases; i++ ) {
- 	aPhrase = (Phrase) phrases.get(i);
- 	System.out.println( aPhrase.***toPipedString***() );
- }

## Document: A Model

- How it’s put together

- Image from: **www.themodelcarmuseum.org/ 49box0.jpg**

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- Abstract. The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

- Title

- Abstract

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

## Document Model: Parts List

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

- Specialist Lexicon

## Document Model: Parts List

**SPECIALIST Lexical Records**
- {base=**capillary**
-  entry=E0015013
- 	cat=adj
- }
- {base=**capillary**
-  entry=E0015014
-     cat=noun
- }

- Sections
- Sentences
- Phrases
- Terms
- Words
- Lexicon Entries

- Specialist Lexicon

## Document Model: Structure

- Document

- Section

- Section

## Document Model: Structure

- One to Many
- Relationship

## Document Model: Structure

- One to Many
- Relationship

## Document Model: Structure

- One to Many
- Relationship

## Document Model: Structure

- One to Many
- Relationship

## Document Model: Structure

- One to Many
- Relationship

## Document Model: Lexical Element

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

- Multi-word term

- single-word term

- vitamin E

## Document Model: Lexical Element

- Document Model: Lexical Element

**     **

- Token:
- E

- Lexical Element:

- Vitamin E

- Token: vitamin

- Token: E

- Token:
- vitamin

- Token:
- vitamin

- Token:
- E

**    ****Lexical Entry:**
- {base=**vitamin E**
-  entry=E0064781
-    cat=noun

- Tagger
- Part of
- speech
- tag

## Document Model: Token

**Token**

- Part of Speech Tag
- Span

**Span**

- Char begin offset
- Char end offset

## Document Model: Phrase

**Phrase**

- String displayTags()
- String displayVariants()
- List getAllVariants()
- UMLS_ConceptPointer getConceptPointer()
- UMLS_ConceptPointer[] getConcepts()
- List getDerivedPhrases()
- ArrayList getFinalMappings()
- List getLexicalElements()
- List getNp()
- String getNpString()
- List getNpTokens()
- String getOriginalString()

**Phrase (cont.)**

- int getPhrasePosition()
- int getSizeOfPhrase()
- String getTrimmedString()
- boolean isOfPhrase()
- boolean isPrepPhrase()
- String toMincoManString()
- String toMoString()
- String toPipedString()
- String toString()
- String toSyntaxString()

## Assembly Instructions

## Term Tokenizer

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- Abstract. The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

**Sentence**

**Lexical Element**

**Token**

- Lexical
- Entry

## Phrase Tokenizer

**SPME determination of volatile aldehydes for evaluation of in-vitro antioxidant activity**Elena E. Stashenko, Miguel A. Puertas, Jairo R. Martínez
- A1 Chromatography Laboratory, Research Center for Biomolecules, School of Sciences, Industrial University of Santander. A.A. 678, Bucaramanga, Colombia
**Abstract:**
- Abstract. The in-vitro antioxidant activity of natural (essential oils, vitamin E) or synthetic substances (tert-butyl hydroxy anisole (BHA), Trolox) has been evaluated by monitoring volatile carbonyl compounds released in model lipid systems subjected to peroxidation. The procedure employed methodology previously developed for the determination of carbonyl compounds as their pentafluorophenylhydrazine derivatives which were quantified, with high sensitivity, by means of capillary gas chromatography with electron-capture detection. Linoleic acid and sunflower oil were used as model lipid systems. Lipid peroxidation was induced in linoleic acid by the Fe2+ ion (1 mmol L-1, 37 °C, 12 h) and in sunflower oil by heating in the presence of O2 (220 °C, 2 h).

**Sentence**

**Lexical Element**

**Token**

- Lexical
- Entry