A simple NLP application for ambiguity resolution | by Laura Gorrieri | Towards Data Science

Ambiguity is one of the biggest challenges in NLP. When trying to understand the meaning of a word we consider several different aspects, such as the context in which it is used, our own knowledge of the world, and how a given word is generally used in society. Words change meaning over time and can also mean one thing in a certain domain and another in a different one. This phenomenon can be observed in homographs — two words that happen to be written in the same way, usually coming from different etymologies — and polysemy — one word that carries different meanings.In this tutorial, we’ll see how to resolve ambiguity in PoS tagging and semantic tagging, using expert.ai technology.
Please check how to install expert.ai NL API python SDK, either on this Towards Data Science article or on the official documentation, here.
Language is ambiguous: not only a sentence could be written in different ways and still convey the same meaning, but even lemmas — a concept that is supposed to be far less ambiguous — can carry different meanings.
For example, the word play could refer to several different things. Let’s take a look at the following examples:I really enjoyed the play.I’m in a band and I play the guitar.
Not only the same word can have different meanings, but it can be used in different roles: in the first sentence, play is a noun, while in the second it’s a verb. Assigning the correct grammatical label to each token is called PoS (Part of Speech) tagging and it’s not a piece of cake.
Let’s see how to resolve PoS ambiguity with expert.ai — first, let’s import the library and create the client:
We’ll see the PoS tagging for two sentences — notice how the lemma key is the same in both sentences, while its PoS changes:
To analyze each sentence we need to create a request to NL API: the most important parameters — shown in the code below as well — are the text to analyze, the language, and the analysis we are requesting, represented by the resource parameter.Please notice that expert.ai NL API currently supports five languages (en, it, es, fr, de). The resource we use is disambiguation, which performs multi-level tagging as the product of the expert.ai NLP pipeline.Without further ado, let’s create our first request:
Now we need to iterate over the PoS of the text and check which one was assigned to the lemma key:
Part of speech for "The key broke in the lock."The            	POS: DETkey            	POS: NOUNbroke in       	POS: VERBthe            	POS: DETlock           	POS: NOUN.              	POS: PUNCT
What is printed above, is a list of PoS following UD Labels, where NOUN indicates that the lemma key is here used as a noun. This should not be the case for its homograph that we see in the second sentence, in which key is used as an adjective:
Part of speech for "The key problem was not one of quality but of quantity."The            	POS: DETkey            	POS: ADJproblem        	POS: NOUNwas            	POS: AUXnot            	POS: PARTone            	POS: NUMof             	POS: ADPquality        	POS: NOUNbut            	POS: CCONJof             	POS: ADPquantity       	POS: NOUN.              	POS: PUNCT
As you can see printed above, the lemma key was correctly recognized as an adjective in this sentence.
One word can also have the same grammatical label and have different meanings. This phenomenon is called polysemy. Being able to infer the correct meaning for each word is to perform semantic tagging.
Words that are more common tend to have more meanings that have been added to them in time. For example, the lemma paper can have multiple meanings, as seen here:I like to take notes on paper.Every morning my husband reads the news from the local paper.
Pointing out the correct meaning of every single lemma is an important task, as one document could change meaning or focus based on that. To do so, we must rely on technology that is well developed and robust, since semantic tagging heavily depends on many pieces of information that come from the text.
For semantic tagging IDs are often used: these IDs are identifiers of concepts, and each concept will have its own ID. For the same lemma, e.g. paper, we will have a certain id x for its meaning as a material, and another y for its meaning as a newspaper.These IDs are usually stored in a Knowledge Graph, that is a graph in which each node is a concept and the arches are the connections between concepts that follow a certain logic (e.g. an arch could link two concepts if one is the hyponym of the other).Let’s now look at how expert.ai performs semantic tagging. We begin by choosing the sentences from which we will compare the two lemmas solution:
And now the request for the first sentence — using the same parameters as the previous example:
Semantic information is found in the syncon attribute for each token: a syncon is a concept, that is stored in expert.ai’s Knowledge Graph; each concept is formed by one or more lemmas, which are synonyms.Let’s see how the information is presented in the document object:
Semantic tagging for "Work out the solution in your head."Work out       	CONCEPT_ID: 63784the            	CONCEPT_ID: -1solution       	CONCEPT_ID: 25789in             	CONCEPT_ID: -1your           	CONCEPT_ID: -1head           	CONCEPT_ID: 104906.              	CONCEPT_ID: -1
Each token has its own syncon, whereas some of them present -1 as concept id: this is the default ID assigned to tokens that do not have any concept, such as punctuation or articles.So, if for the previous sentence we obtain concept id 25789 for the lemma solution, for the second sentence we should obtain another one since the two lemmas have a different meaning in the two sentences:
Semantic tagging for "Heat the chlorine solution to 75° Celsius."Heat           	CONCEPT_ID: 64278the            	CONCEPT_ID: -1chlorine       	CONCEPT_ID: 59954solution       	CONCEPT_ID: 59795to             	CONCEPT_ID: -175             	CONCEPT_ID: -1° Celsius      	CONCEPT_ID: 56389.              	CONCEPT_ID: -1
As expected, the lemma solution corresponds to a different concept id, indicating that the lemma used has a different meaning from the previous sentence.
Please find this article as a notebook on GitHub.
NLP is hard because language is ambiguous: one word, one phrase, or one sentence can mean different things depending on the context. With technologies such as expert.ai, we can solve ambiguity and build solutions that are more accurate when dealing with the meaning of words.