Fundamentals of Text Analytics
By: Sowmya Surampalli
Tags: Naturallanguageprocessing, Text Analytics, AI, NLP, Lexical Analysis

Natural language is the basic form of communication between human beings. With the hugely evolved social media, e-commerce and digitization, there is a lot of scope to make the machines process the natural language and gain insights. These insights help the industries to improve their products, marketing strategies, performance etc.

For example, in the e-commerce websites, it is quite usual to see reviews of a product. People usually write the positives and negative sides of the product. By analyzing these reviews, product companies can take business decisions to improve their backdrop.

For a typical NLP problem, we can give a generalized solution outline as:

Data Cleaning Text Analytics Machine learning model Dashboarding the insights (Optional)

Each of the above steps must be customized as per the problem needs. The scope of this article is to elaborate the text analytics phase.

In the text analytics phase, the data is pre-processed and converted to a format that is understood by any machine learning model. The processing steps can further be categorized as

Lexical Processing Syntactic Processing Symantec Processing

Lexical processing: Lexical Processing focuses on the basic components in the raw text data such as words and sentences. It does not bother about the grammar and meaning of the sentences. Based on the problem’s need, the data must be broken into paragraphs, sentences, or words.

For example, a simple classifier that is designed to classify the emails to be ham or spam can make use of just the words in the email. Emails having words such as “Lottery”, “lucky”, “gift” are more likely to be spam. Lexical processing suffices for these kinds of problem statements where the meaning of text need not be considered.

However, lexical processing itself involves many steps of processing the data such as breaking the raw text into words, converting the words to their root form, removing the unwanted words, spelling correction etc.

Lexical processing is usually insufficient in case of complex applications such as machine translation, automatic chat bots etc. For these kinds of applications, we should consider going with syntactic and semantic processing as well. Because “my cat eats in the morning” and “my cat has been eating since morning” have totally different meanings. Lexical processing cannot understand the difference in these sentences because it deals only with the words.

Syntactic processing: Syntactic processing tries to extract some meaning out of the sentence based on its structure. It focuses on the grammatical syntax of the sentence instead of focusing just on words.

One use case is identifying the subject and object in a sentence. With syntactic processing we can identify who is affecting and who is affected by. This can help in building interactive chat bots and question and answering systems. For example, while posting a question “how can I get the refund?” to an interactive chat bot, the object of the message can be understood, and the machine can look up for the answer in its database.

Semantic processing: There can be cases where the problem statement demands understanding the exact meaning of the sentence where lexical and syntactic processing cannot suffice. In these cases, a more sophisticated processing of the data is essential.

Machine translation can be quoted as an example to understand the requirement of Semantic processing. To translate a sentence into another language, the meaning of the whole sentence must be understood. Translating each word will not yield a meaningful sentence in the other language.

Now, the machine should be capable of identifying synonyms, antonyms, abbreviations etc. It should also consider the words around. The machine should be able to develop an intelligence that ‘cow’ and ‘bull’ come under the same category of animals but just differ in their gender. Manually storing the relations is quite tedious and it would be great if the machine can learn the relationships by itself.

These are the steps involved in the text pre-processing phase of solving a typical NLP problem. The extent of pre-processing the text data depends on the problem’s need. If the problem could be solved by analyzing the words in the data, Lexical processing would suffice. If the structure and meaning must be considered to a little extent, syntactic processing can be done. If ‘understanding the meaning of the sentence’ is the whole problem solver, semantic processing of the data would be required.