Natural Language Processing — Simplified
By: Srini Janarthanam
Tags: Naturallanguageprocessing, Machine Learning, Chatbots, Conversational Ai, AI

What is NLP and how does it make machines smarter at communication?

One of the first things that fascinated me in Artificial Intelligence is for a computer having the ability to speak and have a conversation with humans. When I saw C-3P0 speak on a giant screen, way back when I was a kid, I was mesmerized. I probably didn’t realise it was fake back then, but I wanted in. I wanted to build robots that can speak, that can take instructions, that can have an engaging conversations with me.

Here I am now, in my almost 40s, still building chatbots (not really robots) and struggling to make them work as good as humans in conversation. Let’s set aside my sad story for a bit and come back to our topic — Natural Language Processing.

From the very early days, having a human like conversational behavior was seen as a test of intelligence in artificial computing systems.

From the very beginning, the ability of a computing system to understand and engage in human language conversation has fascinated many people. Early on in history, when the concept of Artificial Intelligence began to take shape, a British mathematician, Alan Turing, who incidentally worked on cracking the ENGIMA code during the second world war, published one of the earliest papers on Artificial Intelligence. In this paper, he put forward the idea of an imitation game, later became popularly known as the Turing test. In this game, an AI system would chat with a human judge using a type-text interface. If such a system is able to fool the judge into thinking that it is in fact human, then it can be deemed to be intelligent. So from the very early days, having a human like conversational behavior was seen as a test of intelligence in artificial computing systems.

Natural Langugage Processing (NLP) is not only concerned with computers making conversations but almost everything to do with language. Can computers process natural language? Can they understand a news article and answer questions relating to the article? Can they recognise words spoken by humans and speak back to them? Can they understand and carry out instructions in the digital and physical world? Can they crunch streams of data and summarize the key points, trends, patterns in a text report? All of these and more come under the ambit of Natural Language Processing.

Natural Language Processing can be understood in terms of grand challenges, tasks and tools.

Challenges

Several grand challenges were identified as the NLP research began to gather steam. Translating text from one natural language to another (e.g. English to French) — Machine Translation, comprehending text and answering questions — Question Answering, recognizing speech — Automatic Speech Recognition, generating speech (Speech synthesis), recognizing text from images — Optical Character Recognition, generating text reports from data — Natural Language Generation, classifying text documents into categories — Text Classification, retrieving relevant documents — Information Retrieval, reasoning with text — Textual Entailment, having meaningful conversations in NL with a robot or a chatbot — Conversational AI are some of the main challenges that researchers in the academia and industry have been working on for a very long time. Some of these challenges may be called by different names between the industry and the academia.

These challenges involve processing natural language, sometimes at several different levels — Phones & phonemes, Morphology, Syntax, Semantics and Pragmatics.

Most of the challenges, either implicitly or explicitly, process language at following levels. These levels come from the field of linguistics — the science of human languages.

Level 1: Phones and phonemes — At this level, we try to understand the input at the acoustic level. Can we discover phones (i.e. smallest unit of sound in spoken human languages that are independent of a specific language) and phonemes (i.e. all phones that are part of a language). Phonemes make syllables, and syllables make words. Understanding phone and phonemes are essential to speech processing tasks.

Level 2: Morphology — The next level deals with words. These may be typed or spoken. Once words are identified, they are broken down into morphemes. Morphemes are smallest units that make up words. For instance, the word ‘booked’ can be broken into ‘book’ and ‘-ed’. The word ‘book’ here is a verb, not a noun. The ‘-ed’ part identifies its tense to be in the past. This kind of analysis simplifies processing and helps the computer to deal with variations of the word (e.g. tense, gender, number). It also allows the system to understand the meaning of the word (e.g. ‘book’, the noun is different from ‘book’ the verb).

Level 3: Syntax — Syntax analysis is where the words are grouped together into phrases that make sense. Phrases are then nested under clauses and clauses under sentences. This creates a tree like representation. Syntactic analysis is one level away from understanding the meaning of a sentence. It however lays the ground work by identifying the key action (i.e. verb) and related actors (i.e. nouns) and the attributes of the action as well as the actors (e.g. adverbs, articles and adjectives).

Level 4: Semantics — Semantics is where we try to understand the meaning of the sentence. The output of semantic analysis is meaning representation of input text. Meaning can be represented in many formats . Many formalisms have been developed for the purpose— predicate logic, discourse representation structures, etc. But off late the most popular way of representing meaning of an utterance in conversational AI is the use of intents and entities.

Level 5: Pragmatics — Finally we put the meaning of the individual utterance or input text into context to understand the contribution of the current text in a holistic manner. This lets us contextualize the input text, based on which output can be processed and generated — response to utterance, text summary, translated sentence, etc.

However not all levels of processing are explicitly done. Sometimes some levels are skipped and other times they are implicitly done (e.g. encoded as a layer of neurons in a multi layer neural network architecture). Another major issue is that in all of the above levels, there is the problem of ambiguity. Words, phrases and sentences can be ambiguous and may mean multiple things at the same time. This kind of fuzziness is what make language difficult to process.

Source: Facebook — Language Nerds

Tasks

One level below challenges are tasks.

NLP tasks are more generic and help us achieve the grand challenges.

Tasks can been seen as challenges broken down and typically serve handle analysis in one of the levels mentioned above. Because all the challenges deal essentially with language, when broken down, they share a number of tasks.

POS tagging — This task is about labeling a sequence of words with their parts of speech.

Named Entity Recognition — This task takes as input a sequence of words and identifies named entities in it such as person names, city names, names of organizations, etc.

Syntax parsing — In this task, an utterance or a sentence is converted into a syntactic structure. It identifies the how the constituent parts of speech (PoS) are related to each other. Another variation of this task is called dependency parsing where the constituents are not identified using their part of speech or grammatical phrases but on the role they play in the sentence (e.g. subject, object, etc).

Intent classification — This task is about identifying the central proposition of the utterance. Utterances are classified into their semantic categories. For instance, utterances like ‘I want to book a ticket’, ‘Can I get tickets to..’, etc will mapped to an intent request_book_ticket because they mean the same idea. This is popularly used in Natural Language Understanding (NLU) modules like IBM Watson Assistant, Google DialogFlow, etc to provide chatbots the capability to understand what the user is saying.

Language Modelling — This task is about creating a model of a language using a corpus of data. A popular approach is called the n-gram model. These models encode the knowledge of how likely sequences of n-words appear in the language. If I asked you to fill the following sentence — ‘How do you __?’, you would say ‘do’. This exactly is what the n-gram model would do. It cal also be used to calculate the likelihood of a sequence of words in a language. For instance, the model would say that ‘How you do do?’ is less likely than ‘How do you do?’. There are other variants (e.g. skip-gram, CBOW, etc) that are used for similar tasks. In general, language models are used to enhance the linguistic knowledge of the system.

Tools

To help with the tasks above, NLP toolkits provide a number of reusable tools. Let’s have brief look at each one of them.

NLP tools are specific to linguistic data and are ways to pre-process and cleanup such data.

Tokenisation — Tokenisation is the task of creating a list of tokens from an input string. Tokens include words and punctuation marks.

Stopwords removal— Stopwords are a list of words in the language that offer little content value to a sentence. Words like ‘a’, ‘an’, ‘the’, ‘I’, ‘you’, etc are very common in any text. So these are removed as the first step when processing text for tasks like information retrieval. Since all words are equally weighted, it doesn’t make sense to weigh stopwords like ‘you’ and content words like ‘language’ the same way.

Stemming — Every words has a root or base form upon which a morphological inflection may be added. Stemmers strip the inflection and extract the root of the give word. For instance, a stemmer would convert ‘swimming’ to ‘swim’, ‘friendly’ to ‘friend’, etc.

Lemmatization — This is very similar to stemming but instead of the base form, Lemmatization identifies the canonical form of the word. For instance, ‘went’ will be converted into ‘go’.

Edit distance — Edit distance is the distance between two similar words in terms of their spelling. For instance, the words ‘book’ and ‘look’ are off by distance 1, because it takes one substitution (‘b’ to ‘l’) to go from one to other. The operations that are counted are — substitution, insertion and deletion. One of the most popular algorithms to calculate edit distance is called Levenshtein distance algorithm.

As you can imagine, these tools are used in several language processing tasks. There are several toolkits that implement these tools — NLTK, Spacy, CoreNLP, etc.

Evolution of NLP

How has the field of NLP evolved?

So far, we can divide this into three distinct phases — rule & knowledge based NLP, shallow machine learning based NLP and deep learning based NLP.

NLP 1.0 — This is the earliest phase on NLP evolution. During this time, the NLP tasks were mostly powered by curated linguistic knowledge bases and handcoded rules. Early NLP systems such as SHRDLU and Eliza were built using such approaches. Since a huge manual effort is involved in creating rules and ontologies, these systems worked in limited toy domains that were called block worlds. In addition to rules, huge knowledge bases like Cyc, WordNet were created that were used to provide semantic and commonsense world knowledge to NLP systems.

NLP 2.0 — Since mid 1980s, machine learning and statistical approaches were introduced to support most NLP tasks. Instead of writing rules and creating linguistic knowledge bases, data was used to train machine learning algorithms to learn the rules automatically. Algorithms like Hidden Markov Models (HMM) were used for sequence tagging tasks like PoS tagging, Named Entity Recognition, etc. Classifiers like Maximum Entropy models, Bayes Nets, logistic regression, etc were used to classify documents into categories. Statistical models were used generate syntactic and semantic parse trees for input sentences. Challenges like machine translation started using approaches like Expectation Maximization to learn alignment between words and phrases between languages. In phase 2, there was a steady migration from purely rule and knowledge based solutions to machine learning solutions based on data.

NLP 3.0 — Advances in Deep Learning techniques have propelled machine learning tasks to the next league. Deep learning refers to new multi layered neural network architectures that can learn hierarchical representations of the world from data. The first layer in deep learning for NLP tasks is often a word embedding layer. Word embedding layer is a neural network that encodes the given input in terms a distributional vector. The principle that is applied here is called the Distributional Hypothesis — words with similar meanings tend to occur in similar contexts. This means that the word embeddings for semantically similar words like ‘creator’, ‘writer’ and ‘author’ would be closer to each other than an unrelated word like, for instance, ‘athlete’. This encoding represents input in a much richer way than earlier encoding approaches like one-hot encoding. These vectors are then used to feed into further layers that either classify them (e.g. topic classification, sentiment analysis) or generate an output sequence (e.g. PoS, NER or translation).

As these networks can be trained on any corpus of text, they can be pre-trained. Availability of pre-trained networks allows engineers to quickly build robust pipelines that can address their NLP challenges with a performance much better than shallow approaches used earlier. This also addresses the problem of curse of dimensionality —the problem of sparse data . Deep learning approaches have shown many fold improvement in many of the standard NLP tasks. With new concepts and architectures like transformers, encoder-decoders and attention, I believe this trend will continue in the near future.

Conclusion

So there you go! An overview of natural language processing (NLP) — the challenges, tasks and tools and a brief look into how the field has evolved over time. Recently, Google introduced its chatbot called Meena using a neural conversational model that can engage with users in open domain conversations just like humans would. Amazon introduced one of theirs a little while later that was focused on customer service conversations. There are many more to come and the field of NLP is now hotter than ever. Stay tuned.