Natural Language Processing Explained
By: Ethan Koch
Tags: Machine Learning, Artificial Intelligence, NLP, Technology, Naturallanguageprocessing

Co-authored by a NLP dream team: Stella Liu (AI Product Manager), David Kearns (Product Manager in Data and AI), Nadine Handal (Data Scientist), Shubham Agarwal (AI Research Scientist), Michael Flores (AI Architect) and me (Machine Learning Developer)

Introduction

While AI has sometimes become synonymous with chatbots like Siri and Facebook Discovery bots, this is only scratching the surface of what the underlying technology — natural language processing (NLP) — can do for your product or your organization.

Google “natural language processing” though and you’ll get high level messaging on how ubiquitous it is or deep technical details on its implementation — co-reference resolution, neural network dependency parser and entity recognizer… anyone?

We (a data scientist, NLP consultant, NLP research scientist and AI product manager with a collective 50+ years of experience in this technology) decided that this is not helpful for people to grasp the potential of this technology and understand how to get started.

We collectively wrote this article for you — a tech-savvy but non-technical reader — to get the inside scoop on what this technology is, common success patterns that the team has seen that work in real-world situations, questions you can ask yourself to vet its relevance to the problem you’re facing and a few links so you can pursue deeper research on your own.

Our goal is to de-mystify NLP and provide a way for you to get started. Let us know if we succeeded in the comments.

What is Natural Language Processing?

Natural Language Processing (NLP) is an interdisciplinary field that spans techniques to process, understand, and analyze human language. NLP enables most of the current state-of-the-art AI applications by providing algorithms that convert the data humans understand (e.g. English, French, speech, tweets, e-mails etc.) into data computers understand and can operate upon. If you find yourself with any sort of written, typed, or spoken data in your business, you should be asking someone: “How can NLP help me turn this data into business value?”

Real world applications of NLP

If you’ve ever found yourself saying “Ok Google” or “Hey Siri”, you’ve experienced NLP without even knowing it. Many of the experiences we have today are supported in some way by NLP. When you request something from a virtual assistant like Siri, the system is using NLP to understand the meaning of your phrase so that it can respond appropriately. The four key patterns of NLP use cases that this article will explore are improving the consumer end user experience, democratizing access to big data in a large organization, speech recognition (translating speech to text) and sentiment analysis.

With NLP, we can make applications easier for users to interact with. For instance, imagine re-creating the experience of filling out a long, lengthy online form on the internet. No one enjoys tediously filling out all of the information that’s required from them. Instead of spending time trying to understand the information the online form needs, end users can be guided through the entire process of completing the form by a chatbot. NLP could also be used to auto-correct misspellings or misinformation. If desired, the entire online form can be replaced by a chatbot that engages the end user in conversation to extract the information needed.

Getting access to data in a large organization is notoriously time-consuming. Business analysts usually have to resort to creating an IT ticket to access the information they need or using SQL. Advances in NLP can help these business analysts go straight to the source and query the database in natural language — asking the computer questions like “What region performed best last quarter?” or “Compare store x and store y revenue in 2010 and 2012.” Salesforce’s WikiSql initiative, Narrative Science and Kueri are all examples that allow users the ability to ‘talk’ to their data.

NLP can also translate speech to text which is the backbone of chatbots. In the business context, translating speech to text helps automate manual business processes. For instance, in healthcare, auditing clinical trial interviews used to be a very lengthy task — analysts had to listen to hours of clinical interview trials in order to determine whether or not the person consented to do the trial. With advances in NLP, these audio interviews can be converted to text and consent can be determined with a simple “crt+f” operation.

Finally, sentiment analysis of text can help businesses tap into human-generated text such as news, blog posts, forums, and social media and understand the consumer in a deeper way. As consumers have started voicing their opinions on these public platforms, businesses can analyze this raw data to make sense of trends across their opinions and take action proactively.

Why NLP is better than traditional programming

To understand when an NLP solution is needed, we have to understand the end goal and the problem we’re trying to solve. While the definition of natural language processing is very broad — covering written and spoken language, the fact that you have data with human language doesn’t always mean you need to use NLP.

For example, if the goal is to identify a specific term “x”, a simple key word search operation can achieve this goal since this does not require deep understanding of what the text is about. On the other hand, if your application needs to assist lawyers in extracting the information they need from a huge collection of legal documents, then simple keyword search is not sufficient. Think of it this way — if you were to hire an assistant, they can’t do their research without proper training. They’d have to understand the legal language, the relationships and concepts that are relevant for their research subject. They will also need to know the domain in which they’re working in such as business law vs international law. NLP helps go beyond simple keyword search and by being more domain-specific. You can implement your own NLP application that can be trained specifically for your domain (like business law) to extract the relevant information from large amounts of textual data even more accurately and efficiently than simple keyword search.

Additionally, traditional programming techniques to organize data involve developing long list of rules (such as the key word search operation) that require hand-tuning by engineers. Thus, when you are dealing with fluctuating environments with changing data, traditional programming techniques quickly become inefficient and time-consuming to maintain. NLP techniques can adapt to fluctuating data, simplify code and perform better.