Conversational A.I. Needs Meaning, Not Keywords: Part 1
By: John Ball
Tags: Word Embeddings, Machine Learning, Linguistics, Nlu, NLP

Understanding is deeper than just word matching

As a cognitive scientist working on human-level conversational A.I., I often ask people why they use parts-of-speech in their model of language. I mean, it duplicates definitions, is a part of the parsing concept that has never been accurately implemented for any human language, and excludes meaning. But it is relentlessly taught to students instead of the meaning-based model.

Similarly, I ask people why they use word embeddings in their conversational systems when it takes something that we have exquisitely detailed knowledge of, a word, and converts it into a meaningless number in one or more dimensions. Worse, the numbers relate to the word’s properties of collocation with other words, not to anything meaningful. It’s not accurate either, because languages don’t work by collocation alone — phrases are key. And it changes depending on how the statistics are gathered and from where. Do we really need to use out-of-context words in conversation?

In science, when a model doesn’t work, it is often tinkered with if there is no known alternative. In the case of the solar system, circular planetary orbits were improved with planetary orbits moving in epicycles, which were improved further with planets following multiple epicycles orbiting the earth. Sadly, no matter how many were added, the model didn’t accurately predict planetary motion because the model was fundamentally wrong.

In the world of conversational A.I., all platforms today seem enamored with intents. Intents allow a developer to map known strings of text to intents: the so-called “intent classification.”

Why use intents? It comes back to word embeddings. We will see the parallels of the solar-system model behind the use of word embeddings for conversation. It is fundamentally the wrong way to manage a conversation for a number of reasons.

Welcome to the world of chatbots.

And welcome to the revolutionary world of conversational A.I. where it’s claimed that “A.I. and sophisticated natural language processing” is in play (in the Figure 1 article)!

Figure 1. From www.consumersadvocate.org/chatbots. Note how the interactions are now “deeper than ever”, presumably due to deep learning. But today’s chatbots don’t offer human-like interactions because they don’t understand.

This series will explore the science and engineering behind the current, disappointing, conversational A.I. and how working without meaning results in a lack of generalization and a lack of understanding.

Introduction

The media reporting around artificial neural networks continues to back the idea that all manner of human-imitative problems are being solved by the “deep learning” breakthrough. But most of the scientists behind that technology acknowledge the severe limitations of it, such as Turing award winner Yoshua Bengio:

“what is missing from current machine learning are understanding and generalizations beyond the training distribution[i]”

This is a severe limitation because human languages center around communications: discourse encodes words (meaning/semantics) in phrases (syntax) to convey unambiguously (in context).

The best model of semantics, syntax and context comes from Role and Reference Grammar (RRG), a linguistic model that was first to explain the world’s languages in a way that even computers can understand. My company, Pat Inc. (Pat), uses my brain theory (Patom theory) to eliminate the combinatorial explosion caused by the parsing model. Sets and lists alone, with phrase template matching, are sufficient to understand human language. Patom theory resolves the combinatorial explosion from computational linguistics with different methods: converting rules to sets, decomposing everything possible, and resolving meaning separately to consolidating phrases. It all follows from the brain theory.

The combination of linguistics and computer science leads to what has been known as Natural Language Understanding (NLU), but which has been stolen by marketing to mean: “I think you meant this, maybe.” Understanding is radically different to today’s claims of NLU by state-of-the-art systems.

NLU is intended to support NLP (Natural Language Processing) by providing meaning, but sadly the concept of NLU is being used to describe “understanding” without understanding the meaning of words. This splits NLU into two concepts: (a) keyword-NLU which ‘understands’ phrases without determining their word’s meaning and (b) meaning-NLU, which does.

Again, my brain theory expects common experiences to be decomposed, like words, meaning words are typically composed of multiple meanings, each of which is to be validated.

Keyword-NLU (state-of-the-art)

In keyword-NLU, finding relevant keywords or words that are statistically similar is good enough, even when a person would strongly disagree with the classification of the sentence.

Meaning-NLU (Pat’s model)

In meaning-NLU, the meaning of the words must be recognized, including validation from predicates. In meaning-NLU systems (like Pat’s system), “the meanings of a word in a sentence is determined by the meanings of the other words in a sentence.”

Let’s explore the state-of-the-art used in “Conversational A.I.” which uses keyword-NLU and contrast its limitations with meaning-NLU. It’s not that keyword-NLU doesn’t have an application. It does. It’s just that it cannot scale from simple chatbot to conversations, because it asks the developer to determine each and every intent in advance for all possible conversations. It leaves the problem of A.I. to the developer, while depriving the developer of meaning.

The Science — “good enough” isn’t good enough

In science, near-enough usually isn’t good enough. Getting the wrong answers is unacceptable in almost every science. To deal with ineffective theory in NLP, we need to refocus our aim at the real target — conversing with machines with speed and accuracy. It starts with meaning, not keywords and should be our primary interface with all devices, regardless of your source language.

When digital computers were first built, they were expected to get the right answers. If 1+1=3 were produced regularly, we would not have gone to the moon because computers could not have controlled the spacecraft. We would not have iPhones either. By contrast, the frustrations in getting NLP to work has led to an acceptance that errors are acceptable. They aren’t.

The goal of NLP is to produce systems that are 100% accurate, not something else. Granted, noisy environments cause clarifying questions. New or unknown words can cause that too, as can ambiguity. But A.I. benchmarking tests continue to be built that the state-of-the-art technology cannot pass. Worse still, the best tools are considered a success when they cannot be used commercially due to inaccuracy.

The General Language Understanding Evaluation (GLUE)[ii] benchmark consolidates a number of different tests. The related paper explains: “The human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data.”

But in the words of the organizers: “… the low absolute performance of our best model indicates the need for improved general NLU systems”

For NLU systems to progress, tests like those we propose at Pat are needed, in addition. In a future article, I will explain how Pat provides a range of tests from simple to advanced in which working NLU systems pass the easiest tests with 100% accuracy — keyword-NLU systems can’t. Human children should pass those tests also.

Facebook A.I. Research: bAbI tasks

In our work on the Facebook A.I. Research team’s (bAbI) tasks, our system scored 100% and found errors in the training datasets — a blind spot for the machine learning methodology. It also answered the system in English, not with artificial keywords that allow tests to pass without a English solution (does that make any sense for an NLP test?).

This testing model, ensuring the system is validated on easy examples before testing more complex examples, is the approach we advocate in order to provide human-level NLU in the future across multiple languages.

If companies release tools that can’t get 100% correct on simple tests, like bAbI what does that say about the platform?

Along the same lines, at Stanford near my office in Palo Alto, their SQuAD[iii] tests seem to get results of up to 90%. You’d think that’s OK, but it means the systems are inaccurate. Sure, the tests are complex, but they are tailored for the contestants to be able to pass without understanding.

Simply getting 90% accuracy without understanding is aiming at the wrong target.

What we are seeing is the difference between search technology (that points humans at web pages they may want) and conversation (where the user gets an answer). Conversation is more useful as it stops humans needing to select the correct documents, and it should exclude bias. It requires a very different approach.

To build useful conversational systems, they must pass a large array of tests with perfection. Anything less becomes intelligence augmentation, not human-imitative A.I. Intelligence augmentation should be checked by humans. That’s OK, but the revolution in NLU only comes with understanding like a human, at least some of the time.

Setting the Scene: a brief history

NLP with meaning-NLU has long been considered the Holy Grail in Silicon Valley. It will be the last device interface we ever use because it uses the fastest method people use to communicate. No more keyboards or keywords — not even glowing holograms floating in midair for actors to press in sci-fi movies. Words are more than enough to do the job rapidly and accurately.

But NLP has always been a failure. It started with Noam Chomsky’s linguistic revolution in which he proposed a model based only on syntax. I call that syntax-first. Lack of success with the related rules-based models were replaced by better statistical approach that, in turn, morphed into a connectionist approach with artificial neural networks which is better still. Fundamentally, however, these approaches to parsing trace back to Chomsky’s original 1957 model. Lack of sucess generally has also led to the use of word embeddings — with scientific justification that J.R.Firth advocated it. But he didn’t, if his writings are to be believed.

Aiming at the right target, trying things, failing and iterating to find the myriad of ways not to do NLP is the key to assembling a working NLU solution.

The science of NLP has failed because of Einstein’s observation: “Everything should be made as simple as possible, but not simpler.”

Syntax-first (like parsing) excludes meaning and context. It is too simple.

Distributional semantics (out-of-context-only?) excludes meaning and syntax. It is also too simple.

Pat’s model based on RRG and Patom theory is just right, like Baby bear’s bed in “Goldilocks.” This system is the minimum needed to realize the goals of voice-first[iv] interaction or its next iteration, voice-only.

The goal is captured clearly in the article: “We’re finally able to communicate with our devices — phones, computers, wearable devices, smart speakers and more — the same way we talk to one another when we need to get things done.” Sadly, the state-of-the-art can’t deliver on this, but the sentiment remains.

Using Meaning instead of keywords

Patom theory models a brain with bidirectional elements. The representation remains in the sensory area while its object form connects back to its sensory pieces. With languages, the sounds of words is therefore auditory recognition, but the thing the word refers to is in its own applicable area — vision (the occipital lobe) for color words, the temporal lobe for objects and the frontal lobe for actions. Connecting different words to the same meaning creates synonyms, recognized independently, with perhaps some additional qualification. Isn’t “whisper” the same as “speak” with a particular manner? It’s bidirectional: words connect to meaning, and meaning connects to words.

Machine-readable Meaning (RRG layers)

RRG provides a language-independent model for a sentence: a layered model. A Semantic Set maps the words in a sentence to such a representation (shortened here) for the example: “The cat ate the rat continuously slowly on the mat today evidently because it was hungry.”

Figure 2. A semantic set showing the layered RRG model — nucleus, core and clause level.

Machine-readable Meaning (RRG juncture)

Perhaps the second most intriguing feature of RRG is its treatment of junctures. How are arguments from the start of a sentence mapped to the other side of the juncture?

In the sentence “John promised the cat to eat the rat” broken down below, it covers two lines (note that the juncture shows that ‘John promised’ covers the second phrase with the shared argument, ‘John’).

Figure 3. A semantic set with an RRG juncture.

By providing the machine-readable meaning of the input, a developer now has the start of the tools needed to perform conversational A.I. In addition to the meaning of the sentence, the tracking of context is also important as languages allow ambiguous sentences that are unambiguous in context. If a brain can easily determine someone’s meaning, there is no need to simplify an element further.

End of Part 1

In the upcoming articles, we will see how today’s conversational A.I. platforms inhibit the use of conversation with their design principles — embracing the meaningless distributional semantics instead of focusing on meaning. We will also go into the detail of the solution: how semantic sets such as those above are arrived at, getting the results of parsing without the uncontrolled combinatorial explosion caused by excluding meaning.

While today’s platforms provide keyword-NLU, the minimum solution for conversation is meaning-NLU. Let’s begin the journey.

[i] https://syncedreview.com/2019/04/16/bengio-and-marcus-at-world-ai-summit-in-montreal/ Synced, April 16, 2019.

[ii] https://gluebenchmark.com/ GLUE benchmark.

[iii] https://rajpurkar.github.io/SQuAD-explorer/ The Stanford Question Answering Dataset, version 2.0.

[iv] https://www.forbes.com/sites/ilkerkoksal/2018/02/01/voice-first-devices-are-the-next-big-thing-heres-why/#56a189de6873 Voice-First, Feb 2018.