Word Vectors and Word Meaning
By: Kourosh Alizadeh
Tags: Word Vectors, Meaning, Word Embeddings, NLP, Theory

Word Vectors and Word Meanings

What does meaning even mean, you know? Photo by Rostyslav Savchyn on Unsplash

This article aims to answer a simple question: do word vectors actually capture the meaning of words?

The answer is… it depends. In particular, it depends on what you think a meaning is. But I’m getting ahead of myself. First, let’s quickly break down what word vectors are. Then we can talk about the theory of meaning that suits them best.

WHAT IS A WORD VECTOR?

A word vector is an attempt to mathematically represent the meaning of a word. In essence, a computer goes through some text (ideally a lot of text) and calculates how often words show up next to each other. These frequencies are represented with numbers. So if the word ‘good’ always shows up next to the word ‘friend’ then part of the word vector for ‘good’ will reflect that connection. A given word will have a vast number of such values, usually in the hundreds and sometimes in the thousands.

Once you have this set of numbers for a word, you can compare the vectors of different words. For example, you could compare the different vectors for the words ‘banana,’ ‘kiwi,’ and ‘raincloud.’ Since the vectors are mathematical objects, you can calculate the numerical similarity of different vectors. And since ‘banana’ and ‘kiwi’ are used in a lot of similar contexts, and are not used in the contexts where ‘raincloud’ shows up, their vectors will be closer to each other and farther from the vector for ‘raincloud.’

From here, one can make an intuitive leap and say that the vector represents the meaning of the word ‘banana.’ After all, the vector can produce analogies (e.g. it can tell you that a banana is to a kiwi as a potato is to a carrot), it can categorize the banana into the right category (e.g. tell you that banana is closer to fruit than to meat or to vegetable), and it can even be used to flag correct and incorrect uses of ‘banana’ in sentences.

But can a set of numbers really be all there is to the meaning of a word? In other words…

IS A WORD VECTOR A MEANING?

At the level of the actual code used to generate word vectors, what we have been calling a word is simply a string — a series of characters with no particular representational content except to us as humans. The goal of word vectors is to model the representational content we humans have assigned to that string so that it can become a word not just for us but for the computer too. In other words, we are trying to teach the computer what the word ‘banana’ means.

Now, there are any number of different theories of meaning. Here, I want to focus on a specific one that can be used to support the idea that word vectors do in fact represent meanings. This is a view advanced by the philosopher Ludwig Wittgenstein. In his book Philosophical Investigations, Wittgenstein proposes that we should understand the meaning of a word as being nothing more than how that word is used. We can call this the ‘meaning-is-use thesis.’

The face of a man with his own theory of meaning. Taken from Wikimedia Commons.

This might sound kind of strange, or else just blatantly obvious, so let’s take a moment to compare it to other ideas so we can see why Wittgenstein’s conception was a bit of a breakthrough. Before you read this, if someone asked you what the meaning of a word was, what would you say? Maybe you’d say it’s whatever is in the dictionary. Or maybe you’d say it’s what the word stands for. What does ‘banana’ mean? It means that literal physical object — the banana.

These perspectives have some serious issues. After all, the dictionary doesn’t create meaning, it just writes down what words meant already; people were meaning stuff with their words long before the first dictionary. And if the word’s meaning is the thing it stands for, then how does some random set of sounds come to stand for anything at all? Why does ‘banana’ mean the physical banana, while ‘cake’ means a cake? There’s nothing special about the sounds that should make them point one way or the other.

Wittgenstein’s idea of meaning as use dodges both of these problems. Meaning isn’t some ethereal link between a sound and an object, nor is it some grand plan handed down by Webster’s dictionary; it’s just a matter of social convention and everyday use. If you ask me for a coffee by saying ‘bingle bangle bongle,’ and then I get you a coffee, well, then ‘bingle bangle bongle’ means ‘please get me a coffee.’ That’s all there is to it.

MEANING-IS-USE AND WORD VECTORS

Perhaps you’ve got some inkling now as to why this theory of meaning is more friendly to word vectors than the others. If meaning was about some magic connection between sounds and objects, then the word vector would have no hope of ever capturing it. No matter how many texts you scan, you’ll never find that link.

Word vectors can, however, tell you exactly how a word is used. In fact, if a computer had read all the text and heard all the words that you’ve heard in your life, it would probably a much more precise and clear representation of how those words were used than you do. If all there is to the meaning of a word is how it is used in a sentence, and word vectors can capture that, then word vectors do in fact capture the meaning of words.

Of course, Wittgenstein’s theory has its opponents. And maybe it doesn’t really matter if a word vector captures the meaning of a word; what matters, you might say, is just whether we can get the computer to use the word correctly or to make accurate predictions.

But if you’re interested in whether work on word vectors (or NLP in general) actually does more than produce some functioning product, then it’s worth taking a moment to consider what a meaning is. Thanks for taking the time to do so with me here.