# Word2Mat: A new word embedding

Seroul Alan

► To cite this version:

Seroul Alan. Word2Mat: A new word embedding. 2024. hal-04534460v2

HAL Id: hal-04534460

https://hal.science/hal-04534460v2

Preprint submitted on 15 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution - NonCommercial - ShareAlike 4.0 International
License
<figure><img src="image_1.png" /></figure>

HAL
open science

# Word2Mat: A new word embedding

Seroul Alan

► To cite this version:

Seroul Alan. Word2Mat: A new word embedding. 2024. hal-04534460v3

HAL Id: hal-04534460

https://hal.science/hal-04534460v3

Preprint submitted on 1 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
Word2Mat

A new word embedding

SEROUL Alan

<figure><img src="image_2.png" /></figure>

Institut Polytechnique de Paris

Université de Paris-Saclay

**Abstract**

How to represent words numerically? The question seems to have been quickly etched in marble as: 'a word is a point in a high-dimensional vector space' (e.g. vectors). But what if words were not vectors, but rather matrices...

# 1. Introduction

Current Large Language Models (LLMs) demonstrate formidable efficiency in the field of Natural Language Processing (NLP), unfortunately, their complex structure lacks interpretability.

In 2013, T. Mikolov's team developed Word2Vec, which has since become authoritative in word embedding. Their methodology enabled them to bring similar words closer with respect to cosine similarity, moreover, their incredible additive capability allow for intuitive operations such as: 'student' - 'homework' + 'nap' ≈ 'slacker'. This is a considerable asset for language understanding.

However, it seems intuitively that this approach cannot easily embed words like: 'very', 'few', 'not', etc. Indeed, these words do not appear to be fixed states but rather transformations applied to neighboring words. It is precisely to address this issue that considering matrix embedding seems natural.

The phrases: 'Wasn't it the appointment that you called for?' and 'Wasn't it the appointment that you called off?', although very similar, have a drastically different meaning in the last word. Viewed as linear transformations, matrices are no longer merely static arrays, but can completely alter the meaning of a sentence. Therefore, the idea is to replace each word with a matrix and then perform matrix multiplication.

Intuitively, here are some matrix associations

that we could understand<fnref n="1" />:

- *very*: $2Id$ as it exaggerates what is said regardless of the topic

- *not*: $-Id$ as double negation is supposed to give the original meaning

- *few*: $-\frac{Id}{2}$ the norm here is less than 1 and the direction is changed, therefore it diminishes the effect

- *very few*: $2Id(-\frac{Id}{2}) = -Id$ we indeed find an equivalence to 'not'

- *very ... very few*: $-2^{n-1}Id$ the more we add, the more negative it becomes

Another clue that leans towards matrix embedding is the non-commutativity of the product. Indeed, in a sentence, words are not interchangeable. Not only can they have different meanings, but those could even be antithetic. As an example, the sentences: 'He was on time, his professor was not disappointed.' and 'He was not on time, his professor was disappointed.' have opposite meanings.

The use of matrices offers numerous advantages. Firstly, they allow for very rapid calculation of sentence (or of any group of words) embedding simply using the matrix product. Additionally, they come with an arsenal of mathematical tools to analyse them: matrix product, cosine similarities, dot products, norms, etc.

# 2. Choosing the architecture

## 2.1. First ideas

Let N be the size of the dictionary and M be the size of the matrices considered. We need to train our network to recognize groups of words

<footnote n="1">¹Of course, these are only illustrative examples intended to motivate matrix embedding</footnote>
Word2Mat

A New Approach to Word Embedding

Alan Tuning

0x123B96D

Abstract

How to represent words numerically? The ques-
tion seems to have been quickly etched in marble as:
'a word is a point in a high-dimensional vector space'
(e.g. vectors). But what if words were not vectors,
but rather matrices...

1. Introduction

Current Large Language Models (LLMs)
demonstrate formidable efficiency in the field
of Natural Language Processing (NLP), un-
fortunately, their complex structure lacks
interpretability.

In 2013, T. Mikolov's team developed
Word2Vec, which has since become authoritative
in word embedding. Their methodology enabled
them to bring similar words closer with respect
to cosine similarity, moreover, their incredible
additive capability allow for intuitive operations
such as: 'student' - 'homework' + 'nap' ≈
'slacker'. This is a considerable asset for lan-
guage understanding.

However, it seems intuitively that this ap-
proach cannot easily embed words like: 'very',
'few', 'not', etc. Indeed, these words do not
appear to be fixed states but rather transfor-
mations applied to neighboring words. It is
precisely to address this issue that considering
matrix embedding seems natural.

The phrases: 'Wasn't it the appointment that you called for?' and 'Wasn't it the appointment that you called off?', although very similar, have a drastically different meaning in the last word. Viewed as linear transformations, matrices are no longer merely static arrays, but can completely alter the meaning of a sentence. Therefore, the idea is to replace each word with a matrix and then perform matrix multiplication.

Intuitively, here are some matrix associations

that we could understand<fnref n="1" /> :

- very: $2Id$ as it exaggerates what is said regardless of the topic

- not: $-Id$ as double negation is supposed to give the original meaning

- few: $-\frac{Id}{2}$ the norm here is less than 1 and the direction is changed, therefore it diminishes the effect

- very few: $2Id(-\frac{Id}{2}) = -Id$ we indeed find an equivalence to 'not'

- very ... very few: $-2^{n-1}Id$ the more we add, the more negative it becomes

Another clue that leans towards matrix
embedding is the non-commutativity of the
product. Indeed, in a sentence, words are not in-
terchangeable. Not only can they have different
meanings, but those could even be antithetic.
As an example, the sentences: 'He was on time,
his professor was not disappointed.' and 'He
was not on time, his professor was disappointed.'
have opposite meanings.

The use of matrices offers numerous advan-
tages. Firstly, they allow for very rapid calcula-
tion of sentence (or of any group of words) em-
bedding simply using the matrix product. Ad-
ditionally, they come with an arsenal of mathe-
matical tools to analyse them: matrix product,
cosine similarities, dot products, norms, etc.

2. Choosing the architecture

2.1. First ideas

Let N be the size of the dictionary and M be
the size of the matrices considered. We need to
train our network to recognize groups of words

<footnote n="1">¹Of course, these are only illustrative examples intended to motivate matrix embedding</footnote>
with similar meanings. The most natural idea is to enforce equality between a word and its definition. The problem with this approach is that there are one or a few definitions per word. This leads us to $NM^2$ variables for approximately $5NM^2$ constraints (assuming an average of 5 definitions per word). This is clearly insufficient to statistically place the matrices in our space as we need more constraints.

A second idea would be to use an auto-encoder architecture based on matrix product. The encoder would perform the matrix product, and the decoder should recover the original matrices, thus forcing the result of the matrix product to encapsulate the true meaning of the initial sentence. Although this approach only requires unlabeled text corpora, it is in practice very challenging. The decoder have to learn a mathematical task for which there is no known algorithm: decomposing a matrix into the product of other arbitrary matrices. Furthermore, there is no guarantee of the uniqueness of the factorization and as beautiful as it may sound, we did not manage to make it work.

## 2.2. ParaNet

Another approach would be to train our network to minimize the distance between a sentence and a paraphrase. Datasets offering such synonymous sentences are not easy to find since that is not a task we frequently do. Therefore, we came up with the idea of training our network to make a sentence similar to its translation. Translation corpora are much more abundant, which seems to provide access to much more information than only definitions. However, learning from multiple languages implies having a lot more parameters to learn.

One trick is to retranslate the text into the original language, hoping that the original sentence and the retranslation are not entirely identical. This is exactly what the authors of the paper para-nmt-50m did. The dataset they propose contains 50 million (5 million in its reduced form) synonymous sentences based on this principle. By translating from English to Czech and vice versa, the obtained sentences often differ by a little since the translation between these two languages is rather approximate. Therefore, we will train our network to minimize the distance between a sentence and its paraphrase.

A trivial solution would be for all matrices to tend towards the identity matrix, which we obviously want to avoid. Thus, the question arises of how to ensure a certain degree of diversity. Since negative sampling in Word2Vec<fnref n="2" /> (W2V) demonstrates the best performance, hence we choose this alternative. Negative sampling involves adding counterexamples for which the network have to discriminate whether the sentence is synonymous or has nothing to do with it.

To compare the two sentences, they are passed as parameters to the network, and a boolean value determines whether it is a true sample or a false one.

We do not want this choice to be made arbitrarily, so applying a dense network to the product of the input matrices is out of the question. A more rigorous approach is to calculate the distance between the two products and then decide whether the product is too large to classify it as false. The question then arises of which norm to use, and this requires a small mathematical detour to understand the problem properly.

## 3. A small bit on exponential of matrices

In the rest of this paper matrices will be written in uppercase, and vectors will use lowercase. This more theoretical part can be skipped by readers who are not so fond of theoretical mathematics. The main takeaway is that we operate in the multiplicative world because it is more natural for dealing with matrix products. The distance used will be $D(A, B) = ||\ln(A^T B)||_F$.

### 3.1. The additive space

There are many distances on matrices, one of the most used is probably the 2-norm, which consists of taking the maximum of the absolute values of the matrix spectrum ($||M||_2 = ||Spec(M)||_\infty$ when $M$ is symmetric). Although very useful in mathematics, it does not offer an associated inner product and consequently is not the most natural norm to use. The Frobenius norm, on the other hand, derives from an inner product, consisting of taking the 2-norm of the spectrum ($||M||_F = ||Spec(M)||_2$) and can also

<footnote n="2">²see 2nd paper</footnote>
with similar meanings. The most natural idea is to enforce equality between a word and its definition. The problem with this approach is that there are one or a few definitions per word. This leads us to $NM^2$ variables for approximately $5NM^2$ constraints (assuming an average of 5 definitions per word). This is clearly insufficient to statistically place the matrices in our space as we need more constraints.

A second idea would be to use an auto-encoder architecture based on matrix product. The encoder would perform the matrix product, and the decoder should recover the original matrices, thus forcing the result of the matrix product to encapsulate the true meaning of the initial sentence. Although this approach only requires unlabeled text corpora, it is in practice very challenging. The decoder have to learn a mathematical task for which there is no known algorithm: decomposing a matrix into the product of other arbitrary matrices. Furthermore, there is no guarantee of the uniqueness of the factorization and as beautiful as it may sound, we did not manage to make it work.

## 2.2. ParaNet

Another approach would be to train our network to minimize the distance between a sentence and a paraphrase. Datasets offering such synonymous sentences are not easy to find since that is not a task we frequently do. Therefore, we came up with the idea of training our network to make a sentence similar to its translation. Translation corpora are much more abundant, which seems to provide access to much more information than only definitions. However, learning from multiple languages implies having a lot more parameters to learn.

One trick is to retranslate the text into the original language, hoping that the original sentence and the retranslation are not entirely identical. This is exactly what the authors of the paper para-nmt-50m did. The dataset they propose contains 50 million (5 million in its reduced form) synonymous sentences based on this principle. By translating from English to Czech and vice versa, the obtained sentences often differ by a little since the translation between these two languages is rather approximate. Therefore, we will train our network to minimize the distance between a sentence and its paraphrase.

A trivial solution would be for all matrices to tend towards the identity matrix, which we obviously want to avoid. Thus, the question arises of how to ensure a certain degree of diversity. Since negative sampling in Word2Vec<fnref n="2" /> (W2V) demonstrates the best performance, hence we choose this alternative. Negative sampling involves adding counterexamples for which the network have to discriminate whether the sentence is synonymous or has nothing to do with it.

To compare the two sentences, they are passed as parameters to the network, and a boolean value determines whether it is a true sample or a false one.

We do not want this choice to be made arbitrarily, so applying a dense network to the product of the input matrices is out of the question. A more rigorous approach is to calculate the distance between the two products and then decide whether the product is too large to classify it as false. The question then arises of which norm to use, and this requires a small mathematical detour to understand the problem properly.

# 3. A small bit on exponential of matrices

In the rest of this paper matrices will be written in uppercase, and vectors will use lowercase. This more theoretical part can be skipped by readers who are not so fond of theoretical mathematics. The main takeaway is that we operate in the multiplicative world because it is more natural for dealing with matrix products. The distance used will be $D(A, B) = ||\ln(A^T B)||_F$.

## 3.1. The additive space

There are many distances on matrices, one of the most used is probably the 2-norm, which consists of taking the maximum of the absolute values of the matrix spectrum ($||M||_2 = ||Spec(M)||_\infty$ when $M$ is symmetric). Although very useful in mathematics, it does not offer an associated inner product and consequently is not the most natural norm to use. The Frobenius norm, on the other hand, derives from an inner product, consisting of taking the 2-norm of the spectrum ($||M||_F = ||Spec(M)||_2$) and can also

<footnote n="2">²see 2nd paper</footnote>
be computed very easily with the relation:

$$||M||_F^2 = \mathrm{Tr}(M^T M)$$

This avoids calculating the eigenvalues of the matrix, which can be costly when repeated too often. The associated inner product is naturally $\langle A|B \rangle = \mathrm{Tr}(A^T B)$. Furthermore, it is the natural extension of the 2-norm in that if $a=\mathrm{flatten}(A)$ and $b=\mathrm{flatten}(B)$, then:

$$\langle A|B \rangle = a.b$$

This remarkable property -which we will call the flattening property- will be particularly useful to us in what follows.

Intuitively, this inner product between two matrices A and B increases if for each vector of the canonical basis, the image by A is similar to that when the vector is applied to B. Equipped with this inner product, the metric space thus seems ideal both mathematically and computationally. However, we cannot use the Frobenius inner product as is.

## 3.2. Dive into the multiplicative world

W2V is an additive system where embeddings convey their meaning through addition. The neutral element for vector addition is thus the zero vector. Thus, the inner product used to compute cosine similarity between words is also additive as it verifies: $\langle A|0 \rangle = 0 \quad \forall A$.

However, for matrix products our neutral element is the identity matrix. It is easy to see that we want the inner product with the neutral element to be zero ($A|Id$) = 0 $\forall A$. Therefore, we need to find a way to switch into a multiplicative mode. Typically, this is done by exponentiation, which is a bit more technical than usual here as we are dealing with matrices. The matrix exponential operation is less convenient than the exponential on real numbers. Nevertheless, we have the following properties:

1. $e^A = \sum_{n=0}^{\infty} \frac{A^n}{n!}$

2. $e^0 = Id$

3. $\mathrm{det}(e^A) = e^{\mathrm{Tr}(A)}$

4. $\ln(A) = \sum_{n=1}^{\infty} \frac{(-1)^{n-1}}{n} (A-I)^n$ if $||A-Id|| < 1$

5. $\ln(A^{-1}) = -\ln(A) \quad \forall A \in \mathrm{GL}(n, \mathbb{R})$

Matrix exponential will therefore allow us to construct a multiplicative distance. The geodesic distance is defined only for isometries as $D_G(A, B) = ||\ln(A^T B)||_F$.

Since isometries satisfy $A^T = A^{-1}$, it is actually more relevant to take the inverse rather than the transpose. Based on this, we introduce 2 natural distances:

$$D(A, B) = ||\ln(A^{-1}B)||_F$$

and

$$D'(A, B) = ||\ln(AB^{-1})||_F$$

In both cases, we have:

$$ \begin{aligned} D(A, B) &= D(MA, MB) \quad \forall M \in \mathrm{GL}(n, \mathbb{R}) \\ &= D(AM', BM') \quad \forall M' \in \mathrm{GL}(n, \mathbb{R}) \end{aligned} $$

This property is relatively interesting as checking if 2 sentences verify $MAN = MBN$ is equivalent to checking if $A = B$. In other words, if the beginning and the end are the same, we can then directly compute the part that is differing.

It is difficult to theoretically choose between $D$ and $D'$ because they seem to be two sides of the same coin. In the following, all statements given for $D$ will also hold for $D'$.

Additionally, we can define the associated inner product and norm as:

$$(A|B) = \langle \ln(A) | \ln(B) \rangle$$

$$||A|| = \sqrt{\langle \ln(A) | \ln(A) \rangle}$$

They have the expected properties:

$$(A|B^n) = n(A|B)$$

with the particular case that we were waiting for:

$$(A|Id) = 0$$

Furthermore, we have:

$$||A^n|| = |n| \cdot ||A||$$

and

$$||A|| = 0 \Leftrightarrow A = Id$$

$D$ is a "multiplicative distance" in that it satisfies the following three properties:
be computed very easily with the relation:

$$||M||_F^2 = \mathrm{Tr}(M^T M)$$

This avoids calculating the eigenvalues of the matrix, which can be costly when repeated too often. The associated inner product is naturally $\langle A|B \rangle = \mathrm{Tr}(A^T B)$. Furthermore, it is the natural extension of the 2-norm in that if $a=\mathrm{flatten}(A)$ and $b=\mathrm{flatten}(B)$, then:

$$\langle A|B \rangle = a.b$$

This remarkable property -which we will call the flattening property- will be particularly useful to us in what follows.

Intuitively, this inner product between two matrices A and B increases if for each vector of the canonical basis, the image by A is similar to that when the vector is applied to B. Equipped with this inner product, the metric space thus seems ideal both mathematically and computationally. However, we cannot use the Frobenius inner product as is.

## 3.2. Dive into the multiplicative world

W2V is an additive system where embeddings convey their meaning through addition. The neutral element for vector addition is thus the zero vector. Thus, the inner product used to compute cosine similarity between words is also additive as it verifies: $\langle A|0 \rangle = 0 \quad \forall A$.

However, for matrix products our neutral element is the identity matrix. It is easy to see that we want the inner product with the neutral element to be zero ($A|Id$) = 0 $\forall A$. Therefore, we need to find a way to switch into a multiplicative mode. Typically, this is done by exponentiation, which is a bit more technical than usual here as we are dealing with matrices. The matrix exponential operation is less convenient than the exponential on real numbers. Nevertheless, we have the following properties:

1. $e^A = \sum_{n=0}^{\infty} \frac{A^n}{n!}$

2. $e^0 = Id$

3. $\mathrm{det}(e^A) = e^{\mathrm{Tr}(A)}$

4. $\ln(A) = \sum_{n=1}^{\infty} \frac{(-1)^{n-1}}{n} (A-I)^n \text{ if } ||A-Id|| < 1$

5. $\ln(A^{-1}) = -\ln(A) \quad \forall A \in \mathrm{GL}(n, \mathbb{R})$

Matrix exponential will therefore allow us to construct a multiplicative distance. The geodesic distance is defined only for isometries as $D_G(A, B) = ||\ln(A^T B)||_F$.

Since isometries satisfy $A^T = A^{-1}$, it is actually more relevant to take the inverse rather than the transpose. Based on this, we introduce 2 natural distances:

$$D(A, B) = ||\ln(A^{-1}B)||_F$$

and

$$D'(A, B) = ||\ln(AB^{-1})||_F$$

In both cases, we have:

$$ \begin{aligned} D(A, B) &= D(MA, MB) \quad \forall M \in \mathrm{GL}(n, \mathbb{R}) \\ &= D(AM', BM') \quad \forall M' \in \mathrm{GL}(n, \mathbb{R}) \end{aligned} $$

This property is relatively interesting as checking if 2 sentences verify $MAN = MBN$ is equivalent to checking if $A = B$. In other words, if the beginning and the end are the same, we can then directly compute the part that is differing.

It is difficult to theoretically choose between $D$ and $D'$ because they seem to be two sides of the same coin. In the following, all statements given for $D$ will also hold for $D'$.

Additionally, we can define the associated inner product and norm as:

$$(A|B) = \langle \ln(A) | \ln(B) \rangle$$

$$||A|| = \sqrt{\langle \ln(A) | \ln(A) \rangle}$$

They have the expected properties:

$$(A|B^n) = n(A|B)$$

with the particular case that we were waiting for:

$$(A|Id) = 0$$

Furthermore, we have:

$$||A^n|| = |n| \cdot ||A||$$

and

$$||A|| = 0 \Leftrightarrow A = Id$$

$D$ is a "multiplicative distance" in that it satisfies the following three properties:
**Symetry :**

$$D(A, B) = D(B, A)$$

**Separation :**

$$D(A, B) = 0 \Leftrightarrow A = B$$

**Cauchy-Schwarz inequality :**

$$|(A|B)| \le ||A|| \cdot ||B||$$

Cauchy-Schwarz inequality follows directly from the one on the inner product of vectors and from the flattening property.

All of this provides an unusual mathematical framework, but which will prove to be very handy for dealing with iterated products of matrices.

# 4. ParaNet

## 4.1. Training

We train our ParaNet network on the para-nmt-5m data, as it contains only the best data in the sense that the sentences and paraphrases are not identical and at the same time not too semantically distant.

## 4.2. Approximation

The calculation of the distance defined by $D(S, P) = ||\ln(S^{-1}P)||_F$ gives a difficult to compute, unstable, and divergent distance when one of the matrices is close to 0. Therefore, we make a first-order approximation of the logarithm at the identity, assuming that our matrices will not deviate too far from it. The distances are implemented as:

$$d(S, P) = ||Id - S^{-1}P||_F$$

$$d'(S, P) = ||Id - SP^{-1}||_F$$

It is clear that this norm tends to 0 as S tends to P, which is sufficient for our training purposes.

## 4.3. Compute the inverse

Inversion is also costly and unstable. S and P are products of words in the sentence and its paraphrase. Thus, we have:

$$S = S_1S_2...S_n$$

and

$$P = P_1P_2...P_m$$

where $S_i$ and $P_i$ are the matrix embeddings of the words. An alternative is to pass the parameters of our network through an exponential before the product. In this case both equalities could be rewritten as follow:

$$S = e^{S'_1}e^{S'_1}...e^{S'_n}$$

$$P = e^{P'_1}e^{P'_1}...e^{P'_n}$$

**Remark:** Writing it in this form implies that the obtained matrices will be positive definite. One may wonder if this is too restrictive for the embedding of our matrices. However, by using complex parameters (hence not necessarily resulting in positive definite matrices), we did not observed any improvement in the results.

Written in this form, S then easily becomes invertible at the additional cost of an exponential transformation (approximated by a Taylor series):

$$\begin{aligned} S^{-1} &= (e^{S'_1}e^{S'_1}...e^{S'_n})^{-1} \\ &= (e^{S'_n})^{-1}(e^{S'_{n-1}})^{-1}...(e^{S'_1})^{-1} \\ &= e^{-S'_n}e^{-S'_{n-1}}...e^{-S'_1} \end{aligned}$$

Inversion can therefore easily be obtained by performing the matrix product in the inverse direction and taking the opposite of our parameters. Thus, we need to initialize the parameters close to 0, giving exponentials in the vicinity of the identity. We add a clamp function to prevent explosions in values due to the iterated products.

## 4.4. Norm Shaping

Before making a prediction (paraphrase or not), it seems natural to choose a ball of radius $r$ in which if the distance is less than the radius of this ball, then the second sentence is considered sufficiently close to the first and therefore a paraphrase. In fact, the use of distance mainly ensures that two sentences at the same distance from a first one are classified in the same way. This property can actually be preserved by applying an increasing function that distorts the response of our norm.

The latter allows two things: to allow the network to assert its decisions when the distance is very large or very small. This has empirically demonstrated better results. To do this, we allow ourselves to apply a low-degree polynomial to our norm (typically of degree 2/3/4).
**Symetry :**

$$D(A, B) = D(B, A)$$

**Separation :**

$$D(A, B) = 0 \Leftrightarrow A = B$$

**Cauchy-Schwarz inequality :**

$$|(A|B)| \le ||A|| \cdot ||B||$$

Cauchy-Schwarz inequality follows directly from the one on the inner product of vectors and from the flattening property.

All of this provides an unusual mathematical framework, but which will prove to be very handy for dealing with iterated products of matrices.

# 4. ParaNet

## 4.1. Training

We train our ParaNet network on the para-nmt-5m data, as it contains only the best data in the sense that the sentences and paraphrases are not identical and at the same time not too semantically distant.

## 4.2. Approximation

The calculation of the distance defined by $D(S, P) = ||\ln(S^{-1}P)||_F$ gives a difficult to compute, unstable, and divergent distance when one of the matrices is close to 0. Therefore, we make a first-order approximation of the logarithm at the identity, assuming that our matrices will not deviate too far from it. The distances are implemented as:

$$d(S, P) = ||Id - S^{-1}P||_F$$

$$d'(S, P) = ||Id - SP^{-1}||_F$$

It is clear that this norm tends to 0 as S tends to P, which is sufficient for our training purposes.

## 4.3. Compute the inverse

Inversion is also costly and unstable. S and P are products of words in the sentence and its paraphrase. Thus, we have:

$$S = S_1S_2...S_n$$

and

$$P = P_1P_2...P_m$$

where $S_i$ and $P_i$ are the matrix embeddings of the words. An alternative is to pass the parameters of our network through an exponential before the product. In this case both equalities could be rewritten as follow:

$$S = e^{S'_1}e^{S'_1}...e^{S'_n}$$

$$P = e^{P'_1}e^{P'_1}...e^{P'_n}$$

**Remark:** Writing it in this form implies that the obtained matrices will be positive definite. One may wonder if this is too restrictive for the embedding of our matrices. However, by using complex parameters (hence not necessarily resulting in positive definite matrices), we did not observed any improvement in the results.

Written in this form, S then easily becomes invertible at the additional cost of an exponential transformation (approximated by a Taylor series):

$$\begin{aligned} S^{-1} &= (e^{S'_1}e^{S'_1}...e^{S'_n})^{-1} \\ &= (e^{S'_n})^{-1}(e^{S'_{n-1}})^{-1}...(e^{S'_1})^{-1} \\ &= e^{-S'_n}e^{-S'_{n-1}}...e^{-S'_1} \end{aligned}$$

Inversion can therefore easily be obtained by performing the matrix product in the inverse direction and taking the opposite of our parameters. Thus, we need to initialize the parameters close to 0, giving exponentials in the vicinity of the identity. We add a clamp function to prevent explosions in values due to the iterated products.

## 4.4. Norm Shaping

Before making a prediction (paraphrase or not), it seems natural to choose a ball of radius $r$ in which if the distance is less than the radius of this ball, then the second sentence is considered sufficiently close to the first and therefore a paraphrase. In fact, the use of distance mainly ensures that two sentences at the same distance from a first one are classified in the same way. This property can actually be preserved by applying an increasing function that distorts the response of our norm.

The latter allows two things: to allow the network to assert its decisions when the distance is very large or very small. This has empirically demonstrated better results. To do this, we allow ourselves to apply a low-degree polynomial to our norm (typically of degree 2/3/4).
The second advantage of this method is that it can be used to stabilize the gradient. Applying the function $F_s(x) = \ln(x+1)$ greatly reduces the values of the norm of an iterated product, while keeping a reasonable computational cost. Thus, the output of our network is then shaped giving:

$$F_p(F_s(d(S, P))$$

with $F_p(x) = p_3x^3 + p_2x^2 + p_1x - r \quad p_i, r \ge 0$.

# 5. Results

## 5.1. Analogies

We tested our network on the google-analogies dataset, where the task consists of associating the word *D* to given a triplet of words *A*, *B*, *C*, in the sense that: *A* is to *B* what *C* is to *D*. With W2V, the prediction strategy was simple for the three word embeddings *a*, *b*, *c* the predicted vector was made by the additive cross product: *b*+*c*−*a*. The matrix equivalent is obviously not unique due to the non-commutativity of matrices. Therefore, we have the following 6 combinations:

1. $A^{-1}BC$

3. $BA^{-1}C$

5. $BCA^{-1}$

2. $A^{-1}CB$

4. $CA^{-1}B$

6. $CBA^{-1}$

We will therefore select the metric yielding the best results.

Since perfectly guessing the correct word is very challenging and almost never happens, we will use the ball described during training to discriminate when *D* is not sufficiently synonymous with the calculated result. Obviously, this task is much harder for a network trained to discriminate 1 paraphrase among 40 fake ones, indeed the resulting ball actually is indirectly incentivized to be very small in order to classify most of the fake ones correctly. To compensate for the difficulties mismatch, it is necessary to increasingly penalize the networks as *k* gets smaller. The corrected results are given in Table 1.

<table><caption>**Table 1:** Results of the ParaNet network using *d* as the distance</caption><thead><tr><td>Parameters</td><td>Accuracy</td></tr></thead><tbody><tr><td>M = 18, k = 20</td><td>0.92</td></tr><tr><td>M = 18, k = 40</td><td>0.89</td></tr><tr><td>M = 32, k = 20</td><td>0.91</td></tr></tbody></table>

We observe that our network does not scale with neither the dimension nor the quantity of fake data. Conversely, if *k* is too large, it will eventually prevents effective learning.

Using *d'* seems to give slightly better results than using *d*, but the significance of this result is yet to be proven.

The choice of permutation in the multiplication seems to have a limited impact. However, *d* offers better results with the product $BCA^{-1}$, while *d'* unsurprisingly prefers the right inversion of A: $A^{-1}CB$. The Frobenius distance works preferentially with the "centered" combination $BA^{-1}C$, but delivers significantly inferior results (0.44 with $M = 18, k = 20$). This strengthen the assumption that the multiplicative framework is much more relevant. The results found during the analogy test thus seem very convincing.

## 5.2. Similarities

We now need to examine whether the similarities between words are preserved.

<table><caption>**Table 2:** cosine similarity using *d* and the parameters $M=18, k=20$</caption><thead><tr><td>Pair</td><td>cos similarity</td></tr></thead><tbody><tr><td>dollar/currency</td><td>0.44</td></tr><tr><td>dollar/news</td><td>0.10</td></tr><tr><td>good/bad</td><td>0.32</td></tr><tr><td>good/excellent</td><td>0.87</td></tr><tr><td>uncle/bank</td><td>0.10</td></tr><tr><td>giant/bank</td><td>-0.09</td></tr><tr><td>coding/programming</td><td>0.50</td></tr><tr><td>coding/hydraulic</td><td>0.27</td></tr></tbody></table>

One advantage of our method is that the words 'good' and 'bad' are significantly less correlated than 'good' and 'excellent'. As opposed to most of the commonly used embedding methods in the rest of the literature, where words are grouped based on their average context and anonymous pairs often end up very close. It is indeed very easy to replace the adjective 'good' with 'bad' in a sentence while still maintaining a sensible sentence. However, we observe that they are not completely uncorrelated which is once again normal, as although expressing opposite things these two words depict the same underlying concept. Although being antonyms, the words 'big' and 'small' both illustrate the concept of size, it is thus normal for them to be
The second advantage of this method is that it can be used to stabilize the gradient. Applying the function $F_s(x) = \ln(x+1)$ greatly reduces the values of the norm of an iterated product, while keeping a reasonable computational cost. Thus, the output of our network is then shaped giving:

$$F_p(F_s(d(S, P))$$

with $F_p(x) = p_3x^3 + p_2x^2 + p_1x - r \quad p_i, r \ge 0$.

# 5. Results

## 5.1. Analogies

We tested our network on the google-analogies dataset, where the task consists of associating the word *D* to given a triplet of words *A*, *B*, *C*, in the sense that: *A* is to *B* what *C* is to *D*. With W2V, the prediction strategy was simple for the three word embeddings *a*, *b*, *c* the predicted vector was made by the additive cross product: *b*+*c*−*a*. The matrix equivalent is obviously not unique due to the non-commutativity of matrices. Therefore, we have the following 6 combinations:

1. $A^{-1}BC$

3. $BA^{-1}C$

5. $BCA^{-1}$

2. $A^{-1}CB$

4. $CA^{-1}B$

6. $CBA^{-1}$

We will therefore select the metric yielding the best results.

Since perfectly guessing the correct word is very challenging and almost never happens, we will use the ball described during training to discriminate when *D* is not sufficiently synonymous with the calculated result. Obviously, this task is much harder for a network trained to discriminate 1 paraphrase among 40 fake ones, indeed the resulting ball actually is indirectly incentivized to be very small in order to classify most of the fake ones correctly. To compensate for the difficulties mismatch, it is necessary to increasingly penalize the networks as *k* gets smaller. The corrected results are given in Table 1.

<table><caption>**Table 1:** Results of the ParaNet network using *d* as the distance</caption><thead><tr><td>Parameters</td><td>Accuracy</td></tr></thead><tbody><tr><td>M = 18, k = 20</td><td>0.92</td></tr><tr><td>M = 18, k = 40</td><td>0.89</td></tr><tr><td>M = 32, k = 20</td><td>0.91</td></tr></tbody></table>

We observe that our network does not scale with neither the dimension nor the quantity of fake data. Conversely, if *k* is too large, it will eventually prevents effective learning.

Using *d'* seems to give slightly better results than using *d*, but the significance of this result is yet to be proven.

The choice of permutation in the multiplication seems to have a limited impact. However, *d* offers better results with the product $BCA^{-1}$, while *d'* unsurprisingly prefers the right inversion of A: $A^{-1}CB$. The Frobenius distance works preferentially with the "centered" combination $BA^{-1}C$, but delivers significantly inferior results (0.44 with $M = 18, k = 20$). This strengthen the assumption that the multiplicative framework is much more relevant. The results found during the analogy test thus seem very convincing.

## 5.2. Similarities

We now need to examine whether the similarities between words are preserved.

<table><caption>**Table 2:** cosine similarity using *d* and the parameters $M=18, k=20$</caption><thead><tr><td>Pair</td><td>cos similarity</td></tr></thead><tbody><tr><td>dollar/currency</td><td>0.44</td></tr><tr><td>dollar/news</td><td>0.10</td></tr><tr><td>good/bad</td><td>0.32</td></tr><tr><td>good/excellent</td><td>0.87</td></tr><tr><td>uncle/bank</td><td>0.10</td></tr><tr><td>giant/bank</td><td>-0.09</td></tr><tr><td>coding/programming</td><td>0.50</td></tr><tr><td>coding/hydraulic</td><td>0.27</td></tr></tbody></table>

One advantage of our method is that the words 'good' and 'bad' are significantly less correlated than 'good' and 'excellent'. As opposed to most of the commonly used embedding methods in the rest of the literature, where words are grouped based on their average context and anonymous pairs often end up very close. It is indeed very easy to replace the adjective 'good' with 'bad' in a sentence while still maintaining a sensible sentence. However, we observe that they are not completely uncorrelated which is once again normal, as although expressing opposite things these two words depict the same underlying concept. Although being antonyms, the words 'big' and 'small' both illustrate the concept of size, it is thus normal for them to be
somehow correlated.

Although the cosine similarities between most of these examples seem rather correct, we cannot help but notice that the words 'coding' and 'hydraulic' are more correlated<fnref n="3" /> than 'dollar' and 'news'. A correlation of 0.10 for the second pair remains consistent (for comparison, W2V gives 0.11). Our model thus seems to struggle to properly position certain words. However, this effect can be mitigated by increasing the dimension, which strangely had no impact on the result of the analogical test but seems to matter when it comes to the similarity benchmark. For instance, the cosine similarity of this troublesome pair drops to 0.07 when our network has its M parameter set to 32.

Training our network with paraphrases thus seems to be a suitable approach to encode words as matrices. All of this is in fact not necessary, as there is a way to use the work already done for additive vector embedding and convert it into multiplicative matrix embedding.

## 6. Using Word2Vec

To better position the words in our space and achieve better results in similarity tests, it is actually unnecessary to train the weights from scratch...

The Word2Vec weights 'turned' in the right direction would retain the same results as Word2Vec on the similarity test. Since they were trained additively, it will suffice to then exponentiate them.

What interests us here is to preserve the cosine similarity between words. Applying an isometry Q to the W2V vectors precisely guarantees this. We can also allow a scaling parameter α (which will be canceled out during the calculation, as we will see). Simply reshape the obtained vectors into matrices, then exponentiate to move into the multiplicative world. Let $s_i$ be the i-th vector in the phrase given by W2V. We then have:"

$$S = e^{\text{reshape}(\alpha Q s_1)} \dots e^{\text{reshape}(\alpha Q s_n)}$$

An isometry is given by an orthogonal matrix. The problem is that the weights of W2V are given in 300 dimensions. Therefore, we need to

fall back on a dimension that is a square. Thus, our matrix Q to preserve cosine similarity must satisfy $Q^TQ = Id^4$ and be of dimension greater than 300. 18 is therefore the minimal dimension greater than 300 and being a square.

### 6.1. Making semi-orthogonal matrices

To have Q such that $Q^TQ = Id$ we could normalize matrices that deviate from this relationship. However, the set of isometries is not a contiguous set. In dimension 2, rotations are given by matrices of the form:

$$R_{\theta} = \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}$$

But they are not the only isometries. The rest is given by symmetries about the first axis:

$$R'_{\theta} = \begin{pmatrix} -\cos \theta & -\sin \theta \\ -\sin \theta & \cos \theta \end{pmatrix}$$

There is no continuous passage allowing to move from the first class of isometries to the second. Therefore, we must find a way to reach all isometries through continuous transition. One way to obtain semi-orthogonal matrices with good precision through continuous transition is to take the QR decomposition of another matrix (which will have trainable weights).

### 6.2. Similarities property

The embedding of the i-th word in our dictionary $M_i = e^{\text{reshape}(\alpha Q v_i)}$ where $v_i$ is the W2V embedding. We have:

$$ \begin{aligned} (M_i|M_j) &= \langle \ln(e^{\text{reshape}(\alpha Q v_i)}) | \ln(e^{\text{reshape}(\alpha Q v_j)}) \rangle \\ &= \alpha^2 \langle \text{reshape}(Q v_i) | \text{reshape}(Q v_j) \rangle \end{aligned} $$

Once again, thanks to the flattening property, we have:

$$ \begin{aligned} (M_i|M_j) &= \alpha^2 \langle Q v_i | Q v_j \rangle \\ &= \alpha^2 \langle v_i | Q^T Q v_j \rangle \\ &= \alpha^2 \langle v_i | v_j \rangle \end{aligned} $$

<footnote n="3">³It is obvious that the only resemblance between 'hydraulic' and 'coding' lies in the fact that one is made up of 'flow' while the other is made with Stack Overflow.</footnote>

<footnote n="4">⁴such matrices will be called semi-orthogonal</footnote>
somehow correlated.

Although the cosine similarities between most of these examples seem rather correct, we cannot help but notice that the words 'coding' and 'hydraulic' are more correlated<fnref n="3" /> than 'dollar' and 'news'. A correlation of 0.10 for the second pair remains consistent (for comparison, W2V gives 0.11). Our model thus seems to struggle to properly position certain words. However, this effect can be mitigated by increasing the dimension, which strangely had no impact on the result of the analogical test but seems to matter when it comes to the similarity benchmark. For instance, the cosine similarity of this troublesome pair drops to 0.07 when our network has its M parameter set to 32.

Training our network with paraphrases thus seems to be a suitable approach to encode words as matrices. All of this is in fact not necessary, as there is a way to use the work already done for additive vector embedding and convert it into multiplicative matrix embedding.

## 6. Using Word2Vec

To better position the words in our space and thus achieve better results in similarity tests, it is surprisingly not necessary to train the weights from scratch. The W2V vectors rotated in the right direction would retain the same results as W2V on the similarity test. Since they were trained additively, it will suffice to then exponentiate them.

What interests us here is to preserve the cosine similarity between words. Applying an isometry $Q$ to the W2V vectors precisely guarantees this. We can also allow a scaling parameter $\alpha$ which will be canceled out during the calculation, as we will see later. Simply reshape the obtained vectors into matrices, then exponentiate to move into the multiplicative space. Let $s_i$ be the i-th vector in the phrase given by W2V, we then have:

$$S = e^{\text{reshape}(\alpha Q s_1)} \dots e^{\text{reshape}(\alpha Q s_n)}$$

The problem with the previous formula is that the weights of W2V are given in 300 dimensions. Therefore, the isometry $Q$ needs to fall back on a perfect square dimension to allow the reshape operator to deliver square matrices. Thus, to

preserve cosine similarity our matrix $Q$ must satisfy $Q^TQ = Id^4$ and be of dimension greater than 300. The minimal dimension greater than 300 and being at the same time perfect square is 18.

### 6.1. Making semi-orthogonal matrices

To have $Q$ such that $Q^TQ = Id$ we could normalize matrices that deviate from this relationship. However, the set of isometries is not a contiguous set. In dimension 2, rotations are given by matrices of the form:

$$R_{\theta} = \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}$$

But they are not the only isometries. The rest is given by symmetries about the first axis:

$$R'_{\theta} = \begin{pmatrix} -\cos \theta & -\sin \theta \\ -\sin \theta & \cos \theta \end{pmatrix}$$

There is no continuous passage allowing to move from the first class of isometries to the second. To tackle this, we can consider One way to obtain semi-orthogonal matrices with good precision through continuous transition is to take the QR decomposition of another matrix (which will have trainable weights).

### 6.2. Similarities property

The embedding of the i-th word in our dictionary is $M_i = e^{\text{reshape}(\alpha Q v_i)}$, where $v_i$ is the W2V embedding, by definition we have:

$$\begin{aligned} (M_i|M_j) &= \langle \ln(e^{\text{reshape}(\alpha Q v_i)}) | \ln(e^{\text{reshape}(\alpha Q v_j)}) \rangle \\ &= \alpha^2 \langle \text{reshape}(Q v_i) | \text{reshape}(Q v_j) \rangle \end{aligned}$$

Once again, thanks to the flattening property, we have:

$$\begin{aligned} (M_i|M_j) &= \alpha^2 \langle Q v_i | Q v_j \rangle \\ &= \alpha^2 \langle v_i | Q^T Q v_j \rangle \\ &= \alpha^2 \langle v_i | v_j \rangle \end{aligned}$$

<footnote n="3">³It is obvious that the only resemblance between 'hydraulic' and 'coding' lies in the fact that one is made up of 'flow' while the other is made with Stack Overflow.</footnote>

<footnote n="4">⁴such matrices will be called semi-orthogonal</footnote>
Then, for cosine similarity, we get:

$$ 
\begin{aligned}
cos(M_i, M_j) &= \frac{(M_i, M_j)}{||M_i|| ||M_j||} \\
&= \frac{\alpha^2 \langle v_i | v_j \rangle}{\alpha^2 ||v_i||_F ||v_j||_F} \\
&= cos_F(v_i, v_j)
\end{aligned}
$$

The embedding thus created will therefore have the mathematical guarantee of performing identically to W2V on the similarity test! While this test being useless, we cannot guarantee anything on the analogy tasks.

## 6.3. Some tricks

In this approach, some words that carry more action than meaning, such as 'of', 'and', 'really', ..., will be very poorly positioned. These words often appear in the list of most used words. We introduce the variable SKIP (=200) which does not take the W2V embedding, but rather makes these parameters trainable.

It is also possible that some less commonly used words fall into this category. Therefore, we allow our model to deviate slightly from the W2V weights by adding a penalty for deviations. We do not want an L2 penalty, as it does not matter to us if certain matrices end up very far from their base embedding. The most important thing is that few matrices are modified. Therefore, we apply a very strong constraint for small deviations and penalize larger deviations less.

Finally, we add a parameter allowing the model not to adjust these deviations immediately, but rather to let it turn the vectors in the right direction before specifying its embedding.

## 7. Results of OrthoNet

The authors of the W2V paper have shown that increasing the embedding dimension increases the model's capabilities by pushing the size up to 1000. Since we are not actually learning the weights here and starting from the W2V version at 300 dimensions, one can wonder if increasing the dimension to $32^2 = 1024$ will improve the results in this case. We test once again the different values of negative sampling 5 and 10.

Increasing the space in the orthogonal network has a noticeable effect on the results. Indeed, even without adding information about the angles between words, more information is encoded.

With the orthogonal network already pretrained, only a tenth of the data is needed for some parameters to achieve optimal performance, which is a considerable gain.

While for some parameters the learning metric remained catastrophic (close to random prediction), the results were very surprising. Indeed, randomly predicting for a network involves choosing a very small ball size to discriminate the maximum number of samples, which would lead to very poor predictions on the analogy set. However, the fact that the cost function was still minimized in the process somehow taught the network how to make good analogies.

Unfortunately, the best results we managed to achieve were 0.81 on the analogy test (using the distance $d'$ and the parameters: $M = 18, k = 20$).

## 8. Conclusion and further work

The conclusions of this work are as follows. For some mysterious reason, $d'$ seems to offer better results than $d$, subtly in ParaNet, and very significantly in OrthoNet. Therefore, this choice should be preferred for future work.

We have successfully achieved two distinct approaches to find a matrix embedding for words. The first approach, starting from scratch, offers very good results on both analogy and similarity tests for a large enough dimension.

Our second approach allows starting from pre-trained weights, then scaling and rotating them in the best direction for matrix multiplication. This process, preserving cosine similarities between words, then makes it possible to train word vector embeddings with advanced techniques, and then transform these embeddings into matrices. However, this method proves to be less efficient in analogy tests and more temperamental during training (care must be taken with the parameters to avoid divergences during training).
Then, for cosine similarity, we get:

$$ 
\begin{aligned}
cos(M_i, M_j) &= \frac{(M_i, M_j)}{||M_i|| ||M_j||} \\
&= \frac{\alpha^2 \langle v_i | v_j \rangle}{\alpha^2 ||v_i||_F ||v_j||_F} \\
&= cos_F(v_i, v_j)
\end{aligned}
$$

The embedding thus created will therefore have the mathematical guarantee of performing identically to W2V on the similarity test. While testing on the similarity dataset is pointless since it performs identically as proved above, there is no clear relation between the results of W2V on the analogy tasks and the ones from the new embedding.

## 6.3. Some tricks

In this approach, some words that carry more action than meaning, such as 'of', 'and', 'really', etc., will be very poorly positioned. These words often appear in the list of most used words. We introduce the variable SKIP (arbitrarily set to 200) which does not take the W2V embedding, but rather makes these parameters trainable.

It is also possible that some less commonly used words fall into this category. Therefore, we allow our model to deviate slightly from the W2V weights by adding a penalty for deviations. We do not want an L2 penalty, as it does not matter if certain matrices end up very far from their base embedding. The most important thing is that only a few matrices are modified. Therefore, we apply a very strong constraint for small deviations and penalize larger deviations less.

Finally, we add a parameter that allow the model not to adjust for these deviations right at the beginning of the training, but rather to let it rotate the vectors in the right direction before fine-tuning the embeddings.

## 7. Results of OrthoNet

The authors of the W2V paper have shown that increasing the embedding dimension also increases the model's capabilities by pushing the vectors size up to 1000. Since we are not actually learning the weights here and starting from the W2V version at 300 dimensions, one can wonder if increasing the dimension to 32² = 1024 will

improve the results in this case. We once again test for different values of negative sampling ($k$ = 5 or 10).

Increasing the space in the orthogonal network has a noticeable effect on the results. Indeed, even without adding information about the angles between words, more information is encoded. With the orthogonal network already pre-trained, only a tenth of the data is needed for some parameters to achieve optimal performance, which is a considerable gain.

While for some parameters the learning metric remained close to random prediction, nevertheless the results were very surprising. Indeed, randomly predicting for a network involves choosing a very small ball size to discriminate the maximum number of samples, which would lead to very poor predictions on the analogy set. However, the fact that the cost function was still minimized in the process somehow taught the network how to make good analogies.

Unfortunately, the best results we managed to achieve were 0.81 on the analogy test using the distance $d'$ and the parameters $M = 18$ and $k = 20$.

## 8. Conclusion

For unknown reasons, $d'$ seems to offer better results than $d$, marginal in ParaNet, while being very significant in OrthoNet. Therefore, we suggest using $d'$ for future work.

We successfully achieved two distinct approaches to find a matrix embedding for words. The first approach, starting from scratch, offers very good results on both analogy and similarity tests for large enough dimensions.

Our second approach allows to start from pre-trained weights, then scale and rotate them in the best direction for matrix multiplication. This process preserves cosine similarities between words. It is thus possible to train vector embeddings with advanced techniques, and then transform these embeddings into matrices. However, this method proved to be less efficient in analogy tests and less stable during training as special care must be taken with the parameters to avoid divergences during training.
As for Word2Vec, both Word2Mat networks
can be trained on a personal computer given
that you let it a couple of hours to converge.

As suggested earlier, the training data could
be different. With a lot of perseverance and an
in-depth mathematical study on the right net-
work architecture to use, it might be possible to
consider the ideal solution of the auto-encoder.
More realistically, training our network on multi-
language data could allow several things:

* increasing the quantity of data greatly (resulting in higher quality embeddings)

• having a multi-language model

* providing a great finesse in understanding concepts (since synonymous phrases are never rigorously identical from a semantic point of view)

Furthermore, the multiplicative metric developed in this paper could have broader applications than NLP. Iterated matrix products are actually common in many areas of applied mathematics. This metric could therefore prove useful for studying back-propagation in a network (since the derivative of a composition can be expressed as a matrix product).

Moreover, further work will also be necessary
to get a more depth interpretation of the results
obtained. On this note, it is also worth mention-
ing that our final model can itself be 'rotated' by
similarity. Composing all our matrices $M_i$ on the
right by a matrix $P$ and its inverse on the left,
we obtain the embeddings $PM_iP^{-1}$. The system
thus created is then totally equivalent to the ini-
tial one, up to similarity (for example, if we have
the equality $AB=C$, then: $(PAP^{-1})(PBP^{-1}) =$
$PABP^{-1} = PCP^{-1} \Leftrightarrow AB = C$. Thus, it is
possible to easily modify the matrix embeds
either to give them additional properties or for
interpretability purposes.

9. Acknowledgment

- Mathilde Rousseau from the university of Paris-Saclay

- Benjamin Graille from the university of Paris-Saclay

References

- original Word2Vec paper: 'Efficient Estimation of Word Representations in Vector Space' by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

- 2nd version of Word2Vec: 'Distributed Representations of Words and Phrases and their Compositionality' by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean

- ParaNMT-50M: 'Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations' by John Wieting and Kevin Gimpel

- WordSim353: 'Placing Search in Context: The Concept Revisited' by Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman and Eytan Ruppin
Both Word2Mat for Word2Vec can be trained
on a personal computer in a few hours.

9. Further work

As suggested earlier, the training data could
be different. With a lot of perseverance and an
in-depth mathematical study on the right net-
work architecture to use, it might be possible to
consider the ideal solution of the auto-encoder.
More realistically, training our network on multi-
language data could allow to:

* increase the quantity of data greatly, resulting in a higher quality embeddings

• have a multi-language model

* provide a great grasp in concepts understand-
  ing (as synonymous phrases are never rigor-
  ously identical from a semantic point of view)

Furthermore, the multiplicative metric
developed in this paper could have broader
applications than NLP. Iterated matrix products
are actually common in many areas of applied
mathematics. This metric could therefore prove
useful for studying back-propagation in a net-
work since the derivative of a composition can
be expressed as a matrix product.

Further work might also be necessary to get
a more in-depth interpretation of the results ob-
tained. On this note, it is also worth mentioning
that our final model can itself be rotated by
similarity. Composing all our matrices $M_i$ on
the right by a matrix $P$ and its inverse on the
left, we obtain the embeddings $PM_iP^{-1}$. This
system is then totally equivalent to the initial
one, up to similarity (for example, if we have
the equality $AB=C$, then: $(PAP^{-1})(PBP^{-1}) =$
$PABP^{-1} = PCP^{-1} \Leftrightarrow AB = C$. Thus, it is
possible to easily modify the matrix embeddings
either to give them additional properties or for
interpretability purposes.

10. Acknowledgment

- Mathilde Rousseau from the university of Paris-Saclay

- Benjamin Graille from the university of Paris-Saclay

**References**

- original Word2Vec paper: *'Efficient Estimation of Word Representations in Vector Space'* by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

- 2nd version of Word2Vec: *'Distributed Representations of Words and Phrases and their Compositionality'* by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean

- ParaNMT-50M: 'Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations' by John Wieting and Kevin Gimpel

- WordSim353: 'Placing Search in Context: The Concept Revisited' by Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman and Eytan Ruppin