Title: REFLECTION-BASED WORD ATTRIBUTE TRANSFER

Abstract: We propose a word attribute transfer framework based on reflection to obtain a word vector with an inverted target attribute for a given word in a word embedding space. Word embeddings based on Pointwise Mutual Information (PMI) represent such analogic relations as --→ king ---→ man + -----→ woman ≈ ---→ queen. These relations can be used for changing a word's attribute from king to queen by changing its gender. This attribute transfer can be performed by subtracting a difference vector --→ man ------→ woman from --→ king when we have explicit knowledge of the gender of given word king. However, this knowledge cannot be developed for various words and attributes in practice. For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it. In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice. We introduce a framework based on reflection mapping that satisfies this property; queen should be transferred back to king with the same mapping as the transfer from king to queen. Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.

1. INTRODUCTION:

Distributed representation (Hinton et al., 1984) is a kind of data representation that can capture data similarities in a vector space. In natural language processing, various studies have been conducted on word embeddings (Mikolov et al., 2013a; b; Pennington et al., 2014; Peters et al., 2018; Bojanowski et al., 2017) . Word embedding models, such as skip-gram with negative sampling (SGNS) (Mikolov et al., 2013b) or GloVe (Pennington et al., 2014) , capture some analogic relations, such as --→ king ---→ man + -----→ woman ≈ ---→ queen. Previous work offer theoretical explanation based on Pointwise Mutual Information (PMI; Church & Hanks (1990) ) for maintaining the analogic relations in word vectors (Levy & Goldberg, 2014b; Arora et al., 2016; Gittens et al., 2017; Ethayarajh et al., 2019; Allen & Hospedales, 2019) .

These relations can be used for transferring a certain attribute of a word, such as changing king into queen by transferring the gender. This task, which is called word attribute transfer, enables us to rewrite He is a boy as She is a girl. Word attribute transfer is expected to be applicable for natural language inference and data augmentation in natural language processing. The above analogic relations can be used, including adding difference vector -----→ woman---→ man to --→ king to transfer --→ king to ---→ queen. This operation requires the explicit knowledge whether an input word is male or female; we have to add a difference vector to a male word and subtract it from a female word for a gender transfer. We also have to avoid changing words without any gender attributes, such as is and a in the example above. Thus, analogy-based word attribute transfer requires explicit knowledge of word attributes, such as king is male, queen is female, and is has no gender attribute. Developing such knowledge is very difficult for various words and attributes in practice.

In this study, we propose a novel framework based on reflection, which enables word attribute transfer by a single reflection-based mapping for a certain attribute. Reflection in geometry is a mapping that exchanges the locations of two vectors in a Euclidean space by a hyperplane called a mirror, which satisfies the above desired property: working as identity mapping when it is applied twice and when it is applied to vectors on the mirror. We apply this reflection mapping to the problem of word attribute transfer by estimating an appropriate mirror that maps word pairs with a binary target attribute (e.g., male and female) and keeps the other words without that attribute using training data. We also extend this approach by introducing parameterized mirrors, which work as different mirrors based on the given input words, to overcome a limitation using a single fixed mirror to represent complex transfer mappings for different words. Experimental results show that the reflection-based method enables such transfers, achieves comparable performance to analogy-based methods with explicit attribute knowledge, even though our proposed method does not use such knowledge.

The following are the contributions of this paper:

• We propose a novel representation learning framework that obtains a vector with an inverted attribute in embedding space without explicit attribute knowledge of the given word. Let x denote a word and let v x denote its vector representation. Here we assume that v x is learned in advance with an embedding model such as SGNS. In this task, we have two inputs, word x and onehot vector z, representing a certain target attribute, and one output, return word t with the inverted attribute of x for z. Let A denote a set of a triplet (x, t, z), e.g., (man, woman, z gender ) ∈ A. Let N denote a set of words without attribute z, e.g., apple ∈ N , when z represents gender. The purpose of this task is to transfer v x to v t by transfer function f z that inverts attribute z of v x . In other words, output v y should be close to the vector of corresponding target word v t , which is typically the nearest neighbor of v t in the word embedding space.

EQUATION

Note that mapping f z transfers word x if it has target attribute z; otherwise f z works as identity mapping. For instance with z gender , given input word man, gender attribute transfer f z gender (v man ) should result in a vector close to v woman . Given input word apple as x, the results should be v apple .

3. ANALOGY-BASED WORD ATTRIBUTE TRANSFER:

Analogy is a general idea for realizing attribute transfer. Several PMI-based word embedding methods (Mikolov et al., 2013c; Linzen, 2016) tackled to embed words into word embedding space to capture the analogic relations. An embedded vector with SGNS or GloVe captures analogic relations (Levy & Goldberg, 2014a; Mikolov et al., 2013c; Linzen, 2016) . For instance, v queen is near the vector obtained on the right side of Eq. 2. By rearranging Eq. 2, Eq. 3 is obtained:

EQUATION

EQUATION

We can transfer the gender attribute by subtracting difference vector v man -v woman from input word vectors, e.g., v king . The analogy-based transfer function is

EQUATION

where d is a difference vector of the given word pair such as man and woman, M is a set of words having a target attribute, and F is a set of words having an inverse attribute, for example, man ∈ M and woman ∈ F for gender attributes. Eq. 4 indicates that the operation changes depending on whether input word x belongs to M or F. For gender words, we subtract difference vector d if x is male, and add it if x is female. Therefore, we need such explicit knowledge. However, this knowledge cannot be developed for various words and attributes in practice.

4. REFLECTION-BASED WORD ATTRIBUTE TRANSFER:



4.1. IDEALIZED TRANSFER WITHOUT EXPLICIT KNOWLEDGE:

What is an idealized transfer function φ z for the word attribute transfer? The following are the idealized natures of such a transfer function:

EQUATION

EQUATION

where m ∈ M and w ∈ F. This function φ z enables to transfer a word without explicit knowledge. Function φ z transfers v m to v w and v w to v m without such explicit knowledge as m ∈ M and w ∈ F. By combining Eqs. 5 and 6, we obtain the following formula:

EQUATION

and

EQUATION

Hence, the idealized transfer function is a mapping that becomes an identity mapping when we apply it twice for any v. Such a mapping is called involution in geometry. For example, φ : v → -v is one example of an involution. Note that the identity map itself, such as φ : v → v, is excluded from the involution.

4.2. REFLECTION:

A reflection is an involution that reverses the location between two vectors in a Euclidean space through an affine hyperplane (mirror). Reflection is an idealized function because every point returns to its original location when reflection is applied twice:

EQUATION

Given vector v in Euclidean space R n , the formula for the reflection in the mirror is given:

EQUATION

where a ∈ R n is a vector orthogonal to the mirror (normal vector) and c ∈ R n is a point through which the mirror passes. a and c are parameters that determine the mirror.

We apply reflection to the word attribute transfer to invert a specific attribute of an input word without its explicit attribute knowledge. We learn a mirror (hyperplane) in a pre-trained embedding space using training word pairs with a common (binary) attribute z (Fig. 2 ). Here since the mirror is uniquely determined by two parameter vectors, a and c, we estimate a and c from target attribute z using fully connected multi-layer perceptrons:

EQUATION

EQUATION

Transferred vector v y is obtained by inverting attribute z of v x by reflection: 

EQUATION

4.4. PARAMETERIZED MIRRORS:

Reflection with a mirror by Eqs. 11 and 12 assumes a single mirror depending only on z. Previous discussion assumed that there will be pairs sharing a stable attribute such king and queen. However, often gendered words don't come in pairs, and gender is far from a stable attribute. For example, actress may be feminine, but actor is clearly neutral in many cases (Fig. 3 ). Thus, actor isn't as obvious a masculine counterpart as king. In fact, it is known that there are biases in gender words in the embedding space (Zhao et al., 2018; Kaneko & Bollegala, 2019) . This phenomenon can occur not only with the gender attribute, but also with other attributes. With this assumption of a single mirror, the mirror must be a hyperplane that goes through the midpoints for all word vector pairs. However, the vector pairs shown on the left of Fig. 3 cannot be transferred well since the single mirror does not satisfy this constraint due to the bias of the embedding space. To solve this problem, we introduce different mirrors for different words. We propose parameterized mirrors determined by input vector v x in addition to attribute z. The following are the definitions of the mirror parameters:

EQUATION

EQUATION

where [•; •] indicates the vector concatenation in the column. The parameterized mirrors are expected to work flexibly on different words. For instance, as shown in Fig. 3 , suppose we learned the mirror (the blue line) that transfers v hero to v heroine in advance. If input word vector v actor resembles v hero , a mirror that is similar to the one for v hero should be derived and used for the attribute transfer.

4.5. LOSS FUNCTION:

Loss function L is defined:

EQUATION

)

where

1 |A| (xi,ti,zi)∈A (v yi -v ti )

2 is a term that draws target word vector v ti closer to corresponding transferred vector v yi and1 

|N | xj ∈N (v yj -v xj )

2 is a term that prevents words without a target attribute from being moved by transfer function f z . Θ represents the set of all the trainable parameters. The parameters in the proposed model are the MLP weights used to determine mirror hyperplanes via a and c. We iteratively update Θ to minimize L:

EQUATION

)

where t is the number of parameter updates at that time.

5. EXPERIMENT:

We evaluated the performance of the proposed reflection-based word attribute transfer using data with some different attributes.

5.1. EXPERIMENTAL SETUP:

We used three different datasets of word pairs with three binary attributes: Male-Female 1 , Singular-Plural and Capital-Country, shown in Table 1 .These word pairs were collected from analogy test sets (Mikolov et al., 2013a; Gladkova et al., 2016) We had no N val because the tuning was conducted with only |A val |. We used word2vec (Mikolov et al., 2013b) 3 and GloVe (Pennington et al., 2014) 4 as the pre-trained embedding model. The embedded vector dimension is n = 300. 

5.2. EVALUATION METRICS:

We measured the accuracy and stability performances of the word attribute transfer. The accuracy measures how many input words in A test were transferred correctly to the corresponding target words. The stability score measures how many words in N test are not mapped to other words. For example, in a gender transfer, given man, the transfer is regarded as correct if woman is the closest word to the transferred vector; otherwise it is incorrect. Given apple, the transfer is regarded as correct if apple is the closest word to the transferred vector; otherwise its stability is incorrect. Here we used cosine similarity to measure the similarity of output vector v y and target vector v t . The accuracy and stability scores are calculated by the following formula:

EQUATION

EQUATION

EQUATION

)

where V is the vocabulary of the word embedding model.

5.3. METHODS AND CONFIGURATIONS:

In our experiment, we compared our proposed method with the following baseline methods:

REF Reflection-based word attribute transfer with a single mirror. We used a fully connected 2-layer MLP with 300 hidden units and ReLU activations to estimate a and c.

REF+PM Reflection-based word attribute transfer with parameterized mirrors. We used the same MLP as the REF.

MLP Fully connected MLP:

v y = M LP ([v x ; z]

). The highest accuracy models are a 2-layer MLP for Capital-Country and 3-layer MLP for the other datasets.

DIFF Analogy-based word attribute transfer with a difference vector: d = v m -v w , where m and w are in the training data of A. We chose d because it achieved the best accuracy in the validation data of A. We determined whether to add or subtract d to v x based on attribute knowledge (Eq. 4).

DIFF + Analogy-based word attribute transfer with a difference vector regardless of the attribute knowledge. d was obtained in the same way as the DIFF. We added d to v x for any input

x: f z (v x ) = v x + d, ∀v x ∈ R n .

DIFF -Analogy-based word attribute transfer with a difference vector regardless of the attribute knowledge. d was obtained in the same way as the DIFF. We subtracted d from v x for any input

x: f z (v x ) = v x -d, ∀v x ∈ R n .

MEANDIFF Analogy-based word attribute transfer with a mean difference vector d: d = 1 |Atrain| (mi,wi)∈Atrain (v mi -v wi ). We determined whether to add or subtract d to v x based on the attribute knowledge (Eq.4).

MEANDIFF + Analogy-based word attribute transfer with a mean difference vector regardless of the attribute knowledge: f z (v x ) = v x + d, ∀v x ∈ R n . d was obtained in the same way as the MEANDIFF.

MEANDIFF -Analogy-based word attribute transfer with a mean difference vector regardless of the attribute knowledge:

f z (v x ) = v x -d, ∀v x ∈ R n .

d was obtained in the same way as the MEANDIFF.

Based on the tuning, we used the Adam optimizer (Kingma & Ba, 2015) with a learning rate of α = 10 -4 (the other hyperparameters were the same as the original one (Kingma & Ba, 2015)), and a batchsize of 62 for male-female, and 32 for the others. These hyperparameters were identical for the learning-based methods: REF, REF + PM, or MLP. We did not use such regularization methods as dropout (Srivastava et al., 2014) or batch normalization (Ioffe & Szegedy, 2015) because they did not show any improvement in our pilot test.

5.4. ACCURACY AND STABILITY:

Table 2 and 3 shows the transfer accuracy and stability score results. Both experiments using GloVe or word2vec obtained similar results. REF + PM achieved the best accuracy among the methods that did not use explicit attribute knowledge. This means that reflection can be used for word attribute transfers even without attribute knowledge. In the stability evaluation, reflection-based methods (REF, REF + PM) and the analogy-based methods with a mean difference vector (MEANDIFF -, MEANDIFF + ) achieved high stability. In particular, reflection-based transfers achieved outstanding stability scores exceeding 99%. The stability of DIFF + and DIFF -was much lower than the other methods. Although MEANDIFF -and MEANDIFF + achieved high stability, their accuracy results were very low. Interestingly, reflection-based transfer with parameterized mirrors (REF + PM) achieved high performance in both accuracy and stability. For example, the accuracy of REF + PM was 41.67%, and the stability was 99.9% in Male-Female (MF), and the accuracy was 58 % and the stability was 99.40 % in Capital-Country (CC). These results show that the proposed method transfers an input word if it has a target attribute and does not transfer an input word even though it does not use explicit attribute knowledge on the input words. MLP worked poorly both in accuracy and stability. In the antonym (AN), while the transfer accuracy by the proposed method was a bit lower than that by MLP, the stability of the proposed method was 100% and that of MLP was really poor (almost 0%).

We investigated the relation between the size of |N train | and the stability of learning-based methods by conducting an additional experiment by varying |N train | from 0 to 50. The stability scores by MLP did not improve (Table 4 ). On the other hand, REF and REF + PM achieved high stability scores with just |N train | = 4 and maintained the accuracy.

5.5. TRANSFER EXAMPLE:

Table 5 shows examples of a gender transfer at the sentence level, where the attribute transfer was applied to words in sentence X = {x 1 , x 2 , ...}. Here since such words as a and . are not in the vocabulary of the original word embedding model, we omitted them from the inputs. MLP made many wrong transfers on words without gender attributes, e.g., the became By Katie Klingsporn, was became she, when became Doughty Evening Chronicle, and woman became girlfriend. DIFF + can transfer if x is female, e.g., it transferred from woman to man, but it could not transfer grandfather and boy. Similarly, DIFF -failed to transfer from female to male. In addition, since the stability of these methods was low, they erroneously transferred. For example, in DIFF -, the and when became she. REF + PM can selectively transfer words with a gender attribute without using explicit gender information. For example, when woman was given, Ref (v woman ) became man without knowledge that woman is a female word, and when man was given, it became woman. When non-attribute word Table 5 : Transfer results when sentence X = {the, ..., boy} was given. Out-of-vocabulary words a and . were not given as input.

X the woman was married when your grandfather was (a) boy ( .)

Ref (x)

the man was married when your grandmother was (a) girl (.) Ref (Ref (x)) the woman was married when your grandfather was (a) boy (.)

MLP:

By Katie Klingsporn girlfriend she fiancee Doughty Evening Chronicle ma'am daughter she (a) mother (.) DIFF + the man was married when your grandfather was (a) boy (.) DIFF - she woman was married she your grandmother was (a) girl (.)

We can transfer some different attributes of words with reflection-based transfer one-by-one. Table 6 shows that the words having different target attributes were transferred by each reflection-based transfer in the order of Male-Female, Singular-Plural, and Country-Capital. Given actress for a Male-Female transfer, it was transferred to actor and to actors for Singular-Plural. Given Tokyo for Male-Female, Singular-Plural, and Antonym, it was not transferred, but it was transferred to Japan for Country-Capital. Given rich for Male-Female, Singular-Plural, and Capital-Country, it was not transferred, but it was transferred to poor for Antonym.

Table 6 : Transfer of different attributes with reflection-based word attribute transfer with parameterized mirrors.

X the rich actress and the poor actor want to stay the beautiful city in Tokyo.

+ Male-Female the rich actor and the poor actress want to stay the beautiful city in Tokyo.

+ Singular-Plural the rich actors and the poor actresses want to stay the beautiful cities in Tokyo.

+ Capital-Country the rich actors and the poor actresses want to stay the beautiful citie in Japan.

+ Antonym the poor actors and the rich actresses want to stay the beautiful cities in Japan.

6. RELATED WORK:

The embedded vectors obtained by SGNS (Mikolov et al., 2013a; b) and GloVe (Pennington et al., 2014) 2019) argued that they proved the existence of such analogic relations without strong assumptions. In our work, we focus on the analogic relations in a word embedding space and propose a novel framework to obtain a word vector with inverted attributes. Style transfers (Niu et al., 2018; Prabhumoye et al., 2018; Jain et al., 2019; Logeswaran et al., 2018; Dai et al., 2019; Zhang et al., 2018) resemble our task. In a style transfer, the text style of the input sentences is changed. For instance, Jain et al. (2019) transferred from formal to informal sentences. Logeswaran et al. (2018) transferred sentences by controlling such attributes as mood and tense. These style transfer tasks use sentence pairs; our word attribute transfer task uses word pairs. Style transfer changes sentence styles, but our task changes the word attributes (contents). Soricut & Och (2015) studied the problem of morphological transformation based on character information. Our work aims more general attribute transfer such as gender transfer and country-capital and is not limited to the morphological transformation.

7. CONCLUSION AND FUTURE WORK:

We proposed a novel representation learning framework based on reflection to invert a certain attribute of a word vector. We proposed a reflection-based method for word attribute transfers without relying on the explicit attribute knowledge of an input word, which is necessary for a simple analogy-based transfer. Experimental results showed that our proposed method can transfer the word attributes if the input word has a target attribute. If not, reflection does not transfer the word. 2019), and entity prediction on an analogic graph embedding space (Liu et al., 2017) , in the field of computer vision, visual analogy (Reed et al., 2015) , or style transfer (Zhu et al., 2017; Liao et al., 2017) with GANs (Radford et al., 2016; Goodfellow et al., 2014) because their latent space holds analogic relations (Radford et al., 2016) .

A TOP THREE ACCURACY AND STABILITY  and 5 suggest not only the mirror parameters of paired words are similar to each other but also the parameters with the attribute form a cluster -words with the same attribute has similar mirror parameter a. The mirrors of paired words are close to each other in the same attribute (Fig. 4 and 5 ). Some MF pairs in Fig. 5 are placed away from the cluster of the MF words. This may come from missing principal components due to the small data size used for PCA. Figs. 6, 7, 8, 9, 10, 11, 12, and 13 are the detailed PCA results for four different attributes: MF, SP, CC, and AN. These results show that a reflection transfers a paired word each other by using a similar mirror. For example, rich and poor use almost the same mirror (Fig. 9 ). On the other hand, different mirrors are used for different word pairs since the mirrors are parameterized. These results shows the effect of the mirror as described in section 4.4.

.' 41 $$ "/ 

:

Note that these gender word pairs are an assumption because they contain socially problematic words.

These datasets and the experimental codes will be released in the future.

https://code.google.com/archive/p/word2vec/

https://nlp.stanford.edu/projects/glove/

Figures and Tables:

Figure 2: Reflection-based word attribute transfer examples.

Figure 3: Mirror estimation methods

have analogic relations. The theory of analogic relations in word embeddings has been widely discussed: Levy & Goldberg (2014b); Arora et al. (2016); Gittens et al. (2017); Ethayarajh et al. (2019); Allen & Hospedales (2019);Linzen (2016).Levy & Goldberg (2014b)  offer the explanation that SGNS factorizes a shifted PMI matrix.Allen & Hospedales (2019) andEthayarajh et al. (

work includes applications to other transfer tasks: sentence by sentence transfer, such Niu et al. (2018); Prabhumoye et al. (2018); Jain et al. (

Figure 4: A PCA result of a (word2vec).

Figure 6: Male-Female (word2vec)

• Our proposed reflection-based word attribute transfer enables us to transfer word attributes in up to 76% for words with target attributes and to avoid changing words without target attributes in over 99% in our experiments.

Table content: <table><tr><td colspan="2">2 WORD ATTRIBUTE TRANSFER</td></tr><tr><td>x</td><td>Invert gender z</td></tr><tr><td>father</td><td>mother</td></tr><tr><td/><td>fathers</td></tr><tr><td/><td>Invert singular/plural</td></tr></table>

Statistics of binary attribute word pair datasets (in the number of word pairs).

Table content: <table><tr><td>Dataset A</td><td colspan="4">Train Val Test Total</td></tr><tr><td>Male-Female (MF)</td><td>29</td><td>12</td><td>12</td><td>53</td></tr><tr><td>Singular-Plural (SP)</td><td>90</td><td>25</td><td>25</td><td>140</td></tr><tr><td>Capital-Country (CC)</td><td>59</td><td>25</td><td>25</td><td>109</td></tr><tr><td>Antonym (AN)</td><td colspan="4">1354 290 290 1934</td></tr></table>

Results in accuracy and stability scores (word2vec).

Table content: <table><tr><td>Method</td><td>Knowledge</td><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td><td/></tr><tr><td/><td>MF</td><td>SP</td><td>CC</td><td>AN</td><td>MF</td><td>SP</td><td>CC</td><td>AN</td></tr><tr><td>REF</td><td>20.83</td><td>0.00</td><td>36.00</td><td>0.00</td><td colspan="4">99.80 100.00 99.80 100.00</td></tr><tr><td>REF + PM</td><td colspan="4">41.67 22.00 58.00 28.79</td><td>99.90</td><td>99.40</td><td colspan="2">99.40 100.00</td></tr><tr><td>MLP</td><td>8.33</td><td>4.00</td><td colspan="2">12.00 35.86</td><td>2.20</td><td>0.00</td><td>2.70</td><td>1.90</td></tr><tr><td>DIFF +</td><td>25.00</td><td>2.00</td><td>32.00</td><td>-</td><td>72.10</td><td>77.90</td><td>53.90</td><td>-</td></tr><tr><td>DIFF -</td><td>25.00</td><td>2.00</td><td>30.00</td><td>-</td><td>49.60</td><td>78.20</td><td>56.30</td><td>-</td></tr><tr><td>MEANDIFF +</td><td>4.17</td><td>0.00</td><td>22.00</td><td>-</td><td>98.60</td><td>99.40</td><td>87.60</td><td>-</td></tr><tr><td>MEANDIFF -</td><td>8.33</td><td>0.00</td><td>14.00</td><td>-</td><td>97.20</td><td>99.30</td><td>92.40</td><td>-</td></tr><tr><td>DIFF</td><td>62.50</td><td>4.00</td><td>64.00</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MEANDIFF</td><td>12.50</td><td>0.00</td><td>36.00</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

Results in accuracy and stability scores (GloVe).

Table content: <table><tr><td>Method</td><td>Knowledge</td><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td><td/></tr><tr><td/><td>MF</td><td>SP</td><td>CC</td><td>AN</td><td>MF</td><td>SP</td><td>CC</td><td>AN</td></tr><tr><td>REF</td><td>12.50</td><td>2.00</td><td>26.00</td><td>0.00</td><td colspan="4">100.00 100.00 100.00 100.00</td></tr><tr><td>REF + PM</td><td colspan="4">45.83 50.00 76.00 33.54</td><td>99.70</td><td>99.10</td><td>99.20</td><td>100.00</td></tr><tr><td>MLP</td><td>4.17</td><td colspan="3">10.00 18.00 36.72</td><td>5.10</td><td>7.00</td><td>5.20</td><td>1.20</td></tr><tr><td>DIFF +</td><td>25.00</td><td>2.00</td><td>26.00</td><td>-</td><td>99.30</td><td>94.20</td><td>99.30</td><td>-</td></tr><tr><td>DIFF -</td><td>25.00</td><td>2.00</td><td>24.00</td><td>-</td><td>100.60</td><td>99.90</td><td>99.50</td><td>-</td></tr><tr><td>MEANDIFF +</td><td>0.00</td><td>0.00</td><td>22.00</td><td>-</td><td colspan="3">100.00 100.00 100.00</td><td>-</td></tr><tr><td>MEANDIFF -</td><td>0.00</td><td>0.00</td><td>0.00</td><td>-</td><td colspan="3">100.00 100.00 100.00</td><td>-</td></tr><tr><td>DIFF</td><td>50.00</td><td>4.00</td><td>44.00</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MEANDIFF</td><td>0.00</td><td>0.00</td><td>0.00</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

Relation among size of |Ntrain| and stability of learning-based methods. Ref (v married ) became married without knowledge that married has no gender attribute. When we applied the reflection-based transfer twice, the transferred word returned to its original word, e.g., Ref (Ref (v woman )) gives woman.

Table content: <table><tr><td/><td/><td/><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td></tr><tr><td/><td/><td/><td colspan="2">|Ntrain|</td><td/><td/><td colspan="2">|Ntrain|</td></tr><tr><td/><td/><td>0</td><td>4</td><td>10</td><td>50</td><td>0</td><td>4</td><td>10</td><td>50</td></tr><tr><td/><td>REF</td><td colspan="4">16.67 20.83 20.83 20.83</td><td colspan="2">98.30 98.80</td><td>99.40</td><td>99.80</td></tr><tr><td>MF</td><td colspan="5">REF + PM 41.67 45.83 20.83 41.67</td><td colspan="3">38.30 98.40 100.00</td><td>99.90</td></tr><tr><td/><td>MLP</td><td>4.17</td><td>8.33</td><td>8.33</td><td>8.33</td><td>0.00</td><td>0.30</td><td>0.30</td><td>2.20</td></tr><tr><td/><td>REF</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td colspan="2">99.90 99.90</td><td>99.90</td><td>100.90</td></tr><tr><td>SP</td><td colspan="5">REF + PM 12.00 22.00 18.00 18.00</td><td colspan="2">98.40 99.40</td><td>99.30</td><td>99.80</td></tr><tr><td/><td>MLP</td><td>4.00</td><td>4.00</td><td>2.00</td><td>2.00</td><td>0.00</td><td>0.00</td><td>0.10</td><td>3.40</td></tr><tr><td/><td>REF</td><td colspan="4">36.00 36.00 36.00 34.00</td><td colspan="2">99.80 99.80</td><td>99.80</td><td>100.00</td></tr><tr><td>CC</td><td colspan="5">REF + PM 58.00 56.00 58.00 54.00</td><td colspan="2">73.80 99.70</td><td>99.40</td><td>99.40</td></tr><tr><td/><td>MLP</td><td>6.00</td><td>6.00</td><td>8.00</td><td>12.00</td><td>0.00</td><td>0.30</td><td>0.50</td><td>2.70</td></tr><tr><td/><td>REF</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td colspan="2">99.90 99.90</td><td>99.90</td><td>99.80</td></tr><tr><td>AN</td><td colspan="5">REF + PM 21.72 27.24 28.62 28.79</td><td colspan="2">95.30 99.20</td><td>99.50</td><td>99.80</td></tr><tr><td/><td>MLP</td><td colspan="4">34.14 35.00 34.31 35.86</td><td>0.00</td><td>0.01</td><td>0.02</td><td>1.90</td></tr><tr><td colspan="2">married was given,</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Male-Female

Table content: <table><tr><td>Method</td><td>Knowledge</td><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td></tr><tr><td/><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td></tr><tr><td>REF</td><td>50.00</td><td colspan="3">20.83 62.50 66.67</td><td>99.80</td><td colspan="3">99.80 99.80 99.80</td></tr><tr><td>REF + PM</td><td>55.56</td><td colspan="3">41.67 58.33 66.67</td><td>99.13</td><td colspan="3">99.00 99.20 99.20</td></tr><tr><td>MLP</td><td>20.83</td><td>8.33</td><td colspan="2">20.83 33.33</td><td>2.20</td><td>2.20</td><td>2.20</td><td>2.20</td></tr><tr><td>DIFF +</td><td>31.94</td><td colspan="3">25.00 33.33 37.50</td><td>75.43</td><td colspan="3">72.10 75.80 78.40</td></tr><tr><td>DIFF -</td><td>31.94</td><td colspan="3">25.00 33.33 37.50</td><td>55.80</td><td colspan="3">49.60 57.30 60.50</td></tr><tr><td>MEANDIFF +</td><td>23.61</td><td>4.17</td><td colspan="2">33.33 33.33</td><td>98.93</td><td colspan="3">98.60 99.10 99.10</td></tr><tr><td>MEANDIFF -</td><td>23.61</td><td>8.33</td><td colspan="2">29.17 33.33</td><td>97.63</td><td colspan="3">97.20 97.80 97.90</td></tr><tr><td>DIFF</td><td>68.05</td><td colspan="3">62.50 66.66 75.00</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MEANDIFF</td><td>47.22</td><td colspan="3">12.50 62.50 66.67</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

Singular-Plural

Table content: <table><tr><td>Method</td><td>Knowledge</td><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td><td/></tr><tr><td/><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td></tr><tr><td>REF</td><td>50.00</td><td>0.00</td><td colspan="2">72.00 78.00</td><td>100.00</td><td colspan="3">100.00 100.00 100.00</td></tr><tr><td>REF + PM</td><td>56.67</td><td colspan="3">22.00 72.00 76.00</td><td>99.70</td><td>99.40</td><td>99.80</td><td>99.90</td></tr><tr><td>MLP</td><td>7.33</td><td>4.00</td><td>8.00</td><td>10.00</td><td>0.13</td><td>0.00</td><td>0.20</td><td>0.20</td></tr><tr><td>DIFF +</td><td>36.67</td><td>2.00</td><td colspan="2">50.00 58.00</td><td>80.27</td><td>77.90</td><td>79.70</td><td>83.20</td></tr><tr><td>DIFF -</td><td>36.67</td><td>2.00</td><td colspan="2">52.00 56.00</td><td>80.27</td><td>78.20</td><td>80.70</td><td>81.90</td></tr><tr><td>MEANDIFF +</td><td>42.00</td><td>0.00</td><td colspan="2">56.00 70.00</td><td>99.47</td><td>99.40</td><td>99.40</td><td>99.60</td></tr><tr><td>MEANDIFF -</td><td>39.33</td><td>0.00</td><td colspan="2">56.00 62.00</td><td>99.50</td><td>99.30</td><td>99.60</td><td>99.60</td></tr><tr><td>DIFF</td><td>49.33</td><td>4.00</td><td colspan="2">70.00 74.00</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MEANDIFF</td><td>52.00</td><td>0.00</td><td colspan="2">76.00 80.00</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

Capital-Country

Table content: <table><tr><td>Method</td><td>Knowledge</td><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td></tr><tr><td/><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td></tr><tr><td>REF</td><td>66.67</td><td colspan="3">36.00 78.00 86.00</td><td>99.80</td><td colspan="3">99.80 99.80 99.80</td></tr><tr><td>REF + PM</td><td>72.67</td><td colspan="3">58.00 74.00 86.00</td><td>99.53</td><td colspan="3">99.40 99.60 99.60</td></tr><tr><td>MLP</td><td>35.33</td><td colspan="3">12.00 40.00 54.00</td><td>2.80</td><td>2.70</td><td>2.80</td><td>2.90</td></tr><tr><td>DIFF +</td><td>39.33</td><td colspan="3">32.00 42.00 44.00</td><td>64.87</td><td colspan="3">53.90 69.60 71.10</td></tr><tr><td>DIFF -</td><td>34.67</td><td colspan="3">30.00 36.00 38.00</td><td>58.63</td><td colspan="3">56.30 58.90 60.70</td></tr><tr><td>MEANDIFF +</td><td>36.00</td><td colspan="3">22.00 42.00 44.00</td><td>89.33</td><td colspan="3">87.60 89.70 90.70</td></tr><tr><td>MEANDIFF -</td><td>34.67</td><td colspan="3">14.00 44.00 46.00</td><td>93.07</td><td colspan="3">92.40 93.10 93.70</td></tr><tr><td>DIFF</td><td>80.00</td><td colspan="3">64.00 86.00 90.00</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MEANDIFF</td><td>70.67</td><td colspan="3">36.00 86.00 90.00</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></table>

Antonym

Table content: <table><tr><td>Method</td><td>Knowledge</td><td colspan="2">Accuracy (%)</td><td/><td/><td colspan="2">Stability (%)</td></tr><tr><td/><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td><td>Mean@3</td><td>@1</td><td>@2</td><td>@3</td></tr><tr><td>REF</td><td>0.94</td><td>0.00</td><td>1.19</td><td>16.21</td><td>99.97</td><td colspan="3">99.90 100.00 100.00</td></tr><tr><td>REF + PM</td><td>38.56</td><td colspan="3">28.79 41.38 45.52</td><td>99.93</td><td colspan="3">99.80 100.00 100.00</td></tr><tr><td>MLP</td><td>41.38</td><td colspan="3">35.86 42.59 45.69</td><td>1..97</td><td>1.90</td><td>2.00</td><td>2.00</td></tr><tr><td colspan="5">B VISUALIZATION OF PARAMETERIZED MIRRORS</td><td/><td/><td/></tr><tr><td colspan="9">Figures below visualize PCA results of a obtained for the test words. We normalized the L2 norm of a to 1 ( a a ). Corresponding word pairs are connected by solid lines. Figs. 4</td></tr></table>

Peer review (iclr2020):
If I understood correctly, in experimental part pre-trained embeddings, taken from word2vec, is a ground truth. Given those embeddings, a system of hyperplanes are trained (every hyperplane refers to a certain word attribute and a region in space, centered around a word, to which reflection is applied). In my opinion, the most natural way to check *the reflection hypothesis* is to train new embeddings where to word2vec objective the loss function (16) is added and to look how perplexity will behave with that additional cost and to look how your accuracy and stability will behave on the test set. It is also interesting to learn how this reflection based attribute transfer can be applied to the same word, but with embeddings that play different role in a model: e.g. input and output embeddings. In fact, input and output embeddings are located in the same space, they can be considered as pairs of words with 1 attribute flipped (i.e. role in a model, input->output). Can an output embedding be obtained from an input embedding via reflections that you describe. How many hyperplanes we need in that case? This is an interesting edge of the problem. In simplest models, there is some evidence, that output embeddings are result of reflections of input embeddings in half the dimensions. It would be interesting to learn your comment on that.