Year: 2019

Abstract:
We propose a word attribute transfer framework based on reflection to obtain a word vector with an inverted target attribute for a given word in a word embedding space. Word embeddings based on Pointwise Mutual Information (PMI) represent such analogic relations as −−→ king − −−→ man + −−−−−→ woman ≈ −−−→ queen. These relations can be used for changing a word’s attribute from king to queen by changing its gender. This attribute transfer can be performed by subtracting a difference vector −−→ man − −−−−−→ woman from −−→ king when we have explicit knowledge of the gender of given word king. However, this knowledge cannot be developed for various words and attributes in practice. For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it. In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice. We introduce a framework based on reflection mapping that satisfies this property; queen should be transferred back to king with the same mapping as the transfer from king to queen. Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.

text: −−→ king − −−→man + −−−−−→woman ≈ −−−→queen. These relations
can be used for changing a word’s attribute from king to queen by changing its gender. This attribute transfer can be performed by subtracting a difference vector −−→man − −−−−−→woman from −−→ king when we have explicit knowledge of the gender of given word king. However, this knowledge cannot be developed for various words and attributes in practice. For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it. In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice. We introduce a framework based on reflection mapping that satisfies this property; queen should be transferred back to king with the same mapping as the transfer from king to queen. Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.

heading: 1 INTRODUCTION
text: Distributed representation (Hinton et al., 1984) is a kind of data representation that can capture data similarities in a vector space. In natural language processing, various studies have been conducted on word embeddings (Mikolov et al., 2013a;b; Pennington et al., 2014; Peters et al., 2018; Bojanowski et al., 2017). Word embedding models, such as skip-gram with negative sampling (SGNS) (Mikolov et al., 2013b) or GloVe (Pennington et al., 2014), capture some analogic relations, such as
−−→ king −−−→man + −−−−−→woman ≈ −−−→queen. Previous work offer theoretical explanation based on Pointwise Mutual Information (PMI; Church & Hanks (1990)) for maintaining the analogic relations in word vectors (Levy & Goldberg, 2014b; Arora et al., 2016; Gittens et al., 2017; Ethayarajh et al., 2019; Allen & Hospedales, 2019).
These relations can be used for transferring a certain attribute of a word, such as changing king into queen by transferring the gender. This task, which is called word attribute transfer, enables us to rewrite He is a boy as She is a girl. Word attribute transfer is expected to be applicable for natural language inference and data augmentation in natural language processing. The above analogic relations can be used, including adding difference vector−−−−−→woman−−−→man to −−→ king to transfer −−→ king to −−−→queen. This operation requires the explicit knowledge whether an input word is male or female; we have to add a difference vector to a male word and subtract it from a female word for a gender transfer. We also have to avoid changing words without any gender attributes, such as is and a in the example above. Thus, analogy-based word attribute transfer requires explicit knowledge of word attributes, such as king is male, queen is female, and is has no gender attribute. Developing such knowledge is very difficult for various words and attributes in practice.
In this study, we propose a novel framework based on reflection, which enables word attribute transfer by a single reflection-based mapping for a certain attribute. Reflection in geometry is a mapping that exchanges the locations of two vectors in a Euclidean space by a hyperplane called a mirror, which satisfies the above desired property: working as identity mapping when it is applied twice and when it is applied to vectors on the mirror. We apply this reflection mapping to the problem of word attribute transfer by estimating an appropriate mirror that maps word pairs with a binary target
attribute (e.g., male and female) and keeps the other words without that attribute using training data. We also extend this approach by introducing parameterized mirrors, which work as different mirrors based on the given input words, to overcome a limitation using a single fixed mirror to represent complex transfer mappings for different words. Experimental results show that the reflection-based method enables such transfers, achieves comparable performance to analogy-based methods with explicit attribute knowledge, even though our proposed method does not use such knowledge.
The following are the contributions of this paper:
• We propose a novel representation learning framework that obtains a vector with an inverted attribute in embedding space without explicit attribute knowledge of the given word.
• Our proposed reflection-based word attribute transfer enables us to transfer word attributes in up to 76% for words with target attributes and to avoid changing words without target attributes in over 99% in our experiments.
2 WORD ATTRIBUTE TRANSFER
Let x denote a word and let vx denote its vector representation. Here we assume that vx is learned in advance with an embedding model such as SGNS. In this task, we have two inputs, word x and onehot vector z, representing a certain target attribute, and one output, return word t with the inverted attribute of x for z. LetA denote a set of a triplet (x, t, z), e.g., (man,woman, zgender) ∈ A. LetN denote a set of words without attribute z, e.g., apple ∈ N , when z represents gender. The purpose of this task is to transfer vx to vt by transfer function fz that inverts attribute z of vx. In other words, output vy should be close to the vector of corresponding target word vt, which is typically the nearest neighbor of vt in the word embedding space.
vt ≈ vy = fz(vx). (1)
Note that mapping fz transfers word x if it has target attribute z; otherwise fz works as identity mapping. For instance with zgender, given input word man, gender attribute transfer fzgender (vman) should result in a vector close to vwoman. Given input word apple as x, the results should be vapple.

heading: 3 ANALOGY-BASED WORD ATTRIBUTE TRANSFER
text: Analogy is a general idea for realizing attribute transfer. Several PMI-based word embedding methods (Mikolov et al., 2013c; Linzen, 2016) tackled to embed words into word embedding space to capture the analogic relations. An embedded vector with SGNS or GloVe captures analogic relations (Levy & Goldberg, 2014a; Mikolov et al., 2013c; Linzen, 2016). For instance, vqueen is near the vector obtained on the right side of Eq. 2. By rearranging Eq. 2, Eq. 3 is obtained:
vqueen ≈ vking − vman + vwoman, (2)
≈ vking − (vman − vwoman). (3)
We can transfer the gender attribute by subtracting difference vector vman − vwoman from input word vectors, e.g., vking . The analogy-based transfer function is
fz(vx) = { vx − d if x ∈M, vx + d if x ∈ F ,
(4)
where d is a difference vector of the given word pair such as man and woman,M is a set of words having a target attribute, and F is a set of words having an inverse attribute, for example, man ∈M and woman ∈ F for gender attributes. Eq. 4 indicates that the operation changes depending on whether input word x belongs to M or F . For gender words, we subtract difference vector d if x is male, and add it if x is female. Therefore, we need such explicit knowledge. However, this knowledge cannot be developed for various words and attributes in practice.

heading: 4 REFLECTION-BASED WORD ATTRIBUTE TRANSFER

heading: 4.1 IDEALIZED TRANSFER WITHOUT EXPLICIT KNOWLEDGE
text: What is an idealized transfer function φz for the word attribute transfer? The following are the idealized natures of such a transfer function:
vm = φz(vw), (5)
vw = φz(vm), (6)
where m ∈M and w ∈ F . This function φz enables to transfer a word without explicit knowledge. Function φz transfers vm to vw and vw to vm without such explicit knowledge as m ∈ M and w ∈ F . By combining Eqs. 5 and 6, we obtain the following formula:
∀m ∈M, vm = φz( φz(vm) ), (7) and
∀w ∈ F , vw = φz( φz(vw) ). (8) Hence, the idealized transfer function is a mapping that becomes an identity mapping when we apply it twice for any v. Such a mapping is called involution in geometry. For example, φ : v 7→ −v is one example of an involution. Note that the identity map itself, such as φ : v 7→ v, is excluded from the involution.

heading: 4.2 REFLECTION
text: A reflection is an involution that reverses the location between two vectors in a Euclidean space through an affine hyperplane (mirror). Reflection is an idealized function because every point returns to its original location when reflection is applied twice:
∀v ∈ Rn, v = Refa,c(Refa,c(v) ). (9) Given vector v in Euclidean space Rn, the formula for the reflection in the mirror is given:
Refa,c(v) = v − 2 (v − c) · a
a · a a, (10)
where a ∈ Rn is a vector orthogonal to the mirror (normal vector) and c ∈ Rn is a point through which the mirror passes. a and c are parameters that determine the mirror.

heading: 4.3 REFLECTION-BASED WORD ATTRIBUTE TRANSFER
text: We apply reflection to the word attribute transfer to invert a specific attribute of an input word without its explicit attribute knowledge. We learn a mirror (hyperplane) in a pre-trained embedding space using training word pairs with a common (binary) attribute z (Fig. 2). Here since the mirror is uniquely determined by two parameter vectors, a and c, we estimate a and c from target attribute z using fully connected multi-layer perceptrons:
a = MLP (z), (11)
c = MLP (z). (12)
Transferred vector vy is obtained by inverting attribute z of vx by reflection:
vy = Refa,c(vx). (13)

heading: 4.4 PARAMETERIZED MIRRORS
text: Reflection with a mirror by Eqs. 11 and 12 assumes a single mirror depending only on z. Previous discussion assumed that there will be pairs sharing a stable attribute such king and queen. However, often gendered words don’t come in pairs, and gender is far from a stable attribute. For example, actress may be feminine, but actor is clearly neutral in many cases (Fig. 3). Thus, actor isn’t as obvious a masculine counterpart as king. In fact, it is known that there are biases in gender words in the embedding space (Zhao et al., 2018; Kaneko & Bollegala, 2019). This phenomenon can occur not only with the gender attribute, but also with other attributes. With this assumption of a single mirror, the mirror must be a hyperplane that goes through the midpoints for all word vector pairs. However, the vector pairs shown on the left of Fig. 3 cannot be transferred well since the single mirror does not satisfy this constraint due to the bias of the embedding space. To solve this problem, we introduce different mirrors for different words. We propose parameterized mirrors determined by input vector vx in addition to attribute z. The following are the definitions of the mirror parameters:
a = MLP ([z;vx]), (14)
c = MLP ([z;vx]), (15)
where [·; ·] indicates the vector concatenation in the column. The parameterized mirrors are expected to work flexibly on different words. For instance, as shown in Fig. 3, suppose we learned the mirror (the blue line) that transfers vhero to vheroine in advance. If input word vector vactor resembles vhero, a mirror that is similar to the one for vhero should be derived and used for the attribute transfer.

heading: 4.5 LOSS FUNCTION
text: Loss function L is defined:
L(Θ) = 1 |A| ∑ (xi,ti,zi)∈A (vyi − vti)2 + 1 |N | ∑ xj∈N (vyj − vxj )2, (16)
where 1|A| ∑ (xi,ti,zi)∈A(vyi − vti) 2 is a term that draws target word vector vti closer to corre-
sponding transferred vector vyi and 1 |N | ∑ xj∈N (vyj − vxj ) 2 is a term that prevents words without a target attribute from being moved by transfer function fz. Θ represents the set of all the trainable parameters. The parameters in the proposed model are the MLP weights used to determine mirror hyperplanes via a and c. We iteratively update Θ to minimize L:
Θt+1 ← arg min Θt L(Θt), (17)
where t is the number of parameter updates at that time.

heading: 5 EXPERIMENT
text: We evaluated the performance of the proposed reflection-based word attribute transfer using data with some different attributes.

heading: 5.1 EXPERIMENTAL SETUP
text: We used three different datasets of word pairs with three binary attributes: Male-Female 1, SingularPlural and Capital-Country, shown in Table 1.These word pairs were collected from analogy test sets (Mikolov et al., 2013a; Gladkova et al., 2016), the Internet and Nguyen et al. (2017) for antonyms of noun2. Since these datasets are very small, we added Gaussian noise (σ = 0.1) to every input vector vx during training to avoid overfit. Random noise was applied independently to every sample in every iteration. For a non-attribute dataset N , we sampled words from the three-million-word vocabulary of the word embedding model. We sampled from 4 to 50 words for training (0 ≤ |Ntrain| ≤ 50) and 1000 words for the test (|Ntest| = 1000). We used a mixed dataset that included both |Ntrain| and |Atrain|. Note thatNtest was sampled and excluded words fromNtrain andAtrain. We had no Nval because the tuning was conducted with only |Aval|. We used word2vec (Mikolov et al., 2013b) 3 and GloVe (Pennington et al., 2014) 4 as the pre-trained embedding model. The embedded vector dimension is n = 300.

heading: 5.2 EVALUATION METRICS
text: We measured the accuracy and stability performances of the word attribute transfer. The accuracy measures how many input words in Atest were transferred correctly to the corresponding target words. The stability score measures how many words in Ntest are not mapped to other words. For example, in a gender transfer, given man, the transfer is regarded as correct if woman is the closest word to the transferred vector; otherwise it is incorrect. Given apple, the transfer is regarded as correct if apple is the closest word to the transferred vector; otherwise its stability is incorrect. Here we used cosine similarity to measure the similarity of output vector vy and target vector vt. The accuracy and stability scores are calculated by the following formula:
δ(vy, t) =
{ 1 if arg max
k∈V
vy·vk ‖vy‖‖vk‖ = t,
0 otherwise, (18)
1Note that these gender word pairs are an assumption because they contain socially problematic words. 2These datasets and the experimental codes will be released in the future. 3https://code.google.com/archive/p/word2vec/ 4https://nlp.stanford.edu/projects/glove/
Accuracy = 1 |Atest| ∑
(xi,ti,zi)∈Atest
δ(vyi , ti), (19)
Stability = 1 |Ntest| ∑
xi∈Ntest
δ(vyi , xi), (20)
where V is the vocabulary of the word embedding model.

heading: 5.3 METHODS AND CONFIGURATIONS
text: In our experiment, we compared our proposed method with the following baseline methods:
REF Reflection-based word attribute transfer with a single mirror. We used a fully connected 2-layer MLP with 300 hidden units and ReLU activations to estimate a and c.
REF+PM Reflection-based word attribute transfer with parameterized mirrors. We used the same MLP as the REF.
MLP Fully connected MLP: vy = MLP ([vx; z]). The highest accuracy models are a 2-layer MLP for Capital-Country and 3-layer MLP for the other datasets.
DIFF Analogy-based word attribute transfer with a difference vector: d = vm − vw, where m and w are in the training data of A. We chose d because it achieved the best accuracy in the validation data of A. We determined whether to add or subtract d to vx based on attribute knowledge (Eq. 4).
DIFF + Analogy-based word attribute transfer with a difference vector regardless of the attribute knowledge. d was obtained in the same way as the DIFF. We added d to vx for any input x: fz(vx) = vx + d, ∀vx ∈ Rn. DIFF − Analogy-based word attribute transfer with a difference vector regardless of the attribute knowledge. d was obtained in the same way as the DIFF. We subtracted d from vx for any input x: fz(vx) = vx − d, ∀vx ∈ Rn.
MEANDIFF Analogy-based word attribute transfer with a mean difference vector d̄: d̄ = 1 |Atrain| ∑
(mi,wi)∈Atrain(vmi − vwi). We determined whether to add or subtract d̄ to vx based on the attribute knowledge (Eq.4).
MEANDIFF + Analogy-based word attribute transfer with a mean difference vector regardless of the attribute knowledge: fz(vx) = vx + d̄, ∀vx ∈ Rn. d̄ was obtained in the same way as the MEANDIFF.
MEANDIFF − Analogy-based word attribute transfer with a mean difference vector regardless of the attribute knowledge: fz(vx) = vx − d̄, ∀vx ∈ Rn. d̄ was obtained in the same way as the MEANDIFF.
Based on the tuning, we used the Adam optimizer (Kingma & Ba, 2015) with a learning rate of α = 10−4 (the other hyperparameters were the same as the original one (Kingma & Ba, 2015)), and a batchsize of 62 for male-female, and 32 for the others. These hyperparameters were identical for the learning-based methods: REF, REF + PM, or MLP. We did not use such regularization methods as dropout (Srivastava et al., 2014) or batch normalization (Ioffe & Szegedy, 2015) because they did not show any improvement in our pilot test.

heading: 5.4 ACCURACY AND STABILITY
text: Table 2 and 3 shows the transfer accuracy and stability score results. Both experiments using GloVe or word2vec obtained similar results. REF + PM achieved the best accuracy among the methods that did not use explicit attribute knowledge. This means that reflection can be used for word attribute transfers even without attribute knowledge. In the stability evaluation, reflection-based methods (REF, REF + PM) and the analogy-based methods with a mean difference vector (MEANDIFF −, MEANDIFF +) achieved high stability. In particular, reflection-based transfers achieved outstanding stability scores exceeding 99%. The stability of DIFF + and DIFF − was much lower than the other methods. Although MEANDIFF − and MEANDIFF + achieved high stability, their accuracy results
were very low. Interestingly, reflection-based transfer with parameterized mirrors (REF + PM) achieved high performance in both accuracy and stability. For example, the accuracy of REF + PM was 41.67%, and the stability was 99.9% in Male-Female (MF), and the accuracy was 58 % and the stability was 99.40 % in Capital-Country (CC). These results show that the proposed method transfers an input word if it has a target attribute and does not transfer an input word even though it does not use explicit attribute knowledge on the input words. MLP worked poorly both in accuracy and stability. In the antonym (AN), while the transfer accuracy by the proposed method was a bit lower than that by MLP, the stability of the proposed method was 100% and that of MLP was really poor (almost 0%).
We investigated the relation between the size of |Ntrain| and the stability of learning-based methods by conducting an additional experiment by varying |Ntrain| from 0 to 50. The stability scores by MLP did not improve (Table 4). On the other hand, REF and REF + PM achieved high stability scores with just |Ntrain| = 4 and maintained the accuracy.

heading: 5.5 TRANSFER EXAMPLE
text: Table 5 shows examples of a gender transfer at the sentence level, where the attribute transfer was applied to words in sentence X = {x1, x2, ...}. Here since such words as a and . are not in the vocabulary of the original word embedding model, we omitted them from the inputs. MLP made many wrong transfers on words without gender attributes, e.g., the became By Katie Klingsporn, was became she, when became Doughty Evening Chronicle, and woman became girlfriend. DIFF+ can transfer if x is female, e.g., it transferred from woman to man, but it could not transfer grandfather and boy. Similarly, DIFF − failed to transfer from female to male. In addition, since the stability of these methods was low, they erroneously transferred. For example, in DIFF −, the and when became she. REF + PM can selectively transfer words with a gender attribute without using explicit gender information. For example, when woman was given, Ref(vwoman) became man without knowledge that woman is a female word, and when man was given, it became woman. When non-attribute word
married was given, Ref(vmarried) became married without knowledge that married has no gender attribute. When we applied the reflection-based transfer twice, the transferred word returned to its original word, e.g., Ref(Ref(vwoman)) gives woman.
We can transfer some different attributes of words with reflection-based transfer one-by-one. Table 6 shows that the words having different target attributes were transferred by each reflection-based transfer in the order of Male-Female, Singular-Plural, and Country-Capital. Given actress for a Male-Female transfer, it was transferred to actor and to actors for Singular-Plural. Given Tokyo for Male-Female, Singular-Plural, and Antonym, it was not transferred, but it was transferred to Japan for Country-Capital. Given rich for Male-Female, Singular-Plural, and Capital-Country, it was not transferred, but it was transferred to poor for Antonym.

heading: 6 RELATED WORK
text: The embedded vectors obtained by SGNS (Mikolov et al., 2013a;b) and GloVe (Pennington et al., 2014) have analogic relations. The theory of analogic relations in word embeddings has been widely discussed: Levy & Goldberg (2014b); Arora et al. (2016); Gittens et al. (2017); Ethayarajh et al. (2019); Allen & Hospedales (2019); Linzen (2016). Levy & Goldberg (2014b) offer the explanation that SGNS factorizes a shifted PMI matrix. Allen & Hospedales (2019) and Ethayarajh et al. (2019) argued that they proved the existence of such analogic relations without strong assumptions. In our work, we focus on the analogic relations in a word embedding space and propose a novel framework to obtain a word vector with inverted attributes. Style transfers (Niu et al., 2018; Prabhumoye et al., 2018; Jain et al., 2019; Logeswaran et al., 2018; Dai et al., 2019; Zhang et al., 2018) resemble our task. In a style transfer, the text style of the input sentences is changed. For instance, Jain et al. (2019) transferred from formal to informal sentences. Logeswaran et al. (2018) transferred sentences by controlling such attributes as mood and tense. These style transfer tasks use sentence pairs; our word attribute transfer task uses word pairs. Style transfer changes sentence styles, but our task changes the word attributes (contents). Soricut & Och (2015) studied the problem of morphological transformation based on character information. Our work aims more general attribute transfer such as gender transfer and country-capital and is not limited to the morphological transformation.

heading: 7 CONCLUSION AND FUTURE WORK
text: We proposed a novel representation learning framework based on reflection to invert a certain attribute of a word vector. We proposed a reflection-based method for word attribute transfers without relying on the explicit attribute knowledge of an input word, which is necessary for a simple analogy-based transfer. Experimental results showed that our proposed method can transfer the word attributes if the input word has a target attribute. If not, reflection does not transfer the word.
Future work includes applications to other transfer tasks: sentence by sentence transfer, such Niu et al. (2018); Prabhumoye et al. (2018); Jain et al. (2019), and entity prediction on an analogic graph embedding space (Liu et al., 2017), in the field of computer vision, visual analogy (Reed et al., 2015), or style transfer (Zhu et al., 2017; Liao et al., 2017) with GANs (Radford et al., 2016; Goodfellow et al., 2014) because their latent space holds analogic relations (Radford et al., 2016).

heading: A TOP THREE ACCURACY AND STABILITY
text: B VISUALIZATION OF PARAMETERIZED MIRRORS
Figures below visualize PCA results of a obtained for the test words. We normalized the L2 norm of a to 1 ( a‖a‖ ). Corresponding word pairs are connected by solid lines. Figs. 4 and 5 suggest not only the mirror parameters of paired words are similar to each other but also the parameters with the attribute form a cluster — words with the same attribute has similar mirror parameter a. The mirrors of paired words are close to each other in the same attribute (Fig. 4 and 5). Some MF pairs in Fig. 5 are placed away from the cluster of the MF words. This may come from missing principal components due to the small data size used for PCA. Figs. 6, 7, 8, 9, 10, 11, 12, and 13 are the detailed PCA results for four different attributes: MF, SP, CC, and AN. These results show that a reflection transfers a paired word each other by using a similar mirror. For example, rich and poor use almost the same mirror (Fig. 9). On the other hand, different mirrors are used for different word pairs since the mirrors are parameterized. These results shows the effect of the mirror as described in section 4.4.

Review summaries:
This paper proposes a method for binary word attribute transfer based on reflection. It applies a single reflection-based mapping that relates the locations of two vectors in a Euclidean space by a hyperplane, and results in an identity mapping when it is applied twice. It's sort of like roundtrip-translation, but for word vector attributes. The paper also proposes a pretty smart idea: the mirror functions are parameterized to take advantage of the fact that inversions differ, even for the "same" word attributes.