IA0248_09isibasi (2021-03-05 14:09)

General Paper

Reﬂection-based Word Attribute Transfer
Yoichi Ishibashi† , Katsuhito Sudoh† , Koichiro Yoshino† and Satoshi Nakamura†−−→−→ +Word embeddings, which often represent analogic relations such as king − −man−−−−−→ ≈ −−−→ can be used to change an attribute of a word, including its gender.
woman
queen,To transfer the gender attribute of king to obtain queen in this analogy, we subtract a−→ − −−−−−→ from king based on the knowledge that king is male.
diﬀerence vector −
man
woman
However, developing such knowledge is signiﬁcantly costly for words and attributes. In
this work, we propose a novel method for word attribute transfer based on reﬂection
mapping without an analogy-based operation. Experimental results show that our
proposed method can transfer the word attributes of the given words without changing
the words that are invariant with respect to the target attributes.
Key Words: Word, Word Embedding, Reﬂection, Word Attribute Transfer

1 Introduction
Distributed representation (Hinton et al. 1984) is a type of data representation that is trainedon objectives to embed similar data samples into closed points in a vector space for capturing
their similarities. In recent natural language processing, several studies have used neural networks. The distributed representation is compatible with neural networks because the representation can capture the features of language data and compress them into low-dimensional vectors.
Word embedding methods handle word semantics in natural language processing (Mikolov et al.
2013a, 2013b; Pennington et al. 2014; Nickel and Kiela 2017; Vilnis and McCallum 2015). Word
embedding models such as skip-gram with negative sampling (SGNS) (Mikolov et al. 2013b) or
global vectors for word representation (GloVe) (Pennington et al. 2014) capture analogic rela−−→−→ + −−−−−→ ≈ −−−→ Previous works (Levy and Goldberg 2014b; Arora
tions such as king − −
man
woman
queen.
et al. 2016; Gittens et al. 2017; Ethayarajh et al. 2019; Allen and Hospedales 2019) oﬀer theoretical explanations based on pointwise mutual information (PMI) (Church and Hanks 1990) for
maintaining analogic relations in word vectors.
These relations can be used to transfer a certain attribute of a word, such as changing kinginto queen by transferring its gender. This transfer can be applied to perform data augmentation;for example, rewriting He is a boy to She is a girl. It can be used to generate negative examples† Graduate School of Science and Technology, Nara Institute of Science and Technology

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

Fig. 1

Examples of word attribute transfer

for natural language inference (Kang et al. 2018). For example, in the Stanford Natural Language
Inference (SNLI) corpus (Bowman et al. 2015), negative examples can be generated by transferring
a hypothesis sentence from entailment to contradiction. We tackled a novel task that changes aword by transferring certain attributes associated with the word, which is called word attributetransfer (Fig. 1).
A naive way for word attribute transfer is to use a diﬀerence vector based on analogic relations,−→−−−−→ − −−→ to −−−→ This requires explicit knowledge whether
such as adding −
woman
man
king to obtain −queen.
an input word is male or female. We have to add a diﬀerence vector to a male word and subtract
it from a female word for achieving gender transfer. We also have to avoid changing words
that are invariant with respect to gender attributes, such as is and a in the example above, as
they are gender-invariant words. Developing such knowledge is signiﬁcantly costly for words andattributes in practice. In this paper, we propose a novel framework for word attribute transfer
based on reﬂection that does not require explicit knowledge of the given words in its prediction.
The contributions of this work are twofold: (1) We propose a word attribute transfer methodthat obtains a vector with an inverted binary attribute without explicit knowledge. (2) The
proposed method demonstrates more accurate word attribute transfer for words that have target
attributes than other baseline methods, while ensuring that the words that do not have target
attributes are unchanged.

2 Word Attribute Transfer Task
In this task, we focus on modeling the binary attributes (e.g., male and female1 ). Let x
denote a word and let vx ∈ Rn denote its n-dimensional vector representation. We assume
that vx is learned in advance using an embedding model, such as a skip-gram. In this task,

1 Gender-speciﬁc words are sometimes considered socially problematic.

on the man-woman relation.

207

Here, we use this as an example based

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

we have two inputs, a word x and vector z ∈ Rn , which represents a certain target attribute,and an output word y. y is the word obtained through the transfer of x according to the target
attribute speciﬁed by z. y should be the same as the reference word t. Note that t is the sameas x when x is invariant based on the target attribute. In this paper, z is an n-dimensional
vector embedded from a target attribute ID by using an embedding function of a deep learning
framework. For example, given a set of attributes Z = {gender, antonym}, we assign diﬀerent
random vectors zgender for gender and zantonym for antonym. Let A denote a set of triplets (x, t, z),e.g., (man, woman, zgender ) ∈ Agender , and N denote a set of invariant words for an attribute z,e.g., (person, zgender ) ∈ Ngender . This task transfers an input word vector vx to an output word
vector vy ∈ Rn by using a transfer function fzattr that inverts the attribute zattr of vx . vy is
expected to be the same as its reference word vector vt ∈ Rn . This is denoted according to thefollowing formula:vt ≈ vy = fz (vx ).

(1)

The following properties must be satisﬁed: (1) attribute words {x|(x, t, z) ∈ A} are transferred totheir counterparts, and (2) invariant words {x|(x, z) ∈ N } are not changed (are transferred backinto themselves). For instance, with zgender , for a given input word man, the gender attributetransfer fzgender (vman ) should result in a vector close to vwoman . When given another input word
person as x, the result should be vperson .

3 Analogy-based Word Attribute Transfer
Analogy is a general idea that can be used for word attribute transfer. PMI-based wordembedding methods, such as SGNS and GloVe, capture analogic relations, as shown in Eq. 2(Mikolov et al. 2013c; Levy and Goldberg 2014a; Linzen 2016). By rearranging Eq. 2, Eq. 3 isobtained:vqueen ≈ vking − vman + vwoman ,≈ vking − (vman − vwoman ).

(2)
(3)

The analogy-based transfer function isfz (vx ) =vx − dif x ∈ M,vx + dif x ∈ F ,

208

(4)

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

where M is a set of words with a particular target attribute (e.g., male) and F is a set of words
with an inverse attribute (e.g., female). d is a diﬀerence vector, such as vman − vwoman . Eq. 4
indicates that the operation changes depending on whether the input word x belongs to M or
F. However, to transfer the word attribute based on analogy, we require explicit knowledge suchas the attribute value (M, F, or others) that is contained by the input word.

4 Reﬂection-based Word Attribute Transfer
4.1

Ideal Transfer Mapping without Knowledge

What is an ideal transfer function fz for the word attribute transfer? The following are theideal natures of a transfer function:∀(m, w, z) ∈ A,

vm = fz (vw ),(5)∀(m, w, z) ∈ A,

vw = fz (vm ),(6)∀(u, z) ∈ N , vu = fz (vu ).

(7)

The function fz enables a word to be transferred without explicit knowledge because the operationof fz does not change depending on whether the input word belongs to M or F. By combining
Eqs. 5, 6 and 7, we obtain the following formulas:∀(m, w, z) ∈ A,

vm = fz ( fz (vm ) ),(8)∀(m, w, z) ∈ A,

vw = fz ( fz (vw ) ),(9)∀(u, z) ∈ N , vu = fz ( fz (vu ) ).

(10)

Hence, the ideal transfer function is a mapping that becomes an identity mapping when we apply
it twice for any v. Such a mapping is called involution in geometry. For example, f : v 7→ −v isan example of an involution.

4.2

Reﬂection

Reﬂection Ref a,c is an ideal function because this mapping is an involution, as shown below:∀v ∈ Rn ,v = Ref a,c ( Ref a,c (v) ).

(11)

Reﬂection reverses the location between two vectors in a Euclidean space through an aﬃne
hyperplane called a mirror. a and c are parameters that determine the mirror. a ∈ Rn is a vector
orthogonal to the mirror and c ∈ Rn is a point through which the mirror passes. Reﬂection is209

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

diﬀerent from inverse mapping. When m and w are paired words, reﬂection can transfer vm andvw between each other with identical reﬂection mapping as shown in Eqs. 5 and 6; however, an
inverse mapping cannot perform this action. Given a vector v in the Euclidean space Rn , theformula for the reﬂection in the mirror is given by
Ref a,c (v) = v − 2

4.3(v − c) · a
a.
a·a

(12)

Proposed Method: Reﬂection-based Word Attribute Transfer

Reﬂection by a Single Mirror We apply reﬂection to the word attribute transfer process. We
learn a mirror (hyperplane) in a pretrained embedding space using training word pairs with binary
attribute z (Fig. 2). Because the mirror is uniquely determined by two parameter vectors, a andc, we estimate a and c from the target attribute z using fully connected multilayer perceptrons(MLPs):
a = MLPθ1 (z),(13)

c = MLPθ2 (z),(14)

where θ is a set of trainable parameters of MLPθ . The transferred vector vy is obtained byinverting the attribute z of vx by reﬂection:vy = Ref a,c (vx ).

Fig. 2Reﬂection-based word attribute transfer with a single mirror

210

(15)

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

Reﬂection by Parameterized Mirrors Reﬂection with a mirror according to Eqs. 13 and14 assumes a single mirror that only depends on z. The previous discussion assumed pairs thatshare a stable pair, such as king and queen.
However, as gender-variant words often do not come in pairs, gender is not suﬃciently stable
to be modeled by a single mirror. For example, although actress is exclusively feminine, actor isclearly neutral in several cases. Thus, actor is not a masculine counterpart such as king. In fact,bias exists in gender words in the embedding space (Zhao et al. 2018; Kaneko and Bollegala 2019).
This phenomenon can occur not only with gender attributes but also with other attributes. The
assumption of a single mirror forces the mirror to be a hyperplane that goes through the midpoints
for all word vector pairs. However, the vector pair actor -actress, shown on the right in Fig. 3,
cannot be transferred well as the single mirror (the green line) does not satisfy this constraint
owing to the bias of the embedding space. To solve this problem, we propose parameterized
mirrors based on the idea of using diﬀerent mirrors for diﬀerent words. We deﬁne the mirror
parameters a and c using the word vector vx to be transferred in addition to the attribute vectorz:a = MLPθ1 ([z; vx ]),(16)

c = MLPθ2 ([z; vx ]),(17)

where [·; ·] indicates the vector concatenation in the column. Parameterized mirrors are expected
to work more ﬂexibly on diﬀerent words than a single mirror because parameterized mirrors
dynamically determine similar mirrors for similar words. For instance, as shown in Fig. 3, let us
assume that we learned the mirror (the blue line) that transfers vhero to vheroine in advance. If
the input word vector vactor resembles vhero , a mirror that resembles the one for vhero should

Fig. 3Reﬂection using parameterized mirrors

211

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

be derived and used for the transfer. Conversely, the reﬂection works as an identity mapping
for a vector on the mirror (e.g., vperson in Fig. 3). That is, the proposed method assumes that
invariant word vectors are located on the mirror. Because we used a 300-dimensional embedded
space in the experiment, we assume that the invariant word vector exists in a 299-dimensional
subspace.
It should be noted that Eq. 11 may not hold for parameterized mirrors. In the reﬂection
with a single mirror, it is true that v = Ref a,c ( Ref a,c (v)). However, this is not guaranteed withthe v-parameterized reﬂection Ref av ,cv (v). This is because the mirror parameters av and cv
depend on an input word vector, as shown in Eqs. 16 and 17. Thus, we exclude this constraint
and employ the constraints given by Eqs. 5–7 for our loss function.
Weight Sharing

In neural networks, weight sharing can reduce the number of trainable weights

and often improve performance (Yang et al. 2018; Lample et al. 2018). The mirror parameters aand c can be deﬁned using a shared MLP as follows:o = MLPθ ([z; vx ]),(18)

a = Wa o,(19)

c = Wc o,(20)

where θ indicates the shared weights. o ∈ Rm is an output vector of MLPθ . Wa ∈ Rn×m andWc ∈ Rn×m are weight matrices corresponding to a and c, respectively.
Loss Function The following properties must be satisﬁed in word attribute transfer: (1) words
with attribute z are transferred and (2) words without it are not transferred. Thus, loss L(Θ) is
deﬁned as:
L(Θ) =

1|A|∑(vy − vt )2 +(x,t,z)∈A

1
|N |

∑

(vy − vx )2 , ,(21)(x,z)∈N

where Θ is a set of trainable parameters (Θ = {θ} for weight sharing and Θ = {θ1 , θ2 } otherwise).
The ﬁrst term draws the target word vector vti closer to the corresponding transferred vector vyi
and the second term prevents words that are invariant with respect to a target attribute frombeing moved by the transfer function. vy is the output of a reﬂection (Eq. 15).

212

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

5 Experiment
We evaluated the performance of word attribute transfer using data with four diﬀerent attributes. We used 300-dimensional word2vec2 and GloVe3 models as the pretrained word embedding. We used four diﬀerent datasets of word pairs with four binary attributes: Male-Female(MF), Singular-Plural (SP), Capital-Country (CC), and Antonym (AN) (Table 1). These wordpairs were collected from analogy test sets (Mikolov et al. 2013a; Gladkova et al. 2016) and theInternet. Antonyms were obtained from the literature (Nguyen et al. 2017). Their datasets werecollected from WordNet (Miller 1995) and Wordnik 4 . The original data by Nguyen et al. (2017)contains synonyms; however, we excluded them and used only the antonyms. We compared themodels that train with attributes individually with the models that train with joint attributes.
The invariant word dataset N were constructed by random sampling from WordNet by excluding
the attribute-variant words in the corresponding set A. We sampled the invariant words for theinvariant portion of the training data by varying their occupancy, i.e., 0, 5, 10, 25, and 50%,
to investigate their eﬀects on the tradeoﬀs between variant and invariant words. We also chose
1,000 invariant words for the test (|Ntest | =1,000).

5.1

Evaluation Metrics

We measured the accuracy and stability performances of the word attribute transfer. The
accuracy measures the number of input words in Atest that were transferred correctly to thecorresponding target words. The stability score measures the number of words in Ntest that were
not mapped to other words. For example, in the MF transfer, given man, the transfer is regarded
as correct if woman is the closest word to the transferred vector; otherwise, it is incorrect. Givenperson, the transfer is regarded as correct if person is the closest word to the transferred vector;

Dataset A
Male-Female (MF)
Singular-Plural (SP)Capital-Country (CC)
Antonym (AN)
Table 1

#Train

#Val

#Test

#Total

106
3624
118
5002

48
776
50
642

48
776
50
642

202
5176
218
6286

Statistics of binary-attribute word datasets

2 https://code.google.com/archive/p/word2vec/
3 https://nlp.stanford.edu/projects/glove/
4 https://www.wordnik.com/

213

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

otherwise, it is incorrect. The accuracy and stability scores are calculated using the following
formula:1
δ(vy , t) =
0

Accuracy =Stability =ifarg max(cos(vy , vk )) = t
k∈V

(22)

otherwise,∑
1|Atest |(23)

δ(vy , x),(24)(x,t,z)∈Atest∑

1|Ntest |δ(vy , t),(x,z)∈Ntest

where V is the vocabulary of the word embedding model and cos(vy , vk ) is the cosine similarityv ·v
measure, which is deﬁned as cos(vy , vk ) = ∥vyy∥∥vkk ∥ .
For the accuracy evaluation in the AN transfer, we used a diﬀerent deﬁnition, as presented
subsequently, to evaluate the accuracy because there are multiple possible candidates for thetransfer in the AN dataset.

δAN (vy , t) =



1

0

AccuracyAN =if

arg max(cos(vy , vk )) ∈ T ,k∈Votherwise,∑
1|Atest |(25)

δAN (vy , T ),(26)

(x,T ,z)∈Atest

where T = {t1 , t2 , ..., t3 } is a set of target words of the input antonym word x.

5.2

Methods and Conﬁgurations

Because these datasets are signiﬁcantly small, we added 300-dimensional Gaussian noise to
every input vector during training to avoid overﬁtting, i.e., vx + gσ , where gσ is the Gaussian noise and σ is the standard deviation of the Gaussian distribution.

The reported re-sults are presented for the best set of hyperparameters evaluated on the validation set for eachmodel after a grid search on the following values: Adam (Kingma and Ba 2015) learning rateα ∈ {0.0001, 0.00015, 0.001, 0.0015} (the other hyperparameters were the same as the original hyperparameters), σ ∈ {0.0, 0.05, 0.1, 0.15, 0.2}, and MLP inner hidden size ∈ {300, 500, 1500, 3000}.
Table 2 lists the best hyperparameters of the proposed method. We did not use regularization
methods such as dropout (Srivastava et al. 2014) or batch normalization (Ioﬀe and Szegedy 2015)
because they did not show any improvement in our pilot test.
In training, the attribute and invariant word data were combined into one training dataset,
where an invariant word (x, z) ∈ N was represented as (x, x, z) ∈ N , similar to an attribute214

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

EmbeddingHyperparametersMF

word2vec

Model
Batch size
Best Iterations
Noise σAdam αActivation Function
Num of MLP layers
Inner hidden size of MLP
Size of |Ntrain |ModelBatch size
Best Iterations
Noise σAdam αActivation Function
Num of MLP layers
Inner hidden size of MLP
Size of |Ntrain |

.

GloVe

Table 2

SP

CC

AN

Ref+PM
512
21000
0.1
0.0001
ReLU
3
300
5%

Ref+PM+Share
512
14000
0.1
0.0001
ReLU
5
1500
25%

Ref+PM
512
20000
0.1
0.0001
ReLU
3
300
5%

Ref+PM
4096
20000
0.1
0.0001
ReLU
5
3000
50%

Ref+PM+Share
512
48000
0.1
0.0001
ReLU
3
300
25%

Ref+PM+Share
512
20000
0.1
0.0001
ReLU
5
1500
10%

Ref+PM
512
24000
0.1
0.0001
ReLU
3
300
5%

Ref+PM
4096
30000
0.1
0.0001
ReLU
5
3000
10%

Hyperparameters for reﬂection-based word attribute transfer

word (x, t, z) ∈ A. Thus, we could simply implement the loss function (Eq. 21) as follows:∑
1
2
L(Θ) = |A∩N
(x,t,z)∈A∩N (vy − vt ) , where A ∩ N is a mini-batch of training data.
|In our experiment, we compared our proposed method with the following baseline methods:
RefThis is a reﬂection-based word attribute transfer with a single mirror. We used a fully
connected MLP with a rectiﬁed linear unit (ReLU) (Glorot et al. 2011) to estimate a andc.

Ref +PM This is a reﬂection-based word attribute transfer with parameterized mirrors. We
used the same architecture of MLP as Ref.
Ref +PM+Share This method consists of a reﬂection-based word attribute transfer with parameterized mirrors. We used the MLP with shared weights to estimate a and c (refer to
Weight Sharing in Section 4.3).
MLP This is a fully connected MLP with ReLU: vy = MLP([vx ; z]). The highest accuracy
models are a ﬁve-layer MLP with 1500 hidden units for SP, ﬁve-layer MLP with 3000
hidden units for AN, and three-layer MLP with 300 hidden units for the other datasets.
The optimal conﬁgurations were as follows: the learning rate for Adam α = 0.00015 forall datasets; moreover, σ = 0.05 for AN and σ = 0.1 for the other datasets.
TransE The word attribute transfer task is similar to link prediction in which (x, z, t) is replaced215

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

with (head, label, tail). In link prediction, given a set of triplets (head, label, tail), theknowledge graph embedding model predicts the tail from the head and label. We applied
TransE (Bordes et al. 2013), a baseline model for knowledge graph embeddings, to word
attribute transfer. We modiﬁed the model to input the word vector into the knowledge
graph embedding model based on the following equations:h = Whead vx ,(27)

t = Wtail vt ,(28)

where h ∈ Rk is a head vector and t ∈ Rk is a tail vector. Whead∈Rk×n and Wtail∈Rk×n
are weight matrices corresponding to the head and tail, respectively. The label vectorl was embedded in the same way as the original TrasnE based on a set of relations{MF, SP, CC, AN}. The optimal conﬁgurations were as follows: the latent dimension
k = 200, learning rate λ for stochastic gradient descent λ = 1.0, and margin γ = 5.0.
TransE was implemented using an open toolkit for knowledge embedding called OpenKE(Han et al. 2018). In the evaluation, when calculating the accuracy and stability in Eqs.
22 and 25, the score function of TransE was used instead of cos(vy , vk ).
Diﬀ

This method consists of analogy-based word attribute transfer with a diﬀerence vector,
d = vm − vw , where m and w are in the training data of A. We chose the d that achieved
the best accuracy in the validation data of A. We determined whether to add or subtract
d to vx based on explicit knowledge (Eq. 4). Here, Diff+ and Diff− transfer word
attributes using a diﬀerence vector regardless of the explicit knowledge. + and − add or
subtract the diﬀerence vector to any input word vector.

MeanDiﬀ

This method includes an analogy-based word attribute transfer with a mean diﬀer-

ence vector d̄, where∑1d̄ = |Atrain(mi ,wi ,z)∈Atrain (vmi − vwi ). We determined whether to add or subtract d̄ to|vx based on the explicit knowledge (Eq. 4).

5.3

Evaluation of Accuracy and Stability

Table 3 lists the accuracy and stability results. Because AN has a many-to-many relationship,a single diﬀerence vector cannot be obtained. Therefore, the analogy methods (Diff, Diff+− ,MeanDiff, and MeanDiff+− ) were not applied to AN. Diﬀerent pretrained word embeddings
by GloVe and word2vec provided similar results. Ref+PM and Ref+PM+Share achieved thebest accuracy among the methods that did not use explicit attribute knowledge. For example,
the accuracy of Ref+PM was 74% for CC; however, the accuracy of MLP was 18%. For most216IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Embeddingword2vec

MethodStability (%)

MF

SP

CC

AN

MF

SP

CC

AN

22.9
22.9
41.7
37.5
10.4
0.0

0.5
0.5
44.2
43.0
40.1
0.2

44.0
42.0
62.0
60.0
18.0
0.0

0.2
0.2
11.2
7.2
12.5
0.0

100.0
100.0
98.5
99.3
5.7
100.0

100.0
100.0
95.9
98.7
95.1
100.0

100.0
100.0
100.0
100.0
9.2
100.0

100.0
100.0
75.2
96.2
92.4
100.0

Ref (joint)
Ref+Share (joint)
Ref+PM (joint)
Ref+PM+Share (joint)
MLP (joint)
TransE (joint)

18.8
18.8
25.0
18.8
16.7
0.0

0.3
0.3
43.4
50.8
38.1
0.3

42.0
44.0
44.0
34.0
8.0
0.0

0.2
0.2
13.2
16.0
14.0
0.0

100.0
100.0
100.0
100.0
95.4
100.0

100.0
100.0
93.7
98.8
98.6
100.0

100.0
100.0
100.0
100.0
97.6
100.0

100.0
100.0
63.7
89.0
97.1
100.0

Diff +
Diff −
MeanDiff +MeanDiff −

22.9
22.9
6.3
8.3

3.2
3.1
0.3
0.3

32.0
32.0
22.0
14.0

—
—
—
—

97.1
87.9
100.0
100.0

93.1
98.8
100.0
100.0

89.5
99.4
99.7
99.9

—
—
—
—

37.5
14.6

6.3
0.5

64.0
36.0

—
—

—
—

—
—

—
—

—
—

Ref (individual)
Ref+Share (individual)
Ref+PM (individual)
Ref+PM+Share (individual)
MLP (individual)
TransE (individual)

10.4
10.4
37.5
39.6
14.6
0.0

0.5
0.4
45.0
42.8
41.1
0.2

24.0
24.0
74.0
72.0
18.0
0.0

0.2
0.0
12.3
11.5
14.2
0.0

100.0
100.0
99.2
99.2
41.7
100.0

100.0
100.0
99.3
99.8
97.6
100.0

100.0
100.0
100.0
100.0
50.3
100.0

100.0
100.0
93.3
94.7
93.4
100.0

Ref (joint)
Ref+Share (joint)
Ref+PM (joint)
Ref+PM+Share (joint)
MLP (joint)
TransE (joint)

12.5
2.1
12.5
12.5
27.1
0.0

0.4
0.4
39.8
47.0
35.2
0.3

24.0
20.0
36.0
36.0
26.0
0.0

0.2
0.0
9.0
9.8
11.1
0.0

100.0
100.0
100.0
100.0
98.0
100.0

100.0
100.0
99.7
97.7
99.7
100.0

100.0
100.0
100.0
100.0
99.4
100.0

100.0
100.0
94.1
59.0
97.6
100.0

Diff +
Diff −
MeanDiff +MeanDiff −

14.6
12.5
0.0
0.0

4.5
4.3
0.3
0.3

22.0
26.0
2.0
4.0

—
—
—
—

100.0
100.0
100.0
100.0

100.0
99.8
100.0
100.0

100.0
99.9
100.0
100.0

—
—
—

27.1
0.0

8.7
0.5

48.0
6.0

—
—

—
—

—
—

—
—

—

Diff
MeanDiff

Table 3

Accuracy (%)

Knowledge

Ref (individual)
Ref+Share (individual)
Ref+PM (individual)
Ref+PM+Share (individual)
MLP (individual)
TransE (individual)

Diff
MeanDiff

GloVe

Reﬂection-based Word Attribute Transfer

✓
✓

✓
✓

Results of accuracy and stability scores. MF, SP, CC, and AN are datasets. Here, “joint”
models are trained with joint attributes and “individual” models are trained with an individual
attribute.

attributes, our proposed methods outperformed the analogy-based transfer. Weight sharing did
not signiﬁcantly improve the performance of the proposed methods. The parameterized mirror
improved the performance of a reﬂection-based transfer, although the learning was unstable(Fig. 4). For stability, reﬂection-based transfers achieved superior stability scores, which exceeded217IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

93% in most cases. We mixed all the attribute datasets and trained models. The best model was
the proposed method trained with an individual attribute dataset. In the joint condition, theMLP demonstrated better performance than that in the individual attribute condition with thehelp of the larger training data. The results show that our proposed methods transfer an input
word if it has a target attribute and does not transfer an input word with better scores than thebaseline methods, even though the proposed methods do not use knowledge of the input words.
In TransE, the accuracy was almost 0% for all the attributes.5 However, in the MF, SP, andCC relations, several reference words were within the top three nearest neighbors, as listed inTable 4. This result is similar not only in this task but also when learning with WN18 (Trouillon
et al. 2016). This is owing to the nature of TransE. While considering AN, the accuracy in thethree nearest neighbors was still low. This can be explained by the poor performance of TransE

Fig. 4Visualization of validation accuracy

word2vec
MethodTransE (individual)
TransE (joint)

Table 4

GloVe

MF
@1 @3

SP
@1 @3

CC
@1 @3

AN
@1 @3

MF
@1 @3

SP
@1 @3

CC
@1 @3

AN
@1 @3

0.0
0.0

0.2
0.3

0.0
0.0

0.0
0.0

0.0
0.0

0.2
0.3

0.0
0.0

0.0
0.0

75.0
72.9

76.3
70.7

80.0
64.0

20.2
20.0

66.7
64.6

80.0
75.6

76.0
72.0

42.2
36.4

Accuracy of the top three nearest neighbors of TransE. A “joint” model is trained with joint
attributes. An “individual” model is trained with an individual attribute.

5 We also experimented with ComplEx (Trouillon et al. 2016), which is a knowledge graph embedding modelin a complex space; however, it is not described in this paper because it is not comparable because of its low
accuracy.

218

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

for one-to-many or many-to-many relationships (Bordes et al. 2013).
In the individual attribute condition, MLP worked poorly, especially in terms of stability for
MF and CC, while it showed better transfer accuracy for AN than the proposed method. We
reviewed the training curves resulting from the MLP, which are shown in Fig. 4 and 5; however,they showed reasonable convergence. This would be due to the training data size in the individual
attribute condition, because MLP stability signiﬁcantly improves in the joint condition.
We also investigated the tradeoﬀ between transfer accuracy and stability by changing thesize of the invariant words and the stability of the learning-based methods by conducting an
additional experiment that varied |Ntrain |. The large size of Ntrain is expected to increase thestability; however, it may also decrease the accuracy. The stability scores demonstrated by the
MLP did not improve (Table 5) for MF and CC. Conversely, the proposed methods achieved
high stability scores with |Ntrain | = 5% and maintained the accuracy. We hypothesized that thehigh stability was owing to the distance between the word and its mirror. If invariant words are
distributed on the mirror, they will not be transferred. We investigated the distance between the
input word vector vx and its mirror (Fig. 6). The result showed that invariant words were close
to the mirror and attribute words were distributed away from it. If the distance between paired
words is signiﬁcantly small, the distance between the word and its mirror is also small. Fig. 7
shows the distribution of the distance between the input vx and the target word vector vt . The
distance between paired words for MF and SP is considerably smaller than that for CC and AN.
Although analogy-based methods achieved high stability, their accuracy results were low. In
particular, the MeanDiff+ and MeanDiff− did not change the original vector. We hypothe-

Fig. 5Visualization of training loss

219

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

Accuracy (%)
EmbeddingMethod|Ntrain |10% 25%

50%

0%

5%

|Ntrain |
10%

25%

50%

20.8
41.7
31.2
6.2

22.9
37.5
35.4
8.3

22.9
35.4
37.5
8.3

18.8
25.0
29.2
10.4

100.0
86.5
78.6
0.0

100.0
98.5
99.4
0.0

100.0
99.6
99.5
0.0

100.0
99.7
99.3
0.8

100.0
91.8
99.8
5.7

0.4
43.3
44.7
42.0

0.5
46.3
43.6
41.0

0.4
44.2
43.7
40.1

0.3
42.4
43.0
36.7

0.3
40.3
38.7
36.0

100.0
53.4
42.3
66.8

100.0
82.4
93.5
86.0

100.0
95.9
94.5
95.1

100.0
99.1
98.7
98.1

100.0
99.5
98.4
99.6

CC

Ref
Ref+PM
Ref+PM+Share
MLP

34.0
62.0
56.0
10.0

34.0
62.0
58.0
12.0

36.0
54.0
58.0
10.0

38.0
54.0
60.0
18.0

44.0
50.0
56.0
18.0

100.0
90.0
86.4
0.0

100.0
100.0
100.0
0.0

100.0
100.0
100.0
0.0

100.0
100.0
100.0
0.6

100.0
99.8
100.0
9.2

AN

Ref
Ref+PM
Ref+PM+Share
MLP

0.0
12.5
13.9
17.0

0.2
12.9
12.8
15.4

0.0
12.3
12.1
15.1

0.0
11.8
12.0
12.5

0.2
11.2
7.2
14.2

100.0
26.8
7.7
1.2

100.0
26.0
20.8
6.2

100.0
34.3
49.7
36.6

100.0
65.7
71.4
92.4

100.0
75.2
96.2
67.2

MF

Ref
Ref+PM
Ref+PM+Share
MLP

10.4
37.5
35.4
4.2

4.2
39.6
31.2
12.5

6.2
37.5
39.6
6.2

4.2
37.5
39.6
8.3

2.1
35.4
35.4
14.6

100.0
89.3
88.7
0.0

100.0
93.2
98.9
0.0

100.0
95.7
97.5
0.0

100.0
99.2
99.2
0.3

100.0
99.6
99.6
41.7

SP

Ref
Ref+PM
Ref+PM+Share
MLP

0.5
46.3
43.9
42.7

0.4
46.6
43.7
41.0

0.5
46.4
44.8
41.1

0.4
44.6
45.0
38.9

0.4
45.0
42.8
36.9

100.0
54.1
52.6
70.0

100.0
94.5
95.5
95.2

100.0
97.8
98.1
97.6

100.0
98.9
99.4
99.3

100.0
99.3
99.8
99.8

CC

Ref
Ref+PM
Ref+PM+Share
MLP

22.0
66.0
70.0
8.0

24.0
74.0
70.0
6.0

24.0
70.0
70.0
8.0

22.0
70.0
72.0
6.0

20.0
74.0
72.0
18.0

100.0
99.9
99.8
0.0

100.0
100.0
99.9
0.0

100.0
99.9
99.9
0.0

100.0
100.0
99.9
1.3

100.0
99.9
100.0
50.3

AN

Ref
Ref+PM
Ref+PM+Share
MLP

0.2
12.1
11.7
16.6

0.2
13.2
12.5
14.5

0.0
12.3
10.7
16.0

0.0
10.9
11.5
14.2

0.0
10.6
8.4
11.7

100.0
16.0
14.0
2.1

100.0
78.0
42.4
53.1

100.0
93.3
73.9
70.5

100.0
96.1
94.7
93.4

100.0
97.1
96.8
97.9

0%

5%

MF

Ref
Ref+PM
Ref+PM+Share
MLP

18.8
35.4
37.5
4.2

SP

Ref
Ref+PM
Ref+PM+Share
MLP

word2vec

GloVe

Table 5

Stability (%)

Relation between the size of |Ntrain | and the stability of methods trained with an individual
attribute

sized that the result can be attributed to the signiﬁcantly small L2 norm of the mean diﬀerence
vector d̄. Table 6 lists the relationship between the MeanDiff performances and the L2 normof the mean diﬀerence vector. The stability was high because the original vector was almost
unchanged even if d was added or subtracted. Conversely, when the L2 norm of d̄ was large, theaccuracy became high as the diﬀerence vectors were similar to each other.

220IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

Fig. 6

Distribution of distance between the input word vector and its mirror |(vx∥−ac∥)·a| learned by
Ref+PM. It can be observed that invariant words are close to the mirror and attribute words
are distributed away from it.

Fig. 7

Distribution of distance between the input word vector and the target word vector ∥vx − vt ∥Diff, Diff+ , and Diff− obtained high accuracy for CC and low accuracy for SP. This is dueto the use of a ﬁxed diﬀerence vector d in Diﬀ. We investigated the mean cosine similarity between∑
1
the diﬀerence vector d and other diﬀerence vectors, i.e., mean = |A|(x,t,z)∈A cos(d(x,t) , d),where d(x,t) is the diﬀerence vector of a word pair (x, t) other than d. We found that the mean
cosine similarity between SP words was almost 0% in Diff− , as listed in Table 6. Thus, when
we use a single diﬀerence vector, several SP words are transferred into inappropriate words.

221

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

5.4

Vol. 28 No. 1

March 2021

Visualization of Parameterized Mirrors

Fig. 8 shows the principal component analysis (PCA) results of the mirror parameter aobtained for the test words. We normalized the L2 norm of a to 1 ( ∥aa∥ ). We compared the
PCA results with the results of the model trained with the joint attributes and the model trained

MeanDiff−
Embedding

Attr

Diff−

Acc

Stb

L2

cos
mean var

Acc

Stb

L2

cos
mean var

word2vec

MF
SP
CC

8.3
0.3
14.0

100.0
100.0
99.9

1.17
0.75
1.82

0.39
0.18
0.61

0.05
0.01
0.02

22.9
3.1
32.0

97.1
98.8
99.4

2.91
3.07
2.76

0.28
0.06
0.49

0.05
0.01
0.02

GloVe

MF
SP
CC

0.0
0.3
4.0

100.0
100.0
100.0

2.29
1.82
3.56

0.38
0.21
0.61

0.03
0.02
0.02

14.6
4.5
22.0

100.0
100.0
100.0

4.61
5.68
6.05

0.29
0.08
0.44

0.04
0.01
0.01

Table 6

Analysis of diﬀerence vectors. L2 is the L2 norm of the diﬀerence vector that the model usedduring the inference time (d for Diff and d̄ for MeanDiff). cos is the distribution of cosine
similarities between the diﬀerence vector (d or d̄) and other diﬀerence vectors.

Fig. 8Two-dimensional principal component analysis projection of the 300-dimensional mirror parameter a. The mirror parameters were estimated by the proposed model (Ref+PM+Share)
trained by each attribute.

222

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

with an individual attribute. Similar results were obtained for both conditions. Fig. 8 suggeststhat the mirror parameters of the paired words are similar to each other and that those with an
attribute form a cluster; words with the same attribute have similar mirror parameters, i.e., a.

5.5Transfer Example

Table 7 lists the gender transfer results for a tiny example sentence. Here, the attributetransfer was applied to every word in the sentence X = {x1 , x2 , ...}. Here, because words suchas a and . are not in the vocabulary of word2vec, we omitted them from the inputs. The MLP
resulted in several incorrect transfers on gender-invariant words, e.g., the became dear madam,
were became laundresses, boyfriend became boyfriend, and boy became mother. Analogy-basedtransfers can transfer only in one direction. Diff+ could transfer if x is female, e.g., it transferredfrom woman to man, but it could not transfer boy. Similarly, Diff− failed to transfer from female
to male. Moreover, as the stability of these methods was low, they resulted in erroneous transfers.
For example, in Diff− , the became Sir. Ref+PM could transfer only gender words without
using explicit gender information. Thus, woman was transferred to man without the knowledge
that woman is a female word. When the gender-invariant word married was input, it was notchanged by reﬂection, without the knowledge that married has no gender attribute.
From Table 8, it can be noted that words with diﬀerent target attributes were transferredby each reﬂection-based transfer. Thus, when daughter was input for a MF transfer, it wastransferred to son and daughters for SP transfer. When Tokyo was input for MF, SF, and AN
tranfers, it was not transferred; however, it was transferred to Japan in the CC transfer. When
stereo was input for MF, SP, and CC transfers, it was not transferred; however, it was transferred
to monaural in the AN transfer.
Input words

the woman got married when you were a boy .

Refthe man got married when you were a boy .

Ref+PMthe man got married when you were a girl .

Ref+PM+Sharethe man got married when you were a girl .

Diff +

Sir man got married when you were a boy .

Diff −

chairwoman woman got married chairwoman you were a girl .

MLPdear madam man dear madam boyfriend dear madam
lazy slob laundresses a mother .

Methods

Table 7Comparison of gender transfers. Each method transfers words in a sentence one by one.

223

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

6 Discussion
In our method, the performance of GloVe was better than that of word2vec. Table 9 lists thescores of the Google analogy test set (Mikolov et al. 2013a) for the diﬀerent embedding methods,i.e., word2vec and GloVe. From in Table 9, it can be noted that the score of GloVe was higher
than that of word2vec. This indicates that the embedding space of GloVe worked better thanthat of word2vec in this task. The performance of the word attribute transfer using reﬂection
probably depends on the analogic space, because transferring will be easy if the pair of transferred
words exists in a similar place in the analogic space.

Mr. Smith and his daughter want to visit the science museumin Tokyo to see the stereo microphone .

Input words

Ref+PM(individual)

Ref+PM(joint)Table 8MF

Ms. Smith and her son want to visit the science museumin Tokyo to see the stereo microphone .

SP

Mr. Smith and his daughters want to visits the science museumsin Tokyo to see the stereo microphones .

CC

Mr. Smith and his daughter want to visit the science museumin Japan to see the stereo microphone .

AN

Mr. Smith and his daughter eliminate to visit the science museumin Tokyo to back the monaural microphone .

MF

Ms. Smith and her son want to visit the science museumin Tokyo to see the stereo microphone .

SP

Mr. Smith and his daughters dos to visits the science museumsin Tokyo to watches the cassette decks microphones .

CC

Mr. Smith and his daughter want to visit the science museumin Japan to see the stereo microphone .

AN

Mr. Smith and his daughter want to visit the zoology museumin Tokyo to see the monaural microphone .

Transfer of diﬀerent attributes with the proposed method (Ref+PM)

Embedding method

Analogy score (%)

word2vec
GloVe

74.01
75.13

Table 9Comparison of analogy scores224

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

7 Error Analysis
We analyzed the attribute words that could not be transferred accurately by models trainedwith the individual attributes. We categorized the failed output words into three error cases.
Case1 The output word was the same as the input word (y = x).
Case2 The attribute of the input word was transferred but was incorrect. For example, in CC
transfer, the transfer result was Beijing when Japan was given.
Case3 Other types of errors.
Table 10 lists the results of the error analysis in word2vec. The results show that most of
the failures in the reﬂection-based transfer was in Case1. We speculated that such unchanged
attribute word pairs tended to be close to each other. Fig. 9 shows the diﬀerence in the distance
between the input word and the target word kvx − vt k in the changed and unchanged attribute
word pairs. Contrary to this hypothesis, it was shown that there was no diﬀerence in the distance
between the changed and unchanged pairs. Table 11 lists examples of Case2 and Case3 in the
proposed method. For example, when given stepbrother as a gender word, the proposed method(Ref+PM) did not output stepsister but provided the result of stepmother. In MF and CC

Cases where transfer failed (%)
Case1 Case2 Case3

MF

Ref
Ref+PM
Ref+PM+Share
MLP

100.0
86.0
88.0
6.0

0.0
12.0
8.0
70.0

0.0
2.0
4.0
24.0

SP

Ref
Ref+PM
Ref+PM+Share
MLP

100.0
64.0
62.0
64.0

0.0
36.0
38.0
34.0

0.0
0.0
0.0
2.0

CC

Ref
Ref+PM
Ref+PM+Share
MLP

100.0
37.5
55.8
2.0

0.0
62.5
44.2
98.0

0.0
0.0
0.0
0.0

AN

Ref
Ref+PM
Ref+PM+Share
MLP

100.0
42.0
94.0
66.0

0.0
10.0
2.0
10.0

0.0
48.0
4.0
24.0

Table 10Error analysis results

225IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Fig. 9Vol. 28 No. 1

March 2021

Distribution of the distance between the input word vector vx and the target word vector vt(Comparisons between the changed and unchanged attribute words)

transfers, the maximum failures using MLP were observed in Case2 and Case3 errors, while theproposed methods demonstrated signiﬁcant failure in the Case1 category. This shows that thereﬂection-based transfer is more stable in MF and CC transfers than MLP.

8 Related WorkThe embedded vectors obtained by SGNS (Mikolov et al. 2013a, 2013b) and GloVe (Pennington
et al. 2014) have analogic relations. The theory of analogic relations in word embeddings has been
widely discussed (Levy and Goldberg 2014b; Arora et al. 2016; Gittens et al. 2017; Ethayarajh
et al. 2019; Allen and Hospedales 2019; Linzen 2016). Levy and Goldberg (2014b) explain that
SGNS factorizes a shifted PMI matrix. Allen and Hospedales (2019) and Ethayarajh et al. (2019)argued that they proved the existence of such analogic relations without strong assumptions. In
our work, we focused on the analogic relations in a word embedding space and propose a novel
framework to obtain a transferred word vector with the target attribute.
Link prediction in knowledge graph embeddings can also be applied to word transfer tasks. In
knowledge graph embeddings (Bordes et al. 2013; Trouillon et al. 2016; Nickel et al. 2011), given a
set of triplets (head, tail, label), tail is predicted from head and label TransE (Bordes et al. 2013) isa knowledge graph embedding model. TransE embeds entity and relation into an embedding space
using a score function. The score function models the analogy-like operation translating fromthe head entity to the tail entity according to the relation. Trouillon et al. (2016) proposed the226IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

Attribute

Error type

Input (x)Target (t)Output (y)

Case2
Case2
Case2

hens
stepbrotheremperorroosters
stepsister
empressoxen
stepmothergoddess

Case3

mare

stalliongelding

SP

Case2
Case2
Case2
Case2
Case2
Case2
Case2
Case2
Case2
Case2
Case2

killers
atoll
gulls
windmill
saxophonetrailsspectaculars
visa
neckties
wagonsoutlookkilleratollsgull
windmills
saxophones
trail
spectacularvisas
necktiewagon
outlooksmurdererislands
heron
paddlewheels
trombones
trailheadextravaganza
passports
jackettractorforecasts

CC

Case2
Case2
Case2
Case2
Case2

Australia
CanadaJamaica
LondonHungary

Canberra
OttawaKingston
England
Budapest

SydneyMontreal
Belmopan
BritainBucharest

Case2
Case2
Case2
Case2
Case2

underbidperfection
rely
penaltysane

overbidimperfectionsuspect, distrustadvantage, reward
insane, crazyovercharged
imperfect
independentacquittal
irrational

AN

Case3
Case3
Case3
Case3
Case3
Case3
Case3

disinherit
elder
unimpressive
unhelpfulstarveextraneous
harmlessleave, will, bequeath
juniorimpressive
helpfulgive, feed
intrinsic
harmfuldisinheriting
niece
solid
sensibleencourage
necessaryimportant

Table 11

Examples of Case2 and Case3 errors in the proposed methodMFknowledge graph embedding model based on complex values for link prediction. Their knowledge
graph embedding model is better suited for modeling a variety of binary relations, including
symmetric and asymmetric relations. Our task diﬀers from link prediction in that when an

227IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

invariant word for attribute z is entered, the model returns the input x.
The style transfer task presented in previous studies (Jain et al. 2019; Logeswaran et al.
2018) resembles ours. In style transfer, the text style of the input sentences is changed. For
instance, Jain et al. (2019) transferred the style from formal to informal sentences. Logeswaran
et al. (2018) transferred sentences by controlling attributes such as mood and tense. These styletransfer tasks use sentence pairs. Our word attribute transfer task uses word pairs. Style transfer
changes sentence styles; however, our task changes the word attributes.
Soricut and Och (2015) studied morphological transformation based on character information.
Our work aims for a more general attribute transfer, such as gender transfer and obtaining theantonym, and is not limited to morphological transformation.

9 Conclusion
This research aimed to transfer word binary attributes (e.g., gender) for applications such asdata augmentation of a sentence.6 We can transfer word attributes using the analogy of word
vectors; however, this process requires explicit knowledge on whether the input word has theattribute or not. However, this knowledge cannot be developed for various words and attributes
in practice. The proposed method uses reﬂection-based mappings to transfer attribute-variant
words into their counterparts while ensuring that attribute-invariant words are unchanged, without using attribute knowledge in the inference time. The experimental results showed that the
proposed method outperformed baseline methods in terms of transfer accuracy for attributevariant words and stability for attribute-invariant words. We speculated that the reason why the
proposed method achieved signiﬁcantly high stability was that invariant words were distributedin the mirrors. We examined the distance between the input word vector and its mirror. The
result showed that invariant words were distributed near the mirror and attribute words were
distributed away from the mirror.

Acknowledgement
We would like to thank the anonymous reviewers for their insightful comments and suggestions. Part of this work was presented at the 58th Annual Meeting of the Association for Computational Linguistics (ACL2020) Student Research Workshop (SRW) (Ishibashi et al. 2020). This

6 Our code and datasets are available at: https://github.com/ahclab/reflection

228IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

work was supported by JST CREST Grant Number JPMJCR1513.

References
Allen, C. and Hospedales, T. M. (2019). “Analogies Explained: Towards Understanding Word
Embeddings.” In Proceedings of the 36th International Conference on Machine Learning,
ICML 2019, 9–15 June 2019, Long Beach, California, USA, pp. 223–231.
Arora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A. (2016). “A Latent Variable Model Approachto PMI-based Word Embeddings.” Transactions of the Association for Computational Linguistics, 4, pp. 385–399.
Bordes, A., Usunier, N., Garcı́a-Durán, A., Weston, J., and Yakhnenko, O. (2013). “Translating
Embeddings for Modeling Multi-relational Data.” In Burges, C. J. C., Bottou, L., Ghahramani, Z., and Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems
26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedingsof a meeting held December 5–8, 2013, Lake Tahoe, Nevada, United States, pp. 2787–2795.
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). “A Large Annotated Corpus
for Learning Natural Language Inference.” In Màrquez, L., Callison-Burch, C., Su, J.,
Pighin, D., and Marton, Y. (Eds.), Proceedings of the 2015 Conference on Empirical Methods
in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17–21, 2015,
pp. 632–642. The Association for Computational Linguistics.
Church, K. W. and Hanks, P. (1990). “Word Association Norms, Mutual Information, andLexicography.” Computational Linguistics, 16 (1), pp. 22–29.
Ethayarajh, K., Duvenaud, D., and Hirst, G. (2019). “Towards Understanding Linear Word
Analogies.” In Proceedings of the 57th Conference of the Association for Computational
Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers,
pp. 3253–3262.
Gittens, A., Achlioptas, D., and Mahoney, M. W. (2017). “Skip-Gram −Zipf + Uniform = Vector
Additivity.” In Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, Volume 1: Long Papers,
pp. 69–76.
Gladkova, A., Drozd, A., and Matsuoka, S. (2016). “Analogy-based Detection of Morphologicaland Semantic Relations with Word Embeddings: What Works and What Doesn’t.” In Proceedings of the Student Research Workshop, SRW@HLT-NAACL 2016, The 2016 Conferenceof the North American Chapter of the Association for Computational Linguistics: Human229

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

Language Technologies, San Diego California, USA, June 12–17, 2016, pp. 8–15.
Glorot, X., Bordes, A., and Bengio, Y. (2011). “Deep Sparse Rectiﬁer Neural Networks.” In
Proceedings of the 14th International Conference on Artiﬁcial Intelligence and Statistics,AISTATS 2011, Fort Lauderdale, USA, April 11–13, 2011, pp. 315–323.
Han, X., Cao, S., Lv, X., Lin, Y., Liu, Z., Sun, M., and Li, J. (2018). “OpenKE: An Open Toolkit
for Knowledge Embedding.” In Blanco, E. and Lu, W. (Eds.), Proceedings of the 2018
Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System
Demonstrations, Brussels, Belgium, October 31–November 4, 2018, pp. 139–144. Association
for Computational Linguistics.
Hinton, G. E., McClelland, J. L., Rumelhart, D. E., et al. (1984). Distributed Representations.
Carnegie-Mellon University Pittsburgh, PA.
Ioﬀe, S. and Szegedy, C. (2015). “Batch Normalization: Accelerating Deep Network Training by
Reducing Internal Covariate Shift.” In Proceedings of the 32nd International Conference on
Machine Learning, ICML 2015, Lille, France, 6–11 July 2015, pp. 448–456.
Ishibashi, Y., Sudoh, K., Yoshino, K., and Nakamura, S. (2020). “Reﬂection-based Word Attribute Transfer.” In Rijhwani, S., Liu, J., Wang, Y., and Dror, R. (Eds.), Proceedings of the58th Annual Meeting of the Association for Computational Linguistics: Student ResearchWorkshop, ACL 2020, Online, July 5–10, 2020, pp. 51–58. Association for Computational
Linguistics.
Jain, P., Mishra, A., Azad, A. P., and Sankaranarayanan, K. (2019). “Unsupervised Controllable
Text Formalization.” In The 33rd AAAI Conference on Artiﬁcial Intelligence, AAAI 2019,
The 31st Innovative Applications of Artiﬁcial Intelligence Conference, IAAI 2019, The 9th
AAAI Symposium on Educational Advances in Artiﬁcial Intelligence, EAAI 2019, Honolulu,
Hawaii, USA, January 27–February 1, 2019, pp. 6554–6561.
Kaneko, M. and Bollegala, D. (2019). “Gender-preserving Debiasing for Pre-trained Word Embeddings.” In Proceedings of the 57th Conference of the Association for Computational
Linguistics, ACL 2019, Florence, Italy, July 28–August 2, 2019, Volume 1: Long Papers,
pp. 1641–1650.
Kang, D., Khot, T., Sabharwal, A., and Hovy, E. H. (2018). “AdvEntuRe: Adversarial Training
for Textual Entailment with Knowledge-Guided Examples.” In Gurevych, I. and Miyao,
Y. (Eds.), Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics, ACL 2018, Melbourne, Australia, July 15–20, 2018, Volume 1: Long Papers,
pp. 2418–2428. Association for Computational Linguistics.
Kingma, D. P. and Ba, J. (2015). “Adam: A Method for Stochastic Optimization.” In 3rd
230

IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA,May 7–9, 2015, Conference Track Proceedings.
Lample, G., Ott, M., Conneau, A., Denoyer, L., and Ranzato, M. (2018). “Phrase-Based & Neural
Unsupervised Machine Translation.” In Riloﬀ, E., Chiang, D., Hockenmaier, J., and Tsujii,
J. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing, Brussels, Belgium, October 31–November 4, 2018, pp. 5039–5049. Association
for Computational Linguistics.
Levy, O. and Goldberg, Y. (2014a). “Linguistic Regularities in Sparse and Explicit Word Representations.” In Proceedings of the Eighteenth Conference on Computational Natural Language
Learning, CoNLL 2014, Baltimore, Maryland, USA, June 26–27, 2014, pp. 171–180.
Levy, O. and Goldberg, Y. (2014b). “Neural Word Embedding as Implicit Matrix Factorization.”
In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8–13 2014, Montreal, Quebec, Canada,
pp. 2177–2185.
Linzen, T. (2016). “Issues in Evaluating Semantic Spaces Using Word Analogies.” In Proceedingsof the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval@ACL
2016, Berlin, Germany, August 2016, pp. 13–18.
Logeswaran, L., Lee, H., and Bengio, S. (2018). “Content Preserving Text Generation with
Attribute Controls.” In Advances in Neural Information Processing Systems 31: Annual
Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December
2018, Montréal, Canada, pp. 5108–5118.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). “Eﬃcient Estimation of Word Representations in Vector Space.” In 1st International Conference on Learning Representations,
ICLR 2013, Scottsdale, Arizona, USA, May 2–4, 2013, Workshop Track Proceedings.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013b). “Distributed Representations of Words and Phrases and their Compositionality.” In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing
Systems 2013. Proceedings of a meeting held December 5–8, 2013, Lake Tahoe, Nevada,
United States., pp. 3111–3119.
Mikolov, T., tau Yih, W., and Zweig, G. (2013c). “Linguistic Regularities in Continuous Space
Word Representations.” In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9–14, 2013,
Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pp. 746–751.
Miller, G. A. (1995). “WordNet: A Lexical Database for English.” Communications of the ACM,
231

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

38 (11), pp. 39–41.
Nguyen, K. A., im Walde, S. S., and Vu, N. T. (2017). “Distinguishing Antonyms and Synonyms
in a Pattern-based Neural Network.” In Proceedings of the 15th Conference of the European
Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain,
April 3–7, 2017, Volume 1: Long Papers, pp. 76–85.
Nickel, M. and Kiela, D. (2017). “Poincaré Embeddings for Learning Hierarchical Representations.” In Advances in Neural Information Processing Systems 30: Annual Conference on
Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA,
pp. 6338–6347.
Nickel, M., Tresp, V., and Kriegel, H.-P. (2011). “A Three-Way Model for Collective Learning
on Multi-Relational Data.” In Getoor, L. and Scheﬀer, T. (Eds.), Proceedings of the 28th
International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA,
June 28–July 2, 2011, pp. 809–816. Omnipress.
Pennington, J., Socher, R., and Manning, C. D. (2014). “Glove: Global Vectors for Word Representation.” In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25–29, 2014, Doha, Qatar, A meeting of SIGDAT,
a Special Interest Group of the ACL, pp. 1532–1543.
Soricut, R. and Och, F. J. (2015). “Unsupervised Morphology Induction Using Word Embeddings.” In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31–June 5, 2015, pp. 1627–1637.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014).
“Dropout: A Simple Way to Prevent Neural Networks from Overﬁtting.” Journal fo Machine
Learning Research, 15 (1), pp. 1929–1958.
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., and Bouchard, G. (2016). “Complex Embeddings for Simple Link Prediction.” In Balcan, M.-F. and Weinberger, K. Q. (Eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York
City, NY, USA, June 19–24, 2016, Vol. 48 of JMLR Workshop and Conference Proceedings,
pp. 2071–2080. JMLR.org.
Vilnis, L. and McCallum, A. (2015). “Word Representations via Gaussian Embedding.” In 3rd
International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA,May 7–9, 2015, Conference Track Proceedings.
Yang, Z., Chen, W., Wang, F., and Xu, B. (2018). “Unsupervised Neural Machine Translation
with Weight Sharing.” In Gurevych, I. and Miyao, Y. (Eds.), Proceedings of the 56th Annual
232IA0248_09isibasi (2021-03-05 14:09)

Ishibashi et al.

Reﬂection-based Word Attribute Transfer

Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia,
July 15–20, 2018, Volume 1: Long Papers, pp. 46–55. Association for Computational Linguistics.
Zhao, J., Zhou, Y., Li, Z., Wang, W., and Chang, K.-W. (2018). “Learning Gender-Neutral
Word Embeddings.” In Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, Brussels, Belgium, October 31–November 4, 2018, pp. 4847–4853.

Yoichi Ishibashi: He is a Ph.D. student at the Nara Institute of Science andTechnology. He received his bachelor’s degree in computer science from Kyoto
Sangyo University in 2018 and his master’s degree in engineering from Nara
Institute of Science and Technology in 2020.
Katsuhito Sudoh: He is an associate professor at the Nara Institute of Scienceand Technology. He received his bachelor’s degree in engineering in 2000 andhis master’s and Ph.D. degrees in informatics in 2002 and 2015, respectively,from Kyoto University. He worked with the NTT Communication Science
Laboratories from 2002 to 2017. He currently works on machine translationand natural language processing. He is a member of ACL, ANLP, IPSJ, andASJ.
Koichiro Yoshino: He received his B.A. degree in 2009 from Keio University,
M.S. degree in informatics in 2011, and Ph.D. degree in informatics in 2014
from Kyoto University. From 2014 to 2015, he was a research fellow (PD)of Japan Society for Promotion of Science. From 2015, he has worked as
an assistant professor in Nara Institute of Science and Technology (NAIST).
He is currently working on areas of spoken and natural language processing,
especially on spoken dialogue systems. Dr. Koichiro Yoshino received the JSAISIG Research Award in 2013 and ANLP outstanding paper award in 2018. He
is a member of IEEE, SIGDIAL, ACL, IPSJ, RSJ, and ANLP.
Satoshi Nakamura: He is a professor at the Nara Institute of Science and Technology, Team Leader of the Tourism Information Analytics Team, AIP Center,
RIKEN, and Honorar professor at the Karlsruhe Institute of Technology, Germany. He received his B.S. from the Kyoto Institute of Technology in 1981 anda Ph.D. from the Kyoto University in 1992. He was the Director of ATR Spoken Language Communication Research Laboratories in the period 2000–2008233

IA0248_09isibasi (2021-03-05 14:09)

Journal of Natural Language Processing

Vol. 28 No. 1

March 2021

and Vice President of ATR in the period 2007–2008. He was the Director General of Keihanna Research Laboratories and Executive Director of Knowledge
Creating Communication Research Center, National Institute of Information
and Communications Technology in 2009–2010. He is currently the directorof the Augmented Human Communication laboratory and a full professor at
the Data Science Center and Information Science Division, Graduate Schoolof Science and Technology, Nara Institute of Science and Technology. He wasan Elected Board Member of the International Speech Communication Association, ISCA, in the period June 2011-2019, and IEEE Signal Processing
Magazine Editorial Board member in 2012-2015, IEEE SPS Speech and Language Technical Committee Member in 2013-2015. He is an ATR Fellow, IPSJ
Fellow, IEEE Fellow, and ISCA Fellow.
(Received July 30, 2020)(Revised November 20, 2020)(Accepted December 26, 2020)234