Reflection-based Word Attribute Transfer

Yoichi Ishibashi, Katsuhito Sudoh, Koichiro Yoshino, Satoshi Nakamura
Nara Institute of Science and Technology{ishibashi.yoichi.ir3, sudoh, koichiro, s-nakamura}@is.naist.jp

Abstract
Word embeddings, which often represent such−−→ −→ −−−−−→
analogic relations as king − −man+ woman ≈−−−→ can be used to change a word’s atqueen,
tribute, including its gender. For transferring
king into queen in this analogy-based manner,
−→ −
−−−−→
we subtract a difference vector −
man−
woman
based on the knowledge that king is male.
However, developing such knowledge is very
costly for words and attributes. In this work,
we propose a novel method for word attributetransfer based on reflection mappings without
such an analogy operation. Experimental results show that our proposed method can transfer the word attributes of the given words without changing the words that do not have thetarget attributes.

1

Introduction

Word-embedding methods handle word semantics
in natural language processing (Mikolov et al.,2013a,b; Pennington et al., 2014; Vilnis and McCallum, 2015; Bojanowski et al., 2017). Such wordembedding models as skip-gram with negative sampling (SGNS; Mikolov et al., 2013b) or GloVe (Pennington et al., 2014) capture such analogic relations−−→ −→ −−−−−→ −−−→
as king− −man+ woman ≈ queen. Previous work
(Levy and Goldberg, 2014b; Arora et al., 2016; Gittens et al., 2017; Ethayarajh et al., 2019; Allenand Hospedales, 2019) offers theoretical explanation based on Pointwise Mutual Information (PMI;
Church and Hanks, 1990) for maintaining analogic
relations in word vectors.
These relations can be used to transfer a certain attribute of a word, such as changing king into
queen by transferring its gender. This transfer canbe applied to perform data augmentation; for example, rewriting He is a boy to She is a girl. It canbe used to generate negative examples for natural
language inference, for example. We tackle a novelxz
Invert genderfatherInvert singular/pluraltmotherfathers

Figure 1: Examples of word attribute transfer

task that transfers any word associated with certain
attributes: word attribute transfer.
A naive way for word attribute transfer is to use a
difference vector based on analogic relations, such−→−−−−→ − −−→ to −−−→
as adding −
woman
man
king to obtain −queen.
This requires explicit knowledge whether an input
word is male or female; we have to add a difference
vector to a male word and subtract it from a female
word for the gender transfer. We also have to avoid
changing words without gender attributes, suchas is and a in the example above, since they are
non-attribute words. Developing such knowledge
is very costly for words and attributes in practice.
In this work, we propose a novel framework for aword attribute transfer based on reflection that does
not require explicit knowledge of the given words
in its prediction.
The contribution of this work is two-fold: (1) We
propose a word attribute transfer method that obtains a vector with an inverted binary attribute without explicit knowledge. (2) The proposed method
demonstrates more accurate word attribute transfer for words that have target attributes than otherbaselines without changing the words that do nothave the target attributes.

2

Word Attribute Transfer Task

In this task, we focus on modeling the binary attributes (e.g. male and female1 ). Let x denotea word and let vx denote its vector representation. We assume that vx is learned in advance1

Gender-specific words are sometimes considered sociallyproblematic. Here we use this as an example from the manwoman relation.

with an embedding model, such as skip-gram. In
this task, we have two inputs, word x and vector z, which represent a certain target attribute,and output word t with the inverted attribute ofx for z. In this paper, z is a 300-dimensional
vector embedded from a target attribute ID using an embedding function of a deep learning
framework. For example, given a set of attributes
Z = {gender, antonym}, we assign different random vectors zgender for gender and zantonym forantonym, respectively. Let A denote a set of triplets(x, t, z), e.g., (man, woman, zgender ) ∈ A, andN denote a set of words without attribute z, e.g.,(person, zgender ) ∈ N . This task transfers input
word vector vx to target word vector vt by transfer
function fz that inverts attribute z of vx :vt ≈ vy = fz (vx ).

(1)

The following property must be satisfied: (1) attribute words {x|(x, t, z) ∈ A} are transferredto their counterparts and (2) non-attribute words{x|(x, z) ∈ N } are not changed (transferredback into themselves). For instance with zgender ,given input word man, gender attribute transfer
fzgender (vman ) should result in a vector close to
vwoman . Given another input word person as x,
the results should be vperson .

3

Analogy-based Word AttributeTransfer

Analogy is a general idea that can be used for word
attribute transfer. PMI-based word embedding,such as SGNS and GloVe, captures analogic relations, including Eq. 2 (Mikolov et al., 2013c;
Levy and Goldberg, 2014a; Linzen, 2016). By rearranging Eq. 2, Eq. 3 is obtained:
vqueen ≈ vking − vman + vwoman ,(2)≈ vking − (vman − vwoman ). (3)The analogy-based transfer function is(vx − dif x ∈ M,
fz (vx ) =vx + dif x ∈ F,(4)

where M is a set of words with a target attribute(e.g., male) and F is a set of words with an inverse
attribute (e.g., female). d is a difference vector,
such as vman − vwoman . Eq. 4 indicates thatthe operation changes depending on whether input
word x belongs to M or F. However, to transfer

the word attribute by analogy, we need such explicit
knowledge as attribute value (M, F or others) thatis contained by the input word.

4

Reflection-based Word AttributeTransfer

4.1

Ideal Transfer without Knowledge

What is ideal transfer function fz for a word attribute transfer? The following are the ideal naturesof such a transfer function:∀(m, w, z) ∈ A,

vm = fz (vw ),(5)∀(m, w, z) ∈ A,

vw = fz (vm ),(6)∀(u, z) ∈ N ,vu = fz (vu ).

(7)This function fz enables a word to be transferredwithout explicit knowledge because operation fz
does not change depending on whether input word
belongs to M or F. By combining Eqs. 5, 6 and 7,
we obtain the following formulas:∀(m, w, z) ∈ A,

vm = fz ( fz (vm ) ), (8)∀(m, w, z) ∈ A,

vw = fz ( fz (vw ) ), (9)∀(u, z) ∈ N ,vu = fz ( fz (vu ) ). (10)Hence, the ideal transfer function is a mapping that
becomes an identity mapping when we apply it
twice for any v. Such a mapping is called involution in geometry. For example, f : v 7→ −v is one
example of an involution.
4.2

Reflection

Reflection Ref a,c is an ideal function because thismapping is an involution:∀v ∈ Rn ,v = Ref a,c ( Ref a,c (v) ). (11)Reflection reverses the location between two vectors in a Euclidean space through an hyperplane
called a mirror. Reflection is different from inverse
mapping. When m and w are paired words, reflection can transfer vm and vw each other withidentical reflection mapping as in Eqs. 5 and 6,
but an inverse mapping cannot. Given vector v in
Euclidean space Rn , the formula for the reflectionin the mirror is given:
Ref a,c (v) = v − 2(v − c) · a
a,
a·a

(12)

where a ∈ Rn is a vector orthogonal to the mirror
and c ∈ Rn is a point through which the mirror
passes. a and c are parameters that determine themirror.

4.3

Proposed method: Reflection-based
Word Attribute Transfer
queen
Ref a,c (v)woman

Mirror
a

king

c

man

VectorFigure 2: Reflection-based word attribute transfer witha single mirror

We apply reflection to the word attribute transfer.
We learn a mirror (hyperplane) in a pre-trained
embedding space using training word pairs withbinary attribute z (Fig. 2). Since the mirror is
uniquely determined by two parameter vectors, aand c, we estimate a and c from target attribute z
using fully connected multi-layer perceptrons:a = MLPθ1 (z),(13)

c = MLPθ2 (z),(14)

where θ is a set of trainable parameters of MLPθ .
Here, θ1 and θ2 are optimized for each attributedataset. Transferred vector vy is obtained by inverting attribute z of vx by reflection:vy = Ref a,c (vx ).
sistermother

queen

king

Reflection

woman

brotherfather(15)

Mirrors

person

man
actor
hero

actress
heroineFigure 3: Reflection with parameterized mirrors

Reflection with a mirror by Eqs. 13 and 14 assumes a single mirror that only depends on z. Previous discussion assumed pairs that share a stablepair, such as king and queen. However, since gendered words often do not come in pairs, gender is
not stable enough to be modeled by a single mirror. For example, although actress is exclusively
feminine, actor is clearly neutral in many cases.
Thus, actor is not obviously a masculine counterpart like king. In fact, bias exists in gender words
in the embedding space (Zhao et al., 2018; Kaneko
and Bollegala, 2019). This phenomenon can occur
not only with gender attributes but also with otherattributes. The single mirror assumption forces themirror to be a hyperplane that goes through the
midpoints for all the word vector pairs. However,the vector pair actor-actress shown on the right in
Fig. 3 cannot be transferred well since the single
mirror (the green line) does not satisfy this constraint due to the bias of the embedding space. To
solve this problem, we propose parameterized mirrors, based on the idea of using different mirrors
for different words. We define mirror parameters
a and c using word vector vx to be transferred inaddition to attribute vector z:
a = MLPθ1 ([z; vx ]),(16)

c = MLPθ2 ([z; vx ]),(17)

where [·; ·] indicates the vector concatenation in thecolumn. The parameterized mirrors are expected towork more flexibly on different words than a single
mirror because parameterized mirrors dynamicallydetermine similar mirrors for similar words. For
instance, as shown in Fig. 3, suppose we learned
the mirror (the blue line) that transfers vhero tovheroine in advance. If input word vector vactor
resembles vhero , a mirror that resembles the one forvhero should be derived and used for the transfer.
On the other hand, the reflection works as an
identity mapping for a vector on the mirror (e.g.,vperson in Fig 3). That is, the proposed method
assumes that non-attribute word vectors are locatedon the mirror. Since we used a 300-dimensional
embedded space in the experiment, we assume
that the non-attribute word vector exists in a 299dimensional subspace.
Here, it should be noted that Eq. 11 may not holdfor parameterized mirrors. In reflection with a single mirror, it is true that v = Ref a,c ( Ref a,c (v)).
However, with the v-parameterized reflection
Ref av ,cv (v), this is not guaranteed. Because mirror parameters av and cv depend on an input word
vector as Eqs. 16 and 17. Thus, we exclude thisconstraint and employ the constraints given by Eqs.
5-7 for our loss function.
The following property must be satisfied in word
attribute transfer: (1) words with attribute z aretransferred and (2) words without it are not transferred. Thus, loss L(θ1 , θ2 ) is defined:
L(θ1 , θ2 ) =

1
|A|

1
+
|N |

X

(vy − vt )2(18)(vy − vx )2 ,(19)

(x,t,z)∈AX(x,z)∈N

where Eq. 18 is a term that draws target word vector
vti closer to corresponding transferred vector vyi
and Eq. 19 is a term that prevents words without
a target attribute from being moved by transfer
function. vy is the output of a reflection (Eq. 15).

5

Experiment

We evaluated the performance of the word attributetransfer using data with four different attributes.
We used 300-dimensional word2vec and GloVe as
the pre-trained word embedding. We used four
different datasets of word pairs with four binary
attributes: Male-Female, Singular-Plural, CapitalCountry, and Antonym (Table 1). These word pairs
were collected from analogy test sets (Mikolov
et al., 2013a; Gladkova et al., 2016) and the Internet. Noun antonyms were taken from the literature(Nguyen et al., 2017). For non-attribute dataset N ,we sampled words from the vocabulary of wordembedding. We sampled from 4 to 50 words fortraining and 1000 for the test (|Ntest | = 1000).
Table 1: Statistics of binary attribute word pair datasets(in number of word pairs)
Dataset A

Train

Val

Test

Total

Male-Female (MF)
Singular-Plural (SP)Capital-Country (CC)
Antonym (AN)

29
90
59
1354

12
25
25
290

12
25
25
290

53
140
109
1934

5.1

Evaluation Metrics

We measured the accuracy and stability performances of the word attribute transfer. The accuracy
measures how many input words in Atest weretransferred correctly to the corresponding targetwords. The stability score measures how many
words in Ntest were not mapped to other words.
For example, in the Male-Female transfer, givenman, the transfer is regarded as correct if woman
is the closest word to the transferred vector; otherwise it is incorrect. Given person, the transfer
is regarded as correct if person is the closest word
to the transferred vector; otherwise it is incorrect.
The accuracy and stability scores are calculated by
the following formula:δ(vy , t) =


1

if

0

otherwise,arg max(cos(vy , vk )) = t
k∈V

(20)

Accuracy =1|Atest |Stability =Xδ(vy , t),(21)(x,t,z)∈Atest

1|Ntest |Xδ(vy , x), (22)(x,z)∈Ntest

where V is the vocabulary of the word embedding
model and cos(vy , vk ) is the cosine similarity meav ·v
sure, defined as: cos(vy , vk ) = kvyykkvkk k .
5.2

Methods and Configurations

In our experiment, we compared our proposed
method with the following baseline methods2 :R EF Reflection-based word attribute transfer witha single mirror. We used a fully connected2-layer MLP with 300 hidden units and ReLU
(Glorot et al., 2011) to estimate a and c.
R EF+PM Reflection-based word attribute transfer with parameterized mirrors. We used thesame architecture of MLP as the R EF.
MLP Fully connected MLP with 300 hidden units
and ReLU: vy = MLP([vx ; z]). The highest
accuracy models in SGNS are a 2-layer MLP
for Capital-Country and 3-layer MLP for theother datasets. The highest accuracy models in GloVe are a 2-layer MLP for SingularPlural and 3-layer MLP for the other datasets.
D IFF Analogy-based word attribute transfer witha difference vector: d = vm − vw , where m
and w are in the training data of A. We chosed that achieved the best accuracy in the validation data of A. We determined whether to
add or subtract d to vx based on the explicit
knowledge (Eq. 4). D IFF+ and D IFF− transfer with a difference vector regardless of the
explicit knowledge. + and − add or subtract
the difference vector to any input word vector.
M EAN D IFF Analogy-based word attribute transfer withPa mean difference vector d̄: d̄ =1
(mi ,wi ,z)∈Atrain (vmi −vwi ). We de|Atrain |termined whether to add or subtract d̄ to vx
based on the explicit knowledge (Eq. 4).
For proposed methods, we used the Adam optimizer (Kingma and Ba, 2015) with α = 10−4 forMale-Female, Singular-Plural and Capital-Country,2
Our code and datasets are available at: https://github.com/ahclab/reflection

Table 2 shows the accuracy and stability results.
Different pre-trained word embeddings GloVe or
word2vec gave similar results. R EF+PM achieved
the best accuracy among the methods that did notuse explicit attribute knowledge. For example, theaccuracy of R EF+PM was 76% in Capital-Country,
but the accuracy of D IFF+ was 26%. For stability,reflection-based transfers achieved outstanding stability scores that exceeded 99%. The results showthat our proposed method transfers an input word
if it has a target attribute and does not transfer an
input word with better score than the baselines otherwise, even though the proposed method does notuse attribute knowledge of the input words. MLP
worked poorly both in accuracy and stability. On
the antonym dataset, although the transfer accuracy
by the proposed method was a bit lower than thatby MLP, the proposed methods stability was 100%and that of MLP was extremely poor: about 1%.
We investigated the relation between the training
data size of the non-attribute words, and the stability of the learning-based methods by conducting
an additional experiment that varied |Ntrain |. The
stability scores by MLP did not improve (Table 3).
On the other hand, R EF+PM achieved high stability scores with |Ntrain | = 0 and maintained theaccuracy. We hypothesized that the high stability
came from the distance between the word and its
mirror. If non-attribute words are distributed on
the mirror, they will not be transferred. We investigated the distance between input word vector vx
and its mirror (Fig. 4). The result shows that nonattribute words are close to the mirror, and attributewords are distributed away from it. In Male-Female
and Singular-Plural, the distance is not significantly
farther than Antonym and Capital-Country. If the
distance between paired words is very small, thedistance between the word and its mirror is also
small. Fig. 5 shows the distribution of the distance between input vx and target word vector vt .

Visualization of Parameterized Mirrors

Figure 6 shows the t-SNE results of mirror parameter a obtained for the test words. Paired mirror,(ax , at ), is connected by a line segment. Fig. 6
suggests that the mirror parameters of the paired
words are similar to each other and that those withthe attribute form a cluster; words with the same
attribute have similar mirror parameters a.
Attribute word
1.5
Density

Evaluation in Accuracy and Stability

5.4

Non-attribute word
1.5

Male-Female

1.0

1.0

0.5

0.5

0.0

1

1.5
Density

5.3

The distance of Male-Female and Singular-Plural ismuch smaller than Capital-Country and Antonym.

0

1

2

3

4

5

0.0

1.0

0.5

0.5
1

0

1

Singular-Plural1

1.5

Capital-Country

1.0

0.0

2 3
Distance

4

5

0.0

0

1

2

3

4

5

2 3
Distance

45Antonym101Figure 4: Distribution of distance between input word
vector and its mirror |(vxk−ack)·a| learned by R EF+PM.
Non-attribute words are close to the mirror, and attribute words are distributed away from it.

Male-Female
Singular-Plural

Capital-Country
Antonym

1.0
0.8

Densityand α = 15−3 for Antonym (the other hyperparameters were the same as the original one (Kingma
and Ba, 2015)). We did not use such regularization methods as dropout (Srivastava et al., 2014)
or batch normalization (Ioffe and Szegedy, 2015)
because they did not show any improvement in ourpilot test. We implemented R EF, R EF+PM and
MLP with Chainer (Tokui et al., 2019), which isone of the best deep learning frameworks.

0.6
0.4
0.2
0.0

0

2

4

6
Distance

8

10

12

Figure 5: Distribution of distance between input word
vector vx and target word vector vt

MF
SP
CC
AN

10

5

0

5

10

15

10

5

0

5

10

15

Figure 6: Two-dimensional t-SNE projection of a

Table 2: Results in accuracy and stability scores: MF, SP, CC, and AN are datasets.
word2vec
MethodKnowledge

GloVe

Accuracy (%)
MF
SP
CC AN

MF

Stability (%)
SP
CC

20.8
41.7
8.3
25.0
25.0
4.2
8.3

0.0
22.0
4.0
2.0
2.0
0.0
0.0

36.0
58.0
12.0
32.0
30.0
22.0
14.0

0.0
28.8
35.9
-

99.8
99.9
2.2
72.1
49.6
98.6
97.2

100.0
99.4
0.0
77.9
78.2
99.4
99.3

99.8
99.4
2.7
53.9
56.3
87.6
92.4

100.0
100.0
1.9
-

12.5
45.8
4.2
25.0
25.0
0.0
0.0

2.0
50.0
10.0
2.0
2.0
0.0
0.0

26.0
76.0
18.0
26.0
24.0
22.0
0.0

0.0
33.5
36.7
-

100.0
99.7
5.1
99.3
100.0
100.0
100.0

100.0
99.1
7.0
94.2
99.9
100.0
100.0

100.0
99.2
5.2
99.3
99.5
100.0
100.0

100.0
100.0
1.2
-

62.5
12.5

4.0
0.0

64.0
36.0

-

-

-

-

-

50.0
0.0

4.0
0.0

44.0
0.0

-

-

-

-

-

Table 3: Relation among size of |Ntrain | and stabilityof learning-based methods

6

R EF
R EF+PMMLP
D IFF +
D IFF −M EAN D IFF +M EAN D IFF −D IFF
M EAN D IFF

X
X0

Accuracy (%)

Stability (%)|Ntrain |4
10|Ntrain |
4
10

50

0

MF

R EF
R EF+PM
MLP

12.5
45.8
0.0

12.5
41.7
4.2

12.5
37.5
0.0

12.5
41.7
4.2

100.0
99.7
0.0

100.0
99.9
0.4

100.0
99.9
1.0

100.0
99.9
5.0

SP

R EF
R EF+PM
MLP

0.0
48.0
4.0

0.0
40.0
6.0

2.0
50.0
6.0

0.0
46.0
10.0

100.0
53.3
0.0

100.0
99.1
0.5

100.0
99.1
1.7

100.0
99.8
7.0

CC

R EF
R EF+PM
MLP

24.0
76.0
16.0

26.0
72.0
10.0

24.0
74.0
14.0

20.0
74.0
18.0

100.0
99.2
0.0

100.0
100.0
0.4

100.0
99.9
1.0

100.0
99.9
5.2

AN

R EF
R EF+PM
MLP

0.0
26.9
29.5

0.0
26.7
29.7

0.0
33.5
36.7

0.0
25.7
36.6

100.0
100.0
0.1

100.0
100.0
0.5

100.0
100.0
1.2

100.0
100.0
4.6

5.5

50

Transfer Example

Table 4 shows the gender transfer results for a tiny
example sentence. Here the attribute transfer wasapplied to every word in the sentence. MLP mademany wrong transfers. Analogy-based transfers
can transfer only in one direction. R EF+PM cantransfer only attribute words. Table 5 shows thatwords with different target attributes were transferred by each reflection-based transfer.
Table 4: Comparison of gender transfers. Each method
transfers words in a sentence one by one.
X

the woman got married when you were a boy.

R EF
R EF+PMD IFF +
D IFF −

the woman got married when you were a boy.
the man got married when you were a girl.
the man got married when you were a boy.
she woman got married she you were a girl.
By Katie Klingsporn girlfriend Valerie Glodowski
fiancee Doughty Evening Chronicle ma’am
Bob Grossweiner & a mother.

MLP

Table 5: Transfer of different attributes with R EF+PMX

the rich actor wants to visit the beautiful city in tokyo.

+ MF
+ SP
+ CC
+ AN

the rich actress wants to visit the beautiful city in tokyo.
the rich actresses wants to visit the beautiful cities in tokyo.
the rich actresses wants to visit the beautiful cities in japan.
the poor actresses wants to visit the beautiful cities in japan.

AN

Accuracy (%)
MF
SP
CC AN

MF

Stability (%)
SP
CC

AN

Related Work

The theory of analogic relations in word embeddings has been widely discussed (Levy and Goldberg, 2014b; Arora et al., 2016; Gittens et al., 2017;
Ethayarajh et al., 2019; Allen and Hospedales,
2019; Linzen, 2016). In our work, we focus on theanalogic relations in a word embedding space andpropose a novel framework to obtain a word vector
with inverted attributes. The style transfer task (Niuet al., 2018; Prabhumoye et al., 2018; Logeswaran
et al., 2018; Jain et al., 2019; Dai et al., 2019; Lample et al., 2019) resembles ours. In style transfer,the text style of the input sentences is changed.
For instance, Jain et al. (2019) transferred fromformal to informal sentences. These style transfer
tasks use sentence pairs; our word attribute transfer task uses word pairs. Style transfer changes
sentence styles, but our task changes the word attributes. Soricut and Och (2015) studied morphological transformation based on character information. Our work aims for more general attributetransfer, such as gender transfer and antonym, andis not limited to morphological transformation.

7

Conclusion

This research aims to transfer word binary attributes (e.g., gender) for applications such as data
augmentation of a sentence. We can transfer theword attribute with analogy of word vectors, but it
requires explicit knowledge whether the input word
has the attribute or not (e.g., man ∈ gender, woman∈ gender, person ∈/ gender). The proposed method
transfers binary word attributes using reflectionbased mappings and keeps non-attribute words unchanged, without attribute knowledge in inference
time. The experimental results showed that the
proposed method outperforms analogy-based and
MLP baselines in transfer accuracy for attributewords and stability for non-attribute words.

References
Carl Allen and Timothy M. Hospedales. 2019. Analogies Explained: Towards Understanding Word Embeddings. In Proceedings of the 36th International
Conference on Machine Learning, ICML 2019, 915 June 2019, Long Beach, California, USA, pages
223–231.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma,
and Andrej Risteski. 2016. A Latent Variable Model
Approach to PMI-based Word Embeddings. Transactions of the Association for Computational Linguistics, 4:385–399.
Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching Word Vectors with
Subword Information. Transactions of the Association for Computational Linguistics, 5:135–146.
Kenneth Ward Church and Patrick Hanks. 1990. Word
Association Norms, Mutual Information, and Lexicography. Computational Linguistics, 16(1):22–29.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing
Huang. 2019. Style Transformer: Unpaired Text
Style Transfer without Disentangled Latent Representation. In Proceedings of the 57th Conference of
the Association for Computational Linguistics, ACL
2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5997–6007.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst.
2019. Towards Understanding Linear Word Analogies. In Proceedings of the 57th Conference of
the Association for Computational Linguistics, ACL
2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3253–3262.
Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney. 2017. Skip-Gram - Zipf + Uniform = Vector
Additivity. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,
ACL 2017, Vancouver, Canada, July 30 - August 4,
Volume 1: Long Papers, pages 69–76.
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the Student Research Workshop, SRW@HLTNAACL 2016, The 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies,
San Diego California, USA, June 12-17, 2016, pages
8–15.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio.
2011. Deep Sparse Rectifier Neural Networks. In
Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13,
2011, volume 15 of JMLR Proceedings, pages 315–
323. JMLR.org.

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by
Reducing Internal Covariate Shift. In Proceedingsof the 32nd International Conference on Machine
Learning, ICML 2015, Lille, France, 6-11 July 2015,
pages 448–456.
Parag Jain, Abhijit Mishra, Amar Prakash Azad, andKarthik Sankaranarayanan. 2019. Unsupervised
Controllable Text Formalization. In The ThirtyThird AAAI Conference on Artificial Intelligence,
AAAI 2019, The Thirty-First Innovative Applicationsof Artificial Intelligence Conference, IAAI 2019,
The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019.,
pages 6554–6561.
Masahiro Kaneko and Danushka Bollegala. 2019.
Gender-preserving Debiasing for Pre-trained Word
Embeddings. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers, pages 1641–1650.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
Method for Stochastic Optimization. In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015,
Conference Track Proceedings.
Guillaume
Lample,Sandeep
Subramanian,
Eric
MichaelSmith,
Ludovic
Denoyer,Marc’Aurelio Ranzato, and Y-Lan Boureau.
2019. Multiple-Attribute Text Rewriting. In 7thInternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9,
2019. OpenReview.net.
Omer Levy and Yoav Goldberg. 2014a. Linguistic
Regularities in Sparse and Explicit Word Representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning,
CoNLL 2014, Baltimore, Maryland, USA, June 2627, 2014, pages 171–180.
Omer Levy and Yoav Goldberg. 2014b. Neural Word
Embedding as Implicit Matrix Factorization. In Advances in Neural Information Processing Systems
27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2177–2185.
Tal Linzen. 2016. Issues in evaluating semantic spaces
using word analogies. In Proceedings of the 1st
Workshop on Evaluating Vector-Space Representations for NLP, RepEval@ACL 2016, Berlin, Germany, August 2016, pages 13–18.
Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. 2018. Content preserving text generation withattribute controls. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems
2018, NeurIPS 2018, 3-8 December 2018, Montréal,Canada., pages 5108–5118.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In 1st International Conference on Learning Representations, ICLR 2013,
Scottsdale, Arizona, USA, May 2-4, 2013, Workshop
Track Proceedings.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S.
Corrado, and Jeffrey Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information
Processing Systems 26: 27th Annual Conference on
Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013,
Lake Tahoe, Nevada, United States., pages 3111–
3119.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.
2013c. Linguistic Regularities in Continuous Space
Word Representations. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree
Plaza Hotel, Atlanta, Georgia, USA, pages 746–751.
Kim Anh Nguyen, Sabine Schulte im Walde, andNgoc Thang Vu. 2017. Distinguishing Antonyms
and Synonyms in a Pattern-based Neural Network.
In Proceedings of the 15th Conference of the European Chapter of the Association for Computational
Linguistics, EACL 2017, Valencia, Spain, April 3-7,
2017, Volume 1: Long Papers, pages 76–85.
Xing Niu, Sudha Rao, and Marine Carpuat. 2018.
Multi-Task Neural Models for Translating Between
Styles Within and Across Languages. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New
Mexico, USA, August 20-26, 2018, pages 1008–
1021.
Jeffrey Pennington, Richard Socher, and Christopher D.
Manning. 2014. Glove: Global Vectors for Word
Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language
Processing, EMNLP 2014, October 25-29, 2014,
Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. 2018. Style Transfer
Through Back-Translation. In Proceedings of the56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers,
pages 866–876.
Radu Soricut and Franz Josef Och. 2015. Unsupervised Morphology Induction Using Word Embeddings. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May
31 - June 5, 2015, pages 1627–1637.

Nitish Srivastava, Geoffrey E. Hinton, Alex
Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neuralnetworks from overfitting. J. Mach. Learn. Res.,
15(1):1929–1958.
Seiya Tokui, Ryosuke Okuta, Takuya Akiba, Yusuke
Niitani, Toru Ogawa, Shunta Saito, Shuji Suzuki,
Kota Uenishi, Brian Vogel, and Hiroyuki Yamazaki
Vincent. 2019. Chainer: A Deep Learning Framework for Accelerating the Research Cycle. In Proceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8,
2019, pages 2002–2011.
Luke Vilnis and Andrew McCallum. 2015. Word Representations via Gaussian Embedding. In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015,
Conference Track Proceedings.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and KaiWei Chang. 2018. Learning Gender-Neutral Word
Embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4847–4853.