Reflection-based Word Attribute Transfer
Authors: Yoichi Ishibashi
Katsuhito Sudoh
Koichiro Yoshino
Satoshi Nakamura
Venue:  (2020)
Keywords: embedding
representation learning
analogy
geometry

Abstract: We propose a word attribute transfer framework based on reflection to obtain a word vector with an inverted target attribute for a given word in a word embedding space. Word embeddings based on Pointwise Mutual Information (PMI) represent such analogic relations as king - man + woman \approx queen. These relations can be used for changing a word’s attribute from king to queen by changing its gender. This attribute transfer can be performed by subtracting a difference vector man - woman from king when we have explicit knowledge of the gender of given word king. However, this knowledge cannot be developed for various words and attributes in practice. For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it. In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice. We introduce a framework based on reflection mapping that satisfies this property; queen should be transferred back to king with the same mapping as the transfer from king to queen. Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.

Under review as a conference paper at ICLR 2020
REFLECTION-BASER WORD ATTRIBUTE TRANSFER
Anonymous authors Paper under double-blind review
ABSTRACT
We propose word attribute transfer framework based on reflection to obtain a word vector with an inverted target attribute for given word in a word embedding space. Word embeddings based on Pointwise Mutual Information (PMI) represent such analogic relations as king man + woman can be used for changing a word's attribute from king to queen by changing its gender. This attribute transfer can performed by subtracting a difference vector man woman from king when we have explicit knowledge of the gender of given word king. However, this knowledge cannot be developed for various words and attributes in practice. For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it. In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice. We introduce framework based on reflection mapping that satisfies this property; queen should be transferred hack to king with the same mapping as the transfer from king queen. Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.
queeñ. These relations
INTRODUCTION
Distributed representation (Hinton al. 1984 is kind of data representation that capture data similarities in a vector space. In natural language processing, various studies have been conducted on word embeddings (Mikolov et al.] 2013a Pennington al. [2014] Peters al.. 2018 Bojanowski ct al.
2017). Word embedding models, such as skip-gram with negative sampling (SGNS) (Mikolov
2013b) or GloVe (Pennington et al. 2014), capture some analogic relations, such king al.
+ woman x queen. Previous work offer theoretical explanation based on Pointwise Mutual Information (PMI; Church & Hanks
mar
(1990)) for maintaining the analogic relations word vectors
(Levy & Goldberg 2014b Arora el al., 2016 Gittens et al. 2017 Ethayarajh et al. 2019: Allen &
Hospedales] 2019).
These relations can be used for transferring certain attribute of word, such as changing king into queen by transferring the gender. This task. which is called word attribute transfer. enables us to rewrite He is a boy as She is a girl. Word attribute transfer is expected to be applicable for natural language inference and data augmentation in natural language processing. The above analogic relations can be used, including udding difference vector woman-ma -mni to king to transfer
king to queen. This operation requires the explicit knowledge whether an input word is male or female; we have to add a difference vector to male word and subtract it from female word for gender transfer. We also have avoid changing words without any gender attributes, such is and in the example above. Thus, analogy-based word attribute transfer requires explicit knowledge of word attributes, such as king is male, queen is female, and is has no gender attribute. Developing such knowledge is very difficult for various words and attributes in practice.
In this study, we propose a novel framework based on reflection, which enables word attribute trans- fer by single reflection-based mapping for certain attribute. Reflection in geometry is mapping that exchanges the locations of two vectors in a Euclidean space by a hyperplane called a mirror, which satisfies the above desired property: working as identity mapping when it is applied twice and when it is applied to vectors on the mirror. We apply this reflection mapping to the problem of word attribute transfer by estimating appropriate mirror that maps word pairs with binary target

---

Under review as a conference paper at ICLR 2020
attribute (e.g., male and female) and keeps the other words without that attribute using training data. We also extend this approach by introducing parameterized mirrors, which work as different mir- rors based on the given imput words, overcome limitation using single fixed mirror represent complex transfer mappings for different words. Experimental results show that the reflection-basse method enables such transfers, achieves comparable performance analogy- malogy--based methods with explicit attribute knowledge, even though our proposed method does not use such knowledge.
The following the contributions of this paper:
We propose novel representation learning framework that obtains a vector with an inverted attribute in embedding space without explicit attribute knowledge of the given word.
Our proposed reflection-bas based word attribute transfer enables us transfer word attributes in up to 76% for words with target attributes and to avoid changing words without target attributes in over 99% our experiments.
2 WORD ATTRIBUTE TRANSFER
T
Invert gender
t
mother
father
fathers
Invert singular/plural
Figure 1: Given word v. and attribute one-hot vector z, word attribute transfer predicts word vi. which the inverted attribute of v...
Let æ denote a word and let v denote its vector representation. Here we assume that v. is learned in advance with an embedding model such as SGNS. In this task, we have two inputs, word and one- hot vector z, representing certain target attribute, and one output, return word t with the inverted attribute of æ for z. Let A denote set triplet (x, z). e.g., (man, woman, &gender) E A. Let ' denote set of words without attribute e.g., apple E W, when z represents gender. The purpose of this task is transfer Va to V. by transfer function fz that inverts attribute z of Vz. In other words, output Vy should be close to the vector of corresponding target word which is typically the nearest neighbor of v, in the word embedding space.
V1 F (v)).
Note that mapping fz transfers word æ if it has target attribute otherwise fz works as identity mapping. For instance with Rgenddr. given input word man, gender attribute transfer [zyender (Vman) should result in vector close Vuoman- Given input word apple as æ, the results should be Vapple-
ANALOGY-BASED WORD ATTRIBUTE TRANSFER
Analogy is a general idea for realizing attribute transfer. Several PMI-based word embedding meth- ods (Mikolov el al.) 2013c Linzen 2016) tackled to embed words into word embedding space to embedded vector with SGNS or GloVe captures analogic relations
capture the analogic relations.
(Levy & Goldberg, 2014a| Mikolov et al. 2013c| Linzen 2016 For instance, Vqueen is near the vector obtained on the right side of Eq. By rearranging Eq. Eq. is obtained:
Yqueen
V Vinng
Vman
Vwoman1
Vking (V
Vwoman).
(3)
We can transfer the gender attribute by subtracting difference vector Vman Vwomnn from input word vectors, e.g., Vking- The analogy-based transfer function is
V2 d if M,
fz(v.)
(4)
Vz +
if a FF
2

---

The following the Under review as a conference paper at ICLR 2020
where d is difference vector of the given word pair such and woman, M is a set of words having a target attribute, and F is set words having an inverse attribute, for example, man E M and woman E FF for gender attributes. Eq. 4 indicates that the operation changes depending whether input word x belongs to MM or F. For gender words, we subtract difference vector d is male, and add it if x is female. Therefore, we need such explicit knowledge. However, this knowledge cannot be developed for various words and attributes in practice.
4 REFLECTION-BASER WORD ATTRIBUTE TRANSFER
4.1 IDEALIZED TRANSFER WITHOUT EXPLICIT KNOWLEDGE
What is idealized transfer function oz for the word attribute transfer? idealized natures of such a transfer function:
Vm (v(Vw
(5)
V <z(Vm))
where m E M and w E F. This function oz enables to transfer a word without explicit knowledge. Function Øz transfers Vm V and Va to Vm without such explicit knowledge MM and w E F. By combining Eqs. and 6 we obtain the following formula:
Vmn E M,,
oz Oz (Vm) )
and
Yw E F, Vi = dz (vz(Vec)
(8)
Hence, the idcalized transfer function is mapping that becomes identity mapping when apply twice for any v. Such a mapping is called involution in geometry. For example, Φ: v + -v is one example of involution. Note that the identity map itself, such as o: y is excluded from the involution.
4.2 REFLECTION
reflection is an involution that reverses the location between two vectors in Euclidean space through an affine hyperplane (mirror). Reflection is an idcalized function because every point returns its original location when reflection applied twice:
Vv E R" v =Refa,cc el Refa,c(v)).
(9)
Given vector v in Euclidean space R", the formula for the reflection in the mirror is given:
Refa.c(v) = a
a,
(10)
where E Ih is vector orthogonal to the mirror (normal vector) and c E IR is a point through which the mirror passes. a and c are parameters that determine the mirror.
4.3 REFLECTION-BANE BASED WORD ATTRIBUTE TRANSFER
We apply reflection to the word attribute transfer to invert specific attribute of input word without its explicit attribute knowledge. We learn a mirror (hyperplane) in pre-trained embedding space using training word pairs with common (binary) attribute (Fig. 2). Here since the mirror is uniquely determined by two parameter vectors, a and c, we estimate a and c from target attribute z using fully connected multi-layer perceptrons: a MLP(z).
(11)
= MLP(z).
(12)
Transferred vector v,, is obtained by inverting attribute of a by reflection:
V, = Refa.c(v.).
(13)
3

---

Under review as a conference paper at ICLR 2020
queen
Refaclv)
bananas,
Refac(v)
Mirror
Mirror
apples
banana
king
oranges
apple
man
orange
(a) Male ++ Female
(b) Singular ++ Plural
Figure 2: Reflection-basce word attribute transfer examples.
sister*
mother
sister
mother
queen
fa
queen
Refac(v)
brother
woman
Mirror
brother
woman
Mirrors
father
father
king
king
man
man
actress
actor
actor
heroine
heroine
hero
hero
(a) Single mirror
(b) Parameterized mirrors
Figure 3: Mirror estimation methods
4.4 PARAMETERIZED MIRRORS
Reflection with a mirror by Eqs. and assumes a single mirror depending only on z. Previous discussion assumed that there will be pairs sharing a stable attribute such king and queen. However, often gendered words don't t come in pairs, and gender far from a stable attribute. For example, actress may be feminine, but actor is clearly neutral in many cases (Fig. 3 Thus, actor isn't as obvious a masculine counterpart as king. In fact, is known that there are biases in gender words in the embedding space (Zhao al., 2018 Kaneko & Bollegala 2019). This phenomenon can occur not only with the gender attribute, but also with other attributes. With this assumption of a single mirror, the mirror must be hyperplane that goes through the midpoints for all word vector pairs. However, the vector pairs shown on the left of Fig. 3 cannot be transferred well since the single mirror does not satisfy this constraint due the bias the embedding space. To solve this problem, we introduce different mirrors for different words. We propose parameterized mirrors determined by input vector v in addition to attribute z. The following are the definitions of the mirror parameters:
a MLP(22; vz]);
(14)
c = ML.((z.vl),,
(15)
where | indicates the vector concatenation in the column. The parameterized mirrors are expected work flexibly on different words. For instance, shown in Fig. 3 suppose learned the mirror (the blue line) that transfers Vhero to Vheroine in advance. If input word vector Vactor resembles Vhero- mirror that is similar to the one for Vhero should be derived and used for the attribute transfer.
4.5 LOSS FUNCTION
Loss function L is defined:
L())= = A
Σ
(Vy.
v
Σ (Vy, EE
V2,)²,
(16)
W
4

---

Under review as a conference paper at ICLR 2020
v2,)² is term that draws target word vector Vt, closer sponding transferred vector V and [VT En,ex((ys, -va,) term that prevents words without target attribute from being moved by transfer function fz. θ represents the set of all the trainable parameters. The parameters in the proposed model are the MLP weights used to determine mirror hyperplanes via and c. We iteratively update θ to minimize C:
where [A] L(..,,,
4 arg min ((O,),
(17)
where t is the number of parameter updates at that time.
5 EXPERIMENT
We evaluated the performance of the proposed reflection-based word attribute transfer using data with some different attributes.
5.1 EXPERIMENTAL. SETUP
We used three different datasets of word pairs with three binary attributes: Male- Stale-Female| Singular- Plural and Capital-Country, shown Table word pairs were collected from analogy test sets (Mikolov et al. 2013a Gladkova et al. 20166), the Internet and Nguyen et al. (2017) for antonyms of nour Since these datasets are very small, we added Gaussian noise (a 0.1) to every input vector v. during training avoid overfit. Random noise was applied independently every sample in every iteration. For a non-attribute dataset N, we sampled words from the three-milion vocabulary of the word embedding model. We sampled from 4 to 50 words for training (0 < 50) and 1000 words for the test ((Nesst = 1000). We used a mixed dataset that included sampled and excluded words from Wirain and Arrain- We had no A because the tuning was conducted with only |Aval-- We used word2vec (Mikolov al. 2013b) and GloVe (Pennington et al. 2014 the pre-trained embedding model. The embedded vector dimension is n 300.
|Nurain S
both |Ntrain and |AAerain Note that Werst
Table 1: Statistics of binary attribute word pair datasets (in the number of word pairs).
Dataset A
Train
Val
Test
Total
Male-Female (MF)
29
12
12
53
Singular-Plural (SP)
90
25
25
140
Capital-Counte (CC)
59
25
25
109
Antonym (AN)
1354
290
290
1934
5.2 EVALUATION METRICS
We measured the accuracy and stability performances of the word attribute transfer. The accuracy measures how many input words in Aust were transferred correctly to the corresponding target words. The stability score measures how many words in Nust are not mapped other words. For example, in a gender transfer, given man, the transfer is regarded correct if woman is the closest word to the transferred vector; otherwise it is incorrect. Given apple, the transfer is regarded as correct if apple is the closest word the transferred vector; otherwise its stability is incorrect. Here we used cosine similarity to measure the similarity of output vector Vy and target vector Vt. The accuracy and stability scores are calculated by the following formula:
if arg max REV
V. VV =
8(v 4)
(18)
otherwise,
Note that these gender word pairs are an assumption because they contain socially problematic words. "These datasets and the experimental codes will be released the future.
"https://code.goog""" wtt.gg/.ccooaaccc/////wwtttto thups::///lp.aa httpns///lp.ssanfood.ddoo www.aaffoot.duuumpiisss/eoeeeee
5

---

Under review as a conference paper at ICLR 2020
Accuracy
Σ
8(Vertts)
(19)
Arest (...
Stability
Σ
(20)
W i.eeN
where V is the vocabulary of the word embedding model.
5.3 METHODS AND CONHGURRATINSS
In our experiment, we compared proposed method with the following baseline methods:
REF Reflection-based word attribute transfer with a single mirror. We used fully connected 2-layer MLP with 300 hidden units and ReLU activations to estimate a and c.
REF+ PM Reflection-baxee word attribute transfer with parameterized mirrors. We used the same MLP as the REF.
MLP
Fully connected MLP: Vy MLP((v=; The highest accuracy models are 2-layer MLP for Capital-Counte and 3-layer MLP for the other datasets.
DIFF
Analogy-based word attribute transfer with difference vector: d = V. Vas where and w are the training data A.. We chose d because achieved the best accuracy in the validation data of A. We determined whether add subtract d to v, based on attribute knowledge (Eq. 4.
DIFF
Analogy-basee based word attribute transfer with a difference vector regardless of the attribute knowledge. d obtained in the same way the DIFF. We added d to va for any input v: (z((v)) ) Vz + d, vV,, E R".
DIEF
Analogy- based word attribute transfer with difference vector regardless the attribute knowledge. dl was obtained in the same way as the DIFF. We subtracted d from v. for any input x: fz(v2>) = Va - d, Vv, E RR.
MEANDIFF Analogy-bar based word attribute transfer with a mean difference vector d: d
Avaini Em,,, DE.AArrin (Vm, Vw.). We determined whether to add or subtract d based on the attribute knowledge (Eq
MEANDIFF Analogy-basee based word attribute transfer with mean difference vector regardless of the attribute knowledge: fz(vz) V2 + Vv, E R", d was obtained in the same way the MEANDIFF.
MEANDIFF
Analogy- based word attribute transfer with mean difference vector regardless of the attribute knowledge: fz(vz) = V2 d. Vv, E R", d was obtained in the same way the MEANDIFF.
Based on the wning, we used the Adam optimizer (Kingma & Ba 2015) with learning rate of 10-1 (the other hyperparameters were the same as the original one (Kingma Ba 2015)), ) and batchsize of 62 for male-female. and 32 for the others. These hyperparameters were identical for the lcarning-based methods: REF, REF PM, or MLP. We did not use such regularization methods as dropout (Srivastava et al. 2014) batch normalization (loffe & Szegedy 2015) because they did not show any improvement in our pilot test.
5.4 ACCURACY AND STABILITY
Table 2 and 3 shows the transfer accuracy and stability score results. Both experiments using GloVe or word2vec obtained similar results. REF + PM achieved the best accuracy among the methods that did not use explicit attribute knowledge. This means that reflection can be used for word attribute transfers even without attribute knowledge. In the stability evaluation, reflection-based methods (REF, REF + PM) and the analogy- based methods with a mean difference vector (MEANDIFF MEANDIFF ") achieved high stability. In particular, reflection-basee stability scores exceeding 99%. The stability DIFF and DIFF was much lower than the other methods. Although MEANDIFF and MEANDIFF achieved high stability, their accuracy results
transfers achieved outstanding

---

Under review as a conference paper at ICLR 2020
Table 2: Results in accuracy and stability scores (word2vec).
Accuracy (%)
Stability (%)
Method
Knowledge
MF
SP
CC
AN
MF
SP
CC
AN
REF
20.83
0.00
36.00
0.00
99.80
100.00
99.80
100.00
REF + PM
41.67
22.00
58.00
28.79
99.90
99.40
99.40
100.00
MLP
8.33
4.00
12.00
35.86
2,20
0,00
2.70
1.90
DIFF
25.00
2.00
32.00
72.10
77.90
53.90
DIFF
25.00
2,00
30.00
49.60
78.20
56.30
MEANDIFF
4.17
0.00
22.00
98.60
99.40
87.60
MEANDIFF
8.33
0.00
14.00
97.20
99.30
92.40
DIFF MEANDIFF
/
62.50
4.00
64.00
12.50
0.00
36.00
Table 3: Results in accuracy and stability scores (GloVe).
Accuracy (%)
Stability (%)
Method
Knowledge
MF
SP
AN
MF
SP
AN
REF
12.50
2.00
26.00
0.00
100.00
100.00
100.00
100.00
REF + PM
45,83
50.00
76.00
33.54
99.70
99.10
99.20
100.00
MLP
4.17
10.00
18.00
36.72
5.10
7.00
5.20
1.20
DIEF
25.00
2.00
26.00
99.30
94.20
99.30
DIFF
25.00
2.00
24.00
100.60
99.90
99.50
MEANDIFF
0.00
0.00
22.00
100.00
100.00
100.00
MEANDIFF
0.00
0.00
0.00
100.00
100.00
100.00
DIFF
50.00
4.00
44.00
MEANDIFF
0.00
0.00
0.00
were very low. Interestingly, reflection based transfer with parameterized mirrors (REF + PM) achieved high performance in both accuracy and stability. For example, the accuracy of REF + PM was 41.67%, and the stability was 99.9% in Male-Female (MF), and the accuracy was 58 and the stability 99.40 % in Capital-Country (CC). These results show that the proposed method transfers an input word if it has target attribute and does not transfer an input word even though it does not use explicit attribute knowledge the input words. MLP worked poorly both in accuracy and stability. In the antonym (AN), while the transfer accuracy by the proposed method was a bit lower than that by MLP, the stability of the proposed method was 100% and that of MLP was really poor (almost 0%).
We investigated the relation between the size of |Nirain| and the stability of learning-based methods by conducting additional experiment by varying [Wrrain| from O to 50. The stability scores by MLP did not improve (Table 4). On the other hand, REF and REF PM achieved high stability scores with just [Atrain| and maintained the accuracy.
5.5 TRANSFER EXAMPLE
Table 5 shows examples of a gender transfer at the sentence level, where the attribute transfer was applied to words in sentence cabulary of the original word embedding model, we omitted them from the inputs. MLP made many wrong transfers on words without gender attributes, e.g., the became By Katie Klingsporn, was be- came she, when became Doughhy_Evening CCrrrnicchrr,,e,eee and woman became girlfriend. DIFFt can transfer if x is female, e.g., transferred from woman to man, but could not transfer grandfather and boy. Similarly, DIFF failed to transfer from female to male. In addition, since the stability of these methods was low, they erroneously transferred. For example, in DIFF ", the and when became she. REF + PM can selectively transfer words with gender attribute without using explicit gender information. For example, when woman was given, Ref (Vmaman) became man without knowledge that woman is female word, and when was given, it became woman. When non-attribute word
{#1, ....... Here since such words as a and are not in the vo-

---

Under review as a conference paper at ICLR 2020
Table 4: Relation among size of Warain and stability of learning based methods.
Accuracy (%)
Stability (%)
Varsin
[Nirain]
10
50
0
4
50
REF
16.67
20.83
20.83
20.83
98.30
98.80
99.40
99.80
MF
REF
PM
41.67
45.83
20.83
41.67
38.30
98.40
100,00
99.90
MI.P
4.17
8.33
8.33
8.33
0.00
0.30
0.30
2.20
REF
0.00
0.00
0.00
0.00
99.90
99.90
99.90
100.90
SP
REF PM
12,00
22.00
18.00
18.00
98.40
99.40
99.30
99.80
ML.P
4.00
4.00
2.00
2.00
0.00
0.00
0.10
3.40
REF
36.00
36.00
36.00
34.00
99.80
99.80
99.80
100.00
CC
REF PM
58.00
56.00
58.00
54.00
73.80
99.70
99.40
99,40
MLP
6.00
6,00
8.00
12.00
0.00
0.30
0.50
2.70
REF
0.00
0.00
0.00
99.90
99.90
99.90
99.80
AN
REF - PM
21.72
27.24
28.62
28.79
95.30
99.20
99.50
99.80
MLP
34.14
35.00
34.31
35,86
0.00
0.01
0.02
1.90
married was given, Ref(v (Vmarriea) became married without knowledge that married has no gender attribute. When we applied the reflection-bart transfer twice, the transferred word returned to its original word, e.g., Re (Vee(vaaaa))) gives woman.
Table 5: Transfer results when sentence X = (the, .... boy) was given. Out-of-vocabulary words a and. were not given input.
X
the woman was married when your grandfather was (a) boy (.)
Ref(x)
the man was married when your grandmother was (a) girl (.)
(Re((e(())
the woman was married when your grandfather was (a) boy (.)
By SlatieeKiinsspom girlfriend she fiancee
ML.
Doughty Evening Chronicle ma' am daughter she (a) mother (.)
DIFF
the man was married when your grandfather was (a) boy (.)
DIFF
she woman married she your grandmother was (a) girl (.)
We transfer some different attributes of words with reflection-basee -bsed transfer one-by-one. Table 6 shows that the words having different target attributes were transferred by each reflection-based transfer in the order of Male-Female, Singular-Phun
and Country-Capital. Given actress for Male-Female transfer, it was transferred to actor and to actors for Singular-Plurat Given Tokyo for Male-Female, Singular-Plure Pluraa,, and Antonym, it was transferred, but it was transferred Japan for
Given rich for Male-Female, Singular-Plural and Capital -outttr-- it was not transferred, but it was transferred to poor for Antonym.
Table 6: Transfer of different attributes with rellection-based word attribute transfer with parameterized mir- ror..
X
the rich actress and the poor actor to stay the beautiful city in Tokyo.
+ Male-Female
the rich actor and the poor actress stay the beautiful city in Tokyo.
+ Singular-Plural
the rich actors and the poor actresses want to stay the beautiful cities in Tokyo.
+ Copital-Country
the rich actors and the poor actresses want stay the beautiful citie in Japan.
+ Antonym
the poor actors and the rich actresses want to stay the beautiful cities in Japan.
8

---

Under review as a conference paper at ICLR 2020
6 RELATED WORK
The embedded vectors obtained by SGNS (Mikolov et al. 2013a b) and GloVe (Pennington al. 2014) have analogic relations. The theory of analogic relations in word embeddings has been widely discussed: Levy & Goldberg (2014b); Arora al. (2016); Gittens et al. (2017); Ethayarajh et al.
(2019): AAlen & Hospedales (2019); Linzen 2016). Levy & Goldberg (2014b) offer the explanation that SGNS factorizes a shifted PMI matrix. Allen & Hospedales (2019) and Ethayarajh et al. (2019)
argued that they proved the existence of such analogic relations without strong assumptions. In our work, focus on the analogic relations in word embedding space and propose a novel framework to obtain a word vector with inverted attributes. Style transfers (Niu et al. 2018 Prabhumoye et al.,
2018 Jain al. 2019 Logeswaran al. 2018 Dai et al. 2019 Zhang al.) 2018) resemble our task. In a style transfer, the text style of the imput sentences is changed. For instance, Jain et al. (2019) transferred from formal informal sentences, [oogeswaran al. (2018) transferred sentences by controlling such attributes mood and tense. These style transfer tasks use sentence pairs; our word attribute transfer task uses word pairs. Style transfer changes sentence styles, but our task changes the word attributes (contents). [Soricut & Och (2015) studied the problem of morphological transformation based on character information. Our work aims more general attribute transfer such as gender transfer and country-cape and is not limited the morphological transformation.
7 CONCLUSION AND FUTURE WORK
We proposed a novel representation learning framework based on reflection to invert certain at- tribute of a word vector. We proposed a reflection-based method for word attribute transfers with- out relying the explicit attribute knowledge of an input word, which is necessary for a simple analogy-based transfer. Experimental results showed that our proposed method can transfer the word attributes if the input word has target attribute. If not, reflection does not transfer the word.
Future work includes applications to other transfer tasks: sentence by sentence transfer, such Niu et al. (2018): PPrabhumoye et al. (2018); Jain et al. (2019), and entity prediction
analogic graph embedding space (Liu et al.] [211)). ) in the field of computer vision, visual analogy (Reed et al. 2015), or style transfer (Zhu et al.; 2017 Liao et al. (017) with GANs (Radford al. 2016 Goodfellow et al. 2014) because their latent space holds analogic relations (Radford et al. 2016
REFERENCES
Carl Allen and Timothy M. Hospedales. Analogies Explained: Towards Understanding Word Em- beddings. In Proceedings of the 36th International Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. //prcced ngs. ml wt....l..pp.......ee.......eeeee www.ppressvvvv v97/av/99/// lenl 9a. html
2019. URL http:
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A Latent Variable Model Approach to PMI-based Word Embeddings. Transactions of the Association for Compu- tational Linguisticss, 4:385-399, 2016.
/index.///e..////ee phpp
tacl/article/viese
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching Word Vec- tors with Subword Information. Transactions of the Association for Computational Linguisties, 5:135-146, 2017. view/999
Kenneth Ward Church and Patrick Hanks. Word Association Norms, Mutual Information, and Lex- icography. Computational Linguistics, 16(1):22-29, 1990.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. Style Transformer:
Unpaired Text Style Transfer without Disentangled Latent Representation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. pp. 5997-6007, 2019. URL https: //t::///wwww.. aclweb. org/
Pothology//P9-160

---

Under review as a conference paper at ICLR 2020
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards Understanding Linear Word Analogies. In Proceedings of the 57th Conference of the Association for Computational Linguis- fics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3253-3262, 2019. URLbttps:: wwwp//wwwwoooo....o ac web. or org/ att.aaathhhooggo 9- 1315/
Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney. Skip-Gram Zipf + Uniform Vector Additivity. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 69-76, 2017. doi: 10.18653//1/P111 1/117-1077 BR https: http:///eciiogggoo..o.oo.. 11..:./111.1.11131///111.1.1 1007
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy- detection of morpho- logical and semantic relations with word embeddings: what works and what doesn't. In Pro- ceedings of the Student Research Workshop. NeW@HLT-NAAC. 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, San Diego California, USA, June 12-17, 2016, pp. 8-15, 2016. URL
https:// /tw::///www... ac wwwwwwlwwbbb.......... www.l/ewbbogggrr......... wergg/anthlloooooo /0og//N66--- N166-20027
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neu- ral Information Processing Systems 27: Anmual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2672-2680, 2014. URL n www.p.c.... cc/ http:/aapperr/ee -ewe::2ggeeerr.iveeeeeee aelveeevdvvraarrr -nets
httt:://
Geoffrey E Hinton, James I. McClelland, David E Rumelhart, et al. Distributed representations. Carnegie-Mellon University Pittshurgh, PA, 1984.
Sergey Toffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 448-456, 2015. URL http:: //proceedings.. mlr
html
Parag Jain, Abhijit Mishra, Amar Prakash Azad, and Karthik Sankaranarayanan. Unsupervised Controllable Text Formalization. In The Thirty-Third AAAI Conference on Artificial Intelligence,
AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pp. 6554-6561, 2019. URL https:
7/aaai. ojs index. hhtp:/AAt//ariicle///ee
Masahiro Kaneko and Danushka Bollegala. Gender Gender-preserving Debiasing for Pre-trained Word Em beddings. In Proceedings of the 57th Conference of the Association for Computational Linguis- fics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 1641-1650, 2019. [RL.tttppe /ww.w//wommoooom web. org/ wwt.aaooooooyy//oo 9- 1160/
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015, UtL/htpp: ott.tt:::/wxxx..oggg/ooooo abs 1412. 6980
Omer Levy and Yoav Goldberg. Linguistic Regularities in Sparse and Explicit Word Representa- tions. In Proceedings of the Eighteenth Conference on Computational Natural Language Learn- ing, CONLL 2014, Baltimore, Maryland, USA, June 26-27, 2014, pp. 171-180, 2014 URL https:/ / www.ww.wwww.boo... hotool /oloo//////111111111
Omer Levy and Yoav Goldberg. Neural Word Embedding as Implicit Matrix Factor- ization. In Advances in Neural Information Processing Systems 27: Annual Confer- ence on Neural Information Processing Systems 2014, December 8-13 2014, Montreal,
Quebec, Canada, pp. 2177-2185, 2014b. URL http::/papprs.a.t..///// nips cc/paper/ 1-77-neural-wozer wewoorddemmeddding-geee cetss-impiiiii www.iiatiiiiiiii www.trix--actttrzzaaiiiio
Jing Liao, Yuan Yao. Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. ACM Trans. Graph.. 36(4): 120:1120::11:1:111 2017. doi: 105//457907.95. 3073683. URL. htt.:tttt////t.m .orillorg/////oo
10

---

Under review as a conference paper at ICLR 2020
Tal Linzen. Issues in evaluating semantic spaces using word analogies. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP. RepEval@ 2016, Berlin, Germany, August 2016, pp. 13-18, 2016. doi: 11111////111116555500011 URL https://doi.
org/10 100.18655.0 181568633/30 /vvl / W1 6- 2503
Hanxiao Liu, Yuexin Wu, and Yiming Yang. Analogical Inference for Multi-relational Embeddings. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Syd- ney, NSW, Australia, 6-11 August 2017, pp. 2168-2178, 2017. WtLLhtt::// htt.:httppppoooo httttp:p/prroeediggss.
mlr. www..press/vil 1iull1110 7d.
Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. In Advances in Neural Information Processing Systems 31: Annual
Conference on Neural Information Processing Systems 2018, NewrIPS 2018, 3-8 December 2018, Montréal, Canada., pp. 5108 5118, 2018. URLAttt::/paapes..oe. ntpp. text gewerattionwiieeee
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Repre- sentations in Vector Space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a. URL
http://arxiv.om org abs/
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionaalty. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Pro- cessing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pp. 3111-3119, 2013b. URL/tttp:/ppppes.niii
cc/paper/
1021-distributee representationnnnooeeeee inns of words and phrases and their compositional
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space
Word Representations. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pp. 746-751, 2013c. URL https: /t//://wwww
aclweb. org/
Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. Distinguishing Antonyms and Synonyms in a Satter--bsed Neural Network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties, FACI EACI.2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pp. 76-85, 2017. URL https: /www/www.....owwwww...... aclweb.
Xing Niu, Sudha Rao, and Marine Carpuat. Multi-task neural models for translating between styles within and across languages. In Proceedings of the 27th International Conference on Computa- tional Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018 pp- 1008- 1021, 2018. httpst / www. ac web. org. http//anhholoolcccoooo 11t///1111111111
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL. pp. 1532-1543, 2014. URL lttpt::///// www.ww aclweb.
org/
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pp. 2227-2237, 2018. URI. https://////.......
web. orgg/
anthology/NI8-1220
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. Style Transfer Through Back- Translaaanl In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 866-876, 876, 2018. doi: 10.18653/v1/P18- 1080. URL / www. aclweb.or. antho Btchology/PP8-1000
11

---

Under review as a conference paper at ICLR 2020
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceed- ings, 2016. URL httt:://aaxiv..rrr. www.oorg///b//oooooooeo......oae /111//11111 511. 06434
Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep Visual Analogy- Making, In Advances in Neural Information Processing Systems 28: Annual Confer- ence on Neural Information Processing Systems 2015, December 7-12, 2015. Montreal,
Quebec, Canada, pp. 1252-1260, 2015. URL http::/papers: n ps cc/paper/ sua visaa ana a ogy-making
5845-deep-ye vi
Radu Soricut and Franz Josef Och. Unsupervised Morphology Induction Using Word Embeddings. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Denver, Colorado, USA, May
June 5, 2015, pp. 1627-1637, 2015. URL htt.http::// htttpps:/www........ /wwww wawww.aclll web. www.oeee..////oooooooooo//o
N15 11867
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting J. Mach. Learn. Res., 15(1): 1929-1958, 2014. [tthttp: acm. org/ciiatiooonn.ooooooooo cfm?id=2mmmm
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang. Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. Style Transfer as Unsupervised Machine Translation. CoRR, abs/ 11818888877 2018. URL
07894
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. Learning Gender-Neutral Word Embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 4847-4853, 2018. URL
https:
ftttp::/wwww.cl... http://wwwwwllllc.cc org.org///aooooo wtg/annbooogg//11--3333
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired Image-to-Image Trans- lation Using Cycle-Consistent Adversarial Networks. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2242-2251, 2017. doi: 10.1100//CCV22017.2444, URL https:://////.. doii oww.lo.rg11...0 1109 ICCV. 2017 244
12

---

Under review as a conference paper at ICLR 2020
A
TOP THREE ACCURACY AND STABILITY
Table 7: Male-Female
Accuracy (%)
Stability (%)
Method
Knowledge
Mean@3
@
@2
@3
Mean @
@
@2
@
REF
50.00
20.83
62.50
66.67
99.80
99.80
99.80
99.80
REF + PM
55.56
41.67
58.33
66.67
99.13
99.00
99.20
99.20
MLP
20.83
8.33
20.83
33.33
2.20
2.20
2.20
2.20
DIFF
31.94
25.00
33.33
37.50
75.43
72.10
75.80
78.40
DIFF
31.94
25.00
33.33
37.50
55.80
49.60
57.30
60.50
MEANDIFF MEANDIFF
23.61
4.17
33.33
33.33
98.93
98.60
99.10
99.10
23.61
8.33
29.17
33.33
97.63
97.20
97.80
97.90
DIFF
68.05
62.50
66.66
75.00
MEANDIFF
47.22
12.50
62.50
66.67
Table 8: Singular-Plurat
Accuracy (%)
Stability (%)
Method
Knowledge
Mean @
@
@22
@
Mcan @
@ 1
@22
@
REF
50.00
0.00
72.00
78.00
100.00
100.00
100.00
100.00
REF PM
56.67
22.00
72.00
76.00
99.70
99.40
99.80
99.90
MLP
7.33
4.00
8.00
10.00
0.13
0.00
0.20
0.20
DIFF +
36.67
2.00
50.00
58.00
80.27
77.90
79.70
83.20
DIFF
36.67
2.00
52.00
56.00
80.27
78.20
80.70
81.90
MEANDIFF +
42,00
0.00
56.00
70.00
99.47
99.40
99.40
99.60
MEANDIFF
39.33
0.00
56.00
62.00
99.50
99.30
99.60
99.60
DIFF
49.33
4.00
70.00
74.00
MEANDIFF
52.00
0.00
76.00
80.00
Table 9: Capital-Country
Accuracy (%)
Stability (%)
Method
Knowledge
Mean 3
@
@
Mcan @
@
@2
@
REF
66.67
36.00
78.00
86,00
99,80
99.80
99.80
99.80
REF +
72.67
58.00
74.00
86,00
99.53
99.40
99.60
99,60
MLP
35.33
12.00
40.00
54.00
2.80
2.70
2.80
2.90
DIFF
39.33
32.00
42.00
44.00
64.87
53.90
69.60
71.10
DIFF
34.67
30.00
36.00
38.00
58.63
56.30
58.90
60.70
MEANDIFF
36.00
22.00
42,00
44.00
89.33
87.60
89.70
90.70
MEANDIFF
34.67
14.00
44.00
46.00
93.07
92.40
93.10
93.70
DIEF
80.00
64.00
86.00
90.00
MEANDIFF
70.67
36.00
86.00
90.00
13

---

Under review as a conference paper at ICLR 2020
Table 10: Antonym
Accuracy (%)
Stability (%)
Method
Knowledge
Mean @
@2
3
Mcan @
@ 1
@2
REF REF + PM
0.94
0.00
1.19
16.21
99.97
99.90
100.00
100.00
38. 56
28.79
41.38
45.52
99.93
99.80
100.00
100.00
MLP
41.38
35.86
42.59
45.69
1.997
1.90
2.00
2.00
B VISUALIZATION OF PARAMETERIZED MIRRORS
Figures below visualize PCA results of a obtained for the test words. We normalized the L2 norm of a to 1 ||a] Corresponding word pairs are connected by solid lines. Figs. and 5 suggest not only the mirror parameters of paired words are similar to each other but also the parameters with the attribute form a cluster words with the same attribute has similar mirror parameter The mirrors of paired words are close to each other in the same attribute (Fig. 4 aand 5}. Some MF pairs in Fig. are placed away from the cluster of the MF words. This may come from missing principal components due to the small data size used for PCA. Figs. 6 7
12 and 13 are the detailed PCA results for four different attributes: MF, SP, CC, and AN. These results show that reflection transfers a paired word each other by using similar mirror. For example, rich and poor use almost the same mirror (Fig. 9). On the other hand, different mirrors are used for different word pairs since the mirrors are parameterized. These results shows the effect of the mirror as described section
9
0.75
0.2
0.00
0.25
-0.50
1.00
0.75
Figure PCA result of (word2vec).
1.00
MF
SP
0.75
CC
AN
0.60
0.25
0.25
1..00
-076
-025
0.50
025
0.75
1.00
Figure 5: A PCA result of (Glo Ve).
14

---

Under review as a conference paper at ICLR 2020
ctor
MF
0.4
0.2
randfather
wughter
0.0
ctress
-0.4
0.4
0.6
Figure fale-Femmae (word2vee)
nember
0.4
hirts
grup
bus
0.0
Jeschers
Juacher gillggs
ailاa Rospiaa
nembers
pospital
-0.2
Juses
guides
gestaurant
0.4
-0.4
-02
0.0
0.2
0.4
0.6
Figure 7: Singular-Plurdd (word2vec)
CC
einut
Burundi
0.3
Swndd
Jllamabadi
Canbern mr
0.2
Alcarage Nivo
Creasks
Dlusk
6.0
Useeti
Syris
-0.1
Retttian
amb
and
Ankara
Tibbaboe
-0.2
iiplii
Veneaa
skuu
uustraloa
-0.4
-0.2
0.0
Figure 8: Capital-Country (word2vec)
15

---

Under review as a conference paper at ICLR 2020
66
AN
0.4
Rabess
departuure privvll
0.0
galfrergence
agunnditty
cooinng
pearside
gurity couuiiy
shickness
-0.4
-0.4
0.0
Figure 9: Antonym (word2vec)
MF
grandfather
0.2
empero
gother
emopss
tandmotthr
00
-0.4
-033
-02
-0.1
0.0
0.2
0.3
0.4
Figure 10: Male-Female (GloVo
SP
golls
artists
0.2
deachers
ebjects
abirt
dighhits pain
artist
@bjjet
0.0
buskoes
program
programs
ddi
-0.2
gestaunant
Juses
story
Sawronts
gun
-0.4
-0.2
Figure 11: Singular-Plure Plural (GloVe)
16

---

Under review as a conference paper at ICLR 2020
jabul
fghnniittnn
CC
0.4
johran
0.3
pakistan
bungkok
solwon
0.1
owndd
haka
agurg
aracaa
baka
iev
acsucisaloo
inillnn
tockhaim
-01
tir spus
budarest wonden
engola
selegesccr
czbelisttn
coatti
nareb
bungery
-0.2
0.0
0.4
Figure 12: Capital-Country (Glo
ousfield
jubility
0.2
ingeligaben
Goofingtng
maad
0.0
geivergenee
ogy
impotence
porality
mmmoriiity
-0.4
dugisss
-0.4
-0.2
0.0
0.2
0.5
Figure 13: Antonym (Glo Vc)
17


Reviews:
[{"review_id": "ryetZIM0KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a method for binary word attribute transfer based on reflection. It applies a single reflection-based mapping that relates the locations of two vectors in a Euclidean space by a hyperplane, and results in an identity mapping when it is applied twice. It's sort of like roundtrip-translation, but for word vector attributes. The paper also proposes a pretty smart idea: the mirror functions are parameterized to take advantage of the fact that inversions differ, even for the \"same\" word attributes. \n\nI am giving the paper a weak accept, because I think idea is really fun, and definitely has legs. I like the idea for this paper quite a bit, it really got me thinking, but I think it isn't quite structured how I would like. Unfortunately, as it is, it also feels a bit spare in terms of contributions. I would like much more discussion/analysis of the parameterizations (and their parameters), or treatment of more binary-attribute linguistic analogy types (antonymy, scalar terms), or both! Another thing that could be added is comparisons across types of word vectors (word2vec alone is a bit narrow). Any of those things would have justified a strong accept. \n\nSuggestions: \n- As I was reading, I was not at all convinced by the \"idealness\" of your transfer function in 4.1 (maybe change it from \"ideal\" to \"idealized\"). \"Ideal\" assumes that there will be pairs sharing a stable attribute. But, often \"gendered\" words don't come in pairs, and \"gender\" is far from a stable attribute. To be more specific, despite your Fig 3a: \"actress\" may be feminine, but \"actor\" is clearly neutral (anyone can be described as an \"actor\" but only women can be described as \"actress\" in most cases); thus \"actress\" is the gendered one. We can go ahead and pair \"actress\" with \"actor\"---they do share a morphological relationship after all---but \"actor\" isn't as obvious a masculine counterpart as \"king\" is for \"queen\". How would you square this with phi(queen)=king and phi(king)=queen, where the identity of counterparts is clearer? Do we ideally want the same invertible method here for both actor-actress and queen-king (I think I don't, hence the function is not \"ideal\"). I wouldn't be surprised if the same situation arises in other analogies. This is, I think, why you brought up parameterization. But you miss the opportunity to clearly motivate why we need to parameterize! You should include more motivation and lay the linguistic facts out in a clearer way that incorporates examples like the one I gave, and preferably before S4.1.\n-The total number of words you chose to use is pretty small, which you acknowledge. I would've loved to see a few more phenomena, perhaps antonymy, part-whole, morphological reinflection (e.g., a generalization of your sing-pl set, datasets exist for past-tense at least), etc. \n-it would have been neat to compare word2vec to other types of vector embeddings, maybe contextualized? They are what everyone's using right now (this suggestion didn't factor heavily into my rating though). Also the framing in your introduction made me think this is where you were going (you started with discussing vectors, as opposed to just jumping in on binary word attribute transfer. I think the paper could just jump right in with binary word attribute transfer, and skip the basic vectors paragraph).\n-I would've liked more discussion of your parameterized mirrors, since that was the neatest/prettiest part, in my opinion. (maybe I missed something, but how did you decide on how many z to use? or did every pair get its own? Do your zs for each attribute correlate at all? you could use something straightforward like CCA to check, would be neat.)\n\nSmall Notes/Musings:\n-on p.1 \"(PMI (citation...\" with nested parentheses is not easy to read, I prefer \"(PMI; citation)\".\n- It is fine to operate on the assumption that gender is binary for now, but you should acknowledge that binary gender is an assumption (and a socially problematic one), that isn't real.\n-in 5.5, I'm not impressed by \"When non-attribute word the was given, Ref(v the) became 'the' without knowledge that the has no gender attribute\", because \"the\" is the most frequent word in English, perhaps show that it works with rare, non-gendered words instead. Incidentally, does it know syntactic category? Because that might be a neat thing to check.\n-what do you think about trinary attributes (cold, warm, hot)? It would be a cool extension.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxXKwzCYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of transferring word attributes between words.  There is a lot of discussion of\nelementary ideas such as reflection about an affine plain and involutions.  The key technical idea in the paper is given in the definition of the objective function in equation (16).  The notation in this equation is unfortunate as the parameter vector Theta does not appear in the loss expression and some guesswork is needed to realize that y_i is hat{t}_\\Theta(x_i).  The discussion of optimization seems strange --- why not just apply a standard optimizer to this loss function?  The experimental results are limited to three attributes one of which --- the capital/country attribute --- seems a relation not an attribute.\n\nBut the most serious problem with this paper is a lack of references to related work.  I would start with the following reference and track papers that reference it.\n\n\"Unsupervised Morphology Induction Using Word Embeddings\" by Radu Soricut, Franz Och, NAACL 2015.\n\nPostscript: I have read the rebuttal and looked at the revised paper. The citation I suggested has been added but with inadequate acknowledgement.  Two of four attributes studied in the revised paper are morphological (gender is morphological in many languages) and the country-capital relation and antonym relation do not seem to be attributes to me.\n\nOn other hand, the results do look promising.  Based entirely on the results I have raised my score weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xS1S3aFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "If I understood correctly, in experimental part pre-trained embeddings, taken from word2vec, is a ground truth. Given those embeddings, a system of hyperplanes are trained (every hyperplane refers to a certain word attribute and a region in space, centered around a word, to which reflection is applied).\n\nIn my opinion, the most natural way to check \"the reflection hypothesis\" is to train new embeddings where to word2vec objective the loss function (16) is added and to look how perplexity will behave with that additional cost and to look how your accuracy and stability will behave on the test set. \n\nIt is also interesting to learn how this reflection based attribute transfer can be applied to the same word, but with embeddings that play different role in a model: e.g. input and output embeddings. \nIn fact, input and output embeddings are located in the same space, they can be considered as pairs of words with 1 attribute flipped (i.e. role in a model, input->output). Can an output embedding be obtained from an input embedding via reflections that you describe. How many hyperplanes we need in that case? This is an interesting edge of the problem. In simplest models, there is some evidence, that output embeddings are result of reflections of input embeddings in half the dimensions. It would be interesting to learn your comment on that.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a way to transform word vectors based on a binary attribute (e.g. male/female) based on reflection, with the property that applying the reflection operator twice, the vector for a word is left unchanged.  By identifying parameterized mirror planes for each word, the proposed method can leave neutral words left unchanged.

The paper received 3 weak accepts.  There was initially one reject, but the revisions convinced the reviewer to update their score to a weak accept.  Overall, the reviewers appreciated the idea of reflection-based binary word attribute transfer.   suggestions, the authors made small improvements to the writing, added missing citations, as well as additional results for another word embedding (GloVE) and another dataset (antonyms).  One of the main remaining weakness of the work, is still the small dataset.  Although somewhat alleviated by the inclusion of the antonym dataset, this is still a weakness of the paper.  

The AC agrees that the paper has an nice idea and is well presented.  However, the work is limited in scope and is likely to be of limited interest to the ICLR community and would be more appreciated in the NLP community.  The authors are encouraged to improve upon the work, and resubmit to an appropriate venue.  

Decision: Reject