Reflection-based Word Attribute Transfer

−−→ king − −−→man + −−−−−→woman ≈ −−−→queen . These relations can be used for changing a word ’ s attribute from king to queen by changing its gender . This attribute transfer can be performed by subtracting a difference vector −−→man − −−−−−→woman from −−→ king when we have explicit knowledge of the gender of given word king . However , this knowledge can not be developed for various words and attributes in practice . For transferring queen into king in this analogy-based manner , we need to know that queen denotes a female and add the difference vector to it . In this work , we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice . We introduce a framework based on reflection mapping that satisfies this property ; queen should be transferred back to king with the same mapping as the transfer from king to queen . Experimental results show that the proposed method can transfer the word attributes of the given words , and does not change the words that do not have the target attributes . 1 INTRODUCTION . Distributed representation ( Hinton et al. , 1984 ) is a kind of data representation that can capture data similarities in a vector space . In natural language processing , various studies have been conducted on word embeddings ( Mikolov et al. , 2013a ; b ; Pennington et al. , 2014 ; Peters et al. , 2018 ; Bojanowski et al. , 2017 ) . Word embedding models , such as skip-gram with negative sampling ( SGNS ) ( Mikolov et al. , 2013b ) or GloVe ( Pennington et al. , 2014 ) , capture some analogic relations , such as −−→ king −−−→man + −−−−−→woman ≈ −−−→queen . Previous work offer theoretical explanation based on Pointwise Mutual Information ( PMI ; Church & Hanks ( 1990 ) ) for maintaining the analogic relations in word vectors ( Levy & Goldberg , 2014b ; Arora et al. , 2016 ; Gittens et al. , 2017 ; Ethayarajh et al. , 2019 ; Allen & Hospedales , 2019 ) . These relations can be used for transferring a certain attribute of a word , such as changing king into queen by transferring the gender . This task , which is called word attribute transfer , enables us to rewrite He is a boy as She is a girl . Word attribute transfer is expected to be applicable for natural language inference and data augmentation in natural language processing . The above analogic relations can be used , including adding difference vector−−−−−→woman−−−→man to −−→ king to transfer −−→ king to −−−→queen . This operation requires the explicit knowledge whether an input word is male or female ; we have to add a difference vector to a male word and subtract it from a female word for a gender transfer . We also have to avoid changing words without any gender attributes , such as is and a in the example above . Thus , analogy-based word attribute transfer requires explicit knowledge of word attributes , such as king is male , queen is female , and is has no gender attribute . Developing such knowledge is very difficult for various words and attributes in practice . In this study , we propose a novel framework based on reflection , which enables word attribute transfer by a single reflection-based mapping for a certain attribute . Reflection in geometry is a mapping that exchanges the locations of two vectors in a Euclidean space by a hyperplane called a mirror , which satisfies the above desired property : working as identity mapping when it is applied twice and when it is applied to vectors on the mirror . We apply this reflection mapping to the problem of word attribute transfer by estimating an appropriate mirror that maps word pairs with a binary target attribute ( e.g. , male and female ) and keeps the other words without that attribute using training data . We also extend this approach by introducing parameterized mirrors , which work as different mirrors based on the given input words , to overcome a limitation using a single fixed mirror to represent complex transfer mappings for different words . Experimental results show that the reflection-based method enables such transfers , achieves comparable performance to analogy-based methods with explicit attribute knowledge , even though our proposed method does not use such knowledge . The following are the contributions of this paper : • We propose a novel representation learning framework that obtains a vector with an inverted attribute in embedding space without explicit attribute knowledge of the given word . • Our proposed reflection-based word attribute transfer enables us to transfer word attributes in up to 76 % for words with target attributes and to avoid changing words without target attributes in over 99 % in our experiments . 2 WORD ATTRIBUTE TRANSFER Let x denote a word and let vx denote its vector representation . Here we assume that vx is learned in advance with an embedding model such as SGNS . In this task , we have two inputs , word x and onehot vector z , representing a certain target attribute , and one output , return word t with the inverted attribute of x for z. LetA denote a set of a triplet ( x , t , z ) , e.g. , ( man , woman , zgender ) ∈ A. LetN denote a set of words without attribute z , e.g. , apple ∈ N , when z represents gender . The purpose of this task is to transfer vx to vt by transfer function fz that inverts attribute z of vx . In other words , output vy should be close to the vector of corresponding target word vt , which is typically the nearest neighbor of vt in the word embedding space . vt ≈ vy = fz ( vx ) . ( 1 ) Note that mapping fz transfers word x if it has target attribute z ; otherwise fz works as identity mapping . For instance with zgender , given input word man , gender attribute transfer fzgender ( vman ) should result in a vector close to vwoman . Given input word apple as x , the results should be vapple . 3 ANALOGY-BASED WORD ATTRIBUTE TRANSFER . Analogy is a general idea for realizing attribute transfer . Several PMI-based word embedding methods ( Mikolov et al. , 2013c ; Linzen , 2016 ) tackled to embed words into word embedding space to capture the analogic relations . An embedded vector with SGNS or GloVe captures analogic relations ( Levy & Goldberg , 2014a ; Mikolov et al. , 2013c ; Linzen , 2016 ) . For instance , vqueen is near the vector obtained on the right side of Eq . 2 . By rearranging Eq . 2 , Eq . 3 is obtained : vqueen ≈ vking − vman + vwoman , ( 2 ) ≈ vking − ( vman − vwoman ) . ( 3 ) We can transfer the gender attribute by subtracting difference vector vman − vwoman from input word vectors , e.g. , vking . The analogy-based transfer function is fz ( vx ) = { vx − d if x ∈M , vx + d if x ∈ F , ( 4 ) where d is a difference vector of the given word pair such as man and woman , M is a set of words having a target attribute , and F is a set of words having an inverse attribute , for example , man ∈M and woman ∈ F for gender attributes . Eq . 4 indicates that the operation changes depending on whether input word x belongs to M or F . For gender words , we subtract difference vector d if x is male , and add it if x is female . Therefore , we need such explicit knowledge . However , this knowledge can not be developed for various words and attributes in practice . 4 REFLECTION-BASED WORD ATTRIBUTE TRANSFER . 4.1 IDEALIZED TRANSFER WITHOUT EXPLICIT KNOWLEDGE . What is an idealized transfer function φz for the word attribute transfer ? The following are the idealized natures of such a transfer function : vm = φz ( vw ) , ( 5 ) vw = φz ( vm ) , ( 6 ) where m ∈M and w ∈ F . This function φz enables to transfer a word without explicit knowledge . Function φz transfers vm to vw and vw to vm without such explicit knowledge as m ∈ M and w ∈ F . By combining Eqs . 5 and 6 , we obtain the following formula : ∀m ∈M , vm = φz ( φz ( vm ) ) , ( 7 ) and ∀w ∈ F , vw = φz ( φz ( vw ) ) . ( 8 ) Hence , the idealized transfer function is a mapping that becomes an identity mapping when we apply it twice for any v. Such a mapping is called involution in geometry . For example , φ : v 7→ −v is one example of an involution . Note that the identity map itself , such as φ : v 7→ v , is excluded from the involution . 4.2 REFLECTION . A reflection is an involution that reverses the location between two vectors in a Euclidean space through an affine hyperplane ( mirror ) . Reflection is an idealized function because every point returns to its original location when reflection is applied twice : ∀v ∈ Rn , v = Refa , c ( Refa , c ( v ) ) . ( 9 ) Given vector v in Euclidean space Rn , the formula for the reflection in the mirror is given : Refa , c ( v ) = v − 2 ( v − c ) · a a · a a , ( 10 ) where a ∈ Rn is a vector orthogonal to the mirror ( normal vector ) and c ∈ Rn is a point through which the mirror passes . a and c are parameters that determine the mirror . 4.3 REFLECTION-BASED WORD ATTRIBUTE TRANSFER . We apply reflection to the word attribute transfer to invert a specific attribute of an input word without its explicit attribute knowledge . We learn a mirror ( hyperplane ) in a pre-trained embedding space using training word pairs with a common ( binary ) attribute z ( Fig . 2 ) . Here since the mirror is uniquely determined by two parameter vectors , a and c , we estimate a and c from target attribute z using fully connected multi-layer perceptrons : a = MLP ( z ) , ( 11 ) c = MLP ( z ) . ( 12 ) Transferred vector vy is obtained by inverting attribute z of vx by reflection : vy = Refa , c ( vx ) . ( 13 ) 4.4 PARAMETERIZED MIRRORS . Reflection with a mirror by Eqs . 11 and 12 assumes a single mirror depending only on z . Previous discussion assumed that there will be pairs sharing a stable attribute such king and queen . However , often gendered words don ’ t come in pairs , and gender is far from a stable attribute . For example , actress may be feminine , but actor is clearly neutral in many cases ( Fig . 3 ) . Thus , actor isn ’ t as obvious a masculine counterpart as king . In fact , it is known that there are biases in gender words in the embedding space ( Zhao et al. , 2018 ; Kaneko & Bollegala , 2019 ) . This phenomenon can occur not only with the gender attribute , but also with other attributes . With this assumption of a single mirror , the mirror must be a hyperplane that goes through the midpoints for all word vector pairs . However , the vector pairs shown on the left of Fig . 3 can not be transferred well since the single mirror does not satisfy this constraint due to the bias of the embedding space . To solve this problem , we introduce different mirrors for different words . We propose parameterized mirrors determined by input vector vx in addition to attribute z . The following are the definitions of the mirror parameters : a = MLP ( [ z ; vx ] ) , ( 14 ) c = MLP ( [ z ; vx ] ) , ( 15 ) where [ · ; · ] indicates the vector concatenation in the column . The parameterized mirrors are expected to work flexibly on different words . For instance , as shown in Fig . 3 , suppose we learned the mirror ( the blue line ) that transfers vhero to vheroine in advance . If input word vector vactor resembles vhero , a mirror that is similar to the one for vhero should be derived and used for the attribute transfer .

Summary: This paper proposes a method for binary word attribute transfer based on reflection. It applies a single reflection-based mapping that relates the locations of two vectors in a Euclidean space by a hyperplane, and results in an identity mapping when it is applied twice. It's sort of like roundtrip-translation, but for word vector attributes. The paper also proposes a pretty smart idea: the mirror functions are parameterized to take advantage of the fact that inversions differ, even for the "same" word attributes.