Reflection-based Word Attribute Transfer
Yoichi Ishibashi, Katsuhito Sudoh, Koichiro Yoshino, Satoshi Nakamura
Nara Institute of Science and Technology (NAIST), Japan

Background：2

Analogy

Analogy in the embedding space• is a operation that transfer word attributes• Change word attributes (e.g. gender)

king − man + woman ≈ queen
queen

woman
king − man + woman

king

king − manman

Background：3

Task overview

Word attribute transfer task
• Get a word vector that inverted attribute of aninput word vector

vmother ⇡ fgender (vfather )zx

Invert genderfatherInvert singular/pluraltmotherfathers

Background：

Application

4

What can word attribute transfer be used for?
• E.g. Data Augmentation
Targetattribute

Input words

Output words

GenderI am his mother.

I am her father.

AntonymNobody has a suit.

Someone has a suit.

CapitalCountry

I live in Japan.

I live in Tokyo.

Transfer function

5

fz transfer words if they have a target attribute z• E.g. man → woman (attribute: gender)

vwoman ⇡ fgender (vman )fz does not transfer words if it does not has atarget attribute z• E.g. person → person (attribute: gender)

vperson ⇡ fgender (vperson )Analogy-based Word Attribute Transfer 6
Analogy-based word attribute transfer
king ­ (man ­ woman) = queen
queen + (man ­ woman) = king

• Add or subtract a difference vector
Problem• Need explicit knowledge whether input word has
the target attribute or notGoal• Transform word attributes without the explicit
knowledge

Proposed method

8

Ideal function
What is an ideal transfer function?
• No explicit knowledge= Transfer any words with the same function

vman = f (vwoman )vwoman = f (vman )vperson = f (vperson )Combine above formulas

vx = f (f (vx ))← Nature of theideal function

9Reflection
Reflection is an ideal function• Transfer any words with the same functionv = Refa,c ( Refa,c (v) )Ref a,c (v) = v

2(vc) · aaa·a• Move two vectors through a hyperplane (mirror)

man = Ref a,c (woman)

Ref a,c (woman)

woman = Ref a,c (man)person = Ref a,c (person)

Reflection

womanMirror

Ref a,c (man)man

Ref a,c (person)

personReflection-based Word Attribute Transfer 10
How to apply to word attribute transfer?
• Transfer an input word vector vx
to a target word vector vt

vt ⇡ vy = Ref a,c (vx )• Learn a mirror for each attributes
Male Female
queen
woman

Singular Plural
Refa,c (v)bananasMirrorMirrorappleskingmanRefa,c (v)orangesbananaapple

orangeReflection-based Word Attribute Transfer 11
How to learn the mirror?
Idea : Estimate a and c by MLP

a … A vector orthogonal to the mirror
c … A point through which the mirror passes
queen
woman

kingmanRef a,c (v)MirroracVector

Two types of mirror estimation② Parameterized mirrors① Single mirror
Estimate from an attribute z

Some pairs are
non-transferable

Estimate from z
and an input word vector vx

Work more flexibly

a = MLP✓1 ([z; vx ])c = MLP✓2 ([z; vx ])a = MLP✓1 (z)c = MLP✓2 (z)sistermother

queen
woman

brotherfatherking

sistermotherRefa,c (v)Mirroractress
heroine

queen
woman

brotherfather
king

man
actor
hero

12

Refa,c (v)Mirrors

man
actor
hero

actress
heroineTwo types of mirror estimation② Parameterized mirrors① Single mirror
Estimate from an attribute z

Some pairs are
non-transferable

Estimate from z
and an input word vector vx

Work more flexibly

a = MLP✓1 ([z; vx ])c = MLP✓2 ([z; vx ])a = MLP✓1 (z)c = MLP✓2 (z)sistermother

queen
woman

brotherfatherking

sistermotherRefa,c (v)Mirroractress
heroine

queen
woman

brotherfather
king

man
actor
hero

13

Refa,c (v)Mirrors

man
actor
hero

actress
heroineExperiments

Purpose
Compare reflection and baselines
Four different attributes
Male-Female, Singular-Plural
Capital-Country, Antonyms
Two pre-trained word embeddings
word2vec (SGNS), GloVe
Two evaluation metrics
Accuracy, Stability

15

16Dataset

Attribute words … Four different binary attributes
Attribute (z)

Train

Val

Test

Example(x, t)

Male-Female (MF)291212(king, queen)

Singular-Plural (SP)90

2525(king, kings)

Capital-Country (CC)59

2525(Japan, Tokyo)

Antonym (AN)

1354

290

290

(good, bad)Non-attribute words
0  |Ntrain |  50
Train

Test|Ntest | = 1000

17Evaluation metrics① Accuracy: Ratio of attribute words transferredwoman ✔
manmother ✕② Stability:

Ratio of non-attribute words not transferred
apple

✔

apple
woman ✕

18Results (Accuracy)

Best method: Reflection with parameterized mirrors→ High performance in both accuracy and stability
Worst method: MLP
GloVeMethod

Accuracy (%)

Stability (%)

MFSPCCANMF

SP

Ref

12.5

2.0

26.0

0.0

100.0 100.0 100.0 100.0

Ref+PM

45.8

50.0

76.0

33.5

99.7

99.1

99.2

100.0

MLP

4.2

10.0

18.0

36.7

5.1

7.0

5.2

1.2

Diff+

25.02.026.0-

99.3

94.2

99.3-

Diff-

25.0

2.0

24.0-

100.0

99.9

99.5

-

MF: Male-Female, SP: Singular-Plural, CC: Country-Capital,

CCAN

AN: Antonym

19Results (Stability)

Best method: Reflection with parameterized mirrors→ High performance in both accuracy and stability
Worst method: MLP
GloVeMethod

Accuracy (%)

Stability (%)

MF

SPCCAN

MF

SP

Ref

12.5

2.0

26.0

0.0

100.0 100.0 100.0 100.0

Ref+PM

45.8

50.0

76.0

33.5

99.7

99.1

99.2

100.0

MLP

4.2

10.0

18.0

36.7

5.1

7.0

5.2

1.2

Diff+25.0

2.0

26.0-

99.3

94.2

99.3-

Diff-

25.0

2.0

24.0-

100.0

99.9

99.5

-

MF: Male-Female, SP: Singular-Plural, CC: Country-Capital,

CCAN

AN: Antonym

20Transfer examples
Reflection with parameterized mirrors (Ref+PM) canselectively transfer words without the knowledge
boy

girl

youyouInput

the woman got married when you were a boy.

Ref

the woman got married when you were a boy.

Ref+PMthe man got married when you were a girl.

MLP

By_Katie_Klingsporn girlfriend Valerie_Glodowski
fiancee Doughty_Evening_Chronicle ma’am Bob_Grossweiner_& a mother.

Diff+

the man got married when you were a boy.

Diff-

the woman got married when you were a girl.

Why is the reflection very stabile?

21

Hypothesis: Non-attribute word distributes on its mirror→ Visualize the distance between a word vector and its mirror

Attribute word (e.g. woman)
Mirrordistance =|(vxc) · a|kakdistance

Non-attribute word(e.g. person)

Distance between the word and its mirror 22
・Attribute words distributed apart from the mirror
・Non-attribute words distributed near the mirror

SummaryBackground• Word attribute transfer task
• Analogy can be used for the transfer• Analogy-based transfer requires the explicit knowledge

Proposed method• Reflection-based word attribute transfer• Reflection is an ideal mapping for word attribute transfer
Experimental results• Reflection-based transfer achieved best performance• Reflection transfers attribute words
e.g. man → woman• Reflection does not transfer non-attribute words
e.g. person → person