# Screening Gender Transfer in Neural Machine Translation

Guillaume Wisniewski, Lichao Zhu, Nicolas Ballier, François Yvon

<figure><img src="image_1.png" /></figure>

## To cite this version:

Guillaume Wisniewski, Lichao Zhu, Nicolas Ballier, François Yvon. Screening Gender Transfer in Neural Machine Translation. BlackBoxNLP 2021, Nov 2021, Punta Cana, Dominican Republic. hal-03993451

HAL Id: hal-03993451

https://hal.science/hal-03993451

Submitted on 16 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
## Information flow within an encoder/decoder architecture

* key steps in interpreting NMT systems

→ which informations are captured by the decoder?
→ which informations are captured by the encoder?
→ which informations are transferred from the source to the target?

* how: study the transfer of gender information from French to English

→ using probes to find where this information is represented;
→ using causal models to determine when this information is used.

## Gender Transfer between French and English

* challenge

→ in French: gender = property of all nouns ⊕ agreement rules within noun phrase
→ in English: gender = only in rare constructs involving human agents and pronoun coreference

* focus on the following pattern:

→ [DET] [N] a terminé son travail.
→ The [N] has finished [PRO] work.

→ [N] = occupational noun either feminine or masculine
→ [DET] = French determiner in agreement with the noun
→ [PRO] = English possessive pronoun

* Dataset of 3,394 parallel sentences following this pattern

→ perfectly balanced between genders

* Hypothetical paths for transferring gender information from French to English

<figure><img src="image_3.png" /></figure>

→ (a) direct influence → cross-lingual attention;
→ (b) indirect influence → monolingual encoding of gender in the representation of the English noun;
→ (c) indirect influence → cross-lingual attention to the French possessive adjective.

## Probing Representations

<figure><img src="image_4.png" /></figure>

* linguistic probe: predict the gender of the French occupational noun from a source/target word representation
  → simple binary classification problem
  → evaluation: accuracy

### In the source

<table><thead><tr><th>layer</th><th>a</th><th>terminé</th><th>son</th><th>travail</th><th>.</th><th>eos</th></tr></thead><tbody><tr><td>1</td><td>80.4%</td><td>75.1%</td><td>80.6%</td><td>76.4%</td><td>59.5%</td><td>73.3%</td></tr><tr><td>2</td><td>85.8%</td><td>80.8%</td><td>81.6%</td><td>78.3%</td><td>87.6%</td><td>88.3%</td></tr><tr><td>3</td><td>89.5%</td><td>88.2%</td><td>89.2%</td><td>82.0%</td><td>86.5%</td><td>87.6%</td></tr><tr><td>4</td><td>90.8%</td><td>89.3%</td><td>90.6%</td><td>85.9%</td><td>85.7%</td><td>85.6%</td></tr><tr><td>5</td><td>90.4%</td><td>89.3%</td><td>90.4%</td><td>85.5%</td><td>86.4%</td><td>85.2%</td></tr><tr><td>6</td><td>91.0%</td><td>89.3%</td><td>90.0%</td><td>86.0%</td><td>86.4%</td><td>85.1%</td></tr></tbody></table>

### Gender information

→ is more present in the deepest layer of the encoder
→ spreads all over the representation of the source tokens
⇒ and not only the tokens involved in our hypothesis

### In the target

* target sentence not as 'regular' as source sentences

(predicted by MT system)

⇒ accuracy of the probe computed over all tokens of the translation hypothesis

<table><thead><tr><th rowspan="2">layer</th><th colspan="2">decoder</th></tr><tr><th>the</th><th>all tokens</th></tr></thead><tbody><tr><td>1</td><td>89.5%</td><td>71.6%</td></tr><tr><td>2</td><td>92.0%</td><td>76.3%</td></tr><tr><td>3</td><td>91.8%</td><td>78.1%</td></tr><tr><td>4</td><td>90.9%</td><td>79.1%</td></tr><tr><td>5</td><td>89.3%</td><td>82.4%</td></tr><tr><td>6</td><td>87.7%</td><td>84.7%</td></tr></tbody></table>

* gender information encoded in all target tokens
  ⇒ even those for which the information is useless

## Manipulating Representations

<figure><img src="image_5.png" /></figure>

* goal: identify if and when gender information is used

* how: intervention → replace the embedding of 'son' by a representation that triggers

→ the generation of 'her' (feminine embedding)
→ the generation of 'his' (masculine embedding)
→ a neutral version of the embedding average of son representation over all sentences

* evaluation: distribution of pronoun in translation hypothesis

<table><thead><tr><th>intervention</th><th>English pronoun</th><th>%</th><th>sentences</th></tr></thead><tbody><tr><td rowspan="3">none</td><td>her</td><td>13.4%</td><td></td></tr><tr><td>his</td><td>57.1%</td><td></td></tr><tr><td>other</td><td>29.5%</td><td></td></tr><tr><td rowspan="3">feminine</td><td>her</td><td>17.3%</td><td></td></tr><tr><td>his</td><td>56.8%</td><td></td></tr><tr><td>other</td><td>25.9%</td><td></td></tr><tr><td rowspan="3">gender-neutral</td><td>her</td><td>13.2%</td><td></td></tr><tr><td>other</td><td>29.4%</td><td></td></tr><tr><td>his</td><td>57.4%</td><td></td></tr><tr><td rowspan="3">masculine</td><td>her</td><td>13.8%</td><td></td></tr><tr><td>other</td><td>29.2%</td><td></td></tr><tr><td>his</td><td>57.0%</td><td></td></tr></tbody></table>

→ representations of son are not the only evidence used during the generation of the translation hypothesis
⇒ path (c) has only a limited influence

## Conclusion

* Contributions: new dataset ⊕ two techniques (probing & manipulating)

* Conclusions:

→ gender information in the representation of all tokens representations built by the encoder and the decoder
→ choice of English pronoun distributed

* future work:

→ generalization to other language & syntactic divergences
→ identify which information is used to choose the English pronoun

## Code & Corpus

https://github.com/neuroviz/neuroviz/tree/main/blackbox2021

## Acknowledgments

This work was partially funded by the NeuroViz project subsidized by the Île-de-France Region, and supported in part by the 2020 émergence research project SPEC TRANS.

## Contact information

guillaume.wisniewski@u-paris.fr