# What ChatGPT tells us about ourselves

Aaron Boussidan, Fanny Ducel, Aurélie Névéol, Karën Fort

► To cite this version:

Aaron Boussidan, Fanny Ducel, Aurélie Névéol, Karën Fort. What ChatGPT tells us about ourselves. Journée d'étude Éthique et TAL 2024, Apr 2024, Nancy, France. hal-04521121

HAL Id: hal-04521121

https://inria.hal.science/hal-04521121

Submitted on 26 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# What ChatGPT tells us about ourselves

Aaron Boussidan, Fanny Ducel, Aurélie Névéol, Karën Fort

► To cite this version:

Aaron Boussidan, Fanny Ducel, Aurélie Névéol, Karën Fort. What ChatGPT tells us about ourselves. Journée Éthique et TAL 2024, Apr 2024, Nancy, France. hal-04521121

HAL Id: hal-04521121

https://inria.hal.science/hal-04521121v1

Submitted on 26 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# What ChatGPT tells us about ourselves

Aaron Boussidan<fnref n="1" />, Fanny Ducel², Aurélie Névéol², Karën Fort³

(1) Université Gustave Eiffel, LIGM (France)

(2) Université Paris-Saclay, CNRS, LISN (France)

(3) Sorbonne Université / LORIA (France)

aaron.boussidan@univ-eiffel.fr, fanny.ducel@lisn.fr

## 1 Introduction

Large Language Models are known to generate various biases, and the scientific community has started to tackle them. There are two main approaches to detect and evaluate stereotypical biases: intrinsically or extrinsically. Intrisic methods aim at uncovering internal biases, located within the model's representations (Bolukbasi et al., 2016; Guo & Caliskan, 2021), whereas extrinsic methods focus on biases that are produced during inference, i.e. the output of a use-case (De-Arteaga et al., 2019; Parrish et al., 2022). Since our goal is to work on ChatGPT, we are compelled to opt for the second approach as OpenAI did not release enough information on the training corpora or the internal model's representations. More precisely, this experiment is an attempt at measuring ChatGPT's biases about various countries in the world. With the help of prompting, generative models can easily be used for other classical NLP tasks. In this experiment in particular, we prompted ChatGPT 3.5 to perform sentiment analysis, by getting it to assign a "positivity score" to every country of the world. We then generated a choropleth (a map where countries are each assigned a color associated to their score) with ChatGPT's rating of each country. The experiment has been conducted in 4 different languages: French, English, Russian and Arabic, to assess potential variations between languages. We briefly present our methodology, and then detail our results, their interpretation, and we conclude by discussing the design of the experiment and its inherent biases.

## 2 Methodology

We used the publicly available version of ChatGPT, using the web interface, between February 21, 2024 and March 6, 2024. For each language, we used the same prompts. Translations into languages other than English were generated using DeepL, and checked for fluency and correctness by native or bilingual speakers. We queried ChatGPT to assign a score between -10 (very negative) and +10 (very positive) to different words:

* Hi ! I will give you words, and I would like you to assign scores between -10 and 10 to those words, based on how positive they are

We then provide it with a list of countries<fnref n="1" />, asking it to assign a score to each of them:

* Please score the following words, and give your results in a .csv file: ['Afghanistan', 'Albania', ..., 'Zambia', 'Zimbabwe', 'eSwatini']

We then generated maps where each country was assigned a color on a red (most negative) to green (most positive) scale.

<footnote n="1">¹We used the list of countries provided by the Python library Geopandas.</footnote>
# What ChatGPT tells us about ourselves

Aaron Boussidan<fnref n="1" />, Fanny Ducel², Aurélie Névéol², Karën Fort³

(1) Université Gustave Eiffel, LIGM (France)

(2) Université Paris-Saclay, CNRS, LISN (France)

(3) Sorbonne Université / LORIA (France)

aaron.boussidan@univ-eiffel.fr, fanny.ducel@lisn.fr

## 1 Introduction

Large Language Models are known to generate various biases, and the scientific community has started to tackle them. There are two main approaches to detect and evaluate stereotypical biases: intrinsically or extrinsically. Intrisic methods aim at uncovering internal biases, located within the model's representations (Bolukbasi et al., 2016; Guo & Caliskan, 2021), whereas extrinsic methods focus on biases that are produced during inference, i.e. the output of a use-case (De-Arteaga et al., 2019; Parrish et al., 2022). Since our goal is to work on ChatGPT, we are compelled to opt for the second approach as OpenAI did not release enough information on the training corpora or the internal model's representations. More precisely, this experiment is an attempt at measuring ChatGPT's biases about various countries in the world. With the help of prompting, generative models can easily be used for other classical NLP tasks. In this experiment in particular, we prompted ChatGPT 3.5 to perform sentiment analysis, by getting it to assign a "positivity score" to every country of the world. We then generated a choropleth (a map where countries are each assigned a color associated to their score) with ChatGPT's rating of each country. The experiment has been conducted in 4 different languages: French, English, Russian and Arabic, to assess potential variations between languages. We briefly present our methodology, and then detail our results, their interpretation, and we conclude by discussing the design of the experiment and its inherent biases.

## 2 Methodology

We used the publicly available version of ChatGPT, using the web interface, between February 21, 2024 and March 6, 2024. For each language, we used the same prompts. Translations into languages other than English were generated using DeepL, and checked for fluency and correctness by native or bilingual speakers. We queried ChatGPT to assign a score between -10 (very negative) and +10 (very positive) to different words:

* Hi ! I will give you words, and I would like you to assign scores between -10 and 10 to those words, based on how positive they are

We then provide it with a list of countries<fnref n="1" />, asking it to assign a score to each of them:

* Please score the following words, and give your results in a .csv file: ['Afghanistan', 'Albania', ..., 'Zambia', 'Zimbabwe', 'eSwatini']

We then generated maps where each country was assigned a color on a red (most negative) to green (most positive) scale.

<footnote n="1">¹We used the list of countries provided by the Python library Geopandas.</footnote>
All of our code, as well as our prompts will be made available upon acceptation.

# 3 Results and first interpretations

## 3.1 A Western perspective?

Looking at the results and data for the four languages (See Figures 1 to 4), a clear global trend appears: North American and European countries are given higher scores, while African countries are given the worst ones. South American and Asian countries tend to be in the medium range. In Oceania, Australia and New-Zealand are evaluated more positively than their neighbours. Some countries are notable outliers: North Korea, Venezuela, Belarus, and Myanmar, for example.

Comparing our generated maps with GDP<fnref n="2" /> or Internet access maps for the world shows some notable similarities. This suggests that the scores obtained seem to reflect the opinions of the users and writers of the Web, who positively perceive themselves and their fellow users. Indeed, we know that Western countries have better access to the Internet<fnref n="3" /> and that, at least for Wikipedia, the majority of users who express themselves online are Western men who are either in their mid-20s or retired<fnref n="4" />. We can easily see similar trends between these aforementioned statistics and the results of our experiment: ChatGPT reproduces, and maybe amplifies, what people write on the Internet.

The model outputs often mention these scores as reflecting a “general opinion”. However, no particular way of scoring countries was integrated to our initial prompt: we only asked how “positive” some “words” are. When asked to score countries individually, the model usually explains how these scores are linked to the “economic level” of the country, its “reputation” or some of its policies: these criteria of scoring have been purely inferred by the model. For example, when prompted to elaborate about the score for Uruguay, the model produces the following explanation:

Uruguay (Score: 5): Uruguay is perceived positively for its stable democracy, progressive social policies (including legalizing cannabis), strong rule of law, environmental sustainability efforts, and relatively high quality of life in Latin America.

Among other things, framing cannabis legalisation as a positive factor to evaluate a country is a political choice. When prompted about the United Arab Emirates, the model produces a list of qualities of the countries, which does not include social policies, that had been factored in previously.

United Arab Emirates (Score: 7): The UAE is viewed positively due to its rapid development, modern infrastructure, economic diversification beyond oil, tourism initiatives, and ambitious projects such as the Burj Khalifa and Palm Islands.

These outputs also seem to be outdated, as across all languages, Russia is given a positive score, whereas the context of the Russia-Ukraine war is known to have negatively impacted people's view of the country<fnref n="5" />. The training data of ChatGPT 3.5 allegedly stops in January 2022, a month before Russia's invasion of Ukraine and the beginning of the war. This reminds us that the model reproduces

<footnote n="2">²https://www.imf.org/external/datamapper/NGDPDPC@WEO/OEMDC/ADVEC/WEOWORLD</footnote>

<footnote n="3">³https://ourworldindata.org/grapher/share-of-individuals-using-the-internet?
time=2021&tab=map</footnote>

<footnote n="4">⁴See the example of Wikipedia editors: https://en.wikipedia.org/wiki/Wikipedia:Who_writes_Wikipedia%3F</footnote>

<footnote n="5">⁵https://www.pewresearch.org/global/2023/07/10/overall-opinion-of-russia/</footnote>
All of our code, as well as our prompts will be made available upon acceptation.

# 3 Results and first interpretations

## 3.1 A Western perspective?

Looking at the results and data for the four languages (See Figures 1 to 4), a clear global trend appears: North American and European countries are given higher scores, while African countries are given the worst ones. South American and Asian countries tend to be in the medium range. In Oceania, Australia and New-Zealand are evaluated more positively than their neighbours. Some countries are notable outliers: North Korea, Venezuela, Belarus, and Myanmar, for example.

Comparing our generated maps with GDP<fnref n="2" /> or Internet access maps for the world shows some notable similarities. This suggests that the scores obtained seem to reflect the opinions of the users and writers of the Web, who positively perceive themselves and their fellow users. Indeed, we know that Western countries have better access to the Internet<fnref n="3" /> and that, at least for Wikipedia, the majority of users who express themselves online are Western men who are either in their mid-20s or retired<fnref n="4" />. We can easily see similar trends between these aforementioned statistics and the results of our experiment: ChatGPT reproduces, and maybe amplifies, what people write on the Internet.

The model outputs often mention these scores as reflecting a “general opinion”. However, no particular way of scoring countries was integrated to our initial prompt: we only asked how “positive” some “words” are. When asked to score countries individually, the model usually explains how these scores are linked to the “economic level” of the country, its “reputation” or some of its policies: these criteria of scoring have been purely inferred by the model. For example, when prompted to elaborate about the score for Uruguay, the model produces the following explanation:

Uruguay (Score: 5): Uruguay is perceived positively for its stable democracy, progressive social policies (including legalizing cannabis), strong rule of law, environmental sustainability efforts, and relatively high quality of life in Latin America.

Among other things, framing cannabis legalisation as a positive factor to evaluate a country is a political choice. When prompted about the United Arab Emirates, the model produces a list of qualities of the countries, which does not include social policies, that had been factored in previously.

United Arab Emirates (Score: 7): The UAE is viewed positively due to its rapid development, modern infrastructure, economic diversification beyond oil, tourism initiatives, and ambitious projects such as the Burj Khalifa and Palm Islands.

These outputs also seem to be outdated, as across all languages, Russia is given a positive score, whereas the context of the Russia-Ukraine war is known to have negatively impacted people's view of the country<fnref n="5" />. The training data of ChatGPT 3.5 allegedly stops in January 2022, a month before Russia's invasion of Ukraine and the beginning of the war. This reminds us that the model reproduces

<footnote n="2">²https://www.imf.org/external/datamapper/NGDPDPC@WEO/OEMDC/ADVEC/WEOWORLD</footnote>

<footnote n="3">³https://ourworldindata.org/grapher/share-of-individuals-using-the-internet?
time=2021&tab=map</footnote>

<footnote n="4">⁴See the example of Wikipedia editors: https://en.wikipedia.org/wiki/Wikipedia:Who_writes_Wikipedia%3F</footnote>

<footnote n="5">⁵https://www.pewresearch.org/global/2023/07/10/overall-opinion-of-russia/</footnote>
## 3.2 Variation between languages

We can also observe some differences across languages. Prompts in French lead to a more negative score for African countries, especially the ones that France colonized<fnref n="6" />. On the contrary, when prompted in English, the model gives positive scores to the Commonwealth nations (e.g. India, Australia). For Arabic and Russian, the results are quite similar and more homogeneous, with a more positive overall average. The rare countries that have negative scores are also in Africa and, especially for the Russian version, in the Middle East.

We can propose different hypotheses to explain the changes in results based on the language used in the prompt. On one hand, it could be related to the cultural differences that are encapsulated in the training corpora. On the second hand, it could be the reflection of the generations' linguistic quality and of the use of toxicity filters and Reinforcement Learning with Human Feedback (RLHF), that differ based on the language.

This experiment is, of course, not a systematic large-scale approach at those biases, but rather a first approach, supported by a visualization method. In particular, the model is non-deterministic in its core: country scores vary from conversation to conversation, and even the average score among all countries varies from conversation to conversation. We have tried to counter-act this by normalising the data, to make it comparable, and by trying to have a larger sample size for English, but a more large scale approach should be taken to further pursue this experiment.

Finally, it might be misguided to take these results as showing that ChatGPT “has an opinion” on countries, or “favors” some over others. When asked to explain its scores, the model gives explanation for why it has chosen its score. A positivity score is fundamentally 1-dimensional, and by forcing the model to choose, we are somewhat forcing discrepancies between countries.

## 3.3 Variation between iterations

Since ChatGPT is non-deterministic, we replicated the experiment on English 15 times, on different user accounts, using different conversation tabs and shuffling the order of the countries in our prompt. Our goal was to observe whether or not scores change drastically across iterations. We then computed the averages, medians, and standard deviations for each country. Our results show that the majority of countries obtain very similar scores across iterations. Nonetheless, it is not the case for 30 countries (See Table A), which present significant standard deviation (>2, when the mean standard deviation is 1.35), e.g. notable score differences across trials. The concerned countries are mostly situated in Africa (e.g. Sierra Leone, Zimbabwe), or, to a lesser extent, in South/Southeast Asia (e.g. Bangladesh, Myanmar). The rest are situated in Eastern-European (Ukraine, Belarus and Russia), the Near East (Lebanon, Saudi Arabia) or South America (Venezuela, Cuba). We can imagine that the notable differences of these 30 countries are related to a less important quantity of data about them, or to very polarized opinions in corpora.

<footnote n="6">⁶https://fr.wikipedia.org/wiki/Afrique_français</footnote>
## 3.2 Variation between languages

We can also observe some differences across languages. Prompts in French lead to a more negative score for African countries, especially the ones that France colonized<fnref n="6" />. On the contrary, when prompted in English, the model gives positive scores to the Commonwealth nations (e.g. India, Australia). For Arabic and Russian, the results are quite similar and more homogeneous, with a more positive overall average. The rare countries that have negative scores are also in Africa and, especially for the Russian version, in the Middle East.

We can propose different hypotheses to explain the changes in results based on the language used in the prompt. On one hand, it could be related to the cultural differences that are encapsulated in the training corpora. On the second hand, it could be the reflection of the generations' linguistic quality and of the use of toxicity filters and Reinforcement Learning with Human Feedback (RLHF), that differ based on the language.

This experiment is, of course, not a systematic large-scale approach at those biases, but rather a first approach, supported by a visualization method. In particular, the model is non-deterministic in its core: country scores vary from conversation to conversation, and even the average score among all countries varies from conversation to conversation. We have tried to counter-act this by normalising the data, to make it comparable, and by trying to have a larger sample size for English, but a more large scale approach should be taken to further pursue this experiment.

Finally, it might be misguided to take these results as showing that ChatGPT “has an opinion” on countries, or “favors” some over others. When asked to explain its scores, the model gives explanation for why it has chosen its score. A positivity score is fundamentally 1-dimensional, and by forcing the model to choose, we are somewhat forcing discrepancies between countries.

## 3.3 Variation between iterations

Since ChatGPT is non-deterministic, we replicated the experiment on English 15 times, on different user accounts, using different conversation tabs and shuffling the order of the countries in our prompt. Our goal was to observe whether or not scores change drastically across iterations. We then computed the averages, medians, and standard deviations for each country. Our results show that the majority of countries obtain very similar scores across iterations. Nonetheless, it is not the case for 30 countries (See Table A), which present significant standard deviation (>2, when the mean standard deviation is 1.35), e.g. notable score differences across trials. The concerned countries are mostly situated in Africa (e.g. Sierra Leone, Zimbabwe), or, to a lesser extent, in South/Southeast Asia (e.g. Bangladesh, Myanmar). The rest are situated in Eastern-European (Ukraine, Belarus and Russia), the Near East (Lebanon, Saudi Arabia) or South America (Venezuela, Cuba). We can imagine that the notable differences of these 30 countries are related to a less important quantity of data about them, or to very polarized opinions in corpora.

<footnote n="6">⁶https://fr.wikipedia.org/wiki/Afrique_français</footnote>
# 4 A flawed and biased experiment

The presented experiment and results are actually fundamentally flawed, as they encapsulate our (the authors’) own biases. As presented by Hovy & Prabhumoye (2021), there are five sources of bias in NLP, including the research design.

In our case, some biases and questionable choices can be found at different levels. First of all, the goal and choice of the experiment itself can raise questions such as: *What are we truly asking the model? What are we trying to measure? What does the provided score mean?* This last question is especially important, as the used prompts only ask for a “score” on a “word”, and we decided to interpret it as the perceived positivity of a country.

We can also raise questions about the provided (over)interpretation of the outputs. ChatGPT is not transparent about its training, so we can not know where the outputs come from and we can not give definite answers. All the explanations are purely hypothetical, relying on the authors’ internal biases and opinions, and could easily be twisted: we could make the results, hence ChatGPT, say anything. This can be used as a reminder that language models do not understand nor produce meaning, but the readers do (Bender & Koller, 2020).

Finally, the choices of implementation that were made (or were not made, when letting default parameters) are also important. The chosen scores scale can be questioned (-10 to 10), as well as the colors of the maps. These colors, red and green, have connotations and invite readers to interpret green as positive and red as negative. Besides, they reduce the accessibility for color-blind people. Moreover, the maps’ design, especially their orientation (Europe in the center, North countries on top, South countries on the bottom), and their projection (Mercator) also represent some political and cultural values (Graham & Dittus, 2022).

In other words, every step of this experiment implied its load of decisions. Some decisions may seem shallow at first sight, whereas they have important consequences and implications. We can hypothesize that different design choices would have led to different results (even more as ChatGPT is non-deterministic and outputs vary between iterations and between small prompt changes), hence different interpretations and an overall different paper.

## References

* BENDER E. M. & KOLLER A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In D. JURAFSKY, J. CHAI, N. SCHLUTER & J. TETREAULT, Éds., *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, p. 5185–5198, Online: Association for Computational Linguistics. DOI: 10.18653/v1/2020.acl-main.463.

* BOLUKBASI T., CHANG K.-W., ZOU J. Y., SALIGRAMA V. & KALAI A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. *Advances in neural information processing systems*, 29.

* DE-ARTEGA M., ROMANOV A., WALLACH H., CHAYES J., BORG S. C., HOULDECHOVA A., GEYIK S., KENTHAPADI K. & KALAI A. T. (2019). Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In *Proceedings of the Conference on Fairness, Accountability, and Transparency*, p. 120–128. arXiv:1901.09451 [cs, stat], DOI: 10.1145/3287560.3287572.
# 4 A flawed and biased experiment

The presented experiment and results are actually fundamentally flawed, as they encapsulate our (the authors’) own biases. As presented by Hovy & Prabhumoye (2021), there are five sources of bias in NLP, including the research design.

In our case, some biases and questionable choices can be found at different levels. First of all, the goal and choice of the experiment itself can raise questions such as: *What are we truly asking the model? What are we trying to measure? What does the provided score mean?* This last question is especially important, as the used prompts only ask for a “score” on a “word”, and we decided to interpret it as the perceived positivity of a country.

We can also raise questions about the provided (over)interpretation of the outputs. ChatGPT is not transparent about its training, so we can not know where the outputs come from and we can not give definite answers. All the explanations are purely hypothetical, relying on the authors’ internal biases and opinions, and could easily be twisted: we could make the results, hence ChatGPT, say anything. This can be used as a reminder that language models do not understand nor produce meaning, but the readers do (Bender & Koller, 2020).

Finally, the choices of implementation that were made (or were not made, when letting default parameters) are also important. The chosen scores scale can be questioned (-10 to 10), as well as the colors of the maps. These colors, red and green, have connotations and invite readers to interpret green as positive and red as negative. Besides, they reduce the accessibility for color-blind people. Moreover, the maps’ design, especially their orientation (Europe in the center, North countries on top, South countries on the bottom), and their projection (Mercator) also represent some political and cultural values (Graham & Dittus, 2022).

In other words, every step of this experiment implied its load of decisions. Some decisions may seem shallow at first sight, whereas they have important consequences and implications. We can hypothesize that different design choices would have led to different results (even more as ChatGPT is non-deterministic and outputs vary between iterations and between small prompt changes), hence different interpretations and an overall different paper.

## References

* BENDER E. M. & KOLLER A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In D. JURAFSKY, J. CHAI, N. SCHLUTER & J. TETREAULT, Éds., *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, p. 5185–5198, Online: Association for Computational Linguistics. DOI: 10.18653/v1/2020.acl-main.463.

* BOLUKBASI T., CHANG K.-W., ZOU J. Y., SALIGRAMA V. & KALAI A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. *Advances in neural information processing systems*, **29**.

* DE-ARTEGA M., ROMANOV A., WALLACH H., CHAYES J., BORG S. C., HOULDECHOVA A., GEYIK S., KENTHAPADI K. & KALAI A. T. (2019). Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In *Proceedings of the Conference on Fairness, Accountability, and Transparency*, p. 120–128. arXiv:1901.09451 [cs, stat], DOI: 10.1145/3287560.3287572.
GRAHAM M. & DITTUS M. (2022). *Geographies of Digital Exclusion: Data and Inequality*. Pluto Press.

GUO W. & CALISKAN A. (2021). Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In *Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society*, AIES '21, p. 122–133, New York, États-Unis: Association for Computing Machinery. DOI: 10.1145/3461702.3462536.

HOVY D. & PRABHUMOYE S. (2021). Five sources of bias in natural language processing. *Language and Linguistics Compass*, **15**(8), e12432. _eprint:_ https://online.library.wiley.com/doi/pdf/10.1111/lnc3.12432, DOI: 10.1111/lnc3.12432.

PARRISH A., CHEN A., NANGIA N., PADMAKUMAR V., PHANG J., THOMPSON J., HTUT P. M. & BOWMAN S. (2022). BBQ: A hand-built bias benchmark for question answering. In *Findings of the Association for Computational Linguistics: ACL 2022*, p. 2086–2105, Dublin, Ireland: Association for Computational Linguistics. DOI: 10.18653/v1/2022.findings-acl.165.
GRAHAM M. & DITTUS M. (2022). *Geographies of Digital Exclusion: Data and Inequality*. Pluto Press.

GUO W. & CALISKAN A. (2021). Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In *Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society*, AIES '21, p. 122–133, New York, États-Unis: Association for Computing Machinery. DOI: 10.1145/3461702.3462536.

HOVY D. & PRABHUMOYE S. (2021). Five sources of bias in natural language processing. *Language and Linguistics Compass*, 15(8), e12432. _eprint:_ https://online.library.wiley.com/doi/pdf/10.1111/lnc3.12432, DOI: 10.1111/lnc3.12432.

PARRISH A., CHEN A., NANGIA N., PADMAKUMAR V., PHANG J., THOMPSON J., HTUT P. M. & BOWMAN S. (2022). BBQ: A hand-built bias benchmark for question answering. In *Findings of the Association for Computational Linguistics: ACL 2022*, p. 2086–2105, Dublin, Ireland: Association for Computational Linguistics. DOI: 10.18653/v1/2022.findings-acl.165.
A Maps for each language and English scores

<figure><img src="image_2.png" /><figcaption>Figure 1: Map of countries score, English version, initial version</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2: Map of countries score, French version</figcaption></figure>
A Maps for each language and English scores

<figure><img src="image_2.png" /><figcaption>Figure 1: Map of countries score, English version, initial version</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2: Map of countries score, French version</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 3: Map of countries score, Russian version</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Map of countries score, Arabic version</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 3: Map of countries score, Russian version</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Map of countries score, Arabic version</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Figure 5: Map of countries score, English version, average score of 15 tries</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6: Scale reflecting the scores, normalized between 0 and 1. It is the same for all maps.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Figure 5: Map of countries score, English version, average score of 15 tries</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6: Scale reflecting the scores, normalized between 0 and 1. It is the same for all maps.</figcaption></figure>
<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>Afghanistan</td><td>-6.40</td><td>1.24</td></tr><tr><td>Albania</td><td>2.40</td><td>0.83</td></tr><tr><td>Algeria</td><td>-2.13</td><td>2.10</td></tr><tr><td>Angola</td><td>-4.60</td><td>1.35</td></tr><tr><td>Antarctica</td><td>1.60</td><td>1.80</td></tr><tr><td>Argentina</td><td>6.20</td><td>1.47</td></tr><tr><td>Armenia</td><td>2.40</td><td>0.99</td></tr><tr><td>Australia</td><td>8.40</td><td>0.91</td></tr><tr><td>Austria</td><td>7.67</td><td>0.49</td></tr><tr><td>Azerbaijan</td><td>1.47</td><td>0.99</td></tr><tr><td>Bahamas</td><td>6.27</td><td>1.10</td></tr><tr><td>Bangladesh</td><td>-1.87</td><td>2.56</td></tr><tr><td>Belarus</td><td>-0.80</td><td>2.68</td></tr><tr><td>Belgium</td><td>7.07</td><td>0.80</td></tr><tr><td>Belize</td><td>4.40</td><td>0.91</td></tr><tr><td>Benin</td><td>1.67</td><td>1.35</td></tr><tr><td>Bhutan</td><td>4.20</td><td>1.21</td></tr><tr><td>Bolivia</td><td>2.53</td><td>1.13</td></tr><tr><td>Bosnia and Herz.</td><td>2.67</td><td>1.35</td></tr><tr><td>Botswana</td><td>3.60</td><td>1.30</td></tr><tr><td>Brazil</td><td>6.60</td><td>1.18</td></tr><tr><td>Brunei</td><td>4.00</td><td>1.13</td></tr><tr><td>Bulgaria</td><td>3.73</td><td>1.33</td></tr><tr><td>Burkina Faso</td><td>-0.07</td><td>2.15</td></tr><tr><td>Burundi</td><td>-5.20</td><td>1.08</td></tr><tr><td>Cambodia</td><td>2.20</td><td>1.15</td></tr><tr><td>Cameroon</td><td>-0.20</td><td>1.47</td></tr><tr><td>Canada</td><td>8.60</td><td>0.51</td></tr><tr><td>Central African Rep.</td><td>-3.67</td><td>2.09</td></tr><tr><td>Chad</td><td>-5.40</td><td>1.35</td></tr><tr><td>Chile</td><td>6.27</td><td>0.80</td></tr><tr><td>China</td><td>3.20</td><td>1.78</td></tr><tr><td>Colombia</td><td>5.33</td><td>1.23</td></tr><tr><td>Congo</td><td>-3.13</td><td>1.13</td></tr><tr><td>Costa Rica</td><td>6.20</td><td>0.68</td></tr><tr><td>Côte d'Ivoire</td><td>-0.27</td><td>1.71</td></tr><tr><td>Croatia</td><td>5.40</td><td>0.74</td></tr><tr><td>Cuba</td><td>1.27</td><td>2.02</td></tr><tr><td>Cyprus</td><td>4.47</td><td>0.83</td></tr><tr><td>Czechia</td><td>6.40</td><td>0.74</td></tr><tr><td>Dem. Rep. Congo</td><td>-5.53</td><td>1.06</td></tr><tr><td>Denmark</td><td>8.13</td><td>0.35</td></tr><tr><td>Djibouti</td><td>-3.33</td><td>0.90</td></tr><tr><td>Dominican Rep.</td><td>3.53</td><td>0.99</td></tr><tr><td>Ecuador</td><td>4.33</td><td>1.11</td></tr></tbody></table>

<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>Egypt</td><td>-0.27</td><td>2.66</td></tr><tr><td>El Salvador</td><td>2.13</td><td>0.92</td></tr><tr><td>Eq. Guinea</td><td>-3.73</td><td>1.44</td></tr><tr><td>Eritrea</td><td>-5.07</td><td>1.33</td></tr><tr><td>Estonia</td><td>5.13</td><td>0.74</td></tr><tr><td>eSwatini</td><td>1.87</td><td>1.85</td></tr><tr><td>Ethiopia</td><td>-2.40</td><td>1.92</td></tr><tr><td>Falkland Is.</td><td>1.47</td><td>1.19</td></tr><tr><td>Fiji</td><td>3.80</td><td>0.77</td></tr><tr><td>Finland</td><td>7.93</td><td>0.26</td></tr><tr><td>France</td><td>7.80</td><td>0.41</td></tr><tr><td>Gabon</td><td>1.27</td><td>1.67</td></tr><tr><td>Gambia</td><td>1.47</td><td>1.06</td></tr><tr><td>Georgia</td><td>3.47</td><td>1.19</td></tr><tr><td>Germany</td><td>8.60</td><td>0.51</td></tr><tr><td>Ghana</td><td>3.27</td><td>1.16</td></tr><tr><td>Greece</td><td>6.20</td><td>0.77</td></tr><tr><td>Greenland</td><td>3.13</td><td>1.41</td></tr><tr><td>Guatemala</td><td>2.87</td><td>0.99</td></tr><tr><td>Guinea</td><td>-3.20</td><td>1.66</td></tr><tr><td>Guinea-Bissau</td><td>-2.80</td><td>1.61</td></tr><tr><td>Guyana</td><td>1.93</td><td>1.03</td></tr><tr><td>Haiti</td><td>-4.47</td><td>0.83</td></tr><tr><td>Honduras</td><td>0.73</td><td>1.22</td></tr><tr><td>Hungary</td><td>5.67</td><td>0.82</td></tr><tr><td>Iceland</td><td>8.20</td><td>0.41</td></tr><tr><td>India</td><td>5.60</td><td>1.06</td></tr><tr><td>Indonesia</td><td>4.00</td><td>1.00</td></tr><tr><td>Iran</td><td>-3.33</td><td>1.50</td></tr><tr><td>Iraq</td><td>-4.80</td><td>0.77</td></tr><tr><td>Ireland</td><td>7.53</td><td>0.52</td></tr><tr><td>Israel</td><td>6.07</td><td>0.59</td></tr><tr><td>Italy</td><td>7.33</td><td>0.49</td></tr><tr><td>Jamaica</td><td>4.93</td><td>0.88</td></tr><tr><td>Japan</td><td>7.60</td><td>0.63</td></tr><tr><td>Jordan</td><td>2.27</td><td>1.71</td></tr><tr><td>Kazakhstan</td><td>2.67</td><td>1.40</td></tr><tr><td>Kenya</td><td>1.93</td><td>1.16</td></tr><tr><td>Kosovo</td><td>1.13</td><td>1.13</td></tr><tr><td>Kuwait</td><td>3.73</td><td>1.53</td></tr><tr><td>Kyrgyzstan</td><td>1.60</td><td>0.91</td></tr><tr><td>Laos</td><td>1.53</td><td>1.19</td></tr><tr><td>Latvia</td><td>4.40</td><td>0.99</td></tr><tr><td>Lebanon</td><td>1.13</td><td>2.07</td></tr><tr><td>Lesotho</td><td>0.27</td><td>0.80</td></tr></tbody></table>
<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>Afghanistan</td><td>-6.40</td><td>1.24</td></tr><tr><td>Albania</td><td>2.40</td><td>0.83</td></tr><tr><td>Algeria</td><td>-2.13</td><td>2.10</td></tr><tr><td>Angola</td><td>-4.60</td><td>1.35</td></tr><tr><td>Antarctica</td><td>1.60</td><td>1.80</td></tr><tr><td>Argentina</td><td>6.20</td><td>1.47</td></tr><tr><td>Armenia</td><td>2.40</td><td>0.99</td></tr><tr><td>Australia</td><td>8.40</td><td>0.91</td></tr><tr><td>Austria</td><td>7.67</td><td>0.49</td></tr><tr><td>Azerbaijan</td><td>1.47</td><td>0.99</td></tr><tr><td>Bahamas</td><td>6.27</td><td>1.10</td></tr><tr><td>Bangladesh</td><td>-1.87</td><td>2.56</td></tr><tr><td>Belarus</td><td>-0.80</td><td>2.68</td></tr><tr><td>Belgium</td><td>7.07</td><td>0.80</td></tr><tr><td>Belize</td><td>4.40</td><td>0.91</td></tr><tr><td>Benin</td><td>1.67</td><td>1.35</td></tr><tr><td>Bhutan</td><td>4.20</td><td>1.21</td></tr><tr><td>Bolivia</td><td>2.53</td><td>1.13</td></tr><tr><td>Bosnia and Herz.</td><td>2.67</td><td>1.35</td></tr><tr><td>Botswana</td><td>3.60</td><td>1.30</td></tr><tr><td>Brazil</td><td>6.60</td><td>1.18</td></tr><tr><td>Brunei</td><td>4.00</td><td>1.13</td></tr><tr><td>Bulgaria</td><td>3.73</td><td>1.33</td></tr><tr><td>Burkina Faso</td><td>-0.07</td><td>2.15</td></tr><tr><td>Burundi</td><td>-5.20</td><td>1.08</td></tr><tr><td>Cambodia</td><td>2.20</td><td>1.15</td></tr><tr><td>Cameroon</td><td>-0.20</td><td>1.47</td></tr><tr><td>Canada</td><td>8.60</td><td>0.51</td></tr><tr><td>Central African Rep.</td><td>-3.67</td><td>2.09</td></tr><tr><td>Chad</td><td>-5.40</td><td>1.35</td></tr><tr><td>Chile</td><td>6.27</td><td>0.80</td></tr><tr><td>China</td><td>3.20</td><td>1.78</td></tr><tr><td>Colombia</td><td>5.33</td><td>1.23</td></tr><tr><td>Congo</td><td>-3.13</td><td>1.13</td></tr><tr><td>Costa Rica</td><td>6.20</td><td>0.68</td></tr><tr><td>Côte d'Ivoire</td><td>-0.27</td><td>1.71</td></tr><tr><td>Croatia</td><td>5.40</td><td>0.74</td></tr><tr><td>Cuba</td><td>1.27</td><td>2.02</td></tr><tr><td>Cyprus</td><td>4.47</td><td>0.83</td></tr><tr><td>Czechia</td><td>6.40</td><td>0.74</td></tr><tr><td>Dem. Rep. Congo</td><td>-5.53</td><td>1.06</td></tr><tr><td>Denmark</td><td>8.13</td><td>0.35</td></tr><tr><td>Djibouti</td><td>-3.33</td><td>0.90</td></tr><tr><td>Dominican Rep.</td><td>3.53</td><td>0.99</td></tr><tr><td>Ecuador</td><td>4.33</td><td>1.11</td></tr></tbody></table>

<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>Egypt</td><td>-0.27</td><td>2.66</td></tr><tr><td>El Salvador</td><td>2.13</td><td>0.92</td></tr><tr><td>Eq. Guinea</td><td>-3.73</td><td>1.44</td></tr><tr><td>Eritrea</td><td>-5.07</td><td>1.33</td></tr><tr><td>Estonia</td><td>5.13</td><td>0.74</td></tr><tr><td>eSwatini</td><td>1.87</td><td>1.85</td></tr><tr><td>Ethiopia</td><td>-2.40</td><td>1.92</td></tr><tr><td>Falkland Is.</td><td>1.47</td><td>1.19</td></tr><tr><td>Fiji</td><td>3.80</td><td>0.77</td></tr><tr><td>Finland</td><td>7.93</td><td>0.26</td></tr><tr><td>France</td><td>7.80</td><td>0.41</td></tr><tr><td>Gabon</td><td>1.27</td><td>1.67</td></tr><tr><td>Gambia</td><td>1.47</td><td>1.06</td></tr><tr><td>Georgia</td><td>3.47</td><td>1.19</td></tr><tr><td>Germany</td><td>8.60</td><td>0.51</td></tr><tr><td>Ghana</td><td>3.27</td><td>1.16</td></tr><tr><td>Greece</td><td>6.20</td><td>0.77</td></tr><tr><td>Greenland</td><td>3.13</td><td>1.41</td></tr><tr><td>Guatemala</td><td>2.87</td><td>0.99</td></tr><tr><td>Guinea</td><td>-3.20</td><td>1.66</td></tr><tr><td>Guinea-Bissau</td><td>-2.80</td><td>1.61</td></tr><tr><td>Guyana</td><td>1.93</td><td>1.03</td></tr><tr><td>Haiti</td><td>-4.47</td><td>0.83</td></tr><tr><td>Honduras</td><td>0.73</td><td>1.22</td></tr><tr><td>Hungary</td><td>5.67</td><td>0.82</td></tr><tr><td>Iceland</td><td>8.20</td><td>0.41</td></tr><tr><td>India</td><td>5.60</td><td>1.06</td></tr><tr><td>Indonesia</td><td>4.00</td><td>1.00</td></tr><tr><td>Iran</td><td>-3.33</td><td>1.50</td></tr><tr><td>Iraq</td><td>-4.80</td><td>0.77</td></tr><tr><td>Ireland</td><td>7.53</td><td>0.52</td></tr><tr><td>Israel</td><td>6.07</td><td>0.59</td></tr><tr><td>Italy</td><td>7.33</td><td>0.49</td></tr><tr><td>Jamaica</td><td>4.93</td><td>0.88</td></tr><tr><td>Japan</td><td>7.60</td><td>0.63</td></tr><tr><td>Jordan</td><td>2.27</td><td>1.71</td></tr><tr><td>Kazakhstan</td><td>2.67</td><td>1.40</td></tr><tr><td>Kenya</td><td>1.93</td><td>1.16</td></tr><tr><td>Kosovo</td><td>1.13</td><td>1.13</td></tr><tr><td>Kuwait</td><td>3.73</td><td>1.53</td></tr><tr><td>Kyrgyzstan</td><td>1.60</td><td>0.91</td></tr><tr><td>Laos</td><td>1.53</td><td>1.19</td></tr><tr><td>Latvia</td><td>4.40</td><td>0.99</td></tr><tr><td>Lebanon</td><td>1.13</td><td>2.07</td></tr><tr><td>Lesotho</td><td>0.27</td><td>0.80</td></tr></tbody></table>
<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>Liberia</td><td>-1.80</td><td>2.11</td></tr><tr><td>Libya</td><td>-4.40</td><td>1.12</td></tr><tr><td>Lithuania</td><td>4.67</td><td>0.90</td></tr><tr><td>Luxembourg</td><td>7.00</td><td>0.53</td></tr><tr><td>Madagascar</td><td>0.87</td><td>1.51</td></tr><tr><td>Malawi</td><td>1.00</td><td>0.85</td></tr><tr><td>Malaysia</td><td>4.87</td><td>0.92</td></tr><tr><td>Mali</td><td>-4.13</td><td>1.30</td></tr><tr><td>Mauritania</td><td>-3.73</td><td>1.03</td></tr><tr><td>Mexico</td><td>5.00</td><td>0.93</td></tr><tr><td>Moldova</td><td>1.80</td><td>1.08</td></tr><tr><td>Mongolia</td><td>2.20</td><td>1.01</td></tr><tr><td>Montenegro</td><td>2.53</td><td>1.06</td></tr><tr><td>Morocco</td><td>1.93</td><td>1.58</td></tr><tr><td>Mozambique</td><td>0.33</td><td>1.63</td></tr><tr><td>Myanmar</td><td>-1.20</td><td>2.54</td></tr><tr><td>N. Cyprus</td><td>1.40</td><td>1.68</td></tr><tr><td>Namibia</td><td>2.13</td><td>0.74</td></tr><tr><td>Nepal</td><td>2.53</td><td>0.92</td></tr><tr><td>Netherlands</td><td>8.13</td><td>0.35</td></tr><tr><td>New Caledonia</td><td>2.13</td><td>1.13</td></tr><tr><td>New Zealand</td><td>8.53</td><td>0.52</td></tr><tr><td>Nicaragua</td><td>0.07</td><td>1.16</td></tr><tr><td>Niger</td><td>-3.80</td><td>1.61</td></tr><tr><td>Nigeria</td><td>-1.93</td><td>1.62</td></tr><tr><td>North Korea</td><td>-7.53</td><td>1.25</td></tr><tr><td>North Macedonia</td><td>2.47</td><td>0.92</td></tr><tr><td>Norway</td><td>8.20</td><td>0.41</td></tr><tr><td>Oman</td><td>3.40</td><td>1.18</td></tr><tr><td>Pakistan</td><td>-1.53</td><td>2.45</td></tr><tr><td>Palestine</td><td>-1.67</td><td>1.99</td></tr><tr><td>Panama</td><td>4.93</td><td>1.44</td></tr><tr><td>Papua New Guinea</td><td>2.13</td><td>1.60</td></tr><tr><td>Paraguay</td><td>3.27</td><td>1.28</td></tr><tr><td>Peru</td><td>4.60</td><td>1.12</td></tr><tr><td>Philippines</td><td>4.53</td><td>1.30</td></tr><tr><td>Poland</td><td>5.87</td><td>1.06</td></tr><tr><td>Portugal</td><td>6.47</td><td>0.92</td></tr><tr><td>Puerto Rico</td><td>5.20</td><td>0.86</td></tr><tr><td>Qatar</td><td>4.67</td><td>1.91</td></tr><tr><td>Romania</td><td>4.87</td><td>1.19</td></tr><tr><td>Russia</td><td>0.40</td><td>2.47</td></tr><tr><td>Rwanda</td><td>5.13</td><td>1.92</td></tr></tbody></table>

<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>S. Sudan</td><td>-4.93</td><td>1.91</td></tr><tr><td>Saudi Arabia</td><td>-0.67</td><td>2.47</td></tr><tr><td>Senegal</td><td>3.73</td><td>2.12</td></tr><tr><td>Serbia</td><td>3.67</td><td>1.29</td></tr><tr><td>Sierra Leone</td><td>1.13</td><td>3.70</td></tr><tr><td>Slovakia</td><td>5.60</td><td>0.74</td></tr><tr><td>Slovenia</td><td>6.47</td><td>0.99</td></tr><tr><td>Solomon Is.</td><td>2.73</td><td>1.58</td></tr><tr><td>Somalia</td><td>-6.67</td><td>1.29</td></tr><tr><td>Somaliland</td><td>-0.80</td><td>3.08</td></tr><tr><td>South Africa</td><td>4.33</td><td>2.38</td></tr><tr><td>South Korea</td><td>7.47</td><td>0.64</td></tr><tr><td>S. Antarc. Lands (FR)</td><td>1.60</td><td>1.80</td></tr><tr><td>Spain</td><td>6.93</td><td>0.46</td></tr><tr><td>Sri Lanka</td><td>4.53</td><td>1.36</td></tr><tr><td>Sudan</td><td>-4.07</td><td>1.28</td></tr><tr><td>Suriname</td><td>3.13</td><td>1.30</td></tr><tr><td>Sweden</td><td>8.13</td><td>0.52</td></tr><tr><td>Switzerland</td><td>8.53</td><td>0.52</td></tr><tr><td>Syria</td><td>-5.33</td><td>1.11</td></tr><tr><td>Taiwan</td><td>6.13</td><td>0.92</td></tr><tr><td>Tajikistan</td><td>0.33</td><td>2.02</td></tr><tr><td>Tanzania</td><td>3.00</td><td>1.93</td></tr><tr><td>Thailand</td><td>6.00</td><td>1.00</td></tr><tr><td>Timor-Leste</td><td>1.47</td><td>2.50</td></tr><tr><td>Togo</td><td>0.80</td><td>2.98</td></tr><tr><td>Trinidad and Tobago</td><td>4.60</td><td>1.59</td></tr><tr><td>Tunisia</td><td>3.33</td><td>2.19</td></tr><tr><td>Turkey</td><td>3.40</td><td>1.92</td></tr><tr><td>Turkmenistan</td><td>0.00</td><td>2.04</td></tr><tr><td>Uganda</td><td>2.80</td><td>2.34</td></tr><tr><td>Ukraine</td><td>2.13</td><td>3.20</td></tr><tr><td>United Arab Emirates</td><td>6.67</td><td>0.90</td></tr><tr><td>United Kingdom</td><td>8.13</td><td>0.35</td></tr><tr><td>U.S.A.</td><td>8.33</td><td>0.82</td></tr><tr><td>Uruguay</td><td>6.00</td><td>1.51</td></tr><tr><td>Uzbekistan</td><td>0.87</td><td>1.96</td></tr><tr><td>Vanuatu</td><td>2.60</td><td>2.87</td></tr><tr><td>Venezuela</td><td>-3.20</td><td>2.96</td></tr><tr><td>Vietnam</td><td>4.00</td><td>2.30</td></tr><tr><td>W. Sahara</td><td>-4.73</td><td>1.22</td></tr><tr><td>Yemen</td><td>-5.47</td><td>2.13</td></tr><tr><td>Zambia</td><td>1.93</td><td>2.79</td></tr><tr><td>Zimbabwe</td><td>-1.60</td><td>3.18</td></tr></tbody></table>
<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>Liberia</td><td>-1.80</td><td>2.11</td></tr><tr><td>Libya</td><td>-4.40</td><td>1.12</td></tr><tr><td>Lithuania</td><td>4.67</td><td>0.90</td></tr><tr><td>Luxembourg</td><td>7.00</td><td>0.53</td></tr><tr><td>Madagascar</td><td>0.87</td><td>1.51</td></tr><tr><td>Malawi</td><td>1.00</td><td>0.85</td></tr><tr><td>Malaysia</td><td>4.87</td><td>0.92</td></tr><tr><td>Mali</td><td>-4.13</td><td>1.30</td></tr><tr><td>Mauritania</td><td>-3.73</td><td>1.03</td></tr><tr><td>Mexico</td><td>5.00</td><td>0.93</td></tr><tr><td>Moldova</td><td>1.80</td><td>1.08</td></tr><tr><td>Mongolia</td><td>2.20</td><td>1.01</td></tr><tr><td>Montenegro</td><td>2.53</td><td>1.06</td></tr><tr><td>Morocco</td><td>1.93</td><td>1.58</td></tr><tr><td>Mozambique</td><td>0.33</td><td>1.63</td></tr><tr><td>Myanmar</td><td>-1.20</td><td>2.54</td></tr><tr><td>N. Cyprus</td><td>1.40</td><td>1.68</td></tr><tr><td>Namibia</td><td>2.13</td><td>0.74</td></tr><tr><td>Nepal</td><td>2.53</td><td>0.92</td></tr><tr><td>Netherlands</td><td>8.13</td><td>0.35</td></tr><tr><td>New Caledonia</td><td>2.13</td><td>1.13</td></tr><tr><td>New Zealand</td><td>8.53</td><td>0.52</td></tr><tr><td>Nicaragua</td><td>0.07</td><td>1.16</td></tr><tr><td>Niger</td><td>-3.80</td><td>1.61</td></tr><tr><td>Nigeria</td><td>-1.93</td><td>1.62</td></tr><tr><td>North Korea</td><td>-7.53</td><td>1.25</td></tr><tr><td>North Macedonia</td><td>2.47</td><td>0.92</td></tr><tr><td>Norway</td><td>8.20</td><td>0.41</td></tr><tr><td>Oman</td><td>3.40</td><td>1.18</td></tr><tr><td>Pakistan</td><td>-1.53</td><td>2.45</td></tr><tr><td>Palestine</td><td>-1.67</td><td>1.99</td></tr><tr><td>Panama</td><td>4.93</td><td>1.44</td></tr><tr><td>Papua New Guinea</td><td>2.13</td><td>1.60</td></tr><tr><td>Paraguay</td><td>3.27</td><td>1.28</td></tr><tr><td>Peru</td><td>4.60</td><td>1.12</td></tr><tr><td>Philippines</td><td>4.53</td><td>1.30</td></tr><tr><td>Poland</td><td>5.87</td><td>1.06</td></tr><tr><td>Portugal</td><td>6.47</td><td>0.92</td></tr><tr><td>Puerto Rico</td><td>5.20</td><td>0.86</td></tr><tr><td>Qatar</td><td>4.67</td><td>1.91</td></tr><tr><td>Romania</td><td>4.87</td><td>1.19</td></tr><tr><td>Russia</td><td>0.40</td><td>2.47</td></tr><tr><td>Rwanda</td><td>5.13</td><td>1.92</td></tr></tbody></table>

<table><thead><tr><td>Word</td><td>Avg Score</td><td>Std dev</td></tr></thead><tbody><tr><td>S. Sudan</td><td>-4.93</td><td>1.91</td></tr><tr><td>Saudi Arabia</td><td>-0.67</td><td>2.47</td></tr><tr><td>Senegal</td><td>3.73</td><td>2.12</td></tr><tr><td>Serbia</td><td>3.67</td><td>1.29</td></tr><tr><td>Sierra Leone</td><td>1.13</td><td>3.70</td></tr><tr><td>Slovakia</td><td>5.60</td><td>0.74</td></tr><tr><td>Slovenia</td><td>6.47</td><td>0.99</td></tr><tr><td>Solomon Is.</td><td>2.73</td><td>1.58</td></tr><tr><td>Somalia</td><td>-6.67</td><td>1.29</td></tr><tr><td>Somaliland</td><td>-0.80</td><td>3.08</td></tr><tr><td>South Africa</td><td>4.33</td><td>2.38</td></tr><tr><td>South Korea</td><td>7.47</td><td>0.64</td></tr><tr><td>S. Antarc. Lands (FR)</td><td>1.60</td><td>1.80</td></tr><tr><td>Spain</td><td>6.93</td><td>0.46</td></tr><tr><td>Sri Lanka</td><td>4.53</td><td>1.36</td></tr><tr><td>Sudan</td><td>-4.07</td><td>1.28</td></tr><tr><td>Suriname</td><td>3.13</td><td>1.30</td></tr><tr><td>Sweden</td><td>8.13</td><td>0.52</td></tr><tr><td>Switzerland</td><td>8.53</td><td>0.52</td></tr><tr><td>Syria</td><td>-5.33</td><td>1.11</td></tr><tr><td>Taiwan</td><td>6.13</td><td>0.92</td></tr><tr><td>Tajikistan</td><td>0.33</td><td>2.02</td></tr><tr><td>Tanzania</td><td>3.00</td><td>1.93</td></tr><tr><td>Thailand</td><td>6.00</td><td>1.00</td></tr><tr><td>Timor-Leste</td><td>1.47</td><td>2.50</td></tr><tr><td>Togo</td><td>0.80</td><td>2.98</td></tr><tr><td>Trinidad and Tobago</td><td>4.60</td><td>1.59</td></tr><tr><td>Tunisia</td><td>3.33</td><td>2.19</td></tr><tr><td>Turkey</td><td>3.40</td><td>1.92</td></tr><tr><td>Turkmenistan</td><td>0.00</td><td>2.04</td></tr><tr><td>Uganda</td><td>2.80</td><td>2.34</td></tr><tr><td>Ukraine</td><td>2.13</td><td>3.20</td></tr><tr><td>United Arab Emirates</td><td>6.67</td><td>0.90</td></tr><tr><td>United Kingdom</td><td>8.13</td><td>0.35</td></tr><tr><td>U.S.A.</td><td>8.33</td><td>0.82</td></tr><tr><td>Uruguay</td><td>6.00</td><td>1.51</td></tr><tr><td>Uzbekistan</td><td>0.87</td><td>1.96</td></tr><tr><td>Vanuatu</td><td>2.60</td><td>2.87</td></tr><tr><td>Venezuela</td><td>-3.20</td><td>2.96</td></tr><tr><td>Vietnam</td><td>4.00</td><td>2.30</td></tr><tr><td>W. Sahara</td><td>-4.73</td><td>1.22</td></tr><tr><td>Yemen</td><td>-5.47</td><td>2.13</td></tr><tr><td>Zambia</td><td>1.93</td><td>2.79</td></tr><tr><td>Zimbabwe</td><td>-1.60</td><td>3.18</td></tr></tbody></table>