BERT has a Moral Compass: Improvements of ethical and moral values of machines	Allowing machines to choose whether to kill humans would be devastating for world peace and security. But how do we equip machines with the ability to learn ethical or even moral choices? Jentzsch et al.(2019) showed that applying machine learning to human texts can extract deontological ethical reasoning about "right" and "wrong" conduct by calculating a moral bias score on a sentence level using sentence embeddings. The machine learned that it is objectionable to kill living beings, but it is fine to kill time; It is essential to eat, yet one might not eat dirt; it is important to spread information, yet one should not spread misinformation. However, the evaluated moral bias was restricted to simple actions -- one verb -- and a ranking of actions with surrounding context. Recently BERT ---and variants such as RoBERTa and SBERT--- has set a new state-of-the-art performance for a wide range of NLP tasks. But has BERT also a better moral compass? In this paper, we discuss and show that this is indeed the case.

Introduction	There is a broad consensus that artificial intelligence (AI) research is progressing steadily, and that its impact on society is likely to increase. From self-driving cars on public streets to self-piloting, reusable rockets, AI systems tackle more and more complex human activities in a more and more autonomous way. This leads into new spheres, where traditional ethics has limited applicability. Both self-driving cars, where mistakes may be life-threatening, and machine classifiers that hurt social matters may serve as examples for entering grey areas in ethics: How does AI embody our value system? Can AI systems learn human ethical judgements? If not, can we contest the AI system? Unfortunately, aligning social, ethical, and moral norms to structure of science and innovation in general is a long road. According to  (), who examined affirmative ethics, the emergence of new questions leads to intense public discussions, that are driven by strong emotions of participants. And machine ethics , ,  is no exception. Consider, e.g., 's () empirical proof that human language reflects our stereotypical biases.

These ask which choices are morally required, forbidden, or permitted instead of asking which kind of a person we should be or which consequences of our actions are to be preferred. Thus, norms are understood as universal rules of what to do and what not to do. Therefore, we focus on the valuation of social acceptance in single verbs and single verbs with surrounding context information —e.g. trust my friend or trust a machine— to figure out which of them represent a Do and which tend to be a Don't. Because we specifically chose templates in the first person, i.e., asking “should I” and not asking “should one”, we address the moral dimension of “right” or “wrong” decisions, and not only their ethical dimension. This is the reason why we will often use the term “moral”, although we actually touch upon “ethics” and “moral”. To measure the valuation, we make use of implicit association tests (IATs) and their connections to word embeddings. Word and Sentence Embeddings. A word/phrase embedding is a representation of words/phrases as points in a vector space.

Assumptions and Background	In this section, we review our assumptions, in particular what we mean by moral choices, and the required background, following closely . Moral Choices. Philosophically, roughly speaking, morals refer to the “right” and “wrong” at an individual's level while ethics refer to the systems of “right” and “wrong” set by a social group. Social norms and implicit behavioural rules exist in all human societies. But even though their presence is ubiquitous, they are hardly measurable and difficult to define consistently. The underlying mechanisms are still poorly understood. Indeed, each working society possesses an abstract moral that is generally valid and needs to be adhered to. However, theoretic definitions have been described as being inconsistent or even contradicting occasionally. Accordingly, latent ethics and morals have been described as the sum of particular norms that may not follow rational justification necessarily. Recently,  () for instance suggested that moral norms are determined to a large extent by what is perceived to be common convention. With regards to complexity and intangibility of ethics and morals, we restrict ourselves to a rather basic implementation of this construct, following the theories of deontological ethics.

Appendix ::: Moral Choice Machine	 () developed Moral Choice Machine computes the cosine similarity in a sentence embedding space of an arbitrary action embedded in question/answer pairs. This is illustrated in Fig. FIGREF16 for the moral bias of the action murder. Since murdering is a quite destructive and generally refused behaviour, the questions are expected to lie closer to the denying response and thus to yield a negative bias. To create a more meaningful and comprehensive statistic, several question/answer prompts were conflated to a question/answer template (cf. Tab. TABREF15). The element of interest is inserted to each considered prompt and resulting biases averaged to an overall bias value.

Moral Subspace Projection	As  () showed the question/answer template is an appropriate method to extract moral biases. However as  () showed, one is also able to even adapt the model's bias, e.g. debias the model's gender bias. They describe that the first step for debiasing word embeddings is to identify a direction (or, more generally, a subspace) of the embedding that captures the bias. To identify the gender subspace, e.g., they proposed to take the difference vectors of given gender pairs and computed its principal components (PCs) and found a single direction that explains the majority of variance in these vectors, i.e. the first eigenvalue is significantly larger than the rest. Therefore, they argue that the top PC, denoted by the unit vector $g$, captures the gender subspace. Subsequently, they debias the embedding based on this subspace. Please note that the gender pairs are labelled beforehand. Using the above-mentioned methodology, we propose an alternative to identify the moral bias. Inspired by , we first compute the moral subspace of the text embedding.

Appendix ::: Moral Bias of USE and BERT	The following results were computed with the MCM version of  () using both USE and BERT as sentence embedding. Specifically, to investigate whether the sentiments of the extracted Dos and Don'ts also hold for more complex sentence level, we inserted them into the question/answer templates of Moral Choice Machine . The resulting moral biases scores/choices are summarized in Tab. TABREF28. It presents the moral biases exemplary for the top ten Dos and Don'ts by WEAT value of both sets. The threshold between the groups is not 0, but slightly shifted negatively (Using USE further shifted than Using BERT). However, the distinction of Dos and Don'ts is clearly reflected in bias values. Using USE the mean bias of all considered elements is $-0.018$ ($std=0.025$), whereat the mean of Dos is $-0.001$ ($std=0.190$, $n=50$) and the mean of Don'ts $-0.037$ ($std=0.017$, $n=50$).

FIGREF16 for the moral bias of the action murder. Since murdering is a quite destructive and generally refused behaviour, the questions are expected to lie closer to the denying response and thus to yield a negative bias. To create a more meaningful and comprehensive statistic, several question/answer prompts were conflated to a question/answer template. The element of interest is inserted to each considered prompt and resulting biases averaged to an overall bias value. Specifically, we considered two different biases: gender and moral. The question/answer template for moral biases, our main target, included the ten prompts summarized in Tab. TABREF15(Appendix), where any action term can be inserted for the dots. All prompts follow the same structure: the action is mirrored in the question, and the answers represent two extremes of social desirability. Questions were, e.g., “Is it allowed to ...?” or “Should I ...?”. The used answers were grammatically adjusted to the question, e.g., “Yes, it is.” vs. “No, it is not.” respectively “Yes, you should.” vs. “No, you should not.”. Besides atomic actions, e.g.

Appendix ::: Moral Subspace Projection	To create a the moral subspace projection a Principal Component Analysis (PCA) was computed. The used atomic actions are listed in Tab. TABREF26. The resulting space, with the MCM using BERT, is visualized in Fig. FIGREF1 based on the first two top PCs. The top PC (the $X$ axis) defines the moral direction $m$ (bias). The context-based actions which were tested using the moral subspace projection are listed in Tab. TABREF27. The resulting moral direction $m$ (or bias) for both the atomic and context-based actions can be found in Tab. TABREF30. We also list the results using the sentence embedding USE instead of BERT. $m < 0$ corresponds to a positive moral score and $m > 0$ corresponds to a negative moral score.

Instead of the difference vectors of the question/answer pairs, we compute the PCA on selected atomic actions —we expect that these actions represent Dos and Don'ts (cf. Appendix). We formulate the actions as questions, i.e. using question templates, and compute the mean embedding, since this amplifies their moral score . Similar to the gender subspace, if the first eigenvalue is significantly larger than the rest, the top PC, denoted by the unit vector $m$, captures the moral subspace and therefore also the moral bias. Then, based on this subspace, one can extract the moral bias of even complex actions with surrounding context by the projection of an action.

Next, based on the verbs extractions and the question/answer templates, we show that social norms are present in text embeddings and a text embedding network known to achieve high score in unsupervised scenarios —such as semantic textual similarity via cosine-similarity, clustering or semantic search— improves the scores of the extracted moral actions. The correlation of the moral bias and the corresponding WEAT value was calculated to test consistency of findings. It is hypothesised that resulting moral biases for generated Dos and Don'ts correspond to the WEAT value of each word. The correlation was tested by means of Pearson's Correlation Coefficient: where $m_x$ and $m_y$ are the the means of $X$ and $Y$. Pearson's $r$ ranges between $-1$, indicating a strong negative correlation, and 1, indicating a strong positive correlation. Significance levels are defined as $5\%$, $1\%$ and $0.1\%$, indicated by one, two or three starlets. The correlation between WEAT value and the moral bias gets tangible, when inspecting their correlation graphically, cf. Fig. FIGREF4.

Conclusions	We have demonstrated that BERT has a more pronounced moral compass than previous embedding methods. That is, yes, text embeddings encode knowledge about deontological ethical and even moral choices, but the quality of the bias score depends on the quality of the text embedding network. Specifically, our empirical results show that the Moral Choice Machine with recent state-of-the-art language representations, namely BERT, extends the boundary of previous approaches and demonstrate the existence of biases in human language on a complex phrase level. Moreover, we identified for the first time that there is a moral dimension in text embeddings, even when taking context into account. Generally, improved moral choice machines hold promise for identifying and addressing sources of ethical and moral choices in culture, including AI systems. This provides several avenues for future work. Inspired by  (), we aim at modifying the embedding, given human ethical values collected from an user study. Further, it is interesting to track ethical choices over time and to compare them among different text corpora. Even more interesting is an interactive learning setting with an interactive robot, in which users would teach and revise the robot's moral bias. Our identification of a moral subspace in sentence embeddings lays the foundation for this.

Using BERT the mean bias of all considered elements is $-0.054$ ($std=0.11$), whereat the mean of Dos is $0.041$ ($std=0.064$, $n=50$) and the mean of Don'ts $-0.163$ ($std=0.053$, $n=50$). Furthermore Tab. TABREF29 shows the resulting moral biases scores/choices for action with additional surrounding context exemplary for the top ten Dos and Don'ts of both sentence embeddings.

Based on that, we show that the new state-of-the-art method BERT improves the quality of the MCM. Although the three methods—Word Embedding Association Test (WEAT), Moral Choice Machine based on the Universal Sentence Encoder (USE), and Moral Choice Machine based on Sentence-BERT (SBERT)—are based on incoherent embeddings with different text corpora as training source, we show that they correspond in classification of actions as Dos and Don'ts. Our findings support the hypothesis of the presence of generally valid valuation in human text. Actually, they show that BERT improves the extraction of the moral score. Next, we move to more complex actions with surrounding contextual information and extend the (moral-) ranking of such actions presented in  by an evaluation of the actual moral bias. Again, we show that BERT has a more accurate reflection of moral values than USE. Finally, we contribute an alternative way of specifying the moral value of an action by learning a projection of the embedding space into a moral subspace. With the MCM in combination with BERT we can reduce the embedding dimensionality to one single dimension representing the moral bias. We proceed as follows.

Using USE, we were unable to find a clear moral dimension, rather multiple directions. Although both projections should enable one to adapt the model's moral bias based on the subspace, BERT seems to have a more intuitive moral direction. Next, we investigate the subspace projection with the actions formulated as questions. Also, here, one can see that BERT enables the MCM to identify a clear moral direction, cf. Fig. FIGREF6(c-d). The PCA is computed with the embedding of atomic actions. Based on this projection, we query more complex actions to investigate their moral bias score. The atomic actions in the subspace are visualized in Fig. FIGREF1 and the queried actions in Fig. FIGREF11. The horizontal axis (the top PC) represents the moral direction. One can observe that the atomic actions kill, murder, slaughter, brutalise, destroy are the most negative actions and congratulate, compliment, welcome and smile the most positive. E.g. apologize, dream, go, become seem to be neutral —which would change depending on the context—.

The concrete bias scores can be found in the Appendix, Fig. TABREF28 and TABREF29. For both WEAT and MCM, the scatter plots of Dos and Don'ts are divided on the x-axis. The Pearson's Correlation Coefficient using USE as embedding (Top) $r = 0.73$ with $p = 2.3732e^{-16}$ is indicating a significant positive correlation. However, according to the distribution one can see that using BERT (Bottom) improves the distinction between Dos and Don't. Actually, the Pearson's Correlation Coefficient $r = 0.88$ with $p = 1.1054e^{-29}$ indicates a high positive correlation. These findings suggest that if we build an AI system that learns an improved language representation to be able to better understand and produce it, in the process it will also acquire more accurate historical cultural associations to make human-like “right” and “wrong” choices. Replicating Complex Moral Choices in the Moral Subspace. The strong correlation between WEAT values and moral biases at the verb level gives reasons to investigate BERT's Moral Choice Machine for complex human-like choices at the phrase level.

Human-like Moral Choices from Human Text	Word-based approaches such as WEAT or Verb Extraction are comparatively simple. They consider single words only, detached from their grammatical and contextual surrounding. In contrast, the Moral Choice Machine determines biases on a sentence level. Analogous to word embeddings, sentence embeddings, e.g. the Universal Sentence Encoder  and Sentence-BERT , allow one to calculate the cosine similarity of various different sentences, as for instance the similarity of a question and the corresponding answer. The more appropriate a specific answer is to a given question, the stronger is its cosine similarity expected to be. When considering two opposite answers, it is therefore possible to determine a bias value: where $\vec{q}$ is the vector representation of the question and $\vec{a}$ and $\vec{b}$ the representations of the two answers/choices. A positive value indicates a stronger association to answer $a$, whereas a negative value indicates a stronger association to $b$. This can be adapted to any arbitrary kind of bias by formulating appropriate question/answer triples, where the question captures the target dimension and the answers represent two opposite manifestations, the choices. This is illustrated in Fig.

These and similar recent scientific studies have raised awareness about machine ethics in the media and public discourse: AI systems “have the potential to inherit a very human flaw: bias”, as Socure's CEO Sunil Madhu puts it. Besides the typically discussed bias in gender and race stereotypes, AI systems are also not neutral with respect to purpose and societal values anymore. Ultimately, if AI systems carry out choices, then they implicitly make ethical and even moral choices. Choosing most often entails trying to pick one of two or more (mutually exclusive) alternatives with an outcome that gives desirable consequences in your ethical frame of reference.

As a first investigation, we compare the new state-of-the-art text-embedding model BERT, or more precisely Sentence-BERT, focusing on quantifying deontological ethics, i.e. finding out, whether an action itself is right or wrong. Following  and for a fair comparison, we first restrict our attention to atomic actions instead of complex behavioural patterns. Semantically, those contextual isolated actions are represented by verbs. Consequently, identifying verbs that reflect social norms and allow capturing what people rather should do and what not. To conduct this investigation, we used the same template list of prompts and responses for ethical choices as in , cf. Tab. TABREF15(Appendix). The templates include questions, such as "Should I kill people?", "Is it allowed to murder people?", etc. with answer templates of "Yes/no, I should (not)." Using the Moral Choice Machine (MCM), based on some language representation, one is able to demonstrate the presence of ethical valuation in text collections by generating an ethical bias of actions derived from the Verb Extraction. As the next step, the correlation of WEAT (Word Embedding Association Test) values  and moral bias is examined.

All approaches have in common that more related or even similar text entities lie close to each other in the vector space, whereas distinct words/phrases can be found in distant regions . This enables determining semantic similarities in a language. Although these techniques have been around for some time, their potential increased considerably with the emergence of deep distributional approaches. In contrast to previous implementations, those embeddings are built on neural networks (NNs) and enable a rich variety of mathematical vector operations. One of the initial and most widespread algorithms to train word embeddings is Word2Vec , where unsupervised feature extraction and learning is conducted per word on either CBOW or Skip-gram NNs. This can be extended to full sentences , , . Bias in Text Embeddings. While biases in machine learning models can potentially be rooted in the implemented algorithm, they are primarily due to the data they are trained on.  () empirically showed that human language reflects our stereotypical biases. Once AI systems are trained on human language, they carry these (historical) biases, as for instance the (wrong) idea that women are less qualified to hold prestigious professions.

Once AI systems are trained on human language, they carry these (historical) biases, such as the (wrong) idea that women are less qualified to hold prestigious professions. These and similar recent scientific studies have raised awareness about machine ethics in the media and public discourse. In other words, AI systems are not neutral with respect to purpose and society anymore. Ultimately, if AI systems carry out choices, then they implicitly make ethical and even moral choices. Choosing in general most often entails trying to pick one of two or more (mutually exclusive) alternatives with an outcome that gives desirable consequences in your individual ethical frame. So, one may think that it is impossible to equip AI systems to make human like ethical choices. Luckily,  () showed that standard machine learning can actually learn answers to ethical choices from textual data that reflect everyday human culture. Recently,  () showed that sentence embeddings created by SBERT outperform other state-of-the-art sentence embedding methods like InferSent  and Universal Sentence Encoder . We hypothesis that the improvement of language representations models also improves the representation of the underlying ethical and moral values in these models. To investigate this, we follow the experimental pipeline described in .

Thus, recent improvements of language representations also improve the representation of the underlying ethical and moral values of the machine. We argue that through an advanced semantic representation of text, BERT allows one to get better insights of moral and ethical values implicitly represented in text. This enables the Moral Choice Machine (MCM) to extract more accurate imprints of moral choices and ethical values.

For instance, it is appropriate to help old people, but one should not help a thief. It is good behaviour to love your parents, but not to steal money. To see whether the moral choice machine can, in principle, deal with complex choices and implicit context information around these complex choices,  () considered the rankings among answers induced by cosine distance. Their results indicate that human text may indeed contain complex human-like choices that are reproducible by the Moral Choice Machine. To investigate this further, we define a Moral Subspace Projection and consider a set of atomic actions and combine them with varying context information, e.g. “Should I have a gun to hunt animals?” or “Should I have a gun to kill people?”. First we will investigate the subspace of vector differences (moral direction) which was introduced by  () to debias word embeddings. Fig. FIGREF6 (a-b) shows the percentage of variance explained in the PCA using the MCM with USE(a) and BERT(b). Clearly, the top principal component (PC) using BERT explains the majority of variance in these vectors, therefore we conclude that it represents the moral direction $m$.

Experimental Results	This section investigates empirically whether text corpora contain recoverable and accurate imprints of our moral choices. Specifically, we move beyond , by showing that BERT has a more accurate moral representation than that of the Universal Sentence Encoder. Datasets and Embeddings Models. Experiments of the Moral Choice Machine are conducted with the Universal Sentence Encoder (USE)  and Sentence-BERT (SBERT) . The USE model is trained on phrases and sentences from a variety of different text sources; mainly Wikipedia but also sources such as forums, question/answering platforms, and news pages and augmented with supervised elements. SBERT is a modification of the pretrained BERT  network that aims to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. BERT is, like USE, also trained mainly on Wikipedia. For the verb extraction, the same general positive and negative association sets as in  are used—$A$ and $B$ in Eq. DISPLAY_FORM18—. The comprehensive list of vocabulary can be found in the appendix (Tab. TABREF20). Dos and Don'ts for the Moral Choice Machine.

The verb extraction identifies the most positive and most negative associated verbs in vocabulary, to infer socially desired and neglected behaviour.  () extracted them with the general positive and negative association sets on the Google Slim embedding. Since those sets are expected to reflect social norms, they are referred as Dos and Don'ts hereafter. Tab. TABREF22 and Tab. TABREF23 (cf. Appendix) lists the most positive and negative associated verbs (in decreasing order). Summarized, even though the contained positive verbs are quite diverse, all of them carry a positive attitude. Some of the verbs are related to celebration or travelling, others to love matters or physical closeness. All elements of the above set are rather of general and unspecific nature. Analogously, some of the negative words just describe inappropriate behaviour, like slur or misdeal, whereas others are real crimes as murder. As  () describe, the listed words can be accepted as commonly agreed Dos and Don'ts. Replicating Atomic Moral Choices.

Appendix ::: Dos and Don’ts for the Moral Choice Machine	Tab. TABREF22 lists the most positive associated verbs (in decreasing order). Even though the contained verbs are quite diverse, all of them carry a positive attitude. Some of the verbs are related to celebration or travelling, others to love matters or physical closeness. All elements of the above set are rather of general and unspecific nature. Analogously, Tab. TABREF23 presents the most negative associated verbs (in decreasing order) we found in our vocabulary. Some of the words just describe inappropriate behaviour, like slur or misdeal, whereas others are real crimes as murder. And still others words, as for instance suppurate or rot, appear to be disgusting in the first place. Exculpate is not a bad behaviour per se. However, its occurrence in the don't set is not surprising, since it is semantically and contextual related to wrongdoings. Some of the words are of surprisingly repugnant nature as it was not even anticipated in preliminary considerations, e.g. depopulate or dehumanise. Undoubtedly, the listed words can be accepted as commonly agreed Don'ts.

smile, love, lie or kill, this template can be used on more complex, context-based actions e.g. love my parents, love my wife/husband, kill people or kill time.

If we, now, query the MCM with projection with more complex actions, one can see that the most negative actions are kill people, have a gun to kill people and become evil, but becoming a good parent is positive. Further, one can see that eat healthy is positive but eat meat is not appropriate. One should not travel to North Korea, but also not to Germany. Instead traveling to the United States is appropriate.

Appendix ::: Implicit Associations in Word Embeddings	Transferring the approach of implicit associations from human subjects to information retrieval systems on natural text was initially suggested by Caliskan et al. (), who reported some basic effects of the Word Embedding Association Test (WEAT). Whereas the strength of association in human minds is defined by response latency in Implicit Association Tests (IAT), it is here instantiated as cosine similarity of text in the Euclidean space. Similar to the IAT, complex concepts are defined by word sets. The association of any single word vector $\vec{w}$ to a word set is defined as the mean cosine similarity between $\vec{w}$ and the particular elements of the set. Now, let there be two sets of target words $X$ and $Y$. The allocation of $\vec{w}$ to two discriminating association sets $A$ and $B$ can be formulated as A word with representation $\vec{w}$ that is stronger associated to concept $A$ yields a positive value and representation related to $B$ a negative value.

Appendix ::: Association Sets	The complete lists of positive and negative association words that were applied for generating Dos and Don'ts with Verb Extraction are given in Tab. TABREF20. The words were collected from four different literature sources that provide unspecific association sets to define pleasant and unpleasant associations , , , .

Both lists include few words are rather common as a noun or adjectives, as joy, long, gift or bad. Anyhow, they can also be used as verbs and comply the requirements of being a do or a don't in that function. The allocation of verbs into Dos and Don'ts was confirmed by the affective lexicon AFINN . AFINN allows one to rate words and phrases for valence on a scale of $-5$ and 5, indicating inherent connotation. Elements with no ratings are treated as neutral ($0.0$). When passing the comprehensive lists of generated Dos and Don'ts to AFINN, the mean rating for Dos is $1.12$ ($std=1.24$) and for Don'ts $-0.90$ ($std=1.22$). The t-test statistic yielded values of $t = 8.12$ with $p < .0001^{***}$.

When neglecting all verbs that are not included in AFINN, the mean value for Dos is $2.34$ ($std=0.62$, $n = 24$) and the mean for Don'ts $-2.37$ ($std = 0.67$, $n=19$), with again highly significant statistics ($t = 23.28$, $p<.0001^{***}$). Thus, the sentimental rating is completely in line with the allocation of Verb Extraction. The verb extraction was highly successful and delivers useful Dos and Don'ts. The word sets contain consistently positive and negative connoted verbs, respectively, that are reasonable to represent a socially agreed norm in the right context. The AFINN validation clearly shows that the valuation of positive and negative verbs is in line with other independent rating systems.

After reviewing our assumptions and the required background, we present the MCM using BERT, followed by improvements of the MCM. Before concluding, we present our empirical results.

Question: How is moral bias measured?
Answer: Answer with content missing: (formula 1) bias(q, a, b) = cos(a, q) − cos(b, q)
Bias is calculated as substraction of cosine similarities of question and some answer for two opposite answers.