Tracking historical changes in trustworthiness using machine learning analyses of facial cues in paintings
Authors: Nicolas Baumard
Julie Grèzes
Lou Safra
Coralie Chevallier
Journal: Nature Communications (2020)
Discipline: Multidisciplinary
Keywords: Social trust; societal outcomes; economic performance; crime rates; inclusive institutions
Affiliations: Laboratoire de Neurosciences Cognitives, Département d'études cognitives, ENS, PSL, Research University, INSERM, Paris, France. ., Laboratoire de Neurosciences Cognitives, Département d'études cognitives, ENS, PSL, Research University, INSERM, Paris, France., Institut Jean Nicod, Département d'études cognitives, ENS, EHESS, PSL Research University, CNRS, Paris, France. .

Abstract:  Social trust is linked to a host of positive societal outcomes, including improved economic performance, lower crime rates and more inclusive institutions. Yet, the origins of trust remain elusive, partly because social trust is difficult to document in time. Building on recent advances in social cognition, we design an algorithm to automatically generate trustworthiness evaluations for the facial action units (smile, eye brows, etc.) of European portraits in large historical databases. Our results show that trustworthiness in portraits increased over the period 1500-2000 paralleling the decline of interpersonal violence and the rise of democratic values observed in Western Europe. Further analyses suggest that this rise of trustworthiness displays is associated with increased living standards. 

Retracted: No

PubPeer comments:
 This article is generating a lot of heat on Twitter. I d just like to focus on one point, which is, it seems to me, the rather naive highlighting of the correlation between  trustworthiness and GDP per capita (Fig. 1B). ![file] About this correlation, the authors state:  Demonstrating that the association between GDP and the rise of trustworthiness is causal would of course require additional data. Well, indeed. (And why do the authors plot  z-score instead of the actual numbers? Has the z-score of the GDP been plotted also?) Given that the  trustworthiness score increases with time, presumably there are about a billion other variables that would also correlate with it, including no doubt the sale of avocados and the number of toothbrushes in circulation. The existence of one or two predictors that correlate less strongly or even don t adds very little to the debate. (And remember,  the difference between significant and non-significant is non-significant .) , 
 We agree a lot of changes have occurred during these centuries. However, we focused on GDP based on existing studies showing a link between economic wealth and social trust at the country level, but also at more local levels (such as neighborhoods) and even at individual level in current societies. We are aware that GDP is only an imperfect measure of economic well-being (for instance, it does not account for inequalities, which may have an important impact on individuals psychology). However, it has been shown to correlate with social trust in current societies and historians and economists have reconstructed this measure for extended periods of time (we did not find similar historical data on GINI for instance). Still, because GDP increased with time it was important for us to test whether adding GDP better accounted for variations in trustworthiness displays in portraits than time alone. That is why we included Bayes Factor analyses in our paper, which supported this hypothesis. Finally, we compared the effect of GDP with the effect of another variable that increased with time: democratization. We used the polity2 index, because it appeared as a complete index which has been computed for quite a long period of time. Contrary to the effect of GDP, the effect of this democratization index was not robust to the inclusion of time as a predictor. In a addition, a second Bayes Factor analysis provided evidence that a model including GDP and time better accounted for our data than a model including democratization and time - further suggesting that the evidenced effect of GDP was not only due to its raw increase with time. Finally, because GDP and the computed scores of trustworthiness displays were on very different scales, we decided to z-score them in the figure as this transformation does not change the shape of the data - however, we could also have plotted the results with two different scales on the graph, but we thought it would be less readable. , 
 The outliers (in fact three?) in the sixteenth century do look as if they might have driven some of that regression, but I would be surprised if the authors hadn t addressed this somehow. https: twitter.com skyglowberlin status 1309383493156302848 , 
 Ignoring the strong framing, there are a number of interesting methodological and interpretational points raised in this thread. https: twitter.com BlancheMinerva status 1309744656692006912 , 
 #3 Concerning the outliers, as explained in the paper, the statistical analyses were conducted taking each portrait as a data point (and not averaging them as it is done on the graph for illustration purposes). Therefore, the 21 portraits corresponding to the outliers (1 in the 1500 decade, 5 in the 1530 decade and 15 in the 1560 decade) did not had much power for biasing our analyses conducted on 1962 portraits. Nonetheless, regressions excluding all the paintings from the 16th century confirmed both the significant effects of time (b = 0.12 u00b1 0.04, z = 6.12, p .001) and of GDP (controlling for time: b = 0.02 u00b1 0.01, z = 3.45, p .001) on the evolution of trustworthiness displays in the National Portrait Gallery paintings. , 
 #4 The question of which cues individuals use to make inferences about others based on their appearance is a central question in the field of first impressions (which does not mean that these cues are accurate for making these inferences). While it has been shown that first impressions from faces have an idiosyncratic component (they are notably influenced by your previous social experiences), existing data provide evidence that they also have a u2018sharedu2019 component (see notably Engell et al., 2007). For instance, Oosterhof Todorov (2008) successfully created avatars that are recognized as varying in their displayed level of trustworthiness by many different Western participant samples. Birkas et al (2014) generalized this effect to East and South Asian participant samples. Going one step further, Walker et al. (2011) found that both Asian and Western faces generated to display varying levels of trustworthiness based on a model built from ratings of Western participants were recognized by participants with both Asian and Western background as varying in their degree of displayed trustworthiness. Going even further, Xu et al. (2012) showed that some facial cues used by Chinese and Canadian participants for evaluating a new face trustworthiness were highly similar. Similarly, Bente et al. (2014) found a strong agreement in the trustworthiness ratings of avatar faces made by participants from Germany and from the United Arab Emirates. Although none of these studies precisely tested whether these facial cues were also used by people from the past to assess trustworthiness from faces, as we say in the paper, they provide evidence that these features are consistently recognized as cues of trustworthiness across individuals and cultures. Cognitive science research on first impressions also revealed that individuals - at least Western participants - are biased to perceive younger faces as more trustworthy than older ones. This effect is not necessarily huge, but it exists. In the sample of images from the Chicago Face Database we analyzed, the correlation between trustworthiness ratings generated by our algorithm and participantsu2019 ratings of facesu2019 age is r = -.12 (t(518) = -2.69, p = .008). It is quite small, but actually, as we report in our paper, very close to the correlation between participantsu2019 ratings of age and of trustworthiness (r = -.12 (t(518) = - 2.75, p = .006). On the other hand, it is important to note that there are multiple candidates mechanisms through which affluence may be related to social trust. Although liberalism may be one of them for some authors, our evolutionary psychology perspective makes us prefer other candidates, notably changes in the perceived costs and benefits associated with the engagement in cooperative activities. This is why our research focused on a proxy of affluence (GDP) and not on measures of economic liberalism. Importantly, while GDP was actively measured in the past decades, it was not the case in more ancient periods. However, economists and historians have made extensive work reconstructing this measure for past periods, which allows for quantitatively testing hypotheses on the effect of resources in history. In our study, we tested our hypotheses twice, once on the National Portrait Gallery paintings, once on the Web Gallery of Art paintings (which allowed us to extend our results to paintings from other European countries). More precisely, instead of merging these two databases, we conducted each analysis independently on these two datasets, reporting the exact results for each analysis (which were significant in both samples) and, in an effort of transparency, providing the codes and data for each analysis and figure as well as a guide for running the scripts. , 
 #2 So the estimated GDP correlates with what your model s estimate of current humans guesses as to how trustworthy people are. Those current humans guesses are themselves a proxy for the attitudes of people who lived hundreds of years ago. That s a lot of caveats that your paper s narrative doesn t pay much attention to. Speaking of Bayes Factor, your paper says Crucially, GDP per capita accounted for the evolution of trustworthiness displays better than a monotonous effect of time (Bayes Factor: **3.38**), which suggests that the observed evolution of trustworthiness displays cannot be reduced to a simple cultural accumulation that would have led to the development of painting techniques making sitters look more trustworthy. We then sought to replicate this result in the Web Gallery of Art database and also found a signiufb01cant positive association between GDP per capita and trustworthiness displays (b = 0.09 u00b1 0.03, z = 3.16, p = 0.002; Table 1; Fig. 2d). This association was robust to adjusting for a monotonous increase of trust displays over time (b = 0.07 u00b1 0.04, z = 1.98, p = 0.048; Table 1). Again, the model including GDP per capita provided a better account of the variations of trust displays than time alone (Bayes Factor: **130.16**) Although a significant association was found between these two variables in the National Portraits Gallery (b=0.03 u00b1 0.01z=5.24,p 0.001), this effect was not robust to the inclusion of time as covariate (b=u22120.01 u00b10.01,z=u22120.50,p 0.250) and the evolution of trust worthiness displays was better explained by GDP per capita than by changes in the institutions (Bayes Factor: **2.75**). Moreover, the positive association between more democratic institutions and higher trust worthiness displays was not replicated in the Web Gallery of Art sample (b=u22120.01 u00b1 0.01z=u22121.96,p=0.051; with time as a covariate:b=u22120.01 u00b1 0.01z=u22120.96,p 0.250; Bayes Factor of the GDP per capita model compared to the democratic institutions model: **6.16**). Changes in affluence precede changes in trustworthiness displays in portraits. Demonstrating that the association between GDP and the rise of trustworthiness is causal would of course require additional No analysis of these numbers is provided, with the only other reference to Bayes Factors being the sentence  Bayes factor analyses were conducted using the BIC approximation, which approximates Bayes factors computed under the unit information prior. A Bayes Factor of 1-3 is generally taken to be  anecdotal evidence for the alternate hypothesis and 3-10  moderate evidence for the alternate hypothesis. 130 by contrast is extraordinarily strong evidence. What do you make of the fact that you found weak evidence of an effect in one dataset and exceptionally strong evidence on the other? Why was this discrepancy not important enough to discuss? Given that you found results of 2.75, 3.38, 6.16, 130.16 does it not seem like perhaps there is something of interest to investigate in the 130 case? And given that the other analyses give only anecdotal to moderate evidence of an effect why is this fact not shared with the readers? Wagenmakers, E., Lee, M. D. (2014). Bayesian Cognitive Modeling: A Practical Course. United Kingdom: Cambridge University Press. , 
 https: twitter.com Calthalas status 1309403512867106816 , 
  First off, they keep talking about u201ctrustworthinessu201d or u201ccomputed trustworthinessu201d when they really mean u201ca measure that is weakly correlated with respondentsu2019 judgments about trustworthiness.u201d   Itu2019s nice to see digital phrenology branching out from criminality and finding the gays, I guess.  https: statmodeling.stat.columbia.edu 2020 10 02 its-kinda-like-phrenology-but-worse-not-so-good-for-the-nature-brand-name-huh-measurement-baby-measurement ,