# A TUTORIAL ON CONDUCTING AND INTERPRETING A BAYESIAN ANOVA IN JASP

Don van den Bergh, Johnny van Doorn, Maarten Marsman, Tim Draws, Erik-Jan van Kesteren, Koen Derks, Fabian Dablander, Quentin F. Gronau, Šimon Kucharský, Akash R. Komarlu Narendra Gupta, Alexandra Sarafoglou, Jan G. Voelkel, Angelika Stefan, Alexander Ly, Max Hinne, Dora Matzke, Eric-Jan Wagenmakers

P.U.F. | « L'Année psychologique »

2020/1 Vol. 120 | pages 73 - 96

ISSN 0003-5033

ISBN 9782130822936

Available online at :

https://www.cairn-int.info/article-E_ANPSY1_201_0073--a-tutorial-on-conducting-and.htm

How to cite this article :

Don van den Bergh et al., « A tutorial on conducting and interpreting a Bayesian ANOVA in JASP », *L'Année psychologique* 2020/1 (Vol. 120), p. 73-96.
# A Tutorial on Conducting and Interpreting a Bayesian ANOVA in JASP

Don van den Bergh<fnref n="1" />,<fnref n="2" />, Johnny van Doorn<fnref n="2" />, Maarten Marsman<fnref n="2" />, Tim Draws<fnref n="2" />, Erik-Jan van Kesteren<fnref n="5" />, Koen Derks<fnref n="2" />,<fnref n="3" />, Fabian Dablander<fnref n="2" />, Quentin F. Gronau<fnref n="2" />, Šimon Kucharský<fnref n="2" />, Akash R. Komarlu Narendra Gupta<fnref n="2" />, Alexandra Sarafoglou<fnref n="2" />, Jan G. Voelkel<fnref n="6" />, Angelika Stefan<fnref n="2" />, Alexander Ly<fnref n="2" />,<fnref n="4" />, Max Hinne<fnref n="2" />, Dora Matzke<fnref n="2" />, and Eric-Jan Wagenmakers<fnref n="2" />

## ABSTRACT

Analysis of variance (ANOVA) is the standard procedure for statistical inference in factorial designs. Typically, ANOVAs are executed using frequentist statistics, where *p*-values determine statistical significance in an all-or-none fashion. In recent years, the Bayesian approach to statistics is increasingly viewed as a legitimate alternative to the *p*-value. However, the broad adoption of Bayesian statistics—and Bayesian ANOVA in particular—is frustrated by the fact that Bayesian concepts are rarely taught in applied statistics courses. Consequently, practitioners may be unsure how to conduct a Bayesian ANOVA and interpret the results. Here we provide a guide for executing and interpreting a Bayesian ANOVA with JASP, an open-source statistical software program with a graphical user interface. We explain the key concepts of the Bayesian ANOVA using two empirical examples.

**Keywords:** Bayes Factor, Analysis of Variance, JASP, Posterior distribution, Hypothesis Test, Tutorial

<footnote n="1">¹Corresponding author</footnote>

<footnote n="2">²University of Amsterdam</footnote>

<footnote n="3">³Nyenrode Business University</footnote>

<footnote n="4">⁴Centrum Wiskunde & Informatica</footnote>

<footnote n="5">⁵Utrecht University</footnote>

<footnote n="6">⁶Stanford University</footnote>

<footnote n="1">¹ Corresponding author: Don van den Bergh, University of Amsterdam, Department of Psychological Methods, Postbus 15906, 1001 NK Amsterdam, The Netherlands. E-Mail: donvdbergh@hotmail.com.</footnote>
# Tutoriel pour réaliser et interpréter une analyse de variance bayésienne dans JASP

## RESUME

L'analyse de variance (ANOVA) est la procédure standard utilisée pour l'inférence statistique dans les plans factoriels. En règle générale, les analyses de variance sont exécutées à l'aide de statistiques fréquentistes, où les valeurs *p* déterminent la significativité statistique en termes de « tout ou rien ». Ces dernières années, l'approche bayésienne de la statistique inférentielle est de plus en plus considérée comme une alternative légitime à la valeur *p*. Toutefois, l'adoption généralisée des statistiques bayésiennes, et en particulier de l'ANOVA bayésienne, est limitée par le fait que les concepts bayésiens sont rarement enseignés dans les cours de statistiques appliquées. Par conséquent, les utilisateurs peuvent ne pas savoir comment effectuer une ANOVA bayésienne et en interpréter les résultats. Nous fournissons ici un guide pour réaliser et interpréter une ANOVA bayésienne avec JASP, un logiciel statistique open-source ayant une interface utilisateur graphique. Nous expliquons les concepts clés de l'ANOVA bayésienne à l'aide de deux exemples empiriques.

**Mots-clés:** Facteur de Bayes, Analyse de la variance, JASP, Distribution postérieure, Test d'hypothèse, Tutoriel

## 1. INTRODUCTION

Ubiquitous across the empirical sciences, analysis of variance (ANOVA) allows researchers to assess the effects of categorical predictors on a continuous outcome variable. Consider for instance an experiment by Strack, Martin, & Stepper (1988) designed to test the *facial feedback hypothesis*, that is, the hypothesis that people's affective responses can be influenced by their own facial expression. Participants were randomly assigned to one of three conditions. In the *lips* condition, participants were instructed to hold a pen with their lips, inducing a pout. In the *teeth* condition, participants were instructed to hold a pen between their teeth, inducing a smile. In the control condition, participants were told to hold a pen in their nondominant hand. With the pen in the instructed position, each participant then rated four cartoons for funniness. The outcome variable was the average funniness rating across the four cartoons. The ANOVA procedure may be used to test the null hypothesis that the pen position does not result in different funniness ratings.

ANOVAs are typically conducted using frequentist statistics, where *p*-values decide statistical significance in an all-or-none manner: if *p* < .05, the result is deemed statistically significant and the null hypothesis is rejected; if *p* > .05, the result is deemed statistically nonsignificant, and
the null hypothesis is retained. Such binary thinking has been critiqued extensively (e.g., Amrhein, Greenland, & McShane, 2019; Cohen, 1994; Rouder, Engelhardt, McCabe, & Morey, 2016), and some perceive it as a cause of the reproducibility crisis in psychology (Cumming, 2014; but see Savalei & Dunn, 2015). In recent years, several alternatives to p-values have been suggested, for example reporting confidence intervals (Cumming, 2014; Gardner & Altman, 1986) or abandoning null hypothesis testing altogether (B. B. McShane, Gal, Gelman, Robert, & Tackett, 2019).

Here we focus on another alternative: Bayesian inference. In the Bayesian framework, knowledge about parameters and hypotheses is updated as a function of predictive success—hypotheses that predicted the observed data relatively well receive a boost in credibility, whereas hypotheses that predicted the data relatively poorly suffer a decline (Wagenmakers, Morey, & Lee, 2016). A series of recent articles show how the Bayesian framework can supplement or supplant the frequentist *p*-value (e.g., Burton, Gurrin, & Campbell, 1998; Dienes & McLatchie, 2018; Jarosz & Wiley, 2014; Masson, 2011; Nathoo & Masson, 2016; Rouder *et al.*, 2016).

The advantages of the Bayesian paradigm over the frequentist *p*-value are well documented (e.g., Wagenmakers *et al.*, 2018); for instance, with Bayesian inference researchers can incorporate prior knowledge and quantify support, both in favor and against the null-hypothesis; furthermore, this support may be monitored as the data accumulate (Stefan, Gronau, Schönbrodt, & Wagenmakers, 2019). Despite these and other advantages, Bayesian analyses are still used only sparingly in the social sciences (van der Schoot, Winter, Ryan, Zondervan-Zwijnenburg, & Depaoli, 2017). The broad adoption of Bayesian statistics—and Bayesian ANOVA in particular—is hindered by the fact that Bayesian concepts are rarely taught in applied statistics courses. Consequently, practitioners may be unsure of how to conduct a Bayesian ANOVA and interpret the results.

To help familiarize researchers with Bayesian inference for common experimental designs, this article provides a guide for conducting and interpreting a Bayesian ANOVA with JASP (JASP Team, 2019). JASP is a free, open-source statistical software program with a graphical user interface that offers both Bayesian and frequentist analyses. Below, we first provide a brief introduction to Bayesian statistics. Subsequently, we use two data examples to explain the key concepts of ANOVA.
## 2. BAYESIAN FOUNDATIONS

This section explains some of the fundamentals of Bayesian inference. We focus on interpretation rather than mathematical detail; see the special issue on Bayesian inference by Vandekerckhove, Rouder, & Kruschke (2018) for a set of comprehensive, low-level introductions to Bayesian inference. The central goal of Bayesian inference is learning, that is, using observations to update knowledge. In an ANOVA we want to learn about the candidate models *M* and their condition-effect parameters *β*. Returning to the example of the facial feedback experiment, we commonly specify two models. The null model describes the funniness ratings using a single grand average across all three conditions, effectively stating that there is no effect of pen position. The parameters of the null model are thus the average test score and the error variance. The alternative model describes the funniness ratings using an overall average and the effect of pen position; in other words, the means of the three condition are allowed to differ. Therefore, the alternative model has five parameters: the average funniness ratings across participants, the error variance, and for each of the three pen positions the magnitude of the effect.<fnref n="2" />

To start the learning process, we need to specify prior beliefs about the plausibility of each model, *p*(*M*), and about the plausible parameter values *β* within each model, *p*(*β*|*M*).<fnref n="3" /> These prior beliefs are represented by *prior distributions*. Observing data *D* drives an update of beliefs, transforming the prior distribution over models and parameters to a joint *posterior distribution*, denoted *p*(*β*, *M*|*D*).<fnref n="4" /> The updating factor—the change from prior to posterior beliefs—is determined by relative predictive performance for the observed data (Wagenmakers *et al.*, 2016). As shown in Figure 1, the knowledge updating process forms a learning cycle, such that the posterior distribution after the first batch of data becomes the prior distribution for the next batch.

<footnote n="2">² Note that one of the four parameters, average funniness rating and the three condition effects, is redundant. That is, we can make identical predictions even when we fix one of the four parameters to zero.</footnote>

<footnote n="3">³ The symbol “|” may be read as “given the” or “conditional on”.</footnote>

<footnote n="4">⁴ The joint posterior distribution describes the posterior probabilities for each parameter to take on values in a particular range. Simultaneously, it describes a discrete probability distribution over the models considered, where models with a higher plausibility after seeing the data have a higher probability. Furthermore, the joint posterior distribution also describes any interaction between parameters and models. For example, in the full model containing all covariates, the posterior distributions for the parameters estimates may be more concentrated around zero.</footnote>
<figure><img src="image_1.png" /><figcaption>**Figure 1.** Bayesian learning can be conceptualized as a cyclical process of up-dating knowledge in response to prediction errors. The prediction step is deductive, and the updating step is inductive. For a detailed account see Jevons (1874/1913, Chapters XI and XII). Figure available at BayesianSpectacles.org under a CC-BY license.</figcaption></figure>

**Figure 1.** L'apprentissage bayésien peut être conceptualisé comme un processus cyclique d'actualisation des connaissances en réponse à des erreurs de prédiction. L'étape de prédiction est déductive, et l'étape de mise à jour est inductive. Pour un compte rendu détaillé, voir Jevons (1874/1913, chapitres XI et XII). Figure disponible sur BayesianSpectacles.org sous une licence CC-BY.

Mathematically, the updating process is given by Bayes' rule:

$$ \widehat{p(\beta, M|D)} = \widehat{p(M)} \times \widehat{p(\beta|M)} \times \widehat{\frac{p(D|\beta, M)}{p(D)}} $$

This rule stipulates how knowledge about the relative plausibility of both models and parameters ought to be updated in light of the observed data. When the focus is on the comparison of two rival models, one generally considers only the model updating term. This term, commonly
known as the *Bayes factor*, quantifies the relative predictive performance of the rival models, that is, the change in relative model plausibility that is brought about by the data (Etz & Wagenmakers, 2017; Jeffreys, 1939; Kass & Raftery, 1995; Wrinch & Jeffreys, 1921):

$$ \underbrace{\frac{p(M_1|D)}{p(M_0|D)}}_{\textit{Posterior model odds}} = \underbrace{\frac{p(M_1)}{p(M_0)}}_{\textit{Prior model odds}} \times \underbrace{\frac{p(D|M_1)}{p(D|M_0)}}_{\textit{Bayesfactor}} \times \frac{p(D|M_0)}{p(D|M_1)}. $$

When the Bayes factor $BF_{10}$ equals 20, the observed data are twenty times more likely to occur under $M_1$ than under $M_0$ (i.e., support for $M_1$ versus $M_0$); when the Bayes factor $BF_{10}$ equals 1/20, the observed data are twenty times more likely to occur under $M_0$ than under $M_1$ (i.e., support for $M_0$ versus $M_1$); when the Bayes factor $BF_{10}$ equals 1, the observed data are equally likely to occur under both models (i.e., neither model is supported over the other). Note that the Bayes factor is a comparison of two models and hence it is always a relative measure of evidence, that is, it quantifies the performance of one model relative to another.<fnref n="5" />

Since the prior and posterior odds are both ratios of probabilities, the Bayes factor can be seen as an odds ratio that quantifies the change in belief from prior odds to posterior odds. The Bayes factor can be presented as $BF_{10}$, $p(D|M_1)$, divided by $p(D|M_0)$, or as its reciprocal $BF_{01}$, $p(D|M_0)$ over $p(D|M_1)$. Typically, $BF_{10} > 1$ is used to quantify evidence in favor of the alternative hypothesis, whereas $BF_{01} > 1$ is used to quantify evidence in favor of the null hypothesis. For instance, $BF_{10} > 1/3$ can be interpreted as “the data are 1/3 times more likely under $M_1$ than under $M_0$”, but for a Bayes factor lower than 1 it is more intuitive to switch numerator and denominator and instead report the results as $BF_{01} = 3$, that is, “the data are 3 times more likely under $M_0$ than under $M_1$”.

The Bayesian paradigm differs from the frequentist paradigm in at least four key aspects. First, evidence in favor of a particular model, quantified by a Bayes factor, is a *continuous* measure of support. Unlike the frequentist Neyman-Pearson decision rule (usually $p < 0.05$), there is no need to impose all-or-none Bayes factor cut-offs for accepting or rejecting a particular model. Moreover, the Bayes factor can discriminate between “absence of evidence” (i.e., nondiagnostic data that are predicted about equally well under both models, such that the Bayes factor is close to 1) and

<footnote n="5">⁵ For a cartoon that explains the strength of evidence provided by a Bayes factor, see https://www.bayianspectacles.org/lets-poke-a-pizza-a-new-cartoon-to-explain-the-strength-of-evidence-in-a-bayes-factor/</footnote>
"evidence of absence" (i.e., diagnostic data that support the null hypothesis over the alternative hypothesis).

A second difference is that, in the Bayesian paradigm, knowledge about models *M* and parameters *β* is updated simultaneously. Consequently, it is natural to account for model uncertainty by considering all models, but assigning more weight to those models that predicted the data relatively well. This procedure is known as Bayesian model averaging (BMA; Hinne, Gronau, van den Bergh, & Wagenmakers, 2019; Hoeting, Madigan, Raftery, & Volinsky, 1999; Jevons, 1874/1913; p. 296 Jeffreys, 1939; p. 365 Jeffreys, 1961). In contrast, many frequentist analyses first select a 'best' model and subsequently estimate its parameters, thereby neglecting model uncertainty and producing overconfident conclusions (Claeskens & Hjort, 2008, Ch 7.4).<fnref n="6" /> Another benefit of BMA is that point estimates and uncertainty intervals can be derived without conditioning on a specific model. This way, model uncertainty is accounted for in point estimates and uncertainty intervals.

A third difference is that the Bayesian posterior distributions allow for direct probabilistic statements about parameters. For example, based on the posterior distribution of *β* we can state that we are 95% confident that the parameter lies between *x* and *y*. This range of parameter values is commonly known as a *95% credible interval*.<fnref n="7" /> Similarly, we can consider any interval from *a* to *b* and quantify our confidence that the parameter falls in that specific range. A fourth difference is that Bayesian inference automatically penalizes for complexity and thus favors parsimony (e.g., Berger & Jefferys, 1992; Jeffreys, 1961; Myung & Pitt, 1997). For instance, a model with a redundant covariate will make poor predictions. Consequently, the Bayes factor, which compares the relative predictive performance of two models, will favor the model without the redundant predictor over the model with the redundant predictor. Key is that, as the words suggest, the *predictive* performance is assessed using parameter values that are drawn from the *prior* distributions.

<footnote n="6">⁶ Although uncommon, it is possible to average over the models in the frequentist framework. To do so, calculate for each model an information criterion such as AIC, and use a transformed version as model weights (Burnham & Anderson, 2002).</footnote>

<footnote n="7">⁷ Note the difference in interpretation compared to the frequentist 95% confidence interval: "if we repeat this experiment an infinite number of times and compute an infinite number of confidence intervals, then 95% of these intervals contain the true parameter value." See also Morey, Hoekstra, Rouder, Lee, & Wagenmakers (2016).</footnote>
## 3. ANOVA

Traditionally, analysis of variance involves—as the name suggests—a comparison of variances. In the frequentist framework, the variance between each level of the categorical predictor is compared to the variance within the levels of the categorical predictor. When the categorical predictor has no effect, the population variances between the levels equals the population variances within the levels, and the sample ratio of these variances is distributed according to a central F-distribution. Under the assumption that the null hypothesis is true, we may then calculate the probability of encountering a sample ratio of variances that is at least as large as the one observed—this then yields the much-maligned yet omnipresent *p*-value.

Instead, the Bayesian ANOVA contrasts the predictive performance of competing models (Rouder et al., 2016). In order to make predictions the model parameters need to be assigned prior distributions. These prior distributions could in principle be specified from subjective background knowledge, but here we follow Rouder, Morey, Speckman, & Province (2012) and use a default specification inspired by linear regression models, designed to meet general desiderata such as consistency and scale invariance (i.e., it does not matter whether the outcome variable is measured in seconds or milliseconds; see also Bayarri, Berger, Forte, & García-Donato, 2012; Liang, Paulo, Molina, Clyde, & Berger, 2008).

## 4. ASSUMPTIONS

Before interpreting the results from an ANOVA, it is prudent to assess whether its main assumption holds, namely that the residuals are normally distributed. A common tool to assess the normality of the residuals is a Q-Q plot, which visualizes the quantiles of the observed residuals against the quantiles expected from a standard normal distribution. If the residuals are normally distributed, then all the points in a Q-Q plot fall on the red line in Figure 2. In contrast to a frequentist ANOVA, where the residuals are point estimates, a Bayesian ANOVA provides a probability distribution for each residual. The uncertainty in the residuals can thus be summarized by 95% credible intervals. The left panel of Figure 2 shows an example where the larger quantiles lie away from the red line, displaying a substantial
deviation from normality. The right panel of Figure 2 shows residuals that are more consistent with what is expected under a normal distribution.

<figure><img src="image_2.png" /><figcaption>**Figure 2.** Q-Q plots of non-normally distributed residuals (left) and approximately normally distributed residuals (right). The vertical bars through each point represent the 95% central credible interval. If the data are perfectly normally distributed, all points fall on the red line. Note that the y-axis of the two panels has a different scale.</figcaption></figure>

**Figure 2.** Tracés Q-Q des résidus non distribués normalement (à gauche) et des résidus distribués à peu près normalement (à droite). Les barres verticales à travers chaque point représentent l'intervalle central de crédibilité à 95%. Si les données sont parfaitement distribuées normalement, tous les points tombent sur la ligne rouge. Notez que l'axe des y des deux tracés a une échelle différente.

Introductory texts discuss additional ANOVA assumptions, most of which follow directly from the normality of the residuals. For some of these assumptions, violations can be difficult to detect visually in a Q-Q plot. An example is sphericity, which is specific to repeated measures ANOVA. One definition of sphericity is that the variance of all pairwise difference scores is equal. In the frequentist paradigm, this assumption is usually assessed using Mauchly's test (but see Tijmstra, 2018). Another example is homogeneity of variances, which implies that the residual variance is equal across all levels of the predictors. Homogeneity of variances can be assessed using Levene's test (Levene, 1961).

The following sections illustrate how to conduct and interpret a Bayesian ANOVA with JASP. JASP can be freely downloaded from https://jasp-stats.org/download/. Annotated .jasp files of the discussed analyses, data sets, and a step-by-step guide on conducting a Bayesian ANOVA in JASP are available at https://osf.io/f8krs/. We should stress that the current
implementation of Bayesian ANOVA in JASP is based on the R package *BayesFactor* (Morey & Rouder, 2015) which is itself based on the statistical work by Rouder *et al.* (2012).

## 5. EXAMPLE I: A ROBOT'S SOCIAL SKILLS

Do people take longer to switch off a robot when it displays social skills? This question was studied by Horstmann *et al.* (2018, see their publication for online access to the complete data set) and we use their data

<figure><img src="image_3.png" /><figcaption>**Figure 3.** Observed log switch-off times for the data of Horstmann *et al.* (2018).</figcaption></figure>

<caption>**Figure 3.** Log des temps mis pour éteindre le robot, à partir des données de Horstmann *et al.* (2018).</caption>
to illustrate the key concepts of a Bayesian ANOVA. In the Horstmann *et al.* (2018) study, 85 participants interacted with a robot. Participants were told that the purpose of their interaction with the robot was to test a new algorithm. After two dummy tasks were completed, the instructor told the participants that they could switch off the robot if they wanted. The outcome variable was the time it took participants to switch off the robot. Here we analyze the log-transformed switch-off times since the Q-Q plot of the raw switch off times showed a violation of normality. Horstmann *et al.* (2018) manipulated two variables in a between-subjects design. First, they manipulated the robots' verbal responses to be either social (e.g., "Oh yes, pizza is great. One time I ate a pizza as big as me.") or functional (e.g., "You prefer pizza. This worked well. Let us continue."). Second, either the robot protested to being turned off (e.g., "No! Please do not switch me off! I am scared that it [sic] will not brighten up again!") or it did not. Therefore, the design of this study is a 2x2 between-subjects ANOVA. The data are shown in Figure 3.

## 5.1. Interpreting the Bayesian ANOVA

### *Model comparison*

The primary output from the JASP ANOVA is presented in Table 1, which shows the support that the data offer for each model under consideration. The left-most column lists all models at hand: four alternative models and one null model. The models are ordered by their predictive performance relative to the best model; this is indicated in the *BF*<sub>01</sub> column, which shows the Bayes factor relative to the best model which features only the objection factor. For example, the data are about 73 times more likely under the model with only the robot's objection as a predictor than under the null model. The prior model probability *P*(M) is 0.2 for all models and the resulting posterior model probabilities are given by *P*(M|D). The *BF*<sub>M</sub> column shows change from prior odds to posterior odds for each model. For example, for the best model with only the robot's objection as a predictor the change in odds is: 0.542/(1 - 0.542) × (1 - 0.2)/0.2 ≈ 4.734, which matches the output of Table 1. The right-most column provides an error percentage indicating the precision of the numerical approximations, which should not be too large.<fnref n="8" />

<footnote n="8">⁸ In many situations, error percentages below 20% are acceptable. If the error is 20%, then a Bayes factor of 10 can fluctuate between 8 and 12. Because Bayes factors between 8 and 12 lead to the same qualitative</footnote>
<caption>**Table 1.** Model comparison for all models under consideration for the data of Horstmann *et al.* (2018).</caption>

<table><caption>**Tableau 1.** Comparaison de modèles pour tous les modèles considérés à partir des données de Horstmann *et al.* (2018).</caption><thead><tr><th>Model</th><th>P(M)</th><th>P(M|D)</th><th>BF<sub>M</sub></th><th>BF<sub>01</sub></th><th>error %</th></tr></thead><tbody><tr><td>O</td><td>0.2</td><td>0.542</td><td>4.735</td><td>1.000</td><td></td></tr><tr><td>O + S + O * S</td><td>0.2</td><td>0.303</td><td>1.736</td><td>1.791</td><td>2.770</td></tr><tr><td>O + S</td><td>0.2</td><td>0.146</td><td>0.682</td><td>3.719</td><td>1.323</td></tr><tr><td>Null model</td><td>0.2</td><td>0.007</td><td>0.030</td><td>73.373</td><td>0.000</td></tr><tr><td>S</td><td>0.2</td><td>0.002</td><td>0.009</td><td>252.495</td><td>0.005</td></tr></tbody></table>

Note: The abbreviations ‘O’ and ‘S’ stand for the robot’s objection and social interaction type, respectively. The term ‘O * S’ stands for the interaction between the two factors. The ‘Model’ column shows the predictors included in each model, the column the prior model probability, the column the posterior model probability, the column the posterior model odds, and the column the Bayes factors of all models compared to the best model. The final column, ‘error’ is an estimate of the numerical error in the computation of the Bayes factor. All models are compared to the best model and are sorted from lowest Bayes factor to highest.

Bayes factors are *transitive*, which means that if the model with only the robot’s objection outpredicts the null model by a factor of *a*, and the null model outpredicts the model with only social interaction type by a factor of *b*, then the model with only the robot’s objection will outpredict the model with only social interaction type by a factor of *a* × *b*. Transitivity can be used to compute Bayes factors that may be of interest but are missing from the table. For example, the Bayes factor for the null model versus the model with only social interaction type can be obtained by dividing their Bayes factors against the best model: 252.495/73.373 ≈ 3.441 in favor of the null model. Note that the Bayes factor is represented as $BF_{01}$ in Table 1; predictive performance of the best model divided by the predictive performance for a particular model. Had we shown $BF_{10}$, we would have needed to take the reciprocal of the previous calculation to obtain the same result.

## Analysis of Effects

The previous section compared all available models. However, as the number of predictors increases, the number of models quickly grows too

<footnote>conclusion, this amount of numerical error is not problematic (see also Appendix B Jeffreys, 1961). When the error percentage is deemed too high, the number of samples can be increased to reduce the error percentage at the cost of longer computation time. For more information, see van Doorn *et al.* (2019).</footnote>
large to consider each model individually.<fnref n="9" /> Rather than studying the results for each model individually, it is possible to average the results from Table 1 over all models, that is, compute the model-averaged results. This produces Table 2, which shows for each predictor the prior and posterior inclusion probabilities, and the inclusion Bayes factor. A prior inclusion probability is the probability that a predictor is included in the model before seeing the data and is computed by summing up the prior model probabilities of all models which contain that predictor. A posterior inclusion probability is the probability that a predictor is included in the model after seeing the data and is computed by summing up the posterior model probabilities of all models which contain that predictor. The inclusion Bayes factor quantifies the change from prior inclusion odds to posterior inclusion odds and can be interpreted as the evidence in the data for including a predictor. For example, Table 2 shows that the data are about 68.6 times more likely under the models that include the robot's objection than under the models without this predictor.

<caption>**Table 2.** Results from averaging over the models in Table 1.</caption>

<table><caption>**Tableau 2.** Résultats obtenus à partir des modèles moyennés du Tableau 1.</caption><thead><tr><th>Effects</th><th>P(incl)</th><th>P(incl|D)</th><th>BF<sub>Inclusion</sub></th></tr></thead><tbody><tr><td>O</td><td>0.6</td><td>0.990</td><td>68.558</td></tr><tr><td>S</td><td>0.6</td><td>0.445</td><td>0.535</td></tr><tr><td>O * S</td><td>0.2</td><td>0.293</td><td>1.659</td></tr></tbody></table>

Note: The abbreviations 'O' and 'S' stand for the robot's objection and social interaction type respectively. The first column denotes each predictor of interest, the column shows the prior inclusion probability, shows the posterior inclusion probability, and shows the inclusion Bayes factor.

Although model-averaged results are straightforward to obtain, their interpretation requires special attention when interaction effects are concerned. In JASP, models are excluded from consideration when they violate the *principle of marginality*, that is, they feature an interaction effect but lack the constituent main effects (for details see Nelder, 1977). This model exclusion rule means that the active model set is not balanced. For example, in Table 2 the inclusion odds for the interaction 'O * S' is obtained by comparing four models without the interaction effect against the one model with the interaction effect. As an alternative, Sebastiaan Mathôd

<footnote n="9">⁹ In general, given *p* predictors there are 2^p* models to consider. If interaction effects are considered, the model space grows even faster.</footnote>
has suggested to compute inclusion probabilities for “matched” models only.<fnref n="10" /> What this means is that all models with the interaction effect are compared to models with the same predictors except for the interaction effect. For example, the model with an interaction effect between ‘O * S’ in Table 2 is compared against the model with the main effects of ‘O’ and ‘S’, but not against any other models. To compute inclusion probabilities for main effects, models that feature interaction effects composed of these main effects are not considered. These models are excluded because they cannot be matched with models that include the interaction effect but not the main effect, since those violate the principle of marginality. Note that without interaction effects, the matched and not matched inclusion probabilities are the same.

Table 3 shows the inclusion probabilities and inclusion Bayes factor obtained by only considering matched models. Comparing Table 3 to Table 2, the prior inclusion probability of the main effects decreased because these are based on one model fewer. The posterior inclusion probabilities of the main effects decreased but that of the interaction effect increased. The inclusion Bayes factor, the evidence in the data for including a predictor, provides slightly more evidence for including the main effect of the robot’s objection and the interaction effect, and somewhat more evidence for excluding the main effect of the social interaction type.

<caption>**Table 3.** Results from averaging over the models in Table 1 but only considering “matched” models (see text for details).</caption>

<table><caption>**Tableau 3.** Résultats à partir des modèles moyennés du tableau 1 mais en considérant uniquement les modèles « appariés » (voir le texte pour plus de détails).</caption><thead><tr><th>Effects</th><th>P(incl)</th><th>P(incl|D)</th><th>BF<sub>Inclusion</sub></th></tr></thead><tbody><tr><td>O</td><td>0.4</td><td>0.6872</td><td>72.76</td></tr><tr><td>S</td><td>0.4</td><td>0.1524</td><td>0.28</td></tr><tr><td>O * S</td><td>0.2</td><td>0.3033</td><td>2.018</td></tr></tbody></table>

Note: The abbreviations ‘O’ and ‘S’ stand for the robot’s objection and social interaction type respectively. The first column denotes each predictor of interest, the column shows the prior inclusion probability, shows the posterior inclusion probability, and shows the inclusion Bayes factor.

<footnote n="10">¹⁰ See also https://www.cogsci.nl/blog/interpreting-bayesian-repeated-measures-in-jasp.</footnote>
## Parameter Estimates

After establishing which predictors are relevant, we can investigate the magnitude of the relations by examining the posterior distributions. Table 4 summarizes the model-averaged posterior distributions of each level ($\beta_j$), using four statistics: the posterior mean, the posterior standard deviation, and the lower and upper bound of the 95% central credible interval. The symmetry in the estimates is a consequence of the sum-to-zero constraint, that is, the posterior mean of O-Yes = -1 × the posterior mean of O-No = 0.265. Table 4 shows that the effect of objection is about 0.265 (95% CI [0.111, 0.418]). A posterior estimate for the observed log response time of a particular group, say the condition where the robot did not object, can be obtained by adding the posterior mean of the intercept (i.e., the grand mean), 1.724, to the posterior mean of the no-objection condition, -0.265, which yields 1.459.<fnref n="11" />

<table><caption>**Table 4.** Summary of the marginal model averaged posterior distributions. **Tableau 4.** Résumé des distributions postérieures moyennées du modèle marginal.</caption><thead><tr><th rowspan="2">Predictor</th><th rowspan="2">Level</th><th colspan="4">95% CI</th></tr><tr><th>Mean</th><th>SD</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td>Intercept</td><td></td><td>1.724</td><td>0.077</td><td>1.569</td><td>1.877</td></tr><tr><td rowspan="2">O</td><td>Yes</td><td>0.265</td><td>0.077</td><td>0.111</td><td>0.418</td></tr><tr><td>No</td><td>-0.265</td><td>0.077</td><td>-0.420</td><td>-0.113</td></tr><tr><td rowspan="2">S</td><td>Functional</td><td>-0.044</td><td>0.071</td><td>-0.186</td><td>0.097</td></tr><tr><td>Social</td><td>0.044</td><td>0.071</td><td>-0.098</td><td>0.185</td></tr><tr><td rowspan="3">O * S</td><td>Yes &amp; Social</td><td>-0.132</td><td>0.072</td><td>-0.278</td><td>0.008</td></tr><tr><td>Yes &amp; Functional</td><td>0.132</td><td>0.072</td><td>-0.009</td><td>0.276</td></tr><tr><td>No &amp; Social</td><td>0.132</td><td>0.072</td><td>-0.009</td><td>0.276</td></tr><tr><td></td><td>No &amp; Functional</td><td>-0.132</td><td>0.072</td><td>-0.278</td><td>0.008</td></tr></tbody></table>

Note: Posteriors are summarized using mean, standard deviation, and 95% central credible intervals (CI).

<footnote n="11">¹¹ This calculation is valid only for the posterior means, not for the other posterior summaries.</footnote>
To summarize, the Bayesian ANOVA revealed that the robot's objec-
tion almost certainly had an effect on switch-off time (*BF*<sub>Inclusion</sub> = 68.558).
We also learned that the data are not sufficiently informative to allow a
strong conclusion about the effect of the robot's social interaction type
(*BF*<sub>Inclusion</sub> = 0.535) or about an interaction effect between objection and
social interaction type (*BF*<sub>Inclusion</sub> = 1.659).

6. EXAMPLE II: POST HOC TESTS ON THE HOUSES
OF HOGWARTS

After executing an ANOVA and finding strong evidence that a
particular predictor relates to the outcome variable, a common question
arises: "Which levels of the predictor deviate from one another?". As
an illustration, consider the data from Jakob, Garcia-Garzon, Jarke,
& Dablander (2019) where 847 participants filled out a 'sorting hat'
questionnaire that determined their assignment to one of the four
Houses of Hogwarts from the Harry Potter books: Gryffindor, Hufflepuff,
Ravenclaw, or Slytherin.<fnref n="12" /> Subsequently, participants filled out the dark
triad questionnaire (Jones & Paulhus, 2014) that was used to derive the
outcome variable: Machiavellism.

In this example, there is only one categorical predictor: The House of
Hogwarts a participant was assigned to. If we compare the model with
this predictor to the null model, we find overwhelming evidence for the
alternative (BF<sub>10</sub> = 6.632 × 10<sup>18</sup>). This is a clear indication that Machiavellism
differs between the members of the four houses. However, this result does
not indicate the houses responsible for the difference. To address that
question, we need a post hoc test.

<footnote n="12">¹² The raw data and original analyses can be found at https://osf.io/rtf74/</footnote>
<figure><img src="image_4.png" /><figcaption>**Figure 4.** Observed Machiavellism scores for each of the four Houses of Hogwarts.</figcaption></figure>

<caption>**Figure 4.** Scores observés de Machiavélisme pour chacune des quatre maisons de Poudlard.</caption>

For ANOVA models, the main component of a post hoc test is a t-test on all pairwise combinations of a predictor's levels. For a Bayesian ANOVA, the main component is the Bayesian t-test. Table 5 shows the Bayesian post hoc tests for the sorting hat data. As with frequentist inference, Bayesian post hoc tests are subject to a multiple comparison problem. To control for multiplicity, we follow the approach discussed in Westfall (1997) which is an extension of the approach of Jeffreys (1938); for an overview of Bayesian methods correcting for multiplicity see for instance de Jong (2019). Westfall's approach relates the overall null hypothesis $p(H_0)$ that all condition means are equal to each comparison between two condition means. That way, the prior probability of the overall null hypothesis can be adjusted to correct for multiplicity, and this influences each individual comparison. The procedure to relate the overall null hypothesis to each comparison is described below.

A condition mean $\mu_i$ is either equal to the grand mean $\mu$ with probability $\tau$, or $\mu_i$ is drawn from a continuous distribution with probability $1 - \tau$. It is key that this distribution is continuous because two values drawn from a continuous distribution are never exactly equal. Thus, the probability
that two condition means $\mu_i$ and $\mu_j$ are equal is $p(\mu_i = \mu_j) = p(\mu_i = \mu) \times p(\mu_j = \mu) = \tau^2$. From this, the probability of the null hypothesis that all J condition means are equal follows: $p(H_0) = p(\mu_1 = \mu_2 = \dots = \mu_j) = p(\mu_1 = \mu) \times p(\mu_2 = \mu) \times \dots \times p(\mu_j = \mu) = \tau^J$. Solving for $\tau$, we obtain $\tau = p$. Thus, the prior probability that two specific magnitudes are equal can be expressed in terms of the prior probability that all magnitudes are equal, that is $p(\mu_i = \mu_j) = \tau^2 = p$. For example, imagine there are four conditions (J = 4) and the prior probability that all condition means are equal is 0.5. Then, the prior probability that two conditions means are equal is: $p(\mu_1 = \mu_2) = \sqrt{0.5/\sqrt{0.5}} \approx 0.414$. In sum, the Westfall approach involves, as a first step, Bayesian t-tests for all pairwise comparisons, which provides the unadjusted Bayes factors. In the next step, the prior model odds are adjusted by fixing the overall probability of no effect to 0.5. The adjusted prior odds and the Bayes factor are then used to calculate the adjusted posterior odds.

Table 5 shows the results for the post hoc tests of the sorting hat example. The adjusted posterior odds show (1) evidence (i.e., odds of about 16) that Machiavellism differs between Hufflepuff and Ravenclaw; (2) evidence (i.e., odds of about 27) that Machiavellism differs between Gryffindor and Hufflepuff; (3) overwhelming evidence (i.e., odds of about $1.04 \times 10^9$, $5.43 \times 10^{16}$, and $5.30 \times 10^9$) that Machiavellism differs between Gryffindor and Slytherin, between Hufflepuff and Slytherin, and between Ravenclaw and Slytherin, respectively; (4) evidence (i.e, odds of $1/0.0432 \approx 23$) that Machiavellism of Gryffindor and Ravenclaw is the same.

<caption>Table 5. Post hoc test for the Sorting House data.</caption>

<table><caption>Tableau 5. Test Post hoc selon l'affectation aux Maisons de Poudlard.</caption><thead><tr><th>Level 1</th><th>Level 2</th><th>Prior Odds</th><th>Posterior Odds</th><th>BF<sub>10,U</sub></th><th>error %</th></tr></thead><tbody><tr><td>Gryffindor</td><td>Hufflepuff</td><td>0.414</td><td>27.2</td><td>65.6</td><td>5.73 x 10<sup>-5</sup></td></tr><tr><td>Gryffindor</td><td>Ravenclaw</td><td>0.414</td><td>0.0432</td><td>0.104</td><td>9.56 x 10<sup>-5</sup></td></tr><tr><td>Gryffindor</td><td>Slytherin</td><td>0.414</td><td>1.04 x 10<sup>9</sup></td><td>2.50 x 10<sup>9</sup></td><td>3.94 x 10<sup>-16</sup></td></tr><tr><td>Hufflepuff</td><td>Ravenclaw</td><td>0.414</td><td>15.5</td><td>37.3</td><td>7.57 x 10<sup>-8</sup></td></tr><tr><td>Hufflepuff</td><td>Slytherin</td><td>0.414</td><td>5.43 x 10<sup>16</sup></td><td>1.31 x 10<sup>17</sup></td><td>3.35 x 10<sup>-23</sup></td></tr><tr><td>Ravenclaw</td><td>Slytherin</td><td>0.414</td><td>5.30 x 10<sup>9</sup></td><td>1.28 x 10<sup>10</sup></td><td>6.36 x 10<sup>-16</sup></td></tr></tbody></table>

Note: The first two columns indicate the houses being compared, the third and fourth column indicate the adjusted prior model odds and posterior model odds respectively, and the fifth column indicates the uncorrected Bayes factor in favor of the alternative hypothesis that the magnitudes differ. The final column shows the numerical error of the Bayes factor computation.
Now that we know which Houses differ, the next step is to assess the magnitude of each House of Hogwarts on Machiavellism score. Rather than examining a table that summarizes the marginal posteriors, we plot the model averaged posteriors for each house in Figure 5. Clearly, Slytherin scores higher on Machiavellism than the other Houses whereas Hufflepuff scores lower on Machiavellism than the other Houses. Table 6 in the appendix shows the parameters estimates of the marginal posterior effects for each house.

<figure><img src="image_5.png" /><figcaption>**Figure 5.** Posterior distributions of the effect of each House of Hogwarts on Machiavellism. Slytherin scores higher on Machiavellism than the other Houses whereas Hufflepuff scores lower on Machiavellism than the other Houses. The horizontal error bars above each density represent 95% credible intervals.</figcaption></figure>

**Figure 5.** Distributions postérieures de l'effet de chaque maison de Poudlard sur le Machiavélisme. Serpentard obtient le plus haut score de Machiavélisme par rapport aux autres maisons, alors que Poufsouffle obtient le score de Machiavélisme le plus bas. Les barres d'erreur horizontales au-dessus de chaque densité représentent des intervalles de crédibilité à 95%.
# 7. CONCLUDING COMMENTS

The goal of this paper was to provide guidance for practitioners on to conduct a Bayesian ANOVA in JASP and interpret the results. Although the focus was on ANOVAs with categorical predictors, JASP can also handle ANOVAs with additional continuous predictors. The appropriate analysis then becomes an analysis of covariance (ANCOVA) and all concepts explained here still apply. For a general guide on reporting Bayesian analyses see van Doorn et al. (2019).

As with all statistical methods, the Bayesian ANOVA comes with limitations and caveats. For instance, when the model is severely misspecified and the residuals are non-normally distributed, the results from a standard ANOVA—whether Bayesian or frequentist—are potentially misleading and should be interpreted with care. In such cases, at least two alternatives may be considered. The first alternative is to consider a rank-based ANOVA such as the Kruskal–Wallis test (Kruskal & Wallis, 1952). This test depends only on the ordinal information in the data and hence does not make strong assumptions on how the data ought to be distributed. The second alternative is to specify a different distribution for the residuals. Using software for general Bayesian inference such as Stan (Carpenter et al., 2017) or JAGS (Plummer, 2003), it is relatively straightforward to specify any distribution for the residuals. However, this approach requires knowledge about programming and statistical modeling and is likely to be computationally intensive. Another limitation of the Bayesian ANOVA is that, especially in more complicated designs, it is not straightforward to intuit what knowledge the prior distributions represent.

Some limitations are specific to JASP. Currently, it is not possible to use post hoc tests to examine whether the contribution of a level differs from zero, that is, to test whether a specific level deviates from the grand mean. It is also not possible to handle missing values in any other way than list-wise deletion. Another limitation relates to sample size planning. The typical planning process involves a frequentist power analysis which provides the sample size needed to achieve a certain rate of correctly detecting a true effect of a prespecified magnitude. A Bayesian equivalent of power analysis is Bayes factor design analysis (BFDA; e.g., Schönbrodt & Wagenmakers, 2018). In a sequential design, BFDA produces the expected sample sizes required to reach a target level of evidence (i.e., a target Bayes factor). In a fixed-n design, BFDA produces the expected levels of evidence, given a specification of the magnitude of the effect. At the moment of writing
BFDA has not been implemented in JASP; an accessible tutorial and a
Shiny app are provided by Stefan et al. (2019).

We believe that the Bayesian ANOVA provides a perspective on the
analysis of factorial designs that can fruitfully supplement or even supplant
the currently dominant frequentist ANOVA. The epistemic advantages of
the Bayesian paradigm are well known (e.g., Jeffreys, 1961; Wagenmakers
et al., 2018) but in order to be adopted in research practice it is essential for
the methodology to be implemented in an easy-to-use software package
such as JASP. In addition to the software, however, practitioners also require
guidance on how to interpret the results, which was the main purpose
of this paper. In general, we hope that the increased use of the Bayesian
ANOVA will stimulate the methodological diversity in the field, and that
it will become more standard to examine the robustness of frequentist
conclusions by comparing them to the Bayesian alternative.

REFERENCES

Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise up against statistical significance. *Nature*, *567*, 305–307.

Bayarri, M. J., Berger, J. O., Forte, A., & García-Donato, G. (2012). Criteria for Bayesian model choice with application to variable selection. *The Annals of Statistics*, *40*, 1550–1577.

Berger, J. O., & Jefferys, W. H. (1992). The application of robust Bayesian analysis to hypothesis testing and Occam's razor. *Journal of the Italian Statistical Society*, *1*, 17–32.

Burnham, K. P., & Anderson, D. R. (2002). *Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.)*. New York: Springer Verlag.

Burton, P. R., Gurrin, L. C., & Campbell, M. J. (1998). Clinical significance not statistical significance: A simple Bayesian alternative to *p*-values. *Journal of Epidemiology & Community Health*, *52*(5), 318–323.

Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M.,

... Riddell, A. (2017). Stan: A probabilistic programming language. *Journal of Statistical Software*, *76*.

Claeskens, G., & Hjort, N. L. (2008). *Model selection and model averaging*. Cambridge: Cambridge University Press.

Cohen, J. (1994). The earth is round ($p < .05$). *American Psychologist*, *49*, 997–1003.

Cumming, G. (2014). The new statistics: Why and how. *Psychological Science*, *25*, 7–29.

de Jong. (2019). A Bayesian approach to the correction for multiplicity. *Unpublished Master Thesis*. Retrieved from https://doi.org/10.31234/osf.io/s56mk

Dienes, Z., & McLatchie, N. (2018). Four reasons to prefer Bayesian analyses over significance testing. *Psychonomic Bulletin & Review*, *25*(1), 207–218.

van Doorn, J., van den Bergh, D., Böhm, U., Dablander, F., Derks, K., Draws, T., ... Wagenmakers, E.-J. (2019). The JASP guidelines for conducting and reporting a Bayesian analysis. *Manuscript Submitted*
for Publication. Retrieved from psyarxiv.
com/yqxfr

Etz, A., & Wagenmakers, E.-J. (2017). J. B. S. Haldane's contribution to the Bayes factor hypothesis test. *Statistical Science*, 32, 313-329.

Gardner, M. J., & Altman, D. G. (1986). Confidence intervals rather than p values: Estimation rather than hypothesis testing. *Br Med J (Clin Res Ed)*, 292(6522), 746-750.

Hinne, M., Gronau, Q. F., Bergh, D. van den, & Wagenmakers, E.-J. (2019). A conceptual introduction to Bayesian model averaging. *Manuscript Submitted for Publication*.

Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999). Bayesian model averaging: A tutorial. *Statistical Science*, 14, 382-417.

Horstmann, A. C., Bock, N., Linhuber, E., Szczuka, J. M., Straßmann, C., & Krämer, N. C. (2018). Do a robot's social skills and its objection discourage interactants from switching the robot off? *PLoS ONE*, 13(7), e0201581.

Jakob, L., Garcia-Garzon, E., Jarke, H., & Dablander, F. (2019). The Science Behind the Magic? The Relation of the Harry Potter "Sorting Hat Quiz" to Personality and Human Values. *Collabra: Psychology*, 5(1).

Jarosz, A. F., & Wiley, J. (2014). What are the odds? A practical guide to computing and reporting Bayes factors. *Journal of Problem Solving*, 7, 2-9.

JASP Team. (2019). JASP (Version 0.9.2) [Computer software]. Retrieved from https://jasp-stats.org/

Jeffreys, H. (1938). Significance tests when several degrees of freedom arise simultaneously. *Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences*, 165, 161-198.

Jeffreys, H. (1939). *Theory of probability* (1st ed.). Oxford, UK: OUP.

Jeffreys, H. (1961). *Theory of probability* (3rd ed.). Oxford, UK: OUP.

Jevons, W. S. (1874/1913). *The principles of science: A treatise on logic and scientific method*. London: MacMillan.

Jones, D. N., & Paulhus, D. L. (2014). Introducing the short dark triad (sd3) a brief measure of dark personality traits. *Assessment*, 21(1), 28-41.

Kass, R. E., & Raftery, A. E. (1995). Bayes factors. *Journal of the American Statistical Association*, 90, 773-795.

Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. *Journal of the American Statistical Association*, 47(260), 583-621.

Levene, H. (1961). Robust tests for equality of variances. In I. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow, & H. B. Mann (Eds.), *Contributions to Probability and Statistics. Essays in Honor of Harold Hotelling* (pp. 279-292). Stanford, California: Stanford University Press.

Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008). Mixtures of priors for Bayesian variable selection. *Journal of the American Statistical Association*, 103, 410-423.

Masson, M. E. J. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing. *Behavior Research Methods*, 43, 679-690.

McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019). Abandon statistical significance. *The American Statistician*, 73, 235-245.

Morey, R. D., & Rouder, J. N. (2015). BayesFactor 0.9.12-4.2. Comprehensive R Archive Network. Retrieved from https://CRAN.R-project.org/package=BayesFactor

Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016). The fallacy of placing confidence in confidence intervals. *Psychonomic Bulletin & Review*, 23, 103-123.

Myung, I. J., & Pitt, M. A. (1997). Applying Occam's razor in modeling cognition: A Bayesian approach. *Psychonomic Bulletin & Review*, 4, 79-95.
Nathoo, F. S., & Masson, M. E. J. (2016). Bayesian alternatives to null-hypothesis significance testing for repeated-measures designs. *Journal of Mathematical Psychology, 72*, 144-157.

Nelder, J. (1977). A reformulation of linear models. *Journal of the Royal Statistical Society: Series A (General)*, 140(1), 48-63.

Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In K. Hornik, F. Leisch, & A. Zeileis (Eds.), *Proceedings of the 3rd international workshop on distributed statistical computing*. Vienna, Austria.

Rouder, J. N., Engelhardt, C. R., McCabe, S., & Morey, R. D. (2016). Model comparison in ANOVA. *Psychonomic Bulletin & Review*, 23, 1779-1786.

Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default Bayes factors for ANOVA designs. *Journal of Mathematical Psychology*, 56, 356-374.

Savalei, V., & Dunn, E. (2015). Is the call to abandon *p*-values the red herring of the replicability crisis? *Frontiers in Psychology*, 6, 245.

Schönbrodt, F. D., & Wagenmakers, E.-J. (2018). Bayes factor design analysis: Planning for compelling evidence. *Psychonomic Bulletin & Review*, 25, 128-142.

Stefan, A. M., Gronau, Q. F., Schönbrodt, F. D., & Wagenmakers, E.-J. (2019). A tutorial on Bayes Factor Design Analysis using an informed prior. *Behavior Research Methods*, 51(3), 1042-1058. doi:10.3758/s13428-018-01189-8

Strack, F., Martin, L. L., & Stepper, S. (1988). Inhibiting and facilitating conditions of

the human smile: A nonobtrusive test of the facial feedback hypothesis. *Journal of Personality and Social Psychology*, 54, 768-777.

Tijmstra, J. (2018). Why checking model assumptions using null hypothesis significance tests does not suffice: A plea for plausibility. *Psychonomic Bulletin & Review*, 25, 548-559.

van der Schoot, R., Winter, S., Ryan, O., Zondervan-Zwijnenburg, M., & Depaoli, S. (2017). A systematic review of Bayesian papers in psychology: The last 25 years. *Psychological Methods*, 22, 217-239.

Vandekerckhove, J., Rouder, J. N., & Kruschke, J. K. (Eds.). (2018). Editorial: Bayesian methods for advancing psychological science. *Psychonomic Bulletin & Review*, 25, 1-4.

Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, A. J., Love, J., ... Morey, R. D. (2018). Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. *Psychonomic Bulletin & Review*, 25, 35-57.

Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016). Bayesian benefits for the pragmatic researcher. *Current Directions in Psychological Science*, 25, 169-176.

Westfall, P. H. (1997). Multiple testing of general contrasts using logical constraints and correlations. *Journal of the American Statistical Association*, 92(437), 299-306.

Wrinch, D., & Jeffreys, H. (1921). On certain fundamental principles of scientific inquiry. *Philosophical Magazine*, 42, 369-390.
# APPENDIX: PARAMETER ESTIMATES FOR THE SORTING HAT DATA

**Table 6.** Parameter estimates for each of the houses in the data of Jakob et al. (2019).

<table><caption>**Tableau 6.** Estimations des paramètres pour chacune des maisons à partir des données de Jakob et al. (2019).</caption><thead><tr><th rowspan="2">Predictor</th><th rowspan="2">Level</th><th colspan="4">95% CI</th></tr><tr><th>Mean</th><th>SD</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td>Intercept</td><td></td><td>26.923</td><td>0.215</td><td>26.46</td><td>27.337</td></tr><tr><td rowspan="4">Sorting house</td><td>Gryffindor</td><td>-0.568</td><td>0.357</td><td>-1.28</td><td>0.140</td></tr><tr><td>Hufflepuff</td><td>-2.610</td><td>0.360</td><td>-3.34</td><td>-1.898</td></tr><tr><td>Ravenclaw</td><td>-0.696</td><td>0.330</td><td>-1.36</td><td>-0.037</td></tr><tr><td>Slytherin</td><td>3.874</td><td>0.418</td><td>3.04</td><td>4.719</td></tr></tbody></table>

Note: The interpretation of each column is identical to that of Table 4.

Cet ouvrage a été mis en pages par JOUVE
1, rue du Docteur-Sauvé, 53101 Mayenne
Nº 2697793T – Dépôt légal : mars 2020