# Guidelines for Experimental Algorithmics:
## 7 Evaluating Results with Statistical Analysis
### Recommendations  
Different statistical methods fulfill different needs. For almost all objectives, both Bayesian and frequentist methods exist, see Table 5. In experimental algorithmics, most hypotheses can be expressed as statements constraining parameters in a statistical model, i.e., \"the average speedup of A over B is at least 20%\". Thus, in contrast to earlier statistical practice, we recommend to approach evaluation of hypotheses by parameter estimation and to only use the classical hypothesis tests when parameter estimation is not possible. The additional information gained by parameter estimates has a couple of advantages. For example, when using only hypothesis tests, small differences in large data sets can yield impressively small \(p\)-values and thus claims of _statistical significance_ even when the difference is irrelevant in practice and the significance is statistical only [15]. Using confidence intervals (Section 7.3.2) or the posterior distribution in addition with a region of practical equivalence avoids this problem [88].  
Below is a rough guideline (also shown in Figure 10) outlining our method selection process. It favors Bayesian methods, since the python library PyMC3 ([PERSON] et al., 2016) offering them fits well into our workflow.  
1. Define a model that captures the parts of the measured results that interest you, see Section 7.1.
2. Using _confidence intervals_ (Section 7.3.2) or _credible intervals_ (Section 7.4), estimate plausible values for the model parameters, including their uncertainty. If this proves intractable and you are only interested in whether a measured difference is due to chance, use a significance test instead (Section 7.3.1).
3. If you want to show that two distributions (of outcomes of algorithms) are _similar_, use an equivalence test, in which you define a region of practical equivalence.
4. If you want to show that two distributions (of outcomes of algorithms) are _different_, you may also use an equivalence test or alternatively, a significance test.
5. If you want to compare how well different hypotheses explain the data, for example compare whether the diameter has an influence on relative scaling, compare the _relative fit_ using a Bayes factor (Section 7.4.2). Bayes factors are also useful when investigating equivalence or difference of more complex models.  
Needless to say, these are only recommendations.  
**Guideline 8**. Statistical methods help a rigorous interpretation of experimental results. The classical approach of using \(p\)-values, while common, is often not the most appropriate method.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{2}{c}{**Frequentist**} & **Bayesian** \\ \hline \multicolumn{2}{c}{**Estimation**} & **Confidence Interval** & **Posterior** \\ \hline \multirow{3}{*}{Hypothesis} & Equivalence & TOST & \multirow{3}{*}{Posterior + ROPE _or_ BF} \\ \cline{2-3}  & Difference & NHST & \\ \cline{2-3}  & Model Selection & AIC & Bayes Factor (BF) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Overview of different statistical methods. TOST stands for _Two One-Sided T-Tests_ and AIC is the _Akaike Information Criterion_([PERSON] et al., 2016), a combined measure for model fit and model complexity.