Package: deepsig
deep-significance: Easy and Better Significance Testing for Deep Neural NetworksContents:interrobang: Why:inbox_tray: Installation:bookmark: ExamplesIntermezzo: Almost Stochastic Order - a better significance test for Deep Neural NetworksScenario 1: Comparing multiple runs of two modelsScenario 2: Comparing multiple runs across datasetsScenario 3: Comparing sample-level scoresScenario 4: Comparing more than two modelsHow to report resultsSample sizeOther featuresGeneral Recommendations & other notes:mortar_board: Cite:medal_sports: Acknowledgements:people_holding_hands: Papers using deep-significance:books: Bibliography:interrobang: Why?Although Deep Learning has undergone spectacular growth in the recent decade,
a large portion of experimental evidence is not supported by statistical hypothesis tests. Instead,
conclusions are often drawn based on single performance scores.This is problematic: Neural network display highly non-convex
loss surfaces (Li et al., 2018) and their performance depends on the specific hyperparameters that were found, or stochastic factors
like Dropout masks, making comparisons between architectures more difficult. Based on comparing only (the mean of) a
few scores,we often cannot
conclude that one model type or algorithm is better than another.
This endangers the progress in the field, as seeming success due to random chance might lead practitioners astray.For instance, a recent study in Natural Language Processing by Narang et al. (2021) has found that many modifications proposed to
transformers do not actually improve performance. Similar issues are known to plague other fields like e.g.,
Reinforcement Learning (Henderson et al., 2018) and Computer Vision (Borji, 2017) as well.To help mitigate this problem, this package supplies fully-tested re-implementations of useful functions for significance
testing:Statistical Significance tests such as Almost Stochastic Order (del Barrio et al, 2017; Dror et al., 2019),
bootstrap (Efron & Tibshirani, 1994) and permutation-randomization (Noreen, 1989).Bonferroni correction methods for multiplicity in datasets (Bonferroni, 1936).Bootstrap power analysis (Yuan & Hayashi, 2003) and other functions to determine the right sample size.All functions are fully tested and also compatible with common deep learning data structures, such as PyTorch /
Tensorflow tensors as well as NumPy and Jax arrays.  For examples about the usage, consult the documentationhere, the scenarios in the sectionExamplesor
thedemo Jupyter notebook.:inbox_tray: InstallationThe package can simply be installed usingpipby runningpip3 install deepsigAnother option is to clone the repository and install the package locally:git clone https://github.com/Kaleidophon/deep-significance.git
cd deep-significance
pip3 install -e .Warning: Installed like this, imports will fail when the clones repository is moved.:bookmark: Examplestl;dr: Useaso()to compare scores for two models. If the returnedeps_min < 0.5, A is better than B. The lowereps_min, the more confident the result (we recommend to checkeps_min < 0.2and recordeps_minalongside
experimental results).:warning: Testing models with only one set of hyperparameters and only one test set will be able to guarantee superiority
in all settings. SeeGeneral Recommendations & other notes.In the following, we will lay out three scenarios that describe common use cases for ML practitioners and how to apply
the methods implemented in this package accordingly. For an introduction into statistical hypothesis testing, please
refer to resources such asthis blog postfor a general
overview orDror et al. (2018)for a NLP-specific point of view.We assume that we have two sets of scores we would like to compare, $\mathbb{S}\mathbb{A}$ and $\mathbb{S}\mathbb{B}$,
for instance obtained by running two models $\mathbb{A}$ and $\mathbb{B}$ multiple times with a different random seed.
We can then define a one-sided test statistic  $\delta(\mathbb{S}\mathbb{A}, \mathbb{S}\mathbb{B})$ based on the gathered observations.
An example of such test statistics is for instance the difference in observation means. We then formulate the following null-hypothesis:$$
H_0: \delta(\mathbb{S}\mathbb{A}, \mathbb{S}\mathbb{B}) \le 0
$$That means that we actually assume the opposite of our desired case, namely that $\mathbb{A}$ is not better than $\mathbb{B}$,
but equally as good or worse, as indicated by the value of the test statistic.
Usually, the goal becomes to reject this null hypothesis using the SST.p-value testing is a frequentist method in the realm of SST.
It introduces the notion of data thatcould have been observedif we were to repeat our experiment again using
the same conditions, which we will write with superscript $\text{rep}$ in order to distinguish them from our actually
observed scores (Gelman et al., 2021).
We then define thep-value as the probability that, under the null hypothesis, the test statistic using replicated
observation is larger than or equal to theobservedtest statistic:$$
p(\delta(\mathbb{S}\mathbb{A}^\text{rep}, \mathbb{S}\mathbb{B}^\text{rep}) \ge \delta(\mathbb{S}\mathbb{A}, \mathbb{S}\mathbb{B})|H_0)
$$We can interpret this expression as follows: Assuming that $\mathbb{A}$ is not better than $\mathbb{B}$, the test
assumes a corresponding distribution of statistics that $\delta$ is drawn from. So how does the observed test statistic
$\delta(\mathbb{S}\mathbb{A}, \mathbb{S}\mathbb{B})$ fit in here? This is what the $p$-value expresses: When the
probability is high, $\delta(\mathbb{S}\mathbb{A}, \mathbb{S}\mathbb{B})$ is in line with what we expected under the
null hypothesis, so we cannotreject the null hypothesis, or in other words, we \emph{cannot} conclude
$\mathbb{A}$ to be better than $\mathbb{B}$. If the probability is low, that means that the observed
$\delta(\mathbb{S}, \mathbb{S}_\mathbb{B})$ is quite unlikely under the null hypothesis and that the reverse case is
more likely - i.e. that it is likely larger than - and we conclude that $\mathbb{A}$ is indeed better than
$\mathbb{B}$. Note thatthe $p$-value does not express whether the null hypothesis is true. To make our decision
about whether or not to reject the null hypothesis, we typically determine a threshold - the significance level
$\alpha$, often set to 0.05 - that thep-value has to fall below. However, it has been argued that a better practice
involves reporting thep-value alongside the results without a pidgeonholing of results into significant and non-significant
(Wasserstein et al., 2019).Intermezzo: Almost Stochastic Order - a better significance test for Deep Neural NetworksDeep neural networks are highly non-linear models, having their performance highly dependent on hyperparameters, random
seeds and other (stochastic) factors. Therefore, comparing the means of two models across several runs might not be
enough to decide if a model A is better than B. In fact,even aggregating more statistics like standard deviation, minimum
or maximum might not be enoughto make a decision. For this reason, del Barrio et al. (2017) and Dror et al. (2019)
introducedAlmost Stochastic Order(ASO), a test to compare two score distributions.It builds on the concept ofstochastic order: We can compare two distributions and declare one asstochastically dominantby comparing their cumulative distribution functions:Here, the CDF of A is given in red and in green for B. If the CDF of A is lower than B for every $x$, we know the
algorithm A to score higher. However, in practice these cases are rarely so clear-cut (imagine e.g. two normal
distributions with the same mean but different variances).
For this reason, del Barrio et al. (2017) and Dror et al. (2019) consider the notion ofalmost stochastic dominanceby quantifying the extent to which stochastic order is being violated (red area):ASO returns a value $\epsilon_\text{min}$, which expresses (an upper bound to) the amount of violation of stochastic order. If
$\epsilon_\text{min} < \tau$ (where \tau is 0.5 or less), A is stochastically dominant over B in more cases than vice versa, then the corresponding algorithm can be declared as
superior. We can also interpret $\epsilon_\text{min}$ as aconfidence score. The lower it is, the more sure we can be
that A is better than B. Note:ASO does not compute p-values.Instead, the null hypothesis formulated as$$
H_0: \epsilon_\text{min} \ge \tau
$$If we want to be more confident about the result of ASO, we can also set the rejection threshold to be lower than 0.5
(see the discussion inthis section).
Furthermore, the significance level $\alpha$ is determined as an input argument when running ASO and actively influence
the resulting $\epsilon_\text{min}$.Scenario 1 - Comparing multiple runs of two modelsIn the simplest scenario, we have retrieved a set of scores from a model A and a baseline B on a dataset, stemming from
various model runs with different seeds. We want to test whether our model A is better than B (higher scores = better)-
We can now simply apply the ASO test:importnumpyasnpfromdeepsigimportasoseed=1234np.random.seed(seed)# Simulate scoresN=5# Number of random seedsmy_model_scores=np.random.normal(loc=0.9,scale=0.8,size=N)baseline_scores=np.random.normal(loc=0,scale=1,size=N)min_eps=aso(my_model_scores,baseline_scores,seed=seed)# min_eps = 0.225, so A is betterNote that ASOdoes not make any assumptions about the distributions of the scores.
This means that we can apply it to any kind of test metric, as long as a higher score indicates a better performance
(to apply ASO to cases where lower scores indicate better performances, just multiple your scores by -1 before feeding
them into the function). The more scores of model runs is supplied, the more reliable
the test becomes, so try to collect scores from as many runs as possible to reject the null hypothesis confidently.Scenario 2 - Comparing multiple runs across datasetsWhen comparing models across datasets, we formulate one null hypothesis per dataset. However, we have to make sure not to
fall prey to themultiple comparisons problem: In short,
the more comparisons between A and B we are conducting, the more likely gets is to reject a null-hypothesis accidentally.
That is why we have to adjust our significance threshold $\alpha$ accordingly by dividing it by the number of comparisons,
which corresponds to the Bonferroni correction (Bonferroni et al., 1936):importnumpyasnpfromdeepsigimportasoseed=1234np.random.seed(seed)# Simulate scores for three datasetsM=3# Number of datasetsN=5# Number of random seedsmy_model_scores_per_dataset=[np.random.normal(loc=0.3,scale=0.8,size=N)for_inrange(M)]baseline_scores_per_dataset=[np.random.normal(loc=0,scale=1,size=N)for_inrange(M)]# epsilon_min values with Bonferroni correctioneps_min=[aso(a,b,confidence_level=0.95,num_comparisons=M,seed=seed)fora,binzip(my_model_scores_per_dataset,baseline_scores_per_dataset)]# eps_min = [0.006370113450148568, 0.6534772728574852, 0.0]Scenario 3 - Comparing sample-level scoresIn previous examples, we have assumed that we compare two algorithms A and B based on their performance per run, i.e.
we run each algorithm once per random seed and obtain exactly one score on our test set. In some cases however,
we would like to compare two algorithms based on scoresfor every point in the test set. If we only use one seed
per model, then this case is equivalent to scenario 1. But what if we also want to use multiple seeds per model?In this scenario, we can do pair-wise comparisons of the score distributions between A and B and use the Bonferroni
correction accordingly:fromitertoolsimportproductimportnumpyasnpfromdeepsigimportasoseed=1234np.random.seed(seed)# Simulate scores for three datasetsM=40# Number of data pointsN=3# Number of random seedsmy_model_scored_samples_per_run=[np.random.normal(loc=0.3,scale=0.8,size=M)for_inrange(N)]baseline_scored_samples_per_run=[np.random.normal(loc=0,scale=1,size=M)for_inrange(N)]pairs=list(product(my_model_scored_samples_per_run,baseline_scored_samples_per_run))# epsilon_min values with Bonferroni correctioneps_min=[aso(a,b,confidence_level=0.95,num_comparisons=len(pairs),seed=seed)fora,binpairs]# eps_min = [0.3831678636198528, 0.07194780234194881, 0.9152792807128325, 0.5273463008857844, 0.14946944524461184, 1.0,# 0.6099543280369378, 0.22387448804041898, 1.0]Scenario 4 - Comparing more than two modelsSimilarly, when comparing multiple models (now again on a per-seed basis), we can use a similar approach like in the
previous example. For instance, for three models, we can create a $3 \times 3$ matrix and fill the entries
with the corresponding $\epsilon_\text{min}$ values.The package implements the functionmulti_aso()exactly for this purpose. It has the same arguments asaso(), with
a few differences. First of all, the function takes a singlescoresargument, which can be a list of lists (of scores),
or a nested NumPy array or Tensorflow / PyTorch / Jax tensor or dictionary (more about that later).
Let's look at an example:importnumpyasnpfromdeepsigimportmulti_asoseed=1234np.random.seed(seed)N=5# Number of random seedsM=3# Number of different models / algorithms# Simulate different model scores by sampling from normal distributions with increasing means# Here, we will sample from N(0.1, 0.8), N(0.15, 0.8), N(0.2, 0.8)my_models_scores=np.array([np.random.normal(loc=loc,scale=0.8,size=N)forlocinnp.arange(0.1,0.1+0.05*M,step=0.05)])eps_min=multi_aso(my_models_scores,confidence_level=0.95,seed=seed)# eps_min =# array([[1.       , 0.92621655, 1.        ],#       [1.        , 1.        , 1.        ],#       [0.82081635, 0.73048716, 1.        ]])In the example,eps_minis now a matrix, containing the $\epsilon_\text{min}$ score between all pairs of models (for
the same model, it set to 1 by default). The matrix is always to be read as ASO(row, column).The function applies the bonferroni correction for multiple comparisons by
default, but this can be turned off by usinguse_bonferroni=False.Lastly, when thescoresargument is a dictionary and the function is called withreturn_df=True, the resulting matrix is
given as apandas.DataFramefor increased readability:importnumpyasnpfromdeepsigimportmulti_asoseed=1234np.random.seed(seed)N=5# Number of random seedsM=3# Number of different models / algorithms# Same setup as above, but use a dict for scoresmy_models_scores={f"model{i+1}":np.random.normal(loc=loc,scale=0.8,size=N)fori,locinenumerate(np.arange(0.1,0.1+0.05*M,step=0.05))}# my_model_scores = {#   "model 1": array([...]),#   "model 2": array([...]),#   ...# }eps_min=multi_aso(my_models_scores,confidence_level=0.95,return_df=True,seed=seed)# This is now a DataFrame!# eps_min =#          model 1   model 2  model 3# model 1  1.000000  0.926217      1.0# model 2  1.000000  1.000000      1.0# model 3  0.820816  0.730487      1.0:newspaper: How to report resultsWhen ASO used, two important details have to be reported, namely the confidence level $\alpha$ and the $\epsilon_\text{min}$
score. Below lists some example snippets reporting the results of scenarios 1 and 4:Using ASO with a confidence level $\alpha = 0.05$, we found the score distribution of algorithm A based on three 
random seeds to be stochastically dominant over B ($\epsilon_\text{min} = 0$).

We compared all pairs of models based on five random seeds each using ASO with a confidence level of 
$\alpha = 0.05$ (before adjusting for all pair-wise comparisons using the Bonferroni correction). Almost stochastic 
dominance ($\epsilon_\text{min} < \tau$ with $\tau = 0.2$) is indicated in table X.:control_knobs: Sample sizeIt can be hard to determine whether the currently collected set of scores is large enough to allow for reliable
significance testing or whether more scores are required. For this reason,deep-significancealso implements functions to aid the decision of whether to
collect more samples or not.First of all, it containsBootstrap power analysis(Yuan & Hayashi, 2003): Given a set of scores, it gives all of them a uniform lift to
create an artificial, second sample. Then, the analysis runs repeated analyses using bootstrapped versions of both
samples, comparing them with a significance test. Ideally, this should yield a significant result: If the difference
between the re-sampled original and the lifted sample is non-significant, the original sample has too high of a variance. The
analyses then returns thepercentage of comparisonsthat yielded significant results. If the number is too low,
more scores should be collected and added to the sample.The result of the analysis is thestatistical power: The
higher the power, the smaller the risk of falling prey to a Type II error - the probability of mistakenly accepting the
null hypothesis, when in fact it should actually be rejected. Usually, a power of ~ 0.8 is recommended (although that is
sometimes hard to achieve in a machine learning setup).The function can be used in the following way:importnumpyasnpfromdeepsigimportbootstrap_power_analysisscores=np.random.normal(loc=0,scale=20,size=5)# Create too small of a sample with high variancepower=bootstrap_power_analysis(scores,show_progress=False)# 0.081, way too lowscores2=np.random.normal(loc=0,scale=20,size=50)# Let's collect more samplespower2=bootstrap_power_analysis(scores2,show_progress=False)# Better power with 0.2556By default,bootstrap_power_analysis()uses a one-sided Welch's t-test. However, this can be modified by passing
a function to thesignificance_testargument, which expects a function taking two sets of scores and returning a
p-value.Secondly, if the Almost Stochastic Order test (ASO) is being used, there is a second function available. ASO estimates
the violation ratio of two samples using bootstrapping. However, there is necessarily some uncertainty around that
estimate, given that we only possess a finite number of samples. Using more samples decreases the uncertainty and makes the estimate tighter.
The degree to which collecting more samples increases the tightness can be computed using the following function:importnumpyasnpfromdeepsigimportaso_uncertainty_reductionscores1=np.random.normal(loc=0,scale=0.3,size=5)# First sample with five scoresscores2=np.random.normal(loc=0.2,scale=5,size=3)# Second sample with three scoresred1=aso_uncertainty_reduction(m_old=len(scores1),n_old=len(scores2),m_new=5,n_new=5)# 1.1547005383792515red2=aso_uncertainty_reduction(m_old=len(scores1),n_old=len(scores2),m_new=7,n_new=3)# 1.0583005244258363# Adding two runs to scores1 increases tightness of estimate by 1.15# But adding two runs to scores2 only increases tightness by 1.06! So spending two more runs on scores1 is better:sparkles: Other features:rocket: For the impatient: ASO with multi-threadingWaiting for all the bootstrap iterations to finish can feel tedious, especially when doing many comparisons. Therefore,
ASO supports multithreading usingjoblibvia thenum_jobsargument.fromdeepsigimportasoimportnumpyasnpfromtimeitimporttimeita=np.random.normal(size=1000)b=np.random.normal(size=1000)print(timeit(lambda:aso(a,b,num_jobs=1,show_progress=False),number=5))# 616.2249192680001print(timeit(lambda:aso(a,b,num_jobs=4,show_progress=False),number=5))# 208.05637107000007If you want to select the maximum number of jobs possible on your device, you can setnum_jobs=-1:print(timeit(lambda: aso(a, b, num_jobs=-1, show_progress=False), number=5))  # 187.26257274800003:electric_plug: Compatibility with PyTorch, Tensorflow, Jax & NumpyAll tests implemented in this package also can take PyTorch / Tensorflow tensors and Jax or NumPy arrays as arguments:fromdeepsigimportasoimporttorcha=torch.randn(5,1)b=torch.randn(5,1)aso(a,b)# It just works!:woman_farmer: Setting seeds for replicabilityIn order to ensure replicability, bothaso()andmulti_aso()supply asseedargument. This even works
when multiple jobs are used!:game_die: Permutation and bootstrap testShould you be suspicious of ASO and want to revert to the good old faithful tests, this package also implements
the paired-bootstrap as well as the permutation randomization test. Note that as discussed in the next section, these
tests have less statistical power than ASO. Furthermore, a function for the Bonferroni-correction using
p-values can also be found usingfrom deepsig import bonferroni_correction.importnumpyasnpfromdeepsigimportbootstrap_test,permutation_testa=np.random.normal(loc=0.8,size=10)b=np.random.normal(size=10)print(permutation_test(a,b))# 0.16183816183816183print(bootstrap_test(a,b))# 0.103General recommendations & other notesNaturally, the CDFs built fromscores_aandscores_bcan only be approximations of the true distributions. Therefore,
as many scores as possible should be collected, especially if the variance between runs is high. If only one run is available,
comparing sample-wise score distributions like in scenario 3 can be an option, but comparing multiple runs willalwaysbe preferable. Ideally, scores should be obtained even using different sets of hyperparameters per model.
Because this is usually infeasible in practice, Bouthilier et al. (2020) recommend tovary all other sources of variationbetween runs to obtain the most trustworthy estimate of the "true" performance, such as data shuffling, weight initialization etc.num_bootstrap_iterationscan be reduced to increase the speed ofaso(). However, this is not
recommended as the result of the test will also become less accurate. Technically, $\epsilon_\text{min}$ is a upper bound
that becomes tighter with the number of samples and bootstrap iterations (del Barrio et al., 2017). Thus, increasing
the number of jobs withnum_jobsinstead is always preferred.While we could declare a model stochastically dominant with $\epsilon_\text{min} < 0.5$, we found this to have a comparatively high
Type I error (false positives). Testsin our paperhave shown that a more useful threshold that trades of Type I and
Type II error between different scenarios might be $\tau = 0.2$.Bootstrap and permutation-randomization are all non-parametric tests, i.e. they don't make any assumptions about
the distribution of our test metric. Nevertheless, they differ in theirstatistical power, which is defined as the probability
that the null hypothesis is being rejected given that there is a difference between A and B. In other words, the more powerful
a test, the less conservative it is and the more it is able to pick up on smaller difference between A and B. Therefore,
if the distribution is known or found out why normality tests (like e.g. Anderson-Darling or Shapiro-Wilk), something like
a parametric test like Student's or Welch's t-test is preferable to bootstrap or permutation-randomization. However,
because these test are in turn less applicable in a Deep Learning setting due to the reasons elaborated on inWhy?, ASO is still a better choice.:mortar_board: CiteUsing this package in general, please cite the following:@article{ulmer2022deep,
  title={deep-significance-Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks},
  author={Ulmer, Dennis and Hardmeier, Christian and Frellsen, Jes},
  journal={arXiv preprint arXiv:2204.06815},
  year={2022}
}If you use the ASO test viaaso()or `multi_aso, please cite the original works:@inproceedings{dror2019deep,
  author    = {Rotem Dror and
               Segev Shlomov and
               Roi Reichart},
  editor    = {Anna Korhonen and
               David R. Traum and
               Llu{\'{\i}}s M{\`{a}}rquez},
  title     = {Deep Dominance - How to Properly Compare Deep Neural Models},
  booktitle = {Proceedings of the 57th Conference of the Association for Computational
               Linguistics, {ACL} 2019, Florence, Italy, July 28- August 2, 2019,
               Volume 1: Long Papers},
  pages     = {2773--2785},
  publisher = {Association for Computational Linguistics},
  year      = {2019},
  url       = {https://doi.org/10.18653/v1/p19-1266},
  doi       = {10.18653/v1/p19-1266},
  timestamp = {Tue, 28 Jan 2020 10:27:52 +0100},
}

@incollection{del2018optimal,
  title={An optimal transportation approach for assessing almost stochastic order},
  author={Del Barrio, Eustasio and Cuesta-Albertos, Juan A and Matr{\'a}n, Carlos},
  booktitle={The Mathematics of the Uncertain},
  pages={33--44},
  year={2018},
  publisher={Springer}
}For instance, you can writeIn order to compare models, we use the Almost Stochastic Order test \citep{del2018optimal, dror2019deep} as 
implemented by \citet{ulmer2022deep}.:medal_sports: AcknowledgementsThis package was created out of discussions of theNLPnorth groupat the IT University
Copenhagen, whose members I want to thank for their feedback. The code in this repository is in multiple places based on
several ofRotem Dror'srepositories, namelythis,thisandthis one. Thanks also go out to her personally for being available to
answer questions and provide feedback to the implementation and documentation of this package.The commit message template used in this project can be foundhere.
The inline latex equations were rendered usingreadme2latex.:people_holding_hands: Papers using deep-significanceIn this last section of the readme, I would like to refer to works already usingdeep-significance. Open an issue or
pull request if you would like to see your work added here!"From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding" (van der Groot et al., 2021)"Cartography Active Learning" (Zhang & Plank, 2021)"SkillSpan: Hard and Soft Skill Extraction from English Job Postings" (Zhang et al., 2022a)"What do you mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification" (Bassignana & Plank, 2022)"KOMPETENCER: Fine-grained Skill Classification in Danish Job Postings
via Distant Supervision and Transfer Learning" (Zhang et al., 2022b)"Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity" (Ulmer et al., 2022)"DBGSL: Dynamic Brain Graph Structure Learning" (Campbell et al., 2022)"An Empirical Study on Cross-X Transfer for Legal Judgment Prediction" (Niklaus et al., 2022):books: BibliographyDel Barrio, Eustasio, Juan A. Cuesta-Albertos, and Carlos Matrán. "An optimal transportation approach for assessing almost stochastic order." The Mathematics of the Uncertain. Springer, Cham, 2018. 33-44.Bonferroni, Carlo. "Teoria statistica delle classi e calcolo delle probabilita." Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8 (1936): 3-62.Borji, Ali. "Negative results in computer vision: A perspective." Image and Vision Computing 69 (2018): 1-8.Bouthillier, Xavier, et al. "Accounting for variance in machine learning benchmarks." Proceedings of Machine Learning and Systems 3 (2021).Dror, Rotem, et al. "The hitchhiker’s guide to testing statistical significance in natural language processing." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.Dror, Rotem, Shlomov, Segev, and Reichart, Roi. "Deep dominance-how to properly compare deep neural models." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.Efron, Bradley, and Robert J. Tibshirani. "An introduction to the bootstrap." CRC press, 1994.Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, Donald B Rubin, John
Carlin, Hal Stern, Donald Rubin, and David Dunson. Bayesian data analysis third edition, 2021.Henderson, Peter, et al. "Deep reinforcement learning that matters." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. "Visualizing the Loss Landscape of Neural Nets." NeurIPS 2018: 6391-6401Narang, Sharan, et al. "Do Transformer Modifications Transfer Across Implementations and Applications?." arXiv preprint arXiv:2102.11972 (2021).Noreen, Eric W. "Computer intensive methods for hypothesis testing: An introduction." Wiley, New York (1989).Ronald L Wasserstein, Allen L Schirm, and Nicole A Lazar. Moving to a world beyond “p< 0.05”,
2019Yuan, Ke‐Hai, and Kentaro Hayashi. "Bootstrap approach to inference and power analysis based on three test statistics for covariance structure models." British Journal of Mathematical and Statistical Psychology 56.1 (2003): 93-110.