# An Extension of the Permutation Test for Dependent Data in Samples with a Small Number of Subjects

Petr Ermolinskiy

<figure><img src="image_1.png" /></figure>

To cite this version:

Petr Ermolinskiy. An Extension of the Permutation Test for Dependent Data in Samples with a Small Number of Subjects. 2024. hal-04558513

HAL Id: hal-04558513

https://hal.science/hal-04558513

Preprint submitted on 2 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_1.png" /></figure>

HAL
open science

# An Extension of the Permutation Test for Dependent Data in Samples with a Small Number of Subjects

Petr Ermolinskiy

► To cite this version:

Petr Ermolinskiy. An Extension of the Permutation Test for Dependent Data in Samples with a Small Number of Subjects. 2024. hal-04558513

HAL Id: hal-04558513

https://hal.science/hal-04558513v1

Preprint submitted on 2 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# AN EXTENSION OF THE PERMUTATION TEST FOR DEPENDENT DATA IN SAMPLES WITH A SMALL NUMBER OF SUBJECTS

A PREPRINT

● **Petr B. Ermolinskii**

Physics Department

M.V. Lomonosov Moscow State University

Moscow, Russia

ermolinskiy.pb15@physics.msu.ru

April 24, 2024

## ABSTRACT

This study introduces an extended permutation test for dependent data with small sample sizes, incorporating measurement errors. This method is especially applicable in single case studies, offering a valuable tool for hypothesis testing without strict distributional assumptions. Additional research is needed to fully assess the effectiveness and validity of this approach.

**Keywords** statistics · permutation · dependent data · small sample size

## 1 Introduction

Permutation tests are non-parametric statistical methods used to evaluate the significance of an observed difference in datasets or between datasets [1]. They are particularly useful when the assumptions of parametric tests are violated, or when dealing with small sample sizes or non-normal data. Permutation tests provide exact p-values and are robust against various distributions and outliers present in the data [2].

In a permutation test, the null hypothesis is tested by creating a distribution of test statistics under the assumption that the null hypothesis is true. This is achieved by randomly reshuffling or permuting the data to create multiple permutations of the dataset. The test statistic is calculated for each permutation, and the distribution of these permuted test statistics forms the null distribution [3]. That is why permutation tests are available for any test statistic, whether its distribution is known or not [4]. For example, one can perform a permutation version of Brunner Munzel or Mann Whitney Wilcoxon test in the python programming language [5].

The basic steps of the permutation test include:

1. Define the null and alternative hypotheses.

2. Compute the observed test statistic (mean, median, or etc.) from the original data.

3. Randomly permute the data (shuffle the labels or values) to create multiple permuted datasets.

4. Calculate the test statistic for each permuted dataset.

5. Compare the observed test statistic with the distribution of permuted test statistics to determine the p-value.

6. Based on the p-value, make a decision regarding the null hypothesis.

It is very important to note that the "permutation" assumes interchangeability of data between samples. In most cases, it means that the variance of different samples should be equal. Using permutation test for samples with unequal variances can lead to an inflated Type I error rate even when the means are equal [6]. In that case, one can assume a more general alternative hypothesis or control the frequency of Type I errors using modified tests or testing algorithms [1].
# AN EXTENSION OF THE PERMUTATION TEST FOR DEPENDENT DATA IN SAMPLES WITH A SMALL NUMBER OF SUBJECTS

## A PREPRINT

● **Petr B. Ermolinskii**

Physics Department

M.V. Lomonosov Moscow State University

Moscow, Russia

ermolinskiy.pb15@physics.msu.ru

April 24, 2024

## ABSTRACT

This study introduces an extended permutation test for dependent data with small sample sizes, incorporating measurement errors. This method is especially applicable in single case studies, offering a valuable tool for hypothesis testing without strict distributional assumptions. Additional research is needed to fully assess the effectiveness and validity of this approach.

**Keywords** statistics · permutation · dependent data · small sample size

## 1 Introduction

Permutation tests are non-parametric statistical methods used to evaluate the significance of an observed difference in datasets or between datasets [1]. They are particularly useful when the assumptions of parametric tests are violated, or when dealing with small sample sizes or non-normal data. Permutation tests provide exact p-values and are robust against various distributions and outliers present in the data [2].

In a permutation test, the null hypothesis is tested by creating a distribution of test statistics under the assumption that the null hypothesis is true. This is achieved by randomly reshuffling or permuting the data to create multiple permutations of the dataset. The test statistic is calculated for each permutation, and the distribution of these permuted test statistics forms the null distribution [3]. That is why permutation tests are available for any test statistic, whether its distribution is known or not [4]. For example, one can perform a permutation version of Brunner Munzel or Mann Whitney Wilcoxon test in the python programming language [5].

The basic steps of the permutation test include:

1. Define the null and alternative hypotheses.

2. Compute the observed test statistic (mean, median, or etc.) from the original data.

3. Randomly permute the data (shuffle the labels or values) to create multiple permuted datasets.

4. Calculate the test statistic for each permuted dataset.

5. Compare the observed test statistic with the distribution of permuted test statistics to determine the p-value.

6. Based on the p-value, make a decision regarding the null hypothesis.

It is very important to note that the "permutation" assume interchangeability of data between samples. In most cases, it means that the variance of different samples should be equal. Using permutation test for samples with unequal variances can lead to an inflated Type I error rate even when the means are equal [6]. In that case, one can assume a more general alternative hypothesis or control the frequency of Type I errors using modified tests or testing algorithms [1].
When the sample size is large, an asymptotically equivalent permutation test can be formed. This involves creating a benchmark distribution through Monte Carlo sampling, which randomly selects a small sample of possible replicates compared to the total number of permutations [7].

However, permutation tests are mainly used in single case studies, i.e., with small sample sizes. Single case studies are commonly used to study clinical interventions because of the impracticality or the impossibility of conducting large-scale trials in clinical studies. They are now a valuable tool for testing various medical, psychological, and other hypotheses[8]. The extension of permutation test for treatment effect in single case studies shows robustness under the null hypothesis and efficacy under the alternative hypothesis [9, 10].

In this work, an extension of the permutation test for dependent data, i.e., paired permutation test, in samples with a small number of subjects is proposed. The basic idea behind the extension is that basically all tests use only mean values, whereas in a medical or other experiment there is some measurement or averaging error when estimating a parameter. By utilizing the error, i.e., standard deviation or other error, for each treatment measurement, better performance of paired permutation test can be expected.

## 2 Algorithm of proposed test with standard deviation

The paired permutation test, similar to other statistical tests for dependent data, aims to identify the difference between two groups: before (first) and after treatment (second). Usually the parameter under consideration for statistics has some kind of measurement error (standard deviation (SD), instrumental error, etc) that is usually omitted when using statistical tests, e.g., Wilcoxon signed-rank test. Further in the text, SD will be taken as the measurement error.

The extension of paired permutation test aims to use the error of parameter in the algorithm and includes the following steps:

1. Compute the observed test statistic, i.e., the average parameter difference by patient in percent between the second and the first group.

2. Permute the data in the first and in the second group to create multiple permuted datasets using mean and SD value. Each value in both the second and the first group can be presented in 3 ways, i.e., (1) mean, (2) mean + SD and (3) mean - SD. For example, 9 combinations can be derived from the data of two samples (see Fig.1).

3. Randomly permute the data to create multiple permuted datasets of the first and the second groups.

4. Calculate the average parameter difference by patient in percent between all permuted datasets.

5. Compare the observed test statistic with the distribution of permuted test statistics to determine the p-value.

6. Based on the p-value, make a decision regarding the null hypothesis.

<figure><img src="image_3.png" /><figcaption>Figure 1: Demonstration of creating new permuted datasets of one group of two values with errors (or SD).</figcaption></figure>

The steps above are implemented in python and the code is available in the GitHub repository [11].
When the sample size is large, an asymptotically equivalent permutation test can be formed. This involves creating a benchmark distribution through Monte Carlo sampling, which randomly selects a small sample of possible replicates compared to the total number of permutations [7].

However, permutation tests are mainly used in single case studies, i.e., with small sample sizes. Single case studies are commonly used to study clinical interventions because of the impracticality or the impossibility of conducting large-scale trials in clinical studies. They are now a valuable tool for testing various medical, psychological, and other hypotheses[8]. The extension of permutation test for treatment effect in single case studies shows robustness under the null hypothesis and efficacy under the alternative hypothesis [9, 10].

In this work, an extension of the permutation test for dependent data, i.e., paired permutation test, in samples with a small number of subjects is proposed. The basic idea behind the extension is that basically all tests use only mean values, whereas in a medical or other experiment there is some measurement or averaging error when estimating a parameter. By utilizing the error, i.e., standard deviation or other error, for each treatment measurement, better performance of paired permutation test can be expected.

## 2 Algorithm of proposed test with standard deviation

The paired permutation test, similar to other statistical tests for dependent data, aims to identify the difference between two groups: before (first) and after treatment (second). Usually the parameter under consideration for statistics has some kind of measurement error (standard deviation (SD), instrumental error, etc) that is usually omitted when using statistical tests, e.g., Wilcoxon signed-rank test. Further in the text, SD will be taken as the measurement error.

The extension of paired permutation test aims to use the error of parameter in the algorithm and includes the following steps:

1. Compute the observed test statistic, i.e., the average parameter difference by patient in percent between the second and the first group.

2. Permute the data in the first and in the second group to create multiple permuted datasets using mean and SD value. Each value in both the second and the first group can be presented in 3 ways, i.e., (1) mean, (2) mean + SD and (3) mean - SD. For example, 9 combinations can be derived from the data of two samples (see Fig.1).

3. Randomly permute the data to create multiple permuted datasets of the first and the second groups.

4. Calculate the average parameter difference by patient in percent between all permuted datasets.

5. Compare the observed test statistic with the distribution of permuted test statistics to determine the p-value.

6. Based on the p-value, make a decision regarding the null hypothesis.

<figure><img src="image_3.png" /><figcaption>Figure 1: Demonstration of creating new permuted datasets of one group of two values with errors (or SD).</figcaption></figure>

The steps above are implemented in python and the code is available in the GitHub repository [11].
<table><caption>Table 1: Calculated p-value for two examples using Wilcoxon signed-rank test and proposed paired permutation test with SD.</caption><thead><tr><th rowspan="2">Statistical test</th><th colspan="2">Example 1</th><th colspan="2">Example 2</th></tr><tr><th>Before treatment</th><th>After treatment</th><th>Before treatment</th><th>After treatment</th></tr></thead><tbody><tr><td>mean</td><td>[20, 33, 20, 24]</td><td>[21, 34, 22, 25]</td><td>[20, 33, 20, 24]</td><td>[21, 34, 22, 25]</td></tr><tr><td>SD</td><td>[2, 2, 2, 2]</td><td>[2, 2, 2, 2]</td><td>[0.5, 0.5, 0.5, 0.5]</td><td>[0.5, 0.5, 0.5, 0.5]</td></tr><tr><td>Wilcoxon signed-rank test</td><td colspan="2">0.063</td><td colspan="2">0.063</td></tr><tr><td>Paired permutation with SD</td><td colspan="2">0.1874</td><td colspan="2">0.045</td></tr></tbody></table>

<table><caption>Table 2: Calculated p-value for the example when SD are different and high in both groups using Wilcoxon signed-rank test and proposed paired permutation test with SD.</caption><thead><tr><th rowspan="2">Statistical test</th><th colspan="2">Example 3</th></tr><tr><th>Before treatment</th><th>After treatment</th></tr></thead><tbody><tr><td>mean</td><td>[40, 45, 51, 60, 38, 44]</td><td>[41, 48, 51.1, 62, 38.5, 46]</td></tr><tr><td>SD</td><td>[1, 2, 0.5, 1, 0.6, 0.3]</td><td>[0.5, 2, 0.9, 0.5, 0.7, 0.1]</td></tr><tr><td>Wilcoxon signed-rank test</td><td colspan="2">0.016</td></tr><tr><td>Paired permutation with SD</td><td colspan="2">0.052</td></tr></tbody></table>

## 3 Examples

In this section, Table 1 and Table 2 summarize three examples of using the proposed paired permutation test with SD. It can be seen from Table 1 that the increased SD values significantly affect the p-value. That is why in a sample with a small number of subjects it is crucial to estimate the error of each value.

To exemplify the significance of the SD, consider a dataset containing 6 subjects each before and after treatment (Table 2), with each value having a relative error of less than 5%. The p-value calculated using the Wilcoxon signed-rank test is less than 0.02, whilst the p-value calculated using the proposed paired permutation with SD is more than 0.05. This may be preliminary evidence that in small samples the Wilcoxon signed-rank test (or a paired test not accounting for SD) may yield false positive statistical significance when the error of values is not negligible.

## 4 Conclusion

Permutation tests offer a flexible and powerful approach for hypothesis testing in statistics, allowing researchers to draw reliable conclusions without strict distributional assumptions. By leveraging permutations of the data, these tests provide a reliable method for assessing the significance of observed effects or differences in a wide range of research fields.

In this work, an extension of the permutation test for dependent data in samples with a small number of subjects utilizing the errors of measurements is proposed and discussed. It is believed to be useful in single case studies. However, further examination of this approach is required.

## References

[1] Holt, C. A., & Sullivan, S. P. (2023). Permutation tests for experimental data. Experimental Economics, 26(4), 775-812. https://doi.org/10.1007/s10683-023-09799-6

[2] Noguchi, K., Konietschke, F., Marmolejo-Ramos, F., & Pauly, M. (2021). Permutation tests are robust and powerful at 0.5% and 5% significance levels. Behavior Research Methods, 53(6), 2712-2724. https://doi.org/10.3758/s13428-021-01595-5

[3] Good, P. (2010). Permutation, Parametric, and Bootstrap Tests of Hypotheses. Springer New York, NY, p.316. https://doi.org/10.1007/b138696

[4] Welch, W. J. (1990). Construction of Permutation Tests. Journal of the American Statistical Association, 85(411), 693-698. https://doi.org/10.1080/01621459.1990.10474929

[5] Charlier, F. (2022). Permutation-based statistical tests in Python. https://pypi.org/project/permutations-stats/

[6] Huang, Y., Xu, H., Calian, V., & Hsu, J. C. (2006). To permute or not to permute. Bioinformatics, 22(18), 2244-2248. https://doi.org/10.1093/bioinformatics/bt1383
<table><caption>Table 1: Calculated p-value for two examples using Wilcoxon signed-rank test and proposed paired permutation test with SD.</caption><thead><tr><th rowspan="2">Statistical test</th><th colspan="2">Example 1</th><th colspan="2">Example 2</th></tr><tr><th>Before treatment</th><th>After treatment</th><th>Before treatment</th><th>After treatment</th></tr></thead><tbody><tr><td>mean</td><td>[20, 33, 20, 24]</td><td>[21, 34, 22, 25]</td><td>[20, 33, 20, 24]</td><td>[21, 34, 22, 25]</td></tr><tr><td>SD</td><td>[2, 2, 2, 2]</td><td>[2, 2, 2, 2]</td><td>[0.5, 0.5, 0.5, 0.5]</td><td>[0.5, 0.5, 0.5, 0.5]</td></tr><tr><td>Wilcoxon signed-rank test</td><td colspan="2">0.063</td><td colspan="2">0.063</td></tr><tr><td>Paired permutation with SD</td><td colspan="2">0.1874</td><td colspan="2">0.045</td></tr></tbody></table>

<table><caption>Table 2: Calculated p-value for the example when SD are different and high in both groups using Wilcoxon signed-rank test and proposed paired permutation test with SD.</caption><thead><tr><th rowspan="2">Statistical test</th><th colspan="2">Example 3</th></tr><tr><th>Before treatment</th><th>After treatment</th></tr></thead><tbody><tr><td>mean</td><td>[40, 45, 51, 60, 38, 44]</td><td>[41, 48, 51.1, 62, 38.5, 46]</td></tr><tr><td>SD</td><td>[1, 2, 0.5, 1, 0.6, 0.3]</td><td>[0.5, 2, 0.9, 0.5, 0.7, 0.1]</td></tr><tr><td>Wilcoxon signed-rank test</td><td colspan="2">0.016</td></tr><tr><td>Paired permutation with SD</td><td colspan="2">0.052</td></tr></tbody></table>

## 3 Examples

In this section, Table 1 and Table 2 summarize three examples of using the proposed paired permutation test with SD. It can be seen from Table 1 that the increased SD values significantly affect the p-value. That is why in a sample with a small number of subjects it is crucial to estimate the error of each value.

To exemplify the significance of the SD, consider a dataset containing 6 subjects each before and after treatment (Table 2), with each value having a relative error of less than 5%. The p-value calculated using the Wilcoxon signed-rank test is less than 0.02, whilst the p-value calculated using the proposed paired permutation with SD is more than 0.05. This may be preliminary evidence that in small samples the Wilcoxon signed-rank test (or a paired test not accounting for SD) may yield false positive statistical significance when the error of values is not negligible.

## 4 Conclusion

Permutation tests offer a flexible and powerful approach for hypothesis testing in statistics, allowing researchers to draw reliable conclusions without strict distributional assumptions. By leveraging permutations of the data, these tests provide a reliable method for assessing the significance of observed effects or differences in a wide range of research fields.

In this work, an extension of the permutation test for dependent data in samples with a small number of subjects utilizing the errors of measurements is proposed and discussed. It is believed to be useful in single case studies. However, further examination of this approach is required.

## References

[1] Holt, C. A., & Sullivan, S. P. (2023). Permutation tests for experimental data. Experimental Economics, 26(4), 775-812. https://doi.org/10.1007/s10683-023-09799-6

[2] Noguchi, K., Konietschke, F., Marmolejo-Ramos, F., & Pauly, M. (2021). Permutation tests are robust and powerful at 0.5% and 5% significance levels. Behavior Research Methods, 53(6), 2712-2724. https://doi.org/10.3758/s13428-021-01595-5

[3] Good, P. (2010). Permutation, Parametric, and Bootstrap Tests of Hypotheses. Springer New York, NY, p.316. https://doi.org/10.1007/b138696

[4] Welch, W. J. (1990). Construction of Permutation Tests. Journal of the American Statistical Association, 85(411), 693-698. https://doi.org/10.1080/01621459.1990.10474929

[5] Charlier, F. (2022). Permutation-based statistical tests in Python. https://pypi.org/project/permutations-stats/

[6] Huang, Y., Xu, H., Calian, V., & Hsu, J. C. (2006). To permute or not to permute. Bioinformatics, 22(18), 2244-2248. https://doi.org/10.1093/bioinformatics/bt1383
[7] Zelený, D. (2022). Monte Carlo permutation test in constrained ordination. https://www.davidzeleny.net/anadat-r/doku.php/en:monte_carlo.

[8] Nickels, L., Fischer-Baum, S., & Best, W. (2022). Single case studies are a powerful tool for developing, testing and extending theories. Nature Reviews Psychology, 1(12), 733-747. https://doi.org/10.1038/s44159-022-00127-y

[9] Solmi, F., Onghena, P., Salmaso, L., & Bulté, I. (2013). Extensions of Permutation Solutions to Test for Treatment Effects in Replicated Single-Case Alternation Experiments with Multivariate Response. Communications in Statistics - Simulation and Computation, 43(5), 1036-1051. https://doi.org/10.1080/03610918.2012.725144

[10] Vroegindeweij, A., Nijhof, L. N., Onghena, P., van de Putte, E. M., Nijhof, S. L., & Houtveen, J. (2023). The Permutation Distancing Test for dependent single-case observational AB-phase design data: A Monte Carlo simulation study. Behavior Research Methods, 56(3), 2569-2580. https://doi.org/10.3758/s13428-023-02167-5

[11] Ermolinskiy, P. (2024). GitHub repository, https://github.com/Petr-Ermolinskiy/Paired-Permutation-Test-With-SD
[7] Zelený, D. (2022). Monte Carlo permutation test in constrained ordination. https://www.davidzeleny.net/anadat-r/doku.php/en:monte_carlo.

[8] Nickels, L., Fischer-Baum, S., & Best, W. (2022). Single case studies are a powerful tool for developing, testing and extending theories. Nature Reviews Psychology, 1(12), 733-747. https://doi.org/10.1038/s44159-022-00127-y

[9] Solmi, F., Onghena, P., Salmaso, L., & Bulté, I. (2013). Extensions of Permutation Solutions to Test for Treatment Effects in Replicated Single-Case Alternation Experiments with Multivariate Response. Communications in Statistics - Simulation and Computation, 43(5), 1036-1051. https://doi.org/10.1080/03610918.2012.725144

[10] Vroegindeweij, A., Nijhof, L. N., Onghena, P., van de Putte, E. M., Nijhof, S. L., & Houtveen, J. (2023). The Permutation Distancing Test for dependent single-case observational AB-phase design data: A Monte Carlo simulation study. Behavior Research Methods, 56(3), 2569-2580. https://doi.org/10.3758/s13428-023-02167-5

[11] Ermolinskiy, P. (2024). GitHub repository, https://github.com/Petr-Ermolinskiy/Paired-Permutation-Test-With-SD