Repository: JuliaStats/HypothesisTests.jl
Pull request #252 (2021-12-28 21:48:16)

Use StableRNG in more tests to avoid breakage in Julia 1.7
Also, introduce an `rng` keyword argument in `ApproximatePermutationTest` to allow passing an `AbstractRNG` object. This avoid the reliance on the global RNG. I decided to use a keyword argument instead of using the first argument as we do for the `rand` methods. I think it's easier to use a keyword argument for functions that take many argument. Please comment if you disagree.

codecov-commenter: # [Codecov](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats) Report
> Merging [#252](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats) (9f8cff2) into [master](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/commit/992fe205c9dcff24869d4b24cf6df847a7983329?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats) (992fe20) will **increase** coverage by `0.23%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/graphs/tree.svg?width=650&height=150&src=pr&token=ztSoTXYVhb&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats)](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats)

```diff
@@            Coverage Diff             @@
##           master     #252      +/-   ##
==========================================
+ Coverage   92.93%   93.16%   +0.23%     
==========================================
  Files          28       28              
  Lines        1641     1712      +71     
==========================================
+ Hits         1525     1595      +70     
- Misses        116      117       +1     
```


| [Impacted Files](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats) | Coverage Δ | |
|---|---|---|
| [src/permutation.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL3Blcm11dGF0aW9uLmps) | `79.31% <100.00%> (+0.73%)` | :arrow_up: |
| [src/HypothesisTests.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL0h5cG90aGVzaXNUZXN0cy5qbA==) | `80.30% <0.00%> (-1.24%)` | :arrow_down: |
| [src/common.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2NvbW1vbi5qbA==) | `100.00% <0.00%> (ø)` | |
| [src/correlation.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2NvcnJlbGF0aW9uLmps) | `100.00% <0.00%> (ø)` | |
| [src/jarque\_bera.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2phcnF1ZV9iZXJhLmps) | `100.00% <0.00%> (ø)` | |
| [src/kruskal\_wallis.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2tydXNrYWxfd2FsbGlzLmps) | `100.00% <0.00%> (ø)` | |
| [src/wald\_wolfowitz.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL3dhbGRfd29sZm93aXR6Lmps) | `100.00% <0.00%> (ø)` | |
| [src/breusch\_godfrey.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2JyZXVzY2hfZ29kZnJleS5qbA==) | `100.00% <0.00%> (ø)` | |
| [src/kolmogorov\_smirnov.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2tvbG1vZ29yb3Zfc21pcm5vdi5qbA==) | `95.77% <0.00%> (+0.06%)` | :arrow_up: |
| [src/durbin\_watson.jl](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL2R1cmJpbl93YXRzb24uamw=) | `98.91% <0.00%> (+0.07%)` | :arrow_up: |
| ... and [10 more](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats). Last update [992fe20...9f8cff2](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/pull/252?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats).

andreasnoack: I've changed the argument from a keyword to a positional one. We can consider adding the keyword argument in a separate PR if needed. I added a test for it and it revealed that we'd have to bump the minimum Julia requirement to 1.3 since `Random.default_rng` wasn't introduced with that version. With a new long term support release out, I don't think it's worth adding version conditional definitions.
pdeffebach: Is there any way I can help to get this PR merged?

Patch:
diff --git a/.github/workflows/TagBot.yml b/.github/workflows/TagBot.yml
index d77d3a0..99a04f5 100644
--- a/.github/workflows/TagBot.yml
+++ b/.github/workflows/TagBot.yml
@@ -9,3 +9,4 @@ jobs:
       - uses: JuliaRegistries/TagBot@v1
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
+          token: ${{ secrets.DOCUMENTER_KEY }}
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5f36b02..a2fceef 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -12,7 +12,7 @@ jobs:
       fail-fast: false
       matrix:
         version:
-          - '1.0'
+          - '1.3'
           - '1' # automatically expands to the latest stable 1.x release of Julia
           - 'nightly'
         os:
diff --git a/Project.toml b/Project.toml
index 3507634..3f6c32f 100644
--- a/Project.toml
+++ b/Project.toml
@@ -18,7 +18,7 @@ Distributions = "0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25"
 Rmath = "0.5, 0.6, 0.7"
 Roots = "0.7, 0.8, 1.0"
 StatsBase = "0.27, 0.28, 0.29, 0.30, 0.31, 0.32, 0.33"
-julia = "1"
+julia = "1.3"
 
 [extras]
 DelimitedFiles = "8bb1440f-4735-579b-a4ab-409b98df4dab"
diff --git a/README.md b/README.md
index ce86879..352af1c 100644
--- a/README.md
+++ b/README.md
@@ -3,14 +3,13 @@
 *HypothesisTests.jl* is a Julia package that implements a wide range of hypothesis tests.
 
 - **Build & Testing Status:**
-  [![Build status](https://github.com/JuliaStats/HypothesisTests.jl/workflows/CI/badge.svg)](https://github.com/JuliaStats/HypothesisTests.jl/actions?query=workflow%3ACI+branch%3Amaster)
-  [![Coverage Status](https://coveralls.io/repos/JuliaStats/HypothesisTests.jl/badge.svg?branch=master)
-  ](https://coveralls.io/r/JuliaStats/HypothesisTests.jl?branch=master)
+[![Build Status](https://github.com/JuliaStats/HypothesisTests.jl/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/JuliaStats/HypothesisTests.jl/actions/workflows/ci.yml?query=branch%3Amaster)
+[![Coverage Status](https://codecov.io/gh/JuliaStats/HypothesisTests.jl/branch/master/graph/badge.svg?token=ztSoTXYVhb)](https://codecov.io/gh/JuliaStats/HypothesisTests.jl)
 
 - **Documentation**: [![][docs-stable-img]][docs-stable-url] [![][docs-latest-img]][docs-latest-url]
 
-[docs-latest-img]: https://img.shields.io/badge/docs-latest-blue.svg
-[docs-latest-url]: http://JuliaStats.github.io/HypothesisTests.jl/latest/
+[docs-latest-img]: https://img.shields.io/badge/docs-dev-blue.svg
+[docs-latest-url]: http://JuliaStats.github.io/HypothesisTests.jl/dev/
 
 [docs-stable-img]: https://img.shields.io/badge/docs-stable-blue.svg
 [docs-stable-url]: http://JuliaStats.github.io/HypothesisTests.jl/stable/
diff --git a/docs/Project.toml b/docs/Project.toml
index d7906fc..3a52a5d 100644
--- a/docs/Project.toml
+++ b/docs/Project.toml
@@ -2,4 +2,4 @@
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 
 [compat]
-Documenter = "0.23"
+Documenter = "0.27"
diff --git a/src/permutation.jl b/src/permutation.jl
index 6e9e034..918dfb7 100644
--- a/src/permutation.jl
+++ b/src/permutation.jl
@@ -27,18 +27,22 @@ function ExactPermutationTest(x::AbstractVector{R}, y::AbstractVector{S},
 end
 
 """
-    ApproximatePermutationTest(x::Vector, y::Vector, f::Function, n::Int)
+    ApproximatePermutationTest([rng::AbstractRNG,] x::Vector, y::Vector, f::Function, n::Int)
 
 Perform a permutation test (a.k.a. randomization test) of the null hypothesis
 that `f(x)` is equal to `f(y)`.  `n` of the `factorial(length(x)+length(y))`
-permutations are sampled at random.
+permutations are sampled at random. A random number generator can optionally
+be passed as the first argument. The default generator is `Random.default_rng()`.
 """
-function ApproximatePermutationTest(x::AbstractVector{R}, y::AbstractVector{S},
+function ApproximatePermutationTest(rng::AbstractRNG, x::AbstractVector{R}, y::AbstractVector{S},
                                     f::Function, n::Int) where {R<:Real,S<:Real}
     xy, rx, ry = ptstats(x,y)
-    samples = [(shuffle!(xy); f(view(xy,rx)) - f(view(xy,ry))) for i = 1:n]
+    samples = [(shuffle!(rng, xy); f(view(xy,rx)) - f(view(xy,ry))) for i = 1:n]
     PermutationTest(f(x) - f(y), samples)
 end
+ApproximatePermutationTest(x::AbstractVector{R}, y::AbstractVector{S},
+                           f::Function, n::Int) where {R<:Real,S<:Real} =
+    ApproximatePermutationTest(Random.default_rng(), x, y, f, n)
 
 function pvalue(apt::PermutationTest; tail=:both)
     if tail == :both


Test patch:
diff --git a/test/anderson_darling.jl b/test/anderson_darling.jl
index 719282f..66ae095 100644
--- a/test/anderson_darling.jl
+++ b/test/anderson_darling.jl
@@ -1,57 +1,57 @@
 using HypothesisTests, Distributions, Test, Random
 using HypothesisTests: default_tail
+using StableRNGs
 
 @testset "Anderson-Darling" begin
 @testset "One sample test" begin
     n = 1000
-    Random.seed!(1984948)
+    rng = StableRNG(1984948)
 
     d = Normal(1000, 100)
-    x = rand(d, n)
+    x = rand(rng, d, n)
     t = OneSampleADTest(x, d)
-    @test t.A² ≈ 0.3078 atol=0.1^4
-    @test pvalue(t) ≈ 0.9321 atol=0.1^4
+    @test t.A² ≈ 1.2960 atol=0.1^4
+    @test pvalue(t) ≈ 0.2336 atol=0.1^4
     @test default_tail(t) == :right
 
     d = DoubleExponential()
-    x = rand(d, n)
+    x = rand(rng, d, n)
     t = OneSampleADTest(x, Normal(mean(d), std(d)))
-    @test t.A² ≈ 11.3969 atol=0.1^4
+    @test t.A² ≈ 11.0704 atol=0.1^4
     @test pvalue(t) ≈ 0.0 atol=0.1^4
     t = OneSampleADTest(x, d)
-    @test t.A² ≈ 0.6036 atol=0.1^4
-    @test pvalue(t) ≈ 0.6446 atol=0.1^4
+    @test t.A² ≈ 0.8968 atol=0.1^4
+    @test pvalue(t) ≈ 0.4162 atol=0.1^4
 
     d = Cauchy()
-    x = rand(Cauchy(), n)
+    x = rand(rng, Cauchy(), n)
     t = OneSampleADTest(x, Normal())
     @test pvalue(t) ≈ 0.0 atol=0.1^4
     t = OneSampleADTest(x, d)
-    @test pvalue(t) ≈ 0.6773 atol=0.1^4
+    @test pvalue(t) ≈ 0.9640 atol=0.1^4
 
     d = LogNormal()
-    x = rand(d, n)
+    x = rand(rng, d, n)
     t = OneSampleADTest(x, Normal(mean(d), std(d)))
     @test pvalue(t) ≈ 0.0 atol=0.1^4
     t = OneSampleADTest(x, d)
-    @test pvalue(t) ≈ 0.1055 atol=0.1^4
+    @test pvalue(t) ≈ 0.9123 atol=0.1^4
 
     d = Uniform(-pi, 2pi)
-    x = rand(d, n)
+    x = rand(rng, d, n)
     t = OneSampleADTest(x, d)
-    @test pvalue(t) ≈ 0.5614 atol=0.1^4
+    @test pvalue(t) ≈ 0.2337 atol=0.1^4
 
-    x = rand(Uniform(0, 1.8), n)
+    x = rand(rng, Uniform(0, 1.8), n)
     t = OneSampleADTest(x, Uniform())
     @test pvalue(t) ≈ 0.0 atol=0.1^4
 
-    x = rand(Exponential(), n)
+    x = rand(rng, Exponential(), n)
     t = OneSampleADTest(x, Exponential())
-    @test pvalue(t) ≈ 0.1086 atol=0.1^4
+    @test pvalue(t) ≈ 0.8579 atol=0.1^4
 end
 
 @testset "k-sample test" begin
-    Random.seed!(948574875)
     samples = Any[
         [38.7, 41.5, 43.8, 44.5, 45.5, 46.0, 47.7, 58.0],
         [39.2, 39.3, 39.7, 41.4, 41.8, 42.9, 43.3, 45.8],
@@ -78,20 +78,20 @@ end
 end
 
 @testset "more tests" begin
-    Random.seed!(31412455)
-    samples = Any[rand(Normal(), 50), rand(Normal(0.5), 30)]
+    rng =  StableRNG(31412455)
+    samples = Any[rand(rng, Normal(), 50), rand(rng, Normal(0.5), 30)]
     t = KSampleADTest(samples...)
     @test pvalue(t) < 0.05
 
-    samples = Any[rand(Normal(), 50), rand(Normal(), 30), rand(Normal(), 20)]
+    samples = Any[rand(rng, Normal(), 50), rand(rng, Normal(), 30), rand(rng, Normal(), 20)]
     t = KSampleADTest(samples...)
     @test pvalue(t) > 0.05
 
-    @test pvalue(OneSampleADTest(vcat(rand(Normal(),500), rand(Beta(2,2),500)), Beta(2,2))) ≈ 0.0 atol=0.1^4
+    @test pvalue(OneSampleADTest(vcat(rand(rng, Normal(),500), rand(rng, Beta(2,2),500)), Beta(2,2))) ≈ 0.0 atol=0.1^4
 
     n = 1000
-    x = rand(Exponential(), n)
-    @test pvalue(KSampleADTest(rand(1000),randn(1000))) ≈ 0.0 atol=eps()
+    x = rand(rng, Exponential(), n)
+    @test pvalue(KSampleADTest(rand(rng, 1000), randn(rng, 1000))) ≈ 0.0 atol=eps()
     @test pvalue(KSampleADTest(x,x,x,x,x,x)) ≈ 1.0
 end
 
diff --git a/test/permutation.jl b/test/permutation.jl
index fbc503e..db6411f 100644
--- a/test/permutation.jl
+++ b/test/permutation.jl
@@ -1,14 +1,20 @@
-using HypothesisTests, Test
+using HypothesisTests, Test, Statistics, StableRNGs
 
 @testset "Permutation" begin
-@test isapprox(pvalue(ExactPermutationTest([1,2,3],[4,5,6],mean), tail=:both), 0.1)
-@test isapprox(pvalue(ExactPermutationTest([4,5,6],[1,2,3],mean), tail=:both), 0.1)
-@test isapprox(pvalue(ExactPermutationTest([1,2,3],[4,5,6],mean), tail=:left), 0.05)
-@test isapprox(pvalue(ExactPermutationTest([4,5,6],[1,2,3],mean), tail=:right), 0.05)
+    @testset "ExactPermutationTest" begin
+        @test pvalue(ExactPermutationTest([1,2,3], [4,5,6], mean), tail=:both) ≈ 0.1
+        @test pvalue(ExactPermutationTest([4,5,6], [1,2,3], mean), tail=:both) ≈ 0.1
+        @test pvalue(ExactPermutationTest([1,2,3], [4,5,6], mean), tail=:left) ≈ 0.05
+        @test pvalue(ExactPermutationTest([4,5,6], [1,2,3], mean), tail=:right) ≈ 0.05
+    end
 
-Random.seed!(12345)
-@test isapprox(pvalue(ApproximatePermutationTest([1,2,3],[4,5,6],mean,100), tail=:both), 0.09)
-@test isapprox(pvalue(ApproximatePermutationTest([4,5,6],[1,2,3],mean,100), tail=:both), 0.04)
-@test isapprox(pvalue(ApproximatePermutationTest([1,2,3],[4,5,6],mean,100), tail=:left), 0.08)
-@test isapprox(pvalue(ApproximatePermutationTest([4,5,6],[1,2,3],mean,100), tail=:right), 0.07)
+    @testset "ApproximatePermutationTest" begin
+        rng = StableRNG(12345)
+        @test pvalue(ApproximatePermutationTest(rng, [1,2,3], [4,5,6], mean, 100), tail=:both) ≈ 0.14
+        @test pvalue(ApproximatePermutationTest(rng, [4,5,6], [1,2,3], mean, 100), tail=:both) ≈ 0.08
+        @test pvalue(ApproximatePermutationTest(rng, [1,2,3], [4,5,6], mean, 100), tail=:left) ≈ 0.05
+        @test pvalue(ApproximatePermutationTest(rng, [4,5,6], [1,2,3], mean, 100), tail=:right) ≈ 0.05
+        # Test that the non-rng method works
+        @test ApproximatePermutationTest([4,5,6], [1,2,3], mean, 100) isa HypothesisTests.PermutationTest{Float64}
+    end
 end