Repository: statsmodels/statsmodels

Issue:
BUG: Fix two bugs in `statsmodels.stats.power.normal_sample_size_one_tail`
- ~~[ ] closes #xxxx~~ (does not apply)
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

This pull request fixes two bugs in `statsmodels.stats.power.normal_sample_size_one_tail`:

1. If the default value of that method's `std_alternative` parameter is used, that method raises an exception because a `float` cannot be multiplied by a `NoneType`. This bug is mitigated by setting the `std_alternative` parameter to zero in the body of that method if its value is `None`.
2. The `power` parameter cannot be less than `alpha`. Admittedly, I don't know why someone would intentionally try to pass arguments satisfying that criterion, but nevertheless, a power less than alpha would be ill-posed. This bug is mitigated by adding a conditional checking this condition; if satisfied, a `ValueError` is raised. The method's documentation has also been updated.

Reviewing the code in `statsmodels.stats.power.normal_sample_size_one_tail` also suggests the implementation might be incorrect: after the mitigations above, assuming `std_alternative` is `None`, and all parameters other than `power` are fixed, the computed sample size will be the same for all valid values of `power`. One possible alternative implementation (omitting docstrings) could be something like:

```python
def normal_sample_size_one_tail(diff, power, alpha, std_null=1.,
                                 std_alternative=None):

    if power < alpha:
        raise ValueError(f"Power cannot be less than alpha; set power >= {alpha} to fix")

    if std_alternative is None:
        std_alternative = 0

    crit_power = stats.norm.isf(power)
    crit = stats.norm.isf(alpha)
    n1 = (std_null ** 2 + std_alternative ** 2) * (crit - crit_power)**2 / (diff ** 2)
    return n1
```
under the assumption that, in the two-sample case, the sample sizes are equal. This expression agrees with a similar expression for the two-sample case on page 490 in Chapter 10 of J. L. Devore, K. N. Berk, *Modern Mathematical Statistics with Applications*, 2nd Edition, in *Springer Texts on Statistics*, and when `std_alternative=0`, it agrees with the expression for the one-sample case on page 441 in Chapter 9 of the same textbook, as well as that on [Wikipedia's page for "sample size determination"](https://en.wikipedia.org/wiki/Sample_size_determination#Cumulative_distribution_function). For the one-tailed, one-sample case, this expression should be exact; for the two-tailed case, it makes the approximation of ignoring the smaller tail's contribution to Type II error. However, changing the implementation would affect some existing tests in `statsmodels/stats/tests/test_proportion.py`, so I have not changed the expression for `n1` in this pull request. If making the change above to the expression for  `n1` is sensible, I would be happy to contribute that change as well.

<details>
**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>



Discussion:
Thanks, I will go through it

The function was not really written as a standalone function.
AFAICS, I wrote it mainly for use with proportion ztest

the intention or usecase for std_alternative is to allow std to differ between alternative and null.
E.g. in standard cases without inherent heteroscedasticity, like OLS/normal, the std would be the same under alternative and null

```
if std_alternative is None:
        std_alternative = std_null
```

second

`if power < alpha:` will break vectorization, which IIRC is allowed here.

There is only one unit test for the function directly and that is in test_proportions

Other unit tests are for samplesize_proportions_2indep_onetail which delegates to normal_sample_size_one_tail
> `if power < alpha:` will break vectorization, which IIRC is allowed here.

If you prefer, I could mask the vector and return `nan` for vector elements in which `power < alpha`. This approach is used in, e.g., `scipy.stats.rv_continuous` to indicate invalid input arguments for methods like `scipy.stats.norm.isf` and would not break vectorization.
> If you prefer, I could mask the vector and return nan for vector elements in which power < alpha

Yes, but it might be better to do it on the return, because any of the arguments could be an array with broadcasting, but there is only one return scalar or array.
> The function was not really written as a standalone function.
> AFAICS, I wrote it mainly for use with proportion ztest

I believe that's the only case under which this function would be valid. If the idea is to compute a sample size for a proportion z-test, then the current interface seems nonintuitive because standard deviations of proportions modeled as binomial distributions depend on their mean values, so the parameters `diff`, `std_null`, and `std_alternative` cannot vary independently of each other. All three of these quantities only depend on two parameters: the proportion used to formulate the alternative hypothesis (e.g., the `p_alt` in `p < p_alt`) and whatever proportion is used for a power calculation:

```python
def normal_sample_size_one_tail_proportion(p_alt, p_power, alpha, power):

    # p_power = proportion used for power calculation
    crit_power = stats.norm.isf(power)
    crit = stats.norm.isf(alpha)
    diff = p_power - p_alt
    std_alt = numpy.sqrt(p_alt * (1 - p_alt))
    std_power = numpy.sqrt(p_power * (1 - p_power))
    n1 = (crit * std_alt - crit_power * std_power) ** 2 / (diff ** 2)
    return n1
```

> the intention or usecase for std_alternative is to allow std to differ between alternative and null.
> E.g. in standard cases without inherent heteroscedasticity, like OLS/normal, the std would be the same under alternative and null

As far as I can tell from the current formula, even if that were the case, it would under-count variance in an unpaired, nonproportion z-test by a factor of 2, and that choice would also restrict the sample size calculation to two-sample tests.
@josef-pkt I've removed the exception-raising conditional in commit [`07342637`](https://github.com/statsmodels/statsmodels/pull/8507/commits/07342637751594f7519fd64826680792afc263e0) and implemented a vectorizable mask that returns `numpy.nan` for elements violating the `power >= alpha` condition in commit [`67765590`](https://github.com/statsmodels/statsmodels/pull/8507/commits/67765590475b050210a8050e109f3798693374f5). Commit [`2af04209`](https://github.com/statsmodels/statsmodels/pull/8507/commits/2af04209a217cfd1ac0c24f7b6a6fbea6bd4ca44) adds a unit test to prevent regression of the "multiplying `float` by `None`" bug. Should be ready for another review.
I'm not remembering the details well enough

should diff be replaced by np.abs(diff) ?

(thinking more)
abs not needed: diff shows up only as `diff**2` (if `(crit * std_null - crit_power * std_alternative)` is positive)

what's "appropriate" tail?

> This uses an explicit computation for the sample size that is required
    to achieve a given power corresponding to the appropriate tails of the
    normal distribution.

I guess there should be a sign restriction on `(crit * std_null - crit_power * std_alternative)`

This function needs more unit tests, e.g. against `normal_power_het`
The function should also apply to z-tests and arbitrary hypothesis tests based on the normal distribution

I wrote normal_sample_size_one_tail as a shortcut function that can be reused for different hypothesis tests, similar to the other functions like normal_power_het (which I use also for Poisson)

`diff`, `std_null`, and `std_alternative` are parameters of the function, so they can be based on any type of hypothesis test with normal distributed test statistic.
The calling function is responsible to provide the appropriate values for those arguments, e.g. as samplesize_proportions_2indep_onetail does for the comparison of 2 independent Binomial proportions.
(I ran out of patience to add it to the recently added poisson rate functions)

For example normal_sample_size_one_tail should work for z-test for mean difference of 2 independent samples, with appropriate std_null.

Aside: When I was looking at the power functions just now, I got confused again how effect size is defined in my functions. At the time I wrote the power classes, I was following mainly R pwr and G*Power+ packages, without understanding the details different definitions of effect sizes. 

Is there a reference for the expression you're using, and the assumptions under which it is valid?
> This function needs more unit tests, e.g. against `normal_power_het`

I don't disagree, but in terms of the changes in this pull request, unit tests have been added to test the changes that have been made to the function. Adding more unit tests of the existing functionality seems better scoped to a separate pull request.
> Is there a reference for the expression you're using, and the assumptions under which it is valid?

It's a generic formula, that shows up in articles that I read mainly in the context of proportions, or cases where std_alternative is different from std_null, Binomial proportion, Poisson rates, and similar where variance depends on the mean or mean parameter.
Cases where the variance is independent of the mean, like z-test or t-test for (normal, ...) means, std_alternative is usually equal to std_null.

I agree that more unit tests and other use cases can wait for a different PR.

One part still to check is whether checking `alpha < power` is enough or if we need `(crit * std_null - crit_power * std_alternative) > 0` when std_null and std_alternative differ

note 
sigma is in the denominator in https://en.wikipedia.org/wiki/Sample_size_determination#Cumulative_distribution_function
That's the same as multiplying the expression by sigma (= std_null = std_alternative) instead
one unrelated fail in quantreg unit tests (in only one environment)
AFAICS no random seed is set
```
  def test_nontrivial_singular_matrix():
        x_one = np.random.random(1000)
...
E       AssertionError: 
E       Not equal to tolerance rtol=0.01, atol=0
```
condition is still unclear

currently if alpha == power (and std_alt=std_null), then nobs = 0, i.e. condition holds for any sample size

What should be the returned sample size if power requirements always hold, np.nan or zero?
nobs=zero would mean, power requirement is always satisfied.
however if nobs=0, then std of test statistic is inf, and we compute inf - inf, which would be nan.
another alternative would be to truncate/clip at nobs=1, i.e. nobs=1 means power requirement is satisfied for any nonzero sample size.
 

unequal std:
current samplesize_proportions_2indep_onetail has always std_null > std_alternative, AFAICS. Currently only value=0 is allowed so props under alternative have larger diff than under the null.
I guess this might not be true if `value != 0`. e.g. noninferiority testing (or superiority testing with a margin ?)

So with unequal std, we can still have positive term/nobs even if alpha=power, AFAICS.
(I didn't try the math yet for the derivation of the sample size formula to see where nonnegativity constraint kicks in.)

The simplest implementation would be using np.maximum(term, 0) where term is that part that needs to be nonnegative.
That would return nobs = 0 as lower bound, and prevent the bug with incorrect square term
Attached are screenshots of the textbook pages I have been citing. Worth noting are the differences in sample size expressions between one-population and two-population tests, as well as the differences between sample size expressions for proportion data versus non-proportion data.

![CF4536CC-55E4-46F4-8570-A990D11076C7](https://user-images.githubusercontent.com/321479/201421973-36afbbe5-ff8b-4844-b244-5038bfd0cb49.png)
![C200C6E2-0CA3-476E-91E0-05E22CB46B5E](https://user-images.githubusercontent.com/321479/201421981-c030e1fd-7b59-4eac-91a5-6b477f76241f.png)
![CDD42579-E6F1-401F-99C9-DBA9AC0EB353](https://user-images.githubusercontent.com/321479/201421984-119205ba-f960-4df5-bbc0-4b64961a83fe.png)
![417B57D7-AB84-4F82-BF10-2B065F54D65F](https://user-images.githubusercontent.com/321479/201421986-b1e802fd-0c7d-4dc2-b982-26e21bf13398.png)
![809BFA04-C74A-4E73-B726-80C9A7ED9923](https://user-images.githubusercontent.com/321479/201421989-89f32857-bfb1-4542-b9bb-380871bdb9ce.png)

The reason `power` cannot be less than `alpha` is because if it were, then the null hypothesis and the alternative hypothesis would have nonempty intersection -- the specific alternative being considered for the false negative condition would not violate the null hypothesis.
> What should be the returned sample size if power requirements always hold, np.nan or zero? nobs=zero would mean, power requirement is always satisfied.

It would mean that the effect size you're using for the power computation doesn't make sense, which is why I chose `np.nan`. In IEEE-754 arithmetic, NaNs propagate through non-Boolean operations on floating-point numbers, which gives users a signal that something aberrant has occurred in their calculations.

All Boolean comparisons between a floating-point number and NaN are false, including `numpy.nan == numpy.nan`, which is why `math.isnan` and `numpy.isnan` exist. Consequently, if an illegal argument has been given to, say, `scipy.stats.norm.isf`, that will return `numpy.nan` when `n1` is computed. The `numpy.where` statement used in the pull request will preserve any `numpy.nan` elements in `n1` that existed upon entry to that line, and can only overwrite elements in `n1` with `numpy.nan` values when the corresponding element of `alpha_minus_power` is positive. If any element of `alpha` or `power` is `math.nan`, then corresponding elements of the Boolean NumPy array `alpha_minus_power` will evaluate to `False`, so those elements of `n1` are not overwritten with `numpy.nan`s in the `numpy.where` statement. But those elements of `n1` should already be `numpy.nan` upon entry to that statement anyway, so those elements of `n1` remain `numpy.nan` upon exiting the function.
Returning a sample size of zero would have to be documented, and that choice would force users to test for that condition, violating Principle of Least Surprise. It would also be inconsistent with what packages like NumPy and SciPy do in many situations (which is return NaN).
> The reason `power `cannot be less than `alpha` is because if it were, then the null hypothesis and the alternative hypothesis would have nonempty intersection -- the specific alternative being considered for the false negative condition would not violate the null hypothesis.

A more precise way to state that condition would be that the effect size used in the power computation would be in the complement of the critical region of the test. I could see returning `numpy.nan` if `power == alpha` because returning zero would be nonsensical; I hadn't given much thought as to whether the critical region should be an open set or a closed set in one dimension.

I ran a quick test using the `pwr` library in R, and setting `power` too close to `alpha` causes a runtime error:

```r
library(pwr)
pwr.t.test(d=0.50, sig.level=0.05, power=0.06, type="one.sample", alternative="two.sided")
```

Result:

```console
Error in uniroot(function(n) eval(p.body) - power, c(2 + 1e-10, 1e+09)) : 
  f() values at end points not of opposite sign
Calls: pwr.t.test -> uniroot
Execution halted
```

which I'd guess is due to numerical integration issues wreaking havoc on Brent's method.

Thanks for the reference and copies

The main reason that I have a small preference for returning nobs=0 instead of nobs=nan, is that the power requirement is satisfied (with inequality, power is weakly large than required).
AFAIR, we also have cases in equivalence testing where no solution (of rootfinding) exists, I don't remember the details, but there is an explicit warning for that case.

Also, if we use the one tail approximation for starting values of full rootfinding, then 0 is a valid, finite value, while nans would have to be captured separately.

slightly strange
nobs = 0 does not cause a warning or error in the power computation for proportions_2indep
```
pres = smprop.power_proportions_2indep(diff, prop2, 0, ratio=1, alpha=alphas,
                                value=0, alternative='two-sided',
                                return_results=True)

pres.power, alphas
(0.17699994280370485, 0.2)
```

The results is close nobs=1e-13, i.e. continuous extension to zero.
I guess it's a numerical side effect of how the the power computation is implemented.
Also the lower bound on power (independent of sample size) is smaller than alpha. 

We will have the document the corner behavior in either case whether it's zero or nan.

Another one,
if we return nobs = 0, then users can np.clip nobs to lower bound one, but we cannot clip nan.

using maximum with 0, for proportion_2indep
```
powers = np.linspace(1e-4, 0.5, 100)
n = smprop.samplesize_proportions_2indep_onetail(diff, prop2, powers, ratio=1,
                                                 alpha=0.2, value=0)
```

![image](https://user-images.githubusercontent.com/440735/201485894-5b9f43a0-c4e7-4496-b10f-a6cfa9329d20.png)


**edit**
actually it's not a function of nobs and power, as function there would be a minimum power level (on x-axis)
it's minimum required power, with corrected labelling:
![image](https://user-images.githubusercontent.com/440735/201486128-de894633-3e8f-48d9-be2b-310051e6ebbb.png)

about "alternative" one versus 2 sided  (I briefly got confused about how this works)

this is handled in `samplesize_proportions_2indep_onetail` which has alternative keyword
```
    if alternative in ['two-sided', '2s']:
        alpha = alpha / 2
```

`power.normal_sample_size_one_tail` was written as helper function and does not have "alternative" keyword
docstring description says
'Use alpha/2 to compute the one tail approximation to the two-sided
    test, i.e. consider only one tail of two-sided test.'

but it's not mentioned again in the alpha parameter explanation.
alpha has the generic, copy-pasted explanation.
> We will have the document the corner behavior in either case whether it's zero or nan.

Agreed. I'll change it to zero. Could possibly be worth a keyword option to toggle behavior re: zero vs nan (or even raising an exception).
@josef-pkt Changed implementation to return `nobs = 0` when `power <= alpha`, and updated the function's docstring to document this behavior in commit [`2de4c9be`](https://github.com/statsmodels/statsmodels/pull/8507/commits/2de4c9be7ce849a47472f18bb436fe491dc20e35).
As far as I can tell, the reason `power` must be at least `alpha` is a consequence of the Neyman-Pearson Lemma regarding uniformly most powerful hypothesis tests for various situations (which includes, under certain assumptions, z-tests). Below is the relevant section of G. G. Roussas, *A Course in Mathematical Statistics*, 2nd Edition, Academic Press, 1997 (pages 329-333):
![8522308F-A760-45BB-B035-391CAD0476E6](https://user-images.githubusercontent.com/321479/202263483-dc07972f-ff70-4698-8e83-29639a6199b8.jpeg)
![8A91AC05-3583-4EAB-8AF0-3613081BB00E](https://user-images.githubusercontent.com/321479/202263490-1b47358d-166f-4881-9e46-d4b27f12ddec.jpeg)
![8FE45883-1957-4CA8-A0A8-052072AFC5A0](https://user-images.githubusercontent.com/321479/202263496-b07817fe-0894-43d1-8738-8ec8bfb7fcce.jpeg)
![45448E79-B9ED-4EE4-A9A0-AD601E76EA69](https://user-images.githubusercontent.com/321479/202263501-fc25e06f-9938-4f54-a0e8-0be7509a8dd2.jpeg)
![B43B24F6-531E-4C10-A2F8-353FA03818AE](https://user-images.githubusercontent.com/321479/202263499-d3fdf18f-1c0a-4102-a063-91a0992bc668.jpeg)


An example power function plot demonstrating that its value cannot be less than alpha is shown in the excerpt from J. Gillard, *A First Course in Statistical Inference*, Springer, 2010 (pages 77-79):

![818AB219-A340-47CB-B4DC-27C61B773FA7](https://user-images.githubusercontent.com/321479/202264135-9a6feac9-a96a-4256-a362-78cdd344392a.jpeg)
![F9B6EE54-019C-4B9A-AC1E-A72F9EA2034B](https://user-images.githubusercontent.com/321479/202264149-58a2aa68-648f-48cf-a2e7-3795f0ec16be.jpeg)
![0F4DF529-4ADC-4A02-97D6-76ED1DC0A33A](https://user-images.githubusercontent.com/321479/202264152-d8c886b6-e266-4433-b908-44b09bcbcbec.jpeg)

Where are just using properties of an existing test, and compute type 1 and type 2 (power) and their relationship to nobs.
Most of our hypothesis tests are not UMP, and I never really looked at Neyman Pearson lemma in detail not used it.

The bug, or one of the bugs, you pointed out at the beginning is that we use a quadrating function where we should only use the positive arm. In the current code we have an incorrect u-shaped function of nobs as function of required power where the initial downward sloping is wrong. That shows a positive nobs, even though it should be nan or zero.

The condition alpha < power is only the lowest point of that quadratic function if std are equal. If std_alt and std_null differ, then the positive arm (nobs > 0) does not start at alpha = power. We have to take into account that those std can differ to find nobs >0 or rule out that we take square of a negative number.

That was all I was trying to explain.

As properties of a specific test, we don't impose alpha < power unless needed for some computation.
The current power functions can be evaluated at points where power < alpha, This happens if we are on the wrong side of a tail the null, It doesn't make much sense to use a hypothesis test like that, but we can compute the type1 and type2 error or power, i.e. rejection probability,  for any point in the parameter space.
for example,
We test that diff =0 against diff > 0
However, we can compute the rejection rate if the true diff is < 0. In that case power < alpha, if rejection rate if diff =0 is true is alpha. 
(It works out in the computation, but we don't really have a reason to do that in practice if we are interested in the "larger" alternative.  I'm using the edge case alpha=power in unit tests quite often.)

> we should only use the positive arm. In the current code we have an incorrect u-shaped function of nobs as function of required power where the initial downward sloping is wrong. That shows a positive nobs, even though it should be nan or zero.

I think I understand that, and will make the suggested change. Thank you for your patience!
Change made re: `np.maximum` in [`16b05325`](https://github.com/statsmodels/statsmodels/pull/8507/commits/16b0532555c60bb280f3554569c3ce270397d995).

Resolving patch:
diff --git a/statsmodels/stats/power.py b/statsmodels/stats/power.py
--- a/statsmodels/stats/power.py
+++ b/statsmodels/stats/power.py
@@ -168,7 +168,7 @@ def normal_power_het(diff, nobs, alpha, std_null=1., std_alternative=None,
 
 
 def normal_sample_size_one_tail(diff, power, alpha, std_null=1.,
-                                 std_alternative=None):
+                                std_alternative=None):
     """explicit sample size computation if only one tail is relevant
 
     The sample size is based on the power in one tail assuming that the
@@ -182,29 +182,39 @@ def normal_sample_size_one_tail(diff, power, alpha, std_null=1.,
     diff : float
         difference in the estimated means or statistics under the alternative.
     power : float in interval (0,1)
-        power of the test, e.g. 0.8, is one minus the probability of a type II error.
-        Power is the probability that the test correctly rejects the Null
-        Hypothesis if the Alternative Hypothesis is true.
+        power of the test, e.g. 0.8, is one minus the probability of a type II
+        error. Power is the probability that the test correctly rejects the
+        Null Hypothesis if the Alternative Hypothesis is true.
     alpha : float in interval (0,1)
         significance level, e.g. 0.05, is the probability of a type I
         error, that is wrong rejections if the Null Hypothesis is true.
+        Note: alpha is used for one tail. Use alpha/2 for two-sided
+        alternative.
     std_null : float
         standard deviation under the Null hypothesis without division by
         sqrt(nobs)
     std_alternative : float
         standard deviation under the Alternative hypothesis without division
-        by sqrt(nobs)
+        by sqrt(nobs). Defaults to None. If None, ``std_alternative`` is set
+        to the value of ``std_null``.
 
     Returns
     -------
     nobs : float
-        sample size to achieve the desired power
+        Sample size to achieve (at least) the desired power.
+        If the minimum power is satisfied for all positive sample sizes, then
+        ``nobs`` will be zero. This will be the case when power <= alpha if
+        std_alternative is equal to std_null.
 
     """
 
+    if std_alternative is None:
+        std_alternative = std_null
+
     crit_power = stats.norm.isf(power)
     crit = stats.norm.isf(alpha)
-    n1 = ((crit * std_null - crit_power * std_alternative) / diff)**2
+    n1 = (np.maximum(crit * std_null - crit_power * std_alternative, 0)
+          / diff)**2
     return n1
 
 


Test patch:
diff --git a/statsmodels/stats/tests/test_power.py b/statsmodels/stats/tests/test_power.py
--- a/statsmodels/stats/tests/test_power.py
+++ b/statsmodels/stats/tests/test_power.py
@@ -14,7 +14,7 @@
 
 import numpy as np
 from numpy.testing import (assert_almost_equal, assert_allclose, assert_raises,
-                           assert_equal, assert_warns)
+                           assert_equal, assert_warns, assert_array_equal)
 import pytest
 
 import statsmodels.stats.power as smp
@@ -772,3 +772,22 @@ def test_power_solver_warn():
                               alternative='larger')
         assert_equal(nip.cache_fit_res[0], 0)
         assert_equal(len(nip.cache_fit_res), 3)
+
+
+def test_normal_sample_size_one_tail():
+    # Test that using default value of std_alternative does not raise an
+    # exception. A power of 0.8 and alpha of 0.05 were chosen to reflect
+    # commonly used values in hypothesis testing. Difference in means and
+    # standard deviation of null population were chosen somewhat arbitrarily --
+    # there's nothing special about those values. Return value doesn't matter
+    # for this "test", so long as an exception is not raised.
+    smp.normal_sample_size_one_tail(5, 0.8, 0.05, 2, std_alternative=None)
+
+    # Test that zero is returned in the correct elements if power is less
+    # than alpha.
+    alphas = np.asarray([0.01, 0.05, 0.1, 0.5, 0.8])
+    powers = np.asarray([0.99, 0.95, 0.9, 0.5, 0.2])
+    # zero_mask = np.where(alphas - powers > 0, 0.0, alphas - powers)
+    nobs_with_zeros = smp.normal_sample_size_one_tail(5, powers, alphas, 2, 2)
+    # check_nans = np.isnan(zero_mask) == np.isnan(nobs_with_nans)
+    assert_array_equal(nobs_with_zeros[powers <= alphas], 0)