Repository: numba/numba

Issue:
numpy.arange and numpy.linspace not accurate in numba
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (using `numba=0.52.0=py38h51da96c_0` from conda-forge which is the latest available version, also tested with `0.51.2` from the defaults channel).
- [x] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python test_numba.py'.

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->

I am getting different results in my code depending on if I run with numba or not. The problem is that the code that numba compiles is extremely simple (calling another compiled numba function in a for loop) but still one of the values comes out different. I have spent some time to create a small reproducible code which is attached below.

[test_numba.zip](https://github.com/numba/numba/files/6049357/test_numba.zip) which contains:
```
- test_numba.py
- env.yml # to reproduce the conda environment though 
#other versions of numba (including the one from the defaults channel) fail too

# 3 numpy arrays inside hdf5 files
- nonsep-epc.hdf5
- nonsep-negatives.hdf5
- nonsep-positives.hdf5

```

The related part of the code is (the `epc` function results are different from `epc.py_func`):
```python
@jit(nopython=True)
def min_weighted_error_rate_threshold(negatives, positives, cost):
    if cost > 1.0:
        cost = 1.0
    elif cost < 0.0:
        cost = 0.0

    return _minimizing_threshold(negatives, positives, cost)


@jit(nopython=True)
def epc(
    dev_negatives,
    dev_positives,
    n_points,
    is_sorted=False,
):
    # if not pre-sorted, copies and sorts
    dev_neg = dev_negatives if is_sorted else numpy.sort(dev_negatives)
    dev_pos = dev_positives if is_sorted else numpy.sort(dev_positives)
    step = 1.0 / (n_points - 1.0)
    alpha = numpy.arange(0, 1 + step, step)
   # something goes wrong from here
    thres = numpy.empty_like(alpha)
    for i, cost in enumerate(alpha):
        thres[i] = min_weighted_error_rate_threshold(dev_neg, dev_pos, cost)

    return thres, alpha
```
with this test function below written to test the `epc` function.
```python
def test():
    def _load(path):
        with h5py.File(path, "r") as f:
            return numpy.array(f["array"])

    positives = _load("nonsep-positives.hdf5")
    negatives = _load("nonsep-negatives.hdf5")
    thresholds_ref = _load("nonsep-epc.hdf5")[2]
    n_points = 100

    dev_negatives = negatives[: (negatives.shape[0] // 2)]
    dev_positives = positives[: (positives.shape[0] // 2)]

    print("Comparing pure Python to reference data.")
    thres, alpha_ref = epc.py_func(
        dev_negatives,
        dev_positives,
        n_points,
        is_sorted=False,
    )
    numpy.testing.assert_allclose(thres, thresholds_ref)
    print("All values are similar")

    print("Comparing sorted in Python to reference data.")
    thres, alpha = epc(
        numpy.sort(dev_negatives),
        numpy.sort(dev_positives),
        n_points,
        is_sorted=True,
    )
    numpy.testing.assert_allclose(alpha, alpha_ref)
    numpy.testing.assert_allclose(thres, thresholds_ref) # code fails here, this is line 181
    print("All values are similar")

    print("Comparing numba variant to reference data.")
    thres, alpha = epc(
        dev_negatives,
        dev_positives,
        n_points,
        is_sorted=False,
    )
    numpy.testing.assert_allclose(alpha, alpha_ref)
    numpy.testing.assert_allclose(thres, thresholds_ref) # it would fail here too if you comment the failing assertion above
    print("All values are similar")
```
If you run `python test_numba.py`, you will get:
```
$ python test_numba.py
Comparing pure Python to reference data.
All values are similar
Comparing sorted in Python to reference data.
Traceback (most recent call last):
  File "test_numba.py", line 197, in <module>
    test()
  File "test_numba.py", line 181, in test
    numpy.testing.assert_allclose(thres, thresholds_ref)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 1 / 100 (1%)
Max absolute difference: 0.4
Max relative difference: 0.06837607
```

If I call `min_weighted_error_rate_threshold` (which is a compiled function) from Python, everything works. But, when it is called from the `epc` function, one of the values turns out to be wrong. I have checked all the computations inside the epc function:
```python
@jit(nopython=True)
def epc(
    dev_negatives,
    dev_positives,
    n_points,
    is_sorted=False,
):
    # if not pre-sorted, copies and sorts
    dev_neg = dev_negatives if is_sorted else numpy.sort(dev_negatives)
    dev_pos = dev_positives if is_sorted else numpy.sort(dev_positives)
    step = 1.0 / (n_points - 1.0)
    alpha = numpy.arange(0, 1 + step, step)
    thres = numpy.empty_like(alpha)
    for i, cost in enumerate(alpha):
        thres[i] = min_weighted_error_rate_threshold(dev_neg, dev_pos, cost)

    return thres, alpha
```
and looks like all the values are correct (according to my tests in the `test()` function) till the code reaches:
```python
    thres = numpy.empty_like(alpha)
    for i, cost in enumerate(alpha):
        thres[i] = min_weighted_error_rate_threshold(dev_neg, dev_pos, cost)
```
Here, numba is just calling another already compiled function, so I don't understand what could go wrong here when numba compiles it. Please let me know if I can provide more information.


Discussion:
I found the cause of the problem.

It boils down to `numpy.arange(0, 1 + step, step)` having different round-off error when comparing the original numpy implementation vs the compiled version in Numba.

```python

In [15]: @njit
    ...: def foo(step):
    ...:     return numpy.arange(0, 1 + step, step)


In [16]: foo(0.010101010101010102) - foo.py_func(0.010101010101010102)
Out[16]:
array([ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  6.93889390e-18,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00, -1.38777878e-17, -1.38777878e-17,
       -1.38777878e-17, -2.77555756e-17, -2.77555756e-17,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  2.77555756e-17,  2.77555756e-17,
        2.77555756e-17,  5.55111512e-17,  5.55111512e-17,  5.55111512e-17,
        8.32667268e-17,  5.55111512e-17,  5.55111512e-17,  5.55111512e-17,
        5.55111512e-17,  1.11022302e-16,  1.11022302e-16,  1.11022302e-16,
        1.11022302e-16,  1.11022302e-16,  1.11022302e-16,  1.11022302e-16,
        1.66533454e-16,  1.66533454e-16,  1.66533454e-16,  1.66533454e-16,
        1.66533454e-16,  1.66533454e-16,  2.22044605e-16,  2.22044605e-16,
        2.22044605e-16,  2.22044605e-16,  2.22044605e-16,  2.22044605e-16,
        2.77555756e-16,  2.77555756e-16,  2.22044605e-16,  1.11022302e-16,
        1.11022302e-16,  1.11022302e-16,  0.00000000e+00,  0.00000000e+00,
       -1.11022302e-16, -1.11022302e-16, -1.11022302e-16, -2.22044605e-16,
       -2.22044605e-16, -3.33066907e-16, -3.33066907e-16, -4.44089210e-16,
       -4.44089210e-16, -4.44089210e-16, -5.55111512e-16, -5.55111512e-16,
       -6.66133815e-16, -6.66133815e-16, -7.77156117e-16, -7.77156117e-16,
       -7.77156117e-16, -8.88178420e-16, -8.88178420e-16, -9.99200722e-16,
       -9.99200722e-16, -9.99200722e-16, -1.11022302e-15, -1.11022302e-15,
       -1.22124533e-15, -1.22124533e-15, -1.33226763e-15, -1.33226763e-15,
       -1.33226763e-15, -1.44328993e-15, -1.44328993e-15, -1.55431223e-15,
       -1.55431223e-15, -1.66533454e-15, -1.66533454e-15, -1.66533454e-15,
       -1.77635684e-15, -1.77635684e-15, -1.88737914e-15, -1.88737914e-15,
       -1.88737914e-15, -1.99840144e-15, -1.99840144e-15, -1.99840144e-15])
This problem exists on the latest commit (4ddedd6c56073713ceacafe561de13b6ab3804dc) on mainline as well.
Thank you for debugging this. I was testing if alpha changes too, but I was testing using `allclose` which made me miss this. After reading the documentation of [numpy.arange](https://numpy.org/doc/stable/reference/generated/numpy.arange.html) which mentions that:

> When using a non-integer step, such as 0.1, the results will often not be consistent. It is better to use numpy.linspace for these cases.

I changed the code to use `numpy.linspace` instead:
```python
alpha = numpy.linspace(0, 1.0, n_points)
```
~~and now I get the same behavior with and without numba. I am not sure if you want to keep this issue open, I will let you decide.~~
but I still get different behavior. See comment below.
Actually, I still see a bug here:
```python
@njit
def foo(n_points):
    return numpy.linspace(0.0, 1.0, n_points)
foo(100) - foo.py_func(100)
array([ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00, -6.93889390e-18,  0.00000000e+00, -1.38777878e-17,
        0.00000000e+00,  0.00000000e+00, -1.38777878e-17, -1.38777878e-17,
        0.00000000e+00,  0.00000000e+00, -2.77555756e-17,  0.00000000e+00,
        0.00000000e+00, -2.77555756e-17,  0.00000000e+00, -2.77555756e-17,
       -2.77555756e-17,  0.00000000e+00, -2.77555756e-17, -2.77555756e-17,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00, -5.55111512e-17,
       -5.55111512e-17,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00, -5.55111512e-17, -5.55111512e-17, -5.55111512e-17,
        0.00000000e+00,  0.00000000e+00, -5.55111512e-17, -5.55111512e-17,
       -5.55111512e-17, -5.55111512e-17,  0.00000000e+00,  0.00000000e+00,
       -5.55111512e-17, -5.55111512e-17, -5.55111512e-17, -5.55111512e-17,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00, -1.11022302e-16,
        0.00000000e+00,  0.00000000e+00, -1.11022302e-16,  0.00000000e+00,
       -1.11022302e-16,  0.00000000e+00,  0.00000000e+00, -1.11022302e-16,
        0.00000000e+00, -1.11022302e-16,  0.00000000e+00, -1.11022302e-16,
        0.00000000e+00,  0.00000000e+00, -1.11022302e-16,  0.00000000e+00,
       -1.11022302e-16,  0.00000000e+00, -1.11022302e-16, -1.11022302e-16,
        0.00000000e+00, -1.11022302e-16,  0.00000000e+00, -1.11022302e-16,
       -1.11022302e-16,  0.00000000e+00, -1.11022302e-16,  0.00000000e+00,
       -1.11022302e-16,  0.00000000e+00, -1.11022302e-16, -1.11022302e-16,
        0.00000000e+00, -1.11022302e-16,  0.00000000e+00, -1.11022302e-16,
       -1.11022302e-16, -1.11022302e-16, -1.11022302e-16,  0.00000000e+00,
       -1.11022302e-16,  0.00000000e+00, -1.11022302e-16, -1.11022302e-16,
        0.00000000e+00, -1.11022302e-16,  0.00000000e+00,  0.00000000e+00])
```

Resolving patch:
diff --git a/numba/np/arrayobj.py b/numba/np/arrayobj.py
--- a/numba/np/arrayobj.py
+++ b/numba/np/arrayobj.py
@@ -3907,8 +3907,7 @@ def impl(start, stop=None, step=None, dtype=None):
         arr = np.empty(nitems, true_dtype)
         val = _start
         for i in range(nitems):
-            arr[i] = val
-            val += _step
+            arr[i] = val + (i * _step)
         return arr
 
     return impl
@@ -3929,15 +3928,19 @@ def linspace(start, stop):
 def numpy_linspace_3(context, builder, sig, args):
     dtype = as_dtype(sig.return_type.dtype)
 
+    # Implementation based on https://github.com/numpy/numpy/blob/v1.20.0/numpy/core/function_base.py#L24 # noqa: E501
     def linspace(start, stop, num):
         arr = np.empty(num, dtype)
         if num == 0:
             return arr
         div = num - 1
-        delta = stop - start
-        arr[0] = start
-        for i in range(1, num):
-            arr[i] = start + delta * (i / div)
+        if div > 0:
+            delta = stop - start
+            step = delta / div
+            for i in range(0, num):
+                arr[i] = start + (i * step)
+        else:
+            arr[0] = start
         return arr
 
     res = context.compile_internal(builder, linspace, sig, args)


Test patch:
diff --git a/numba/tests/test_array_methods.py b/numba/tests/test_array_methods.py
--- a/numba/tests/test_array_methods.py
+++ b/numba/tests/test_array_methods.py
@@ -922,6 +922,16 @@ def test_arange_throws(self):
                     self.assertIn("Maximum allowed size exceeded",
                                 str(raises.exception))
 
+    def test_arange_accuracy(self):
+        # Checking arange reasonably replicates NumPy's algorithm
+        # see https://github.com/numba/numba/issues/6768
+        @jit(nopython=True)
+        def foo(step):
+            return np.arange(0, 1 + step, step)
+
+        x = 0.010101010101010102
+        self.assertPreciseEqual(foo(x), foo.py_func(x))
+
     def test_item(self):
         pyfunc = array_item
         cfunc = jit(nopython=True)(pyfunc)
diff --git a/numba/tests/test_dyn_array.py b/numba/tests/test_dyn_array.py
--- a/numba/tests/test_dyn_array.py
+++ b/numba/tests/test_dyn_array.py
@@ -1234,7 +1234,7 @@ def test_error_handling(self):
             dfunc = nrtjit(self.py_kw)
             dfunc(d, k=3)
 
-class TestNdArange(BaseTest):
+class TestLinspace(BaseTest):
 
     def test_linspace_2(self):
         def pyfunc(n, m):
@@ -1252,6 +1252,16 @@ def pyfunc(n, m, p):
                             (1+0.5j, 1.5j, 5), (1, 1e100, 1)],
                            exact=False)
 
+    def test_linspace_accuracy(self):
+        # Checking linspace reasonably replicates NumPy's algorithm
+        # see https://github.com/numba/numba/issues/6768
+        @nrtjit
+        def foo(n, m, p):
+            return np.linspace(n, m, p)
+
+        n, m, p = 0.0, 1.0, 100
+        self.assertPreciseEqual(foo(n, m, p), foo.py_func(n, m, p))
+
 
 class TestNpyEmptyKeyword(TestCase):
     def _test_with_dtype_kw(self, dtype):