Repository: numba/numba

Issue:
Unicode equality overload regression
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

For more general "how do I do X?" type questions, please speak to us in real
time on https://gitter.im/numba/numba or post to the Numba mailing list
https://groups.google.com/a/continuum.io/forum/#!forum/numba-users.

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I am using the latest ~~released~~ master version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->
The overload for `eq` of unicodes on current master always returns `False` if one of the types is unicode but the other one is not. This is too generic and does not allow defining comparison with unicodes for extension types (e.g. String arrays). Here is a simple example:

```python
import operator
import numba
from numba.extending import overload, register_model, models, unbox, NativeValue, typeof_impl

class Foo:
    pass

class FooType(numba.types.Type):
    def __init__(self):
        super(FooType, self).__init__(name='Foo')


foo_type = FooType()
register_model(FooType)(models.OpaqueModel)


@typeof_impl.register(Foo)
def _typ_foo(val, c):
  return FooType()


@unbox(FooType)
def unbox_foo(typ, obj, c):
  return NativeValue(c.context.get_dummy_value())


@overload(operator.eq)
def foo_eq(a, b):
    if a == FooType():
        return lambda a, b: True


@numba.njit
def f(a):
    return a == "A"


print(f(Foo()))  # prints False instead of True
```


Discussion:
Thanks for the report. This change was made to support the use of heterogeneous container types in a wider range of scenarios. I think this raises the question of, how, in general, does this sort of thing get supported. We have to be able to define type based equality to permit more dynamic (albeit still static) typing and ensure things like `'a' == 12` work. One obvious way is via the `resolve_value_type/get_call_type/get_function` idiom, but this requires having a typing/codegen context around and associated hassle of `@intrinsic`. I'm wondering again whether `@overload` should store a reference to these precisely for this sort of scenario?
Ideally, extension functions like `@overload` should not be generic, and should just handle the input types specifically. To make it easier in this case, maybe there could be a concept of "default overload" that is used if other overloads failed. But specifying all basic types isn't too bad and can ensure robustness.
> Ideally, extension functions like `@overload` should not be generic, and should just handle the input types specifically.

I in part agree. I agree in that being generic in this case is not ideal. I'm struggling to agree about spelling out numerous of type combinations vs. doing some sort of secondary resolve. Whilst it's easy to say "one input must be string, string literal or unicode char sequence" trying to say "other input can be any of \<literally the entire *unextended* type system apart from those other three things\>" is not so easy. Perhaps if there were a set of all types, that might help, but really this is something to think about for when declarative typing arrives.

> To make it easier in this case, maybe there could be a concept of "default overload" that is used if other overloads failed. But specifying all basic types isn't too bad and can ensure robustness.

Maybe "default overload" could work, but we'd probably need to set up a special place to go look for those to prevent interference with normal resolution.



@ehsantn @sklam this https://github.com/numba/numba/pull/4983 implements the discussed "fix" from the core developer meeting. Having actually written this I'm not sure I'm comfortable with it, I still kind of more want to be able to spell out the set difference of `(all internal types - locally defined accepted types)` and use that in the `@overload` guard instead.

Resolving patch:
diff --git a/numba/types/abstract.py b/numba/types/abstract.py
--- a/numba/types/abstract.py
+++ b/numba/types/abstract.py
@@ -39,6 +39,15 @@ class _TypeMetaclass(ABCMeta):
     and hashing), then looking it up in the _typecache registry.
     """
 
+    def __init__(cls, name, bases, orig_vars):
+        # __init__ is hooked to mark whether a Type class being defined is a
+        # Numba internal type (one which is defined somewhere under the `numba`
+        # module) or an external type (one which is defined elsewhere, for
+        # example a user defined type).
+        super(_TypeMetaclass, cls).__init__(name, bases, orig_vars)
+        root = (cls.__module__.split('.'))[0]
+        cls._is_internal = root == "numba"
+
     def _intern(cls, inst):
         # Try to intern the created instance
         wr = weakref.ref(inst, _on_type_disposal)
@@ -207,6 +216,12 @@ def cast_python_value(self, args):
         raise NotImplementedError
 
 
+    @property
+    def is_internal(self):
+        """ Returns True if this class is an internally defined Numba type by
+        virtue of the module in which it is instantiated, False else."""
+        return self._is_internal
+
 # XXX we should distinguish between Dummy (no meaningful
 # representation, e.g. None or a builtin function) and Opaque (has a
 # meaningful representation, e.g. ExternalFunctionPointer)
diff --git a/numba/unicode.py b/numba/unicode.py
--- a/numba/unicode.py
+++ b/numba/unicode.py
@@ -425,6 +425,8 @@ def len_impl(s):
 
 @overload(operator.eq)
 def unicode_eq(a, b):
+    if not (a.is_internal and b.is_internal):
+        return
     accept = (types.UnicodeType, types.StringLiteral, types.UnicodeCharSeq)
     a_unicode = isinstance(a, accept)
     b_unicode = isinstance(b, accept)
@@ -443,6 +445,8 @@ def eq_impl(a, b):
 
 @overload(operator.ne)
 def unicode_ne(a, b):
+    if not (a.is_internal and b.is_internal):
+        return
     accept = (types.UnicodeType, types.StringLiteral, types.UnicodeCharSeq)
     a_unicode = isinstance(a, accept)
     b_unicode = isinstance(b, accept)


Test patch:
diff --git a/numba/tests/test_types.py b/numba/tests/test_types.py
--- a/numba/tests/test_types.py
+++ b/numba/tests/test_types.py
@@ -5,7 +5,14 @@
 from __future__ import print_function, absolute_import
 
 from collections import namedtuple
+import contextlib
 import gc
+import importlib
+import os
+import operator
+import shutil
+import sys
+import uuid
 try:
     import cPickle as pickle
 except ImportError:
@@ -16,11 +23,13 @@
 
 from numba import unittest_support as unittest
 from numba.utils import IS_PY3
-from numba import sigutils, types, typing
+from numba import sigutils, types, typing, errors
 from numba.types.abstract import _typecache
 from numba.typing.templates import make_overload_template
-from numba import jit, numpy_support, typeof
-from .support import TestCase, tag
+from numba import jit, njit, numpy_support, typeof
+from numba.extending import (overload, register_model, models, unbox,
+                             NativeValue, typeof_impl)
+from .support import TestCase, tag, temp_directory
 from .enum_usecases import Color, Shake, Shape
 
 
@@ -28,9 +37,11 @@
 
 Rect = namedtuple('Rect', ('width', 'height'))
 
+
 def gen(x):
     yield x + 1
 
+
 class Dummy(object):
     pass
 
@@ -74,6 +85,7 @@ def test_equality(self):
         class DummyTemplate(object):
             key = "foo"
         a = types.BoundFunction(DummyTemplate, types.int32)
+
         class DummyTemplate(object):
             key = "bar"
         b = types.BoundFunction(DummyTemplate, types.int32)
@@ -155,7 +167,7 @@ def test_cache_trimming(self):
         cache = _typecache
         gc.collect()
         # Keep strong references to existing types, to avoid spurious failures
-        existing_types = [wr() for wr in cache]
+        existing_types = [wr() for wr in cache]  # noqa: F841
         cache_len = len(cache)
         a = types.Dummy('xyzzyx')
         b = types.Dummy('foox')
@@ -173,9 +185,9 @@ def check(arrty, scalar, ndim, layout):
         scalar = types.int32
         check(scalar[:], scalar, 1, 'A')
         check(scalar[::1], scalar, 1, 'C')
-        check(scalar[:,:], scalar, 2, 'A')
-        check(scalar[:,::1], scalar, 2, 'C')
-        check(scalar[::1,:], scalar, 2, 'F')
+        check(scalar[:, :], scalar, 2, 'A')
+        check(scalar[:, ::1], scalar, 2, 'C')
+        check(scalar[::1, :], scalar, 2, 'F')
 
     def test_array_notation_for_dtype(self):
         def check(arrty, scalar, ndim, layout):
@@ -186,9 +198,9 @@ def check(arrty, scalar, ndim, layout):
         dtyped = types.DType(scalar)
         check(dtyped[:], scalar, 1, 'A')
         check(dtyped[::1], scalar, 1, 'C')
-        check(dtyped[:,:], scalar, 2, 'A')
-        check(dtyped[:,::1], scalar, 2, 'C')
-        check(dtyped[::1,:], scalar, 2, 'F')
+        check(dtyped[:, :], scalar, 2, 'A')
+        check(dtyped[:, ::1], scalar, 2, 'C')
+        check(dtyped[::1, :], scalar, 2, 'F')
 
     @tag('important')
     def test_call_notation(self):
@@ -235,6 +247,7 @@ def ol():
             types.Function((template1, template2))
         self.assertIn("incompatible templates:", str(raises.exception))
 
+
 class TestNumbers(TestCase):
     """
     Tests for number types.
@@ -252,9 +265,9 @@ def test_minval_maxval(self):
         self.assertEqual(types.int8.minval, -128)
         self.assertEqual(types.uint8.maxval, 255)
         self.assertEqual(types.uint8.minval, 0)
-        self.assertEqual(types.int64.maxval, (1<<63) - 1)
-        self.assertEqual(types.int64.minval, -(1<<63))
-        self.assertEqual(types.uint64.maxval, (1<<64) - 1)
+        self.assertEqual(types.int64.maxval, (1 << 63) - 1)
+        self.assertEqual(types.int64.minval, -(1 << 63))
+        self.assertEqual(types.uint64.maxval, (1 << 64) - 1)
         self.assertEqual(types.uint64.minval, 0)
 
     def test_from_bidwidth(self):
@@ -302,7 +315,9 @@ def check(ty, dtypes, ndim, layout, indexers=None):
             self.assertEqual(ty.dtypes, dtypes)
             views = [types.Array(dtype, 0, "C") for dtype in dtypes]
             if len(views) > 1:
-                self.assertEqual(ty.yield_type, types.BaseTuple.from_types(views))
+                self.assertEqual(
+                    ty.yield_type,
+                    types.BaseTuple.from_types(views))
             else:
                 self.assertEqual(ty.yield_type, views[0])
             if indexers is not None:
@@ -342,9 +357,9 @@ def check(ty, dtypes, ndim, layout, indexers=None):
         self.assertFalse(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((a, e, e, c64))
         check(ty, (f32, i16, i16, c64), 1, "C",
-              [('flat', 0, 1, [0]), # a
-               ('0d', 0, 0, [1, 2]), # e, e
-               ('scalar', 0, 0, [3]), # c64
+              [('flat', 0, 1, [0]),  # a
+               ('0d', 0, 0, [1, 2]),  # e, e
+               ('scalar', 0, 0, [3]),  # c64
                ])
         self.assertFalse(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((a, f))
@@ -360,24 +375,26 @@ def check(ty, dtypes, ndim, layout, indexers=None):
         check(ty, (f32,), 2, "C", [('flat', 0, 2, [0])])
         self.assertFalse(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((b, c))
-        check(ty, (f32, c64), 2, "C", [('flat', 0, 2, [0]), ('indexed', 0, 2, [1])])
+        check(
+            ty, (f32, c64), 2, "C", [
+                ('flat', 0, 2, [0]), ('indexed', 0, 2, [1])])
         self.assertTrue(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((d,))
         check(ty, (i16,), 2, "C", [('indexed', 0, 2, [0])])
         self.assertTrue(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((b, c, d, d, e))
         check(ty, (f32, c64, i16, i16, i16), 2, "C",
-              [('flat', 0, 2, [0]), # b
-               ('indexed', 0, 2, [1, 2, 3]), # c, d, d
-               ('0d', 0, 0, [4]), # e
+              [('flat', 0, 2, [0]),  # b
+               ('indexed', 0, 2, [1, 2, 3]),  # c, d, d
+               ('0d', 0, 0, [4]),  # e
                ])
         self.assertTrue(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((a, b, c, d, d, f))
         check(ty, (f32, f32, c64, i16, i16, f32), 2, "C",
-              [('flat', 1, 2, [0]), # a
-               ('flat', 0, 2, [1]), # b
-               ('indexed', 0, 2, [2, 3, 4]), # c, d, d
-               ('indexed', 1, 2, [5]), # f
+              [('flat', 1, 2, [0]),  # a
+               ('flat', 0, 2, [1]),  # b
+               ('indexed', 0, 2, [2, 3, 4]),  # c, d, d
+               ('indexed', 1, 2, [5]),  # f
                ])
         self.assertTrue(ty.need_shaped_indexing)
 
@@ -387,18 +404,18 @@ def check(ty, dtypes, ndim, layout, indexers=None):
         self.assertFalse(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((c, b, c, f))
         check(ty, (c64, f32, c64, f32), 2, "F",
-              [('flat', 0, 2, [0, 2]), # c, c
-               ('indexed', 0, 2, [1]), # b
-               ('indexed', 0, 1, [3]), # f
+              [('flat', 0, 2, [0, 2]),  # c, c
+               ('indexed', 0, 2, [1]),  # b
+               ('indexed', 0, 1, [3]),  # f
                ])
         self.assertTrue(ty.need_shaped_indexing)
         ty = types.NumpyNdIterType((b, c, c, d, d, a, e))
         check(ty, (f32, c64, c64, i16, i16, f32, i16), 2, "F",
-              [('indexed', 0, 2, [0, 3, 4]), # b, d, d
-               ('flat', 0, 2, [1, 2]), # c, c
-               ('flat', 0, 1, [5]), # a
-               ('0d', 0, 0, [6]), # e
-              ])
+              [('indexed', 0, 2, [0, 3, 4]),  # b, d, d
+               ('flat', 0, 2, [1, 2]),  # c, c
+               ('flat', 0, 1, [5]),  # a
+               ('0d', 0, 0, [6]),  # e
+               ])
         self.assertTrue(ty.need_shaped_indexing)
 
 
@@ -579,11 +596,11 @@ def assert_matches(arr, val, exact):
 
     def test_kind(self):
         def tkind(A):
-            return A.dtype.kind=='f'
+            return A.dtype.kind == 'f'
         jit_tkind = jit(nopython=True)(tkind)
         self.assertEqual(tkind(np.ones(3)), jit_tkind(np.ones(3)))
         self.assertEqual(tkind(np.ones(3, dtype=np.intp)),
-                            jit_tkind(np.ones(3, dtype=np.intp)))
+                         jit_tkind(np.ones(3, dtype=np.intp)))
 
     def test_dtype_with_type(self):
         def impl():
@@ -600,5 +617,177 @@ def impl():
         self.assertEqual(impl(), jit_impl())
 
 
+class TestIsInternalTypeMarker(TestCase):
+    """Tests the use of the Type metaclass init correctly setting the flag on
+    the `is_internal` attr of a concrete Type class
+    """
+    source_lines = """
+from numba import types
+
+class FooType(types.Type):
+    def __init__(self):
+        super(FooType, self).__init__(name='Foo')
+"""
+
+    # this is largely copied (with some modifications) from test_jit_module
+    @contextlib.contextmanager
+    def create_temp_module(self, source_lines=None, **jit_options):
+        # Use try/finally so cleanup happens even when an exception is raised
+        try:
+            if source_lines is None:
+                source_lines = self.source_lines
+            tempdir = temp_directory('test_extension_type')
+            # Generate random module name
+            temp_module_name = 'test_extension_type_{}'.format(
+                str(uuid.uuid4()).replace('-', '_'))
+            temp_module_path = os.path.join(tempdir, temp_module_name + '.py')
+
+            with open(temp_module_path, 'w') as f:
+                lines = source_lines.format(jit_options=jit_options)
+                f.write(lines)
+            # Add test_module to sys.path so it can be imported
+            sys.path.insert(0, tempdir)
+            test_module = importlib.import_module(temp_module_name)
+            yield test_module
+        finally:
+            sys.modules.pop(temp_module_name, None)
+            sys.path.remove(tempdir)
+            shutil.rmtree(tempdir)
+
+    def test_create_temp_module(self):
+        sys_path_original = list(sys.path)
+        sys_modules_original = dict(sys.modules)
+        with self.create_temp_module() as test_module:
+            temp_module_dir = os.path.dirname(test_module.__file__)
+            self.assertEqual(temp_module_dir, sys.path[0])
+            self.assertEqual(sys.path[1:], sys_path_original)
+            self.assertTrue(test_module.__name__ in sys.modules)
+        # Test that modifications to sys.path / sys.modules are reverted
+        self.assertEqual(sys.path, sys_path_original)
+        self.assertEqual(sys.modules, sys_modules_original)
+
+    def test_create_temp_module_with_exception(self):
+        try:
+            sys_path_original = list(sys.path)
+            sys_modules_original = dict(sys.modules)
+            with self.create_temp_module():
+                raise ValueError("Something went wrong!")
+        except ValueError:
+            # Test that modifications to sys.path / sys.modules are reverted
+            self.assertEqual(sys.path, sys_path_original)
+            self.assertEqual(sys.modules, sys_modules_original)
+
+    def test_externally_defined_type_is_external(self):
+
+        with self.create_temp_module(self.source_lines) as test_module:
+            FooType = test_module.FooType
+            self.assertFalse(FooType().is_internal)
+
+            # set up an extension type
+            class Foo(object):
+                pass
+
+            register_model(FooType)(models.OpaqueModel)
+
+            @typeof_impl.register(Foo)
+            def _typ_foo(val, c):
+                return FooType()
+
+            @unbox(FooType)
+            def unbox_foo(typ, obj, c):
+                return NativeValue(c.context.get_dummy_value())
+
+            # function to overload
+            def false_if_not_array(a):
+                pass
+
+            # Set up an overload which will accept all types irrespective of
+            # whether they are from Numba's closed type system
+            @overload(false_if_not_array)
+            def ol_false_if_not_array(a):
+                if isinstance(a, types.Array):
+                    return lambda a : True
+                else:
+                    return lambda a : False
+
+            @njit
+            def call_false_if_not_array(a):
+                return false_if_not_array(a)
+
+            self.assertTrue(call_false_if_not_array(np.zeros(10)))
+            self.assertFalse(call_false_if_not_array(10))
+
+            # The extension type was accepted
+            self.assertFalse(call_false_if_not_array(Foo()))
+
+            # Now do the same sort of overload but put in a guard based on the
+            # use of internal types
+
+            def false_if_not_array_closed_system(a):
+                pass
+
+            @overload(false_if_not_array_closed_system)
+            def ol_false_if_not_array_closed_system(a):
+                if a.is_internal:  # guard
+                    if isinstance(a, types.Array):
+                        return lambda a : True
+                    else:
+                        return lambda a : False
+
+            @njit
+            def call_false_if_not_array_closed_system(a):
+                return false_if_not_array_closed_system(a)
+
+            self.assertTrue(call_false_if_not_array_closed_system(np.zeros(10)))
+            self.assertFalse(call_false_if_not_array_closed_system(10))
+
+            with self.assertRaises(errors.TypingError) as raises:
+                call_false_if_not_array_closed_system(Foo())
+            estr = str(raises.exception)
+            self.assertIn("Invalid use of Function", estr)
+            self.assertIn("false_if_not_array_closed_system", estr)
+            self.assertIn("(Foo)", estr)
+
+    def test_mixin_against_real_example(self):
+        # See issue #4970, this checks that unicode eq/ne now ignores extension
+        # types.
+
+        with self.create_temp_module(self.source_lines) as test_module:
+            FooType = test_module.FooType
+            self.assertFalse(FooType().is_internal)
+
+            # set up an extension type
+            class Foo(object):
+                pass
+
+            register_model(FooType)(models.OpaqueModel)
+
+            @typeof_impl.register(Foo)
+            def _typ_foo(val, c):
+                return FooType()
+
+            @unbox(FooType)
+            def unbox_foo(typ, obj, c):
+                return NativeValue(c.context.get_dummy_value())
+
+            @overload(operator.eq)
+            def foo_eq(a, b):
+                if a == FooType():
+                    return lambda a, b: "RAN CUSTOM EQ OVERLOAD"
+
+            @overload(operator.ne)
+            def foo_ne(a, b):
+                if a == FooType():
+                    return lambda a, b: "RAN CUSTOM NE OVERLOAD"
+
+            @njit
+            def f(a):
+                return a == "A", a != "A"
+
+            self.assertEqual(("RAN CUSTOM EQ OVERLOAD",
+                              "RAN CUSTOM NE OVERLOAD"),
+                             f(Foo()))
+
+
 if __name__ == '__main__':
     unittest.main()