Repository: numba/numba

Issue:
TypeError in warnings machinery
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

For more general "how do I do X?" type questions, please speak to us in real
time on https://gitter.im/numba/numba or post to the Numba mailing list
https://groups.google.com/a/continuum.io/forum/#!forum/numba-users.

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I am using the latest ~~released~~ master version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [ ] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->
I see the error below on master which I believe is due to the introduction of deprecated warnings. `WarningsFixer` tries to sort warnings [here](https://github.com/numba/numba/blob/master/numba/errors.py#L425), but Numba warnings don't have comparison implemented. This happens only if previous attributes of the warning data structure happen to be the same. I haven't been able to create a small reproducer for this yet.

```
../numba/numba/compiler.py:1017: in compile_ir
    lifted_from=lifted_from)
../numba/numba/compiler.py:385: in compile_ir
    return self._compile_ir()
../numba/numba/compiler.py:893: in _compile_ir
    return self._compile_core()
../numba/numba/compiler.py:873: in _compile_core
    res = pm.run(self.status)
../numba/numba/compiler_lock.py:32: in _acquire_compile_lock
    return func(*args, **kwargs)
../numba/numba/compiler.py:254: in run
    raise patched_exception
../numba/numba/compiler.py:245: in run
    stage()
../numba/numba/compiler.py:501: in stage_nopython_frontend
    self.locals)
../numba/numba/compiler.py:1109: in type_inference_stage
    warnings.flush()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.errors.WarningsFixer object at 0x7f368b3c2cf8>

    def flush(self):
        """
        Emit all stored warnings.
        """
>       for (filename, lineno, category), messages in sorted(self._warnings.items()):
E       TypeError: Failed in bodo mode pipeline (step: convert to distributed)
E       Failed in nopython mode pipeline (step: nopython frontend)
E       '<' not supported between instances of 'type' and 'type'

../numba/numba/errors.py:425: TypeError
```


Discussion:
Thanks for the report. I've a feeling that this may always have been a problem but was not ever encountered because warnings were off by default.
This reproduces:
```python
from numba import errors
import warnings

wfix = errors.WarningsFixer(errors.NumbaWarning)
with wfix.catch_warnings('foo', 10):
    warnings.warn(errors.NumbaWarning(''))
    warnings.warn(errors.NumbaPerformanceWarning(''))

wfix.flush()
```
two warnings of different category are caught, but their contents is identical. How this is arising in practice I'm not sure!?
The way I got this was through a complicated exec/compile/transform/compile path...
If you can look with `pdb` as to the contents of those warnings when flush fails it'd be useful to find out where they are coming from, thanks.
Below is the warning contents by printing `self._warnings`.
```python
defaultdict(<class 'set'>,
{('<string>', 4, <class 'numba.errors.NumbaWarning'>):
{'Function "agg_update" was compiled in object mode without forceobj=True.\n\nFile "<string>", line 1:\n<source missing, REPL/exec in use?>\n', '\nCompilation is falling back to object mode WITH looplifting enabled because Function "agg_update" failed type inference due to: Untyped global name \'__update_redvars\': cannot determine Numba type of <class \'numba.ir.UndefinedType\'>\n\nFile "<string>", line 2:\n<source missing, REPL/exec in use?>\n'},
('<string>', 4, <class 'numba.errors.NumbaDeprecationWarning'>):
{'\nFall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.\n\nFor more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit\n\nFile "<string>", line 1:\n<source missing, REPL/exec in use?>\n'}})
```
@ehsantn thanks, I think the identical synthetic source and line number are the likely cause as it probably leads to comparison of the type itself.
Can we add a generic comparison operator to the base class that just compares on `id()`?  Once the sorting gets to that level, it doesn't really matter what the ordering is, it just has to be there.
Would `str(__class__)` then `id()` be ok, it'd then group warnings by class?
oh yeah, that would work better.

Resolving patch:
diff --git a/numba/errors.py b/numba/errors.py
--- a/numba/errors.py
+++ b/numba/errors.py
@@ -24,6 +24,7 @@ class NumbaWarning(Warning):
     """
     Base category for all Numba compiler warnings.
     """
+
     def __init__(self, msg, loc=None, highlighting=True, ):
         self.msg = msg
         self.loc = loc
@@ -422,7 +423,15 @@ def flush(self):
         """
         Emit all stored warnings.
         """
-        for (filename, lineno, category), messages in sorted(self._warnings.items()):
+        def key(arg):
+            # It is possible through codegen to create entirely identical
+            # warnings, this leads to comparing types when sorting which breaks
+            # on Python 3. Key as str() and if the worse happens then `id`
+            # creates some uniqueness
+            return str(arg) + str(id(arg))
+
+        for (filename, lineno, category), messages in sorted(
+                self._warnings.items(), key=key):
             for msg in sorted(messages):
                 warnings.warn_explicit(msg, category, filename, lineno)
         self._warnings.clear()


Test patch:
diff --git a/numba/tests/test_warnings.py b/numba/tests/test_warnings.py
--- a/numba/tests/test_warnings.py
+++ b/numba/tests/test_warnings.py
@@ -5,19 +5,21 @@
 import numba.unittest_support as unittest
 from numba import jit
 from numba.errors import NumbaWarning, deprecated, NumbaDeprecationWarning
+from numba import errors
+
 
 class TestBuiltins(unittest.TestCase):
 
     def check_objmode_deprecation_warning(self, w):
         # Object mode fall-back is slated for deprecation, check the warning
         msg = ("Fall-back from the nopython compilation path to the object "
-                "mode compilation path has been detected")
+               "mode compilation path has been detected")
         self.assertEqual(w.category, NumbaDeprecationWarning)
         self.assertIn(msg, str(w.message))
 
     def test_type_infer_warning(self):
         def add(x, y):
-            a = {}
+            a = {} # noqa dead
             return x + y
 
         with warnings.catch_warnings(record=True) as w:
@@ -40,6 +42,7 @@ def add(x, y):
 
     def test_return_type_warning(self):
         y = np.ones(4, dtype=np.float32)
+
         def return_external_array():
             return y
 
@@ -80,10 +83,9 @@ def return_external_array():
             # No more warning
             self.assertEqual(len(w), 0)
 
-
     def test_no_warning_with_forceobj(self):
         def add(x, y):
-            a = []
+            a = [] # noqa dead
             return x + y
 
         with warnings.catch_warnings(record=True) as w:
@@ -96,7 +98,7 @@ def add(x, y):
 
     def test_loop_lift_warn(self):
         def do_loop(x):
-            a = {}
+            a = {} # noqa dead
             for i in range(x.shape[0]):
                 x[i] *= 2
 
@@ -132,7 +134,8 @@ def do_loop(x):
 
     def test_deprecated(self):
         @deprecated('foo')
-        def bar(): pass
+        def bar():
+            pass
 
         with warnings.catch_warnings(record=True) as w:
             warnings.simplefilter('always')
@@ -143,5 +146,27 @@ def bar(): pass
             self.assertIn('bar', str(w[0].message))
             self.assertIn('foo', str(w[0].message))
 
+    def test_warnings_fixer(self):
+        # For some context, see #4083
+
+        wfix = errors.WarningsFixer(errors.NumbaWarning)
+        with wfix.catch_warnings('foo', 10):
+            warnings.warn(errors.NumbaWarning('same'))
+            warnings.warn(errors.NumbaDeprecationWarning('same'))
+
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            wfix.flush()
+
+            self.assertEqual(len(w), 2)
+            # the order of these will be backwards to the above, the
+            # WarningsFixer flush method sorts with a key based on str
+            # comparison
+            self.assertEqual(w[0].category, NumbaDeprecationWarning)
+            self.assertEqual(w[1].category, NumbaWarning)
+            self.assertIn('same', str(w[0].message))
+            self.assertIn('same', str(w[1].message))
+
+
 if __name__ == '__main__':
     unittest.main()