Repository: numba/numba
Issue category: Performance Issue

Calling a @jit function with a type that cannot be "fingerprinted" uses excessive memory
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (most recent is
  visible in the release notes
  (https://numba.readthedocs.io/en/stable/release-notes-overview.html).
- [x] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

This code:

```python
from numba import jit

@jit("none(unicode_type)")
def foo(x):
    pass

while 1:
    foo("x")
```

uses increasingly more and more memory as time goes by (order of megabytes a second). It seems like the issue is that types which end up going through `typeof_pyval` because they cannot be "fingerprinted":
https://github.com/numba/numba/blob/301ba23116864ef66a2caccfab7c7123e7bcbcfc/numba/core/dispatcher.py#L679-L695
will end up being added to this list:
https://github.com/numba/numba/blob/301ba23116864ef66a2caccfab7c7123e7bcbcfc/numba/core/dispatcher.py#L694
on the dispatcher object itself. This list doesn't leak, it is cleared on finalisation, but it grows with every call to the dispatcher with a type that triggers the `typeof_pyval` fallback. I think the list itself exists as a way to keep references alive between computing a type and compilation, but it has the noted side effect if a function is called repeatedly.

I think the the list can be made into a `set`, as types are hashable in general. This may not solve _all_ of the problem as the hash of some types could be `id` based, but it will deal with the most common cases. I have a patch for this locally and it seems to resolve the issue, I'll make a PR hopefully for 0.61.
Calling a @jit function with a type that cannot be "fingerprinted" uses excessive memory
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (most recent is
  visible in the release notes
  (https://numba.readthedocs.io/en/stable/release-notes-overview.html).
- [x] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

This code:

```python
from numba import jit

@jit("none(unicode_type)")
def foo(x):
    pass

while 1:
    foo("x")
```

uses increasingly more and more memory as time goes by (order of megabytes a second). It seems like the issue is that types which end up going through `typeof_pyval` because they cannot be "fingerprinted":
https://github.com/numba/numba/blob/301ba23116864ef66a2caccfab7c7123e7bcbcfc/numba/core/dispatcher.py#L679-L695
will end up being added to this list:
https://github.com/numba/numba/blob/301ba23116864ef66a2caccfab7c7123e7bcbcfc/numba/core/dispatcher.py#L694
on the dispatcher object itself. This list doesn't leak, it is cleared on finalisation, but it grows with every call to the dispatcher with a type that triggers the `typeof_pyval` fallback. I think the list itself exists as a way to keep references alive between computing a type and compilation, but it has the noted side effect if a function is called repeatedly.

I think the the list can be made into a `set`, as types are hashable in general. This may not solve _all_ of the problem as the hash of some types could be `id` based, but it will deal with the most common cases. I have a patch for this locally and it seems to resolve the issue, I'll make a PR hopefully for 0.61.


Functions to edit:
numba/core/dispatcher.py:_DispatcherBase.__init__
numba/core/dispatcher.py:_DispatcherBase._compile_for_args
numba/core/dispatcher.py:_DispatcherBase.typeof_pyval

Patch:
diff --git a/docs/upcoming_changes/9757.bug_fix.rst b/docs/upcoming_changes/9757.bug_fix.rst
new file mode 100644
index 00000000000..17a239ef57c
--- /dev/null
+++ b/docs/upcoming_changes/9757.bug_fix.rst
@@ -0,0 +1,6 @@
+Fix excessive memory use/poor memory behaviour in the dispatcher for non-fingerprintable types.
+-----------------------------------------------------------------------------------------------
+
+In the case of repeated dispatch on non-fingerprintable types, the dispatcher
+now uses memory in proportion to the number of unique types seen opposed to in
+proportion to the number of types in total.
diff --git a/numba/core/dispatcher.py b/numba/core/dispatcher.py
index b63e58fa054..d195b8b4965 100644
--- a/numba/core/dispatcher.py
+++ b/numba/core/dispatcher.py
@@ -209,7 +209,7 @@ def __init__(self, arg_count, py_func, pysig, can_fallback,
         # but newer python uses a different name
         self.__code__ = self.func_code
         # a place to keep an active reference to the types of the active call
-        self._types_active_call = []
+        self._types_active_call = set()
         # Default argument values match the py_func
         self.__defaults__ = py_func.__defaults__
 
@@ -441,7 +441,7 @@ def error_rewrite(e, issue_type):
             # ignore the FULL_TRACEBACKS config, this needs reporting!
             raise e
         finally:
-            self._types_active_call = []
+            self._types_active_call.clear()
         return return_val
 
     def inspect_llvm(self, signature=None):
@@ -691,7 +691,7 @@ def typeof_pyval(self, val):
         else:
             if tp is None:
                 tp = types.pyobject
-        self._types_active_call.append(tp)
+        self._types_active_call.add(tp)
         return tp
 
     def _callback_add_timer(self, duration, cres, lock_name):