Repository: numba/numba

Issue:
Variable name uniqueness issue in inlining (introduced in Numba 0.54)
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (most recent is
 visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [ ] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->
`mk_unique_var` was removed from the line below in Numba 0.54. This is causing variable name conflict issues in Bodo due to heavy use of inlining. Reproducer is not easy to provide but possible if necessary (doesn't seem necessary to me at the moment). What was the reason for removing `mk_unique_var`?

https://github.com/numba/numba/blob/7f72315f0bf672729b5a6eaf004aa9061e8bf0f5/numba/core/inline_closurecall.py#L398


Discussion:
It was removed in this PR: https://github.com/numba/numba/pull/7031
> Reproducer is not easy to provide but possible if necessary (doesn't seem necessary to me at the moment). 

Can you provide some more detail about the conditions that cause the problem please?
Variable names of the inlined function clash with the caller.
@ehsantn thanks for reporting this. I think it'd be good to have a reproducer so it's possible to see what is going wrong and from there work out the best way to fix it. A reproducer would also be used as part of the unit tests for any proposed fix with view of preventing similar issues in future. Thanks.

IIRC `mk_unique_var` was removed in favour of using the `scope.redefine` to handle https://github.com/numba/numba/issues/6969#issuecomment-840909275.
```python
import numpy as np
from numba import jit, generated_jit, prange

my_jit = jit(cache=True, nogil=True, nopython=True, inline="always")


def factory():

    def impl(col):
        for i in range(col.size):
            col[i] *= 2
        
    return my_jit(impl)


@generated_jit(cache=True, nogil=True, nopython=True, parallel=True)
def outer(a):

    inner = factory()

    def impl(a):
        for i in prange(a.shape[0]):
            inner(a[i])

    return impl


if __name__ == "__main__":

    a = np.arange(100).reshape(10,10)
    outer(a)
```
I am not sure if this is identical, @ehsantn. Is the problem in your case also related to `prange`? I am also running into issues.
The traceback for the above is:
```
Traceback (most recent call last):
  File "test.py", line 31, in <module>
    outer(a)
  File "/venvs/nbenv/lib/python3.8/site-packages/numba/core/dispatcher.py", line 485, in _compile_for_args
    error_rewrite(e, 'unsupported_error')
  File "/venvs/nbenv/lib/python3.8/site-packages/numba/core/dispatcher.py", line 423, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.UnsupportedRewriteError: Failed in nopython mode pipeline (step: convert to parfors)
Overwrite of parallel loop index

File "test.py", line 10:
    def impl(col):
        for i in range(col.size):
        ^
```
Is this a MWR (based on the above)?
```python
import numpy as np
from numba import njit, prange

@njit(inline="always")
def bar(x):
    for i in range(x.size):
        x[i] += 1

@njit(parallel=True)
def foo(a):
    for i in prange(a.shape[0]):
        bar(a[i])

a = np.ones((10, 10))
foo(a)
```
if so, I think the issue is that the inliner renames the variable `i` in the `bar` function as a version of `i` and adds it to the scope. Then in parfors, the failure comes from here:
https://github.com/numba/numba/blob/7f72315f0bf672729b5a6eaf004aa9061e8bf0f5/numba/parfors/parfor.py#L2673-L2684
where the scope is checked and if it is in the scope the unversioned name is returned and this collides with the unversioned induction var name.

Since #7031, in the inliner uses `scope.redefine` to create a variable with a versioned name and put it in scope, whereas previously the `mk_unique_var` was creating a variable with a similarly versioned name but does not put it into the scope.

I think the code ends up looking a bit like:
```python
import numpy as np
from numba import njit, prange

@njit(parallel=True)
def foo(a):
    for i in prange(a.shape[0]):
        for i in range(a.shape[1]):
            a[i] += 1

a = np.ones((10, 10))
foo(a)
print(a)

```
which is valid and fails the same way under parfors.
Thanks for simplifying @stuartarchibald - I threw together that example in a hurry. I am not surprised that that is the source of the error - it is consistent with what I imagined.

Is this to be considered a bug, it is it a user error? I ask because ensuring uniqueness of variable names in complicated code could be challenging.
> Thanks for simplifying @stuartarchibald - I threw together that example in a hurry. I am not surprised that that is the source of the error - it is consistent with what I imagined.

No problem.

> Is this to be considered a bug, it is it a user error?

It's a bug, but one of those bugs where a decision will have to be made about where to fix it. Parfors should probably just handle this use case, but that is easier said than done. CC @DrTodd13 what do you think?

> I ask because ensuring uniqueness of variable names in complicated code could be challenging.

@JSKenyon What's the reason for setting `inline='always'` in your code? Obviously it should "just work" but if you can get away without using it that'd be a work around for this case. Also, it's entirely unreasonable for Numba to expect users to implement a version of SSA (a bit like unique naming) in their code to ensure "uniqueness" of variables, Numba needs to fix this!!

@ehsantn I'm not convinced that the above is the same as the problem you are seeing?

> @ehsantn I'm not convinced that the above is the same as the problem you are seeing?

Apologies if I derailed the issue!

> @JSKenyon What's the reason for setting `inline='always'` in your code? Obviously it should "just work" but if you can get away without using it that'd be a work around for this case. Also, it's entirely unreasonable for Numba to expect users to implement a version of SSA (a bit like unique naming) in their code to ensure "uniqueness" of variables, Numba needs to fix this!!

It is not a roadblock for me at present. The `inline='always'` was there because in my real code this is about 7 loop levels deep and I wanted to be sure that the function call was elided. This may be an error on my part - I would be lying if I said I perfectly understood the if and when's of inlining.

> > @ehsantn I'm not convinced that the above is the same as the problem you are seeing?
> 
> Apologies if I derailed the issue!

Not at all, thanks for reporting it, I'm fairly sure it's a related problem, but just perhaps not exactly the one @ehsantn can see. I've opened https://github.com/numba/numba/issues/7380 for more discussion.

> > @JSKenyon What's the reason for setting `inline='always'` in your code? Obviously it should "just work" but if you can get away without using it that'd be a work around for this case. Also, it's entirely unreasonable for Numba to expect users to implement a version of SSA (a bit like unique naming) in their code to ensure "uniqueness" of variables, Numba needs to fix this!!
> 
> It is not a roadblock for me at present. The `inline='always'` was there because in my real code this is about 7 loop levels deep and I wanted to be sure that the function call was elided. This may be an error on my part - I would be lying if I said I perfectly understood the if and when's of inlining.

I see, there's a reasonable chance that LLVM would inline it anyway. There's some notes on what the Numba inliner does (and what LLVM does) here: https://numba.readthedocs.io/en/stable/developer/inlining.html#notes-on-inlining. Feel free to post on https://numba.discourse.group/ if you've any questions about it. Thanks!
@stuartarchibald your reproducer captures the issue well. I just tried it and adding `mk_unique_var` fixes the issue. The problem is name conflicts during inlining. Bodo has a lot of IR nodes inlined in various ways in several transformations in general.

Just adding an extra `mk_unique_var` in this line doesn't seem to hurt anything. Is the intention to break existing code so everyone is forced to use the `Scope` object properly because it is super critical now?
This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.
From a Numba public meeting a few weeks back... It was decided to explore fixing this (and improving debugging) by doing something similar to what LLVM does in this situation, which is prefix the inlined IR with the function name. It seems like this would solve the above, be correct with respect to `Scope` behaviours, helps debugging IR and also aids DI metadata generated in that inlined functions are likely to be easily identifiable and this can be expressed as such.

Resolving patch:
diff --git a/numba/core/inline_closurecall.py b/numba/core/inline_closurecall.py
--- a/numba/core/inline_closurecall.py
+++ b/numba/core/inline_closurecall.py
@@ -53,6 +53,20 @@ def callee_ir_validator(func_ir):
                 raise errors.UnsupportedError(msg, loc=stmt.loc)
 
 
+def _created_inlined_var_name(function_name, var_name):
+    """Creates a name for an inlined variable based on the function name and the
+    variable name. It does this "safely" to avoid the use of characters that are
+    illegal in python variable names as there are occasions when function
+    generation needs valid python name tokens."""
+    inlined_name = f'{function_name}.{var_name}'
+    # Replace angle brackets, e.g. "<locals>" is replaced with "_locals_"
+    new_name = inlined_name.replace('<', '_').replace('>', '_')
+    # The version "version" of the closure function e.g. foo$2 (id 2) is
+    # rewritten as "foo_v2". Further "." is also replaced with "_".
+    new_name = new_name.replace('.', '_').replace('$', '_v')
+    return new_name
+
+
 class InlineClosureCallPass(object):
     """InlineClosureCallPass class looks for direct calls to locally defined
     closures, and inlines the body of the closure function to the call site.
@@ -395,7 +409,9 @@ def copy_ir(the_ir):
         var_dict = {}
         for var in callee_scope.localvars._con.values():
             if not (var.name in callee_freevars):
-                new_var = scope.redefine(var.name, loc=var.loc)
+                inlined_name = _created_inlined_var_name(
+                    callee_ir.func_id.unique_name, var.name)
+                new_var = scope.redefine(inlined_name, loc=var.loc)
                 var_dict[var.name] = new_var
         self.debug_print("var_dict = ", var_dict)
         replace_vars(callee_blocks, var_dict)
@@ -599,7 +615,9 @@ def inline_closure_call(func_ir, glbls, block, i, callee, typingctx=None,
     var_dict = {}
     for var in callee_scope.localvars._con.values():
         if not (var.name in callee_code.co_freevars):
-            new_var = scope.redefine(mk_unique_var(var.name), loc=var.loc)
+            inlined_name = _created_inlined_var_name(
+                callee_ir.func_id.unique_name, var.name)
+            new_var = scope.redefine(inlined_name, loc=var.loc)
             var_dict[var.name] = new_var
     debug_print("var_dict = ", var_dict)
     replace_vars(callee_blocks, var_dict)
diff --git a/numba/core/ir_utils.py b/numba/core/ir_utils.py
--- a/numba/core/ir_utils.py
+++ b/numba/core/ir_utils.py
@@ -1714,12 +1714,15 @@ def _create_function_from_code_obj(fcode, func_env, func_arg, func_clo, glbls):
     * func_clo - string for the closure args
     * glbls - the function globals
     """
-    func_text = "def g():\n%s\n  def f(%s):\n    return (%s)\n  return f" % (
-        func_env, func_arg, func_clo)
+    sanitized_co_name = fcode.co_name.replace('<', '_').replace('>', '_')
+    func_text = (f"def closure():\n{func_env}\n"
+                 f"\tdef {sanitized_co_name}({func_arg}):\n"
+                 f"\t\treturn ({func_clo})\n"
+                 f"\treturn {sanitized_co_name}")
     loc = {}
     exec(func_text, glbls, loc)
 
-    f = loc['g']()
+    f = loc['closure']()
     # replace the code body
     f.__code__ = fcode
     f.__name__ = fcode.co_name
@@ -1730,7 +1733,7 @@ def get_ir_of_code(glbls, fcode):
     Compile a code object to get its IR, ir.Del nodes are emitted
     """
     nfree = len(fcode.co_freevars)
-    func_env = "\n".join(["  c_%d = None" % i for i in range(nfree)])
+    func_env = "\n".join(["\tc_%d = None" % i for i in range(nfree)])
     func_clo = ",".join(["c_%d" % i for i in range(nfree)])
     func_arg = ",".join(["x_%d" % i for i in range(fcode.co_argcount)])
 
@@ -2243,7 +2246,7 @@ def convert_code_obj_to_function(code_obj, caller_ir):
                    "variable '%s' in a function that will escape." % x)
             raise TypingError(msg, loc=code_obj.loc)
 
-    func_env = "\n".join(["  c_%d = %s" % (i, x) for i, x in enumerate(freevars)])
+    func_env = "\n".join(["\tc_%d = %s" % (i, x) for i, x in enumerate(freevars)])
     func_clo = ",".join(["c_%d" % i for i in range(nfree)])
     co_varnames = list(fcode.co_varnames)
 


Test patch:
diff --git a/numba/tests/support.py b/numba/tests/support.py
--- a/numba/tests/support.py
+++ b/numba/tests/support.py
@@ -28,7 +28,11 @@
 
 from numba import testing
 from numba.core import errors, typing, utils, config, cpu
-from numba.core.compiler import compile_extra, compile_isolated, Flags, DEFAULT_FLAGS
+from numba.core.compiler import (compile_extra, compile_isolated, Flags,
+                                 DEFAULT_FLAGS, CompilerBase,
+                                 DefaultPassBuilder)
+from numba.core.typed_passes import IRLegalization
+from numba.core.untyped_passes import PreserveIR
 import unittest
 from numba.core.runtime import rtsys
 from numba.np import numpy_support
@@ -996,3 +1000,21 @@ def run_in_subprocess(code, flags=None, env=None, timeout=30):
         msg = "process failed with code %s: stderr follows\n%s\n"
         raise AssertionError(msg % (popen.returncode, err.decode()))
     return out, err
+
+
+class IRPreservingTestPipeline(CompilerBase):
+    """ Same as the standard pipeline, but preserves the func_ir into the
+    metadata store after legalisation, useful for testing IR changes"""
+
+    def define_pipelines(self):
+        pipeline = DefaultPassBuilder.define_nopython_pipeline(
+            self.state, "ir_preserving_custom_pipe")
+        # mangle the default pipeline and inject DCE and IR preservation ahead
+        # of legalisation
+
+        # TODO: add a way to not do this! un-finalizing is not a good idea
+        pipeline._finalized = False
+        pipeline.add_pass_after(PreserveIR, IRLegalization)
+
+        pipeline.finalize()
+        return [pipeline]
diff --git a/numba/tests/test_closure.py b/numba/tests/test_closure.py
--- a/numba/tests/test_closure.py
+++ b/numba/tests/test_closure.py
@@ -5,7 +5,8 @@
 import unittest
 from numba import njit, jit
 from numba.core.errors import TypingError, UnsupportedError
-from numba.tests.support import TestCase
+from numba.core import ir
+from numba.tests.support import TestCase, IRPreservingTestPipeline
 
 
 class TestClosure(TestCase):
@@ -438,6 +439,40 @@ def bar(a, b):
         msg = "Calling a closure with *args is unsupported."
         self.assertIn(msg, str(raises.exception))
 
+    def test_closure_renaming_scheme(self):
+        # See #7380, this checks that inlined (from closure) variables have a
+        # name derived from the function they were defined in.
+
+        @njit(pipeline_class=IRPreservingTestPipeline)
+        def foo(a, b):
+            def bar(z):
+                x = 5
+                y = 10
+                return x + y + z
+            return bar(a), bar(b)
+
+        self.assertEqual(foo(10, 20), (25, 35))
+
+        # check IR. Look for the `x = 5`... there should be
+        # Two lots of `const(int, 5)`, one for each inline
+        # The LHS of the assignment will have a name like:
+        # closure__locals__bar_v2_x
+        # Ensure that this is the case!
+        func_ir = foo.overloads[foo.signatures[0]].metadata['preserved_ir']
+        store = []
+        for blk in func_ir.blocks.values():
+            for stmt in blk.body:
+                if isinstance(stmt, ir.Assign):
+                    if isinstance(stmt.value, ir.Const):
+                        if stmt.value.value == 5:
+                            store.append(stmt)
+
+        self.assertEqual(len(store), 2)
+        for i in store:
+            name = i.target.name
+            regex = r'closure__locals__bar_v[0-9]+.x'
+            self.assertRegex(name, regex)
+
 
 class TestObjmodeFallback(TestCase):
     # These are all based on tests from real life issues where, predominantly,
diff --git a/numba/tests/test_copy_propagate.py b/numba/tests/test_copy_propagate.py
--- a/numba/tests/test_copy_propagate.py
+++ b/numba/tests/test_copy_propagate.py
@@ -10,7 +10,7 @@
 from numba.core.ir_utils import (copy_propagate, apply_copy_propagate,
                             get_name_var_table)
 from numba.core.typed_passes import type_inference_stage
-from numba.tests.test_ir_inlining import InlineTestPipeline
+from numba.tests.support import IRPreservingTestPipeline
 import numpy as np
 import unittest
 
@@ -124,7 +124,7 @@ def test_impl(a):
             b = a + 3
             return b
 
-        j_func = njit(pipeline_class=InlineTestPipeline)(test_impl)
+        j_func = njit(pipeline_class=IRPreservingTestPipeline)(test_impl)
         self.assertEqual(test_impl(5), j_func(5))
 
         # make sure b is the target of the expression assignment, not a temporary
diff --git a/numba/tests/test_ir_inlining.py b/numba/tests/test_ir_inlining.py
--- a/numba/tests/test_ir_inlining.py
+++ b/numba/tests/test_ir_inlining.py
@@ -7,8 +7,8 @@
 from itertools import product
 import numpy as np
 
-from numba import njit, typeof, literally
-from numba.core import types, ir, ir_utils, cgutils
+from numba import njit, typeof, literally, prange
+from numba.core import types, ir, ir_utils, cgutils, errors
 from numba.core.extending import (
     overload,
     overload_method,
@@ -25,29 +25,11 @@
 from numba.core.datamodel.models import OpaqueModel
 from numba.core.cpu import InlineOptions
 from numba.core.compiler import DefaultPassBuilder, CompilerBase
-from numba.core.typed_passes import IRLegalization, InlineOverloads
-from numba.core.untyped_passes import PreserveIR
+from numba.core.typed_passes import InlineOverloads
 from numba.core.typing import signature
 from numba.tests.support import (TestCase, unittest, skip_py38_or_later,
-                                 MemoryLeakMixin)
-
-
-class InlineTestPipeline(CompilerBase):
-    """ Same as the standard pipeline, but preserves the func_ir into the
-    metadata store"""
-
-    def define_pipelines(self):
-        pipeline = DefaultPassBuilder.define_nopython_pipeline(
-            self.state, "inliner_custom_pipe")
-        # mangle the default pipeline and inject DCE and IR preservation ahead
-        # of legalisation
-
-        # TODO: add a way to not do this! un-finalizing is not a good idea
-        pipeline._finalized = False
-        pipeline.add_pass_after(PreserveIR, IRLegalization)
-
-        pipeline.finalize()
-        return [pipeline]
+                                 MemoryLeakMixin, IRPreservingTestPipeline,
+                                 skip_parfors_unsupported)
 
 
 # this global has the same name as the global in inlining_usecases.py, it
@@ -101,7 +83,7 @@ def check(self, test_impl, *args, **kwargs):
             assert isinstance(k, str)
             assert isinstance(v, bool)
 
-        j_func = njit(pipeline_class=InlineTestPipeline)(test_impl)
+        j_func = njit(pipeline_class=IRPreservingTestPipeline)(test_impl)
 
         # check they produce the same answer first!
         self.assertEqual(test_impl(*args), j_func(*args))
@@ -490,6 +472,43 @@ def bar(x):
         self.check(impl, inline_expect={'foo': True, 'boz': True,
                                         'fortran': True}, block_count=37)
 
+    def test_inline_renaming_scheme(self):
+        # See #7380, this checks that inlined variables have a name derived from
+        # the function they were defined in.
+
+        @njit(inline="always")
+        def bar(z):
+            x = 5
+            y = 10
+            return x + y + z
+
+        @njit(pipeline_class=IRPreservingTestPipeline)
+        def foo(a, b):
+            return bar(a), bar(b)
+
+        self.assertEqual(foo(10, 20), (25, 35))
+
+        # check IR. Look for the `x = 5`... there should be
+        # Two lots of `const(int, 5)`, one for each inline
+        # The LHS of the assignment will have a name like:
+        # TestFunctionInlining_test_inline_renaming_scheme__locals__bar_v2.x
+        # Ensure that this is the case!
+        func_ir = foo.overloads[foo.signatures[0]].metadata['preserved_ir']
+        store = []
+        for blk in func_ir.blocks.values():
+            for stmt in blk.body:
+                if isinstance(stmt, ir.Assign):
+                    if isinstance(stmt.value, ir.Const):
+                        if stmt.value.value == 5:
+                            store.append(stmt)
+
+        self.assertEqual(len(store), 2)
+        for i in store:
+            name = i.target.name
+            basename = self.id().lstrip(self.__module__)
+            regex = rf'{basename}__locals__bar_v[0-9]+.x'
+            self.assertRegex(name, regex)
+
 
 class TestRegisterJitableInlining(MemoryLeakMixin, InliningBase):
 
@@ -1059,7 +1078,7 @@ def setUp(self):
 
     def check_method(self, test_impl, args, expected, block_count,
                      expects_inlined=True):
-        j_func = njit(pipeline_class=InlineTestPipeline)(test_impl)
+        j_func = njit(pipeline_class=IRPreservingTestPipeline)(test_impl)
         # check they produce the same answer first!
         self.assertEqual(j_func(*args), expected)
 
@@ -1081,7 +1100,7 @@ def check_method(self, test_impl, args, expected, block_count,
 
     def check_getattr(self, test_impl, args, expected, block_count,
                       expects_inlined=True):
-        j_func = njit(pipeline_class=InlineTestPipeline)(test_impl)
+        j_func = njit(pipeline_class=IRPreservingTestPipeline)(test_impl)
         # check they produce the same answer first!
         self.assertEqual(j_func(*args), expected)
 
@@ -1455,6 +1474,43 @@ def foo(z):
 
         self.assertEqual(foo(10), (11.3, 11.3))
 
+    @skip_parfors_unsupported
+    def test_issue7380(self):
+        # This checks that inlining a function containing a loop into another
+        # loop where the induction variable in boths loops is the same doesn't
+        # end up with a name collision. Parfors can detect this so it is used.
+        # See: https://github.com/numba/numba/issues/7380
+
+        # Check Numba inlined function passes
+
+        @njit(inline="always")
+        def bar(x):
+            for i in range(x.size):
+                x[i] += 1
+
+        @njit(parallel=True)
+        def foo(a):
+            for i in prange(a.shape[0]):
+                bar(a[i])
+
+        a = np.ones((10, 10))
+        foo(a) # run
+        # check mutation of data is correct
+        self.assertPreciseEqual(a, 2 * np.ones_like(a))
+
+        # Check manually inlined equivalent function fails
+        @njit(parallel=True)
+        def foo_bad(a):
+            for i in prange(a.shape[0]):
+                x = a[i]
+                for i in range(x.size):
+                    x[i] += 1
+
+        with self.assertRaises(errors.UnsupportedRewriteError) as e:
+            foo_bad(a)
+
+        self.assertIn("Overwrite of parallel loop index", str(e.exception))
+
 
 if __name__ == '__main__':
     unittest.main()