Repository: numba/numba

Issue:
Added support for PyInstaller frozen caching
Previously, attempts to use caching with a frozen application raised a RuntimeError, with the following code:

```
RuntimeError("cannot cache function %r: no locator available "
                               "for file %r" % (qualname, self._source_path))
```

This is due to the frozen application architecture, where a frozen application uses a bootloader which either identifies resources from a stable folder (one-folder), or a temporary directory (one-file), and this has the added advantage of working with both.

**Patch 1**

In both the one-folder and one-file example, `os.path.exists(py_file)` will return False, since the file is either bundled into an executable within a folder or provided as an extension of site-packages inside a temporary folder (prefixed with 'site-packages'). Therefore, a simple check if `os.path.exists(py_file) and not getattr(sys, 'frozen', False)` is enough to override the function exit. I therefore redefined the `_SourceFileBackedLocatorMixin.from_function` for `_UserWideCacheLocator`, so the in-tree compilation is not attempted, but user-wide cache is only if the function is frozen.

**Patch 2**

Normally, not on Windows systems, the numba compiler looks for the abspath to the file, to avoid conflicting installations of the same module, doing:

```
if os.name != 'nt':
    cache_subpath = os.path.abspath(cache_subpath).lstrip(os.path.sep)
```

Since there is no real path to the file, and `os.abspath` returns a location that is specific to a relocatable executable, I skipped this code block if the code is frozen.

```
 if not (os.name == "nt" or getattr(sys, 'frozen', False)):
    cache_subpath = os.path.abspath(cache_subpath).lstrip(os.path.sep)
```

**Patch 3**

Finally, since `self._pyfile` does not point to a real file, it's important to look at the source time stamp of the executable, and not the virtual file. I therefore redefined `get_source_stamp` to get file metadata from the executable and not the virtual file.

Original:

```
def get_source_stamp(self):
    st = os.stat(self._py_file)
```

New:

```
def get_source_stamp(self):
    if getattr(sys, 'frozen', False):
        st = os.stat(sys.executable)
    else:
        st = os.stat(self._py_file)
```

Finally, this should minimally conflict with any other installs, since in-tree caching is favored for user installs or running from source, global installs use the system path or a local path, and iPython uses a separate cache, and frozen applications preface all imported functions with "site-packages", and all frozen functions using solely the executable name:

```
>>> inspect.getfile(package.function)
'site-packages/package/function.py'
>>> inspect.getfile(local_function)
'myscript.py'
```



Discussion:
@pitrou: Can you review this?

Hi,

Thanks for the patch! Can you change the docstring for `UserWideCacheLocator` to mention it also handles frozen applications?

Hi,

Is the update to the class docstring sufficient, or should I go into more detail?

I think it's good enough. As a sidenote, though, since there are no automated tests, there is always a risk of a regression in the future. I presume it would be complicated to write such a test?

Actually, no it would be pretty easy. The source path is determined by the writable attribute `__file__` for modules and by the `func_code` attribute for functions, as shown by the inspect function:

**Inspect getfile**

``` python
def getfile(object):
    """Work out which source or compiled file an object was defined in."""
    if ismodule(object):
        if hasattr(object, '__file__'):
            return object.__file__
    ...
    if isfunction(object):
        object = object.func_code
    ...
    if iscode(object):
        return object.co_filename
```

**Demonstration**

``` python
>>> import inspect
>>> import module
>>> inspect.getfile(module)
'package.py'
>>> module.__file__ = 'site-packages/module.py'
>>> inspect.getfile(module)
 'site-packages/module.py'
>>> inspect.getfile(module.function)
'package.py'
>>> module.function.func_code = compile('pass', 'site-packages/module.py', 'exec')
>>> inspect.getfile(module.function)
 'site-packages/module.py'
```

The only other factor is `sys.frozen`, which is also a writable attribute. Combined, it would be easy to create a test case as follows:

**Test Case**

frozen_test.py

``` python
'''Simple cache location test'''

import inspect
import sys
import unittest

from numba.caching import _UserWideCacheLocator

from module import function

class _UserWideCacheLocatorTest(unittest.TestCase):
    '''Test user-wide caching for frozen applications'''

    def test_frozen(self):
        old_code = function.func_code
        code_obj = compile('pass', 'site-packages/module.py', 'exec')
        try:
            function.func_code = code_obj

            source = inspect.getfile(function)
            # doesn't return anything, since it cannot find the module
            # fails unless the executable is frozen
            locator = _UserWideCacheLocator.from_function(function, source)
            self.assertIsNone(locator)

            sys.frozen = True
            # returns a cache locator object, only works when executable is frozen
            locator = _UserWideCacheLocator.from_function(function, source)
            self.assertIsInstance(locator, _UserWideCacheLocator)

        finally:
            function.func_code = old_code
            del sys.frozen


if __name__ == "__main__":
    unittest.main()
```

module.py

``` python
'''Dummy module'''

def function():
    '''Do nothing'''

```

I will update my pull request to merge these changes.

> I will update my pull request to merge these changes.

Great, looking forward to it!

@pitrou Do you mind if I merge that test into the test branch? Should I make a new file. I looked over the test examples and it seems my unittest is quite different from the test dispatchers numba uses.

Hi! There are caching tests already in `numba.tests.test_dispatcher` so I suggest you add your test case there.


Resolving patch:
diff --git a/numba/caching.py b/numba/caching.py
--- a/numba/caching.py
+++ b/numba/caching.py
@@ -162,7 +162,10 @@ class _SourceFileBackedLocatorMixin(object):
     """
 
     def get_source_stamp(self):
-        st = os.stat(self._py_file)
+        if getattr(sys, 'frozen', False):
+            st = os.stat(sys.executable)
+        else:
+            st = os.stat(self._py_file)
         # We use both timestamp and size as some filesystems only have second
         # granularity.
         return st.st_mtime, st.st_size
@@ -224,8 +227,8 @@ def get_cache_path(self):
 
 class _UserWideCacheLocator(_SourceFileBackedLocatorMixin, _CacheLocator):
     """
-    A locator for functions backed by a regular Python module, cached
-    into a user-wide cache directory.
+    A locator for functions backed by a regular Python module or a
+    frozen executable, cached into a user-wide cache directory.
     """
 
     def __init__(self, py_func, py_file):
@@ -234,17 +237,33 @@ def __init__(self, py_func, py_file):
         appdirs = AppDirs(appname="numba", appauthor=False)
         cache_dir = appdirs.user_cache_dir
         cache_subpath = os.path.dirname(py_file)
-        if os.name != "nt":
+        if not (os.name == "nt" or getattr(sys, 'frozen', False)):
             # On non-Windows, further disambiguate by appending the entire
             # absolute source path to the cache dir, e.g.
             # "$HOME/.cache/numba/usr/lib/.../mypkg/mysubpkg"
             # On Windows, this is undesirable because of path length limitations
+            # For frozen applications, there is no existing "full path"
+            # directory, and depends on a relocatable executable.
             cache_subpath = os.path.abspath(cache_subpath).lstrip(os.path.sep)
         self._cache_path = os.path.join(cache_dir, cache_subpath)
 
     def get_cache_path(self):
         return self._cache_path
 
+    @classmethod
+    def from_function(cls, py_func, py_file):
+        if not (os.path.exists(py_file) or getattr(sys, 'frozen', False)):
+            # Perhaps a placeholder (e.g. "<ipython-XXX>")
+            # stop function exit if frozen, since it uses a temp placeholder
+            return
+        self = cls(py_func, py_file)
+        try:
+            self.ensure_cache_path()
+        except OSError:
+            # Cannot ensure the cache directory exists or is writable
+            return
+        return self
+
 
 class _IPythonCacheLocator(_CacheLocator):
     """


Test patch:
diff --git a/numba/tests/dummy_module.py b/numba/tests/dummy_module.py
new file mode 100644
--- /dev/null
+++ b/numba/tests/dummy_module.py
@@ -0,0 +1,4 @@
+'''Dummy module'''
+
+def function():
+    '''Do nothing'''
diff --git a/numba/tests/test_dispatcher.py b/numba/tests/test_dispatcher.py
--- a/numba/tests/test_dispatcher.py
+++ b/numba/tests/test_dispatcher.py
@@ -8,6 +8,7 @@
 import sys
 import threading
 import warnings
+import inspect
 
 import numpy as np
 
@@ -19,6 +20,7 @@
 from .support import (TestCase, tag, temp_directory, import_dynamic,
                       override_env_config, capture_cache_log)
 from numba.targets import codegen
+from numba.caching import _UserWideCacheLocator
 
 import llvmlite.binding as ll
 
@@ -544,11 +546,11 @@ def foo(the_array):
         # Call inspect_cfg(signature)
         cfg = foo.inspect_cfg(signature=foo.signatures[0])
         self._check_cfg_display(cfg)
-       
+
     @unittest.skipIf(config.IS_WIN32 and not config.IS_32BITS, "access violation on 64-bit windows")
     @unittest.skipIf(config.IS_OSX, "segfault on OSX")
     def test_inspect_cfg_with_python_wrapper(self):
-        # Exercise the .inspect_cfg() including the python wrapper. 
+        # Exercise the .inspect_cfg() including the python wrapper.
         # These are minimal tests and do not fully check the correctness of
         # the function.
         @jit
@@ -562,7 +564,7 @@ def foo(the_array):
         foo(a1)
         foo(a2)
         foo(a3)
-        
+
         # Call inspect_cfg(signature, show_wrapper="python")
         cfg = foo.inspect_cfg(signature=foo.signatures[0],
                               show_wrapper="python")
@@ -945,6 +947,28 @@ def test_same_names(self):
         f = mod.renamed_function2
         self.assertPreciseEqual(f(2), 8)
 
+    def test_frozen(self):
+        from .dummy_module import function
+        old_code = function.__code__
+        code_obj = compile('pass', 'tests/dummy_module.py', 'exec')
+        try:
+            function.__code__ = code_obj
+
+            source = inspect.getfile(function)
+            # doesn't return anything, since it cannot find the module
+            # fails unless the executable is frozen
+            locator = _UserWideCacheLocator.from_function(function, source)
+            self.assertIsNone(locator)
+
+            sys.frozen = True
+            # returns a cache locator object, only works when executable is frozen
+            locator = _UserWideCacheLocator.from_function(function, source)
+            self.assertIsInstance(locator, _UserWideCacheLocator)
+
+        finally:
+            function.__code__ = old_code
+            del sys.frozen
+
     def _test_pycache_fallback(self):
         """
         With a disabled __pycache__, test there is a working fallback