Repository: sanic-org/sanic

Issue:
`AttributeError: Can't pickle local object 'register.<locals>._handler'` sanic 19.12.2; python3.8.2; sanic-cors==0.10.0.b1
Hi.

On OSX 10.14.6, python 3.8.1

    conda list sanic                                                                                                
    # packages in environment at /Users/efarrell/miniconda3/envs/LG-py38:
    #
    # Name                    Version                   Build  Channel
    pytest-sanic              1.1.2                    pypi_0    pypi
    sanic                     19.12.2                  py38_0    conda-forge
    sanic-cors                0.10.0b1                 pypi_0    pypi
    sanic-openapi3e           0.6.2                    pypi_0    pypi
    sanic-plugins-framework   0.9.0b1                  pypi_0    pypi

I cannot start an API due to 

    Traceback (most recent call last):
      File "API/src/main.py", line 613, in <module>
        main(sys.argv)
      File "API/src/main.py", line 195, in main
        app.go_fast(**sanic_run_env)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/site-packages/sanic/app.py", line 1169, in run
        serve_multiple(server_settings, workers)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/site-packages/sanic/server.py", line 997, in serve_multiple
        process.start()
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
        return Popen(process_obj)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
        super().__init__(process_obj)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
        reduction.dump(process_obj, fp)
      File "/Users/efarrell/miniconda3/envs/py38/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
        ForkingPickler(file, protocol).dump(obj)
    AttributeError: Can't pickle local object 'register.<locals>._handler'




Discussion:
For triaging: this is now a problem because Python 3.8 changes the default mode of `multiprocessing` on MacOS to `spawn` instead of `fork`. This requires all parameters passed to Sanic worker processed to be picklable, and now there is an object that isn't.

This could be worked around by forcing `fork` mode instead of the new default, but there really shouldn't be objects that cannot be pickled there because Windows cannot fork and probably could not spawn any workers then either.

Hi @Tronic - many thanks for this. We write code on a mixture of OSX and Linux, and deploy to Linux, so in our particular instance, not being able to fork on Windows is not going to cause a problem.

We _do_ have complex objects being passed to the workers, but I had _though_ that those with connections to databases (which we know cannot be pickled) were created only in `after_server_start` handlers. Will check. Many thanks.
@Tronic a comment related to this:
around the release of Sanic 18.12 we wanted to get multiprocessing working on Windows, so I put in some effort and made _all_ of Sanic pickle-able.

I thought I added tests to ensure a sanic app could be fully pickled and unpickled without error.

Looks like now some code has been introduced at some point that has a local sub_function, which isn't pickleable, or perhaps its only a problem on Python 3.8, which I think our test suite isn't running properly yet.
@ashleysommer Would it be viable to move most app initialisation to within workers, instead of pickling it from the main process? Only server socket opening really needs to be in the master process, while app definition and other user code could be run in workers. Getting this done would need quite a bit of refactoring, but barring that, it could be the clean solution to this. Doing so would avoid troublesome pickling and the `freeze_support` hack of `multiprocessing` that tries to avoid the re-execution of main function and other such code when child processes are run (but it still re-runs any module level code reached prior to that freeze function call).
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this  is incorrect, please respond with an update. Thank you for your contributions.

This still needs to be fixed.
@Tronic I have some (hopefully good) news. 

Our project was busy for a few months but last week we got around to putting effort back into our py38 upgrade. Somewhere along the lines we had also re-checked all code to ensure that calls to set up connections to external databases (we have three external connection pools to three different kinds of database) were made in `@bp.listener("after_server_start")` methods in our blueprints. There had been at least one "before_server_start" before.

We were happily surprised to find that due to ensuring we created our connections "after_server_start" we can start the sanic `19.12.2` API just fine on OSX `10.14.6` with py3.8.2 (and sanic-cors `0.10.0.post3` - but I do not believe we did anything special with that).

At _this_ stage I can think of two possible next steps:

1: a note in the docs (and a review of existing examples) to state that connections to databases should be created "after_server_start"
2: a note in the docs that states that "after_server_start" happens after the main thread creates the workers but before allowing them to take traffic. It seems somewhat obvious in hind-sight, but I struggled for a little to grok this.

@Tronic - what's the best way to proceed? 

I don't think that before vs. after server start should make a difference for pickling. Both are done in worker processes, and only `await loop.create_server(...)` occurs between them. Also, what happens inside those handlers should not affect anything.

According to your traceback, the `_handler` function defined within `sanic.static.register` is what cannot be pickled, which makes sense because pickling AFAIK doesn't work for anything defined inside functions. Why this is *now* working is unknown to me, and more time would be needed to fully investigate. Perhaps different Python or Sanic versions, or perhaps somehow it no longer uses that `_handler` wrapper.
Ah - I clearly still don't understand the inner working yet then.  

I can't relate the string `AttributeError: Can't pickle local object 'register.<locals>._handler'` to our code - at least not directly nor in an obvious manner. Do you happen to have any pointers as to how to chase this particular thing down? 
Are you no longer using `app.static` (for serving static files)? That would explain it because this appears to be the only place where that is used.
I believe this could be fixed quite easily by rewriting `sanic/static.py` to use `functools.partial` to pass `use_modified_since`, `use_content_range`, `stream_large_files` and `content_type` to `_handler` instead of it being a local function.

Pinging anyone who cares: pull requests are welcome!

Resolving patch:
diff --git a/sanic/static.py b/sanic/static.py
--- a/sanic/static.py
+++ b/sanic/static.py
@@ -1,3 +1,4 @@
+from functools import partial, wraps
 from mimetypes import guess_type
 from os import path
 from re import sub
@@ -15,6 +16,89 @@
 from sanic.response import HTTPResponse, file, file_stream
 
 
+async def _static_request_handler(
+    file_or_directory,
+    use_modified_since,
+    use_content_range,
+    stream_large_files,
+    request,
+    content_type=None,
+    file_uri=None,
+):
+    # Using this to determine if the URL is trying to break out of the path
+    # served.  os.path.realpath seems to be very slow
+    if file_uri and "../" in file_uri:
+        raise InvalidUsage("Invalid URL")
+    # Merge served directory and requested file if provided
+    # Strip all / that in the beginning of the URL to help prevent python
+    # from herping a derp and treating the uri as an absolute path
+    root_path = file_path = file_or_directory
+    if file_uri:
+        file_path = path.join(file_or_directory, sub("^[/]*", "", file_uri))
+
+    # URL decode the path sent by the browser otherwise we won't be able to
+    # match filenames which got encoded (filenames with spaces etc)
+    file_path = path.abspath(unquote(file_path))
+    if not file_path.startswith(path.abspath(unquote(root_path))):
+        raise FileNotFound(
+            "File not found", path=file_or_directory, relative_url=file_uri
+        )
+    try:
+        headers = {}
+        # Check if the client has been sent this file before
+        # and it has not been modified since
+        stats = None
+        if use_modified_since:
+            stats = await stat_async(file_path)
+            modified_since = strftime(
+                "%a, %d %b %Y %H:%M:%S GMT", gmtime(stats.st_mtime)
+            )
+            if request.headers.get("If-Modified-Since") == modified_since:
+                return HTTPResponse(status=304)
+            headers["Last-Modified"] = modified_since
+        _range = None
+        if use_content_range:
+            _range = None
+            if not stats:
+                stats = await stat_async(file_path)
+            headers["Accept-Ranges"] = "bytes"
+            headers["Content-Length"] = str(stats.st_size)
+            if request.method != "HEAD":
+                try:
+                    _range = ContentRangeHandler(request, stats)
+                except HeaderNotFound:
+                    pass
+                else:
+                    del headers["Content-Length"]
+                    for key, value in _range.headers.items():
+                        headers[key] = value
+        headers["Content-Type"] = (
+            content_type or guess_type(file_path)[0] or "text/plain"
+        )
+        if request.method == "HEAD":
+            return HTTPResponse(headers=headers)
+        else:
+            if stream_large_files:
+                if type(stream_large_files) == int:
+                    threshold = stream_large_files
+                else:
+                    threshold = 1024 * 1024
+
+                if not stats:
+                    stats = await stat_async(file_path)
+                if stats.st_size >= threshold:
+                    return await file_stream(
+                        file_path, headers=headers, _range=_range
+                    )
+            return await file(file_path, headers=headers, _range=_range)
+    except ContentRangeError:
+        raise
+    except Exception:
+        raise FileNotFound(
+            "File not found", path=file_or_directory, relative_url=file_uri
+        )
+
+
 def register(
     app,
     uri,
@@ -56,86 +140,21 @@ def register(
     if not path.isfile(file_or_directory):
         uri += "<file_uri:" + pattern + ">"
 
-    async def _handler(request, file_uri=None):
-        # Using this to determine if the URL is trying to break out of the path
-        # served.  os.path.realpath seems to be very slow
-        if file_uri and "../" in file_uri:
-            raise InvalidUsage("Invalid URL")
-        # Merge served directory and requested file if provided
-        # Strip all / that in the beginning of the URL to help prevent python
-        # from herping a derp and treating the uri as an absolute path
-        root_path = file_path = file_or_directory
-        if file_uri:
-            file_path = path.join(
-                file_or_directory, sub("^[/]*", "", file_uri)
-            )
-
-        # URL decode the path sent by the browser otherwise we won't be able to
-        # match filenames which got encoded (filenames with spaces etc)
-        file_path = path.abspath(unquote(file_path))
-        if not file_path.startswith(path.abspath(unquote(root_path))):
-            raise FileNotFound(
-                "File not found", path=file_or_directory, relative_url=file_uri
-            )
-        try:
-            headers = {}
-            # Check if the client has been sent this file before
-            # and it has not been modified since
-            stats = None
-            if use_modified_since:
-                stats = await stat_async(file_path)
-                modified_since = strftime(
-                    "%a, %d %b %Y %H:%M:%S GMT", gmtime(stats.st_mtime)
-                )
-                if request.headers.get("If-Modified-Since") == modified_since:
-                    return HTTPResponse(status=304)
-                headers["Last-Modified"] = modified_since
-            _range = None
-            if use_content_range:
-                _range = None
-                if not stats:
-                    stats = await stat_async(file_path)
-                headers["Accept-Ranges"] = "bytes"
-                headers["Content-Length"] = str(stats.st_size)
-                if request.method != "HEAD":
-                    try:
-                        _range = ContentRangeHandler(request, stats)
-                    except HeaderNotFound:
-                        pass
-                    else:
-                        del headers["Content-Length"]
-                        for key, value in _range.headers.items():
-                            headers[key] = value
-            headers["Content-Type"] = (
-                content_type or guess_type(file_path)[0] or "text/plain"
-            )
-            if request.method == "HEAD":
-                return HTTPResponse(headers=headers)
-            else:
-                if stream_large_files:
-                    if type(stream_large_files) == int:
-                        threshold = stream_large_files
-                    else:
-                        threshold = 1024 * 1024
-
-                    if not stats:
-                        stats = await stat_async(file_path)
-                    if stats.st_size >= threshold:
-                        return await file_stream(
-                            file_path, headers=headers, _range=_range
-                        )
-                return await file(file_path, headers=headers, _range=_range)
-        except ContentRangeError:
-            raise
-        except Exception:
-            raise FileNotFound(
-                "File not found", path=file_or_directory, relative_url=file_uri
-            )
-
     # special prefix for static files
     if not name.startswith("_static_"):
         name = f"_static_{name}"
 
+    _handler = wraps(_static_request_handler)(
+        partial(
+            _static_request_handler,
+            file_or_directory,
+            use_modified_since,
+            use_content_range,
+            stream_large_files,
+            content_type=content_type,
+        )
+    )
+
     app.route(
         uri,
         methods=["GET", "HEAD"],


Test patch:
diff --git a/tests/test_multiprocessing.py b/tests/test_multiprocessing.py
--- a/tests/test_multiprocessing.py
+++ b/tests/test_multiprocessing.py
@@ -87,3 +87,14 @@ def test_pickle_app_with_bp(app, protocol):
     request, response = up_p_app.test_client.get("/")
     assert up_p_app.is_request_stream is False
     assert response.text == "Hello"
+
+@pytest.mark.parametrize("protocol", [3, 4])
+def test_pickle_app_with_static(app, protocol):
+    app.route("/")(handler)
+    app.static('/static', "/tmp/static")
+    p_app = pickle.dumps(app, protocol=protocol)
+    del app
+    up_p_app = pickle.loads(p_app)
+    assert up_p_app
+    request, response = up_p_app.test_client.get("/static/missing.txt")
+    assert response.status == 404