Repository: tornadoweb/tornado

Issue:
autoreload: Fix argv preservation
`autoreload` currently has a wrapper mode (e.g. `python -m tornado.autoreload -m tornado.test`) for scripts, and an in-process mode (enabled by `Application(..., debug=True)`). It's useful to combine these, since the wrapper can catch syntax errors that cause the process to abort before entering its IOLoop. However, this doesn't work as well as it should, because the `main` wrapper only restores `sys.argv` if the process exits, meaning the `-m tornado.autoreload` flags are lost if the inner autoreload fires. The original argv needs to be stored in a global when `autoreload` is `__main__`, so that it can be used in `_reload()`.
fix autoreload argv perservation
#1983 
I think I just follow your ideas. The testcase is done by two process one is to touch the `__init__.py` file and the other is the `autoreload` process, but I not sure if it is appropriate because it kind of integration test not unit test.

I have another version of testcase which work for python2 because for python>=3.4, the `tornado._reload()` will run the module use current `spec` which is `testapp` not the `tornado._reload()`.

```
class AutoreloadTest(unittest.TestCase):

    def test_reload_module_with_argv_preservation_only_for_python2(self):
        main = """\
import os
import sys
from tornado import autoreload

# This import will fail if path is not set up correctly
import testapp
print(autoreload._original_argv)
if 'TESTAPP_STARTED' not in os.environ:
    os.environ['TESTAPP_STARTED'] = '1'
    sys.stdout.flush()
    autoreload._reload()
else:
    sys.stdout.flush()
    autoreload.add_reload_hook(lambda: os._exit(0))
    autoreload._reload()
"""

        # Create temporary test application
        path = mkdtemp()
        os.mkdir(os.path.join(path, 'testapp'))
        open(os.path.join(path, 'testapp/__init__.py'), 'w').close()
        with open(os.path.join(path, 'testapp/__main__.py'), 'w') as f:
            f.write(main)

        # Make sure the tornado module under test is available to the test
        # application
        pythonpath = os.getcwd()
        if 'PYTHONPATH' in os.environ:
            pythonpath += os.pathsep + os.environ['PYTHONPATH']

        p = Popen(
            [sys.executable, '-m', 'tornado.autoreload', '-m', 'testapp'], stdout=subprocess.PIPE,
            cwd=path, env=dict(os.environ, PYTHONPATH=pythonpath),
            universal_newlines=True)
        out = p.communicate()[0]
        self.assertEqual((str([os.path.join(os.path.dirname(os.path.abspath(tornado.autoreload.__file__)),
                                            'autoreload.py'), '-m', 'testapp']) + '\n') * 2, out)
```


Discussion:
This would be really handy addition and would enable fast iteration within running container.
I think we can set `setattr(sys.modules['__main__'], '__original_argv__', [....])` in `main()`, so we can use the original argv in the `_reload()`. The idea is the same as the `sys.modules['__main__'].spec`. Is that a way out ?
I don't think we need to use `setattr()` on `sys.modules`, we just need to use the `global` keyword. I have a branch that does this; I can't remember why I never pushed/merged it. 
I think using the `global _original_argv` in `main()` won't work. If I type the command `python -m tornado app/app.py`, in the `main()`, the namespace of `__name__` is `__main__` and in the `_reload()`, the namespace of `__name__` is `tornado.autoload`. The `__main__` has its own namespace, so we cannot refer to the same `_original_argv` in `_reload()`. Maybe we can solve this by:
1) Use module level variables.
2) Use `import inspect` to change the variable in different namespaces.
 



Ah, that's right. The `__main__` namespace is tricky. My branch doesn't use `setattr()`, but it does `import tornado.autoreload` and set attributes there:

```diff
autoreload 531fe3fe520e4df515df75107348f6a6ba488fd2
Author:     Ben Darnell <ben@bendarnell.com>
AuthorDate: Sun Jul 10 10:03:01 2016 -0400
Commit:     Ben Darnell <ben@bendarnell.com>
CommitDate: Sun Jul 10 10:03:01 2016 -0400

Parent:     973e83e3 demos/benchmark: use current() instead of instance()
Merged:     autoreload branch1.2 branch3.0 branch3.2 branch4.1 branch4.3
Containing: autoreload
Follows:    v4.4.0b1 (2)

autoreload: improve use as main

1 file changed, 18 insertions(+), 4 deletions(-)
tornado/autoreload.py | 22 ++++++++++++++++++----

modified   tornado/autoreload.py
@@ -113,6 +113,8 @@ _watched_files = set()
 _reload_hooks = []
 _reload_attempted = False
 _io_loops = weakref.WeakKeyDictionary()  # type: ignore
+_autoreload_is_main = False
+_original_argv = None
 
 
 def start(io_loop=None, check_time=500):
@@ -224,12 +226,17 @@ def _reload():
             not os.environ.get("PYTHONPATH", "").startswith(path_prefix)):
         os.environ["PYTHONPATH"] = (path_prefix +
                                     os.environ.get("PYTHONPATH", ""))
+    argv = [sys.executable]
+    if _autoreload_is_main:
+        argv.extend(_original_argv)
+    else:
+        argv.extend(sys.argv)
     if not _has_execv:
-        subprocess.Popen([sys.executable] + sys.argv)
+        subprocess.Popen(argv)
         sys.exit(0)
     else:
         try:
-            os.execv(sys.executable, [sys.executable] + sys.argv)
+            os.execv(sys.executable, argv)
         except OSError:
             # Mac OS X versions prior to 10.6 do not support execv in
             # a process that contains multiple threads.  Instead of
@@ -242,8 +249,7 @@ def _reload():
             # Unfortunately the errno returned in this case does not
             # appear to be consistent, so we can't easily check for
             # this error specifically.
-            os.spawnv(os.P_NOWAIT, sys.executable,
-                      [sys.executable] + sys.argv)
+            os.spawnv(os.P_NOWAIT, sys.executable, argv)
             # At this point the IOLoop has been closed and finally
             # blocks will experience errors if we allow the stack to
             # unwind, so just exit uncleanly.
@@ -269,7 +275,15 @@ def main():
     can catch import-time problems like syntax errors that would otherwise
     prevent the script from reaching its call to `wait`.
     """
+    # Remember that we were launched with autoreload as main.
+    # The main module can be tricky; set the variables both in our globals
+    # (which may be __main__) and the real importable version.
+    import tornado.autoreload
+    global _autoreload_is_main
+    global _original_argv
+    tornado.autoreload._autoreload_is_main = _autoreload_is_main = True
     original_argv = sys.argv
+    tornado.autoreload._original_argv = _original_argv = original_argv
     sys.argv = sys.argv[:]
     if len(sys.argv) >= 3 and sys.argv[1] == "-m":
         mode = "module"
```

But even this can be avoided, I think, if we just change the end of the file to:

```
if __name__ == '__main__':
    # Ensure that any global variables accessed by main() are in the module namespace
    # instead of the __main__ namespace
    import tornado.autoreload
    tornado.autoreload.main()
```


Resolving patch:
diff --git a/tornado/autoreload.py b/tornado/autoreload.py
--- a/tornado/autoreload.py
+++ b/tornado/autoreload.py
@@ -107,6 +107,9 @@
 _reload_hooks = []
 _reload_attempted = False
 _io_loops = weakref.WeakKeyDictionary()  # type: ignore
+_autoreload_is_main = False
+_original_argv = None
+_original_spec = None
 
 
 def start(check_time=500):
@@ -214,11 +217,15 @@ def _reload():
     # __spec__ is not available (Python < 3.4), check instead if
     # sys.path[0] is an empty string and add the current directory to
     # $PYTHONPATH.
-    spec = getattr(sys.modules['__main__'], '__spec__', None)
-    if spec:
-        argv = ['-m', spec.name] + sys.argv[1:]
+    if _autoreload_is_main:
+        spec = _original_spec
+        argv = _original_argv
     else:
+        spec = getattr(sys.modules['__main__'], '__spec__', None)
         argv = sys.argv
+    if spec:
+        argv = ['-m', spec.name] + argv[1:]
+    else:
         path_prefix = '.' + os.pathsep
         if (sys.path[0] == '' and
                 not os.environ.get("PYTHONPATH", "").startswith(path_prefix)):
@@ -226,7 +233,7 @@ def _reload():
                                         os.environ.get("PYTHONPATH", ""))
     if not _has_execv:
         subprocess.Popen([sys.executable] + argv)
-        sys.exit(0)
+        os._exit(0)
     else:
         try:
             os.execv(sys.executable, [sys.executable] + argv)
@@ -269,7 +276,17 @@ def main():
     can catch import-time problems like syntax errors that would otherwise
     prevent the script from reaching its call to `wait`.
     """
+    # Remember that we were launched with autoreload as main.
+    # The main module can be tricky; set the variables both in our globals
+    # (which may be __main__) and the real importable version.
+    import tornado.autoreload
+    global _autoreload_is_main
+    global _original_argv, _original_spec
+    tornado.autoreload._autoreload_is_main = _autoreload_is_main = True
     original_argv = sys.argv
+    tornado.autoreload._original_argv = _original_argv = original_argv
+    original_spec = getattr(sys.modules['__main__'], '__spec__', None)
+    tornado.autoreload._original_spec = _original_spec = original_spec
     sys.argv = sys.argv[:]
     if len(sys.argv) >= 3 and sys.argv[1] == "-m":
         mode = "module"


Test patch:
diff --git a/tornado/test/autoreload_test.py b/tornado/test/autoreload_test.py
--- a/tornado/test/autoreload_test.py
+++ b/tornado/test/autoreload_test.py
@@ -1,14 +1,19 @@
 from __future__ import absolute_import, division, print_function
 import os
+import shutil
 import subprocess
 from subprocess import Popen
 import sys
 from tempfile import mkdtemp
+import time
 
 from tornado.test.util import unittest
 
 
-MAIN = """\
+class AutoreloadTest(unittest.TestCase):
+
+    def test_reload_module(self):
+        main = """\
 import os
 import sys
 
@@ -24,15 +29,13 @@
     autoreload._reload()
 """
 
-
-class AutoreloadTest(unittest.TestCase):
-    def test_reload_module(self):
         # Create temporary test application
         path = mkdtemp()
+        self.addCleanup(shutil.rmtree, path)
         os.mkdir(os.path.join(path, 'testapp'))
         open(os.path.join(path, 'testapp/__init__.py'), 'w').close()
         with open(os.path.join(path, 'testapp/__main__.py'), 'w') as f:
-            f.write(MAIN)
+            f.write(main)
 
         # Make sure the tornado module under test is available to the test
         # application
@@ -46,3 +49,64 @@ def test_reload_module(self):
             universal_newlines=True)
         out = p.communicate()[0]
         self.assertEqual(out, 'Starting\nStarting\n')
+
+    def test_reload_wrapper_preservation(self):
+        # This test verifies that when `python -m tornado.autoreload`
+        # is used on an application that also has an internal
+        # autoreload, the reload wrapper is preserved on restart.
+        main = """\
+import os
+import sys
+
+# This import will fail if path is not set up correctly
+import testapp
+
+if 'tornado.autoreload' not in sys.modules:
+    raise Exception('started without autoreload wrapper')
+
+import tornado.autoreload
+
+print('Starting')
+sys.stdout.flush()
+if 'TESTAPP_STARTED' not in os.environ:
+    os.environ['TESTAPP_STARTED'] = '1'
+    # Simulate an internal autoreload (one not caused
+    # by the wrapper).
+    tornado.autoreload._reload()
+else:
+    # Exit directly so autoreload doesn't catch it.
+    os._exit(0)
+"""
+
+        # Create temporary test application
+        path = mkdtemp()
+        os.mkdir(os.path.join(path, 'testapp'))
+        self.addCleanup(shutil.rmtree, path)
+        init_file = os.path.join(path, 'testapp', '__init__.py')
+        open(init_file, 'w').close()
+        main_file = os.path.join(path, 'testapp', '__main__.py')
+        with open(main_file, 'w') as f:
+            f.write(main)
+
+        # Make sure the tornado module under test is available to the test
+        # application
+        pythonpath = os.getcwd()
+        if 'PYTHONPATH' in os.environ:
+            pythonpath += os.pathsep + os.environ['PYTHONPATH']
+
+        autoreload_proc = Popen(
+            [sys.executable, '-m', 'tornado.autoreload', '-m', 'testapp'],
+            stdout=subprocess.PIPE, cwd=path,
+            env=dict(os.environ, PYTHONPATH=pythonpath),
+            universal_newlines=True)
+
+        for i in range(20):
+            if autoreload_proc.poll() is not None:
+                break
+            time.sleep(0.1)
+        else:
+            autoreload_proc.kill()
+            raise Exception("subprocess failed to terminate")
+
+        out = autoreload_proc.communicate()[0]
+        self.assertEqual(out, 'Starting\n' * 2)