Repository: Kinto/kinto

Issue:
Make serve recipe is failing
Seems related to #1087
```
gsurita-30820:kinto gsurita$ make serve 
.venv/bin/kinto --ini config/kinto.ini init
Select the backend you would like to use: (1 - postgresql, 2 - redis, default - memory) 
INFO   Created config /Users/gsurita/kinto/config/kinto.ini
.venv/bin/kinto --ini config/kinto.ini migrate
/Users/gsurita/kinto/kinto/core/initialization.py:567: UserWarning: HTTPS is not enabled
  warnings.warn('HTTPS is not enabled')
INFO   Running kinto 6.0.0.dev0. 
echo '{"name":"kinto","version":"5.3.1-118-g5fc2dedf","source":"https://github.com/gabisurita/kinto/","commit":"5fc2dedfc0a75d9f337ccdbcc935fa051f377164"}' > version.json
.venv/bin/kinto --ini config/kinto.ini start --reload
usage: kinto [-h] [-n NAME] [-s SERVER_TYPE] [--server-name SECTION_NAME]
             [--reload] [--reload-interval RELOAD_INTERVAL] [-b] [-v] [-q]
             [config_uri] [config_vars [config_vars ...]]
kinto: error: unrecognized arguments: http_port=8888
make: *** [serve] Error 2
```


Discussion:
https://github.com/Pylons/pyramid/blob/master/HISTORY.txt#L45
It seems like `pserve` had a [massive rewrite upstream](https://github.com/Pylons/pyramid/pull/2805) which breaks us in a couple of ways.

- It seems like moving from optparse to argparse (https://github.com/Pylons/pyramid/pull/2864/) changed the semantics of an argument string like `['pserve', 'config.ini', '--reload', 'http_port=8080']`. For some reason, the argument parser fails to parse this, but it will parse correctly if `--reload` comes either before the INI file or after the variable.

- It seems like the implementation of `pserve` used to re-invoke the same command as `sys.argv`, but now specifically recursively invokes `pserve` from inside a "monitored" reloader process. The second (inner) execution then tries to parse `kinto serve` arguments with `pserve` and totally fails.

It seems like we'll have to rewrite our `kinto serve` code to make it work with Pyramid 1.8. Since master is kind of broken right now, I think we should revert the change until someone has time to do that. In the meantime, @gabisurita points out that if you remove `--reload` from the `Makefile`, then everything works OK (but without reloading). We could make that change instead and create an issue to track it.
After investigation, it seems that `--reload` call `hupper` which takes `sys.argv` instead of the previous argv.
I made a pull-request upstream that should handle the matter: https://github.com/Pylons/pyramid/pull/2962/files

Resolving patch:
diff --git a/kinto/__main__.py b/kinto/__main__.py
--- a/kinto/__main__.py
+++ b/kinto/__main__.py
@@ -23,21 +23,6 @@ def main(args=None):
 
     parser = argparse.ArgumentParser(description="Kinto Command-Line "
                                                  "Interface")
-    # XXX: deprecate this option, unnatural as first argument.
-    parser.add_argument('--ini',
-                        help='Application configuration file',
-                        dest='ini_file',
-                        required=False,
-                        default=DEFAULT_CONFIG_FILE)
-
-    parser.add_argument('-q', '--quiet', action='store_const',
-                        const=logging.CRITICAL, dest='verbosity',
-                        help='Show only critical errors.')
-
-    parser.add_argument('--debug', action='store_const',
-                        const=logging.DEBUG, dest='verbosity',
-                        help='Show all messages, including debug messages.')
-
     commands = ('init', 'start', 'migrate', 'delete-collection', 'version')
     subparsers = parser.add_subparsers(title='subcommands',
                                        description='Main Kinto CLI commands',
@@ -49,6 +34,20 @@ def main(args=None):
         subparser = subparsers.add_parser(command)
         subparser.set_defaults(which=command)
 
+        subparser.add_argument('--ini',
+                               help='Application configuration file',
+                               dest='ini_file',
+                               required=False,
+                               default=DEFAULT_CONFIG_FILE)
+
+        subparser.add_argument('-q', '--quiet', action='store_const',
+                               const=logging.CRITICAL, dest='verbosity',
+                               help='Show only critical errors.')
+
+        subparser.add_argument('-v', '--debug', action='store_const',
+                               const=logging.DEBUG, dest='verbosity',
+                               help='Show all messages, including debug messages.')
+
         if command == 'init':
             subparser.add_argument('--backend',
                                    help='{memory,redis,postgresql}',
@@ -145,11 +144,20 @@ def main(args=None):
                                          parsed_args['collection'])
 
     elif which_command == 'start':
-        pserve_argv = ['pserve', config_file]
+        pserve_argv = ['pserve']
+
         if parsed_args['reload']:
             pserve_argv.append('--reload')
+
+        if level == logging.DEBUG:
+            pserve_argv.append('-v')
+
+        if level == logging.CRITICAL:
+            pserve_argv.append('-q')
+
+        pserve_argv.append(config_file)
         pserve_argv.append('http_port={}'.format(parsed_args['port']))
-        pserve.main(pserve_argv)
+        pserve.main(argv=pserve_argv)
 
     else:
         print(__version__)


Test patch:
diff --git a/tests/test_main.py b/tests/test_main.py
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -23,7 +23,7 @@ def setUp(self):
             pass
 
     def test_cli_init_generates_configuration(self):
-        res = main(['--ini', TEMP_KINTO_INI, 'init', '--backend', 'memory'])
+        res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
         assert res == 0
         assert os.path.exists(TEMP_KINTO_INI)
 
@@ -31,24 +31,22 @@ def test_cli_init_returns_if_file_exists(self):
         with open(TEMP_KINTO_INI, 'w') as f:
             f.write("exists")
         with mock.patch('sys.stderr', new_callable=StringIO) as mock_stderr:
-            res = main(['--ini', TEMP_KINTO_INI, 'init',
-                        '--backend', 'memory'])
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             assert res == 1
             content = mock_stderr.getvalue()
             assert '{} already exists.'.format(TEMP_KINTO_INI) in content
 
     def test_cli_init_asks_for_backend_if_not_specified(self):
         with mock.patch("kinto.__main__.input", create=True, return_value="2"):
-            res = main(['--ini', TEMP_KINTO_INI, 'init'])
+            res = main(['init', '--ini', TEMP_KINTO_INI])
             assert res == 0
         with open(TEMP_KINTO_INI) as f:
             content = f.read()
         assert 'redis' in content
 
     def test_cli_init_asks_until_backend_is_valid(self):
-        with mock.patch("kinto.__main__.input", create=True,
-                        side_effect=["10", "2"]):
-            res = main(['--ini', TEMP_KINTO_INI, 'init'])
+        with mock.patch("kinto.__main__.input", create=True, side_effect=["10", "2"]):
+            res = main(['init', '--ini', TEMP_KINTO_INI])
             assert res == 0
             with open(TEMP_KINTO_INI) as f:
                 content = f.read()
@@ -57,8 +55,8 @@ def test_cli_init_asks_until_backend_is_valid(self):
     def test_fails_if_not_enough_args(self):
         with mock.patch('sys.stderr', new_callable=StringIO) as mock_stderr:
             with pytest.raises(SystemExit) as excinfo:
-                main(['--ini', TEMP_KINTO_INI])
-            assert 'INI_FILE' in mock_stderr.getvalue()
+                main([])
+            assert 'arguments are required: subcommand' in mock_stderr.getvalue()
             assert excinfo.value.code == 2
 
     def test_cli_init_installs_postgresql_dependencies_if_needed(self):
@@ -72,9 +70,8 @@ def psycopg2_missing(name, *args, **kwargs):
 
         with mock.patch('builtins.__import__', side_effect=psycopg2_missing):
             with mock.patch('pip.main', return_value=None) as mocked_pip:
-                with mock.patch("kinto.__main__.input", create=True,
-                                return_value="1"):
-                    res = main(['--ini', TEMP_KINTO_INI, 'init'])
+                with mock.patch("kinto.__main__.input", create=True, return_value="1"):
+                    res = main(['init', '--ini', TEMP_KINTO_INI])
                     assert res == 0
                     assert mocked_pip.call_count == 1
 
@@ -89,15 +86,13 @@ def redis_missing(name, *args, **kwargs):
 
         with mock.patch('builtins.__import__', side_effect=redis_missing):
             with mock.patch('pip.main', return_value=None) as mocked_pip:
-                with mock.patch("kinto.__main__.input", create=True,
-                                return_value="2"):
-                    res = main(['--ini', TEMP_KINTO_INI, 'init'])
+                with mock.patch("kinto.__main__.input", create=True, return_value="2"):
+                    res = main(['init', '--ini', TEMP_KINTO_INI])
                     assert res == 0
                     assert mocked_pip.call_count == 1
 
     def test_main_takes_sys_argv_by_default(self):
-        testargs = ["prog", "--ini", TEMP_KINTO_INI, 'init',
-                    '--backend', 'memory']
+        testargs = ['prog', 'init', '--ini', TEMP_KINTO_INI, '--backend', 'memory']
         with mock.patch.object(sys, 'argv', testargs):
             main()
 
@@ -107,20 +102,18 @@ def test_main_takes_sys_argv_by_default(self):
 
     def test_cli_migrate_command_runs_init_schema(self):
         with mock.patch('kinto.__main__.scripts.migrate') as mocked_migrate:
-            res = main(['--ini', TEMP_KINTO_INI, 'init',
-                        '--backend', 'memory'])
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             assert res == 0
-            res = main(['--ini', TEMP_KINTO_INI, 'migrate'])
+            res = main(['migrate', '--ini', TEMP_KINTO_INI])
             assert res == 0
             assert mocked_migrate.call_count == 1
 
     def test_cli_delete_collection_run_delete_collection_script(self):
         with mock.patch('kinto.__main__.scripts.delete_collection') as del_col:
             del_col.return_value = mock.sentinel.del_col_code
-            res = main(['--ini', TEMP_KINTO_INI, 'init',
-                        '--backend', 'memory'])
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             assert res == 0
-            res = main(['--ini', TEMP_KINTO_INI, 'delete-collection',
+            res = main(['delete-collection', '--ini', TEMP_KINTO_INI,
                         '--bucket', 'test_bucket',
                         '--collection', 'test_collection'])
             assert res == mock.sentinel.del_col_code
@@ -128,22 +121,38 @@ def test_cli_delete_collection_run_delete_collection_script(self):
 
     def test_cli_start_runs_pserve(self):
         with mock.patch('kinto.__main__.pserve.main') as mocked_pserve:
-            res = main(['--ini', TEMP_KINTO_INI, 'init',
-                        '--backend', 'memory'])
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             assert res == 0
-            res = main(['--ini', TEMP_KINTO_INI, 'start'])
+            res = main(['start', '--ini', TEMP_KINTO_INI])
             assert res == 0
             assert mocked_pserve.call_count == 1
 
+    def test_cli_start_with_quiet_option_runs_pserve_with_quiet(self):
+        with mock.patch('kinto.__main__.pserve.main') as mocked_pserve:
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
+            assert res == 0
+            res = main(['start', '-q', '--ini', TEMP_KINTO_INI])
+            assert res == 0
+            assert mocked_pserve.call_count == 1
+            assert mocked_pserve.call_args[1]['argv'][1] == '-q'
+
+    def test_cli_start_with_verbose_option_runs_pserve_with_verbose(self):
+        with mock.patch('kinto.__main__.pserve.main') as mocked_pserve:
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
+            assert res == 0
+            res = main(['start', '-v', '--ini', TEMP_KINTO_INI])
+            assert res == 0
+            assert mocked_pserve.call_count == 1
+            assert mocked_pserve.call_args[1]['argv'][1] == '-v'
+
     def test_cli_start_with_reload_runs_pserve_with_reload(self):
         with mock.patch('kinto.__main__.pserve.main') as mocked_pserve:
-            res = main(['--ini', TEMP_KINTO_INI, 'init',
-                        '--backend', 'memory'])
+            res = main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             assert res == 0
-            res = main(['--ini', TEMP_KINTO_INI, 'start', '--reload'])
+            res = main(['start', '--reload', '--ini', TEMP_KINTO_INI])
             assert res == 0
             assert mocked_pserve.call_count == 1
-            assert '--reload' in mocked_pserve.call_args[0][0]
+            assert '--reload' in mocked_pserve.call_args[1]['argv']
 
     def test_cli_can_display_kinto_version(self):
         with mock.patch('sys.stdout', new_callable=StringIO) as mock_stdout:
@@ -153,24 +162,21 @@ def test_cli_can_display_kinto_version(self):
 
     def test_cli_can_configure_logger_in_quiet(self):
         with mock.patch('kinto.__main__.logging') as mocked_logging:
-            main(['-q', '--ini', TEMP_KINTO_INI, 'init',
-                        '--backend', 'memory'])
+            main(['init', '-q', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             mocked_logging.basicConfig.assert_called_with(
                 level=mocked_logging.CRITICAL,
                 format=DEFAULT_LOG_FORMAT)
 
     def test_cli_can_configure_logger_in_debug(self):
         with mock.patch('kinto.__main__.logging') as mocked_logging:
-            main(['--debug', '--ini', TEMP_KINTO_INI, 'init',
-                  '--backend', 'memory'])
+            main(['init', '--debug', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             mocked_logging.basicConfig.assert_called_with(
                 level=mocked_logging.DEBUG,
                 format=DEFAULT_LOG_FORMAT)
 
     def test_cli_use_default_logging_logger(self):
         with mock.patch('kinto.__main__.logging') as mocked_logging:
-            main(['--ini', TEMP_KINTO_INI, 'init',
-                  '--backend', 'memory'])
+            main(['init', '--ini', TEMP_KINTO_INI, '--backend', 'memory'])
             mocked_logging.basicConfig.assert_called_with(
                 level=logging.INFO,
                 format=DEFAULT_LOG_FORMAT)