Repository: EducationalTestingService/rsmtool

Issue:
Wishlist for interactive generation
Few things to add to interactive generation:
- Add `-i` flag (in addition  to ``--interactive``)
- Make subgroups an option rather than flag: "Do you have subgroups?" I always forget to set the flag.
- Add a question at the end: "Save output to a file?" 

Discussion:

Patch:
diff --git a/doc/automated_configuration.rst b/doc/automated_configuration.rst
index 50c673d6..4f714ac0 100644
--- a/doc/automated_configuration.rst
+++ b/doc/automated_configuration.rst
@@ -60,6 +60,8 @@ We end with a list of important things to note about interactive generation:
 
 - Optional fields will accept blank inputs since they have default values that will be used if no user input is provided. In some cases, default values are shown underlined in parentheses. 
 
+- You can also use ``-i`` as an alias for ``--interactive`` and ``-g`` as an alias for ``--subgroups``. So, for example, if you want to interactively generate a configuration file with subgroups for ``rsmtool``, just run ``rsmtool generate -ig`` instead of ``rsmtool generate --interactive --subgroups``.
+
 - The configuration files generated interactively contain comments (as indicated by ``// ...``). While RSMTool handles JSON files with comments just fine, you may need to remove the comments manually if you wish to use these files outside of RSMTool.
 
 Non-interactive Generation
@@ -87,7 +89,7 @@ Note the two comments demarcating the locations of the required and optional fie
 
 Just like interactive generation, non-interactive generation is supported by all 5 tools: ``rsmtool``, ``rsmeval``, ``rsmcompare``, ``rsmpredict``, and ``rsmsummarize``. 
 
-Similarly, to include subgroup information in the reports for ``rsmtool``, ``rsmeval``, and ``rsmcompare``, just add ``--subgroups`` to the command. Note that unlike in interactive mode, this would *only* add subgroup-based sections to the ``general_sections`` list in the output file. You will need to manually edit the ``subgroups`` option in the configuration file to enter the subgroup column names.
+Similarly, to include subgroup information in the reports for ``rsmtool``, ``rsmeval``, and ``rsmcompare``, just add ``--subgroups`` (or ``-g``) to the command. Note that unlike in interactive mode, this would *only* add subgroup-based sections to the ``general_sections`` list in the output file. You will need to manually edit the ``subgroups`` option in the configuration file to enter the subgroup column names.
 
 Generation API
 ~~~~~~~~~~~~~~
diff --git a/doc/contributing.rst b/doc/contributing.rst
index e0b73c89..4e5b97f7 100644
--- a/doc/contributing.rst
+++ b/doc/contributing.rst
@@ -144,6 +144,8 @@ Here are some advanced tips and tricks when working with RSMTool tests.
 
 3. In the rare case that you *do* need to create an entirely new ``tests/test_experiment_X.py`` file instead of using one of the existing ones, you can choose whether to exclude the tests contained in this file from updating their expected outputs when ``update_files.py`` is run by setting ``_AUTO_UPDATE=False`` at the top of the file. This should *only* be necessary if you are absolutely sure that your tests will never need updating.
 
+4. The ``--pdb-errors`` and ``--pdb-failures`` options for ``nosetests`` are your friends. If you encounter test errors or test failures where the cause may not be immediately clear, re-run the ``nosetests`` command with the appropriate option. Doing so will drop you into an interactive PDB session as soon as a error (or failure) is encountered and then you inspect the variables at that point (or use "u" and "d" to go up and down the call stack). This may be particularly useful for tests in ``tests/test_cli.py`` that use ``subprocess.run()``. If these tests are erroring out, use ``--pdb-errors`` and inspect the "stderr" variable in the resulting PDB session to see what the error is.
+
 .. rubric:: Footnotes
 
 .. [#] For older versions of conda, you may have to do ``source activate rsmtool`` on Linux/macOS and ``activate rsmtool`` on Windows.
diff --git a/rsmtool/configuration_parser.py b/rsmtool/configuration_parser.py
index 0a7575a0..633c309a 100644
--- a/rsmtool/configuration_parser.py
+++ b/rsmtool/configuration_parser.py
@@ -9,20 +9,16 @@ and creating configuration objects.
 :organization: ETS
 """
 
-import functools
 import json
 import logging
 import re
-import warnings
 
 from copy import copy, deepcopy
 from collections import Counter
-from configparser import ConfigParser
 from os import getcwd
 from os.path import abspath
 
 from pathlib import Path
-from ruamel import yaml
 
 from skll import Learner
 from skll.metrics import SCORERS
diff --git a/rsmtool/utils/commandline.py b/rsmtool/utils/commandline.py
index 9edea101..a823d539 100755
--- a/rsmtool/utils/commandline.py
+++ b/rsmtool/utils/commandline.py
@@ -160,7 +160,8 @@ def setup_rsmcmd_parser(name,
     # Setting up options for the "generate" subparser #
     ###################################################
     if uses_subgroups:
-        parser_generate.add_argument('--subgroups',
+        parser_generate.add_argument('-g',
+                                     '--subgroups',
                                      dest='subgroups',
                                      action='store_true',
                                      default=False,
@@ -178,7 +179,8 @@ def setup_rsmcmd_parser(name,
                                       "using the generated configuration "
                                       "as-is will be suppressed.")
 
-    parser_generate.add_argument('--interactive',
+    parser_generate.add_argument('-i',
+                                 '--interactive',
                                  dest='interactive',
                                  action='store_true',
                                  default=False,
@@ -783,6 +785,11 @@ class ConfigurationGenerator:
         sys.stderr.write(" - you may still need to edit the generated configuration\n")
         sys.stderr.write("\n")
 
+        if not self.use_subgroups:
+            sys.stderr.write("IMPORTANT: If you have subgroups and didn't specify the '-g' "
+                             "option, exit now (ctrl-d) and re-run!\n")
+            sys.stderr.write("\n")
+
         # instantiate a blank dictionary
         configdict = OrderedDict()
 


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 86b51ab0..6e3645ab 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -160,6 +160,7 @@ class TestToolCLI:
             proc = subprocess.run(shlex.split(cmd, posix='win' not in sys.platform),
                                   check=True,
                                   cwd=working_dir,
+                                  stderr=subprocess.PIPE,
                                   stdout=subprocess.DEVNULL,
                                   encoding='utf-8')
             # then check that the commmand ran successfully
@@ -172,7 +173,7 @@ class TestToolCLI:
             proc = subprocess.run(shlex.split(cmd, posix='win' not in sys.platform),
                                   check=True,
                                   stdout=subprocess.PIPE,
-                                  stderr=subprocess.DEVNULL,
+                                  stderr=subprocess.PIPE,
                                   encoding='utf-8')
             ok_(proc.returncode == 0)
             self.validate_generate_output(context,
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 598c0ad0..fe7d9019 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -963,7 +963,19 @@ class TestSetupRsmCmdParser:
                                                 subgroups=True)
         eq_(parsed_namespace, expected_namespace)
 
-    def test_generate_subparser_with_subgroups_but_no_flag(self):
+    def test_generate_subparser_with_subgroups_option_and_short_flag(self):
+        """
+        test generate subparser with subgroups option and short flag
+        """
+        parser = setup_rsmcmd_parser('test', uses_subgroups=True)
+        parsed_namespace = parser.parse_args('generate -g'.split())
+        expected_namespace = argparse.Namespace(subcommand='generate',
+                                                interactive=False,
+                                                quiet=False,
+                                                subgroups=True)
+        eq_(parsed_namespace, expected_namespace)
+
+    def test_generate_subparser_with_subgroups_option_but_no_flag(self):
         """
         test generate subparser with subgroups option but no flag
         """
@@ -1009,6 +1021,17 @@ class TestSetupRsmCmdParser:
                                                 quiet=False)
         eq_(parsed_namespace, expected_namespace)
 
+    def test_generate_subparser_with_only_interactive_short_flag(self):
+        """
+        test generate subparser with only the short interactive flag
+        """
+        parser = setup_rsmcmd_parser('test')
+        parsed_namespace = parser.parse_args('generate -i'.split())
+        expected_namespace = argparse.Namespace(subcommand='generate',
+                                                interactive=True,
+                                                quiet=False)
+        eq_(parsed_namespace, expected_namespace)
+
     def test_generate_subparser_with_subgroups_and_interactive_flags(self):
         """
         test generate subparser with subgroups and interactive flags
@@ -1021,6 +1044,30 @@ class TestSetupRsmCmdParser:
                                                 subgroups=True)
         eq_(parsed_namespace, expected_namespace)
 
+    def test_generate_subparser_with_subgroups_and_interactive_short_flags(self):
+        """
+        test generate subparser with short subgroups and interactive flags
+        """
+        parser = setup_rsmcmd_parser('test', uses_subgroups=True)
+        parsed_namespace = parser.parse_args('generate -i -g'.split())
+        expected_namespace = argparse.Namespace(subcommand='generate',
+                                                quiet=False,
+                                                interactive=True,
+                                                subgroups=True)
+        eq_(parsed_namespace, expected_namespace)
+
+    def test_generate_subparser_with_subgroups_and_interactive_short_flags_together(self):
+        """
+        test generate subparser with short subgroups and interactive flags together
+        """
+        parser = setup_rsmcmd_parser('test', uses_subgroups=True)
+        parsed_namespace = parser.parse_args('generate -ig'.split())
+        expected_namespace = argparse.Namespace(subcommand='generate',
+                                                quiet=False,
+                                                interactive=True,
+                                                subgroups=True)
+        eq_(parsed_namespace, expected_namespace)
+
 
 class TestBatchGenerateConfiguration: