Repository: IDSIA/sacred
Pull request #332 (2018-07-07 02:51:22)

Add a command print_named_configs
I found this command very useful, especially when an experiment contains multiple ingredients which define (nested) named configs. This refers to #239 and may be used to print the available named configs when the user requests an unknown config.

This command is added to the main experiment and prints a list of available named configurations. Docstrings for config scopes are copied and printed as additional information for the named config colored in grey. Here's an example of what the output looks like:

```
Named Configurations (doc):
  named_config1
  ingredient2.named_config3
    """
      A docstring with
      mutliple

      lines.

      """
  ingredient2.dict_config
  ingredient1.named_config2   # A beatiful docstring (this is colored in grey)
```

Btw: This is my first pull request so please correct me if I am doing something wrong.


Patch:
diff --git a/README.rst b/README.rst
index 0c3aeaa..aa3941e 100644
--- a/README.rst
+++ b/README.rst
@@ -178,14 +178,6 @@ License
 -------
 This project is released under the terms of the `MIT license <http://opensource.org/licenses/MIT>`_.
 
-
-Citing Sacred
--------------
-`K. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’, 
-in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56 
-<http://conference.scipy.org/proceedings/scipy2017/klaus_greff.html>`_.
-
-
 .. |pypi| image:: https://img.shields.io/pypi/v/sacred.svg
     :target: https://pypi.python.org/pypi/sacred
     :alt: Current PyPi Version
diff --git a/docs/command_line.rst b/docs/command_line.rst
index 0e51404..e6f8a1f 100644
--- a/docs/command_line.rst
+++ b/docs/command_line.rst
@@ -137,7 +137,7 @@ The name of the command has to be first on the commandline::
 
     >>> ./my_demo.py COMMAND_NAME with seed=123
 
-If the COMMAND_NAME is ommitted it defaults to the main function, but the name
+If the COMMAND_NAME is omitted it defaults to the main function, but the name
 of that function can also explicitly used as the name of the command.
 So for this experiment
 
@@ -263,6 +263,32 @@ The format for exporting the config is inferred from the filename and can be
 any format supported for :ref:`config files <config_files>`.
 
 
+.. _print_named_configs:
+
+Print Named Configs
+-------------------
+
+The ``print_named_configs`` command prints all available named configurations.
+Function docstrings for named config functions are copied and displayed colored
+in **grey**.
+For example::
+
+    >> ./named_config print_named_configs
+    INFO - hello_config - Running command 'print_named_configs'
+    INFO - hello_config - Started
+    Named Configurations (doc):
+      rude   # A rude named config
+    INFO - hello_config - Completed after 0:00:00
+
+If no named configs are available for the experiment, an empty list is printed::
+
+    >> ./01_hello_world print_named_configs
+    INFO - 01_hello_world - Running command 'print_named_configs'
+    INFO - 01_hello_world - Started
+    Named Configurations (doc):
+      No named configs
+    INFO - 01_hello_world - Completed after 0:00:00
+
 Custom Commands
 ---------------
 If you just run an experiment file it will execute the default command, that
diff --git a/docs/settings.rst b/docs/settings.rst
index ad092d4..90808fd 100644
--- a/docs/settings.rst
+++ b/docs/settings.rst
@@ -3,8 +3,8 @@
 Settings
 ********
 
-Some of Sacred's general behaviour is configurable via ``sacred.SETTINGS``.
-Its entries can be set simply by importing and modifying it using dict or attribute notation:
+Some of Sacreds general behaviour is configurable via ``sacred.SETTINGS``.
+It's entries can be set simply by importing and modifying it using dict or attribute notation:
 
 .. code-block:: python
 
@@ -14,7 +14,7 @@ Its entries can be set simply by importing and modifying it using dict or attrib
 
 Settings
 ========
-Here is a brief list of all currently available options.
+Here a brief list of all currently available options.
 
 
 * ``CAPTURE_MODE`` *(default: 'fd' (linux/osx) or 'sys' (windows))*
@@ -36,13 +36,13 @@ Here is a brief list of all currently available options.
     make sure no config key contains an equals sign
   * ``IGNORED_COMMENTS`` *(default: ['^pylint:', '^noinspection'])*
     list of regex patterns to filter out certain IDE or linter directives
-    from in-line comments in the documentation.
+    from inline comments in the documentation
 
 * ``HOST_INFO``
 
   * ``INCLUDE_GPU_INFO`` *(default: True)*
-    Try to collect information about GPUs using the nvidia-smi tool.
-    Deactivating this can cut the start-up time of a Sacred run by about 1 sec.
+    Try to collect information about GPUs using the nvidia-smi tool
+    Deactivating this can cut the start-up time of a sacred run by about 1 sec.
   * ``CAPTURED_ENV`` *(default: [])*
     List of ENVIRONMENT variable names to store in the host-info.
 
@@ -50,7 +50,7 @@ Here is a brief list of all currently available options.
 * ``COMMAND_LINE``
 
   * ``STRICT_PARSING`` *(default: False)*
-    disallow string fallback, if parsing a value from command-line failed.
+    disallow string fallback, if parsing a value from command-line failed
     This enforces the usage of quotes in the command-line. Note that this can
     be very tedious since bash removes one set of quotes, such that double
     quotes will be needed.
diff --git a/examples/named_config.py b/examples/named_config.py
index b1b65d1..fde8af0 100755
--- a/examples/named_config.py
+++ b/examples/named_config.py
@@ -9,6 +9,7 @@ ex = Experiment('hello_config')
 
 @ex.named_config
 def rude():
+    """A rude named config"""
     recipient = "bastard"
     message = "Fuck off you {}!".format(recipient)
 
diff --git a/sacred/commands.py b/sacred/commands.py
index 86bc6fd..79ec86c 100644
--- a/sacred/commands.py
+++ b/sacred/commands.py
@@ -6,14 +6,14 @@ from __future__ import division, print_function, unicode_literals
 import pprint
 import pydoc
 import re
-from collections import namedtuple
+from collections import namedtuple, OrderedDict
 
 from sacred.config import save_config_file
 from sacred.serializer import flatten
 from sacred.utils import PATHCHANGE, iterate_flattened_separately
 
 __all__ = ('print_config', 'print_dependencies', 'save_config',
-           'help_for_command')
+           'help_for_command', 'print_named_configs')
 
 BLUE = '\033[94m'
 GREEN = '\033[92m'
@@ -62,6 +62,49 @@ def print_config(_run):
     print(_format_config(final_config, config_mods))
 
 
+def _format_named_config(indent, path, named_config):
+    indent = ' ' * indent
+    assign = path
+    if hasattr(named_config, '__doc__') and named_config.__doc__ is not None:
+        doc_string = named_config.__doc__
+        if doc_string.strip().count('\n') == 0:
+            assign += GREY + '   # {}'.format(doc_string.strip()) + ENDC
+        else:
+            doc_string = doc_string.replace('\n', '\n' + indent)
+            assign += GREY + '\n{}"""{}"""'.format(indent + '  ',
+                                                   doc_string) + ENDC
+    return indent + assign
+
+
+def _format_named_configs(named_configs, indent=2, hide_path=None):
+    lines = ['Named Configurations (' + GREY + 'doc' + ENDC + '):']
+    for path, named_config in named_configs.items():
+        if hide_path is not None and path.startswith(hide_path + '.'):
+            path = path[len(hide_path) + 1:]
+        lines.append(_format_named_config(indent, path, named_config))
+    if len(lines) < 2:
+        lines.append(' ' * indent + 'No named configs')
+    return '\n'.join(lines)
+
+
+def print_named_configs(ingredient):
+    """
+    Returns a command function that prints the available named configs for the
+     ingredient and all sub-ingredients and exits.
+
+     The output is highlighted:
+       white: config names
+       grey:  doc
+     """
+
+    def print_named_configs():
+        """Print the available named configs and exit."""
+        named_configs = OrderedDict(ingredient.gather_named_configs())
+        print(_format_named_configs(named_configs, 2, ingredient.path))
+
+    return print_named_configs
+
+
 def help_for_command(command):
     """Get the help text (signature + docstring) for a command (function)."""
     help_text = pydoc.text.document(command)
diff --git a/sacred/config/config_scope.py b/sacred/config/config_scope.py
index d6954c9..ece27af 100644
--- a/sacred/config/config_scope.py
+++ b/sacred/config/config_scope.py
@@ -30,6 +30,7 @@ class ConfigScope(object):
         self._func = func
         self._body_code = get_function_body_code(func)
         self._var_docs = get_config_comments(func)
+        self.__doc__ = self._func.__doc__
 
     def __call__(self, fixed=None, preset=None, fallback=None):
         """
diff --git a/sacred/experiment.py b/sacred/experiment.py
index 56775b1..4236b2e 100755
--- a/sacred/experiment.py
+++ b/sacred/experiment.py
@@ -14,7 +14,8 @@ from sacred.arg_parser import format_usage, get_config_updates
 from sacred.commandline_options import (
     ForceOption, gather_command_line_options, LoglevelOption)
 from sacred.commands import (help_for_command, print_config,
-                             print_dependencies, save_config)
+                             print_dependencies, save_config,
+                             print_named_configs)
 from sacred.config.signature import Signature
 from sacred.ingredient import Ingredient
 from sacred.initialize import create_run
@@ -82,6 +83,7 @@ class Experiment(Ingredient):
         self.command(print_config, unobserved=True)
         self.command(print_dependencies, unobserved=True)
         self.command(save_config, unobserved=True)
+        self.command(print_named_configs(self), unobserved=True)
         self.observers = []
         self.current_run = None
         self.captured_out_filter = None
diff --git a/sacred/ingredient.py b/sacred/ingredient.py
index 98b3b34..5238750 100644
--- a/sacred/ingredient.py
+++ b/sacred/ingredient.py
@@ -283,6 +283,23 @@ class Ingredient(object):
             for cmd_name, cmd in ingred.gather_commands():
                 yield cmd_name, cmd
 
+    def gather_named_configs(self):
+        """Collect all named configs from this ingredient and its sub-ingredients.
+
+        Yields
+        ------
+        config_name: str
+            The full (dotted) name of the named config.
+        config: ConfigScope or ConfigDict or basestring
+            The corresponding named config.
+        """
+        for config_name, config in self.named_configs.items():
+            yield self.path + '.' + config_name, config
+
+        for ingred in self.ingredients:
+            for config_name, config in ingred.gather_named_configs():
+                yield config_name, config
+
     def get_experiment_info(self):
         """Get a dictionary with information about this experiment.
 
diff --git a/sacred/observers/file_storage.py b/sacred/observers/file_storage.py
index a491d6c..a4df676 100644
--- a/sacred/observers/file_storage.py
+++ b/sacred/observers/file_storage.py
@@ -215,7 +215,7 @@ class FileStorageObserver(RunObserver):
         try:
             metrics_path = os.path.join(self.dir, "metrics.json")
             saved_metrics = json.load(open(metrics_path, 'r'))
-        except IOError:
+        except IOError as e:
             # We haven't recorded anything yet. Start Collecting.
             saved_metrics = {}
 
diff --git a/sacred/stdout_capturing.py b/sacred/stdout_capturing.py
index a0b86db..f216d88 100644
--- a/sacred/stdout_capturing.py
+++ b/sacred/stdout_capturing.py
@@ -127,11 +127,11 @@ def tee_output_fd():
             # this is done to avoid receiving KeyboardInterrupts (see #149)
             # in Python 3 we could just pass start_new_session=True
             tee_stdout = subprocess.Popen(
-                ['tee', '-a', target.name], preexec_fn=os.setsid,
-                stdin=subprocess.PIPE, stdout=1)
+                ['tee', '-a', '/dev/stderr'], preexec_fn=os.setsid,
+                stdin=subprocess.PIPE, stderr=target_fd, stdout=1)
             tee_stderr = subprocess.Popen(
-                ['tee', '-a', target.name], preexec_fn=os.setsid,
-                stdin=subprocess.PIPE, stdout=2)
+                ['tee', '-a', '/dev/stderr'], preexec_fn=os.setsid,
+                stdin=subprocess.PIPE, stderr=target_fd, stdout=2)
         except (FileNotFoundError, (OSError, AttributeError)):
             # No tee found in this operating system. Trying to use a python
             # implementation of tee. However this is slow and error-prone.


Test patch:
diff --git a/tests/test_commands.py b/tests/test_commands.py
index fdf0dbc..4018c3a 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -4,11 +4,15 @@
 from __future__ import division, print_function, unicode_literals
 
 import pprint
+from collections import OrderedDict
 
 import pytest
+from sacred import Ingredient, Experiment
 from sacred.commands import (BLUE, ENDC, GREY, GREEN, RED, ConfigEntry,
                              PathEntry, _format_config, _format_entry,
-                             help_for_command, _iterate_marked, _non_unicode_repr)
+                             help_for_command, _iterate_marked, _non_unicode_repr,
+                             _format_named_configs, _format_named_config)
+from sacred.config import ConfigScope
 from sacred.config.config_summary import ConfigSummary
 
 
@@ -161,3 +165,54 @@ def test_help_for_command():
     help_text = help_for_command(my_command)
     assert "my_command" in help_text
     assert "This is my docstring" in help_text
+
+
+def _config_scope_with_single_line_doc():
+    """doc"""
+    pass
+
+
+def _config_scope_with_multiline_doc():
+    """Multiline
+    docstring!
+    """
+    pass
+
+
+@pytest.mark.parametrize('indent, path, named_config, expected', [
+    (0, 'a', None, 'a'),
+    (1, 'b', None, ' b'),
+    (4, 'a.b.c', None, '    a.b.c'),
+    (0, 'c', ConfigScope(_config_scope_with_single_line_doc), 'c' + GREY
+     + '   # doc' + ENDC),
+    (0, 'd', ConfigScope(_config_scope_with_multiline_doc),
+     'd' + GREY + '\n  """Multiline\n    docstring!\n    """' + ENDC)
+])
+def test_format_named_config(indent, path, named_config, expected):
+    assert _format_named_config(indent, path, named_config) == expected
+
+
+def test_format_named_configs():
+    ingred = Ingredient('ingred')
+    ex = Experiment(name='experiment', ingredients=[ingred])
+
+    @ingred.named_config
+    def named_config1():
+        pass
+
+    @ex.named_config
+    def named_config2():
+        """named config with doc"""
+        pass
+
+    dict_config = dict(v=42)
+    ingred.add_named_config('dict_config', dict_config)
+
+    named_configs_text = _format_named_configs(OrderedDict(
+        ex.gather_named_configs()), hide_path=ex.path)
+    assert named_configs_text.startswith('Named Configurations (' +
+                                            GREY + 'doc' + ENDC + '):')
+    assert 'named_config2' in named_configs_text
+    assert '# named config with doc' in named_configs_text
+    assert 'ingred.named_config1' in named_configs_text
+    assert 'ingred.dict_config' in named_configs_text
diff --git a/tests/test_ingredients.py b/tests/test_ingredients.py
index 1aa72de..7ac7e5f 100644
--- a/tests/test_ingredients.py
+++ b/tests/test_ingredients.py
@@ -256,6 +256,23 @@ def test_gather_commands(ing):
     assert ('tickle.foo', foo) in commands
 
 
+def test_gather_named_configs(ing):
+    ing2 = Ingredient('ing2', ingredients=[ing])
+
+    @ing.named_config
+    def named_config1():
+        pass
+
+    @ing2.named_config
+    def named_config2():
+        """named config with doc"""
+        pass
+
+    named_configs = list(ing2.gather_named_configs())
+    assert ('ing2.named_config2', named_config2) in named_configs
+    assert ('tickle.named_config1', named_config1) in named_configs
+
+
 def test_config_docs_are_preserved(ing):
     @ing.config
     def ing_cfg():
diff --git a/tests/test_run.py b/tests/test_run.py
index 8903c0b..9adc527 100755
--- a/tests/test_run.py
+++ b/tests/test_run.py
@@ -266,5 +266,4 @@ def test_captured_out_filter(run, capsys):
     run.main_function.side_effect = print_mock_progress
     with capsys.disabled():
         run()
-        sys.stdout.flush()
     assert run.captured_out == 'progress 9'