Repository: Pyomo/pyomo

Issue:
Fix keyword argument word wrapping/indentation/type specification for docstrings generated through `ConfigDict`
## Summary

The method `pyomo.common.config.add_docstring_list` (or alternatively, `pyomo.common.config.ConfigDict.generate_documentation(...,width=78,...)`), useful for generating/extending the docstring of a function/method whose arguments are handled through a  `ConfigDict`, does not seem to properly handle the case in which the `description` or `doc` string for a `ConfigValue` entry is of length exceeding the `width` argument to `ConfigDict.generate_documentation`. As a result, arguments are not documented as expected in the online docs, and readability of docstrings in a terminal setting may be reduced.

### Rationale

Documentation of the `contrib.pyros.PyROS.solve()` method, which contains arguments for which the descriptions are relatively long, are affected. (See Read the Docs build for #2707) Possibly also affects documentation of any method for which the docstring is modified using `ConfigDict.generate_documentation()`.

### Example
Imports:

```python
from pyomo.common.config import add_docstring_list, ConfigDict, ConfigValue
```

Suppose we define a function `test_func` through:

```python
def test_func(string, reps=1, **kwds):
    """
    Do what this test function does.

    Arguments
    ---------
    string : str
        Str to display.
    reps : int, optional
        Number of print repetitions.

    Returns
    -------
    repeated_str : str
        Repeated string.

    """
    ...
```

and decide to create a corresponding `ConfigDict` through:

```python
def generate_func_config():
    config = ConfigDict(description="test_func")

    ...
    config.declare(
        name="print_it",
        config=ConfigValue(
            default=False,
            domain=bool,
            description="Print things.",
            doc=(
                """
                True to print the repeated string, False otherwise.
                The remainder of this docstring is filler, intended
                to demonstrate the issue of rendering very lengthy
                ConfigDict entry descriptions/docs.
                The number of characters in this string (excluding
                whitespace) exceeds
                256, which is the value for the argument `width`
                passed to config docstring generator.
                """
            ),
        )
    )
    return config
```

If we then update the docstring for `test_func` with:
```python
test_func.__doc__ = add_docstring_list(func.__doc__, config_dict, indent_by=4)
```

Then the new docstring for `test_func` appears as
```
    Do what this test function does.

    Arguments
    ---------
    string : str
        Str to display.
    reps : int, optional
        Number of print repetitions.

    Returns
    -------
    repeated_str : str
        Repeated string.

    Keyword Arguments
    -----------------
    ...
    print_it
      True to print the repeated string, False otherwise. The remainder of this docstring is filler, intended to demonstrate the issue of rendering v
ery lengthy ConfigDict entry descriptions/docs. The number of characters in this string (excluding whitespace)
    exceeds 256, which is the value for the argument `width` passed to config docstring generator.

```

which may be problematic since the online ReST documentation is sensitive to indentation. In particular, the keyword arguments section of this docstring may be rendered roughly to:

> **Keyword Arguments**
> - ...
> - **print_it** --True to print the repeated string, False otherwise. The remainder of this docstring is filler, intended to demonstrate the issue of rendering very lengthy ConfigDict entry descriptions/docs. The number of characters in this string (excluding whitespace)
> - **exceeds 256, which is the value for the argument `width` passed to config docstring generator.**

instead of to the desired result:

> **Keyword Arguments**
> - ...
> - **print_it** -- True to print the repeated string, False otherwise. The remainder of this docstring is filler, intended to demonstrate the issue of rendering very lengthy ConfigDict entry descriptions/docs. The number of characters in this string (excluding whitespace) exceeds 256, which is the value for the argument `width` passed to config docstring generator.

which is what would occur if the docstring appeared as, say:
```
    Do what this test function does.

    Arguments
    ---------
    string : str
        Str to display.
    reps : int, optional
        Number of print repetitions.

    Returns
    -------
    repeated_str : str
        Repeated string.

    Keyword Arguments
    -----------------
    ...
    print_it
      True to print the repeated string, False otherwise. The remainder of this docstring is filler,
      intended to demonstrate the issue of rendering very lengthy ConfigDict entry descriptions/docs.
      The number of characters in this string (excluding whitespace)
      exceeds 256, which is the value for the argument `width` passed to config docstring generator.

```

which may also be more readable than in a terminal.

We can consider enhancing further by (optionally) specifying the acceptable data types for the argument `print_it`, perhaps by adding an optional `acceptable_types` argument to the `ConfigValue` used for generating the documentation of this specific argument, such that the Keyword Arguments section of the docstring looks like:

```
    Keyword Arguments
    -----------------
    ...
    print_it : bool
      True to print the repeated string, False otherwise. The remainder of this docstring is filler,
      intended to demonstrate the issue of rendering very lengthy ConfigDict entry descriptions/docs.
      The number of characters in this string (excluding whitespace)
      exceeds 256, which is the value for the argument `width` passed to config docstring generator.
```

This may be more useful for arguments for which the acceptable types are not more immediately obvious, though we would need to be careful to ensure the docstring style (e.g. standard Python, numpy, sphinx, etc.) is consistent across all arguments to the function.

### Description

- Fix word wrapping and indentation for `ConfigDict` keyword argument descriptions/docs to make docstrings more readable in online docs and terminal.
- Allow for optional documentation of argument type specification.


Discussion:

Resolving patch:
diff --git a/pyomo/common/config.py b/pyomo/common/config.py
--- a/pyomo/common/config.py
+++ b/pyomo/common/config.py
@@ -28,23 +28,29 @@
 import ply.lex
 import re
 import sys
-from textwrap import wrap
+import textwrap
 import types
 
 from pyomo.common.collections import Sequence, Mapping
-from pyomo.common.deprecation import deprecated, relocated_module_attribute
+from pyomo.common.deprecation import (
+    deprecated,
+    deprecation_warning,
+    relocated_module_attribute,
+)
+from pyomo.common.errors import DeveloperError
 from pyomo.common.fileutils import import_file
+from pyomo.common.formatting import wrap_reStructuredText
 from pyomo.common.modeling import NOTSET
 
-logger = logging.getLogger('pyomo.common.config')
+logger = logging.getLogger(__name__)
 
 relocated_module_attribute(
     'PYOMO_CONFIG_DIR', 'pyomo.common.envvar.PYOMO_CONFIG_DIR',
     version='6.1')
 
 USER_OPTION = 0
-ADVANCED_OPTION = 1
-DEVELOPER_OPTION = 2
+ADVANCED_OPTION = 10
+DEVELOPER_OPTION = 20
 
 def Bool(val):
     """Domain validator for bool-like objects.
@@ -214,7 +220,7 @@ class In(object):
         values are passed to ``domain.__contains__()``, and if ``True``
         is returned, the value is accepted and returned.
 
-    cast: callable, optional
+    cast: Callable, optional
         A callable object.  If specified, incoming values are first
         passed to `cast`, and the resulting object is checked for
         membership in `domain`
@@ -337,7 +343,7 @@ def domain_name(self):
 
 
 class Module(object):
-    """ Domain validator for modules.
+    """Domain validator for modules.
 
     Modules can be specified as module objects, by module name,
     or by the path to the module's file. If specified by path, the
@@ -350,21 +356,25 @@ class Module(object):
 
     Parameters
     ----------
-    basePath: None, str, ConfigValue
+    basePath : None, str, ConfigValue
         The base path that will be prepended to any non-absolute path
         values provided.  If None, defaults to :py:attr:`Path.BasePath`.
 
-    expandPath: bool
+    expandPath : bool
         If True, then the value will be expanded and normalized.  If
         False, the string representation of the value will be used
         unchanged.  If None, expandPath will defer to the (negated)
         value of :py:attr:`Path.SuppressPathExpansion`.
 
+    Examples
+    --------
+
     The following code shows the three ways you can specify a module: by file
     name, by module name, or by module object. Regardless of how the module is
     specified, what is stored in the configuration is a module object.
 
     .. doctest::
+
         >>> from pyomo.common.config import (
         ...     ConfigDict, ConfigValue, Module
         ... )
@@ -372,6 +382,7 @@ class Module(object):
         >>> config.declare('my_module', ConfigValue(
         ...     domain=Module(),
         ... ))
+        <pyomo.common.config.ConfigValue object at ...>
         >>> # Set using file path
         >>> config.my_module = '../../pyomo/common/tests/config_plugin.py'
         >>> # Set using python module name, as a string
@@ -379,6 +390,7 @@ class Module(object):
         >>> # Set using an imported module object
         >>> import os.path
         >>> config.my_module = os.path
+
     """
     def __init__(self, basePath=None, expandPath=None):
         self.basePath = basePath
@@ -503,7 +515,7 @@ class DynamicImplicitDomain(object):
     ``pyomo/common/tests/config_plugin.py``:
 
     .. literalinclude:: /../../pyomo/common/tests/config_plugin.py
-       :lines: 10-
+       :start-at: import
 
     .. doctest::
        :hide:
@@ -550,20 +562,6 @@ def __call__(self, key, value):
         return self.callback(key, value)
 
 
-def add_docstring_list(docstring, configdict, indent_by=4):
-    """Returns the docstring with a formatted configuration arguments listing."""
-    return docstring + (" " * indent_by).join(
-        configdict.generate_documentation(
-            block_start="Keyword Arguments\n-----------------\n",
-            block_end="",
-            item_start="%s\n",
-            item_body="  %s",
-            item_end="",
-            indent_spacing=0,
-            width=256
-        ).splitlines(True))
-
-
 # Note: Enum uses a metaclass to work its magic.  To get a deprecation
 # warning when creating a subclass of ConfigEnum, we need to decorate
 # the __new__ method here (doing the normal trick of letting the class
@@ -616,7 +614,7 @@ def from_enum_or_string(cls, arg):
 .. doctest::
 
     >>> from pyomo.common.config import (
-    ...     ConfigDict, ConfigList, ConfigValue, In,
+    ...     ConfigDict, ConfigList, ConfigValue
     ... )
     >>> config = ConfigDict()
     >>> config.declare('filename', ConfigValue(
@@ -700,7 +698,7 @@ def from_enum_or_string(cls, arg):
    Module
    Path
    PathList
-
+   DynamicImplicitDomain
 
 Configuring class hierarchies
 =============================
@@ -1239,6 +1237,298 @@ def _default_string_dict_lexer(value):
 _default_string_dict_lexer._lex = None
 
 
+def _formatter_str_to_callback(pattern, formatter):
+    "Wrapper function that converts formatter strings to callback functions"
+
+    if not pattern:
+        pattern = ''
+    if '%s' in pattern:
+        cb = lambda self, indent, obj: self.out.write(indent + pattern % obj.name())
+    elif pattern:
+        cb = lambda self, indent, obj: self.out.write(indent + pattern)
+    else:
+        cb = lambda self, indent, obj: None
+    return types.MethodType(cb, formatter)
+
+def _formatter_str_to_item_callback(pattern, formatter):
+    "Wrapper function that converts item formatter strings to callback functions"
+
+    if not pattern:
+        pattern = ''
+    if '%s' in pattern:
+        _item_body_formatter = lambda doc: pattern % (doc,)
+    else:
+        _item_body_formatter = lambda doc: pattern
+
+    def _item_body_cb(self, indent, obj):
+        _doc = obj._doc or obj._description or ""
+        if not _doc:
+            return ''
+        wraplines = '\n ' not in _doc
+        _doc = _item_body_formatter(_doc).strip()
+        if not _doc:
+            return ''
+        _indent = indent + ' ' * self.indent_spacing
+        if wraplines:
+            doc_lines = textwrap.wrap(
+                _doc,
+                self.width,
+                initial_indent=_indent,
+                subsequent_indent=_indent,
+            )
+            self.out.write(('\n'.join(doc_lines)).rstrip() + '\n')
+        else:
+            self.out.write(_doc.rstrip() + '\n')
+
+    return types.MethodType(_item_body_cb, formatter)
+
+
+class ConfigFormatter(object):
+    def _initialize(self, indent_spacing, width, visibility):
+        self.out = io.StringIO()
+        self.indent_spacing = indent_spacing
+        self.width = width
+        self.visibility = visibility
+
+    def _block_start(self, indent, obj):
+        pass
+
+    def _block_end(self, indent, obj):
+        pass
+
+    def _item_start(self, indent, obj):
+        pass
+
+    def _item_body(self, indent, obj):
+        pass
+
+    def _item_end(self, indent, obj):
+        pass
+
+    def _finalize(self):
+        return self.out.getvalue()
+
+    def generate(self, config, indent_spacing=2, width=78, visibility=None):
+        self._initialize(indent_spacing, width, visibility)
+        level = []
+        lastObj = config
+        indent = ''
+        for lvl, pre, val, obj in config._data_collector(1, '', visibility, True):
+            if len(level) < lvl:
+                while len(level) < lvl - 1:
+                    level.append(None)
+                level.append(lastObj)
+                self._block_start(indent, lastObj)
+                indent += ' ' * indent_spacing
+            while len(level) > lvl:
+                _last = level.pop()
+                if _last is not None:
+                    indent = indent[:-indent_spacing]
+                    self._block_end(indent, _last)
+
+            lastObj = obj
+            self._item_start(indent, obj)
+            self._item_body(indent, obj)
+            self._item_end(indent, obj)
+        while level:
+            _last = level.pop()
+            if _last is not None:
+                indent = indent[:-indent_spacing]
+                self._block_end(indent, _last)
+        return self._finalize()
+
+
+class String_ConfigFormatter(ConfigFormatter):
+    def __init__(self, block_start, block_end, item_start, item_body, item_end):
+        self._block_start = _formatter_str_to_callback(block_start, self)
+        self._block_end = _formatter_str_to_callback(block_end, self)
+        self._item_start = _formatter_str_to_callback(item_start, self)
+        self._item_end = _formatter_str_to_callback(item_end, self)
+        self._item_body = _formatter_str_to_item_callback(item_body, self)
+
+
+class LaTeX_ConfigFormatter(String_ConfigFormatter):
+    def __init__(self):
+        super().__init__(
+            "\\begin{description}[topsep=0pt,parsep=0.5em,itemsep=-0.4em]\n",
+            "\\end{description}\n",
+            "\\item[{%s}]\\hfill\n",
+            "\\\\%s",
+            "",
+        )
+
+
+class numpydoc_ConfigFormatter(ConfigFormatter):
+    def _initialize(self, *args):
+        super()._initialize(*args)
+        self.wrapper = textwrap.TextWrapper(width=self.width)
+
+    def _item_body(self, indent, obj):
+        typeinfo = ', '.join(filter(
+            None,
+            [
+                'dict' if isinstance(obj, ConfigDict) else obj.domain_name(),
+                'optional' if obj._default is None else f'default={repr(obj._default)}',
+            ]
+        ))
+        # Note that numpydoc / ReST specifies that the colon in
+        # definition lists be surrounded by spaces (i.e., " : ").
+        # However, as of numpydoc (1.1.0) / Sphinx (3.4.3) / napoleon
+        # (0.7), things aren't really geared for nested lists of
+        # parameters.  Definition lists omit the colon, and
+        # sub-definitions are rendered as normal definition sections
+        # (without the special formatting applied to Parameters lists),
+        # leading to less readable docs.  As they tolerate omitting the
+        # space before the colon at the top level (which at lower levels
+        # causes nested definition lists to NOT omit the colon), we will
+        # generate non-standard ReST and omit the preceeding space:
+        self.out.write(f'\n{indent}{obj.name()}: {typeinfo}\n')
+        self.wrapper.initial_indent = indent + ' ' * self.indent_spacing
+        self.wrapper.subsequent_indent = indent + ' ' * self.indent_spacing
+        vis = ""
+        if self.visibility is None and obj._visibility >= ADVANCED_OPTION:
+            vis = "[ADVANCED option]"
+            if obj._visibility >= DEVELOPER_OPTION:
+                vis = "[DEVELOPER option]"
+        itemdoc = wrap_reStructuredText(
+            '\n\n'.join(filter(
+                None, [vis, inspect.cleandoc(obj._doc or obj._description or "")]
+            )),
+            self.wrapper,
+        )
+        if itemdoc:
+            self.out.write(itemdoc + '\n')
+
+    def _finalize(self):
+        return inspect.cleandoc(self.out.getvalue())
+
+ConfigFormatter.formats = {
+    'latex': LaTeX_ConfigFormatter,
+    'numpydoc': numpydoc_ConfigFormatter,
+}
+
+
+def add_docstring_list(docstring, configdict, indent_by=4):
+    """Returns the docstring with a formatted configuration arguments listing."""
+    section = 'Keyword Arguments'
+    return (
+        inspect.cleandoc(docstring)
+        + '\n' + section + '\n' + '-'*len(section) + '\n'
+        + configdict.generate_documentation(
+            indent_spacing=indent_by, width=256, format='numpydoc')
+    )
+
+
+class document_kwargs_from_configdict(object):
+    """Decorator to append the documentation of a ConfigDict to the docstring
+
+    This adds the documentation of the specified :py:class:`ConfigDict`
+    (using the :py:class:`numpydoc_ConfigFormatter` formatter) to the
+    decorated object's docstring.
+
+    Parameters
+    ----------
+    config : ConfigDict or str
+        the :py:class:`ConfigDict` to document.  If a ``str``, then the
+        :py:class:`ConfigDict` is obtained by retrieving the named
+        attribute from the decorated object (thereby enabling
+        documenting class objects whose ``__init__`` keyword arguments
+        are processed by a :py:class:`ConfigDict` class attribute)
+
+    section : str
+        the section header to preface config documentation with
+
+    indent_spacing : int
+        number of spaces to indent each block of documentation
+
+    width : int
+        total documentation width in characters (for wrapping paragraphs)
+
+    doc : str, optional
+        the initial docstring to append the ConfigDict documentation to.
+        If None, then the decorated object's ``__doc__`` will be used.
+
+    Examples
+    --------
+
+    >>> from pyomo.common.config import (
+    ...     ConfigDict, ConfigValue, document_kwargs_from_configdict
+    ... )
+    >>> class MyClass(object):
+    ...     CONFIG = ConfigDict()
+    ...     CONFIG.declare('iterlim', ConfigValue(
+    ...         default=3000,
+    ...         domain=int,
+    ...         doc="Iteration limit.  Specify None for no limit"
+    ...     ))
+    ...     CONFIG.declare('tee', ConfigValue(
+    ...         domain=bool,
+    ...         doc="If True, stream the solver output to the console"
+    ...     ))
+    ...
+    ...     @document_kwargs_from_configdict(CONFIG)
+    ...     def solve(self, **kwargs):
+    ...         config = self.CONFIG(kwargs)
+    ...         # ...
+    ...
+    >>> help(MyClass.solve)
+    Help on function solve:
+    <BLANKLINE>
+    solve(self, **kwargs)
+        Keyword Arguments
+        -----------------
+        iterlim: int, default=3000
+            Iteration limit.  Specify None for no limit
+    <BLANKLINE>
+        tee: bool, optional
+            If True, stream the solver output to the console
+
+    """
+    def __init__(
+            self,
+            config,
+            section='Keyword Arguments',
+            indent_spacing=4,
+            width=78,
+            visibility=None,
+            doc=None,
+    ):
+        if '\n' not in section:
+            section += '\n' + '-'*len(section) + '\n'
+        self.config = config
+        self.section = section
+        self.indent_spacing = indent_spacing
+        self.width = width
+        self.visibility = visibility
+        self.doc = doc
+
+    def __call__(self, fcn):
+        if isinstance(self.config, str):
+            self.config = getattr(fcn, self.config)
+        if self.doc is not None:
+            doc = inspect.cleandoc(self.doc)
+        elif fcn.__doc__:
+            doc = inspect.cleandoc(fcn.__doc__)
+        else:
+            doc = ""
+        if doc:
+            if not doc.endswith('\n'):
+                doc += '\n\n'
+            else:
+                doc += '\n'
+        fcn.__doc__ = (
+            doc
+            + f'{self.section}'
+            + self.config.generate_documentation(
+                indent_spacing=self.indent_spacing,
+                width=self.width,
+                visibility=self.visibility,
+                format='numpydoc',
+            )
+        )
+        return fcn
+
+
 class ConfigBase(object):
     __slots__ = ('_parent', '_name', '_userSet', '_userAccessed', '_data',
                  '_default', '_domain', '_description', '_doc', '_visibility',
@@ -1613,7 +1903,7 @@ def generate_yaml_template(self, indent_spacing=2, width=78, visibility=0):
             os.write(comment)
             txtArea = max(width - field - len(comment), minDocWidth)
             os.write(("\n" + ' ' * field + comment).join(
-                wrap(
+                textwrap.wrap(
                     obj._description, txtArea, subsequent_indent='  ')))
             os.write('\n')
         return os.getvalue()
@@ -1622,80 +1912,49 @@ def generate_yaml_template(self, indent_spacing=2, width=78, visibility=0):
     def generate_documentation(
             self, block_start=None, block_end=None,
             item_start=None, item_body=None, item_end=None,
-            indent_spacing=2, width=78, visibility=0,
+            indent_spacing=2, width=78, visibility=None,
             format='latex'):
-        _formats = ConfigBase.generate_documentation.formats
-        if block_start is None:
-            block_start = _formats.get(format, {}).get('block_start','')
-        if block_end is None:
-            block_end = _formats.get(format, {}).get('block_end','')
-        if item_start is None:
-            item_start = _formats.get(format, {}).get('item_start','')
-        if item_body is None:
-            item_body = _formats.get(format, {}).get('item_body','')
-        if item_end is None:
-            item_end = _formats.get(format, {}).get('item_end','')
 
-        os = io.StringIO()
-        level = []
-        lastObj = self
-        indent = ''
-        for lvl, pre, val, obj in self._data_collector(1, '', visibility, True):
-            if len(level) < lvl:
-                while len(level) < lvl - 1:
-                    level.append(None)
-                level.append(lastObj)
-                if '%s' in block_start:
-                    os.write(indent + block_start % lastObj.name())
-                elif block_start:
-                    os.write(indent + block_start)
-                indent += ' ' * indent_spacing
-            while len(level) > lvl:
-                _last = level.pop()
-                if _last is not None:
-                    indent = indent[:-1 * indent_spacing]
-                    if '%s' in block_end:
-                        os.write(indent + block_end % _last.name())
-                    elif block_end:
-                        os.write(indent + block_end)
+        if isinstance(format, str):
+            formatter = ConfigFormatter.formats.get(format, None)
+            if formatter is None:
+                raise ValueError(f"Unrecognized documentation formatter, '{format}'")
+            formatter = formatter()
+        else:
+            # Assume everything not a str is a valid formatter object.
+            formatter = format
+
+        deprecated_args = (block_start, block_end, item_start, item_end)
+        if any(arg is not None for arg in deprecated_args):
+            names = ('block_start', 'block_end', 'item_start', 'item_end')
+            for arg, name in zip(deprecated_args, names):
+                if arg is None:
+                    continue
+                deprecation_warning(
+                    f"Overriding '{name}' by passing strings to "
+                    "generate_documentation is deprecated.  Create an instance of a "
+                    "StringConfigFormatter and pass it as the 'format' argument.",
+                    version='6.5.1.dev0',
+                )
+                setattr(
+                    formatter,
+                    "_" + name,
+                    _formatter_str_to_callback(arg, formatter)
+                )
+        if item_body is not None:
+            deprecation_warning(
+                f"Overriding 'item_body' by passing strings to "
+                "generate_documentation is deprecated.  Create an instance of a "
+                "StringConfigFormatter and pass it as the 'format' argument.",
+                version='6.5.1.dev0',
+            )
+            setattr(
+                formatter,
+                "_item_body",
+                _formatter_str_to_item_callback(item_body, formatter)
+            )
 
-            lastObj = obj
-            if '%s' in item_start:
-                os.write(indent + item_start % obj.name())
-            elif item_start:
-                os.write(indent + item_start)
-            _doc = obj._doc or obj._description or ""
-            if _doc:
-                _wrapLines = '\n ' not in _doc
-                if '%s' in item_body:
-                    _doc = item_body % (_doc,)
-                elif _doc:
-                    _doc = item_body
-                if _wrapLines:
-                    doc_lines = wrap(
-                        _doc,
-                        width,
-                        initial_indent=indent + ' ' * indent_spacing,
-                        subsequent_indent=indent + ' ' * indent_spacing)
-                else:
-                    doc_lines = (_doc,)
-                # Write things out
-                os.writelines('\n'.join(doc_lines))
-                if not doc_lines[-1].endswith("\n"):
-                    os.write('\n')
-            if '%s' in item_end:
-                os.write(indent + item_end % obj.name())
-            elif item_end:
-                os.write(indent + item_end)
-        while level:
-            _last = level.pop()
-            if _last is not None:
-                indent = indent[:-1 * indent_spacing]
-                if '%s' in block_end:
-                    os.write(indent + block_end % _last.name())
-                else:
-                    os.write(indent + block_end)
-        return os.getvalue()
+        return formatter.generate(self, indent_spacing, width, visibility)
 
     def user_values(self):
         if self._userSet:
@@ -1711,17 +1970,6 @@ def unused_user_values(self):
             if obj._userSet and not obj._userAccessed:
                 yield obj
 
-ConfigBase.generate_documentation.formats = {
-    'latex': {
-        'block_start': "\\begin{description}["
-            "topsep=0pt,parsep=0.5em,itemsep=-0.4em]\n",
-        'block_end': "\\end{description}\n",
-        'item_start': "\\item[{%s}]\\hfill\n",
-        'item_body': "\\\\%s",
-        'item_end': "",
-    }
-}
-
 
 class ConfigValue(ConfigBase):
     """Store and manipulate a single configuration value.
@@ -1732,7 +1980,7 @@ class ConfigValue(ConfigBase):
         The default value that this ConfigValue will take if no value is
         provided.
 
-    domain: callable, optional
+    domain: Callable, optional
         The domain can be any callable that accepts a candidate value
         and returns the value converted to the desired type, optionally
         performing any data validation.  The result will be stored into
@@ -1858,7 +2106,7 @@ class ConfigList(ConfigBase, Sequence):
         otherwise the default is cast to the domain and forms a default
         list with a single element.
 
-    domain: callable, optional
+    domain: Callable, optional
         The domain can be any callable that accepts a candidate value
         and returns the value converted to the desired type, optionally
         performing any data validation.  The result will be stored /
@@ -2039,7 +2287,7 @@ class ConfigDict(ConfigBase, Mapping):
         were not prevously declared using :py:meth:`declare` or
         :py:meth:`declare_from`.
 
-    implicit_domain: callable, optional
+    implicit_domain: Callable, optional
         The domain that will be used for any implicitly-declared keys.
         Follows the same rules as :py:meth:`ConfigValue`'s `domain`.
 
@@ -2354,9 +2602,9 @@ def _data_collector(self, level, prefix, visibility=None, docMode=False):
                 level += 1
         for key in self._decl_order:
             cfg = self._data[key]
-            for v in cfg._data_collector(
-                    level, cfg._name + ': ', visibility, docMode):
-                yield v
+            yield from cfg._data_collector(
+                level, cfg._name + ': ', visibility, docMode
+            )
 
 # Backwards compatibility: ConfigDict was originally named ConfigBlock.
 ConfigBlock = ConfigDict
diff --git a/pyomo/common/formatting.py b/pyomo/common/formatting.py
--- a/pyomo/common/formatting.py
+++ b/pyomo/common/formatting.py
@@ -14,9 +14,11 @@
 
    tostr
    tabular_writer
+   wrap_reStructuredText
    StreamIndenter
 """
 
+import re
 import types
 from pyomo.common.sorting import sorted_robust
 
@@ -239,3 +241,141 @@ def write(self, data):
     def writelines(self, sequence):
         for x in sequence:
             self.write(x)
+
+
+_indentation_re = re.compile(r'\s*')
+_bullet_re = re.compile(
+    r'([-+*] +)' # bulleted lists
+    r'|(\(?[0-9]+[\)\.] +)' # enumerated lists (arabic numerals)
+    r'|(\(?[ivxlcdm]+[\)\.] +)' # enumerated lists (roman numerals)
+    r'|(\(?[IVXLCDM]+[\)\.] +)' # enumerated lists (roman numerals)
+    r'|(\(?[a-zA-Z][\)\.] +)' # enumerated lists (letters)
+    r'|(\(?\#[\)\.] +)' # auto enumerated lists
+    r'|([a-zA-Z0-9_ ]+ +: +)' # definitions
+    r'|(:[a-zA-Z0-9_ ]+: +)' # field name
+    r'|(?:\[\s*[A-Za-z0-9\.]+\s*\] +)' # [PASS]|[FAIL]|[ OK ]
+)
+_verbatim_line_start = re.compile(
+    r'(\| )' # line blocks
+    r'|(\+((-{3,})|(={3,}))\+)' # grid table
+)
+_verbatim_line = re.compile(
+    r'(={3,}[ =]+)'  # simple tables, ======== sections
+    # sections
+    + ''.join(r'|(\%s{3,})' % c for c in r'!"#$%&\'()*+,-./:;<>?@[\\]^_`{|}~')
+)
+
+
+def wrap_reStructuredText(docstr, wrapper):
+    """A text wrapper that honors paragraphs and basic reStructuredText markup
+
+    This wraps `textwrap.fill()` to first separate the incoming text by
+    paragraphs before using ``wrapper`` to wrap each one.  It includes a
+    basic (partial) parser for reStructuredText format to attempt to
+    avoid wrapping structural elements like section headings, bullet /
+    enumerated lists, and tables.
+
+    Parameters
+    ----------
+    docstr : str
+        The incoming string to parse and wrap
+
+    wrapper : `textwrap.TextWrap`
+        The configured `TextWrap` object to use for wrapping paragraphs.
+        While the object will be reconfigured within this function, it
+        will be restored to its original state upon exit.
+
+    """
+    # As textwrap only works on single paragraphs, we need to break
+    # up the incoming message into paragraphs before we pass it to
+    # textwrap.
+    paragraphs = [(None, None, None)]
+    literal_block = False
+    verbatim = False
+    for line in docstr.rstrip().splitlines():
+        leading = _indentation_re.match(line).group()
+        content = line.strip()
+        if not content:
+            if literal_block:
+                if literal_block[0] == 2:
+                    literal_block = False
+            elif paragraphs[-1][2] and ''.join(paragraphs[-1][2]).endswith('::'):
+                literal_block = (0, paragraphs[-1][1])
+            paragraphs.append((None, None, None))
+            continue
+        if literal_block:
+            if literal_block[0] == 0:
+                if len(literal_block[1]) < len(leading):
+                    # indented literal block
+                    literal_block = 1, leading
+                    paragraphs.append((None, None, line))
+                    continue
+                elif (
+                        len(literal_block[1]) == len(leading)
+                        and content[0] in '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
+                ):
+                    # quoted literal block
+                    literal_block = 2, leading
+                    paragraphs.append((None, None, line))
+                    continue
+                else:
+                    # invalid literal block
+                    literal_block = False
+            elif leading.startswith(literal_block[1]):
+                paragraphs.append((None, None, line))
+                continue
+            else:
+                # fall back on normal line processing
+                literal_block = False
+        if content == '```':
+            # Not part of ReST, but we have supported this in Pyomo for a long time
+            verbatim ^= True
+        elif verbatim:
+            paragraphs.append((None, None, line))
+        elif _verbatim_line_start.match(content):
+            # This catches lines that start with patterns that indicate
+            # that the line should not be wrapped (line blocks, grid
+            # tables)
+            paragraphs.append((None, None, line))
+        elif _verbatim_line.match(content):
+            # This catches whole line patterns that should not be
+            # wrapped with previous/subsequent lines (e.g., simple table
+            # headers, section headers)
+            paragraphs.append((None, None, line))
+        else:
+            matchBullet = _bullet_re.match(content)
+            if matchBullet:
+                # Handle things that look like bullet lists specially
+                hang = matchBullet.group()
+                paragraphs.append(
+                    (leading, leading + ' '*len(hang), [content]))
+            elif paragraphs[-1][1] == leading:
+                # Continuing a text block
+                paragraphs[-1][2].append( content )
+            else:
+                # Beginning a new text block
+                paragraphs.append((leading, leading, [content]))
+
+    while paragraphs and paragraphs[0][2] is None:
+        paragraphs.pop(0)
+
+    wrapper_init = wrapper.initial_indent, wrapper.subsequent_indent
+    try:
+        for i, (indent, subseq, par) in enumerate(paragraphs):
+            base_indent = wrapper_init[1] if i else wrapper_init[0]
+
+            if indent is None:
+                if par is None:
+                    paragraphs[i] = ''
+                else:
+                    paragraphs[i] = base_indent + par
+                continue
+
+            wrapper.initial_indent = base_indent + indent
+            wrapper.subsequent_indent = base_indent + subseq
+            paragraphs[i] = wrapper.fill(' '.join(par))
+    finally:
+        # Avoid side-effects and restore the initial wrapper state
+        wrapper.initial_indent, wrapper.subsequent_indent = wrapper_init
+
+    return '\n'.join(paragraphs)
diff --git a/pyomo/common/log.py b/pyomo/common/log.py
--- a/pyomo/common/log.py
+++ b/pyomo/common/log.py
@@ -18,6 +18,7 @@
 #
 # Utility classes for working with the logger
 #
+import inspect
 import io
 import logging
 import re
@@ -27,10 +28,9 @@
 from pyomo.version.info import releaselevel
 from pyomo.common.deprecation import deprecated
 from pyomo.common.fileutils import PYOMO_ROOT_DIR
+from pyomo.common.formatting import wrap_reStructuredText
 
 _indentation_re = re.compile(r'\s*')
-_bullet_re = re.compile(r'(?:[-*] +)|(\[\s*[A-Za-z0-9\.]+\s*\] +)')
-_bullet_char = '-*['
 
 _RTD_URL = "https://pyomo.readthedocs.io/en/%s/errors.html" % (
     'stable'
@@ -96,7 +96,9 @@ def __init__(self, **kwds):
                 raise ValueError('unrecognized style flag "%s"'
                                  % (kwds['style'],))
         self._wrapper = textwrap.TextWrapper(width=kwds.pop('wrap', 78))
-        self.hang = kwds.pop('hang', ' '*4)
+        self._wrapper.subsequent_indent = kwds.pop('hang', ' '*4)
+        if not self._wrapper.subsequent_indent:
+            self._wrapper.subsequent_indent = ''
         self.basepath = kwds.pop('base', None)
         super(WrappingFormatter, self).__init__(**kwds)
 
@@ -117,93 +119,49 @@ def format(self, record):
             for k,v in _orig.items():
                 setattr(record, k, v)
 
+        # We want to normalize the incoming message *before* we start
+        # formatting (wrapping) paragraphs.
+        #
         # Most of the messages are either unformatted long lines or
         # triple-quote blocks of text.  In the latter case, if the text
         # starts on the same line as the triple-quote, then it is almost
         # certainly NOT indented with the bulk of the text, which will
         # cause dedent to get confused and not strip any leading
-        # whitespace.  This attempts to work around that case:
-        #
-        #if not (_msg.startswith('\n') or _indentation_re.match(_msg).group()):
-        #    # copy the indention for the second line to the first:
-        #    lines = _msg.splitlines()
-        #    if len(lines) > 1:
-        #        _msg = _indentation_re.match(lines[1]).group() + _msg
+        # whitespace.
         #
-        # The problem with the above logic is that users may want a
-        # simple introductory line followed by an intented line (our
-        # tests did this!), and cannot specify it without adding an
-        # extra blank line to the output.  In contrast, it is possible
-        # for the user to fix the scenario above that motivated this
-        # code by just indenting their first line correctly.
-        msg = textwrap.dedent(msg).strip()
+        # A standard approach is to use inspect.cleandoc, which
+        # allows for the first line to have 0 indent.
+        msg = inspect.cleandoc(msg)
 
         # Split the formatted log message (that currently has _flag in
         # lieu of the actual message content) into lines, then
         # recombine, substituting and wrapping any lines that contain
         # _flag.
         return '\n'.join(
-            self._wrap_msg(l, msg, _id) if self._flag in l else l
-            for l in raw_msg.splitlines()
+            self._wrap_msg(line, msg, _id) if self._flag in line else line
+            for line in raw_msg.splitlines()
         )
 
-    def _wrap_msg(self, l, msg, _id):
-        indent = _indentation_re.match(l).group()
-        wrapped_msg = self._wrap(l.strip().replace(self._flag, msg), indent)
+    def _wrap_msg(self, format_line, msg, _id):
+        _init = self._wrapper.initial_indent, self._wrapper.subsequent_indent
+        # We will honor the "hang" argument (for specifying a hanging
+        # indent) unless the formatting line was indented (e.g. because
+        # DEBUG was set), in which case we will use that for both the
+        # first line and all subsequent lines.
+        indent = _indentation_re.match(format_line).group()
+        if indent:
+            self._wrapper.initial_indent = self._wrapper.subsequent_indent = indent
+        try:
+            wrapped_msg = wrap_reStructuredText(
+                format_line.strip().replace(self._flag, msg), self._wrapper
+            )
+        finally:
+            # Restore the wrapper state
+            self._wrapper.initial_indent, self._wrapper.subsequent_indent = _init
         if _id:
-            wrapped_msg += f"\n{indent}    See also {RTD(_id)}"
+            wrapped_msg += f"\n{indent}{_init[1]}See also {RTD(_id)}"
         return wrapped_msg
 
-    def _wrap(self, msg, base_indent):
-        # As textwrap only works on single paragraphs, we need to break
-        # up the incoming message into paragraphs before we pass it to
-        # textwrap.
-        paragraphs = []
-        verbatim = False
-        for line in msg.rstrip().splitlines():
-            leading = _indentation_re.match(line).group()
-            content = line.strip()
-            if not content:
-                paragraphs.append((None, None))
-            elif content == '```':
-                verbatim ^= True
-            elif verbatim:
-                paragraphs.append((None, line))
-            else:
-                matchBullet = _bullet_re.match(content)
-                if matchBullet:
-                    paragraphs.append(
-                        (leading + ' '*len(matchBullet.group()), [content]))
-                elif paragraphs and paragraphs[-1][0] == leading:
-                    paragraphs[-1][1].append( content )
-                else:
-                    paragraphs.append((leading, [content]))
-
-        base_indent = (self.hang or '') + base_indent
-
-        for i, (indent, par) in enumerate(paragraphs):
-            if indent is None:
-                if par is None:
-                    paragraphs[i] = ''
-                else:
-                    paragraphs[i] = base_indent + par
-                continue
-
-            par_indent = base_indent + indent
-            self._wrapper.subsequent_indent = par_indent
-            if not i and self.hang:
-                self._wrapper.initial_indent = par_indent[len(self.hang):]
-            else:
-                self._wrapper.initial_indent = par_indent
-
-            # Bulleted lists get indented with a hanging indent
-            bullet = _bullet_re.match(par[0])
-            if bullet:
-                self._wrapper.initial_indent = par_indent[:-len(bullet.group())]
-
-            paragraphs[i] = self._wrapper.fill(' '.join(par))
-        return '\n'.join(paragraphs)
-
 
 class LegacyPyomoFormatter(logging.Formatter):
     """This mocks up the legacy Pyomo log formating.


Test patch:
diff --git a/pyomo/common/tests/test_config.py b/pyomo/common/tests/test_config.py
--- a/pyomo/common/tests/test_config.py
+++ b/pyomo/common/tests/test_config.py
@@ -37,6 +37,7 @@
 from io import StringIO
 
 from pyomo.common.dependencies import yaml, yaml_available, yaml_load_args
+
 def yaml_load(arg):
     return yaml.load(arg, **yaml_load_args)
 
@@ -46,6 +47,8 @@ def yaml_load(arg):
     Bool, Integer, PositiveInt, NegativeInt, NonPositiveInt, NonNegativeInt,
     PositiveFloat, NegativeFloat, NonPositiveFloat, NonNegativeFloat,
     In, ListOf, Module, Path, PathList, ConfigEnum, DynamicImplicitDomain,
+    ConfigFormatter, String_ConfigFormatter,
+    document_kwargs_from_configdict, USER_OPTION, DEVELOPER_OPTION,
     _UnpickleableDomain, _picklable,
 )
 from pyomo.common.log import LoggingIntercept
@@ -1743,28 +1746,85 @@ def test_generate_custom_documentation(self):
   endBlock{flushing}
 endBlock{}
 """
-        test = self.config.generate_documentation(
-            block_start= "startBlock{%s}\n",
-            block_end=   "endBlock{%s}\n",
-            item_start=  "startItem{%s}\n",
-            item_body=   "item{%s}\n",
-            item_end=    "endItem{%s}\n",
-        )
+        with LoggingIntercept() as LOG:
+            test = self.config.generate_documentation(
+                block_start= "startBlock{%s}\n",
+                block_end=   "endBlock{%s}\n",
+                item_start=  "startItem{%s}\n",
+                item_body=   "item{%s}\n",
+                item_end=    "endItem{%s}\n",
+            )
+        LOG = LOG.getvalue().replace('\n', ' ')
+        for name in ('block_start', 'block_end', 'item_start', 'item_end', 'item_body'):
+            self.assertIn(
+                f"Overriding '{name}' by passing strings to "
+                "generate_documentation is deprecated.",
+                LOG
+            )
+        self.maxDiff = None
         #print(test)
         self.assertEqual(test, reference)
 
-        test = self.config.generate_documentation(
-            block_start= "startBlock\n",
-            block_end=   "endBlock\n",
-            item_start=  "startItem\n",
-            item_body=   "item\n",
-            item_end=    "endItem\n",
-        )
+        with LoggingIntercept() as LOG:
+            test = self.config.generate_documentation(format=String_ConfigFormatter(
+                block_start= "startBlock{%s}\n",
+                block_end=   "endBlock{%s}\n",
+                item_start=  "startItem{%s}\n",
+                item_body=   "item{%s}\n",
+                item_end=    "endItem{%s}\n",
+            ))
+        self.assertEqual(LOG.getvalue(), "")
+        self.maxDiff = None
+        #print(test)
+        self.assertEqual(test, reference)
+
+        with LoggingIntercept() as LOG:
+            test = self.config.generate_documentation(
+                block_start= "startBlock\n",
+                block_end=   "endBlock\n",
+                item_start=  "startItem\n",
+                item_body=   "item\n",
+                item_end=    "endItem\n",
+            )
 
         stripped_reference = re.sub(r'\{[^\}]*\}','',reference,flags=re.M)
         #print(test)
         self.assertEqual(test, stripped_reference)
 
+        reference = \
+"""startBlock{}
+  startBlock{network}
+  startBlock{scenario}
+  startBlock{scenarios}
+  startBlock{impact}
+  startBlock{flushing}
+    startBlock{flush nodes}
+    startBlock{close valves}
+"""
+        with LoggingIntercept() as LOG:
+            test = self.config.generate_documentation(
+                block_start="startBlock{%s}\n",
+                block_end="",
+                item_start="",
+                item_body="",
+            )
+        LOG = LOG.getvalue().replace('\n', ' ')
+        for name in ('block_start', 'block_end', 'item_start', 'item_body'):
+            self.assertIn(
+                f"Overriding '{name}' by passing strings to "
+                "generate_documentation is deprecated.",
+                LOG
+            )
+        for name in ('item_end'):
+            self.assertNotIn(
+                f"Overriding '{name}' by passing strings to "
+                "generate_documentation is deprecated.",
+                LOG
+            )
+        self.maxDiff = None
+        #print(test)
+        self.assertEqual(test, reference)
+
 
     def test_generate_latex_documentation(self):
         cfg = ConfigDict()
@@ -1802,6 +1862,16 @@ def test_generate_latex_documentation(self):
 \\end{description}
             """.strip())
 
+    def test_empty_ConfigFormatter(self):
+        cfg = ConfigDict()
+        cfg.declare('field', ConfigValue())
+        with self.assertRaisesRegex(
+                ValueError, "Unrecognized documentation formatter, 'unknown'"):
+            cfg.generate_documentation(format="unknown")
+
+        self.assertEqual(
+            cfg.generate_documentation(format=ConfigFormatter()), ''
+        )
 
     def test_block_get(self):
         self.assertTrue('scenario' in self.config)
@@ -2606,6 +2676,105 @@ def test_declare_from(self):
                 ValueError, "only accepts other ConfigDicts"):
             cfg2.declare_from({})
 
+    def test_docstring_decorator(self):
+        @document_kwargs_from_configdict('CONFIG')
+        class ExampleClass(object):
+            CONFIG = ConfigDict()
+            CONFIG.declare('option_1', ConfigValue(
+                default=5,
+                domain=int,
+                doc='The first configuration option',
+            ))
+            SOLVER = CONFIG.declare('solver_options', ConfigDict())
+            SOLVER.declare('solver_option_1', ConfigValue(
+                default=1,
+                domain=float,
+                doc='The first solver configuration option',
+                visibility=DEVELOPER_OPTION,
+            ))
+            SOLVER.declare('solver_option_2', ConfigValue(
+                default=1,
+                domain=float,
+                doc="""The second solver configuration option
+
+                With a very long line containing
+                wrappable text in a long, silly paragraph
+                with little actual information.
+                #) but a bulleted list
+                #) with two bullets
+                """,
+            ))
+            CONFIG.declare('option_2', ConfigValue(
+                default=5,
+                domain=int,
+                doc="""The second solver configuration option
+                with a very long line containing
+                wrappable text in a long, silly paragraph
+                with little actual information.
+                """,
+            ))
+
+            @document_kwargs_from_configdict(CONFIG)
+            def __init__(self):
+                "A simple docstring"
+
+            @document_kwargs_from_configdict(
+                CONFIG, doc="A simple docstring\n", visibility=USER_OPTION
+            )
+            def fcn(self):
+                pass
+
+
+        ref = """
+Keyword Arguments
+-----------------
+option_1: int, default=5
+    The first configuration option
+
+solver_options: dict, optional
+
+    solver_option_1: float, default=1
+        [DEVELOPER option]
+
+        The first solver configuration option
+
+    solver_option_2: float, default=1
+        The second solver configuration option
+
+        With a very long line containing wrappable text in a long, silly
+        paragraph with little actual information.
+        #) but a bulleted list
+        #) with two bullets
+
+option_2: int, default=5
+    The second solver configuration option with a very long line
+    containing wrappable text in a long, silly paragraph with little
+    actual information."""
+        self.assertEqual(ExampleClass.__doc__, ref.lstrip())
+        self.assertEqual(ExampleClass.__init__.__doc__, "A simple docstring\n" + ref)
+
+        ref = """
+Keyword Arguments
+-----------------
+option_1: int, default=5
+    The first configuration option
+
+solver_options: dict, optional
+
+    solver_option_2: float, default=1
+        The second solver configuration option
+
+        With a very long line containing wrappable text in a long, silly
+        paragraph with little actual information.
+        #) but a bulleted list
+        #) with two bullets
+
+option_2: int, default=5
+    The second solver configuration option with a very long line
+    containing wrappable text in a long, silly paragraph with little
+    actual information."""
+        self.assertEqual(ExampleClass.fcn.__doc__, "A simple docstring\n" + ref)
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/pyomo/common/tests/test_log.py b/pyomo/common/tests/test_log.py
--- a/pyomo/common/tests/test_log.py
+++ b/pyomo/common/tests/test_log.py
@@ -346,6 +346,36 @@ def test_verbatim(self):
             "  - including a\n"
             "    long list\n"
             "  - and a short list \n"
+            "\n"
+            "And a section\n"
+            "~~~~~~~~~~~~~\n"
+            "\n"
+            "  | and\n"
+            "  | a line\n"
+            "  | block\n"
+            "\n"
+            "And a\n"
+            "quoted literal::\n"
+            "\n"
+            ">> he said\n"
+            ">\n"
+            "> and they replied\n"
+            "\n"
+            "this is\n"
+            "outside the quote\n"
+            "\n"
+            "indented literal::\n"
+            "\n"
+            "    Here is\n"
+            "       an indented\n"
+            "\n"
+            "    literal\n"
+            "    with a blank line\n"
+            "\n"
+            "Finally, an invalid::\n"
+            "\n"
+            "quote\n"
+            "block\n"
         )
         logger.setLevel(logging.WARNING)
         logger.warning(msg)
@@ -362,7 +392,35 @@ def test_verbatim(self):
             "    And some internal non-verbatim\n"
             "      - including a long list\n"
             "      - and a short list\n"
+            "\n"
+            "    And a section\n"
+            "    ~~~~~~~~~~~~~\n"
+            "\n"
+            "      | and\n"
+            "      | a line\n"
+            "      | block\n"
+            "\n"
+            "    And a quoted literal::\n"
+            "\n"
+            "    >> he said\n"
+            "    >\n"
+            "    > and they replied\n"
+            "\n"
+            "    this is outside the quote\n"
+            "\n"
+            "    indented literal::\n"
+            "\n"
+            "        Here is\n"
+            "           an indented\n"
+            "\n"
+            "        literal\n"
+            "        with a blank line\n"
+            "\n"
+            "    Finally, an invalid::\n"
+            "\n"
+            "    quote block\n"
         )
+        self.maxDiff = None
         self.assertEqual(self.stream.getvalue(), ans)
 
 class TestLogStream(unittest.TestCase):