Repository: hylang/hy

Issue:
set up Sphinx autodoc to put our docstrings in our reference docs.
There is considerable duplication of documentation between the reStructuredText docs and Hy's internal docstrings. I propose we move as much as reasonably possible into the docstrings themselves, to make the references more accessible at the repl, and for language tools like emacs. @jkitchin has started this effort with #1040.

I think Sphinx can fetch docstrings [programmatically](http://www.sphinx-doc.org/en/stable/ext/autodoc.html) from Python objects rather than parsing them from text. This means that it could theoretically work for Hy docstrings as well, though I've never seen this done before.

Our documentation effort is lagging behind Hy's actual capabilities. Part of this is due to the extra effort required to document things in two places, and then keep them in sync afterwards.

The new setup will require considerable reorganization of the documentation. For each function (and potentially class, module etc.) we'll have to take the better of the two or combine them somehow.

Our testing is similarly lagging behind. While Hy's doctests shouldn't completely replace unit tests (the priority must be understandable examples, rather than completeness), it is better than the nothing we sometimes get. More importantly, making our reference examples into doctests in our docstrings will ensure that they actually work as intended. Currently, many of our example docs are broken. They should all be automatically testable, even those from the tutorial.

Doctests show a Python repl session and don't currently work with a Hy repl, but we need our Hy examples to be in Hy, not Python. See #1019.

Hy lacks Python's (raw) triple quotes. Hy's strings can certainly represent any possible text given enough backslashes, but this will make it difficult to write reStructuredText (which also needs backslash escapes) and Hy doctests (which will certainly use quoted strings themselves) in our docstrings. I've proposed adding heredocs as string literals to the reader (#1042) to resolve this. 

Docstring comes out in expansion of defmacro!
Macros defined with `defmacro!` put too much in the expansion. This isn't obvious unless you have `--spy` on or use `disassemble`. I also see a lot of `None`s for some reason. It seems to be the result of each form in the `defmacro!` definition, instead of just the final one.
```Python
=> (defmacro! foo [] "doc" 42)
from hy.core.language import interleave
import hy
from hy import HyExpression, HySymbol

def _hy_anon_fn_1():
    None
    return HyExpression(((([] + [HySymbol('do')]) + [HyExpression((([] + [HySymbol('setv')]) + list((interleave([], []) or []))))]) + list((['doc', 42] or []))))
hy.macros.macro('foo')(_hy_anon_fn_1)
<function _hy_anon_fn_1 at 0x000001E6E05F4F28>
=> (foo)
None
'doc'
42
42
=> (disassemble '(foo))
from hy.core.language import disassemble
from hy import HyExpression, HySymbol
disassemble(HyExpression(([] + [HySymbol('foo')])))
"Module(body=[Expr(value=Name(id='None')), Expr(value=Str(s='doc')), Expr(value=Num(n=42))])"
```


Discussion:
I am not sure you can get the autodoc feature of Sphinx to work with hy the way it does with Python. There is a lot of redirection between hy functions/macros and where the doc strings are. For example, here https://github.com/jkitchin/hyve/blob/master/hylp.hy#L103 I expanded some code you posted earlier to get the Python object for a given hy symbol, so that I can use the python tools for accessing docstrings, source files and linenumbers. I am pretty sure Sphinx would see things from the Python side in using the autodoc features.

On the other hand it is not that difficult to write code that would generate html docs from these docstrings. pydoc already has a lot of this framework. So there probably could be some code in the Sphinx docs that generates documentation for the hy functions and macros.

The benefit of this approach would be getting things the way you want them, e.g. sorted, or separated by categories, etc... The downside is more than one line of .. autodoc: ...

I'd have to re-read the Sphinx documentation. We might have to experiment to find the best approach.

> I expanded some code you posted earlier

I used the or/false short-circuiting trick to avoid evaluation. You don't need it in your macro (you evaluate `sym` anyway for `do-import`), so It could be simplified:

``` Hy
(defmacro get-python-object [sym]
  "Get the Python object for the symbol SYM.
  SYM is a function or module.
  If SYM has a . in it, import the base module, unless the dot is the first character. We can't get these yet."
  `(do
     (do-import ~sym)
     (try
       (->> ~sym
            (.get hy.compiler._compile_table '~sym)
            (.get (get hy.macros._hy_macros nil) '~sym)
            (.get (get hy.macros._hy_macros "__main__") '~sym)
            (.get (get hy.macros._hy_macros "__console__") '~sym)
            (.get (vars hy.core.shadow) '~sym)
            (.get (vars hy.core.language) '~sym))
       (except [e NameError] None)))
```

I tried out autodoc a bit. It can get us surprisingly far, but it chokes on function names that are not valid Python identifiers. Sphinx is supposed to be pretty extensible though. We might be able to implement [our own extension](http://www.sphinx-doc.org/en/stable/extdev/index.html) for Hy files. This would also be useful tooling for third-party Hy users to have.
My Hy domain sphinx extension has autodoc working for almost everything we'd need for hy's core library and I've started the overhaul that would close this issue. It's going to be a massive pull request though since it'll touch the docstrings of almost every hy builtin, contrib, and extras modules. Do y'all have any guidance for how you'd want to approach the pull request? it's just a tad bigger than any pull request I've done for hy before lol. Also any requests for things y'all want for the hy docs while I'm at it? So far I have a `Cheatsheet.rst` with categories and links to each builtin/contrib method and a unified `API.rst` with autodoc'd methods for `hy.core.language`, `macros`, `bootstrap`, and `shadow`. Lastly, I'm using `sphinx.ext.napoleon` to make the docstrings easier to write and follow a consistent format. Are we okay with this?

Here's what the docstring for the `when` macro look like now.

```clj
(defmacro with [args &rest body]
  "Wrap execution of `body` within a context manager given as bracket `args`.

  ``with`` is used to wrap the execution of a block within a context manager. The
  context manager can then set up the local system and tear it down in a controlled
  manner. The archetypical example of using ``with`` is when processing files.
  ``with`` can bind context to an argument or ignore it completely, as shown below:

  Examples:
    >>> (with [arg (expr)] block)
    >>> (with [(expr)] block)
    >>> (with [arg (expr) (expr)] block)

    The following example will open the ``NEWS`` file and print its content to the
    screen. The file is automatically closed after it has been processed.

    >>> (with [f (open \"NEWS\")] (print (.read f)))

    ``with`` returns the value of its last form, unless it suppresses an exception
    (because the context manager's ``__exit__`` method returned true), in which
    case it returns ``None``. So, the previous example could also be written

    >>> (print (with [f (open \"NEWS\")] (.read f)))

    Shorthand for nested with* loops:

    >>> (with [x foo y bar] baz)
    (with* [x foo]
      (with* [y bar]
        baz)).
  "
  (_with 'with* args body))

```
> Do y'all have any guidance for how you'd want to approach the pull request

Just make sure the massive copy-paste from the docs files into docstrings are in a separate commit from the more interesting code or documentation changes. My philosophy is that changes that create a lot diff lines, but don't involve any real programming that a reader would want to review, should be separated, by commits, from the real programming.

I haven't used the napoleon syntax before, but I think it's fine so long as it supports all the reST constructs that we need.
> Just make sure the massive copy-paste from the docs files into docstrings are in a separate commit from the more interesting code or documentation changes. My philosophy is that changes that create a lot diff lines, but don't involve any real programming that a reader would want to review, should be separated, by commits, from the real programming.

A lot of the heavy lifting is done by `hydomain` so I don't think there should be many code changes :crossed_fingers: I'll make sure any are in their own commits though 

> I haven't used the napoleon syntax before, but I think it's fine so long as it supports all the reST constructs that we need.

I've used Napoleon (Google style docstrings) for pretty much every python project I work on and have found it much easier to navigate than reST style. You can still use all the regular reST constructs if you want, it just makes certain things easier to write and read. [example](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)

I suspect this is due to a compiler mutation #1542.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -3,11 +3,18 @@
 # This file is execfile()d with the current directory set to its containing dir.
 
 import re, os, sys, time, html
-sys.path.append(os.path.abspath(".."))
 
-extensions = ['sphinx.ext.intersphinx']
+sys.path.insert(0, os.path.abspath('..'))
+
+extensions = [
+    'sphinx.ext.napoleon',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.autodoc',
+    'sphinxcontrib.hydomain',
+]
 
 from get_version import __version__ as hy_version
+
 # Read the Docs might dirty its checkout, so strip the dirty flag.
 hy_version = re.sub(r'[+.]dirty\Z', '', hy_version)
 
@@ -33,6 +40,7 @@
     hy_descriptive_version += " <strong style='color: red;'>(unstable)</strong>"
 
 exclude_patterns = ['_build', 'coreteam.rst']
+add_module_names = False
 
 pygments_style = 'sphinx'
 
@@ -51,5 +59,79 @@
 html_context = dict(
     hy_descriptive_version = hy_descriptive_version)
 
+highlight_language = 'clojure'
+
 intersphinx_mapping = dict(
     py = ('https://docs.python.org/3/', None))
+# ** Generate Cheatsheet
+import json
+from pathlib import Path
+from hy.core.language import partition
+
+def refize(spec):
+    role = ':hy:func:'
+    if isinstance(spec, dict):
+        _name = spec['name']
+        uri = spec['uri']
+        if spec.get('internal'):
+            role = ':ref:'
+    else:
+        uri = spec
+        _name = str.split(uri, '.')[-1]
+    return '{}`{} <{}>`'.format(role, _name, uri)
+
+
+def format_refs(refs, indent):
+    ref_groups = partition(map(refize, refs), fillvalue='')
+    return str.join(
+        ' \\\n' + ' ' * (indent + 3),
+        [str.join(' ', ref_group) for ref_group in ref_groups],
+    )
+
+
+def format_row(category, divider_loc):
+    return '{title: <{width}} | {methods}'.format(
+        width=divider_loc,
+        title=category['name'],
+        methods=format_refs(category['methods'], divider_loc)
+    )
+
+
+def format_table(table_spec):
+    table_name = table_spec['name']
+    categories = table_spec['categories']
+    longest_cat_name = max([len(category['name']) for category in categories])
+    table = [
+        table_name,
+        '-' * len(table_name),
+        '',
+        '=' * longest_cat_name + ' ' + '=' * 25,
+        *(format_row(category, longest_cat_name) for category in categories),
+        '=' * longest_cat_name + ' ' + '=' * 25,
+        ''
+    ]
+    return '\n'.join(table)
+
+
+# Modifications to the cheatsheet should be added in `cheatsheet.json`
+cheatsheet_spec = json.loads(Path('./docs/cheatsheet.json').read_text())
+cheatsheet = [
+    '..',
+    '    DO NOT MODIFY THIS FILE. IT IS AUTO GENERATED BY ``conf.py``',
+    '    If you need to change or add methods, modify ``cheatsheet_spec`` in ``conf.py``',
+    '',
+    '.. _cheatsheet:',
+    '',
+    'Cheatsheet',
+    '==========',
+    '',
+    *map(format_table, cheatsheet_spec),
+]
+Path('./docs/cheatsheet.rst').write_text('\n'.join(cheatsheet))
+
+
+# ** Sphinx App Setup
+
+
+def setup(app):
+    app.add_css_file('overrides.css')
diff --git a/hy/compiler.py b/hy/compiler.py
--- a/hy/compiler.py
+++ b/hy/compiler.py
@@ -1936,57 +1936,57 @@ def hy_eval(hytree, locals=None, module=None, ast_callback=None,
     are set properly.  Try `fix_missing_locations` and related functions in the
     Python `ast` library.
 
-    Examples
-    --------
-       => (eval '(print "Hello World"))
-       "Hello World"
-
-    If you want to evaluate a string, use ``read-str`` to convert it to a
-    form first:
-       => (eval (read-str "(+ 1 1)"))
-       2
-
-    Parameters
-    ----------
-    hytree: HyObject
-        The Hy AST object to evaluate.
-
-    locals: dict, optional
-        Local environment in which to evaluate the Hy tree.  Defaults to the
-        calling frame.
-
-    module: str or types.ModuleType, optional
-        Module, or name of the module, to which the Hy tree is assigned and
-        the global values are taken.
-        The module associated with `compiler` takes priority over this value.
-        When neither `module` nor `compiler` is specified, the calling frame's
-        module is used.
-
-    ast_callback: callable, optional
-        A callback that is passed the Hy compiled tree and resulting
-        expression object, in that order, after compilation but before
-        evaluation.
-
-    compiler: HyASTCompiler, optional
-        An existing Hy compiler to use for compilation.  Also serves as
-        the `module` value when given.
-
-    filename: str, optional
-        The filename corresponding to the source for `tree`.  This will be
-        overridden by the `filename` field of `tree`, if any; otherwise, it
-        defaults to "<string>".  When `compiler` is given, its `filename` field
-        value is always used.
-
-    source: str, optional
-        A string containing the source code for `tree`.  This will be
-        overridden by the `source` field of `tree`, if any; otherwise,
-        if `None`, an attempt will be made to obtain it from the module given by
-        `module`.  When `compiler` is given, its `source` field value is always
-        used.
-
-    Returns
-    -------
-    out : Result of evaluating the Hy compiled tree.
+    Examples:
+      ::
+
+         => (eval '(print "Hello World"))
+         "Hello World"
+
+      If you want to evaluate a string, use ``read-str`` to convert it to a
+      form first::
+
+         => (eval (read-str "(+ 1 1)"))
+         2
+
+    Args:
+      hytree (HyObject):
+          The Hy AST object to evaluate.
+
+      locals (dict, optional):
+          Local environment in which to evaluate the Hy tree.  Defaults to the
+          calling frame.
+
+      module (str or types.ModuleType, optional):
+          Module, or name of the module, to which the Hy tree is assigned and
+          the global values are taken.
+          The module associated with `compiler` takes priority over this value.
+          When neither `module` nor `compiler` is specified, the calling frame's
+          module is used.
+
+      ast_callback (callable, optional):
+          A callback that is passed the Hy compiled tree and resulting
+          expression object, in that order, after compilation but before
+          evaluation.
+
+      compiler (HyASTCompiler, optional):
+          An existing Hy compiler to use for compilation.  Also serves as
+          the `module` value when given.
+
+      filename (str, optional):
+          The filename corresponding to the source for `tree`.  This will be
+          overridden by the `filename` field of `tree`, if any; otherwise, it
+          defaults to "<string>".  When `compiler` is given, its `filename` field
+          value is always used.
+
+      source (str, optional):
+          A string containing the source code for `tree`.  This will be
+          overridden by the `source` field of `tree`, if any; otherwise,
+          if `None`, an attempt will be made to obtain it from the module given by
+          `module`.  When `compiler` is given, its `source` field value is always
+          used.
+
+    Returns:
+      Result of evaluating the Hy compiled tree.
     """
 
     module = get_compiler_module(module, compiler, True)
diff --git a/hy/lex/__init__.py b/hy/lex/__init__.py
--- a/hy/lex/__init__.py
+++ b/hy/lex/__init__.py
@@ -21,17 +21,12 @@
 def hy_parse(source, filename='<string>'):
     """Parse a Hy source string.
 
-    Parameters
-    ----------
-    source: string
-        Source code to parse.
+    Args:
+      source (string): Source code to parse.
+      filename (string, optional): File name corresponding to source.  Defaults to "<string>".
 
-    filename: string, optional
-        File name corresponding to source.  Defaults to "<string>".
-
-    Returns
-    -------
-    out : HyExpression
+    Returns:
+      out : HyExpression
     """
     _source = re.sub(r'\A#!.*', '', source)
     res = HyExpression([HySymbol("do")] +
@@ -51,12 +46,9 @@ def __init__(self, source, filename):
 def tokenize(source, filename=None):
     """ Tokenize a Lisp file or string buffer into internal Hy objects.
 
-    Parameters
-    ----------
-    source: str
-        The source to tokenize.
-    filename: str, optional
-        The filename corresponding to `source`.
+    Args:
+    source (str): The source to tokenize.
+    filename (str, optional): The filename corresponding to `source`.
     """
     from hy.lex.lexer import lexer
     from hy.lex.parser import parser
@@ -106,7 +98,14 @@ def parse_one_thing(src_string):
 
 def mangle(s):
     """Stringify the argument and convert it to a valid Python identifier
-    according to Hy's mangling rules."""
+    according to Hy's mangling rules.
+
+    Examples:
+      ::
+
+         => (mangle "foo-bar")
+         'foo_bar'
+    """
     def unicode_char_to_hex(uchr):
         # Covert a unicode char to hex string, without prefix
         if len(uchr) == 1 and ord(uchr) < 128:
@@ -145,7 +144,14 @@ def unicode_char_to_hex(uchr):
 def unmangle(s):
     """Stringify the argument and try to convert it to a pretty unmangled
     form. This may not round-trip, because different Hy symbol names can
-    mangle to the same Python identifier."""
+    mangle to the same Python identifier.
+
+    Examples:
+      ::
+
+         => (unmangle "foo_bar")
+         'foo-bar'
+    """
 
     s = str(s)
 
@@ -173,6 +179,59 @@ def read(from_file=sys.stdin, eof=""):
 
     Can take a given input buffer to read from, and a single byte as EOF
     (defaults to an empty string).
+
+    Reads the next Hy expression from *from-file* (defaulting to ``sys.stdin``), and
+    can take a single byte as EOF (defaults to an empty string). Raises ``EOFError``
+    if *from-file* ends before a complete expression can be parsed.
+
+    Examples:
+      ::
+
+         => (read)
+         (+ 2 2)
+         HyExpression([
+           HySymbol('+'),
+           HyInteger(2),
+           HyInteger(2)])
+
+      ::
+
+         => (eval (read))
+         (+ 2 2)
+         4
+
+      ::
+
+         => (import io)
+         => (setv buffer (io.StringIO "(+ 2 2)\\n(- 2 1)"))
+         => (eval (read :from-file buffer))
+         4
+         => (eval (read :from-file buffer))
+         1
+
+      ::
+
+         => (with [f (open "example.hy" "w")]
+         ...  (.write f "(print 'hello)\\n(print \"hyfriends!\")"))
+         35
+         => (with [f (open "example.hy")]
+         ...  (try (while True
+         ...         (setv exp (read f))
+         ...         (print "OHY" exp)
+         ...         (eval exp))
+         ...       (except [e EOFError]
+         ...         (print "EOF!"))))
+         OHY HyExpression([
+           HySymbol('print'),
+           HyExpression([
+             HySymbol('quote'),
+             HySymbol('hello')])])
+         hello
+         OHY HyExpression([
+           HySymbol('print'),
+           HyString('hyfriends!')])
+         hyfriends!
+         EOF!
     """
     buff = ""
     while True:
@@ -190,6 +249,22 @@ def read(from_file=sys.stdin, eof=""):
 
 
 def read_str(input):
+    """This is essentially a wrapper around `read` which reads expressions from a
+    string
+
+    Examples:
+      ::
+
+         => (read-str "(print 1)")
+         HyExpression([
+         HySymbol('print'),
+         HyInteger(1)]
+
+      ::
+
+         => (eval (read-str "(print 1)"))
+         1
+  """
     return read(StringIO(str(input)))
 
 


Test patch: