Repository: spack/spack

Issue:
The build time for docs needs to be reduced
Very often the documentation badge on `develop` says *failing*. The cause is the time it takes to build the docs, that may exceed the maximum allowed on readthedocs (15 min.) An example of a failing build is [here](https://readthedocs.org/projects/spack/builds/6518192/).

### Expected Result

A  documentation badge that says *passing* - in line with the "docs" unit test that passed on Travis

### Actual Result

Sometimes we have a *passing* result, sometimes a *failing*, depending whether or not readthedocs can build our documentation in less than 15 min.


Discussion:
This will be difficult. We can't build in parallel as there are all kinds of cross-links everywhere. If we replaced all of the calls to `programoutput` with the expected output, it would speed things up a bit, but then things will get rapidly out of date. There's also a lot of docstrings to scrape, which I think is the bulk of the time.
Actually, it looks like the majority of the time is spent reading `package_list.rst`. If we dropped that, we could probably save 1/3 of the time it takes to build the docs. When I build locally on my laptop, it only takes 6 minutes, so I'm surprised it's timing out. Perhaps it's just a `pip` problem like with Travis lately?
>  When I build locally on my laptop, it only takes 6 minutes, so I'm surprised it's timing out. Perhaps it's just a pip problem like with Travis lately?

I don't think so. Maybe there's a way to check that, but I believe it's more likely people offering the services are somehow oversubscribing their machines to maximize the usage. It's the same thing we see (or at least I see) on Travis: unit tests on my laptop take consistently around 220sec. while on Travis there's more variance and 500sec. is not an unusual wall-clock time.
Does anyone have any other ideas on how to speed up the documentation builds? We haven't had a single build pass in 5 days now. The best idea I've come up with so far is to drop `package_list.rst` altogether, but this is rather drastic. @tgamblin?
> > When I build locally on my laptop, it only takes 6 minutes, so I'm surprised it's timing out. Perhaps it's just a pip problem like with Travis lately?
>
> I don't think so. Maybe there's a way to check that, but I believe it's more likely people offering the services are somehow oversubscribing their machines to maximize the usage. It's the same thing we see (or at least I see) on Travis: unit tests on my laptop take consistently around 220sec. while on Travis there's more variance and 500sec. is not an unusual wall-clock time.

The documentation tests only take 9 minutes on Travis, so I don't think it's a server problem. It looks like they run multiple commands to build the HTML docs and PDF docs, so I think that's the problem. I wonder if we can turn off the PDF generation?
It looks like we can turn off PDF builds. Want me to try that? PDFs are cute for releases, but I don't think we need to constantly update our PDFs.
> Does anyone have any other ideas on how to speed up the documentation builds? We haven't had a single build pass in 5 days now. The best idea I've come up with so far is to drop package_list.rst altogether, but this is rather drastic. @tgamblin?

Maybe we can generate it once on Travis, and deploy it somewhere?

Resolving patch:
diff --git a/lib/spack/docs/conf.py b/lib/spack/docs/conf.py
--- a/lib/spack/docs/conf.py
+++ b/lib/spack/docs/conf.py
@@ -68,9 +68,10 @@
 #
 # Generate package list using spack command
 #
-with open('package_list.rst', 'w') as plist_file:
+with open('package_list.html', 'w') as plist_file:
     subprocess.Popen(
-        [spack_root + '/bin/spack', 'list', '--format=rst'], stdout=plist_file)
+        [spack_root + '/bin/spack', 'list', '--format=html'],
+        stdout=plist_file)
 
 #
 # Find all the `cmd-spack-*` references and add them to a command index
diff --git a/lib/spack/spack/cmd/list.py b/lib/spack/spack/cmd/list.py
--- a/lib/spack/spack/cmd/list.py
+++ b/lib/spack/spack/cmd/list.py
@@ -23,12 +23,14 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
 from __future__ import print_function
+from __future__ import division
 
 import argparse
 import cgi
 import fnmatch
 import re
 import sys
+import math
 
 from six import StringIO
 
@@ -113,23 +115,35 @@ def name_only(pkgs):
     colify(pkgs, indent=indent)
 
 
-@formatter
-def rst(pkgs):
-    """Print out information on all packages in restructured text."""
+def github_url(pkg):
+    """Link to a package file on github."""
+    url = 'https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/{0}/package.py'
+    return url.format(pkg.name)
+
+
+def rst_table(elts):
+    """Print out a RST-style table."""
+    cols = StringIO()
+    ncol, widths = colify(elts, output=cols, tty=True)
+    header = ' '.join('=' * (w - 1) for w in widths)
+    return '%s\n%s%s' % (header, cols.getvalue(), header)
 
-    def github_url(pkg):
-        """Link to a package file on github."""
-        url = 'https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/{0}/package.py'
-        return url.format(pkg.name)
 
-    def rst_table(elts):
-        """Print out a RST-style table."""
-        cols = StringIO()
-        ncol, widths = colify(elts, output=cols, tty=True)
-        header = ' '.join('=' * (w - 1) for w in widths)
-        return '%s\n%s%s' % (header, cols.getvalue(), header)
+def rows_for_ncols(elts, ncols):
+    """Print out rows in a table with ncols of elts laid out vertically."""
+    clen = int(math.ceil(len(elts) / ncols))
+    for r in range(clen):
+        row = []
+        for c in range(ncols):
+            i = c * clen + r
+            row.append(elts[i] if i < len(elts) else None)
+        yield row
+
+
+@formatter
+def rst(pkg_names):
+    """Print out information on all packages in restructured text."""
 
-    pkg_names = pkgs
     pkgs = [spack.repo.get(name) for name in pkg_names]
 
     print('.. _package-list:')
@@ -183,6 +197,102 @@ def rst_table(elts):
         print()
 
 
+@formatter
+def html(pkg_names):
+    """Print out information on all packages in Sphinx HTML.
+
+    This is intended to be inlined directly into Sphinx documentation.
+    We write HTML instead of RST for speed; generating RST from *all*
+    packages causes the Sphinx build to take forever. Including this as
+    raw HTML is much faster.
+    """
+
+    # Read in all packages
+    pkgs = [spack.repo.get(name) for name in pkg_names]
+
+    # Start at 2 because the title of the page from Sphinx is id1.
+    span_id = 2
+
+    # HTML header with an increasing id span
+    def head(n, span_id, title, anchor=None):
+        if anchor is None:
+            anchor = title
+        print(('<span id="id%d"></span>'
+               '<h1>%s<a class="headerlink" href="#%s" '
+               'title="Permalink to this headline">&para;</a>'
+               '</h1>') % (span_id, title, anchor))
+
+    # Start with the number of packages, skipping the title and intro
+    # blurb, which we maintain in the RST file.
+    print('<p>')
+    print('Spack currently has %d mainline packages:' % len(pkgs))
+    print('</p>')
+
+    # Table of links to all packages
+    print('<table border="1" class="docutils">')
+    print('<tbody valign="top">')
+    for i, row in enumerate(rows_for_ncols(pkg_names, 3)):
+        print('<tr class="row-odd">' if i % 2 == 0 else
+              '<tr class="row-even">')
+        for name in row:
+            print('<td>')
+            print('<a class="reference internal" href="#%s">%s</a></td>'
+                  % (name, name))
+            print('</td>')
+        print('</tr>')
+    print('</tbody>')
+    print('</table>')
+    print('<hr class="docutils"/>')
+
+    # Output some text for each package.
+    for pkg in pkgs:
+        print('<div class="section" id="%s">' % pkg.name)
+        head(2, span_id, pkg.name)
+        span_id += 1
+
+        print('<dl class="docutils">')
+
+        print('<dt>Homepage:</dt>')
+        print('<dd><ul class="first last simple">')
+        print(('<li>'
+               '<a class="reference external" href="%s">%s</a>'
+               '</li>') % (pkg.homepage, cgi.escape(pkg.homepage)))
+        print('</ul></dd>')
+
+        print('<dt>Spack package:</dt>')
+        print('<dd><ul class="first last simple">')
+        print(('<li>'
+               '<a class="reference external" href="%s">%s/package.py</a>'
+               '</li>') % (github_url(pkg), pkg.name))
+        print('</ul></dd>')
+
+        if pkg.versions:
+            print('<dt>Versions:</dt>')
+            print('<dd>')
+            print(', '.join(str(v) for v in reversed(sorted(pkg.versions))))
+            print('</dd>')
+
+        for deptype in spack.all_deptypes:
+            deps = pkg.dependencies_of_type(deptype)
+            if deps:
+                print('<dt>%s Dependencies:</dt>' % deptype.capitalize())
+                print('<dd>')
+                print(', '.join(
+                    d if d not in pkg_names else
+                    '<a class="reference internal" href="#%s">%s</a>' % (d, d)
+                    for d in deps))
+                print('</dd>')
+
+        print('<dt>Description:</dt>')
+        print('<dd>')
+        print(cgi.escape(pkg.format_doc(indent=2)))
+        print('</dd>')
+        print('</dl>')
+
+        print('<hr class="docutils"/>')
+        print('</div>')
+
+
 def list(parser, args):
     # Retrieve the names of all the packages
     pkgs = set(spack.repo.all_package_names())


Test patch: