Repository: spotify/luigi

Issue:
Docs hosted on readthedocs.io are missing content
I've noticed that a few pages in the contribs documentation appear not to have been built properly, and are missing all of the actual content:

http://luigi.readthedocs.io/en/stable/api/luigi.contrib.hadoop.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.hadoop_jar.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.hive.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.opener.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.redshift.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.s3.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.scalding.html
http://luigi.readthedocs.io/en/stable/api/luigi.contrib.sge.html

These are all of the pages in `luigi/contrib`, but I have no idea if this is isolated to just here. The docstrings are definitely there.


Discussion:
Interesting... Thanks for raising @birdcolour 

@Tarrasch , i have no experience with our readthedocs sphinx implementation. Do you have any ideas or recommendations here?
Actually, looks like @jcrobak did some of the sphinx setup. Maybe he can help too
Hmm... Feel free to take a look @birdcolour, I unfortunately don't have the time to dive in.
It looks like the "API Documentation" section, which describes the steps required for new packages didn't make it through a migration 5899a3d231a8f59216e370f4bd552aef43e34e88

tl;dr: `pip install sphinx && sphinx-apidoc -o doc/api -T luigi` will rebuild`.

> With that said, `tox -e docs` is failing for me related to https://github.com/sphinx-doc/sphinx/issues/2443

Edit: _nevermind, I was on a stale master_ - seeing a different error on HEAD.

HTH!
Hey, any update on this fix? 
I have not yet taken an indepth look yet. @lizmeister321 would you like to take a look and possibly contribute a fix? :)
If i build docs locally, i get all the correct api html files
Might be related: the `sphinx-build` command from `tox.ini` fails in `luigi/configuration/cfg_parser.py` because `LuigiConfigParser.get` has no docstring, and so Sphinx takes the inherited docstring from the stdlib `ConfigParser.get` which contains odd unbalanced quotes (open with \` and close with ') that trigger a warning in Sphinx's parsing. Since the build has warnings as errors enabled the docs build aborts. Same thing happens if you try to override `has_option` or another method of `ConfigParser` that uses the same quote convention without supplying a new docstring without that convention.

If you add a docstring (empty or otherwise) to `LuigiConfigParser.get` to avoid the inheritance the build succeeds. Not sure if it's a configparser bug because the quote convention is weird, a Sphinx bug because it can't parse docstrings in the stdlib, or something that should just be fixed in this project to avoid inheriting this specific docstring.

The docstring value in question on my system, for reference:
> "Get an option value for a given section.\n\n        If \`vars' is provided, it must be a dictionary. The option is looked up\n        in \`vars' (if provided), \`section', and in \`DEFAULTSECT' in that order.\n        If the key is not found and \`fallback' is provided, it is used as\n        a fallback value. \`None' can be provided as a \`fallback' value.\n\n        If interpolation is enabled and the optional argument \`raw' is False,\n        all interpolations are expanded in the return values.\n\n        Arguments \`raw', \`vars', and \`fallback' are keyword only.\n\n        The section DEFAULT is special.\n        "

And the build error:
> Warning, treated as error: 
> docstring of luigi.configuration.LuigiConfigParser.get:3:Inline interpreted text or phrase reference start-string without end-string.
Ah, sorry @dlstadther, that only applies if you try to build the docs in Python 3, and RTD is building in Py2 so it's a different problem.

Going through the build log, to fix these pages it looks like it's a simple case of boto3 not being installed in the RTD environment.

Adding `install_requires.append('boto3')` to `setup.py` alongside sqlalchemy in the RTD check should fix this properly for the RTD build on Python2.7.
Hey @jethron Could you link me to the build log you're referring to?
https://readthedocs.org/projects/luigi/builds/7832345/

Is one of the latest builds that passed, it has started failing since some of the latest merges. At the bottom of the 9th step you see a bunch of import failures for botocore.
Ah yes I recall there's some silent failures happening on that site. I think we're on the right track.

Resolving patch:
diff --git a/luigi/contrib/s3.py b/luigi/contrib/s3.py
--- a/luigi/contrib/s3.py
+++ b/luigi/contrib/s3.py
@@ -31,9 +31,6 @@
 import warnings
 from multiprocessing.pool import ThreadPool
 
-import botocore
-from boto3.s3.transfer import TransferConfig
-
 try:
     from urlparse import urlsplit
 except ImportError:
@@ -54,6 +51,13 @@
 
 logger = logging.getLogger('luigi-interface')
 
+try:
+    from boto3.s3.transfer import TransferConfig
+    import botocore
+except ImportError:
+    logger.warning("Loading S3 module without the python package boto3. "
+                   "Will crash at runtime if S3 functionality is used.")
+
 
 # two different ways of marking a directory
 # with a suffix in S3
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -13,7 +13,6 @@
 # the License.
 
 import os
-import sys
 
 from setuptools import setup
 
@@ -41,18 +40,16 @@ def get_static_files(path):
     'tornado>=4.0,<5',
     # https://pagure.io/python-daemon/issue/18
     'python-daemon<2.2.0',
+    'enum34>1.1.0;python_version<"3.4"',
 ]
 
 if os.environ.get('READTHEDOCS', None) == 'True':
     # So that we can build documentation for luigi.db_task_history and luigi.contrib.sqla
     install_requires.append('sqlalchemy')
     # readthedocs don't like python-daemon, see #1342
-    install_requires.remove('python-daemon<3.0')
+    install_requires.remove('python-daemon<2.2.0')
     install_requires.append('sphinx>=1.4.4')  # Value mirrored in doc/conf.py
 
-if sys.version_info < (3, 4):
-    install_requires.append('enum34>1.1.0')
-
 setup(
     name='luigi',
     version='2.7.9',


Test patch: