Repository: statsmodels/statsmodels

Issue:
DOC: Doc clean-up and additions
It feels like I'm on a doc binge. But writing docs is often a good way to better understand the tool you are using. 

After playing with the website for a bit, I thought to myself that it would be nice if we could clean-up some of the docs. There's some neat features of stats models that I think the docs could advertise more. I cleaned up the docs and removed some complicated instructions for things that are out of date (I doubt any 32 bit windows user is developing, they'll just install from conda or pip wheels) so I removed some things that are old. 

I'm happy to take a stab at adding some other docs--even if just a stub for now--if it can encourage us to organize some of this better. Things like robust covariance are neat and the fact that stats models make it so easy should be a relatively big selling point. Alas, we don't have docs/examples that really show it off, so at least starting it might be a good idea. If there are others, let me know. 


Discussion:

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -269,7 +269,8 @@
 epub_title = u'statsmodels'
 epub_author = u'Josef Perktold, Skipper Seabold'
 epub_publisher = u'Josef Perktold, Skipper Seabold'
-epub_copyright = u'2009-2017, Josef Perktold, Skipper Seabold, Jonathan Taylor, statsmodels-developers'
+epub_copyright = u'2009-2019, Josef Perktold, Skipper Seabold, ' \
+                 u'Jonathan Taylor, statsmodels-developers'
 
 # The language of the text. It defaults to the language option
 # or en if the language is not set.
diff --git a/statsmodels/base/optimizer.py b/statsmodels/base/optimizer.py
--- a/statsmodels/base/optimizer.py
+++ b/statsmodels/base/optimizer.py
@@ -95,6 +95,33 @@ def _fit(self, objective, gradient, start_params, fargs, kwargs,
                 epsilon
                     If fprime is approximated, use this value for the step
                     size. Only relevant if LikelihoodModel.score is None.
+            'lbfgs'
+                m : int
+                    The maximum number of variable metric corrections used to
+                    define the limited memory matrix. (The limited memory BFGS
+                    method does not store the full hessian but uses this many
+                    terms in an approximation to it.)
+                pgtol : float
+                    The iteration will stop when
+                    ``max{|proj g_i | i = 1, ..., n} <= pgtol`` where pg_i is
+                    the i-th component of the projected gradient.
+                factr : float
+                    The iteration stops when
+                    ``(f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr * eps``,
+                    where eps is the machine precision, which is automatically
+                    generated by the code. Typical values for factr are: 1e12
+                    for low accuracy; 1e7 for moderate accuracy; 10.0 for
+                    extremely high accuracy. See Notes for relationship to
+                    ftol, which is exposed (instead of factr) by the
+                    scipy.optimize.minimize interface to L-BFGS-B.
+                maxfun : int
+                    Maximum number of iterations.
+                epsilon : float
+                    Step size used when approx_grad is True, for numerically
+                    calculating the gradient
+                approx_grad : bool
+                    Whether to approximate the gradient numerically (in which
+                    case func returns only the function value).
             'cg'
                 gtol : float
                     Stop when norm of gradient is less than gtol.
@@ -344,10 +371,11 @@ def _fit_bfgs(f, score, start_params, fargs, kwargs, disp=True,
     return xopt, retvals
 
 
-def _fit_lbfgs(f, score, start_params, fargs, kwargs, disp=True,
-                   maxiter=100, callback=None, retall=False,
-                   full_output=True, hess=None):
+def _fit_lbfgs(f, score, start_params, fargs, kwargs, disp=True, maxiter=100,
+               callback=None, retall=False, full_output=True, hess=None):
     """
+    Fit model using L-BFGS algorithm
+
     Parameters
     ----------
     f : function
diff --git a/tools/fold_toc.py b/tools/fold_toc.py
--- a/tools/fold_toc.py
+++ b/tools/fold_toc.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
-import sys
 import re
+import sys
 
 # Read doc to string
 filename = sys.argv[1]
@@ -32,27 +32,27 @@
     post_n = re.sub('#', str(i), post)
     doc = re.sub(pre, post_n, doc, count=1)
 
-## TOC entries
+# TOC entries
 pre = '<li class="toctree-l1">'
 post = '<li class="liClosed"> '
-doc =  re.sub(pre, post, doc)
+doc = re.sub(pre, post, doc)
 
 # TOC entries 2nd level
 pre = '<li class="toctree-l2">'
 post = '<li class="liClosed"> '
-doc =  re.sub(pre, post, doc)
+doc = re.sub(pre, post, doc)
 
 # TOC entries 3rd level
 pre = '<li class="toctree-l3">'
 post = '<li class="liClosed"> '
-doc =  re.sub(pre, post, doc)
+doc = re.sub(pre, post, doc)
 
 # TOC entries 4th level
 pre = '<li class="toctree-l4">'
 post = '<li class="liClosed"> '
-doc =  re.sub(pre, post, doc)
+doc = re.sub(pre, post, doc)
 
 # Write to file
-f = open(filename, 'w')
+f = open(filename, 'w', encoding='utf8')
 f.write(doc)
 f.close()


Test patch:
diff --git a/docs/source/dev/test_notes.rst b/docs/source/dev/test_notes.rst
--- a/docs/source/dev/test_notes.rst
+++ b/docs/source/dev/test_notes.rst
@@ -3,20 +3,33 @@
 Testing
 =======
 
+Setting up development environment locally
+------------------------------------------
+Follow our :ref:`installation instructions <install>` and set up a suitable
+environment to build statsmodels from source. We recommend that you develop
+using a development install of statsmodels::
+
+    python setup.py develop
+
+This will compile the C code and add statsmodels to your activate python
+environment by creating links from your python environemnt's libraries
+to the statsmodels source code. Therefore, changes to pure python code will
+be immediately available to the user without a re-install.
+
 Test Driven Development
-~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------
 We strive to follow a `Test Driven Development (TDD) <https://en.wikipedia.org/wiki/Test-driven_development>`_ pattern.
 All models or statistical functions that are added to the main code base are to have
 tests versus an existing statistical package, if possible.
 
 Introduction to pytest
-~~~~~~~~~~~~~~~~~~~~~~
+----------------------
 Like many packages, statsmodels uses the `pytest testing system <https://docs.pytest.org/en/latest/contents.html>`__ and the convenient extensions in `numpy.testing <http://docs.scipy.org/doc/numpy/reference/routines.testing.html>`__.  Pytest will find any file, directory, function, or class name that starts with ``test`` or ``Test`` (classes only). Test function should start with ``test``, test classes should start with ``Test``. These functions and classes should be placed in files with names beginning with ``test`` in a directory called ``tests``.
 
 .. _run-tests:
 
 Running the Test Suite
-~~~~~~~~~~~~~~~~~~~~~~
+----------------------
 
 You can run all the tests by::
 
@@ -67,11 +80,11 @@ at different levels:
     pytest statsmodels/regression/tests/test_regression.py::test_ridge
 
 How To Write A Test
-~~~~~~~~~~~~~~~~~~~
+-------------------
 NumPy provides a good introduction to unit testing with pytest and NumPy extensions `here <https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt>`__. It is worth a read for some more details.
-Here, we will document a few conventions we follow that are worth mentioning. Often we want to test 
-a whole model at once rather than just one function, for example. The following is a pared down 
-version test_discrete.py. In this case, several different models with different options need to be 
+Here, we will document a few conventions we follow that are worth mentioning. Often we want to test
+a whole model at once rather than just one function, for example. The following is a pared down
+version test_discrete.py. In this case, several different models with different options need to be
 tested. The tests look something like
 
 .. code-block:: python
@@ -87,7 +100,7 @@ tested. The tests look something like
 
         def test_params(self):
             assert_almost_equal(self.res1.params, self.res2.params, 4)
-        
+
         decimal_tvalues = 4
         def test_tvalues(self):
             assert_almost_equal(self.res1.params, self.res2.params, self.decimal_tvalues)
@@ -105,7 +118,7 @@ tested. The tests look something like
             data = sm.datasets.spector.load()
             data.exog = sm.add_constant(data.exog)
             cls.res1 = sm.Probit(data.endog, data.exog).fit(method='newton', disp=0)
-            
+
             # set up results
             res2 = Spector()
             res2.probit()
@@ -117,23 +130,23 @@ tested. The tests look something like
         def test_model_specifc(self):
             assert_almost_equal(self.res1.foo, self.res2.foo, 4)
 
-The main workhorse is the `CheckDiscreteResults` class. Notice that we can set the level of precision 
-for `tvalues` to be different than the default in the subclass  `TestProbitNewton`. All of the test 
+The main workhorse is the `CheckDiscreteResults` class. Notice that we can set the level of precision
+for `tvalues` to be different than the default in the subclass  `TestProbitNewton`. All of the test
 classes have a ``@classmethod`` called ``setup_class``. Otherwise, pytest would reinstantiate the class
 before every single test method. If the fitting of the model is time consuming, then this is clearly
 undesirable. Finally, we have a script at the bottom so that we can run the tests should be running
 the Python file.
 
 Test Results
-~~~~~~~~~~~~
-The test results are the final piece of the above example. For many tests, especially those for the 
-models, there are many results against which you would like to test. It makes sense then to separate 
+------------
+The test results are the final piece of the above example. For many tests, especially those for the
+models, there are many results against which you would like to test. It makes sense then to separate
 the hard-coded results from the actual tests to make the tests more readable. If there are only a few
 results it's not necessary to separate the results. We often take results from some other statistical
 package. It is important to document where you got the results from and why they might differ from
 the results that we get. Each tests folder has a results subdirectory. Consider the folder structure
 for the discrete models::
-    
+
     tests/
         __init__.py
         test_discrete.py
@@ -142,7 +155,7 @@ for the discrete models::
             results_discrete.py
             nbinom_resids.csv
 
-It is up to you how best to structure the results. In the discrete model example, you will notice 
-that there are result classes based around particular datasets with a method for loading different 
-model results for that dataset. You can also include text files that hold results to be loaded by 
+It is up to you how best to structure the results. In the discrete model example, you will notice
+that there are result classes based around particular datasets with a method for loading different
+model results for that dataset. You can also include text files that hold results to be loaded by
 results classes if it is easier than putting them in the class itself.
diff --git a/docs/source/dev/testing.rst b/docs/source/dev/testing.rst
--- a/docs/source/dev/testing.rst
+++ b/docs/source/dev/testing.rst
@@ -1,12 +1,18 @@
 Testing on Build Machines
 -------------------------
 
-There are currently several places that statsmodels is automatically built and tested against different dependency and Python versions and architectures. Check these logs periodically, make sure everything looks okay, and fix any failures.:
+There are currently several places that statsmodels is automatically built and
+tested against different dependency and Python versions and architectures.
+Check these logs periodically, make sure everything looks okay, and fix any
+failures:
 
 * `Travis CI <https://travis-ci.org/statsmodels/statsmodels/builds>`_
-* `Daily testing on Ubuntu via Python(x,y) <https://code.launchpad.net/~pythonxy/+recipe/statsmodels-daily-current>`_
-* `NiPy testing on SPARC Boxes <http://nipy.bic.berkeley.edu/waterfall?category=statsmodels>`_
+* `Appveyor <https://ci.appveyor.com/project/josef-pkt/statsmodels/branch/master>`_
 
-The test coverage pages are here.
+The test coverage pages are here:
 
 * `Coveralls <https://coveralls.io/github/statsmodels/statsmodels>`_
+* `codecov <https://codecov.io/gh/statsmodels/statsmodels>`_
+
+We use `Codacy <https://app.codacy.com/project/josef-pkt/statsmodels/dashboard>`_
+to monitor code quality.