Repository: bokeh/bokeh

Issue:
Investigate the use pytest instead of nose
Pytest differs from nose in that you can just `assert` anything, and when something fails, pytest will apply some voodoo to come up with an error message that makes sense (e.g. show diff of lists if you assert that two lists are equal). This makes writing tests much easier, because you don't need all the `assert_x` functions.

For asserting exceptions, there is a `raises()` function, that can also be used as a context manager. And that's in most cases the only function you ever need in a test.

Some examples:

```
assert some_number == 3
assert some_list == [2, 3, 4]
raises(ValueError, foo, 3, 4, bar=8)
with raises(ValueError):
    foo(3, 4, bar)
```

We can set this up for Travis in a PR and see how we like it. pytest is compatible with unittest and nose, so transition can be smooth. No need to rewrite our existing tests.



Discussion:
not only... niceties like `monkeypatch`, `tmp_dir` and other built-in (but not only) fixtures. Plugins and hooks model. Great integration with pdb for debugging purposes, etc..  (the list could go.. IIRC you could even test javascript with it :-) ) makes pytest my favorite test framework so far :-)

As I commented on another issues earlier the most of the work I'd expect is on the automation machinery of tests (but I don't have enough background on the whole tests implementation/automation to ensure it).

I've set pytest up with Travis before, so I can give it a go.

Sound nice... let's see how the experiment goes...

> No need to rewrite our existing tests.

Not now, but slowly over time. If we are going to transition to all new tests being `pytest` then eventually all tests should be `pytest`. Question I meant to ask at the presentation I saw but did not have a chance to: How does `pytest` handle floating point almost-equal assertions?

afaik it doesn't. You need to be explicit on your test (or use external methods that do that). That said, it can ingest `unittest` tests (so all `unittest.TestCase` and friends) and you should be able to use those.

In `numpy.testing` there is `assert_almost_equal` and `assert_allclose`.
Or you can do `assert abs(x-y) < 0.0001`.

Yeap.. we already use some of those used on `bokeh.charts` tests

worth mentioning that if I run:

``` Bash
conda install -n pybk27 pytest
py.test -v -s
```

the output is the same as running our current tests... :tada: 


Resolving patch:
diff --git a/scripts/devdeps.py b/scripts/devdeps.py
--- a/scripts/devdeps.py
+++ b/scripts/devdeps.py
@@ -60,7 +60,7 @@ def depend_check(deps_name, *args):
                       'pdiffer'                    : 'pdiff'
                      }
 
-    dev_deps = ['bs4', 'colorama', 'pdiffer', 'boto', 'nose', 'mock', 'coverage',
+    dev_deps = ['bs4', 'colorama', 'pdiffer', 'boto', 'pytest', 'mock', 'coverage',
                 'websocket']
     depend_check('Dev', *dev_deps)
 
diff --git a/scripts/runtests.py b/scripts/runtests.py
deleted file mode 100644
--- a/scripts/runtests.py
+++ /dev/null
@@ -1,4 +0,0 @@
-from bokeh import test
-
-if __name__ == "__main__":
-    test()
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -507,7 +507,7 @@ def parse_jsargs():
         # cli
         # 'click>=3.3',
         # tests
-        # 'nose>=1.3.0',
+        # 'pytest'
         # 'mock>=1.0.1',
         'colorama>=0.2.7'
     ]


Test patch:
diff --git a/bokeh/server/tests/auth_test.py b/bokeh/server/tests/auth_test.py
--- a/bokeh/server/tests/auth_test.py
+++ b/bokeh/server/tests/auth_test.py
@@ -3,7 +3,7 @@
 from contextlib import contextmanager
 
 from werkzeug.exceptions import Unauthorized
-from nose.tools import assert_raises
+import pytest
 
 from ..app import bokeh_app
 from ..models.user import User
@@ -23,6 +23,7 @@ def test_login_required():
     def test(x):
         return x
     with patch_current_user(lambda : None):
-        assert_raises(Unauthorized, test)
+        with pytest.raises(Unauthorized):
+            test()
     with patch_current_user(lambda : User):
         assert test(1) == 1
diff --git a/bokeh/util/testing.py b/bokeh/util/testing.py
--- a/bokeh/util/testing.py
+++ b/bokeh/util/testing.py
@@ -1,7 +1,8 @@
 """ Functions to help with testing Bokeh and reporting issues.
 
 """
-from __future__ import absolute_import
+from __future__ import absolute_import, print_function
+
 
 def skipIfPy3(message):
     """ unittest decoractor to skip a test for Python 3
@@ -20,6 +21,7 @@ def skipIfPyPy(message):
     from .platform import is_pypy
     return skipIf(is_pypy(), message)
 
+
 def print_versions():
     """ Print the versions for Bokeh and the current Python and OS.
 
@@ -37,52 +39,36 @@ def print_versions():
            pt.python_implementation(), pt.platform())
     print(message)
 
-def runtests(verbosity=1, xunitfile=None, exit=False):
-    """ Run the full Bokeh test suite, and output the results of the tests
-    to sys.stdout.
 
-    This function uses nosetests to discover which tests to run, and will
-    run tests in any 'tests' subdirectory within the Bokeh module.
+def runtests(args=None):
 
-    Args:
-        verbosity (int, optional) :
-            Acceptable values are 0 (less verbose) to 2 (most verbose)
+    """
+    Run the Bokeh tests under the bokeh python directory using pytest.
 
-        xunitfile (str, optional) :
-            Write xunit-style XML test results to a given filename. This
-            is useful for running tests on a CI server. (default: None)
+    Does not run tests from bokehjs or examples.
 
-        exit (bool, optional) :
-            Whether to return or exit. If True, call sys.exit with an
-            error code after the tests are finished. (default: False)
+    Args:
+        args(list, optional): List of command line arguments accepted by py.test
+        e.g. args=['-s', '-k charts'] prevents capture of standard
+        out and only runs tests that match charts. For more py.test options see
+        http://pytest.org/latest/usage.html#usage.
 
     Returns:
-        int : Nose return code
-
+        int: pytest exitcode
     """
 
-    import nose, os, sys
-
-    argv = ['nosetests', '--verbosity=%d' % verbosity]
-
-    # Output an xunit file if requested
-    if xunitfile:
-        argv.extend(['--with-xunit', '--xunit-file=%s' % xunitfile])
-
-    # Set the logging level to warn
-    argv.extend(['--logging-level=WARN'])
+    import pytest
+    import os
 
-    # Add all 'tests' subdirectories to the options
-    rootdir = os.path.join(os.path.dirname(__file__), "..")
-    for root, dirs, files in os.walk(rootdir):
-        if 'tests' in dirs:
-            testsdir = os.path.join(root, 'tests')
-            argv.append(testsdir)
-            print('Test dir: %s' % testsdir[len(rootdir)+1:])
+    try:
+        import faulthandler
+        faulthandler.enable()
+    except ImportError:
+        # We can live without in python 2.7
+        pass
 
-    # print versions (handy when reporting problems)
-    print_versions()
-    sys.stdout.flush()
+    # change to the bokeh python source directory, for test collection
+    rootdir = os.path.join(os.path.dirname(__file__), os.pardir)
+    os.chdir(rootdir)
 
-    # Run the tests
-    return nose.main(argv=argv, exit=exit)
+    return pytest.main(args=args)
diff --git a/scripts/jenkins_test b/scripts/jenkins_test
--- a/scripts/jenkins_test
+++ b/scripts/jenkins_test
@@ -1,3 +1,3 @@
 #!/bin/bash
 
-nosetests --with-coverage --cover-package=bokeh
+py.test -s --cov=bokeh
diff --git a/scripts/test_matrix.py b/scripts/test_matrix.py
--- a/scripts/test_matrix.py
+++ b/scripts/test_matrix.py
@@ -102,7 +102,7 @@ def version_check(env_name, expected_ver):
 
 
 def run_tests(env_name):
-    """Run bokeh.test() in a given environment.  Writes results to a tmpfile that will be returned as a string
+    """Run py.test in a given environment.  Writes results to a tmpfile that will be returned as a string
     in the event of a failure.
     """
 
@@ -111,9 +111,9 @@ def run_tests(env_name):
     tmpfile = open(file_name, "w+")
 
     if os.name == 'nt':
-        command_string = 'activate %s & python -c "import nose, os, sys, bokeh; bokeh.test(exit=True)"' % env_name
+        command_string = 'activate %s & py.test -s --cov=bokeh"' % env_name
     else:
-        command_string = 'source activate %s; python -c "import nose, os, sys, bokeh; bokeh.test(exit=True)"' % env_name
+        command_string = 'source activate %s; py.test -s --cov=bokeh"' % env_name
 
     result = call_wrapper(command_string, shell=True, stderr=tmpfile)
 
@@ -163,84 +163,84 @@ def logger(failure_list, logfile='logfile.txt'):
 
     envs = {
         "py27_conda_clean"    : {
-            "init"    : "python=2.7 nose mock",
+            "init"    : "python=2.7 pytest pytest-cov mock",
             "install" : '; '.join([
                     # install latest version from dev channel
                     "conda install --yes -c bokeh/channel/dev bokeh",
                     # install dependencies needed for testing
                     "conda install --yes  -c auto websocket-client",
-                    "conda install --yes  -c bokeh nose mock blaze abstract-rendering beautiful-soup "
+                    "conda install --yes  -c bokeh pytest pytest-cov mock blaze abstract-rendering beautiful-soup "
                     "ipython scipy websocket multiuserblazeserver pillow",
                 ])
             },
         "py27_conda_update"   : {
-            "init"    : "python=2.7 nose mock bokeh=%s" % preversion,
+            "init"    : "python=2.7 pytest pytest-cov mock bokeh=%s" % preversion,
             "install" : '; '.join([
                     "conda update --yes -c bokeh/channel/dev bokeh",
                     # install dependencies needed for testing
                     "conda install --yes -c auto websocket-client",
-                    "conda install --yes -c bokeh nose mock blaze abstract-rendering beautiful-soup "
+                    "conda install --yes -c bokeh pytest pytest-cov mock blaze abstract-rendering beautiful-soup "
                     "ipython scipy websocket multiuserblazeserver pillow",
                 ])
             },
         "py27_pip_clean"      : {
-            "init"    : "python=2.7 nose mock pip",
+            "init"    : "python=2.7 pytest pytest-cov mock pip",
             "install" : '; '.join([
                 "pip install --pre -i https://pypi.binstar.org/bokeh/channel/dev/simple"
                 " bokeh --extra-index-url https://pypi.python.org/simple/",
                 # install dependencies needed for testing
-                "pip install nose mock blaze abstract-rendering beautifulsoup4"
+                "pip install pytest pytest-cov mock blaze abstract-rendering beautifulsoup4"
                 " ipython scipy websocket-client multiuserblazeserver",
                 ])
             },
         "py27_pip_update"     : {
-            "init"    : "python=2.7 pip nose mock bokeh=%s" % preversion,
+            "init"    : "python=2.7 pip pytest pytest-cov mock bokeh=%s" % preversion,
             "install" :  '; '.join([
                 "pip install --upgrade --pre -i "
                 "https://pypi.binstar.org/bokeh/channel/dev/simple "
                 "bokeh --extra-index-url https://pypi.python.org/simple/",
                 # install dependencies needed for testing
-                "pip install nose mock blaze abstract-rendering beautifulsoup4"
+                "pip install pytest pytest-cov mock blaze abstract-rendering beautifulsoup4"
                 " ipython scipy websocket-client multiuserblazeserver",
                 ])
             },
         "py34_conda_clean"    : {
-            "init"    : "python=3.4 nose mock",
+            "init"    : "python=3.4 pytest pytest-cov mock",
             "install" : '; '.join([
                     # install latest version from dev channel
                     "conda install --yes -c bokeh/channel/dev bokeh",
                     # install dependencies needed for testing
-                    "conda install --yes  -c bokeh nose mock blaze abstract-rendering beautiful-soup "
+                    "conda install --yes  -c bokeh pytest pytest-cov mock blaze abstract-rendering beautiful-soup "
                     "ipython scipy multiuserblazeserver pillow",
                 ])
             },
         "py34_conda_update"   : {
-            "init"    : "python=3.4 nose mock bokeh=%s" % preversion,
+            "init"    : "python=3.4 pytest pytest-cov mock bokeh=%s" % preversion,
             "install" : '; '.join([
                     "conda update --yes -c bokeh/channel/dev bokeh",
                     # install dependencies needed for testing
-                    "conda install --yes -c bokeh nose mock blaze abstract-rendering beautiful-soup "
+                    "conda install --yes -c bokeh pytest pytest-cov mock blaze abstract-rendering beautiful-soup "
                     "ipython scipy multiuserblazeserver pillow",
                 ])
             },
         "py34_pip_clean"      : {
-            "init"    : "python=3.4 nose mock pip",
+            "init"    : "python=3.4 pytest pytest-cov mock pip",
             "install" : '; '.join([
                 "pip install --pre -i https://pypi.binstar.org/bokeh/channel/dev/simple"
                 " bokeh --extra-index-url https://pypi.python.org/simple/",
                 # install dependencies needed for testing
-                "pip install nose mock blaze abstract-rendering beautifulsoup4"
+                "pip install pytest pytest-cov mock blaze abstract-rendering beautifulsoup4"
                 " ipython scipy websocket-client multiuserblazeserver",
                 ])
             },
         "py34_pip_update"     : {
-            "init"    : "python=3.4 pip nose mock bokeh=%s" % preversion,
+            "init"    : "python=3.4 pip pytest pytest-cov mock bokeh=%s" % preversion,
             "install" :  '; '.join([
                 "pip install --upgrade --pre -i "
                 "https://pypi.binstar.org/bokeh/channel/dev/simple "
                 "bokeh --extra-index-url https://pypi.python.org/simple/",
                 # install dependencies needed for testing
-                "pip install nose mock blaze abstract-rendering beautifulsoup4"
+                "pip install pytest pytest-cov mock blaze abstract-rendering beautifulsoup4"
                 " ipython scipy websocket-client multiuserblazeserver",
                 ])
             },
diff --git a/sphinx/source/docs/dev_guide/testing.rst b/sphinx/source/docs/dev_guide/testing.rst
--- a/sphinx/source/docs/dev_guide/testing.rst
+++ b/sphinx/source/docs/dev_guide/testing.rst
@@ -12,33 +12,37 @@ be viewed at: https://travis-ci.org/bokeh/bokeh.
 
 To run the just the python unit tests, run the command:
 
-.. code-block:: python
+.. code-block:: sh
+
+    py.test -m 'not (js or examples)' 
+
 
-    import bokeh
-    bokeh.test()
+To run just the examples, run the command:
+
+.. code-block:: sh
+
+    py.test -m examples 
 
 To run just the BokehJS unit tests, execute:
 
 .. code-block:: sh
 
-    gulp test
+    py.test -m js
 
-in the `bokehjs` subdirectory of the source checkout.
 
-Additionally, there are "examples tests" that check whether all the examples
-produce outputs. This script is in the `examples` directory and can be run by
-executing:
+Or, in the `bokehjs` subdirectory of the source checkout.
 
 .. code-block:: sh
 
-    test -D
+    gulp test
+
 
 You can run all available tests (python and JS unit tests and example tests)
 from the top level directory by executing:
 
 .. code-block:: sh
 
-    nosetests
+    py.test
 
 .. note::
     Currently this script does not support Windows.
@@ -47,3 +51,17 @@ To help the test script choose the appropriate test runner, there are some
 naming conventions that examples should adhere to. Non-IPython notebook
 example scripts that rely on the Bokeh server should have 'server' or
 'animate' in their filenames.
+
+To run any of the tests with coverage use the following:
+
+.. code-block:: sh
+
+  py.test --cov=bokeh
+
+To run any of the tests without standard output captured use:
+
+.. code-block:: sh
+
+  py.test -s
+
+See the py.test documentation at http://pytest.org/latest/ for further information on py.test and it's options.
diff --git a/tests/test_bokehjs.py b/tests/test_bokehjs.py
--- a/tests/test_bokehjs.py
+++ b/tests/test_bokehjs.py
@@ -1,10 +1,13 @@
 from __future__ import print_function
 
 import os
+import pytest
 from os.path import join
 import unittest
 import subprocess
 
+
+@pytest.mark.js
 class TestBokehJS(unittest.TestCase):
 
     def test_bokehjs(self):
diff --git a/tests/travis/test_examples.py b/tests/travis/test_examples.py
--- a/tests/travis/test_examples.py
+++ b/tests/travis/test_examples.py
@@ -1,6 +1,7 @@
 from __future__ import absolute_import, print_function
 
 import subprocess
+import pytest
 
 from os.path import join, dirname, pardir
 from unittest import TestCase, skipIf
@@ -12,6 +13,7 @@
 else:
     is_IPython = True
 
+@pytest.mark.examples
 class TestExamples(TestCase):
 
     def test_examples(self):