Repository: nltk/nltk

Issue:
Move from nose to pytest or nose2
https://nose.readthedocs.io/en/latest/ -- nose is on life support. I personally prefer pytest, but nose2 may also be considered. Has this been discussed very much yet?


Discussion:
I've been wanting to propose switching to pytest, but I was hesitant to rip the rug from under the feet of people used to the older setup just for the sake of my convenience.

However, this situation with nose EOL may be a good excuse to take the leap. +1 from me.

Resolving patch:
diff --git a/nltk/corpus/__init__.py b/nltk/corpus/__init__.py
--- a/nltk/corpus/__init__.py
+++ b/nltk/corpus/__init__.py
@@ -482,7 +482,7 @@ def demo():
     # demo()
     pass
 
-# ** this is for nose **
+# ** this is for unit testing **
 # unload all corpus after tests
 def teardown_module(module=None):
     import nltk.corpus


Test patch:
diff --git a/nltk/test/Makefile b/nltk/test/Makefile
--- a/nltk/test/Makefile
+++ b/nltk/test/Makefile
@@ -9,7 +9,7 @@ HTML = $(TESTS:.doctest=.html) # $(IPYNB:.ipynb=.html)
 IPYNB = $(wildcard *.ipynb)
 
 .doctest.errs:
-	python ./runtests.py $< > $@
+	pytest $< > $@
 
 .doctest.html:
 	rst2html.py $< > $@
diff --git a/nltk/test/childes.doctest b/nltk/test/childes.doctest
--- a/nltk/test/childes.doctest
+++ b/nltk/test/childes.doctest
@@ -4,6 +4,12 @@
 
 Read the XML version of the CHILDES corpus.
 
+Setup
+=====
+
+    >>> from nltk.test.childes_fixt import setup_module
+    >>> setup_module()
+
 How to use CHILDESCorpusReader
 ==============================
 
diff --git a/nltk/test/childes_fixt.py b/nltk/test/childes_fixt.py
--- a/nltk/test/childes_fixt.py
+++ b/nltk/test/childes_fixt.py
@@ -1,16 +1,14 @@
 # -*- coding: utf-8 -*-
 
-
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
     import nltk.data
 
     try:
         nltk.data.find("corpora/childes/data-xml/Eng-USA-MOR/")
     except LookupError as e:
-        print(e)
-        raise SkipTest(
+        pytest.skip(
             "The CHILDES corpus is not found. "
             "It should be manually downloaded and saved/unpacked "
             "to [NLTK_Data_Dir]/corpora/childes/"
-        ) from e
+        )
diff --git a/nltk/test/classify.doctest b/nltk/test/classify.doctest
--- a/nltk/test/classify.doctest
+++ b/nltk/test/classify.doctest
@@ -5,6 +5,9 @@
  Classifiers
 =============
 
+    >>> from nltk.test.classify_fixt import setup_module
+    >>> setup_module()
+
 Classifiers label tokens with category labels (or *class labels*).
 Typically, labels are represented with strings (such as ``"health"``
 or ``"sports"``.  In NLTK, classifiers are defined using classes that
diff --git a/nltk/test/classify_fixt.py b/nltk/test/classify_fixt.py
--- a/nltk/test/classify_fixt.py
+++ b/nltk/test/classify_fixt.py
@@ -2,10 +2,10 @@
 
 
 # most of classify.doctest requires numpy
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
 
     try:
         import numpy
-    except ImportError as e:
-        raise SkipTest("classify.doctest requires numpy") from e
+    except ImportError:
+        pytest.skip("classify.doctest requires numpy")
diff --git a/nltk/test/corpus.doctest b/nltk/test/corpus.doctest
--- a/nltk/test/corpus.doctest
+++ b/nltk/test/corpus.doctest
@@ -2196,4 +2196,8 @@ access to its tuples() method
     [('NUM:dist', 'How far is it from Denver to Aspen ?'), ('LOC:city', 'What county is Modesto , California in ?'), ...]
 
 
+Teardown
+========
 
+    >>> from nltk.corpus import teardown_module
+    >>> teardown_module()
diff --git a/nltk/test/corpus_fixt.py b/nltk/test/corpus_fixt.py
deleted file mode 100644
--- a/nltk/test/corpus_fixt.py
+++ /dev/null
@@ -1,3 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from nltk.corpus import teardown_module
diff --git a/nltk/test/discourse.doctest b/nltk/test/discourse.doctest
--- a/nltk/test/discourse.doctest
+++ b/nltk/test/discourse.doctest
@@ -9,6 +9,12 @@ Discourse Checking
     >>> from nltk.sem import logic
     >>> logic._counter._value = 0
 
+Setup
+=====
+
+    >>> from nltk.test.childes_fixt import setup_module
+    >>> setup_module()
+
 Introduction
 ============
 
diff --git a/nltk/test/discourse_fixt.py b/nltk/test/discourse_fixt.py
--- a/nltk/test/discourse_fixt.py
+++ b/nltk/test/discourse_fixt.py
@@ -3,14 +3,14 @@
 
 # FIXME: the entire discourse.doctest is skipped if Prover9/Mace4 is
 # not installed, but there are pure-python parts that don't need Prover9.
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
     from nltk.inference.mace import Mace
 
     try:
         m = Mace()
         m._find_binary("mace4")
     except LookupError as e:
-        raise SkipTest(
+        pytest.skip(
             "Mace4/Prover9 is not available so discourse.doctest is skipped"
-        ) from e
+        )
diff --git a/nltk/test/gensim.doctest b/nltk/test/gensim.doctest
--- a/nltk/test/gensim.doctest
+++ b/nltk/test/gensim.doctest
@@ -4,6 +4,9 @@
 =======================================
 Demonstrate word embedding using Gensim
 =======================================
+    
+    >>> from nltk.test.gensim_fixt import setup_module
+    >>> setup_module()
 
 We demonstrate three functions:
 - Train the word embeddings using brown corpus;
diff --git a/nltk/test/gensim_fixt.py b/nltk/test/gensim_fixt.py
--- a/nltk/test/gensim_fixt.py
+++ b/nltk/test/gensim_fixt.py
@@ -1,10 +1,10 @@
 # -*- coding: utf-8 -*-
 
 
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
 
     try:
         import gensim
-    except ImportError as e:
-        raise SkipTest("Gensim doctest requires gensim") from e
+    except ImportError:
+        pytest.skip("Gensim doctest requires gensim")
diff --git a/nltk/test/gluesemantics_malt.doctest b/nltk/test/gluesemantics_malt.doctest
--- a/nltk/test/gluesemantics_malt.doctest
+++ b/nltk/test/gluesemantics_malt.doctest
@@ -7,6 +7,9 @@
  Glue Semantics
 ==============================================================================
 
+    >>> from nltk.test.gluesemantics_malt_fixt import setup_module
+    >>> setup_module()
+
     >>> from nltk.sem.glue import *
     >>> nltk.sem.logic._counter._value = 0
 
diff --git a/nltk/test/gluesemantics_malt_fixt.py b/nltk/test/gluesemantics_malt_fixt.py
--- a/nltk/test/gluesemantics_malt_fixt.py
+++ b/nltk/test/gluesemantics_malt_fixt.py
@@ -1,11 +1,11 @@
 # -*- coding: utf-8 -*-
 
 
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
     from nltk.parse.malt import MaltParser
 
     try:
         depparser = MaltParser("maltparser-1.7.2")
     except LookupError as e:
-        raise SkipTest("MaltParser is not available") from e
+        pytest.skip("MaltParser is not available")
diff --git a/nltk/test/inference.doctest b/nltk/test/inference.doctest
--- a/nltk/test/inference.doctest
+++ b/nltk/test/inference.doctest
@@ -5,6 +5,9 @@
 Logical Inference and Model Building
 ====================================
 
+    >>> from nltk.test.inference_fixt import setup_module
+    >>> setup_module()
+
     >>> from nltk import *
     >>> from nltk.sem.drt import DrtParser
     >>> from nltk.sem import logic
diff --git a/nltk/test/inference_fixt.py b/nltk/test/inference_fixt.py
--- a/nltk/test/inference_fixt.py
+++ b/nltk/test/inference_fixt.py
@@ -1,14 +1,14 @@
 # -*- coding: utf-8 -*-
 
 
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
     from nltk.inference.mace import Mace
 
     try:
         m = Mace()
         m._find_binary("mace4")
-    except LookupError as e:
-        raise SkipTest(
+    except LookupError:
+        pytest.skip(
             "Mace4/Prover9 is not available so inference.doctest was skipped"
-        ) from e
+        )
diff --git a/nltk/test/nonmonotonic.doctest b/nltk/test/nonmonotonic.doctest
--- a/nltk/test/nonmonotonic.doctest
+++ b/nltk/test/nonmonotonic.doctest
@@ -5,6 +5,9 @@
 Nonmonotonic Reasoning
 ======================
 
+    >>> from nltk.test.nonmonotonic_fixt import setup_module
+    >>> setup_module()
+
     >>> from nltk import *
     >>> from nltk.inference.nonmonotonic import *
     >>> from nltk.sem import logic
diff --git a/nltk/test/nonmonotonic_fixt.py b/nltk/test/nonmonotonic_fixt.py
--- a/nltk/test/nonmonotonic_fixt.py
+++ b/nltk/test/nonmonotonic_fixt.py
@@ -1,14 +1,14 @@
 # -*- coding: utf-8 -*-
 
 
-def setup_module(module):
-    from nose import SkipTest
+def setup_module():
+    import pytest
     from nltk.inference.mace import Mace
 
     try:
         m = Mace()
         m._find_binary("mace4")
     except LookupError as e:
-        raise SkipTest(
+        pytest.skip(
             "Mace4/Prover9 is not available so nonmonotonic.doctest was skipped"
-        ) from e
+        )
diff --git a/nltk/test/portuguese_en.doctest b/nltk/test/portuguese_en.doctest
--- a/nltk/test/portuguese_en.doctest
+++ b/nltk/test/portuguese_en.doctest
@@ -18,6 +18,9 @@ Chapter 1 of the NLTK book contains many elementary programming examples, all
 with English texts.  In this section, we'll see some corresponding examples
 using Portuguese.  Please refer to the chapter for full discussion.  *Vamos!*
 
+    >>> from nltk.test.portuguese_en_fixt import setup_module
+    >>> setup_module()
+
     >>> from nltk.examples.pt import *
     *** Introductory Examples for the NLTK Book ***
     Loading ptext1, ... and psent1, ...
@@ -563,3 +566,9 @@ and print them in descending order of frequency:
     ter 249
     dois 231
 
+
+Teardown
+---------
+
+    >>> from nltk.corpus import teardown_module
+    >>> teardown_module()
diff --git a/nltk/test/portuguese_en_fixt.py b/nltk/test/portuguese_en_fixt.py
--- a/nltk/test/portuguese_en_fixt.py
+++ b/nltk/test/portuguese_en_fixt.py
@@ -1,10 +1,8 @@
 # -*- coding: utf-8 -*-
-from nltk.corpus import teardown_module
 
+def setup_module():
+    import pytest
 
-def setup_module(module):
-    from nose import SkipTest
-
-    raise SkipTest(
+    pytest.skip(
         "portuguese_en.doctest imports nltk.examples.pt which doesn't exist!"
     )
diff --git a/nltk/test/probability_fixt.py b/nltk/test/probability_fixt.py
--- a/nltk/test/probability_fixt.py
+++ b/nltk/test/probability_fixt.py
@@ -6,9 +6,9 @@
 
 
 def setup_module(module):
-    from nose import SkipTest
+    import pytest
 
     try:
         import numpy
-    except ImportError as e:
-        raise SkipTest("probability.doctest requires numpy") from e
+    except ImportError:
+        pytest.skip("probability.doctest requires numpy")
diff --git a/nltk/test/pytest.ini b/nltk/test/pytest.ini
new file mode 100644
--- /dev/null
+++ b/nltk/test/pytest.ini
@@ -0,0 +1,13 @@
+[pytest]
+doctest_optionflags=ELLIPSIS NORMALIZE_WHITESPACE IGNORE_EXCEPTION_DETAIL
+
+# --doctest-continue-on-failure allows the test to always
+# get to the teardown, if it exists
+addopts = --doctest-glob *.doctest --doctest-continue-on-failure
+
+# Other options for creating valid teardowns for doctests:
+# 1. Turn doctests into unittest tests
+#    - https://docs.python.org/3/library/doctest.html#unittest-api
+# 2. Use sphinx doctests
+#    - https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html
+# 3. Convert the tests that require setup/teardown into pytest tests with fixtures
diff --git a/nltk/test/runtests.py b/nltk/test/runtests.py
deleted file mode 100755
--- a/nltk/test/runtests.py
+++ /dev/null
@@ -1,75 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-import sys
-import os
-import nose
-from nose.plugins.manager import PluginManager
-from nose.plugins.doctests import Doctest
-from nose.plugins import builtin
-
-NLTK_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
-sys.path.insert(0, NLTK_ROOT)
-
-NLTK_TEST_DIR = os.path.join(NLTK_ROOT, "nltk")
-
-if __name__ == "__main__":
-    # there shouldn't be import from NLTK for coverage to work properly
-    try:
-        # Import RedNose plugin for colored test output
-        from rednose import RedNose
-
-        rednose_available = True
-    except ImportError:
-        rednose_available = False
-
-    class NltkPluginManager(PluginManager):
-        """
-        Nose plugin manager that replaces standard doctest plugin
-        with a patched version and adds RedNose plugin for colored test output.
-        """
-
-        def loadPlugins(self):
-            for plug in builtin.plugins:
-                self.addPlugin(plug())
-            if rednose_available:
-                self.addPlugin(RedNose())
-
-            super(NltkPluginManager, self).loadPlugins()
-
-    manager = NltkPluginManager()
-    manager.loadPlugins()
-
-    # allow passing extra options and running individual tests
-    # Examples:
-    #
-    #    python runtests.py semantics.doctest
-    #    python runtests.py --with-id -v
-    #    python runtests.py --with-id -v nltk.featstruct
-
-    args = sys.argv[1:]
-    if not args:
-        args = [NLTK_TEST_DIR]
-
-    if all(arg.startswith("-") for arg in args):
-        # only extra options were passed
-        args += [NLTK_TEST_DIR]
-
-    # Activate RedNose and hide skipped test messages from output
-    if rednose_available:
-        args += ["--rednose", "--hide-skips"]
-
-    arguments = [
-        "--exclude=",  # why is this needed?
-        # '--with-xunit',
-        # '--xunit-file=$WORKSPACE/nosetests.xml',
-        # '--nocapture',
-        "--with-doctest",
-        # '--doctest-tests',
-        # '--debug=nose,nose.importer,nose.inspector,nose.plugins,nose.result,nose.selector',
-        "--doctest-extension=.doctest",
-        "--doctest-fixtures=_fixt",
-        "--doctest-options=+ELLIPSIS,+NORMALIZE_WHITESPACE,+IGNORE_EXCEPTION_DETAIL",
-        # '--verbosity=3',
-    ] + args
-
-    nose.main(argv=arguments, plugins=manager.plugins)
diff --git a/nltk/test/segmentation_fixt.py b/nltk/test/segmentation_fixt.py
deleted file mode 100644
--- a/nltk/test/segmentation_fixt.py
+++ /dev/null
@@ -1,11 +0,0 @@
-# -*- coding: utf-8 -*-
-
-
-# skip segmentation.doctest if numpy is not available
-def setup_module(module):
-    from nose import SkipTest
-
-    try:
-        import numpy
-    except ImportError as e:
-        raise SkipTest("segmentation.doctest requires numpy") from e
diff --git a/nltk/test/semantics.doctest b/nltk/test/semantics.doctest
--- a/nltk/test/semantics.doctest
+++ b/nltk/test/semantics.doctest
@@ -5,6 +5,10 @@
 Semantics
 =========
 
+    >>> # Setup tests by setting the counter to 0
+    >>> from nltk.sem import logic
+    >>> logic._counter._value = 0
+
     >>> import nltk
     >>> from nltk.sem import Valuation, Model
     >>> v = [('adam', 'b1'), ('betty', 'g1'), ('fido', 'd1'),
diff --git a/nltk/test/semantics_fixt.py b/nltk/test/semantics_fixt.py
deleted file mode 100644
--- a/nltk/test/semantics_fixt.py
+++ /dev/null
@@ -1,7 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# reset the variables counter before running tests
-def setup_module(module):
-    from nltk.sem import logic
-
-    logic._counter._value = 0
diff --git a/nltk/test/translate.doctest b/nltk/test/translate.doctest
--- a/nltk/test/translate.doctest
+++ b/nltk/test/translate.doctest
@@ -240,3 +240,8 @@ Here are some examples:
     *Statistical Machine Translation*, Cambridge University Press
 
 
+Teardown
+========
+
+    >>> from nltk.corpus import teardown_module
+    >>> teardown_module()
diff --git a/nltk/test/translate_fixt.py b/nltk/test/translate_fixt.py
deleted file mode 100644
--- a/nltk/test/translate_fixt.py
+++ /dev/null
@@ -1,3 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from nltk.corpus import teardown_module
diff --git a/nltk/test/unit/conftest.py b/nltk/test/unit/conftest.py
new file mode 100644
--- /dev/null
+++ b/nltk/test/unit/conftest.py
@@ -0,0 +1,15 @@
+import pytest
+
+
+@pytest.fixture(autouse=True)
+def mock_plot(mocker):
+    """ Disable matplotlib plotting in test code """
+
+    try:
+        import matplotlib.pyplot as plt
+
+        mocker.patch.object(plt, 'show')
+    except ImportError:
+        pytest.skip(
+            "mock_plot imports matplotlib which doesn't exist!"
+        )
diff --git a/nltk/test/unit/test_seekable_unicode_stream_reader.py b/nltk/test/unit/test_seekable_unicode_stream_reader.py
--- a/nltk/test/unit/test_seekable_unicode_stream_reader.py
+++ b/nltk/test/unit/test_seekable_unicode_stream_reader.py
@@ -91,7 +91,7 @@ def test_reader():
             try:
                 # skip strings that can't be encoded with the current encoding
                 string.encode(encoding)
-                yield check_reader, string, encoding
+                check_reader(string, encoding)
             except UnicodeEncodeError:
                 pass
 
@@ -120,7 +120,7 @@ def test_reader_on_large_string():
             def _check(encoding, n=1000):
                 check_reader(LARGE_STRING, encoding, n)
 
-            yield _check, encoding
+            _check(encoding)
 
         except UnicodeEncodeError:
             pass
diff --git a/nltk/test/wordnet.doctest b/nltk/test/wordnet.doctest
--- a/nltk/test/wordnet.doctest
+++ b/nltk/test/wordnet.doctest
@@ -602,3 +602,11 @@ Patch-1 https://github.com/nltk/nltk/pull/2065  Adding 3 functions (relations) t
     Synset('french.n.01')
     >>> wn.synsets("slang")[1].in_usage_domains()[18]
     Synset('can-do.s.01')
+
+
+-------------
+Teardown test
+-------------
+
+    >>> from nltk.corpus import wordnet
+    >>> wordnet._unload()
diff --git a/nltk/test/wordnet_fixt.py b/nltk/test/wordnet_fixt.py
deleted file mode 100644
--- a/nltk/test/wordnet_fixt.py
+++ /dev/null
@@ -1,7 +0,0 @@
-# -*- coding: utf-8 -*-
-
-
-def teardown_module(module=None):
-    from nltk.corpus import wordnet
-
-    wordnet._unload()
diff --git a/web/dev/local_testing.rst b/web/dev/local_testing.rst
--- a/web/dev/local_testing.rst
+++ b/web/dev/local_testing.rst
@@ -14,8 +14,8 @@ NLTK testing
 4. Make sure all NLTK data is downloaded (see ``nltk.download()``);
 
 5. run 'tox' command from the root nltk folder. It will install dependencies
-   and run ``nltk/test/runtests.py`` script for all available interpreters.
-   You may pass any options to runtests.py script separating them by '--'.
+   and run ``pytest`` for all available interpreters.
+   You may also pass any pytest options here (for example, `-v` for verbose).
 
 It may take a long time at first run, but the subsequent runs will
 be much faster.
@@ -30,7 +30,6 @@ that failed in the last test run::
 
     tox -e py36 -- -v --failed
 
-
 Run tree doctests for all available interpreters::
 
     tox -- tree.doctest
@@ -48,9 +47,9 @@ In order to skip numpy & friends, use ``..-nodeps`` environments::
 It is also possible to run tests without tox. This way NLTK would be tested
 only under single interpreter, but it may be easier to have numpy and other
 libraries installed this way. In order to run tests without tox, make sure
-``nose >= 1.2.1`` is installed and execute runtests.py script::
+to ``pip install -r test-requirements.txt`` and run ``pytest``::
 
-    nltk/test/runtests.py
+    pytest nltk/test/
 
 
 Writing tests
@@ -71,7 +70,7 @@ unittests. Test should be written as unittest if some of the following apply:
 * test deals with non-ascii unicode and Python 2.x support is required;
 * test is a regression test that is not necessary for documentational purposes.
 
-Unittests currently reside in ``nltk/test/unit/test_*.py`` files; nose
+Unittests currently reside in ``nltk/test/unit/test_*.py`` files; pytest
 is used for test running.
 
 If a test should be written as unittest but also has a documentational value
@@ -85,13 +84,13 @@ There are some gotchas with NLTK doctests (and with doctests in general):
 * Don't write ``+ELLIPSIS``, ``+NORMALIZE_WHITESPACE``,
   ``+IGNORE_EXCEPTION_DETAIL`` flags (they are already ON by default in NLTK).
 
-* Do not write doctests that has non-ascii output (they are not supported in
+* Do not write doctests that have non-ascii output (they are not supported in
   Python 2.x). Incorrect::
 
       >>> greeting
       u'Привет'
 
-  The proper way is to rewrite such doctest as unittest.
+  The proper way is to rewrite such a doctest as a unittest.
 
 * In order to conditionally skip a doctest in a separate
   ``nltk/test/foo.doctest`` file, create ``nltk.test/foo_fixt.py``
@@ -100,10 +99,10 @@ There are some gotchas with NLTK doctests (and with doctests in general):
       # <a comment describing why should the test be skipped>
 
       def setup_module(module):
-          from nose import SkipTest
+          import pytest
 
           if some_condition:
-              raise SkipTest("foo.doctest is skipped because <...>")
+              pytest.skip("foo.doctest is skipped because <...>")
 
 * In order to conditionally skip all doctests from the module/class/function
   docstrings, put the following function in a top-level module namespace::
@@ -111,10 +110,10 @@ There are some gotchas with NLTK doctests (and with doctests in general):
       # <a comment describing why should the tests from this module be skipped>
 
       def setup_module(module):
-          from nose import SkipTest
+          import pytest
 
           if some_condition:
-              raise SkipTest("doctests from nltk.<foo>.<bar> are skipped because <...>")
+              pytest.skip("doctests from nltk.<foo>.<bar> are skipped because <...>")
 
   A good idea is to define ``__all__`` in such module and omit
   ``setup_module`` from ``__all__``.
@@ -150,7 +149,7 @@ If the code requires some external dependencies, then
 
 * tests for this code should be skipped if the dependencies are not available:
   use ``setup_module`` for doctests (as described above) and
-  ``nltk.test.unit.utils.skip / skipIf`` decorators or ``nose.SkipTest``
+  ``@pytest.mark.skipif / @pytest.mark.skip`` decorators or ``pytest.skip``
   exception for unittests;
 * if the dependency is a Python package, it should be added to tox.ini
   (but not to ..-nodeps environments).