Repository: nltk/nltk

Issue:
Failing tests in DocStrings
Checking the Python files in NLTK with "python -m doctest" reveals that many tests are failing. In many cases, the failures are just cosmetic discrepancies between the expected and the actual output, such as missing a blank line, or unescaped linebreaks. Other cases may be real bugs.

If these failures could be avoided, it would become possible to improve CI by running "python -m doctest" each time a python file was modified.

Here is an updated list, after applying #2990, and unzipping those corpora marked unzip="1":

> nltk/parse/corenlp.py: 9 passed and 33 failed.
nltk/parse/stanford.py: 1 passed and 16 failed.
nltk/corpus/reader/framenet.py: 79 passed and 9 failed.
nltk/tokenize/regexp.py: 14 passed and 7 failed.
nltk/draw/table.py: 0 passed and 6 failed.
nltk/tokenize/simple.py: 7 passed and 6 failed.
nltk/text.py: 16 passed and 5 failed.
nltk/tokenize/stanford_segmenter.py: 3 passed and 5 failed.
nltk/draw/util.py: 1 passed and 4 failed.
nltk/tag/hunpos.py: 1 passed and 4 failed.
nltk/tokenize/__init__.py: 6 passed and 4 failed.
nltk/translate/ibm1.py: 12 passed and 4 failed.
nltk/tag/brill_trainer.py: 23 passed and 3 failed.
nltk/tokenize/destructive.py: 8 passed and 3 failed.
nltk/tokenize/treebank.py: 32 passed and 3 failed.
nltk/tokenize/util.py: 27 passed and 3 failed.
nltk/translate/meteor_score.py: 8 passed and 3 failed.
nltk/corpus/reader/categorized_sents.py: 5 passed and 2 failed.
nltk/corpus/reader/opinion_lexicon.py: 3 passed and 2 failed.
nltk/corpus/reader/reviews.py: 8 passed and 2 failed.
nltk/metrics/agreement.py: 8 passed and 2 failed.
nltk/parse/dependencygraph.py: 13 passed and 2 failed.
nltk/parse/evaluate.py: 6 passed and 2 failed.
nltk/parse/util.py: 5 passed and 2 failed.
nltk/tag/stanford.py: 2 passed and 2 failed.
nltk/tokenize/stanford.py: 3 passed and 2 failed.
nltk/corpus/__init__.py: 1 passed and 1 failed.
nltk/corpus/reader/comparative_sents.py: 5 passed and 1 failed.
nltk/corpus/reader/__init__.py: 1 passed and 1 failed.
nltk/corpus/reader/pros_cons.py: 2 passed and 1 failed.
nltk/probability.py: 39 passed and 1 failed.
nltk/sentiment/util.py: 7 passed and 1 failed.
nltk/stem/arlstem2.py: 3 passed and 1 failed.
nltk/stem/rslp.py: 3 passed and 1 failed.
nltk/stem/snowball.py: 6 passed and 1 failed.
nltk/tag/__init__.py: 12 passed and 1 failed.
nltk/tokenize/casual.py: 9 passed and 1 failed.
nltk/translate/ibm2.py: 18 passed and 1 failed.
nltk/translate/ibm3.py: 23 passed and 1 failed.
nltk/translate/ibm4.py: 25 passed and 1 failed.
nltk/translate/ibm5.py: 22 passed and 1 failed.


Discussion:
Is anyone working on this? I would be happy to pick it up and improve the tests.
Not to the best of my knowledge. Feel free to pick this up @Madnex!
Hi! I have managed to fix most of the tests. However I have issues with the following ones:

1. `nltk/parse/corenlp.py: 9 passed and 33 failed.` 
I am getting errors indicating that the local server is not running that is needed for parsing.

2. `nltk/tag/hunpos.py: 1 passed and 4 failed.` 
The hunpos-tag file and the en_wsj.model file were not found

3. `nltk/parse/dependencygraph.py: 13 passed and 2 failed.`
I am getting the error `Exception: Cannot find the dot binary from Graphviz package`.
 
4. `nltk/draw/table.py: 0 passed and 6 failed. `
When I try to run the code I get an error `AttributeError: 'SpaceWidget' object has no attribute 'tk'`

Are those errors related to my configuration? Or do you also encounter them?


Additionally, the errors in those four files I did not fix: 
- nltk/parse/stanford.py: 1 passed and 16 failed. (deprecated)
- nltk/tag/stanford.py: 2 passed and 2 failed. (deprecated)
- nltk/tokenize/stanford.py: 3 passed and 2 failed. (deprecated)
- nltk/tokenize/stanford_segmenter.py: 3 passed and 5 failed. (deprecated)

There I get deprecation errors / warnings. Should they be ignored / removed?
Good work!

I've experienced issues 1 through 3 that you listed. These make some sense, as these services simply require some additional setup. That said, the 4th one is odd. That might be indicative of some actual bugs.

As for the deprecated errors, we'll end up wanting to include the doctests in the CI check, so we should probably ensure that all tests pass. In that case, we might want to either remove the tests for the deprecated work, or modify them such that they're skipped. (I prefer the latter).

I would hope to use `pytest --doctest-modules .\nltk\` to run the doctests easily, but I'm running into issues there, likely because of our LazyLoaders:
```
...\lib\doctest.py:939: in find
    self._find(tests, obj, name, module, source_lines, globs, {})
.env39\lib\site-packages\_pytest\doctest.py:522: in _find
    doctest.DocTestFinder._find(  # type: ignore
...\lib\doctest.py:995: in _find
    for valname, val in obj.__dict__.items():
E   RuntimeError: dictionary changed size during iteration
```

Are you aware of a way to easily run the doctests perhaps?
Okay, so is there a way to include issues 1-3 in the tests for the CI if we need specific setups?
For the issue 4 I think it is a bug then. I will try to see where it's going wrong exactly.

I will add a `+SKIP` for the deprecated tests as suggested.

I don't know how to run the doctests all at once. Maybe @ekaf has more info on this?
Issues 1-3 can be included in the CI, but only with a lot of additional work. I would argue that it's not worth it. We have done something similar in the past for some of the "regular" tests. See our CI job which downloads and caches third party resources:
https://github.com/nltk/nltk/blob/3b72fa2b3dea0088cd6d15d4409ce52318a3d1a0/.github/workflows/ci.yaml#L53-L72
And the script it actually executes: https://github.com/nltk/nltk/blob/develop/tools/github_actions/third-party.sh

---

However, I think the best solution here is to have a conditional skip. 
For example, see the following snippets:
https://github.com/nltk/nltk/blob/3b72fa2b3dea0088cd6d15d4409ce52318a3d1a0/nltk/test/inference.doctest#L4-L9
https://github.com/nltk/nltk/blob/3b72fa2b3dea0088cd6d15d4409ce52318a3d1a0/nltk/test/inference_fixt.py#L1-L10

In the doctest, we import the code below, which performs a lookup step to check whether the required resource is installed/downloaded. If it is not, then we skip the test. This is a bit of a hacky solution sadly. I'm not sure what the most convenient implementation for this is.
> I would hope to use `pytest --doctest-modules .\nltk\` to run the doctests easily, but I'm running into issues there, likely because of our LazyLoaders:

> I don't know how to run the doctests all at once. Maybe @ekaf has more info on this?

I've looked into the reason that `pytest --doctest-modules .\nltk\` was failing. It should be fixed after #3033. That way, we will be able to modify the following line from `nltk/test` to `nltk`:
https://github.com/nltk/nltk/blob/3b72fa2b3dea0088cd6d15d4409ce52318a3d1a0/.github/workflows/ci.yaml#L128

And then it will automatically work with all doctests.
Wow, thanks for the quick fix 👍 
Of the 2 doctests failures in File "nltk/parse/dependencygraph.py", only the first is caused by missing the dot binary from Graphviz package.

The second failure reveals a genuine problem in the module: 

> File "nltk/parse/dependencygraph.py", line 461, in dependencygraph.DependencyGraph.contains_cycle
> Failed example:
>     cyclic_dg.contains_cycle()
> Expected:
>     [3, 1, 2, 4]
> Got:
>     [1, 2, 4, 3]
> 

There could be non-determinism in the implementation of _contains_cycle()_, eventually caused by iterations over randomly ordered dictionary keys, or is the expected doctest output just wrong?

Resolving patch:
diff --git a/nltk/corpus/reader/wordnet.py b/nltk/corpus/reader/wordnet.py
--- a/nltk/corpus/reader/wordnet.py
+++ b/nltk/corpus/reader/wordnet.py
@@ -1178,7 +1178,8 @@ def __init__(self, root, omw_reader):
             warnings.warn(
                 "The multilingual functions are not available with this Wordnet version"
             )
-            self.omw_langs = set()
+
+        self.omw_langs = set()
 
         # A cache to store the wordnet data of multiple languages
         self._lang_data = defaultdict(list)
@@ -1278,9 +1279,8 @@ def _load_lang_data(self, lang):
         if lang in self._lang_data.keys():
             return
 
-        if self.langs() == ["eng"]:
-            self.add_provs(self._omw_reader)
-            self.omw_langs = set(self.provenances.keys())
+        if self._omw_reader and not self.omw_langs:
+            self.add_omw()
 
         if lang not in self.langs():
             raise WordNetError("Language is not supported.")
@@ -1308,12 +1308,16 @@ def add_provs(self, reader):
             file_name, file_extension = os.path.splitext(langfile)
             if file_extension == ".tab":
                 lang = file_name.split("-")[-1]
-                if lang in self.provenances.keys():
+                if lang in self.provenances.keys() or prov in ["cldr", "wikt"]:
                     # We already have another resource for this lang,
                     # so we need to further specify the lang id:
                     lang = f"{lang}_{prov}"
                 self.provenances[lang] = prov
 
+    def add_omw(self):
+        self.add_provs(self._omw_reader)
+        self.omw_langs = set(self.provenances.keys())
+
     def add_exomw(self):
         """
         Add languages from Extended OMW
@@ -1326,6 +1330,7 @@ def add_exomw(self):
         """
         from nltk.corpus import extended_omw
 
+        self.add_omw()
         self._exomw_reader = extended_omw
         self.add_provs(self._exomw_reader)
 


Test patch: