internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: nltk/nltk

Issue:
SpanishStemmer error with certain terms
All these terms:
['tgue','argue', 'lgue', 'ungue', 'igue' ]

Give this error when you try to stem any of them:
warning: range error for conversion to character value
Traceback (most recent call last):
  File "./spanishstemmer.py", line 13, in <module>
    singles.append(stemmer.stem(words.decode('utf-8')))
  File "/usr/local/lib/python2.7/dist-packages/nltk/stem/snowball.py", line 3554, in stem
    if word[-2:] == "gu" and rv[-1] == "u":
IndexError: string index out of range



Discussion:
@he7d3r would you mind taking a look at this please?

I confirmed this problem using Python 3.4

``` python
>>> from nltk.stem.snowball import SnowballStemmer
>>> SnowballStemmer('spanish').stem('tgue')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<path>/lib/python3.4/site-packages/nltk/stem/snowball.py", line 3554, in stem
    if word[-2:] == "gu" and rv[-1] == "u":
IndexError: string index out of range
```

At that point of the code, `rv` is the empty string (in all 5 examples), so `rv[-1]` causes the error.

Thanks @he7d3r. Can you suggest a fix please?

Looks like replacing `rv[-1] == "u"` by any of these should work:
- `rv[-1:] == "u"`
- `rv.endswith("u")`
- `rv and rv[-1] == "u"`

(I don't know which one is more "Pythonic")

BTW: it seems [that if/else block](https://github.com/nltk/nltk/blob/f422e337313b81581a5c13345f6803f3510c2adc/nltk/stem/snowball.py#L3550-L3557) can be simplified a little, since `word = word[:-len(suffix)]` is executed in both cases.


Resolving patch:
diff --git a/nltk/stem/snowball.py b/nltk/stem/snowball.py
--- a/nltk/stem/snowball.py
+++ b/nltk/stem/snowball.py
@@ -3551,7 +3551,7 @@ def stem(self, word):
                     word = word[:-len(suffix)]
                     rv = rv[:-len(suffix)]
 
-                    if word[-2:] == "gu" and rv[-1:] == "u":
+                    if word[-2:] == "gu" and rv.endswith("u"):
                         word = word[:-1]
                 else:
                     word = word[:-len(suffix)]


Test patch:
Repository: nltk/nltk

Issue:
BLEU score returns 1 (perfect match) instead of zero
Hi, taken from bleu implementation:

``` python
    @staticmethod
     def compute(candidate, references, weights):
        candidate = [c.lower() for c in candidate]
        references = [[r.lower() for r in reference] for reference in references]

        p_ns = (BLEU.modified_precision(candidate, references, i) for i, _ in enumerate(weights, start=1))
        s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns) if p_n)

        bp = BLEU.brevity_penalty(candidate, references)
        return bp * math.exp(s)
```

This function incorrectly returns BLEU score 1 when the candidate has no alignment to any of the references. In this case, `p_ns` will be all zeros because there is no overlap for any n-grams, which will make `s` zero, which will return `1` at the end.

There should be a special case check for the case when there is zero alignment and return zero correctly. 



Discussion:

Resolving patch:
diff --git a/nltk/align/bleu.py b/nltk/align/bleu.py
--- a/nltk/align/bleu.py
+++ b/nltk/align/bleu.py
@@ -132,7 +132,13 @@ def compute(candidate, references, weights):
         references = [[r.lower() for r in reference] for reference in references]
 
         p_ns = (BLEU.modified_precision(candidate, references, i) for i, _ in enumerate(weights, start=1))
-        s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns) if p_n)
+        p_ns_nonzero = list(filter(None, p_ns))
+
+        if not p_ns_nonzero:
+            # There is zero aliment, so the score is 0
+            return 0
+
+        s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns_nonzero))
 
         bp = BLEU.brevity_penalty(candidate, references)
         return bp * math.exp(s)


Test patch:
diff --git a/nltk/test/bleu.doctest b/nltk/test/bleu.doctest
new file mode 100644
--- /dev/null
+++ b/nltk/test/bleu.doctest
@@ -0,0 +1,17 @@
+
+==========
+BLEU tests
+==========
+
+>>> from nltk.align.bleu import BLEU
+
+
+If the candidate has no alignment to any of the references, the BLEU score is
+0.
+
+>>> BLEU.compute(
+...     'John loves Mary'.split(),
+...     ['The candidate has no alignment to any of the references'.split()],
+...     [1],
+... )
+0
Repository: nltk/nltk

Issue:
fes doctests breaking
The new functionality in #832 is causing some errors [reported by Jenkins](https://nltk.ci.cloudbees.com/job/nltk/TOXENV=py26-jenkins/lastCompletedBuild/testReport/nltk.corpus.reader.framenet/FramenetCorpusReader/fes/). The problem is simply that the outputs aren't in a canonical order. @nschneid would you please take a look?



Discussion:

Resolving patch:
diff --git a/nltk/corpus/reader/framenet.py b/nltk/corpus/reader/framenet.py
--- a/nltk/corpus/reader/framenet.py
+++ b/nltk/corpus/reader/framenet.py
@@ -1206,17 +1206,17 @@ def fes(self, name=None):
         consistent across frames.)
         
         >>> from nltk.corpus import framenet as fn
-        >>> fn.fes('sound')
-        [<fe ID=328 name=Sound>, <fe ID=323 name=Sound_source>, ...]
-        >>> [(fe.frame.name,fe.name) for fe in fn.fes('sound')]
-        [('Make_noise', 'Sound'), ('Make_noise', 'Sound_source'), 
-         ('Cause_to_make_noise', 'Sound_maker'), ('Sounds', 'Sound_source'), 
-         ('Sounds', 'Location_of_sound_source'), ('Sounds', 'Component_sound'), 
+        >>> fn.fes('Noise_maker')
+        [<fe ID=6043 name=Noise_maker>]
+        >>> sorted([(fe.frame.name,fe.name) for fe in fn.fes('sound')])
+        [('Cause_to_make_noise', 'Sound_maker'), ('Make_noise', 'Sound'), 
+         ('Make_noise', 'Sound_source'), ('Sound_movement', 'Location_of_sound_source'), 
          ('Sound_movement', 'Sound'), ('Sound_movement', 'Sound_source'), 
-         ('Sound_movement', 'Location_of_sound_source'), ('Vocalizations', 'Sound_source'), 
-         ('Vocalizations', 'Location_of_sound_source')]
-        >>> set(fe.name for fe in fn.fes('^sound'))
-        set(['Sound', 'Sound_source', 'Sound_maker'])
+         ('Sounds', 'Component_sound'), ('Sounds', 'Location_of_sound_source'), 
+         ('Sounds', 'Sound_source'), ('Vocalizations', 'Location_of_sound_source'), 
+         ('Vocalizations', 'Sound_source')]
+        >>> sorted(set(fe.name for fe in fn.fes('^sound')))
+        ['Sound', 'Sound_maker', 'Sound_source']
         >>> len(fn.fes('^sound$'))
         2
         


Test patch:
Repository: nltk/nltk

Issue:
Treat "ü" as "u" in the Portuguese stemmer
I believe the accented letter "ü" should be treated as "u" in some cases. For example, the stem of the following words should be the same (i.e. "aguent"):
- aguenta
- agüenta
- aguentam
- aguentamos
- aguentar
- aguentava
- agüentava
- aguente
- agüente
- aguento
- agüento
- aguentou
- agüentou

However, the ones with "ü" are being converted to "agüent".



Discussion:
Is there any reference to support this? Isn't it a separate and more general step to normalise forms across different versions of [Portuguese orthography](http://en.wikipedia.org/wiki/Reforms_of_Portuguese_orthography)?

Consider for example this:
https://github.com/apache/lucene.net/blob/master/src/contrib/Analyzers/BR/BrazilianStemmer.cs#L306-L310

Is there a normalization step like the one you mentioned for some other language with different orthographies?


Resolving patch:
diff --git a/nltk/stem/snowball.py b/nltk/stem/snowball.py
--- a/nltk/stem/snowball.py
+++ b/nltk/stem/snowball.py
@@ -2489,7 +2489,9 @@ def stem(self, word):
         step2_success = False
 
         word = (word.replace("\xE3", "a~")
-                    .replace("\xF5", "o~"))
+                    .replace("\xF5", "o~")
+                    .replace("q\xFC", "qu")
+                    .replace("g\xFC", "gu"))
 
         r1, r2 = self._r1r2_standard(word, self.__vowels)
         rv = self._rv_standard(word, self.__vowels)


Test patch:
Repository: nltk/nltk

Issue:
Bug in DependencyGraph._tree
Running the dependencygraph doctests produces an error, probably due to recent changes in `DependencyGraph`. It shows up in https://github.com/nltk/nltk/blob/develop/nltk/parse/dependencygraph.py#L265

@dimazest – would you be able to look at this please?

```
Traceback (most recent call last):
  File "dependencygraph.py", line 627, in <module>
    demo()
  File "dependencygraph.py", line 421, in demo
    malt_demo()
  File "dependencygraph.py", line 451, in malt_demo
    tree = dg.tree()
  File "dependencygraph.py", line 281, in tree
    return Tree(word, [self._tree(i) for i in deps])
  File "dependencygraph.py", line 281, in <listcomp>
    return Tree(word, [self._tree(i) for i in deps])
  File "dependencygraph.py", line 265, in _tree
    word = node['word']
KeyError: 'word'
```



Discussion:
There's a related problem for the non-projective dependency parser:

```
Traceback (most recent call last):
  File "nonprojectivedependencyparser.py", line 766, in <module>
    demo()
  File "nonprojectivedependencyparser.py", line 724, in demo
    nonprojective_conll_parse_demo()
  File "nonprojectivedependencyparser.py", line 743, in nonprojective_conll_parse_demo
    for parse_graph in npp.parse(['Cathy', 'zag', 'hen', 'zwaaien', '.'], ['N', 'V', 'Pron', 'Adj', 'N', 'Punc']):
  File "nonprojectivedependencyparser.py", line 465, in parse
    g_graph.connect_graph()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/nltk-3.0.0-py3.4.egg/nltk/parse/dependencygraph.py", line 110, in connect_graph
    node1['deps'].append(node2['address'])
AttributeError: 'collections.defaultdict' object has no attribute 'append'
```

I'll try to have a look by the end of the week.

Thanks. It's one of the few things to resolve before 3.0.2 can be released.


Resolving patch:
diff --git a/nltk/parse/dependencygraph.py b/nltk/parse/dependencygraph.py
--- a/nltk/parse/dependencygraph.py
+++ b/nltk/parse/dependencygraph.py
@@ -16,6 +16,7 @@
 from __future__ import print_function, unicode_literals
 
 from collections import defaultdict
+from itertools import chain
 from pprint import pformat
 
 from nltk.tree import Tree
@@ -263,7 +264,7 @@ def _tree(self, i):
         """
         node = self.get_by_address(i)
         word = node['word']
-        deps = node['deps']
+        deps = list(chain.from_iterable(node['deps'].values()))
 
         if deps:
             return Tree(word, [self._tree(dep) for dep in deps])
@@ -276,9 +277,10 @@ def tree(self):
         ``Tree`` constructor. Dependency labels are omitted.
         """
         node = self.root
+
         word = node['word']
-        deps = node['deps']
-        return Tree(word, [self._tree(i) for i in deps])
+        deps = chain.from_iterable(node['deps'].values())
+        return Tree(word, [self._tree(dep) for dep in deps])
 
     def triples(self, node=None):
         """


Test patch:
Repository: nltk/nltk

Issue:
Translation metrics in Python 2.7
The [CI Server](https://nltk.ci.cloudbees.com/job/nltk/TOXENV=py27-jenkins,jdk=jdk8latestOnlineInstall/lastCompletedBuild/testReport/) reports test failures for  `nltk.translate.ribes_score.kendall_tau` and `nltk.translate.ribes_score.spearman_rho`.



Discussion:
@alvations – would you mind taking a look please?


Resolving patch:
diff --git a/nltk/translate/ribes_score.py b/nltk/translate/ribes_score.py
--- a/nltk/translate/ribes_score.py
+++ b/nltk/translate/ribes_score.py
@@ -6,7 +6,7 @@
 # URL: <http://nltk.org/>
 # For license information, see LICENSE.TXT
 """ RIBES score implementation """
-
+from __future__ import division
 from itertools import islice
 import math
 


Test patch:
Repository: nltk/nltk

Issue:
Tox fails with "ERROR: Failure: ImportError (No module named 'six')"
When I try to run the tests with Tox (on Ubuntu) from within a local clone of the repo, it manages to install the dependencies but blows up when trying to import things from within NLTK.

I imagine I can work around this by figuring out how to manually run just the tests I care about, but it's inconvenient.

I'm not sure whether I'm doing something dumb or whether the Tox setup is broken; if the former, the CONTRIBUTING docs should probably mention what needs to be done besides just running Tox; if the latter, it should probably be fixed.

Here's the full output (had to pastebin it due to GitHub's post length limit):

http://pastebin.com/ENuCLnv6



Discussion:

Resolving patch:
diff --git a/nltk/tokenize/api.py b/nltk/tokenize/api.py
--- a/nltk/tokenize/api.py
+++ b/nltk/tokenize/api.py
@@ -11,7 +11,7 @@
 """
 
 from abc import ABCMeta, abstractmethod
-from six import add_metaclass
+from nltk.six import add_metaclass
 
 from nltk.internals import overridden
 from nltk.tokenize.util import string_span_tokenize


Test patch:
diff --git a/nltk/test/probability.doctest b/nltk/test/probability.doctest
--- a/nltk/test/probability.doctest
+++ b/nltk/test/probability.doctest
@@ -69,33 +69,33 @@ ConditionalFreqDist
 -------------------
 
     >>> cfd1 = ConditionalFreqDist()
-    >>> cfd1[1] = FreqDist('abbbc')
+    >>> cfd1[1] = FreqDist('abbbb')
     >>> cfd1[2] = FreqDist('xxxxyy')
     >>> cfd1
     <ConditionalFreqDist with 2 conditions>
 
     >>> cfd2 = ConditionalFreqDist()
-    >>> cfd2[1] = FreqDist('bccd')
-    >>> cfd2[2] = FreqDist('xxyyyzz')
+    >>> cfd2[1] = FreqDist('bbccc')
+    >>> cfd2[2] = FreqDist('xxxyyyzz')
     >>> cfd2[3] = FreqDist('m')
     >>> cfd2
     <ConditionalFreqDist with 3 conditions>
 
     >>> r = cfd1 + cfd2
     >>> [(i,r[i]) for i in r.conditions()]
-    [(1, FreqDist({'b': 4, 'c': 3, 'a': 1, 'd': 1})), (2, FreqDist({'x': 6, 'y': 5, 'z': 2})), (3, FreqDist({'m': 1}))]
+    [(1, FreqDist({'b': 6, 'c': 3, 'a': 1})), (2, FreqDist({'x': 7, 'y': 5, 'z': 2})), (3, FreqDist({'m': 1}))]
 
     >>> r = cfd1 - cfd2
     >>> [(i,r[i]) for i in r.conditions()]
-    [(1, FreqDist({'b': 2, 'a': 1})), (2, FreqDist({'x': 2}))]
+    [(1, FreqDist({'b': 2, 'a': 1})), (2, FreqDist({'x': 1}))]
 
     >>> r = cfd1 | cfd2
     >>> [(i,r[i]) for i in r.conditions()]
-    [(1, FreqDist({'b': 3, 'c': 2, 'a': 1, 'd': 1})), (2, FreqDist({'x': 4, 'y': 3, 'z': 2})), (3, FreqDist({'m': 1}))]
+    [(1, FreqDist({'b': 4, 'c': 3, 'a': 1})), (2, FreqDist({'x': 4, 'y': 3, 'z': 2})), (3, FreqDist({'m': 1}))]
 
     >>> r = cfd1 & cfd2
     >>> [(i,r[i]) for i in r.conditions()]
-    [(1, FreqDist({'c': 1, 'b': 1})), (2, FreqDist({'y': 2, 'x': 2}))]
+    [(1, FreqDist({'b': 2})), (2, FreqDist({'x': 3, 'y': 2}))]
 
 Testing some HMM estimators
 ---------------------------
Repository: nltk/nltk

Issue:
FrameNet: Subcorpus name with non-ascii character can't be handled
This is probably an issue with the underlying xml reader, but here are the repro steps:

```
from nltk.corpus import framenet as fn
minimum = fn.lu(14294)
id = minimum.ID  # okay
sc = minimum.subCorpus  # throws UnicodeDecodeError
```

Looks like one of the elements in the xml file is `<subCorpus name="01-£10,000-(1)"/>`



Discussion:
I think this will be fixed with impending updates.

@rleegoldman would you please see if you can reproduce this error with the current HEAD revision? https://github.com/nltk/nltk/archive/develop.zip

This is still a problem (in Python 2.7).  The problem is in `_handle_lusubcorpus_elt()`, specifically:

```
            sc['name'] = str(elt.get('name'))
```

The type of `elt.get('name')` is either `str` or, if it contains a non-ASCII character, `unicode`, so the extra call to `str()` either is redundant or will fail.  A very simple solution is to just take out the call to `str()`, thusly:

```
            sc['name'] = elt.get('name')
```

Then `sc['name']` will also either be a `str` or a `unicode` object, as appropriate.  Would that cause any problems?  @wooters?  


Resolving patch:
diff --git a/nltk/corpus/reader/framenet.py b/nltk/corpus/reader/framenet.py
--- a/nltk/corpus/reader/framenet.py
+++ b/nltk/corpus/reader/framenet.py
@@ -1886,7 +1886,7 @@ def _handle_lusubcorpus_elt(self, elt):
         """Load a subcorpus of a lexical unit from the given xml."""
         sc = AttrDict()
         try:
-            sc['name'] = str(elt.get('name'))
+            sc['name'] = elt.get('name')
         except AttributeError:
             return None
         sc['_type'] = "lusubcorpus"


Test patch:
Repository: nltk/nltk

Issue:
Is generate() coming back? 
Would really love to see this function back again. The book says it will be reinstated in a later version, but when?



Discussion:
+1 .  Given the recent RNN generation from @karpathy (https://github.com/karpathy/char-rnn), it might be very easy to create a `generate()` algorithm but support of any neural net implementation is all very unstable, API changes almost every 2-3 weeks =(


Resolving patch:
diff --git a/nltk/text.py b/nltk/text.py
--- a/nltk/text.py
+++ b/nltk/text.py
@@ -443,6 +443,13 @@ def dispersion_plot(self, words):
         """
         from nltk.draw import dispersion_plot
         dispersion_plot(self, words)
+        
+    def generate(self, words):
+        """
+        Issues a reminder to users following the book online
+        """
+        import warnings
+        warnings.warn('The generate() method is no longer available.', DeprecationWarning)
 
     def plot(self, *args):
         """


Test patch:
Repository: nltk/nltk

Issue:
Bleu tests failing
BLEU tests are currently failing

```
tox -e py34 nltk/test/unit/translate/test_bleu.py
```
```
py34 runtests: commands[1] | python runtests.py unit/translate/test_bleu.py
.........E
======================================================================
ERROR: test_corpus_bleu (nltk.test.unit.translate.test_bleu.TestBLEUvsMteval13a)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "nltk/test/unit/translate/test_bleu.py", line 187, in test_corpus_bleu
    nltk_bleu = corpus_bleu(references, hypothesis, weights=(1.0/i,)*i)
  File "nltk/translate/bleu_score.py", line 139, in corpus_bleu
    assert len(list_of_references) == len(hypotheses), "The number of hypotheses and their reference(s) should be the same"
TypeError: object of type 'map' has no len()

----------------------------------------------------------------------
Ran 10 tests in 0.006s

FAILED (errors=1)
```

@alvations Could you please check this issue, if you have time?

P.S. If other test failures or import errors get in the way, you might find it useful to create a new branch and merge the following pending PRs: #1497, #1499, #1500, #1501


Discussion:
Ah this is a python2 and python3 difference. `map()` materializes the list in python2 but in python3 doesn't. I'll fix that in a while.

Note that there is a deprecation warning from `nltk.util.ngrams` that's kind of irritating but it'll take longer to fix. It will show up in Python3.5 with:

```
~/git/nltk$ python3 -m unittest nltk.test.unit.translate.test_bleu
....../usr/lib/python3.5/collections/__init__.py:617: PendingDeprecationWarning: generator 'ngrams' raised StopIteration
  _count_elements(self, iterable)
....
----------------------------------------------------------------------
Ran 10 tests in 0.282s
OK
```


Resolving patch:
diff --git a/nltk/translate/bleu_score.py b/nltk/translate/bleu_score.py
--- a/nltk/translate/bleu_score.py
+++ b/nltk/translate/bleu_score.py
@@ -265,14 +265,15 @@ def modified_precision(references, hypothesis, n):
     :return: BLEU's modified precision for the nth order ngram.
     :rtype: Fraction
     """
-    # Extracts all ngrams in hypothesis.
-    counts = Counter(ngrams(hypothesis, n))
+    # Extracts all ngrams in hypothesis
+    # Set an empty Counter if hypothesis is empty.
+    counts = Counter(ngrams(hypothesis, n)) if hypothesis else Counter()
 
     # Extract a union of references' counts.
     ## max_counts = reduce(or_, [Counter(ngrams(ref, n)) for ref in references])
     max_counts = {}
     for reference in references:
-        reference_counts = Counter(ngrams(reference, n))
+        reference_counts = Counter(ngrams(reference, n)) if reference else Counter()
         for ngram in counts:
             max_counts[ngram] = max(max_counts.get(ngram, 0),
                                     reference_counts[ngram])


Test patch:
Repository: nltk/nltk

Issue:
Unnecessary slow down on DecisionTreeClassifier
Just a minor unnecessary calculation that might become huge on large trees.

https://github.com/nltk/nltk/blob/develop/nltk/classify/decisiontree.py#L269

Even if verbose is false you are building that description string. Wouldn't it be a good idea to have that string building inside the if condition of verbose?



Discussion:

Resolving patch:
diff --git a/nltk/classify/decisiontree.py b/nltk/classify/decisiontree.py
--- a/nltk/classify/decisiontree.py
+++ b/nltk/classify/decisiontree.py
@@ -266,12 +266,12 @@ def best_binary_stump(feature_names, labeled_featuresets, feature_values,
                 if stump_error < best_error:
                     best_error = stump_error
                     best_stump = stump
-        if best_stump._decisions:
-            descr = '{0}={1}'.format(best_stump._fname,
-                               list(best_stump._decisions.keys())[0])
-        else:
-            descr = '(default)'
         if verbose:
+            if best_stump._decisions:
+                descr = '{0}={1}'.format(best_stump._fname,
+                                         list(best_stump._decisions.keys())[0])
+            else:
+                descr = '(default)'
             print(('best stump for {:6d} toks uses {:20} err={:6.4f}'.format \
                    (len(labeled_featuresets), descr, best_error)))
         return best_stump


Test patch:
Repository: nltk/nltk

Issue:
Better documentation for HMM unsupervised training
I had some problems using unsupervised training for hmms.
The docs say:

```
        :param unlabeled_sequences: the unsupervised training data, a set of
            sequences of observations
        :type unlabeled_sequences: list
```

When i read this i thought about a List [ word_1, ..., word_n ] < a unlabeled sequence > but the correct way is [ (word_1, tag_1),...,(word_n,tag_n) ] < a labeled sequence imo > .
I think we should explain this better to avoid confusion.



Discussion:
I would like to fix this. 


Resolving patch:
diff --git a/nltk/tag/hmm.py b/nltk/tag/hmm.py
--- a/nltk/tag/hmm.py
+++ b/nltk/tag/hmm.py
@@ -842,9 +842,11 @@ def train(self, labeled_sequences=None, unlabeled_sequences=None,
         :rtype: HiddenMarkovModelTagger
         :param labelled_sequences: the supervised training data, a set of
             labelled sequences of observations
+            ex: [ (word_1, tag_1),...,(word_n,tag_n) ] 
         :type labelled_sequences: list
         :param unlabeled_sequences: the unsupervised training data, a set of
             sequences of observations
+            ex: [ word_1, ..., word_n ]
         :type unlabeled_sequences: list
         :param kwargs: additional arguments to pass to the training methods
         """


Test patch: