internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: NVIDIA/apex

Issue:
Got unexpected keyword argument 'channel_last' when using apex.parallel.convert_syncbn_model
As https://github.com/NVIDIA/apex/blob/b2f63c48408e1110bb1c7ad7bf6310141da30616/apex/parallel/__init__.py#L41 shows, `apex.parallel.convert_syncbn_model` pass `channel_last` to `SyncBatchNorm`, but `SyncBatchNorm` does not accept. I'm not sure why this function adds a `channel_last` keyword, PyTorch uses channel first tensor, while TensorFlow uses channel last. Simply remove channel_last from the function call fix this issue.


Discussion:
I should have done a better job maintaining the API 😭 

channel_last flag is supported by our optimized BN implementation (Requires installation with `--cuda_ext`). You are encouraged to use the optimized version as it provides better perf with fused kernels.

I'll update the primitive op implementation so that it accepts this dummy argument. (as suggested by @egaburov)

Resolving patch:
diff --git a/apex/parallel/optimized_sync_batchnorm.py b/apex/parallel/optimized_sync_batchnorm.py
--- a/apex/parallel/optimized_sync_batchnorm.py
+++ b/apex/parallel/optimized_sync_batchnorm.py
@@ -55,7 +55,7 @@ class SyncBatchNorm(_BatchNorm):
         >>> inp = torch.randn(10, 14, 14, 100).cuda()
     """
 
-    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True, track_running_stats=True, process_group=None, channel_last = False):
+    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True, track_running_stats=True, process_group=None, channel_last=False):
         super(SyncBatchNorm, self).__init__(num_features, eps=eps, momentum=momentum, affine=affine, track_running_stats=track_running_stats)
         self.process_group = process_group
         self.channel_last = channel_last
diff --git a/apex/parallel/sync_batchnorm.py b/apex/parallel/sync_batchnorm.py
--- a/apex/parallel/sync_batchnorm.py
+++ b/apex/parallel/sync_batchnorm.py
@@ -48,7 +48,9 @@ class SyncBatchNorm(_BatchNorm):
 
     warned = False
 
-    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True, track_running_stats=True, process_group=None):
+    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True, track_running_stats=True, process_group=None, channel_last=False):
+        if channel_last == True:
+            raise AttributeError("channel_last is not supported by primitive SyncBatchNorm implementation. Try install apex with `--cuda_ext` if channel_last is desired.")
 
         if not SyncBatchNorm.warned:
             print("Warning:  using Python fallback for SyncBatchNorm, possibly because apex was installed without --cuda_ext.  The exception raised when attempting to import the cuda backend was: ", self.syncbn_import_error)


Test patch:
Repository: NVIDIA/apex

Issue:
Failing optim_wrapper due to missing Scaler argument
When creating an optimizer and wrapping it via ``amp_handle.wrap_optimizer(optim)``, the handle [`calls the OptimWrapper`](https://github.com/NVIDIA/apex/blob/master/apex/amp/handle.py#L148), who wraps the optimizer and tries to instantiate a loss scaler per loss. 

The `OptimWrapper` tries to [instantiate the loss-scaler without an argument](https://github.com/NVIDIA/apex/blob/master/apex/amp/opt.py#L16), but the loss-scaler [needs an argument `loss_scale`](https://github.com/NVIDIA/apex/blob/master/apex/amp/scaler.py#L28), which causes the whole wrapping process to fail.


Discussion:

Resolving patch:
diff --git a/apex/amp/opt.py b/apex/amp/opt.py
--- a/apex/amp/opt.py
+++ b/apex/amp/opt.py
@@ -13,7 +13,7 @@ def __init__(self, optimizer, amp_handle, num_loss):
         self._num_loss = num_loss
         self._loss_idx = 0
         self._skip_next = [False] * num_loss
-        self._loss_scaler = [LossScaler() for _ in range(num_loss)]
+        self._loss_scaler = [LossScaler('dynamic') for _ in range(num_loss)]
 
     @contextlib.contextmanager
     def scale_loss(self, loss):


Test patch:
Repository: NVIDIA/apex

Issue:
Using --keep-batchnorm-fp32 with --opt-level O1 results in error
 python main_amp.py -a resnet50 -j 20 -b 128 --opt-level O1 --keep-batchnorm-fp32 True /data/imagenet/images/

results in the following error:

`Traceback (most recent call last):
  File "main_amp.py", line 506, in <module>
    main()
  File "main_amp.py", line 153, in main
    loss_scale=args.loss_scale
  File "/opt/conda/lib/python3.6/site-packages/apex/amp/frontend.py", line 333, in initialize
    _amp_state.opt_properties.keep_batchnorm_fp32 = keep_batchnorm_fp32
  File "/opt/conda/lib/python3.6/site-packages/apex/amp/frontend.py", line 73, in __setattr__
    "keep_batchnorm_fp32 was {}".format(keep_batchnorm_fp32))
NameError: name 'keep_batchnorm_fp32' is not defined
`

Is it something expected?


Discussion:

Resolving patch:
diff --git a/apex/amp/frontend.py b/apex/amp/frontend.py
--- a/apex/amp/frontend.py
+++ b/apex/amp/frontend.py
@@ -70,7 +70,7 @@ def __setattr__(self, name, value):
                     if self.opt_level == "O1" and value is not None:
                         warn_or_err("With opt_level O1, batchnorm functions are automatically patched "
                                     "to run in FP32, so keep_batchnorm_fp32 should be None." +
-                                    "keep_batchnorm_fp32 was {}".format(keep_batchnorm_fp32))
+                                    " keep_batchnorm_fp32 was {}".format(value))
                     if value == "False":
                         self.options[name] = False
                     elif value == "True":
@@ -78,7 +78,7 @@ def __setattr__(self, name, value):
                     else:
                         assert (value is True or value is False or value is None),\
                             "keep_batchnorm_fp32 must be a boolean, the string 'True' or 'False', "\
-                            "or None, found keep_batchnorm_fp32={}".format(keep_batchnorm_fp32)
+                            "or None, found keep_batchnorm_fp32={}".format(value)
                         self.options[name] = value
                 elif name == "master_weights":
                     if self.opt_level == "O1" and value is not None:


Test patch:
Repository: NVIDIA/apex

Issue:
Does Apex really support PyTorch 0.4.1?
In Readme.md it says:

> PyTorch 0.4 or newer.

But sample code [examples/simple/distributed](https://github.com/NVIDIA/apex/tree/master/examples/simple/distributed) goes down with error `AttributeError: 'Linear' object has no attribute 'named_buffers'`, because `named_buffers` method of torch.Module was introduced after 0.4.1 version.

Is there any build, that supports pytorch 0.4.1? If not, at least maybe the readme.md should be changed?


P.S.: I'm using apex with CUDA and C++ libraries, installed with command `pip install -v --no-cache-dir .`


Discussion:
Hi @WrathOfGrapes,

thanks for letting us know about this issue!
The `named_buffers()` call slipped in and creates the backward incompatibility.
I'll check what changes would be needed to successfully run with PyTorch `0.4` again.

Resolving patch:
diff --git a/apex/amp/_initialize.py b/apex/amp/_initialize.py
--- a/apex/amp/_initialize.py
+++ b/apex/amp/_initialize.py
@@ -89,7 +89,16 @@ def check_params_fp32(models):
                         "you chose. Use model.to('cuda') to use the default device.".format(
                         name, param.type()))
 
-        for name, buf in model.named_buffers():
+        # Backward compatibility for PyTorch 0.4
+        if hasattr(model, 'named_buffers'):
+            buf_iter = model.named_buffers()
+        else:
+            buf_iter = model._buffers
+        for obj in buf_iter:
+            if type(obj)==tuple:
+                name, buf = obj
+            else:
+                name, buf = obj, buf_iter[obj]
             if buf.is_floating_point():
                 if 'Half' in buf.type():
                     warn_or_err("Found buffer {} with type {}, expected torch.cuda.FloatTensor.\n"


Test patch:
Repository: NVIDIA/apex

Issue:
'function' object has no attribute '__self__' in Pytorch 1.3
After upgrading to Pytorch 1.3.0, my training script crashes with `AttributeError`. Relevant lines from the stack:
```
Traceback (most recent call last):
...
  File "/usr/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 299, in __init__
    super(CosineAnnealingLR, self).__init__(optimizer, last_epoch)
  File "/usr/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 58, in __init__
    self.optimizer.step = with_counter(self.optimizer.step)
  File "/usr/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 40, in with_counter
    instance_ref = weakref.ref(method.__self__)
AttributeError: 'function' object has no attribute '__self__'
```

The `step` method of the Amp optimizer is simply patched with a function, but is not made a method again in https://github.com/NVIDIA/apex/blob/088985936518be7e25795a30d8ab33affa9db6ed/apex/amp/_initialize.py#L235-L245, therefore it does not have a `__self__`.


Discussion:

Resolving patch:
diff --git a/apex/amp/_initialize.py b/apex/amp/_initialize.py
--- a/apex/amp/_initialize.py
+++ b/apex/amp/_initialize.py
@@ -3,6 +3,7 @@
 import functools
 import numpy as np
 import sys
+from types import MethodType
 import warnings
 from ._amp_state import _amp_state, warn_or_err, container_abcs
 from .handle import disable_casts
@@ -236,13 +237,13 @@ def new_fwd(*args, **kwargs):
             # Disable Amp casting for the optimizer step, because it should only be
             # applied to FP32 master params anyway.
             def patch_step(old_step):
-                def new_step(*args, **kwargs):
+                def new_step(self, *args, **kwargs):
                     with disable_casts():
                         output = old_step(*args, **kwargs)
                     return output
                 return new_step
 
-            optimizer.step = patch_step(optimizer.step)
+            optimizer.step = MethodType(patch_step(optimizer.step), optimizer)
 
     if optimizers_was_list:
         if models_was_list:


Test patch:
Repository: NVIDIA/apex

Issue:
RuntimeError: "GeluCUDAKernelImpl" not implemented for 'Half'
PyTorch 1.2 introduced the `gelu` activation function. Unfortunately, this leads to terminal errors when using with AMP.

Trace (`self.activation` is `gelu`): 

```
Traceback (most recent call last):
  File "predict.py", line 282, in <module>
    predictor.predict()
  File "predict.py", line 74, in predict
    fig = trainer.train()
  File "/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/TransformerTrainer.py", line 232, in train
    self._process('train', epoch)
  File "/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/TransformerTrainer.py", line 124, in _process
    preds = self.model(input_ids, attention_mask=input_mask)
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 442, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bram/Python/projects/transformer-classifiers/transformer_classifiers/models.py", line 140, in forward
    cls_output = self.activation(cls_output)
  File "/home/bram/.local/share/virtualenvs/transformer-classifiers-x27iJBv7/lib/python3.7/site-packages/torch/nn/functional.py", line 1126, in gelu
    return torch._C._nn.gelu(input)
RuntimeError: "GeluCUDAKernelImpl" not implemented for 'Half'
```


Discussion:
Try adding `'gelu',` to the fp32 list [here](https://github.com/NVIDIA/apex/blob/master/apex/amp/lists/functional_overrides.py#L40) and reinstall
```
$ pip uninstall apex # repeat if necessary
$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```
If that fixes it in your code I will commit.

Resolving patch:
diff --git a/apex/amp/lists/functional_overrides.py b/apex/amp/lists/functional_overrides.py
--- a/apex/amp/lists/functional_overrides.py
+++ b/apex/amp/lists/functional_overrides.py
@@ -37,7 +37,8 @@
     'softmin',
     'log_softmax',
     'softmax',
-
+    'gelu',
+    
     # Normalization
     'layer_norm',
     'group_norm',


Test patch:
Repository: NVIDIA/apex

Issue:
_amp_state determines whether running in distributed at import
It looks like the `_amp_state.py` module determines whether pytorch is running in distributed mode at the import level. The `distributed` only seems to be used in `maybe_print`. See code snippet:

https://github.com/NVIDIA/apex/blob/37cdaf4ad57ab4e7dd9ef13dbed7b29aa939d061/apex/amp/_amp_state.py#L38-L52

This causes a couple issues:

1. It will only support the `env://` initialization of torch distributed
2. It will fail if amp is imported before launching the distributed training

Neither of these is an issue for most, since most people launch via `torch.distributed.launch`. However, it can be an issue if you define your own distributed launch function or use `torch.multiprocessing.spawn`. I can't see a good reason to do it this way anyway, as it appears this variable is only used in the `maybe_print` function. I'll submit a pull request to fix this. Let me know if I'm missing something though.


Discussion:

Resolving patch:
diff --git a/apex/amp/_amp_state.py b/apex/amp/_amp_state.py
--- a/apex/amp/_amp_state.py
+++ b/apex/amp/_amp_state.py
@@ -1,5 +1,5 @@
 # This is a "header object" that allows different amp modules to communicate.
-# I'm a C++ guy, not a python guy.  I decided this approach because it seemed most C++-like.  
+# I'm a C++ guy, not a python guy.  I decided this approach because it seemed most C++-like.
 # But apparently it's ok:
 # http://effbot.org/pyfaq/how-do-i-share-global-variables-across-modules.htm
 import os
@@ -35,12 +35,9 @@ def warn_or_err(msg):
         #                    "hard_override=True to amp.initialize.")
 
 
-distributed = False
-if 'WORLD_SIZE' in os.environ:
-    distributed = int(os.environ['WORLD_SIZE']) > 1
-
-
 def maybe_print(msg, rank0=False):
+    distributed = torch.distributed.is_initialized() and \
+        torch.distributed.get_world_size() > 1
     if _amp_state.verbosity > 0:
         if rank0:
             if distributed:


Test patch:
Repository: nltk/nltk

Issue:
Can't train a TnT tagger in Python 3
If you do:

``` python
from nltk.tag import tnt
tnt.demo()
```

... then it fails during training with an error like this, at least in Python 3:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alex/nltk/nltk/tag/tnt.py", line 484, in demo
    tagger.train(sents[200:1000])
  File "/home/alex/nltk/nltk/tag/tnt.py", line 166, in train
    self._compute_lambda()
  File "/home/alex/nltk/nltk/tag/tnt.py", line 199, in _compute_lambda
    for history in self._tri.conditions():
  File "/home/alex/nltk/nltk/probability.py", line 1871, in conditions
    return sorted(self.keys())
TypeError: unorderable types: str() < tuple()
```

Python 2.7 works for me, though.



Discussion:

Resolving patch:
diff --git a/nltk/tag/tnt.py b/nltk/tag/tnt.py
--- a/nltk/tag/tnt.py
+++ b/nltk/tag/tnt.py
@@ -140,7 +140,7 @@ def train(self, data):
             self._unk.train(data)
 
         for sent in data:
-            history = ['BOS', 'BOS']
+            history = [('BOS',False), ('BOS',False)]
             for w, t in sent:
 
                 # if capitalization is requested,


Test patch:
Repository: nltk/nltk

Issue:
Featstruct.py: Use of variable before check
In _parse_seq_value (line 2324) m.group(1) is attempted before checking if m is not None. This results in a crash with no information rather than a ValueError. Move this line after the check.



Discussion:

Resolving patch:
diff --git a/nltk/featstruct.py b/nltk/featstruct.py
--- a/nltk/featstruct.py
+++ b/nltk/featstruct.py
@@ -2321,8 +2321,8 @@ def _parse_seq_value(self, s, position, reentrances, match,
 
             # Comma or looking at close paren
             m = re.compile(r'\s*(,|\+|(?=%s))\s*' % cp).match(s, position)
-            if m.group(1) == '+': seen_plus = True
             if not m: raise ValueError("',' or '+' or '%s'" % cp, position)
+            if m.group(1) == '+': seen_plus = True
             position = m.end()
 
 ######################################################################


Test patch:
Repository: nltk/nltk

Issue:
Remove prob_parse from ParserI
The ParserI methods `prob_parse` and `prob_parse_sents` are not used anywhere, and are not defined anywhere, so I propose to remove them from `nltk/parse/api.py`



Discussion:
+1


Resolving patch:
diff --git a/nltk/parse/api.py b/nltk/parse/api.py
--- a/nltk/parse/api.py
+++ b/nltk/parse/api.py
@@ -26,7 +26,6 @@ class ParserI(object):
 
     Subclasses may define:
       - ``grammar()``
-      - either ``prob_parse()`` or ``prob_parse_sents()`` (or both)
     """
     def grammar(self):
         """
@@ -95,22 +94,6 @@ def iter_parse(self, sent):
         else:
             raise NotImplementedError()
 
-    def prob_parse(self, sent):
-        """
-        :return: A probability distribution over the possible parse
-        trees for the given sentence.  If there are no possible parse
-        trees for the given sentence, return a probability distribution
-        that assigns a probability of 1.0 to None.
-
-        :param sent: The sentence to be parsed
-        :type sent: list(str)
-        :rtype: ProbDistI(Tree)
-        """
-        if overridden(self.prob_parse_sents):
-            return self.prob_parse_sents([sent])[0]
-        else:
-            raise NotImplementedError
-
     def parse_sents(self, sents):
         """
         Apply ``self.parse()`` to each element of ``sents``.  I.e.:
@@ -141,13 +124,3 @@ def iter_parse_sents(self, sents):
         """
         return [self.iter_parse(sent) for sent in sents]
 
-    def prob_parse_sents(self, sents):
-        """
-        Apply ``self.prob_parse()`` to each element of ``sents``.  I.e.:
-
-            return [self.prob_parse(sent) for sent in sents]
-
-        :rtype: list(ProbDistI(Tree))
-        """
-        return [self.prob_parse(sent) for sent in sents]
-


Test patch:
Repository: nltk/nltk

Issue:
MaltParser not completely updated to new API
Because of API changes, MaltParser objects fail when raw_parse() is called, since MaltParser.raw_parse internally calls parse() with two arguments (in addition to self).

Error occurs in malt.py, line 127, though other issues may exist.



Discussion:
@heatherleaf are you able to take a look at this please?


Resolving patch:
diff --git a/nltk/parse/dependencygraph.py b/nltk/parse/dependencygraph.py
--- a/nltk/parse/dependencygraph.py
+++ b/nltk/parse/dependencygraph.py
@@ -284,7 +284,7 @@ def to_conll(self, style):
             elif style == 4:
                 lines.append('%s\t%s\t%s\t%s\n' % (word, tag, head, rel))
             elif style == 10:
-                lines.append('%s\t%s\t%s\t%s\t%s\%s\t%s\t%s\t_\t_\n' % (i+1, word, lemma, ctag, tag, feats, head, rel))
+                lines.append('%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t_\t_\n' % (i+1, word, lemma, ctag, tag, feats, head, rel))
             else:
                 raise ValueError('Number of tab-delimited fields (%d) not supported by CoNLL(10) or Malt-Tab(4) format' % (style))
         return ''.join(lines)
diff --git a/nltk/parse/malt.py b/nltk/parse/malt.py
--- a/nltk/parse/malt.py
+++ b/nltk/parse/malt.py
@@ -113,6 +113,18 @@ def parse_sents(self, sentences, verbose=False):
         tagged_sentences = [self.tagger.tag(sentence) for sentence in sentences]
         return self.tagged_parse_sents(tagged_sentences, verbose)
 
+    def parse(self, sentence, verbose=False):
+        """
+        Use MaltParser to parse a sentence. Takes a sentence as a list of words.
+        The sentence will be automatically tagged with this MaltParser instance's
+        tagger.
+
+        :param sentence: Input sentence to parse
+        :type sentence: list(str)
+        :return: ``DependencyGraph`` the dependency graph representation of the sentence
+        """
+        return self.parse_sents([sentence], verbose)[0]
+
     def raw_parse(self, sentence, verbose=False):
         """
         Use MaltParser to parse a sentence. Takes a sentence as a string;


Test patch:
Repository: nltk/nltk

Issue:
Replace Portuguese suffixes by Spanish suffixes in the Spanish snowball stemmer
I'm creating this issue to track the regression introduced while fixing #754 (see comments on 986d103), so we don't forget it.



Discussion:

Resolving patch:
diff --git a/nltk/stem/snowball.py b/nltk/stem/snowball.py
--- a/nltk/stem/snowball.py
+++ b/nltk/stem/snowball.py
@@ -3343,8 +3343,8 @@ class SpanishStemmer(_StandardStemmer):
                         "les", "los", "nos", "me", "se", "la", "le",
                         "lo")
     __step1_suffixes = ('amientos', 'imientos', 'amiento', 'imiento',
-                        'aciones', 'uções', 'adoras', 'adores',
-                        'ancias', 'logias', 'encias', 'amente',
+                        'aciones', 'uciones', 'adoras', 'adores',
+                        'ancias', 'log\xEDas', 'encias', 'amente',
                         'idades', 'anzas', 'ismos', 'ables', 'ibles',
                         'istas', 'adora', 'aci\xF3n', 'antes',
                         'ancia', 'log\xEDa', 'uci\xf3n', 'encia',
@@ -3475,11 +3475,11 @@ def stem(self, word):
                             word = word[:-2]
                             rv = rv[:-2]
 
-                    elif suffix in ("logia", "logias"):
+                    elif suffix in ("log\xEDa", "log\xEDas"):
                         word = word.replace(suffix, "log")
                         rv = rv.replace(suffix, "log")
 
-                    elif suffix in ("ução", "uções"):
+                    elif suffix in ("uci\xF3n", "uciones"):
                         word = word.replace(suffix, "u")
                         rv = rv.replace(suffix, "u")
 


Test patch: