internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: OpenNMT/OpenNMT-py

Issue:
bug in tally_parameters
Hi - there's a bug in _tally_parameters because `elif 'decoder' or 'generator' in name:` will always evaluate to True, since a string is True.  I'd submit the fix, but I've already got a fork with some unrelated changes on the file I don't want to submit :)


Discussion:
Good catch, but this code should still return correct parameter counts. Anything that isn't counted for the encoder should be counted for the decoder anyway so the current code is just an obfuscated if/else. The parameter counts will only be off iof you have an `NMTModel` that contains a submodule which is not called `encoder`, `decoder`, or `generator`, and which should not be counted toward the decoder parameters displayed in the log file.

Resolving patch:
diff --git a/onmt/train_single.py b/onmt/train_single.py
--- a/onmt/train_single.py
+++ b/onmt/train_single.py
@@ -34,7 +34,7 @@ def _tally_parameters(model):
     for name, param in model.named_parameters():
         if 'encoder' in name:
             enc += param.nelement()
-        elif 'decoder' or 'generator' in name:
+        elif ('decoder' or 'generator') in name:
             dec += param.nelement()
     return n_params, enc, dec
 


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Bug introduced by #1114
@bpopeters 
Last PR #1114 introduced a bug in the vocab preprocess.
I did not dig but it leads to an issue where when you translate, each generated sentence ends with a <unk>

I think there might be a mismatch wih EOS.

Can you look into it please ?
Thanks.


Discussion:
What does each generated sentence end with? I just preprocessed, trained, and translated with a small model and the output looks normal.
sorry github deleted it: < unk >
Here is the issue (doing itos in vocab.pt files) :
before your PR:
[
  "< unk >",
  "< blank >",
  "< s >",
  "< /s >",
...]
after your PR:
[
  "< unk >",
  "< blank >",
  null,
  null,
...]


I haven't been able to reproduce this bug. What options are you using? Is that the source vocabulary or target vocabulary? What's your command line?
  python3 preprocess.py -shard_size 1000000 \
	-train_src $TRAIN_SRC \
	-train_tgt $TRAIN_TGT \
	-valid_src $VALID_SRC \
	-valid_tgt $VALID_TGT \
        -src_seq_length $seq_len \
        -tgt_seq_length $seq_len \
        -share_vocab \
	-save_data exp/dataset.$sl-$tl
It's the `-share_vocab` that causes it. I'll have a fix soon.

Resolving patch:
diff --git a/onmt/inputters/inputter.py b/onmt/inputters/inputter.py
--- a/onmt/inputters/inputter.py
+++ b/onmt/inputters/inputter.py
@@ -387,8 +387,8 @@ def build_vocab(train_dataset_files, fields, data_type, share_vocab,
 def _merge_field_vocabs(src_field, tgt_field, vocab_size, min_freq):
     # in the long run, shouldn't it be possible to do this by calling
     # build_vocab with both the src and tgt data?
-    specials = [src_field.unk_token, src_field.pad_token,
-                src_field.init_token, src_field.eos_token]
+    specials = [tgt_field.unk_token, tgt_field.pad_token,
+                tgt_field.init_token, tgt_field.eos_token]
     merged = sum(
         [src_field.vocab.freqs, tgt_field.vocab.freqs], Counter()
     )


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
The comment in onmt.modules.average_attn may be wrong
The comment in onmt.modules.average_attn.AverageAttention.forward says that it gets a input of size `[batch_size x input_len x model_dim]`, and outputs a gating_outputs and a average_outputs both with size `[batch_size x 1 x model_dim]`. However, in my test the two outputs are both with size [batch_size x input_len x model_dim].
```python
import torch
import onmt.modules.average_attn as Att
model = Att.AverageAttention(3)
a = torch.arange(24).reshape(2,4,3).float()
print("intput size:", a.size())
o1, o2 = model(a)
print("o1 size:", o1.size())
print("o2.size:", o2.size())
```
intput size: torch.Size([2, 4, 3])
o1 size: torch.Size([2, 4, 3])
o2 size: torch.Size([2, 4, 3])


Discussion:
@francoishernandez  when you have time if you can have a look.

Resolving patch:
diff --git a/onmt/modules/average_attn.py b/onmt/modules/average_attn.py
--- a/onmt/modules/average_attn.py
+++ b/onmt/modules/average_attn.py
@@ -65,6 +65,9 @@ def cumulative_average(self, inputs, mask_or_step,
                 used to compute the cumulative average.
             cache: a dictionary containing the cumulative average
                 of the previous step.
+
+        Returns:
+            a tensor of the same shape and type as :obj:`inputs`.
         """
         if layer_cache is not None:
             step = mask_or_step
@@ -85,8 +88,9 @@ def forward(self, inputs, mask=None, layer_cache=None, step=None):
         Returns:
             (`FloatTensor`, `FloatTensor`):
 
-            * gating_outputs `[batch_size x 1 x model_dim]`
-            * average_outputs average attention `[batch_size x 1 x model_dim]`
+            * gating_outputs `[batch_size x input_len x model_dim]`
+            * average_outputs average attention
+                `[batch_size x input_len x model_dim]`
         """
         batch_size = inputs.size(0)
         inputs_len = inputs.size(1)


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
torchaudio has to be optional
@bpopeters 
The last change https://github.com/OpenNMT/OpenNMT-py/pull/1144/files
made torchaudio a requirement, not an optional one as it should be.

Can you fix it please ?
Thanks.


Discussion:

Resolving patch:
diff --git a/onmt/inputters/audio_dataset.py b/onmt/inputters/audio_dataset.py
--- a/onmt/inputters/audio_dataset.py
+++ b/onmt/inputters/audio_dataset.py
@@ -3,9 +3,6 @@
 from tqdm import tqdm
 
 import torch
-import torchaudio
-import librosa
-import numpy as np
 
 from onmt.inputters.dataset_base import DatasetBase
 
@@ -21,6 +18,9 @@ def sort_key(ex):
     @staticmethod
     def extract_features(audio_path, sample_rate, truncate, window_size,
                          window_stride, window, normalize_audio):
+        import torchaudio
+        import librosa
+        import numpy as np
         # torchaudio loading options recently changed. It's probably
         # straightforward to rewrite the audio handling to make use of
         # up-to-date torchaudio, but in the meantime there is a legacy


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Source/Target length distinction
## Preprocess parameters
Removed parameter `-seq_length`
New parameters `-src_seq_length` and `-tgt_seq_length`

---

## Training speed token/s
In both LUA/PyTorch OpenNMT, the training process prints a speed, in token/sec, but:

* LUA OpenNMT is printing source token/sec
* PyOpenNMT is printing target token/sec

This can lead to important differences, especially when src/tgt sequence length are different (e.g. summarization), and therefore lead to false conclusion about performances.

See also: [pytoch/example/issue#75](https://github.com/pytorch/examples/issues/75)


Discussion:

Resolving patch:
diff --git a/onmt/inputters/dataset_base.py b/onmt/inputters/dataset_base.py
--- a/onmt/inputters/dataset_base.py
+++ b/onmt/inputters/dataset_base.py
@@ -50,16 +50,6 @@ class DatasetBase(Dataset):
         the same structure as in the fields argument passed to the constructor.
     """
 
-    def __getstate__(self):
-        return self.__dict__
-
-    def __setstate__(self, _d):
-        self.__dict__.update(_d)
-
-    def __reduce_ex__(self, proto):
-        # This is a hack. Something is broken with torch pickle.
-        return super(DatasetBase, self).__reduce_ex__()
-
     def __init__(self, fields, src_examples_iter, tgt_examples_iter,
                  filter_pred=None):
 
@@ -90,6 +80,15 @@ def __init__(self, fields, src_examples_iter, tgt_examples_iter,
 
         super(DatasetBase, self).__init__(examples, fields, filter_pred)
 
+    def __getattr__(self, attr):
+        # avoid infinite recursion when fields isn't defined
+        if 'fields' not in vars(self):
+            raise AttributeError
+        if attr in self.fields:
+            return (getattr(x, attr) for x in self.examples)
+        else:
+            raise AttributeError
+
     def save(self, path, remove_fields=True):
         if remove_fields:
             self.fields = []


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Gold score computation bug
The current gold score computation has an off-by-one error. The current code gathers the logps of tgt_in = tgt[:-1], but it should be tgt[1:]. This is because tgt[:-1] is the input to the LSTM (i.e. [START] + words, without an [END] token) and tgt[1:] is the output from the LSTM (i.e. no start, with an [END] token). 

Patch to fix is below 
```
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
index dd23a82..a5ba1e9 100644
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -803,7 +803,7 @@ class Translator(object):
             memory_lengths=src_lengths, src_map=src_map)
 
         log_probs[:, :, self._tgt_pad_idx] = 0
-        gold = tgt_in
+        gold = tgt[1:]
         gold_scores = log_probs.gather(2, gold)
         gold_scores = gold_scores.sum(dim=0).view(-1)
```

A related issue is that the current random sampling code has scores that do not correspond to logps for the sentence. An easy fix to this is the following, which just accumulates all the scores instead of taking the last one.
```
diff --git a/onmt/translate/random_sampling.py b/onmt/translate/random_sampling.py
index ddc073b..19b70aa 100644
--- a/onmt/translate/random_sampling.py
+++ b/onmt/translate/random_sampling.py
@@ -116,8 +116,12 @@ class RandomSampling(DecodeStrategy):
 
         self.ensure_min_length(log_probs)
         self.block_ngram_repeats(log_probs)
-        topk_ids, self.topk_scores = sample_with_temperature(
+        topk_ids, topk_scores = sample_with_temperature(
             log_probs, self.sampling_temp, self.keep_topk)
+        if self.topk_scores is not None:
+            self.topk_scores += topk_scores
+        else:
+            self.topk_scores = topk_scores
 
         self.is_finished = topk_ids.eq(self.eos)
 
@@ -145,6 +149,7 @@ class RandomSampling(DecodeStrategy):
             return
         is_alive = ~self.is_finished.view(-1)
         self.alive_seq = self.alive_seq[is_alive]
+        self.topk_scores = self.topk_scores[is_alive]
         if self.alive_attn is not None:
             self.alive_attn = self.alive_attn[:, is_alive]
         self.select_indices = is_alive.nonzero().view(-1)
```


Discussion:
Can you please submit a PR ?
thanks!

Resolving patch:
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -806,7 +806,7 @@ def _score_target(self, batch, memory_bank, src_lengths,
             memory_lengths=src_lengths, src_map=src_map)
 
         log_probs[:, :, self._tgt_pad_idx] = 0
-        gold = tgt_in
+        gold = tgt[1:]
         gold_scores = log_probs.gather(2, gold)
         gold_scores = gold_scores.sum(dim=0).view(-1)
 


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Queue full error - Multi-GPU 1M custom dataset
```
Traceback (most recent call last):
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/OpenNMT-py/train.py", line 127, in batch_producer
    q.put(b, False)
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/queues.py", line 83, in put
    raise Full
queue.Full
[2019-06-04 07:32:18,291 INFO] Step 1200/100000; acc:  79.56; ppl:  1.82; xent: 0.60; lr: 1.00000; 33996/13919 tok/s;    401 sec
Traceback (most recent call last):
  File "train.py", line 196, in <module>
    main(opt)
  File "train.py", line 78, in main
    p.join()
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "train.py", line 180, in signal_handler
    raise Exception(msg)
Exception: 

-- Tracebacks above this line can probably
                 be ignored --

Traceback (most recent call last):
  File "/workspace/OpenNMT-py/train.py", line 138, in run
    single_main(opt, device_id, batch_queue, semaphore)
  File "/workspace/OpenNMT-py/onmt/train_single.py", line 139, in main
    valid_steps=opt.valid_steps)
  File "/workspace/OpenNMT-py/onmt/trainer.py", line 224, in train
    self._accum_batches(train_iter)):
  File "/workspace/OpenNMT-py/onmt/trainer.py", line 162, in _accum_batches
    for batch in iterator:
  File "/workspace/OpenNMT-py/onmt/train_single.py", line 116, in _train_iter
    batch = batch_queue.get()
  File "/opt/conda/envs/learn-dev/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/opt/conda/envs/learn-dev/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 109, in rebuild_cuda_tensor
    event_sync_required)
RuntimeError: CUDA error: unknown error
```


Discussion:
Can you please post your command line ?

Resolving patch:
diff --git a/train.py b/train.py
--- a/train.py
+++ b/train.py
@@ -124,7 +124,7 @@ def next_batch(device_id):
 
         # hack to dodge unpicklable `dict_keys`
         b.fields = list(b.fields)
-        q.put(b, False)
+        q.put(b)
         b = next_batch(device_id)
 
 


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
when "len(minibatch) == overflowed",  this line will return a empty list []. That cause a exception.
https://github.com/OpenNMT/OpenNMT-py/blob/668c3ef362995c55633fde592354160fec1d1efd/onmt/inputters/inputter.py#L511


Discussion:
overflowed==1, batch_size==32
Can you post a command line to reproduce this?
-data
E:/************
-save_model
E:/********
-layers
1
-rnn_size
1024
-word_vec_size
1024
-max_grad_norm
0
-optim
adam
-encoder_type
transformer
-decoder_type
transformer
-position_encoding
-dropout
0.2
-warmup_steps
8000
-learning_rate
2
-decay_method
noam
-label_smoothing
0.1
-adam_beta2
0.998
-batch_size
32
-batch_type
tokens
-normalization
tokens
-max_generator_batches
2
-train_steps
48500
-share_embeddings
-param_init_glorot
-world_size
1
-gpu_ranks
0
-valid_batch_size
16
-valid_steps
4000
-save_checkpoint_steps
4000
(I know this batch size is too small when batch_type is tokens.)
This 32 token batch_size does not make sense.
However @francoishernandez we may test and make sure it does not yield an empty minibatch

Resolving patch:
diff --git a/onmt/inputters/inputter.py b/onmt/inputters/inputter.py
--- a/onmt/inputters/inputter.py
+++ b/onmt/inputters/inputter.py
@@ -506,11 +506,16 @@ def batch_size_fn(new, count, sofar):
                 yield minibatch
                 minibatch, size_so_far = [], 0
             else:
-                yield minibatch[:-overflowed]
-                minibatch = minibatch[-overflowed:]
-                size_so_far = 0
-                for i, ex in enumerate(minibatch):
-                    size_so_far = batch_size_fn(ex, i + 1, size_so_far)
+                if overflowed == len(minibatch):
+                    logger.warning(
+                        "An example was ignored, more tokens"
+                        " than allowed by tokens batch_size")
+                else:
+                    yield minibatch[:-overflowed]
+                    minibatch = minibatch[-overflowed:]
+                    size_so_far = 0
+                    for i, ex in enumerate(minibatch):
+                        size_so_far = batch_size_fn(ex, i + 1, size_so_far)
     if minibatch:
         yield minibatch
 


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
OverflowError when running summarization
Hi there,

Task: summarization

Problem: when testing the model (running the ``` translate.py ``` script), I keep getting a ``` OverflowError ``` on the 10000-th test datum.

Has anyone else had this issue?

``` Traceback (most recent call last):
  File "translate.py", line 6, in <module>
    main()
  File "/home/khev/research/OpenNMT-py/onmt/bin/translate.py", line 49, in main
    translate(opt)
  File "/home/khev/research/OpenNMT-py/onmt/bin/translate.py", line 33, in translate
    attn_debug=opt.attn_debug
  File "/home/khev/research/OpenNMT-py/onmt/translate/translator.py", line 405, in translate
    pred_words_total)
  File "/home/khev/research/OpenNMT-py/onmt/translate/translator.py", line 843, in _report_score
    name, math.exp(-score_total / words_total)))
OverflowError: math range error

```


Discussion:
can you try #1639 ?
@pltrdy didn't you have the same issue?
Yes it's an issue. I have not been using sharding a lot, therefore it causes no trouble (the issue is raised after everything is done in my case) but yeah, it need to be fixed.

This said, it happens in pretty much everycase, so I'm not sure what to do (other than just ignoring the exception)



Resolving patch:
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -3,8 +3,8 @@
 from __future__ import print_function
 import codecs
 import os
-import math
 import time
+import numpy as np
 from itertools import count, zip_longest
 
 import torch
@@ -730,7 +730,9 @@ def _report_score(self, name, score_total, words_total):
         if words_total == 0:
             msg = "%s No words predicted" % (name,)
         else:
+            avg_score = score_total / words_total
+            ppl = np.exp(-score_total / words_total)
             msg = ("%s AVG SCORE: %.4f, %s PPL: %.4f" % (
-                name, score_total / words_total,
-                name, math.exp(-score_total / words_total)))
+                name, avg_score,
+                name, ppl))
         return msg


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Fix overflow error on translation perplexity
Now calculate `ppl` with `numpy` instead of `math` which converts overflow to `inf` and prints a warning (which make sense imo).
Fix #1655 


Discussion:

Resolving patch:
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -731,7 +731,7 @@ def _report_score(self, name, score_total, words_total):
             msg = "%s No words predicted" % (name,)
         else:
             avg_score = score_total / words_total
-            ppl = np.exp(-score_total / words_total)
+            ppl = np.exp(-score_total.item() / words_total)
             msg = ("%s AVG SCORE: %.4f, %s PPL: %.4f" % (
                 name, avg_score,
                 name, ppl))


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Logger is not passed to translator
When building the translator, there is a logger created but not passed to the translator:
https://github.com/OpenNMT/OpenNMT-py/blob/35cf4f0ae774a4aa500318879a1a4d53408ac129/onmt/bin/translate.py#L18
This results in a log file that only contains a single entry:
https://github.com/OpenNMT/OpenNMT-py/blob/35cf4f0ae774a4aa500318879a1a4d53408ac129/onmt/bin/translate.py#L24


Discussion:

Resolving patch:
diff --git a/onmt/bin/translate.py b/onmt/bin/translate.py
--- a/onmt/bin/translate.py
+++ b/onmt/bin/translate.py
@@ -15,7 +15,7 @@ def translate(opt):
     ArgumentParser.validate_translate_opts(opt)
     logger = init_logger(opt.log_file)
 
-    translator = build_translator(opt, report_score=True)
+    translator = build_translator(opt, logger=logger, report_score=True)
     src_shards = split_corpus(opt.src, opt.shard_size)
     tgt_shards = split_corpus(opt.tgt, opt.shard_size)
     shard_pairs = zip(src_shards, tgt_shards)


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Alpha channel and grayscale in image-to-text with -image_channel_size=3
For training image to text, the argument `-image_channel_size=3` imply that the images already have the good number of channel. However, some of my images are black and white and saved with only one channel or saved in RGB but with the alpha channel.
I could fix it with a change in `onmt/inputters/image_dataset.py` [here](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/inputters/image_dataset.py#L78):

from this:
```
            if self.channel_size == 1:
                img = transforms.ToTensor()(
                    Image.fromarray(cv2.imread(img_path, 0)))
            else:
                img = transforms.ToTensor()(Image.open(img_path))
```
to this:
```
            if self.channel_size == 1:
                img = transforms.ToTensor()(
                    Image.fromarray(cv2.imread(img_path, 0)))
            else:
                img = transforms.ToTensor()(
                    Image.fromarray(cv2.imread(img_path, 1)))
```
The flag in `cv2.imread` with value of 1 tell cv2 to convert to RGB no matter what the original image is.

Should I do a PR ?


Discussion:
sure feel free to send a PR

Resolving patch:
diff --git a/onmt/inputters/image_dataset.py b/onmt/inputters/image_dataset.py
--- a/onmt/inputters/image_dataset.py
+++ b/onmt/inputters/image_dataset.py
@@ -75,7 +75,8 @@ def read(self, images, side, img_dir=None):
                 img = transforms.ToTensor()(
                     Image.fromarray(cv2.imread(img_path, 0)))
             else:
-                img = transforms.ToTensor()(Image.open(img_path))
+                img = Image.open(img_path).convert('RGB')
+                img = transforms.ToTensor()(img)
             if self.truncate and self.truncate != (0, 0):
                 if not (img.size(1) <= self.truncate[0]
                         and img.size(2) <= self.truncate[1]):


Test patch: