internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: OpenNMT/OpenNMT-py

Issue:
report_stats is being reset after every batch (ignoring report_every)
After every batch, report_stats = Statistics() is executed in Trainer.py, so its not aggregating the stats anymore. That line should be moved into report_func or something.


Discussion:
@JianyuZhan can you check this.

Resolving patch:
diff --git a/onmt/ModelConstructor.py b/onmt/ModelConstructor.py
--- a/onmt/ModelConstructor.py
+++ b/onmt/ModelConstructor.py
@@ -168,15 +168,17 @@ def make_base_model(model_opt, fields, gpu, checkpoint=None):
         generator.load_state_dict(checkpoint['generator'])
     else:
         if model_opt.param_init != 0.0:
-            print('Intializing parameters.')
+            print('Intializing model parameters.')
             for p in model.parameters():
                 p.data.uniform_(-model_opt.param_init, model_opt.param_init)
+            for p in generator.parameters():
+                p.data.uniform_(-model_opt.param_init, model_opt.param_init)
         model.encoder.embeddings.load_pretrained_vectors(
                 model_opt.pre_word_vecs_enc, model_opt.fix_word_vecs_enc)
         model.decoder.embeddings.load_pretrained_vectors(
                 model_opt.pre_word_vecs_dec, model_opt.fix_word_vecs_dec)
 
-    # add the generator to the module (does this register the parameter?)
+    # Add generator to model (this registers it as parameter of model).
     model.generator = generator
 
     # Make the whole model leverage GPU if indicated to do so.
diff --git a/onmt/Trainer.py b/onmt/Trainer.py
--- a/onmt/Trainer.py
+++ b/onmt/Trainer.py
@@ -133,10 +133,9 @@ def train(self, epoch, report_func=None):
                     dec_state.detach()
 
             if report_func is not None:
-                report_func(epoch, i, len(self.train_iter),
-                            total_stats.start_time, self.optim.lr,
-                            report_stats)
-                report_stats = Statistics()
+                report_stats = report_func(
+                        epoch, i, len(self.train_iter),
+                        total_stats.start_time, self.optim.lr, report_stats)
 
         return total_stats
 
diff --git a/train.py b/train.py
--- a/train.py
+++ b/train.py
@@ -67,18 +67,24 @@ def report_func(epoch, batch, num_batches,
     """
     This is the user-defined batch-level traing progress
     report function.
+
     Args:
         epoch(int): current epoch count.
         batch(int): current batch count.
         num_batches(int): total number of batches.
         start_time(float): last report time.
         lr(float): current learning rate.
-        report_stats(Statistics): a Statistics instance.
+        report_stats(Statistics): old Statistics instance.
+    Returns:
+        report_stats(Statistics): updated Statistics instance.
     """
     if batch % opt.report_every == -1 % opt.report_every:
         report_stats.output(epoch, batch+1, num_batches, start_time)
         if opt.exp_host:
             report_stats.log("progress", experiment, lr)
+        report_stats = onmt.Statistics()
+
+    return report_stats
 
 
 def make_train_data_iter(train_data, opt):


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
#layers for encoder equals #layers for decoder
I just noticed that for the default RNN encoder, the `enc_layers` parameter is ignored and the number of layers of the encoder is equal to the number of layers of the decoder (see [this line](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/ModelConstructor.py#L70)).

Is there some reasoning behind this or is it an error?
#layers for encoder equals #layers for decoder
I just noticed that for the default RNN encoder, the `enc_layers` parameter is ignored and the number of layers of the encoder is equal to the number of layers of the decoder (see [this line](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/ModelConstructor.py#L70)).

Is there some reasoning behind this or is it an error?


Discussion:
Oops, I believe this is a bug, due to typo or somehow, it is written as `dec_layers`.  Thanks for reporting.

Maybe in most cases people just set it the same as `dec_layers`, so it hasn't been noticed. Will fix it.
Oops, I believe this is a bug, due to typo or somehow, it is written as `dec_layers`.  Thanks for reporting.

Maybe in most cases people just set it the same as `dec_layers`, so it hasn't been noticed. Will fix it.

Resolving patch:
diff --git a/onmt/ModelConstructor.py b/onmt/ModelConstructor.py
--- a/onmt/ModelConstructor.py
+++ b/onmt/ModelConstructor.py
@@ -67,7 +67,7 @@ def make_encoder(opt, embeddings):
         return MeanEncoder(opt.enc_layers, embeddings)
     else:
         # "rnn" or "brnn"
-        return RNNEncoder(opt.rnn_type, opt.brnn, opt.dec_layers,
+        return RNNEncoder(opt.rnn_type, opt.brnn, opt.enc_layers,
                           opt.rnn_size, opt.dropout, embeddings)
 
 


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Explicit check when training with share_embeddings and not share_vocab
Hey, Whenever I run training with share_embedding flat I get the following error:

```RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1502009910772/work/torch/lib/THC/THCCachingHostAllocator.cpp:258```

Any idea what can cause this? how can fix this!

Thank.


Discussion:
Did you used `share_vocab` during preprocessing? I think it's mandatory and I remember of this error.

Still, I encountered this kind of device-side assert error on several occasion for various reasons (on experimental code)

Oh yeah, need this at preprocessing. We should have an explicit check. 
Will do this after prerpocess sharding work. 
Just wonder currently both options are by default `True`, if not explicitly assigning a `False` value, how would @mataney encounter this problem?
@pltrdy and @srush thanks! will run preprocessing again now.
@JianyuZhan they are both with `store_true`, doesn't it mean that only when you are passing `--share_vocab` it is assigned as True?!
@mataney, that means it is the default value if you don't explicitly specify a value for it.
@JianyuZhan 

![image](https://user-images.githubusercontent.com/11559198/34255455-714f992c-e659-11e7-88b4-8f0313859043.png)
It looks like when `action='store_true'` the default is False.

So I think you have to pass `share_vocab`
@mataney, ah,  right, sorry I mess up with the `default` arg... Yep, you are right.

Resolving patch:
diff --git a/onmt/ModelConstructor.py b/onmt/ModelConstructor.py
--- a/onmt/ModelConstructor.py
+++ b/onmt/ModelConstructor.py
@@ -143,13 +143,17 @@ def make_base_model(model_opt, fields, gpu, checkpoint=None):
 
     # Make decoder.
     tgt_dict = fields["tgt"].vocab
-    # TODO: prepare for a future where tgt features are possible.
     feature_dicts = onmt.io.collect_feature_vocabs(fields, 'tgt')
     tgt_embeddings = make_embeddings(model_opt, tgt_dict,
                                      feature_dicts, for_encoder=False)
 
-    # Share the embedding matrix - preprocess with share_vocab required
+    # Share the embedding matrix - preprocess with share_vocab required.
     if model_opt.share_embeddings:
+        # src/tgt vocab should be the same if `-share_vocab` is specified.
+        if src_dict != tgt_dict:
+            raise AssertionError('The `-share_vocab` should be set during '
+                                 'preprocess if you use share_embeddings!')
+
         tgt_embeddings.word_lut.weight = src_embeddings.word_lut.weight
 
     decoder = make_decoder(model_opt, tgt_embeddings)


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
tools/embeddings_to_torch.py fails when some word features are included in the preprocessing step
When there are some word features appended to each token in the source text, it seems that the `tools/embeddings_to_torch.py` script cannot extract correct vocabulary from the dataset.
```
$ python tools/embeddings_to_torch.py -emb_file /path/to/word.vectors.txt -dict_file dataset.vocab.pt -output dataset.emb
Traceback (most recent call last):
  File "tools/embeddings_to_torch.py", line 94, in <module>
    main()
  File "tools/embeddings_to_torch.py", line 62, in main
    enc_vocab, dec_vocab = get_vocabs(opt.dict_file)
  File "tools/embeddings_to_torch.py", line 24, in get_vocabs
    enc_vocab, dec_vocab = [vocab[1] for vocab in vocabs]
ValueError: too many values to unpack (expected 2)
```


Discussion:

Resolving patch:
diff --git a/tools/embeddings_to_torch.py b/tools/embeddings_to_torch.py
--- a/tools/embeddings_to_torch.py
+++ b/tools/embeddings_to_torch.py
@@ -21,7 +21,7 @@
 
 def get_vocabs(dict_file):
     vocabs = torch.load(dict_file)
-    enc_vocab, dec_vocab = [vocab[1] for vocab in vocabs]
+    enc_vocab, dec_vocab = vocabs[0][1], vocabs[-1][1]
 
     print("From: %s" % dict_file)
     print("\t* source vocab: %d words" % len(enc_vocab))


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
translate.py error with -n_best option (n > 1)
This happens with **-n_best, n> 1** and **-verbose**.
```
$ python translate.py -model model.pt -src source.txt -n_best 10 -output pred10best.txt -replace_unk -verbose
Loading model parameters.
PRED SCORE: -6.3616

BEST HYP:
Traceback (most recent call last):
  File "translate.py", line 116, in <module>
    main()
  File "translate.py", line 97, in main
    output = trans.log(sent_number)
  File "/home/user/OpenNMT-py/onmt/translate/Translation.py", line 116, in log
    for score, sent in zip(self.pred_score, self.pred_sents):
AttributeError: 'Translation' object has no attribute 'pred_score'
```
translate.py error with -n_best option (n > 1)
This happens with **-n_best, n> 1** and **-verbose**.
```
$ python translate.py -model model.pt -src source.txt -n_best 10 -output pred10best.txt -replace_unk -verbose
Loading model parameters.
PRED SCORE: -6.3616

BEST HYP:
Traceback (most recent call last):
  File "translate.py", line 116, in <module>
    main()
  File "translate.py", line 97, in main
    output = trans.log(sent_number)
  File "/home/user/OpenNMT-py/onmt/translate/Translation.py", line 116, in log
    for score, sent in zip(self.pred_score, self.pred_sents):
AttributeError: 'Translation' object has no attribute 'pred_score'
```


Discussion:

Resolving patch:
diff --git a/onmt/translate/Translation.py b/onmt/translate/Translation.py
--- a/onmt/translate/Translation.py
+++ b/onmt/translate/Translation.py
@@ -113,7 +113,7 @@ def log(self, sent_number):
 
         if len(self.pred_sents) > 1:
             print('\nBEST HYP:')
-            for score, sent in zip(self.pred_score, self.pred_sents):
+            for score, sent in zip(self.pred_scores, self.pred_sents):
                 output += "[{:.4f}] {}\n".format(score, sent)
 
         return output


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Special symbols in merged vocab 
[Source Code](https://github.com/OpenNMT/OpenNMT-py/blob/81d94563870d0b65c8414b576a4e7a44ce5b63b2/onmt/io/IO.py#L90)

\<unk> is missing in the special symbols, 
After merged, itos looks like ['\<blank>', '\
', '\
' ... ], which has wrong index.

Special symbols in merged vocab 
[Source Code](https://github.com/OpenNMT/OpenNMT-py/blob/81d94563870d0b65c8414b576a4e7a44ce5b63b2/onmt/io/IO.py#L90)

\<unk> is missing in the special symbols, 
After merged, itos looks like ['\<blank>', '\
', '\
' ... ], which has wrong index.



Discussion:

Resolving patch:
diff --git a/onmt/io/DatasetBase.py b/onmt/io/DatasetBase.py
--- a/onmt/io/DatasetBase.py
+++ b/onmt/io/DatasetBase.py
@@ -6,6 +6,7 @@
 
 
 PAD_WORD = '<blank>'
+UNK_WORD = '<unk>'
 UNK = 0
 BOS_WORD = '
'
 EOS_WORD = '
'
diff --git a/onmt/io/IO.py b/onmt/io/IO.py
--- a/onmt/io/IO.py
+++ b/onmt/io/IO.py
@@ -7,7 +7,7 @@
 import torchtext.data
 import torchtext.vocab
 
-from onmt.io.DatasetBase import PAD_WORD, BOS_WORD, EOS_WORD
+from onmt.io.DatasetBase import UNK_WORD, PAD_WORD, BOS_WORD, EOS_WORD
 from onmt.io.TextDataset import TextDataset
 from onmt.io.ImageDataset import ImageDataset
 from onmt.io.AudioDataset import AudioDataset
@@ -87,7 +87,8 @@ def merge_vocabs(vocabs, vocab_size=None):
     """
     merged = sum([vocab.freqs for vocab in vocabs], Counter())
     return torchtext.vocab.Vocab(merged,
-                                 specials=[PAD_WORD, BOS_WORD, EOS_WORD],
+                                 specials=[UNK_WORD, PAD_WORD,
+                                           BOS_WORD, EOS_WORD],
                                  max_size=vocab_size)
 
 


Test patch:
diff --git a/test/test_preprocess.py b/test/test_preprocess.py
--- a/test/test_preprocess.py
+++ b/test/test_preprocess.py
@@ -57,8 +57,8 @@ def test_merge_vocab(self):
 
         self.assertEqual(Counter({'c': 6, 'b': 4, 'a': 2, 'e': 2, 'f': 1}),
                          merged.freqs)
-        # 3 specicials + 2 words (since we pass 2 to merge_vocabs)
-        self.assertEqual(5, len(merged.itos))
+        # 4 specicials + 2 words (since we pass 2 to merge_vocabs)
+        self.assertEqual(6, len(merged.itos))
         self.assertTrue('b' in merged.itos)
Repository: OpenNMT/OpenNMT-py

Issue:
copy_loss_by_seqlength results in overflow over value of 255 because of sum over boolean tensor
Hi,
The below code results in overflow over value of 255. See related discussion in pytorch [sum of boolean tensor](https://discuss.pytorch.org/t/sum-of-a-boolean-tensor/5719)
https://github.com/OpenNMT/OpenNMT-py/blob/3bd519070ad3b092dc081d2e72d0cd129bd085c7/onmt/modules/CopyGenerator.py#L210-L214

The line 210 should be `tgt_lens = batch.tgt.ne(pad_ix).float().sum(0)`
Thanks.



Discussion:
Thanks for noticing - could you submit a PR for this? 
Sure, will send a PR. 

Resolving patch:
diff --git a/onmt/modules/CopyGenerator.py b/onmt/modules/CopyGenerator.py
--- a/onmt/modules/CopyGenerator.py
+++ b/onmt/modules/CopyGenerator.py
@@ -207,7 +207,7 @@ def _compute_loss(self, batch, output, target, copy_attn, align):
             # Compute Loss as NLL divided by seq length
             # Compute Sequence Lengths
             pad_ix = batch.dataset.fields['tgt'].vocab.stoi[onmt.io.PAD_WORD]
-            tgt_lens = batch.tgt.ne(pad_ix).sum(0).float()
+            tgt_lens = batch.tgt.ne(pad_ix).float().sum(0)
             # Compute Total Loss per sequence in batch
             loss = loss.view(-1, batch.batch_size).sum(0)
             # Divide by length of each sequence and sum


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Program crashes if there is an empty line in the vocabulary
When preprocessing, the program crashes if the vocabulary file contains an empty line (or a line containing only one space, as happened to me).

The error is:

```
Traceback (most recent call last):
  File "preprocess.py", line 195, in <module>
    main()
  File "preprocess.py", line 188, in main
    build_save_vocab(train_dataset_files, fields, opt)
  File "preprocess.py", line 165, in build_save_vocab
    opt.tgt_words_min_frequency)
  File "/kore-ns-groups/dh/aprosio/simp/OpenNMT-py-master/onmt/io/IO.py", line 265, in build_vocab
    word = line.strip().split()[0]
```

The problem can be solved with a minor code insertion.
In both lines 262 and 273 in [IO.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/io/IO.py) this code should be added:

```
if not line.strip():
    continue
```

I think that this can be included in the next version.


Discussion:
Perhaps there's a way that this problem can be solved in a way that also resolves issues with empty lines during the translation step, as I mentioned here: https://github.com/OpenNMT/OpenNMT-py/issues/743
Thanks for reporting! Will fix.

Resolving patch:
diff --git a/onmt/io/IO.py b/onmt/io/IO.py
--- a/onmt/io/IO.py
+++ b/onmt/io/IO.py
@@ -262,6 +262,8 @@ def build_vocab(train_dataset_files, fields, data_type, share_vocab,
             'src vocab %s not found!' % src_vocab_path
         with open(src_vocab_path) as f:
             for line in f:
+                if len(line.strip()) == 0:
+                    continue
                 word = line.strip().split()[0]
                 src_vocab.add(word)
 
@@ -273,6 +275,8 @@ def build_vocab(train_dataset_files, fields, data_type, share_vocab,
             'tgt vocab %s not found!' % tgt_vocab_path
         with open(tgt_vocab_path) as f:
             for line in f:
+                if len(line.strip()) == 0:
+                    continue
                 word = line.strip().split()[0]
                 tgt_vocab.add(word)
 


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
patch to fix 0.4.1 compatibility
quick fix of #921 to make it work with pytorch 0.4.1



Discussion:

Resolving patch:
diff --git a/onmt/trainer.py b/onmt/trainer.py
--- a/onmt/trainer.py
+++ b/onmt/trainer.py
@@ -153,7 +153,7 @@ def train(self, train_iter_fct, valid_iter_fct, train_steps, valid_steps):
                     if self.norm_method == "tokens":
                         num_tokens = batch.tgt[1:].ne(
                             self.train_loss.padding_idx).sum()
-                        normalization += num_tokens
+                        normalization += num_tokens.item()
                     else:
                         normalization += batch.batch_size
                     accum += 1
@@ -167,7 +167,7 @@ def train(self, train_iter_fct, valid_iter_fct, train_steps, valid_steps):
                         if self.n_gpu > 1:
                             normalization = sum(onmt.utils.distributed
                                                 .all_gather_list
-                                                (normalization.cpu()))
+                                                (normalization))
 
                         self._gradient_accumulation(
                             true_batchs, normalization, total_stats,


Test patch:
diff --git a/onmt/tests/rebuild_test_models.sh b/onmt/tests/rebuild_test_models.sh
--- a/onmt/tests/rebuild_test_models.sh
+++ b/onmt/tests/rebuild_test_models.sh
@@ -50,7 +50,7 @@ $my_python preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train
 $my_python train.py -data data/data -save_model /tmp/tmp -batch_type tokens -batch_size 1024 -accum_count 4 \
  -layers 4 -rnn_size 256 -word_vec_size 256 -encoder_type transformer -decoder_type transformer -share_embedding \
  -train_steps 10000 -gpuid 0 -max_generator_batches 4 -dropout 0.1 -normalization tokens \
- -max_grad_norm 0 -optim sparseadam -decay_method noam -learning_rate 2 -label_smoothing 0.1 \
+ -max_grad_norm 0 -optim adam -decay_method noam -learning_rate 2 -label_smoothing 0.1 \
  -position_encoding -param_init 0 -warmup_steps 100 -param_init_glorot -adam_beta2 0.998
 #
 mv /tmp/tmp*e10.pt onmt/tests/test_model.pt
diff --git a/onmt/tests/test_models.sh b/onmt/tests/test_models.sh
--- a/onmt/tests/test_models.sh
+++ b/onmt/tests/test_models.sh
@@ -230,7 +230,7 @@ transformer(){
                     -dropout 0.1 \
                     -normalization tokens \
                     -max_grad_norm 0 \
-                    -optim sparseadam \
+                    -optim adam \
                     -decay_method noam \
                     -learning_rate 2 \
                     -position_encoding \
Repository: OpenNMT/OpenNMT-py

Issue:
valid_batch_size
In the train_single.py file , lines 120 to 124
```
def train_iter_fct(): return build_dataset_iter(
        lazily_load_dataset("train", opt), fields, opt)

    def valid_iter_fct(): return build_dataset_iter(
        lazily_load_dataset("valid", opt), fields)
```
should be changed
```
def train_iter_fct(): return build_dataset_iter(
        lazily_load_dataset("train", opt), fields, opt)

    def valid_iter_fct(): return build_dataset_iter(
        lazily_load_dataset("valid", opt), fields, opt, is_train=False)
```
If it doesn't, it will not use `valid_batch_size`.



Discussion:
Good point, wondering why it never popped before..., can you please PR ?
First, if it doesn't, it will not use `valid_batch_size`, and it will use `opt.batch_size`. 
In the inputter.py file. lines 466 to 472.

```
def build_dataset_iter(datasets, fields, opt, is_train=True):
    """
    This returns user-defined train/validate data iterator for the trainer
    to iterate over. We implement simple ordered iterator strategy here,
    but more sophisticated strategy like curriculum learning is ok too.
    """
    batch_size = opt.batch_size if is_train else opt.valid_batch_size
```

I know I get it, just asking you if you can send a PR .... thanks

Resolving patch:
diff --git a/onmt/train_single.py b/onmt/train_single.py
--- a/onmt/train_single.py
+++ b/onmt/train_single.py
@@ -121,7 +121,7 @@ def train_iter_fct(): return build_dataset_iter(
         lazily_load_dataset("train", opt), fields, opt)
 
     def valid_iter_fct(): return build_dataset_iter(
-        lazily_load_dataset("valid", opt), fields, opt)
+        lazily_load_dataset("valid", opt), fields, opt, is_train=False)
 
     # Do training.
     trainer.train(train_iter_fct, valid_iter_fct, opt.train_steps,


Test patch:
Repository: OpenNMT/OpenNMT-py

Issue:
Support gold target scoring in the new beam search



Discussion:
Thanks

Resolving patch:
diff --git a/onmt/train_single.py b/onmt/train_single.py
--- a/onmt/train_single.py
+++ b/onmt/train_single.py
@@ -137,6 +137,10 @@ def valid_iter_fct(): return build_dataset_iter(
         lazily_load_dataset("valid", opt), fields, opt, is_train=False)
 
     # Do training.
+    if len(opt.gpu_ranks):
+        logger.info('Starting training on GPU: %s' % opt.gpu_ranks)
+    else:
+        logger.info('Starting training on CPU, could be very slow')
     trainer.train(train_iter_fct, valid_iter_fct, opt.train_steps,
                   opt.valid_steps)
 
diff --git a/onmt/translate/translator.py b/onmt/translate/translator.py
--- a/onmt/translate/translator.py
+++ b/onmt/translate/translator.py
@@ -379,7 +379,7 @@ def _fast_translate_batch(self,
         results["batch"] = batch
         if "tgt" in batch.__dict__:
             results["gold_score"] = self._score_target(
-                batch, copy.deepcopy(dec_states), memory_bank, src_lengths)
+                batch, copy.deepcopy(memory_bank, src_lengths))
         else:
             results["gold_score"] = [0] * batch_size
 
@@ -697,9 +697,9 @@ def _run_target(self, batch, data):
 
         #  (2) if a target is specified, compute the 'goldScore'
         #  (i.e. log likelihood) of the target under the model
-        return self._score_target(batch, dec_states, memory_bank, src_lengths)
+        return self._score_target(batch, memory_bank, src_lengths)
 
-    def _score_target(self, batch, dec_states, memory_bank, src_lengths):
+    def _score_target(self, batch, memory_bank, src_lengths):
         tgt_in = inputters.make_features(batch, 'tgt')[:-1]
         tt = torch.cuda if self.cuda else torch
         gold_scores = tt.FloatTensor(batch.batch_size).fill_(0)
diff --git a/train.py b/train.py
--- a/train.py
+++ b/train.py
@@ -25,7 +25,7 @@ def main(opt):
     if opt.truncated_decoder > 0 and opt.accum_count > 1:
         raise AssertionError("BPTT is not compatible with -accum > 1")
 
-    if len(opt.gpuid) > 1:
+    if opt.gpuid:
         raise AssertionError("gpuid is deprecated \
               see world_size and gpu_ranks")
 


Test patch: