Repository: OpenNMT/OpenNMT-py

Issue:
[BUG] dynamic_dict/copy_attn does not work with accum_count > 1
I am getting the following error when I tried to train transformer model on CNNDM dataset for summarization.

> Loading train dataset from data/cnndm/CNNDM.train.3.pt, number of examples: 33018
> Traceback (most recent call last):
>   File "train.py", line 502, in <module>
>     main()
>   File "train.py", line 494, in main
>     train_model(model, fields, optim, data_type, model_opt)
>   File "train.py", line 257, in train_model
>     train_stats = trainer.train(train_iter, epoch, report_func)
>   File "/home/wasiahmad/workspace/projects/OpenNMT-py/onmt/Trainer.py", line 178, in train
>     report_stats, normalization)
>   File "/home/wasiahmad/workspace/projects/OpenNMT-py/onmt/Trainer.py", line 311, in _gradient_accumulation
>     trunc_size, self.shard_size, normalization)
>   File "/home/wasiahmad/workspace/projects/OpenNMT-py/onmt/Loss.py", line 123, in sharded_compute_loss
>     loss, stats = self._compute_loss(batch, **shard)
>   File "/home/wasiahmad/workspace/projects/OpenNMT-py/onmt/modules/CopyGenerator.py", line 191, in _compute_loss
>     batch, self.tgt_vocab, self.cur_dataset.src_vocabs)
>   File "/home/wasiahmad/workspace/projects/OpenNMT-py/onmt/io/TextDataset.py", line 116, in collapse_copy_scores
>     src_vocab = src_vocabs[index]
> IndexError: list index out of range

The training works fine for the first part of the data but whenever it loads the second part (ex., `CNNDM.train.3.pt` file), it gives index out of range error. Any idea what is hapenning? 

I have downloaded the dataset from the link mentioned [here](http://opennmt.net/OpenNMT-py/Summarization.html) and did the preprocessing without any error. Please help.








Discussion:
Hi, same problem for me. Anyone knows why?
I tried a lot but couldn't find the reason behind this error.
@vince62s @wasiahmad what was the eventual resolution?
did you try with the lates code + pytorch 0.4 + torchtext 0.3 ?
i tried it on master as of 3 days ago, currently rerunning now
I tried but didn't work and then stopped trying. It was around 1.5 months ago!!
okay, I can replicate this. We'll look into it.
Thanks to @pltrdy who helped figuring it out.
The issue is as follow:
When accum_count > 1, when using copy_attn (and therefore dynamic_dict) there is a mismatch if the batches within the "true_batch" = accum_count x batch are accross different shards of the dataset.

For instance, at the end of the first shard, if we have accum_count = 4, we can have 2 batches loaded from the first shard and the last 2 from the second shard.

In this context, we have an issue here: https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/modules/copy_generator.py#L195-L197

where self.cur_dataset.src_vocabs may refer to the wrong shard.

This is not easy to solve.

In the meantime, use only accum_count = 1 with copy_attn and if you can use more GPU.

I have tried to go back in time and in fact it seems to have never worked.
So I am a bit confused that the tuto specification with copy_attn + accum_count = 4

@srush  @sebastianGehrmann  can you check the config of the model you posted for summarization ?

Thanks.




Resolving patch:
diff --git a/onmt/inputters/inputter.py b/onmt/inputters/inputter.py
--- a/onmt/inputters/inputter.py
+++ b/onmt/inputters/inputter.py
@@ -440,23 +440,19 @@ def __len__(self):
         assert self.cur_iter is not None
         return len(self.cur_iter)
 
-    def get_cur_dataset(self):
-        """ Return the current dataset settings """
-        return self.cur_dataset
-
     def _next_dataset_iterator(self, dataset_iter):
         try:
-            self.cur_dataset = next(dataset_iter)
+            cur_dataset = next(dataset_iter)
         except StopIteration:
             return None
 
         # We clear `fields` when saving, restore when loading.
-        self.cur_dataset.fields = self.fields
+        cur_dataset.fields = self.fields
 
         # Sort batch by decreasing lengths of sentence required by pytorch.
         # sort=False means "Use dataset's sortkey instead of iterator's".
         return OrderedIterator(
-            dataset=self.cur_dataset, batch_size=self.batch_size,
+            dataset=cur_dataset, batch_size=self.batch_size,
             batch_size_fn=self.batch_size_fn,
             device=self.device, train=self.is_train,
             sort=False, sort_within_batch=True,
diff --git a/onmt/modules/copy_generator.py b/onmt/modules/copy_generator.py
--- a/onmt/modules/copy_generator.py
+++ b/onmt/modules/copy_generator.py
@@ -153,10 +153,6 @@ def __init__(self, generator, tgt_vocab,
                  eps=1e-20):
         super(CopyGeneratorLossCompute, self).__init__(
             generator, tgt_vocab)
-
-        # We lazily load datasets when there are more than one, so postpone
-        # the setting of cur_dataset.
-        self.cur_dataset = None
         self.force_copy = force_copy
         self.normalize_by_length = normalize_by_length
         self.criterion = CopyGeneratorCriterion(len(tgt_vocab), force_copy,
@@ -194,7 +190,7 @@ def _compute_loss(self, batch, output, target, copy_attn, align):
         scores_data = scores.data.clone()
         scores_data = inputters.TextDataset.collapse_copy_scores(
             self._unbottle(scores_data, batch.batch_size),
-            batch, self.tgt_vocab, self.cur_dataset.src_vocabs)
+            batch, self.tgt_vocab, batch.dataset.src_vocabs)
         scores_data = self._bottle(scores_data)
 
         # Correct target copy token instead of <unk>
diff --git a/onmt/trainer.py b/onmt/trainer.py
--- a/onmt/trainer.py
+++ b/onmt/trainer.py
@@ -145,8 +145,6 @@ def train(self, train_iter_fct, valid_iter_fct, train_steps, valid_steps):
                     if self.gpu_verbose_level > 1:
                         logger.info("GpuRank %d: index: %d accum: %d"
                                     % (self.gpu_rank, i, accum))
-                    cur_dataset = train_iter.get_cur_dataset()
-                    self.train_loss.cur_dataset = cur_dataset
 
                     true_batchs.append(batch)
 
@@ -221,9 +219,6 @@ def validate(self, valid_iter):
         stats = onmt.utils.Statistics()
 
         for batch in valid_iter:
-            cur_dataset = valid_iter.get_cur_dataset()
-            self.valid_loss.cur_dataset = cur_dataset
-
             src = inputters.make_features(batch, 'src', self.data_type)
             if self.data_type == 'text':
                 _, src_lengths = batch.src


Test patch: