Repository: OpenNMT/OpenNMT-py

Issue:
LM prior fails if the LM checkpoint is not converted in CT2
It seems that the conversion is necessary. With a non converted checkpoint I have this error:
```shell
Traceback (most recent call last):
  File "/nmt-pipeline/onmt/trainer.py", line 501, in _gradient_accumulation
    self.optim.backward(loss)
  File "/nmt-pipeline/onmt/utils/optimizers.py", line 355, in backward
    self._optimizer.backward(loss, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/apex/contrib/optimizers/fp16_optimizer.py", line 140, in backward
    scaled_loss.backward()
  File "/usr/local/lib/python3.8/dist-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1976, 32768]], which is output 0 of LogSoftmaxBackward0, is at version 3; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "/usr/local/bin/nmt-pipeline", line 33, in <module>
    sys.exit(load_entry_point('NMT-pipeline', 'console_scripts', 'nmt-pipeline')())
  File "/nmt-pipeline/nmt_pipeline/bin/run_pipeline.py", line 79, in main
    pipeline.run()
  File "/nmt-pipeline/nmt_pipeline/bin/run_pipeline.py", line 55, in run
    step.run()
  File "/nmt-pipeline/nmt_pipeline/steps/step.py", line 30, in wapper_function
    results = func(*args, **kwargs)
  File "/nmt-pipeline/nmt_pipeline/steps/step.py", line 149, in run
    return self._run()
  File "/nmt-pipeline/nmt_pipeline/steps/train.py", line 230, in _run
    train_main(opts)
  File "/nmt-pipeline/onmt/bin/train.py", line 50, in train
    train_process(opt, device_id=0)
  File "/nmt-pipeline/onmt/train_single.py", line 185, in main
    trainer.train(
  File "/nmt-pipeline/onmt/trainer.py", line 287, in train
    self._gradient_accumulation(
  File "/nmt-pipeline/onmt/trainer.py", line 514, in _gradient_accumulation
    raise exc
  File "/nmt-pipeline/onmt/trainer.py", line 501, in _gradient_accumulation
    self.optim.backward(loss)
  File "/nmt-pipeline/onmt/utils/optimizers.py", line 355, in backward
    self._optimizer.backward(loss, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/apex/contrib/optimizers/fp16_optimizer.py", line 140, in backward
    scaled_loss.backward()
  File "/usr/local/lib/python3.8/dist-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1976, 32768]], which is output 0 of LogSoftmaxBackward0, is at version 3; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

```


Discussion:
can you post the options you used for training ? it should accept both ".pt" file (for non ct2 converted LM) and ct2 format.

I think your error is not related to LM prior. Seems to be a apex or cuda version mismatch. try with adam instead of fusedadam just to make sure this is unrelated.
Grabbed from @l-k-11235 's config:
```yaml
_multilingual: false
accum_count:
- 2
accum_steps:
- 0
adam_beta2: 0.998
attention_dropout:
- 0.1
average_decay: 0.0005
batch_size: 2048
batch_type: tokens
bucket_size: 8192
case_strategy: pos
data:
  ...
decay_method: noam
decoder_type: transformer
dropout:
- 0.1
dropout_steps:
- 0
dump_preds: model_TM
dump_transforms: false
encoder_type: transformer
gpu_ranks:
- 0
keep_checkpoint: 50
label_smoothing: 0.1
learning_rate: 2
lemmatization_strategy: pos
lm_prior_model: models_LM/models_LM_step_10.pt
max_grad_norm: 0
max_relative_positions: 20
model_dtype: fp16
n_sample: 0
optim: fusedadam
overwrite: true
param_init: 0
param_init_glorot: true
report_every: 50
save_checkpoint_steps: 5000
save_data: /nas-labs/MT/pytorchwork/Recipes/toy-lina/LM_prior_en-fr_V3/models_TM/onmt_run
save_model: /nas-labs/MT/pytorchwork/Recipes/toy-lina/LM_prior_en-fr_V3/models_TM/models_TM
scoring_debug: true
seed: 2345
share_decoder_embeddings: true
share_embeddings: true
share_vocab: true
skip_empty_level: silent
src_onmttok_kwargs: '{''mode'': ''aggressive'', ''joiner_annotate'': True, ''preserve_placeholders'':
  True, ''case_markup'': True, ''soft_case_regions'': True, ''preserve_segmented_tokens'':
  True}'
src_seq_length: 200
src_subword_model: /nas-labs/MT/pytorchwork/Recipes/toy-lina/LM_prior_en-fr_V3/subwords/subwords.bpe
src_subword_type: bpe
src_vocab: /nas-labs/MT/pytorchwork/Recipes/toy-lina/LM_prior_en-fr_V3/build_vocab/onmt_run.vocab.src
src_words_min_frequency: 5
tgt_onmttok_kwargs: '{''mode'': ''aggressive'', ''joiner_annotate'': True, ''preserve_placeholders'':
  True, ''case_markup'': True, ''soft_case_regions'': True, ''preserve_segmented_tokens'':
  True}'
tgt_seq_length: 200
tgt_subword_model: /nas-labs/MT/pytorchwork/Recipes/toy-lina/LM_prior_en-fr_V3/subwords/subwords.bpe
tgt_subword_type: bpe
tgt_words_min_frequency: 5
train_metrics:
- BLEU
- TER
train_steps: 130000
transformer_ff: 4096
transforms:
- onmt_tokenize
- filtertoolong
valid_batch_size: 2048
valid_metrics:
- BLEU
- TER
valid_steps: 5000
warmup_steps: 6000
word_vec_size: 1024
world_size: 1
```

If @vince62s can't replicate, probably something wrong in our setup.
Two ideas:
- some cuda OOM that would not be properly caught;
- some optimizer incompatibility indeed, as mentioned above (but not sure why this would fail on the pt case, and not CT2), and some first quick tests don't seem to do the trick.

maybe you could set torch.autograd.set_detect_anomaly(True) as mentionned but I really think there is a pytorch version mismatch somewhere. What version did you use ?
I reproduced the error, looking into it.

Resolving patch:
diff --git a/onmt/transforms/misc.py b/onmt/transforms/misc.py
--- a/onmt/transforms/misc.py
+++ b/onmt/transforms/misc.py
@@ -99,10 +99,12 @@ def get_prefix_dict(cls, opts):
             if 'infer' not in prefix_dict.keys():
                 prefix_dict['infer'] = {}
             prefix_dict['infer']['src'] = opts.src_prefix
+            logger.info(f"Get prefix for src infer: {opts.src_prefix}")
         if hasattr(opts, 'tgt_prefix'):
             if 'infer' not in prefix_dict.keys():
                 prefix_dict['infer'] = {}
             prefix_dict['infer']['tgt'] = opts.tgt_prefix
+            logger.info(f"Get prefix for tgt infer: {opts.tgt_prefix}")
 
         return prefix_dict
 
diff --git a/onmt/utils/loss.py b/onmt/utils/loss.py
--- a/onmt/utils/loss.py
+++ b/onmt/utils/loss.py
@@ -224,7 +224,7 @@ def _compute_lm_loss(self, output, target):
         lm_outs, _ = self.lm_prior_model(src, None, src_len,
                                          with_align=False)
         lm_scores = self.lm_prior_model.generator(
-            self._bottle(lm_outs)) / self.lm_prior_tau
+            self._bottle(lm_outs)).detach().clone() / self.lm_prior_tau
         # again we use raw probs to rescale with tau and apply log_softmax
         lm_scores = F.log_softmax(lm_scores.to(torch.float32), dim=-1)
         lm_scores[:, self.vocab[DefaultTokens.UNK]] = -50


Test patch: