Repository: OpenNMT/OpenNMT-py

Issue:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
I am trying to train a Transformer model with the parameters in [http://opennmt.net/OpenNMT-py/FAQ.html#how-do-i-use-the-transformer-model](url) and I get the following error: _RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation_
I run Python 3.6.1 and the most recent version of OpenNMT (cloned yesterday)


Traceback is as follows:

```
Traceback (most recent call last):
  File "train.py", line 513, in <module>
    main()
  File "train.py", line 505, in main
    train_model(model, fields, optim, data_type, model_opt)
  File "train.py", line 263, in train_model
    train_stats = trainer.train(train_iter, epoch, report_func)
  File "/home/roberto.dessi/openNMT-py/onmt/Trainer.py", line 182, in train
    report_stats, normalization)
  File "/home/roberto.dessi/openNMT-py/onmt/Trainer.py", line 315, in _gradient_accumulation
    trunc_size, self.shard_size, normalization)
  File "/home/roberto.dessi/openNMT-py/onmt/Loss.py", line 121, in sharded_compute_loss
    for shard in shards(shard_state, shard_size):
  File "/home/roberto.dessi/openNMT-py/onmt/Loss.py", line 276, in shards
    torch.autograd.backward(inputs, grads)
  File "/home/roberto.dessi/.virtualenvs/opennmt/lib/python3.6/site-packages/torch/autograd/__init__.py", line 89, in backward
    allow_unreachable=True)  # allow_unreachable flag

```

I'm using __version__ = '0.4.0' and cuda = '8.0.61'
The command I used is: 

`python train.py -data data/binarized/preprocessed -save_model models/ -gpuid 1 -layers 6 -rnn_size 512 -word_vec_size 512 -encoder_type transformer -decoder_type transformer -position_encoding -epochs 50  -max_generator_batches 32 -dropout 0.1 -batch_size 4096 -batch_type tokens -normalization tokens  -accum_count 4 -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2 -max_grad_norm 0 -param_init 0  -param_init_glorot -label_smoothing 0.1`

The output of `less data/binarized/preprocessd` is: `preprocessed.train.1.pt  preprocessed.valid.1.pt  preprocessed.vocab.pt`


Thanks in advance!


Discussion:
Did you modify the code?
I haven't run with Transformer yet but I think you modified some lines of code. This error is needed to check carefully every line of code. In your code, some variables for computations that are needed to be backward are inplace operations.
An example of inplace operation is:
`a += b`
It has to be:
`a = a + b`
This is my experience with this error. Hope this helps.
Thanks for reporting this issue! I'm able to reproduce it with your options, working to fix it now.

Resolving patch:
diff --git a/onmt/modules/Transformer.py b/onmt/modules/Transformer.py
--- a/onmt/modules/Transformer.py
+++ b/onmt/modules/Transformer.py
@@ -29,8 +29,8 @@ def __init__(self, size, hidden_size, dropout=0.1):
         self.w_1 = nn.Linear(size, hidden_size)
         self.w_2 = nn.Linear(hidden_size, size)
         self.layer_norm = onmt.modules.LayerNorm(size)
+        self.dropout_1 = nn.Dropout(dropout)
         # Save a little memory, by doing inplace.
-        self.dropout_1 = nn.Dropout(dropout, inplace=True)
         self.relu = nn.ReLU(inplace=True)
         self.dropout_2 = nn.Dropout(dropout)
 


Test patch: