Repository: OpenNMT/OpenNMT-py

Issue:
Inconsistent tensor types when using average attn in Transformer
Commit ID: `6b93582`
I'm trying to train a transformer model using the following command:
```
python train.py -data data/data -save_model data/model \
-layers 2 -rnn_size 256 -word_vec_size 256 \
-encoder_type transformer -decoder_type transformer \
-position_encoding -max_generator_batches 32 -dropout 0.2 \
-batch_type tokens -normalization tokens -accum_count 4 \
-optim adam -adam_beta2 0.99 -decay_method noam \
-warmup_steps 8000 -learning_rate 2 -max_grad_norm 0 \
-param_init 0 -param_init_glorot -label_smoothing 0.1 \
-batch_size 8192 -report_every 1 -save_checkpoint_steps 5 \
-self_attn_type average
```
I get the following traceback:
```
Traceback (most recent call last):
  File "train.py", line 37, in <module>
    main(opt)
  File "train.py", line 24, in main
    single_main(opt)
  File "/home/vikrant/src/OpenNMT-py/onmt/train_single.py", line 128, in main
    opt.valid_steps)
  File "/home/vikrant/src/OpenNMT-py/onmt/trainer.py", line 175, in train
    report_stats)
  File "/home/vikrant/src/OpenNMT-py/onmt/trainer.py", line 282, in _gradient_accumulation
    self.model(src, tgt, src_lengths, dec_state)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/models/model.py", line 52, in forward
    memory_lengths=lengths)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/decoders/transformer.py", line 217, in forward
    step=step)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/decoders/transformer.py", line 86, in forward
    layer_cache=layer_cache, step=step)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/modules/average_attn.py", line 97, in forward
    if layer_cache is None else step, layer_cache=layer_cache)
  File "/home/vikrant/src/OpenNMT-py/onmt/modules/average_attn.py", line 47, in cumulative_average_mask
    weights = torch.ones((1, inputs_len)) / torch.arange(1, inputs_len + 1)
RuntimeError: Expected object of type torch.FloatTensor but found type torch.LongTensor for argument #2 'other'
```

The command works fine with `-self_attn_type` set to `scaled-dot` or omitted (as it's the default).
I do not have a GPU on my machine, if that helps.
Am I using it wrong?
Thanks in advance.
Inconsistent tensor types when using average attn in Transformer
Commit ID: `6b93582`
I'm trying to train a transformer model using the following command:
```
python train.py -data data/data -save_model data/model \
-layers 2 -rnn_size 256 -word_vec_size 256 \
-encoder_type transformer -decoder_type transformer \
-position_encoding -max_generator_batches 32 -dropout 0.2 \
-batch_type tokens -normalization tokens -accum_count 4 \
-optim adam -adam_beta2 0.99 -decay_method noam \
-warmup_steps 8000 -learning_rate 2 -max_grad_norm 0 \
-param_init 0 -param_init_glorot -label_smoothing 0.1 \
-batch_size 8192 -report_every 1 -save_checkpoint_steps 5 \
-self_attn_type average
```
I get the following traceback:
```
Traceback (most recent call last):
  File "train.py", line 37, in <module>
    main(opt)
  File "train.py", line 24, in main
    single_main(opt)
  File "/home/vikrant/src/OpenNMT-py/onmt/train_single.py", line 128, in main
    opt.valid_steps)
  File "/home/vikrant/src/OpenNMT-py/onmt/trainer.py", line 175, in train
    report_stats)
  File "/home/vikrant/src/OpenNMT-py/onmt/trainer.py", line 282, in _gradient_accumulation
    self.model(src, tgt, src_lengths, dec_state)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/models/model.py", line 52, in forward
    memory_lengths=lengths)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/decoders/transformer.py", line 217, in forward
    step=step)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/decoders/transformer.py", line 86, in forward
    layer_cache=layer_cache, step=step)
  File "/home/vikrant/src/OpenNMT-py/.env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vikrant/src/OpenNMT-py/onmt/modules/average_attn.py", line 97, in forward
    if layer_cache is None else step, layer_cache=layer_cache)
  File "/home/vikrant/src/OpenNMT-py/onmt/modules/average_attn.py", line 47, in cumulative_average_mask
    weights = torch.ones((1, inputs_len)) / torch.arange(1, inputs_len + 1)
RuntimeError: Expected object of type torch.FloatTensor but found type torch.LongTensor for argument #2 'other'
```

The command works fine with `-self_attn_type` set to `scaled-dot` or omitted (as it's the default).
I do not have a GPU on my machine, if that helps.
Am I using it wrong?
Thanks in advance.


Discussion:
My guess is what happened is that the default return type of `torch.arange` changed from `torch.FloatTensor` to `torch.LongTensor` between 0.4.0 and 0.4.1. To fix it, you can change line 47 of average_attn.py to `weights = torch.ones((1, inputs_len)) / torch.arange(1, inputs_len + 1).float()`. I'll see if there are any similar bugs elsewhere in the code too.
My guess is what happened is that the default return type of `torch.arange` changed from `torch.FloatTensor` to `torch.LongTensor` between 0.4.0 and 0.4.1. To fix it, you can change line 47 of average_attn.py to `weights = torch.ones((1, inputs_len)) / torch.arange(1, inputs_len + 1).float()`. I'll see if there are any similar bugs elsewhere in the code too.

Resolving patch:
diff --git a/onmt/modules/average_attn.py b/onmt/modules/average_attn.py
--- a/onmt/modules/average_attn.py
+++ b/onmt/modules/average_attn.py
@@ -43,8 +43,9 @@ def cumulative_average_mask(self, batch_size, inputs_len):
             * A Tensor of shape `[batch_size x input_len x input_len]`
         """
 
-        triangle = torch.tril(torch.ones((inputs_len, inputs_len)))
-        weights = torch.ones((1, inputs_len)) / torch.arange(1, inputs_len + 1)
+        triangle = torch.tril(torch.ones(inputs_len, inputs_len))
+        weights = torch.ones(1, inputs_len) / torch.arange(
+            1, inputs_len + 1, dtype=torch.float)
         mask = triangle * weights.transpose(0, 1)
 
         return mask.unsqueeze(0).expand(batch_size, inputs_len, inputs_len)
diff --git a/onmt/modules/embeddings.py b/onmt/modules/embeddings.py
--- a/onmt/modules/embeddings.py
+++ b/onmt/modules/embeddings.py
@@ -23,8 +23,8 @@ class PositionalEncoding(nn.Module):
     def __init__(self, dropout, dim, max_len=5000):
         pe = torch.zeros(max_len, dim)
         position = torch.arange(0, max_len).unsqueeze(1)
-        div_term = torch.exp((torch.arange(0, dim, 2) *
-                             -(math.log(10000.0) / dim)).float())
+        div_term = torch.exp((torch.arange(0, dim, 2, dtype=torch.float) *
+                             -(math.log(10000.0) / dim)))
         pe[:, 0::2] = torch.sin(position.float() * div_term)
         pe[:, 1::2] = torch.cos(position.float() * div_term)
         pe = pe.unsqueeze(1)


Test patch: