RuntimeError: expected scalar type Half but found Float
Author: jroakes
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
bert-large-uncased

Language I am using the model on (English, Chinese....):
English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

I am using a modified version of the run_lm_finetuning.py and amp at optimization level o1.  Level O2 runs without issue.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

I am finetuning on a dataset of sentences, with a min length of 10 tokens, using padding and the mask_tokens function given by the repo. Input and labels are padded per specs and of type=LongTensor (torch.Size([4, 200]) torch.Size([4, 200])) with batch size of 4.

## To Reproduce

Steps to reproduce the behavior:

1. When I run without amp, training works as intended.  If I train at amp level O2, training runs as intended.
2. Running with amp level O1 leads to an error: RuntimeError: expected scalar type Half but found Float
3. I have verified that there is no model.eval() and that scaled_loss and clip_grad_norm_ calls are the same as prescribed in the example.
3. Examples of model initialization, train loop, and error are below.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Training to complete without error.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:  Google Colab
* Python version:  3.6
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? Y
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

**Model/Optimizer code:**
```
    model = model_class.from_pretrained(args.model_name_or_path, from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
    model.to(args.device)

    # Prepare optimizer and schedule (linear warmup and decay)
    #no_decay = ['bias', 'LayerNorm.weight']
    no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': args.weight_decay},
        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
        ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, weight_decay=args.weight_decay)
    
    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=args.warmup_steps, t_total=t_total)
    
    if args.fp16:
        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)
```

**Import Part of Train Loop:**
```
        for step, batch in enumerate(epoch_iterator):
            inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else mask_labels(batch, tokenizer, args)

            inputs = inputs.to(args.device)
            labels = labels.to(args.device)
            
            model.train()

            loss, _ = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps

            if args.fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()

            tr_loss += loss.item()
            
            if (step + 1) % args.gradient_accumulation_steps == 0:
                print('Clipping Grad Norm. fp16:', args.fp16)
                if args.fp16:
                  torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
                else:
                  torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
                scheduler.step()  # Update learning rate schedule
                optimizer.step()
                optimizer.zero_grad()
                global_step += 1
```



**Error:**
```
RuntimeError                              Traceback (most recent call last)

<ipython-input-7-627529edcd83> in <module>()
     10 ft_data.fp16_opt_level = 'O1'
     11 
---> 12 run_finetune(ft_data)
     13 
     14 print('Finetuned BERT model loaded.')

12 frames

<ipython-input-6-39df7e7a4180> in run_finetune(args)
    337         torch.distributed.barrier()
    338 
--> 339     global_step, tr_loss = train(args, train_dataset, model, tokenizer)
    340     logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
    341 

<ipython-input-6-39df7e7a4180> in train(args, train_dataset, model, tokenizer)
    193             model.train()
    194 
--> 195             loss, _ = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
    196 
    197             if args.n_gpu > 1:

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, masked_lm_labels)
    767 
    768         sequence_output = outputs[0]
--> 769         prediction_scores = self.cls(sequence_output)
    770 
    771         outputs = (prediction_scores,) + outputs[2:]  # Add hidden states and attention if they are here

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, sequence_output)
    417 
    418     def forward(self, sequence_output):
--> 419         prediction_scores = self.predictions(sequence_output)
    420         return prediction_scores
    421 

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, hidden_states)
    406 
    407     def forward(self, hidden_states):
--> 408         hidden_states = self.transform(hidden_states)
    409         hidden_states = self.decoder(hidden_states) + self.bias
    410         return hidden_states

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, hidden_states)
    388         hidden_states = self.dense(hidden_states)
    389         hidden_states = self.transform_act_fn(hidden_states)
--> 390         hidden_states = self.LayerNorm(hidden_states)
    391         return hidden_states
    392 

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py in forward(self, input)
    157         if self.elementwise_affine:
    158           return FusedLayerNormAffineFunction.apply(
--> 159               input, self.weight, self.bias, self.normalized_shape,self.eps)
    160         else:
    161           return FusedLayerNormFunction.apply(input, self.normalized_shape, self.eps)

/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py in forward(ctx, input, weight, bias, normalized_shape, eps)
     23     bias_ = bias.contiguous()
     24     output, mean, invvar = fused_layer_norm_cuda.forward_affine(
---> 25         input_, ctx.normalized_shape, weight_, bias_, ctx.eps)
     26     ctx.save_for_backward(input_, weight_, bias_, mean, invvar)
     27     return output

RuntimeError: expected scalar type Half but found Float (data<c10::Half> at /usr/local/lib/python3.6/dist-packages/torch/include/ATen/core/TensorMethods.h:1821)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f0840be9273 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::Half* at::Tensor::data<c10::Half>() const + 0x3ee (0x7f08298ccf8e in /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #2: cuda_layer_norm(at::Tensor*, at::Tensor*, at::Tensor*, at::Tensor*, int, int, c10::ArrayRef<long>, at::Tensor*, at::Tensor*, double) + 0x4c5 (0x7f08298ca745 in /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
```
There are a total of 63 frames which output in the error that have been truncated here.


Comments:
I've encountered this problem as well

Seems like an apex error (apex should be converting the tensors to half).
Maybe try to update or reinstall apex following carefully the required step for installation? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.