Attribute errors with pytorch_transformers tests
Author: tobimichigan
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (from the official repo):

Language I am using the model on (English, Yoruba,Igbo, Hausa etc):

The problem arise when using:
* [ ] the official example scripts: (give details): the test run script


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)=>yes


## To Reproduce

Steps to reproduce the behavior:

1.python -m pytest -sv ./pytorch_transformers/tests/


<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

`pytorch_transformers/tests/modeling_transfo_xl_test.py::TransfoXLModelTest::test_transfo_xl_lm_head FAILED
pytorch_transformers/tests/modeling_transfo_xl_test.py::TransfoXLModelTest::test_transfo_xl_model FAILED
pytorch_transformers/tests/tokenization_xlnet_test.py::XLNetTokenizationTest::test_tokenizer_no_lower PASSED

=================================== FAILURES ===================================
__________________ TransfoXLModelTest.test_attention_outputs ___________________

self = <pytorch_transformers.tests.modeling_transfo_xl_test.TransfoXLModelTest testMethod=test_attention_outputs>

    def test_attention_outputs(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        for model_class in self.all_model_classes:
            config.output_attentions = True
            config.output_hidden_states = False
            model = model_class(config)
            model.eval()
>           outputs = model(**inputs_dict)

pytorch_transformers/tests/modeling_common_test.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
    outputs = self._forward(input_ids, mems=mems, head_mask=head_mask)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = TransfoXLModel(
  (word_emb): AdaptiveEmbedding(
    (emb_layers): ModuleList(
      (0): Embedding(10, 32)
      (1):... LayerNorm(torch.Size([32]), eps=1e-05, elementwise_affine=True)
      )
    )
  )
  (pos_emb): PositionalEmbedding()
)
dec_inp = tensor([[19, 69, 72, 42, 32, 34, 52, 38, 81, 71, 81, 47, 44],
        [22, 12,  3, 26, 63, 25, 64, 52, 79, 71, 17, 16,...   [82, 26, 62, 95, 55, 79,  8, 90, 33, 83, 64, 53, 68],
        [ 7, 57, 63, 40, 74, 77, 50, 77, 19,  7, 53, 38, 19]])
mems = [tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0.,...   [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]]])]
head_mask = [None, None, None, None, None]

    def _forward(self, dec_inp, mems=None, head_mask=None):
        qlen, bsz = dec_inp.size()
    
        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads] (a head_mask for each layer)
        # and head_mask is converted to shape [num_hidden_layers x qlen x klen x bsz x n_head]
        if head_mask is not None:
            if head_mask.dim() == 1:
                head_mask = head_mask.unsqueeze(0).unsqueeze(0).unsqueeze(0).unsqueeze(0)
                head_mask = head_mask.expand(self.n_layer, -1, -1, -1, -1)
            elif head_mask.dim() == 2:
                head_mask = head_mask.unsqueeze(1).unsqueeze(1).unsqueeze(1)
            head_mask = head_mask.to(dtype=next(self.parameters()).dtype) # switch to fload if need + fp16 compatibility
        else:
            head_mask = [None] * self.n_layer
    
        word_emb = self.word_emb(dec_inp)
    
        mlen = mems[0].size(0) if mems is not None else 0
        klen = mlen + qlen
        if self.same_length:
            all_ones = word_emb.new_ones(qlen, klen)
            mask_len = klen - self.mem_len
            if mask_len > 0:
                mask_shift_len = qlen - mask_len
            else:
                mask_shift_len = qlen
            dec_attn_mask = (torch.triu(all_ones, 1+mlen)
>                   + torch.tril(all_ones, -mask_shift_len)).bool()[:, :, None] # -1
E           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError

>           outputs = model(**inputs)

pytorch_transformers/tests/modeling_common_test.py:185: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
    outputs = self._forward(input_ids, mems=mems, head_mask=head_mask)
E           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError
_________________ TransfoXLModelTest.test_hidden_states_output _________________

self = <pytorch_transformers.tests.modeling_transfo_xl_test.TransfoXLModelTest testMethod=test_hidden_states_output>

    def test_hidden_states_output(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        for model_class in self.all_model_classes:
            config.output_hidden_states = True
            config.output_attentions = False
            model = model_class(config)
            model.eval()
>           outputs = model(**inputs_dict)

pytorch_transformers/tests/modeling_common_test.py:249: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
    outputs = self._forward(input_ids, mems=mems, head_mask=head_mask)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError
__________________ TransfoXLModelTest.test_transfo_xl_lm_head __________________

self = <pytorch_transformers.tests.modeling_transfo_xl_test.TransfoXLModelTest testMethod=test_transfo_xl_lm_head>

    def test_transfo_xl_lm_head(self):
        self.model_tester.set_seed()
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
>       output_result = self.model_tester.create_transfo_xl_lm_head(*config_and_inputs)

pytorch_transformers/tests/modeling_transfo_xl_test.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pytorch_transformers/tests/modeling_transfo_xl_test.py:142: in create_transfo_xl_lm_head
    lm_logits_1, mems_1 = model(input_ids_1)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1349: in forward
    transformer_outputs = self.transformer(input_ids, mems=mems, head_mask=head_mask)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
E           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError
=============================== warnings summary ===============================
-- Docs: http://doc.pytest.org/en/latest/warnings.html
======= 5 failed, 206 passed, 10 skipped, 36 warnings in 171.71 seconds ========
`

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Seamless execution!!!
## Environment
==============NVSMI LOG==============

Timestamp                           : Sat Aug 31 11:09:33 2019
Driver Version                      : 418.67
CUDA Version                        : 10.1

Attached GPUs                       : 1
GPU 00000000:00:04.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla

* OS:Ubuntu 18.04
* Python version:3.6
* PyTorch version:1.1.0
* PyTorch Transformers version (or branch):https://github.com/huggingface/pytorch-transformers
* Using GPU ?yes
* Distributed of parallel setup ?no
* Any other relevant information:

Why do these attribute errors occur?

Comments:
Hi, could you try updating your pytorch version to 1.2.0 ?

Same issue here. Pytorch==1.2.0, python==3.6.2

What exactly is your issue @ukliu ?

> What exactly is your issue @ukliu ?

I was going through the pytorch-transformers tutorial at https://github.com/ukliu/pytorch-transformers 

<img width="870" alt="Screen Shot 2019-09-30 at 3 48 10 PM" src="https://user-images.githubusercontent.com/14615401/65910949-bfdddb00-e399-11e9-9970-f73d8e6f388b.png">

All others seems fine, but TransfoXLModel gives an error of  AttributeError: 'Tensor' object has no attribute 'bool'

That's mainly a pytorch version issue. You can upgrade your pytorch or change the type to torch.uint8 rather than call the .bool() function.