noamaanMulla-03/transformers-issues huggingface dataset:
[DOCS] fix hardcoded path in examples readme
Author: saippuakauppias
Pull request: yes

I think, after merge this PR - need to rebuild docs, right?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=h1) Report
> Merging [#2964](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/94ff2d6ee8280c5595b92c1128c0f18e44925e56?src=pr&el=desc) will **decrease** coverage by `1.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2964/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2964      +/-   ##
==========================================
- Coverage   77.11%   76.08%   -1.04%     
==========================================
  Files          98       98              
  Lines       15977    15977              
==========================================
- Hits        12321    12156     -165     
- Misses       3656     3821     +165
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2964/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2964/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.71% <0%> (-10%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2964/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.48% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2964/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `96.03% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2964/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=footer). Last update [94ff2d6...4e2066b](https://codecov.io/gh/huggingface/transformers/pull/2964?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks! cc @LysandreJik
Does Pre-Trained Weights Work Internally in pytorch? 
Author: shashankMadan-designEsthetics
Pull request: no
Labels: name: wontfix

I am using bert’s pretrained model in from_pretrained and coming across it’s fine tuning code we can save the new model weights and other hyper params in save_pretrained.
My doubt is in there modeling_bert code there is no explicit code that takes the pre-trained weights in acount and then trains as it generally takes attention matrices and puts it in a feed forward network in the class `BertPredictionHeadTransform`
```
class BertPredictionHeadTransform(nn.Module):
    def __init__(self, config):
        super(BertPredictionHeadTransform, self).__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        if isinstance(config.hidden_act, str) or (sys.version_info[0] == 2 and isinstance(config.hidden_act, unicode)):
            self.transform_act_fn = ACT2FN[config.hidden_act]
        else:
            self.transform_act_fn = config.hidden_act
        self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)

    def forward(self, hidden_states):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.transform_act_fn(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        print('BertPredictionHeadTransform', hidden_states.shape)
        return hidden_states

```
And here i do not see any kind of “inheritance” of the pre-trained weights…
So is it internally handled by pytorch or am i missing something in the code itself?



Comments:
`BertPredictionHeadTransform` is never used in itself AFAIK, and is only part of the full scale models (e.g. `BertModel`). As such, the weights for the prediction head are loaded when you run BertModel.from_pretrained(), since the `BertPredictionHeadTransform` is only a module in the whole model.

> `BertPredictionHeadTransform` is never used in itself AFAIK, and is only part of the full scale models (e.g. `BertModel`). As such, the weights for the prediction head are loaded when you run BertModel.from_pretrained(), since the `BertPredictionHeadTransform` is only a module in the whole model.

Thanks, Bram. As you said `BertModel` does take `BertPreTrainedModel` in `super`. I did notice it, But It's just that my mind doesn't get around how/where and when those weights are getting used exactly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bart: fix layerdrop and caching shapes for generation
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=h1) Report
> Merging [#2969](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c36416e53c29da8b6193f4a36d7b024c5f513495?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2969/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2969      +/-   ##
==========================================
- Coverage   77.14%   77.14%   -0.01%     
==========================================
  Files          98       98              
  Lines       16006    16003       -3     
==========================================
- Hits        12348    12345       -3     
  Misses       3658     3658
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/2969/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `84.58% <100%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=footer). Last update [c36416e...0198f22](https://codecov.io/gh/huggingface/transformers/pull/2969?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
fix xlm roberta tokenizer mask id
Author: andompesta
Pull request: yes

As per issue #2508 the xlm_roberta_tokenizer has an error in the mask_id computation.
The sp_model already contains all the special tokens (bos, pad, eos, unk) but the mask id, which, according to the model specification should be 250001 instead of 250004:
```
self.fairseq_tokens_to_ids["<mask>"] = len(self.sp_model) + self.fairseq_offset
```
instead of 
```
self.fairseq_tokens_to_ids["<mask>"] = len(self.sp_model) + len(self.fairseq_tokens_to_ids)
```

Comments:
Hey @stefan-it, mind taking a look at this?

Hi,
I get the following error when trying to adapt the same code as BERT for Masked LM with XLM-Roberta For Masked LM, where I've replaced **'[MASK]'** with **'\<mask>'** and '**[CLS]'** and **'[SEP]'** with **\
** and **\
** respectively.

```
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485 
   1486 

RuntimeError: index out of range: Tried to access index 250004 out of table with 250001 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

I'm sorta new to this. Can I know whether this is related to the above discussed issue? and is much obliged to know how this error can be fixed.
How to add the output word vector of bert to my model
Author: e-tuanzi
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Hello, I am a student who is learning nlp. 
Now I want to use the word vector output by bert to apply to my model, but **I can't connect the word vector to the network**. Could you give me an example program or tutorial about this which use textCNN or LSTM. You can sent e-mail to **vvain0208@163.com** or reply me, please.
Thank you for your kind cooperation!
<!-- A clear and concise description of the question. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I think you can see [here](https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/). In more details, this tutorial uses **BERT** as **feature extractor** and on-top they have used a **Logistic Regression** model from [Scikit-learn](https://scikit-learn.org/stable/) for the **sentiment analysis** task. 

Question: which is the problem in details? Are you not able to connect the feature vector extracted by BERT to a custom classifier on-top? Is the shape of the feature vector fixed?

> ##  Questions & Help
> Hello, I am a student who is learning nlp.
> Now I want to use the word vector output by bert to apply to my model, but **I can't connect the word vector to the network**. Could you give me an example program or tutorial about this which use textCNN or LSTM. You can sent e-mail to **[vvain0208@163.com](mailto:vvain0208@163.com)** or reply me, please.
> Thank you for your kind cooperation!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Question about output pipeline(feature-extraction)
Author: Stuffooh
Pull request: no

Hi,

I'm new to python and transformers so please bare with me. I have the following code to use the pipeline wrapper of transformers.

```
from transformers import (
    pipeline
)
import h5py

nlp = pipeline('feature-extraction', model='bert-base-cased', config='bert-base-cased', tokenizer='bert-base-cased', device=-1)

test = nlp("PersonA: Hi . PersonB: How are you doing ? PersonA: I 'm doing alright thank you very much.")

h5f = h5py.File('test.h5', 'a')
h5f.create_dataset('name', data=test)
h5f.close()
```

The above proof of concept works fine for me however I have two questions.

1. If I look at the shape of the h5py dataset it is (1, 270, 768). For my purpose I require the shape (270, 768). How can I make sure the output gets saved to h5 in this format?

2. device=-1 means the code will be executed on CPU correct?

Could someone please help me out with these two? ;)

Comments:
Length of special_tokens_mask doesn't align with the input_ids
Author: hinthornw
Pull request: no
Labels: name: Core: Tokenization

name: Should Fix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
data = tokenizer.encode_plus("Hello, world!", add_special_tokens=False, return_special_tokens_mask=True)
assert len(data['input_ids']) == len(data['special_tokens_mask'])
>> AssertionError
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Expect there to be no assertion error. The mask should be of the same shape as the input_ids.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.4.1
- Platform: Linux
- Python version: 3.5.2
- PyTorch version (GPU?): 1.2.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
unreadable codes in for utils_glue
Author: ghost
Pull request: no

Hi
Previously the function "convert_examples_to_features" was implemented very nicely, and now this is implemented with so many nested function calls, which are all very hard to read and IMO unreadable codes. Could you return this implementation back to previous readable version?
thanks

Comments:
Hi, thank you for your comment. Github allows you to look at older versions of the [code](https://github.com/huggingface/transformers/blob/v1.0.0/examples/utils_glue.py).
Delete untested, broken Model2LSTM
Author: sshleifer
Pull request: yes

If you need it back `git checkout c36416e5`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=h1) Report
> Merging [#2968](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c36416e53c29da8b6193f4a36d7b024c5f513495?src=pr&el=desc) will **decrease** coverage by `1.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2968/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2968      +/-   ##
==========================================
- Coverage   77.14%   76.13%   -1.02%     
==========================================
  Files          98       98              
  Lines       16006    15998       -8     
==========================================
- Hits        12348    12180     -168     
- Misses       3658     3818     +160
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/2968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `25.37% <ø> (-1.3%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.71% <0%> (-10%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.48% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `96.03% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2968/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=footer). Last update [c36416e...05560e2](https://codecov.io/gh/huggingface/transformers/pull/2968?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
PreTrainedEncoderDecoder does not work for LSTM 
Author: pruksmhc
Pull request: no
Labels: name: Core: Modeling

name: seq2seq

name: Should Fix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
If we want to have a BERT-based encoder and LSTM encoder, that is not currently possible with the current huggingface implementation, mostly because torch.nn.LSTM does not contain a config class variable. 

## Stack Trace 
  File "/beegfs/yp913/anaconda3/envs/jiant_new/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 349, in from_pretrained
    model = super().from_pretrained(*args, **kwargs)
  File "/beegfs/yp913/anaconda3/envs/jiant_new/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 153, in from_pretrained
    decoder.config.is_decoder = True
  File "/beegfs/yp913/anaconda3/envs/jiant_new/lib/python3.6/site-packages/torch/nn/modules/module.py", line 539, in __getattr__
    type(self).__name__, name))
AttributeError: 'LSTM' object has no attribute 'config'

## To reproduce

You can reproduce this by:
import transformers
from transformers.modeling_encoder_decoder import  Model2LSTM
model =  Model2LSTM.from_pretrained("roberta-large", decoder_config={"hidden_size":512, "input_size":1024, "num_layers": 2})

(When you initialize Model2LSTM like the above it runs into a separate error. I believe a ** is missing from the Model2LSTM decoder LSTM initialization).



Comments:
I put this as a bug because the code as-is does not hint that Model2LSTM does not work. 
https://github.com/huggingface/transformers/blob/90ab15cb7a8fcf8bf58c05453ddf1aa6a4fa00c1/src/transformers/modeling_encoder_decoder.py
It would be great to say that LSTM is not currently supported there. 

Indeed, my initial comment was a mistake. I'm looking into it now.
T5 - Finetuning of an EncoderDecoder Model
Author: f-lng
Pull request: no
Labels: name: wontfix

Hello,

I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model? 

As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?

Could you point me to a script / command / piece of code for finetuning T5?

Comments:
As I know, there is **no** Python scripts for fine-tuning T5 model, **at the moment**.
Besides the source code you can see in this library, you can see the PR #1739 which implements T5 model.

> Hello,
> 
> I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model?
> 
> As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?
> 
> Could you point me to a script / command / piece of code for finetuning T5?

The same question.  #1739 was merged. First of all, In T5_INPUTS_DOCSTRING is said:
```
            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
            (a) For sequence pairs:
                tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
            (b) For single sequences:
                tokens:         [CLS] the dog is hairy . [SEP]

```

At second, it looks like T5Model can work in encoder mode only. So, it's possible to treat it as usual LM:

```
        tokenizer = T5Tokenizer.from_pretrained('t5-small')
        model = T5Model.from_pretrained('t5-small')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids)
        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```

Maybe @thomwolf can clarify how better to fine-tune T5 for classification tasks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Return overflowing tokens if max_length is not given
Author: BramVanroy
Pull request: no
Labels: name: wontfix

## 🚀 Feature

The tokenizer can return overflowing tokens in the `encode_plus` method if 1. `return_overflowing_tokens=True`; and 2. if `max_length` is given.  I imagine that it could also be useful to allow to return the overflowing tokens when a max_length is not given but when the input is longer than the model's max_seq_len. As an example: if the input is 600 tokens long, and the model supports up to 512, then the tokenizer will cut down the input to 512 anyway, so the superfluous 88 tokens can then be returned. 

Example showing that the overflowing tokens are not returned, even though the input is trimmed. Would expect the trimmed tokens to be returned in the `overflowing_tokens` field.

```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = 'I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas.'

encoded_inputs = tokenizer.encode_plus(text,
                                       return_tensors='pt',
                                       pad_to_max_length=True,
                                       return_overflowing_tokens=True,
                                       return_special_tokens_mask=True)
print(encoded_inputs.keys())
print(encoded_inputs['input_ids'].size())
# dict_keys(['special_tokens_mask', 'input_ids', 'token_type_ids', 'attention_mask'])
# torch.Size([1, 512])
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.