noamaanMulla-03/transformers-issues huggingface dataset:
How to calculate memory requirements of different GPT models? 
Author: Henry-E
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

What's the best way to calculate how much GPU ram is needed to finetune a given GPT size? The GPT-xl model is 6.5GB worth of parameters but clearly this isn't the full story when it comes to fine-tuning. I've tried running a batch size of 1 on a 11GB 1080ti but that ran out of memory. My next step is to test it on a 2080ti with FP16 training enabled to further reduce memory requirements. It would be great if there was some back of the envelope calculation to see in advance if this would work or if instead I should go straight to 16GB Titan V. Or maybe even that's too small!


Comments:
Current commercial GPU VRAM sizes are not large enough to process the XL model. You'll need probably around 32-64 GB of RAM in order to finetune successfully. So for now, you'll have to train on a CPU with a lot of memory attached.

How did you come to that number though? 

In Thomas Wolf's medium post on training large models he mentioned gradient checkpointing for models that couldn't fit a batch onto a single gpu. Is it possible to implement something like for GPT-2 and friends?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bias should be resized with the weights
Author: LysandreJik
Pull request: yes

Created a link between the linear layer bias and the model attribute bias. This does not change anything for the user nor for the conversion scripts, but allows the `resize_token_embeddings` method to resize the bias as well as the weights of the decoder.

Added a test.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=h1) Report
> Merging [#2521](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6c32d8bb95aa81de6a047cca5ae732b93b9db020?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2521/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2521      +/-   ##
==========================================
+ Coverage   73.24%   73.25%   +<.01%     
==========================================
  Files          87       87              
  Lines       15008    15011       +3     
==========================================
+ Hits        10993    10996       +3     
  Misses       4015     4015
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2521/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `64.42% <100%> (+0.17%)` | :arrow_up: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2521/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `87.72% <100%> (+0.02%)` | :arrow_up: |
| [src/transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/2521/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hbGJlcnQucHk=) | `78.91% <100%> (+0.05%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=footer). Last update [6c32d8b...b7832ab](https://codecov.io/gh/huggingface/transformers/pull/2521?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix misleading RoBERTa token type ids
Author: LysandreJik
Pull request: yes

RoBERTa does not actually make use of token type ids. When feeding the output of `encode_plus` used with a pair of sequences to the model directly, it crashes as it cannot handle token type ids that have a value of 1.

This fix returns a list of zeros as the token type ids instead.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=h1) Report
> Merging [#2432](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9261c7f771fccfa2a2cb78ae544adef2f6eb402b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2432/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2432   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       15001    15001           
=======================================
  Hits        10988    10988           
  Misses       4013     4013
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2432/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=footer). Last update [9261c7f...7e3feb9](https://codecov.io/gh/huggingface/transformers/pull/2432?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> 
> 
> RoBERTa does not actually make use of token type ids. When feeding the output of `encode_plus` used with a pair of sequences to the model directly, it crashes as it cannot handle token type ids that have a value of 1.
> 
> This fix returns a list of zeros as the token type ids instead.

I encountered the same problem. Thank you for your solution, I figure out what's wrong with my problem now.
Integration with a retriever Model
Author: rajo19
Pull request: no
Labels: name: wontfix

How can I integrate BERT to a retriever model?

Comments:
Have a look at the ParlAI library and in particular [these great models based on BERT](https://github.com/facebookresearch/ParlAI/pull/1331) by @samhumeau.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey @rajo19,
if you want to do Question Answering at scale with a Retriever + Reader pipeline, it might be worth checking out our new [haystack](https://github.com/deepset-ai/haystack/) project. It builds upon transformers and you can use all the QA models from [here](https://huggingface.co/models) as a reader. 

Very interesting, thanks for sharing @tholor! cc @mfuntowicz
How to use transformers-cli serve , how to set up on the server side?
Author: zhoudoufu
Pull request: no
Labels: name: Core: Pipeline

## ❓ Questions & Help

Hi, I would like to make the transformers based models running as a server on a remote machine, as the way bert-as-server did.
I suppose I could call the transformers-cli serve command on the server side, but I haven't find much clue on how to make it running on the client part. 

BTW. I am trying to run the serve cmd with localhost like :
transformers-cli serve --task feature-extraction --model distilbert --config distilbert-base-uncased --tokenizer distilbert 
and failed for ValueError: Can't find a vocabulary file at path  [cached dir file].
and I tried with transformers/src/transformers/__main__.py with same parameters and got same error.
Would you please give me a snippet on how to make the transformers-cli serve work on both sides?



Comments:
Hi @zhoudoufu ! You need to fully specify the model:

```bash
transformers-cli serve --task feature-extraction --model distilbert-base-uncased --config distilbert-base-uncased --tokenizer distilbert-base-uncased
```

Then you should be able to call:

```bash
curl -X POST "http://localhost:8888/forward" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"inputs\":\"My name is Morgan\"}"
```

Let us know :) 

It works, thanks @mfuntowicz
Saving full tensor output of hidden states instead of truncated output in lm_finetuning.py script
Author: Stuffooh
Pull request: no

Hi,

The past few weeks I have been playing around with the "run_lm_finetuning.py" script to finetune a custom dataset and extract it's features by setting 'output_hidden_states=True' and saving the features of the second last layer by changing the code of the script as follows:

`model.train()`
`outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)`
`loss = outputs[0]  # model outputs are always tuple in transformers (see doc)`
`torch.save(outputs[-1], 'output.pt')`

The tensor data gets truncated as follows and I have not been able to figure out yet how to save the full tensor output:

> [[ 0.0656, -0.1678, -0.4601,  ...,  0.0111,  0.0955,  0.7106],
         [ 0.7000, -0.5496,  0.6127,  ...,  0.0038,  0.3024, -0.2240],
         [ 0.1105,  0.3366,  0.1706,  ..., -0.1861, -0.0499,  0.0265],
         ...,
         [-0.3434, -0.1283, -0.0637,  ..., -0.2911, -0.7759,  0.0511],
         [ 0.3330,  0.3573, -0.2226,  ...,  0.4622, -0.6238, -0.5374],
         [ 1.1726,  0.0471, -0.0415,  ...,  1.3879, -0.3199,  0.2052]]]

I have been trying to figure it out by myself because I know the problem is my lack of experience with Python and Pytorch but really been hitting a wall trying to figure this one out.

Can anyone point me to the right direction how to save the full tensor output?

Comments:
Hi, how are they truncated? When you call `tensor.shape`, is it not the shape you're expecting?

@LysandreJik thanks for mentioning tensor.shape. I was so convinced the data was truncated and forgot to check the shape to confirm. Because of your hint I realized the output that gets printed is truncated but the actual data itself is not and is fully there.

Thanks ;)
Save only Bert Model after training a Sequence Classification Task/ LM finetuning Task. 
Author: kaminocode
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
1) How do I save only the BERT Model after finetuning on a Sequence Classification Task/ LM finetuning Task
2) How to load only BERT Model from a saved model trained on Sequence Classification Task/ LM finetuning Task

Comments:
By saving only BERT, do you mean saving only the transformer and not the classification layer as well?

@rahulbaburaj  you can use the code snippet below, change the 'bert-base-uncased' to your fine-tuned model directory.

```python
# load config
conf = BertConfig.from_pretrained('bert-base-uncased', num_labels=2)
# load a sequence model
bsm = BertForTokenClassification.from_pretrained('bert-base-uncased', config=conf)
# get bert core model
bcm = bsm.bert
# save the core model
bcm.save_pretrained('the output directory path')
# you also need to save your tokenizer in the same directory
```

@FacingBugs Thank you.
AttributeError: 'BertForTokenClassification' object has no attribute 'named_configeters'
Author: Dhanachandra
Pull request: no

```
    model = BertForTokenClassification.from_pretrained(bert_model)
```
Error: AttributeError: 'BertForTokenClassification' object has no attribute 'named_configeters'
When I initialized the model as 
```
model = BertForTokenClassification.from_pretrained(bert_model, 2)
```
Error: TypeError: from_pretrained() takes 2 positional arguments but 3 were given
```


Comments:
Hi, what is the variable `bert_model` you're showing? Could you provide more information e.g. the version of transformers, your version of python, your version of torch?
modified method simple_accuracy(), before:(preds == labels).mean() This…
Author: 7will10
Pull request: yes

modified method simple_accuracy(),
before:
        it's (preds == labels).mean() 
        This will cause an exception[AttributeError: 'bool' object has no attribute 'mean'],
then after update:
        change to accuracy_score(labels,preds), 
        use this method accuracy_score() in package sklearn.metrics

Comments:
How to get the output in other layers from Bert?
Author: zy4bvb
Pull request: no

## ❓ Questions & Help
I want to analyze the information that every bert layers contains.  But i found the BertModel only output the sentence embbeding and the CLS embbeding.
<!-- A clear and concise description of the question. -->


Comments:
Have a look at the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertmodel), particularly at the point about 'outputs'.  You'll see that when you use `output_hidden_states=True`, you'll get _all_ outputs back, like so:

```python
model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
```

> Have a look at the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertmodel), particularly at the point about 'outputs'. You'll see that when you use `output_hidden_states=True`, you'll get _all_ outputs back, like so:
> 
> ```python
> model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
> ```

Thank you!
Adding model type to config.json
Author: perdix
Pull request: no

## Feature
Add `model_type` to the *config.json* to define the model_type and make it independent from the name

## Motivation
Currently, the model type is automatically discovered from the name. So if it is a Bert model, the autoloader is choosing the right methods if the name contains `bert`. If not, an error would be thrown.
This is somehow cumbersome and error prone and it restricts the naming of the models. Why not just add this info as a attribute in the config.json? 
Other suggestions are welcome!

## Info
I would happily start working at a PR if others agree as well.


Comments:
Is now solved by this [PR](https://github.com/huggingface/transformers/pull/2494)
Thanks a lot!

Yes, thanks for the contribution @perdix!
[Question] Help needed to understand how torch.distributed.barrier() works
Author: hlums
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I have been trying to understand how torch.distributed.barrier()  is being used in the examples in this repo. I posted [this stackoverflow question](https://stackoverflow.com/questions/59760328/how-does-torch-distributed-barrier-work). Maybe someone from the huggingface team can help answering it? 

Comments:
I've provided an answer on Stack Overflow. Please close the issue here on Github. Thanks.

Thanks @BramVanroy for the detailed answer!
For HuggingFace GPT2DoubleHeadsModel, is there a way to directly provide a hidden state for an input? 
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,
Say I have two custom trained HuggingFace GPT2DoubleHeadsModels (Model 1 and 2).
 I want to take an hidden state of m-th layer of model 1, and use that hidden state as my input for model 2.
Is this possible with HuggingFace GPT2DoubleHeadsModels?

Some coding example would be a great help!

Thank you,


Comments:
Hi! You have access to the models' internals so you could definitely rewrite the main loop function to handle such a use-case!

You can access the hidden layers as follows:

```py
blocks = model.transformer.h  # this is a list of "block"s
block[0]  # contains the MLP/LayerNorm/attention layers
```

I'd recommend taking a look at the `forward` method of the [`GPT2DoubleLMHeadModel`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_gpt2.py#L642) to 
understand how the `transformer` is managed, and to take a look at the [loop inside the `GPT2Model`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_gpt2.py#L451) which you will probably have to replicate a little to achieve what you want to do.

Hello, 
Thank you very much for your reply.
So if I am understanding this correctly, the part of the loop inside GPT2 model below is what I am supposed to use to generate output based on a given hidden state, am I correct?
```
outputs = block(hidden_states,
                            layer_past=layer_past,
                            attention_mask=attention_mask,
                            head_mask=head_mask[i])
```
Also, I would like to clarify what ‘block’ actually refers to in this code. Is ‘block’ same as GPT-2 ‘layer’? It’s not very clear to me what `block[1]`, `block[2]`, etc. represents.

Thank you again,

Yes.

Hello,

What exactly is the `block`? Does `block[0]` refer to the first gtp-2 layer, and `block[1]` refers to the second gtp-2 layer and so on? How can I utilize `block` to have access to the uppermost output layer of a gtp-2?

Thank you,

Yes, `block` refers to the GPT-2 layers. `block[0]` is the first layer, ... `block[-1]` is the last layer.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix BasicTokenizer to respect `never_split` parameters
Author: DeNeutoy
Pull request: yes

`never_split` was not being passed to `_split_on_punc`, causing special tokens to be split apart. Failing test (in first commit) demonstrates the problem.

Comments:
Not sure how to fix that last CI build, seems unrelated?

Unrelated Heisenbug, relaunched the CI

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=h1) Report
> Merging [#2557](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/23a2cea8cb95864ddb7e7e80e126e4f083640882?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2557/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2557      +/-   ##
==========================================
+ Coverage    74.6%   74.61%   +<.01%     
==========================================
  Files          87       87              
  Lines       14802    14802              
==========================================
+ Hits        11043    11044       +1     
+ Misses       3759     3758       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2557/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `96.62% <100%> (+0.42%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=footer). Last update [23a2cea...c0afe26](https://codecov.io/gh/huggingface/transformers/pull/2557?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thank you sir!
question about tokenizer changes original sequence length
Author: zhaoxy92
Pull request: no

Hi, I am working on an entity classification task where I know the entity boundary and the context.
when I use tokenizer to encode the entire sequence, some token got split up to word pieces that will change the original length of the sequence.  I want to extract the states only for the entity, but since the seq length changed, does that mean I need to recalcuate the boundary? Is there a way to automatically do this? Thanks!

Comments:
Is RoBERTa's pair of sequences tokenizer correct with double 

Author: keldLundgaard
Pull request: no

In RoBERTa's build_input_with_special_tokens, the comment says 
 
```    
A RoBERTa sequence has the following format:
single sequence: 
 X 

pair of sequences: 
 A 

 B 

```

I find the double `

` very peculiar. Can you please verify that it should not be `

` (as a normal XML tag).

Thank you for the amazing work here.

Comments:
Yes, this is how RoBERTa was trained.
is RoBERTa-base.json in s3  wrong?
Author: skyf0cker
Pull request: no

Q:
when i open the json file downloaded from this url:
`https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-vocab.json`,
i found there is so many wrong strange code in it,like this:
> {"
": 0, "<pad>": 1, "
": 2, "<unk>": 3, ".": 4, "Ġthe": 5, ",": 6, "Ġto": 7, "Ġand": 8, "Ġof": 9, "Ġa": 10, "Ġin": 11, "-": 12, "Ġfor": 13, "Ġthat": 14, "Ġon": 15, "Ġis": 16, "âĢ": 17, "'s": 18, "Ġwith": 19, "ĠThe": 20, "Ġwas": 21, "Ġ\"": 22, "Ġat": 23, "Ġit": 24, "Ġas": 25, "Ġsaid": 26, "Ļ": 27, "Ġbe": 28, "s": 29, "Ġby": 30, "Ġfrom": 31, "Ġare": 32, "Ġhave": 33, "Ġhas": 34, ":": 35, "Ġ(": 36, "Ġhe": 37, "ĠI": 38, "Ġhis": 39, "Ġwill": 40, "Ġan": 41, "Ġthis": 42, ")": 43, "ĠâĢ": 44, "Ġnot": 45, "Ŀ": 46, "Ġyou": 47, "ľ": 48, "Ġtheir": 49, "Ġor": 50, "Ġthey": 51, "Ġwe": 52, "Ġbut": 53, "Ġwho": 54, "Ġmore": 55, "Ġhad": 56, "Ġbeen": 57, "Ġwere": 58, "Ġabout": 59, ",\"": 60, "Ġwhich": 61, "Ġup": 62, "Ġits": 63, "Ġcan": 64, "Ġone": 65, "Ġout": 66, "Ġalso": 67, "Ġ$": 68, "Ġher": 69, "Ġall": 70, "Ġafter": 71, ".\"": 72, "/": 73, "Ġwould": 74, "'t": 75, "Ġyear": 76, "Ġwhen": 77, "Ġfirst": 78, "Ġshe": 79, "Ġtwo": 80, "Ġover": 81, "Ġpeople": 82, "ĠA": 83, "Ġour": 84, "ĠIt": 85, "Ġtime": 86, "Ġthan": 87, "Ġinto": 88, "Ġthere": 89, "t": 90, "ĠHe": 91, "Ġnew": 92, "ĠâĢĶ": 93, "Ġlast": 94, "Ġjust": 95, "ĠIn": 96, "Ġother": 97, "Ġso": 98, "Ġwhat": 99, "I": 100, "Ġlike": 101, "a": 102, "Ġsome": 103, "S": 104, "Ã«": 105, "Ġthem": 106, "Ġyears": 107, "'": 108, "Ġdo": 109, "Ġyour": 110, "Ġ-": 111, "Ġ1": 112, "\"": 113, "Ġif": 114, "Ġcould": 115, "?": 116, "Ġno": 117, "i": 118, "m": 119, "Ġget": 120, "ĠU": 121, "Ġnow": 122, "Ġhim": 123, "Ġback": 124, "ĠBut": 125, "ĠâĢĵ": 126, "Ġmy": 127, "Ġ'": 128, "Ġonly": 129, "Ġthree": 130, ";": 131, "Ġ2": 132, "The": 133, "1": 134, "Ġpercent": 135, "Ġagainst": 136, "Ġbefore": 137, ...

**this really make me confused .
hope for your help,and i will appreciate it!**


Comments:
Yes this file is correct.